Hello everyone and welcome to my talk on transport-level testing of NVMe devices using VFIO. My name is Klaps. I'm a sub-netware engineer with Samsung electronics and on my database basis I work on open source. I'm a co-maintainer of the QEMO emulated NVMe device and I also work in various other areas of the open source NVMe ecosystem. We'll be talking about VFIO today and some of the core concepts that we're going to have to go through to understand what we're going to work with is we're going to have to go through some low-level stuff in NVMe and PCI Express. Then we're talking a little bit about the IOMMU and how it works and the VFIO kernel framework. We'll use this to try and write a driver in less than 30 minutes which will be able to do low-level commands submission to NVMe devices and we'll also see how we can interact with the controller memory buffer which is a PCI Express-based feature. So NVMe one slide. NVMe is non-volatile memory express and it's a storage interface that's designed to exploit the low latency and inherent parallelism of NAND flash memory. Now some core terminology here is the controller which is basically just a PCI Express function that acts as the interface to the host and then there's the command queues and the command queues are the core feature that makes NVMe tick. The host will write the submission queue entries that describe a command to a dedicated submission queue and the controller will write completion queue entries to another dedicated queue called completion queues. So while this might be easy we won't be able to do with one slide we're going to have to need way more than that and because by the end of this talk we'll actually be having a rudimentary NVMe driver. So we will focus on core concepts which are the queues, the submission and completion queue entries, PCI doorbells and core controller configuration and we will only be talking about a single NVMe command, the identify command which can tell the host about various stuff about the controller which means that we will skip all other commands like IO and you can read these up in the spec or look these up in the spec afterwards if you're interested. We also won't be talking about specific namespace types, the standard NVM namespace type and shown namespace type if you're interested in this I suggested you go see my talk from last year as an open source summit which have a talk specifically on the shown namespace implementation in QEMO. So let's first look at the actual submission queue entry. It's a 64 byte payload and it describes everything that the controller needs to actually execute a command. There's a lot of fields in here but we are only specifically interested in three of these for what we'll be doing in this talk. So the first one is the command identifier which is a host chosen unique ID for a command and then we have the opcode which together with the actual submission queue that we submit stuff to uniquely identifies the command that the controller should execute and then there's the data pointer which describes the payload that's associated with the command. Now the payload or the data pointer is a 16 bytes so but it actually consists of two 64 bit memory addresses what we call physical region pages that always point to something that are of the size of the host operating systems page size. So the queues that are central to NVMe are basically circular buffers that are defined by two pointers. There's the tail pointer which is incremented when we add something to the queue and then there's the head pointer which is incremented when we are removing entries from the queue or reading entries from the queue. And as you can see in the example figure here we have a queue where we have three entries in it which and the reader has not read any of them so the head pointer points to the first location and for the producer or the writer to know where to write the next entries we have the tail point to the point to the next empty slot in the queue. So the queues are in that way locked free because the reader will never read past the tail pointer and the writer will never write past the head pointer. So the host and the controller can act both as queue readers and writers depending on the actual queue where for instance in the case of a submission queue the host is the writer and the controller is the reader and for completion queues the at the other way around. So the queues are normally allocated in main system memory and with this means that from the host point of view writing a command or reading a completion entry is as simple as a memory copy. So this brings us to some questions. So first thing is how does the controller or the device actually know where to find these queues and how do we get the device to fetch and execute these commands when we have written something to it and how do we know if the command was executed and when it was actually executed. So to configure the device until different stuff the PCI device can expose a set of configuration registers we call this the PCI configuration space and it's mapped into into the host memory address space. This means that the host can go into this address space and configure the PCI device and read various stuff about the device in this configuration area and the device can also expose memory areas to be mapped into the host address space we call these the base address registers and they contain an address which the host configures and this tells the device that at this location we will be reading these configuration registers that the device expose. These are different from the PCI registers so this is custom registers that are unique to the device type say NVMe. So in the case of NVMe the device will expose an area of memory called the memory base address register and this memory location contains a basic controller configuration and is used for the basic controller configuration and communication between the host and the device. So it looks something like this when we look at it in tabular form. So there are a bunch of registers here there's the controller capabilities register which is a read-only register where the device can tell the host about various capabilities and features that it support and features that it does not support then there's the version register that tells it about what version of the specification are we implementing and then there's the control configuration registers where the host can set up the device in a certain way with certain parameters that are needed for the device to operate. We also have these two special registers called the admin submission queue base address and admin completion queue base address. These are special registers that tell the device where to fetch admin submissions, admin commands and where to post completion entries for these commands. So now we know how to basically bootstrap the device by configuring this controller configuration register. Now how do we get the device to actually fetch and execute the command? For this we use something called a PCI doorbell. A PCI doorbell is just a common name for write only memory map register. We can read from them but the behavior of that is undefined and so from the point of view of the host we will only be writing these registers. As you can see here they are also located in the M bar or the first bar on the device and it starts at the address of the memory bar plus 4k so the address of 1,000 looping hex. The tail doorbells are four byte wide and they come in two forms. They come in a tail doorbell and a head doorbell and we shall see how these are used. So when we write this we call it ringing doorbell and the way we do this is that when the host has written one or more entries to the submission queue we sort of can kick the device and tell it now you need to start executing stuff. We do this by writing the new value of the tail pointer to this register. So we can see here that it's basically just a write to a memory location at a certain offset with a certain value. So the value tells the device that I have inserted entries up to this point in the queue and you may now fetch them from your previously head position. In this case this tells the device that it can execute and fetch and execute entries 0, 1 and 2. So what happens when the queue is full? In this case that's when the host has produced enough submission queue entries that the tail pointer goes to 7 in this case and there's no more room in the queue. When that happens the controllers somehow have to notify the host that it has executed these commands and that the host may reuse the slots in the queue. Note also that is the responsibility of the host not to override the non-fetched entries in the queue. But to understand this we have to look at how NVMe piggybacks this information in completion queue entries. So specifically how do we know if a command was executed? To do that we use the completion queue entries and these are posted or written by the controller whenever it has finished executing a command. The completion queue entry is 16 bytes so it's smaller but it contains just enough information for the host to know about the status of the command. Was it a success? Did the controller encounter some kind of error? Maybe it didn't understand the command, the opcode that we requested or maybe we encountered a memory error or something like that. It will always add the complete command identifier which is used by the host to know what command this relates to. It also includes the identification of the completion of the submission queue that we originally submitted this identifier to and it also includes a new value of the head pointer. That is, this is the new value of the internally maintained head pointer to the controller which tells the host that I have now executed these commands so you may reuse the slots. To see there's an action when the controller picks it back something and gives a new head pointer location of four then the host will know that it can now reuse entries zero, one, two, and three for new commands. How do we know that when these commands are executed or have been executed? Since there's no doorbell register that can be ringed on the host to inform that the completion queue contains entries, we have two options. The host can choose to simply keep reading the memory location of the completion queue and wait for something called a phase change. This is a polling technique. We can also rely on interrupts that are maybe generated by controller or a combination of these two things. Let's first look at what polling means and what it means to look for a phase change. There's a special bit in the completion queue entry called the phase bit. The phase bit is inverted by the controller whenever it writes or overrides an existing entry in the completion queue. When we begin operation with a completely empty queue the phase bit is zero and the host may keep reading this memory location, the tail, and keep reading it until the phase bit changes. So when the host writes something it inverts this phase bit, goes to one. The host will immediately notice this and then it knows that this is a new entry and then it will advance the tail position and it will check the next entry. If the entry or the phase bit has changed again then we know that this is not a new entry and we stop the polling or we wait for another interrupt. So while polling allows very low latency it usually comes at the cost of dedicating an entire call to keep doing this constantly. So another way to do this is to rely on interrupts and it is the standard way of working with NVMe devices. So in an interrupt based system we rely on the controller to generate an interrupt which basically says that something's up because the interrupt doesn't carry any information other than something's up you need to take action. Now depending on the configuration and depending on the capabilities of the device the interrupt may indicate that there is new stuff in a specific completion queue or it may just indicate that there's new stuff in some completion queue. So we still use the phase bit because when we get this interrupt we don't know how many completion queue entries the queue holds. So we still keep reading the completion queue until we see the phase change in the phase bit once again. So finally we need some way for the controller or the host to actually indicate to the controller that it has read this completion queue entry and that the controller may reuse this slot in the completion queue because the controller is under the same restrictions that it cannot override the entries until they are acknowledged by the host. So we do this with another doorbell we call this the head doorbell and it's also registered in the bar and basically what happens is that we tell the device the new location of the head pointer, the maintained head pointer but notice that the entry is not zero or cleared in the memory it remains in place which means that the phase bit value remains in place. So at the next point when we go around through the circular queue and we come to this position then the controller will again invert the phase bit and we notice a new entry. So that's basically it for the low-level NVMe stuff that we're going to need. Now in part two we'll look at the VFIO or IOMU and how to actually use this and try to do these steps manually on a device. Okay so what we've been going through right now is basically what the kernel driver or the user space framework does for you. So you don't need to worry about these details to do amazing stuff with an NVMe device. You can use the block layer abstractions of your operating system coupled with a high performance synchronous programming model like raw IOU ring or a slightly higher framework like XNVMe or you can use user space frameworks such as SPDK for even more control. But neither XNVMe in its pass-through mode or SPDK allows manipulation at the actual transport level. Now all the abstractions of these frameworks are at the command level which means that if you look at the API the SPDK lowest level commands is about writing a command an admin command or an IO command and it's about simply waiting and processing the completions. This is similar in XNVMe where you can pass through a command to the device and you can wait and you can look at the completion queue. So this means that there's no manipulation of the queues directly and one of the things is that both XNVMe and SPDK relies on a concept where we always have a one-to-one correspondence between submission queues and completion queue. NVMe actually supports that multiple submission queues can result in completion queue entries going to one completion queue so you can have a many to one relationship between submission queues and completion queues. This is not supported by any of these frameworks. You also have no specific control of the interrupt vector configuration so you will always I believe that the SPDK will try to attach SPDK actually doesn't use interrupt vectors it uses polling on all the queues but you cannot set up a set of completion queues to use one interrupt vector or another set to use one vector each and stuff like that and you basically also have no low-level controller configuration abilities at all these frameworks. So this makes them a little less suitable for doing really low-level inspection of your devices and testing of your devices and the code paths in the device. So if we wanted to actually do some of the stuff that we learned in part one how would we go about doing this? So we could go to the standard NVMe kernel driver in Linux and we could modify it to do some custom stuff we could add a custom iOctil to do a specific test and stuff like that or we could simply disable all security features of the kernel and just go completely crazy as root and do this through the iOuPCI generic driver or we could actually utilize the virtual functions iO framework to do this kind of custom stuff but in a very safe way. So how low can we actually go? So the problem is that interacting with devices at the level of the registers had traditionally been a job that is for the kernel to do. It's a very privileged activity reserved for kernel and kernel headers but the virtual function iO framework actually changes this at least sort of. So Vfio is a driver framework in itself and the VfioPCI is a driver that the PCI devices will attach to a boot bound to instead of the normal driver like NVMe. Now the point here is that we retain the kernel retains authority of the device it's still in charge of the device it still protects the device from user space but the driver the VfioPCI driver provides full access to this device from user space which means that we can actually read and write from the PCI configuration space and we can read and write to the bars and it gives us very fine grained interrupt control to configure it as for pin based legacy pin based interrupts or MSI and MSIX configurations. Now it all relies on IOU based DMA translations which allows the driver to limit what the user can actually program the device to do to to make it safe and it also confines this DMA to the DMA of the device to the address space of the user space process. This is essential since traditionally the IOU devices have been working with physical addresses such that you could you could basically instruct the device to perform a DMA into another physical address which may belong to another process or something like that but the IOU provides a translation between the IOU devices and the CPU it basically does what we call it adds something called IOU virtual addresses which are sort of like virtual addresses a virtual address space but for the IOU devices that match to physical devices that can then go into the system memory. So it's basically just what the memory management unit does for virtual for regular virtual memory when the CPU is operating. So and as we can see here in the in the diagram you have a bunch of IOU devices behind this IOU and they will work with IOU virtual addresses. Now we can actually have multiple IOU devices behind the IOU and in this case the IOU devices can technically depending on how the motherboard or how it's actually wired up these devices may actually be able to interact with each other. So we can still have one device that could read or write to the other device through DMA. So the VFIO framework and the IOU framework in the kernel has the concept of an IOU new group which is basically the lowest granularity in which we can ensure safety. So for a perfectly isolated IOU device it needs to be in a singular group with no other IOU devices otherwise we need to accept that the security or the isolation granularity is at the level of the group marked here. So to to work with VFIO there's a lot of boilerplate involved. So VFIO worked with something a concept called a container and when the first thing that you have to do when you start working with VFIO is to create this container and then you can verify the VFIO capabilities like what's the API version supported by the kernel is there actually an IOU that we can use and then we have to determine the group of the device the IOU or VFIO IOU group and we need to verify that this group is viable and what viable means is that each of the devices in the group has to be either not bound to any driver or at least everyone or all of them bound to the VFIO PCI driver. So when the group is determined to be viable we can add it to the container and the cool thing here is that since we are since we we can ensure that different groups interaction between different groups will always go through the MMM IOU we can add more groups to the same container and work with all of them independently. When all this is done we can enable the IOU and we can retrieve a list of the ranges that the IOU virtual addresses can have and that we can use and then finally we can actually open the device we get a device handle which is a file descriptor and we can start configuring it that means we can set up the memory regions the PCI configuration space any bars that we're interested in and we can configure the interrupts of the device. So the API or the user space API from the kernel is very minimal but also extremely extensible for future features and capabilities. It looks something like this when you work with it all which means that it's a bunch of IOPTL calls for instance this we have an example here where we just query the group for the status of the group and we can check if it's viable and then there's an example here of how to set up interrupts as you can see you set up a data structure of a certain size you set up a bunch of fields you but what you do here is that you set up the exact way that the interrupt should be configured and then you issue another IOPTL to actually configure this for the device. Now as you can see all of this is agnostic to PCI it's an abstraction on the device which uses this common framework. So there is a pretty good documentation on the kernel website for the actual VFIO framework but there's also a lot of good code in the QEMO source code. The QEMO source code has extensive support for device pass through this means unbinding a device on the host and basically binding it or passing it through to the machine and what happens here is that basically the virtual machine or the hypervisor becomes a user space driver that manages the device and passes it through to the guest operating system. This is done by the VFIO subsystem in QEMO. QEMO also has an NVMe driver for using raw NVMe devices as block storage and this is VFIO based so there's a lot of good stuff in there. The VFIO helpers we call a library but the object in QEMO contains a bunch of various very nice helpers to actually work with this. So in general in QEMO there is a bunch of great code to learn from and to look through to actually understand better how this works very deep. But one thing that we don't have is that there's no lib VFIO. There's no user space, there's no library available that actually tries to unify all of these best practices that are especially in QEMO code and all these utility functions. There's no library that does this. So I've been working on a library called, so far just called VFIO in my local repository and it's a set of utility libraries written in C to work with the VFIO user library, user API. The library contains functions to do the basic VFIO device initialization of the spoiler plate code and it includes extremely simple naive and fixed IO VA allocator. It just helps you allocate IO virtual addresses but you cannot free them. You can just allocate a new one and then the allocator just makes sure that you don't reuse these virtual addresses. So in some sense, while this can give you unused IO virtual addresses you still have to manage them yourself but it's a plugin-based so you can add another allocator of your own writing if you want a more advanced allocator.  That's support for or utility functions for setting up the interrupt configuration and there's utility functions for doing DMA mapping of memory. So all of the boiler plate stuff that I talked about previously can basically be done in one single API call here. It's called the VFIO-PCI init and it will initialize a VFIO-PCI state from a PCI ID. And then there's another call to configure the IRQ basically setting up a Unix kernel event file descriptor and match it to a specific interrupt vector on the device. Now this is all intended to be open sourced, I'm still working on it but my intention is to open source it, no less than Q3 this year. So besides just basic VFIO functionality, it also includes a bunch of indianese specific functionality because that's what I'm working on.  So it also has very low level command submission. So the call API is at the level of posting a command to the submission queue. It's at the level of kicking the device, riding the doorbell, ringing the doorbell with the kick call. There's a peak call which allows you to peak at the top entry in the completion queue and then there's acknowledgement function that allows you to acknowledge all the entries in the completion queue. It also has some mid-level convenience functions such as the post-kit weight acknowledge combined command that basically allows you to synchronously submit a command and wait for its completion while giving you a reference to the completion queue entry. So when we go about how to try and emulate all this stuff, what we can do is try to use QEMO which is a super nice platform to experiment with this kind of stuff because we need an IOMMU. Most platforms, most workstations already have this but it's nice to be able to work in a virtual environment. So setting this up requires you to configure QEMO with the Q35 machine type and you need to enable the kernel IRCOOT chip to be split or completely user-based dream and then you need to add the Intel IOMU device, the virtual IOMU and then as usual you add your NVMe controller and you add a namespace to it and then you set up your regular boot drives, configure the CPU type, amount of memory and so on network and so on. So when you have all this up and running, the first thing you do with the library is initialize the device at the PCI node. You do this with the VFIO PCI unit call and then at that point we can now map the controller registers. So in this case we're interested in mapping the M-bar or the first bar, bar zero that we talked about previously. We're mapping the controller registers and the doorbells separately as you can see here. We're mapping the first bar zero. We're mapping the first 4k of bar zero which is the controller registers and then we're mapping the doorbells which are the next 4k bytes on the bar. When we have this mapped we can basically reset the device which means that we write a certain value to the controller configuration register and the device resets and is ready for use. Then we can empty some memory. We use regular virtual memory using the MIMMAP which guarantees that you get a nice page aligned set of memory. Then we assign some IO virtual address. This is something we choose because we control the mapping. So in this case we just choose the null address. It is a valid address and this actually found a bug in the QEMO emulated device because it didn't actually accept the address of zero for say the admin queue or stuff like that. But the zero address is a valid address, hot address, so no problem there. Then we call the VFIO DMA MAP function which maps the virtual address to the IO virtual address and from there on we can use the IO virtual address in commands to the device. So what we need to do as we learned was to set up or bootstrap the device with the admin queues. So we assign two IO virtual addresses at the zero and address one thousand here. We memory map space for the admin queues and then we map them. The next thing we do is that we tell the device about the size of these queues. We do that writing to the admin queue attribute register and then we inform the controller about the actual address of the queues through the admin submission queue address register and the admin completion queue address register. Then we set up an event file descriptor and we assign that to interrupt vector zero which always corresponds to the admin queue. When all of that is done we can enable the controller which writes the configuration register, the cc register and from here on out the controller is ready to use. So let's try to execute an identified command and get some information about the device. Again we allocate some memory of a certain size, in this case the identified data structure given returned by the device is 4k in size and we allocate that. Again we choose an arbitrary virtual address, we can choose that ourselves so we can use the allocator from the library and then we map that address. Then we set up our command. We need to have a data structure available that actually maps to the submission queue entry that we saw in the beginning of the talk and we set it up. We set the opcode which identifies the command to be executed, we choose our controller or a command id and then we set up the data pointer. In this case we just use the first address in the data pointer and set that to be assigned by a virtual address and then specifically for the identify command there's a field called the controller namespace selection which chooses the sub command. In this case we are asking the device for information about general information about the controller. So to write the command we simply use a mem copy as we discussed. We write it to the virtual address, the address space that we see as the user space process and we write it at the current tail of the submission queue address and at the size of the command. Then we update the tail pointer and we take care of wrap around because it's a circular queue and then the final thing we do is that we write the doll bill and we use the utility function again here to do a memory map IO write of 32 bits to the doll bill address and writing the new tail value. So when this happens the controller will pick up the the command and it will execute it and it will generate an interrupt when it's done. So to do that we simply read from the event file descriptor which is a blocking file IO so we block here until it's actually until we get the interrupt and as soon as we have the interrupt then we can read the completion. What this means is that we start reading the completion queue, the admin completion queue at the current head position and then we wait for this face bit or we accept allow we expect a face bit change here. When we read that we constantly increase the head pointer value that we maintain internally and when the face bit changes again we know that we're done. So the final thing we do is that we inform the controller that we have acknowledged and we have read the completion queue entry and we write that back to the completion queue head doll bill. So let's look at this with an actual demo of how this works. So here I've booted up a virtual machine like I described previously and over here we have the trace log output of QE mode that allows us to see what it's actually going on on the emulator unit device. Over here I have my VFIO and the first thing that we'll do is that we'll check that we actually have devices and as we can see here we have a NVMe controller device, a QE mode emulated NVMe device which currently contains two namespaces and we're not really care about that right now. So if we look at the LSPCI we'll see that we have the controller here and the first thing that we do is that we unbind the device from the NVMe driver and we bind it to the VFIO PCI driver. So as we can see here VFIO loads up and what actually happens is that the controller is shut down and it's in a sort of a clean state. We don't know this for sure but in this case that's the kernel nicely shuts down the device as we unbind it. So the next thing we can do now is we can try to run this IdentifyExample and just to see the actual code here we can see that what the Identify is supposed to do is to initialize the PCI device, the writeouts of stuff from the registers in this case, the writeout version of the device. We will map some memory, use the allocator to reserve an IOVA, we'll map that virtual address then we'll set up the Identify command and then we'll use this convenience function to post, kick, wait and acknowledge the command. Finally when the command has been executed we're going to see what the device actually responded to us and print out something from it. In this case we're going to print out the vendor ID of the device. So running this we see that we get a bunch of debug information but over here in the trace is actually what's most interesting. So what we see here is that we see the library configuring the device, specifically it's configuring the admin completion queue and admin submission queue addresses and finally it's enabling the controller and there's some other stuff here that just is part of the initialization but finally here we see that the host or the application we have here is writing the submission queue doorbell and the controller picks this up, notes that it's an Identify command, it executes the Identify command, it maps the address that we have assigned in this case address 2000 and it writes stuff back into this and then it includes a completion and raising an interrupt on vector 0 and when the application here has read that completion queue entry again writes the completion queue doorbell with the new value of the head pointer. So we see here that we could read the register and get the actual specification implemented by the controller and we also read out the vendor ID of the device. Okay now so let's do some bit more tricky stuff. So higher end NVMe devices support something called the controller memory buffer. It's basically a region of general purpose read write memory that is located on the device and it is exposed through a bar, it might be an exclusive bar that means it's at offset zero or it can be an offset into a memory location exposed through a bar. There are two address spaces for this area there's the PCI express address range which is basically like physical addresses and then there's all IOBA addresses and then there's controller virtual controller address range. Now the the area because it's on a bar can be mapped like any other bar in this case we know that it's a 16 kilobyte controller memory buffer it's located on bar two at offset zero so we map the bar like we would map the M bar and we get a pointer to this virtual memory. So we can write it through the PCI express address range by simply writing directly to the pointer with a MIM copy or typecasting and write it then greatly into it. We can also instruct the controller to carry to execute a command and then post and then write the result of that command into the controller memory buffer instead of writing it into host memory. We do this by setting up like a virtual virtual address for the controller this is what we call a controller base address and this address is only visible by the controller so this is not an address that's backed by physical memory or something like that it's only available inside the controller and the controller uses this address to know if it's supposed to be doing a DNA operation or writing into its own internal memory. So we simply choose something in this case we could choose the maximum IODA address so we know for sure that we're not using an address that might be used for DMA so in this case we can use the max IODA address we can add one and we can align it to page size. Then what we do is write a certain register on the device called the cmd memory space configuration register and we write this address into that register and we enable it by setting the two lower bits to one in this case. Then we again set up the identify command but in this in this case the the the memory pointer that we ask the device to write the result into is this controller base address which doesn't have any backing in an actual memory on the host and then we use our convenience function to a post kick wait and acknowledge the command. Now when we have written this into the controller memory at controller memory buffer we can read out the data again we could either ask the controller to to read it from the to read it from the memory but we can also like just use our virtual address reference to read directly on the device and ask the kernel to perform the memory map IO for us. So in this case we just take the address of the controller memory buffer we typecast it to a identify controller data structure and then we can simply access the fields as we would any other data structure. So let's see this connection. So back in our demo setup now we're going to be using this example program called cmb and as I showed in the in the previous slides the the the only particular different thing we're going to be doing here is that we're going to be mapping this address we use it we look into a bunch of other registers on the device to to get information about the size of the controller memory buffer that's what we do here we read some specific registers that defines the size and and and it gives us a and we also can get the physical address of the actual device and or the actual the the bar and which we won't be using but but but it's nice to to to know what it is here in this example and then we we map the cmb we simply assign a controller base address and we write out what the result of that is we inform it the controller base address to the controller then we execute the command and then we write out the version field of of the result of that identified command so running the example and we'll see here that the controller setup here or what we're doing here is that we're setting up the this is all this stuff where we're actually setting up the the cmb and we see here that it says it's located on a physical address this is the physical address that the operating system has assigned to the bar and but we won't be using that because we're not interested in using the physical address so instead we assign the controller base address and we just choose something in this case we choose something that that's this address plus one and the the last i o b a range available and then we carry out the command and then we can write out the version field which tells us that it's yes this is specification version 1.4 okay so rounding up on my talk there's some key takeaways here i wanted to to highlight i think the the key thing here is that you can actually write a driver like you don't have to be a kernel hacker or kernel expert to to write a pc i driver you with a relatively brief overview of the device specification you can do this driver stuff from user space and as as we've seen even with going through the theory you actually wrote a very simple driver that can execute commands compliant with the specification in less than 45 minutes but i also think that that one of this what this shows is that we need some kind of lipo i i think it would be nice that we had something that we could all use that the community would maintain so i'll be outsourcing this as a start and then we'll see where we go from there i'm very happy to collaborate on this if anyone else is interested on interested in trying to get something like this up and running so i'm gonna say thank you thank you for for attending my talk i i hope you enjoyed it and if you have any questions i'm available for q&a both on email but also after the talk thanks again goodbye


Welcome to Monet Café. I'm artist Susan Jenkins. I'm glad you've joined me today for another in the beginner series using watercolor paper with watercolor and soft pastel. This is a photo that I got from my Patreon group where my patrons share their own photos and this was from A.R. Mason, a lovely photo indeed. And if you think you're doing a double take and saw this video before, it's probably because I did another painting with the same reference image using pastels alone and a pastel paper. And in this video, once again using the same reference image, I'm going to create my own sanded pastel surface that's so much more affordable. I always feel this is great for beginners because these things get expensive. I'll be using a new pastel, that little pink one, which is a harder pastel, a set of watercolor. Whatever set of watercolor you have is fine.  It has lots of colors available. Any piece of watercolor paper, I'm using a 140 pound Kansan. The final painting, I believe it's like an 8 by 10. I'm using the smooth side this time. Watercolor paper has a usually a more textured side and then one that's more smooth. So all I'm going to do right now is just kind of sketch in my image to get started with the watercolor. I have a mat that I know the exterior is 8 by 10. It just makes it so much easier for me to be able to mark around the mat rather than measuring it out. And I leave myself a little bit of a border just so I'm not so strict with having to stay so tight in an 8 by 10 format. Now, here's the reference image and I'm basically speeding this up a bit to make the video a little bit shorter. But really, I talk about this a lot in my videos. It's just my little method of measuring. I really just kind of look at the perimeters. If you're working with a drawing and an image that's proportionately the same. In other words, my drawing is 8 by 10 or my surface that I've measured out. My reference image is 8 by 10. So I know that my negative spaces and general composition is going to be the same proportionately. So I really look at big shapes first and where things fall. So I often divide my paper in halves or sometimes thirds and compare that to the image that I'm drawing from. But notice I didn't get into any of the petals right now. I'm just drawing again the big shapes, general ideas of where the vase is, where these flowers are. And I also got in a few of their centers because I like to know kind of where the flowers are facing. All right, so I've just got a flat brush with some water and I've got my little set available and I decided that since the last painting I did with the same reference image was very blue, the background was kind of blue-gray, that I wanted to warm it up. So I thought I would make a yellow-orange background as if the sun was shining. So first I'm really just wetting it. I know I've got a little bit of that orange on my brush but I'm wetting the whole surface. Now if you do this and your paper curls, like this is doing, that's kind of naturally what's going to happen, turn your watercolor paper over and I put a paper towel down so I wouldn't get my front of my paper dirty from my board and just brush water on the backside of the paper and it reverses out the curl. So it flattens it out quite a bit. Now just so you know what we're doing here is we're doing a watercolor underpainting and then after we're done with the watercolor portion, we're going to be turning this into a surface that we can use with soft pastels. Again, much more affordable than your typical sanded pastel papers. Okay, so this is a wet-on-wet watercolor technique. What happens is when you put the water down it allows for it to just wash over the whole thing without a lot of brush strokes. So I'm just kind of working in some areas. I went ahead and covered the whole vase even with the yellow because we know that if there's yellow light shining through it, it is going to affect the vase. Even though I'm going to be putting down some teal colors for the vase, it would look very fragmented or separated if I didn't give some sort of that color in the background of the vase as well. And I think that's kind of a common mistake we do sometimes when painting, especially with watercolor, is we forget that the light and the color usually is consistent throughout the theme and it helps to unify your subject matter. Now, as you can see, all I'm doing is it's still wet-on-wet technique. I decided to just go ahead with these pinks. Again, it's a warmer color. This is more of like a complementary underpainting because I know I'm going to have the blues and the vases or the vase and some of the lighter colors and the flowers. I just thought these colors would work well as an underpainting. An underpainting is just a painting underneath a painting. And I know a lot of people have questions as to why, but really it, for one, it fills in a lot of empty space. Two, it gives you a mood or a color to set the theme. And three, if you choose the right colors, especially like with the complementary colors, like I'm doing here, complementary just means the opposite on the color wheel. Like a teal vase, look on the opposite of the color wheel, you're probably going to see like the oranges and the more warmer colors. So if you, if your subject matter is cooler, it's good to put a warmer color underneath. Not always, but it just makes those colors really vibrate and sing and makes your painting much more visually interesting. So I'm actually really kind of liking just this watercolor here.   So now I'm using more of a deeper magenta to go in there. The vase definitely has some darker stems and things reaching down. Shadows, there's almost in the reference image. There's almost some blacks. I hardly ever use black. I make my own mixture of black with watercolor or I use really dark, darks in with my pastels to represent black color and value is all determined by what it's next to.  If it's really dark next to a really light another surface or value that it's next to. So again, just still playing around with this watercolor.  I know I've had some of you guys request that I do some more watercolor alone videos and I'm definitely going to be doing that. I've got a list of painting tutorials that I'm going to be doing. And as soon as I get done with those, I'll be working on some pretty basic watercolor tutorials. I'm glad you guys are liking the beginner videos because I really feel my heart is a lot to teaching the beginners. You know, I just have a passion for that. Now what I'm doing here is again, I'm just kind of developing the flowers. I'm still not doing any individual petals. I'm working more on the value that just means lightness or darkness of the individual flowers. The ones at the top, you see they're much lighter and the ones in the front have that shadow on them because the light is coming from behind. Now all I'm doing here is blowing it dry just because I'm impatient. And after I get it blown dry is when I'm going to take this surface and turn it into a surface where I can apply soft pastel. Now with everything dry, we can use this product, which I'm shaking up that is called Clear Gesso made by Liquitex. And it's a product that has a little bit of grit to it, which turns it into a sanded paper. It's important to get the clear gesso because it's the one with the grit, not regular gesso, and it's clear. So you're not going to cover up this watercolor that you've already done. Now there's different methods you can use for applying this. And I've recently found out that this little roller brush, a foam brush, works so well for applying a smooth coating to this. Now sometimes I want a lot of texture, but if you're looking for more of a consistent surface, this brush works really great. I've recently discovered that I used to put my gesso in a dish, but you know why not just pour it over the painting. So one maybe slight disadvantage to the brush technique is that it does soak up a lot of the gesso. You need more of it to get it down good, but it's a great technique. It works very well. Here's a painting from a previous video where I used a very textured brush. You saw the brush I just showed to apply the clear gesso. Same exact technique. This is watercolor paper. I did an underpainting with these warm tones. I applied the clear gesso with a coarse brush and I got all this gorgeous texture.  And for this painting I will once again be using the 120 hafstic set of unison pastels. I know I'm using the heck out of these pastels, but for these beginner focused videos I like to share that you can get sets where you can complete a whole painting. Now this is some sandpaper that's an 800 fine grit, like hardware store sandpaper. I'm not going to do it for my painting, but I do know that some artists sand down the surface to even get it smoother after the clear gesso has dried. Now it is dry now and there is a little bit of warping, but I find that once I finish the painting it's pretty much all flattened out. So you might have to work around a little bit of the bumps. I just want you guys to be aware of that. But for the most part you can, it will flatten out at the end. I'm choosing my pastels that I want to use. I know I've got that gorgeous vase and I love the way this set is arranged according to color and value. It makes it very easy for the beginner to get it like a row of color that's increasing or decreasing depending on how you look at it in value. I know this vase has, it may look very teal in the photo, but without some other colors to punch it up and give it some interest, it will look very flat. So I'm using some purples. I'm using the teals of course and I'm using some rich gorgeous blues. I'll be starting with the darkest dark first in the vase and again it's where those shadowy areas are or where all those stems are all kind of joined together in the vase. And notice I turn my pastel sideways sometimes. I recently in my, on my Patreon page had a video for my patrons that talked about the differences of line versus edge and it's too much to describe here. But basically when you use edge you often can use the side of your pastel rather than the point or lifting your pastel up to create more of a line. And by using edges and soft edges we often create a more painterly look. Now it's going to, I want to let you beginners know this. I just had recently had a question regarding pressure and that someone felt they used so much pressure that their painting just turned to mud really quickly and they couldn't get any more layers. And so definitely focus on a light touch. I know we have the tendency to want to get the color to look rich or fill up the tooth of the, or all those empty spaces of the paper more quickly than we should. But keep in mind don't worry about these splotchy spaces behind your color, okay?  Now this is just a chamois cloth. I have found that when I create my own pastel papers this way a chamois cloth that you can get at and you know pretty much any store. I got these at the dollar store. I bought a huge sheet of it for a dollar and it works as a great little blending tool. And so early on is usually when I will do blending not later on in a painting and usually it is just to kind of, you know, like I said, cover some of those blank spaces or to create a softer impression. And I know that the left side of that vase is more see-through so that's why I didn't put any dark right there and sort of see where it is right there. Now what you just saw was all real-time. I'm speeding this up just a bit and going to finish out some of these colors in the vase, pulling a few more colors, some more purples, more of a dull teal color. And just watch as I gradually layer different colors and values. And again, don't press too hard, keep a really light touch for this phase because you can always go back and add more pressure, but you can't really undo once you've muddied your surface. I mean, there are techniques you can do to erase pastel. Oh, I'm still using some blending with the little chamois cloth. But again, there are techniques you can do to erase them or so-called erase them. You can use a stiff bristle brush if you've got an area that you need to kind of get back some of the tooth to be able to layer. And I always recommend go outside if you're going to do this instead of doing it in your studio where you could get a lot of pastel dust. If I've got a big area like that, I definitely go outside and then I give my painting a good tap or whack on the back to get all that dust out of there. But anyway, it's always better to not rely on that. Try to get your strokes efficient and correct to begin with and keep a light touch. So that's my point there. Now, I'm gradually just, all I'm doing with this reference photo is looking at color and shape. I'm not even thinking about this being a vase. I'm not thinking about the stems of the flowers. I am literally just looking at color and value and making shapes. Some of those little brighter teal areas you saw where they look kind of dotted. That's just some of the areas where I saw some of the brighter colors. I know it's like light coming through the vase and again, just working on shapes and values and colors, of course. But again, I'm using the chamois cloth gently. I'm not overblending.  They'd start to get combined together and kind of get muddy, you know. So that's a general kind of soft look to this face. And now that I have the basic color and value in, now I can start working a little bit more strategically with some of the colors and then eventually deeper shadows and then more highlights. And pay attention in this video. You'll see how, oh, gotta have some coffee. You'll see how the vase right now looks very flat. And then I move on to other areas of the painting often because I want to have a cohesive feel. So I'll work it to a certain degree. And sometimes I will have already moved on past this, I mean, before I got to this stage, but I kind of wanted to get some of the specifics of the vase in before I moved on. But typically it is better to work all over your painting versus get so caught up in one little area. Again, you won't have the cohesiveness of your painting. Sometimes you'll just frustrate yourself. And I just feel like it really helps the painting overall for your values to be more correct if you don't get stuck on one little part of the painting. All right, so now you can see how I'm just gradually getting some of the darks in. But notice how early on when I added those darks, I didn't focus on getting them as dark as they are now. In other words, I wanted to get the basic values in, not a lot of pressure. So I don't over muddy. I want this vase to be very vibrant and gorgeous with color. So again, I know I'm kind of overemphasizing that, but keeping your pressure light at the beginning is important. You can always, always press harder later and get more colors as long as you haven't muddied up those gorgeous colors. And that's one of the beauties about pastels. I really feel, I think it's actually true and it's been said that pastels are the most vibrant. They really do have the most pigment of any medium that you could use. It's got more pigment and less binder than, say, acrylics or oils or watercolor is pretty vibrant, but you're adding water so you're diluting it, you know. So pastels just have that gorgeous brilliance and vibrance. I think that's why so many artists like it. And I like it for multiple reasons.  Now you saw me just tapping that because I'm working flat for filming purposes. I know some of you guys like that every so often if you work flat, pick up your pastel paper and tap it off. Again, if you want to go in another room or outside or whatever, I don't find the dust. I don't get that much of a problem. I recently had somebody ask me about that and the toxicity of pastels. I know there are pastels, there are warnings on pastels about toxicity, but myself and many other artists, I know haven't had a problem with that. But if you're more sensitive or you're just concerned about that, there are things you can do. You can get a mask. You can also get, for your hands, it's called, what's it called? Gloves in a bottle? Let me look at it real quick. Yeah, it's called Gloves in a Bottle and it is made by, well it's all in one shielding lotion, but you can find it. That's the name of it, Gloves in a Bottle. You put it on your hands and it's literally like an invisible pair of gloves. Now I'll be working on the lighter areas of the flowers. I'm going to use a cooler blue color palette for these. And once again, I am not going to focus on individual petals just yet. I still want to get the values in correctly. And those upper flowers are going to be the lighter ones. And the ones, more of the ones like where you see I've drawn the centers in already, they're more in shadow. In the reference photo, you can see they, probably because they're facing the viewer more, the light's coming from from the back. And so not as much light is getting inside of the flower. So I am just blocking in some shapes here. And again, same as the vase, you don't want to give too much pressure here. And you just want to focus on shapes and values and, you know, color with regard to color temperature and, you know, your basic color palette that you've chosen to use. So now I'm, I'm working in different shadowy areas where I know there's more, more shadow and depth. And using the chamois cloth every so often, but I, I'm still, I'm working all over the whole body of flowers in the vase and just zoning out kind of and forgetting they're even flowers. You know, I know that might sound weird, but sometimes when our brain tells us what something is, we tend to paint an interpretation that we have of that thing rather than maybe how that thing actually looks in this particular situation. So it's a really good idea. I don't recommend you always zone out when you paint, but that's just my term that I use to help you to think about the shape and the value versus what your brain is going to tell you it is. So I know I sound like I talk about that all the time, but it really is so important. It's kind of like little kids when you tell them to paint a sun and they paint a circle with these little rays and most likely it's probably because they've seen it painted that way before or in their young life without a lot of information. They haven't really analyzed what the sun looks like and how, how it can look differently at times. So we're trying to break out of that pattern of painting what we think something looks like rather than what it really looks like. All right, enough of that. Okay, so I'm gradually getting lighter as these flowers get it, get higher up and kind of further back where that sunlight is actually hitting them more than the ones closer to the viewer and, or closer to the vase and facing forward. All right, so I'm going to paint for a little while here. Just kind of pay attention to the values and the shapes and I'll be back. I am starting to differentiate some of the shape into petals and, you know, there's, I really could have just blocked it in and done the darker petals on top of it. That's often how I work and where most pastel artists work is like that layering method, but this one's going to be, that one and a couple of the other ones that are facing forward are going to be more of my focal point. And so I'm going to kind of differentiate between some positive and negative shapes. At least that was my thought. Okay, now I'm using the shaming cloth and I'm blending it a little more because I felt like they were a bit too chunky. And in hindsight, if I did this again, I probably would have done a softer approach and not distinguish the petals as quickly as I did. I mean, you do definitely develop the petals as you go, but I think I got them a little bit too fragmented, I guess I could say, in this particular flower. So, hey, in these videos, I like to share when I would have done something different myself. But notice how this is still very blue right now. It's still a very dark value because it's in shadow. And by the end of the painting, I, and even right here, I'm adding some lighter values on those tips where the sun would be kind of reaching over or peeking through. And I'm really referring to the reference photo a lot. You know, I'm not imagining this. I'm looking at how the light is really behaving on this face of flowers. And I actually want to paint this again after watching this video. I find that's a great thing for beginners to do is take the same reference photo and paint it not just to vary it up and changing and having fun with color palettes. But also, I mean, even if you try to paint it the exact same way, you learn something every time you paint something. And I feel like we're always learning. I mean, that's why I feel like I want to paint this again. I got the flowers too heavy looking with the darker values too quickly. I mean, I'm still happy with it. It was a fun little painting, but I see things that I could have done differently. So I really advise you to do that. That's why I often advise to paint smaller and do studies more often because sometimes we put pressure on ourselves as artists or beginning artists to think we have to get or be so good so quickly. And it just doesn't happen that way. So embrace your mistakes and your failures. And when you see something that's not so awesome, like I'm looking at myself with some of these flowers, I end up kind of correcting it at the end. But I know that, you know, it could have been better, you know? But that's okay because we're human and we're all learning. And as I often say, the glory is not or shouldn't be just in the final product, but in the experience. I mean, we sometimes think that the physical object is the goal and it being a beautiful painting. But there is such a beauty in creating and the process and the intangible aspects of art that I really want to focus on that myself. Because when we focus so much on the physical, this is not just an art, but in life, we can become very materialistic and we can become very disappointed, you know, in things. There's so much more in life intangibly that can be experienced. I play around with the background a little bit more here. I am speeding up this last part of the video. And again, per your request, I am trying to focus on certain aspects that I can share instead of getting overwhelming with talking about every single thing that I'm doing in the whole painting. And even again, some of the things that I thought I could have done better to help you guys know or learn even more about correct and incorrect ways to do things. So I warm up the flowers a little bit and so just watch to music as this painting develops. And I hope you learned something. I hope you'll hang out till the end. I hope you'll subscribe. And if you'd like to become part of my Patreon fun on my Patreon page, you can become a patron of mine which supports this channel. I want to focus that that is the main reason for supporting my Patreon page is to keep Monet cafe keep me able to financially bring free videos to Monet cafe. But I always feel that, you know, if you're going to spend $5, there's two levels five and $10, you can pick whichever one you want just as a support level, but they both provide the same thing. But I feel like if someone's going to support this channel that I really want to give them a little more. So we I actually am giving probably too much of my time right now. We have so many neat things that we're doing. We have homework assignments. I have a story time. I read from different art books for you guys. And it's just a wonderful way for me to get to know you guys a little bit more too. So anyway, if you'd like to become a patron, you'll see the link posted. Enjoy the rest of this video. And I'm just so happy you joined me. I'll say bye at the end, though. So hang out till the end. Get a little more work on this. And then I tried to really lighten it up. I felt like I had lost some of the color intensity. So I hope you enjoyed, learned something. And I hope you'll come back. All right, guys, happy painting and have a blessed day.