Good morning everyone and thank you very much for joining us for presentation on fuzzing a host-to-guest attack surface of fuzzing virtual UPCI drivers in the context of the protected KVM. So it's a big pleasure to you or give this presentation at KVM forum in person in Dublin and especially to co-author it with Bill Deacon who is an active upstream kernel developer and he's sitting here and he's also leading the protected KVM project at Google to enable KVM for Android. My name is Eugene Rudionov. I'm a member a security engineer on Android Red Team where we do offensive security research for Android and Pixel mostly focusing on their low-level firmware and this presentation today is a collaboration of the two teams, feature team and Red Team where Red Team was tasked with performing in-depth review of for Android protected KVM to proactively identify any security issues in it and make sure that Android protected KVM is as secure as possible at the release time. A quick introduction what Android protected KVM is. So essentially your Android 13 comes with your virtualization services which enables a native support for launch and virtual machines with mutual distrust and the privilege there are two key words here. Mutual distrust that's your guest VM no longer trusts the host so basically your Android protected KVM provides security guarantees to the host even if the guest even if the host is compromised even if the attacker is able to run code in the privileged context on the AP and vice versa there is protection from misbehaving guests to escape in the host and the guests are running outside of trust zone so they don't have trust zone privilege which we believe greatly reduce attack surface. There will be a number of other presentations at KVM forum so I won't be here in details or covering various aspects and Quentin will be doing a technical deep dive for protect KVM tomorrow so he'll provide a lot of informational level implementation details and we will move with their looking at actual security concepts offer Android protected KVM and its security pillars. So at the very bottom we have secure boot and Android verified boot which is responsible for authenticating their system boot partition making sure that we're booting the authentic kernel and the hypervisor hypervisor is the keystone of the Android protected KVM as it actually enforces the isolation of their memories between guests and host and their mark on the next session will be providing deep dive information on how this memory isolation works and on top of which we have an attestation and sealing to actually enable the confidential computing in Android so sealing enables virtual machines to have secret data on the per VM basis and attestation enables external parties to attest that the code running in the guest is running in a good security configuration and finally on top of which we have the actual software running on the host side and in the guest on the host side we have a virtual machine monitor and set of the vertical backend drivers and in the guest we have the actual payloads and boot loaders and due to the mutual distrust the guest receives untrusted input from the host and their software is software if there are any vulnerabilities in the software layer even if the all the all the underlying blocks are implemented correctly and perfectly secure exploiting those vulnerabilities can undermine security properties of the Android protected KVM and this is what we are wanted to mitigate and focus in this presentation so basically in our in the scope of our engagement we focus on the holistic picture to make sure that all the layers are implemented correctly and don't have any flaws but in this particular presentation we will be speaking about their software layer and we will be prioritizing host to guest attacks so with that here is how the attack surface for Android protected KVM looks like from the standpoint where the attacker is able to run privileged code in the host we have different layers we have L3 running the most privileged code on the AP then we have a hypervisor layer which exposes a set of hypercalls to the host and to the guest on the side of the protected VM we have a set of the initial boot loaders executed so we have PVM firmware this is a stage one boot loader the very first code which runs in the year Android protected VM once it's launched and the micro droid loader is the next stage with order so both of them are based on you boot but your the feature team is actively investigating the possibility of writing PVM firmware in Rust as an additional defense and depth strategy to make the whole design even more secure as a kernel we have generic kernel image Android kernel and as a user space payload exception level zero we have micro droid manager payload which provides the actual services on the whole side we have cross VM as a virtual machine monitor virtualization server and we are assuming that the attacker runs code either at exception level zero with root privileges or it basically compromise the kernel and everything or in orange here is the attack surface exposed to the year to the attacker and in particular we highlighted here the vertio boxes here available both and in the boot in U-boot and GKI this is this is the main communication vehicle between the host and the guest and this is what we focused in this in this presentation to really mitigate this attack surface covered with fuzzing so why fuzzing vertio is interesting so vertio is their specification we sure describes different implementation modes the respect mode split mode just a quick overview for for the split mode vertio works by having a shared buffers between host and guest so there is a shared buffer called the script or table which is a table of descriptors and every descriptor describes a I operation it has a physical address of the buffer a length of the buffer flags and the next field which is used to chain descriptors in the descriptor table in the chain of descriptors if there are more than one descriptor is required for the operation if we have time and then of the presentation I will show one of the issues which we identified with this fuzzing effort exploiting this next field in U-boot and we also have a avail buffer and use buffer avail buffer is used to communicate from the guest to host which descriptors are in are used by the guest and at the same time used is used to communicate backwards by the host which descriptors have been processed so for instance if if a guest would like to perform an I operation it it finds the very first available descriptor in the in the descriptor table initializes it with the proper values and puts it in the avail buffer it signals to the host host reads it from the avail buffer determines which descriptors it needs to process it handles the operation and it marks in the used buffer which descriptors it's processed and signals back to the year to the guest and as we can see here there is a lot of pointer arithmetics here there is a lot of signed unsigned integers there is conversion from little engine to big engine and this is a generally a error prone code so or and what happens and the whole protocol is also very non-trivial and if the attacker is able to modify those values with the full right access they can exploit vulnerabilities in the guest get code execution or exploit confidential data so this is really what we don't want and what we want to mitigate and here is how the virtio stack looks like in the protected guest so or we have the following devices we have a virtio console virtio block virtio fs and diesel devices the box highlighted in purple they are not present in you boot but they're present in GKI so nevertheless this is the attack surface that we are interested in everything works on top of the virtio PCI transport and there is there's ring this ring driver for handling the ring buffers for virtio before we go to the fuzzing a few words about the existing efforts on fuzzing on a hard in virtio so for Linux mainline host to guest is not in you attack vector there are cases where the Linux is deployed in confidential or a computer environments and there is a set of hardening patches so and those patches are not from Google so this these are industry patches from into other companies and reference at the bottom of the slide however this attack surface is new for Android and more particular in for Android protected KVM and that's why we are concerned about it and additionally here the the when we looked at the year implementation of the virtio in you boot it wasn't hardened against this attacks and because of this engagement and fuzzing effort there was a set of hardening patches submitted and upstream to the year you boot repository to your make sure we're not dropping any zero days today so why fuzzing well bottom line up for we don't have too much options for a project of such complexity as Linux kernel for the solutions which can provide continuous security and be at the same time like very efficient from the standpoint of their soundness and correctness and levels false positives in addition to any other benefits which fuzzing provides the year the existing de facto standard tools c-scolar since bots already identified a tremendous number of security issues in Linux kernels or that was there one of their options that we started to look at however when it tried to apply to our fuzzing virtio we encountered some challenges first of all the year the overall approach of c-scolar is to fuzz everything inside of the guest VM so basically kernel under test runs in the virtual machine and there is a C's fuzzer component which generates mutated data input which are sent to C's executor which translates them to a series of c-scoles executed over kernel under test there is a coverage guided feedback feedback to C's fuzzer so this is how it works and this design works perfectly well when you are fuzzing from top to the bottom so basically you are attacking kernel from the c-scolar interface however we we have our data coming from the other side from bottom to the up from the hardware and this needed some modifications for the c-scolar so this is not here an ultimate limitation of the c-scolar there are some successful projects such as USB fuzzing effort published two years ago where there was additional instrumentation additional c-scoles provided in the c-scolar to do this and there are some other efforts such as KFL and KFX for doing that however our team has a prior experience prior art with the tool which is called LKL so we decided to leverage it for fuzzing virtio in this case and LKL is their Linux architecture port for Linux kernel which enables building Linux code as a user space library and link it to the application so we're essentially on the right hand side we can see how the LKL application looks like at a high level so at the top we have the actual payload functionality then all the boxes highlighted in green are what LKL is so there is a c-scole API layer there is Linux kernel which is unmodified Linux code built for LKL architecture for yes generic LKL archcode is here and at the bottom we have a host environment portability layer this is a layer which provides a set of callbacks for memory location for treating threads for like synchronization primitives or to make it run on various in various environments such as POSIX, Win32 or Mac and this enables a very convenient tool for doing kernel unit tests and their and fuzzing so or and here's just an overview how LKL can be used in your C program so or what you need to do build just Linux kernel for LKL as a static library and links and link it statically to your application so or at this line of code we are essentially initializing the kernel that takes to arguments LKL host ops which is the host environment portability layer providing set of callbacks for memory location creating threads synchronization primitives and then goes your command line argument as we are starting kernel without any other processes we need to mount file systems by ourselves to your make sure that we are able to your access device devices and for instance if we would like to work with device you hid we would need to create node manually by invoking Mac make node add a syscall and there LKL underscore C's prefix indicates that we are invoking the syscall not from the host kernel but inside LKL and the same goes for open syscalls or once we create a node we are able to open this device and we are using LKL underscore C's to indicate that we are invoking it in the LKL kernel not the host kernel so overall as we have everything within the the same user space process this enables a very interesting use case for fuzzing where we can fuzz the Linux kernel without any virtual machine any virtual machines running inside and inside of user space using a coverage guided feedback fuzzer such as libfuzzer as a result we are we can get quite high performance and scalability on x86 cores because at this point fuzzing Linux kernel becomes very similar to fuzzing user space applications as an additional benefit we can also use their existing tools for debugging user space applications such as GDB for crash data application stack trace analysis and reproducing test cases and as we are building it for an LKL architecture we have a possibility to muck out the hardware interfaces such as PCI bus and this is actually what we are using in this in this fuzzing effort because LKL already comes with the implementation of the virtio back end drivers that's why enabling fuzzing virtio front end drivers using LKL was there not that complicated so we can reuse already existing code in LKL to fuzz this interface obviously there are some limitations so LKL doesn't support symmetric multiprocessing which makes it not very efficient to save the least for catching risk conditions and their other concurrency related issues and obviously as we are building code for x86 we might encounter some false positives true negatives for the code which is originally intended to work on our 64 architecture here is another view of the fuzzers developed in the course of this works or a few interesting highlights is we are testing Android 13 5.10 kernel this is the kernel under test and we focused on working with virtio ring because this is the actual the protocol which we wanted to cover with fuzzing the fuzzer work works by getting the mutated inputs or filling the ring buffers which I showed previously with their this mutated inputs and then calls into the guest and gets coverage feedback so we have case sample crash detection we also fuzz the virtio PCI as a transport layer so this fuzzer has some limitations so the work is still in progress we have currently it working for on the probe on the probe path and we are leveraging their possibility to mock out their MMO interfaces in LKL because we are basically we have control over it so we can we can we can provide their mutated data directly over their MMO interface exposed to your PCI bus and we have also a virtio block fuzzer which fuzzers their block configuration block for virtio block device and right in those fuzzers identified a number of security and stability issues here is an example one of the issues which was identified by virtio block fuzzer so we have here an out of bounds right on stack where in block read the full page function where there is a local array of fixed sized AARR and this array is being used in this while loop where there is like index NR and we're assigning block head structures and there the fuzzer generate interesting input where we go out of bouncer and this issue is pretty dangerous because by using out of bounds right on stack we can override the return address and this potentially can lead to code execution in the context of the guest the root cause of the issue was the unusual block size generated by the fuzzer so this is this is the block size and the interesting bit is that we get I block beats set to 32 and the shift left shift of 32 bit integer one to 32 bit lefts is undefined behavior in C and interestingly here on x86 architecture the result is very non-intuitive if you shift one 32 bits left you get again one and yeah this is there how the issue manifested itself in the code probably this won't be reproducible on R64 I believe because it's gonna be it might be zero but I don't know if there is any cases covering what will happen if the create empty buffers will be passed as zero as a second argument if you were some fuzzer and vertigo drivers stack on U-boot because U-boot is also used in the PVM firmware in microdroid bootloader to interesting devices where they block and a vertigo console for debug outputs are in the course of this work we developed a set of all feature team developed a set of patches which enabled fuzzing in U-boot in sandbox mode so basically your U-boot has a sandbox mode for unit tests where it builds for x86 architecture and there by leveraging the sandbox mode we're able to fuzz the vertigo interface with a San enabled and there was a number of security and stability issues identified addressed by this patches which are merged upstream so everything is fixed and here is one of the example of the issues which were identified again in this fuzzing effort as you remember in the very beginning of the presentation I showed you that there is next field in the descriptor table which is fully attacker controlled so or in this case we can see that this value is read into the index in the variable I which is not sanitized and passed to the year a weird Q detached descriptor function which in turn calls the bounce buffer stop where state argument is computed based on the index I so basically if I is out of bounds state is also out of bounds and fully attacker controlled and this dangerous main CPI operation with the fully attacker controlled arguments makes code execution pretty straightforward and again this is one of the examples of the issues caught and identified with this fuzz an effort in you boot and finally this brings us to the conclusion so or and of course of this effort we developed a number of fuzzers which are running and inside Google on the continuous fuzzing infrastructure internally this is not that we develop fuzzers run it for some time and then forget about it no the fuzzers are running 24 7 and as new code has been added to the repositories has been checked in by the fuzzers picked up and they're being fast tested those fuzzers identified a number of security and stability issues which are addressed proactively before release of the year Android protected KVM and as their as as Google launches their Android protected KVM and virtualization framework this makes the certain improvements for Android platform makes their code more transparent more updatable everything as part of the ASP repository and this is a great opportunity to your leverage community support for fuzzing because there we really hope that your Android protected KVM will inspire new cases and by contributing to fuzzing we can harden the harden this implementation altogether a few words in the future work so or well we're never done with fuzzing we have our plans for improving existing fuzzers developing new fast targets and we're actively working on the upstream in LKL to your Android comma kernel so or LKL is not part of the upstream Linux mainline there was a number of the upstreaming efforts the last one started in 2020 but there is still a considerable amount of work to do it to upstream LKL as part of the UML and to share the results of our fuzzing work and our fuzz and tools with the open source community we're currently working on the first upstreaming into the Android common kernel and this is it thank you very much for your attention I think there's some time for a little night to take any questions which one so we're oh yes so basically the question is how we how can we run the year in the needed improvements for improving fuzzing for virtualization I I think I think we definitely need to your more hands in writing fuzzers covering other virtual drivers because currently we have good coverage for virtual block virtual virtual ring virtual PCI there are some other drivers which are would be great to cover is virtual virtual console and vsoc driver is definitely something that is on our list to fuzz and I think I think also different fuzzing tools they bring different dimension to fuzzing because fuzzing is not deterministic process and generally fuzzers are trying to solve a very big so basically they're doing search in a very big space and different fuzz and engines provide different different strategies so basically your c-scolar is one way of fuzzing LKL is another way of fuzzing I think it would be interesting also to explore other other fuzz engines with respect to your how well they're doing for virtio stack so I think that would be your probably number one from from the top of my head I have a question from the chat I'll repeat it says hello may I ask besides virtio what about attack surface from host KVM like shadow VCPU state sync between host and guest could it bring sensitive data leakage so or this attack surface is not covered with fuzzing but your this attack surface went through the manual code reviews audits or we did look at this attack vector and indeed yeah this is absolutely different attack better that we are wanted to mitigate one of the requirements so or one of the reasons why we're focusing on we're fuzzing on virtio because we would like to have our fuzzers running in their continuous infrastructure and there is there is there are existing fuzzing engines such as KFX and KFL for fuzzing the which might be interesting for fuzzing this direction however I think there might be a little bit problematic to run continuously in our fuzzing infrastructure so that's why we decided to prioritize fuzzers that we we can schedule running 20 per seven and we we covered it with the manual code review but this is an interesting topic or if it's worth considering in in the fall of work yes so we'll is adding that the hypervisor was written in with the security in mind while we're using a lot of virtio drivers in the guest which were written under assumption that the host is is fully trusted so our host has full control over the memory and this is an interesting shift in the attack surface which we really wanted to prioritize to the best of my knowledge we haven't received any pushback from the so you're speaking about upstream and fixes for thank you very much