Good morning, everyone.  Just to start by introducing ourselves, my name is Jonathan West. I am a senior software engineer at Red Hat, coming to you from Toronto, Ontario, Canada. Hello, everyone. My name is Sharma Jean. I'm a software engineer and a maintainer of Argo Projects. And we thought we'd start out by giving you a brief elevator pitch, say 90 seconds or so, on what application sets are, how they work, and some of the features that you might find interesting there. So application sets are a new Kubernetes custom resource and a controller that's responsible for managing that custom resource, which works alongside an existing Argo CD install. So you specify, with your application set, a template, a generator, which is used to customize that template. And then the application set goes and creates applications based on the template and generator combination you specify. So you can think of application sets as being kind of like a factory for Argo CD application. You specify the template, the generator, and then the application set controller automatically outputs a bunch of applications that meet your specific application specifications.  So it's very easy to manage. Those cluster deployments can be customized using data from external data sources. So for example, as you scale up to add new clusters, as you add new Git repositories, as you add new applications to your existing Git repositories, those will automatically be detected by application sets and deployed to your cluster using the template. It does automatically react to new external changes on the fly. So as I said, infrastructure changes correspond to cluster changes, and that's both quick and customizable. Finally, application sets are based on Argo CD applications. So for folks that are already familiar with Argo CD and know and love it, you're going to get all the power that comes with Argo CD with Argo CD application sets. So speaking of harnessing the power of Argo CD, I will now hand it over to Shama to talk about Argo CD and GitOps. Thank you, Jonathan. So to begin with, I'll start by giving a brief introduction of GitOps. I'm sure many of you might already be aware of it. GitOps is a way of implementing continuous deployment of cloud native applications. It is a workflow that emphasis on Git being the single source of truth. There are two major components of that workflow. First is that the desired state of your software is stored in form of files on Git. And because we are talking about cloud native, your software environment will be stored in Kubernetes and you'll be storing Kubernetes manifest on Git. And the second part is that there is a GitOps agent, a piece of automation software, whose sole responsibility is to make sure that the desired software state keeps on running on the RENTAN environment. Hence, Argo CD comes into picture. So Argo CD is one such GitOps agent. Now what is Argo CD? So Argo CD is a GitOps-based continuous delivery tool for Kubernetes, which uses Git as source of truth. It is declarative and supports a variety of configuration management tools such as caseinit, caseinit, customize, and any custom config management tool configured as config management plugin. It is implemented as Kubernetes controller and CRD. And it provides a Kubernetes dashboard with powerful sets of source management features. It makes sure that application's current life state is same as the desired state. So the diagram here explains how Argo CD works. First, you will define your application's manifest in Git repository and then connect that repository with Argo CD. And then applications can be synced programmatically either via GRPC or REST API or they can be synced via Argo CD's UI or CLI. Now let's get to a quick demo. I'm sorry. So this is Argo CD's user interface. Here you can see various applications that this Argo CD instance is currently managing. Let's get started by creating a simple application. So for demo purposes, I'll be using this Git repository available as this Git repository of sample applications available as part of our group project. And today we'll be deploying customized application. So as you can see, Argo CD detected all potential applications in the given Git repository that can be deployed in the Kubernetes cluster. For destination cluster, I'll select the cluster where I want to deploy my application and the namespace. Here are some parameters that you can set for your application depending on the configuration management tool being used. Argo CD detected customized specification file in the repository path and hence it is providing some customized specific options. So our application is now created. Argo CD application is a logical grouping of related resources managed and deployed as a unit. In the middle you can see the resource tree and this is showing that there are four resources currently being managed in Git. Two service objects, one deployment and one stateful set. And each item in the resource tree is selectable and by clicking on them you can see further details. This is the diff of the actual and the desired state of this resource. And this diff that you are currently seeing means that this resource is yet to be deployed. So I'll go ahead and deploy our application by clicking on sync button. So now our application is being deployed. As you can see, Argo CD provides real time activity of the application. And it also shows relations between the objects. For example, this deployment created a replica set which in turn created three pods. And you can see further details such as pod summary, pod events and pod logs. Now Argo CD provides mainly two statuses. First is the health status which is the overall health summary of your application. And it is Argo CD gets started by aggregating the health of the application's resources. And the second is the sync status which is the overall synchronization status of the application. It lets you know if your application is in sync with Git or if there is any configuration drift. Now our application is deployed and is in sync with Git. So I will now go ahead and deploy another version of our application. And because this is GitOps, I'll make that change in Git. After making that change, I'll go ahead and refresh my application. As you can see, Argo CD detected the change and our deployment is now out of sync. And here you can see that the difference diff is in the version. It should be now at 0.2, but it's currently at 0.1. So I'll go ahead and sync it once again. So now our Argo CD application is deployed again and is in sync with Git. So that was a quick demonstration of Argo CD. I do want to mention some key features of Argo CD are as shown on the screen. Argo CD provides some many more powerful features that you can check out on its GitHub page. Now I'll hand over my screen to Johnson. Here's a demonstration of the Argo CD web UI. So this is the Kubernetes representation of applications as we were seeing in the Argo CD web UI. So like this is the central custom resource that Argo CD uses to do GitOps. So because this is GitOps, we have a source repository that contains the Kubernetes resources within a Git repository. In this case I'm deploying a desktop from the desktop path. So that looks like this. So just simple Git repository with a deployment resource and a service. So nice and simple. And I want to deploy that to a Kubernetes cluster. So I've got my source. These are the resources that I want to deploy. And this is the destination for those resources. So the guestbook namespace and the particular cluster, which is just going to be my local cluster. So you can think of Argo CD and Argo CD applications as being the glue between Git and Kubernetes. So Argo CD looks at your Git repository, looks at all the manifests that are there, whether it just be like in this example, a set of simple YAML files, or maybe something like Helm, Customize, and so on. And applies them to the destination, applies them to the cluster. Once they're applied, it's up to Argo CD to keep those in sync. So as we saw with Shama's demo, if the Git repository changes, it's up to Argo CD to make sure that what's in Kubernetes is always consistent with what's in Git. But notice that in the application CR, there's only one source and one destination. So you can't say, I want to specify, you know, I want to deploy this one Git repository to 10 clusters for like multi-cluster deployments. You can't say, oh, I want to deploy these 10 Git repositories, this one cluster for something like a monorepo. It's just a one-to-one mapping between a single cluster and a single Git repository path. Wouldn't it be great if we could say do many-to-many relationships between Git repositories and clusters? And I'm happy to say with application sets, you can do exactly that. You can deploy two multiple clusters. You can deploy from a bunch of Git repositories all from a single application set, managing them as a single unit. And I will show you exactly how to do that as a demo in a few moments. So the application set controller is open source, of course. It's an Argo CD sub-project hosted within the Argo Proge Labs organization. And it's built from contributions from many different folks, including individual contributions from folks from Red Hat, SNCC, Intuit, MLB, Alibaba Cloud, and more. It integrates alongside an existing Argo CD installation. It's very easy to install alongside that. And then they work together in tandem to do deployments. So how exactly does an application set allow you to deploy from many different Git repositories to many different clusters? Well, it starts out with the application set custom resource, which is the main custom resource of the application set project. In here, there are a couple of fields, which correspond to the two major concepts of application sets. We see generators, which are responsible for generating template parameters, and template, which is responsible for using those parameters. So let's start with template. You might recognize this from something I showed you a couple of slides ago. Notice that we've got the source here, which is a Git repository, and the destination, which is a Kubernetes cluster. And that sounds familiar because we saw that a few slides ago with the Argo CD application resource. And that's because this template field here is just an Argo CD application, a templated Argo CD application. So the exact same set of fields that you get with Argo CD applications, you get here. So you get all the power of Argo CD. But there is something new, and that's these curly braced parameter values here. So two for cluster, one for URL. And those allow you to customize the contents of the application of the deployment to a target cluster with template parameters. So if we wanted to deploy to a different Kubernetes server, or from a different path than Git, we can customize that within the template. So where do those parameters come from, you might ask? Well, they come from generators. Generators are responsible for producing the parameters that templates consume. In this case, I'm using the list generator. It's one of six generators that are currently within the application set controller, and the list generator is very simple. Within the YAML, you specify a set of values. In this case, we've got a value called cluster, value called URL, they're both strings, and those are being inserted into the corresponding values within the template. So this means take these values, put it in the template, and then that causes this Git repository to be deployed to this cluster. So far, so good. So that's all courtesy of the list generator. The list generator is definitely the simplest generator. It's literally just a set of simple keys and values. So everything that you want the template to do is going to be harder coded upfront into the applications and resource. It's the least reactive, but you can already start to see some of the power here. So in the last slide you saw we had a single cluster. Now we've got two. So just by adding these two extra lines of YAML, it now allows us to deploy our guest book application to a second cluster. So the first one was, let's say, an engineering team cluster for development. Now just by adding these couple of lines, we can deploy to the engineering teams production cluster. These two sets of values will be inserted into the template, one template for each set of values, and that will cause Argos CD to deploy to each of those different clusters. So you can already start to see the power of generators, even just the simple list generator example. So what are generators? Generators are responsible for generating parameters which are then rendered into the template, into the fields of the applications and resource. Generators are usually named for and based on the particular external data sources that they're generating template parameters for. So for example, the list generator is just a literal list of values you define within the applications of resource itself. The cluster generator is generating values based on the clusters that are defined in Argos CD, and I'll show you that in a moment. And the git generator is using files or directories from a git repository in order to generate those template parameters. So looking at the diagram, you can see the general pattern that generators follow. So the list generator produces a set of parameters. Those parameters. So in this case I've got a cluster field and a URL field and a couple of rows corresponding to each cluster I want to deploy to. So those sets of parameters flow into the template. These are replaced based on the keys from the parameters as we saw. And for each row, you get a corresponding application. So for the engineering dev cluster, that's going to be one Argos CD application called engineering dev guest book deploying to the dev cluster. One application called engineering prod guest book deploying to the prod cluster, again corresponding to each of these rows. It's up to Argos CD from there to actually look at these application resources and do the the deployments to Kubernetes from the git repositories that are included in those individual applications. So that's the applications that data flow in a nutshell generators produce parameters templates consume parameters. And then the application set controller turns those rendered templates into Argos CD applications. And from there it's the responsibility of Argos CD to actually deploy those to the cluster to look at the gate repository compare it with what's in Kubernetes and keep them in sync. So that's enough talking for the time being. Let me show you a demo of all that. I'm going to jump over to my Argos CD installation here, and you'll notice that I have no applications defined at the moment. And I'm going to bring up my application set resource. This is nearly exactly the same as the one I just showed you. So it's a list generator with a couple of values one for engineering dev one for engineering prod. So we have this book and we're doing it from the application set repo customizing the server field the namespace field and the name. So I can apply this example to our Argos CD namespace, and you're going to see that the application set controller is going to pick that up and automatically generate Argos CD applications for each of those one for each cluster that we want to deploy to. Next we had the prod cluster, the dev cluster. And here you see the application set has read the application set CR and generated the corresponding Argos CD applications. So, because I have am using manual sync, I can go in and click sync. And Argos CD is going to do the actual deployment. So this Argos CD is automated sync to automatically have it deploy applications as soon as they're created by the application set. So just as we saw with Shama's demo, we've got our service and deployment created the replica sets going it's creating pods, and our CD is doing its thing, and keeping the Kubernetes cluster namespace in sync with the repository that we have to find for it. As I said, you can customize applications based on the contents of the application set template. So I can go into our list. And, for example, just make a simple change. Let's say I want to rename our applications by adding demo in front of them, for example. Normally what I would have to do if I want to do this within the web UI is I would have to like go into app details, like edit, make the change that I want, save that, and so on. And I would have to do that for every single application that's defined here. I can just edit this single resource, the single application set, adjusting the template, saving that, applying it, and it will automatically take effect for all of the applications that we have to find. And you see it now says demo for each of them, so prod and guestbook. And this is our old one. It will be deleted as soon as it finishes syncing in order to stay in sync with the new template. So just like that. And there it goes. So just like that, you can see how I'm able to make changes to the applications that resource, and it will automatically affect all of the child applications. So I can put this back, and I can do something like say change the namespace that they're deploying to, so let's say three, four, apply that, and nice and fast, guestbook three, guestbook four. Finally, we can just delete the parent application set, and it will automatically delete all the child applications. So let's sum up everything that we just saw. So when you create an application set resource, the application set controller automatically detects that. If you have generators specified that are based on external sources like cluster resources, like our CD clusters like it, the application set controller automatically detects those inserts them into the template of the application set. And then the application set template is rendered in target CD applications. So as we saw, in that particular example, the list generator produce two applications, as soon as the application set was created. As you modify the application set resource, it automatically updates all the child applications. And when you delete the application set resource, it deletes all of the applications. So the application set and its child applications are always kept in sync. So that way, all you need to worry about is what's in the contents of the application set. You don't need to worry about the contents of the individual applications ever deviating from what you have defined within the template. The actual deployment of applications remains the responsibility of our go CD. So as we saw in the Shama's demo, our go CD will see your applications. And as soon as you click sync, we're using automated sync. Our go CD will automatically start syncing them, but it's still our go CDs responsibility to actually deploy those application sets. The only response that those applications. The only responsibility of the application set controller is to create these the actual connection to the cluster and say identifying deployments and services and doing all that comparison and all that good get off stuff. That still remains the domain of our go CD. So as I mentioned, the power of templates comes from the generators that feed into them. There are many different generators for many different use cases. We already saw the list generator where you had to hard code a set of values in order to insert them into the template, but they're much more powerful generators, starting with the cluster generator. You'll recall with the list generator, we actually had to hard code the particular clusters that we wanted to deploy to. But what if you could automatically detect the clusters that are available. So within our go CD, within the settings. There is this clusters section where you can see all of the clusters that are go CD has access to. In my case, it's very simple. I'm just running on my laptop. So there's the one, but if you're doing multi cluster deployments, you can have like, I don't buy 1050, you know, as many as you want with our go CD managing all of those. So what if rather than needing to hard code them as we did with the list generator, we could just pull all of those clusters that were already defined within our go CD. And that's exactly what the cluster generator does. It looks at the clusters defined within our go CD. And all you need to do is just specify this simple one line of YAML, and it will automatically detect those and start deploying your applications to them from get ops. So pulling the name and server values from our go CD and deploying your applications. Now let's say you've got prod clusters and dev clusters, maybe you only want to deploy to developments, you can target a subset of clusters very easily using cluster annotations and see the docs for that. And as new clusters are added or removed, the application set controller automatically detects that and updates the corresponding applications. Next we've got the get directory generator. Many folks follow the get ops workflow of having a set of Kubernetes resources for their applications defined within a directory. This is known as the mono repo approach. So let me show you what that looks like real quick. So within this cluster add ons directory. I've got a couple of applications I want to deploy one called Argo workflows, and one called Prometheus operator. Argo workflows is the Argo projects workflow controller for orchestration. Prometheus operator is for application monitoring workflows is customized and Prometheus operator is helm. Wouldn't it be great if we could just scan this get folder for all the applications contained within it and automatically deploy the applications that are found here. So, you know, not surprisingly, that is exactly what the get directory generator will do. It will scan through the path you have defined for applications contained within that get repository within the path and deploy them to the cluster. And again that's great for folks that have multiple workloads or components or microservices. And they have Kubernetes resources defined for them within a single get repository. This will find all of them and deploy all of them all from the single application set. Some folks prefer the opposite approach when they're doing get ops. So rather than having one single get repository containing the resources for multiple applications. Instead they have a one to one relationship between application and repository. So one application and it's Kubernetes resources go in one repository one to one. So for them we have the SCM provider generator, which will scan a GitHub or get lab organization. In this case I'm using GitHub, and it will look for all the repositories within the organization that match particular name. So in this case example apps. So if I had like example apps one example apps two example apps three, the SCM provider will scan through my organization automatically find all those applications that match that path, or that name and deploy them to the Kubernetes cluster, using the template values that are generated by the SCM provider. So this is great for folks that prefer to do get ops by having one get repository per application. And this allows them again to manage that all the deployment of those all from a single application set. The third get base generator is the get file generator. Real quick, this allows you to define Jason or YAML files within a get repository and have them automatically processed and parsed and inserted into the application template by the application set. This allows you to have more fine grain control over individual fields of the template versus other generators. For the last two generators. We really start getting into the kind of meta generators, especially with the matrix generator, the matrix generator combines the output of two generators, allowing you to gain the advantages of both. So what does that mean. So let's say that, you know, you like the idea of using the get directory generator to find every application within a path of your repository. And you like the idea of using the cluster generator to deploy to every cluster that's defined within our CD. But what if you want to do both. What if you want to find every application and deploy it to every cluster. So, before you couldn't before you could just do either one or the other, you either, you need to pick one of those two generators so either use the get generator to find all the applications, or use the cluster generator to deploy to all the clusters. Fortunately, with the matrix generator, you no longer have to choose, you can now have both of them together. So we've got the get generator here, the cluster generator here, the get generators going to run. It's going to find all of your applications under the path you specify cluster generators going to run find all your clusters. And then the parent matrix generator is going to combine the two. So let's say you have five applications, 10 clusters, it's going to generate all 50 so five times 1050 individual combinations, and then automatically deploy them. So with this matrix generator using these two child generators, you can deploy every application and get to every argue CD cluster. So you can really start to see the power of the matrix generator here. It'll work with any two generators. So just as another example, if you're following the organization with many repositories get ops approach, you could have it stand through, you can have the SM providers scan through your organization to find all the repositories. You could have the get file generator with Jason files to finding every environment that you want to deploy to and automatically combine them and then deploy them to both in the exact same way. So the possibilities really are endless. Finally, we've got the cluster decision resource generator, which is definitely the most complex generator, but probably also the most powerful. It allows you to outsource the logic of which clusters to place applications on to an external custom resource.  But once you do that, it allows you to have those custom rules to find exactly where your applications are deployed to on which clusters and the applications that control are using this generator will automatically pick those up. So that is a lot of work. Fortunately, the open cluster management folks already have an example of this with their placement rule rule custom resource where they have their own set of fancy complex rules for where applications should go. And it's easy to hook up that placement rule custom resource into application sets using this generator, but this generator will work with any custom resource that meets the particular generic API contract that the generator is looking for. So hopefully all these generators have piqued your interest in how you can use them to handle your own individual Kubernetes deployment use cases. At Red Hat, application sets are built into our Red Hat OpenShift GitOps product, along with Argo CD, so the two are bundled together for use on OpenShift, and you can learn more about how to deploy using Argo CD application sets and OpenShift GitOps in these two blog posts. This is an introduction to application sets in OpenShift GitOps by myself and Dhawan Ahmed, and getting started with application sets by Christian Hernandez. And over to you. Yeah, so yeah, as Jonathan mentioned that with application sets you can automatically deploy a Kubernetes resource to multiple clusters from Git repository and manage all those relationships as a single application sets here. So leveraging that at Intuit, we adopted application sets to automate the add-ons installation customized across multiple Kubernetes clusters, one such add-on being Argo Rollouts. So Argo Rollouts is a rollout controller of Argo projects, and it is a progressive delivery controller that provides advanced update policies such as blue, green and canary for Kubernetes clusters. So at Intuit, each cluster is allocated an Argo Rollout instance, which in turn maps to an Argo CD application. And in order to automate the Rollouts installation, we selected the Git file generator of application sets, as Jonathan mentioned before. Hence, whenever an Argo Rollout is needed to be installed on a new cluster, a folder containing the respective configuration file is added to the Git repository as you can see on the screen. And the application set controller then detects the newly added configuration file and creates an Argo CD application which in turns installs the Argo Rollout instance on that new cluster. So yeah, that's all about application sets. To find out more about application sets, please visit our GitHub page or read our blog or connect with us on Slack. Thank you everyone for watching and have a great time. Thanks everyone.


This study found that the Cybex norm registered by HUMAC and the C-station by physiometer are reliable and valid tools for measuring plantar flexor strength. Both devices showed strong correlations between measurements taken with each other, indicating that they can be used interchangeably. Additionally, the C-station had high inter-rater reliability, meaning that it could be used repeatedly without significant variation in measurement. These findings suggest that the C-station may be a viable alternative to the more expensive and cumbersome IKD, making it easier and faster to measure plantar flexor strength. This article was authored by Seth O'Neill, Alice Weeks, Jens for example Norgaard, and others.


I guess we can start. So welcome to my talk about GPU pass through this beehive. Before I start with my presentation, let me introduce myself. So I'm Corvin Kröner, I'm a software developer working for back-of-automation and I'm focusing on x86 technologies and hypervisor technologies. And I'm a through with this source computer since 2022. Yeah, and at back-of-we're doing industrial automation. So we have a product line ranging from industrial PCs of our IOT terminals and also motor drives and yeah, all those is controlled by our control software. And yeah, that's the motivation because we like to use GPU pass through with beehive because we are using a Windows system for more than 25 years. And yeah, our control system which is software which is called Twinkat is integrated into Windows and runs on Windows. And but now we want to run our control system in FreeBSD but our customers are used to Windows and want a Windows interface and want to run their Windows tools and Windows. So we like to have a machine which has this user interface, a Windows virtual machine and yeah, underneath of it, FreeBSD should run with our controls software. Okay, so yeah, for my talk, I would like to start with a live demonstration then shortly discuss the current state and then discuss how to use GPU pass through that so that everyone else can test it. And at the end, a short question on some more. Okay, so let's start with a live demonstration. So therefore I have a system. Okay, so I have a system which is connected to KVM switch. And yeah, I've also connected to shell over SSH to control the system. So in the first step, which we can look up is that this system is running FreeBSD 14. So it's as it's currently not released, it's an alpha version of FreeBSD. But as you will see, yeah, the GPU pass who still works on FreeBSD 14. It's not fully working on FreeBSD 14 because one thing you have to do is update the firmware of Behive. So the firmware is in the ports tree under this utils edk2. And there I have a public branch on our Git repository. Yeah, which is at GitHub dash back off dash FreeBSD ports. And there I have a branch which updates edk2 port to enable fully enable GPU pass through. Okay, so first of all, before we can start GPU pass through, it's like any other PCI pass through. We have to detach. If you look at our PCI devices, we have a slot to the VGA PCI device, so our graphics card. So in the first step, like if you're doing normal PCI pass through, you have to detach the driver. So I'm detaching it from the USB device and the graphics device. And as a next step, you have to set the PPP driver which is required for the PCI pass through. So I'm doing it now for both devices. And then I've prepared a small script to run the GPU pass through. And yeah, the most important line is here where you add the graphic card, which is that slot two to the behave command line. There are some options which are required, but I will talk about them later. So and if you now start the VM, you can see at the KVM switch that, yeah, first of all, the UFI starts up. And yeah, after some time, so it's a Windows virtual machine. So we have to write a moment until it boots. Okay, so one moment. Okay, so here we are, here's the Windows desktop. And now to check that we're really running on a graphics card, we can, for example, start a graphical load. So I've picked up the firmware benchmark. And if we start the stress test, so yeah, I'm not sure if it's large enough, but you should see here that our frames per second are around 200. And yeah, it wouldn't be possible if we have an emulated graphics card. So as you can see, this should work. Okay, so yeah, so, oh. So in preparation of my talk, I did a screenshot of this. And as you can see here, we have 200 frames per second. So the GPU pass-through is working. Okay, so let's start with what is supported on previous defaulting. And so the short answer is on porting, we are supporting AMD graphic cards and Intel graphic cards, but not Nvidia graphic cards. But yeah, there are some issues. So for example, on the Nvidia graphic card, many of the cards have a hardware issue that they can't reset the card properly. And yeah, this means that if you start up your virtual machine, it works once, but if you reboot your UVM, it won't work properly. And on Intel, as I already said, you have to update the EDK2 port, but there's already some patches online and I'm working on merging this too. And a bit longer answer is if you're looking at AMD and Intel's integrated and dedicated graphic cards. So I've tested on the Ryzen V1000 system if GPU pass-through is working on integrated AMD cards, but it seems to fail. And it even fails on QEMU, so it looks like other hypervisors have the same issues too. And for the Intel dedicated graphic cards, I don't know which the current state is because I've never tested it and never heard that somebody tries to do it. I do not even know if other hypervisors support it, so we have to see what's the current state there is. Okay, so let's start with our how-to. So if one of you would like to use GPU pass-through, and the good answer is it's basically the same as GPU pass-through, that's really nice because I think most of the behind users have already used PCI pass-through and know how it works. But yeah, it's mostly like PCI pass-through. So it's not always that easy. Yeah, because the problem is it has often some constraints which you have to take care of. And if you do that, yeah, it's like any other PCI pass-through. Okay, so on those constraints, we try a ROM file, a grob driver, a female config interface, and a proper behave configuration. And yeah, to understand this more, I will now continue to explain what your ROM, grob, and female config is. So let's start with ROM. So it's a driver with a chip on the hardware itself. So if you have a graphics card, there's a small flash chip on it, and if you plug in the graphic card in your system, the UFI reads these flash chip and executes the driver. And it's responsible to initialize your hardware because you can't build a UFI BIOS which is capable of initializing any kind of hardware, especially complex hardware like GPUs or some complex NICs. And it can also add a UFI runtime driver. For example, if you want to pick the boot from a NIC or if you want to have graphics output, you need the ROM driver. And that's the answer for what is a grob driver? It's basically the graphics driver for the UFI. So if you build a system, you first start with the UFI stage, then with the boot loader stage and then the OS stage. And to get graphical output in the early stages of the boot, you need the grob driver. Yeah, and then at some later stage, the OS driver continues to be utilizing the GPU. Yeah, and the grob driver is mostly included in the ROM of the graphics card. And so it's not really required for GPU password, but it's really useful because when you're installing an operating system, you mostly have the problem that the installer don't have a graphics driver. So you can't install an operating system without a grob driver. And for example, if you want to use the loader or grob menu on your virtual machine, you also have to use the grob driver. Okay. And last but not least, the FEMA config, it's basically just an interface which is used for host and guest communication. So it was developed by QEMU and it's used to pass some information from the hypervisor to the guest and it's mainly used by the firmware and that's why it's called firmware configuration. So for example, you can add pass ACPI tables to the guest or you can specify a boot order or make many much more things with it. Okay. So now that we know the basics, let me start with the AMD GPU password which is a bit simpler and easier than an Intel GPU password. So the first step, you have to extract the ROM from your graphics card and you're on Linux, you can use the ZSFS to extract this. On Windows there's a GPU Z tool and unfortunately on previously it's not supported yet or I'm not aware of any method for that. So yeah, if you have a graphics card, somehow extract it from another system and then you can continue. And then in the next step, you have to call Beehive and yeah, so let's start with the basic Beehive command which just adds an image basically, it sets some flags and so on. So the first step, we have to generate ACPI tables because if the ACPI tables don't meet your hardware, yes, the GPU password won't work properly. So you have to add the minus A flag. Then you have to always boot with UFI. The reason for that is to make the ROM accessible for the operating system. It has to be executed by the UFI firmware. So we have to use the boot ROM. And there you have also to specify the firmware config option because otherwise the firmware config option is mainly useful to pass ACPI tables probably to the guest. And yeah, as the last step, you can just, you just have to add your password device just as you normally do. And there, don't forget to add the ROM. So there's a ROM option. At the path to the ROM you have extracted in the first step and that's it. Okay, let's go on to Intel. The first step is the same, extract the ROM, but on Intel it's a bit difficult because sometimes it's not possible and there's also the Akron hypervisor which is a hypervisor utilized by Intel and there they say you should ask your mainboard vendor or Intel to receive a drop driver. Yeah, so you have to try if this works, maybe you're lucky, maybe not, I don't know, but it's not required that you have the ROM file but it's really helpful. Yeah, this is good news. Okay, so it's the next step. You also have to call B-Hive and you also start with a basic B-Hive command. Yeah, you also have to generate ACPI tables. You also have to use a UFI ROM. You also have to use FEMA config. You also had to add the password device and now it starts to get a bit more complicated because on Intel systems, the GPU is always connected to slot two and some drivers are expecting that the device is connected to slot two. So it's really required that you assign the device to slot two, otherwise some drivers refuse to work. Okay, yeah, then you can also add the ROM file similar to AMD and also it's also required by Intel GPUs is an LPC device which mostly is always added to B-Hive, but yeah. And here it's also important to have it at slot 31 because otherwise some driver, yeah, just fail. And what is also important is that the LPC bridge has the same PCI-IDs like your host system because some driver are checking the PCI-IDs of the LPC device to identify your platform you're running on and if they see all this platform is not supported, they refuse to work. So yeah, therefore we have those config options in B-Hive to match the host PCI-IDs. Okay, and then that's it. So thanks for your attention, Michael. What is the state of OVMF support at the firmware as well? Yes, so I've already posted some patches on Fabricator to solve these issues with Intel GPU passthrough. As I showed in my live demonstration, I have a public branch where you can just fetch the port street from and build your EDK too. So yeah, hopefully we can soon merge this and then it will be available on 14. Yes? So the only issue you've added to NVIDIA is the reset problem? What are the issues with NVIDIA? So what I found out on NVIDIA is that you have to add some quirks to B-Hive. So if you're looking in the QEMU source code, they are doing the same. Because for example, the config space of the PCI device is mirrored into the MMIO space. So you have to trap these special MMIO space for NVIDIA. I, yeah, but I've tested it a bit, but not much. And I'm not sure what is failing currently. And it could be that legacy PCI interrupts are not supported by B-Hive yet. And because, yeah, it's required by some graphic cards. So it requires some further investigation. No. Sorry? Oh, repeat. So the question was if we can achieve these options with the VMB Hive package. And I don't think so. Because, yeah, you need really fine grain options for that. I'm not quite sure if VMB Hive already supports the optional format. Yeah, so it would require some patches to VMB Hive. Yes? Are you working on some kind of, like, you know, I was thinking about the virtual VR, for instance, like having the VM use the graphics malware for like, the event, like, yes, how is this related? Is it related in the north? And are you, because I know there is some work about virtual VR, is it something that, I don't know if this is related to that or... So, yes, this part completely passes the, this part completely passes the GPU to the guest. So the guest had full access to the device and there are some technologies, I think, like SRIOV to pass, to split a device and assign it to multiple users. But this is for one guest only? Yes, this is for one guest only. And SRIOV would be for multiple guests, for example. And I think, I'm not sure about the current state on Behive, if Behive supports this. But, yeah, if it supports SRIOV for other devices like NICS or something like that, it should support it for GPUs too, yeah. But, yeah. And besides that, for Intel, there's two technologies which are called GVTG and GVTS to somehow share the graphics card, but I'm not currently working on that. Okay, so any other questions? Then, thank you for your attention.


The enemy has a vote, he's smart, he's training, he's coming for us, so we need to prepare like he is. This training adds to the overall mentality of the Marine Corps because it's investing time and resources into those small unit leaders of sergeants, it's corporals, using the resources of sesame gear, the jungle, combat town, putting them in an environment where they're uncomfortable, training they haven't received, we're putting them in a kinetic environment that forces them to lead, to step up, to think about all the contingencies that go into combat, and I think that will produce the dividends of what we want to see in the next generational leaders. We put 55 Marines through a grinder of decisions, uncertainty, chaos, the fog of war with evident constancy, psychological and physical and emotional, and I think that is setting us up probably more so for a peer on peer threat. What makes a Marine Corps fire team lethal is that confident, trained, aggressive corporal, you've got three other Marines, you give them a task and a purpose, you give him the ability to achieve the end state, and I think he'll surprise you, he's properly trained, he's been prepared for combat, you put the decision making in their court, and I think you'll see what the NCO Corps is really made of.