I'm just testing my mic. Can everybody hear me? I guess so. So welcome, folks. Thanks for taking the time to attend this presentation. I'm going to be talking about Windows in an OpenShift world. It's about enabling Windows workloads in OpenShift. All right, let me see how this works. Oh, awesome. So before we start to do this deep, diverse type presentation, I'd like to show off hands on how many people have used a Linux container. It's almost 100%. Now, how many people have used a Windows container? I am quite surprised. But it's Antonio, so I'm sure. Yeah, so in many ways, Windows containers are sort of new to the game, even though Microsoft has been working on this for a good five to six years. So let's start with a quick history on Windows containers. There are some differences. When you compare them with Linux containers, there are a couple of ways you could run a Windows container. And that differentiation is based on isolation modes. The first mode that people typically think about when you think about containers is process isolation. Process isolation means that you have namespaces, and then you use those namespaces to sort of protect containers from each other for multi-tenancy purposes, et cetera, et cetera. However, Windows has no construct called namespaces. So because of that, they came out with this silver construct, which they call a silo. It's very close to namespaces, but doesn't work exactly in the way a namespace works. So that took care of namespacing, and what do you do about controlling how much resource a particular container is using? To handle this scenario, what they did is they came out or they didn't even come out. They have a construct called JobObjects, and a JobObject can be used to control a set of Windows processes.  The other thing with process isolation is, of course, you all know with containers, the kernel is shared. So when the container is running, it is actually sharing the kernel that's on the host. However, a key difference here is that Microsoft claims of giving you no security guarantees for using containers with process isolation and a multi-tenant environment. So to get over this, there's also an added restriction in this whole world. The fact is that your host and container kernel, they have to match. So if you created your container on Windows Server 2019, you can only run it on Windows Server 2019 if you're using process containers. So server is 2016, you have to run it on server 2016. So to get over these sort of restrictions that you have, what Microsoft did is they said, oh, well, we'll help you and we'll give you security guarantees if you use hyper-reisolation. So what is hyper-reisolation? Hyper-reisolation takes virtualization and they basically run the VM in a very highly optimized virtual machine. So what do you mean by a very highly optimized virtual machine? So what they're actually doing is instead of booting the virtual machine from scratch, which is you start from real mode, you go to protected mode, 32-bit, then you transition into long mode and then there's a whole slew of drivers that you have to do with keyboard, video, mouse and all the other fun stuff. They said, oh, let's try something different. Let's now boot the kernel at a spot where it's already running in long mode, at a 64-bit mode. So that saves a bunch of time and then what they did is they paired down the device model so you don't have all the devices you would have in a particular VM in this little, highly optimized virtual machine that they're running. So that's a huge advantage because what we're used to with containers is if I do Podman and just run a container, it comes up immediately. So they were trying to get as close as possible they could with that even though they were using virtualization, so that's the reason for all this optimization. And with that, you have another advantage. Your host kernel and your container kernel need a match because hey, you have a VM here, you can just run everything within that VM so a mismatch can happen, but there's still some restrictions. Say you have a container that was built for Server 2019. You can't take that and run it on a Windows Server 2016 host. Microsoft says, no, we're not gonna love that. So that's some restrictions that folks are going to have to live with. So I've been saying container kernel images and you must be all wondering like, what is going on here? Why am I even talking about a kernel when it comes to a container image? To explain that, let's first start with what Microsoft decided to do to help their application developers containerize their software.  So everything that you require to run your application is not provided by the kernel like it happens in the monolithic Linux kernel. The Windows kernel is split into pieces. You have the kernel DLL that runs in the most privileged mode and then a lot of the other services are actually provided by user mode DLLs. So they said, we'll give you these base images which is the building block for all your containers. So you have to start from one of these base images. So how do you pick one of these base images? Basically, really depends on your application. Say you want to write a, you have a traditional .NET application.  Say you have a more complex application that requires the full set of Windows APIs that's out there. Then you would pick the Windows version of this container image. And they also have something called IoT Core. If you're writing an IoT app, you typically use this. The reason for all this is actually, so I was mentioning these DLLs that run in user mode. There's a very tight coupling between those libraries and the kernel. And to make things worse, that ABI is not public and it's not stable. So there's a lot of reverse engineering that happens to discover those ABI's. But if you take advantage of it and you decide to use it, there is no, Microsoft says we give you no guarantees that we'll honor those ABI's because you're not supposed to use it. So because of this sort of tight coupling between those user mode libraries and kernel, you have to be careful about what image that you pick when you're containerizing your application. The other advantage is the other reason for having a kernel is because of hyper-reisolation, you're running in a VM. If you want to run in a VM, you have to boot from a kernel. So end result, your container image not only contains your application and any libraries that it needs, it also needs the kernel in there. So it comes sort of with the game if you're running Windows applications. Now that I have the kernel in there, these user mode libraries that my application depends on, you can take the most basic Windows image out there and you're gonna find that it has a pretty large disc footprint and this can actually cause problems in the Kubernetes space because what happens is you have the Kubelet that runs inside a Windows node and you try to launch an application that's bringing down a 10 GB container image. The Kubelet after a while will say, hey, you know what? I've been waiting for too long. I think there's something going wrong. I'm gonna stop trying. So you have to do something smart. You either increase the timeout that the Kubelet will wait in these scenarios or you have to preheat or pre-download those container images upfront. The other quirk with Windows container images is that it doesn't support the latest tag. So you have to be specific. If you wanna use a Windows Server core, you have to specify a tag that says, I don't know. I think the tag is LTSC 2019 which indicates that this is a 2019 instance of Windows Server core and they have similar ones for if you wanna run Windows Server 2016 containers. So yeah, that's another difference that you would have to get used to in this world. All right, so now let's get on to a more targeted talk about what we are doing with Windows workloads in OpenShift. The question many people ask me is like, why are you doing this? Like we are Red Hat, like why do we care about Windows and what's this whole relationship with Microsoft? Well, the answer is customers. Customers have basically come and told us that, hey, we have large Windows applications, large .NET applications that we would love to run in OpenShift because we like what you did with our Linux applications. We were able to manage and orchestrate all of that using OpenShift and we love it. We wanna do the same for our Windows applications. So that's why the Windows container team was formed with an OpenShift and we have started going down this path of saying, hey, what's the best way of enabling Windows workloads with an OpenShift? So our initial target was, we wanted to start very simple. We wanted to hit just the basics, just enough so that we can say that, hey, we are getting our feet wet in this whole new sort of environment. So the first thing we wanna do is you take a Windows Server instance that's running somewhere and then you'd say, hey, I wanna add this to my OpenShift cluster. This is sort of our model that we call bring your own host model, which means that the customer is basically in charge of creating this Windows VM. They'll create the Windows VM, they'll install the container on time in it, they'll typically attach it to the cluster VPC or the cluster network, and then we can add it to the cluster. And to start things off, we decided to only support process isolation and that's because if you want to do hyper-V isolation, it means you need virtualization support and in many clouds, that means you need to turn on nested virtualization. So we do not wanna go down that path yet and so to kick things off, we are only going to deal with process isolation in the short term. And then of course, once you have the node attached to a OpenShift cluster, we wanna of course deploy a workload to the Windows Worker node. And the way we do this is how do you differentiate between nodes, right? You have Linux workers, you have Windows workers. What we do is when we bring up the node, we apply a taint to that node and then we tell people who wanna run a part on that node or a deployment, you say in the pod space, you say that you can tolerate this taint so that you get targeted scheduling happening on those Windows nodes only. So you don't have this confusion of a Linux workload trying to land on the Windows node and dying. The other key difference here is you cannot use the default OpenShift SDN network type if you wanna run Windows workloads in that environment because what they do in that case is they are actually, we are actually carving out a piece of the network for Windows, what do you call, communication. And this was done not just by Red Hat, we have a partnership with Microsoft. So the OpenShift SDN team worked closely with Microsoft to get all these pieces in place. And where the Windows container team comes here is we don't really develop the networking side, we just glue those pieces together. And then of course you wanna be able to route traffic between pods. This is both Linux and Windows pods and Windows and Windows pods. And then of course route traffic to application users who actually wanna use the service that's running inside the Windows node. So this is our architecture. And since we wanted to start really simple, what we decided to do was we'll write an Ansible Playbook that is going to initialize the preparation of this Windows VM. And the way it does it is there's a bunch of binaries that is needed for this process. It's going to go and pull those binaries from a release location and then copy those binaries to the Windows VM and then launch it in whatever manner that is required. One of these binaries that gets copied over is what we call the Windows machine config bootstrapper. It's a one-time use binary in the sense that it's not a demon or something that runs in the background.  And what this model allows us to do is you can even look at this and say, okay, I can now just take that Playbook and convert it into an operator if I want to in the future. But for the sake of simplicity and because there are so many unknowns in this space, we decided to take this approach. So let's now talk a little bit about some of those pieces. So one of these pieces is the Windows machine config bootstrapper. Now this is the binary that we're gonna copy onto the Windows node by the Playbook and then it's going to do some stuff. So what does it do? It's going to take the worker ignition. Worker ignition is, you can think of it as a packaged file that contains some information that's pertinent to that cluster. And so some of the things we extract out of that worker ignition package is the Kubelet configuration. I do want to call out that we're just doing this for the time being, moving forward, we're gonna construct our own Kubelet configuration which is very specific to Windows. We also, what we get from the worker ignition is we get kube configs. We actually get a couple of them. We get what's called a bootstrap kube config to bootstrap the node and then we get the main cluster kube config because there are some dependencies on those kube configs by some other pieces of the software that we have inside that node. And of course we need search, so we pull them all out. And then what the WMCB does is it configures and runs the Kubelet as a Windows service. This is sort of the first step in saying, hey, you know, I'm now joining the cluster and you can start deploying applications to me.  And if you remember in the last few slides I had said, we apply those taints because it's the taint that's going to tell us, okay, this is a Windows node. So when you write your part spec or your deployment spec, you're gonna say, hey, you should be able to tolerate this stain and that will cost your application to land on the right node. The other piece of work that the Windows machine configurator does is configure the container-native interface, the CNI, and that's required for networking. How all of this is tied together is by the Windows scale-up playbook. Like I had mentioned before, it downloads all the required binaries from the release location, the WMCB, and the way the Ansible playbook is written is a part of it will execute on the Ansible host, mainly to download some of these binaries. But there's also part of it that will actually run on the Windows VM. And things like getting the worker ignition, you can't do that from outside the cluster, you have to do it from something within the cluster. So we grab the worker ignition file on the Windows node itself. We also pull the cubelet from an upstream location and copy that over. And then we also copy this binary called the hybrid overlay. The hybrid overlay is what configures the plumbing that's needed for networking to work for Windows within that cluster. This is a piece of software that was actually written by someone from Microsoft. So shout out to Chaslin, who helped us a lot here. And this was done in conjunction with folks from our OpenShift SDN team. Shout out to Dan Williams and Jacob for helping out with this. And the hybrid overlay, once it runs, it means that the basic etchiness, which is the host networking services pieces in Windows, is ready. You have your OpenShift networks created within that Windows node. And at that moment, what happens is, we can go to the next step of asking the cubelet to be configured to work with CNI, which I'm gonna call right after this. The other binary that we download is the cube proxy. I'll talk about this in the next slide. And of course, when I was talking about network configuration, there's the CNI package that needs to be done. So it collects all these files and then it copies them over to the Windows node. Once the files are copied, it starts to remotely execute these files. It first says, okay, hey, WMCB, go configure the cubelet. It then launches the hybrid overlay. Hybrid overlay talks to etchiness, does all the network plumbing that's required. As I said, the OpenShift etchiness networks are created at this point. And now we have a second step with the WMCB, the Windows Machine Config Bootstrapper, which then runs to configure CNI, which is a container native interface. For this, we sort of relaunch the cubelet with added options for configuring CNI plugins. And then the cube proxy is run. Cube proxy exists on the Linux side too, but it's sort of an optional component. But with Linux, while it's optional, with Windows, it's required. If you do not run cube proxy on your Windows node, you will not be able to allow an application to talk to the outside world. So say you create a Kubernetes service that's running behind a load balancer.  The other thing that happens when the WMCB is configuring the cubelet in the first step is the node will start to generate certs, CS call certificate signing requests, and somebody needs to sign those requests and approve them so that the node can then, the cluster can say, okay, you are a valid node and you can join the cluster. So the scale up playbook also takes care of approving those certs. All right, so now comes the sort of the exciting and a little scary part of the talk for me is the demo. I've been praying to the network gods, the demo gods. They have not been happy with me today morning. I was trying this out and the network here is a little flaky. I do have a backup video to show, but it's not that exciting. So I'm gonna give it a shot, fingers crossed and we'll see if it works. And I'm gonna sit for this. There's no way I can stand to do this, sorry folks. All right. So what I did is to make the network's life easier. I brought up a instance on AWS where I'm gonna run the playbook. The reason for doing this is the network is so slow I need to download these binaries, copy them over, and it was like most of the time I tried it with the local network it was timing out. So before we go down this path, let me show you something about the cluster. So the cluster that I'm running is a OpenShift latest and greatest 9thly version 4.4. You can see that the list of nodes that I have at the moment.  What else can I show you? I also wanna show you the host file that I'm gonna pass on to the Ansible playbook. This basically lists what the address of the Windows node is and the admin password. You're free to copy the admin password because I'm gonna kill this whole cluster the minute this talk is done. So we have all those pieces ready and the next step is going to be running this playbook. So let's go ahead and do that. So what I have here is I'm passing the host file to the main playbook, which is we've called it main.yaml, it has a list of tasks there. And one of the things we pass on to that is the location of the WMCB that we wanna use. There are two ways this playbook can run. One is you can either execute the playbook in such a way if you have Go configured, it can actually pull the Windows machine configure operator by repository and do the build for you and then run. But we don't want all customers to have to do this so you can even pass a release location that you have and it'll pull it down and use that binary instead. So let's kick off this run. And as you can see, one of the things it's actually doing now is it's downloading the binary. So this is gonna take, I don't know, hopefully five or 10 minutes. So what I'm gonna do in that meanwhile is keep going with my presentation and then we can sort of come back to this. All right, so how do I get back to my presentation? Here it is. I'm also gonna, I also should talk about what I'm trying to show in the demo, right? So I'm going to be running on OpenShift, a 4.4 cluster that I, like I showed you. It's configured with OVN hybrid networking. I already have a Windows Server 2019 instance running on AWS and that's the host file that I showed you. It had information about that particular instance and that instance was connected to the cluster's VPC. Now, in the background what we've been seeing is that instance is added to the cluster by running the WSU Playbook. Hopefully it'll succeed and if it succeeds, we'll get to play some Pac-Man. So it's a short and simple, hopefully short and simple demo. Hopefully it'll work. And if the demo doesn't work, I do have a link to a video that'll show you the whole thing but that's not so exciting. And while the demo is running, I do want to talk about what the team is thinking about next steps. We do want to move to an operator model. We do want to move away from just running the Ansible Playbook. We want to make it much easier for customers to gain access to running Windows workloads. We do not want them to go through the pain of creating a Windows VM, installing the required runtime in there. We want to actually take over most of those pieces and the easiest way to do that is actually move to the operator model. It will streamline that whole process for us. At the same time, what we have at the moment is just basic functionality as far as running Windows workloads goes as I described before. So in addition to that, OpenShift has amazing logging functionality. So we want to actually plug in into that so that Windows and Linux, they're on the same plane as far as OpenShift goes. The same goes for monitoring. We have a monitoring framework that's part of the cluster. We do want to plug into that so that no one is going to say that, oh, I'm not able to get certain features that's present for Linux but is not present for Windows. The other thing that we want to do is container native storage. We want to add support for that. And the way we want to do that is, Red Hat has a container native storage team that does take care of storage. They would do some of those pieces. We would be the clue and hopefully we'll get some traction from either Microsoft or somewhere upstream that will help with some of the bits in Windows that is needed here. So this is what the operator architecture is going to look like as we move forward. So we're gonna have what we're gonna most likely call Windows Machine Config operator. And what that operator is going to do is gonna watch for a couple of resources. One is we will create a Windows operator CRD. CRD is a custom resource definition. It's a way to extend the Kubernetes API for different applications. So we will create our own CRD, which we'll call Windows operator. And that's basically gonna describe most, at the moment we think it's gonna just describe the release binaries that need to be used, that you need to go and pull from external sources. Then what'll happen is, it's also gonna watch for machine objects. Machine objects is something very open ship specific. It's a object that describes worker nodes and master nodes. So we would add some features to that machine object that indicates that this is a Windows worker. And then the operator is actually gonna sit and watch for that particular object to see if that object has been created or not. The reason for doing that is, the minute that object gets created, we know a Windows VM has appeared on the cluster. And it's wanting to join the cluster. In addition to the object being created, it's also gonna give us a way to get the admin password of that Windows VM because it's not that easy to use keys to get to the Windows node. So we'll most likely do some further implementation to the machine API that's part of OpenShift that will give us access to the secret in Kubernetes terms in like a secret CR. And so we'll use that secret and then we're pretty much going to do exactly what we did using the Ansible Playbook. We're gonna copy the binaries over, run WMCB. So this is why I was saying that if you look at the original picture, you could have taken that Ansible Playbook and just dropped in this operator and everything should have still worked. So hopefully that's the model we are shooting for. I have actually rushed through my presentation. So let's go back to the demo and see if it's still working. Awesome, it did not work. So what would have actually happened is actually the place where it failed is it was trying to launch the hybrid overlay. And for some reason, there was an issue there. But what would have happened is I'll quickly run through what happens with the demo. So what I'm actually showing here is I'm basically showing the cluster version like I showed before. I'm gonna show you the list of nodes here. And actually what I did, I did miss showing you one of those things. This is the network configuration and you can see the type is OVN Kubernetes. And we also have to change what we call the network operator to make it a hybrid OVN Kubernetes version. And once this needs to be done up front before adding the Windows node, these are the nodes that are already existing. This is again like I mentioned before is the host file that you pass on to the Ansible Playbook. And again, we're gonna kick off and run the Ansible Playbook here. So I'll quickly go through what the Playbook is doing. Like I mentioned before, it downloads the WMCB from a release location. It then also picks up the Qubelet node binary and then from within the Qubelet node binary it takes the Qubelet.exe and downloads it locally. It also grabs the Qtroxy binary that is required and then it goes and fetches the CNI plugins. That's basically a tar.gc file which is then unzipped and will be moved over. And then what it does is it starts to copy these required files onto the Windows node. And that's the thing that takes a bunch of time so I can sort of speed that up a little bit. And here it's trying to get the Ignition file and once it gets the Ignition file it also needs to figure out what cube version the cluster is using so that it uses the right Qubelet binary and it pulls the right Qubelet binary. Once it finishes doing that it's going to pull the hybrid overlay make sure that the hybrid overlay is Shaw matches and then the first step is it's going to run the Bootstrapper. Once the first Bootstrapper operation is done by the WMCB it's going to generate a CSR. So what the playbook will do at that point is it's going to wait for some time for the CSR to show up in the cluster and the minute it finds that the CSR is present it's going to approve it. And then it's going to wait for the next step where once the Bootstrap CSR has been approved another CSR gets generated which is the Node CSR and now it's going to approve that too. And these are key factors for sort of authenticating that this Node is allowed to join the cluster. And once the CSRs have been applied the first part of Bootstrapping this Windows instance is done. And the rest of the work that happens beyond this point is mainly networking. So as you can see here the Bootstrap node has been applied the CSRs have been approved and now it starts to do the networking pieces. The first thing it does is it checks if the hybrid overlay is already running. The reason why it's doing this is it's ensuring that this playbook can be run over and over again against the same Windows instance without any issues. Once the hybrid overlay is up and running you can start doing the CNI configuration. The CNI configuration actually uses some of the annotations that the hybrid overlay once it finishes running applies to the Node object. So it takes some information from there creates the CNI configuration and then it reruns WMCB with different set of options and those set of options is what applies the CNI command line parameters to the KubeLit and then the KubeLit starts to run in its final state as a service and once that's done the next step is it has to configure the KubeProxy that I spoke about so that it allows external network communication to happen and then this KubeProxy what it does is once that's configured as a Windows service that's pretty much all the work that the WMCB has to do and you will see now that the Windows Node has been added and if you describe the Node you will see that it says the OS image being used is Windows Server 2019 and the KubeLit has started, the KubeProxy has started and at this point you can actually add Windows workloads onto this instance. So Windows workload which what I was hoping to show would look something like the following. So if you look at this YAML we have a service that is basically a regular Kubernetes service there's no difference here if you compare it to a Linux service. The key difference is going to be on the deployment side where you will describe things like, I'm looking for that. So as you can see here see the image that's being used is a Windows Server Core image and this is the tag I was talking about before you have to specify an LTSC 2019 sort of tag otherwise things won't work. This is the command that's going to be passed to this is going to be the command that's going to be passed to this Windows Server Core instance and the other thing that I was talking to you about was the toleration that it needs to be specified and so we say that oh the key that we're looking for is OS and the value is Windows so that's a toleration that maps to the taint that we have applied to the node and the same thing we also have a node selector here where we mention what OS type is done. So the end of this if we were able to launch this what do you call launch this deployment you would be able to access this Windows service from an external source. So that's pretty much all I have. I was hoping to show more if the cluster actually came up but unfortunately it did not not the cluster but the node. So let's go back to my slides. I can try. In fact I can just run it in the background while I can maybe open up the floor for questions. All right so while this is running so this is not GA. This is not even like available for tech preview at the moment or dev preview. This is something that the team is playing around with and working out. We might pass it on to certain solution architects to see if they can show this off to customers but we're not planning on releasing this anytime soon. More work needs to be done to stabilize it make it more feature perfect and make it on par with Linux workloads. So we're not ready to GA this. This is I don't know what we can call this very early alpha preview type things that we want to get a head start on this. So at this point I think the floor is open to questions. If anyone has. So the question was what about the licensing for the Windows? So if you remember very early on we had said this is going to be sort of a bring your own host model. So the licensing is sort of on the customer. If they have a Windows license they can use it or pay AWS or Azure for those licenses. And so we don't want to be taking ownership of paying for those licenses either. So it's mostly on the customer side as far as things like licensing goes. Sure. So that's the area that we have not even touched so far. At the moment, whatever default that the container runtime is using we're just going with it. But that's another piece we need to work with the OpenShift Storage team to figure out. And I did forget to mention something about container runtimes. At the moment we're using Docker because that's what the like you can find instance AMIs on Amazon provided by Microsoft that's having the Docker runtime. But the plan upstream is to go with container D. Crycontainer D as their runtime. And what Microsoft is saying is with in the whole Kubernetes space they're gonna go with Crycontainer D and they will only support Hyper-V isolation. So we do have to think about what we're gonna do at that point we might have to use some of the I3 instances on AWS that provides what is called what is that called nested virtualization. There are these instances called Nitro instances where Amazon uses their own version of the hypervisor to allow nested virtualization to happen in a more performant manner. So we do have to think about those things. Yeah, so this is a problem that you're seeing that it's looking for a CSR and not being able to find. So I think running the playbook the CSR has already been approved. So we actually hopefully ignore. I think I'm using a version of our playbook that's actually gonna ignore the CSR. Any more questions? Okay, so the question was what are you gonna do in an offline environment, right? Because we're downloading these libraries from release locations. What are we gonna do in that scenario? The answer is when we go to the operator model we're gonna have a source of truth within the cluster itself. Because we don't wanna keep using different binaries that haven't been certified and things like that. So most of the binaries that we require will come from within the cluster itself. So that way you don't have to reach outside and pull these things. They'll be all packaged as part of the operator. So when you run the operator you basically have it. All right, so the question was is this open shift specific or are we planning to make this part of upstream? So if the question is about the playbook and things like that, that's very open shift specific. We are only planning to use that against open shift clusters. It's very sort of tuned so that it makes it easy in an open shift way. But there is nothing preventing us from modifying that playbook to make it upstream friendly too. There's nothing, we're not doing anything funky or special there. This could easily be adopted for upstream use cases. The thing with upstream use cases is we need to figure out what the networking model is gonna be there. It could be anything, right? Here we're being a little prescriptive saying that we will only support a certain networking model that we're familiar with. So as we start supporting different networking models, that will become an easier thing for us to support in the upstream. Any more questions? The other thing I, let me quickly go back to my slides. So actually the piece that I was saying might be a problem is actually not a problem. I was using the latest and greatest of the playbook so that actually figures out that, okay, this is actually a node that we have already approved so it's sort of ignoring that whole process and it's gonna keep going. But I think the place it's gonna fail is the hybrid overlay which means there's some networking funkiness that is happening on the cluster which is what I was running into earlier in the morning. So I'm not so confident that that'll work, but we'll see. But I am trying to get back to my slides. I have a couple of more things that I could show in this time. Was there somebody, yeah, sure. Oh, that's a good question. I actually haven't thought about OKD but I would say that if you're able to do this against OCP, you should be able to do it against OKD. I can't think of anything that would prevent us from being able to use this with OKD. So it should be there. All right, let me see if I can get back to my slides. I have actually lost my slides. No, I'm still running Fedora. That's one thing to say that I'm working on Windows is another thing to say that I'm gonna run Windows on my laptop. I just wanted to show what I think are, yeah, so these are the couple of links that I've put out there. One is for our repo, which is the Windows machine config operator. We're calling it the Windows machine config operator. It'll soon be renamed to Windows machine config bootstrapper and we're gonna start afresh when you go down the operator model. SIG Windows is the special interest group in Kubernetes for upstream work. So if any of you folks are interested in contributing in this space, please join SIG Windows. They have a Slack channel on the Kubernetes Slack. There are also weekly meetings where people discuss about work that's happening in this area. So really useful. I highly suggest folks interested in Windows workloads to go and try this out. And of course I have to put in a little bit of a plug for Red Hat, right? We are hiring, we're hiring all over the world. In particular, the Windows container team is hiring in Boston. So if someone wants to work out of Boston, come and look me up after and I'm willing to take in your resume. So that was my plug. If I have a little bit more time, I had a couple of slides that I thought wouldn't make it. And also, yeah, so it failed at the same spot trying to configure the hybrid overlay. So I must have done something wrong maybe or there's some networking glitch happening at the moment. But sorry folks that I couldn't show you a live enough demo. So Rancher has come out with a release. So has AWS. It's all, again, just like us very early, they're not putting any support statements out there. So very early in the game. And that's why I think we're also getting involved as quickly as we can because we sort of want to get up to speed before the rest of the industry catches up. So the question was, is Microsoft Azure doing it? No, not yet at least. There might have plans to release it, but not yet. I was going to show some more slides but I think I'm slowly running out of time. So yeah, there's five minutes. I could maybe show those slides. They're more of historical interest and actually hidden them. I mean, this is just to show you the kind of evolution that's been happening in the Windows Server space. Microsoft started with Windows Server 2016 where they started to introduce process isolation, Hyper-V isolation. They had like a built-in docker for Windows Server. They slowly moved and in the next step, which was I think 1709, they came out with these container images for Nano Server and Server Core. They also had platform level support for Linux containers so you can theoretically run Linux containers on a Windows node, but it actually again happens way within a machine. That is something that we're not really looking at at least with an open shift. If someone wants to run a Linux workload, you just run it on a Linux worker node and they did the same by the time they got to Server 2019 is where they had more of the networking pieces in place, more of the container storage pieces in place. They also added support for some of the networking plugins like Calico and Flannel. So in fact, if you go look upstream, you would see that all the networking stuff is done through Flannel, but we decided to go the HyperDovion route. If you want technical details and more architecture about the networking stuff, you should ping the SDN team. So while they were doing this evolution on the Windows server side, the same sort of evolution was happening in the Kubernetes area too. I know this infograph kind of throws a lot of information out there. It's more to just show you the kind of stuff they were doing over a timeframe. So this stops I think at 113 and we're like at 118 or 119 now at this point. But so they've been working on this almost since the 1.8 or 1.6 timeframe. So Microsoft has been pretty hard at work in trying to enter this whole container and Kubernetes orchestration space for a while. All right, so that's pretty much all I had. I again apologize for not being able to show a live demo. I think I sort of asked for a little too much, trying to add an old live. But hopefully maybe it found a bug in our code so it's all good. All right, nothing else, thank you folks.


Abstract silicon is the most scalable optoelectronic material but has suffered from its inability to generate directly an efficiently classical or quantum light on chip. To address this issue, researchers have developed an all-silicon quantum light source based on a single atomic emissive center embedded in a silicon-based nanophotonic cavity. This device exhibits a more than 30-fold enhancement of luminescence, a near-unity atom-cavity coupling efficiency, and an eight-fold acceleration of the emission from the all-silicon quantum emissive center. The results open immediate avenues for large-scale integrated cavity quantum electrodynamics and quantum light matter interfaces with applications in quantum communication and networking, sensing, imaging, and computing. This article was authored by W. Regem, Y. Gnbyev, W. Karani, and others.


right, this needs to be mute, is we need to ret-reclaimed is zero. Reclaimed plus equals one, self.retired.count fetch sub reclaimed like so. This complains because we cannot borrow and dot next is mutable. Only one error left. Old retire. Right, so here the problem is we don't know how to retire this because really this has to be like, maybe this can just be Fox from RAW. Right, so the deleter here is a little awkward. Like how do you specify the deleter? It arguably doesn't even need to be a trait. The advantage of it being a trait is that it doesn't have to be a, it doesn't have to be just like a function pointer. We could associate state with the deleter two, which is valuable if someone wants to bring their own allocator, for example. And so the delete can't really just be a freestanding function. Right, so if we said it's gonna be this, then this is now gonna be a star mute deleter, din deleter, and then retire is gonna take a deleter of type D or reference to one, just a D. Maybe we're gonna require that the deleter is moved. Or that it's like static or something. Right, like, okay, the deleter we get in here, I guess we're claim is gonna be deleter as star mute, din deleter, which is why this has to be, yeah, so the question is what do we want retired to actually take? Like do we want someone to have to pass in an instance of a deleter? Or should we just say you just have to be generic over it and then it just has to be a sort of stateless closure? We could make this interface a little simpler, right, by saying we also have a retire with drop in place that just takes a pointer and all it's gonna do is call drop in place on the thing. But it is tempting to sort of allow the user to pass in a deleter, but the interface is a little bit weird here, because the deleter they give us has to be, it like has to be behind a pointer because otherwise we can't stick it in the list with other things that might use other deleters, right? So this is why we can't just stick the deleter here because then we'd have a D and retired is not generic over D. It could be a pointer, but then this has to be a pointer and if this is a pointer, then what is it even a pointer to? Like maybe it's a static, but if it's static, then why not just have it be a static method instead? Like why not just have it be a function pointer if it can't hold state anyway? There's like a, it could be a star mute din deleter that gets passed in here and it's up to the caller to make sure that this is like valid for long enough, but long enough is really poorly defined here, right? It's like the lifetime of the domain because you don't know when it's gonna be reclaimed. That might be, it might not be until the domain actually gets destroyed, which might be a super long time from here. So it's hard to assign a lifetime to this. You know, I think we're just gonna go with what we had here for now. I think this is just gonna be a function pointer.  Oh, actually no, I do wanna deleter trait just because it documents the interface better, but I don't want it to do this. And then the question now becomes what do I pass to retire? So one thing we certainly could do is have a, there are too many things in this file now, but we could have a pub struct drop in place. So maybe this is a pub mod deletors and impold deleter for drop in place, delete, mute and drop, which does the standard mem standard pointer, drop in place of pointer. And similarly we'll have a drop box which does a just box from raw of pointer. This is unsafe and this is unsafe and these are safe by the contract on retire. Which is has pointer object retire. And then we're gonna have to go back to retire and add here column is guaranteed that pointer is a valid reference, column is guaranteed that self is no longer accessible to reader and caller must guarantee that D is a valid deleter for self. Right, so if you're using the box deleter or drop as box or whatever we ended up calling it, then the part of the unsafe promise you need to make in order to call retire is that the star mute you pass in used to be a box. And I guess we could document this on here, right? Which is sort of safety can only be used on values that were originally derived from box. And similarly for drop in place can only be used, I guess drop in place is just like always safe. So this one is actually always safe to use given requirements on retire on pass pointer object retire, but may lead to memory leaks if the pointer type itself needs drop. Can you interact with the leader behind a local generic function? You behind a local generic function? I mean, yes, it can be any function pointer. So I think what you're suggesting if I understand you correctly is that here we do FN drop box which takes a pointer which is a star mute didn't drop and it does unsafe box from raw pointer this and then we pass old retire, oops, drop box. This drop box is a function item, not a type. Function item types cannot be named directly. Yeah, so this is what we're going to do. I mean, it is true that we could have, we can maybe do this and then say that retire, the generic over the leader and takes an instance, or maybe it's not generic over the leader, maybe it just take, well, I guess it would have to be, it takes a deleter and I guess it would have to be. This and then the leader is the D and then this is the deleter, right? Like D delete, like the argument is unused which is a little weird, but maybe it makes it nicer to work with the interface. It is true that now trade deleter is not implemented. Right, so this is where maybe we don't actually want the deleter trait, is that nicer? Like I'm not quite sure, right? So if we comment this out like this and then say that D is going to be an FN star mute in drop, right? Then we do the same here and then we say this is just D and then I guess actually this then is D. This has to be in, this is this, it's not generic anymore and then this same thing is not generic anymore, it just takes a function pointer that the deleter, like that, is that nicer? And then let's actually call this deleter to have it sort of match up, so this is going to be deleter and this is going to be deleter. Maybe that is nicer, maybe you're right. It doesn't mean that like people need to write these closures themselves, but maybe that's the nicer way to do things anyway and then here safety, safety because by the safety guarantees of retire and because retired object and because it's only used when retiring box objects or something like that. Why not pointer read instead of box from raw in drop? Because pointer read would just give us back the T like the din drop, but we want box from raw because we want to run the destructor of box so that we free the memory as well instead of just running the drop for the backing type, right? So imagine this is a box file, right? Like imagine this is really a box file then if we just call standard, well, if we just called standard pointer read for this we would get back a file, we would drop the file but we wouldn't drop the box. So the file will be closed but the memory used by the box would never be freed. That's why we need to turn it back into a box first. Could you drop and place a string? You could, but actually what would drop and place string even do? I mean, if you mean if you have a star mute string can you drop and place it? You absolutely can drop and place a star mute string and it would drop the string correctly but it wouldn't drop whatever this pointer type was. Like if this originally was a box string then you would end up dropping the string but not the box. As you would call the destructor for string which deallocates the heap memory used by the string but you wouldn't call the drop method for box which frees the memory associated with the box itself. So that would be a memory leak. Oh, yeah. So one thing we could do actually in order to keep the deleter as a trait is impold deleter for fn star mute din drop. Delete pointer star mute din drop. No, that won't work. We would need a self and we don't get a self. Maybe we do get a self. Yeah, the problem is we don't get to store anything except a single function pointer. It's like awkward because it's really just because of the lifetimes that this gets awkward. Like you need to make sure that the deleter is valid for is valid for the entire duration of the domain. So I guess we could just require that it's static, right? Like, okay, here's one way to get at this which is delete of, see, maybe it just takes self. Yeah, so we do this, we do this, we do, so this is just gonna be this of pointer and then I guess this is like a little bit weird, but let's go with it for now. And then what we do is on retire, we take a deleter and that's gonna be a tick static din de, din de, din deleter actually. So it doesn't even need to be generic over D and same thing here, this is gonna take a tick static din deleter. This is still, this is more general than what we had before, not by much because of the static bound but it is maybe a little nicer. And then what we do is we go here and we say this is gonna be a tick static din deleter. And now I think, actually, I wonder, I think closures that don't capture from their environments are considered function pointers. Like if I do this, yeah, I was afraid of that. That's fine. I mean, it can just be this instead. I think expected one argument found zero. That sounds like a lie. I mean, I feel like I did something silly, like the syntax is like not valid. The trait bound fn star mute din drop plus static drop box implements deleter is not satisfied. Let me, I want to, I don't believe that. I want to see the output. Oh, something else failed. Deleter dot delete. That's fine. Now there are two v tables but that's sort of what we would need this to be anyway to allow the deleter to be somewhat stateful. Like in the future, this might not need to be tick static, right? It's just going to be tick of the domain which might be something we can find a way to express. At least this way, we're not building our way into a corner where we won't be able to add it in the future. Yeah, the C++ library uses this like, it basically constructs a function that calls the deleter which is basically equivalent to what we're doing here. Except that you would have to, in that case, you would have to allocate. Could the reclaim API be like, reclaim T? No, because you no longer know the T once it's in the retire list. The retire list contains all sorts of different types. We can maybe do something here with like type IDs but it gets real wonky and I don't know that it gives us much of a benefit here. I wonder why did I mess up my trade implementation here? Is that what it wants? Could a delete closure be saved along with a pointer in the linked list? That's basically what we're doing with the vTable, right? Restoring a closure is really just a function pointer. A function pointer with some associated state.  So storing the vTable is sort of equivalent because the vTable is the function pointer for a closure. So the reason I didn't write, the reason I did not write this for f where f is fn, the reason I didn't write that is because it's a blanket implementation which means that no one will be able to implement this trait themselves for their own type. So that's why I didn't write that. I'm a little confused why this isn't allowed because like this function signature looks an awful lot like this. This question is what does this mean? I'm guessing that's the name used for it to just help me figure out which one, but are you planning on making a fuzzing harness for this? That'd be really cool. One thing that's challenging about fuzzing stuff like this is that because it's so concurrent, you sort of need to fuzz the concurrency. That's not always trivial. One way that it might be doable is to like, you generate a, you basically fuzz a command sequence and a schedule and then you execute that. That could be really neat, right? So you generate like, here's a sequence of 100 commands and you generate a schedule among like three threads, a schedule among those commands and then you run them. But it's just really hard to fuzz concurrency like this because you don't really have control over the relative ordering of events across the different threads. Maybe we could do it with something like, like Lume here might work really well. Like testing those with Lume seems like a really cool idea. I think we still have a little bit to go before we get to that, but I think that's probably something we wanna do. Yeah, Lume would give you assurance that you're actually handling every schedule, which is basically exactly designed for something like this. But why, oh, why, oh, why does this not implement deleter? Like if I do let X is static din deleter is equal to drop box, like does that complain? That complains. If I here do underscore, does that complain? Okay, so it is static. If I do fn star mute din drop, oh. Like am I just being dense here? Like why, these are the same. Hey compiler, right? Like if I just do this, these are the same and we've already confirmed that it's static because I can do this and it doesn't complain. So it must be tic static. So this part is not a problem. I think it's because it's an inner function for some reason. Like if I'd clear it out here and then do out drop box. Huh, no, not that either. So it's not because it's an inner function either. That's weird. This feels like a bug. All din pointers have drop in their V table. Yeah, I know that, but I can't, but look. If I don't take drop as a bound here, then I'm not allowed to cast to a din drop. So I mean, I guess I can just do like, what else would it be than din drop, right? Like din, there's no other trait here. And I can't drop this because if I drop this, I'm not allowed to cast into a din drop in the first place. But see, this is trying to cast it to the function type and I'm not allowed to do that. Okay, what if I do this? Like, can I do this? Okay, and now can I do, why is, this is bizarre. Why is tick static of that? Can that be and X? What? So if I give Y here, X here, and get rid of this. Okay, this is a more helpful error message. So this is just saying, you can't take the, you can't take the address of X because X is on the stack. That's fine, but the address of drop box, I guess maybe it's just like, it's just drop box itself is the function pointer. I feel like this is really what it should be. Oh, what did I do? Right, I should not have deleted that. And now the problem, right. I think what's going on here is, this is not a meaningful type because this is saying, like drop box is the address of the drop box function already. So what even is the address of the address of drop box? Like that's not a meaningful sentiment. And I think that's what it's trying to tell me. It's telling it very poorly, but I think that's what it's trying to say. But if I try to actually just pass in the address, then it doesn't know that this is already a, it doesn't know that this is already a pointer type that can just be turned into a DIN. Like I'm saying that I need to take a reference and this isn't the reference. So maybe this really just needs to take a, and we need to cast it to a V-table pointer. It's just really not nice.  I think that's what the compiler is trying to get at here, that this isn't a fat pointer and you don't have a way to give me a static fat pointer to it. But maybe if I do like, okay, static foo is an FN of star mu DIN drop is equal to drop box. Can I now give and foo? It's so dumb. It's so dumb. Yeah, because this is a pointer to a pointer. It wants a pointer to a thing that implements the trait and that means in our case, a pointer to a function pointer. But this is clearly not an ergonomic interface. But we can't have that be the interface. I mean, another option here is the like retire just takes, but the reason we have to do this, I think, is because we need to construct a fat pointer to pass the retire and the raw function pointer doesn't have any data associated with it. So there isn't really a translation, but by constructing a reference here, it sort of, I guess, understands which input to use. I really feel like this is a bug that the compiler should be able to handle this for you. Okay, so maybe what we do here is like, so silly, it's so silly. But maybe what we do is for, at least for the time being, we say that this is a pub static. And then we do the same thing for drop in place, right? And then we document the pub static. Can I say that a static is unsafe? I don't think so. Right, and then now in the test, I can have this be, I guess the leader's drop box. In fact, I can go even further than that and say, oh, this is even worse. What I'm actually gonna do here is do pubfn drop in place, takes nothing and returns a tick static dindaliter. And it does this. How do you like them apples as it goes, as the saying goes, and then drop box is gonna be static dindaliter. And it does this and it returns drop box. And this one is unsafe. Nice. Now we actually do get to document all of our oddities. And then down here, this is now gonna be the leader's drop box. It's real stupid, but it's the best way I know it for us to work through this weirdity. It does not work with a cast. I already tried it for the cast. What cast are you proposing to use? Oh, you're saying, all right, let me, so your proposal is down here that we can use instead of this, all right, drop in place, but whatever, as fn star mute dindrop reference. I think you're lying. Oh, like this. But why? Why? This should not be necessary. This cast should not be necessary. I don't know if this is better, actually. The reason I don't know whether I think it's better, like, okay, first of all, why on earth is this work? Or rather, why is the explicit coercion necessary here? Because that's already the signature. This shouldn't be necessary. But at the same time, I kind of like this because it lets me document these separately. Like, I could provide drop in place in Dropbox as just function pointers into liters, but there wouldn't be a way to annotate that one was unsafe and the other was not. Although I guess retire is unsafe anyway. So maybe, okay, so maybe, maybe you're right. Maybe what we can do is where is my deletors? Like, I don't want that cast to have to be there in user code. Can I make it a static, oops, drop in place? Like, if I do this, I guess that is another way to do it. And then specifically say, allow uppercase static. What's it called? Non-uppercase globals. Non-uppercase globals. And then can I do the same thing here probably? Dropbox, Dropbox2, Pub, Static, Dropbox. And then this, I don't need this anymore, that's fine. And now this can be this. Okay, now I'm happy. Great. Yeah, it's just like, the ergonomics would be too bad if the caller had to actually give this cast. The pointer came from box, so it's valid. Two, the deleter is valid for box type. And three, the old value is no longer accessible. And I guess let's number these so that it's easier for people to document their safety. Nice. Do we even have any more to-dos? Get rid of the sub-quirement, better heuristics. Okay, so I guess maybe it's time for us to run the test. Okay, feel good passes. So if I do assert equals this, 42. And I guess down here, assert equals x, 42 still. And I guess I'm going to do the swap down here. Oh, so here's what I'll do. If I create an H, oh, it's confused. And then I load this H here. I assert that I can still read 42 there. I can still read 42 here. And what else can I do? And then if I now do, I guess old retire all, we're going to need some like eager reclaim. So that's fine. And at this point, after calling whatever that is, I should still be able to read that. And then if I then drop H at that and then run old eager reclaim, like to do check, actually reclaimed. So I loaded before the swap, so I should still see 42. I should still see 42 all the way up there. And I guess after the swap, if I do an H two and load that, then X two should be a 9001. And here, if I drop H, not drop H two. Actually, I guess there's also an argument here for, I create a thing that loaded, and then I guess this is H temp. Like I'm going to have one that loads the old value, but then goes away. And I want to make sure that doesn't hold up the reclamation either. So at this point, eagerly reclaiming shouldn't work because I still have this holder. Same here. Here, I should be able to eagerly reclaim. And then here, I should check that it actually was reclaimed. Here we can do something around like the magical drop implementations or something, right? And eager reclaim is something we can add on, actually, what do we even, it's not entirely clear what we call eager reclaim on because we've already given a well retire. So this I guess should be shared domain. I think this has to be on the domain. So there's a pubfn eager reclaim on self, which just does a bulk reclaim. And I messed something up. It's borrowed. That's fine. Does that all pass? Okay, that all passes. That doesn't tell us very much. We don't actually assert that things aren't dropped because it could be that it like, this retire actually drops the 42, but the memory isn't reused for anything. So it doesn't end up being a problem. So what we're going to do is add a little sneaky type that's going to be like count drops. So we're going to have a struct count drops, which is going to hold an arc of atomic U size. We're going to impl count drops, and that's going to be self.zero.fetch add. We could probably use like a mocking library here to make this a little nicer, but I just want something that's simple in this case. So 42 and count drops drops is an arc new of atomic U size new zero. This is going to take drops assert equals my x dot zero. This is just going to nice. This is going to be count drops, I guess count drops of drops over here. Let drops 9001, and this is going to be, I guess drops 42, just so we can keep them distinct, that's all fine. And then here, we're going to assert equal count, no, drops 42 load, is still equal to zero. Here it should still be zero. Here it should probably still be zero. Here though it should be one, and the 9001 drop should still be zero, and this can be zero. And borrow of moved value, that's right, because this has to be arc clone of this, and this has to be arc clone of this. Oh my, I can't type anymore. All right, let's see if that does the right thing. That does the right thing, wow. Did we just write error free code the first time around? I don't quite believe it. You can detect that with zero wise being implemented on the type.  I guess we could like, Imple count drops fn new, which returns to self, which is a self arc new, but I don't really wanna do that. It seems more worth that it's. Yeah, I don't believe this yet. There is, so we're also gonna want some compile tests here, some compile failure tests for things like this being invalid. And for this, oops, where's the other, this one being invalid? That's neat. I mean, that seems like it works. That's a little bit scary. I mean, remember, we haven't done anything about the, like we only have this shared domain business, which is something that we would want to fix up. I guess actually, like if I now do say, drop h2, and then this, then this should still be zero. So silly test, but there are almost certainly things missing. Like for example, there are memory barriers that are not documented in the paper, but the implementation has. There's also all of these memory orderings that need to be corrected because currently they're all sequentially consistent. And in reality, that's not what we want them to be. What does eager reclaim do? Oh, eager reclaim is like, if you specifically want to make sure that you free memory before you do something else, eager reclaim is what you would call, right? So it's, I don't have a thing to, I don't have a thing to retire right now, but I still want you to reclaim things that have been retired in the past, right? Because remember, a call to retire might not succeed. Like it might not be able to reclaim the item yet. So eager reclaim is a way for you to say, I now want to reclaim things that I retired in the past, even if you have to block in order to do so. So arguably actually, eager reclaim should say, this should be true, this should be false, and this should be like block as a bool. And then like, there should be some kind of here, like a, where's the best place to do this? I'm gonna go with down here. Actually, it's really sort of around here somewhere, but I'm gonna for now just say, if block, then return self dot bulk reclaim. If, if tail dot is some, and block. We, caller wants to reclaim everything, but somewhere left, so try again. And here we can do like thread, yield now. All right, we can be a little bit nice about retrying. Oh, we can do like thread, yield now. All right, we can be a little bit nice about retrying. Oh, fine. I guess if remaining is not null, then do it again. And I guess this could actually be a parameter here, right? Like, do you want to actually block or do you want to just try? And so here we could say, in all these cases, we don't actually want to block, because we sort of know that blocking shouldn't be necessary. And maybe eager reclaim returns how many things were reclaimed. Maybe that's nice. So this is gonna be return zero, and this is gonna be return reclaimed, and this is gonna be reclaimed. So now our test can actually be like assert eek and zero, and we can do the same thing. Here it should be one, and here it should be zero. Great. And then the one thing, okay, so we are getting pretty close to time. One thing I do want to run though is cargo Miri test. I want to see if Miri spots something silly here. Like Miri is particularly good at spotting incorrect things in this kind of unsafe code, and it might spot something silly we did. Like ultimately I want loom test here too. The following memory was leaked. That's fine. So we are expecting this to leak memory at the moment. There's certainly like a info drop for has pointer domain to do. But that's fine. I'm not too concerned about that because it's sort of a it's a known memory leak. And that doesn't bother me. Arguably, yeah, that's fine. Like we are gonna have to implement drop for has point domain. I think, can I even do this? Like if I do this, will it panic? No, great. Yeah, cause destructors for domain, destructors for statics are never run. Great. The recursion of bulk reclaim should be added to reclaim results. You are completely right. Which actually means that it can't be tail recursive, which is well said. We could make it be tail recursive actually. Reclaimed. Did I even use a claim for anything? I don't think I did. So this way it can be tail recursive because this is still just a recursive call. If this was instead reclaimed plus, then it couldn't be tail recursive because you need to run something after the function returns. It won't necessarily be turned tail recursive anyway. Allow tail recursion by passing down reclaimed. Yeah, Rust doesn't guarantee tail recursion, but it is nice to write things that can be tail recursive once we get something like become, which is a keyword that's reserved. But even so, like, if we didn't do this, it couldn't be tail recursive, even if the compiler sometimes made it.  Miri will work perfectly fine just here. The challenge with Miri and with sanitizers in general is that they will only check the code that you wrote, like the code that actually ran, whereas very often the concurrency bugs are in particular interleavings that you haven't run, right? And that's where a library like loom comes into play where loom will ensure that every possible interleaving gets executed or approximately, which means you're much more likely to find these kinds of problems. And then you can run Miri on top of loom, but then things run really slow. I could make it a loop, but making this whole thing as loop is actually really painful because the whole thing ends up being very long. That's why I didn't wanna do that. So I intentionally wanted to do it this way instead. Great. I think now that we actually have a test that passes and code with no to-dos except features, I think that's a good place for us to end today. I guess I will push this code to GitHub. It's not super interesting yet. Like it's a pretty simple implementation of hazard pointers. And there are obviously things that we want to add. Actually, I just realized one thing that we missed, I think. Yeah, spotted a thing that we missed. Oh, no, nevermind, nevermind.  Like I'm guessing for the next stream, what we'll probably do is implement domains so that they actually work so that there's not just the one static domain. Maybe we'll take a look at like the other features. We'll certainly fix up all the memory orderings and the atomic barriers and stuff that are necessary. And then probably write a little bit of a test suite with Loom because I'm just like not convinced that this actually is correct. And that should give us some more confidence in the correctness of the implementation. Let's see. Yeah, it's a haphazard implementation of hazard pointers. That's right. That's very funny. And of course, this is like better heuristics for this once in a while. Like it seems like the Facebook library has a much smarter implementation here that we'll want to investigate now that we have the basic structure up and running. But I think that's all some things we're gonna save for next time. So I guess, are there any sort of tail end questions from chat on everything that we did today and sort of the path we're gonna go next? And then after that, I think we're gonna call it quits for the day. Ibrahim, I know you were very much screaming at the screen earlier to get me to realize the cast for the function border. I'm glad I caught it eventually. I do think we ended up at a good compromise though of having the, there's too much code on screen now. I really need to split this up. Of having, where even is it? Having it so that the caller doesn't need to do the cast. It's a little bit of like an ugly trick here, but I think it's the right thing to do. All right, if there are no further questions I think we'll end it there. Thanks everyone for watching and I'll see you in two to three weeks when we do another one. So long for well.


I want to train up the youth. I want to be a great teacher. Hello, my name is Travis, Travis T. Thomas. I'm a music producer, an artist, and an educator. You know, I feel like as an artist, I consider myself as a progressive, innovative person. So my artistry grows as I grow. I kind of feel like, you know, I don't really like to put myself in any type of box or any type of genre. I just like to be creative. You know, I like to reflect who I am and what I am through music and through art. My inspiration is a long list. You know, definitely my family. You know, my mother in particular inspired me a lot. She really was a driving force behind me, just getting into music and motivating me. So, you know, believe in myself and know that I had a boundless potential. So I think life, family, and just music inspired me. When my mother died, April 1, 2011, I realized that I hadn't become that thing that she wanted me to be. So in honor of her, I decided to be that thing and start to stop allowing doubt and fear and all the things that people allow to keep into their mind and stop them from facing their own, you know, sweeping their own destiny. And I said, I'm going to do it. In the year she died, I put up my first album, Bixie Motions, you know, because that's how I was feeling. You know, I was, it was bittersweet, but that was the defining moment for me to just decide to step out of the background and just be the artist. How do I come up with my beats? Sometimes I may hear something in a record and that might trigger something. I might say, okay, let me try to do something. Sometimes I just sit at the piano and just play. Sometimes I just throw out a rhythm. Sometimes I just sing. So just, I guess I'm inspiration driven. So a lot of times it has to be something that inspires me and that, that really inspiration will lead to the next. My message is always to inspire. My message is to be you. Really, I'm speaking through myself, you know, to myself first and hoping that the listener, you know, takes that and applies it to them, you know, because I understand how it is to, you know, to want to do something and want to be something, but just not know the steps to take.  So I'm always trying to live people up. I'm always trying to motivate people to be great in their own way. So my message is always going to be, to be great, to be a light, to be an example, to be honest. What is my next move? I want to train up the youth. I want to be a great teacher for them so that they can have some knowledge and have a foundation strong enough to prepare them to, you know, the next level. I also want to put out more music. I want to do a couple beat tapes for the, for the culture, for the art. I really feel like as a producer, our craft has been in the sense taken for granted.  So I just want to be a leader in that aspect. So that's really what I'm focusing on. I am Travis, Travis T. Thomas, and this is Five Questions.


Hi guys, it's Monica and hey, it's Sheila. It's Sheila, oh my gosh, my lovely, lovely friend, Sheila. I'm so happy to be sitting next to you. It has been so much fun having Monty and Jay here and when Monica stepped out of that car, I was just astounded. Her weight loss is dramatic. It's dramatic. I mean, dramatic. I don't, well, thank you. I don't really think it's dramatic, but thank you so much. I think I've lost more inches than pounds. I'm trying not to be a, you know, like addicted to the scale, but I get on it every day and it's like crazy. Don't do that because then it doesn't move and you kind of get discouraged, but I can see a difference in my clothes and stuff like that. You have nothing to be discouraged about there. It's dramatic. Oh my gosh, thank you, thank you so much. So I just want to say that a lot of people have been asking about Sheila and tell her when you see her that we said hello and then I've had other people various points and times. How's Sheila, how's she doing? Is she going to come back to YouTube? Probably not. Probably not, but she's doing good. But I'm doing good and I do, Michelle and I still talk with some of y'all and I value y'all, but I don't think I'll be coming back to YouTube. No, no, no. I don't think you will either, but you never know. Never say never because we're women or we can change our mind. I have been away from YouTube so long that YouTube changed my name. I am no longer simply Sheila. I am life with Lily002 something. I noticed it a couple of months ago and that was some changes YouTube made a while back with their inactive people. YouTube has made a number of different changes and some of them have been not good at all. And that's the frustration I think that we all have with YouTube is that it's so hard when something happens there's not a person you can talk to. You can send messages and you might be dealing with a chatbot, but you can't really talk to a person.  This is true, the little person on YouTube. Absolutely, this is so true. So Jay and I have been, so those of you that have followed that, Jay and I have been on sort of an extended trip and I'm not gonna say vacation because it hasn't been a vacation. We've both been working almost every day a little bit. I've been teaching, he's been teaching, but we've been doing it in a different location. And one of the things that when we left Florida to come up to South Carolina, that I just could not wait to do was to get together with Sheila and her husband, David. And I was like, couldn't wait to see her. You know, at first we thought she was gonna come over to Myrtle and then that didn't happen and Jay and I decided we were gonna do an M&J adventure and explore a little bit of North Carolina. Yes. So we came here and Sheila and David kind enough to host us, put us up over a night, two nights actually. And... I ask them to be a chance to tell you you're leaving next Wednesday, right? No. No, we're leaving Sunday.  And this morning we went out and walked together. We did. You know, I did a little over two miles, I think, something like that. Even though it's colder here, it's still a lot nicer than in New Hampshire right now for sure with all that snow, right? Yeah. Absolutely. But you look fantastic. Thank you. And you look like you've lost a lot of weight too. No. Well, I think you have. You look, your body for sure. I think the walking is just toning things up. You know, I haven't lost a lot of weight. But David and I had COVID, Lily is getting into your video. That's okay. David and I had COVID and it was rough this time and it's really messed up my blood sugar. So I've been walking a lot. And I think maybe it's just, things aren't being lost, they're just being rearranged a little. And I, well, I've lost some of them, but I also feel that I am being rearranged too and toning and stuff like that. But just trying to stay more physical, you know, hitting 70 this year was hard for me.  But hitting 70, I couldn't have believed it. I said, how can I be 70? I don't feel 70. I don't think I look 70. I don't think I act 70. You don't. But I'm 70. And that's like so scary. We had a discussion over at my mom's on Easter Sunday. I said that Jay and Mani were coming and mom was like, who is Mani? Oh, you met them at the beach last year. I'm like, well, we'd met them before. And I said something to David. I said, Manika's 70 now. He goes, Manika, it's not 70 years old. And I'm like, yeah. And he goes, no, she's not. And I pulled up a picture and I showed it to mom and she said, that woman is not 70 years old. Oh, that made my day when you told me that.  It just made my day to hear that and your mom too. How sweet. Well, he swore up and down until you told him last night that yes, you were 70. She's not 70. You have misunderstood. That was funny. We went out to dinner last night and it was an Italian restaurant, right? And we had, I had some really good cheese sticks. I just love those mozzarella cheese sticks. And Jay had the buffalo chicken wings. And what else did I have? Oh, I had this cheesy garlic bread. And onion rings. And onion rings. Yeah, and I probably gained 20 pounds, but no, I walked it off. You walked it off this morning. I walked it off. Anyways, we are gonna be here. I don't know what we're doing tonight, but we're just kind of venting out and chatting and getting caught up. And I wanted to share a little bit of my lovely, lovely Sheila. And I was so happy when she agreed to come on camera. I didn't have to really force her. No. Which was really nice. So, and she looks beautiful. She looks, you look a lot younger. No, I don't. Yes, you do. Yes, you do. What do you, how old are you? I'll be 62, the end of the month. She's a baby. 62. Some days I'm 62 and some days I'm 16 and some days I'm 123, you know? Yeah. But in either case, she is the baby and the youngest and I have a little circle here today. No, David's. David's the baby. David's the baby. Oh, that's so funny. David's the baby. He's the baby. Oh, gosh. Well, you're both babies then. Well, I can't, okay. We're the babies, we're the babies. I would have done anything Monica asked me to do just to get her to spend time with her. If she had asked me to go push a peanut down the driveway with my nose, she would have done it. She would have done it down the driveway with my nose. She would, oh my gosh, no, we'd never, never ask you to do something we really didn't want to do. I might force you to do something a little bit that I think you want to do, but just need to know it. But anyways, so one of the things that I did, this trip is I ended up buying a lot of elf products. And I think I mentioned in my empties that I was using a lot of elf, but I bought this fairly new, at least new to me that I saw which is the Cameo blush. And I put it on heavy today. So I might look like a, I tried to rub it off, but it actually is really nice. I really like it. So I don't know if you can see, you know? I probably isn't picking it up. But anyways, I thought it was really nice. I liked the fact that I could rub it in, right?  But I really, really like it. So I can't remember what I paid for this, but I bought it in Ulta. There's a brand new Ulta in Myrtle Beach at the outlets that is a beautiful Ulta, huge. And so I bought it in Ulta and I also bought, and these things cost like four bucks. This is their lip gloss. This I really like. I think this is the peach flavor. You have the peach. I have, do you have the peach? Yes, it's the peach. It's like a balm, but it's not gummy and it's not sticky and it stays on overnight. I really like it. I really like it. I am addicted to lip balm. I am addicted to lip balm to a certain degree because I've had such dry, dry lips ever since I've been on the medication for the breast cancer stuff. It's like, I'm dry. You know, my lips are dry.  So I'm always putting something on my lips just to, you know, feel like they're moved there. I can move or whatever. It's like unbelievable. But this I thought was really, I liked how it went on and it felt really good on my lips. So if you guys see it, you know, pick it up for four bucks, my gosh. It feels really, really good. It can't go wrong. It's not, it's not a plumper. It doesn't tingle, it doesn't numb and the flavor or scent is not overpowering. It's just pleasant. I know right. But mainly it feels really good. I said to Jay, how was it for kissing? And he said, when he kissed me, he thought it was nice. Well, there you go. And the other thing, I think I mentioned this a number of times. I have, and I bought this at home. This is the, and I know you have the other version or you had the other one, but this is the Centagebel Invisible Sunscreen and it's 35 SPF and it's the ELF. It almost goes on to me. It's not shiny, it's not greasy. Cause I think there's another one that is greasy, right? That you had tried? It's the dupe for the super goop. And there are two different ones and the only one our Walmart had was the other one and it was greasy and pilly and horrible. This is not greasy. I actually use this as a primer. So I put this on and it kind of like, you know, spills in a little bit of the holes at times on my, the pores on my nose. So I use it as a primer, but this is pretty well the first step that I do. And after I do my regular skincare and then I will put whatever makeup on. But I'll tell you, since I've been doing the Red Light Therapy Mask, I haven't, other than going on camera, I really haven't used a whole lot of foundation. I think the Red Light Therapy, the one that I have is the Candy Way one. I took it on our cruise. I mean, I put that on, I was sitting out in the balcony while we were on the cruise, just watching the ocean go by was the funniest thing and I'm just sitting there like this with the mask on. I really think that that has helped my whole skin tone even out more and become more like thicker, you know, type of thing, right? And I love, love, love that mask. I got that in April. My mom was in the hospital. My mom had just passed away when I started to use it in last April and I used it pretty consistently. And I think by sort of maybe September, October, I was really seeing a difference in my skin tone from that mask. So, and then the other thing was the Nara. I think the laser on that. And I don't have, I have it here with me. I don't have the pro.  But this I really, really have seen an improvement. I don't know if this video is gonna be after the update or before the update on the Nara but I filmed it update on it and you may already have seen it or not. But I was amazed on the photo that I took on the left side of my face, which is the driving side. Yes. And it was naked skin and she really saw me naked skin this morning too, right? It is a big, huge, dare I say again, dramatic difference in those two pictures. Everything about Monica this trip. Money is just dramatic. Oh my gosh. Well, the picture, I'll show it again here. That was taken in December 28th and then the second picture I had mentioned it, you know, March 3rd, but it was really April 3rd. I was shocked when I saw, I mean, you can still see the, there are still wrinkles there, but they're like so much better and not nearly as deep as they were. I was amazed at the difference. So yeah, I'm loving this thing. Now the nearer and the nearer pro is the only difference the coverage area or is there a difference in the laser? I think I'm not 100% sure, but I believe there is a little bit difference in the laser as well as the coverage area. So I was looking at, if I did the nearer pro I could go through my face, because like my jowls, I would really like to work them. And it just, it takes a long time because you figure there's 40 pulses that you are getting and you have to move. And then you go to a different area. And so it can take a while in some days. I don't have all that time, but I do this mostly in the morning, but I have also done it at night when I didn't do it in the morning. So the coverage area on the nearer pro looks to be about four times the size of that coverage. It's more like this size. More oblong or something. Yeah, yeah. Well, nearer, you need to send her the nearer pro because she's really giving you a good little shout out here. And it is dramatic. It's, to me it was really, really dramatic. And I do fully believe that for me, anyways, it's a combination of a lot of different things.  Right. Your body may look better, right? But your face may look a little bit more wrinkly. So I really was on a slow weight loss, trying to be really slow because I wanted my skin to recover. But I was amazed that my skin, even after losing, I mean, pounds-wise, it's 38 pounds, but it's been since November of 2022 that I started. So it's been a slow process. But I was surprised that my skin hasn't fallen as much as it could have. Oh my gosh. With losing close to 40 pounds, but your face doesn't look, and I, you know, let me get my words together.  No, I'm surprised because in the past, I've seen that happen.  So I guess the point that I'm trying to bring forward is that I think it's the accumulation of all the skin care that I've been doing. I think the red light therapy mask, every day I have that on. And I think that has really, really helped the overall, you know, in toning, and then the narrow for that position, you know, really working on the certain spots I think has been really good. So lots of different things that, and I have that zip device. I brought that. That's like the new face toning. Yeah, I've been trying to do that a little bit, but I'm not as consistent. My Marionette lines, I used to say they had grandchildren. Well, now they have great, great grandchildren and I'm not doing nothing. My biggest skin care item right now is sunscreen because I'm outside a lot. I mean, I want to do something to those, but I haven't even been using Retin-A. And I thought it's gonna be warm weather. I don't like using Retin-A in the winter and I'm gonna get back on the Retin-A and I'm gonna try and take care of this old face a little bit better. I think you look absolutely wonderful, especially if you're not doing anything. But yeah, get back on your Retin-A. I think that'll be something you'll be happy you do later on, maybe not now, but later on, you know, when you look at it and go okay. It's just like, not having the oomph, right? Yeah, it's, we were sick from January the 1st through the middle of February and I stopped doing anything.  And I think that's understandable when you don't feel good, you're sick and all that. Yeah, you know, and I had about a food poisoning too, not too long ago or about a month ago was something like that. And I was like, ugh, I couldn't, I couldn't, like three days move, you know, because it was so bad. But I guess I'm just addicted to skincare. So, and I'm happy I'm addicted to skincare. Well, you know, I even, the floor and bath CBD oil had taken away the veins on my face. And I'm in the bathroom the other day and I'm like, what's on my face? And it's the veins are back on my face.  So I've got to, I gotta get back into a routine. I've got to. Yeah, you were using that CBD. Oh, it was wonderful. It, I had a lot of broken capillaries on both cheeks and it made a big difference. Yeah, I was using it, I really, really liked it a lot. I was using a lot of the CBD products. And then I think once YouTube removed my video for mentioning it, I was like, kind of bummed about it. I should not have mentioned that in this video. No, I don't think that's going to be an issue. I don't think so, you know? But you, and when YouTube removed it, that whole was the whole dedicated video in CBD. So it's completely different.   And look what happened. I don't even want to mention it, but they removed, you know, Natalie. Yeah, you just need to edit that out. I don't want to get, I don't want to get my BFF in trouble. You're not going to, you're not going to. So anyways, I have no idea how long this video is, but I thought, you know, Sheila looks awesome even if she doesn't want to say she looks awesome. She's been doing a lot of things right. She's been walking and maybe you haven't been doing all your, you know, skincare stuff. You're still taking care of your skin because the most important thing is that you're using sunblock. Oh, yeah. I think to me that is the biggest anti-aging, anti-aging tool you can use is sunblock.  Then I have a setting spray with sunblock in it and then I'm reapplying sunblock during the day. I mean, you figure since, I've been in Florida and then South Carolina and I do have a tan. I know I have a tan, especially in my arms and stuff like my legs, but you know, been wearing a lot of hats and straw hats and baseball caps and sunblock all the time. You know, if this, if this was 10 years ago, 15 years ago, I would have been sunburned and dark is anything after, you know, all this time in the sun. I get up in the morning and at seven o'clock I'm putting on the sunscreen I don't like. Like it doesn't, I don't like to wear it, you know, under any makeup, it doesn't play nice. And I put that on and I have on a ball cap and my sunglasses and gloves on my hands. And then I have a hoodie over my ball cap. I do think sunscreen is the most important thing. So I am doing that. And I've always said that, you know, you can use Retin-A, you can use Nara, the red light therapy, all of these other devices and vitamin C's and all of that stuff, you can spend a fortune on that. And the one thing that you miss is if you don't do it and doesn't even cost that much it is your sunscreen and you make everything else not work.  So this will be my talk for sunscreen. Yes, yes. Sunscreen, yes, absolutely. So any parting words? There's someone very special out there that about twice a month I get a lovely card in the mail. I get emails, she's very special to me. Jacks, this was just for you. Okay, so hopefully she'll watch this. She'll watch it. Okay, she'll watch it. All right. It was for Jacks and for Moni. I'm glad you didn't make me push a peanut down the driveway with my nose. I never would do that. But I love you and I'm so happy to have you on my camera, my screen with me. It has been awesome. It has been just awesome. Jay was texting Moni during the last part of the drive and Jay was texting me, we're on this road, we're on this road, we're on this road. And when he got to our road, I'm standing at the door and pacing between the side patio door and the front window, side patio door and the front window.  It's drastic. Oh my gosh, I don't see it that way.  Oh, thank you. And I love you. Thank you with my little wig on. She saw my gray hair. I won't show you my gray hair, but she saw my gray hair. It is awful. It is like so great all over the place. I'm like, I'm not ready for that. So I'll wear my wig, I'll wear my topper, I'll wear a baseball cap. And when I get home, I'm gonna go see my hairdresser and see what she can do. So, yeah, that's the plan. Well, it's been great. And, you know, maybe the next time Monty and Jay come up, they can stay till Wednesday. Ooh, Wednesday. They can stay till Wednesday or Thursday or a month. Or a month. You could just park your motorhome out there. Yeah, I said to Jay, could we fit the IV in this driveway? Yes, you can. And he said, I think we could. Yeah. So you never know, right? You never know.  Maybe do another video. Maybe we can do a live stream. I can be in New Hampshire and you can be down here. I'll do that with you. Oh, see? She's committed. So anyways, guys, thank you all so much. I will see you in my next video and Sheila will see you some day soon too, right? Some day soon. Bye-bye. Bye-bye. Bye. I don't know what to do. Just smile. Now another little smile. That kind of smile? Yeah. So I'll just take away a screen shot of that and that'll be the thumbnail. Yeah. That'll be the thumbnail.


What's up everybody? It's Jason here with jaspyscasebrakes.com 2019-20, Panini, Corkings, Vasswell, 4 box, Pikachu number 12 just sold out And we randomized the pelicans to one customer in this break So all you had to do is buy one team in the break, you get one entry, you bought two teams, you get two entries, etc, etc So we copy the names in the customer list here Paste them and customer number one after Full 12, 6 and a 6 gets the pelicans One, two 3, 4, 5, 6, 7, 8, 9, 10, 11, 12 Steven P. Anemoneuno Congratulations man, you just won the pelicans and that was 12 times Which, what did you have in here?  Um, Ryan recently just bought the pistons at number 12 Maybe it was a different number, maybe it was number 11? I don't even think number 11, you had the pistons I had just seen the order confirmation for Ryan, for number 12 Ryan just got the pistons, Kenneth got the pistons at number 11, Steven got the pistons at number 10 Pistons in Mega Box number 4, it was taken 12 hours ago, Gabe got the pistons at number 9 Maybe you're thinking of a different breakdown Yeah, I don't have any of your orders with the pistons here Or at least recently since last Wednesday Chet bought the, you didn't select no pistons The only order for Chet was the Eagles in score number 4 Or score number 2, sorry You bought the Buffalo Bills in Contenders draft picketing number 10 And you bought one spot in select You could double check that order confirmation But that's all I see, I don't see no pistons under that order So maybe you thought you put them in there and it might have just been taken out or booted But yeah, sorry, I don't see your name in there for the pistons You could double check yourself Alright, so there you go Oh, you removed them? Yeah, yeah, sorry No problem man Cellopack number 12 sold, nice We need those We need those to sell so we can do that mixer That mixer is really loaded right now guys, it honestly is It's a little bit more pricier straight up, but I mean you have a choice of prism in there You have a lot more better boxes Diable Level 1 Ah, gotcha, no worries man, it's all good I mean I do make mistakes sometimes though So, woo, Zion 5 out of 49, a little piece of his shoe First steps We're not a piece of his shoe, just a photo of his shoe Look at that, Zion Williamson The first card I've seen of Zion in this case 5 out of 49 So Pelican's going to Stephen Sneakerheads are going to love this card too Impressions Of Nasir Little Or Impressionists Portland Is this an RJ Barrett? Wow, this box is loaded RJ Barrett autographed heir apparent The Knicks, Brian Zeller has the Knicks There you go Maestro Anthony Davis Donovan Mitchell Angoga Bitazi Level 2 Alright, next one I don't think we've hit the case yet Either, I forget the name of it I forget the name I'll notice it though right away Yeah, Spencer Jasper's case brings out GMO If you want to email us And let us know what happened And we'll take care of that for you Cody Martin Larry Martin, Lucas Amnick We got Mark Price for the Warriors Emergent 99 Cody Martin Artists Maestro, Steph Curry And Lucas Amnick, Level 2 Next one I don't see two autographs from memorabilia cards But we only got an autograph from memorabilia Pashaw We got for the Pacers, Goga Romeo Lankford to 179 Celtics And Dino Raja for the Celtics And we got a Maestro LeBron James Look at that And Romeo Lankford and Luka Donchich Alright, last box guys, good luck PJ Washington Jr Buddy Heald Admiral Skollfield Look at that, we got a Thunder Derrick Fischer autograph That's cool Number 25 Thunder, going to Steven Apprentice Artist John Moran That's for the Grizzlies Matthew And a Jackson Hayes Level 3 With a Yanis Maestro Alright you guys, and there you go There were the four boxes Fischer Auto Raja Jackson Hayes Level 3 Apprentice Artist Luka Donchich Maestro LeBron James R.J. Barrett Auto And a First Steps Zion Williamson Number to 49 Which I believe is a huge hit right there as well And there you go Pick a G number 12 in the books Jasperis.com