So my name is Gurney Hunt. I work at IBM Research with a group of people, some of whom have already spoken like Mimi. Used to work with Dave. Dave moved on to perhaps greener pastures at GE and others that you've seen. So what I'm going to be talking about is a product that's about to come out in IBM. IBM is obviously changing because we're talking about things that you can't buy before you can buy them, but not telling you when you'll be able to buy them. But it's close enough that we're going to talk about it. This work was done as part of, is derived from work that we did under a contract to the US and Canadian governments. That's what all that fancy words on the left says that our contract requires us to put on all derived publications of which this is one. So I'm only going to speak from my view, although it represents fairly closely what's going on if you have any and I'll answer any question I can or tell you I can't answer it. So basically I'm going to go over these four things. I'm going to give you an introduction to protected execution facility. I'm going to give you an overview of the architecture. I'm going to give you some lower level details, and then I'll give you a quick summary. So the team that's working on this is IBM Research, IBM Cognitive Systems, otherwise known as Power, and the Linux Technology Center. Some of the people from the Linux Technology Center who have contributed to this are objective is to deliver this technology on for open power and power systems and all the software and firmware associated with it is is either already open-sourced or will be open-sourced. So the basic idea is that there's a, there's, it's, we're, we have a model that's similar to the model that AMD introduced if you heard their talk earlier in the conference. Basically, it's being increasingly difficult to guarantee the security of systems that are in a virtualized environment. So we've come up with our approach to giving you a way of having, having a virtual machine on, on top of a hypervisor without necessarily having to trust the hypervisor. When we're saying that, we're not saying that hypervisors are bad guys. We're basically saying that our customers are asking as we try, as we try to move more and more people to cloud computing, especially the, especially the people that we look at, they, they want to use cloud computing, but they don't want to give their stuff to the cloud provider or the cloud providers administrator or anyone that has access to the cloud system. So our objectives are introduce something we call secure virtual machines. We want to protect the secure virtual machine against attack. We want to protect the confidentiality and confidentiality and integrity of the secure virtual machine. And we're going to integrate with the trust of computing tooling. So what we're doing is dependent on the secure and trusted boot stuff that Nina and Al talked about earlier in the conference. We innate in our approach to enable secrets to be embedded inside of the secure virtual machine. And we, we get secure virtual machines by converting existing virtual machines into secure virtual machines with new tooling that's yet to be open sourced. The whole, the whole idea is that instead of having to trust all of the hypervisor, you have to trust your host boot, Opal, which we're talked about in the secure booting stack. And then we're adding a new piece of code called the protected execution ultrvisor or ultrvisor for short. And then, and that's it. And the hardware. So we're relying on the open source ecosystem. We're not limiting the amount of protected memory you can put in a machine and any existing and, and the way we do this has no impact impact on the applications you run in the virtual machine. And these foils will be on the deck. Okay, so this is our quick overview. We got our hardware down at the bottom. We got this protected execution ultrvisor on top of it, which is firmware. We have Linux KVM on top of that. And we have both normal virtual machines and secure virtual machines running on top of Linux KVM. As a little red arrows indicate you can't access the secure virtual machines memory from a normal virtual machine or from Linux KVM. The system doesn't allow that. So how's it going to work? So we start with, if you want to create one, oh, well, I should tell you that we still allow the hypervisor to do pretty much everything it normally does for a normal virtual machine, for the secure virtual machine, and I'll talk a bit about how we make that work in a minute, although it may sound familiar.  You have to collect the public keys of the authorized machines, where one of the things we're introducing in here is every machine will have a public private key pair. The private key of the machine will remain inside the TPM and the public key will be available to the owner of the machine. We make it possible for you to use the same public key on more than one virtual machine or to have every machine you own have a different public key, that's your choice. We do that not by stuff that we've created, but by exploiting the capabilities of the TPM 2.0. Our tooling confirms that your file system is encrypted. The gotcha here is that you'll get only the protections for your encrypted file system that the encryption form that you use allow. We recommend that you use an encrypted file system that gives you integrity protection of which there is one for Linux, but we will support any encrypted file system you use. I think in this talk I probably talk about using DMCrypt, but DMCrypt doesn't have an integrity protection, but there are encrypted file systems that do. The tooling builds some integrity information and outputs an SVM. I'll give you the format of it in just a little bit. To let you know what's going on, it starts like any normal VM as a normal virtual machine, nothing special. During the brute process, the SVM executes an intersecure mode syscall instruction, which is a new instruction we inserted into the instruction set. That instruction is actually executed right at the end of prominent. The ultravisor receives the SM instruction and it points to some encrypted information and it's included with the secure virtual machine that enables the ultravisor to check the integrity of the secure virtual machine. Okay, so what does it do? It grabs the entire blob of the secure virtual machine that's in memory at that point and moves it off to secure memory, which cannot be referenced by the hypervisor. Once it gets it to secure memory, it moves the whole thing off, including the secure blob. Once it gets to secure memory, it opens up the secure blob, assuming it has permission. If it doesn't have permission, it fails right there. It then opens up the integrity information and checks the integrity of everything that it moved over to make sure that it's the same as it was when the user created the secure virtual machine. If it is the same as it is when the user created the secure virtual machine, it shifts the secure virtual machine into secure mode, which I'll explain in a minute, and then it resumes execution and the machine starts to run. Now, I have, there's an important point that during execution of the secure virtual machine, the ultravisor receives all interrupts from the SMN. It saves the SMN and only reflects information required to process the internet. Basically, the ultravisor sits between the secure virtual machine and the hypervisor, and interrupts come into the ultravisor that are intended for the hypervisor, or asynchronous interrupts that would occur while the SVM was running. And what the ultravisor does is it saves off all of the state of the secure virtual machine, which is all the registers. Floating point, vector registers the whole nine yards, puts them into a structure that's in secure memory, and it puts other state into the register state. If it's an asynchronous interrupt, then reflects the asynchronous interrupt to the hypervisor. In our case, Linux KVM, and when the hypervisor's done, it has to call, it's, there's some patches out, so go call back and we will restart the secure virtual machine. If the interrupt is because of an H call coming out of the SVM intended for the hypervisor to do some work for it, we do exactly the same thing, except we leave in the registers reflected to the hypervisor, all of the state that the hypervisor needs to do its job, otherwise it couldn't do its job, and then it comes back. And then we take this stuff out. Now, we'll get into that. We're going to go now at a slightly lower level and give in more details of that architecture. Okay, so the base principles are the following. Previously in power, we had hypervisor mode, supervisor mode and problem state. Now we have ultravisor mode, hypervisor mode, supervisor mode and problem state. Ultravisor is at the bottom because it owns the machine. So it's the highest privilege mode in the machine at this point. So this allows us to do these things. We're saying that we're minimizing the trusted base because you only have to trust the stuff below the hypervisor. And with the TPM and stuff, you can find out that this is in fact actually running our stuff and it's a machine and all this other stuff. If you're doing all the remote attestation and you're concerned and you're on a remote cloud provider. We introduced this notion of secure memory, which is only accessible by secure virtual machines and the ultravisor. The other way to say it is it's only, there's a new bit in the MSR, the machine status register that says you're in secure mode. And you can only reference something that's in secure memory if that bit is on. And the only thing that can turn that bit on is the ultravisor. So for those of you who are really into hardware architectures, you realize that the machine therefore has to boot in secure mode with secure mode turned on. So it comes into host boot in Opal if you're familiar with our stack with secure mode on. And at the correct point Opal loads the ultravisor and when the ultravisor goes back to Opal, the secure mode bit is off. And then Opal loads, starts the booting of the guest kernel just to put it in perspective from what you saw about the secure boot on power. We introduce ultravisor mode and we enable secure virtual machines. So here's our overview and we'll work through this in a couple of more slides in a little bit more detail. As I told you at the bottom, we're relying on TPM where we embed our private key. Whoever we get our RAM from, we have the CPU with protected execution facility modifications. We have attached the secure memory associated with those modifications. We have normal memory that anybody in the CPU can read including things in running in secure mode. We have NuFirm where we call the protected execution organizer. We run Linux KVM hypervisor and on top of that we have virtual machines, either secure virtual machines or normal virtual machines. They both come from whatever storage the system has and we create a new tool there where you give the public key so you can convert a machine from one to the other. Protected execution facility refers to the changes made to our open power architecture and in particular each machine has a public-private key pair. Protected execution organizer refers to the open-source firmware that will be open-source that's not open-source yet. I said all of this so we won't say it again. Starting from the bottom up, the private key remains in the TPM. There's a function of the TPM for those who are familiar where you can say give me a public-private key pair, keep the private key and give me the public. That's what we use so the TPM generates the key. The alt-visor does not, for those of you who are familiar with the kernel, the TPM device driver remains in Linux KVM. Therefore, we're going to have a TSS inside of the alt-visor so that we can appropriately share the device driver with the VMs that are running up above. However, again, exploiting the trusted computing base, we can set up a secure channel to the TPM because we have a shared secret. And we can therefore talk through Linux KVM to the TPM to do the work that we need to do as the alt-visor. So the only thing that you can get is the denial of service attack. You can't get a loss of secrets because the information we're passing back and forth will be properly encrypted. The hardware separates the memory into normal memory and secure memory. And after boot, only SVMs and the alt-visors run in secure memory or can reference secure memory. When ESM call is received, if the calling SVM has not been modified, the alt-visor will transition it into secure mode. So what's going on with the hypervisor is we're slightly more privileged. It has to be para-virtualized to run with the alt-visor. In our research project, we built a version of this architecture where the alt-visor did not have to know that the alt-visor was there. In other words, we virtualized the hypervisor. We did that just for... Why not? We had a small team. It made it easy to do. We didn't have to have hypervisor skills. We got it up and running. We could run virtual machines. And the performance would kick us out of the market because it was really bad. I won't give you the numbers unless you really need to know them. We tried it. We did it. We know how to do it.  And that pushed us into a para-virtualization model. And in the model that we're running now, the impact on normal virtual machines is nearly zero when they're booting. The overhead for secure virtual machines is a single digit and a small single digit. By our projections, we're still working to get the final numbers on the actual hardware. So if the hypervisor needs to update, the partition scope page table will have to ask the alt-visor. And if it's running in SVM, it will have to ask the alt-visor to complete the return. If it wants to go back to an SVM, and we're going to update HMM to help manage the secure memory. I think those patches are already pushed out. At the VM level, they run on the same hardware, and we use Grubb. We built a prototype using Petiboot, figured out how to use Grubb, so we set Petiboot aside for now. SVMs and VMs both get services from the hypervisor, as I indicated earlier. The alt-visor sanitizes everything that goes from the secure virtual machine to the hypervisor. An SVM can share unprotected memory with the hypervisor. It has to. If you couldn't have what we call normal memory and a secure virtual machine, the secure virtual machine wouldn't be able to communicate with anybody, because whenever the hypervisor can look at the secure virtual machine's memory, it only sees it encrypted. So you're sending a packet off to some other remote system. It's been encrypted by the alt-visor, and you don't have the key, so the remote system can't do it. So that says that in order to do IO and things into and out of the alt-visor, you've got to do bounce buffering. And you have to bounce buffer through memory that the hypervisor, and its subsystem can reference. We do not point the hypervisor at memory and secure memory, because it can't reference it. When I say it can't reference it, I'll be blunt. The hardware will not allow it. We have changed. We went through every single subsystem in the power chip, and we looked at them and we analyzed them deeply to determine whether or not they could be made secure by our definition of secure. And every subsystem that we couldn't make secure in the first round cannot reference secure memory. If it tries, you'll get a machine fault. So you have to move data from secure virtual machines to the normal virtual machines. You have to bounce buffer. That's the cost. We create them with new tooling, and the secure VM starts executing as a normal VM, as I've said, over and over. But it executes, this ESM instruction is a syscall instruction level three in power. There used to be syscalls level one and two, one. And now we've introduced level three, which goes up to the alt-visor. A revocation. I talked about this at LSS in Vancouver, but I didn't talk about revocation, so I'll name up in the questions, so I'll just talk about it briefly now. Revocation means disabling an SVM from executing on a machine where it was previously authorized. For all intents and purposes, the SVM is encrypted. I'll get to the format in a minute. Just think of it as an encrypted blob. Obviously not completely encrypted, otherwise it couldn't start as a normal virtual machine, but think of it that way for now. In the case where the SVM is multiply authorized to say, so I told you you had to have all the public keys. So I can take and build this SVM object and I can embed stuff in this encrypted blob inside of it and I can embed keys for more than one machine. So with that, when the SVM starts up, the alt-visor has to look in that list and it's fairly quick to say, well, which one of these am I supposed to be able to decrypt? And if it doesn't see its identity in there, it fails. If it sees its identity in there, it uses the TPM to decrypt that blob, looks at the results, and then proceeds on with the execution. So if you're revoking, you're saying, well, do I want to revoke it on every machine that I've authorized it for or do I want to revoke it on a single machine? So those are the two questions that we're sort of looking at and thinking about. But who is revoking the access? Well, there's really three choices. There's the user of the SVM, the creator of the SVM, or the owner of the infrastructure. Those are like the three primary parties who would want some revocation. And should revocation be reversible? All right. For users, nobody's forcing a user to use an SVM. If they happen to have one and they don't want it anymore, they can just erase it. So that's simple. That's a complete revocation. It's not reversible, but it works. The creator of the SVM, we have a model that exists today in software called the license server model that gives the creator of the SVM the ability to grant and remove licenses on an SVM and to even have a revocable. If I revoke your license because you didn't pay me my fee and you pay the fee, then I can reinstate it. If you write your SVM, and you can do this because the SVM is essentially encrypted so you can embed in secrets and call home or call wherever and say, am I still authorized to run? And if you're authorized to run, you'll run. And if you're not authorized to run, you won't run. And we have integrity protection on our stuff. So if somebody starts tooling bits trying to break it, it won't run anyway. So that revocation is handled. What's not handled gracefully and what we've done is enabling the infrastructure owner to decide I don't want that SVM to run because he can kill it but he kills them all because he changes his public-private key set and that gets rid of all of them. We realize that's fairly heavy-handed so we're looking at various alternatives, all of which will involve some form of a revocation list and therefore will probably not be reversible depending on which way we go. If you have comments or questions on that, we can talk about it later. In the first release, we'll not support suspend and resume and migration. That's to make it easier to get the first release out. And we'll also not support over-committed SVM memory in the first release. Our architecture and design allows the hypervisor to page SVM memory. We're just deferring that till we get it running. Then we'll put that in. And we will not in the first release support dedicated devices to SVMs. That's because in the timeframe in which we had to design the hardware, we couldn't figure out how to make it work well so we said, nope. So you can't dedicate a device to a virtual machine. What can you do? You can do all the virtual IO you want to. We have the virtual IO subsystem for virtual machines. It works well. And we're modifying it to the bounce buffering so that's a modification to the kernel and all the virtual IO devices will be supported. We understand that that's a limitation. It may not be acceptable to some customers and in future releases those are things that we will consider and address. It does not support transaction memory currently. So an application, if it uses transaction memory and it runs in an SVM, it will crash. A few more lower level details. All right. Here's our contents of the ESM blob. In this case, I've illustrated this particular one is encrypted for three different machines, A, B, and C. And there's the symmetric key that decrypts this blue thing wrapped under that's been encrypted with the public key of each of those machines. The verification information contains integrity information for the kernel, the init ram FS, and our tasks. It may contain some symmetric key blobs and it does contain the passphrase for the encrypted file system. Remember I said that your main disk is encrypted. Artulie, make sure you've encrypted it one way. If you haven't encrypted it, we'll encrypt it with whatever we decide is the default encryption method and stick the passphrase in there. This passphrase is stuck in this blob and is inside of the ultravisor. Net, one of the mods we have to make to the kernel is to use an ultra call to get the passphrase so that you can mount your disk and see the contents. Otherwise, you can do whatever you want with it but it will all be encrypted. So, since it's locked up in here, you can't, as the, this came up in the KVM form which is concurrent with this one, you can as KVM play around with ESVM and then emulate it because as KVM you won't be running in secure mode and you won't be able to get the passphrase for the disk out of this blob that's associated with the secure virtual machine. So how do we boot? This picture refers to booting a virtual machine on top of Linux KVM. Sloth is not part of our design. It's what people normally use. Wait, yes. So what happens is Sloth eventually passes control of the grub which is in the prep partition. It is not encrypted. That's why we can start unencrypted. It goes in it and it goes and eventually it does, okay, which one of these kernels do I want to boot? It does its thing. You could in an ESVM have multiple kernels that you wanted to boot. You don't have to have just one. When we had design choices, we aired on the side of flexibility in this case. So you could have just one. You could have more than one. In any case, they'll all be Z-images. Grub boots is the image just fine. So you'll get your Grub menu. You'll have your default. You'll have however much time you've allowed yourself to switch to a different one if you want to. Inside of that Z-image is the blob. So you start booting that. And that gives you a slash boot file system, which is unencrypted, but we have some integrity information on the things that we need to know about that we'll check. And then you have the real root file system, which is encrypted. And I've explained what we do there. So we start as a normal virtual machine. At the end of prominent, we say, switch me to secure mode. You get copied into secure memory. We search for the properly wrapped symmetric key after we copy you into secure memory. If it's not found, execution fails. If it's found, we decrypt the verification information. If decryption fails, if decryption fails, verification fails. Using the verification information to confirm the integrity information of the kernel, the init ramfs, and the R test information. If that is successful, if all that, if the integrity means if the bits have not been modified, then we pass control to the SVM in secure mode and the pass phase is available through the Ultra call. The ESM Ultra call is the only one you can make from a normal virtual machine. It makes it easier. And we have some Ultra calls for the hypervisor, and we check to make sure that if you're coming into the Ultra visor, you're coming in from the right location. All right. So we're going to make this open source. So we've already, in each case, I'm walking through the things that we think we're going to have to change in order to make this work. We had to change prominent around so that this notion of letting you run through prominent unencrypted works as long as prominent doesn't make changes that would cause the integrity verification to fail. Originally it did. These patches change that. So it doesn't do that anymore. So we can do that. There's a wrapper. And so we've made some changes to let the ESM stuff, the ESM blob be added to the Z image that changes up. We're probably going to have to make some form of a change to grub. We're not to grub the core of grub, but to either some of the scripts or some of the things around it. Those changes have not been pushed up, but they will be shortly. I've, this is a summary of what we call, of the ESM calls available from the SVM to the ultra visor. Or from the hyper visor to the ultra visor, scum, read and write scum. Scum is scum registers of the registers in the power architecture that the hyper visor and other entities use to control the configuration of the hardware. The entire machine can be reconfigured, and not the entire machine, but a good part of the machine can be reconfigured using the scum registers. And for those of you who are hardware, you realize this means you could bypass almost any security feature you wanted to. Which is why you can't write a scum register without the ultra visor looking at it and said, do I like this? If I think it's okay, it goes. And if I don't, you get told you were successful, but it doesn't happen.  I guess we're concerned about security. So we can, you can pick up, you can page in and page out. You can write, you can write the partition table. You, UV return is the call that the hyper visor uses is to tell the ultra visor, I want to start this secure virtual machine. We had, we were originally going to not let the hyper visor know that the machine was a secure virtual machine or not, we decided to change that since we didn't think it could really hurt it, so we make it explicit. Registering and unregistering memory slots is about getting and sharing memory. We can, you can terminate us, SVM terminate I think is coming out of the SVM. Share and unshare, oh, sorry. Well, we've got, we've got, in power we have something called VPA and then, and a couple of other things that we have to share with the hyper visor. The shared pages is the share page and unshare page. The memory slots are the other things. And we have the ESM. We also have some things so that secure virtual machines can share memory across from each other, but they're not on this list right now. For KVM, there's some special H calls we needed. We needed start it, finish it, terminate it, page in, page out, and then TPM com is just, we need to add this H call, so the ultra visor will reflect TPM com to the hyper visor when it needs to talk to the TPM and the response will come back to the ultra visor. The ultra visor reflects it to the hyper visor as if it's coming from the SVM, but it's really the ultra visor that's doing it. We have to modify HMM because we want to use HMM.  We're trying to exploit much of what Linux is out there right now. So we've got a proposal set there that will allow HMM to manage moving memory between things between secure and insecure. Of course, the ultra visor will get in and encrypt or decrypt as necessary as those things happen. Oh, I got vert.io here. We have a set of initial patches out for vert.io. This is a set that was designed, but we're debating a simpler set of patches for vert.io that rely on the DMA ops, the DMA ops structure that's in the vert.io that might be simpler than the ones that are out there, and we're working with the vert.io maintainer to zero in on exactly the set of patches. The sorts of things that we've been modifying and looking at modifying are similar to the things at AMBs and looking at modifying. And then we're talking, we have a set of changes out for VPA, which is a power specific thing. Let's see. Now, just to get a little more into the hardware, this gives you the point, this is what we changed at a high level. We added an address bit that indicates the memory is secure and not secure. It's a high order address bit, much higher than any machines that we have exist for a while. We added an MSRS bit that indicates the process is running secure. We added, we now, because we have that, we now have, these are your, what state you're in if you're in secure mode, these are what state you're in if you're not secure, and we have one that's reserved for future capabilities that we're looking at. We added a bunch of new registers like SMF CTRL, which tells the system whether SMF is enabled, hardware is designed so it comes up with SMF enabled. If it gets, if SMF gets disabled, it cannot be re-enabled without rebooting the machine.  We added a bunch of ultra-visor specific registers, URMOR, USSR, USR01, USPRG01, which are similar to the special registers that we make available for hypervisors on power. We added a new instruction, URFID, which the ultra-visor can use, which allows it to return to something and flip on the SBIT, a convenient feature. And, okay, so, and these other things that I mentioned earlier, all the hypercalls go to the ultra-visor, all the interrupts go to the ultra-visor when you're in secure mode, and we reflect stuff to the hypervisor. Just a quick summary. We protect you from the hypervisor or other software system admins. The security domain is the VM at rest and transit or executing. We don't change the applications that run in SVM. We have some new K-config operations. KVM must be para-virtualized. Those patches are being put together. Where your secure memory has integrity and confidentiality protection, you can embed secrets. We're limited by the available memory, and all will be open source. There's a bunch of links to papers and things that are related to this, and do you have any questions? Questions? What? Reference slide. Okay, so one of the things that we have in SCX is the ability for enclave to prove that it runs inside secure enclave and not inside emulated environment. Is there such thing in this system? Well, so that if you have an enclave running on a cloud... Yeah, I know what you're talking about. Basically what we're going to do is be exploiting the TPM for that so that you can know that you're actually talking to a system that has a real TPM that was on an IBM platform and then you can use a remote attestation for that feature. It's similar to the attestation that you have in the enclave because what you've done is you've put in an attestation capability so that the enclave can attest to the user. We're just going to exploit the attestation from the TPM. It's actually equivalent to a hardware chip in a way. Yeah, so we're just exploiting the attestation in the TPM. Questions? You're saying that everything will be open source. Does it mean ultravizer also? Does it mean what? Ultravizer also. Ultravizer will be open source. Okay, great. Could you please, Mikova, Mandra and Siar, could you please tell if there are any key differences from ARM monitor for ultravizer? From what? ARM monitor. ARM monitor? Yes. I'm not that familiar with ARM monitor, so... Right. Let's take that question offline. So does Bova system have protection against somebody installing something to the random access memory bus? Is the secondary memory encrypted? Is there protection against hardware tampering? Okay, so we do not have a hardware encryption on our memory right now. It's fairly difficult to tamper with the hardware memory, but that's something that will be coming, but it won't be in the first release. And probably for power and open power, when we do hardware encryption, it's going to be sort of like what you see in Z-Series. If you're familiar with that, we now have this pervasive encryption in Z-Series. So the encryption of the memory that protects the encrypted memory will be orthogonal. The hardware part of that encryption will be orthogonal, and the ultravizer will still have its own independent encryption. Monty Weissman, General Electric. You mentioned that the VMs are going to have a tunnel down of the physical TPM, so they're going to be sharing the same physical TPM? Is that the proposal? We don't in our... No, we don't have the VMs share the physical TPM. The ultravizer has access to the physical TPM as the hypervisor has access to the physical TPM. The operating system that boots on the hardware has a device driver for the physical TPM. And it has a TSS in it that allows the virtual machines above it to share access. So I guess if the virtual machine is written to access the physical TPM, yes, we'll all be using the same one. But we're not bypassing... There's only one device driver in the entire system. We're not putting another one in. So kind of get to where Yoriko's going then. I mean, the TPM's key hierarchy works very well for virtualization and sharing, but the PCRs do not. So how are you able... If you have these multiple VMs, how are you going to be able to attest to launching each one of these VMs and be able to attest what operating system launched and then kind of a follow-up to that, if they turn on IMA, how do they all share the same... If they're all sharing the same physical TPM, how would they do that? Seems like they have to go to a VTPM. Okay. I'm not sure I've got all of your question. The ultravizer actually won't... The ultravizer gets booted as part of the firmware. And it actually won't start if the PCRs are not in the correct state. So if the PCRs are not in the correct state, in other words, you're trying to boot something else, the secret for that machine is locked permanently. That's... The only other thing we need, the ultravizer needs the TPM for, is using the private key to decrypt things. That's it. We don't ever reference any other PCRs or anything. So as we're coming up, we have a fairly fancy policy that allows us to set a password for access to the private key. And so once on the boot of every SVM, the ultravizer uses... That password is created when the firmware is booting up before you get to the boot of the base operating system. And as part of creating that password, the PCRs that it's based on are extended to so nobody else can do it. And then the ultravizer has that password, and then every time it wants to start an SVM, it asks the TPM to decrypt this blob and hand it back and then keep going. So we're not... We don't care about the PCRs or anything else after that. That's all we use it for. We use it for holding our secret and decryption, and that's it. Okay, so you really do have VTPMs for individual... Yeah, the VTPM is up in KVM. Oh, I see. We're not taking that away. We're not using the VTPM. So each VM has its own VTPM. Every VM, including SVMs, would have their own VTPM. Exactly. But I think that probably addresses Yarkov's question as well. Okay. I think we have to... If there are more questions, maybe there is a break right now so you can come talk to the speaker. So let's thank the speaker.


That's so disgusting, you can't even come out and speak like a whole human being. There might be more different people that you really want to protest around. No. It's been very clear. Wait, Senator, Senator, one question. Have you talked to any of the Americans? I'm a Palestinian. That's not, that's an inhumane view. Being pro-genocide is not a view. We're the staff and he has all the hostages and pictures in there for the Israelis and you know I feel it's very one-sided that he doesn't have anything for the Palestinians so we'd like to either bring him some or ask him why he doesn't. He's shown a lot of disregard for Palestinian lives and especially given that he had been on being a progressive he's let a lot of people down initially supported him we're excited for his candidacy as allies of Palestinians as well as some of us being Palestinian here here to talk to him and let him know that supporting genocide is not okay and that supporting killing of babies is not okay. I would like to also tell Senator Perlman that the majority of American people are asking for a ceasefire that the overwhelming majority of the Democrats are asking for a ceasefire and I know for sure that hundreds of thousands of Arabs and Muslims and friends of the Palestinian people in Pennsylvania are asking for a ceasefire. I'm not sure where he's getting his advice from that's ill-advised not to support a ceasefire at this point. I am Josie. I'm a 17-year veteran of the United States Army also a military intelligence officer. I have a background in understanding what's going on in the Middle East and I'm here to ask you to call for a ceasefire. This is not self-defense. I understand exactly what defense is. This is not self-defense. This is a genocide. The ICJ has ruled it as a plausible genocide. The US support has ruled it as a plausible genocide. There is no hiding. Veterans are waking up. Americans are waking up. We are prepared to take to the streets and continue to protest. We are not going to be silent in the night. We are going to keep talking and keep talking until you call for a ceasefire. Senator Federman, ceasefire now. Ceasefire now. Ceasefire now. Ceasefire now.


I'm Alan Flour from HCL. I'm joined by Thomas Davos. We're here to show you something quite interesting today. This is the first time you would have seen this in our industry. We're at an event called CubeCon. And it's easy to think that in the world of container orchestration platforms, Kubernetes is the only platform that counts. In fact, there are two significant container orchestration platforms in our industry. Kubernetes and Cloud Foundry. How many people here have ever heard of a project called Cloud Foundry? A lot of people. What's happened in our industry over the last two or three years is the gravitational pull towards Kubernetes has been overwhelming. It is now the de facto orchestration platform. An open-shift, of course, is the market's most dominant containerisation platform today. What we're going to show you now is fully automated migration from Cloud Foundry to open-shift. If there is one request that we've heard more than anything else in our industry over the last two years, it's from Cloud Foundry users who are desperate to leave a stagnating legacy platform and move all of their workloads to open-shift. You may be familiar with products like Pivotal Cloud Foundry. You may have heard of products like VMware Tanzu. You may be an IBM Bluemix or Cloud Foundry user. These are all commercial distributions of Cloud Foundry. So what we've built is a fully automated journey to take you away from Cloud Foundry to open-shift. We're announcing today at this show support for all of the major commercial Cloud Foundry implementations, Pivotal Cloud Foundry, VMware Tanzu, and so on, but we also support migration to open-shift from any other Kubernetes platform. This is fully automated, no human intervention migration, which we are going to demo to you. You're going to see a demo where we migrate stateful as well as stateless production workloads from a live Cloud Foundry platform. It's running in our lab today in Dallas, and we're going to show you how we move from Cloud Foundry to open-shift. The open-shift instance we're using is running on Google. This is a fully automated journey. We automate absolutely everything. You only have to make three clicks and we do the work for you. We do complete vulnerability scanning, so we're checking for the consistency and the security of any workloads that come across to open-shift. This is a fully automated journey. The best thing I can say to you is rather than me talk about it, we're going to show you this. It is so straightforward. Thomas, if you'd like to join me and maybe take over and maybe just describe what we're going to show, I think. No, not yet. Now you can hear me. Are we awake? Perfect. That's good. Thank you, Alan, so much for the introduction over here. I would like to demonstrate over here live how we can migrate a Cloud Foundry workload onto open-shift. This is the demo setup I have over here for you. We have the Kubernetes migration platform, KMP, deployed on GCP, first and foremost. We have KMP tags with a couple of agents deployed on a VM, and the purpose of those agents is to keep connectivity going with, in this case, TAS, like Alan was saying in Dallas, into our cloud-native labs. On the right-hand side, you see the target, which is open-shift. I'm going to bring up now a drawing application, which is running on TAS. I'm going to try to move the workload and the data onto open-shift. I can't see this very well. This is the Kubernetes migration platform. I have currently defined two agents, as I earlier said. The first one is the open-shift one, and you can see here the endpoint. The endpoint is connected to open-shift. It does the automatic discovery over here. I also have an endpoint configured here connected to VMware Tanzu. You can see I have over here one application deployed. This is a Spring Boot application. It has a back-end or managed service of my SQL. This is the drawing application I was talking about, and hopefully I will try to bring in my creative skills over here and do some drawing over here. It's a bit more difficult with this mouse, but okay. Hello, KubeCon. Let's try. Anyone else has better artistic skills than myself? I think we get it. All right, okay, perfect, that's good. This is kind of the application. I am storing the state of this fantastic drawing into my SQL. Next, I need to kind of go to my Kubernetes migration platform. I am going to define a plan, and a plan is nothing more than defining what do I migrate from and to. Over here, I give it a friendly name, OpenShift Commons, and I want to migrate all the applications deployed into the default space. In this case, there's only one application to OpenShift. I have my plan created, and the next thing I need to do is just simply click on Migrate. So now the migration has started. The intelligence behind the Kubernetes migration platform has started. What are we doing over here? We are first and foremost discovering what applications we have running. In this case, one application, this is a drawing application. We are getting all the artifacts, we are getting all the binaries, and we are downloading this into a central place. From here, we are creating a new docker file in order to create a container image. The container image is currently being created in the background. Once that has been created, we will store it in Docker Hub. The reason we are storing this in Docker Hub is because we need to create, as well, a set of manifest files to deploy into OpenShift. We are creating those manifest files from the ground up with a reference to the new container image that we have created. We also have discovered that we have a MySQL database. This is a little bit more trickier. We need a request from Client Foundry the endpoint of that managed service. We have discovered that we have a MySQL database. We also have support for Postgres. We also have support for Redis at this moment in time. Once we know what type of database we have, we get all the metadata, database tables. We get also the data out of it. We store it again into a central place. Now we have all the artifacts. Now we can start recreating this in OpenShift. We are having a sync module that syncs the data between that central place into Kubernetes. We have created a new instance into a namespace on Kubernetes, in this case, on OpenShift. Migration is currently happening. The other thing that we are doing is creating a number of Kubernetes manifest files. We are trying to match the storage, networking, endpoints, egress, DNS, all these kind of good things onto OpenShift. We have a lot of intelligence built in there in order to map whatever was there deployed into Client Foundry and also in OpenShift. I suspect that this migration will take about three minutes. The same kind of automatic migration we do as well for Kubernetes or any commercial Kubernetes platform onto OpenShift. Whilst this is running, are there any questions? No? Okay, let's see. There's one question here. Left-hand side at the rear. Tell me? Sorry, I can't hear you very well. We're just migrating the binary. We're getting all the artifacts, the JAR files, the WAR files, whatever has been deployed. We are recreating the original container that was been running into Client Foundry and are we creating a new container in which we store this in Docker Hub and not the source code. We're going to have to migrate the source code manually if that's what you would like to do. I guess that goes into your normal software engineering process anyway. It's completed. Here you can see after three minutes and 19 seconds the migration has been completed successfully. The only proof in the putting over here now is that we go to my OpenShift cluster and see if this one actually happened successfully. Let me open this one. Yes, indeed, I do have now a drawing app deployed into a namespace. Let me have a look at the endpoint over here. Here you go. Application fully migrated, automated in three minutes and 19 seconds. Thank you. It's encrypted into a central repository. That's right. We are here in the coming three days as well at KubeCon. We are at Boot G1 as part of HGL, so if you have any more questions either for the Client Foundry 2 Kubernetes migration or you have further questions around any commercial kind of Kubernetes migration onto OpenShift, we are more than welcome to show this more in detail or if you have more detailed questions, feel free to come over. Thank you so much.


We are live except oh no, we're not we're recording right now Well, we are recording. I just yeah, we're live wait with Okay It's what 11 to November 2nd already Wow So We've been working away on getting the skills into their individual shapes And we're starting to think about the skill interaction stuff and Along the way we've noticed that our Wi-Fi setup process Which we've known for a while is is a little buggy Wasn't wasn't really finished, but we thought it was good enough turns out. It's not good enough So we're gonna take a step back and make that actually work properly. So that's kind of where I see you see us overall right now in the meantime, I believe we've started to solicit feedback from the community about at least some elements of the skill interaction Thoughts that we're we're having and So looking forward to that Well, actually I've already seen some of the feedback on Derek's videos. So that's good and We will I think we'll keep producing videos like that in terms of you know Setting expectations for what how we want the system to work There's also some additional Documentation that we're working on I don't know where we are in the process of sharing that with the community But we've got some some ideas about how to Start to add a little bit more formality to the message-dress messages and the things that are happening within the system So that it's a little bit more Easy to monitor and keep track of what's going on Example that we went over in Hawaii to that document just to have something How it works and then we'll probably be able to pass it on the community. Okay good Yeah, so and just in terms of like high level where are we I think I think that's it And so why don't you guys take it away in terms of individual statuses?  I'm the only one not muted. So yeah, sure. Um Today I was a little bit of everything. I had an interview with the candidate. I Responded to some PR Comments gas had on my latest PR for the GUI for the Wi-Fi skill and I Fixed I've already just met a PR that fixes the skill date On the Device when you when you have a dev kit my new methodology is I found out that the Mycroft skills repo is actually checked out on the device MSM uses it I've checked out cloned on the device and I found where it is and I'm using Get logs to find the most recent commit on that dirt on that repository and Get the time out for that and then that's the time Okay, that'll be great at least to determine a stick Yes, everything should be the same for everybody this way So yeah, I will submit that PR today and it's only a few lines of code So guys of you I'll probably Throw that your way if you look at it tonight pretty straightforward that was it tomorrow and Tomorrow I'm gonna get back to working on that document. We want to share the community about the organizing the code And then hopefully get going through with that and then a day or two and we'll be able to share that out and I'll start working on that Okay, great Ken. Oh wait, no, sorry, sorry Ken Derek. I forgot Derrick's Yeah, okay So yeah, on the software side we you know, we met with Lucissus this morning or we sink There's a couple things where there's at least one thing there and you take a look at this The load there's a loading animation Kind of issue where we've got all different types of loading animations. We've created through getting it. It's kind of confusing So I was gonna create the ideal You know what we'd like to see and see if they can help us get that sorted out Let's see I did do I had an issue where weather does not seem to be working on my device So I log a bug on that It's not on the home screen and it's does not No intense pick up whether it just goes straight to Wikipedia and gives you the definition of weather so something's going on there The The other thing that I saw in a couple things in the chair that I need to do I would Like to split out the mute functionality, I think into its own ticket I think Michael suggests to maybe doing that for the home screen Just so that we can You know, it's got a couple things I'd like to do with it, you know, I'd like you know, for example the LEDs to show Something as well as on the screen But we've talked about this new edge, you know, both Chris mentioned this to the new edge illumination concept being a good consistent way to show that as well So I think there's just enough there that we could just maybe split it off to its own ticket Yeah, that makes perfect sense. Just read a note that we're changing the scope of you know a comment in that ticket where yeah, you know signing off on splitting the new functionality into a new new task and Then of course, there's the big Wi-Fi process That I actually have just read I don't know you work on it. Yeah, but I know that's that's gonna take some some thinking In terms of the UI the gooey side But yeah, mostly today is still kind of working on hardware stuff Kind of unpacking we had a meeting Aztec last night and so I unpacking Some stuff from that meeting and and starting to plan Did I miss something on the Wi-Fi skill? Why are we changing that UI design Well, these would be things that we're we would need to add based on our conversation At the summit so things like oh, okay, what happens when connectivity is lost what happens? Okay when You know things like that like so this would be new stuff. Okay, that makes more sense. Thank you. Yeah You've got everything in there that was currently defined. So So yeah, a lot a lot of hardware stuff today, but that's been good. We're getting some plans laid out for that All right, cool. All right, Ken So, Michael, did you see you were called out in the dev channel? There's some feedback for you in there Oh, let's check it out So I was working on the NTP stuff and I was in the enclosure. I was actually moving the enclosure up so that it comes up right after the bus But there's really no way to synchronize that because there's services right now But anyway, I tried to change the date And realize that you can't change the date or time on the mark to I suspect container permission issues So I've reached out Panicor on that I'm Trying to switch networks from our container using the debuffs. I realized I'm gonna have to replicate the whole access point stuff. So I've been working on figuring out how to I mean, I it's pretty easy to get the access point up and everything but Figuring out how to get the login profile set up so it fits out each TML and you hit it with Not any authentication and that kind of stuff. That's where I was at and then that was like the first half of the day and then Yeah All right, so who hasn't done status yet there I went first Yes, I did a hell of a lot of translation stuff. So apologies to everyone's inboxes and courtesy of Goldifruit Who does some excellent Docker? Images of my craft He did sir. He was doing some benchmarking of the Yobos precise light plug-in against, you know precise you know standard and You know as you expect it's it's Significantly low resource. So I asked him to do the same for for the for the my craft one Which you know, he didn't know about because we haven't really told many people about it And what became apparent there is that Yeah, as I said in the chat like where we're starting off at like, you know, super low like 8% CPU usage But it's just like slowly creeping up over the course of the first 30 40 minutes and Which makes sense as to why the the service eventually just crashes because it's you know eating too much stuff and Can't handle it. So there's something in there that's You know some reinforcement loop that's Out of hand that we need to find and and nicks So yes We can either do that or we can Which seems like it still seems like a better approach to To only run the TF light runtime as Ken, you know Did all the work on on doing that Rather than, you know, the the OVOS way is to have like the the precise traditional method of having the separate runner and and everything So this should theoretically be be even leaner, but I'm gonna wait and see. Yeah, we've got a memory or a process leak or something like that. Yeah. Yeah, okay Yeah, I suspect the memory leak. I'll get to that one Other than that just lots of lots of things Yeah, I am trying to to use the new PR processing project board public project board and push that out to some of the community channels to To let people have a look at that We have a lot of outstanding PRs in Microsoft, so It's no small feat to like Try and get that backlog down But yeah, the hope is that this will particularly for things that are like current and ready to actually merge that they won't slip through the cracks and disappear that will kind of keep on top of the front of the queue and clearing that out, you know reduces noise and hopefully that aids processing of the remainder of the queue as well Yeah, we'll take a Yeah, we need to keep on top of that. This is kind of a new process for us. So Definitely want to get better at Setting the expectations with the community about, you know, what we're working on what our priorities are you know and and Where we're looking for help and you know and where we're not looking for help, frankly, you know There's there's I think that you know the the plug-in process that was started a few years ago I think is something that we definitely want to need to build on and and I think that, you know We want to move in a direction where You know a lot of the things I think that, you know, some of our community members want to contribute Maybe don't belong in core, right? Maybe they belong in a plug-in and you know, if the architecture doesn't support the right kind of plug-in for those things then Then that's what I would rather work on rather than Try to build more functionality in the core that's not necessarily You know, if it's not necessarily where that should go. Yeah, so in my opinion core is already, you know, kind of a nightmare and You know, we've had plans for you know a year and a half to Untangle some of that mess, but we just frankly have we just barely That are preventing us from accepting PRs in a timely way, but But you know, I'm adamant to be frank that we're not just going to accept all PRs because it creates a you know a bigger test burden on us and it creates You know uncertainty around you know performance and things like that so You know, we need to be pretty measured about how we accept certain kinds of submissions at this point Because we're trying to take my crop from a place where it's you know, it's a great Development tool and lots of people use it to something that is production ready and it's a whole different set of requirements so You know, so We can't just accept everything without without a fair bit of scrutiny, so Communicate that message through the forums recently as well Certain things but you know particularly if it like you know, there's a The XDG Skills PR, you know, there's some more stuff that needs to happen to make Microskills kit use XDG paths for example and Yeah, like I still believe that you know becoming XDG compatible is good for the long-term prospects of my crops across different things But it it doesn't and that was already happening there. We shouldn't be leave that half done We shouldn't yeah. Yeah, we want to follow through with that if we started close that out for sure Yeah, it's just you know spending time working on Us working on MSK improvements to make XDG work does not in any way get us towards the mark to shipping so So yeah, they communicated that and hopefully a good way and Yeah, hopefully we we can get that moving with some community with some more community support We'll see yeah, exactly Yeah, I mean if it weren't for the pandemic, I think we would have had a completely different Forum experience. This is something that we've we wanted to do last year and again this year, but It's just not possible to get everybody together for you know a convention style Meetup and and hash out a lot of these issues, which you know internally we've found to be extremely valuable even just with our small team So I think you know I'm I'm trying to think about you know, when's the right time to To have something like that even if it's just a virtual event. Yeah, that's useful. Yeah, totally All right Well any other news people want to share? All right Well, let's make sure we are very clear with our communication on the Wi-Fi stuff You know we did spend a lot of time hashing out through requirements and making sure that You know we can have a well-defined boot-up sequence and And from there it will it will be a lot easier to define Once that's working it'll be a lot easier for us to define what happens when we lose connectivity because we'll have you know We'll have a well-defined system for Responding to connectivity loss internet loss You know and and from there we can you know add in the the paths we need to Trigger like changing Wi-Fi networks or whatever if we want to go that way So let's let's be clear on communicating communicating that over to panic or and You know, we need to measure measured way, but you know his document, right? What's that? Which is simply Chris V? Providing them with his document, correct? Yeah, but I think we need to engage in a dialogue with them as well The document will define will define the steps as all it does. So, yeah, we're you know, I'm sure I will be some Some further communication need to happen after they read that and try to get digested Yeah, I mean there's there's what we need on our side in order to get things to work But then there's the okay Well, how are we gonna implement it if they were gonna implement a message bus thing that would just send us the specific messages We require then we'd be done right And the document assumes that there's a message bus being used on their side, right? So, you know, if they if they can't go that way or are willing to for whatever reason then, you know We need to figure out. Okay. What's the implementation path? Is it building a translator for the D bus over to the message bus on our side? You know, how do we solve Ken's NTP? You know time-setting problem things like that. So So I think we should talk through the whole process with them maybe we even need to set up a Conference call to go over the document after they've had a couple days to look over it I was kind of hoping it would just be you know Here's a here's a set of requirements throw it over the wall and it magically happens, you know But if it's not that then let's get together and instead of you know going back and forth with you know Text messages are you know in my opinion a terrible way to communicate Most things So let's get together and have a face-to-face, you know After they've had a chance to you know digest the document so All right, so Chris you're gonna take a lead on that I'll lead on which I am digging lead on getting the document to them Yeah, and set up a communication because it's really between you and Ken and them Yeah, all right cool And Yeah, and if we do you know if we do these one step at a time it should be pretty straightforward and shouldn't you know, hopefully it will not You know we won't be tripping over ourselves as we do it so All right, thanks guys One other thing real quick I wanted to know if I want another minor version of core Oh, okay Mostly because there's a bunch of testy stuff in there that is going to start holding up a lot of our skill work From making it into the marketplace, which it should be getting its way into for too long So any I know I know I just when I say that I feel back if I say that I'm basically putting the hours of work on Gaza's plate, but I I Do think we're at a point now. It's probably worth doing so we don't hold up all this stuff Getting in the marketplace Anybody else have any thoughts on that? Well, my my thought is if you have specific things that are being held up You know that need to get out to the mark twos then absolutely we should do that But you know, I don't think we should do it just because Maybe sometime in the future it'll be useful, you know, so I'm not sure where exactly you're at It will like it is you know because There to get things in the marketplace they need to pass all the all the testing and so because these test steps aren't available in In production my craft then they won't pass and therefore they you know If we put them in a marketplace, then everything will just fail forever more Yeah, I think I think it is a good thing to do Is there anything else is it particularly VK wise that we're thinking will be like that we have on the carts because if there Is anything else we Should try and get that in before we cut the release. I haven't touched VK since our we finished our skill sprint I don't anything There's nothing else that's glaringly Breaking or you know in need I think I can think of but I also think we'll have you know, we'll have a major revision here soon and So maybe you should just go yeah, but I think if someone else comes up, maybe we just you know draw a line in the same Yeah Yeah Cool, okay So new new minor release eminence them. Yep Okay, they'll tell me more stuff in there and just to speak a stuff because there have been things committed to them, but oh Yeah I might do I might get the mark twos updated to 21 Dot two dot one proper before we do that So we can all test it. Yeah, and so that way we can we can Get back to at least you know pegging the stable releases But yeah, I'll add a ticket to cut a new minor release All right, cool. Well then All right, it just occurs to me that the ticket I've got assigned to can maybe you should be reassigned at Chris if he's I think I misunderstood who was in contact with Pantacore Regarding the the Wi-Fi changes so If you guys sort that out Well, it sounds like I've done I'm defining them and Ken's implementing them. What kind of what it sounds like right now. Oh well Well, yeah, exactly. We have have the super high-level vision of what we're looking for It's really simple I mean that the high-level vision is very simple. We we have a well-defined signal that does not report that The internet is available until The internet has actually been reached and the time has been synchronized using NTP. Yeah, that's it But so, you know, what would the what is the on-screen experience look like? if the Yeah, anyway, because there's the there's the whole like on-screen Wi-Fi selection, you know Stuff all that seems fine. There was a couple little tweaks we wanted to make Just in terms of the user experience like the user not knowing like oh well You know when you when you type in your SSID you select an SSID and you put in your password It just kind of goes dark and doesn't tell you what's happening after that like that's a little tweaks Yeah, you're talking about the instructional UI like there is it there is also an entire UI for like Wi-Fi network management like on the screen and You know, I mean on the wait on the mark to screen or on the on the mark to screen Yeah, we should probably talk about this and get it out what Derek was talking about so Yeah, it's two ways Connect right there's the way we do it now and then there's a way you could do it using the soft keyboard and so Derek had the grand vision of Merging them and incorporating them somehow And somehow that's crept into this discussion No, that's not part of this discussion. Well, that's that's what I'm saying is like, you know I that is not part of the discussion as far as I'm I'm aware That's I agree right so then if that's not part of the discussion, what do we discuss what we're discussing is the ability to recover From network failures. That's step two step step one Is to be able to know where we are in our network establishment process so we can order our boot sequence Way we want right then step to deal with connectivity loss. Yeah, in other words recovery from fatal Correct. So that's it from a high level. There's step one step. That's what we're trying to accomplish step three problem So all right Well and step four step three step something step n plus one is You know is where I see this the on-screen stuff coming back in is like Where you know imagine I'm connected to a network and I want to change networks Like, you know, what do I do? do I that's That's part of that's that will be handled as part of the I lost my internet connection. That's that's defined in the same ticket Well, I Mean Yes, they are different. They're different functional requirements, but they rely on the same Underlying processes. So maybe we could split it into two tickets, but it's in the ticket right now as I defined it Well, you know, you lose if the user can Choose a new Wire wireless network because they lost the old network then why not let them choose a new wireless network because they want to right? So they're they're both both of those functions are defined in there Just we don't want to get into a tail chasing where we say well We're in step two and we could solve that here, but let's not because in step four. We're gonna solve that No, and I definitely don't want to I'm definitely not saying that I'm like I'm saying the step four is is Next project in the future like step one. Yeah, I mean other important right now Yeah, and they're those those are ones that are necessary just for the basic user experience, right? Yeah, if we had an on-screen display and you didn't have to use your phone to Typing your SSID and password and stuff that'd be a better experience on the mark, too, but it's not a scalable you know and You know for devices that for example don't have a display and so we're just you know, we're trying to solve You know the more general problem first Right, and I think it's all began because of Derek's input regarding well What happens after I push connect on my phone? Because when we were out there doing the focus groups, it was clear that people were lost They didn't know if something else was being expected of them Whether they needed to do something else whether it worked or whether it didn't how did they know if it worked It didn't and so then what happened was that was this intermediate Status messages during that process that were defined that would enable us Give better messaging to the user during that process. That's how this all got kicked off Well, and right and also correct the error that was preventing them from actually, you know getting to the next step You know So, you know looking at your mark to and saying well, I don't know it doesn't seem to be doing anything I guess I'll power cycle it and then hey it magically works. That's a crappy user experience So, you know, yeah, I guess I'm just flagging this, you know Step four and beyond stuff because like that's already been communicated to Pentacore as well And so like if we're then coming to them talking about Wi-Fi management Without the context of the other stuff, then they might also be like what the hell are you guys talking about you told us something completely different before you know Yeah, this is this is not this has nothing to do with on-screen Wi-Fi setup stuff at all so and in fact it in you know In the perfect world for me It wouldn't even be It was certainly wouldn't be implemented through, you know the pull-down menu as it is right now It would be actually just through the skill and be a skill graphical interface for that But we'll get there later like I think you know the thing that we talked about I mean originally it just showed up on boot and like it had the Wi-Fi setup You know on the screen I know but I think I think we should be able to boot all the way up into Running on the device without Wi-Fi at all even if you require Wi-Fi for TTS like the device should boot and at least tell you Hey, I need a Wi-Fi, right? There's no reason why it should go into like a weird state. It should just be you know Should be interactive in some way And so we should be able to use skills and things like that to do things like set up your Wi-Fi and whatnot Now I don't want to re-implement the AW Connect just to do it For now, but you know ultimately we kind of we we already have that because we We boot the entire system and then run the Wi-Fi Connect skill Okay, show the instructions. So like Yeah, well, we should be able to do that without errors getting thrown. Yes Yeah, you know Without the TTS engine, you know throwing up all over the place because like it doesn't have an internet connection Whatever so anyway, we'll get there Okay, I think we've beaten this horse to death So Chris you're gonna get in touch with Panicor and we'll chat tomorrow. Maybe they can deal with it tonight. All right, all right