I appreciate that everybody. My name is Luke Hines and we're going to be doing a talk on a new project called Keyline. So I'll go into the ins and outs of what Keyline is and what it covers. So a quick intro. I'm a security engineer. I work at Red Hat in the Office of Technology. I've been around security for a good number of years now, mainly tall in and vulnerability management and research. So I recently joined the Product Security Committee for Kubernetes, but I've done stuff in open stack and open daylight and various open source security committees. I live in Wiltshire in the UK. It's a rural area. It's mainly software developers and farmers. That's about as much as you get there. I'm a keen runner as well, so if anybody wants to talk running, I could probably talk about that all night. So let's define the problem. What is it that we're actually looking at here? Because without a problem you can't really have a solution, can you? So essentially we're talking about remote IoT trust. Now the key take from here is the remote part. So with IoT, consider that devices are often in physically easily accessible places. So they can be tampered with. They're not within a secure place like a data centre or an office. A lot of the time we might be talking about units that are up in the roof or outside and situated in areas where it's very difficult to maintain a level of physical security. And from that physical security, the actual software security itself because people can access that device. So if we consider the security, I want to see if I can turn this round because I can't see my own slides. OK, so first of all we have physical protections. So if we have a unit, it's not very economical to have a human watch over it all the time. So security guards, CCTV, all of these things are not going to work. There is various types of tamper prevention that you can use. So adhesives can be used to seal the box. You get special security bits, torcs. These are like screwdrivers. Typically, I don't know whether this is a universal term, but we have a Philips and a flat head. And they have these sort of torcs where there are these variations of screw heads. Again, really, they're not really going to provide any decent level of security. And then you have tamper monitoring, which is effectively where there's trip switches. So if you open a device, it will trigger off an alert. Now the problem with physical protection schemes is it's a gamercat and mouse. So eventually somebody's going to get hold of it. There's going to be a design flaw that's not originally anticipated. And somebody's going to be able to open the unit and bypass the security protections that you have. And if we look at safe breaking, the history of safe breaking, it's just a continuous cycle of more security and then somebody circumvents that security. Then we also have software security. So we have mandatory access control discretionary. So we have our standard permissions, SC Linux or APARMA labels. And we can have cryptographic assurity. So we can sign objects. We can look at the hash and then compare it and then make a decision that somebody's not tampered with that particular hash to suggest us trust date.  And there's many. I mean, we could go on all day about the various sort of software security solutions that there are. But again, the problem is the software trust is, it resides either in the memory or in the disk. So there's a private key that will be resident in the memory or be in a disk. And so that can be accessed. It can be tampered. It can be spoofed and so forth. And then the other aspect is you're at the mercy of the lower levels of the stack. So what do I mean by that? If we take, for example, that we're talking about a remote device here, essentially you have a chain of trust from the firmware, the bootloader or the shim.  So again, if we think about, we've got a device that we've remotely instantiated and provisioned, you can really only see the latter stages of that instantiation. So you can log in with a shell session, you can move around, you can change accounts. But you can't really get an implicit level of trust in the lower levels of the stack itself. You have to kind of make a loose level of trust in that. So this brings us on to a hardware root of trust, aka trusted platform modules. Now I'm not really going to go into the deep ins and outs of TPMs. There's probably a few people here that may well know them even better than I do. So we're just going to do a 101 just to bring those up to speed that are new to the technology. So it's a specialized chip. It's not a crypto accelerator. It's just has a very simple engine that can perform certain operations such as signing and hashing and so forth. It has an RSA key pair that it's inaccessible to software. It's only a particular bus that can connect to perform requests for various operations. There's actually multiple keys. There's an intent to T key. There's a station key and you can create your own keys from that. But essentially the key part to consider here is that the private counterpart is siloed and is within that chip and it can't be accessed. The TPM is able to hash critical sections of firmware and software. So by hash we're talking effectively just creating a cryptographic hash to show the state of that particular object. But the extra part that a TPM can do is it can make those hashes public and it will sign them with the private key, which remember is physically inaccessible within the TPM chip itself. So using the public key you can verify that the hash list has been signed by a TPM and it's not been tampered with. Because obviously if it's been tampered then it would break the cryptography. So that's effectively something that we call an event log. And with this event log being made public you can then do something called remote attestation. So you can remotely, outside of the device, look at the hash measurements, be sure that a TPM actually signed that hash list and then verify the integrity of the system remotely. So some of the usages for a TPM, it's quite commonly used for disk encryption. So if a device was removed for example from a laptop the TPM would no longer be present so he wouldn't be able to get the data from the drive. It's also used for password key protection, machine identification. It's been used in gaming to stop people cheating and then as said earlier platform integrity, which is what we're interested in here, that's a particular use of the TPM that we're using in KeyLine. Okay so what is KeyLine? So KeyLine is a project that provides open source remote attestation. It was originally devised in MIT Lincoln Labs, they have a security team there. They came up with the white paper, the cryptographic relations that we used and they put together some early code as a proof of concept. And since then it's become an open source project which various different people are working on. So KeyLine provides a measured boot. So again the TPM will measure various artefacts such as the firmware, the bootloader, your init ramifest, your kernel, your modules and then that list of hashes is made public and then you can verify that nobody has tampered with any of those particular objects. Now there's a bit more to it than that. There's an extend operation, it's a one-way hash so hashes are concatenated together and then they're rehashed. And this provides a level of assurity because it's very difficult to go back on a one-way function. It also provides remote runtime attestation. So in the kernel there's integrity measurement architecture. And it's been something in the kernel since I think Linux 2.6, kernel 2.6. And IMA is effectively it's a subsystem where whenever a certain action occurs and a system call happens, so I have here BPRM, M-MAP, SE Linux labels are changed, files are executed, there's various policies that you can set up. And what will happen is when that occurs the hash will be captured, it will be put into a security FS and then the TPM will perform an extend on that hash so it will build up that cryptographic hash tree again. And one of the things we do with keyline is we can continuously monitor that list as it's populated and compare it to a white list of values that we consider good. So we have like a golden state that we want the system to be in. And we can then tell if somebody is remotely executed something that we don't have white listed or they've changed something or they've changed an SE Linux label, then we can know within a matter of seconds that the machine is being compromised and then start to take an action. We also can do encrypted payloads. So what we do here is effectively we monitor the machine so we can monitor the boot, monitor the runtime with IMA and then if the cryptographic state of that machine is as we expect it to be, then we can unload an encrypted payload on that machine. So that could be effectively, that could be some certificates, some secrets, perhaps a config file that has database, password strings, any sort of sensitive data. And we also have a revocation framework. So when a node fails a state, there's a series of actions that you can take afterwards. And I'll go into those a bit more. We've got some more details on those. So this is the architecture. So the way to consider this is that over to your right, we effectively have the remote domain. So this is where our IoT devices. And then over to your left, we have on-premise. So these would be systems that you have within your local control, perhaps within your home network. First of all, we have the verifier. So the verifier continuously monitors the integrity state of the agent which runs on the actual node that we wish to monitor. This is where all of the cryptographic verification happens for the boot and runtime and so forth. And for anybody interested in TPMs, this continuously polls a TPM quote which is where effectively we request a quote from the TPM on the current cryptographic state of the machine. We have a register where we keep the public keys that the TPM manufacturers provide that can be used to attest the signatures that we get back. And we also register the agent there. We have a simple database. And then we have a revocation service which is the framework where we can kick off specific events that occur should a node fail its integrity state. And then we can also interface with a certificate authority. So, for example, CFSSL. So, again, if a machine fails its state then we could revoke a certificate which in turn would perhaps strip down all the TLS or IPsec connections.  Some of you may have noticed in the middle it says HTTP. That's intentionally put there because we don't have any secrets at all that pass across this connection. So this can actually be open because it's effectively all that comes back as a TPM quote with a nonce. And if somebody tried to tamper with that it would break the cryptography and the node would be seen as failed. So we have no reason to protect that connection at all. Okay, so it can be deployed in many different architectural models. So a single site to a single node or device. A single site to many, many, many different devices or nodes. A multi-site, multi-node so you could connect multiple data centres together and then have a whole many, many relationship of nodes. It works very well within multi-tenant, within a cloud scenario where I would be a cloud consumer. I would have a workload that has a sensitive angle to it and I can effectively ask a cloud provider can I trust your hardware so I can remotely test their hardware. If I can trust it then I can schedule my load to execute on that particular hypervisor. And multiple can attest a single node so you could effectively have a user attest a machine and then the provider of that machine attest the machine as well. Okay, so let's have a look at some of the use cases. So the first one is where we actually bootstrap the machine and then we tag on, it's not mandatory but we're going to tag on an encrypted payload. Okay, so again we have our, at the bottom we've got our machine that we're monitoring and this is running the key lime agent. And this has a TPM chip and then at the top we have the verifier which performs the cryptographic verification and then we have the register where we register the node and we keep the public TPM keys. And then over to the far right here we have our user and they're going to use something called the key lime talent which is effectively just a CLI application that we provide but there's REST APIs for all of this as well so you could develop your own system to integrate with key lime. And what's going to happen is using this key lime talent the user is going to create a key just check on my keys. Okay, I don't know why. I've bought it out of full screen mode because my keys disappeared, I've got my keys back. Right, so effectively what's going to happen is we're going to create a key and this is called the bootstrap key and this is going to be cryptographically split into two pieces. Now we're going to call these v and u. First of all we're going to delegate to the verifier that we want to monitor the integrity of a machine. So we're going to send them the V counterpart of the key that's been split into two. Then we're going to send the u half to the device itself which is running the key lime agent. We're then going to ask the verifier to perform an integrity check so it's going to perform a TPM quote onto the agent which is then going to communicate to the TPM itself using a TPM2 software stack. What's going to happen is if that integrity is shown to be sound in that we know that nothing is being tampered with it cryptographically checks out then the second part of the key is going to be provided to the agent who can then put these together and then has the bootstrap key to be able to unencrypt the payload. Now this payload it could be something that we send over the wire or it could be stored within an OS image so it could be embedded into the image itself whether that be an ISO, QCAL2 or whatever format you have. So the keys are recombined by the agent and then it's able to execute the payload and safely have the secrets delivered to the device. So for example if this device failed the verification then the counterpart of the key is not going to be made available to the agent and the agent is not going to be able to do anything with that information so effectively if somebody hacked the machine they're not going to be able to get hold of your secrets. So that's the first use case that we have which is an encrypted payload. The second one is what I spoke about earlier which is continuous remote attestation. So for this one what we have is a whitelist let me give you an example so you can see what I'm talking about. So this is a whitelist, it's pretty simple we've got a hash and then we have the POSIX path to the file. This was generated by unarchiving an init ramfs of a stock operating system. So as I said we've got the hash and the file itself. This is IMA, the IMA Integrity Measurement Architecture. It stores this on SecurityFS and again there's a few more labels but effectively we've got a hash and then we have the POSIX path to the object that's been measured. Now what happens is every time a syscall is made a various syscall the IMA will update this list so it will generate a new hash but that hash will be created and extended by the TPM so you have a hardware root of trust. Now using these two we have a point of comparison between the current state of the machine and our whitelist, the state that we expect the machine to be in. So as we can see again we have the keyline tenant our CLI application and we've got the whitelist. So what we do is we send the whitelist to the verifier. Remember the verifier is on premise. We're not sending it out into the dangerous world where the agent resides. So we send this to the verifier. The verifier generates some nonce so that we don't have any susceptibility to replay attacks and so forth and then it performs a TPM quote to the device and the device will return the quote and then the verifier will perform that cryptographic comparison differential between the expected state and the current state. Now if that fails then there's a series of actions that we can take which we'll look at next. But just to refresh we have the whitelist which is the golden state. We have IMA which populates a list based on the current state which is updated in real time every time an event occurs and key line remotely attests the system state IMA against the golden state and that happens continuously. We typically work around it's a configurable interval but our poll is, I think the default is every two seconds and one verify can do that to thousands of devices so it's very light traffic we're talking it's a very small get request and we did do some benchmarking and we took it up to 2,000 devices against one verifier. Okay so we look at the verification framework so what we're talking about here is effectively where a device fails its trust state. Again we have the verifier at the top and then at the bottom we have the key line agents and the difference is this connects into a certificate authority and then what happens when the device fails its state the verifier will send out a revocation event to all of the other nodes that are alongside the failed node and it will tell it to perform some local actions we'll look at what local actions are in the next slide and at the same time should a node fail remember we had that bootstrap key that we created that could be part of a certificate authority so a request could be made into a certificate authority to revoke the certificate so then if you built up a TLS structure or some IPsec tunnels based on that certificate authority that node would then effectively be cut off. Okay so let's just have a little bit more of a deep dive on what the revocation framework is so it's a custom framework so you can effectively you can come up with anything I mean the world's your oyster if you can write some simple bash or some Python you script it yourself and as I say you know anything that you can script can be kicked off locally on the machines so a good example would be a node fails the verifier sends out a revocation event which is signed so that you know that it's the actual verifier that's generated that revocation event and you could tell all of the local machines apart from the failed machine this machine has failed knock out its entry from authorized keys that's something that will be relatively simple to script another one would be again a node fails and the verifier calls your certificate authority to make a certificate revocation and again the example that I used earlier this would invalidate all of the TLS connections and then effectively you'd cut the node off because it's been compromised okay a little bit more about the project itself so we've had some really nice organic growth people have found the project from various Google searches and they've come along they've shown an interest and they've actually turned into people that are contributed so we're a kind of a multi vendor project there's a couple of independent developers that have come along and started working on the project as well so as you can see we've this is a kind of like an auto generation metric of a github project's current state so as you can see there's an increase in year on year commits we've got a young but established code base developed by a large development team the first commit was made in October 2016 which is when MIT uploaded the code to their github repository and then it says the most recent one was an hour ago we're very welcome in and friendly to people that want to come along and get involved you don't have to be a security expert you don't have to even be a developer we need people that just try out the solution tell us how to improve it perhaps write documentations we're very open to any help that we get we do mark issues as good first issues and this has attracted some people to the project and we have a contributions guide which will tell you exactly how to make a poor request what sort of issues we need help with and we are a friendly community I like to think we're a welcoming community since the project was opened by MIT we've worked on trying to have as much continuous integration automation testing as we can so every time somebody makes a patch we instantiate a container which runs a TPM emulator and then all of those scenarios that I've just outlined where a machine is a tested and we fail and so forth we carry out functional equivalents of those tests within a container if the test fails then Travis will tell the poor request that is failed and the developer will need to fix it make a git commit amend send up the patch again and we'll retest it we also are in the midst of doing the same with all of our upstream dependencies so we use the TPM to software stack there's a software stack, a set of tools and a resource manager that they develop and they have various versions and of course their master branch so we're going to start testing against that so that we can capture any breakages as they happen we have all our documentation that's auto generated by Sphinx and is rendered on to read the docs and we're going to also set up build testing so as far as coverage at the moment we're working on a lot of the standard Linux distributions so Fedora, Ubuntu, Debian, Raspberry Pi, Buster release has recently been verified and we're also open to working with as many distributions that we can so if somebody came forward and they wanted to get it working on X we as a community would support them we'll discuss things over an issue we'll troubleshoot it with you and so forth so we currently have weekly meetings every meeting the agenda is a GitHub issue that way we can easily link into issues and pull requests and do an at a sign to a particular person and so forth and that helps us easily track everything and we also meet once a week Wednesday 1500 UTC we have a Gitter channel where we meet and we discuss the various items that we need to go through and this Gitter channel is also, it's there 24-7 so if somebody was trying to achieve something with a system or something didn't work you'd be able to jump in there quite a few of the core developers hang out on that channel and you'd be able to get support so that's the other key thing that we wanted for this project really was that when new people come along they try to get it to work something blows up, they get an exception they have support people that will rally around and help them to get that to work so what's coming next at the moment we're working on VTPM support so a VTPM is a virtual TPM so effectively a virtual TPM could be rendered within a virtual machine or a container now the key thing with a virtual TPM you don't have that hardware root of trust so effectively the keys for the VTPM are stored within the disk but VTPMs are very useful because you can create them at scale and every container or virtual machine can run its own local TPM so one of the things we've been looking at doing is extending that hardware trust from the hardware TPM into the virtual TPM so at the moment we're working on this with Boston University and one of the professors from MIT that originally worked on the project and we're effectively going to pool all of the quotes together build them into a Merkle tree and then have one central hardware request quote request to the TPM which will then effectively allow us to pool a quote of thousands and thousands of devices into a single operation because one of the things with hardware TPMs they're not designed for handling multiple requests they typically expect to just work with one piece of metal machine themselves so this would allow us to have mass scale and it extends the cryptographic trust from the hardware TPM to the VTPM the other thing we're working on at the moment is KeyLime is developed in Python and we're porting the agent to Rust so the agent is the part that runs on the remote machine so that's in a more hostile environment and we're going for Rust quite simply because it's very performant there's no runtime garbage collection and the security Rust has a very strict compiler around memory safety and thread safety so you tend to deal with a lot of your security technical debt when you're actually trying to get your code to compile so last of all just to round up you know we're a young project we're looking for people to get involved anybody you know you don't have to be a security crypto expert or even a developer architects users people that have got TPMs people that are producing their own boards anybody we welcome you to come along and get involved we have a website where you can get hold of everything easily like our documentation find out where the repositories are how to get a simple system up and running and then as mentioned earlier we have a community chat channel where you can jump in 24-7 and ask any questions so talking of questions we have a few minutes should anybody have anything they'd like to ask sure you're probably going to need a mic is this one good? one, two thanks for the talk the connection to the TPM is usually via ISCA-C on the board so that would be subject to local attacks is there some way to authenticate that if you have a trusted boot or something on my SOC this part is looked after by the TPM software stack so they have a built in a session authentication system so we're quite lucky what we do is we effectively wrap around the command set that they provide and that allows us to put in commands to create a session and they've got lots of things in there to prevent dictionary attacks and replay attacks and stuff like that so luckily that because we sit on top of a quite actively developed stack that provides a resource manager a set of tools and a software stack itself the TSS is actually all part of that system there does that answer your question? someone else? sure thank you for sharing these community insights that was very interesting one question was that who is sponsoring the CIE infrastructure and who is maintaining and operating the pipeline so nobody at present it's a free travis they provide a free account for open source projects and I tend to set it up myself so I sponsor it but not with cash just time okay sure in the beginning you mentioned a few times that the TPM is doing measuring of various code at boot I'm a bit surprised because I thought the TPM is not doing DMA and controlling all the buses so how exactly is the TPM measuring anything? effectively what happens is so the TPM never makes a decision on the trust state all it does is it signs the object itself so this is where I'm not I'm not a TPM expert but from what I understand there's a core root of trust there's a CRTM which is like a seed and then it will take the next part of the boot chain whatever that might be the firmware and it will take a hash of that and it will add the two hashes together so it will concatenate them and then it will sign those and then the next part will be measured okay and then that will continue until you have a complete one way hash function which will allow you to effectively replay the boot is that what you were asking about or something around the communication? so the TPM is not not the thing on which the security depends but rather another component which reads all the data and feeds it to the TPM so the TPM even if the TPM is working correctly you have no guarantee that the rest of the system hasn't been tampered with because the rest is reading the data so there's two within the context of Keyline there's two parts that feed the TPM so one of them is IMA which is part of the Linux subsystem okay I wouldn't really be able to tell you about the attack vectors in there but it's been around for about I think close to nine ten years that code now so it's pretty well audited and that came out in kernel 2.6 I think so that's IMA which measures the run time okay and what will happen is IMA if it senses a TPM then it will send its measurements into the TPM into a platform configuration register which is a particular part of the TPM where a hash is stored now if it's the boot then there is in fact a shim which can do the same thing it can if a TPM is present then the shim will extend its measurements into the TPM itself around the protections on that I wouldn't be able to comment I'm relying on the kind of the maintainers and the people that are developing that so we're more of an application that's sitting on top of that software stack but it's very widely audited I believe so I'd be fairly confident in the security of that no exploits have been reported since TPM 1.2 which was depreciated quite a long time ago and I think that's the case of somebody was I think they had an oscilloscope and a solder in iron that sort of attack no worries anybody else have a question sure so you have been talking about having a certificate for the key in the TPM on many embedded systems you also have certificates and keys for things like VPN tunnels or something like that is managing and renewing certificates of that kind also in scope for keyline or is that something that another component would do so that would be managed by a certificate authority so keyline not something that we do at present all we would be able to do is we could effectively tie the certificate into a CA in the aspect that the machine is compromised you can revoke and invalidate that certificate you could do an encrypted payload which would have your your SSH keys your TLS certificates and so forth so if the trust state of the machine is good then a key will the second part of the key will be release which would allow the device to unlock its secrets so you could do a follow up provision that's quite possible where you could restage some more secrets but again you'd ensure a non compromised state on the machine first and you could tie that into a CA as well yep okay anybody else cool okay so thanks everybody for your help sorry about me having to keep Ben round I couldn't see the slides on my laptop so that's why I had to keep craning my head round but I'm here all week so do come and grab me if anybody wants to learn some more details or find out a bit more about the project I'm here all week as well I'm also here for the Linux security summit where I'm hopefully going to do a demo of Keyline, we'll do a live demo if we can so thank you


What's up everyone? It's your girl Kenty here from Layton Sports Cards and I'm ripping a personal year for John S. We've got a 21-22 Panini Revolution basketball hobby. Alrighty John, best of luck on your box. Thanks as always for your order. Let's see what we can get you. A couple more. Personals just came through. Thank you. Thank you. Let us know if you have any questions. If you have a budget you're looking to spend, don't know what to get. Let me know what you're interested in. I can definitely help you find a product that fits your budget and your interests. Always willing to help you find something you like. All right, here we go John. Good luck. Let's see a galactic. Scope of Carl Anthony Towns and Santi Aldama Rookie. We've got D'Angelo Russell on the impact that is 78 of 149. That's a cool looking card.  Nice job and vortex, Kawhi Leonard. See an auto too. I can check for you New Hampshire. We have John Wall on the Sunverse 43 out of 60 and Villy Rookie between 75 and 170. Specific sport? Are you open to anything? We've got Astro of Seth Curry and Shockwave of Jonathan Kominga. Definitely will help. Yeah, let me know if there's a specific sport or anything. I'll look out for this break for you. Davion Mitchell Rookie Revolution and Rookie for the Bulls, Io Dosu Mu. WWE Revolution is pretty cheap. Yeah. I might be able to do it cheapest too most. We've got Jeremy Grant on the fractal and vortex of Damian Lillard. Is what those are like 99 or something like that? Yeah. We've got Franz Wagner on the Rookie Astro. Nice one. They are Magic and Keon Johnson Rookie. New Hampshire, that should be filtered by Price, low to high right there for you. If you click that link. We've got Astro Jordan Pool and Rookie Revolution, Franz Wagner. Nice way to end that with some nice Wagner's there for you, John. Congrats. Thanks for hopping in and we will get these right out to you.


Hi you guys, I am so excited about today's video because this is like years and years and years in the making. So who remembers the Blue Drits Needle Organizer that they discontinued? I feel like I got mine right on the tail end of them being available. I showed it for a couple of times in videos that I was working on and then no one could get it anymore. People fell in love with it and then couldn't get it. And then like a couple of years after that I found the little Madam So pillow that you like put your needles into and that didn't really, it wasn't the same. It's not the same. But I have found not one but two new options for needle organizers that are very similar to the one that Drits used to offer but even kind of better in a way. Here they are. Both of them are from Colonial Needle and I'm gonna open up each one and I don't know, kind of explore them with you. I've never seen these before. I literally just ordered them, got them in the mail, thinking, hoping, praying. Oh my God, I hope this is very, very close. And so we're about to find out together but I think it's gonna be good. Okay, so this one is the Machine Needle Pack Two. The other one is Machine Needle Pack One. I don't know why I'm going backwards but welcome to my life. And you have this cutie little pouch with like this, I mean, really pretty kind of design on it. It opens up here and then the back falls down. So that's how you have it. I think you could probably also go this, oh yes, you can. And then a belt crows. I'm so cute. How about the pattern matching? That's not bad. Okay, but the whole point of this thing is we have our types of needles going down the side and this is like quilted, it's like plush. And then we have the sizes going across here. So you would take your needle case, your needle cases go in here. One, I think there's eight of them. Yeah, you can put eight cases of needles in here, right? You can take the one out of your machine because what I like to do and why the needle organizers are so helpful is when I put in a brand new needle and I like make one garment with it, I'm like, okay, it's been used, but it's not dead yet. And if I need to switch out for my next project, I'm like, I don't wanna throw this away, needles are expensive, but I can't put it back in the case that it came in because then I won't remember that I already used it. So I wanna store it somewhere where it tells me, okay, if I use this, it'll be in my second use and then after that is probably done. So with this, you could take your machine needle and you could store them in here, right? So you'd leave the packs here and then you'd use one for a project and then you're gonna go switch out to a different needle, you take that one out of your machine and you put it in here. And then over time, you'd have a whole bunch of them that you would have used one time and then here's your second time in here.  You want it to kind of go sideways because it can go through, it can go through the back. There's not any plastic or in there. You can also go sideways and out, I don't know, you can store it in there however you want. But yeah, this one's really compact. Like I said, oh, actually, this one down here is bigger for the ones that are multi-packs. That's really thoughtful. So you have six single packs and then room for one multi-pack.  Great little thing for traveling, no? Really cute, really cute. Okay, so yeah, like I said, that's machine needle pack two. Okay, and then yeah, it says the same stuff on the back. So if you wanna screenshot that, that's what it is. Okay, machine needle pack two, very cute, very handy. I also did see on their website, I wanna mention really quickly that they have the same thing for hand sewing needles. So if you're into like needles for needle point, I don't know what those things are. They have one of these for that, one of each of these for that as well. It's not just sewing machine needles, it's also hand sewing needles if you need that. Okay, so this is needle pack, no number, a compact caddy for machine needles. So here it is, it comes attached to this thing with like tacky, you know that tacky stuff, but it doesn't stick. And this one's, this one's, okay, intense. Is leather bound, can you see that? Or probably pleather bound. It's definitely not as cute, but same idea with the folding up like this and then you pin it down like that and then yeah, the design is pretty much the same except this one, you have sleeves up here that are see-through, that's different from that other one. This is all, these are all see-through and then you have like this much space for each of these. So yeah, you can put your machine needle pack, hold on. This one comes with needles too, the other one did not come with needles. So you can put your machine needle pack in there. I think you can only fit one. Yeah, and maybe it's even meant to be laid down. You can lay them down like that. So they're in there nice and good. And then same deal with the ballpoint needle, a ballpoint pin to mark what you've got now. So you just wanna slide it behind. There's like two layers here, you wanna slide it behind. This is not cushy, there's no quilting in here at all. It's just two layers of like plastic kinda. And then if we wanna do what the needles look like in there. Okay, so we're gonna pretend that we just used this needle in our sewing machine. So I just took this out of my machine and I'm gonna find universal size 70 and I'm going to slide this in there. Okay, yeah, that's really interesting. So there's 70, let me see what it's like when we put more in there.   Oh, these are, sorry, I just wanna figure out, these are twin needles. These are twin needles, that didn't have that. Obviously this is meant for you to like write in yourself what extra needles you have and then use those things for that. But I don't think you're storing like 50 universal needles here, right? You're really only using them once because then once you take out, like let's say I put in a stretch needle for my neck, I used universal size 70 the first time and then I swapped it out for a stretch needle, size 70, took the universal one out, put it in here. Then when I take the stretch one out and replace it with like a denim needle, that's gonna go in here. And then eventually I'm gonna use the universal 70 again and I'm just gonna take it from here and put it in my machine. So you really don't need like a ton of space to do, to put needles, just one for each type, which I think that there's plenty of space here to do that. This one does not have, oh, it does have twin needles, but it's just the same size as everything else. This one also has lots more options in addition to the usual ones. It also has leather, metallic, quilting, sharps, top stitch and wing, never even heard of wing. It has those. And then if you have anything outside of this, there's not one, there's not an option for you. So if you use a lot of like maybe specialty needles, that would be this would be the best option. If you use more of the general ones, this would be the best option. Yeah, I think you're definitely gonna be able to fit more of them in here because imagine if you went diagonal like that. Okay, and then this is size 70, universal 70 right here. So this would go in here like this. Yeah, so you could probably get two of them in there. Like if you were taking them out of your serger, you could fit two of them in here for sure. This one probably only one per type per size. But I think these are like really, really, really close to what we had before and the best that I've seen on the market yet for organizing your machine needles. I love that they are, they close up. That's actually even better than the Dritz one because the Dritz everything was just open. It was just like a plastic board kinda. So they close up, you can easily travel with this if you wanted to take it. Especially actually the hand sewing ones, if you do that and you like to do that when you travel, one of these for that would be perfect. But for sewing, it would be great to take to class, especially if you don't know what you're gonna be making in class and you just wanna have an assortment of needles or if you're taking some kind of class, it's like, you know, how to use all the functions on your sewing machine and you needed a variety of needles to do that. You could do that here.  So I hope that helps fill the void of machine needle organizers that we've all felt for years and years and years once they took the other one away. These are super good options and yeah, happy to show you guys what's available out there now to get your machine needles organized. I have a link in the description box where you can grab these. Any questions you have about how to use them or if you wanna see more needles in them or whatever it is, just leave them in the comments. And yeah, that's gonna do it for me today but thank you Colonial Needle for offering not one but two different machine needle organizing options. We've needed this. So thank you. All right, happy shopping.


Okay, all right. So I'm Andrew Harding. Like Amir mentioned, I'm a spy. I've been a spy maintainer for a couple of years now, and recently I was made a Spiffy maintainer, participated in that effort for quite some time. Today we're going to have a little bit of a deep dive on Spiffy Inspire. And both Evan and myself are staff engineers at VMware. I'll let Evan introduce himself though. Thank you, Andrew. Yes, I'm Evan. I spoke a little bit earlier on the Spiffy Update thing. You know, the goal for today, the goal for this talk is really, you know, we've got a full day ahead of us, and we're really only just getting started. And, you know, there's likely to be kind of a lot of jargon thrown around and a lot of terms and things like that that might be specific to Spiffy Inspire. And so the goal of this talk is really to kind of familiarize everyone with it and do as deep a dive as is necessary. But we hope, as is necessary, to set folks up to really understand and grow up the presentations for the rest of the day. You know, we also recognize that, you know, this might not be new to some of you. We've tried our best to make the presentation as interesting as possible for those who already may know some of this stuff. And we do have a lot to share with you. So, you know, I hope we don't go too far over time here. But I'm going to, in the interest of time, I'm going to let that be that. And I'll pass it back to Andrew to log into the agenda and kick things off. Okay. So, yeah, like Kevin said, we're just going to do a quick overview to help people maybe understand the rest of the talks that are coming today. We're going to start with a quick Spiffy refresher. They're going to go over the Spiffy Inspire project goals and then dive right into how Spire specifically tries to attain those goals through the agent through no delmerco data station, how Spire manages its keys and rotation strategies.  So right off the bat, starting with Spiffy. There we go. Kelsey talked about some of these topics. So hopefully this will be a very light refresher. We're going to start off with the Spiffy ID. Again, this is the heart of Spiffy and forms the way that we structure identity for services. And it's a URI and it's got a couple components. The authority component represents the trust domain for the identity and the path component represents, you know, the particular, you know, entity identity within that trust domain. And the trust domain is in Spiffy nomenclature is essentially namespace and provides a boundary. So trust main boundaries can be along security boundaries. This could be like, you know, different environments like production versus staging, or even other sort of workloads or systems that you might have some sort of, you know, requirement around security isolation for. It could also just be as simple as an administrative administrative boundary. Like you've got a couple different teams who want to manage their own independent Spiffy implementations or deployments. And so this could be like, you know, billing versus sales versus human resources. And the idea is that trust domains have signing authorities within them. And those signing authorities are responsible for issuing those secure identities for identity within that trust domain. And the secure identity inside Spiffy is codified in what's known as the Spiffy variable identity document. And this document contains a Spiffy ID and it's again signed by an authority within the trust domain. And we've got specifications out that define this type of document for both X5 and 9 certificates and Jock tokens. So we've got our game, we've got it embedded into a, you know, assigned document. Now let's talk about how we verify those documents. And we did that with materials that are found in what's called the Spiffy bundle. So again, this is a collection of public key material from the authorities for trust domain. And it's used to validate SVITs that belong to that trust domain. If you're reading through, like, the documentation or the specifications, you'll also see this called the trust domain bundle or just trust bundle. These three terms are used interchangeably. So building up, now we've got our ID, we've got assigned document over that ID, we can verify it with bundles. Let's talk about how workloads receive this cryptographic material. And that is done through the Spiffy workload API. So the Spiffy workload API is something that unauthenticated workloads talk to and again provides SVITs and bundle materials and streams new materials to the workload as those materials change. And because it's an authenticated API, in other words, you know, workloads don't have to bring some sort of identity with them or secret with them in order to authenticate against this API. It solves the secret zero problem for the workloads. The last thing we'll talk about in relation to Spiffy is another mechanism to retrieve the bundles for a trust domain. And that is the Federation API. And we've talked about this a few times already today. And again, this is just a very quick way for trust domains to exchange public key material so they can authenticate each other's SVITs. It's a one-way relationship. You know, when you contact this API, you're asserting trust in the bundle material that you pull off that API, but it doesn't go the other way around. So you can authenticate their identities, but the person you're federating with can't authenticate yours unless they also perform this sort of one-way Federation step to obtain your public key materials. So in a nutshell, Spiffy gives us cryptographic verifiable, secret zero solving, frequently rotated, federatable, namespaced, uniform identity. And that's quite a list. It's huge. And you might be asking yourself, well, you know, maybe on my infrastructure runs inside of a very homogeneous environment where I have, you know, maybe some of these checkboxes checked off, all the ones that are important to me. So like, what's really Spiffy bringing to the table? And so for Spiffy and its project goals, it's not about, you know, services that are running in a single cloud environment, or maybe this other cloud environment or that other cloud environment, right? Or this organization, that organization, or get another organization, or maybe your services that are running bare metal or inside virtual machines or running inside these containerized environments. It's really about an identity substrate that provides all of those benefits that can span all these different environments. So that's it for a Spiffy recap. Let's talk about Spire now. Spire's whole goal in life as a Spiffy implementation, the first Spiffy implementation, is really to light up that workload API that we talked about that gives you those, you know, bundle and SFID materials in as many different environments as possible and to provide a sense of uniformity around management of those identities. And it does this starting with the Spire agent. This is kind of the natural place to start for Spire's support of Spiffy because the agent is what lives alongside workloads and implements that workload API and provides those cryptographic materials to workloads in that API. Spire agent itself doesn't start out with any of those materials. Those materials are centrally managed and signed by the Spire server. The Spire server acts as the centralized signing authority inside of your trust domain. And so there's this mechanism through which Spire agent is able to reach out to Spire server and obtain the materials that it will later on feed down to workloads to the workload API and it caches those materials in an internal cache. And so as bundles are prepared and updated by, you know, inside of your trust domain by Spire server and as SFIDs are minted, those are sent down to the Spire agent and cached. And of course, as, you know, those materials change and rotate, Spire agent is able to reach out and continue updating those materials inside of its cache. Again, like making those available to workloads downstream via the workload API. Um, you know, now these are cryptographically signed identities. These are, you know, and they're associated private keys. And so there's, you know, these are security sensitive materials and Spire server isn't just going to hand them out to anybody. And so there's a process through which Spire agent is able to bootstrap and authenticate against Spire server and I'll kick it over to Evan to talk about that next. Thank you, Andrew. Um, yeah.  How do you get the first credential? You probably don't want to bake it into your workloads or bake it into your nodes and deploy them. So how do you solve this ideally at runtime? And so, you know, Spire and Spiffy both look to solve that problem for workloads. Spire in particular, which, which uses this agent, this agent also has to have a solution to the secret zero problem. So when a new agent comes up or a new node comes up online, how do you know the identity of that node or agent and how do you, in order to like, in turn, authorize the issuance of, of estates to it. So we have this process that is called node attestation. And this is a way that Spire server can, can find out the identity of a new agent or a new node without that agent or node having to have had any kind of thing baked into it or any kind of preexisting secret. You can see on the right, there's one node attestor plugin for the agent. It is platform specific, generally. And I'll let you have Spire server where it has multiple node attestor plugins. So a Spire server can manage agents across different types of infrastructure. You know, in this example, we have AWS and Google Cloud as well as a bare metal TPM based attestor. So I'm gonna rock you through a couple of examples of how this actually works. In the AWS case, the agent tickles this node attestor and this node attestor knows how to reach out and talk to AWS. So in this case, we reach out to the AWS meta data service and we grab a document that is signed by AWS that AWS makes available only to this node. And that document has the instance ID and other identity related information for this particular node. So the agent plugin reaches out and grabs this thing and then shoots it over to Spire server under a TLS protected connection. Spire server receives this document and then passes it down to its node attestor that pairs with the agent one. And this node attestor knows not only how to validate that document that it got from AWS, but also to call AWS APIs and perform an extra set of validations. Is this a new node? There's some anti tampering checks that occur there. You could write whatever logic you want in there really. That's the beauty of this of this pluggable system. But once the Spire server has effectively been convinced that the agent is running on instance ID 1234, for example, Spire server will issue the agent its own Svid. And this Svid identifies the agent uniquely within the trust domain. And this identity that we issue, the agent is derived from the attestation. So in this case, we issue an identity that's a function of this AWS account number on this AWS instance ID. And in order to just demonstrate the flexibility of this node attestation mechanism, I have one more example, which uses a TPM. TPM stands for Trusted Platform Module. It's a small little chip that is soldered onto most motherboards these days. And it provides an enclave, if you've heard that term before, secure enclave for holding private keys and making other kinds of security assertions about the state of the hardware. So there's this node attestor TPM, this TPM base node attestor rather that knows how to reach out to this TPM. And what it can do is it can grab a certificate that is burned into the TPM from its manufacturer. And it sends the certificate over to the Spire server, which is then of course passed to the TPM based node attestor on Spire server. That node attestor is configured with this manufacturer CA certificate. So it knows how to validate the certificate that yes, this is a valid certificate from the TPM manufacturer we expect. Inside that certificate and a lot and a message sent along with it are some public keys. And the private keys that are paired with those public keys are actually burned into this TPM hardware. So what the server does is it is then in a position to issue a challenge. So it can take a little nox or a small little randomly generate a secret. It encrypts it with this public key that it received, which it now knows to be burned into this TPM by way of the certificate it received and sends it back to the agent. So the agent receives this challenge and passes it down into its node attestor plugin, which then passes it back into the TPM to be solved. So the private key that the TPM is holding on to is able to solve this challenge and send back in clear text the bit of information that Spire server generated we can see here. At this point, Spire server has got a pretty good idea of the identity of the hardware that this agent is running on. And it knows that particular key that is burned into this TPM. And same as before, it uses that information to issue an identity to the agent. And in this case, the agent's identity is bound to the identity of this TPM and the hardware that it is running on. So that's as fast as I can tell the story of node attestation. The end result here is that we've gone from an agent or a node that just comes up on the network with no prior knowledge to, okay, now we know exactly the identity of this hardware of this agent that we're communicating with. So that solves the secret zero problem for the agent. But what about the workload, you know, Andrew described that we solve this problem for the workload two via the workload API. So there has to be some, some magic there. And so to solve that, we do a very similar kind of approach and Spire agent that we call workload attestation. So this is able to take a workload that we have no prior knowledge of that it has no credentials and we're able to identify it. So how do we do that? Inside Spire agent, we have this library called peer tracker. Peer tracker is a platform specific implementation of some logic that is able to introspect the kernel that the agent is running on to find out the, to find out like which process is calling it. So when the workload calls this workload API, they do all these special lookups and we're able to figure out the process ID and some other attributes associated with the process. Once we have this information, we pass the process info back into these workload attestors, which are similar to node attestors. One big difference here is that the agent can load multiple workload attestors and we fan out across all of these. So in this example, we have, you know, one for UNIX that knows how to talk to the Linux kernel, we have one for Docker that knows how to talk to the Docker demon, and we have one for Kubernetes that knows how to talk to the Kubla. So we just patch each of these plugins and they go and they collect information about this process that's calling us and they return what's called selectors. These selectors describe that calling process. In this case, we have, you know, a username, we have the Shasam of the workload binary. We have the Docker image ID is in Kubernetes related information. This is pretty much all we need in order to positively identify this workload. What is the shape of this workload exactly? What is the identity? Now we are in a position where we can issue it an SVID, a key, and the associated bundle. So he's done a lot of time so far talking kind of about the agent, how the agent gets an identity, how the agent issues identity to workloads. But of course, there's the server component that Andrew mentioned earlier. It's got to manage these keys. It's got to actually mint these SVIDs. It's got a bunch of responsibilities on its shoulder. So I'll pass it back to Andrew to take a look at some of those internals. Thanks, Evan. All right. So again, SPIR server is the centralized signing authority for SVIDs inside the trust domain. And it accomplishes by having a signing authority for each SVID type. So inside SPIR server, there's a distinct authority for X519 and Jot SVIDs. Now these pairs here, the X519 authority is represented as an X519 CA certificate and it's accompanying private key. This CA certificate can be self-signed. It can also belong to a larger existing PKI scheme and be signed by an authority inside that existing PKI. The Jot authority is just a simple asymmetric key pair. It's in charge of signing the Jot SVIDs. And one thing to note here is that across these two authorities, the private key material is not directly managed by SPIR server itself. And this is a security consideration to kind of separate out the private key management from SPIR server itself and offload that to what is known as the key manager plugin. The key manager plugin is a simple interface that is more or less loosely based off of a subset of PKCS 11. And through this interface, SPIR server manages multiple key slots for private keys and can create these keys and use these key slots to sign arbitrary data. There are a couple of key manager plugins that are built into SPIR. The top two you see there are memory and desk. We've touched on this earlier when Augustine gave his update that there are community efforts in place to develop a TPM based key manager as well as something that hits AWS's key management servers. Getting back, we talked just a second ago about how the X519 authority can be part of a larger existing PKI. And the way that that's accomplished inside SPIR is that the use of an... Well, Evan, I think we just... Let's see. Yep. Sorry. Hold on. I got kicked out of my SSO session right in the middle of our presentation. One second. Is that better? I have my side. I need to share. Hold on. Hold on. Okay. Yep. Let's see. Virtual conferencing at its finest. Yes, yes. Sorry, everyone. Hey, it's real time and it's interactive. Beats pre-retro. Yeah, absolutely. Okay. Where's my mouse? Let's see. Okay. Is that better? All right. I'm back. Perfect. Thank you. All right. So the way that SPIR accomplishes interacting with this sort of existing upstream PKI is through the upstream authority plugin. And again, this is a very simple interface that just provides enough functionality for SPIR to interact with that upstream PKI for the two different authority types that it manages. Specifically, as an X519 authority is prepared, the CSR for that intermediate CA certificate that SPIR wants to be signed is sent upstream through the Mint X519 CA RPC, where it is signed by the upstream authority and then shipped back. And if we want to talk about JOT, as the JOT authority is prepared, the public key material is published upstream, again, through the upstream authority published JOT key RPC. And the idea here is that mechanisms inside that upstream PKI can then disseminate that key to interested parties who want to validate JOT SPIR transmitted by this particular SPIR server instance. There's a whole bunch of upstream CA implementations here. I won't go over most of these, but I will mention this last one, the SPIR upstream authority plugin. Evan's going to dive into the details of this later, but this is essentially where SPIR is acting as the upstream authority for a downstream SPIR server. It enables some interesting resiliency and isolation benefits. So again, we've talked about how these authorities are prepared and maybe participating inside of an upstream PKI, but we haven't really talked about how the public key material from these authorities makes its way back to agents and down through the workloads via the workload API. And so essentially what happens is SPIR manages the storage back end, and this is known as the data store, and it's involved in storing all sorts of different things, which we won't get into now. We're mostly concerned at this point with the trust bundle for the trust domain. As these authorities are prepared, the public key material is appended to the bundle inside the data store. And like we talked about way back when we first introduced the SPIR agent, the SPIR agent is at some frequency connecting to the server and synchronizing down bundle material and getting SFID assigned and rotated. And so as part of that synchronization process, we can see that it pulls the bundle from the data store and directly through the SPIR server. And as the X519 and JOT authorities are rotated, again, the new public key material is appended to that bundle inside the data store over and over as SPIR lives and breathes and does its rotation strategy. And again, those materials are periodically pulled from the data store through the SPIR server down to the SPIR agent and out to the workloads. Now, there's this rotation of X519 and JOT authorities. This happens at a configurable cadence. And we've seen how the public key material from those newly prepared authorities is stuffed inside the data store and eventually makes its way down to agents. But because the agent is not getting a continuous stream of updates and is kind of pulling in at some frequency, there's an interval in which an X519 authority has been prepared and its public key material has been published to the data store, but where an agent has yet to pull that data store for that key material. And so at that point, you can imagine that if that newly prepared authority was immediately start to immediately assigned to start minting ESFIDS, that if those ESFIDS made their way to an agent who has yet to pull for that bundle update, the agent would be unable to verify those ESFIDS. And so SPIR implements an interesting rotation strategy to kind of prevent this and mitigate this situation. And it accomplishes this by actually having two authorities per ESFIDS type. So I lied a little bit early in the presentation. The first set of authorities is considered the active set. And the active set sits alongside the prepared set. And the active set is the one that is involved in signing the ESFIDS. So any authority that's in that active slot is the authority that's chosen to sign incoming ESFIDS requests. And of course, its public key material exists in the data store and is propagated out to agents as they sync. And at some point, maybe these authorities are going to retire somewhat soon. SPIR is going to decide it needs to rotate in a new authority key pair for both the X519 and JOT authorities. But when it does so, it doesn't just replace the active authorities. Instead, it prepares a new set of authorities in advance and sticks those in the prepared slot. And during this time, of course, the public key material is added to the bundle. And the active slot is still the one in charge of minting these different identities. And so there's an interval of time here where our active slot is minting identities. And our prepared slot has been prepared in its public key material placed in the bundle where it's now propagating down to agents in advance of ever being used to sign ESFIDS. So after ample time has elapsed, that has allowed the public key material for the newly prepared authorities to propagate throughout the system and make its way all the way down through agents and into workloads. That active key pair can then be retired. Now the bundle material stays the same. We don't prune out the old authority key material quite yet. We want to leave it in there for a bit of time because there could still be, first of all, the rotation event happens or the activation step happens before that old authority has actually expired. And so there could still be ESFIDS floating around inside your system that have been signed with that old now retired authority that you'd still need to validate. So we give some time before we end up pruning that key material out of the bundle. And again, this process is repeated as Spire continuously monitors and rotates these authorities to maintain freshly rotated authority material. The cadence that we do this at is, it's a pretty simple strategy. It's based on how much time is left on the active authority. So when the active authority has half of its lifetime left, that's when we go ahead and prepare a new authority. And then when the active authority has one sixth of its lifetime left, that's when we go ahead and activate the new authority. And you can imagine a space of time between that halfway mark and that one sixth mark, that is the time that we give that prepared bundle material to propagate out to the trust domain before we start minting ESFIDS with the newly prepared authority. And there's some, some caps in there to like prevent, prevent some weird times with really long live authorities. But I'm happy to talk about that. I won't take the time to talk about that right now. Let's see. So we've talked a lot about, you know, Spire's responsibility, Spire server's responsibilities in particular. It's doing a lot of different things. It's signing ESFIDS. It's rotating authorities. It's publishing stuff upstream. And it's obviously a big, a big point of failure. If something goes wrong, Spire server that has large implications on the, you know, on their ability to put, to push identity out to our workloads. And so Evan next is going to talk about, sorry, I'm messing around with slides here. Evan's going to take a minute to talk about, you know, some deployment strategies for Spire to try and mitigate those failure modes. Thanks, Andrew. I'm going to speed through this as fast as possible because we're already well over our, our a lot of time here. So, you know, this is kind of the, the simplest deployment we can imagine, right. And as Andrew mentioned, Spire server becoming unavailable is particularly problematic if all the workloads are depending on having valid assets to communicate with each other. The good news is that there, this is not the most terrible thing in the world of the Spire server where it could fail. I mean, it's not ideal. But, you know, Andrew mentioned previously that the Spire agent does have a cache and the agent knows what, what identities it can issue and it fetches those in an advance and it caches them. So, the Spire server goes away. You know, we can't get new ESFIDS, we can't rotate expiring ESFIDS, but the agent can still perform workload attestation and can still serve ESFIDS to workloads from its cache without contacting the server. So, you know, it's survivable in a steady state, but again, it's not ideal. The very simple kind of approach just to addressing this is to scale the Spire servers horizontally. You can have as many of them as you like that this obviously addresses performance issues as well. We don't have any, any notion of active or passive, each server, you know, has the full authority. And they do have kind of a shared, shared data store though. If you were to say, hey, I want to put one of these in each, in like different failure domains, like one in each region or one in each availability zone or something, you know, having this to stripe a data store across those things is not ideal. And so another tool we have in our tool chest is what we call nested Spire. I've had a couple mentions of it today. This is where Spire uses another Spire server as its upstream authority. Downstream Spire servers do know attestation and workload attestation the same as a regular workload does. So, you could have, for instance, one Spire server in AWS and another Spire server in Azure and both of them kind of roll up to this global root level. So this allows you to scale across these different failure domains and to manage the failure of, of, you know, different tiers of Spire servers. So if this global kind of tier were to go away for some reason, you know, the local Spire servers can still perform signing operations, can still rotate workload attestants, they cannot rotate their own signing keys like their job keys or CA certificates. So, you know, that's, that's what we look to the upstream servers for. But, you know, you can imagine if you have a one-week lifetime on those or a two-week lifetime on those, you've got a significant amount of time to get that central kind of cluster back up and running. The final tool we have in our tool trust is federation. You know, this is where we have a different set of Spire servers that is in a completely different trust domain as a completely different set of authorities. And then the Spire servers between each other, they exchange public keys. This is good for managing failure domains. It's also good for managing security domains. If the Spire server and trust domain bar on the right-hand side here were to go down, it does not affect identity issues and trust domain through. If it were to be compromised, it also doesn't affect security of identity issues and trust domain through. So in summary, very quickly, we learned about all these major kind of Spire code paths. This is all that the major works that are really important to understand, like how Spire works under the covers. And we learned particularly about node and workload attestation with Spire, how we go from not knowing who anything is to knowing what things are. We learned about key management and rotation, how all of that is managed by Spire server and how Spire agent receives those and figures out which workloads to give them to. And we also learned about some of the Spire failure modes and different deployment patterns and techniques that you can use to mitigate some of the concerns that come up with this kind of technology. If you want to learn more, you can check out the Spiffy website at spiffy.io. These are the two main GitHub repos. Spiffy and Inspire GitHub repos. We also have our Slack channel. It's a very welcoming community, so we hope to see you there. I know we're already very much over time, so we'll take questions in the chat. I'll be there and Andrew will be there to answer them ASync. And we also have a session later today, a networking session, in which we can kind of talk about anything that might have come to mind during this presentation. So thank you for bearing with us in this much longer than planned talk, and we really, really hope that it's helped to kind of set the stage for the rest of the day. Thank you, everyone. Yeah, thank you.