Thank you, everyone, for coming to my talk. My name is Yeremi Piotrowski. I work for Microsoft. And the title of this talk is the next episode in workload isolation, Confidential Containers. At the start of this talk, I'd like to talk just briefly about myself, just so that you know where I'm coming from and what my background is. So I started working in embedded Linux systems and hardware security modules. I'm one of the maintainers of the flat card container Linux project. I've been working for Microsoft in Azure for the last two years. And for the last year, I've been working on Confidential Containers and within the Confidential Containers project. If you want to reach me after the talk with questions, follow up anything, that's my email. My SCED profile contains other socials, but that's a reliable one. Or Slack. I'm on the CNCF Slack, on the KubeCon Slack. And the focus of this talk will be a project called Confidential Containers. We often refer to it as COCO just because it's nice, short, and cute. And COCO is a CNCF sandbox project. It is built on top of Cata Containers, a project with which many of you may be familiar. Cata Containers is used to isolate workloads using VMs. And Confidential Containers isolates workloads using confidential VMs. It's a fairly young project, so I think it's been around for some two years in the CNCF since a year. We're currently at the 0.50 release, which happened last week. It was very exciting. And the goal of the project is to enable cloud-native confidential computing by leveraging trusted execution environments to protect containers and data. And so this talk will really talk about the different aspects of that mission. If you want to find out more about the project, about the community, that's the GitHub link to the community repo. It links to the various documentation repos and code. And the thing about Confidential Containers is that every single thing about it is interesting. There are challenges everywhere. Everything could be a 30-minute talk. And so I can't cover everything, maybe for the better. But there was a talk earlier today, like right before lunch, by Fabiano Fidenzio and Jens Freiman, also to Confidential Containers community members. If you weren't there, review it offline later. It was great. And give you an intro into the architecture of the Confidential Containers project. If you're more interested in Cata containers itself, there's a talk later in the same track, I guess, in the same room about tackling hard multi-tenancy with Cata containers. Now, there are various hardware implementations of Confidential Containers and Confidential Computing. And the one that I am most familiar with, so that's why it's going to serve as the example in this talk, is called AMD-SEV-SNP. SEV-SNP stands for Secure Encrypted Virtualization, Secure Nested Paging, which is quite a mouthful, but it makes sense. So SEV-SNP, or SNP for short, is a Confidential Computing technology that allows us to run virtual machine-based trusted execution environments. And across all the Confidential Computing technologies, you'll find similar properties of which I will focus on three. One is that we want to guarantee confidentiality and integrity of both code and data. Confidentiality is implemented by encrypting memory, but confidentiality alone is not enough. You can still compromise a trusted execution environment if you can modify encrypted pages from the hypervisor. So that's why we need integrity as well. The second aspect that is important to Confidential Computing is attestation, and more specifically, remote attestation. And I realize it's a complex word. It's difficult to grasp. It took me a long time to come up with a simple way to remember what it really is about. It's about obtaining evidence that we can forward to someone else. They can verify this evidence and decide whether to trust us with something. By the way, don't ask me about the meaning of provenance. The third aspect of Confidential Computing that is important is that the hypervisor is outside of the trusted computing base. So this is often known as TCB. So the essence of the trusted execution environment is that we only trust what is inside this environment. It's nothing outside, especially not the hypervisor. There are various ways to create virtual machine-based teas. Intel has their own technology called Intel Trust Domain Extensions. And if you were to study that one, you would find that it solves the exact same issues in similar ways, in different ways, but it has equivalent security properties. And so in this slide, I also wanted to show you a brief system architecture of the system for an AMD S&P node. There's an AMD secure processor on the chip that is the hardware root of trust. It manages the encryption keys for memory. The encryption keys are unique per virtual machine. And there is an additional table called the RMP table that manages the integrity aspects of the Confidential Computing Platform. And so let's look a bit deeper into what attestation is about. So the goal of attestation in the context of Confidential Computing is to unlock secrets after performing attestation. And the way this is implemented follows an RFC standard defined by the Internet Engineering Task Force, called the RATS protocol, Remote Attestation Procedures. The capitalization is a bit messed up, but that's the way it's supposed to be. And so this defines the interactions between various parties so that attestation can be performed and so that we can trust the results of it. And so here you see two models that are defined in the RFC that are fairly popular. Let's start with the left one. It's the so-called passport model. The attester is the T. That's the trusted execution environment. And the T obtains evidence that it forwards to a verifier. The verifier verifies this evidence, returns some attestation results that can be relayed to a relying party. Why relying party? Because it relies on the evidence or on the attestation results. There's another model in the so-called background check model where the attester communicates with the relying party. And the evidence is checked in the background. And so both of these approaches have merit. Depends on the use case, which one will be more applicable. But I wanted to also show you some examples of what this looks like in the wild, like in the real world. So you'll see same models, but like concrete components implementations. On the left is what it looks like when using Microsoft Azure Attestation Service. The Confidential VM talks to the Attestation Service, transfers hardware-specific evidence, which is then verified. A token is returned to the Confidential VM. This token can then be used to pass to the Azure Key Vault managed HSM in addition to any other authentication, which then uses that token to make policy decisions and then return keys to the Confidential VM. And the funny thing about this left part is that every single one of these components actually runs in a trusted execution environment. So Attestation Service and the Key Vault, both of them, are also using confidential computing technology. On the right, you'll see the architecture of how the Confidential Containers Project is doing things. It's following the background check model. So the Confidential Container or VM talks to the so-called Generic Key Broker Service, which performs the Attestation Service function internally. Now, we want to look a bit into what the Attestation evidence looks like for this specific case. So for SNP, the evidence is called the Attestation Report. It contains a limited amount of data, but I'll focus on four fields that are present in this Attestation Report. The first one is called the Launch Measurement. And the Launch Measurement contains a single value that represents the initial state of the memory of the trusted execution environment. So in this case, you see it's like 48 bytes. It's essentially a hash. And we'll talk about how this is constructed in a sec. The second important field or useful field is called the host data. And it is used to tie data to Attestation before the T is launched. So anything that is additionally included in the Attestation Report before the launch of the T, the T cannot affect. It's fixed. And you see in this example, it's zero. But if you want to know about use cases for this, we have a demo coming up at the Microsoft booth on Friday at 11. We also presented some use cases in the community calls for the Confidential Containers Project. So you can review them later. The third field that I wanted to talk about in the Attestation Report is called Report Data. And this is used to tie data to the Attestation Report at runtime. So this is under the control of the trusted execution environment. When it performs Attestation, it can pass additional data to be included, although limited. This is basically 64 bytes worth of. So this will be a hash of some sort. And there are two major use cases. One is some proof of freshness. So the relying party may or the verifier may request a challenge response kind of scheme where it gives a challenge to the T. It expects this to be included in the Attestation Report so that it knows that the T is actually in control of the Attestation procedure. The other use case is to include some kind of key. And then the meaning behind this is that this key is held inside the T, actually a component, like a public key from a key pair. And so someone who has the Attestation Report can then tie this public key that is also given to the Attestation Report and use it to encrypt data that will only be decryptable inside the T. The fourth property is what makes remote attestation possible. So the whole Attestation Report is signed by a unique key inside the AMD secure processor. And this key is also part of a certificate chain that chains all the way up to the Hardware Manufacturers Certificate Authority. So we get the Attestation Report. We can obtain the key that was used to sign it. And we can check the certificates. And we know this was generated inside of a trusted execution environment. So the launch measurement, as I mentioned, represents the code that was used to launch the trusted execution environment. And so the way this is done is the hypervisor, while launching the trusted execution environment, talks to the secure processor who holds the encryption keys and asks it to encrypt every page into the address space of the T. And while doing so, every single page is measured. And as I said, measurement in this context just means that it's reduced into a single value. And the way this is done is you can see in the second bullet point. So we start with some initial value of this launch measurement. It's all zeros in this case. And then we concatenate that with some metadata about what is being loaded, the address that we're loading things into, the hash of that data. We hash the result, and that becomes the next launch measurement. And we perform this measurement iteratively as we load more data, more code. And this formula has several interesting properties. If we change the address at which we're loading something, it changes the launch measurement. If we change the data, the launch measurement changes. If we change the order, we also change the launch measurement, even though we might not want that, but it would be impractical to do it otherwise. There are other special kinds of data that is used to start the VM that is also part of the launch measurement, for example, the initial register state of the virtual CPUs. This is important because if we were to start the T at a different address, we could invoke completely different behavior. And so this is really all about being able to pre-compute a launch measurement. And then if the same measurement occurs elsewhere, we know it was the same code. It hasn't been modified. I know this was a lot to digest, but let's look at some deployment options of what a real-world cocoa setup looks like. So here we have a bare-metal system. The system is a physical server with an AMD CPU. And it's a Kubernetes node. So it has a kubelet, a container D. The kubelet talks to container D. Container D talks to the kata runtime. The kata runtime talks to the hypervisor and spins up a confidential VM. It spins up the VM with some guest firmware, a kernel, and some inner RD, all of which are measured. So part of the measurement, the inner RD of the confidential guest contains the kata agent, some other kata bits or cocoa bits, like image management code, attestation code. And this Linux guest running kata agent can then communicate to the AMD secure processor in the bottom corner to perform attestation, to communicate with the key broker service component. It gets a key. It can use that key to decrypt container images that it has fetched and start pods. So this is like a reference deployment. But it has to be performed on bare-metal. So you need your own hardware. Probably your own data center. The hardware is hard to access in some cases still. And you have to manage a lot of things underneath the operating system, firmware versions, bios updates, these kinds of things are still essential. So what we came up with is the second deployment option. And the problem here is really naming, as always. So I don't know if I should call it nested confidential VM or confidential nested VM. Probably the second is the applicable one because there's only one nested VM in this picture, even though there may be multiple, but in the stack, right? The T is a nested VM that is confidential, hosted inside a Kubernetes node, running on top of, in this case, the Hyper-V hypervisor. Inside the Kubernetes node, it's the exact same picture as before, but it can be much smaller. It can be managed using the existing workflows. And this is also the deployment option that we will be using in the confidential pods offering on top of Azure Kubernetes service that was just announced yesterday. So if you're interested, look, open up QR code. There's a sign up sheet and you can find out a bit more. We're also gonna be using this in the confidential containers CI because it's so practical. And this way, we can have multiple of these Kubernetes nodes on the same hardware. Each of them can spin up a certain amount of confidential VMs, each hosting a confidential pod. And if you think about it for a second, right? I mentioned the properties of confidential computing is the hypervisor is outside of the TCB, right? So who cares that there's another hypervisor underneath, right? The hypervisor is already not part of the TCB. And for me, this was like a Eureka moment when I got this working. So I was personally involved in getting this working with KVM inside the Kubernetes node. I got this working, I compared the launch measurement between bare metal and this setup and they match exactly. So the hardware in that case tells me no, none of the hypervisors interfered with the process with the security of the system, tampered with any of the memory and the CPU attests to that fact. So not every cloud or every hypervisor has this capability. So we also support something called peer pods in the Cocoa project. You start with a Kubernetes node running somewhere. It can be in the cloud, it doesn't have to be. You know, there's a Kubelet, a container D, a Cata runtime, and it talks to a new demon called the cloud API adapter. The cloud API adapter talks to cloud APIs, as the name suggests, and spins up confidential VMs, which are available in several of the clouds. I know Azure has an offering for that. And inside this confidential VM, we start the usual Carabit and then start the workload after performing the same attestation as in the previous two cases. If you're interested in this deployment, right? We published a blog post together with Red Hat last week with a real life use case, something like Spark running on OpenShift with confidential pods. So check out the QR code, or we're also gonna be demoing this at the Red Hat booth tomorrow at 14. So be there. Yeah, there is a difference here compared to the previous two cases, which is we are not in control of the hypervisor bringing up the confidential guests any longer, right? The hypervisor is specific to the cloud platform. There may be differences in the guest firmware, so the measurement may differ a bit, right? But it will still be confidential. And so where is this whole story kind of headed, right? Where are we headed with this confidential computing idea? So eventually it'll be easier to consume thanks to projects like the Cocoa Project. It will integrate nicely with the secure supply chain story. So you'll be able to generate a software bill of materials for your code. You'll be able to deploy your code in trusted execution environments. You have assigned software bill of materials. You'll be able to pass that to some kind of key vault, right? And it will base the decision of whether to trust your workload on the contents of the software bill of materials, which will be attested by the hardware that it matches what was launched, right? Yeah. So at this point I wanna come to the conclusion, which is that I think this will be a part of a lot of multi-tenant architectures or zero trust architectures. It matches the trust model that we have from a user's point of view when deploying things, which is we don't wanna trust anything outside our workload. And I'm sure lots of you are in industries with regulations, with working with privacy data or personal data, working with financial data, working with compliance and audits, right? And confidential computing is really an answer that closes, shuts down the discussion, right? Like what's the scope that I'm evaluating, right? The scope ends at the boundary of the trusted execution environment. COCO, fantastic project, healthy community, multi-vendor engagement, integrates confidential computing technologies with containers and with Kubernetes. COCO supports multiple hardware, trusted execution environments. You know, this is a list for now. We hope there will be more. And there's a spectrum of deployment options available to you, right? In your own data center, bare metal, nested confidential VMs or confidential nested VMs, as well as just standalone confidential VMs. You know, this is a spectrum, right? Depending on your circumstances, different options will be available, you know? Nested virtualization has overheads, yes. But it's also flexible and you know, you don't have to pay for the whole node. Confidential VMs are likely where we're gonna see things like confidential GPUs show up first. If you wanna find out more about, you know, the COCO project, this is a link to a blog post, also on the Red Hat blog from some confidential containers, community members talking a bit more about the attestation flow. And yeah, that was it. Hope you enjoyed it. We'll be taking questions on the mics on the sides. That's a link to give some feedback. I also have other community members here to answer your questions, if you have any. Thank you. Anyone? Anyone? Questions? Mics? No? Sorry, I got a question. Yeah. Actually too, but let's start with the quickest one. Are there any plans of bringing these to other cloud providers rather than Red Hat in the future? Which part? The whole stack. Well, the whole stack is not bound to Red Hat in any way. Yeah, sure, but currently there's plans of implementing, actually them weighing in, as you mentioned it, tomorrow. So. The demo is actually running in Azure. So it's about, so it takes a while to implement this in a cloud. Let's put it this way, right? So confidential VMs, I know we have them in Azure. I'm sure Google has some, IBM has some, but the stack is cloud neutral, right? So this was just what is available at the time. I think things like TDX from Intel are available in Alibaba cloud, right? But may not work in cloud API adapter yet if you're talking about the thing that we will demo at the Red Hat booth. Yeah. Yeah, so, yeah. Any more questions? Thank you. Please. I had a question. So suppose you launch a different micro-VM with Kata, for example, Firecracker, which is also supported, right? So how will that work? It's been nested virtualization because then Firecracker does not really work there, right? Or is it the default micro-VM which you launch with Kata all the time? So Firecracker is the, can launch micro-VMs, but it won't be used for the, you know, the Kubernetes node that hosts that. So I know... So, you know, like when you launch the micro-VM with Kata agent, right? Yeah, yeah, yeah. And replace the micro-VM in the Kata configuration to, for example, run Firecracker, right? So I'm just wondering because with Hyper-V, the nested virtualization, it does not, by default, support it. So is it only, I mean, the use case you showed for default micro-VM supported by Kata? That's what I... So I know nothing about support in Firecracker. I know we, this will be supported with Hyper-V, with KVM, Cloud Hypervisor, and QEMU, right? Like a combination of the four. I mean, Microsoft will support the Hyper-V and Cloud Hypervisor variant, right? Okay, all right, thank you. Yeah. There may be a follow-up answer from Fabiano. Can I get this mic tool on? So the work that has been done on the VMM sites is up to the VMM developers, right? So we have QEMU quite invested on that. We have Cloud Hypervisor quite invested on that. We would love to have contributions coming from Firecracker, but right now we have none. So that's the reason Firecracker will, or at least right now, doesn't work with the TE environment. If contributions can come, like I'm pretty sure that I, at least, would be happy to see that happening, but this is a conversation to have with the maintainers of Firecracker. Yep. Yeah. Can I have a question on the left? Sure. Can you hear me? Yeah. Thanks for the talk. Thanks. I was wondering if you might say a word or two on the compatibility with other native encrypted containers, like Singularity, for example, that maybe aren't running container D? Unfortunately, I know nothing about Singularity, so I can't say anything about that. Okay, thanks. Sorry. Does anyone know anything about Singularity? No? Any other questions? Yeah. Can you just, thank you for your presentation. Can you just explain a little bit more how your solution can contribute to creating a robust software build of material? So the idea that I have in mind, right, is that you tie your software build, so based on your software, right, you have a software build of materials, you can pre-compute a measurement, right? Then later, when you launch your workload, the hardware will calculate the actual measurement again. And if the measurements match, right, you know that the running state matches the software build of materials, right? So it'll be like an integration eventually when the ecosystem gets there. Thank you for the talk. Did you do any measurements on the impact on performance, for example, with memory latency when encrypting and decrypting everything, whether that has any meaningful or non-meaningful impact on that one? Thank you. There is an impact on certain workloads, but I don't have benchmarks. I mean, ideally, this would be available everywhere on all the time, right? But it does hurt some workloads at this time. Over time, with different hardware generations, it'll get better, hopefully. But I have no benchmarks. Sorry. And I mean, the same question came up earlier in the Confidential Container Session. There's also other kinds of overheads, like resources have to be allocated upfront. It's much harder to do things like paging out pages, so you reserve a block of memory so you can't over-provision as much. So there's overheads. Can you speak louder? Sorry. Sure. Is that better? Yeah. Just so I understand correctly, the root of the trust of all of this is with the processor manufacturers, right? Yes. So they're putting the key in there and a compromise into their key chain would mean all of this stack becomes untrusted. Correct. Okay. Right, we place the trust in the hardware, right? In the implementation, in the vendor, yes. Well, it's not quite just that, right? Because we're already having to trust the hardware. If you're running on some sort of CPU, you're trusting that that works. Yes. But you're changing the trust model from, I've trusted that they've implemented hardware correctly, which also we know that they don't always do, and we've had serious hardware security bugs, but we're trusting them AMD and Intel and potentially others. To manage keys correctly. Yeah. Yes. I sure hope they're using keys somewhere in there, you know, key injection process. Yes. Okay. Yes. Thanks. More questions, right side, left side? No? If not, I'll still be here, and I'll be at the booth. So if you wanna talk, we have several people from the project here. If you have a use case, I'd love to hear about it. Thank you. Thank you.


So I think I'm ready to go. I guess you can, sorry, say that again. Share your screen. Thank you. All right, let me, I gotta share the screen and start the presentation at the same time. A little bit more complicated. All right, can you see that now? Okay, perfect. Okay, so you, as seen on the screen, I'm Steve, I'm here talking today about modeling relational DB, our data with Cosmos DB. Just kind of a quick intro. My name's Steve Faulkner. I'm at South Pole Steve on the internet. If you see that anywhere, that's me on GitHub, on Twitter, on Facebook. I did go to the South Pole once. So that's me at the South Pole. That's a topic for a different talk though for another time. So you'll have to ask me about that later in the comments. I do work at Microsoft. So I absolutely work on Cosmos DB where I maintain the JavaScript SDK. So don't worry, this is not JavaScript specific. I'm not gonna be doing any complex JavaScript stuff. This is very applicable to all users of Cosmos DB for all the SDKs. But if there's any JavaScript users out there, welcome, shout out. And again, I'm talking about modeling relational data with Cosmos DB. So in order to do that, I kinda gotta go back in time a little bit. I gotta start at the beginning of time, talk about the history of databases and kinda how we got here. Best I can tell, Googling around the internet, the SQL database was really invented in around 1974. So it was invented with sort of the technology that was available at that time. So to give you a head a little bit, if anybody wants to guess in the comments, the price per gigabyte in 1981, relative to the same time period, it was $700,000. So I think this is the Apple II. So this is what it cost us for one gigabyte of data at that time. You couldn't even store a full gigabyte on one system, but this is what it would have cost you. Fast forward to 2017 and the price per gigabyte, two and a half cents. So clearly some things have changed since SQL was invented. And so really a lot of these new technologies out there, like Cosmos and other NoSQL databases, they're optimizing for different things. SQL was heavily optimized for storage, whereas NoSQL was optimized for speed. And that's kind of what I'm maybe talking about today is like how the way we've modeling data is gonna change for NoSQL. Specifically, I'm gonna talk a lot about Cosmos. Cosmos is Azure's NoSQL database product. It's globally distributed. It's a document store. It has this fantastic ability to scale out, take your data and distribute it between different regions and copy it between different regions. Here's kind of like the important bits that I think are important as a developer. You read and write JSON. It's just a document database. It has this massive scale and performance it's capable of. No schema, so it's schema-less compared to a NoSQL, or a SQL database is gonna be a little bit different. And it's also partitioned. So it scales out and I'm gonna get into partition in a little bit, but it has a system where your data can be, exist on different machines. It also has a SQL API, which is really cool. And everything is indexed by default, which is also a really neat feature and pretty unique to Cosmos. Digging in a little bit more into some of these aspects, the multiple APIs. So we have, these are our main APIs at this current moment. We have SQL, it's kind of like the primary original API of Cosmos. There's a tables API, there's a Cassandra API. So you can talk to Cosmos like you would any other Cassandra system. This is really great if you're coming from an existing workload or existing application that uses Cassandra, you can just use Cosmos via all your Cassandra drivers, which is really cool. Same thing for Mongo and Gremlin. These have been built on the open source tools that are already out there. So you can use all of our driver or the open source drivers and existing database SDKs for talking with those tools. We also have some cool things in the works. I believe this just hit public preview, Jupyter Notebooks, as well as Spark is on the horizon and SCD as well. But today I'm really gonna be focused on the SQL API. I'm gonna be talking about how to do relational modeling there. It's kind of the one that frequently gives people the most hiccups. So the first thing to understand is it's a SQL API, but it is not a SQL database. This definitely trips people up, they come to Cosmos, they start raising really complex SQL queries and either they don't work or they're really not very performant because under the hood it's still not a SQL database. We're just kind of exposing this SQL layer on top of your documents. Something to definitely be aware of. That leads us into Cosmos queries. So Cosmos queries, they're SQL. This is probably the most basic one you can run. Select start from C. You can select individual attributes. You can also select filters based on where clauses. You can do things like picking a do-at that's greater than today. You can order by. There's a bunch of different stuff. This is not the full list of SQL that we support, but this is kind of the big ones people tend to use in practice. Select, where, distinct, order by. Count, max, and min, and average are all aggregate queries. So there's some support for aggregate queries, which is pretty cool. We got some other cool stuff on the horizon I want to mention. Multi-order by is coming offset and limit. We already have out. Distinct we already have out. I mentioned that earlier and then correlated subqueries as well. So kind of beyond the lookout for those things that are relatively new to Cosmos. But because it's still a NoSQL system under the hood and it's a document database, we still have the same kind of gotchas that can happen with any NoSQL system. And those are two things. The first thing is scans. And Cosmos because we index everything by default, we kind of tend to avoid this problem. It's still possible to do a scan, but it usually doesn't happen out of the box. So other NoSQL products can worry about this. We don't really run into it too much. But the big one is cross partition queries. This is both one of the unique features of Cosmos as well as one of the Achilles heels that people will have in their application. Cross partition queries is when you're talking to multiple physical machines in order to execute an individual query. Now I'm going to jump a little more into partitioning and how that actually works. So you can understand both what I mean by cross partition queries and also understand how those are executed by Cosmos. So imagine the Cosmos backend. It has machines in different regions potentially, but even in one region, multiple machines that are storing your data. And what ends up happening is that a hash of your partition key that you provide as part of your data comes in and that determines where that data lives, which machine it lives on. And then ID is the thing that uniquely identifies that document on an individual machine. This is all transparent to you. You don't have to really worry about machines. You don't have to have instances of Cosmos running. It's all serverless, but it's important to kind of understand how it's working by in the hood so we can do things in a performant way. So the important kind of takeaway here is that a document is uniquely identified by ID and partition key. So if I do a query like this, select star from C where C.partitionkey equals foo. What's gonna happen behind the scenes is Cosmos is gonna receive that query, that's gonna hash that partition key and it's gonna go find the individual machine that all that data lives on. This is why partition keys and understanding them and where they fit into the system is very important because it means this is a very performant query. Every single time this query for any given partition key is executed, it's only gonna run on one machine, all of your data for that query is co-located, it can be very fast, it's not gonna have any fancy processing to do either on the backend or on the SDK side. But what happens if you do the reverse? So we call this a cross partition query. You query on something where you're filtering by something that is not a partition key. And what happens is the client then kind of takes over and it ends up finding a list of all your partitions from the backend and then it goes executes them against each physical partition.  You are having a cross partition query in your application and you go from 1,000 users to 100,000 users and suddenly that query will slow way down because it's having to execute way more queries against individual queries against individual partitions. Even if some of those partitions don't contain any of your data. Also this can cause problems on the client because it's executing all of these requests in parallel and there's ways to tune this a little bit with the various SDKs but we often see people running into client issues as well because they're having to process all these results and burning up CPU cycles and doing all kinds of stuff like that. So cross partition queries, something to be avoided. Quick recap of what I've talked about so far because it's really important before we get into the relational modeling part of this but SQL API does not necessarily mean that it's SQL. The partition key is very important to understand and you're going to learn to love it as we get through this presentation and we want to avoid cross partition queries. So that kind of gets to the meat of what I'm going to talk about today which is the relational data modeling. I'm going to use a simple example of a blog so there's some key bits here that I don't want to lose so I'm going to try to keep the example really simple. We're going to have users, they're going to have posts and posts are going to have comments. If you're coming from a SQL background you might have a system already like this that has tables and you might just hop over to Cosmos and create three different containers, a user's container, a post container, a comments container. And we need to provide partitioning information so Cosmos knows what to do. So maybe on users you'll partition ID. This you can do this, you can have your partition key and your ID be the same thing. It's actually, if it's possible for your system to work that way, highly recommended, it'll be very performant because your data will be evenly spread across lots of partitions. Maybe on posts we would partition on user ID in order to figure out like all the posts for an individual user so we could provide that in order to just query an individual user's posts. Similarly a comments container could be partitioned on post ID so we can get all the comments for an individual post. You don't want to do this. I have misled you probably over the last 15 seconds. This is not at all the approach that you want to take to modeling data in Cosmos and it's tricky because this is sort of the reverse of how you were taught to model data especially coming from the SQL background. I mean it's how I was taught to model data and I was doing a lot of post-gres work and it took me a while to understand that this is actually an anti-pattern and I'm going to go into why before I sort of talk about exactly how we should be doing things. There's a lot of cross-partition queries that will crop up here. That's why this is going to be a problem. Let's look at one example, our posts, they are partitioned on user ID. Imagine if we want to find one post. Well, how do we do that? We only have the post ID, we don't have the user ID, we don't have enough information to uniquely identify that post. It's still possible with Cosmos to make this query but the problem is, is it's going to result in a cross-partition query. It's going to, if you have hundreds of thousands of posts that are stored across several machines, it's going to talk to every one of those machines in order to find an individual post. And you can imagine that that would be very slow as well as problematic if you have users who like are coming to your blog and they come right to a post page and they don't have user information and you don't have user information you just need to find a post by ID. This is the kind of situation you will run to as you become your wildly successful blog become successful you will have this problem. Comments, another example of where we're going to run it at cross-partition queries. We decided originally to partition it on post ID. It seemed to make a lot of sense at the time. We still can't find an individual comment, has the same problem as posts. But what if we want to find all the comments where they belong to a user, right? We might want to give them ability to see that on a page or edit a list of comments or see a list of comments even. These are all going to be problematic and it's because we sort of brought this SQL style thinking which I kind of like, there's a lot of things that go into SQL style data modeling. But in my opinion, these are kind of the critical bits to understand, in SQL we rely on databases to drive the schema or to store the schema information. We normalize our data. So we want to keep only one copy of it when at all possible. We have usually pretty rigid consistency with SQL. We have very strong guarantees, we have transactions, things like that. And as a result of normalizing our data, we also have many different tables, right? And that's sort of what led us down this path of doing lots of cross-partition queries. No SQL thinking, right? So this is now the new approach we're going to take. And this is what I'm going to talk about throughout the time.  First off, the application is going to drive the schema, right? Cosmos doesn't have a schema, it's not going to enforce relationships. We're relying on an application to enforce those kinds of things. We're going to de-normalize data. So we're going to take data and potentially store it multiple times. And then we're going to have flexible consistency. So one thing that I definitely run into with users is that they say, well, I can't use transactions so this isn't going to work. And it turns out that I think in practice, transactions are not always necessary, right? If we write a couple of documents so we need them both to succeed, we can just retry that, right? That's actually works for a lot of systems. If you're maybe writing credit card transactions or something like that, you should look at doing something differently, but flexible consistency is an important term here. Cosmos does have the ability to do transactions, but let's go to a single partition. And lastly, we're going to start with a single container. We're going to put every single piece of data into one single container. It's going to look like this. And I'm going to sort of explain how this is going to work here in a minute. But our data is just all going to live in our data container and our partition key is just going to be partition key. I've sort of taken it and shrunk it down to just PK everywhere to help stuff fit on slides. But you can really use whatever you want to call it here like underscore partition key, PK is fine too. It really doesn't matter and Cosmos will let you set whatever you would like. So let's go back in time now and revisit some of our things and how we're going to model them in Cosmos. So finding an individual post. This was problematic before as it was causing a cross partition query. And what we're going to do is when we create a post, we're going to do what we did for users. We're going to store the ID and the partition key as the same value, right? This allows us to do really efficient fast point reads on any of our top level objects. We're going to do the same thing for comments, right? Post comments and users all now will have this document that stores, it's kind of the primary document for the top level entity and it's going to store all of the information we need and we're going to be able to get it fast because we can uniquely identify it in the system. We know both ID and partition key because it was the same thing. It's going to be fast. Now let's get into the relationships between these documents. If I have a user and I want to find all their posts, that's a good starting point. What we're going to do here is we're actually going to store two documents, right? So this is the first thing that we have to kind of wrap our heads around is that when we create a post, we're actually going to create two different documents and cause that. The first is going to be the post itself with all the post information like the title and the publish that date and any other kind of post stuff. But then we're going to store a second document that links the post to the author. And the partition key here is going to be users-one-posts. You can see this matches up with the author ID on the other side. And then the ID here is a pointer to the same ID as posts. So this document, because it's the same partition key, is going to serve as basically creating a collection of all the user's posts. So we're storing that information kind of in two places, right? We're not relying on the database to maintain that relationship. We're doing it at the application level. If I created three posts, you can imagine that I would create three documents that look like this, all having the same partition key, but then having different ID pointing to the kind of full post document. So now when I do this query, I'll do select c.id from c where partition key equals user-one-posts. And the important thing to understand here is that this is providing the partition key information. So we know that going back to our kind of previous example of how Cosmos is handling this on the backend, that query is going to execute against a single partition.  And this is sort of the beauty of this style relational modeling is that we might end up doing more queries because we don't have joins and things like that, but all of our queries are gonna be very efficient and very fast because they're guaranteed to either be point reads or they're gonna execute on a single machine. So this is what the return value would actually look like. We give an array back of all the post IDs and then we might go fetch those posts individually in parallel in order to get more information from them. Let's expand a little bit more. So if I wanna find the user's posts that are in the draft state, I'm gonna do the same kind of thing. I'm gonna create two different documents when I create a post and I'm gonna again duplicate information. So before I'm sort of duplicating the relationship information, now I'm actually duplicating information that belongs to the post. So this state, I'm gonna store in both places and what this allows me to do again is have a query that is scoped to an individual partition key and the state information I can also query on because I know that that's gonna execute on one single backend partition. There's, I gotcha though, right? So if we do that when we create the post, anytime we update the post, we also have to update both pieces of information in both places. So this is sort of where it gets into like the application becomes more responsible for maintaining the data consistency and the data integrity, right? You're trading speed for having forcing your application to do a little more work. And this isn't a bad or a good thing, this is just a trade-off, right? And it's understanding these trade-offs where you decide for your application what makes sense to do. Now, let's do something like a little more complicated than that even. So what if we wanna find a user by an individual email? Right now, this would be a cross-partition query. We would only, the only way we can uniquely identify users is by ID. Again, we're gonna store information twice. We're gonna have two documents. The first one is gonna be the actual user document with all the information. And the second one is going to be using the partition key almost like a secondary index, right? So I would store my email in there in a composite way so that the partition key is user-email-my-email. This is actually not my email, just to be clear. There's some others to you before me at Microsoft, apparently, so I couldn't get it. But this is important to understand here that this is allowing us to use partition keys as a secondary index on these attributes. So I can now, instead of doing this, which is a cross-partition query, right? Select ID from where C.email is Steve at Microsoft. Now, if I do this query, I can actually do it as a single partition query, and then I don't have to worry about, you know, like that query taking a long time. There are definitely other things you can do. I've sort of just touched on the very basics today, but I think there's a lot to take away there because it really sort of requires you to reverse your thinking about how to store relationships and databases. But there are some more advanced techniques. Using change feed, as Santosh talked about earlier, is a really good example of how to kind of do more advanced things here. You can use Azure Functions plus the change feed in order to update documents later and sort of propagate out some of these changes to documents, so you don't have to do them all right in your application or like all at the time you're writing a document. You might lazily update some information in other parts of your database. Materialized documents are something we really didn't cover on, but you can take documents and sort of materialize examples of them. The best example I think for a blog is if you wanted to have like a truncated version of the blog, maybe a little summary that showed up at the beginning, we could potentially store this separately from the regular blog information and we would use Azure Functions to generate this truncated version of the blog every time somebody updates it. The last one is splitting out containers. So I talked a little bit about this at the beginning that you want to put everything in one container, but at some point it might make sense for you to split containers into multiple things. But I think I like trying to tell people to use this as an optimization, right? If you find some query is hitting one partition, even if it's not cross partition, but it's still like heavily hit by your end users and so you can't really cache it or something like that. If you need to make that query work better, you can split that information out to a different container and there's no reason that you can't do that for any part of your data. But I think it's better to start with one container and then sort of optimize over time. So critical takeaways, I'm just gonna kind of recap a bunch of stuff that I've talked about and see what people have to say and questions afterwards. But first off, start with a single container. I just realized I was talking about splitting them, but start with a single container. I think that's the best choice for most people. What is the best partition key to use? Well, just use something called partition key because you wanna think about your partition key in a very flexible way. It is gonna act as a identifier, it's gonna act as a way to index things, it's gonna act as a way to index things with like composite values. You can really do a lot when you just start stuffing data in your partition key. Top-level entities, all of them, all of the top-level models should just have ID and partition key as the same thing, right? It's only when you're doing the secondary relationship or index and operations or relationships that you're sort of having them be separate. Otherwise, this evenly spreads all of your data across all of your partitions for just point reads, which is exactly what you wanna do. You wanna, that's gonna allow you to take advantage of the speed and sort of SLA that Cosmos has. Use partition key to index your relationships and your attributes, which I gave several examples of. And then lastly, optimize as the data size grows. So I gave a lot of examples here of where you wanna avoid cross-partition queries, but that doesn't mean you couldn't start with a cross-partition query and then later optimize it away. I think just to jump back a little bit, this is probably the best example. Find by email, for a small number of users, this might not be a problematic query at all. This is a cross-partition query, but if you only have a few partitions, might be slow or fast enough for a while, but when you start getting more users, you're gonna wanna optimize it into this kind of a query, which you can do. You can use things like change feed in order to replicate your data and sort of update these indexes later after the fact. Oops. I believe that's it actually. So that's at the end. I'm happy to take questions from the comments. Thank you so much, Steve. So we have a couple of questions from people on the channel. The first one is, is there an upper limit recommended for the number of documents in a given partition? Yes. So in Cosmos, one partition can hold 10 gigabytes. And one of the beauties of Cosmos is it will sort of handle that behind the scenes for you. It will split partitions in order to move documents between them and stuff like that, which is really cool. But you still can only hold a maximum of 10 gigabytes in a single partition. This can be problematic. There's different ways of handling it. You can, the best example I've come up with is like a chat application. Like most relationships in a chat application are gonna fit under that 10 gigabyte limit. But if an individual channel has more than 10 gigabytes of documents, then you're gonna have a problem. So more than 10 gigabytes of messages in one channel. And there are definitely ways to do it. You end up sort of adding like a number to the end of the partition key name. And then if somebody scrolls way back, they end up going on to one of the old partitions. But you sort of like move your partitioning forward to accommodate that. There's some other strategies as well that probably go beyond this talk. But 10 gigabytes is kind of where this, you have to start doing fancier things. Another question we got is, why is this better than relational database, more scalable? Yeah, so I would say that that's probably the primary thing people are trying to get out of it. It's more scalable. I don't want to disparage traditional SQL relational databases. They're great and they can be great for lots of use cases. And I've used them a lot. Really, Cosmos excels when you have large amounts of data and you really have really strict requirements about performance. With a no SQL database, you sort of are optimizing for the fact that you want to maintain that speed over time and as your application grows, versus if you have a SQL database and suddenly your application grows to the point where it needs to be sharded, you're going to have all these same problems because you're effectively partitioning your SQL database and you're going to have to work through all these problems anyways. So for small applications equals great, but as you sort of scale, it's very, I want to say that Cosmos really shines when you get to those upper levels. Really cool. All right, perfect. I believe we're at time. But if you have, if people want to get more information, where should they go? So on Twitter is the place I'm likely to respond the fastest I'm way too much on Twitter, but also ask Cosmos DB at Microsoft.com is something that goes to the entire team and we pay really close attention to. So if you have specific questions about your architecture or how you're doing partitioning and strategies and stuff like that, feel free to send an email there and somebody will get back to you. It's a great like direct line into the Cosmos team. All right, thank you so much, Dean. Okay, and we're going to get ready for the next speaker.


To the editors, November 25th, 1950, enclosed, please find my case study for peer-reviewing publication. Sincerely, Irene Ibis, MD. The New England tapeworm, Tania Nova Britannia, is a sesdoid parasite, presently treated with antihemetics, which have contraindications and limited effectiveness. The patient presented C-figure 1, with complaints of fatigue, weight loss, and diarrhea. A routine examination indicated an infestation of Tania Nova Britannia. In a less enlightened age, this was dire, but modern science brings a brighter prognosis. Foreign objects would be introduced as antigens, stimulating an antibody response, ejecting the parasite. The therapy would be administered over the course of a week, affecting a complete return to health. At night, our eyes turn toward the moon, yearning for the most virtuous rabbit, chosen from those who recently lost us. Welcome to our perfect world, a jewel on the wine-dark sea, now graced by strange-looking guests. Come lately here to study us. Low wind and crack your cheeks. By the pricking of my nose, something foolish this way goes. Good morrow, Fair Herald. What news? Of men, or so they call themselves. Love like a shadow flies when substance love pursues, pursuing that that flies and flying what pursues. Love like a shadow flies when substance love pursues. We have seen one of their kind before. Gentle scum-crisp, master of song. Oh, it's you. Praise Mother Moon. Dear, my heart. But I fear their kind may not be likewise kind. Kind stumptuous taught me these trills while he sought refuge here from war. A treasure here may prove you wrong, so lend a nose and fathom hence. What testament of good or ill has fortune spilled from careless purse? What a piece of work is a man. How noble in reason. How infinite in faculties, informant moving how express and admirable. In action how like an angel. In apprehension how like a god. I am relieved. What wonders await? Mother Moon, preserve us. But preserve us not like that. What monsters among us.  Your smile. Your smile said. Thou unnecessary letters. My lord, you will give me leave. I will tread this unfolded villain into water. Winged ship drops shining gift into our paws. Lord. We have some medical professionals here who could help with the details. On the other screen, how did you do it? You know what I mean. Yeah. Well, it's the magic of cinema. Did you drink of life or what? Well, it was everything with film or digital film in this case. The documentary maker, Don Penny Baker said that film is truth 24 times a second. And that has to do with the frame rate and it isn't film specifically because it's digital but still it's built on the foundation of film. And the frame rate is now 29.9 times a second but the principle holds. And we're counting on the persistence of vision and we can help that along by pushing movements of the creatures in real time and that can be transliterated into the perceived time of all the frames stringing together. These were made with molds of silicone and there was an original model that was made out of a non-reactive clay and then there was a mold that was made from that and then the silicone resin was poured into it but there's a lot more to it than that. Anyway, the idea is to have all the parts of it merged together and create the illusion of life, of reality, of movement, of light and dark and narrative. Can you show some of the moving parts in the eyes and other things that... I don't want to take the eyes out. They're very hard to... Oh, okay, here's a box of eyes. The eyes were done by a technique called replacement which is having them made in a static position and then taking them out and putting them back another piece that's built slightly differently and the difference between those two shows up as the movement that depicts the transition. Some of these have other techniques. There are gears here. This opens and closes the jaw, this worm gear on this and it pulls some fishing cable, some fishing line through this sheath and the mouth moves in response to that and we control that with this worm gear. Talk about how you walk the fur. Oh, right, that's the dangerous part. The fur, after the silicone is made, it's very difficult to glue things to silicone. So I used an electrostatic generator, a Van de Graaff generator which was about 200 kilovolts and there was a kitchen strainer that I poured some lint into and you can get really nicely colored lint.  Well, some of it was. Now, I didn't use fake fur. I did use some, but this was glued down. From a fabric store. There were some metal springs and cable sheaths that I used so that the structure was flexible and stays in place. So that one is the one that you did with the Van de Graaff? A lot of them are. This and this and this used the static generator. Don't ask me to plug this in because I'm a little skeptical that it could survive it. But in this case there was also an electric motor that communicated movement through these three cable sheaths and there was a transmission inside the thorax that would move the ears up and down and we underscored that movement by using a very long exposure so that it would blur the movement of the ears. Slowing down the camera exposure. And the rest of it is the training of these talented actors and vocalists who taught their skills, their musical skills, their singing, their dramatic talents to these small synthetic rubber creatures. Do you want to introduce the people in the audience? Oh, yes, yes. Several of the voices on this was Goldie Vox who was right here with us who just did a wonderful job on this in breathing life into all of them. And the factotum who came to the rescue with innovation and coaxing movements and like out of the creatures was my friend Chris Burke who has been involved in these projects over the years. And after a few years it all came to life. How many ears were you working on it? I don't know, three I think. No, more like seven. I think it was three. Tell us a little bit about the source. Well, Picasso said that good artists borrow and great artists steal. I thought that was after a while. You sure? No. Well, it's more like Picasso to recommend his stealing. But there was a source material for this which was an obscure book written by a German anatomy professor. And it didn't get a lot of notice. It was a parody on scientific treatises. And it was a review of the American edition, the first English translation in Scientific American when I was in, I think, eighth grade. And it just caught my notice and stuck with it. And I made everything here based on the descriptions in the book. And I kind of hope that the current copyright holder doesn't see a quick way to suing me for infringement. But fortunately, there's fair use right, I think, I hope. No, they're going to go running.  Thanks. It's E-L-S. I'm curious about the one that had all the arms and it played with the music. Yeah, well, that's this. Yeah, this was supposed to be a pipe organ. Yeah, I know. And keeping the puppets with a good balance between their flexibility and their stiffness to hold the poses, wiggle their tails and their noses. And there is a lot of fishing running through these cable guides. All you have to do is take 7,000 really good pictures in a row and you've got it. Oh, and having a good dialogue writer helped. Where do you go now? We don't have Radio Shack anymore. Where do you go? Yeah, well, there is no more Radio Shack. You do it electronically. Oh, OK. Well, there are places that sell silicone. There are places that sell the metal that's used in the skeletons. There's metal sheathing that isn't a spring but it maintains its shape, it's stiff. And then you can run the cables through it and you can use this to hold their limbs. Oh, yeah, it's a place in Connecticut that makes this. And you can push the fishing line through it and then use that to make them move. You make the move easy for you? Well, let me see. No, I don't know that I feel like plugging this in because it might pull the puppet apart. Yeah, but this, you wouldn't know it, but Jill was able to get into this costume. I was substantially smaller if you didn't know that. And the terrific job she did. I don't know if anyone asked this already. How do you make the puppets? Did someone ask that yet? Well, it's a pretty standard casting method. Make an original, do sketches, of course, and use cheap clay to do it at first to see how the things look in three dimensions. You have to use a clay that doesn't have any tin in it or sulfur because clay has a lot of those elements because what that will do is it will inhibit the cure in the silicone. And I use a material called hydrocal, which is a very tough plaster to make the receiving mold. And then I put the halves or as many parts as I make together and pour in the mold material, which is a platinum-based silicone. And somewhere I should have it. A lot of the color is dried pigments because there's a really good range for it. And use a silicone adhesive to glue some of the fur on. This is one way of doing it. You can see here I marked the direction of the grain in this and held down the material with silicone glue and had to pin everything in shape so that there wouldn't be any lumps. Do you have a vision of what you want these animals to look like and then go looking for the stuff that'll make it work? Or do you collect stuff as you go and does that suggest things? Well, both, really. For example, this creature's ears were modeled after my dog's ears, who unfortunately is no longer with us. He should be here. How long does it usually take to make one of your creations? Well, they have all different levels of complexity. And it's a long process. And I usually do things by stages rather than doing one puppet and letting it go. So as long as it takes to sketch out the idea, that's the first thing. Before you do anything, you should do sketches from different angles. You can use inexpensive clay just to see how it looks in three dimensions. And the skeleton in it is, like I said, brass tubing, which is cut and soldered and it's pinched into hinges. But the metal work isn't really that complex. In fact, I think it's sometimes a little bit sloppy. Blobs of solder and the like. So for this one to work, we had this which connects to these cables and cable guides, which go into the transmission in here. This is the eccentric to have the motion reciprocating. But there was a small transformer in three volts. And when I was doing the shooting, I'd be looking through the camera and watching the movement and then pushing this button and it would move the cables. And an interesting thing is a lot of the tempo in doing the animation, which as I said showed up at 29 frames per second, translated from actual regular middle of everything movement, seemed to fit perfectly with the tempo of some of the late symphonies of Dmitri Shostakovich. So I would be doing the animation in time to some of the symphonies and it made it a lot easier because it's about rhythm. It's about the visual rhythm that we're creating when the creatures are waving their arms around or leaping about or talking to one another or gesticulating. It seems to translate. I really appreciated that you called us to invite us and I'm so happy to see the final result. I've seen a little bits and bobs here and there and it's just really cool that the magic come together.  Thank you for inviting me. Excellent source for materials to make all these... It's kind of an advertisement. No. It's the version that it takes a village, really. Yeah. So all right, well, that's great because Reynolds has been with me for the whole process and I appreciate it. So that's another important thing is to get somebody who knows the materials and the processes and is willing to help you along. Very, very impressive creatures that you've created. And to think about all the pieces that have to come together. Yeah, that's funny. Yeah, another thing you can do if you're just trying to put something together in a few steps is in Genesis. There's some instructions. Modesty aside. Do you want to say anything about the photography itself? The photography? Oh, it's blurry. It's scratchy. The lights are falling. The gels are coming off. The photography is terrible. You've seen better images through the bottom of a soda pop bottle.


Hi everyone, Joe for JasperJasperysCaseRakes.com coming at you with a box of 2023 Leaf Exotic Baseball. Random team break number two. A few things happening here. First of all, we're gonna give away eight extra Leaf spots within this break. Then we'll do the break itself with the second dice roll, third and final dice roll. We'll give away a nine of those off-season baseball mixer spots. That is almost full. We just need to move, do I have an aisle? Let's take a look. We just need to move six spots straight up. No more fillers and then we're in breaksville. Big thanks to this group for making it happen. So we only sold 22 spots outright. So top eight after the first dice roll, top eight after six. Good luck. One, two, three, four, five and six. So top eight, look at that. Steven, we've got a few extra spots there. So do you, Mark, Michael, Nathan, Justin and Michael again. All right, so that's after six, right? So top eight. So extra spots going your way. So there you go. A little buy one, get one for you. And you're still part of the break itself. So let's do that with the second dice roll. All baseball teams are in. There are your names. Let's roll it, randomized names and it teams four to six, 10 times. One, two, three, four, five, six, seven, eight, nine and 10. Four to six, 10 times for the teams. One, two, three, four, nine and 10. Cardinals down in nationals after 10. It's just a little filler break. So there's not going to be a trade window, but there are your teams right there just as is. There's the second half right there. It's alphabetized by team. Print and rip, print it and rip it. All right, there's the exotic baseball three metal parts per box. There is the final printout, hot off the presses. Thanks everybody for hanging with me on a Monday, breaking with me on a Monday. Always appreciated. And get those full spots. We can do that baseball mixer tonight. First one here is Dylan Lesko. Let's go. Three out of eight. Number up there. Is that a Padre? I think he's a Padre, right? Dylan Lesko is a Padre. So that's going to go to Travis in the San Diego Padres. Nice low number on that as well. I think he's one of their kind of higher end prospects. Yeah, first round or 15th overall. All right, a couple more in here. We have Gavin Cross. Gavin Cross a Royal maybe? Show me Royals. I guess the uniform kind of helped give it away. He's pretty good too. First round or ninth overall in the 2022 MLB draft. Kansas City, Michael. And that's four out of nine. Nice low number there. And the third one is, oh wow, nice. Wade Boggs, four out of 10. That's pretty cool. Red Sox uni right there. That's how we'll kind of do these. And Boston, that's for Mark. All right, not bad. Now let's see who's going to win. Let's see who's going to win those mixer spots. Let's go back to the list. We're giving away nine off season mixer spots. Let's gather your names, one through 30 and everyone in between. New dice, new list. Top nine after six. Three and a three, six the hard way. Good luck, one, two, three, four, five, and sixth and final time. So Travis, very close. No cigar here. Big thanks, 10 through 30. But ultimately sad times. Justin, you see yourself right there. You're in the mixer. So is Stephen, Andrew, Mark, Nathan, Michael, Stephen, Stephen, and Andrew. Congrats to the top nine. We'll see you in that break. That mixer hopefully a little bit later tonight. Jasper'scasebreaks.com, bye-bye.