Yeah, thank you, Pushka. Welcome to From Threats to Trust, Safeguarding Sensitive Data in Kubernetes. Hope this is the talk you all are here for. Otherwise, feel free to run. Quick introduction about me. I'm Moritz Eckert. My day job, I work at a startup from Germany called Edge for Systems. And there I have the pleasure to work with a wonderful team on open source, confidential computing software. We also happen to be the organizer of the biggest conference in that space called the Open Confidential Computing Conference. Yeah, personally, my background is in the software security, vulnerability discovery, binary exploitation kind of side of things. I spend a fair bit of my lifetime in captured-the-flag competitions with Team Shellfish. And yeah, if I'm not in front of a laptop screen, I like to ride my bike a lot and also enjoy a good cup of coffee. I can definitely recommend the IntelliGenser here in Chicago, so if you're into good coffee, you're welcome, definitely check that out. But enough about that boring stuff. Today I wanna go on a journey about trust, especially trust in our model of cloud computing and especially on the infrastructure side of things. So trust can run very deep. If you think about the trust model in our cloud and the infrastructure side, think of the iceberg meme here, right?  That's something you trust.  But then of course, your application is not standalone in most cases. You have some dependencies, your code is stored somewhere in a repository. Some people, some machines have access to that repository and then likely there's some kind of pipeline that takes your code from the repository to a production deployment. And all of this, of course, comes with a certain amount of trust, right? Do you trust the code dependencies? Usually, I guess most of you will implement some form of supply chain security. You have some access control to your GitHub repository. So you have all of this controls to reduce the trust in these components. Then we can go one step further.  Do you actually trust the platform you're deploying in? Do you trust your cloud service provider? If you have the container runs on GKE, do you trust Google to maintain the security of that cluster? And if you don't only have code dependencies, you might use some kind of software service like a database. You potentially trust that database vendor, that provider of that service to keep your data secure. And that, of course, also includes the employees and whoever and their devs and their developers who have access to the system and trust them that they keep their credentials secure, that they potentially implement two-factor authentication and so forth. And, of course, we can go deeper. We can go deeper in the software stack, right? Do you trust the hypervisor? Do you trust your container runtime that it's not backdoored? How do you trust that the Kubernetes you're running on is actually the release Kubernetes and not some malicious, I don't know, modified backdoored version? How do you do that? Hopefully, or likely, you use the latest Kubernetes release signing with the six-door project and verify that signatures and make sure that you're actually using the correct binaries that were released officially. And we can go once to further. Do you trust that signatures? Who has access to the keys that sign those releases? Who has access to the machines where the signature process is happening, right? And if you run on the cloud, you might trust the hypervisor that is running there. It's not being maliciously used by the cloud service provider or anybody else. But do you trust the other cloud tenants that they're not exploiting the virtualization stack and attack you horizontally or vertically from there? And we can go deeper and deeper, right? Do you trust the data center employees that have physical access to the machines that they don't access the memory, even though you might consider that unlikely, right? And finally, you trust the actual hardware. You trust your CPU that it does decapitation correctly. Otherwise, yeah, you need to trust something. And you might say, well, that's a bit paranoid, right? Do I really need to consider trusting my hypervisor? Do I really need to think about the data center employee? Are these likely scenarios for me? Is that a threat model I need to consider? I think that's fairly important. Talking about the threat model and talking about where trust matters. I have a couple of cases here. One thing we've been working with was, or is investigative journalism. So folks that are dealing with information, sensitive information and information about the sources, the actual content that it has information about very powerful actors, like organized crime, human trafficking rings, potentially some government, so some state actors. Think of if you're a whistleblower or somebody that is highly vulnerable, who and what can you trust? So safeguarding the information, if you have some form of cloud deployment and be one of those vendors, what would be your deployment model and what kind of security measures you have to do to keep that data secure? And you can say, well, these are the paranoid. Naturally, these people need to be paranoid. Maybe something a bit more practical. And there, one of the cases, or a usual case would be more in the regulated industries, like healthcare, financial sector, everything that deals with information that is potentially sensitive should stay confidential. And one of the cases we have in the EU and specifically in Germany is the case of the electronic health records. So whenever you go to a doctor or hospital, you get some scans or some other information about your body, this can be stored. And nowadays, this can be stored electronically in some kind of cloud system, so it can be accessed from all over the place, and people can actually have one concise place for all of this information. And with health information, it's not like some credit card information that once lost, you can replace that credit card the next day you have a new number. Once this data is gone, it's gone forever.  And health records are one example where we already see there's very, at least in Germany, there's very tight regulation. It's very tightly regulated on how you can implement such a system, who needs to be excluded from that data, and that goes down to the provider of the data that's being stored. And the provider of the system itself, where we see this regulation now is catching up in other sectors as well. So in the financial industry, we now are, it's now being worked out the Digital Operation Resilience Act that specifically mentions that data needs to be protected in all states, while transiting through a network and addressed, but also in use, or if that's not possible, if you can't encrypt it, it needs to be processed in some separate, isolated environment. And to be fair, these are fairly European cases where we see much, much more regulation in terms of these industries. But another case I brought today is a bit more recent, and maybe a bit more global, that we see is in the now upcoming AI and large language model hype. You've all probably seen the JetGPT case. When we look behind the hood, what happens there and how this is implemented, we usually see some of this kind of pattern where we have a large language model that is owned by someone, or might be open source, but currently is still largely proprietary and owned by someone who has VIP of this model of the weights. And then we have folks privately, but also in a corporate enterprise context that want to use that model. And at the same time, you might want the model to be trained on some more specific information, like your local knowledge hub, your documents, so that the model can provide you with a more targeted help for your specific cases, like generating a summary of your data, your reports, whatever. And they're not only about a regulated angle, but more from a privacy concerning angle. We see that the left hand side here, the request is very, very hesitant to use that, especially in an enterprise scenario. So the question is, how can we make sure that all the requests, the prompts that flow into that model, how can we keep that secure if it contains personal identified information or some kind of IP? How can the model owner ensure that whenever they deploy that, that their weights, their model is being contained, is being secured, and how can it securely access a knowledge hub without revealing the data inside to the model owner? So these are cases where we see that we need to be able to isolate the data that's being processed from the actual infrastructure, from the service provider, from whoever processes the data. So the question is, how can we achieve that? How can we safeguard that data from these potential threats? And one remark I want to make is that we can debate about that we can just go and run our entire stack in an on-prem in a private cloud scenario. And I think that's actually the case for a lot of these scenarios where people are forced to stay in their own private data center, in their own private cloud. And the question is, is that desirable? Is that something we want? Where everybody needs to maintain their own stack, the security of the stack, and or do we wanna profit more from the left-hand side? Do we want to actually being able to run those use cases in the public cloud and maybe have somebody else take care of the software stack, the security, for example, keep our hypervisor up-to-date? Is that something we want to offload to somebody that does that just specifically? And then we can profit from the scalability and all the rest in the public cloud. And I would highly advocate that. The left-hand side, in a general case, is probably what we want. And the right-hand side is only a workaround because we can't establish the trust in the public cloud. And that's what I would like to focus now is how can we get there? How can we establish this trust in a public cloud scenario? And as hinted by this Dora regulation, the missing piece is essentially that we can also protect our workloads, our data while it's being in use. So we already have the encryption addressed. It's pretty much a solved problem. We also, everybody uses TLS and some other forms of transport encryption. What's missing is that while the data is processed, while this data is being executed on the actual in the cloud, that there it's accessible and vulnerable to being accessed by some malicious actor. And there are different ways how we can protect data in use. There's multi-party computation, homomorphic encryption. What I would consider the most practical and usable one today is confidential computing. And that's the one I want to present. So confidential computing, the rough ideas that we have a hardware-based solution for isolating workloads in a environment such as the cloud by keeping it encrypted in use. So making creating a private isolated space in that stack and making this verifiable and accessible from the outside. That's the rough idea. In terms of our trust stack, the goal is that we can reduce this entire iceberg so that we trust the hardware that creates that isolated environment and our application. So we can eliminate the rest of the stack and don't get me wrong, this doesn't mean that confidential computing is a solved-it-all solution. We still want to use all of the other tooling and all of the other things like supply chain security and testing and the fuzzing and so forth protecting the front door of application. But in terms of trust and isolating against the infrastructure, the desire is that we don't need to trust all these intermediate steps, we only want to trust the hardware and the things we provide our application. And these isolated environments that we create with confidential computing are usually called trusted execution environments and you can define them by usually three properties. One of course is the encryption in use, the memory encryption and then there is some form of identity, some kind of way we can measure the identity of a workload. So that we can attest it remotely, we can verify remotely, that's exactly what we expect to be running in that environment. The latest generation of hardware usually does it by isolating VMs. So there's an extension to the processor that allows us to create an isolated VM. So the memory pages of that VM are isolated are only accessible from within that virtual machine contacts, not from the hypervisor, not from another VM or any other thing below in the stack. And that we can verify that entire context from the outside. And this is available in most of the server CPUs that you can get out there today, like Intel and AMD have solutions in their arm, has a specification that's yet to be seen in silicon, but that's already there. IBM has secure execution and there's also a risk five specification for that. They all are slightly different, but more or less apply that concept of isolating an entire VM. One note to that, what we've seen, I've mentioned the AI use case, and with AI we usually need to talk about GPUs. Now with the latest generation of Nvidia GPUs, they've released a concept where they have something in practice that allows to extend that concept also to the GPU. So think of an isolated VM and now we can establish a secure channel to the GPU and also have the GPU's memory being isolated for that context. So have the same properties as we have for the VM. And that means from within the VM we can securely use the GPU and process the data securely inside without losing the trust here. Okay, so we've seen the left hand side. We've seen that we can create a trust execution environment by isolating VMs, rating isolated VM in the cloud. So the question is now, okay, how does it help us with trust? How do I verify that whatever is inside that context is actually what I expected there and this is actually isolated from the cloud? How do we gain this trust? The first thing is we need to understand how the identity of such trust execution environment could look like. In the case of confidential VM, it can roughly look like this. We can have, it's usually being referred as a measured boot approach. That means we go from a well-defined state, the initial memory state of that VM, which is usually the firmware that's being loaded by the hypervisor. And we want to go all the way up in the boot chain so that in the end we have a booted VM that is verifiable from the outside. And the way it works is that every component in the boot chain is measured by the previous component. So the hardware loads the initial memory layout, loads the firmware, measures that firmware, creates a hash of the memory pages, stores that hash in a safe space, and transfers execution. And this continues. So the firmware loads the next component, potentially the bootloader, stores its hash, the bootloader then holds the kernel, the kernel loads an init-rd, and the init-rd might load a root file system and store the hash of that root file system, and then we transfer execution to system D or whatever our init is, and yeah, from there we can start, for example, a container on time in that VM. What we get then is a unique chain of measurements, specifically for that VM. So when we obtain that chain of hashes, we can verify that was exactly the image boot that we expected to be there in this VM. So given that we have this concept of a measured boot, what the CPU and the CPU features like an A and B SCV until TDX is what providers with is generating a report for that environment, and this report, of course, it contains some metadata, the firmware version of that CPU and other stuff, it can contain this chain of hashes, so that report is unique to that VM and we can identify the VM from there, and then it signs it with a key that's burned inside the CPU. So this key is only accessible by the CPU, so that we know that this was actually not generated by someone randomly, but this was actually generated by a CPU from, let's say, AMD that has access to that key. So the process of attesting that environment then is, we boot up this confidential VM in the cloud, we obtain this attestation statement, and then we need to verify it. So we need some ground truth about the measurements we expect. We need to be able to verify the signature, so there must be a certificate change with the hardware vendor that makes it possible to verify that signature. And once this is done, we can establish a secure connection to that environment, so think of this VM booted up. We use that remote attestation statement to make a key exchange, and then we can securely communicate with that environment and potentially send our data inside for processing. There's a proposal in RFC for remote attestation called RATS. If you're interested in the details and the inner workings, feel free to check that out. Yeah, what do we get now? We get the capability of creating isolated, secured environments inside a context like the public cloud. We can verify that from a mode. So essentially, we can create our own private space in the cloud, but not the cloud provider, and no other parts of the cloud stack has access to, and that's something fairly cool. Questions, okay, great. Now, how do I use it? Where can I use it? Where you can use it is pretty much most cloud providers have some form of confidential computing offering. It goes from basic bare metal machines with the CPUs that have the features to what's probably more interesting is infrastructure as a service, so you can create confidential VMs based on that, and then of course, higher level services as well. So the whereas is not a big deal. I would say everywhere in most clouds. The question is how? How do we use it? Since we're at the KubeCon, I think that's the most relevant question, and now we go to the juicy part of that presentation. How do we make use of this feature? Quick revision of the Kubernetes stack. Super simple here in this case. Yeah, you know, we package our application containers. These are usually grouped and managed in an entity called POT, and this POT is scheduled by a control plane by an orchestrator into some compute resources, some notes, and these are usually, when we talk about a cloud use case, usually VMs. So every node is a VM, and then we schedule our POTs inside. So how do we apply confidential VMs to that stack? I like to present three options here. You can probably find more fine-grained versions in between, but three main cases. One is we consider that as, consider our Kubernetes as one environment, one concise environment that want to isolate against the cloud provider, so we can deploy our stack inside while being isolated. That's one case, I would call this case the confidential cluster, and one implementation of that is called constellation, we'll see that in a second. The second case is maybe one step away is we don't consider the control plane, so we only take our worker nodes and package our worker nodes inside confidential VMs. You will get that from the hyperscalers, like a GKE with confidential worker nodes or an AKS. We'll also see that in a second in more detail. And then a slightly different concept is instead of protecting our cluster or protecting our nodes, we want to protect the workloads themselves. So protecting individual POTs instead of a cluster. The interesting question is how do we match this VM-based isolation to POTs? And also that we'll see. So the confidential cluster, as I said, the entire cluster should be isolated as a whole. What we need to do is run the control plane, and this could be multiple instances, of course, run that in confidential VMs, run the workers in confidential VMs. That's the rough idea. If everything runs inside a confidential VM, this whole thing can be seen as one shielded context. And then from within this context, I mean, it's just Kubernetes, so nothing really changes there. What are the challenging bits about that? First of all, of course, attestation. So how, roughly, how would attestation work in this context? So think of this. You want to create a cluster in the public cloud. You want to run all of the nodes in confidential VMs, and you want to attest this entire cluster so that every node inside this cluster has the correct identity and is running inside a confidential VM. We can, of course, as we've seen, do remote attestation with a single node. So we can connect to one node, test this node based on a ground truth like the expected measurements. And then, of course, we could repeat that process with all nodes, but this would be very tedious. Let's say you think more into the future of day two operations. You need to upgrade your cluster, roll out a new node image. You need to do this all over again or think of a scaling event for that cluster. So one alternative would be, instead of doing this sequentially, you do this more recursively and then verify the cluster transitively. So you verify the control plane or the first node of that control plane, then the control plane verifies itself so all of the other instances and just the control plane verifies all the worker nodes. Of course, therefore, the control plane needs also access to some verification, right? So it knows the ground truth and it knows it can verify this remote attestation statements, but given that, you can verify the entire cluster by basically doing this recursively. And then we can bootstrap this together. They can of course exchange credentials in that process and then create a cluster and all of the nodes are mutually authenticated. And with this concept, we need to also revise some soft problems like networking. If we don't trust, if you don't wanna trust the club provider, of course you need to make sure that all the networking is encrypted as well. So we keep the data secured in all states. And you can think of a lot of options here, right? You can implement this on the application layer. You could implement this via service mesh. Probably the most comfortable one would be to implement this on the CNI layer so that no matter how the application is being deployed, the connection between nodes is encrypted automatically. Yeah, there was an interesting talk on that from a colleague of mine at the Cilium con here, just focusing on how this can be done, for example, with Cilium. And then, yeah, storage, encryption address. Again, a soft problem, you would say. I can ask my club provider to encrypt the storage, the persistent volumes. The question would be if we do that on the server side, so we ask the club provider to encrypt the storage, we can say, hey, I have a key management system. So I keep control of my key, but yeah, if the club provider can encrypt the storage, they can also, of course, not encrypt the storage, so you trust that the storage is actually encrypted. So to keep the concept and to keep this isolated, we need to do this on the client side. Again, a lot of options here, application layer, but we can also decouple that, for example, into the CSI layer, encrypt the storage there, and the encryption needs to happen inside the confidential context. If we talk about maybe an external storage, like an S3 bucket, you can think of concepts like a proxy, but does the encryption inside this cluster for you. So yeah, we need to also revise those concepts in order to keep the state encrypted in all states, and make sure that the encryption happened in a context that we control. And yeah, as I said, there is an implementation. We do an open source project called Constellation that implements this concept of a confidential cluster. You can check that out on GitHub, play with that. There's also, for mentioning that, there's also a concept from the Reddit folks that presented a confidential cluster concept for OpenShift that follows similar patterns now. So that is the one option. I said there's maybe one step below, there's the confidential worker nodes. And to be clear, I don't think that's a good concept. The question is, of course, what's your threat model? But what that means is only your worker nodes are running inside confidential VMs. What you gain is the runtime encryption for whatever's running in that nodes. So consider the data center employee that I had on the iceberg. They couldn't go to the physical hardware, pull out that memory, and extract anything, because it's encrypted. But most of the other components in the stack, this could be a fun exercise for capture flag competition. I give you access to the control plane, you have full access to the control plane, all of the worker nodes are running inside confidential VMs. How do you gain access to those workers? Things would be, for most of you, probably a couple of minutes, since there is no isolation between control plane and the nodes themselves. What's more interesting though is the confidential containers project. That's also an open source project, and the idea here is that we isolate individual services, or individual pods, individual containers. Again, it's the infrastructure. So the main challenge here is then how do we get this confidential VM concept into isolating individual pods. And the idea that's being followed currently is based on the idea of cutter containers. If you're not familiar with that project, the idea for them is running every pod inside its own VM to isolate the hardware against that pod, so that you can run potentially malicious containers that have not such an easy way of escaping their environment and attacking the host. We turn that around, so we protect the pod from the infrastructure from the hardware. So that means instead of running every pod in its own VM, we run every pod in its own confidential VM. We can do this either nested or by a remote hypervisor concept where we say just create a confidential VM somewhere else in the cloud. We need to take care of the connection to that context. But nevertheless, the more demanding challenges and the thing that's currently being worked on is now that we, again, we don't trust the control plane here. We only trust the concept of the pod. How do we take care of orchestration and attestation? So consider you don't trust the API server. You don't trust the ACT. How do you provide a deployment file? Like, let's say you have a helm file, you deploy that. It goes through the API server. It could do whatever. So we need to restrict the access of the control plane to those pods to ensure the confidentiality whatever ever runs inside there. And then we also need, of course, make sure all the IO problems we also solve the confidential cluster concept are solved. So how can I consume storage? How do I do a networking? How do one container can trust another container and could securely communicate with each other and make sure that the other containers are also running inside a confidential context? So these are challenges we need to solve and for different environments, different solutions are being discussed. Unfortunately, I don't have the time to go through all the details. But if you're interested in that, there was a great talk from Magnus from Microsoft on the open source summit in September that basically focused just on confidential containers. So yeah, check that out if that's interesting. Yeah, again, confidential containers is a GitHub open source GitHub project. You can find it. Just to mention it, Intel also released a project called ACAN or ACON that is specifically for Intel TDX and follows similar pattern as confidential containers isolating individual containers based on Intel TDX. Okay, summary. So first of all, we've seen why this is interesting. Where is the threat model of protecting against infrastructure-based threat, isolating ourselves from the service provider? Where's that relevant? We've seen how we can solve the problem of providing that isolation in the public cloud, different concepts for different use cases, either isolating an entire cluster or isolating individual workloads. And we've seen that we can pretty much go ahead and implement that today on most cloud providers. Some references for further, for yeah, exploring that further. There was interestingly, there was a Kubernetes block article, the Kubernetes security block, I think it is, written by a bunch of people. There's the Confidential Computing Consortium that's part of the Linux Foundation, oops. That has some material on the topic. And of course, you have the open source projects. I think that's the most practical start and getting really hands on trying that stuff in practice. And if you're more into, oh, in our conference, the OC3 is coming up next year in March, where we have a bunch of topics on just confidential computing. If you're more into books, there's Mike Brazil's book on trust in computer systems. These are just for reference. But I'd like to conclude with a statement from Akrozinovich, the Azure CTO at OC3 last year, where he was setting, we're heading towards a fully confidential cloud. Just think a very promising vision and the kind of vision that changes the model we think about cloud computing today, where you can actually exclude the data from the service provider. And that's a very promising future. And there's a lot of problems, a lot of challenges to be solved. And yeah, if you find that interesting, as you've seen a lot of open source stuff going on, get involved. Hope that inspired somebody here. Thank you. To so many of questions. Yeah. Hi. You talked about the boot flow where everything, the first thing attests the second thing and all the way up to the OS disk. Secure boot solves all the problems until VM or machine boots. But what, how do you, what is that tool or component that does the attestation of the root disk, you know? As in traditional tools don't have it, so what are you doing? You mean, let's say you go up to the unit, or how does the attestation of the root disk work? Yeah, yeah, right now it only stops, as in Secure boot only stops before system disk starts. Right. What's that handover look like? So there are different ways people do that. To, for constellation, for example, the way it works is that the root disk is mounted as read only and integrity protected by DMVarity. So the DMVarity root hash is included in the, in it or D itself that's being measured. So you essentially hash that component and contains the root hash, which is then used for DMVarity to ensure the integrity of the root disk. Does that solve the answer? More questions? So for revocation procedures, how do you deal with that? So you have a, in the middle of the run, production run, the root is compromised or sign certificate is compromised. How do you do the checkup that during the... Okay, so the question is, for when we saw the attestation model, the attestation segment is signed by the CPU assuming that we can't trust that signature anymore or the firmware or whatever of the CPU and there's an update of that. How do we handle that? So with a constellation cluster, you would need to recreate another cluster and update, of course need to update the ground truth with that new certificate. And in place upgrade would not be so sufficient due to, yeah, you can't trust anything in that cluster anymore, yeah. So essentially we need to have another process that monitors the trustworthy of the signing certificate or the signing root. Say again, please. So we need to maintain another monitoring process, right? That would say, oh, is this signature still trusted? Within the cluster, there's no built-in mechanism to do that, right? No, the cluster itself does not have a reconciliation loop that automatically updates itself about the ground truth. So let's say, for usual cases where we just say like we want to deploy a new image because we updated the container runtime, we can push the ground truth into that cluster and tell it to only accept new nodes based on that new container runtime. Of course, the trust model here means if you consider that cluster being compromised already, you lost, right? But considering that we want to just deploy a security update, then we can do that. But given that scenario you described where we don't trust the CPU firmware itself anymore, well, then we need to update the security firmware and then the firmware of the processor and then go from there, yeah. I have one more question. So in the model where you are bringing confidentiality only for workloads or pods, how do you deal with logs and metrics? How do you divide it between, okay, this is application specific and this is cloud specific? Yeah, the short answer is this is not a solve problem. Yeah, logs and how you deal with observability and metrics in this case is there's no existing solution that does that perfectly for you. Of course, as you say, you can deploy the usual observability stack and then try to restrict the access or restrict what's being locked and try to infer what's application, what might be sensitive information, what not, but generally observability and confidentiality are, yeah. You need to level out and see what for that your threat model works or not. But are you seeing any patterns that are being employed right now or are users comfortable with giving away, let's say, specific metrics and then only keeping logs with themselves or some kind of mixed model? Yeah, so for constellation, that's where the most insights, since we have some production deployments there, is that usually the main goal is isolating against the cloud service provider. So if you have a metrics endpoint in what you call a safe space, then you're usually fine, right? And you log to that. Or you consider logs that are low-level enough that they don't contain sensitive information about the application itself. For confidential containers, this goes very much into the details now, but one of the concepts that's being discussed is, I think, I'm not sure what it's called, the data hub or something. So something that sits in below and takes care of any IO-related stuff or like storage and stuff, you can think of implementing some kind of logging probably also there. There was a research paper from IBM about how you can make a service mesh like Istio compatible with that concept, restricting the access from the Istio control plane to the data plane in the service mesh or the envoy sidecar service mesh.


Why narcissistic women end up dissatisfied and alone? The divorce rate is around 50% but maybe the other 50% just hasn't been divorced yet. The average duration of a marriage is 7 to 8 years but the first high risk period is the initial two years of the marriage known as the honeymoon period. The Philippines and the Vatican are the only two countries where divorce is illegal. 30 is the average age for couples that get divorced for the first time. The average time to remarry after divorce is three years. One in three divorces starts with online affairs according to a report. A husband and wife in Italy became the oldest married couple. Divorce rates around the world included them with their 77 years of marriage. However, the couple divorced after the 99-year-old husband learned that his 96-year-old wife had an affair in the 1940s. Increasing numbers of women are chosen to be single. 71% of older women say that they aren't looking to date right now compared with 42% of men aged 40 and older. Narcissistic women end up dissatisfied and alone and sadly, judging by the statistics, this could even be said about most women. So if you are a man who is looking to date, you need to be very careful because there is a high chance that the woman you meet will end up discarding you. Statistics show that one in three divorces starts with online affairs and that doesn't include unmarried couples. Where there is probably a great of risk, because there is generally less investment, there is less at stake. 39% of relationships start online, which may be a contributing factor to the high divorce rate because there's no way for you to test and examine them to see who they are. Narcissistic women are playing a role. They've molded themselves into an image of what they think you want to see and it's often based on someone to be persona. They're pretended to be someone different in order to trick and attract you. They're assuming a false identity and personality on the internet and often in person as well, which is not going to be good for you, but divorce and people struggling to find healthy relationships is actually very good for businesses because it creates a problem which gives businesses an opportunity to provide you with a solution. But the solution is never as cut and dried as you might like it to be because they don't really care about the results. You may think that you've met a good woman. You may think that you've met the woman of your dreams because you don't know what is going on behind the scenes.  I've seen women's Tinder accounts and they are thousands of likes and messages.  You have a good job. You have it together. But you're still not finding anything. You're not being recognized by women because everything is set up in their favor. You don't have a chance. Even if you do message a woman online, it's just going to get lost in the sea of other men she has messaging her. She may not even see it. So it's not going to have the impact that you may expect it to have because there's millions of men who are in the same position and women are overwhelmed with all of the attention and many of them are not even interested in dating anyone. Many of them are already in relationships even while they're on dating apps. So they're just using you for attention. They're not even taking it seriously.  They're just looking to use women for their own needs. And narcissistic women are onto this and they're using it to their advantage. They're using it to deal with men in a skilful and dishonest way because they have so many of them competing for their attention. And they don't have to do anything. So they don't really care about them. They don't value them because they have thousands of likes and messages. So it doesn't matter how good of a man you are. What difference does it make if they can't see it?  So any effort you put into these women is just going to be a waste of your time because you're competing against guys who will do anything for these women. Guys who will give up all of their time. Guys who will spend their last penny. Guys who know how to get these women's attention. Guys who display the image of a decent guy. They're going to put you out of the game because these guys are professionals at this. They have nothing better to do with their lives. So you're not going to outdo them because they have nothing else to do with their time. They can invest all of their time into indulging in a series of sexual relationships without committing themselves to anyone while you're at work or at the gym. While you're focusing on improving yourself and making your life better. So these women won't even see you.  Which is why they're not acknowledging you. They're not giving you any attention. They're not responding to you because it's all a game. It only works for people who are narcissistic. People who are only in it for themselves. People who are driven to get their needs met at all costs rather than people who are actually looking to build something with another person. Which is why the divorce rate is so high. Which is why narcissistic women end up dissatisfied and alone. Because they were only looking for situations that suited their needs. Which resulted in them engaging in behavior that was intended to deceive men. It resulted in false connections because they were pretended to be something they're not. Narcissistic women are playing the game. They're not really serious about anything. They know that most of these men and their DMs are just looking for sex but they're still on there. They're still taking pictures in provocative clothing and seductive poses. So if they saw it as a problem they probably wouldn't engage in those online activities anymore. They would find a genuine way to connect with people but they don't do that. They choose to remain on there. They choose to engage with guys who are only looking for sex because they enjoy the attention. So you're competing with the lowest of the low. You're competing with guys who are the most immoral and socially inferior of all.  Guys from the lowest social class because these are the types of guys who are pursuing these women. So even if you do manage to establish a connection they're just going to put you in the box of those other guys because they've already had so many interactions they've been around the block too many times. They have a lot of experience so it's going to be very difficult for you to excite or surprise them. They're not going to see you as someone special or unique even though that may be the case because anything you say they've already cured it a thousand times. Anything you do it's already been done by these master manipulators. They got there before you so there's nothing you can say or do to stand out from the crowd. You're just a needle in a haystack so nothing you do is going to influence them. Nothing you do is going to have an effect. You will feel invisible. You will feel like they don't notice you because they've put you all in the same boat. They've assumed that you're all the same. You're all identical in kind based on the small fraction of guys that they have dealt with which are reprobates guys who lack quality and character guys who are thirsty for sex so they never find the men that they are looking for because they're getting the lowest of the low so they assume that all men are the same because the good men aren't trying to compete for their attention they have more important things going on in their lives they have better things to do but narcissistic women want you to fight for their attention they want you to chase them and the only men who are going to do that are men who are needy and desperate men who don't have anything else going on in their lives men who don't have other options which are the types of men that they end up with and then they assume that all men are that way they assume that all men just want to have sex with them and they enjoy the attention they feel like they're the best thing since life's bred so it enables them to maintain their false image which means that you are going to meet a lot of fake people a lot of narcissistic women who are being whatever they think people want to see which is why a lot of them are having Botox and plastic surgery so a lot of what you're seeing isn't even real you don't know who you're dealing with it's just a fantasy and when you want out or you're trying to build something real they're not going to want anything to do with you because they need the attention so if they have the choice they're going to settle for all of their likes and messages than to settle with you they will discard a perfectly good relationship just for some attention so of course most women are ending up dissatisfied and alone because they're not taking men seriously it's all a game to them but not all women are like this there are good women out there but they're like a needle in a haystack you're going to find women who have already engaged with desperate thirsty men women who have already been around the block women who have had a lot of experience and know how to play the game and there's nothing you can tell those women they're not going to treat you any different they've heard it all before so they're going to treat you the same they're only going to stick with you if they can get what they want from you and if they can't get what they want they're going to find someone else you're competing with master manipulators you're competing with desperate thirsty men you're competing with the lowest of the low so they just put you in the same boat as the breast they assume you're all the same because they've had plenty of experience with those types of men so they expect you to be the same as everyone else and if you're not going to be what they want they're not bothered they don't care because they've got hundreds of other men under their control waiting for their command but these men aren't getting anything out of it they're giving away their money and attention for free which is why there's more sexually perverted men there's more men who are only driven by their sex drive who are constantly pursuing these women and there's no way you can compete with that if you're looking for a relationship if you show respect to them they're just going to see it as a weakness they're just going to see it as though you want something from them they're going to treat you as any other guy they're going to treat you as a creep and that is why in the end these women settle for guys they don't really want they settle for less than what they think they deserve but their eyes are always wondering they have a tendency to look at and have sexual thoughts about other men while they're already in a relationship they never find what they're looking for they never find satisfaction and most of them end up alone thank you for watching i hope this video resonates with you please like comment share and subscribe if you would like to donate my paypal link is in the video description coaching inquiries you can email me at coaching at narksurvivor.co.uk thank you for watching and i'll talk to you soon


Welcome to day two of KubeCon Chicago. It's great to be here. My name is Hemant, and today, we're here to talk to you about the hardest incident we ever had to deal with at Datadog. What is Datadog? Datadog is a cloud monitoring, security, and observability company. And at Datadog, we run Kubernetes at a large scale. We run hundreds of self-managed Kubernetes clusters, which help us manage tens of thousands of Kubernetes nodes. And some of our Kubernetes clusters have 4,000-plus nodes in them. So what happened? What was this hardest incident? So on March 8, 2023, our platform experienced a global outage, and our users were unable to access the platform for around 10 hours. And it would take us another 12 hours to recover the last major service. And this happened because we lost 60% of our compute capacity across multiple cloud providers distributed in several availability zones in a span of one hour. This is a story of how we lost almost everything everywhere, all at once, at Datadog. So during the incident, our internal Datadog employees realized that they were not able to access the Kubernetes control planes. And our admin teams were having trouble trying to even SSH into the nodes to try and debug what was happening. So at this point, it seemed like a widespread network outage. So did we roll out a bad change, fleet-wide? No. We actually have explicit policies in place to prevent our engineers from deploying code too fast. So how did this happen? In fact, we run completely independent stacks with no explicit dependencies between them. So there's no way we could have deployed something that can impact all of our data centers at the same time. Luckily for us at Datadog, we have a strong culture of incident response. So it was pretty easy for us to get all hands on deck pretty quickly. Our engineers who build these services own them end to end. And as you can see from the screenshot, we were having around 400-plus engineers working in different breakout rooms trying to figure out what is happening. Soon enough, one of our engineers figured out that we were able to recover some of the last nodes by simply restarting them from Google Cloud Console or via the GCP API. And once we recovered those nodes, we were able to SSH into them and take a closer look at the system logs. And that system logs told us that there was an unattended upgrade on these nodes. And soon after the unattended upgrade, these nodes were losing network connectivity. So what are unattended upgrades? Unattended upgrades are our legacy way of getting critical security updates onto our nodes. And at Datadog, at the same time, we've also been working very hard to build a node lifecycle automation platform that can replace thousands of nodes every day with minimal impact to applications. And it does that in a Kubernetes native way. But as you can imagine, this is a long migration. So at this point in time, we had both of these systems enabled. And if you're interested in learning more about this node lifecycle automation platform, my colleagues are giving a talk later today. Do check it out. So using this node lifecycle automation platform, we were upgrading our fleet from Ubuntu 20.04 to Ubuntu 22.04. And as you can see from this graph, we started our migration in November 2022. And by March 2023, 90% of our fleet was upgraded to Ubuntu 22.04. So we wanted to take a closer look at what exactly was included in these unattended upgrades. Turns out there was a patch to a vulnerability in system D that was being addressed in these unattended upgrades. But this patch had nothing to do with networking. And all we know so far is any node that receives these patches seems to be broken. And we know that this had something to do with system D. So we tried restarting just system D. And that alone was enough to break networking consistently on one of our recovered nodes. So is there a relationship between networking and system D? Yes. There's actually a component called network D system D that manages default networking on these nodes. So we tried restarting just system D network D. And that was enough to break networking on these nodes as well. And after some more investigation, we realized that this was happening only on Ubuntu 22.04 nodes. So we wanted to look at what really changed between Ubuntu 20.04 and 22.04. And we started digging into the commit history of system D network D. And we discovered this very interesting commit. According to this, system D network D would wipe out any IP rules that were installed by anyone but system D network D. And now things started to make a lot more sense. In order to understand how this would impact Datadog, let's take a closer look at how our Kubernetes networking looks like. So at Datadog, we use Cilium for our container networking. And in Kubernetes, we know that every part gets its own IP address and its own network namespace. So most CNI plugins install a few route table entries or IP rules to make sure that the kernel can do the necessary routing. And they can send and receive packets. So on system D network D restart, all these IP rules that were installed by Cilium were completely wiped out. And that explains why these nodes were losing network connectivity. So it turns out system D network D never needs to restart in the happy flow. And a patch for a completely unrelated CVE in system D triggered a restart in system D network D. And that restart of system D network D broke networking on these nodes. And on top of that, unattended upgrades run everywhere on our fleet between 6 AM and 7 AM. And that explains why we lost so much compute capacity in such a short span of time. So how did we recover from this? In order to walk us through what happened next, I would like to invite Laurent, principal engineer at Datadog, to walk us through it. So as you can imagine, recovering from this incident was pretty challenging. And it involved multiple steps. As Hemant was saying, all our regions were impacted. But they were impacted differently. For instance, on GCP, the web pages were not even loading. And on AWS, we had a web page, but with limited contents. So we decided to focus first on GCP. And the first step we had to go through was to get our Kubernetes clusters in a healthy state. So as Hemant explained, we had discovered pretty early on in the incident that we could simply fix a GCP node by rebooting it. That's simple, right? So first, well, we had to recover our Kubernetes control plane so we could know what the state of the clusters was. And this is what we started to do early in the morning. We started to recover the nodes of the Kubernetes control planes. And at this moment, we started to get a full picture of the impact of the incident, because we discovered that 60% of the nodes were not ready and down. And at this time, of course, we had to recover all these nodes. And this graph is showing all the nodes we had to restart. And each color is a different cluster. And of course, you can imagine that we have thousands of nodes in this environment, and it took us time. I said earlier that the AWS regions were in a better state because the web page was loading. So very quickly after we started working on fixing the GCP region, we started looking at the AWS ones. And we discovered that instances were running and healthy, but they were very recent. I mean, that had been up for only a few hours at best. And by investigating a little bit more, we actually discovered that the instances had been replaced. And the reason for this is, if you're familiar with GCP and AWS, on GCP, when you run an instance in a managed instance group, there's no health checks. So if the instance is failing its network, it's just going to remain there without the network. And if you restart it, it's going to be fixed in our case. On AWS, however, the autoscaling groups will detect the instances as being unhealthy, and the autoscaling group will actively replace them. That's amazing, right, because we auto-healed. Well, except we actually run a lot of data stores at Datadog. And these data stores quite often use local disk. Once again, if you're familiar with AWS, you will know that when you have local disk and you replace the instance, you lose the data. And here is a list of example of data stores we use. So we use a lot of open source ones and internal ones. Of course, all of them are very resilient to the loss of a single node or a few nodes. But given the impact of this incident, we had lost 60% of our nodes, you can imagine that in many cases, we had lost current and actually lost data. We don't store any source of truth in the data source so we were able to rebuild the data, either from backups or for other source of data. But of course, it took us time. So at this time, our Kubernetes cluster were healthy and we had a pretty good idea of what the state of the word was. However, we had accumulated a lot of backlog because, of course, our users kept sending us data. And so we need to process all this backlog. And to do this, we required scaling up many of our applications to process all this in a timely fashion. And luckily, at Datadog, we're cloud-first. And cloud is elastic. Well, the problem is, at our scale, elasticity is not always an easy promise. And a very good example is, for instance, that instances don't run in a vacuum. They run in subnets. And these subnets have a finite size. And we're making sure, always, that we can absorb the capacity of the loss of an ability-by-t zone. However, we scaled so fast during this incident that we actually ran out of IPs in some subnets. Another problem we faced is Kotas. Of course, we are being very proactive about the Kotas for the cloud providers at Datadog. And one of the very typical Kotas we're careful about is the maximum number of instances you can run in a project on GCP, the red line on this slide. And as you can see, during the incident, we always remain below the line. However, you can see a flat blue line in the middle. And it turns out, during that incident, we actually discovered a Kotas we didn't know about, which is the maximum number of instances you can have in a peering group when you have multiple VPCs peered together. Once we had discovered this, we had a very strong relationship with GCP, and we were able to create a ticket and very quickly get the Kotas raised. But, of course, it took us time. Another example here is we run Kotas ourselves. However, we heavily depend on Shopify or APIs. And, of course, when you scale very fast, you're going to ask a lot from these APIs. And in this example here, we can see an extreme one where we actually have a controller that is responsible for allocating network interfaces to nodes. And this controller was trying to allocate interfaces. But, of course, we were creating hundreds, thousands of nodes. And at some point, this controller was starting to get really limited. And because it was actively retrying, we got into a state where about 100% of this request were being reclimited. To recover from this, we had to decrease the pressure by reducing the number of nodes requiring additional network interface and by also asking AWS to increase the limits which they also did pretty fast.  Now that we had healthy clusters that were able to get additional nodes, we need to recover data, like all the applications we have. And, of course, we wanted to do this as fast as possible. So we wanted to do this in parallel. However, as you can imagine, we have a very complex Microsoft architecture. And as you can see in this example, the chain of dependencies is often pretty complex. And so we did our best to do things in parallel. But, of course, we had to make sure that we were recovering things in order in order to recover data back. So we learned a lot of lessons from this incident. And we're going to share a few today, right? The first thing is we're doing our best to make sure that our regions are isolated. But the problem is they're in the same operating system with the same configuration. And in this case here, this is why we had a shared phase, a shared phase, sorry, where all the instances worldwide failed at the same time. Another thing we learned the hard way is various simple abstractions, like instances and autoscaling, can actually leak and have certain differences between providers. I mentioned the difference between the AWS autoscaling group and the managed instant group in GCP, which, of course, it's a small difference. But in our case, it made debugging and understanding what was happening a bit harder. I mentioned the complex chain of dependencies we have between our applications. And, of course, during this incident, we had to restart the doc from scratch, right? And we don't do that often. So we actually discovered dependencies we didn't know about, which, of course, also made recovery a bit harder. And, finally, and this one is kind of obvious, right? Like, when you run at scale, everything is a challenge because when you want to do large-scale operations, you're going to put so much pressure on the underlying dependencies that you're going to face very interesting issues. So we really want to do better in the future. We want to continue to decrease the blast ranges and make sure that our regions are very well isolated, that our zones within a region are very isolated. We also want to degrade more gracefully, right? Because if we have limited capacity, we want to prioritize the data that matters most. And it's usually live data because it allows you to see what's happening in your systems and to have monitors. And, finally, we already do a lot of game days and chaos tests. But, of course, we've never tried something that big. But when we do this test, usually, what we do is we try and fail a zone, right? But nothing of that scale. And so we plan to do much larger, a much larger-scale chaos tests to make sure that we exercise our resiliency but also our incident response process. Something we really wanted to mention is we wanted to thank our partners there. So, first, the cloud providers were mentioned earlier because they were extremely helpful and they went out of their ways to help us that day. We also wanted to thank the Celium community, of course, because the Celium team very quickly created a patch to make sure that IP roles created by the Celium agents would not be replaced and deleted by SystemDNetworkD to make sure that this incident would not happen to anyone again. Thank you very much. It was a bit challenging to summarize this incident in 15 minutes. So, if you're curious and if you want more details, we wrote a very detailed blog post. And also, we're going to be around for the conference, Hemant and I, and also many other people from Datadug. Thank you very much. Thank you, everyone. Have a great day. See you soon.