Warning, the following awesome presentation contains slides with flashing images for storytelling purposes. Any events described may or may not be fictional. Hey, you must be new to this co-working space. Let me introduce myself via the form of a musical number. I'm just Ali, engineering skills X10. Is it my life to write YAML again and again and again and again? People love me here partially because I'm so great at singing in public places when everyone else is trying to work. But with my 10x ability, there is no we in this works today. It's an Aussie works co-working space. I work for Big Corp and I'm kind of a big deal there. But I'm sure you already knew that. Big Corp sent me to KubeCon 2023 in Chicago last year and I'm still wearing the t-shirt. There's been no security incidents since I've been back and I think this t-shirt has given me good luck. That's why I haven't washed it and you've probably noticed that smell. What? Now that smell? No, that must be my lunch. I've been microwaving some fish and I get so many compliments here about my food such as Aussie, you microwaving fish again? Sorry, co-working space is it. So what's that? You're just getting into Kubernetes too. Let me show you the perfect cluster that I single-handedly created. So this is Aussie's production cluster. We already have Big Corp running production workloads. I don't know what everyone has gone on about actually. Kubernetes isn't so hard. I've moved all of Big Corp's microservices into containers. Lift and shift, baby. And these are runners' parts. You can see them as at the bottom of the cluster. I've set up services to manage the networking between our services so all microservices can talk to each other. Big Corp already had a surge of demand after we put the cluster in place and we scaled to meet their needs. HPA! All this from a three-day conference, a technical masterpiece. Everyone in this co-working space is so lucky to have me here today. I am the perfect engineer. Right, I need to go to talk to management. I deserve a huge promotion. I'm going to go and do a warrior pose next to some random person so they know how important I am while I'm having these conversations. Hey, watch where you're going. Oh, excuse me. I'm so sorry. Hi, I'm Nova. I'm here in this co-working space today too.  He needs to be brought down a peg or two, am I right? I think I'm the one to do the job. You see, I look pretty friendly and I really am deeply kind. For example, I buy my friend's lavish birthday gifts with money that I make stealing data from people in the co-working spaces like this one. People really should be more careful. So basically, I hang out here, I pretend to work, but really I'm watching people and sometimes I see someone who gets a bit excited and leaves a desk with their computer unlocked. So I have this tool called a rubber ducky. It looks like an innocent USB flash drive, but to the computer, it's a keyboard typing at superhuman speeds. So I'm going to plug it into Aussie's computer and I'm going to cause some chaos. So while I do some stuff with this rubber ducky that I'll tell you about later, I'm also going to do more to mess with them in the short term.  So then he can't easily stop my running application. So now let's pull up my app in his browser. Okay, great. Now I'll lock his computer. Oh my God. Look at this lock screen. This guy is something else. Okay, I got to get back to my desk before someone sees me. Yeah, okay. So management said they're going to call me back later because they're kind of busy right now apparently, right? Let me just unlock my computer and wait. Nova rules Aussie drills. First of all, that's not true. The pillow was already wet before I went to sleep actually. And why is that URL pointing at big core? Nova has somehow accessed my new cluster. Well, I know Nova. I know all of them. Some fool got pwned by Nova the other day. It was a talk of a co-working space. Apparently Nova got some pictures of them and they transferred Bitcoin monies to Nova's wallet. And Nova is still asking for more. There was an investigation here to find out who Nova is, but no one's found anyone wearing a dark hoodie and an anonymous mask. I have to hurry up and delete this site before anyone else sees it. Shoot, I'm getting an error. I can't delete their workloads. The error message looks like my RBAC has changed. That's super weird. I gave myself most privileges so I can run this cluster. What's changed? Like here I have an offline config file of the cluster admin on my USB key that I have in a wallet alongside a photo of myself. All right, let's plug that in. Switch to cube config file. Great. I have cluster admin access again by using admin profile. I'll use this profile to make changes to the cluster to harden it, and I should be careful about managing this admin account though. We might need to access this in a break glass emergency, but we should have processes in place to make sure we know who can access it and when. Maybe having it in my wallet isn't the best idea. So we could use a secrets manager to store this securely, but equally having this offline in the safe might be a good option too. So I'm still using the admin account and I'm able to remove Nova's website. And the site's me for well, I better try harder. Let's update my R back so instead of having most privilege, I have least privilege when I connect the cluster with my RZ account. Enough to do the work when I'm supposed to do and nothing more. Right. Thinking about it, the API shouldn't be available to everyone on the public internet either. We already have strong security in place to ensure who can access the internal networks of our cluster via our cloud provider. So using my cloud provider, I will put the cube API onto a private network to ensure that all my systems can no longer access it on, but people in big core can and they can interact with it if needed. This is more secure, but I've hit a problem. How do I manage what's running in my cluster if I don't have direct access to cube API? I can only view the cluster now with my RZ account, but how do I create, apply, delete or patch my workloads? I heard about this thing called GitOps actually. It's like infrastructure as code, but for my cluster. So I can put all my YAML into a Git repo and then I can use GitOps to monitor that repo for any changes to the cluster configuration. I learned about CNCF GitOps tools like Argo CD, Flux and the Carval Cap controller that can help me achieve this when I was in Chicago last year. Right. Then the GitOps tool will be the one to interact with the cube API to deploy all the company's workloads. This all happens within the private network. But how does that help here? Well, instead of a cluster having configuration sent in, the cluster can go out and check a Git repo. Instead of giving away a key to the cluster to be able to run any workloads, I can use GitOps to read access to the Git repo and let that update the cluster via the cube API. Then I just need to secure the Git repo. Instead of the cube API being the threat boundary, it then becomes the Git repo. Right. Actually, I'm also going to use Cubescape. So Cubescape is a CLI tool that scans clusters, and Helm charts detecting misconfigurations. So I'm going to look into these results now to see if there's anything else that can change my cluster. I wonder what's going on over there. He seems pretty busy. But I've been watching the big, my big corp website in his browser, and sadly he's taken it down. His loss. It was a masterpiece. So remember I used that innocent looking rubber ducky hot plug to interact with Ozzie's computer? Let me tell you what I did exactly. First, I made a copy of his cube config file. So now I know the IP address of that cluster he made and he won't stop talking about. Actually, since I have this IP address, maybe I can put my website back up of his cube API server still public. Hold on. Since I know the IP address of Ozzie's cluster, I can use this software called inMap to scan the ports and it tells me he's made his cube API server private. Bummer. Oh, well, there's still a lot I can and I will do. So back to the matter at hand. With rubber ducky hot plug, I was able to procure the IP address of Ozzie's cluster with the cube config file. I was also able to get Ozzie's private SSH key. So now I have his identity. I also got his SSH config file, so I know where Ozzie's private SSH key can be used. And the cherry on top, I took the git config files, which show me all of the URLs of all of the git repositories that Ozzie's contributed to lately. So with this information, I can cause some real trouble making git commits as Ozzie. Huge promotion. Yeah, Ozzie, yeah, yeah, yeah. All wasn't by promotion. Apparently, BigCorp is losing money on each sale on the website. Instead of selling our product for $100, we're selling it for something around $15. It costs us $40 to make the product, so we're losing money right now. I linked our sales to a spreadsheet for the finance team. Let's check that out. Is your microphone working? Welcome back to YouTube. I see that instead of making around $100 per sale, everything is $13.37. What kind of idiotic, rookie liability co-working of mine changed the price of our product? Let me check the git commits for the payment service. It was me. I didn't change that code. I've been here all this time. You've all seen that. Here is my commit code that sets the value to $13.37. Someone must have my identity. All right. Let me just revert the git for commit. If only I could remember how to revert a git commit. Let me check chat GPT for one moment. Great. I think that's how I do it. That looks okay to me. Let's enter. Has that worked? No. Be quiet, please. I'm doing some super serious work here. I'm looking at some messaging. I feel terrible for causing you any inconvenience. Every day, it seems. Wait. Wait. Someone just committed this to me again. The price is back to $13.37. They've reverted my reverse. How did I set up my git account? I made sure I was secure and I used a private key. That's only on this machine. A private key is better than a password, but how has someone got access to my private key? I'm going to have to revoke my SSH key to stop any further commits. I've removed that from our git repo.  I'm more about that in a bit, but now let's check the log, and that looks good to me. We've reverted the reverse of the reverse. Perfect. Another way to sign our commits is to use a tool such as GPG. This provides a way for me to manage and rotate keys, and I could store it on a uber key rather than having it on my laptop. But in this instance, I used a tool called git sign, which provides a certificate to sign a git commit when I verify myself with OAuth, like you would on Google, for instance. This is then stored on SickStore so that the commit can be verified against my identity publicly. So now let's go back to the spreadsheet and back to $100, and we're okay again. Well, gee, captain, it's only Wednesday. Okay, this shows how important git is to us, not just my git commits, but access to git in general. If someone has my password, they could add a random key and commit as me again. I'll also set up a multi-factor authentication. So that way, to gain access to the repo, people will need to know my username, my password, and have access to my multi-factor authentication device. I should have got this set up. Dave, day two of a conference? I should have set this up long ago. I guess I was just too busy thinking how good I was. Wait, what? Back to me already? Ozzy wishes. I'm three steps ahead. Ozzy's about to notice that his cluster is running some random container image for his message queue instead of the company-approved image. Why is my cluster running some random container image for the message queue instead of the company-approved image? Well, that's weird. Our messaging seems to be working just fine. There haven't been any health status alerts. This must have been from a commit earlier on. Well, Nova seems to be a great multi-tasker. Or is there more than one Nova in here? All right, focus, focus, focus. We need to look into the message queue. It looks like that an image is being pulled from a random registry off of public internet. We put all our images from a registry we own with our cloud provider. Why would we pull a random image? Let's describe the message queue. Looks like the image name is Nova Images Generic Message Queue App. Tag 1, 3, 3, 7. Nova! All right, let's see what this message queue container is running. I can see that data is being sent to a random address, and I don't recognize that address at all. It's not part of the big corp IP. Oh, no. Nova's Nefarious Message Queue App is also sending information outside of a cluster. This can't be good. Why would it need to send information out to the internet? Right, I'll add Kubernetes network policies in place to ensure that the message queue doesn't have egress access. The message queue app should be taken requests from our website app only and passing it to our database app inside the cluster. There is no reason that any part of the message queue should be sending data out of a cluster. So, while Ozzie's been distracted with prank websites and product prices changing, I've been doing my real money-making attacks. So this whole time, the company message queue has been running my image from my container image registry, my message queue app, instead of the company-approved one. And my app seems normal, but I'm able to capture all of the information that's being moved through the message queue. And I'm going to sell it for big money. Wait, what? The information stopped being sent. Ugh! Ozzie must have put a Kubernetes network policy in place. As much as I hate to admit it, Ozzie's starting to get better at Kubernetes security. Lucky for me, Ozzie didn't notice the other container I put into his cluster.  Right, let's fix this message queue first. I'll restore the big court message queue by converting the git commit that changed it to Nova's application. And I've already set network policies. I've clearly defined which apps in my system are allowed to access egress, not the message queue app. But I can do better than that. Let's take a step back and look at our cluster again. Our cluster is an open network once inside the cluster. What if someone is inside a cluster? How could they intercept network calls between pods? It'd be good to have encryption set up. Like how HTTPS is used to prevent person-in-middle attacks within a co-working space. My cluster is like a co-working space for my apps. What happens if one of the apps starts intercepting internal network requests? So I'll add a service mesh like Istio, LinkD, Kuma, or Silium. And a service mesh is used to set up a network inside of our cluster. And it has some additional features that we didn't have before. We can encrypt all traffic between pods so that internal traffic can't get intercepted. This is what we call mutual TLS, or MTLS. So as well as MTLS, we could add authentication so it ensures that the services communicating with each other are who they can aim to be. This prevents impersonation attacks. We could also use automatic certificate management, which simplifies the operational overhead of MTLS. And like a certificate mismanagement was exploited in a famous hack in 2017 that cost a company half a billion dollars. Service mesh can further enhance our network policies, so this reduces the attack server scan by ensuring only authorized services can connect to each other. And finally, a service mesh can add auditing and monitoring, allowing for better anomaly detection so we can start tracing unauthorized or suspicious requests. I guess a service mesh could be kind of a big deal here. Now, what about those workloads coming into our cluster? If someone can access a cube API, they can apply whatever YAML they like. Then our cluster will run it. That's what Nova did. Let's see about using a cluster-level policy tool like Konovo or OPA to put a mission controller in place to make sure our workloads meets our requirements. Then we'll add a rule that any image in the cluster must come from big corpse internal registry. This prevents random images from being run in the cluster. Right, done, and I am feeling great. Now, what's that? The policy that I just added notified me that there's a problem. The policy fails for another image being used in the cluster. Another image is coming from a registry that is outside of the cluster while the image is failing. Nova images generic buildup, 1, 3, 4, 7, again. No, but not over again. When I was still able to commit as Aussie into Aussie's Git repo, so back before he set up multi-factor authentication, I started running a container that was disguised to look like Aussie's company's build service. It's a common rookie mistake for orgs to give extra privileges to their build service containers, and it's a mistake that I now intend to exploit. Aussie isolated his build service in a CI CD namespace of the cluster. Here, let me show you. There, so here's the build service in the CI CD namespace. Aussie put measures in place to isolate the CI CD namespace thinking it will protect him from attack. So from that privileged running build service container, I'll use a tool called NSEnter to connect to a different Linux namespace on the host machine. And now I connect to the host process on the node, and that gives me a little more privilege to gain access to all of the machine as root, and now I can see everything on the entire machine. Ha, ha, ha, ha, ha, ha, ha. Yes, maybe. Quiet, please. I'm going through a serious security incident right now. So back to Nova images. Generic build app 1337... Wait, 1337 is a hackathon for lead as an elite.  Not only is Nova running a random image, Nova has given a number of privileges to that container, including the process namespace of the host machine. But thinking about it, if you run a container with privileges, I'm pretty sure you'll be able to move laterally onto a machine running the containers. If that's the case, I'm not in a good position right now. Right. Let's update our runtime policies to prevent this kind of profile from being used nefariously. And using a runtime security tool with EBPF, like Falco or Cubama, all communications happening on the kernel and enforce policies for iCreate. EBPF is a technology that allows code to run in the Linux kernel without changing kernel source code or loading kernel modules. EBPF has a lot of the same features, service mesh, but for the kernel. It gives us lots of power, but as we all know, with great power comes... for the internet, great responsibility. The kernel is outside of a Kubernetes cluster, but I can track what is going on in the cluster at the kernel level. Just the cluster, it'll be everything on the machine. So here I am. I have access to this whole machine in Ozzy's cluster as root. So let me see if I can listen in on any of the network traffic going across the network. Ugh, the traffic's all encrypted. Drats, Ozzy must have preemptively added service mesh. Dang, he's getting good. Well, with EBPF, I can track what's going on in Ozzy's cluster at the kernel level. And it's not just Ozzy's cluster, it's everything on Ozzy's entire machine. So here's what I'm going to do. I'm going to use this EPF-based tool called BoopKit, which, as a side note, BoopKit was created by Chris Nova, who's a cloud-native security researcher, a wonderful human and friend. She inspired my name for this talk today, but she used her hacking skills for good, not for evil, like me. So right, I'm going to use BoopKit so that I can manipulate Ozzy's running kernel. The first thing I'll do is to create a backdoor. That way I can get back here easily at any time without having to break out of a container again. Next, I want to use BoopKit to see the system calls on Ozzy's Kubernetes node. Perhaps there's some data here that I can sell or hold for ransom. But Ozzy's getting good at the security stuff, so I'm going to go old school and I'm going to leave a calling card. A fork bomb. So if I lose access to this cluster, the process will get started and BigCorp, and more importantly, Ozzy will pay. Okay, for those of you who don't know, a fork bomb is a bash function that gets executed recursively. So it's a denial-of-service attack where a process continually replicates itself and depletes system resources. So basically, it crashes the whole system to restore starvation. So this is what you get for microwaving fish, Ollie! Wait, what? It didn't work. Oh no, I just lost access to the cluster. Oh, I didn't have enough time to set up the fork bomb. No! No! Completely shut out. Dang it! It's like you're having a rough day. Oh man, this guy, he really had to come into him.  You're not mic'd up again. I am. I'm just softly spoken in times of need. Okay. Right, okay, here's what happened. I looked to add Falco onto the node, but noticed that something didn't look right. It felt like someone was already there and looking to modify something or other, and I was at my depth, so I just called BigCorp Security and we put plans together to quarantine the cluster. The engineering team provisioned a new cluster at the same time. This new cluster is built with everything we've put in place over the last, I'd say, 21 minutes, and we've now directed all customer traffic to our new cluster. We can use forensics now on the old cluster to try to figure out how far and over it got. The best part is we have a brand new cluster which has been hardened. So can I tell them about the security features of the new cluster? Sure, but how do you know what I'm doing over here? Never you mind. I don't, so go ahead. So first of all, the Cube API isn't public. It can only be accessed from within BigCorp. The next security feature is that we have least privilege RBAC. So we have one heavily guarded accounts that can actually access the Cube API, and then regular users have only view-only access. But then, how do the regular users make changes? We do that because of GitOps. So the GitOps tool is the one interacting with the Cube API, and this moves the threat boundary out to the Git repo. And the GitOps tool has read-only access to the cluster. So the GitOps tool is pulling to the, not from the cluster, the GitOps repo, yeah. Pulling from the Git repo, yeah. Thank you. Applying to Cube API, and it applies the workloads to the cluster, and it keeps the workloads in sync with the state as defined in the Git repo. Then we set up multi-factor authentication. So someone shouldn't just be able to get in with an SSH key or with a password. We also have to have something, a secondary device, working out. We set up Kubernetes network policies. So in our story, this is what Ozzy used to stop me from being able to send data outside of the cluster and collect it and use it for nefarious purposes. So with the Kubernetes network policy defining exactly which applications are allowed to go outside of the cluster. Then we have service mesh, which can be used for a lot of things. But in this case, we talked about MTLS, which encrypts all of the traffic that moves within the cluster between any two applications. We also set up cluster-level policies using a tool like OPA, which stands for Open Policy Agents, or Kyverno. And in our case, we made a rule for the cluster that only images that are company-approved images that are in our registry are allowed to be run within our cluster. If you try to run any type of other type of container image, it will fail. And then finally, we set up runtime security. So this is at the kernel level. So this is watching system calls and making sure there's nothing going on that isn't expected. Yeah, pretty much. That's what you did. Just a wild guess. Good. So this was my cluster at the beginning. And looking back on it, well, I feel a bit embarrassed that I thought it was actually that good. But equally, I didn't know 30 minutes ago what I do now. So this is closer to what a hardened cluster should look like. And although there's room still for improvement, we can continue to do so. I know this looks intense, but when we break it down, it all starts to make sense. Think about it. And I finally understand what they mean by onion layer security. It's not about having a single layer or peeling that layer and then crying myself to 5 p.m. each day. But it's about having strength in depth. It's about having lots of layers of security to prevent lots of different attacks. So you've got the clicker, yeah? Yeah. Do we do that again? So hey, everyone. My name is Lewis Denham-Parrie. I'm the CEO of CubeCon Talks as this one. When I'm not being overly dramatic, I work for an amazing company called ChainGuard. And at ChainGuard, we're here rebuilding our images... Well, your images... We're rebuilding images. I'm making them all secure by default. So please come find us. And it is my absolute distinct pleasure to introduce you to the star of the show today. The amazing creative mind who put all of our ideas into these beautiful slides. Thank you. I am a developer advocate at VMware. I host three streaming shows, so if you're interested, I put stickers down here. I do lots of goofy stuff. This is kind of my thing, actually, it's your thing, too. That's how we work so well together. I also have... I started in tech about two or three years ago, and I've given two... two CubeCon keynotes in that time. I've given no CubeCon keynotes. Now I'm just super bragging. Okay, so that's me.  She really... When we put together... Thank you. When we put together the abstract for this talk, we did, from the very beginning, name our characters Nova and Ozzy. And we didn't realize it at the time, but Lewis and I both have very positive experiences with Chris Nova. And so we kept... When she passed, we decided to keep her name in just to homage to her as part of the talk. Yeah, so we jumped on the call, and like we haven't spoken about how Chris influenced our careers, and we just chatted about it. And for me, I met Chris back in London in 2017 when I was getting started at a Clive Native conference. I didn't know anyone else there, and I felt a little bit lost to say the least. But Chris invited me to go to... For a curry with her and her friends in Brooklyn. Great place if you're ever in London. Where I learned many of the things that she loved including mountaineering, or mountains, sorry. I'd often see her at conferences like this one, and I was always amazed that she remembered who I was. Yeah. But most importantly, she always made me feel welcome and made me feel part of the community. And I hope to pass a feeling on to others, a feeling of welcomeness. As I remember clearly, I came here for the technology, but I stayed for the community. I met Chris Novak, my very first cube con, which was only two years ago in LA. And we, as part of a group called SIG Bouldering, one morning a bunch of people who liked to boulder went to the bouldering gym together. And I spent that morning with her, and we went out to lunch, and then just she and I spent time in the pool. And I was nobody. I'd done zero talks. I didn't know anything from anyone. And she was so, so kind and kind and good, Deverel. And also how to put community first. So Lewis and I both definitely want to pay that forward. So to that, thank you for your time. Don't microwave fish and coworkers in spaces. And please be constructive with your feedback. We have two minutes if you there are questions. There's a microphone over there but Otherwise I've got my own questions right now. Excellent. Thank you so much, y'all. Thank you.


Alright guys, I have to admit there is something really bad going on with humanity now. I went to a dinner the other night and one of the first times I've ever been to a dinner in my life with people over the age of like 28 or 30 and everyone, besides me, pulled out their phone when there was a quiet lull in the conversation.  Now in this video I want to share how to stop using your damn phone so much especially because you're missing out on life which is what happens ironically when you're not on your phone. Hey guys, Alex Hine here, author of the book Master of the Day. So I'll tell you what, the first day that I went on with a girl that I dated for five years from 25 to 30, she pulled out her phone during like a very brief conversational pause after ordering food and I waited for like five seconds. And I don't know if she was on Instagram or texting her friend back but I pulled the phone out of her hand. First date, keep in mind. And I said, do me a favor, don't ever do that again. And she gave me this really like okay psycho look. And the funny thing though is, number one it worked out, we ended up dating for five years but number two, she became like a vigilante of people not using their phone while they're around their friends and she was a lot younger than me. So I was like 25, she was 20 but all the times when we would go out, a lot of her friends would be on their phone all the time.  But unfortunately since then it's only gotten worse because now the generation beneath me has been born with cell phones and born with technology and they're missing out on this thing called life. Kids now don't even know what not virtual reality is because they've lived in Instagram world or in all the other social media worlds. So how do you actually stop using your phone so much?  The first thing is lower the friction, right? It's very easy to use your phone number one if you have it on you all the time. So what I would start doing is what I wanted to get off my phone more is I would dedicate times of the day or days of the week where I would just not bring it with me. So if you go to school or you go to work, you could just as easily leave your phone for example, leave it in your bag for one class to begin with. Or if you're at work you could leave your phone in your car and then when you really need it you could go during your lunch break for example, then check and reply to all the social media stuff or your texts. So the first thing is lower the friction by leaving it somewhere else for a period of time. Now the second habit is actually to increase the friction on doing the things you don't want. So if you're on Facebook and Instagram just start by deleting the apps. Like yeah of course you can just go to Instagram.com and Facebook.com but just delete the apps is going to you know it's going to give you that like I really want to go to yeah screw it. Let me just focus on this. All right. And if that doesn't work put your phone on airplane mode during certain work blocks. So I've shared in some of my videos on productivity here that when I'm working not only is my phone not on me it's either at home in another room or it's on airplane mode with me. So I'm a big believer that don't rely on willpower and discipline that if you've already gotten to the point where you have to just be like no do not click like on the picture of the girl in the bikini you know if you get to that point you're probably already out of luck. The way you chop that friction barrier you either increase it or decrease it is by leaving the phone at home putting on airplane mode and removing a lot of the apps.  I know it sounds crazy but if you live a life that actually excites you guess what when you go to dinner with interesting people and you have interesting things to talk about you're not going to want to pull out your phone. If you're on a date with a pretty girl or a pretty guy and you know and you're really interested like what are the chances you're really going to pull out your phone or if you're going to a really cool meeting for people that want to climb Mount Everest like are you really going to pull out your phone when they're like don't do these three things or you will die on Mount Everest probably not.    Alright comment there below. Now the best way to stay in touch is start that personal development and weight loss challenge at modernhealthmonk.com forward slash youtube and in addition you can get the last two videos that just came out right here and right here and I'll see you soon.


Alright, so I didn't want to do this, but here we are. The title says it all. Patreon, the popular crowdfunding platform that every YouTuber in existence promotes at the end of their videos, is being used to fund neo-Nazis, and the company is doing nothing despite me bringing it to their attention. Now I get to spend time putting this whole thing together and shooting it out there for the whole internet to see and react to. That won't be exhausting at all. I have notes on my computer here, since this is a lot more extensive than usual, but it has a lot more information, and I just want to make sure I get everything right, so I'll be looking down a lot, but I'll also be putting up visuals, so just pretend I'm not doing that. The TLGW of this is that there is a book publisher called Castalia House that takes funding through Patreon, and it's run by neo-Nazis, who docs people, harass critics, and publish works that violate the website's guidelines. Despite all of this, reporting them has done nothing, and Patreon hasn't responded to my emails, so I guess I have to make this big public spectacle of this all. Check the pinned comment for links to overwhelm Patreon with reports and emails until they do something. Ideally, Patreon would enforce its own damn rules without being forced to do so, but if the last few years have taught us anything, it's that business ethics is a myth concocted by PR departments. The only way an internet company will ban users is if it becomes unprofitable to keep them around, and oftentimes the only way that happens is to generate a public outcry that damages their brand. So that's the goal here. I'm trying to generate a public outcry to get Patreon to do the job it should already be doing. Okay, so this all started when someone requested that I read this book. It's called Victoria, a novel of fourth-generation war. I heard it was a terrible right-wing fantasy, not unlike True Allegiance, and those are a dime a dozen, but I thought it might have been fun to mock it.  It's disgusting. Literally, the first page talks about the government executing gay people and it's phrased as putting flames to the faggots, and it only gets worse from there. The long and short of it is that this book is about the US government collapsing because there are too many Jews, black people, and non-fundamentalist Christians running around, and the big strong main character man makes a new country in New England. That darn political correctness prevents the government from properly dealing with all those felonious Negroes and drug-dealing Mexicans. Well, political correctness and the coterie of Jews that secretly runs all the world's governments and businesses. And I am not putting words in the author's mouth here. All of this is put in here with this exact phrasing. There's no subtext or reading between the lines here. It's all front and center. As an aside, a coterie of Jews sounds like the title to the worst young adult novel ever written. There are scenes where all the minorities are thrown out of the country or killed, the city of Atlanta is bombed, killing millions of civilians, and women are forced back into being baby makers. All of this is portrayed as unequivocally good. Both the characters and narrator make sure to point that out. Hell, there is a line about how marital rape isn't real and wives are obligated to give their husbands release at all times. I believe it comes right after all the Puerto Ricans are thrown out of the country. Most anyhow, the ones you aren't thrown out are executed.  Admittedly, it is funny at a few points. Like how the few black people that are allowed to stay can only do so if they agree to be sharecroppers and they're all led by this group called the Council of Responsible Negroes. That sounds like something from 1870, so it did make me laugh at just how over-the-top it is. Even setting aside the political stuff, which you can't because it suffuses every sentence, this book is terrible. The dialogue just regurgitates talking points. None of the characters have any personality and despite the author's supposed expertise in warfare, the main characters only ever charge right into battle and win through the power of friendship. It's an awful book in every way. I even found a few spelling errors in there. You might be tempted to think that this is satire, that it's someone making fun of the sort of person who would enjoy this, but looking into the people who made it, I can tell you it's 100% sincere.  Thomas Hobbes is a pen name for a fella named William S. Lindt. The name might sound familiar as he was one of the people who developed the concept of fourth generation warfare. Beyond that, he is an insane person.  For starters, when you Google him, this picture comes up, but that's not Lindt. Lindt never served in the military, though he sure acts like he did. He's always been very open about being a member of the far right.  At times, he's even been more naked about it than that, since he once claimed that Black Americans were better off as slaves and has openly worked with people who claim Israeli Jews secretly run the American government. Now, you might be familiar with cultural Bolshevism, which was a conspiracy theory created by Adolf Hitler and the Nazi Party to describe the way that Jews and socialists were working in secret to destroy the West by making weird paintings and songs, basically. That'll make us all gay and wreck the family, which will then collapse Western civilization.  That's what it is.  One of the biggest names that first promoted the cultural Marxism rebranding in the late 90s and early aughts was William S. Lind, alongside other human pawn scum like Pat Buchanan, Andrew Breitbart, and Jordan Peterson. Lind is also a monarchist for some reason. You might have heard of Veribus, people who are obsessed with the German army from World War II and won't let facts get in the way of their militaristic fantasies. Lind is their rarer cousin, the Kaiserbu. Kaiserbu's are basically the same, only they obsess over the Prussian and later German imperial army. He seems to think that the general staff was the height of military progress and that the dissolution of the German empire was the biggest tragedy in world history. His words, not mine. That's not even the worst Lind has to offer. Back in 2011, some prick named Anders Breivik murdered 77 people in a series of terrorist attacks in Norway.     That guy in Norway, my plan for white supremacy.  Before the attacks, he published a 1500-page manifesto explaining his views. Most of it was just copy-pasted from the writings of other far-ray crazies, including 27 pages directly from William S. Lind. Yep. Dude inspired at least one terrorist attack. But now onto the main reason for this video. The publisher for Victoria, as well as other far-right dribble, is Castalia House. Castalia is run by someone named Theodore Beale, who usually writes under the name Vox Day. I say it's run by him, really, Castalia is just him. It hasn't published anything from writers other than Beale since 2018, as far as I can tell. And even before that, it didn't offer much in the way of editing or marketing. It was just a way for Beale to gather like-minded victims of fetal alcohol syndrome under a single name before they all realized it was stupid and left. Beale is a failed musician, a failed video game developer, and failed sci-fi author who realized he can only achieve success by appealing to extreme elements in society and getting them to give you their money. So now he does that on his blog with a design straight out of 2004. Because if he shares their views, they'll pretend he knows how to write.   He's a holocaust denier. He's claimed that genital mutilation and honor killings are good for women. He said Jews living in America can't be trusted because of dual loyalties. All religions but Christianity should be outlawed. You know the drill. If you've had a laugh at the whole Sigma male thing that began recently, Beale was the one who started that. If you're unfamiliar, sometimes insecure men call themselves alpha males because they think if they act dominant then they'll get all the sex they want. And of course, alphas are on top of the social hierarchy that they invented, therefore they're superior to others.  Sigma males are just alphas who choose to exist outside of the social hierarchy, so they don't get as much pussy. Now all the self-proclaimed alphas have an excuse for why their cocks are as dry as the Beneville Salt Flats. It's because they choose to live that way. Totally. It's totally your choice to not have any friends or meaningful relationships, Ted.  Just like that time he tried to rig the Hugo Awards to get his book recognized. I'm more focused on his defense of the Taliban trying to murder Malala Yousafzai and that time he admitted to being a rapist. Doing stuff like all of this, even outside of Patreon, is explicitly against their rules. That's why they banned Lauren, immigration is bad but it's fine for me to immigrate to Australia, Southern, Milo the pedo, Carl of Swindon, and Owen Dennis Prager hates anti-Semitism but is fine with me on his show, Benjamin. The difference is that all those generated bad press for the company and this hasn't so far. But hey, maybe you can somehow argue that his situation is different. Maybe you think that he should be allowed to say abhorrent things. And I kind of agree, I don't think the government should come in and stop him from publishing books or writing books or whatever like that. It's just that it is against Patreon's rules and those need to be enforced evenly or they don't exist at all. You know what else is against their rules? Doxing and harassment, both of which Beale has done. In 2013, a woman left a negative review of his book on Amazon so he tracked down her address and published it on his blog telling his followers to go bother slash threaten her. Pretty sure that's what got Onision banned. And for the sake of levity, here's his reaction to getting banned. What am I gonna do? What am I gonna do? I'm coming to kombucha and I have no more Patreon.  As of this writing, Castalia rakes in $5,305 a month which is around $63,660 every year. That's after Patreon takes its cut. Remember, this is mostly going to one guy. I don't know how much he makes off his shitty blog or peddling his books but he's certainly not some starving artist. He's made a decent living with his grifting which matches his family history since his father ran a successful computer company before being arrested for tax evasion and possibly trying to murder the judge that presided over his trial. It's a weird story. But Beale came from wealth and he's kept it by pandering to the lowest common denominator not through hard work or skill. I gathered up most of this info two weeks ago and sent in reports to Patreon. They did nothing. Then I wrote up an email complete with links and screenshots that proved all of this and sent it into their guidelines department. They still did nothing. I got a generic email response that said they don't share details related to their investigations and since then there's been silence. I even got a few people on Discord to send in the same email and nothing has happened. It's pretty clear by now that Patreon won't act unless we force them to and the only way to do that is with social pressure. So I guess let's do that. I put a copy of the email I wrote on a Google Doc complete with links, screenshots and the address to Patreon's guideline violation address. I tried to stick the worst of it in there but there's still plenty of material. So if you guys want to add some more on there, feel free. And for some reason I decided to link that down below so if someone wanted to send the company an email letting them know how upset they are with this that would be super easy to do. I also linked Kastalia's Patreon page so you can easily report it and send in links to Beal's nasty behavior to Patreon. You know, if you feel like it. The whole point of this is to make so much noise that the higher-ups at Patreon become afraid of the hit their reputation will take if they don't take care of the problem.  Share this video, or rather spam it all over the place until it annoys everyone enough to get something done because that's the only way they'll listen. And before you get your panties in a bunch, this is not censorship. Beal and Lynde can write whatever they want and try to sell it to other sexually frustrated losers. They just don't get to flagrantly break Patreon's clearly stated rules. At least, they shouldn't be able to. But we've already gone over how capitalists will look the other way when it comes to fascism.  Do you know why? Because he's not a threat to me. He's a sad, middle-aged man looking for validation and attention. That's why he tried to rig the hugos. That's why he started the Sigma Male craze. He wanted to pretend he's a misunderstood badass who voluntarily left society.  Most right-wing rifters started as failed artists. Ben Shapiro wanted to be a screenwriter. Steven Crowder wanted to be a comedian. Steve Bannon wanted to be a producer. And Ted Beal wanted to be a sci-fi writer. They all sucked at what they tried to do and rather than work to get better, they blamed their problems on political enemies. Conservatives and right-wingers are not discriminated against in Hollywood or the arts. Clint Eastwood's career has been going strong for 66 years and he's pretty open about being conservative. None of his views are hateful. It's just stuff that a lot of people would disagree with. You know why his career is unaffected? Because he's good at his job.  He was even a mayor once. He's one of the most successful people in the history of the film industry. That's because he has skill and talent that these other losers don't and on some level they understand that. It's much easier to whip up a crowd by telling them what they want to hear than to spend time honing your craft. These people are allergic to hard work because they'd always had things handed to them and never had to struggle. Not really. They come from upper middle to upper class families. If I wanted to scare the liberals, I'd call them petit bourgeoisie but that's not the point. The point is that this is where fascism springs from. Mediocre, rich douchebags who don't think they've been given enough and that they're entitled to more.   That's more fun. It's way more fun.  I made this because I felt it was something that needed to be done and because I'm not ashamed of my beliefs.  Ted Beale and his ilk and the people who run Patreon are. That's the closest thing to a greater point I can glean from all this. Hopefully that makes it worth your time. Be sure to spam that hashtag and I might as well take an opportunity to trigger them some more so trans women are women. God isn't real. Gender and sex are different things. Race is a social construct and the American Civil War was fought over slavery. All of those are verifiable facts and facts do not care about your feelings.  Follow me on Twitter.


Hello. My name is Derek Parker. Today I'm going to be talking about deterministic debugging specifically with a tool called DELV. Now, before I start getting into a lot of these concepts and kind of really getting into some of the kind of nitty details and things like that, who has heard of or is aware of or knows about the concept of like deterministic debugging or some sort of record and replay type thing? Okay. So one, two-ish. That's perfect. That's absolutely perfect. One of the reasons why I chose to give this talk is kind of specifically for that reason. I feel like this is a debugging technique that is very underutilized, underdeveloped, and underappreciated, really. I think it's very powerful for even a generic class of bugs, but especially powerful for certain bugs which may be hard to reproduce. That's kind of the sweet spot of this style of debugging. But there's also other benefits that kind of layer on top of it. And I'm looking forward to presenting this to everybody, and hopefully everybody will walk away with an appreciation of this style of debugging. Now, it is deterministic debugging with DELV. If you haven't heard of DELV, I'll explain it a little bit more later, but it's specifically a debugger for the Go programming language. However, not everything in this talk is going to be specifically about Go. A lot of the concepts and even some of the tooling that I discuss are relevant to any programming language, C, C++, Rust. You can apply these concepts to anything, really. But I'm going to specifically talk about it in the context of Go for reasons which I'll explain a little bit later. Now it's going to stop working. There we go. Okay. So as I mentioned, my name is Derek Parker. I am a senior software engineer at Red Hat. You can find me on Twitter at DirkDerek. At Red Hat, I work on the Go language, so I work on stuff in the Go compiler, upstream work. I do work for the Go tool chain for REL specifically. And I also work on DELV, which is a project that I created. We now are going to be shipping that as part of the Go tool set package that we ship in REL. And so I maintain it as part of my job at Red Hat. As with our previous speaker, I'm also previously CoreOS, although I went to Red Hat on purpose, not as part of the acquisition. And as mentioned, I'm the creator of DELV, the Go debugger. I maintain it now with one other co-maintainer and the community at large. So I want to give everybody an idea of kind of the pace of this talk, everything that we're going to go through and all of the major points that we're going to hit. So the first thing that we're going to talk about is what deterministic debugging is, which I think is very important, since it's kind of a concept that is not very well known and that's going to set the stage for a lot of the rest of the talk. Second, we're going to talk a little bit about how it works under the hood. I don't know about you all, but I love to know how things work under the hood. I just like to get those kind of details. So in any talk that I do, I try to do a deep dive into whatever subject I'm presenting. So we'll get into a little bit of a deep dive of how all of this works, which hopefully everybody will find as interesting as I do. Following that, we'll talk a little bit about how DELV leverages all of these concepts for debugging Go programs and some of the pros and cons of that and how you can do it in your daily workflow for finding kind of hard to reproduce issues or just having a more sane debugging experience. And then at the very end, I'll do a little bit of a live demo. Nothing crazy, but just showing off some of the concepts, kind of going through some of the commands, how to use them, just more practical and showing off examples of how everything works. So first off, what is DELV? And second off, what is that weird little thing in the corner? So I have a five-month-old son, so I've been watching a lot of Toy Story. This is a new character in the Toy Story movies, and he always asks, what is things? So I thought it'd be funny to put in the slide. The context is lost on anybody. It's all right. It's more for me. It doesn't matter. So let's talk a little bit about what DELV is. Another kind of poll of the room, who knows about or has used DELV. Cool. So a decent amount of people. So for those of you who don't know, as I mentioned a few times already, DELV is a debugger for the Go programming language. There's plenty of debuggers out there. Why another one? So DELV does certain things that are Go-specific. So it's able to do specific coordination with the Go runtime for doing things such as making function calls in a Go process as it's being debugged. It also knows about the concept of Go routines as a first-class citizen. So as you're stepping through the execution of your program, you want to make sure not only that you're saying on the same thread, you want to say on the same Go routine, which is a concept that's held within the Go runtime, not exposed by the operating system or kernel. And so DELV specifically knows how to figure out what Go routine you're on and follow it, even as it's kind of multiplex and moved across threads. So that's a very important concept. It also knows about Go types and things like that. Supports all major operating systems but for right now only works on X86, 64 and ARM64. We're working on adding more arch support in the coming year. So yeah, I kind of explained this already. So I also want to give a shout out first to my co-maintainer for DELV, Alessandro. He actually did a lot of the plumbing to get some of this stuff to work that I'm going to talk about today. I just get to come up and brag about it. But he did a lot of the, you know, I just reviewed a lot of the pull requests and things like that. But my co-maintainer did a lot of the plumbing hooking all of this stuff up. So I wanted to give credit where credit is due. Okay. So what is deterministic debugging? This is another reference that might be lost on people but hopefully not. Has anybody seen the movie Groundhog Day? Yeah, a couple people. So for the people who haven't, it's a movie where the actor Bill Murray, he goes to this town, punks its honey, and it's on Groundhog Day. Groundhog Day is a tradition where a groundhog comes out of the ground. If he sees a shadow, I think it's more winner or something I can never remember. Anyways, in the movie he wakes up and every day is Groundhog Day for like years. Like this just happens for a really, really long time. So that's kind of what deterministic debugging is. It's replaying the same execution over and over and over and over again and nothing changes. Every time you replay it, it's exactly the same. You're executing the same exact sequences of instructions. Memory layout is the same. Thread IDs are the same. We try to remove all kinds of non-determinism so that every single run is the same. So essentially you're debugging the same exact sequence of events over and over and over again. It's predictable. You're always going to hit the bug. So if you have a bug that maybe only hits one out of a hundred times or one out of whatever, that one time that it actually hits, you can record that and replay it endlessly so that you can figure out where the bug is without having to wait another 99 runs until you hit that bug again. So that's the main benefit of this. It's hard to find bugs. You're able to record them once, replay them indefinitely, and you don't have to waste a lot of time trying to reproduce them. So it's like Bill Murray waking up in his Groundhog Day every single day. So how is this possible? How can you actually record a program and replay it? Every time you run a program, it's something different. The addresses and variables are slightly different. If you do get time of day, it's going to give you the actual time of day. If you call for a random number, it's going to get you a new random number, hopefully. How do you take out all of these variables which may or may not be leading to your bug so that you can reproduce it every single time? So the solution here is called record and replay. So essentially what we're doing is we're running a process, executing it, and recording certain things about the execution of that process, saving that information, and then replaying that back once we replay the process again. So the goal is to take out or record as many sources of nondeterminism as we possibly can so that when that process is played back and, say, a syscall happens or something like that, we can just replay the result of that as opposed to actually having to make the syscall or actually having to do certain things. Basically making your program more of like a black box. So we're just kind of replaying inputs into it as opposed to it actually making some of these syscalls and things like that which could be different on every single run. So I'll go into a little bit more detail of how this actually works under the hood, but basically we can accomplish this using a combination of things like ptrace, hardware performance counters for measuring where we are in the program so that we can do things like sending signals at the right place and scheduling threads, and even doing a little bit of second ppf filtering to mostly for performance. So there's a few different ways that this has been tried kind of over the years. This is like a relatively new-ish technology, but people have been working at it for a while. So you have kind of some of the heavyweight solutions such as like VM recording. So recording everything that happens within a virtual machine. So VM workstation actually used to do this, but it doesn't anymore. They took out support for it. It's clunky. It records more than you want. Likely you only want to record maybe one process or a group of processes that kind of form from that initial process tree. So the traces ended up being huge and it's just overall clunky. You don't really want to play the state of the whole virtual machine, the kernel, everything like that. Another attempt at this is user space reporting. So there's products out there such as UndoDB, TotalView, ReplayEngine, and Mozilla RR. I'll go into it a little bit more, but what we end up focusing more on is Mozilla RR. So some of these other tools and technologies do kind of undesirable things such as instrumenting code and things like that which I've never really been a fan of in the grand scheme of debugging as I feel like instrumenting the code kind of fundamentally changes it. You don't really want to change the code. You just want to debug it, right? So I think anything that's intrusive like that, I've never really been kind of a big fan of. So nothing is perfect. There's always problems with some of the implementations that exist. So I'll kind of go through some of them. I touched on them a little bit already. So with VM recording, you have to debug within a virtual machine that actually supports the feature. There's not really a lot that exists and I think the ones that you have to pay for. But like I said, even VMware Workstation, they took the feature out. So there's not really much that supports it anyways. It produces large traces. This can be a problem for if you need to store a lot of these things or move them around from computer to computer and debug locally or something like that. It can be very large and unwieldy. Yeah, and not widely supported, as I mentioned, it's kind of been taken out of some products. So user space recording, you're not recording kernel execution. That doesn't actually end up being a problem, as I'll explain later. We don't really need to record kernel execution just what the result of what we asked the kernel to do was. Some require code instrumentation. Again, something that I'm not a big fan of. I don't like fundamentally changing the code. And some require kernel extensions which kind of destroys portability. You want to be able to just kind of take this piece of software, put it on a server, put it wherever, and just kind of be able to let it do its thing without having to worry about if you have certain things installed. So there's a lot of downsides. Sounds terrible. Why am I even talking about this? What's the solution? So the solution for now is Mozilla RR. Mozilla RR gets around a lot of these problems. So it tends to be very lightweight. It doesn't do any kind of code instrumentation. It doesn't run your program within a VM and record that. It's just kind of like a lightweight program that you can throw around different places and point it at a program and tell it to record it. So how do we actually eliminate sources of non-determinism? During the execution of a program, there's a lot of things that can happen that are different with each run, right? The result of a sys call, getting the time of day, things like that. How do we, even thread scheduling and when interrupts are fired and things like that, how do we kind of get rid of all of this non-determinism so that we can replay the process execution back faithfully? So let's go through kind of some of the sources of non-determinism and what we can do about them. So certain CPU instructions, right? It turns out a lot of the CPU instructions are deterministic, right? You give them an input, it's going to do something on that input and then put the result of that someplace. Every time it's going to be the same computation. But there are a few CPU instructions that are different every single time. RD-rand, generates random numbers, hopefully going to be different every time, right? We hope. So in the case of Mozilla RR, they kind of end up patching, they find that it wasn't used that much, like on Linux, DevRandom is used, so libstandard C++ uses it, so they kind of patch it out of programs that use it. RDTSC, reading from a timestamp counter, obviously that's going to be different every time we do it. So that instruction can actually be trapped at the CPU level. We can record the result of that and replay it every time it gets trapped again. CPU ID, so this is mostly deterministic, but it does return the index of the running core. So on newer kernels, this can actually be trapped, but RR can kind of use some like CPU affinity to pin to a certain core, so we can get rid of this non-determinism as well so that you can run it on the same core. Thread scheduling. So as your program is running, different threads are running, right? And they can be scheduled at different times by the kernel. So how do we get around that? So RR runs only one thread at a time. So this could be a downside for large, highly parallel programs, which go programs kind of do tend to be. So in practice you could see a lot of slowdown during recording and replay, but we're doing this for debugging anyways, and it's still a relatively new feature, so ideally this will get better in the future. So we run a single thread at a time, and RR kind of does the thread scheduling. So all of the threads are managed by ptrace. When a thread enters a syscall, we know if it's a blocking syscall or not, and we need to run another thread, and so on and so forth. So we can kind of control thread scheduling and play that back in a deterministic fashion. So syscalls, right? System calls can return file descriptors, map memory into the process, things like that. They return data to user space using registers and memory and things like that. So how do we record everything that's happening in syscalls? And how do we do it in a performant way, too? So again, under the hood, RR is using ptrace. So each system call is trapped. Some are filtered by a second BPF filter, which kind of goes into this interception library that is injected where kind of common syscalls can just be recorded all without having to do like a ptrace context switch, so it improves performance a little bit. But for the rest ones, we can just use ptrace stop when we hit a syscall, record the values of it, and then play it back. This even holds true with exacting programs. So RR can play back recordings where a process execs and forks another process. It will record that other process as well and play back that as part of the subtree. So it's pretty powerful. So memory layout. RR, when replaying, will reproduce, recreate the memory layout that was there initially, meaning all variables are going to have the same memory address and so on and so forth. So it removes that kind of nondeterminism as well. Shared memory. Again, RR does single thread at a time, so within the same process, there's not too much raciness between shared memory, so it kind of eliminates that as well. So it's not the best for detecting race conditions and things like that, but it avoids sharing memory issues between threads. There are issues where some applications will share memory with maybe like a device driver or like X or something like that or pulse audio, but RR specifically disables things like that. Signal handling. So signals can be delivered to your program at any time, your process. How do we replay that? So under the hood we're using performance counters to track when that signal was delivered combined with certain register values like the PC value so that we can replay that and send that back at the same time when we're replaying the execution. So let's talk about some sources of nondeterminism that we can't control. So those were all things that we could control. Now we're talking about things that we can't control. So code executed outside user space in the kernel. Turns out, as I mentioned, that doesn't really matter. We just record the result of whatever it does. Hardware failures. If your hardware fails, there's really nothing that we can do about it. We can't make it fail again when we're replaying it. So that's just, you know, something that won't be able to be reproduced. So let's kind of go over some of the pros and cons of this style of debugging. So the pros, it's pretty low overhead. It's relatively performant. The largest hit you're going to get is from running everything within a single, kind of as like single threaded and especially for go programs that you can incur a decent amount of slow down. But I feel the benefits outweigh the negatives here for being able to kind of consistently reproduce a bug. Another yeah, then that's the next pro. You'll always be able to faithfully reproduce your bug. For things that happen intermittently, this is extremely important. But even for bugs that you can reproduce fairly easily, the ability to kind of, one of the things that you can do with these recordings that you can't do with just a typical process is do reverse execution. So instead of having to exit a debug session and start all over again or anything like that, you can get to the source of the bug and work your way back, or you can get to where the bug is exhibited and try to work your way back to find a solution instead of always having to try to work your way forward to get to try to piece together what went wrong. You can start at the moment that things went wrong and kind of work your way backwards if you want to, which could be a little bit easier for being able to find your bug. You can record a trace and replay it anywhere. So there are some limitations on where you can record. Some VMs don't support it like virtual box. You need access to the hardware performance counters for this specific style of recording and replaying. But being able to record it maybe on a server or something like that and then debug it locally on your laptop can be pretty powerful. And then you don't have to have access to that program every time. That trace is always going to be there. You can always look at it and run through it again and debug it. As I mentioned, you can execute programs in reverse, which tends to be pretty powerful. And you can replay from the beginning and execute the same sequence of events. So you can just kind of restart the process over and over again, and it will always be the same. And I'll show a little bit of that during the demo. Okay. So some of the cons that I had mentioned, performance hit for highly parallel programs. This can be a downside for go, but not always. I mean, this is only really if you're doing a lot like actually in parallel. So in practice, there is some slowdown, but you can definitely get away with it. Another con maybe depending on your view of the world is that it only works on Linux. So there's currently no hard set plans to port this over to OS X or Windows or anything like that. It does use specific things of the Linux kernel, the proc file system and things like that to do what it does. So it's a little bit less portable that way. And also as I mentioned, it only works with like VMs or Windows or hardware that supports performance counters. This is needed for a lot of the replay and recording certain events that happen. Great. So that's kind of an overall explanation of what it is and some of the details of how it does certain things under the hood. If you're interested about, that certainly wasn't like an exhaustive representation of what it does under the hood, but if you're interested in learning more about like how technology like this is implemented, RR did deliver a white paper, which kind of goes over a lot of the design and a lot of the low level functionality that it does, why it does what it does and how it does all of this recording and stuff like that. So if you're interested in learning more, I would seek that out. It's pretty easy to find just off of the actual website. So let's talk a little bit about usage with Delve. So how do we use this with Delve and how does Delve leverage it? So it turns out Delve, we made it so that it can use certain different back ends, right? So we've written our own native back end for your typical debugging flow that's completely written in go. We wrote it from scratch. But we can also plug it in using the GDB serial protocol to RR. And we also talked to, say for example, LDB on Darwin OSX, because they make a lot of changes, kernel changes and things like that that can be difficult to keep up with. But that's neither here nor there. So Delve can talk to different back ends and RR happens to be one of them. So what happens is you get the power of RR and being able to record and replay a program coupled with Delve running on top of it. So you get all of Delve's knowledge of go internals, go execution model, knowledge of types and things like that on top of this record replace system. So it ends up being really powerful for go. There we go. Okay. So how do we use it with Delve? First, be using Linux. So you have to be running on a Linux machine or on a Linux server or something like that. As I mentioned, this is all pretty specific to Linux. The first thing you're going to want to do is actually install Mozilla RR. So if you just go to the GitHub, their releases page, they have RPMs, dabs, tar balls, and you can also build it from source. They have documentation on how to do that. It's fairly easy and straightforward. And then once you have it installed, we have kind of three different options for using the RR back end through Delve. I'll go through each of them in a little bit of detail and then I'll do a little bit of demoing. So option one requires a little bit more work, a little bit more setup work. So it's a little bit more manual and I'll run through it in the demo and everything like that. But it can be one of the more flexible options for doing this. So the first thing that you're going to want to do is run, go build, and that flag at the end basically disables optimizations, certain optimizations, some variables, certain inlining, and things like that. Go has gotten a lot better with providing debug information for optimized variables, but it's still not perfect. So the best thing to do is just kind of cut that all out. You have to actually force go to produce unoptimized binaries. It optimizes by default. So produce unoptimized binaries for the sake of debugging. You can do this on just regular standard go binaries and as I said, with modern go compiler, the debug and dwarf information that they put into the resulting binaries has gotten a lot, a lot better. So you actually could have a really, really good experience. But if you do have control over it, running unoptimized can produce a slightly better result. So after you've built your binary, just run RR and the path to your binary, what it's going to do is it's going to execute your program, and during the execution of your program, it's going to record it. Once your program terminates, it'll save that recording to a trace directory somewhere in your computer. It's configurable. By default, it throws it, I forget exactly where, but there's a default directory, but you can control the output of it. Once that's done, the trace is there, and then you can use delve to replay that trace. So it'll load up the trace and start a debug session. From there, you can treat it as like a normal debug session. You can start setting break points. You can continue everything like that. But then once you hit break points, you can do things like reverse execution. So you can rewind from one break point to another break point. Or you can restart without actually having to restart the entire process. You're really just restarting the recording. So it's like hitting play from beginning on Netflix or something like that. It's really, really easy, and you're still replaying the same exact sequence of events every single time. So that's option one. Option two is a little bit easier, but could be frustrating for intermittent bugs. So one of the main entry points into delve is the debug command. So basically what that does is it will compile your program for you, turning off optimizations, and then it will exact that program for you, and then begin debugging it and land you in a debug session immediately. So it's pretty much like one command does everything that you would want to do to start a debug session. And it sets everything up in the most ideal situation so that you're going to have the best debugging experience. The problem with this could potentially be, as I said, if it's an intermittent bug, you're going to have to run this a lot. And if it doesn't fail, you're going to have to exit the debug session, try to run it again, and do this over and over again until it fails. Once it finally fails, then you can start figuring out where the problem is. Also, with this, by default, we delete the trace directory. So after you end the session, the trace directory is going to be gone unless you copy it out during execution. So I don't believe we currently have an option to save that, which could be a good thing to add. But as of right now, it's going to clean everything up afterwards. So if you want to replay over and over again after this session, you're going to have to kind of manually copy it out. So option number three, which solves some of the pain points with that approach is using some scripting within Dell. So you can script the debugger, excuse me, using a language called Starlark, which is a Python ask language that was originally created for the build system basal. So it's kind of been adopted, made it to slightly more of a general-purpose language, but it's kind of a sandbox-safe language, really good for scripting things like this, like the debugger. So you don't have to memorize all this. I can show where this is, but we have an example of this in the documentation. So you can write just a little script, a little command. So basically what this does is it creates a new command within Dell called flaky. And it starts the process, and essentially it continues to run it over and over and over and over and over and over and over again automatically until the process fails. Once it fails, it stops and puts you into a debug session. And you're going to be debugging that recording. So it rerecords the program until it fails. Once it actually fails, it lands you into a debug session debugging that recording and kind of replaying that. So as I mentioned, I can show some of the documentation on that. I can show it now, actually. But I'd like to kind of switch over to a little bit of the demo portion before questions just to kind of show some of this stuff in practice. Is that big enough for everybody? Okay, cool.  It just says, it just prints some stuff, gets the date, and prints that out. So if we run that, you can see every single time it's printing out slightly different information.  Every time it changes. So the first method that I showed you, we can do just by running go build. We have the resulting binary there. So we can do rr on that. So it prints here where it saved the trace directory to. So then we can just do replay with that trace directory. And then we can start issuing commands. So let's set a breakpoint at main. Continue. And you can see the address that it was set at. And we can say go routines, see where they're at, the addresses, the threads. So we have thread ending in 948. And if we restart and continue back to that breakpoint, hit threads, we have the same exact threads, thread IDs and addresses and everything like that that we had last time. So everything is just kind of being replayed exactly as it was. So if we start stepping through the program a little bit with next, we can print like the result of date, which it kind of prints it like expanded out. But you can see the wall time there. We can print the address of it. And then if we restart, maybe I can make this a little bit smaller so that you can see. If we restart and then continue next, next, next a little bit, prints the address of it, you're going to see the address is exactly the same. So EF0, yeah, EF0. So the address is the same. The memory layout, everything is exactly the same every time we run this. So that's kind of the first way to run it. So if we just remove that, I also mentioned the back end flag. So instead of doing all of that, you can just issue one command. It basically did everything that I just did but in one single step. So now we're debugging a recording. And we can essentially go through the whole process and do the whole thing all over again. But we have to issue less commands. And I won't actually run through the script but I don't know if I actually have Wi-Fi. But the script, if you go on our GitHub and go to documentation, CLI, Starlark at the very end, so it gives you a breakdown of the whole scripting API but it also gives some example scripts to kind of get you started. And at the very end is the script that I showed of rerunning a flaky program over and over and over and over again until it hits the bug and then being able to debug that recording. So, yeah. Anyways, that's pretty much it. I was trying to get my thank you slide up big but it doesn't really matter. That's it. And I think we have a little bit more time for questions and stuff. So, yeah, if anybody has any questions or anything like that. So the question was if there's a syscall to like read from a file or read from like a network socket or something like that, is that also recorded as well? Yeah. Yes, all of that is recorded. So file reads, anything like that. I probably should have maybe put some more interesting stuff like that into the demo. But yeah, file reads, anything like that is going to be, is going to be recorded and replayed back exactly the same. So you're saying, so the question was if the program calls like date twice, right, and then you try to call it manually? Yeah. Oh, so like, like doing a function call through the debugger, right? Yeah. So doing a function call through the debugger, you won't get a recorded output. There's no pre-recorded output for that. You'll get the actual time of day. But you know, the two date calls will be different. But they'll be different correctly if that makes any sense. So the earlier date call will be the earlier date call, the later date call will be the later date call. But they'll be the same as they were when the original process ran, if that makes sense. So you like, you change the value of a variable or something like that. Yeah. So it's up to the debugger up until that point to faithfully replay what happened, right? But you can still modify the contents of memory, right, if you want to. You can make it nondeterministic if you want to or change things if you want to. In the end, the address of that variable is still going to be the same. Just the contents of it are going to be different. So that's still possible. Yeah. I guess it's sort of two questions. I get it for command lines, but I think OpenShift will deny ptrace and all that stuff quite tall. So I know there's debug containers on Kubernetes, so there's the general debugger use in Kubernetes. The question would also be, I guess my question is, do you have specific war stories of like above that you found this really useful for as opposed to print up debugging or like none recorded debugging? Sure. So I, so as far as OpenShift and things like that, I believe there's a way to like inject certain debuggers and stuff like that into the containers and maybe remove some of the second stuff that's blocking ptrace or whatever. Not 100% sure on that, but there might be a way. As far as my own personal war stories, I mostly spend a lot of my time working on a debugger now, so I don't tend to debug the debugger too much, but I mean off the top of my head, I can see, you know, I know that there's things that earlier in my professional career that if I would have known about this, this could have been valuable, especially for different processes running on servers and things like that, crashes happening intermittently, being able to record that and kind of take it back to my laptop and figure out what was going on. I know that there's been several times in my career where I would have found that to be beneficial, but I think the problem is kind of raising awareness that something like this is even possible. If nobody knows that it's even possible to record a process and replay it, you'd be more hard pressed to find a use case for it if you don't even know that that's a tool that you can reach for. And if you run into a situation where this might be useful, at least now you have it in your back pocket. So if you're running multi- process, you actually have distinct process trees running multiple threads, so each process is going to run single-threaded but Mozilla can follow several different processes and trace them. So if they're running, I believe what will happen is every process will be running single-threaded, but there will still be multiplexing or parallelism on actual the processes that it's running, if that makes sense. So if you're trying to debug interactions between distinct processes and some of that results on maybe like a race condition or something like that, that involves some sort of parallelism, this could still be useful for you. And sorry, just to repeat the question, he's asking, is it useful for recording and replaying multiple processes if the bug happens to be kind of like a parallelism sinking issue maybe with locks or something like that. Anybody else? All right, thank you. So your question was how can you move in reverse during the debug session? So there's a few different ways. So with Delve specifically right now, we have a rewind command which will, if you're at a breakpoint, it'll rewind you into like the last, whatever breakpoint happens in reverse execution, right? So if you have a breakpoint on line 10 and you have another breakpoint on line 5 or whatever in the same function and you reverse, it'll go back to line 5 when it hits that breakpoint. Another feature that we have currently that I'm currently reviewing right now and we have a pull request up for right now is doing even finer grain stuff like reverse next or reverse step where you can reverse step in and out of functions and things like that or like reverse step instruction, right? So a little bit more fine-grained execution in reverse but it's all definitely possible and yeah, that finer grain stuff will be available in Delve pretty soon but right now you can basically rewind to the last breakpoint so yeah. Cool and if anybody has any more questions I'll be around for a little while so we can take it offline or whatever and thank you.