Check one two three check check one two hello Check one two ABCD EFG H I J K L M N O P Q R S T U V W X Y Z Hello check one two three four five six seven eight 910 1112 What you know, I can't drink like caffeinated beverages after what happened yesterday Yeah, yes, hello Okay I feel like I'm hitting it. I feel like I move and it it bumps into my face. That makes a lot of noise I mean All right, I'll try not to mess up this arrangement got to be the hair You know, I'm okay using a handheld as well, I think this will work What no, I just don't want the the mic to make noise when I like move around and it bumps into my face I'll just not not move around too much No Me on the podium it says not just to move around too much anyways, so okay Are you gonna astroturf this presentation with fake questions I I don't know how to juggle these people have nothing to worry about Oh She asked that at your own talk out row I Okay, this is about the time so we should get started Welcome to another great day at scale Hope you guys are enjoying yourselves so far. Well, this is the first first talk of the first Saturday, but before we get started got to say a little word from our sponsor who's sponsoring this room We want to say thank you to our sponsor build kite for supporting the systems and infrastructure track Build kite is a CI CD platform that enables teams to build and deliver software at scale Not this scale, but And so I like to introduce our speaker Anita Jean and she's about to kill it We are happy for it give her a big hand Baker feel welcome Hello, everyone. My name is Anita John. I'm a software engineering manager at Metta I support the Linux user space team which covers a system D and basically anything in the Linux ecosystem at this point Yeah, I know dangerous And today I'm going to be talking about how I debugged a memory regression in system D. Journal D This is something we found when upgrading our fleet from 247 to 248 I am also an active committed to system D. So if you'd like to ask questions about system D after This talk Because this talk is not about system D. That's the one at six o'clock. Yeah, I'm feel free to do so Okay, here's a rough agenda of what I'm going to be talking about I'm going to set the stage talk about how we discovered the issue Then I'm going to go into how I investigated and tried to track down the root cause Because the root cause was pretty unintuitive. I'm going to spend some time explaining in detail What happened and then close it off with some hindsight thoughts and takeaways So setting the stage This is what our production fleet looked like at the time when we ran into the issue So our entire production fleet runs sent to us stream 8 I was also able to reproduce this issue on Fedora 34 So this occurred while upgrading system D from 247 to 248 So this was quite a few versions ago because the latest version is now 251 upstream our System D build comes from the CentOS hyper scale SIG my teammate David will be talking about hyper scale SIG tomorrow So you can go Listen to that if you want to hear more about how we build packages for hyperscalers But the idea behind this build is that we base it off the Fedora raw hide spec in this case those forces in the 248.2 I also mentioned here that we use kernel 5.6 at the time This is only relevant if you want to run some of the BPF tools that I'm going to talk about in this talk So here's what we found Um, I got a notice one day from the fleet efficiency team that system D journal D's average memory usage Had grown since May 24th from 17 megabytes to over 50 megabytes on average This is pretty non-trivial for Our production fleet because we're running millions of servers and we want to kind of eat out all of the memory utilization that we can And so they wanted to figure out what was going on At the time I was rolling out system D 248 I was pretty close to the end of the rollout at this point So about 90% of the fleet and I thought it was pretty strange that I didn't see this issue earlier And that this didn't actually line up with the rollout that I was doing The way this rollout worked is that I actually had to roll back a few times So I would have expected to see like up and down spikes in the memory usage if this was correlated with the rollout But in this case, I didn't actually see that So what was going on? Trust me verify the graphs don't lie probably so I hopped on one of the host I Hopped on one of the hosts that was exhibiting the issue and I ran p-map to Look at the memory mapping for system D journal D And what I noticed that is the anonymous memory was over 300 megabytes for system D journal D Which is pretty crazy because normally it's like 20 megabytes Anonymous memory is important here because this is memory. That's not backed by a file system This is memory. That's actually being used in the stack and heaps. So it's actually in the RAM So I saw so I was like, well, was it actually the rollout in this graph here? It shows some of the Anonymous memory growth in system D journal D The axes are pretty small, but it was a very slow growth about a hundred kilobytes per hour So we were like Well, if this memory growth is not showing up in 247 then it must be something in 248 But why didn't we see this as part of the rollout because normally when we upgrade system D It's a live upgrade and the daemons will restart into the new daemons as part of the upgrade But we noticed that for system D journal D This wasn't actually happening and so you know since the upgrade from 248 and up This is actually fixed and you can actually reboot into system D journal D Which should be running the correct daemon. So for 247 we just kind of manually Upgraded the system D journal D daemons Restarted them into the correct version and that's when we were able to confirm that indeed It was the rollout and there was something in 248 that was happening Okay, now that we know that the issue was the rollout we have to figure out what changed The first thing I do because I'm actually very in tune with what's going on in system D So I just go straight to the git logs and look at what's changed in the journal So there weren't actually that many changes in the journal code itself between 247 and 248 So I had about five commits to look through Looking at the code for each one. I didn't actually see anything in particular that struck out Maybe some of the a map cache stuff Was using more memory, but nothing like obvious at this point And so I was thinking, you know, maybe it's a memory link If you have like all this growth in your application, it's usually some kind of memory leak, right? But so I ran VAL grant as people do against journal D and you notice that pretty much everything's freed There's like these 8,000 bytes that are still reachable but that's actually unrelated and is kind of Used in every version for the mem pool cache, but for the most part nothing, you know Everything's freed everything's the allocated as expected. So what's going on? I'm pretty much a generalist in this space So I had no idea what to do when we're Debugging a memory regression in something like system D journal D I work in other parts of system D, but journal was kind of a mystery At this point. So I decided to just take those few Journal commits that we had from the previous two slides and do a new system D build Based on each of those commits and deploy them on hosts and monitor for the memory regression So theoretically if we wait a few days Eventually it will see on our charts that the memory growth should be growing on You know the bad commit, right? While that was going on I decided to try and figure out You know if can I figure this out faster? Can I use some tools to debug this issue? My favorite tool of choice is usually just to jump straight to S trace Unfortunately S trace is not very good for debugging a memory regression because a lot of the memory is allocated early on by lip see when the application starts and so the kernel you will see like S break pretty early on in the application But the memory isn't actually Allocated and mapped to physical memory until it's faulted. So that would not show up in S trace You know memory allocations in the kernel if you were trying to use that What I decided to use instead was ebp f ebp f allows you to extend the kernel capabilities without changing kernel source or loading modules and it's become a great feature for tracing profile and monitoring and so in this case I would be able to kind of live Trace what's going on in journal hopefully get some stack traces and figure out what's going on there And of course meta is a founding member of the ebp f foundation So we have a lot of maintainers and others who could help if I had issues Like I mentioned before I'm a generalist So I didn't really know how to debug the where I should actually go and like what bpf features I should be using So I you know I did some googling has people do and I found Brendan Greg's website for Debugging memory leaks and memory growth And that's what led me to try out the bcc tool suite which has a lot of pre-built Applications for tracing and debugging memory leaks and stuff So the first tool I tried was step count The idea here is that if I could count the calls to Malik Maybe I could see that journal D was doing more calls to Malik than it was before and that could explain Why we had this memory growth and so on the right here. You can see I call stack count with dash U for user stack and Malik to count the calls to Malik I give it the PID to system D journal D And what I got here was one of the counts of the stack traces which lead to Malik calls Unfortunately when I did this between 247 and 248 the number of allocations are pretty much the same. So That was a theory that didn't work out. I Was still pretty hung up about the memory leak thing even though Val Grand was like, you know, there's no memory leaks So bcc has a tool called Memleak that you can run and what it does is it tracks The allocations and D allocations that are happening in your program So for example, if you have something that's been Malik for a really long time and it's not been freed during the lifetime of the application That kind of thing would show up in the stack traces that Memleak returns and So on the right here, I have the top 10 allocations happening in system D journal D I look for things that were not being freed within 10 minutes because the way Our configuration was set up. We should be rotating out the journal files quite frequently Unfortunately, this was also a dead end because I noticed that the allocations were pretty much similar between 247 and 248 All right, so let's take a step back and look at what we know so far You know a lot of stuff happens in debugging you have a lot of theories some of it doesn't work out But what we know so far is that There were pretty much no leaks. I've used like multiple tools at this point. There are no extra leaks no extra allocations And so You know, where should I go from here, right one thing I did learn from the Using those BCC tools is that the Allocation path so like from getting to the journal stack where we enter the code It starts from a client context function and then it ends up in an allocation function That's actually in system D's core libraries. So for example, I listed Readful virtual file, which is one of the key like allocation functions that journal D is Ends up in when it actually calls Malik and realloc and so using this information I decided to change my bisect strategy and start looking instead at the allocation function So we go to git log once again. I look this time in the file IO functions Because this is where the read full virtual file function lives And so there are actually quite a few more I omitted a lot of them But this is just an example. I did the same thing. I did system D builds on each of those commits deploy them to machines In our fleet and then just watched for the memory regression You know and at this point bisect prevails and it takes us to the root cause commit The funny or like painful part of this commit The funnier like painful part of this commit is that it was actually the last commit right before 248 was tagged. So If I had done like a whole bisect across all the commits and start from the center it would have taken quite a while But I'll explain this more in detail about what the issue was but the summary of what happened was Normally if we were reading something from proc we'd start with 4,000 bytes and then increase the buffer if we need more of it, but the the function at change to start from a four megabyte buffer instead Reallocate down to a smaller buffer as we read from proc and we didn't need like the larger buffer anymore So, you know, if you just think about intuitively, it's like, I mean that that's fine Right, like you take a larger buffer you bring it down You return that memory back to libc. So what's the issue here, right?  So things like the length of the command line and stuff Those get allocated in like, you know the heap just to store the command line and stuff If for some reason it's larger than the initial size of the buffer we set up we'll call realloc and Double the size of the buffer if the buffer doesn't fit then libc will just find a new space in the heap and we You know, alec new space the old one gets free If for some reason that We need to allocate even more space because the data we're reading exceeded that buffer. We double it once again You know, same thing keep calling realloc and we actually double this up to three times most Files will not most reads will not take up like all 32,000 bytes but This is something that could happen in the code and so the initial commit was trying to fix an issue where The maximum we actually want to allocate was not 32,000 We want to actually cap out at four megabytes and so the commit was trying to fix this issue and also Optimize it so that we wouldn't have to reallocate up three times. So we wouldn't have to retry three times They thought we could just Allocate everything at once and then just deallocate it down So logically, it's like, you know, it makes sense. We could save a few calls to realloc We get the correct maximum that we're aiming for and Everything's good, right? So here's what happens in system knee journal d248 This is with the blimp commit. We have the large four megabyte buffer Most of the time we're reading stuff from pocket fest. It's actually quite small. So a for 4,000 page aligned realloc now and We have a small buffer again, so that looks good, right? Let's pretend we have something like this on the heap, right? We have lots of little buffers scattered out throughout the heap because as the application is running We allocate and we free memory and we create all these free memory chunks in the heap unfortunately when you're looking for a You know an area in the heap to store this huge buffer You know it couldn't fit between these gaps anymore. So instead we have to go towards the end of the heap and Alec Malik this like big buffer there And so now we have this large space being taken up at the end of the heap and Then you know normally like I said pocket vest we read data. We don't need the buffer that big so it gets realloc down The problem here is that if we keep doing this multiple times The way realloc works is that it tries not to copy and move the buffer to a more optimal location in the heap Instead it just keeps bringing it down from where it's already allocated So if we do this then it starts to create a lot of Fragmentation in the heap. So if we were to free Like one of the chunks in front of it. We start to create even more gaps in the heap and they won't be able to store like the large buffer anymore and we won't be able to optimally use the memory I think another point I should bring up here is that You know, even though we have all these free areas in the heap To the to the kernel like in the application that memory is still mapped to a physical mapping And that's why it says that the applications using all this memory that memory That's like in between those allocated chunks is still being taken up by libc in your application They're saving it for later uses. They're not they're actually mapped And so the kernels like your applications using 300 megabytes. You could realistically just be using 20 and so You know the fix I ended up merging was to partially revert Some of the initial blame commit We go back to retrying three times to increase the size of the buffer But I fixed the but I do also address the initial problem Which was they wanted to cap out at four megabytes instead of thirty two thousand. So that's also been integrated in the fix And as far as I know meadow was kind of the first to detect and fix this issue Kind of a benefit to the community that we run system D upstream or like system D latest tag across all of our machines, so Things that only show up at scale. We're able to Fix pretty quickly Okay, closing it off with things that I learned in hindsight So I want to know how I could have detected this issue faster, right? If I see this issue again, it should not take like Weeks or like days to buy a second issue and track it down Things that I should have noticed is that if I instead looked at the rate of change of an honest memory between System D journal D 247 and 248. Sorry. I mislabeled that it should say journal D I would see that in 247. There was basically No rate of change in the anonymous memory, but in 248 I'd be able to see these spikes of like four megabyte allocations So if I instead looked at this graph initially and like thought about it more clearly I probably would have gone in the right direction sooner Another thing I want to bring up because it's kind of cool is mtrace So this is part of glipsy. It's a function and a command line tool You can insert it in the in like the start of your program and it will track like allocations and deallocations In my case, I think this was pretty similar to using BCC's memleak. So You know if you have the choice, you don't want to modify code I would probably use memleak, but I thought it was just cool to call out that there's also a tool in Lipsy that you can Do this programmatically in your code Okay, another one that I thought was really cool that would have been super helpful for this investigation is massif It comes as part of valgrant and tracks kind of like the peak allocations happening in the heap for your application so On the left you see that the peak memory allocation for systemd journal d 247 was around 100 kilobytes But on the right for systemd journal d 248 you see that the peak is four megabytes I've omitted here stack traces So massif also shows you stack traces and that would have been super helpful for tracking down Where these allocations were happening and why this peak was happening And I think if I knew that I could have nailed down the the blame commit a lot sooner Okay You know because I am a software engineer by trade I wanted to know was there like another way I could have done this was there another fix that would have been equally You know as optimal for the heap So one idea was to instead of using realloc which tries to keep the start of the buffer where it is What if we just free and call malloc again just copy the data to another part of the heap And so if you look at the graph on the right I actually show three different builds of systemd journal d So the first one that's like going all the way up. That's the systemd journal d 248 with the memory regression And then in the middle closer to the bottom is the fix that I actually upstreamed This brings us back to baseline systemd journal d 247 levels and then at the bottom is the Hypothetical fix that I'm listening here on the slide. Like what if we instead just malloc mem copy free the buffer It is slightly more optimal I Didn't end up going with this because it would have been pretty inefficient if we have to keep calling mem copy to copy data across the heap You know, you have to balance like memory optimization also performance And so I think it was just fine the the version that we ended up upstreaming Okay, takeaways Definitely invest in observability monitoring tracing all that good stuff. I'm really fortunate that meta has a lot of teams That makes this super easy. I could just throw these builds on the host and I would automatically get these graphs I didn't have to set up anything. Oh Useful stack traces. Oh, yeah, one thing I didn't mention is that a lot of these BPF tools they require Frame pointers so most builds that come straight from your distribution don't Include packages with frame pointers So if you want to get like the really nice stack traces that I had in the BCC tools I run You should make sure your application has frame pointers but otherwise It's really hard to debug anything without stack traces Okay, um, and yeah Always be learning. I love debugging stuff because it allows me to pick up new skills learn about new tools There are all these things in Valgrind beyond like the memory leak checkers that I didn't know about That I now do and I can keep that in mind for if I have another issue like this in the future And of course, I want to keep getting better using EPPF. There's lots of workshops here Tracing is just kind of super valuable nowadays for figuring out how to debug and trace down issues All right, that's all for my talk. Thank you everyone for coming and now take questions Yes, sir So this gentleman mentioned that for embedded systems, we really want to decrease kind of the memory fragmentation and gaps And is there anything in Linux that does this honestly? I'm not quite sure when I was doing this investigation I learned that there were actually lots of knobs that you can toggle in like g-lip see and Malik to change How the allocations work like whether you want to Cash more free buffers and things like that. I'm not sure if there's like a tool Specifically that could analyze gaps for you Anywhere there's be something like that Yeah, Neil's right You mentioned that there's probably something in Valgrind that would do this and especially the KDE folks have been looking into like memory efficiency and things Like that. So Trying and see if I ground have something Yes over there Yeah, this gentleman brings up a good point We could also use in the future amendment and advice calls to see what memory allocations are happening Any other questions or comments? Oh for sure So we so I didn't go into like the I'm sorry the this gentleman mentioned like We rolled out 248 to the in fleet entire fleet. Maybe we should have started an RC tier I didn't go into the details of how the rollout phases work, but we always start in the RC tier But kind of the one of the downsides of the RC tier is that we don't run like all of our production workloads in it And so we don't get all of the signal we need Sometimes it could be rolling out to 50% and that's when we start to see a lot of the signal in these things Yes, oh RC stands for release candidate in our case. We have Just a bunch of machines that we run our initial binaries such as the kernel System D a lot of the widely distributed binaries who run those first in our You can think of it kind of a test here, but some of them do run production workloads And that's where we try and see if anything blows up immediately Yes I think the cons team doesn't want me to say anything about that And also I'm pretty far moved from like the dollar aspect of the machines. I Think legally we're allowed to say millions of machines But yeah, we were it's just this is like the entire fleet like we roll out the latest system D to like all millions of them And it runs our production bare metal fleet Another audience member here just mentioned that we can use alternative memory allocators like Jay Malik Especially an embedded system that helps with memory fragmentation and stuff Yeah, actually when I started this investigation, I was like No way this can't be system D Because upstream we run multiple CIs for like a sand and just if we saw a memory leak, we should be able to detect it immediately I think for us the Linux user space the kernel teams the container teams We're pretty well in tune. We communicate pretty frequently amongst ourselves And so if we notice an issue in one system where we kind of work together Communicate and figure out is it this rollout? Is this this other rollout? And so that kind of helps narrow down the scope a lot being able to communicate effectively between teams and I think The being able to have a very like Generalist skill set like the one I did in this case. I was able to pinpoint Which system had the issue? I think being able to work across kind of a breath of system sometimes just to narrow down like It's not my applications your application and having some of the data to back that up that really helps That's a good point. I didn't address the issue where you mentioned We could be upgrading like tons of packages at once like when we jump between Distribution versions and we have a lot of package upgrades happening Especially you know for system D was a lot easier because we do try to stick very closely with upstream and the fewer Diversions you have between versions You know the quicker you're able to detect the issue which is why I like running the latest system D. It's such a big Space that it covers. I want to be able to detect issues faster Like I wouldn't want to be running 239 and then trying to debug an issue that we now see when upgrading to 251 But for like the big distribution upgrades for sure we do those a lot slower We kind of detect issues as we roll out a lot more slowly across the fleet But it's kind of the same thing. We just keep communicating between teams We do a lot of communications with upstream to resolve issues as well and things like that I mean, I'm happy to do it as long as the Linux user space team is still here You generate a video and if you're running and hold their stable stable version It might be more stable in some end but you do some issues You can't really get those upstream easily. So you have to end up getting a lot more At my workplace, we have personal experience dealing with those where we run very popular distribution everywhere and We've gotten to a point where we maintain somewhere between 8 to 12 percent back forth overrides because of the main package set because The vendor itself can't do anything to fix it. There's not a whole lot that can that they're willing to commit to and Nobody else wants to really work on it because it's so far you know it's Long-term Anyone else have questions about This talk or system D in general. Okay. Well, thank you everyone for coming Hello, that's a very loud for a very small room so Try not to be too loud. It's very hard for me. I mostly want to be very loud So hi, I'm Michelle my pronouns are she her and I work in Netflix. I'm a senior software engineer See this excellent So today, I'm really excited to talk to you about Netflix workstations Throughout this presentation. You'll see some before and after shots of work That was actually done on the workstations. You'll see some close-up beauty shots some De-aging for some of our older celebrities and some large environmental destruction So things like explosions and dust some really cool vfx work. They got them I'll be happy to answer any questions at the end as time permits Or if you find me, you know throughout the day in the hallway, feel free to stop me and if nothing else you can find me on LinkedIn, but Definitely put something in the message.  I'm like, yeah, you're tricking me So one question I will answer right now is that I have no power to renew or green light your favorite show if I did There would be a show called cake with Michelle my number one favorite show where I just eat cake and tell you How great it is be such a good show. It would be really boring for everyone else, but it'd be really great for me So but what I can Do is help artists and creators make the Netflix originals. We all know and love my team production infrastructure engineering or Pi is responsible for helping to make sure they can work every day all day and not worry about the infrastructure powering their workstations So here's our agenda for today. We're gonna learn about what technology we used What was the problem that we solved? Who are the users? How did we actually make Netflix workstations? What did we learn and what comes next in the Netflix workstations 2.0? So before I talk about the problem and why we did this I wanted to give a quick teaser trailer of the technology that we used if you are unfamiliar with any of them Don't worry I'll be kind of defining all these services as I go and if you're like a super expert on any of these Please stick around and answer my questions. It'd be really helpful for my next sprint So at the top is go having a cup of Java salt for configuration and spinnaker for our fleet management Images are made with packer and we use a lot of AWS services But I wanted to highlight EC2 as the actual instances DynamoDB for the data and SQS to support our event driven microservices architecture Those EC2 instances come with nice DCV Which is one of the ways we actually remote in and display it to the user and the other one being Teradigi and Since we're all about open source here I wanted to point out some of the open source software here, which is packer and Salt and spinnaker which are all really great And if you have any questions about them specifically that you don't learn today Happy to answer how to get started with them So what was the problem? Historically artists had machines built for them at their desk and only had access to the data and applications when they're in the office This system allowed for fully on-prem solutions that stopped at the door There are a few reasons why that's not feasible. Some of them you may be well aware of Is the vast shift to work from home in 2020? We actually already started the project in late 2019 as if we were psychics But it really accelerated our timeline to get people working as fast as possible While people are beginning to come back to the office in some regions the work landscape has irrevocably changed obviously So the second reason and our initial project mandate Why we started in 2019 is that Netflix wants to make content at an unprecedented rate I've worked in entertainment almost my entire career, which is about 12 years now And I've never worked on this many Entertainment projects at the same time. So we need that technology to work at this scale and be able to do it more efficiently There are simply not enough artists in a single geographic location to support the Netflix scale of content So productions need to meet artists where they are instead of trying to bring everyone to one giant imaginary office and Finally by taking down traditional geographic barriers to entry instead of making everyone move to LA as wonderful And much as we love it here the industry and Netflix becomes more accessible to all types of people Losing access to great talent because I live a hundred miles away from the nearest office or might have mobility Restrictions or other things that they want to work from home. It's just not something we want to do So why not ship a computer is the question. I'm always asked the first question The North Star for Netflix workstations was to provide the infrastructure for artists to get a one-click experience You go from sitting down to working on a shot But that can also be accomplished again by building a computer and literally just sending it to them So when I do that instead There are two answers their flexibility and security among many others so Production needs change fast and furious new cameras and new software make it an ever-evolving space By creating cloud-based workstations adding anything from new features to security patches can be done quickly a User can switch from one operating system to another by opening a new window instead of a new computer Instead of waiting for a package in the mail. They can get a more powerful GPU or larger storage with configuration change We've had it across like this all the time like oh, I've run out of storage. Give me some more Like okay, I've attached some more to your new workstation go to town instead of waiting for something to come in the mail Workstations like many cloud-based applications are not designed to be permanent Instead artists are renting a workstation instead of buying it The downside to flexibility of course is accounting for other features that come with a single desktop Things like persistent user settings when working on a desktop User and software settings are just sitting on the hard drive Whatever the software puts them with cloud-based solutions We had to be more methodical and save crucial settings to persistent storage associate that with a user and Attach that storage to all future workstations so you have to be a little more think through what are the downsides when you do something That's more flexible For security asset security is critical inside studios Files have traditionally been passed around many different ways from FTP to email to literally someone getting on a plane with a hard drive and flying across the ocean So every time a file gets moved it creates an access point for interception By having all the assets on cloud storage along with a remote workstation Files do not have to pass through any intermediate service. No one is downloading those files to their home computer It's all in the cloud It also allows for this granular control of a files lifecycle and access so you can grant it for the short time They're working on it and then take it back instead of being like please delete this from your computer pretty pretty pleased and They never have to leave the original storage So when designing Netflix workstations we did balance the needs of three significant personas artists technical directors and pipeline engineers and you might not have heard those titles before So I'm gonna kind of go into detail. They are the very specific animation VFX job titles So and then there's also the many supporting teams such as storage and networking and security all the people that make these go So let's look at dig a little deeper into these personas Artists want to focus on their art and not on technology They are the main end users of a workstation They want to get to work immediately and not for some wait for something to boot up or to figure out what files They need to download or anything. They just want to do their job The artist workflows can vary wildly depending on the step of the production process They're on but their basic needs are the same you want to think of you know building a movie is like moving a Factory one item from one team to another So it's moving the shots of the film from one team to another and they're all Adding things to it to get to that final image that we all enjoy So they have source files that include anything from say like original film plates that they just recorded You know like a celebrity jumping away from an imaginary explosion To reference images to something much more tactical like a lidar scan to get a good idea of the 3d environment that was filmed in the artists then use digital content creation or DCC apps to create new images that could be final drawings or the blueprint for a render These new files then have to be shared for review So all these steps would ideally be completed Directly on the Netflix workstation which includes other tools and plugins to manage that whole pipeline from every team to the next team such as task management and getting reviews and things getting approved and final delivery and all the many many steps in the Movie making factory So technical directors are both Artistic and technical expertise Instead of working directly on shots they work daily to improve the artist experience They want to get the artists working faster and easier and not worrying about these things So they need to be able to create scripts and plugins for the artists and also debug their problems So they need to do Half an artist job and half a developer's job and troubleshoot any issues So they need a lot of observability tools a lot of access things like that They decide on the software and ensure the artist on that correct configuration for their task Things like oh, this is the version of this software We're using for this project because this plug-in works here and this will be really great for this You know that sort of thing really in the details for everything So pipeline engineers focus exclusively on creating pipelines or workflows so they handle that whole flow Their portfolio can include things like file orchestration or license management or anything else needed for the artist to complete their work They glue the pieces of the pipeline together with Netflix workstations kind of in the middle They make sure the artist can work as efficiently and smoothly as possible and also the technical directors have everything They need to actually develop on workstations as well not just Make the cool images So my team started with a strong emphasis on building a complete White-Gov product for the artist to use we thought we'd build one thing and everyone would just use it and it'd be great That's never true. I don't know why we thought that Plan was always to move towards a more self-service model Which is good because we got things in their hands We got to see how they actually worked and get that iteration going I'll talk a lot more about iteration as we go on So the first step was adding features To allow the technical directors to start to customize the experience and the next step is Moving towards a platform so the pipeline engineers control and manage their own fleets, right? They're managing everything about it and we just make the tools available to them So they can provide a very curated artist experience for all the teams and really customize what they see So how do we make them now we're gonna get start to get into technology Starting a brand-new project can involve making hundreds of decisions both big and small It can be very overwhelming wondering of a choice made now will ruin an engineer's day a year from now Well, I'm here to tell you not to worry too much about that because it definitely will like don't worry about it It's a hundred percent gonna happen So have you ever worked on a legacy project and thought wow the engineer that wrote this was a genius and Everything makes sense and it's totally perfect and there's no weird problems with this. That's like every project, right? So of course not and the best part about this project is I got to be the person you're mad at because I got to start with An MP repo and build something and now everyone gets to be mad at me two years later. It's fantastic So as the engineer that bought that built in the you know in that greenfield place I'm the worst but it's because I worked within the constraints of what I knew then Not what we know now on how people would actually use what we built so The our techniques we did use to reduce the bad choices that future engineers had to live with For all of my technical talks actually I like to include this underlying theme of do less but accomplish more that is Focus on solving the problems and only build what you absolutely have to if there's a if there's a service or an Open-source tool or something's already built to solve part of your problem Just use it and you know you out. There's obviously things anyone to do to vet it But it's important that you focus on the business problems and not reinventing the wheel, right? So you can actually get get things done. So Netflix actually values a culture freedom responsibility. You'll see that in our culture deck and And yet a commonly used design philosophy that the engineers use is that of the paved path It's tools and practices that are widely adopted in the company and supported Engineers can choose any tool or build any service as part of that freedom But then you have that responsibility to support and maintain it and I mean much easier If there's already like 200 people already using it and I've already figured out all the problems and you can just lack them It's the best That kind of matched my philosophy as well and guided most of the team's choices Of course, sometimes you can't sometimes you're making new things Sometimes you're googling something and there's literally no answers anywhere because no one has tried to do this yet And you're like, how is this possible? I have to write a new thing the worst But you know, though, that's when I put a premium on What tools can I use to build this and if there's solid community adoption as well as if I build something? Can I use it in multiple problems or am I just solving one problem? And if I'm just solving one problem, what's the smallest amount of work I can do to solve this problem? And if it's multiple problems, what do I need to do to make this flexible? It's okay We're casual here No problem Okay Tell them not to call again. You're very busy It's very interesting Unless you want to like put them in the phone so they can listen because it's so good. I Mean I would recommend it you call all your friends and be like you have to hear this talk the slides There's not much on them. You won't miss it. It's just like cool pictures that I downloaded from Netflix originals Cuz I was like these are the TV shows. I like Okay, so this one actually has a lot of text on it Which makes it even funnier that I made those jokes that I'm like, oh, there's no Texas It's fine except this one has like ten lines So in order explain the technology behind this project. I kind of wanted to break it down into main components So the first component is fleet management Which is making sure the workstations are available and tracking their life cycle has so I'm gonna acquire this workstation Is this workstation still building has this workstation is it a zombie now? Which I think is the great name for Something that we've lost contact with it's obviously gone rogue and eat all the other workstations The second component is configuration management Which makes sure the workstations can be designed for any artist needs whether it's installing an application or configuring the registry Or an environment variable you can do pretty much anything to configure a computer You know and we managed to find all the edge cases to do that I've learned a lot more about PowerShell than I ever wanted to know The you know the following two package and image management are both a subset of configuration management But they're slightly different package management handles those individual package like a single software installation While image management handles collections of packages such as specifying, okay These ten packages are required for a compositing artist You would think installing a single piece of software is not complicated Well, it's surprisingly complicated a lot of things aren't as built for this type of Cloud-based environment as you think so you have to be You know very Deliberate and how you install things So it's kind of together both the package and the image they ensure configurations are immutable And you need a clear history for stability and observability like who installed this software What was installed exactly? What was the actual lines of code they use because that's the only way to dig into what why this didn't install correctly and why it's not starting up so The fifth component is user access or role-based access control and is how the right user gets the right workstation with the right files So we're not back into that world where people have access to the wrong images and the wrong movies and Then finally wrote remote display protocols is how the user connects to and actually interacts with the workstation remotely So let's start with fleet management So Fleet management handles the lifecycle of a given workstation as well as the pools of workstations with identical configurations Pools are kind of similar to an auto-scaling group in AWS, but the control plane controls the scaling So that we're not You know auto-scaling or removing things that people are currently middle of working on we need a lot more information instead of letting it automatically scale and We do that with the control plane and that's a collection of Java spring boot services Spring boot is great allows for the easier creation of job applications by abstracting away some of the common tasks Coming from Python to Java is really nice having spring because I didn't have to learn something like I just get it to work and Has a really strong paved path in Netflix and so many services use it There's many people I can ask questions to So that control plane tracks the fleet needs and alerts spinnaker to execute them Spinnaker describes itself as an open source Multi-cloud continuous delivery platform that helps you really software changes with high velocity So it's a common tool for releasing and maintaining services in Netflix and it's on the paved path So we actually reuse this to control the workstation fleet as well So most people are using it. Oh, I'm gonna deploy my service. I'm gonna make sure you know do this red-black deploy get things working We also said hey, we can use the same tool to deploy instances It had different configurations with these different images and control what's happening there So it's great that we can reuse this this tool Spinnaker has this concept called pipelines Which you've already heard me use pipelines in another way. So this will be fun. I'll probably use it five more ways Definitely nice overloaded word, but they're in this case They're instructions for creating the pools of workstations, which it calls clusters So pipelines can be accessed through an API and allows for variables and expressions So we set up a variable a variety of variables that can mix and match for each use case For example an artist can need a GPU when doing graphic intensive work or extra large storage to handle file management Some artists need one workstation to support the compositing surf software will others require a different one to do different software it To minimize latency and get the workstations as closely as possible We also support being able to select a growing list and regions and availability zones So we try to use the same pipeline to do everything But here's all the variables launch it here launch it there launch it with this that sort of thing And that control plane direct spinnaker pipeline around whenever it needs either, you know A whole pool of them I need a hundred more with a total no configuration Or I just need like five more workstations because I've a couple more artists coming and it connects to the API to Terminate once a user is done with it or remove whole pools if they're like we're done with this configuration We're gonna do a whole new one get rid of those and Then the workstation has a goal line agent Which provides the heartbeat and other information about a workstation back to control plane The agent let's the control plane know when the workstation still up and in correct state or if it needs attention We check we chose go lying here because it was easy to build For different operating systems, which we knew was an early requirement. We're gonna go all over the place Let's make sure that we can put our agent on every single type of workstation So configuration management starts at the control plane But the details of software installation and configuration is handled by assault We needed a system that can manage hundreds of thousands of workstations while being highly flexible and easy for anyone to jump in And create a new configuration So not just us as the engineers, but the technical directors and everyone we want them to think hey I have a new software package. How do I install it? What's an easy installation language and that's where we chose so because it's all YAML and ginger based So it's a little easier to read than writing straight code. It also uses Python which is a very common language for people in VFX and animations. They're also really familiar with that There's a bunch of built-in modules from installing a package to file management copy this file over here It also allows for logic statements to handle situations as Mount this storage only in this region or only run the script if this file doesn't exist Please don't reinstall this software five times even if this accidentally runs five times And it's a it can get complex but at the base It looks like this. We're just saying hey install this package and This is the equivalent Here of just saying like a YAM install Lighting 10.1 on a CentOS instance or app get install lighting 10.1 in Ubuntu The module is OS agnostic and should find the right installer per OS Today, there are hundreds of different packages written for the workstation from Installing the software editing the registry and any environment variables lots of different things It was selected over other configuration management tools such as Ansible or Terraform question I get often because it was used recently in our Netflix projects and got high marks as well as being Python based So this is an opportunity create bridges to the other engineers. We knew we'd be working on this because we knew they'd be familiar with by that So initially we sort formulas directly Salt formulas allowing for fast changes and quick deployment Everything was always the latest and no ability to lock to a specific version Which sounds really great, right? That seems like something that would not cause any problems that anyone could just push them code and Everyone get a different version depending on when they loaded the code. That seems right It was a Wild West, but it was part of being able to move quickly So to help manage that we created a package management system that kind of wraps around those all formulas to enforce some rules The first is a template for all the formulas to make it easier to avoid naming conflicts and Enforcing rules such as like making sure that user information is available before this runs or this formula is a PowerShell script So don't run it on Linux. I won't do anything And the second is that all the packages are immutable to use a package on a workstation It needs to be published to a version if any changes are made it needs a new version This makes it very clear what was run. It also allows for checks at the publishing stage Syntax and testing this just sound very familiar. We just basically just did get you know Like we wanted to make sure there was some version control in our on our packaging The structure made it easier for more of the technical directors and other engineers to contribute to library of formulas I know exactly what code was run on the artist workstation while not impeding their progress They don't have to come talk to me every time they want to publish something They just have to publish it. So there's a version number. We can track down what they did So let's talk more about image management The image management process has gone through multiple iterations as we learn more about the users and observed how they actually use the project and This section that kind of uses the case study of like how you iterate through a project and make it better and better as You learn more about how people use things and how when the project is better So a workstation goes through three Configuration stages. We have the bake the run and the acquire The different iterations will show how we move the logic around to better serve the user needs once we knew how they wanted to use it So I added the actual logic of each phase to the chart and we'll be moving through the iterations all of the deployed workstation start within a MI and AMI is an image used to launch a workstation AWS The bake stage is the process of turning a series of instructions to an immutable image if you're familiar with Docker It's just like that. You just here's the script turn it into an image. So every time I launch this instance. It's the exact same instance So then can be used to come up very quickly when it takes a minute or two For for an EC2 instance to come up once you have that AMI However, we found the process of baking can take a long time We wanted to ensure that when we're early stages, we could quickly release new features We have to wait 90 minutes every time we try to iterate through a bug We don't want to be held up by those long waits while it rebate So we only included minimal logic in the base stage that includes just the operating system as well as some You know critical tools such as that go agent like hey, this is all we need to get this workstation up Spinnaker then creates the workstation during the run stage Once the workstation up that agent starts running as well as the control plan. No, it's available. There's a new workstation During this first generation. There was no other configuration happening at this stage The acquire stage is when a user asks for a workstation like I want to get to work now most of the software installation including those DCC apps and all the user configuration happened during the acquire stage it was kind of a just-in-time approach like It's funny. I just went and got a sandwich of jersey mikes and now I get to use a sandwich metaphor Which is I refuse to go if we weren't going to order in advance Which is crazy because I'd let these people have their workstations and they had to wait 30 minutes There was no ordering in advance for them. So I wouldn't accept it for sandwiches They shouldn't accept it for the workstations. So we iterate in The long wait times led to us moving the software configuration in the installation to the run stage So this created a pool of workstations that had specific configurations and just the basic OS However, there were two risks to this method But one great thing which is that the workstation wasn't available to acquire until everything was already installed So when they went to acquire it be fast Because they didn't know it was even being made The first risk was making up many many pools with slight variations That turned out not to be an issue because in practice the usage patterns of the artists were not as diverse as initially thought The same team tended to use the same configuration for extended periods That meant then there were many pools the number did not grow to be unmanageable and The teams tended to actually just switch instead of using this a month multiple configurations So they'd be like oh, we're all using this configuration next week. We're using this configuration. Let's get rid of the original one the Second risk was inefficiency there were always idle workstations ready to go for artists which meant they were sitting around just like lonely and unused and wasting compute and Racking up that AWS build which we all love so much That was an intentional design design decision in favor of usability over efficiency Of course, that means it's time for a third iteration The third and current but probably not the last iteration is pushing the software installation to the bake stage This is like having a library of blueprints instead of a bunch of empty houses Real estate is not wasted because unlike houses where stations can be built from baked images very quickly as they are needed But don't forget. This is what we tried initially and because of long bake times. We didn't like it But the process has now come full circle instead of abandoning baking We improve the process develop better tooling that allows for concurrent and easier bakes and a layering system So you're not always Doing a 90-minute bake. You're just adding on one piece of new software, right? And you only do a full rebake when there's something that you really need to clean up or you need a new operating system Or something that's critical So we also kept the other option as a backup because one of the many Lessons from bringing up a new project is kind of hedge your bets You know, let the old one slowly go away while you choose the new features and see how it goes And maybe you need to go back the old features if you've haven't done the new features correctly real slow roll the technical directors who handle the configuration choices are encouraged to give some time between creating a new one and Letting an artist get a workstation So that's also an improvement in like the education process like you shouldn't expect things right away Like it's always gonna take some time to install so once you make your choices let things happen and and Then you'll get your work stations as Kind of with all projects. There are no perfect choices just a bunch of trade-offs to maximize the user experience Cost and technical efficiency So we try to do here and we'll see if maybe the next time I give this talk the new version there'll be iteration four five and six So a crucial part of any remote workstation experience is connecting to it We currently use two different remote display protocols nice DCV and terror DG When deciding on which remote display protocol to use User must know how they want to connect This is not decided by the artist, but rather by which workflow they're using So generally the pipeline engineers who get the artist team up and running kind of decide on the workflow based on what they need Terror DG is a commercial solution consisting of an agent software hardware client and gateway components Nice DCV is an AWS service provided by any EC2 instance and allows for a browser-based connection and application streaming Application streaming allows for a very streamlined experience. How many times can I say streaming and but also work in Netflix? Explained so artists can jump right into their tasks without Launching an entire desktop. There's like it just be one piece of software So imagine you just just wanted to go into Slack that browser were just up slack in it You'd get another workstation for another application and just be that while with terror DG It's like here's a window. It has an entire desktop on it and your computer doesn't really have to do anything So for both of these the latency and peripheral connectivity were vital features For example of an artist can't use their wake-up tablet or their two giant monitors They won't be able to do their work as well in a remote workstation VFX and animation is very Graphic and detail oriented work. You can't have any latency. You can't have color disparities. You need to be able to see things accurately Or otherwise we'll get like a blue hole or something, but like only in one shot So initially all the users acquired an access network workstations through a UI Key components needed to allow for improved self-service are in the UI This includes giving the TDs the ability to create configuration and control access to them with that role-based access control or our back the UI is also for operators to visually check the health and usage of workstations such as checking logs when a support request comes in or Making sure an artist is on the correct configuration when they say hey, this isn't working correctly There are also like the bigger dashboards to check on your fleets So there's common questions like are we out of licenses that we need to buy more license for the software? Or did the auto scaler go haywire and we actually requested a thousand workstations with only steam installed for some reasons that have any Artist software, you know that may have happened that you know here they're there But you know we got rid of them before anyone noticed Over the past year we realized that stakeholders were less interested in an opinion in a product and more interested in a robust and customizable platform We're continuing our add abilities to control and customize the artist experience while improving the platform such as Reliability observability flexibility all those big production level things now that kind of out of that alpha stage but focusing on these and our strong partnerships and then Netflix workstations can be leveraged for an even greater impact So it's kind of like getting more open-source contributors that I have any to do it all yourself We just say here's the platform build what you'd like on top of it So I'm just going to put it all together The workstation starts with baking the AMI. This is where the majority of the configuration happens Then Spinnaker handles the fleet management by turning the AMI into a workstation during the run stage After that the agent starts up and makes the workstation available for acquisition Finally once a user requests the workstation they connect to it via the remote display protocol When they're all done with it the workstation gets terminated and the cycle starts again This system is designed for artists and everyone who support them Artists can be on board in a matter of their geographic location to accommodate any dynamic production schedules They can get any workstations They need with a configuration change instead of having to wait for a new one to arrive in the mail and Textual directors can quickly modify control configurations to maintain stability while pushing out new tools Yeah, I just had recent things. Just go in. Oh This file is missing. This is wrong. Let's let me remote in real quick and get that for you And now you're ready to go and you have to worry about me being sitting next to you and Those pipeline engineers can build on top of the platform to customize the experience to a specific artist workflow and have Whatever you're doing whether it's one frame a thousand frames this software that software You can have the system ready for you So here's what it looks like today for a user to request the workstation It goes by like really fast But it repeats so you can see it happening over and over again They select the workstation and the configuration they need they wait a very reasonable amount of time For the workstation to prep then they select the connection in this case. It's nice ECV So they'll be opening it in the browser and then there it is a desktop with a few tools already open automatically So that's another thing you can figure like hey when this loads up Automatically open these tools because these are the ones they're going to use and if you can catch it You can see deadline another AWS tool that we use for rendering right at the bottom of the task bar See very reasonable amount of time I didn't edit that into the gif that it went so fast definitely So what do we learn here? You always want to debrief your projects to make them better and that's what we did So everything was new when the project started these were new roles new team and you know an empty repository We made the best choices with the information we had then we pivoted when real-world users showed us a different way Here's some of what we learned about deciding artist driven technology So nothing can be taken from granted when making a significant leap in technology at your company The new technology of cloud-based workstations must at the very least match the old experience of dedicated on-prem workstations Or the user won't use it the having the exact same Experience is the table stakes the things are used to have to be there at least For example, they rely on our software settings to reign a place from one workday to the next But if it's not that's not a given on cloud-based workstations to be more methodical about it Therefore, it's like super essential to learn everything that people are doing So nothing is missed the user's workflow and then improving it because why can't it's in the user new thing if it's not better than their old thing? We focus first on technical flexibility instead of the artist experience But artists need custom workflows not standard solutions by initially building the entire product We kind of painted ourselves in the corner We were on the hook for making everything but unable to deliver that needed customizations because we had limited engineers and Said we're pivoting to a platform to build a strong core and allow for a partner customization Empowering others to develop with you gives your teens more leverage than they can accomplish as individuals The more the system makes it easier for others to Contribute the better the adoption and the iteration will be people do things that you never expected them to do Which is great because I need all to do it and they're doing it The engineers need to help each other It's Critical to empathize with your users when an artist calls me. I know it is a worse part of their day That's what I've known the entire time I've worked in entertainment They want to focus on their art and not on whatever it is you broke today because you push something Observability and testing can seem like things that can be cut when timelines can press say you thought you had a year to build Remote workstations and then you've only worked there two months and everyone is home now But they quickly come back and haunt you Especially when you're on call So we committed to improving in those areas and adding more robust automatic testing clearer failure mode reporting All the like good stuff you want in production that you were like, oh, we don't need this right now Everyone doesn't need we just need to get this to work But you need to get it to work And finally addressing one use case at a time from the micro to the macro level Addressing one operating system making sure that works or addressing one You know user persona first and then moving on to the next team and really hardening as you go From the small tickets and the small pull requests and small releases It you know things we try to abstract too early and be like, oh this will work for all use cases never like got it Right and we end up having to rewrite it anyway So it's better to just for us to do one thing at a time and then you see oh this is how All these five teams are using the same way and now we can I'm sure I things out And know you're on the right track So what's next like the Christmas Prince universe? Crossovers and spin-offs make things more fun. I hope you all enjoy the many Christmas Prince movies the Netflix provides Netflix workstations are kind of one part of a technology ecosystem designed to help kind of creators Make feature films and television at an unprecedented rate So you have new things every week to watch and you always stay a Netflix subscriber It consists of many other teams that help our infrastructure in new and exciting ways We'll be collaborating to plug workstations into more and more workflows So more things are automated more things are done quicker and easier and we always know what's going on So our talent worldwide can be part of making diverse content So thank you for listening It's been fun and I want to thank my many sitting colleagues without whom there'd be no Netflix workstations and I think we have time for questions. I think we have about 10 minutes. So if there are any questions, let me know So he asked how do you get the same frame per second in Netflix workstations? So one of the things we do is just being local regions like making sure that there's a workstation as close to the artist as possible We know someone's gonna be working in this country or the city Let's get let's get in this availability zone The second is we're really relying on our remote display providers to get that work and get that latency down So, you know, you see both those things like making sure you don't feel the latency when you're close enough Because like I have to test workstations obviously that are all over the world And I will notice it. I'll notice a latency if I'm working on a workstation Across the world, but the artist won't because it's right next to them And then relying on the actual graphics and getting the pixels there as fast as possible That's our our great partners at Teradici and ICCV Luckily, I don't have to do everything.  Yeah Yeah, so it there's there's a lot Where so he talked to us about color management obviously colors very important to getting your film all correctly So some of that is the back-end technology that you use in film production, right? So there's this idea of color spaces and those are all math So that's happening as part of the process behind the scenes when you're generating the image You're deciding okay, this entire film is gonna be it's gonna use this math formula So that when they all come out at the end all the images are look Match together, so that's part of it and that's kind of like outside of Netflix workstations But it's just how movies work right the other part of it is relying on our Remote display protocol partners and then also on the monitors and the environment that the user is in So sometimes people ask me like hey can people work on their workstation outside, and I'm like no They're doing color-sensitive work. Of course. They're not working outside Like that's the difference between like making a remote workstation for a developer Of course you can work outside as long as you can see your IDE and have access to stack overflow You can work outside, but there's a lot of things that go into making the color correct and if our RDP protocols aren't right and things are off or if Their environment isn't correct or if the the color space or that math isn't correct That could all lead to different color options So there's a lot of people working hard making sure that is a hundred percent accurate and they need really nice monitors and a nice room when I Used to work inside a VFX facility. It was actually no lights allowed So like imagine like a normal office with like cubicles But like everyone's just like sitting in the dark like it's like severance So it's actually pretty funny But like they have to right because they can't have that glare on their monitor when making feature films Yeah, he asked what changed we would make if we had more variables and configuration that we needed to do I think we're already doing that now, which is Letting the users control the configurations and build their own stuff So when we first started they'd be like I want to work station with this on it And I'd literally like download the software and figure out how to install it and set up the formula and package it for them And then give it to them now. I'm like, okay. Here's our platform Here's how you here's where you put, you know any files you need on s3 Here's where you configure it in this repository. Here's how you build an image go to town test it try it out yourself so by kind of Outsourcing is that people want to use it. They can make as many configurations as they want and we're there to make the tools easier He asked if it also increases the support workload and it kind of depends Right like we're on the hook for the infrastructure They're making sure that works and the basics work and when they go to log in they don't get a connection error But if they've installed software and that software doesn't run correctly We just bring it back to them and say hey you run this I see this is your code if you if there's like confusion On how to use the tools that's part of our support burden. That's part of like oh I noticed you tried to do this and you got an error Did that error make sense? Do you know how to fix it? But most of the time? You know we try to have good messaging so people can see like oh I Try to install the software and it says file doesn't exist And it says where an S3 it look for it and the files in there Maybe I should upload the file to S3 where I said it should be you know things like that where it's like We're trying to be more of a platform and a tool system and not debug every piece of software because There's a lot of software that takes a lot of expertise to install correctly It's just some DC yet see apps that are just like you don't know how much time I spent in the first round Trying to get this thing to install and it's like wild to install correctly on the cloud versus a desktop So and into an image. It's very different installing on an image and it's installing on a machine. That's Always gonna be running and you just leave it so there's a lot of I was called image-making like the devil in the details So having more people involved in configuring their own configurations actually helps to disperse that Knowledge because they should be experts on the software. They want to use Go ahead back there Yeah, so there's Teradici and nice DCV. Those are the two you can also just like RDP into them if you're like a power user. I do I use that sometimes when I'm debugging I just use them the one I really like it's called like Royal TSV has a lot of options for promoting in But that's not like for like the artist experience. The artist experience We use the Teradici client or the nice DCV client because that's a that's a really nice streamlined experience where they we actually generate Short-term use codes for them Every time they want to log on so we're constantly checking their credentials because we you know security is so important in movies So it's like a different experience But like as a power user you can kind of get in any way you want you can use Microsoft remote desktop to go in I'm sorry. Can you say again? We haven't tried that. No Teradici was kind of one that's been used other Entertainment companies have used it well before and the you know So we wanted to use something that people were a little familiar with and it worked well for us And then the nice DCV we just liked for the application streaming and the fact that it like Comes free with an ac2 instance that we were already using so if we run into other use cases where those two don't work We might explore other other options, but that's part of the like You know, it's in my general philosophy not to build new features until someone actually needs it So he'll sometimes ask like oh You know, they'll get I'll get an error and I'll be like, oh, were you trying to do something? Do you need a new feature on the bike? No, I'm like great Do it do it the other way then do it the normal way so Yeah, so we focus primarily on studio But there's a lot of different teams at Netflix that are interested in this But it's interesting because studio is like a very specific Area where they need that color accuracy. They need that a lot of storage movies are very big You know, we need to not lose anything It's also very different than like if you think about like streaming like you're watching Netflix That's like 200 million users. So everything has to be very Resilient but like a single user is less important than the group But if you think about Netflix workstations It's an internal tool for artists and if one artist cannot work that is an emergency Like they get to call me at three in the morning if they can't do their work for that day because you can't like Let people lose a work day. That's awful. Like they're not getting paid. The work isn't getting done So it's a different. It's very different experience Yeah Yeah, we're definitely exploring it, but we really want to make sure that we don't stretch So we asked if like Netflix games was gonna use this and it's definitely something that is possible But we don't want to stretch too far until we're really sure so, you know As I kind of mentioned in the beginning we stretched really far in March April May 2020 And now we're like make sure everything works really well And then we're gonna say okay, how do we get new users like what other groups want to jump in to this and Without changing too much so something like games is actually very similar to VFX animation So it might fit better than say like for engineers who have a totally different workflow But engineers and TDs actually don't have that different workflow. So it's an interesting interesting What to focus on in any project user expansion versus Technology hardening and you always don't want to go too far in one direction where you get too many users and something Nothing works or you spend too much time on technology hardening and you've actually gone in the wrong direction Because your users if you'd expanded users you would have gotten more information So we're trying to balance it a little better now that we're not in a rush But definitely yeah, I want as many users as possible Go ahead Yes, he asked about zero clients and I said yes, and that's about all I know about zero clients So there hopefully there's no follow-up questions. I've heard the phrase zero clients and they've used them. I I Believe it's like when you have like a very thin client right where it's like so like I get into workstations on my MacBook But that's like super overkill because I already have a MacBook But if I just had like I imagine a computer just like a little tiny thing and instead of an entire computer that all it does Is log into remote workstations? That yeah, that's another use case that people actually often use because they had a regular computer Why would they need a remote workstation? Oh? Sorry Guy in the gray shirt and then a guy in it black shirt Yes But okay, if you think about it The computers that they were using other desks were also very expensive artists usually don't have like nice cheap computers You can't like give an artist a Chromebook. I don't know if anyone's ever is like seeing like a box desktop It's very common. It's like they have to be so substantial anyway that Using the cloud isn't making as big a difference. So if you're exchanging your Chromebook for an EC2 instance, you'll see a much bigger leap in costs Then you'll see if you're exchanging your powerhouse gaming rig For an EC2 instance you see what I mean like there was there's not traveling as much in cost But one of the initiatives like I talked about the efficiency of not having idle instances sitting around is To do things like that is to like lower our Unneeded costs. It's needed to have an EC2 instance that has a great graphics card and a lot of storage That's a needed cost having extra wide idle workstations that no one's using as an unneeded cost So that's what we focus on is it makes sure of any extra costs Are we don't have instead of the ones that are actually necessary? Okay Yeah, definitely. So there's a There's so many things like, you know, we use this UI and you saw it a little bit in the in their In the GIF but that's just like the artist view and like the admin view is very different So you can see things like open a workstation and see all the grass of things like CPU and memory and that sort of thing So like if you see something spiking and starting to hit the limit You'll get that alert before they do seeing the storage before it hits the limit So you get the alert before they do seeing, you know, hey, this person Logged in five times over 20 minutes. What's that about? Like seeing these patterns of usage is really important to spot like patterns of problem or You know even grander like hey 30 people tried to log in and 30 people had failures All right, something majors down then then seeing that so it's about both the macro and that micro level so we added a lot of Logging and graphs and things like that to the individual Workstation page in the UI where you can see all the details about it and make it a lot easier to debug and then we are also starting to learn those patterns by aggregating the data and Seeing over time how people are using them That answer your question Yes, we call that rendering it happens all the time So what an artist usually makes they don't actually usually make the final image They actually make a blueprint. So this is all it's like artistry and math at the same time So they say they get like um, they'll get like a film plate they'll get the original film and then they'll say like add this in here add this in here add this in here But they'll be working on a very low resolution version of it And then what rendering does is it takes that image and it takes it the selections They've made and does a bunch of math and creates that like final beautiful image. So that's like that's like the headless Fortune Yeah, you don't need like a whole workstation to do that It's actually a super common VFX way of doing things and if you look there's like a The very fast and the gif I featured deadline which is a monitoring tool that helps you monitor render So that when you're on a workstation and you say launch my render to somewhere else in the cloud That does the render you can see it monitored on your workstation So that is a it's extremely common process. That's how all movies get made, but it's a little different because it's really headless Okay, we allocate a pool so that they're always available He asked about we were allocating a pool for spot instances or reserved And it's more that it's about pricing and more about that an artist can come in and immediately have something. Oh We're at time someone's just during and I didn't know what he was talking about But it looks like we're at time So I'm gonna I'm gonna stop with the questions, but so great to have all these questions I love all the interest feel free to find me out like hanging out outside or again on LinkedIn Leave a message so I know you're not a robot Thank you Can you hear it now? Oh, there we go. Hey go. Thanks. I like the feedback. So if we can make this interactive, that'd be great So anyway, the point is that I am not going to apologize for anything other than the slides So Some people did help me get them at the last minute to have some nice Look and feel and all that kind of stuff, but the content is definitely put on there very rushed All right, so I don't want to apologize any further. Let's just just go ahead and get started Chaos is a feature not a bug. The point is that we don't want to Have it happen to us. We want to get out in front of it, right? So Some of the topics I'm going to cover here are a little bit of intro to chaos especially around like reliability and resilience You got to talk about that first chaos engineering, of course in fault injection I'm gonna talk a little bit about litmus chaos. It's the open-source project that my company harness acquired and the team that comes with it so we now have a really nice staff of chaos experts the company that we acquired was called chaos chaos native and litmus chaos is the open-source project that they submitted to the CNCF and it's now in incubating And we're talking a little bit about game days and or chaos days, you know game days can mean different things, right? So specifically a chaos tech game day So first, oh another thing. Let me do this Let me change this a little bit. I have some notes in here be helpful if I did this in Presenter view. I think you still should be able to see well now you can see what I see, right? How can I do this in a way that's a little more useful to you? Let's see here. Let's see. Hold on. Let me go back and put me in a different view Okay, I should probably get prepared for this before I do start my presentation. Hold on one second Let me just stop this for a second because I can't figure out how to do this Now let's see if I go into presenter view still doing that, huh? Oh, okay Thanks Let's go Let's stop mirroring. Is that what I do? Okay, so we're back to not stop mirroring. Okay. Now we go into Let's see. Do I move this over? Now I can go into See what I'm doing over there. Does that do it? All right. Now what happens if I go slide by slide? Oh All right, thanks for hanging with me on that Okay, so these are some of the subjects we're going to cover. I already cover that slide All right. So first before we start talking about chaos engineering. It's it's important to understand the core You know site reliability engineering Concepts that we're going to be applying this to So for example, and again sorry for the mix up on the slides here or the mess For example, it's never really been easier to create large-scale applications that are distributed We've got Kubernetes. We've got all these tools that help us do this and make bigger and grander and more scalable websites, right and So, you know by example of this we've now got our infrastructures in the cloud. We've got an amazing programming language support there are there's lots and lots of open source projects and components that we can use and We've got lots and lots of services now And so it is easier than ever to build an incredibly large and complicated website and so you know, I remember in the I've been in the space for a while or a long time. I've been in DevRel since early days of APIs and I remember in the early days you couldn't Get somebody to use your API because they would tell you why on earth what I want to add Your downtime to my downtime, right? That's that's how it was back then downtime was significant and You know these days We don't think that way as much you still do a little bit But you know, we don't really think too many too hard about using somebody else's service through an API Because their downtime is probably pretty reasonable, you know, if they're especially if they're of any size And so that's not that big of a deal. So we'll add that to our service. However, this creates a lot of opportunity for issues and so You know, nothing is 100% Reliable and so the larger your application the more of these components and these services that you add that creates more service area of more problems that could occur Even even like You know Amazon S3 or Google storage or Azure storage not those Have downtime to even though it seems like they don't right because night night how many nights will they have these days? But there are times when they've been head outages and so that's where resilience comes in, right? So There is no guarantee that these things won't go down and so you know These are the services that our systems are built upon and so any of this infrastructure can go offline at any point in time and You need a plan you need to plan that your applications and service will go down and so there will be disruptions and And to deal with this you think about your those components as being We said they have some kind of reliability right and then if your individual component goes down Then how do you deal with that is really the resilience? So how resilient is your application when some parts of it go down? So resiliency is the ability to withstand certain types of failure in your components or the pieces and yet remain functional to some level and So you can't be resilient if the components that make up your system Are not reliable or you know have some sort of known reliability And so this talk we're going to talk about how you can make your deployments more reliable through chaos engineering All right who am I my name is Dave Nielsen and I work as the director senior director of developer relations or community and advocacy at harness and Some of the things you should know just about harness is that while we started off as a purely CI or I'm sorry CD company It's just advancing okay Doing continuous delivery we then acquired an open-source project called drone and drone is Like the best kept secret, you know, it was created when Docker came out and the brad the creator of drone felt that Continuous integration should be done in containers Because all of those tests and all the different steps in the pipeline. You don't we don't want them to Impact each other especially if you if any of you have used like Jenkins, you know those different steps along the way Sometimes share co-basers or the or the platform the programming language And so upgrading any of your steps along the way could be pretty Difficult, but if everything's running in containers and you just want to update one of those steps You just update the container Could be a different version of the programming language. It can be a different programming language all together and so He built that that's what drone became and You choose your plug-ins is what we call those and you set it up with yaml and you're off and running It's it's very simple But anyway more or less a one-man show I mean he definitely had some others helping him and he had a great group of volunteers building all these plug-ins, but Two years or almost two years ago. We acquired drone and but it had already reached hundreds of millions of downloads with this One one-plus person show that had never raised any funding pretty amazing I used to work at Redis and There's a similar story there with Salvatore the creator of Redis and it's just really amazing when these things happen So I'm really excited to be Helping to promote drone Which is a different subject, but that is how we got our open source Community going or that's how that's most of our open source community and then we acquired litmus chaos Which has a similar story, but more recent and And so now we have two open source projects as a part of our community. Okay, so So failures are inherent to complex systems. And so what do we mean by complex? Basically, you know, you've got Even in a like a mono repo it can be very complex, but now that we've got all the different systems that are You know microservices different teams Becomes so much more complex. It's almost, you know exponentially more complex and so a system would be made up of multiple components each component might have its own reliability, but Really what we're talking about is, you know, how does the overall system behave? Sometimes there might be a rippling effect and you just don't know how The other components in your system might behave if one system goes down So that's a complex system where you just don't really know how all of them will behave if one thing changes And so There's lots of examples where there's been outages and where why have these outages occurred? Well, so for example salesforce.com, you know, they had and they're a big company. They know what they're doing and yet Boom they had a DNS outage and that's what does it go? I forget how the quote goes, but it's something like It's never DNS It was DNS. There you go. Thank you. That was it. Thank you That's the one I was looking for It's not DNS. Okay. It was DNS. All right So it seems doesn't DNS seem like one of those things would be ridiculously reliable, but in yet, you know, it's a complex system, too And that was when that one was took over a week to recover from so Clearly they had not done a chaos experiment on this problem I don't think we're taking a week to recover Cloudflare this one was interesting because this occurred when they ran a script bash script and but they Didn't use pipe fail. And so what happens when you don't use pipe fail if the one of the commands Has an error and you're expecting it to result in some data It's just gonna continue on and continue to do the Run the script but there's no data. So what happens is it goes and replaces data with no data and in this case That caused a big problem for them Because they basically passed in a zero row array into the next Step which then replaced all the data with nothing. So and that was a pretty big outage Then Atlassian had one. This is another example of where This is an example of where There was miscommunication. They were supposed to delete some accounts They passed in the wrong data like the one team gave the wrong data to the other team And then on top of that they had a script that said delete these accounts but Or mark them for deletion but in the same script was no delete them permanently and so they ran the wrong one and So not only did they pass in the wrong account information. They then deleted them permanently. So That was a difficult one for them to recover from So there's lots of different ones. This is one I personally experienced. I was at PayPal many years ago and We were Some of you may have had this experience So I remember sitting in the in the room when the consulting company that we had hired to tell us how to make a big change in our application basically PayPal was able to get to 50 million users on One database Yeah, so we are running on an oracle database on the biggest baddest you know Sun Solaris box that they could possibly have and The consulting company came and it said, you know In about six months There's nothing we can do to make this going faster and we're gonna run out of capacity And so the company literally stopped marketing everything because it'd be much worse for some financial type website to Not be able to respond to people and their money Then it would be to not have as many users and so we literally tried to put on the brakes But PayPal was so viral That even with zero marketing it just continued to grow and So we hit that sick month six months mark right about the time we were doing the update But it was a I mean that's a race condition. We were really worried about The website you know the database falling apart so anyway all the testing was probably not done very well and There were so many dependencies because we took what was a single database and we built APIs on top of it, but those APIs were really kind of All I can say is that we took processes and tried to split them out across multiple copies of the database and Once we put that into production. We didn't know what to expect and it turned out there was rolling blackouts essentially So about every couple days the website would just crash So I think More testing and perhaps chaos would have helped there as well. So there's a lot of examples and these examples are very expensive This is just a quick chart. I'm not gonna even tell you what the numbers are other big, you know, 30 million dollars per Outage that kind of thing. So the problem is is that there's It's so it's very hard to test all of the different combinations of things that need to test and so a Lot of times what happens is that we do the things that we know to test and what we think are important and at some point there's just not enough time and so We may not test things in fact I put this up here because Sometimes you just say I don't know what the problem is I don't know what the results would be if I don't do this I'm just not going to decide to do it now but the problem is you actually are making a choice you're making a choice not to do that and So You don't know what's going to happen But when some people say, you know, we're not running experience. I'm saying yes, you are you just don't know what they are And so there's a lot of different things that happen out there and I won't spend much time on the slide But yeah, there's there's just a ridiculous number of things that can happen. I remember You know the test that we used to do a long time ago just simply unplugging things, you know That was a pretty easy and cheap one to do Which kind of works but There's just so many things that can happen. You certainly can't just do them all yourself All right, so what are some real-world examples of things that might happen? Well, first of all, I was doing some research in the last week on all of this and one of the things I ran across is that Now this is over the past 25 years So it's definitely a lot of things have changed in that period of time But over the past 25 years according to the research I read a percent of all load losses Were in data centers were human cost All right, and I said this the very first person in our SRE team that I ran across I mentioned this to him He's like, oh, yeah Like what he's like? Oh, yeah, my first day of the job. I tripped over cord took out the entire data center I was like, oh my gosh really he's like, yeah, but if things, you know We've gotten better at this kind of stuff those cords aren't laying around as much but anyway There's other issues that that are real-world examples like you know resource exhaustion and you know I was at Redis and one of the things that we would exhaust is memory. Okay, and that would cause a problem, right? So That would be one type or or just your database fills up the hard drive, right? Or your log is fill up the hard drive. I mean, there's all these things hardware failure hardware failures, of course network latency Sudden increase or decrease in usage that could cause problems. I mean increase you can understand but even sometimes going down can cause problems Downstream dependencies, of course functional bugs. Those are some of the ones I mentioned possibly race conditions state transmission errors between senders Unusual combinations of interservice communications and I named this one This is not this is just me making things up here But the Dunning-Kruger problem when one node thinks it has all the correct information But it doesn't So it thinks it's doing all the right things, but it's not Okay, sir, some of these examples. How many of you have experienced possibly one of these examples Okay. Yeah, any example of the you see up you don't see up here Yeah, what's that air quotations? Oh Air conditioning, okay? Like heat. Yeah. Yeah Yes Excellent Yeah, air conditioning actually my first ever job was when I was in high school I went into a mortgage business and my job is to do the backup of the tapes You know, they're this cold room with all the tapes in it and I'd go on there and whatever So when I graduated I found somebody to replace me and at one point I Remember I didn't actually know that person that well, but I knew the people at the mortgage company And so at one point I'm like, hey, how'd that go with that guy? And he's like, yeah, not so good Like what happened? He's like, well, he got cold one day. So we turned off the air conditioning And they didn't figure that out for almost two weeks so they basically ended up with roughly about a week's worth of no backup So they were lucky that nothing terrible happened. Yeah, so he didn't last long Yeah, exactly. So all right now that we've talked about all that What is chaos engineering? All right, so It's the practice of subjecting application. I'm gonna read to you here It's the practice of subjecting applications and services to real world stresses and failures in order to build and validate resilience to unreliable conditions and missing dependencies, okay, so That's what it is. This is this is from a Microsoft website. I think later on I talk about some Azure stuff I'm trying to error some AWO stuff trying to you know be fair to folks And and the next one is fault injection. All right, so what is fault injection? So it's the act of introducing an error to a system different different faults such as network latency or loss or of access to storage can be used to forget and sorry target system components causing scenarios that An application or service must be able to handle or recover from so chaos engineering is more like the overall practice And then as a part of that you're going to run these individual experiments. Those are the fault injections Okay, so now what about what about chaos engineering? Where does it come from? Was anybody around when Adrian Cockroff was giving those early talks? anybody Yeah, I happened to have been there I Work in Silicon Valley, and I run Silicon Valley DevOps And I also ran this event called cloud camp and between those two events. I gave lots and lots of people opportunity to talk and Adrian submitted a talk to cloud camp. I think it was cloud camp and I'd already known him from talking about other things Of course we accepted his talk and he gave I believe one of the very first talks about chaos monkey at cloud camp and Over the years, I've had him speak on different topics But I remember I quoted him on a couple things on Twitter and I think I got hundreds of likes like it was Definitely like a moment. I'm like, oh So that's kind of cool, but it goes even further back to maybe Jesse Robbins He was one of the ones who was actually doing some of the things like you know, just physically unplugging hardware, you know to create Tests, you know at Amazon and then over the time The Netflix ended up not only having the chaos monkey, which basically did shut down instances That's basically what I was doing, but then they would do other things, right? They would create other monkeys that had different skills like and they had the one that would shut down the whole Region, you know, that was the chaos gorilla or they had ones that would they had ones that would you know Kind of make it look like the server was slowly responding so over time right so it slowed down its responsiveness and so they had all these different monkeys and And they ended up calling that the the Simeon army and this was a big deal in the time It was very very innovative and a lot of us were like wow I hope I can someday, you know have a website that would deserve this kind of attention But you know this concept definitely grew and grew and grew and then around 2014 they coined the term chaos engineer and then 2020 Amazon added their Reliability which is including chaos engineering to the AWS well architected framework Lemus chaos was donated at CNCF in 2020 as well and then and then now it's at the incubation level all right, so This is very short and very biased history, but This is my history with with chaos And so folks are not definitely not just talking about anymore and they're not they're not it's no longer just chapters There's like multiple books about this has become a big deal and by the way Yeah, there's some really good books in here So and if you haven't already checked this out I think the AWS well architect framework is a good one to go look at specifically there is the Reliability pillar and there's a lot on this subject there and some on chaos engineering. Yeah, I'll make them available. Yeah, you bet How am I gonna make them available? At the end I have my contact information so you can email me. That's probably the easiest way Okay, so why are we seeing this now? All right. Well So more and more and more I think the way I think of these things is that it just takes time for some of these higher level more long-tailed concepts to come to attention But as we solve more of the immediate problems We are able to solve the story turning our attention to the ones that may not happen today, but may happen tomorrow And so also I think during COVID I think there were a whole bunch of new websites that sped up really quick and Had a lot of problems and there were a lot of people using them So I think maybe all of a sudden we had a new appreciation for You know reliability and then finally I think just as Kubernetes makes it possible to spin up more Container images and your application might grow and have more surface area because it's easier It's also easier to spin up an entire copy of your application and run chaos on that Which was not something that was really possible before the cloud or Extremely expensive because you basically to own a copy of all of the hardware, right? By the way, this is also something we discovered as a part of Continuous delivery is which again what our company does because when we run you can run canary deployments You can do rollouts, but if you want to do a blue-green you're doing an entire copy of your infrastructure And so that's something that is relatively recent So it's kind of a similar thing and by the way if you roll out an entire copy to do a blue-green deployment Isn't that a great time to run a chaos experiment? All right, you got the whole thing right there, and you haven't switched it yet to beat production Perfect time Okay, so also. I think what's happening is that As it becomes more important, you know, there's there's People who've been doing this for a long time and they're like we now all of a sudden have our moment in the sun and in the light And you know we want a name for what we're doing, and so I'm I'm making this up today I'm now introducing to you dev reliability ops Okay, I'm joking, but No You know just like dev sec ops there are some people in the dev ops space would be saying Isn't dev sec ops just dev ops? Right shouldn't we be doing security as a part of dev ops like does there have to be a whole separate thing? And I think it's the same thing with chaos is I think it's there shouldn't be dev reliability ops It just should be part of dev ops, you know, maybe we'll spend more time on it than we used to but That's the way I look at it. Also. I think it's important now for I think from what I've seen and read is that Chaos is actually a very good team-building exercise to run those kind of experiments Because they do require Working together and understanding each other Creating compassion for like some of the people who have to deal with some really big issues whereas others may not so I Think it's great for internal Teamwork not just for customers All right, so here I just want to mention. Yeah, let this chaos is also we're starting to see more open-source projects one of them is litmus chaos and It's a great tool for running your experiments, so I'll talk a little bit more about it, but this project started actually from another The team that create created chaos native actually where it was the same team or mostly the same team between From open EBS and they had done some experiments to kind of Do chaos per it basically built one or two experiments on their own for their own product to test it and that eventually became What is chaos native and so that's where it came from And also one thing that's interesting is like though they can't one of things that's interesting about KF chaos native is that or Letmus chaos is that it's very declarative in a way that you describe what you want to occur and That's become popular. It's becoming more and more popular. So I think folks kind of gravitated folks who were interested in more declarative type of Pipeline creation were attracted to this Okay, and you can learn more about letmus chaos. It's it is a CNCF in community project There's lots of users and it's growing on so on and so forth The project yeah project's growing a couple things. I'll just point out which I can't really see here, but I'll try to get close so Well, there you go So scheduling all of these experiments is is actually really difficult Especially if you're trying to decide in which order your experiments go and all that kind of stuff So that's one of the main things that you'll get from an open-source project like a limits chaos Is you can actually schedule how in what order you want the experiments to occur? All right, so let's get more into some of the key resilience practices. This is where we're gonna get more into chaos and game days, so Yeah, let me back up here for a second so the way I look at the the field of chaos is Being a part of the resilience practice. Okay, so it's more than just testing your individual components and seeing how things respond It's also about just finding out is your entire team able to respond to these issues Because it's not just about Seeing something broke figuring out how you respond to that one thing with a technical fix it also could be Let's say it happens in the middle of the night and the person who gets the information Do they have what they need to be able to respond? Are they trained to be able to do it that they what they need to do it's a lot more than just follow these steps to fix the problem and especially if there's some sort of cascading effect that affects more teams and so That's why I put here like what are the steps to The key practice for resilience rather than just chaos because it's more than just the chaos It could be for example Game days which I'll talk about a little bit more in a minute and If you're doing a game day or even if you know for just real life you should have some sort of disaster recovery plan, right and So that becomes more about it on the on the technical side and then on the people side There's more about the team culture building So first let's talk about chaos engineering and game days all right, so Some some things you probably already know is that if if you're confident that your processes work and that if you're To deploy your application more often you're gonna do it more often, right? If you're not worried that you're not prepared For a failure for example, okay, you'll deploy more often the other thing that happens is if you deploy more often Then you also know that there are there's more Perceived risk All right, even though I think the evidence proves that when you deploy more often.  So why would deploying? less often be likely to create more problems more changes but Go and those are both the right answers, but why most more change more changes what? per release Right, so more changes per release and we make more changes per release. There's more dependencies and From my own personal experience. I remember having a situation where somebody came to me and said hey We ran into a bug in something you wrote four months ago And so not only is it that you get more bugs or more errors But then the developers have to go back and remember like what was I doing then like I don't even have that environment anymore I've got to go figure out how to set that up So there's just so many other issues that pile on and cascade to waiting to deploy So I just want to make sure those of you who are maybe newer Know that all right, what did I leave off here? This is about where my slides getting a little bit wonkier All right, so let's see okay, so there's a book that's out called accelerate and basically one of the quotes that I like was that build systems build systems that are designed to be deployed easily can Detect and tolerate failures and can have various components of the system update it independently I think this just goes without saying but One of the nice things about having these Pieces work more independently as you have less of a effect upon each other Immediate effect, but you still even though if you do this as a best practice You still need to test the entire system because there are times when perhaps for example a System goes down and you don't notice the immediate effect because there's a cue somewhere All right, but now that cue is starting to back up And at some point that cue might get to a point where it fails Or it doesn't necessarily fail, but the backup is so long that there's never going to be a recovery all right, so Even though those systems might be completely independent That failure is still going to occur They're not completely independent Also, this is one of my another quote. I like which is People keep talking about platform engineering taking over devops All right, this is kind of like one of these things where it's like the pendulum keeps swaying one way or the other And so pro tip every time someone explains you that devops is just CI CD basically just tools All right, one way you can explain them really quickly that it's not just about tools ask them how game days fit in All right, because a game day is your experience.  There's another thing we're coining today partial devops I'm just kidding. All right, so what is the best way to understand your system? All right science science is the best way to understand your system and chaos engineering is essentially science All right, so You're you're basically gonna purposely run an experiment and see how it recover how it turns out So let's like out the basics here. All right, so first of all As I said in the title cast engineering is a feature not a bug In other words, you're planning this you're not waiting to find out. Okay, so Again kind of like a little bit of a summary here, but I wanted to reword this because this was not my original material but So but basically these are the four steps. Okay, so you're gonna carefully apply These failures with explicit hypothesis, this is the science part of it, right? So you come up with a hypothesis and you can test your hypothesis All right, so then You also want to Start off with a small blast radius right now They'll tell you that when you're doing when you're about ready to do chaos engineering Some people will say well, I'm not ready to do chaos engineering yet The answer is the question should be well, why not they're like why I need to prepare for that But that doesn't make sense because the chaos engineering is what helps you identify what you need to prepare Right or what you need to fix however You probably shouldn't just run experiments without much thought because the blast radius I mean you could take down Way more than you had intended Okay, so you have to start I would recommend we were all recommends starting small and then You got a communicative course to who is going to be impacted by that Right. You don't want to run an experiment on something in production That's gonna impact folks who aren't ready for that and They need to have a well-defined process and that goes about saying so When this occurs who's ready for the problems some problems are not going to be Anticipated we need to be ready for an unanticipated issues Okay, so in a nutshell Can't really read this here on my screen.  It could be your By the way, is anybody know in a blue-green deployment, which one is first? Is it For sure. Okay Because I think when they first came up with it I was reading when they first came up with blue green. I don't think they had chosen one in the beginning. They were That's why it's blue Okay So I was reading that about how they came up with it and they're like we just wanted two different colors But I do like that idea of having blue be first Because green feels like the newer one, right? Yeah What's that? It is but I'm telling you if you go back and read about it They didn't choose green because it was green, but anyway, it doesn't matter. I go with I'm going with what you said I think that's right. All right, so you could choose your targeted system and again a blue-green environment would be great One to do You can observe the results In that targeted system you want to learn from them Okay, and then you want to select the systems to well, I'm sorry you select the system to test first And then you run your experiments. Okay, that's a little bit out of whack All right, so So you know apply the scientific method, but how all right? So have you guys ever tried to teach what the scientific method is to other people? I did it with my 11 year old it was it was it was kind of interesting To teach her she was learning it in school, but you know when you're learning at school. I don't think they have a real Real world issue to deal with and so I was just trying to give her an experience, you know, let's come up with a hypothesis, right and It's kind of personal so I won't get into it But it was highly entertaining because when kids they come up with like the craziest results Right, they they don't have any real-world experience. So they come up with with results that are Just completely unbelievable And it's fun to test because obviously those things don't happen My daughter's a little bit on the fearful side. So we applied it to something that she was you know dealing with in sports and We were able to prove that it's not as scary as she thought Not the world didn't end when she tried it so But in in your your real-world experiments with with your team, you're gonna come up with a hypothesis and So first you want to be able to measure How the system changes during the experiment and it's important to understand How they behave now, so that's the first thing you need to do is document like what is happening right now So I'm comfortable understanding what I should be expecting And so you want to come up some baseline metrics, you know, it could be just something like response time or whatever, but some baseline baseline Then you want to think about okay Does this actually work the way I think it's gonna work like I have some assumptions about how this should work But let me actually be thoughtful about this and understand how this system actually works and And then you need to actually go and validate that Okay, because you're guessing most likely unless you're the one to build the system So you go and validate is this actually how it works and then you can start thinking about what kind of failures could I inject? to help me prove That it actually does indeed behave that way So one of the things you could do is simply run experiments to prove that the system behaves the way you think it does That's an excellent place to start Okay, because if you don't truly understand how the system behaves when you run your experiments and you get the results back It is really hard to understand what the impact was. What was the meaning? What are you now going to go and fix and change so you really do have to understand how it began out behave Or how it actually works by the way, I was just in a I was just in one of these Sort of like think tank type sessions you know I'll explain it just a group of people who are paid to sit in a room and answer questions about things My specialty is how Different Platforms have evolved over the years and one of the topics they brought up is how do you? How do you innovate? Okay, especially in an established industry and the one the one solution that was proposed that I liked was You get your product Which could be your application and you get the people who know the most about it in a room and then you document every single step of the process every single one and It's exhausting you could take a week All right, you document every single one and for every single one you asked do we need that? Is that necessary and You're ruthless about it And in this process You'll probably eliminate some stuff and you'll probably come up with some ideas about innovation But you could just do this with your own engineering team and your own application to understand what's happening So you can then design Experiments, and I was thinking about this in the case in the context of this talk and Probably we almost out of time No way, okay Geez, okay Sorry, I've got a lot more slides, but okay, so with the context of this talk I was thinking about how you go about designing a game day and why you're gonna get people together to do this and Simply understanding how your system works is is a great benefit You're gonna come up with parts of your process. You can eliminate or improve or Innovate all right. We do CICD. You know how many steps are involved in CICD? a lot But if you think about how you can eliminate them one of my projects and But as soon as I change it to how about the ten click employment deployment project people? Oh, we can do that Right, so it makes you think differently Okay, so anyway, let me let me move along here. So You got to understand your blast radius of the experiments that you've chosen and Make sure you set up abort conditions All right, because things could totally get out of control and you better have a plan for that and then Analyzing the results and then sharing the results. Okay, if you're involved in dev ops, you should know sharing is very important So that's basically the steps. All right, so again, there's a cycle There's also a great Netflix ebook out there if you email me I'll send you a link to it which goes through a lot of this in a lot more detail and one of the things they say that it was different than List that I listed which is what we came up with was The start small but then quickly increase the scope because small may not be useful Big is useful, but you have to be careful about how big you go But if you quickly increase the scope, you should be able to get there pretty quick And so of course automation is very important You want to be able to automate this some people actually have this a part of their CICD pipeline So it's actually something that they're doing every time they're running these experiments every time because if you're big enough Your application is big enough more teams. You don't know what's actually changed. And so you can't actually think Holistically, oh that changed. I better change now. I need to run that experiment You might just need to run them all the time. It's kind of like a build process with your continuous integration All right, so I'm just gonna be a quick screenshot here of the different types of experiments You might do if you're using chaos for example, so we call them experiments And these experiments are now becoming broadly adopted so beyond just chaos between the beyond litmus chaos for example, AWS has now started to use litmus chaos experiments as a part of their FIS service Okay, so these kind of experiments are not just useful if you're using Let us chaos, but they're also useful if you're going to use AWS fault injection service All right, I know I'm out of time here, but Okay, so if you want to contribute, please contribute go check out our experiments And if you want to contribute to one of them, you might find it somebody's running it in AWS for example, right for a quick demo So there's a very if you how many of you seen the Bank of Anthos? Sample app anybody okay. Well, it's a really good one. It's created by Google. There's lots of reasons why it's good I'm gonna skip that right now But basically what happens is This is a very simple example where you have an application and here's all the dependencies very very simple example Banks banks have outages to now they usually have them all separated into individual components one of the services might simply be a Balanced service Okay, because you're using that balance all over the place right what if the balance service goes down? Okay, pretty simple example Okay, so how does this actually work? I Don't know how the how other projects work, but I know how let miss chaos works What they basically are doing is they're putting a proxy and that proxy is is In between every communication between systems or between components So everything is going through the proxy and so if you're making a call to the resilience. I'm sorry to the balanced service You can set up an experiment that intercepts that request and responds as if it would if it had failed Okay, and so that's essentially what's happening in these chaos experiments They're intercepting the request and they're responding as if it had failed and this is even for databases databases They're starting to get slow it'll actually intercept the request and have a timer and Then respond and they'll set it up So it's slowing down over time and so you can actually see What happens if your database is slowly getting slower? Okay In this particular case, it's just a failure of service and so you want to be able to respond You don't want your whole application to go down. You probably don't want to put like three dashes or whatever that is, right? So you're gonna set it all up So that now I know if that's down. I don't want my page to break. I don't want the dashes You're probably gonna want to put some better information there And for that fact you don't want them to be able to stop you don't want them to have to stop doing other work And I happen to know that banks cash this piece of information because when I was at Redis one of the things that was Interesting is a lot of banks would they didn't want to replace Oracle But that morning when everyone checks your phone to see your balance They have this huge spike like around 9 a.m. And they just wanted to cash all that So they just take all that balance what your current balance is and they put it in memory and So that would be a good use case I should share this with my Redis friends and say look you should run a chaos experiment And then they should use Redis to cash that balance so that even if it goes down They're still gonna be able to see like That amount and what you would you do you would say here's the balance You don't just leave it at that because that could be very out of date So you would say here's the balance as of this time Okay, so therefore you're protecting yourself But that's the kind of thought you can go through is if you understand how the systems are working And you understand what's happening behind the scenes now you can come up with a hypothesis You can test it can a cash actually solve this problem it can but wait a minute what other issues are we dealing with? What about latency? This is no longer the accurate information. It's put up the right That additional you know caveat. This is as of this time and now maybe that works Okay, so there's obviously more experiment experiments you can do All right I'm over time, but let me just take one one minute here. I was gonna talk more about game days here so game days Game day you're basically doing this in a protected environment you can do with production as well But usually you're gonna set up a protected environment and run through This whole process with your team, but I wanted to share with you just real quick and if you want the notes to this This is from our Karthik is the co-creator of litmus gas And they can't really really read this real quick, but I just want to share with you these thoughts Okay, because these are like from actually running game days. All right, so obviously define your radius Your blast radius to find the failure. So there's minimal damage So you you kind of can guess what kind of damage there will be so start off with minimal damage Try to do one variable at a time because then when you get the results You know that the result was because of that one variable that makes it much much easier to then go and correct Use a scientific method. All right Simulate as many real-world problems as possible. Here's some examples Resources this is these are like the popular ones that he knows it came to his mind Resources becoming slow noisy neighbor, you're on a server Maybe you have multiple of your own services running on the same physical machine. All right noisy neighbor can eat up resources Availability zones may go down instances may become unresponsive of shut sudden shutdowns This is one that I think everyone should be testing for black swan events. Oops misspelled that These are hard to come up with But it's a great exercise to think through because sometimes you'll identify some what you thought were black swan events that are not Black swan events. All right, so it's great to think about those Buildings going down regions going down AWS will tell you which like where their Regions are Google does not give you that much information. So if you're on Google, you may need to do a different type of test If you can test in production because that's where the most unknowns are right, that's where That's organic right people traffic all those things you don't have in a test environment Try to repeat them continuously. You don't know what's changed automate tests so that you can okay This is I think probably one of the most important things to do automate the tests Okay, it's it's super hard if you can't reproduce what you did so you got to automate the tests and Just keep in mind like I want to end on this thought which is that human beings are most likely the cause Okay, that's just the way it is We're out there Mix them up with these systems and we caus