Our next speaker is a dear friend of mine, Robert East. We will talk to us about David, an 81 decoder. Hello. Do you hear me? Yeah. OK, I wonder why half of the room is here, because you know probably better than David about than me. But let's try anyway. So yeah, oops. OK, so yeah, that doesn't work. Yeah, I'm using KD with Wayland and sometimes I crash it with a LibreOffice. Yes? Why does the clicker not work? Yeah, it works. OK, so I'm the president of Vidolla and I work on various open source projects, including VLC. It's 264, David, a FEMPEG, and others where I maintain some DVD stack and a bit of Blu-ray stacks. I'm annoying to many of those projects when they screw up. So I'm not probably the biggest developer there, but I'm doing everything else. No? Yes? So AV1. So VP9 is AV1 just VP9++. VP9 is a semi-failure. It's a good codec, but no one uses it. When you ask YouTube, they don't even have many people actually submitting video in VP9. And there are many reasons for that. The reason is mostly because there was no spec for a long time. There was a codec, but no specs. There was absolutely no ecosystem, except a few open source tools. Have you ever watched an anime in VP9? Maybe, right? One? OK, amazing. You're the only one, right?  But you've never been on the Piat Bay, right? And so a VP9 encoding anime, right? So the format is a failure. One of the reasons also is that H264 was so good and so well integrated that it worked. So they decided, except YouTube and Netflix, no one actually used VP9 largely. So for AV1, they decided to not just do VP10, but to do something that was actually open with an actual ecosystem. And I think that's more or less they managed to do it. It's interesting that basically there is three companies that actually worked on AV1, Mozilla and Cisco, plus Google. And there is an actual Alliance for Open Media, which is open, and where people actually contribute. Well, we cannot do say that about VP9. So everyone has a lot of amazing results. We are talking of 20% to 25% better than VP9, and the same around HEVC. And you can discuss about so many cases. But that's the rough numbers that you should remember, 20% better than HEVC, the good number, 25. And there is a lot of things happening on the ecosystem. There is at least three open source encoders. One is going to be presented just after my talk. There is the LibreOM encoder, which is like the reference one, which is extremely, extremely, extremely slow and a difficult code base. There is a new one that is happening that's called SVT-EV1, which is basically done by Intel. It's difficult to talk about that, because six months ago it was nowhere. They were announced at IBC six months ago and they're advancing fast. It's actually usable to produce some videos. But a lot of non-open source people are working on AV1. There is Eve that is a very good encoder about AV1, extremely slow also. Atem, harmonic, bit moving also are doing a lot of things around AV1. There are actual hardware encoders like NG Codec is working on. And some other mostly Chinese companies working on FPGAs or ASICs for AV1. There are actual deployments of AV1 almost since day one. YouTube, Netflix, Facebook, but quite a few cloud vendors who are selling a lot of things about encoding video have an option to encode to AV1. So people care. And you know what, there are specs and there are other specs to put that inside. MP4, MKV, TS and all the stuff like that. Amazing, right? And this year is a year of AV1 because hardware is coming. Intel has more or less announced that they were doing it this year with their Jenton chips. Nvidia has also more or less said that they were for the next chip, so probably around when they announced product which is GNC usually. AMD also, but like the Samsung TV at CS had AV1 decoding, I'mlogic, Podcom and a few others I forgot show chips that were actually usable, produced that can decode AV1. And all of those can decode both 8-bit and 10-bit which is quite important because like you remember for H.264, lots of people just had how 8-bit decoders and not 10-bit. And 2020 is also the year of the competition. Like you probably heard about VVC which is coming in July. EVC which is coming between April and July which is basically a version of a VVC with less patents. There is something called MPEG-5 LC EVC, ECV something like that, which is actually not a codec, but just like post and the pre filters. And the AOM community is already talking about research on AV2. So things are going fast. So is that an actual competition? I don't think so. Most in like VVC is amazing in terms of technical quality. Most of the improvements are based on H.CVC and H.CVC have three or four patent pools so like you can expect that. Since H.264 had two, H.CVC, VVC is going to have five or six or seven. Means like no one can actually deploy that because it's insane those many patent pools. And there is so many people who are now big companies who are just turning into patent roles like Nokia who are outside of patent pools anyway. And so the question is are improvements good enough to justify the cost or like H.CVC is not deployed anywhere than broadcast. So everything that is online is basically skipped that. So I guess it's going to be the same for VVC. VVC which is like oh, we're not really so much patented but then you remove some of the gains. So like if you remove so many of the gains you're going to be at the level of every one then why not choose every one, right? And then like other stuff are actually not codecs and could be applied to open source codecs. So I think that even if competition is coming they're going to have, that's their last big shot and they might have like difficulties. So David. David is an AV1 decoder as the name says. The idea was that we actually need to have a good and fast software decoder because like a lot of people are not going to have hardware decoders and until people can decode then you're screwed because your codec is not being to play anywhere. And the problem is that everything you're going to see it's true for AV1, for VVC and all the new codecs. They are very complex. Like a lot of code is required to write an AV1 decoder because we tried to take any small gains. So there is lots of tools and you take 1% here, 1% here, 1% here, but that's a huge code base compared to H264 or VP8 or VP9. So the idea was like we need to have a very good software implementation that is extremely fast and like every cycle counts because if we are going to deploy that it's going to be billions of people who will still have their correct and actual machine that they have today and not their new machines. And if we don't have that then everyone will fail. The idea was to use basically the people from VLC, FFMPEG and H264 who actually know how to write C and know how to correctly write portable and cross-platform tools and don't use CMake or some weird configure stuff like libvpx and libaom that basically is impossible to support to so many platforms. And one of the goal was to have a small binary size because for YouTube or Facebook, when they ship the decoder in their Android apps, they actually care about the binary size. And that was for example the mistake that was done in FFVP9 which was a FFMPEG VP9 decoder that was done before by basically the team that basically did David and they didn't care that much about the binary size and that was an issue. So we launched it last year. Now almost one year and a half ago in October 2018 and we had like a release quite soon after and it's been improving quite a bit. Just a bit of history. Like Announce was in October like already like three months after we had like the first release which was already four times faster than the reference and the only decoder at that time. And there was of course focused on 64 bits and then like after three months we did another release which was focused on ARM. Like we are already twice as fast as LibyOM on ARM 64 and the same we went every release we focused on like less important platforms like ARM32 and then SSE3 and then even SSE2 and so on and so on and so on. So we have like one release every two or three months which is quite nice. Performance is amazing, right? Like we're talking about three to five times faster than the reference decoder. The reference decoder has assembly in it, right? It's not like we're comparing C to non-C versions, right? It's assembly against assembly. When we started, Ronald who wrote a lot of the code of David said, yeah, we might be two and a half times faster. No, we are a lot faster. And we even faster on SSE2 where we did not write as much assembly as we did for the other platforms. Then ARM. So ARM, you can see that I'm comparing to the new, oh yeah, wow, I got a lead. The new one, which is Libji, everyone, which is another decoder wrote by Chrome and totally not in a non-invented year syndrome fashion because they really wanted to have one that will be faster than David on ARM. So that's the blue one. And as you can see, well, David is quite a bit faster already and they're improving, but they're not getting close to us. So we're talking two and a half to four times faster than the other decoders. That's a question that was asked quite a bit in the past which was what is the complexity of doing AV1 decoding? So here you can see in yellow, which is the FFH264 decoder, so the decoder of H264 inside FFMPEG, then you have in red the VP91, and then in green you got HEVC and David. What you can see more or less is that, of course, VP9 and H264 are way easier to decode, but that when you spend enough time on David, on AV1, you can do decoders around the same complexities and FFH264.  Sure, we could maybe make the HEVC decoder a bit faster, but not so much faster. So everyone is not that complex to decode in terms of CPU, but as there are many tools, it's long to code. Okay, are there some weird stuff on the David? Yes, there are two, oops. There are two things that are weird for decoder. The first one is actually to dual pass decoder, not encoder, you see I just made the mistake. It's quite hard to have that. The first part is basically to analyze a few things to be able to schedule after. And also to do a lot of passing because there is a lot of things to pass at the entry point of an AV1 stream, OBU and other stuff that were probably not the best designed ever, but they were in a rush. And the second one decodes. And we have what is more important for most of the people is that we have a dual threading model. So we have a frame threading model, so you start decoding one frame before the next one. But at the same time, so we have a slice or tile threading models where you can basically, because most of the video you're going to see in AV1 are encoding slice or tiles. So you can start decoding the first row, the second row, the further row and so on. So when you try to get the best performance out of David, you need to basically set both how many styling thread and how many framing threads you're going to have. And the thing is we might add more threads for the filters. So we need to do something about that to have everything automatic and works at the best. And we would like not to use machine learning to decide that, even though it's completely in fashion to do that. The rest looks like a normal decoder like you have in DBV Codec except a bit bigger. So the question is why is David faster than the competition? And there are like three main reasons. The first one is like, we're seeing the single thread C version. And you see that the C is quite well optimized and well written because already the C version in single thread is fast. And more is coming because there is one big part of David that is not optimized in the C version that is coming soon. In two weeks? Yeah, sure. In two weeks. The threading is quite amazing. So is this the number of threads? Yes, this is 2,000 threads. That is a graph that was done by some of the Mozilla teams. And you see that basically like David's actually like scale with threading quite a bit. And if you see LibéOm or EJV1, they just like cap around four or eight threads and then they don't improve while David can still improve, which means that like it's actually good. Oops, of course. And we write actually low level code. That means C, so LibéOm is in C++ so there is no C++ overhead. We write handwritten assembly, not intrinsics. Yeah, but intrinsics are easier. Yes, they are easier. But you lose between 10 to 15% that's what we've seen lately on the various threads on the FFM. Like mailing lists, intrinsics are slower, almost always. So no interesting handwritten assembly. So basically David is like, the C version is faster in single thread. We can scale better with threads and we write lower level code, which means that of course no one is going to be David ever. So basically in an AV1, in David there is like eight stage that are basically that we managed to ASM correctly. The first one is the MSAC, which is basically an entropy decoder. There is inverse transform, the motion compensation, the intra prediction, and then the full after is our loop filters. Loop filter, loop restoration, the famous CDEF and the film grain. Film grain is quite debatable because a lot of people don't like that. But so when you now start and run David and you're perfect you realize that the parts that are basically non-ASM are around 25% of the runtime. And that's mostly on what we call the FMV and the decode coefficients. So that's for AVX2, FSC3, ARM34, and I'm 64, I'm 32. You can see that basically the optimization done for AVX2 are full and there is not much to gain on writing better optimizations. There is just like a few tools like in 4.4.4 or some intra prediction that are not done. For SAC3, SAC3 is probably one of the most difficult assembly to write because we need to support both 32 bits which has absolutely no registers and 64 bits and you need to care about the Windows calling convention and the Linux calling convention and of course Mac which is another mess. So that's quite a bit difficult but that was mostly done. There is some film grain that is done but it's not merged yet. So next release it's merged. And as you can see, we did some parts in SAC2 because that was easier. Mostly it's an entropy decoder which there is no in AVX2. Some of the motion compensation, one of the loop restoration and the CLEF. ARM64, you can see that mostly everything is done except the film grain and ARM32 has still quite a bit of work to be done. The entropy coding, the invest transform which is large to write in for everyone because it's quite large and the intrapredictions like there is DC, H and V that are done but the rest are not. Which brings me to that. Here you can see basically the, whoops. You can see the X264 is an encoder, it's extremely fast, it's blah, blah, blah. We wrote a lot of assembly and when you look at the graph of X264 you see that around 25% of the code base is assembly and the rest is C. And of course like if you look, X264 is 68,000 lines of code and around you see 37 lines of code of assembly. And when you take the whole Liby codec which is half a million lines of code in C, you see that it's 80,000 lines of code of assembly. But David is weird. It's only 25,000 lines of C but it's already 64 kilo lines of code of assembly. That's like, that's a lot more than X264 and almost as much as what you have in Liby codec because there is the 10 bits arm assembly and X26 assembly that is coming soon and it's going to make that just for one decoder to be right more assembly than the whole Liby codec. We worked on something weird last summer. So if you listen to me like one year ago, I would say like, yeah, everything related to GPGPU is idiotic, it's too slow, there is too much latency. I've said that quite a few times and other people of the community said that too, right? It's very difficult because then you have the latency to upload the textures and get it back and it's quite difficult. A lot of tools are QDA based and which is not open source at all because NVIDIA doesn't like open source. But we said like, what about we try anyway? In David, the film grain is already a GLSL shader, right? You have a C version that is optimized but what we advise people is to actually do that in the player because the film grain is really after that your decoding is done. But the question was, can we do more? Can we do CF loop restoration and other loop filters? And it's very difficult because you cannot really know if it's going to be faster but do you actually care about being faster or do you not? What you mostly care about is the consumption, right? How much battery drain is it going to be on your phone, on your underage phone?  It's only for gamers who care about 144 DS. But for video, you don't care about that. What you care is to not drain your battery when you're watching your video on Instagram or in Snapchat or whatever you're watching. So we had a GSOC who did that during the summer. The guy had no idea about most of everyone. And so he did basically the C-Def and the loop restorations of both SGR and Vinner in Vulcan shaders tested on an Android Huawei phone. I think. Yeah, Huawei P20. And what you see is that, what we saw is that we didn't get any speed increase. However, for like basically the same decoding time, so we put basically a VLC running and playing the file on loop, you see that basically we get 20% less battery drain on that just by using GPU computer shaders. Which was not expected. So what is the future? The next work on David is going to be, of course, 10 bits. And then decide what we can do about GPGPU and how can we move that to the next level. Thank you. And I'm taking questions. Am I allowed? Yeah. How much time do we have? We have a lot of questions. I'm not sure we'll be able to answer all of them. But let's start. How does the dual pass requirements fit with the current load agency trend in the streaming world? I don't think it matters most, much on the way it's done. It could be problematic in some cases, but I don't think it matters the way it's implemented. Maybe you should just read that also, but that should be okay. Are the Cisvell patents pulled for everyone valid? I would say no. They are surely everyone patent pull patents that exist, because there is a patent on absolutely everything. I think it's very, very, very small things, so they don't matter at all.  All of this is for 8 bits, which is nice and fast. What about 10 bits and HDR? Yeah, it's coming. Probably first we'll be ARM64 and then X86. Is it possible to use AV1 as an intracordemt? Yes. Is it a good idea? I'm not sure. Why not? I think stuff like FFV1 are easier to do that. I mean, I don't think everyone was done for that, but there are people around there that might know better. AVIF. Yeah, AVIF for images, but... What are the major companies missing from AOM and how to push DVC? I would say that some of the people who are in AOM, like Samsung and Apple, are important, and we don't know exactly what they're doing, because those are big companies. Who is currently using David? Oh, that's a good question. So every version of Chrome, every version of Firefox that is shipped, every version of VLC, Fempeg, and most players based on Fempeg use David, so that's around everyone that is shipping everyone, except Android. What kind of operations do you use from SSC3? But I'm not in SSC2. Just thinking... Ask that guy. If you want more details, ask him. Can David use GPU acceleration? I think you answered that. Yeah, no, that's not OpenCL, not QDA. This is compute shaders. We're using Vulkan compute shaders and not. When do you think we can expect wide AV1 support in cheap low-end devices? End of the year. September, you will have cheap or cheaper Android, like 200 to $300 devices that can decode AV1, 8-bit, 10-bit, 1080p. Well done. Thank you, David. Thanks.


I'm going to get started in just a second.  All right. So welcome. This is going to be a talk about WordPress workflows. It's geared towards beginners, but I'm hoping that there'll be a little something for everyone. I'll take questions at the end. I have quite a bit of material to go through. So if you don't get to ask yours, definitely come and find me afterwards and we can talk about this. I'd be happy to. All right. So just a little bit about me. But you know, who's giving this talk? My name is Anne. You can find me at redrambles on Twitter. I'm a mom of four, two in their 20s, two under eight. I don't know what we were thinking either. I'm a self-taught developer. I teach and I like to mentor a lot and I believe in that relationship. I think that there's a lot to be gained both as a mentor and as a mentee. So that's something that I strongly believe in. All right. So before I dive into the workflow bit, I'd like to share a story with you of how I got started. So I got started with WordPress several years ago now, and I think it was a fairly common beginning. I had a small business. I needed a website. So I did a bit of research and I quickly settle on WordPress as the best solution. So I read some articles, I look at some videos, try to get the lay of the land until I feel confident enough to go ahead and get a site up on a live server somewhere and start working on it.  So they set up the files, the database, the link everything for you and they just hand you the key. So I log in and I carefully choose my theme. It was one of the first responsive ones. I remember that. Then I lovingly enter all my content in my images, all six pages. I was very pleased. I get my custom domain, I hook everything up, everything's going swimmingly. I guess just as I'm about to break out the bubbly and call all my friends and say, check out my website. I did this thing. I was like, there was this one line of text in the footer. That was really annoying me. I just could not figure out how to change it. I wanted to either make some edits to that line or just take it out. I hadn't found a footer field anywhere in any of the pages that I had been editing so far. So being excruciatingly aware that I was very new to WordPress, I figured I just missed it. So I do a little research and I find out about this thing called screen options. So I make sure that all the fields are visible in all of the pages that I had been editing so far, but still no footer field.  So I click on every single thing, a child menu, a sub-child menu, click, click, click, click, click until, well, at some point, look what I've found. I remember being like, what? There's an editor? How come I didn't find this before? Has this been here the whole time? This was before the days of the big warning.  You want to do this? I'm paraphrasing but more or less, that's what it says. But there was no such thing back in the day and I was, so I clicked on it and in fairness, I did hesitate. I had a moment where I was like, oh, I feel like I'm intruding on a private moment or maybe, I'm witnessing the underbelly of WordPress, maybe I shouldn't be here. And so I was about to bail, but then my eyes slid down the page and they fell on the word footer.php. So you understand after all that, I wasn't gonna leave that alone. Of course I was gonna click on footer.php, so I did. And okay, it wasn't a huge file but there was a lot in there that I hadn't seen before but I did find that sentence, it was there. And so I was like, yes, problem solved. I make a little change, I save, I refresh the site. The good news is that that particular sentence was totally gone, not there anymore. The not so great news was, so was my entire footer. The whole thing was gone. No more cute little footer widgets. That little footer menu I liked, it's totally gone. So I was like, okay, what happened?  So naturally I go back, I go back in the editor and I try to undo what I did and that doesn't work. And so I think I lose my mind a little bit there. And I start to click around different files inside the editor, making little changes. Of course, by the time I refresh the site a second time, that's what I get. A big ol' nothing. White screen. And I remember frantically refreshing and refreshing and hard refreshing, clearing the cache, different browser, incognito, my phone, anything that would deny this white screen and alas, it remained. And I think at this point, it's probably when I made a strangled, mulling sound, because my partner ran into the room and was like, what is going on? And what is going on, everybody, is that my workflow sucked. It was really bad. So we're gonna talk about workflows now. So that hopefully you are spared such a situation. So that was my intro segue. So these are the items we're gonna look at today and disclaimer, any one of these could warrant a full talk. So it's going to be a brief, higher level overview. But my goal is that you are going to understand what these are, enough about what these are and why they deserve a spot in your workflow. All right, so let's get this out of the way first. Like, what is a workflow? What does that mean for me? It's a process by which you make edits to a live site. So it could be just getting that site up there, live to begin with, but every subsequent change that you're going to make, the how you go about doing that, that's what I consider part of your workflow. They vary a lot. Some of them are very simple, two or three steps only. Others can be a dozen steps or more. So it'll really depend on what kind of project you have, do you work with a team, how comfortable you are with many of these tools. So we're gonna look at one that's a pretty solid middle-of-the-road workflow. So again, this is what we just saw before in little icons. I'm just gonna take a breath, thank you. So we're gonna look at local development environments, solid text editor, the command line, WPCLI, Git, GitHub, staging sites, and a recovery strategy. Okay, why? Why would we wanna use a workflow? Well, if my little story didn't convince you, it does make you more efficient. Fewer errors slip into production, which is a big one, and it increases your confidence as a developer, which is probably my favorite reason. Let's begin with local. A local development environment, so what is that? It's a server environment that exists on your computer, right? So WordPress uses PHP. PHP is a server-side language. You need a server to work with WordPress. This allows you to set up a server environment on your machine instead of having to use the one that's provided by your hosting company. Okay, so why do you want one on your machine? Well, because what happens on your computer stays on your computer. So that white screen is between you, your computer, and the bottle of wine. Nobody else needs to know about the white screen. You can just take a breather and then fix it and then come back to it later, and it's all good. It's a lot faster, right? So you have direct access to the source files, you just edit them in your editor of choice, save, refresh, and boom, you can see whether or not you've been successful instead of having to move things over to a server. You can work offline, so for some people, this is a big one if they travel a lot or if they work in other areas, cafes, or places where Wi-Fi may be a little shoddy, so this is a plus. You don't have to pay for hosting, this can be free. There are some local development environments that will cost, but most of them are free. It allows for a much wider range of tools, some of which we'll discuss today, and it definitely helps grow your confidence as a developer, so you can break things, fix them, you can experiment, right? What happens if I comment this out or if I change this template tag for this one or what if I remove the WP underscore footer tag, bad idea, you'll see what happens. On your local environment, you can do that. So it gives you that safe space to grow your skills and deepen your understanding and knowledge of WordPress. You have a lot of options. There's some that are more beginner friendly, and there's some that are more advanced. The ones that are advanced are basically, they need a bit more configuration and they usually heavily rely on the command line. So you can find these, most of these are free, and if you're not already working locally, I would, I'm happy to help you with that if you want later on. If you were to press me as to which of the steps that we're looking at today were the most important, this would be the one that I would choose. All right, next up. Okay, we have a nice, safe, private space to work, but what are we gonna use to work with our code? We need a text editor. So I presume that most of you already use one, but it's such an important part of the workflow that I'm just gonna take a moment to talk about it. So text editors are IDEs, which is short for integrated development environment, which is basically a text editor with superpowers, a bit more functionality. It just allows you to edit code. And if you are either thinking of picking one up, if you're just getting started as a developer, or if you're thinking of maybe switching to another one, there's a few different things that you might wanna keep an eye out for. Cross platform is always a plus. Rich plugin package environment is nice. Speed and lightweight is particularly important, at least to me. I really dislike having to wait on the text editor to open something, save something. It interrupts the flow of my thinking and you really want that text editor to be sort of like an extension of your brain. So you wanna have something snappy. Customizable, some people really like. Git integration is very handy, but there's a little asterisk here. I don't know if you can see it, but down there it says does not replace the command line. So what I mean by that is that text editors, I'll use the git integration to have some general overarching information. What branch I'm on, what files have changed since I last committed that sort of thing. It's super useful, of course, if you need to resolve merge conflicts. But anything else, any git command I will be using in the command line. And last is terminal integration. So you actually can just work in the command line in the actual text editor if you'd like. Some people like to remain in one environment, so this one might be something to consider. Lots of options again. Here's a few, we've got visuals to your code. That's what I'm using right now. Sublime text, Atom, Coda, PHP Storm, that's an IDE. Brackets, there's lots more. But so basically you'll just wanna consider when you're choosing this, which of those points are most important to you and make sure that they meet those standards. Quick recommendation, when you do make a choice, try to stick to it for about three months if you can. I know it feels really long. Because there's always like a shiny new editor that you might wanna try or something that your friend is using that looks cool. But you wanna get out of the where is this thing again or how do I toggle the sidebar or where's the fuzzy search and all of that stuff. You want to be able to just have that be second nature so you can truly be productive with your text editor. And that only comes as a virtue of time. So you need to give enough time for you to feel that mastery of your text editor. There's a lot of tips and tricks and best practices out there like a stunning amount. People who have been using your text editor for way longer have written stuff about it. Lots of gems that you can find that helps you level up and leverage that knowledge quickly and put it to use. And if you do a search again with that in mind but with your particular language like PHP for example, again you're gonna find some really good tips. Like this is a great theme because the syntax highlighting is awesome or whatever. Lots of good things that you can find with that that allows you to leverage the strengths of your editor. Okay, we have a safe space. We have something to work with. Now we're gonna jump to the command line. Why bother? So I get a lot of pushback about this sometimes because folks think there's so many great GUI apps out there. Why would you bother with the command line? So GUI is short for graphical user interface so what you would typically use with the mouse. And the truth is that understanding the basics of the command line is absolutely essential to becoming a skilled developer and being a WordPress developer is really no exception. There's a lot that you can do in the tunnel so like a ton, basically nearly everything. But we can look at some of the most common things so it is the most flexible and powerful way to interact with your computer. You can manipulate files extremely quickly. You can download things directly on the command line. You don't even have to open your browser. Use SSH so that's gonna be really important depending on where it is that you're hosting. Search features are excellent. You can do things like turn on or turn off hidden files on a Mac and little things like see what's running on your computer if it's getting really kinda slow or sluggish. You can run a quick top and then see what's eating up all my memory and then maybe kill that process if you need to. And all right, really why bother? Well, there are some things that you can only do in the terminal so some of those like you will need to install software as a super user sometimes. You might need to run some quick scripts but things like bash scripts, package managers or tools like WPC-LI, you're going to need the command line in order to use. So here's a little joke that I'm hoping you get. Yay, some laughter. For those of you who may not get this, sudo is short for super user do. It's kind of like the nuclear command. It's like telling your terminal you really, really, really, really mean it. For me, it's kind of likening it to when my mom used my full name for something. You know, she'd be like, and clean your room. And, and Jennifer Cascarano, clean your room now. And I knew I had to move because I'd be in trouble. So sudo is kind of like that. They really mean it. So it's not all business. There's a few fun things you can do. You can have your computer speak out loud, even sing, kind of. If you use this, if you use a Mac, this is for a Mac. I highly recommend that you pop this in. I'll have slides to share later. It's very funny. You can also ask the kids to do things with the computer voice, which seems to be more effective at times. Pro tip. You can play games if you like these types of games. So that can be fun too. So this is just my attempt to woo you to use the command line if you do not already. All right. Next up on our list, WP CLI. How am I doing for time? Okay? Okay. WP CLI. So what is that? It is a command line tool, see, useful, for interacting with and managing your WordPress sites. So here you have just a little example, very simple command, WP theme list. That's exactly what it says. It lists the themes that are on that particular site. It shows me which ones they are, which one is active. If they need updating, apparently they all do. What version they are. And you can do a ton of stuff, but this is what it's gonna look like, the output of WP CLI in your terminal. So this is one of the best arguments in favor of using the command line with WordPress development. So what can you do with that? You can check which plugins are active, update core themes, plugins, export your database, run a search replace, which includes serialized data. Yes.  You can do an enormous amount of stuff with WP CLI. I also have a couple of tips. I'm not sure if you can see them well, otherwise on the slides later you will be. Aliases here are super useful because some of these commands tend to be quite long. So if you have one or two that you keep using, you'll wanna map them to an alias, and an alias is basically kind of a shortcut. It's a name that you give, it's like a variable, right? It's a name that you give to a long command and you could just use that name instead. I have a couple here to show you examples that I use pretty frequently. The first one is just to check everything that might need an update. So I have an alias that will map to this command and it'll just immediately tell me, does core need updating, does do my plugins need updating, do my themes need updating, and I'll know right away. Backing up a database, also nice, you can give it some different options, like I'm excluding a couple tables there, but there's other things that you can do. And I also have an alias for that command. So if I were gonna run that command right now, the result of that would be something like success exported to backup underscore 2018-08-11.sql. So that'd be pretty handy, I use that one a lot. And before running a search replace, just to double check what's in the home or the site URL field in the options table. A quick query of the database, again, WPCLI can do that for you. All right, moving on to Git. We have a pretty good workflow so far, but now we're gonna add Git and it's gonna be, it's gonna get even better. So what is it? Distributed version control system. Basically, it keeps track of your code history, right? Git's number one priority is to protect the integrity of the history of your project, so it tracks changes. It takes multiple snapshots of your project at different points in time, points in development. It's very, very light and fast.  How does it help you, right? Why would you wanna use Git? Well, you can save snapshots of your project at any point in time. I know I just said this, but let's just think about that for a second. This is kind of like a checkpoint or a save point in a video game. You get somewhere, your character is the way it is at that time, has the inventory they have at that time, the experience they have at that time, and then you make progress.   Let's go back to this save point and instead, I'm gonna make some different choices, you know, like maybe don't burn down the mill or whatever it is that you did. And you can just go from there. So this is, you can do the same thing with your code. You can go back to a previous version, take it into a different direction. You have a lot of control, and these save points are called commits, right, in GitLand, and you can have as many of these as you want. So that's an extremely powerful feature. You can isolate changes by creating branches. So that is something else, probably the second most important point. Let's say you're developing a feature that you know is gonna take you a couple days, maybe longer, maybe you're not sure how it's gonna turn out. Maybe you don't know if the client is going to approve. You don't want to input this code or add this code to your master branch, which is basically your main deployable, ideally bug-free code because you don't know where you're going with this code yet. So you create a branch and you basically develop there in parallel with the master branch that can continue to evolve. And on your branch, you can sync up with the master branch as you are working. And only if and when everything goes well with this feature, then you can merge it into the master branch. So it allows you to really just have that extra safety net of not polluting your good code with something that might break. It makes it easy to collaborate. So that's also a huge, huge thing. If you're not working, if you're working as a freelancer, might not be that important to you, but if you're working with a group, get is essential. It can be used to deploy instead of FTP. So this is a lot less error prone. It's a lot faster. And it also means that you're not going to accidentally overwrite other people's changes. Super important. And can be used to just reveal a massive amount of information about your project as a whole, right? So how did this particular file evolve over time? It can tell you. What are the differences between this version of the project and this one? Or what changed over the course of two weeks? Git can give you all of that information. So it's really, really good just to gather that kind of data. Some considerations. Git is not GitHub. This is often kind of mixed up at the beginning. They're related for sure, but GitHub is a service that will host your repositories. This is what makes collaboration really great, but Git is that version control software itself that we were just talking about. You'll want to use a git ignore file. So that's something, again, you can spend some time researching. There's a lot of opinions about this. There's a lot of examples that you can find online. I kind of like to think of Git as like the eye of Mordor, you know, like sweeping the value of your code with its burning eye. Does anybody know this reference? Please say yes. And then Frodo in his invisibility cloak, that's our git ignore. You're like, nope, not this, not this, not this. So the eye won't see those particular files. So one example would be your wpconfig.php file. You don't want anyone to see that. That's not something you version control. It's also environment specific. You don't want to be moving that around somewhere else. So there's a lot that you can put in that or there's a little, it depends what it is that you're versioning exactly, but you will need to use one. There is a git workflow, meta workflow in workflow. So yeah, if you are working with other people, they are going to have a particular way that they like to work with Git. And they're going to basically, it's going to be one of the first conversations you're going to have with them. Are they going to use feature branches?  I mean, these are different ways. If you're working alone, that's 100% your call, you decide how it is that you want to work with Git. But if you are in a team, you're going to have that to consider as well. And stick to the command line. That is my opinion. I know not everybody agrees with me on that one. There are some really, really great guries to use git like tower, source tree, Kraken, I think it Kraken. There's some really, oh, git Kraken. I just got it. There's some really, really good ones out there. But the terminal is the only tool that's going to give you the full breadth of Git's abilities. It was made to be used on the command line. And if you are using these GUIs, the problem is that you're not really learning how Git works under the hood. They abstract things, right? So you're going to click on something to have this happen, which might actually involve two or three or more Git commands under the hood, but you're not aware of that necessarily. So then if something goes wrong, it's difficult, it's thorny to try to undo and find out what it is exactly that caused the problem. So personally, I would recommend if you really, really want to use a GUI for Git, get to know it first. Use it in the command line first. Know what you're doing, and then move to the GUI you'll probably use it for more kind of run-of-the-mill commands and go to the command line anyway for something that's more specific. But that's what I would recommend if you really want to use that. But otherwise, the command line is great. It can be your friend. All right, time is still good. Thank you. We are now at staging sites. So what's that? It's a clone of your live website sorry that lives on the same server. So it's not necessarily an exact clone. You might still be working on things there and that's fine. But the point is, is that it lives on the same server. It's the bridge between your development, which is your local development environment and your production site. Now I'm aware that there are some that will use more steps. There'll be local another development environment as staging and a production sometimes more. We're just gonna go with three for now. So you would traditionally go from local to staging to live. But sometimes it looks like local staging, local staging, local staging and then live. But generally that's the flow. All right, why? We already have a local development environment. Why do we need the staging thing? Well, it can test in the same environment as your production site. So even if you're using a very advanced, configurable local development environment where you can make a bunch of choices about which version of PHP you're using and all these different things to try to emulate the server as closely as possible and that's great. It's never gonna be as good as being on that server. Just to be 100% sure that what you're testing is actually what you're going to get with the production site. It's a space to get feedback. So your staging site can be accessed by your client or whoever else that you're working with. And so they can go there and click around, test things and then give you their opinion so that you can keep working on whatever it is that you're doing. You can password protect it. It's fairly easy to do. So if it's something that you only want to open up for your client at certain stages in your development process, that's totally fine. And you can just keep it behind a curtain for the rest of the time. It's also, it's another full backup, which is good to have. Okay, last step, recovery strategy. What if something still goes wrong? So it's unlikely we have a really, really solid workflow right now, but it can still totally happen. And if it does, you want to read it. You want to have a plan. You want to have a strategy in place to make it as effortless as possible to just restore the site and then find out what happened. So in order to do that, you're going to want to make sure that you have full regular backups. So by full, I mean the files and the database. Regular depends on you if you want to make them daily or less. I like the daily thing, but I know not everybody agrees. So you'll have to make that call according to your site. You want to automate the process so you don't want to have to babysit this because it will definitely slide at some point. So you just want to make sure that it is happening without you having to intervene and test it. For every environment that you have, test it at least once because it's not when the thing goes down that you want to kind of find out whether it's going to work. You want to know that it does. Some recommendations to consider with recovery strategies are managed hosting. So a lot of these will have recovery. Solutions for you in place. They'll make daily backups, for example, of all your environments. There are plugins that can do that for you too, such as UpDraftPlus, for example. Having a backup off server is also a good idea because even though they're very good nowadays, they're not infallible. So knowing that you've got something that doesn't depend on your server is a good strategy. And if your local development environment is up to date, that also acts as a backup, something else. Quick, quick word about hosting. Managed hosting and unmanaged hosting in both of these environments, you can use that workflow that we talked about. It's just that for manage is going to be a little bit easier. They already have a staging site out of the box for you in most cases. Git deploy is made easy for you in most cases. And they have a dependable recovery strategy. Whereas if you have an unmanaged hosting platform, you're going to want to have to do a lot of these things yourself. Do your own staging site. Subdomains are usually a good solution there. If you have SSH access, you can set up Git deploy. Otherwise, you can still use Git, but you'll need to use FTP to move over your files. You can do your own recovery strategy, again, probably with a plugin. So it's totally possible to do it then. It's just going to take a little bit of work. My drawing skills are abysmal, but we are now done. We have an awesome, very, very solid workflow. And if you do implement all of these steps, you're in great shape. There are a few things that we did not talk about. Task runners, right? Dependency manager, such as composer. Unit tests, preprocessors, CDCIs. So these are all things that in my mind belong in a more advanced workflow, but if you're interested and you want to fold some of these into your current workflow, there's no reason why you can't. The main takeaway then for today, workflows are incremental, okay? So you do not have to sit down and be like, I gotta do all these things now. You can just pick one thing that you're not doing and then give it some time so that you're comfortable with what it is, fold it into that workflow and test it out, get used to it, become comfortable with it. And once you're good there, incorporate one more thing. So it's something that you can do a bit at a time and then you win at life. Thank you. Thank you. Thank you. Thank you. Thank you. Okay, great. I guess I sped through that faster than I thought. Yeah. Any questions? Does anybody have any questions? Yes. What's your workflow involving changes made to the database? Because for files it's simple to me, but what happens when both your client works with his website and changes stuff like font colors or whatever and then you have made changes to the database too. Okay. So you're making changes on the live site both of you at the same time? No, I mean... Are we too close? Maybe. Let's say you make changes locally as a developer to the settings of the WordPress site. Okay, yes. Because you have changes to make. The website is already live. Okay. So you grab a copy, work locally. Yeah. And then you want to put the changes in place. Yes. But your client already made changes at the same time after you made your backup. Okay. And worked on the backup. So what's happening with the database? How do you, what's your workflow? So like that tiny window, there's still a possibility that you guys are making changes at the exact same time. Yeah, well it may be a few days. Yeah. So I'll always back up the live site first and I'll just, I'll compare and make sure that there haven't been any changes since I last made changes. I'll usually let my client know too, like, hey, I'm going to be doing this in the next 15 minutes. Could you please not touch the site? That's not like 100%. Great. But then I'll make a backup. I'll do staging. I'll make sure that it's okay. I'll have them check it. And then I'll move from staging to live. So generally by then, if there's anything awful, have found it out before it gets to live. But everything's been backed up first. So in case there's anything, we can just always go back. But usually, yeah. Usually that's pretty foolproof because you're making, I'm making sure that it's not different from the live site. And then I'm pushing to staging. The client and myself are looking at it on staging. When I get the approval, staging goes to live. Okay. Any other questions? Yep, over here. Thank you, Megan. Hello. Hi. It's a very solid work floor. Thanks for the, I just have one question you spoke about WPCLI. So what would change? Because now we have like a WordPress admin where I can, there's a page where I can see all the updates and then update everything. What's the different, why should I use like this one versus what WordPress admin is already giving me? So like in terms of workflow, like. That's a great question. I mean, it is just on its own. It's a lot faster than having to go into the admin and click, click, click, find what it is that you're looking for. You can find it much faster by just running the command in WPCLI. But there's also a lot that you can do with WPCLI, like just directly query the database to find some information about something specific. And these are things that you can't do unless you're like messing around PHP, my admin or whatever it is that you use. So, and I strongly recommend that you take a look at the documentation because every time I do, that was in my little like bubble on that page. And it was like kind of small to read. But every time I do, I find out this new amazing thing that WPCLI can do that I now fold into my workflow. But it can do a lot for you and it's not just about making things faster. It can do things that you wouldn't otherwise be able to do easily in the admin. Thank you. Any other questions? Well, thank you very much, everybody.


I don't think I had an expectation of how I would have felt, but I think that I guess I would say I feel pretty normal compared to other spring trainings that I've gone through in the past. I feel like I'm bouncing back well from the days that I pitch, you know, going through my routine the next four days and stuff like that. So, you know, obviously that's encouraging. I think that, you know, at some point in spring usually kind of go through that period where you kind of got to push through a couple, couple days of fatigue, things like that. And I'm sure that's coming. But so far, you know, everything feels good, both physically and recovery-wise. Thanks, Corey. Yep. You go next to Sweeney Murdy. Corey, what's it been like to compare notes with, with Garrett Cole, who likes to talk pitching a lot in the year? Yeah, it's been fun. Um, you know, whether it's him talking to me about my stuff or vice versa, um, I think there's been some good conversations there. Uh, you know, I think that it's, it's productive to talk to the people about stuff. I think that, you know, it can, it can open your eyes to other things or make you think about something in a different way than maybe you have previously. So, you know, I enjoy talking about pitching with other guys. And I think it's, um, you know, the more you can do it throughout the season and facing other teams, especially, you know, teams within the division that you face all the time, uh, learn from each other, pick each other's brains, uh, whether it's Garrett or whether there's any of the pitchers. Um, you know, I think it's, it's definitely a productive thing to do. Is there anything that, you know, in the last few weeks, as you've talked to, whether it's Matt Blake, Sam Breen, any of the guys who are on that side of it. Is there anything you learned or just kind of discovered that could maybe influence how you develop here? Or is it mostly just been about you getting back to what you were? Uh, I wouldn't say getting back to what I was, but I would say that maybe, um, you know, some things that I used to do. Kind of what, what allowed me to do them and what we can do now to kind of try to, excuse me, to try to, um, go down that road again. Uh, you know, not necessarily trying to recreate what I was however many years ago, but just trying to, you know, pitch shapes, things like that, separation between different pitches. Um, I think it's just, it's, it's interesting to me, you know, all the, all the different tools that they can show you, whether it's video or numbers or things like that. To look at all that stuff and kind of be able to compare and contrast, you know, if it was different, why was it different? This is why it was the same, that sort of stuff. And is there anything you're doing now that is in relation to keeping stamina from the long season after missing so much time? Or is it mostly just about getting ready for the start of the season and making sure your, your pitches are how many you could be for that part of the year? Yeah, I think honestly right now I'm just trying to get ready for the season.  Um, if, if that's something that they want to discuss, then I'm fine to discuss it. But in my mind right now, I'm just trying to go out there and, and prepare for the season and, you know, try to get myself as locked in as I can. Um, you know, whether it's executing pitches, whether it's the routine in between starts. Um, just trying to make it as normal as possible and not try to think about the fact that I haven't, you know, thrown a lot in the last couple of years.


So I'm Drew Gallatin. I do free BST performance work for our CDN servers at Netflix. And usually, I give talks about how to make things better.  Basically, how do I make things worse? So just to give you a bit of background, there's 800 gigs in the title. And we will get to 800 gigs. And we got to 800 gigs last fall, just after the last year of BSTCon. And that kind of motivated me to look back and see how much the various optimizations that we and the free BST community have made over the years have helped us. So I want to emphasize the free BST community, because most of the things I'm talking about today weren't done at Netflix. Most of them were done by the great people in the community that are some of which are here at this conference. And the other thing you're going to notice is that there's only one optimization between you and no chaos a lot of times.  We run the NGINX web server. And we have the easiest job in the world, because we just serve static files via send file. This is the hardware I'm going to be talking about for most of the talk. There's 800 gig stuff at the end. But it's a lot easier to abuse a 400 gig machine. There's a lot less of a blast radius if I crash something. So I do most of my experiments on a 400 gig machine, which is an older Rome-based AMD with sufficient networking and memory bandwidth. So I came up with a metric compared to these different optimizations, where basically I call it the gigabits per second per CPU metric. And that way we can compare optimizations where, say, the max bandwidth is a lot lower than it could be. So the first thing I'm going to talk about, and I'm going to start with the best setup, and I'm just going to keep making things worse. So the first thing I'm going to talk about is, like, this is how we actually run things, where we're running Nick Kernel TLS with send file, and all of our optimizations are enabled. And we get about 375 gigs at about 53% CPU. And with my new metric, we can call that 7.1.  And I'm going to show you flame graphs. So this is a flame graph. And the thing to take away from this is that what you're really looking at for slowness, if I can get to my laser pointer, which is stuck in my pocket, what you're really looking for is you can see these plateaus. Basically everywhere there's a plateau you're spending time and it's an opportunity to optimize something. And you see there's not a lot of great big plateaus in this. This is what we look like in our optimal setup. So first I'm going to talk about send file and Kernel TLS. So this is my favorite picture. So basically what this picture depicts is the data flow path. And in this case, it's the data flow path if we weren't using send file and Kernel TLS. So to serve 400 gigs, basically we have to read 50 gigabytes a second or 400 gigabytes a second from storage. And then we need to read it out of the kernel with a copy out so it gets read and written to user space. Then in user space, it gets read and written to encrypt it. And then to put it back in the kernel, it gets read and written again. And then the NIC sends it. And so that means we need basically one, two, three, four, five, six, seven, eight of these 50 gigabytes a second or 400 gigabytes a second. If you go back a few slides, like I said, the memory bandwidth on this box is about 150 gigabytes a second. So that's quite a difference. And so now first what I'm going to talk about is what is send file? So I'm going to be talking about how we get rid of these lines. So first, what's send file? Basically it's something that was come up with in the 90s or so where essentially you give the kernel some file descriptors, a file and a socket. And the kernel, you tell the kernel to send this much from this file descriptor on this socket. And the nice thing is no data goes to user space and the web server never has to even look at the data it's sending. So the first problem we ran into with this, like way back in the early 2000 aughts before we were doing, before we were even doing TLS, is when an Nginx worker is blocked, it can't serve as other requests. And so you can imagine if you're going to a spinning disk, it could be 10 milliseconds, 100 milliseconds before the disk B comes back.  And so there's a bunch of different solutions to prevent Nginx from blocking like this. One, we used the time was AIO. And since then, they've added thread pools. So in reaction to that, some folks from Nginx, FreeBSD, and Netflix came up with this thing called async send file, where essentially you don't need to sort of cover up the latency. What happens is basically, send file becomes fire and forget. And when you do send file, basically, the Nginx web server will allocate buffers, pages to store the data and attach them to the socket buffers and then initiate a disk read request. And so at the top of this diagram, we have a very primitive depiction of a socket buffer. And then TCPs is sort of the gateway here of what's good to send out to the internet. Here's the web server. And it's making a request to the disks to read some stuff. And it's allocating pages in the socket buffer. And now these pages have been allocated, but they're marked not ready. That's what those little stop signs mean. And once the disk read comes in, in the context of the interrupt handler, we go ahead and we mark all these pages in the socket buffers ready. And then finally, when they're all there, the TCP is allowed to send it, and then they go out in the layer. And the nice thing about this is that this is all done with existing kernel threads. Since the marking of ready and the triggering of the TCP stack is done out of the disk interrupt handler. And that was a really clever solution. And it's tremendously useful. So now, what's kernel TLS? Basically, I've talked about this at every conference for the last three or four years. We moved bulk crypto into the kernel. And the whole reason we did this is to preserve the send file pipeline. And so now, here's the original diagram we started with. And with send file and kernel TLS, we can get rid of a bunch of these arrows. And now things become a whole lot more possible. We still have four of the arrows, which means we still have 200 gigabytes a second of bandwidth we're using. So that means that we still can't get the full 400 gigs, but we're in a lot better shape than we used to be. Now, what's Nick kernel TLS? Basically, that allows us to remove these remaining two lines, because then Nick is all of a sudden going to do the crypto.  Now we're down to needing 100 gigabytes of bandwidth, which is less than 150. And so now, 400 gigs is possible on this system. And so now, here's where I start making things worse. So what happens if we disable kernel TLS and these things send file? I was expecting, basically, elevated CPU and memory bandwidth usage. But what I actually found was lock intention in AIO. So we got to about 40 gigs, and we were just stuck spinning on locks. So you can see the ideal performance metric of 7.1 gigabits per second per CPU down to way less than 1 with AIO. And the flame graph, I was talking about how plateaus are bad. And whenever you see these plateaus with lock delay, you know you've got lock intention problems. And you're spending all your time just spinning on locks. So let's move away from that. Let's not solve a problem with AIO, because we're not really going to use it. So let's try the Linux solution. The solution's more commonly found in Linux of using the async thread pools. So this did a little bit better. This got to be at 90 gigs or so. And this is more what I was expecting, because we have the additional data copies. And so I was expecting to see a lot of time spent access to memory. And that's what we see. So you notice that it's way better than AIO. We still know we're near as good as the optimal configuration. And now you see these plateaus are copy in, copy out. And there's a mem copy in NGINX. And it's doing the crypto. So there's a lot of these plateaus are just basically accessing memory. So that's kind of what I expected. Now for the next trick, we disable send file, but we still use kernel TLS. And this is somewhat surprising that it actually got worse. But the reason it got worse is because, at least my assumption of the reason it got worse is because when we're doing the crypto in user space rather than the kernel, the data that's being encrypted is hot in the cache from being read in from the kernel. And so this is, again, a little bit worse. And you can see still the same lots of plateaus. But you also see we're doing kernel TLS. That's what this KTLS stuff means. So then let's try to make things better. Let's disable send file, but we use kernel TLS. And so now we're back to about 95 gigs. And in addition to some lock intention, we're still accessing memory more than I thought we would. There's some extra mem copy in Nginx for SSL, which I don't understand and I haven't tried to hunt down. But if I was actually using this, I would hunt it down. But for the purpose of this talk, I just kind of recorded it. And so you can see it's a little bit better than it used to be, but still not great. So it just sort of goes to show you you can have all these great things like kernel TLS or Nick kernel TLS. And it won't help you at all if you don't use it right. And so again, lots of plateaus for memory copies and copy and copy out. And for some reason, Nginx is doing a huge mem copy, which I don't understand. Same thing I would hunt down if we were actually using this mode. Now I'm going to talk a little bit about how we run software kernel TLS. And we don't run software kernel TLS just out of the box from previous DB install. The ISALK mod. And what's ISAL? Basically, Intel wrote some really hand-coded assembly that can do a bunch of stuff like compression for storage. And it was originally intended for storage, but it also does bulk crypto. And the thing that we like about this is that it uses non-temporals. And so what that allows is it allows you to avoid this read-modify-write situation, where if you're trying to write, say, eight bytes to a 64-byte cache line, the way a CPU works is it will read the data from that cache line into memory, insert the eight bytes somewhere in the middle of the cache line, and then write it back. Well, the problem is that it's going to read all those things. It reads reading that for no reason, because we're really going to be replacing all those cache lines. And non-temporals allow us to just write it. And so that saves us a memory bandwidth. So you can see that this next one is before we discovered ISAL, and we're running with send file kernel TLS, but no ISAL. And you can see it's less than ideal, but we don't have the thing we need to compare it with, which is, if we do run with ISAL, so we go from about 180 to about 240 gigabits. And you can see in the flame graph, the difference is we have the AES, the GCN that's in the kernel. And there's some extra mem copies, again, which if we were using this, I would hunt down. But for now, I'm just recording them. But you see with ISAL, it's a different crypto routine, and there's no mem copies. So now we're kind of getting out of my comfort zone, and I'm going to talk about some virtual memory optimizations. And one of the most important things we, optimizations, at least to Netflix, to our workload, is the basically the UMA per CPU page cache. And this was, I think, originally conceived of at Netflix, at least, and it was probably independently conceived of at other places. Randall Stewart, and I think Scott Long, wrote this sort of pre-UMA hand-coded thing. I think it may partially have been up for my salon. I don't really remember. But the idea was there was a per CPU pool of free pages. And as you'll see later, one of the biggest sources of lock and tension we have is accessing the page queues. And so if you can bypass those queues and just not take any locks and just have a per CPU pool, then that should give you some better performance. More recently, it was up-streamed when it was, I think, Mark Johnson did the work. And it was basically managed by UMA, because UMA is really good at having per CPU caches. And the critical thing is it only works for free pages. It doesn't work for things in the inactive queue or the active queue. So it's getting limited to free stuff. So for the next experiment, I'm going to disable the page cache. Basically, everything is configured optimally with Nick, Colonel, TLS, Sunfile. All this stuff we were talking about in the previous section is enabled. And you wind up with some really bad performance, even worse than some of the things in the first section. And that's because you've got all this lock and tension when we are freeing pages and allocating them. So another incredibly useful optimization for Mark Johnson was VM batch queues. And basically, VM batch queues are essentially a way to relieve some of the lock and tension on all the different page queues. Instead of taking a lot, processing one page, adding it to a queue, and then releasing the lock, what happens is you store up in a per CPU area a number of pages. And then once you finally reach the maximum number of pages, you take the lock and then push them all as a group to the page queues. And so when you disable this, at least with our workload, you lose quite a bit of performance. So you go from 375 to about 280 or so with CPU maxed out. And things look better than before, but you still see lots of lock and tension when we are freeing things to the, I believe it's to the inactive queue here. You see the page queue batch. That's just because the way I disabled this was I hacked the size to be zero, so that it was always just submitting things as soon as it got them. This batch size was zero. Another really important optimization for us is SF and no cache. And that is a flag to send file. And the idea is that it causes send file to just directly free the page into the free list or the per CPU free pool rather than to try to go through the inactive queue. And we do this because we have a gauge of how popular various content is. And so if somebody is watching something that we don't think anybody else is going to watch that same chunk of that same video and that same encoding in the next couple minutes, when the client requests it, we have the web server market with SF and no cache. And that means, like I said, that we just throw it away. And so that relieves a lot of contention on the inactive queue by making better use of this per CPU free pool. And so if we disable that, again, we wreck performance. So we go down to 120 gigs to 55% CPU. One of the things that's interesting here is that in most cases, when we have bad performance, the CPU is maxed out. But it's not maxed out in this case. But the problem is that the clients are just running away in terror because Nginx is not answering their requests in a timely fashion. And so we get to about 120 gigs and the Nginx keeps getting delayed by lock and tension. So we keep going somewhere else. And here's kind of a running table of how we can break things. So with no page cache, it's really terrible. No batch queue. It's not great, but not horrible. And with no cache, it's pretty bad. And again, here with no cache, you'll end up seeing lots of, even with batch queues, you end up seeing lots of contention on the inactive queue. And you wind up spinning on locks again and spending all the kernel time on that. So I'm going to segue to a different architecture. So we've recently, in the last year or two, been playing with ARM64. And if you notice a lot of our flame graphs, we spend a lot of our time in page management. And one of the things that was just enabled in ARM64 in the last few months was 16k pages. I just want to thank Andrew Turner for that. And also Warner and Chuck and Kirk for figuring out some weird implications of 16k pages and the file systems and the disk drivers. And so by using 16k pages, we see a huge performance improvement on our ampere box. We go from about 345 gigs to about 368 gigs. And if you notice, the CPU is basically maxed and is down to having substantial idle time. And I didn't write out all the details of this machine, but this machine is essentially identical to the AMD, except it's an ampere, 80 core, 3 gigahertz, 128 gigs of RAM with sufficient TLS off load knicks to be able to push 400 gigs. And so just to compare the same metric I'm using, this is the first time we've seen it on ARM. We go from a little over 4 to around 5 when we enable 16k pages. And just to show you the difference in the flame graph, this is 4k pages. And you see time spent in the VM page functions. And we see some big plateaus, big plateaus. And then if you go to 16k pages, notice how everything got a little bit narrower. That's us recovering CPU time and kind of doing more work with less effort, which is always a good thing. All right, so now I'm going to talk about some network stack optimizations that are pretty important to us. And some of these are pretty basic things. Like one of the most basic things is LRO. So TCP is pretty expensive to run. So when a packet comes in off the wire, there's a lot of processing that needs to happen. And one of the things that I did the first LRO in FreeBSD, and it was very simplistic, it was later moved out of the driver I maintained and into the actual TCP stack. But basically, I did the initial LRO because I was working for a 10 gig NIC company. And with a standard 1500 byte MTU in 2006, you couldn't receive anywhere close to 10 gigs. And with LRO, basically it batches up packets as they're being received. So that if you're receiving, it works on eight connections. And it finds a packet for the first connection, appends it to a chain, but the next packet for that comes in, it gets appended to a chain. And you keep building these packets on these eight connections. When the chain gets to the maximum size, or you start seeing other connections, you flush the chains. So that enabled us on some horribly weak 2006 single core AMD box to get from, I think it was like 2.5 gigs a second to 10 gigs a second with some idle time. And there's been a lot of advancements made to it since then. But the whole advantage of it is, essentially, you avoid running the TCP stack eight times, 16 times, 40 times, whatever your maximum aggregation size is. And that just saves a lot of CPU time. And so if we disabled LRO, we see a little bit of a drop in performance. We see about 330 gigs at 65% CPU. And one of the things I noticed is that when we do this, the NIC starts to drop some packets. And so that impacts our health. I haven't talked about that. That's kind of a net look specific thing. And basically, health is kind of a signal to the load generator that we're not healthy, and it should send traffic elsewhere. And so lots of things will impact health. High CPU usage, dropping packets, high disk latency, high engine X response latency, whatever. But basically, by dropping health, it's telling clients to go somewhere else. And so with LRO disabled, we reach a max of about 330 gigs before the clients start going somewhere else. And you can see that's probably one of the optimizations that has the least effect because it's pretty much the highest bar of anything we've seen so far. And here is a flame graph. And it shows spending a lot of the time in the TCP stack out of the network interrupt handler. And so the next thing we're going to talk about is RSS accelerated LRO. And Hans Peter came up with that from LNOTS. And if you remember, I was talking about LRO. It's got a limited number of connections it can process. It keeps track of like eight connections at once. Well, in a workload like Netflix's, we might have a NIC with, say, 32 queues. And we might have, say, 100,000 connections. And so that's roughly 3,000 connections per queue. And so the odds that a whole bunch of packets on the same connection are going to arrive close enough together so that they can be processed altogether is pretty minuscule. So Hans came up with this super clever idea where what you do is you gather all the packets up in a big bunch before you submit them to LRO. And then before you submit them, you sort them by the RSS hash that came off of the network card and their arrival time. And so what happens is you transform this mishmash of packets that are all from different connections into runs of packets from the same connection. And when you do that, LRO can actually process them. And essentially, if we disable this, I mean, the number's a little bit different. We get to 70% CPU. And here's one of the reasons why I've got this metric. Because you can see that this is essentially if we disable all of LRO, we're here. And if we disable just the RSS assist, that makes LRO pretty ineffective. And it's almost the same as disabling. You can see this big gap between these two, which is bigger than the bandwidth gap would indicate. And again, we see lots of activity and the TCP stack coming out of the network interrupt handler. And another thing which is super important is TSO. And so the principle is the same. The difference is that this requires cooperation from the network card. And pretty much all network cards, since the early 2000s have supported this, even the cheap things. And so the idea is that like LRO, we just reduce the number of trips to the network stack. And so the TCP stack decides he's got the space to send, say, 64K to a client rather than looping through and sending one packet, 1500 bytes, one packet, 1500 bytes, 40 sometimes. It just sends a huge, gigantic 64K packet down through the lower parts of the network stack to the network driver. And then the network card itself is responsible for replicating the IP header and replicating the TCP headers and splitting that 64K TCP chunk up into a bunch of valid 1500 byte Ethernet frames. And so on the host side, that avoids a lot of trips through the network stack and also avoids allocating a lot of unbuff headers and a lot of, you know, queue manipulations in the socket buffers. And so if we disable TSO, we get to about 180 gigs at 85% CPU. And this number took a little while to get because I needed to disable interrupt coalescing just because we're putting so much more pressure on the transmit descriptor rings. And as you can see, there's a lot more time spent in, oh, I actually, oh no, there it is. You can see a lot more time spent in the network driver. I don't think we've ever seen Ether output and the Mellanox transmitter chains showing up this prominently. That's just because, like I said, we're calling them probably 10, 20 times more frequently. Because you don't always have 64K things to send and the network guys hate it actually, actually sending 64K at once. They like to send at most, say, eight because they worry about buffers overflowing on the client side. So now let's disable both TSO and LRO to reduce all of our batching to zero. And that takes it down a little bit. We get down to 170 gigs and again the same IRQ coalescing constraint. And so this is no TSO and this is no TSO and LRO. And you can again see lots of network stack stuff, lots of prominent Ether output and Mellanox driver calls. And everything is much worse just because we're running things so much more often doing so much more work for no reason. Now I'm gonna get to the interesting part of the talk, the headline part of the talk. So last year I really wanted to give some results on an 800 gig box and I couldn't because it got stuck in some kind of shipping snafu. But that was a year ago and so I've had a year to play with the thing. It's a Dell R7525 with two Milan 64 core CPUs. And the important thing is that it has three links between the sockets and that will become important later. It has half a terabyte of RAM and enough Mellanox Kinect X6 Nix to drive 800 gigs and enough storage to feed it. So the first time we fired the thing up we got 420 gigabits out of it which is not very impressive. And if you remember from my Numa talks from a few years ago there's a bunch of different ways you can run I can configure Numa machines to run the Netflix workload. And we're running what I'm calling the network siloing mode where basically connections are hashed by the incoming Nix to a particular Numa node and we try to do all that work local to the node that we can. So during this test the CPUs were mostly idle but we were dropping packets like crazy. And so what AMD guessed was that the links between the CPUs were downlinking to buy two just to save power. It took the CPU load as a signal for it should be saving power. And so you can force them to not downclock or downtrain. And by doing that it's called dynamic link with management. If you turn that off we get up to 500 gigs. And that's again network siloing mode. And the important thing to know here is that means that the data from the NVMe drives is getting directly DNAed from the Numa node where the NVMe drives are to a different Numa node where the Nix are. And one thing we noticed using AMD's profiling tools which are available on their website by the way so anybody can use them is that the XGMI link was very, very unevenly utilized. There were 15 gigabytes a second on one link and then 4 gigabytes a second on another and 2 gigabytes a second on a third. And so what AMD eventually told us was that when you are doing DMA across the Numa bus which we're doing because we're doing the network siloing then the XGMI link that's chosen based on the location on the originating CPU socket of the device doing the DMA. And the problem is that there's essentially four I.O. chips in a I.O. quadrants in an AMD CPU and it hashes by like what quadrant the I.O. device is in. And unfortunately with this Dell things are very uneven and most of the NBME drives are in the same I.O. quadrant which is why we're seeing such even use of the XGMI links. And so one of the things we can do to improve this is remember how I said we're DMA'ing to the Nix Numa node. If we decided to DMA to the NBME drives Numa node then this goes away and then Nix are much more evenly distributed and so when we're hashing based on the Nix location then the XGMI links are going to be more evenly utilized. So that's what I did. I flipped things so that now we're doing DMA directly to the node that NBME is on. And that helped things a little bit. That got us up to about 670 gigs. And like I hope the XGMI was much more even at 10, 10 and 7. Now doing this is problematic for a lot of reasons and I'm going to make things more problematic in a second and then I'm going to go over all the problems. So now at this point I decided to turn on full dis-centric styling which if you saw my talk from a couple of years ago it basically means that in addition to DMA'ing stuff to the Numa node where the Nix live I actually migrate the TCP connections to that node. So we essentially tear down KTLS on the original node and then we move the TLS connection to software and then we move the TLS connection to hardware on the new node. And so now no matter what node the TCP packets are coming in into they're always going to go out of the same node that has the NBME storage for the connection. So the idea is that there'll be like zero Numa crossings for bulk data which will help with this XGMI stuff. Now there's a lot of problems with doing it this way which is wondering why I would never actually run it this way in production because when you're running the network siloing mode the traffic is basically a distributed or sharded by the hash that's done by LACP to the NIC ports and that's a hash over the IP and port ranges and so that's a hash over this huge hundreds of thousands or millions of possible things and so the hashes end up being really, really evenly distributed so your load is very, very evenly distributed between Numa nodes but all of a sudden if you do disk centric siloing it depends on where your content is and that's you're choosing between say 8 or 16 NBME drives and 8 or 16 is a lot less entropy than hundreds of thousands or millions and so if you have a title that's really popular then all of the traffic for that is going to end up on one Numa node if you're doing disk centric siloing but if you're doing network centric siloing it would be distributed all over even across the Numa nodes so the problem with this is if you have a really popular title or you have just a hot disk then that will translate into a hot Numa node and you'll have maybe one Numa node which is reaching bandwidth limits on the NICs and another Numa node which is just loathing along and I mentioned we move TLS sessions that's kind of expensive, that's something I'd rather not do and the other problem is that once we do this affinities for a lot of things are now wrong especially we have TCP acts coming in on the wrong Numa node it's getting processed across the Numa bus which adds latency to the processing the TCP pacing is being done in the wrong node I mean that's something I could probably fix if I had to but I think I already talked about the uneven sharding but we got some good results from it just for the science experiment so we can get to 731 gigabits in this configuration and that's because doing the disk centric siloing almost all the load off the XGMI links except for just CPU cross traffic and at this point we're limited by network output drops not CPU and the cause of the drops is basically due to those disk centric siloing problems I was mentioning in some cases one node is just being pushed too hard with respect to how much memory is being used the other thing I didn't mention is that if there's a hot node our software which decides what content is popular isn't Numa aware and so it will make this cash or no cash decision just globally based on how much memory is in the system not on how much memory is on each Numa node you wind up with the page demon kind of going crazy on one node because it's not enough stuff is being directly freed and the other problem is like I said with hot content you wind up with one node doing more and one node doing less so all the nicks on one node in this experiment were doing 94 gigs and all the nicks on the other node were doing 89 gigs and so I think that's about it and I have time for questions now I have way more time than I thought I think I must have rushed through this because I was worried about the whole thing fitting yes I can repeat the question if I can hear you that's actually one of the reasons why I give this talk is because you know quick is essentially let me go back to the slides quick is essentially I probably should have done an explicit thing so let's see where's my quick right now quick is basically the dumb way because there's no send file for it everything's happening in user space but we're still reading things through the kernel and so quick is essentially the TLS in user space case and so that destroys the performance also and the other bad thing about quick is that it's the combination of that with from the networking section with TSO and LRO disabled now there's some hardware vendors working on quick offload I was actually talking to a small company which I think last time I talked to them was last winter and I think they've actually gone bankrupt out of business unfortunately but they actually had a quick offload solution which did essentially kernel TLS and like send file for quick but until you know solutions like that get popular I have really huge concerns about quick for performance reasons is there something preventing you from duplicating hot data to other numous zones I'm sorry I missed the critical part of the question is there something preventing me from what duplicating data to other numous zones that's hot data hot content can you take your mask off just for a second is there something preventing you from copying hot content to other numous zones no there's not it's basically development work and the way we view these NUMA platforms now is essentially as research vehicles to see what problems we're going to hit internally with our infrastructure and with our other software when we get to these bandwidths so for example this machine is essentially a prototype of a future looking machine based around the next generation DDR5 which more than doubles the memory bandwidth and PCA Express Gen 5 and so we were hoping that I would have results from a machine like that to share because I really didn't want to talk about NUMA and unfortunately a lot of this became about NUMA but we just don't have those machines yet we don't have the PCA Express Gen 5 you know 400 gig NICs yet we don't have these platforms based on DDR5 yet but the interesting thing that we found with this science experiment is what was really killing us was two things was logging and with some local stuff we have to do socket buffer resource management so we sort of treat NGINX as a one NGINX worker and we give him a quota of how much socket buffers he can use and that code was kind of getting a little bit in the way along with some just the way we do logging what I wanted to ask is so you said that with the disk space disk based siloing you have the problem that the incoming connection might be on the wrong NIC and you have costs with moving it and probably stuff going on going out on the wrong NIC so have you thought about exposing that with multiple IPs and making the network aware with BGP or something else aware on which NIC it should be interrogated for specific something? That's one solution but the IPv4 address space is limited and these are all internet facing addresses so we'd rather not burn more IP addresses and I mean there's a lot of solutions that we could use if we really had to and I said this is just a our research vehicle is not designed to pre-production and in fact another thing we could do is that after my first talk in 2019 about NUMA somebody came up to me and made me aware that there were extensions to LACP where you could notify the router that you wanted to move the connection we don't have it implemented in all routers firmware at least as far as our network team was aware so it's not a solution I could use today but there are a lot of things that could be done to make this centric siloing real given its current status as a research vehicle it's just not worth the extra effort that's one of the reasons why I've never upstreamed any of this stuff because I upstreamed some research stuff in 1999 which for zero copy sockets which ended up haunting me for years so no more research stuff and upstream anything else? final chance alright thank you


Let us continue with our theme of unpacking ethnography. In this section, we would like to discuss its other major component, the concepts that form the fundamental structure of ethnographic research. Along with the process, the method is composed of certain principles that guide its process and give ethnography its particular way of seeing. These ideas and concepts define how and what we see when we set out on our investigation and how we understand or interpret our findings. The approach by which we arrive at our data is as important as the data itself. These principles are the core characteristics or attributes of the ethnographic method. Let us understand these concepts in greater detail. As we have seen so far, the experiences and learnings that form the raw material of our research are gathered in the field. That is, in the context where the phenomena we are studying occurs naturally or where they commonly occur. This makes ethnography a naturalistic form of research. Moreover, it has to be conducted in the time in which the phenomena is taking place. It cannot be a study of something from the past or a forecasting of the future. It is in the here and now, therefore, synchronic, meaning a research situated in the present. This is why it emphasizes being present at the immediate site and at the time of the occurrence. When we experience events in the context as they occur, we also understand the various perspectives of others who are involved in it and how they experience the event. To be present is not simply to just turn up on the field. An important characteristic of ethnographic practice is to pay attention to the most minute and mundane details of our participants' lives. Here let us turn to someone who is often credited with laying down some of the founding principles of ethnographic practice. Malinovsky was an Austrian anthropologist who did some of his most significant work in the Trobrian Islands off the coast of New Guinea in the early 1900s. He put forth the idea that the work of the ethnographer is to gather and make sense of what he called the imponderblia of actual life. Let us hear it from the man himself. The imponderblia of everyday life is a series of phenomena of great importance which cannot possibly be recorded by questioning or computing documents but have to be observed in their full actuality. And to illustrate what he means, here is an example of his own recordings. I would get out from under my mosquito net to find around me the village life beginning to stir, or the people well advanced in their working day according to the hour and also to the season, for they get up and begin their labours early or late as work presses. As I went on my morning walk through the village, I could see intimate details of family life, of toilet, cooking, taking of meals. I could see the arrangements for the day's work, people starting on their errands or groups of men and women busy at some manufacturing tasks. Quarrels, jokes, family scenes, events usually trivial, sometimes traumatic but always significant. Form the atmosphere of my daily life as well as of theirs. But why this insistence on imponderblia? Why is it so central to our work? Let us see if you can find the answer to that in our discussion on the contextual and descriptive nature of ethnography. We have spoken about how ethnography is naturalistic and synchronic and can only be gathered if we are present in the setting and at the time when the phenomena occurs. This means that ethnography is inextricably bound to context. It is therefore a contextual form of research. As ethnographers, we do not simply just list down the events that we observe. Our task is to analyze our observations and derive meaning from them. This can only happen if we expand our understanding of the context. Context here does not only refer to what we see or sense as the immediate physical settings of the phenomena. It is also made up of intangible factors such as social hierarchies and dynamics, beliefs and behaviors. The history of a place or a group forms part of their context and so does their material world such as what technologies do they use, what kind of trades are they involved in. For a community of porters, the context is composed of their environment where and how they live. It consists of the craft they practice and how they make a living from it.  It includes their position in the social structure of their village or town, their class and caste position for instance. And it includes the social structure of the community itself, what the gender relations are, what the family structure is like and so on. To understand this expanded idea of context a bit more, let us bring in another key figure in ethnography, Clifford Geertz. Geertz articulated and even redefined much of ethnography and its way of seeing in the 1970s and 1980s. Geertz insisted that in understanding the context of a community, ethnographers must look at the meanings that underlie their everyday practices. These meanings, he said, signify how people experience and perceive their world. Geertz referred to the collection or the intricate web of such meanings as culture. Culture has been defined in many different ways. So the ways in which we dress, our food habits, our ways of interacting with others, all these small ways of behaving constitute the culture that we are a part of. But more significantly, the ways in which we think, feel and form beliefs, these factors that define our ways of dressing, eating and so on, are what Geertz terms culture. Take for example, our collective history of being colonized. That forms a part of our culture. In case of a smaller group like an educational institute, the collective importance given to punctuality or the comfort or cordiality that students feel in approaching their teachers, that is part of culture too. It is this culture that translates into the observable, perceivable behavior such as the conversations that take place between students and teachers. Geertz suggests that in doing ethnography, we need to understand the underlying culture that gives rise to observable behavior. This is important, he says, because human behavior can be seen as a set of actions, each action symbolizing some underlying meaning. This meaning is based on how we see as a social group. Collectively understand that action, what it symbolizes, what associations we build around it. We observe behaviors and interactions and interrogate them for what they are, for what they reveal, the social structure, the symbolism, the associations connected to them. These form the context in which the action is to be understood. By understanding these, we try to understand the meaning of the observed behaviors. The context is made of imponderblia that makes up our everyday worlds. The underlying beliefs, norms and hierarchies that form the social structure and the web of meanings, which is culture. Let us take an example of a behavior and reflect on how we may understand its meaning. Consider the infamous Indian head wobble. The action itself can mean multiple things. It sometimes means yes, sometimes no, sometimes maybe. Sometimes it's just a way of acknowledging another's presence. Or what they said. And ever so often, it is just an absent-minded, hmm. As ethnographers, though, how do we decide for its meaning? What all do we need to understand in order to make sense of a head wobble exchanged between two persons? Write down all the factors that you can think of. In order to understand what a particular head wobble means, we need to know the various meanings that the wobble has in that culture. We need to also know the immediate context in which that particular wobble took place. It's imponderblia. And we might also need to understand something of the persons between whom the wobble was exchanged. All of these small and large immediate and invisible factors form the context of the wobble and define its meaning. In our everyday lives, we experience many such small behaviors and gestures. We understand their meaning without a thought because we belong to this culture and speak the same cultural language. We are embedded in the context. As ethnographers, however, we must step outside of what we know and explore how these meanings are constructed.