n page the conversion is ascii and ebsidic and As it turns out the name DD does not stand for disk duplicator or anything like that the original name comes from JCL in IBM language on mainframes and it's for data definitions in JCL because the IBM mainframe guys who were collaborating with the Unix guys needed to trade tapes and stuff sometimes and in order to Be compatible with an IBM system you needed to use DD to convert the text between different character sets and then write out an ebsidic tape for your IBM using colleagues and Obviously we don't use a lot of ebsidic it still exists. It's probably not super important day-to-day to anybody in this room The thing I want to flag about this is that there's this legacy utility that had a very specific use case in the 1970s That nobody intended to use for backing up USB flash drives because USB wouldn't exist for decades That we still use because it turned out to be kind of useful for something. It wasn't intended to And so there's this enormous amount of legacy stuff with character set conversion in this utility that we never used for character set conversion that is still maintained and could still go wrong and stuff and we're going to revisit how deep a rabbit hole of Weird legacy backwards compatibility we can find in the modern stack So I just sort of wanted to flag that as part of what is going to be the theme cat Doesn't just concatenate files whatever it claims It also has a bunch of conversion and processing stuff the most interesting thing that I found in the man page when I Actually sat down and read the man page for cat if that it'll do line numbers Which most people Never bothered to read the man page for cat so they forget that it has all of these extra flags But that guy in your code review who says cat is only for concatenating multiple files Doesn't use it for line numbers, which is a totally documented valid feature And you should give them a little guff for that during your next code review I All right I also found article on history of cat that goes into a lot more details and then the useless use of cat file my cat an Original cat written in assembly because all of the original utilities were written in assembly before the C compiler was stable And they had to do ports So the next utility that I wanted to work on was LS pretty simple one of the inspirations for why does this take so long? Getting a little more complex than cat. I didn't Focus on implementing every last feature of LS I wanted to start with just listing files and then the ability to maybe do a long listing where you have to make it actually Stat the file to see features about it rather than just looking at the name I didn't implement all the sort features and stuff like that LS is Thousands of lines long when you actually pick it apart and it's very complicated and doing all of it is surprisingly difficult but All right, so the W utils philosophy is to just by start by doing the dumbest simplest thing and then see what happens because like I said very lazy so I'm using C++ you can use boost or in C++ 17 There's library from boost for file system access that is part of the core language now Almost they change some of the spellings on things just to make it difficult But it looks like this in either boost or C++ 17 Give me a directory iterator for a path then iterate over the directory and I want to print it out It's pretty straightforward this is For the default invocation of LS basically equivalent to thousands of lines of C in the new core utils version One thing I will flag C out has a really bad reputation for performance But I didn't bother to use printf turns out the main reason that see out has a really bad reputation for performance is because it has Legacy interlocks for compatibility with printfs because they each hold their own separate buffers if you disable the Backwards compatibility with C so that printf and see out don't intermingle their output by accident Turning off that locking makes the performance just as good as printing out in C. It's fine Again, there's this legacy that is there by default that if you throw it away then everything just suddenly starts working faster And it's great So next step benchmark it figure out like how much slower is this how? Because like obviously going through boost there's this abstraction layer on top of posix LS is talking directly to low-level system APIs So there's going to be some cost using this higher-level API So timing a directory that has a million files in it just to see how long it takes to run LS new LS is Invoked as Regular LS without LS dash dash color equals auto that slows it down a lot because it has to stat every file So if you want, you know turbo charge your LS without having to build your own just an alias LS in the shell And it goes a lot faster that took four seconds and then mine using this complicated abstraction layer that's in C++ object oriented simpler API, but much bigger 1.4 seconds it was much much faster was almost four times as fast by doing the dumbest simplest thing So I think well I was disappointing. I Thought I was gonna have to learn a lot about LS in order to beat it But that was when I sort of started to learn that a lot of this legacy stuff is not optimal on modern systems and if you just Start by doing The simplest dumbest thing in a modern system that is designed for modern systems It just goes faster pretty much for free and you get these negative cost abstraction layers that yeah They're big and they're a bunch of code and it's a pain to link boost your application You have to add include path, but the benefits versus writing 5,000 lines of C from using this abstraction layer are that You wind up with a lot less code you have a lot less maintenance cost You have a lot fewer places for bugs. Oh, and it just happens to go four times as fast for free. So great So next step make it a little more complicated I want to add color output to my LS. This seems like you know at that point I'm in the big leagues for LS implementations And there's a couple of ways to do this. You can blast out antsy characters. There's these control codes You can do you can check if is a TTY standard out because you only want to do it if it's interactive not after piping to something But like that won't work on Windows You can check the term environment variable Which will say which terminal you're using to make some judgments about how to do color and whether or not to do color And then like there's an official library the term cap library and utility called t-put that uses it You can use that and then the requirements that I wanted to do was to do color in a correct portable way My false works fine on Windows. There's not a lot of complexity there So if I can make a totally portable thing that is using just core language features and C++ Then ideally it would be a portable thing that'll work on any OS even if I use something new besides Linux some other Implementation of POSIX so blasting out antsy the core problem with this is little bobby tables You're doing in band signaling a Terminal intermixes the actual content with the color stuff and that's kind of grody Using something like HTML. It's separated out into like CSS where you can make judgments about colors That's not just in band. There is a standard. Unfortunately. There are several standards Another xkcd reference and the way that color handling is defined it changes between all the revisions of the standard because as Systems evolved since 1979. I think the first antsy color standard was Systems became able to display more than eight colors, so they had to make some changes And even if there is a standard a lot of old terminals had color standards that were common enough to be de facto Standards that aren't compatible with the antsy standards Windows support Windows 10 theoretically does support antsy colors But only in like the latest revision of windows and if the terminal is running in a certain way and like not PowerShell Which does colors differently? That's not great terminal emulator support even on Linux is super inconsistent and You can leave the wrong colors because there's this stateful thing where you print out a control code saying become red and the text printed from that point on is red and then your application ends and the terminal doesn't know that your application ended because it just sees a stream of incoming text and it was never told return to the default color so then Everything printed after that just remains red because there is this implicit state it is only stored in the terminal or terminal emulator in the modern world and You cannot query that state to find out what the color is supposed to be or what it has been changed to at least not in a consistent way so Maybe blast out antsy if you check is a tty you Can avoid breaking some things and then like tty's really don't exist on windows only kind of sort of pty's kind of do and then What is a tty? It turns out that is a tty. It's a teletype Checking the term variable This is something a lot of applications do and it's why so many command line utilities break weirdly when you use a different terminal emulator And some like oh, it doesn't give you color out. It gives me color output on my system Oh, are you using a slightly different revision? Oh, you set it to X term to 56 color instead of X term color Linux rather than X term things like that There is also a separate color term variable separate from the term variable that you can use to decide if you want to do color so You cannot make rational judgments about how to output color based on those variables it is Not merely a very difficult thing It is impossible partly because modern terminals like console or GNOME terminal that you're likely to be using Just claim to be X term because everybody just gave up on using newer versions of the term variable because it didn't work anyway So why bother? continuing with it I've been writing this in C++ like for file system access there was this convenient library boost that did file system stuff And I could write like two lines of code. It worked great Is there a great library for C++ that we'll just sort of handle this and who cares how it works? No No, not at all There's some that claim to there's some that kind of seem to under some circumstances One of the ones that I kind of like the API for that I found on github that Shall rename remain blame this blameless Use term cap, but you only looked in one directory, but not all the directories. So it understood X term dash color as your term environment variable But if your environment variable was set to X term or X term dash 256 color it would just fail horribly and think it wasn't allowed to do color output Despite the fact that X term 256 color Loves color output even more than X term color because there's this legacy of files being divided in different directories And if you try and roll your own thing to Parse all of it out you're gonna screw it up because it's massively complicated So you use term cap It's not portable. It's not something that really exists on Windows, but it exists on Linux. It's pretty stable. It's mature You can use the utility called t-put that uses this term cap library It's a demonstration of the t-put set f2 that makes green text set f5 That makes magenta text and then set f3 according to the documentation outputs yellow text That's not yellow though. So what is documented has very little relationship to what actually happens Here's a Wikipedia guide to color keys in certain ANSI APIs and what they tend to come out as with certain things so it's a Real crap shoot like what color you're actually going to output when you try and output a particular color because This table goes off the screen further That's not even a full screenshot and that just says Ubuntu is one of the columns There's a lot of terminal emulators on Ubuntu and I don't even know which one they're claiming that is for so I went to this term cap database That's the thing was what's that? Thousands of files referencing different terminals That you will never use that you've never seen that you will never encounter none of your users will ever sshd your machines using I found Almost 3,000 files in user share term info But remember how I said there was that library that failed when you set it to X term That big path with thousands of files isn't big enough to cover X term which is a reasonably common version because everything claims to be an X term even if it's not tons of legacy So one of the things that you can do using the information from this term cap database You can use stty which is a related thing for setting your teletype configuration settings You can put GNOME terminal or console into uppercase only mode as a legacy compatibility mode to terminals that did not support lowercase So that works fine color doesn't work yet Legacy hardware that you won't use and I sort of fell down a rabbit hole trying to understand how far does this legacy rabbit hole go? This is a photo of Buffalo Bill who became famous during the 1800s for his Wild West show Okay, don't worry. There is not an entry in the term cap database that goes back to the 1800s But it goes back pretty far It goes back to 1963 Now those of you who are Unix historians will note that zero Started in 1970 you can only represent 1963 with a signed time value because this is from before Unix existed and obviously decades before Linux existed But when Unix was first being developed The guys who were working on it got the crappy old terminals that were hand-me-downs from the more important work So they got the very first ASCII terminals that only supported uppercase Consequently like Unix v6 boots by default into only uppercase mode and in gnome terminal You can still flip it to uppercase only mode for compatibility with a 1963 terminal If you set term equals TTY 33 that still works the term cap database will look up the valid control codes for using it And if you had this terminal connected to a machine and SSH to any Linux box in the cloud that you provisioned five minutes ago It will still work and have valid control codes for talking to this TTY model 33 the ASCII standard originally created in 1963 did not get uppercase and lowercase differentiated until 1968 so This was the beta version of ASCII that we still have backwards compatible support for and this is why color is hard So the 1968 version of ASCII it got adopted as Standards get lots of names. It's got a CCIT CCIT version number and things like that one of the names for the spec is ITA 5 That was when they added the lowercase to flip your gnome terminal set STTYL case and Just to point this out If you run LS you get this mojibake gibberish stuff with characters it doesn't recognize That's not a problem with uppercase versus lowercase That's a problem with color if you unalias LS so it's not trying to output color the uppercase output works fine So in 2019 we have perfect support for uppercase only terminals But outputting color will put the gibberish that's all misaligned and has weird characters in it So that's neat and a hundred percent not useful to anybody So I mentioned that it was ITA 5 is the adopted version of ASCII there were ITA versions 1 through 4 that came before ASCII ASCII is a 7-bit code Which causes all sorts of problems with needing to like you you encode to go through Not 8-bit clean systems with old protocols It is a 7-bit code because the code that preceded it the old versions were 5-bit codes and That terminal there is a World War 2 era system using ITA 2 Which is a precursor to ASCII that evolved in a very linear line to ASCII It is made by the teletype corporation the same as that TTY 33 that we maintain backwards compatibility for in Linux to this day It ran at 45.5 BOD Which is impressively fast for for World War 2 given that you know a home user would still be using 300 BOD modem in the 80s and I found a video on YouTube where somebody had made a USB adapter to go from USB to a World War 2 era TTY 19 Because the serial protocol is so consistent between these really early systems the serial ports used for modern computers Were directly inspired by these clocked serial protocols that used a Single-wire for data exactly the same as RS232 This dates back to before we really talked about these things in binary terms so instead of one you have mark as a signal because You didn't plug it into a computer and you weren't going to do math with it yet because nobody had a math a computer that worked with the terminal interactively and With passive components you don't need a microcontroller or anything to do the character set conversion If you set your Linux box to uppercase only mode you can build a very dumb adapter Does not require a processor does not require any CPU power to convert between the character sets and still use this as a terminal on a modern system This predates the information age this predates stored program computing this predates Shannon's limits for signal processing It still works fine. I've seen it happen on YouTube ITA one is the ultimate original version of this ITA code series of character sets that we can trace ASCII back to with some compatibility carriage return in line feed only originated in 1901 but The code was originally developed in 1870 by man named Monsieur Baudot. He's French And his name is where we get the word Baud from because he invented this clocked serial communication almost exactly to RS232 that we would use today and the number of bits per second that it would use Was something that he invented prior systems didn't have that kind of consistent clock the way we would use today on a serial communication system Fun facts ITA one had a bell symbol. So if you've ever wondered gee, why does ASCII have letter ding? It's because on the original international telegraph alphabet They had a ding that you could ring a bell so a guy could come into the room to see a message get printed on a paper tape By one of the really early typing systems. It also has a null character You'll note the keyboard the keys are helpfully labeled in the super logical order four five one two three Because you use the three keys there with your right hand two keys with your left hand and then you just had to cord binary you had to memorize the binary code for everything and just know the combinations and Null was if you didn't have any keys pressed The original interpretation null was I'm not sending anything if you didn't have any keys pressed It wouldn't send any high voltage signals down the line and it would just be waiting at low voltage That's not something that is a useful representation in ASCII But the null character in ASCII is still all zeros so that you can send nothing down a serial line and Just leave it as low voltage That's where this comes from And because you could input totally arbitrary binary sequences on the original version of the 1870s keyboard While the character set didn't officially define carriage return or new line or anything like that You could enter the codes from the later 1901 revision of the spec on the 1870 keyboard So if you connected this through an adapter that did voltage conversion and character set conversion You could enter a command like LS and then enter the code for new line And this would still work with a modern System if you set TTY 33 as your term variable while you were SSHing So Yeah, I just took you back to 1870 for What you can get to work with term cap more or less with some passive electronics But not needing to write any drivers use a microcontroller or do any really exotic things because of the backwards compatibility support that's still there and This picture of Buffalo Bill that I found on Wikipedia is undated It's fairly clear so it's plausible that it was from the 20th century and this picture of Buffalo Bill is actually Decades newer than the oldest thing that you could use as a terminal on a modern Linux box This is why my LS implementation does not support color It's impossible Even if you use LS with weird modes like I showed you just get weird gibberish in your terminal There is no good way to do it. We have failed as a people With centuries to work on the problem literally it goes back to the 1800s all right, so That is my whirlwind tour of LS one of the other utilities that I wanted to work on was CP All right, you want to copy a file? That's a pretty simple thing to do so Standard file system is the API that I use for my LS implementation Works great. I tested that there is Method file system copy underscore file Works great totally follows the WU Tills philosophy of do the simplest thing because this is just Weekend hobby project for fun But I already told you just use standard file system or boost file system to do things with file system If you're gonna do it so repeating that for CP is like it's not fun. It's cheating so There is a lower-level interface that is only Linux specific and it's a lot more interesting called send file which The C++ standard file system method and send file which is a Linux specific API invert the order of source and dest Because in all the years that we've had units We've never agreed which order of the source and the destination go in when you're copying files depending on what you're doing So that's fun Also, just kind of a random fact one of the things that you send is the size of the file to that function The size of the file is an off T type rather than a size T It's not an offset So everything is backwards and all the types are wrong But it goes fast and it's an interesting thing and it gives me an excuse to kind of talk about some storage stuff So if you time my really simple versions that are each one line long With send file you have to open the file yourself and then send file handles to the send file API With the standard file system API you just send file names. So it's a little simpler. There's a couple extra lines in the Send file version that are trimmed from the slide Second and a half to copy a gigabyte file with new CP and then Half or two thirds 0.8 seconds using the really simple dumb ways that are not thousands of lines long And fun work. I noticed during the testing new CP takes a lot longer if the file already exists So the less important the work is the longer it takes because you've already copied the file And you're doing it over and over while you're testing your thing to prepare for a talk It just starts slowing down if you don't delete it yourself between the runs That I don't 100% understand why but new CP has a lot of different modes for compatibility with different scenarios And it will do things in extensive you have sparse files and things and the existence of the file that you're copying over It's not just the amount of time to remove the file and then copy over it It over thinks it and makes bad choices So how does storage work what's going on with this copy? You've seen that I Am in the software life cycling community where I have an interest in legacy. I have an interest in performance I'm going to give you a complete history of storage that is a hundred percent as in-depth as my history of terminals There is rust that gets dizzy and not This is my complete history of storage devices and as a first-order approximation when you're trying to write performant code This is 99.99% of everything that you need to know hard drives there is a Magnetic domain it goes in a circle and the iron oxide gets dizzy with tapes There's iron oxide on a tape and it goes in a circle on a different axis and seeking to your data takes a lot longer Historically there was drum memory which rather than being a disc that went in a circle. It was a cylinder And it went in a circle and the rest got dizzy and it takes a long time for it to come around to the other side Of the circle because it's a mechanical device everything else SSDs that use NAND flash nor flash 3d cross-point battery-backed RAM Basically all the same thing from a programming perspective because there is not a physical device going in a circle that you have to wait For it to come back around because if you're anything like me, you are unlucky And that means that the data that you want is at the wrong point in the circle And you're going to have to wait a very long time for it to come around When it's in chips It's fine So what does that mean for performance terms if you're trying to actually take away something useful other than buffalo bill pictures from this talk Well for dizzy rust Find busy work in your software Regardless of whether it's tape or disc. They're different by orders of magnitude. It's fundamentally the same thing when you issue a read You're telling the colonel go do this and then if you wait for it doing nothing You're going to have to wait for a little piece of rust Work its way around and maybe multiple times if there's a bunch of indirection in the file system and things like that and if you Dispatch that into some sort of a background process You can find some other work to do like if you're looking for copying multiple files You can background the read of the first file and go looking for the second file things like that depending on the exact application What busy work you can find? While you're doing an asynchronous read will vary But generally speaking if your rest gets dizzy find busy work because you're gonna have to wait for it And use big buffers use lots of lots of memory. The memory is faster than the disk so if you can dedicate some RAM to things like page cache, it's gonna work a lot better and Copying between buffers is a lot faster than copying out to disk So even if you're wasting a bunch of RAM and you're wasting a bunch of CPU cycles copying between buffers in RAM Like who cares if you can avoid some disk because you're ordering things to be more efficient totally worth it For other avoid busy work get to it You do not have to wait for a piece of rust to go around a circle and complete a rotation So while yes, there is latency on SSDs It's really interesting how much there is depends on the controller depends on the connection Whether it's SATA or SAS or NVMe things like that all super important when you're getting super into the weeds But generally speaking it's going to be so fast that if you are spinning up a background thread just for a single read job The cost of spinning up a new thread is going to exceed the value of the interlocks and waiting for it and doing this and doing that Oh, are you done it might done because it will be done And then you will just have all of this overhead is spinning up a thread creating a New buffer and coordinating locking between all of it So just do it and then crucially avoid copies and that explains why send file is so much faster than GNU CPU Because send file is not implemented in the C standard library. It has actually implemented in the kernel The way GNU CPU works You say kernel Please give me the data in this file and the kernel reads the file Ideally it's already in page cache. Maybe it has to go to the disk You don't actually care, but it gets into page cache. It's copied from page cache Into your address space. You go great, and then you probably copy it Into another buffer in your address space, and then you go kernel Here's that thing you gave me you can have it back, but put it in a different file now So you've got three copies that it goes through where you roundtrip it from page cache to your application To a destination buffer back to kernel On the other hand with send file because you're saying kernel I don't want any of this you deal with it The kernel copies from one place in the page cache to another place in the page cache, and then it gets flushed out to disk There's only one copy. That's a third as many copies. I'm bad at math, but a third is less as a result it goes a lot faster and whenever you're dealing with non-mechanical storage you always want to avoid copies because While your SSD is much slower than your memory bus you ideally want to use your memory bus for other things than just a bunch of Unnecessary copies particularly if you're using numus systems, which you may not be on your development workstation or laptop But secretly you will be on your machine in the cloud when you ship code to run somewhere in Amazon They're big systems with multiple CPUs So you may wind up having a copy data between physical nodes in a numus system And if you just do anything to avoid those copies then they're free because anything you don't do Doesn't take any time to execute. Yeah, that's It's got bored. I talked too long All right So takeaways you've learned a lot about Buffalo Bill He lived until 1917 Which it turns out is long enough that he was aware of the first tank battles in 1916 during World War one and People drove drove Model T forwards to his funeral And he makes an interesting metaphor It was kind of silly to put a picture of him in the presentation because he has nothing to do with this But he is a guy that we conceptually imagined being purely in this period of the Old West as if he just stopped existing when the 20th century happened and he actually lived well into World War one he was aware of Aerial combat with fighter planes happening in the European theater. He was aware of tank battles He was aware of all of this stuff that we don't think of somebody from the Wild West in the 1800s knowing about and That serves as a good metaphor for the fact that backwards compatibility is a nightmare of Lovecraftian knot that takes you back to the Stone Age if you keep following the thread and will haunt you forever because literally in 1870 there were still parts of the United States that Had Indian tribes that were unconquered and they were literally Stone Age people because they Didn't have large infrastructure for spelting iron or making bronze So when I took you back to the 1870s in parts of California, I literally technically took you back to the Stone Age So in band signaling is always wrong I always want to emphasize that it is the reason for sequel injection It is the reason that color is hard in terminals It is the reason that there are exploits in SCP protocol Unfortunately the bullet point right after that is that in band signaling is inevitable because we need to maintain Backwards compatibility with these legacy systems, and if we have a signal we will use it So one of the takeaways is if you are going to be doing in band signaling that could potentially be Mixed in with your actual data Be thoughtful about how you use your metadata Because like the metaphor of Buffalo Bill it will be around way longer than you think and like DD It will be used for things you did not expect and you will make someone's life terrible In band signaling is inevitable The cheapest code to maintain is also the fastest code to run is also not code Anytime that you can delete some complicated legacy systems from complicated legacy infrastructure Delete a bunch of code from your application. That can no longer make your test suite fail That can no longer get stuck waiting for IO. That can no longer be the cause of a vulnerability because you're doing in band signaling On and on whenever you can use a really simple solution that will always be better Don't be afraid of code that runs on a computer For a lot of this stuff writing C++ even though that's theoretically a super low-level language. It's very scary You can use super high-level API's that aren't that bad to use in practice and you can have very compact code If you're not chasing hyper specific API's and super specific Things and trying to handle every edge case in C++ a lot of times just running code on a computer is pretty easy and pretty efficient and then Yeah, just when in doubt prefer simple solutions All right, so that is my talk that is my tour through the weird history of some things that I learned while I was implementing my own user land and You've been exposed to more legacy than you ever needed or wanted to so are there any questions So on your journey, do you ever look at on your wild wild journey? Do you ever look at busy box? Yes, I'm aware of it and it's super cool on embedded systems. I didn't specifically Dig into the history of it or benchmark my stuff against it, but yeah New core utils is definitely not the only available implementation of any of this user land stuff busy box works great probably Better than you knew in some cases because it is more recent as well How do you disable the see out printf the legacy support and why isn't that a default? Ah? Because legacy when C++ was first becoming popular everybody had legacy code bases in C and If you started using this cool new C++ library, but you've got gibberish It would theoretically result in sort of like if you had two threads outputting to the same pipe You would get intermingled output from the see out buffer and the printf buffer So nobody would have adopted C++ at the time if they had done that and Now we're stuck with it probably forever. Maybe how do you disable it? There is a link in the slide deck if you Got a link to the slide deck on my website If you go to willrosegrants.com it's up there In I clicked through a couple of slides that had just links On the slides and one of them is linked to that and you can see me use it in the WTills repository Which is also a link from my website It's like two lines of codes. It's actually really simple and just Don't do this and then it doesn't it goes faster. It's great So So DD copies files and CP copies files too. Why do we need to? Because DD converts and formats files and incidentally happens to copy them The reason that DD is really still popular is that so IBM mainframes Mainly used fixed record sizes for all of their data formats so DD has a bunch of settings for reading a certain number bytes and block sizes and Offsets and things for doing all of its manipulations with its character set conversion Assuming that you need to convert to only uppercase ebsidic That's an interesting late legacy that dates back to IBM's use of punch cards Where you would allocate a certain number of rows on the punch card and then when they move to electronic systems They sort of adopted that for their default data format with fixed record lengths as a result When you were trying to interface with hardware you could say read from this device file because in Unix everything is a file and read exactly one megabyte and that Ability to set alignments block sizes Buffer sizes things like that doesn't exist in CP could Nobody ever added it because we had DD Long after we stopped using it for converting between uppercase and lowercase and it was useful You were able to write programs and more efficient C++ libraries Was it as a pop don't those libraries don't some of those libraries also have a lot of legacy code. Yeah Yeah, yeah So something like the boost file system library boost itself is massive It's a ton of different libraries. Some of the libraries in boost haven't been touched since C++ 11 and if you try and use like boost graph library You're gonna party like it's 1999 man. It's great fun. Any other questions?  What's next? Not done. It's a fun thing to tinker with now and then At some point I would very much like to write my own shell or something like that It's a big project to try and tackle all What makes up user land, but you know if you pick at it one by one on a weekend Over time. I'm sure I will wind up adding more to it. I Probably won't bother to write a C compiler because The GCC guys have been working on that for decades and they're still not done There was a question back here somewhere No, I guess we're done. Well, thank you very much. Will that was very entertaining and Thank you all And if you stick around for half an hour, you'll get the next round of talks, which I believe is about fast string processing with HTTP Hello. Hi everyone Okay, I think we have a quorum here. So we're gonna get started Welcome everyone to the developer track here at scale This afternoon. We have dummies talking to us dummies talking to us about Working in serverless and GraphQL to build reliable and scalable apps I'll take Q&A at the end to bring around the mic so everybody gets a chance to be heard But beyond that, let's take it away. Tell me Let me just set up my mic. Can can everyone hear me? All right. Cool Great to be here. This is my first scale And I was looking at the schedule today and I saw I saw that there's something about game night I'm going to be here at every scale now. That's that's game night All right, so I'm gonna talk a little bit about building applications that are kind of resilient and scalable out of the box Exploiting GraphQL and serverless as a way to kind of architect such applications easily It's a fairly simple idea. It's just kind of a neat way of putting things together Before I get started just to quickly sample the audience. How many of us are front-end developers? How many of us are back-end developers? Not the application developers, but more on the DevOps and the sysadmin side of things. All right, you're gonna have a rough time, but So all right, so I'll start with a quick intro. Okay, wait, no wait, I forgot to ask How many of you have heard of GraphQL? All right, how many of you have used GraphQL already? Cool. All right. This would be a good intro and how many of you have used serverless functions of the service? All right, cool. Awesome. Okay, so quick intro about myself. I am Tanmay I am I created a open-source project called Hasura and we created a company around that a while back which is Hasura.io, but a fairly young open-source project, but fairly rapidly growing and and and we work in this kind of GraphQL and quote-unquote serverless space. So to kind of motivate the the problem that we have in a way is if you if you think about putting together an application on the Putting together a full application, right? The traditional architecture that you might have Which is probably what you won't have at scale But what you might have initially when you start doing things and when you move from like monolith to services is that you have a front-end app and this talks to an API and The API kind of talks to multiple other services, right? So the API for example talks to a database or the API talks to a payment service or You know, I'm talking about like a food delivery app It talks to maybe a restaurant service to To figure out if the restaurant is open or can take delivery can take orders or not And then maybe to a delivery service that has complicated business logic for figuring out how we should actually assign the delivery person And do the delivery, right? so so this is kind of like this is what you might have and and and This will start becoming a problem very quickly and you will start missing or at least I would start missing the monolith As soon as because you're kind of creating more services. So obviously you need to start worrying about It's it's not really scalable It's not really as scalable as you thought it would be right You have to worry about scaling each of these services independently or the scalability of your API server Actually depends on the scalability of these other services. So it's not like you've got API being scalable for free If you think about it being resilient That also became more of a challenge than when you just had a monolith you now are dependent on every single network link Let's say for example, the restaurant microservice fails. Do you roll back the payment? Do you you can't you can't you can't undo you can undo a database transaction what but you can't undo an API call, right? And and and so kind of building in that resilience after you have this setup When you realize that there are arbitrary network failures is is a bit of a pain, right? And also microservice is a little bit 2015 now. So so maybe we can maybe we can do something new, right? so so one of the first things when people start to think about Making things more scalable or more resilient is To kind of async as many things as you can, right? so that at least failures that happen are not synchronous and and You you when you do this that the expression on the on the things face is exactly the expression that you end up with because Asyncing all the things is not is not fun But if you async stuff, this is kind of what things might look like, right? You have an app the app talks to an API Now the API kind of talks to I'm calling it a database, but it's basically your state, right? And does some operations on the state Which atomically kind of creates events that are captured and then these events get delivered to services And then the services asynchronously process stuff and maybe write back to a stateful layer, right? So they don't again don't have to write back to the same database but state, right? So so they kind of events get triggered things happen you write back to state, right? So and so instead of kind of synchronously talking to multiple services You asynchronously talk to other services, right? And this is the kind of this is the first step towards setting up an async architecture And this is nice because because you have a persistent kind of event system and you're persisting your state you're getting resiliency kind of You're building in resiliency into your system because if something fails if there's a network transient failure or a Hard failure then you basically have as long as you've captured events you can You can theoretically kind of redo things again, right? and and and that's kind of the first step so In in that light and there is there is there are still problems with this and we'll come back to that later But in that light if we think about If you think about serverless or when I say serverless I I mean functions as a service I don't mean the CNCF definition of serverless, which is a little bit complicated But the the simpler definition of serverless, which is just the functions of service piece is And let's take a look at what that is for those of you have not used serverless or functions of service You write a small function typically in a language like no JS This is where most of the serverless action is and so you write like a small function this function When it runs it responds with a JSON that contains hello world, right? That's it. I just write the function I don't do anything else and then You have a cloud vendor or a serverless vendor or your on-prem vendor. You have some tooling you use that tool to say Deploy this function call it hello world and trigger this function every time there's an HTTP call So attach it to an HTTP endpoint And so as soon as you run that command what you get is you get an HTTP endpoint Every time you call that HTTP endpoint an event will be triggered by your vendor the or the cloud vendor Which will then kind of trigger this serverless function. It'll provision the resources to kind of create this function And and and run this function and then take the response and then send it back to you, right? So as soon as you do this you can curl that endpoint and you get a response back, right? This is this is the serverless experience this is scalable Quote unquote because you don't have to deal And and you don't actually get to deal with anything you don't you don't have to worry about You don't have to worry about what happens when there are a thousand invocations of the function because when there are a thousand invocations of the function There are a thousand instances of these functions that run, right? But this kind of scalability that you get where you don't have to do any of the underlying operations of kind of maintaining an API Server because it's just a function or the freedom that you get when you don't have underlying VMs to manage for those servers for those servers or containers to manage And when you don't have to worry about this you kind of think that It's freedom where you just get to write code and you just get to deploy but this is This is not really This is not really freedom. This is it's it's weird because it's You're forcing the developer that's writing code to not do things that they would have done when they were writing a Traditional API server, right? So you can't do global variables You can't think about what happens when there are con-con requests to the same function. There is no you you cannot There is no state. There is no mechanism for you to actually capture state across invocations of the same function And so when you take that out of the function, obviously that function is scalable So it's not it's not it's not magical. It's not even special It's just the fact that there's a little piece of code that runs and each piece of code runs independently And that's it. But in a sense, it's freeing if you have an architecture around it Because you kind of get to scale for free, which means that you're not really thinking about I guess the font sizes might be a little bit small But you get to scale for free because you don't actually have to worry about Managing infrastructure. You don't have to worry about monitoring infrastructure to figure out how to scale stuff And and and as a developer you get almost end-to-end ownership of Of the entire life cycle of your business logic, right? You write the code you deploy the code the code will work the code will scale forever if something If something happens, I own the code. I change the code. I redeploy it. I still get complete ownership And and and because there is no ops things are happening for free I'm using a lot of air quotes because all of this comes at a cost But this mental model is weird, right? So it's kind of like the way I think of the way I think of serverless functions is that it's kind of like Nano services the reason why we move from a monolith to microservice For me the way I think about this is that it's not just about It's with monoliths monolith scale Facebook and hence nobody can argue with that right but But it's it's that ownership doesn't scale right and that's the reason why you chunk out into so into having different microservices that the Management or the the the ownership that you have of this entire code base across a large orb will not scale And so it's easier to think of ownership when you break it up into services, right? and if you Perhaps take that line of thought to an extreme you you get serverless functions where it's complete like it's hyper ownership And this developer doesn't need to talk to anybody else And just does their own work inside a little block of code Manages and manages it end to end entirely, right? So And So you sort of get freedom and you sort of get scalability, right? And for the purpose of this talk, we'll call this being scalable We'll come back to this we'll come back to more aspects later Resiliency around this is also very interesting because of the model the model that a serverless function has is that When the serverless function is running, it's not it's not something that is running Already that is kind of auto scaling up and down the thing doesn't exist And it is called into existence when there's an event, right? the the idea of it being event driven is is built into is built into the paradigm, right and I Mean for better or worse and and what that allows us to do is is think about Making our serverless functions resilient fairly easily because if you think about it each function Think of each function as just being ephemeral because it is And and it doesn't have any knowledge of state, right? And if it doesn't have any knowledge of state, you have to trigger them when certain things happen So you trigger them on events. So technically if you can replay events or you can retry them or Whenever failures happen or whenever there's a back-off or whatever you want to do you should be able to build in At least a certain amount of resiliency Whenever there are transient failures, right? It comes at the cost of you having to think about your serverless function being idempotent Which means that if I try it multiple times things should still work Which is a problem, but which is kind of sort of what you already which you have to do when you have an event-driven system anyway, right? and and and the tooling is getting better to not have to worry about this so much but But in a sense because each function by itself can't too much and each function is triggered on an event if you can persist events and replay events you should be able to move towards it being resilient and And and a combination of it being resilient and it being scalable by the cloud vendor So as a developer now, I don't have to care about the ops of anything, right? Technically. It should just always keep working I and I can ignore all kinds of transient failures and so and so so when you think about So when you think about writing your code in serverless functions, you're getting some sort of resiliency and scalability out of the box, right? You're not you're not explicitly thinking about it You're not explicitly adding it to your application after you've built the application So let's take a food ordering app as an example, right? If this is our state This is our model that is there somewhere in the state, right? It's not it's not necessarily a table that has multiple columns It's it's the abstract data model that we have in our state, right? And so this order this order object has is created by a particular user has a validation state Once the order is validated. It has a payment state once it's paid for, you know It's approved you check whether the restaurant is open or whatever or maybe you check that before doesn't matter And then you check if you want to assign a delivery agent to it and you assign to the agent, right? So this this order object is going through various states and if you think about and if you think about Doing all of this business logic with serverless functions The the way you might want to think about it is you think about your entire business logic is state machine, right? So you have a new order and where all of these kind of validation state paid state approval state Delivery person state is all false Then you run a serverless function That does the order validation and so the state changes to validate it equal to true and maybe some other stuff again apologies for the font size, but If you can just make out that the blue is changing to red That's kind of one state change right and similarly you can kind of keep going to state changes So the next serverless function comes in does your payment stuff or validates that the payment is done does the approval stuff? Run some machine learning algorithm to figure out who the right delivery person for for you is based on your rating Based on your entire life history And then and then kind of assigns a delivery person to you based on their entire life history And then and then you kind of move to this final state where the order is being Is being processed and is actually being sent to you, right? So so so that's kind of So so if you have your state and you have a mechanism to trigger serverless functions whenever there are state changes And your serverless functions will drive those state changes, right? So that's kind of one way to think about running your business logic with serverless functions For your for your for your application logic, right? The the cost of setting up the system is that you get to write code Inside serverless functions easily, but the cost is that you need to build a generic event system that captures events Well that delivers events right and if you have this generic event system Either you might have it or you might use a tool like how sort of what? But you might also but but you can set it up yourself You can do a bunch of things on the cloud vendors are doing this tooling right and and so if you have like a good eventing system Which is generic and a good event capture system, which is generic or which works for your use case You should be able to start moving stuff to serverless functions fairly easily, right? There is There is another cost to this Which we'll come to but at a high level just to kind of make this clear for to kind of just draw Put the two architecture side by side you had the kind of earlier architecture where there were service level dependencies Between your API service and other microservices and now you've taken the back portion of it We still not talking about the front-end app yet and that's where the problem is But you've taken that kind of back-end portion of it and converted to an event-driven system where things or your API's or your services Are triggered asynchronously whenever events happen, right? The problem is that whenever stuff happens on the back end Right. How does the front-end? How does a front-end app know? What is what has happened right and this is historically always been the problem with doing stuff asynchronously And you can replace front-end app with any client, right? The it's it's great It's great to say that we're making things asynchronous But it's very painful for the end user or the developer that's building the experience for the end user to consume Asynchronous information because finally on my application I need to see the state changes that happen as well or maybe some Representation or abstraction of the state changes that need to happen on my app, right? And so when stuff is happening asynchronously on the back end How do I how do I get that info on the app, right? Because the app is designed to make an API call the API call does ten things and I get a response And then I just show that response, right? But now I did an action that forked up a bunch of things When am I gonna get the API response? How am I gonna show that final thing on the app, right? and it's it's painful and Which kind of leads us into Talking a little bit about graphql. So so that that final piece. We're going to we're going to try to see We're gonna see how graphql is going to be helpful. So before we do that. We need to understand take a little bit of a introduction into what graphql is right, so Back in the day from my point of view We were all using rest API's right and so this is maybe what a rest API look like forget everything I said about the back end and now let's come to the app and the API ignore what the API does So let's say I'm building a user profile page on my app if I'm building a user profile page on my app and I make an API call to fetch the user info Let's say I fetch the ID and the name so I do slash API slash user ID equals one I get a json response that says ID is one name is Elmo. I render this on my on my page. It's great But then tomorrow, I realize that we want to make the user profile page a little bit richer. So let's add Let's add more information here I also want to show the last the last known address for this user on their profile page So then I make another API call To fetch slash address user ID equals one So I'm making two API calls and then I get that information and I show it on my app And this is not ideal because in this in this toy example I was making two API calls, but in a in a more serious example You might be making five or six API calls, right? and and As a room full of back-end developers Y'all are not very empathetic to front-end developers Right because this might sound like it's not a problem, but this is a huge problem for a front-end developer because writing API calls and and the last portion of the front-end app that you want to work on is the bit that does the API calling Right because that's just always the worst you make an API call the response comes something happens on the UI You make ten API calls like ten things happen on the UI independently It's just it's just the world's most painful experience to build when you have to make multiple API calls, right? And I don't know if you have this experience But it's also super irritating when you load up a page and it loads up really quickly But then there's an API call that happens in the background you click on this button But the button shifted and it's replaced by another button, which is which is a destructive action, right? And every time that happens to me, I just get so I get I feel like punching my screen And so what you do is that as a front-end developer you go talk to your back-end developer Hopefully your friends and you say can you can you give me one API call? Right, don't do two API calls Just give me one API call and give me all of the info in that API call and then the back-end developers like really You already have it. I mean what has called two functions. I don't want to do this work I have other more important work to do but then like you know You you kind of convince the back-end developer the back-end developer goes on leave for a few weeks comes back whatever Maybe a month later. You have one API endpoint that gives you all of this information, right? And and you're happy because now you're just making one API call for your entire page And this is great till you realize that Till you realize that you want to fetch different amounts of data For a mobile site and different amounts of data for the desktop site on desktop I want to show you your user profile. I want to show you the last 10 addresses I want to show you the last 10 orders one on the mobile site I just want to show you the user profile one order Maybe the last five orders in just one address and then as soon as you do that You again have to go to your back-end developer and say Yeah, I'm making this API call and I'm fetching like 500 kb of data every time Which is fine when you're on this great Wi-Fi connection But it's terrible on mobile and just speeds are slow. Can you add another parameter called like fields? And I'll give you a comma-separated list of the like the actual fields that I want So I'll do like I'll say user info id equals one and Feels equals id comma name comma address comma whatever right and then you kind of come up with this syntax yourself And then you decide how to pass the commas and then you're not allowed to have commas in your names Which you don't anyway probably But but whatever right like it's a thing. It's the thing that you've now built with you and the app back-end developer And and and now you've done this right which is just a nightmare Process so what the good people at Facebook did is they said let's not do any of this Let's reinvent Everything and call it graphql and what we'll do with graphql is we'll not have multiple API endpoints We'll just have one API endpoint and this will be a post API endpoint And it's typically a slash graphql endpoint and we'll post queries to it.  You you just kind of make this query where if you can see on the left in this gif if I query for user ID The json just contains ID one if I query for user ID and name I get user ID and name So this language on the left is called graphql right and if you look at graphql It's basically like json with the values removed and the quotes removed right It's kind of it's the it's the skeleton of the response that you want And this is why front-end developers love it because it's just so easy to see What you know what you want to do and what this query means and the back-end developers hate it because now they have to You know build a graphql parser and whatnot Because you need to pass this query. This is a new language So you need to pass this query figure out what to do, but there's great tooling around it So you don't really have an excuse, but Or you can use also again, but now this is okay But so so you can do so you can kind of build your own graphql server and your graphql server will pass this graphql Query it will do the same thing that you did always with the API call But the contract is that your server will now return a response that has the same shape as What was requested by the user right and this is graphql or this is one core part of graphql that makes graphql awesome, right and Get rest calls are replaced by this notion of graphql queries and post put delete patch Rest calls are replaced by this notion of what is called a mutation right and they both look fairly similar So so that's what graphql does right? But graphql also does something else and that's the bit that's super interesting to us Let's say on the back end. I have this order object, which is changing Hopefully you can notice that like there are two colors that come in and this is a table So this is an order object table and I have and I have the status of this order is changing because Asynchronously something goes and updates this order table this value this order row right As an example right so there is an order that was created after sometime the payment suddenly just turns to true Because something happened asynchronously or the fact that it's dispatched suddenly turns to true because it was actually dispatched as an order, right? So this object mutated on the back Asynchronously now when this happened what I want to do on the front end is build an experience that shows you on your app When you have the order page open that shows you payment tick Delivery tick right even if I close the page and come back to it, you know, wherever I left off You whatever happened in the middle. I just kind of does it shouldn't matter and I should be able to kind of show you the latest state The way I need to write so I'm consuming this information asynchronously from the back end And whenever this changes, I should be able to build the UI for doing that. So before graph QL What you would have done with a typical API is the first option was that you would have done polling The only thing worse than making multiple API calls for front-end developer is to pull for the same API called multiple times That's possibly the only thing that's that's worse than just doing multiple So this is a fairly straightforward idea I will just keep refetching the order object every x seconds and I will pretend that this is real time Because I'm how's how's the end user ever going to know, right? And so doesn't matter and surprisingly it actually works fairly well in most cases It just is bad because a it destroys your soul a little bit every time you set up polling on the front-end app And you set up a cleaner because you can't keep polling even when the person navigates away from that page, right? otherwise you'd be pulling for everything and it also feels bad because On on mobile you're eating up a little bit of extra network You're eating up extra battery every time you do polling, right? So that's one the second option is web sockets and the reason why web sockets is not fun Is because you move straight from like yuck the nightmare There's this there's no middle ground because web sockets is just the world Is very is very bad And it's it's it's been a part of the HTTP spec now for a while, but it's it's incredible how The lack of documentation of the spec and how browsers deal with web sockets is astounding Like the the way browsers deal with web socket connections on cores requests is very different Browsers will not send custom HTTP headers Overcores no matter what you do, but it will work if you're on an Android app It won't work if you're running it in a browser There's all kinds of crazy things that have that's happening and you don't want to deal with that socket, right? so What graphql says is that you know what we'll come up with a spec for doing this real-time business as As a part of the graphql spec itself. We had queries for fetching data We had mutations for mutating data and we'll have subscriptions for watching data, right? Or for watching on events or listening to events It's almost as if like there was a new rest verb called watch and that's called a graphql subscription, right? And so with graphql If you remember that order object Subscribing to it is as simple as let me load that. Yeah, so if you can see the bit on the right here You open up a subscription and you say I want to subscribe to an order where the ID is 57 and The shape of the event should contain the payment info the latest payment info the latest dispatched info Every time this changes Send me send this event to me or send this if every time there's an event So either you can have a live query type thing or you can have an event type thing whatever you prefer but Every time this changes send it to me as a front-end developer. This is all that I do I Don't I don't deal with web sockets underneath it. It uses web sockets because graphql clients use web sockets underneath it And if you're building a graphql server, you still have to write a web socket server So best of luck for that, but but at least as a front-end developer. I don't have to do anything To consume real-time information I just have to write a graphql query and replace the query keyword with a subscription and I'm done, right? I don't I don't have to do anything else Which is which is amazing because now I can kind of consume information. That's happening asynchronously fairly easily, right? So if you kind of put this idea together with what we thought about when we did like event-driven serverless Your app now talks to a graphql API The instead of a rest API the graphql API is Closely linked to an eventing system and has synchronous logic whatever logic you need to be that you think is synchronous kind of goes into inside This API it talks to your state your stateful layer And and that's your graphql query that you're running from the app, right when it talks to your stateful layer Eventing happens and that triggers serverless functions to to kind of do stuff And so you're doing your you're processing stuff in this case payment and restaurant and and delivery and whatnot and And then whenever that happens Your app can open up a graphql subscription to say I've created an order Now I subscribe to this order and then you chill Right and then whatever happens whenever it happens However many failures happen whether it happens in the next few milliseconds whether it happens the next day. It doesn't matter, right? Everything just keeps working That's kind of what the system looks like what you can do is The cost of the system is as is fairly apparent is that you have to build a real-time graphql server You have to build an eventing mechanism And and then you have to convince your back-end developers to move to serverless best of luck with all of the three but But the the kind of the other way of looking at this and and the reason why I'm talking about this also here Is that's kind of what we've been building is that orange box can be replaced by an open-source component called hasura Which is which is what we build which gives you a real-time graphql API and an eventing system on top of Postgres Which kind of helps you set up this kind of architecture out of the box, right? And and in fact this pattern is something that we're learning from have seen from our users And that's kind of where the inspiration of the stock up comes from as well. So what I'm going to do next is do a Do two demos one demo for you to kind of look at what an application That is built like this looks like and the second demo time permits a demo of actually setting this up with with hasura as well so Kind of a little bit of a prayer to the demo gods now for all of this to work if you guys have phones If you all have phones, please choose to take out your phone then scan this QR code or head to this URL And I'm keeping my fingers crossed So I'm just keeping this it's called for those of you want to type it in it's bit.ly slash three-factor hyphen app And if you can scan the QR code you can scan the code. All right, so Let me open this up. So this is kind of what you should What do you should land up on? And this is the food delivery app that we've been talking about right, so I'm going to place an order as Tg 17x right and So now what happened is so this is a react app The react app is speaking to a GraphQL API to fetch your past orders. I don't have any past orders I'm going to place a new order Then it fetches like the menu items and I order a doughnut and haggis So it's the best Congress a new thing and and and then I subscribe to the order and The order validation is a AWS Lambda. It happens asynchronously Now it's blocked in the user. The payment is a synchronous function because I need to talk to the stripe API does Stripe payment gets processed updates the database and says payment is done I get that kind of tick asynchronously then the next Lambda kicks in actually it's on Google Cloud Functions And then the next thing kicks in which is a delivery assignment thing, which is again on Google Cloud Functions, right? and As we can see on this chart that I have also built is That you will see these state changes Happen as you folks are placing orders, right? So the red line represents the total number of orders placed the brown line below that so that that let me The line below that represents people who paid for their order or the orders that have gotten validated Once the order gets validated the green lines represent people paying for their orders You can notice that people don't like paying for their orders, which is why it's substantially lagging But as soon as people pay for their orders that state changes triggers the next serverless function Which goes and does the state change for Assigning or approving whether the restaurant is open and whatnot or can't take your order and then triggers the delivery function, right? So so that's kind of how how it works when I don't do this demo to you I pay nothing for this business logic to be running And then when I do do this demo to you You can do I mean you can we can have like thousands of orders get placed at the same time and I don't have to worry about it, right? I Don't have to worry about scaling it right in a sense in a sense still a toy demo the the other portion that I'm going to show you is Kind of simulating what it looks like if this was if we had a failure, right? So if you can see the screen I am going and placing an order for Dosa and idly which are two South Indian food items delicious and then as soon as I did that I placed a thousand orders in one Shot so I have a UI button that places a thousand orders and then after placing the thousand orders these The validation start kicking in right so more and more of these orders are getting validated And now to kind of simulate failure what what I'm going to do is once these orders get validated I'm going to hit pay for all of these orders and And so I paid for all thousand of them in one shot So you noticed a full state transition where all of them got paid And then I go to AWS and I throttle and I throttle the function And and this is kind of like a network failure, right the LAN cable got disconnected in the data center And so you notice that like 500 or whatever got processed and then it just flattened out, right? And this is this is not a failure of the delivery assignment function. It's a failure of the Of the order validation of the restaurant approval function, which is the third step not the fourth step So third step is failed and so we're kind of flattened out on the third step And we're panicking and I'm calling up like various people and now they're like looking at cables They're still looking at cables right and now they kind of figure out that there's a network failure So let's reconnect those cables, which is me going to AWS and unthrottling it And saying unlimited whatever don't throttle this anymore. And so as soon as I kind of remove that throttle And AWS is back these continue from where they left off, right? And and and this was Having this experience for Everybody in the organization including my application developers. I didn't have to I didn't have to think about it More than I usually would just just by building in this style I I just wrote my application the first time in this weird three-factor asynchronous way And and whether there were failures or not or whether there were thousands of orders or not things keep working. So that is Not my order history. Sorry viewing history, right? Cool. All right. So so that's kind of what that's kind of what An application that's kind of built of this built like this looks like I have a little bit more time so I'm going to spend the next five minutes and show you a live demo of how Hasura itself works And and kind of fits into this helps you build in this pattern. So let me Pray to the gods again because this is a Live demo from scratch. All right, so so has to rise kind of a simple Docker service what we do It's an it's an open source engine. What we do is we You you take the Docker container you point it to a Postgres database and it generates it generates your real-time GraphQL API for you and And then it does eventing as well, right?  So what I'm doing right now when I click on this deploy is that I'm asking Heroku to provision the Hasura container for us and A free Postgres database Heroku does this for free bless their hearts and Kind of allows us to to run this container and to run a free instance of Postgres It takes about it takes a few seconds to deploy. It's a tiny Docker image about 20 MB Docker image and that's kind of what The UI looks like and I'll I'll I'll go into and we'll kind of look at a few GraphQL queries to understand it as well so that's your GraphQL app endpoint so mobile and web apps that you're building will connect to that endpoint will make post requests to this endpoint And we'll try out a few GraphQL queries here, right? I'm going to go ahead and create a table in my database just for trying things out So I create a profile table that has an ID and a name and Creating some random names here and so I do standard database stuff I'm just creating the aspect ratio is a little bit screwed up, but I'm just created table and I've gotten to rows inside it and what housework gives us is a GraphQL API so I can query a profile you know ID name Auto-complete my way through it and I get that json response, right? and So what we're doing internally is we're a compiler so we compile GraphQL into SQL queries We add right access control rules and whatnot. I'm not using any access control right now So any of you can go and destroy my entire database don't do that but But but this is kind of what a GraphQL query looks like and to kind of show you a few more examples of what we can do with GraphQL what I'm going to do is I'm going to just import an existing database so that we have a We have slightly more models to play around with so that's my database URL and a All right, so that's me connecting to Postgres directly and that's the table that I have in Postgres I've done this via the UI so the tables already here. I also have a Database dump. I'm just going to import that into my database, right? So So now this is kind of creating all of those tables and now it's starting to import data, right? So as this happens if I go back to The house for UI It'll tell us that there are these tables. These are not exposed over GraphQL. So I'm going to expose them over GraphQL There are foreign keys. These foreign keys are GraphQL are hints of GraphQL relationships And so I'll say yes convert all of them into relationships and now as soon as I do that what I can start doing is query albums, this is a music database Let me show you what the schema looks like Chinook that that's the So that's the schema that I have here. It's a music database. I have albums albums have artists albums have tracks And and as soon as I import that here, I can do albums on my API ID title And I can find the artist and the name, right? And I'm getting that response here, right? So whatever 300 elements or so In the in the shape that I wanted to write and I can This is the way you'd make queries to like say that I only want. Let's say the first 10 and You'll only get 10 responses, right? So so that's kind of what GraphQL looks like Underneath it just to kind of make this make sense for you. This is a post-end point. I'm sending this as a String to the server the server takes the string and and response with this The experience as a front-end developer is amazing because if I make an error, let's say I misspell something and I call it Whatever TLE right I get a little red underline if you can see and then if I hover over it Says can't query feel TLE. Did you mean title? Yes, I did mean title There's AI in this in this API experience, right? So so I'm I'm I'm the as a front-end developer I like using GraphQL, right? What I can also do is query for the GraphQL API in this case is made by Hasura But you can have your own as well. So what I'm going to do now is query for the music tracks I'm going to make an aggregate query to fetch the count, right? So if you notice you'll see that every time I run the query, you're seeing that the count is changing, right? And the count changes because well, I'm importing data, right? And now what I can do is I can convert this to a subscription And you'll see that this data just keeps changing, right? So as a front-end developer, I don't care I don't care. There is something changing on the back end. Can I make a subscription to it? Yes, I can I will subscribe to it. It works. I'm done, right? And so now if anything changes on the back end, I'm kind of getting that information real time on the app And I can kind of deal with it, right? So so this is kind of the GraphQL portion of it The other portion of it, which is the eventing side of things is We allow you to create event triggers. And so this means that you can create a trigger called You know, whatever. Let's say echo Every time on the profile table, there's an insert We'll go call this webhook, which is httpbin.org slash post Right, so now if I go in and insert something, so this gives us an excuse to try out a mutation So let's do a GraphQL mutation to insert data. This is like a post So I'm going to insert profile I'm going to insert objects name Whatever asdf returning ID, right? So I inserted a new profile. I'm going to insert a bunch more Right, I inserted these Every time there was an insert, I had an event trigger attached to it Which means that I can go in and these events get delivered They did not get delivered Presumably because I made an error in my url endpoint But and and I can now set up retry logic on these to ensure that they get delivered multiple times Or I can respect a back off header so that you can implement your own kind of exponential back off But that's the payload that gets sent That this is the data that was inserted. This was the person who inserted it and and the response says that Whatever this not found because I had a bad url, but But you get the idea, right? So so this is kind of what this looks like and I Yeah Let me just convert this to a full screen again, right? So so that's kind of what we do But but this is just one implementation of doing this with Postgres If you have your kind of own setup and you have a more complicated data data layer setup You'd probably have to piece these things together yourself But but this architecture and this idea of using real-time graphql as an api of having eventing on the state which is Atomic so that you're capturing stuff reliably which is reliable so that you're delivering stuff reliably And then having that trigger async serverless functions I call this the three-factor pattern because it reminds you of 12 factor And it sounds simpler, but it's not simpler. It's much worse And so and and that's kind of that's kind of it, but Feel free to please do reach out to me if you have any questions and check the architecture pattern out at threefactor.app Check hasura at hasura.io. We're a young project. It's open source. Please try it out. You know, give us feedback I also have funky stickers. So please come and collect some funky stickers If if you all are here over the next one or two days and you guys are doing or exploring stuff with graphql and serverless Let's I would love to talk about that. I'd love to learn from you From your implementation and what kind of problems you're running into especially with serverless How you're dealing with state how you're dealing with contracts Are you putting multiple serverless functions in the same repo? I do you have multiple repos for each of them. What your local dev CICD stuff looks like Would love to talk to you about that. So Feel free to reach out, but that's that's it from my side and we can do questions Yeah Let me put you the mic sir wind here Thank you. That was great. Just two questions here. So with graphql some Folks that I know that implemented it they ran into performance problems when it was under high load I'm wondering if you've seen anything with that and then my second question I haven't implemented much serverless stuff, but Is it possible as a best practice? Is it common to implement a microservice? With serverless that does that even make sense? Yeah. Yeah, um I'll I'll I'll take the graphql part first. So so graphql and performance is a big problem because uh So there are there are a few ways to alleviate it But what happens with graphql frequently is that the way you implement a graphql server Is if you were implementing a rest server, you would have URLs that are mapped to functions, right? Like a typical rails app or a jango app or a flask app. You have a url that maps to a function in a graphql server the The graph of the query. So if I do query albums artists album has a function an artist has a function So whenever I make that query the album function is called the artist function is called data is kind of aggregated Put together and you get that response. Um unfortunately in in, um In in bad implementations, what will end up happening is that you'll end up making n plus one queries So let's say for example, I'm fetching albums and artists. There are 10 albums So what you'll do is you'll make one request to fetch the 10 albums for each of the 10 albums You'll make one request to fetch the artist Right, so you're making 11 requests to your database or to other services to aggregate this data into a json api.  Um, and and this is why And this is painful to implement and there are ways to get around this and there are ways that you can't get around this But the way that we get around this for example is that when you make this query You can see that it's super fast. Although it feels like it's on a server And you can make like a thousand of these requests per second in 50 megs of ram and we do this because we compile this to a single SQL query Um, and so like however large your graph kill queries we compile that to one One SQL query we inject the right access control rule so that you're looking at your data as an application user And that kind of solves a lot of the performance problems that you get Um, there are other kinds of performance problems with graphql that you will also see with the rest api That are not unique to graphql, but those you'd solve in the same way that you were doing on rest You do some server side caching instead of querying from the database you query from redis, whatever, right? Um, the the second portion of that question about serverless is Yes, um, it's an emerging pattern to think of your microservice And deploy your microservice with serverless to think of serverless as just a new container Right that that's like like we had we had VMs. We had containers now We have serverless, which is this new thing and we'll just package our entire microservice and deploy it as a serverless function And then it'll hit the main function and then it'll delegate to one of the routes inside the microservice Yeah, that works. I don't think there's I mean, I see some people doing that. Um, there's also a very systematic way of doing that There's something called zappa Yes, yes, but but but with docker you can still reason about so for example Let's say you want to create a request counter You can have a counter variable That you keep as a global variable that gets incremented in memory, right? Because you have you have the notion of Something persisting across invocations So in serverless, you don't have that So if your microservice is written in a way that it does not depend on the fact that it is running continuously even when there are no requests Nothing's happy. It's you you don't rely on the fact that it's running Like you don't have this notion of a global variable that you can keep or a temporary file system that you can save to And so if your microservice is written in that sense in in that style Sure, it's kind of like serverless and and there are tools emerging like zappa that help you convert a flask microservice Into a bunch of serverless functions automatically but But but serverless is not really like the functions approach is a slightly different mental model.  You can't refer to a global variable. You can't do anything It's just that one function. Everything has to happen inside that function, right? Um, and so so so that's the change that's required But you might be able to get away with wrapping your entire microservice into serverless function You first Yeah No, the problem the problem is observables will help you handle the event once you get the event That is that is how observable observables will help you. So graphql client for angular js uses Uses observables, right? But the problem here is what is the mechanism to capture events on the client? That's the problem Observables help you once you have the event. How will you get the event? How will it return how will it return stuff like imagine that order page I closed my order page I opened it again Like how will I how will it the server push data to me? That's what that's yeah So here when I created the order after I created the order I got multiple events, right? I got orders validated done orders paid done something else has happened done So so that mechanism is is the problem Yeah, yeah, and a common pattern for this is called back end for front end bff So you build a rest api that aggregates over other rest apis. Yeah, but like I mean Yeah, we mute. Yeah, like soundcloud had this bff approach But you're still building those apis with graphql the idea is that the the graphql developer doesn't Handle multiple routes. I build one graphql server and then you as a front end developer can query whatever you want I don't care and in production. We'll whitelist it in production We won't allow you to query whatever you want in development. You can query for whatever you want then we whitelist it so that Um, it's not a problem Yeah, so we have a serverless see if you have one is uh With using serverless and everything's its own function. How do you manage the? repositories for those functions and maintain it without having to have the likes etc and another one would be What have you done for concerns of like connection pooling where it's a normal thing for api servers, but serverless It's gonna have to connect every time. Yeah. Yeah. Um, great questions Uh, we have two blog posts out that talk about exactly these both of these problems. Um, I prefer the the way we solve the structuring code organization approach is I'm a big fan of mono repos And uh, and so we just have we just put all of the functions In the right kind of folders in the mono repo and then we have ci cd automation that picks up the right functions from the right places Deploys to wherever Like aws or google cloud or wherever it needs to be deployed Maybe at a different endpoint or whatever right sets up the environment variables goes updates things Uh, so that works fine. Uh, but but I I think the answer to that question really is How are you thinking about the ownership of your serverless functions the repo ownership and the business logic ownership should Should be closely linked, right? So if you think it's this case where you want to have one repo per function go for it But if you feel like no, it's I I feel like this team is managing a bunch of these serverless functions together That's fine as well for connection pooling. We did a few very interesting experiments. Um We ran aws lambda, uh, which would then connect to the which would connect to an rds instance directly Without going through a connection pool, it'll just like open up a new connection every single time This is the worst possible thing to do, but it was just to see what happens, right? Um, and so very quickly it errors out like you start making 100 requests you make 100 invocations It will fail. So the way aws encourages you to solve this problem is by creating a a connection pool variable Uh, that is outside the scope of the function and and and what ends up happening is because the because the cloud vendor is Is running those containers Is recycling those containers They actually keep that that they keep that variable Um in memory So the connection maintains across invocations of your serverless function, but you don't have any control over it So maybe you get connection pooling. Maybe you don't So the way we've solved this problem is We very frequently for rds deploy a pg bouncer Um, and then the and it just runs on an ec2 and the pg bouncer just connects to rds I don't know why rds doesn't give you a good pg bouncer by default like I think digital oceans managed database offering has a Has an inbuilt like connection pooling system But as soon as you add pg bouncer you can do we we we tried out 5000 invocations in a second on Uh on lambda which we're connecting through pg bouncer. No errors Like complete zero error rate, uh aurora. You can't connect to aurora directly aurora is the serverless thing It doesn't work. It errors out faster than rds postgres does It's it's better to just have pg bouncer. So yeah database, right simple structure This is exactly I have it but small things changes Instead of a microservices the microservices is built on grp services. You're right instead of being a cdp The api gateway i'm keeping it as grpc gateway So now I would like to use your hasura framework. Got it. How do I plug in these are options to plug in On the grpc gateway right as a frontend Um, as so your frontend is making grpc calls right now? No, okay. My frontend is going to call the rest api calls.  No, no, no the grpc gateway Supports the rest api calls. Oh got it. Got it. Okay. So it's like a Proxy got it. Got it. Got it. Yeah, sure. Um, so we Yeah, so so there are there are two three ways to do this So on the frontend for example, like you do graphql the graphql query string is sent over http The graphql query string can also be sent over grpc But grpc is not a very common frontend to gateway thing But let's say you want to keep that as rest and after the gateway you want to do grpc stuff the we don't We don't do grpc yet But I think if you I think if you raise an issue on the repo we should be able to add grpc support fairly easily You can yeah, you would in if you have to use it the way you currently are using it without replacing a component You would have to keep hasura in front of grpc Uh, and yeah, you'd have to keep it in front of grpc But the ideal way of doing it is for hasura to support grpc Correct, which is what you should not do and which is why you what you should do is open an issue and say support grpc And we'll support grp. That's the that's the right way to do it Thank you, uh, how do you deal with the authorization right? Um So we took inspiration from postgres's low-level security um postgres rls, uh, we re-implemented Yeah, we re-implemented postgres rls at the api layer So you can give declarative configurations on top of your models To specify what access people have so for example, I can go to the profile table and say permissions uh user can only access data where the id is equal to Some session user id right i'm doing all of this for the ui But you can do this declaratively as well like whatever i do through ui is just a is a declarative setup as well And then you kind of say whether you want access to these columns you want to limit Um, you want to rate limit like the number of responses that you get Whatever size limit the responses that you get. Um, so that's kind of one way of doing it. Um But this is basically what the api layer does is that whenever you get a header which contains an authorization header or a cookie um Like our system resolves that and converts that into session variables like user id and role And then the permission rules Uh refer to these variables So the sequel that we create instead of when you query for query profile id name We convert it to select id comma name from from profile where id equal to cookie dot user id So that's the that's the compiled sequel. Uh, it works Any other questions? All right. Well, oh, sorry. We got one more I always think they're all done There's one left Did you have any problems with Start times for the serverless like For serverless? Yes. Um, yes It's uh, so so yes and no so if you have consistent load Like you all you right. Let's say you have a website and you're just always getting Uh, 10 api calls every instant Then this is not a problem because the cloud vendors keep it warm for you But if you don't have sustained traffic and it's spiky Then what ends up happening is that it takes time for the first it it can take They don't even give you a guarantee the cloud vendor tells you it will take Maybe 200 milliseconds, but it can actually be anything. There is no sla on how much time it takes to start Uh, in our case, this is not a problem because it's async anyway The serverless function is running asynchronously anyway, right like something the the api layer the graphql api layer is not serverless The graphql api layer is a container which is one to an autoscale not zero to an autoscale Um, so so that is running always and the event that triggers serverless That can the serverless function can start whenever it wants. It doesn't matter because whenever it asynchronously happens Uh, I will get the update on my ui because it's asynchronous. I don't actually care about there's no hard limit on The user experience is not made better or worse. Does that make sense? There is no gateway here, right? I call the server that first request ends. I create the order That request is stopped. It's ended. I create the order. I get order ID Done now. I open up a web socket to say subscribe order ID So that first request is ended within like 10 milliseconds It's create order ID. I instantly get the response back and now it's stopped. Does that make sense? We're at the hour so i'm gonna cut it off there and let our speaker get away. Thank you very much Uh, okay presentation and for answering all our questions. Thank you. Thank you See you folks at game night Hello everyone, welcome to the developer track. We'll get started here with our next talk We've got alexander here talking about fast htp string processing Without further ado, I'll let him get started and get you guys into it and we'll talk We'll have questions at the end.  Thank you Okay, thank you for introduction Good evening. See huckers So let me start from introduction who are my what we do and so on so we develop custom software in High performance network in area more than 10 years and usually we Uh, the area includes security applications like the application for those VPNs Whatever usually we develop the stuff in the linux kernel and provide some Performs optimization of existing solutions and so on one example of Such project which we proud of is positive Technologies application firewall. The project was mentioned by gardener magic quadrant in 2015 in visual nerve And also we develop tempest f double u this project which I will be talking more about this presentation And this application delivery controller open source, of course, so it's embedded into linux cpip stack And if we talk about application delivery controller, then usually this Aggregation of hp accelerator and firewall which provides you efficient load balancing Web cache Whatever you need from hp accelerator Plus additional filtration of d-dose attacks Web securities and so on In the case of tempest tempesta, it's since it's built into linux cpip stack. It's very fast. It's comparable with kernel bypass Approaches and at the same time it's very flexible. You can manage it Just like you manage Apache nginx Whatever So if we compare tempest f double u with nginx, we typically see Two to three times better performance and if we compare it with sr star an example of dptk based Web server, we show very similar numbers for For cpu cores just begin of the graph We started to work on tempest f w when we started To work on positive technologies application firewall that time we had to develop web application firewall this Kind of application firewall which works on hp layer and provides you web security filters out as much as possible web Attacks attacks also It protects from Attacks from on application layer and usually such applications are built on top of usual Hp proxies h a proxy nginx, whatever and that time we realized that usual Hpx Proxies aren't good for Fit creation logic. There could be perfect for Application logic for example to deliver content to manage it to mangle it or whatever you need But not for security aspects And that time we started tempest f w W inspired from net filter. So net filter is a kernel firewall working in cp ip stack So we just made a continuation of cp ip stack to htp and now we Have tempest f w. I want to make a disclaimer that I will talk A lot about nginx during my presentation and the reason Why I choose only exactly nginx is that nginx is very simple and everybody knows it So it's just a good reference for Presentation during the presentation. I will concentrate on hpm processing especially crucial for high performance hpm processing and security aspects I will Consider two main questions. The first one is Hp state machine efficient state machine and Hp strings processing and You might Wondering why do we talk about hp strings if we have hp2 after all hp 1.1 is about strings And hp2 is mostly about binary data However, if I would start from the last reason actually The state of the art that hp2 was Adopted in most hp servers in addition to existing hp 1.1. It means that we have a hp 1.1 server and we have a special decoder for hp2 and when we decode hp2 then we pass the Decoded data into standard hp 1.1 Parse this just how how the most server behave. The second thing is that we have Dynamic tables compression in hp2. However, they are limited in size and some of the headers Can easily outsize Dynamic tables the example of the headers cookie uli and Referee and We might note that this Hp fields are very crucial for security So let's have a look at dummy hp float against the nginx Access log switched off. We see hp parser as Hot spot in the profile and I want to mention that in this exactly Profile we see that the profile is flat. It means that if we optimize the bottlenecks then we want to get dramatic performance improvement in this case So, uh, how usual hp Parsers work typically we have a loop over ingress data and we have Couple of switch statement the first one is for states the second one for Input characters. So we have a auxiliary variable state. We have an ingress Character b we start from the loop and first statement. We pass to current state and current Character, so we typically made a jump And probably we get the instruction cache miss here next we go to the end of the loop and we repeat the Next cycle with new state and here we go. We just We made made the spin, but we actually Just moved down for the next state So ideally we should Get what we have on the Right, but actually we are just spinning all over the loop. That's not good Uh, if we look at the request line parsing in engine x, this is a snippet from real code and There are a couple of notes about how engine x works the first of all it works with copied network i o It means that all the data of the requests are in memory and if we process Get method then it means that the method is fully in realm We don't need to bother with various data chunks and Now engine x uses the knowledge it spins over the data character by character calculates the length of current token then after that Runs the switch on the length of that token This we now we're going to see what gcc optimizes for such state machines. So when we have a Uh switch statement we have a true Uh possibilities how the statement is organized. The first one is on the left Is that we have a enum of sequential numbers from 0 to 26 And in this case we instead of scanning Case statements in the inside the switch one by one. Uh, we can run a look up Table. So as a middle of assembly snippet, we have a look up table. It's just a sequential array of pointers and With the first jump we just Jump by address of From the look up table. So with two memory Deferences we get to the required state the second Possibility for switch statement is uh not Consequent states. So in this enum we have various paths States and now we cannot organize look up table because all the Offsets and the tables are very different and the Hydres index in the table is quite too large So we fall back to binary search and gcc compiles your Switch statement into binary search over the states Next let's have a look at nginx Parse typically it's about nine kilobytes of size It's like Bit more than three times less than cpu layer one Cash size and it In total it implements only a bit more than 80 states nginx path is very simple This because it needs to do only basic Technization and do not care about strict hp validation but For security reasons we need to know What the headers are we need to understand which headers are passed to backend servers and so on so We need to provide much more sophisticated HP parses to carefully analyze what the headers are so in tempester currently we have more than 500 states and Timpester uses zero copying ior it means that all the Functions about ior aren't Aren't so important in profile. So Large and Accurate hp parses becomes the even more Strong bottleneck of the system And the bad thing about Zero copy ior is not Is that currently we need to bother with data chunks we can get HP methods separated in a couple of chunks so we cannot compute the size of the Talking so We started to do the same as regular does we just use direct Direct labels and jumps plus two switch statements. So the corner The the main point of this Technique is state a definition. We have a case statement and we have a direct label and now we Execute a switch statement only the first entrance to the function And the bad thing about this Technique is that now if in engine x case we have a four or while loop and we have We have a program counter. We have a data counter inside the Loop header Now we need to move all the logic inside the move marker and Now the state machine executes the code on each state. So typically such state machines generate much larger code. However, just here on optimization purpose does Something similar. So if we look at the assembly code of state machine with 20 or 30 states We might see that about five or six states. We will have the same logic. I mean increment of data pointers to check the bounds and so on Next thing which we did is to use gc extension labels as values and now we remember labels address in parser And now we can remove switch statement at all. So we don't execute any lookup tables or binary search and now we use the auxiliary variable only once when we start to pass HP request. If we have several data chunks of the request then we still use the auxiliary variable only once Let's have a look at the benchmark. I just want to mention that in the benchmark we usually set Task to a particular CPU. We choose the smallest numbers because we need to remove all the system overheads and so on So in states in state machine with very small number of states The approaches are almost indistinguishable. However, with a lot of states we see dramatic performance improvement two or more times If we will look at branch misses in such state machine, we see that While switch statement state machine is two times less than the go-to-driven machine. It still has much more branch misses and it's Very very interesting that While code of state machine is almost 1000 bytes of code is much larger than layer one CPU instruction cache. We still see twice less number of instruction cache misses. This is because We do less jumps and we decrease probability of cache miss even in larger state machine The next point which we observe it in the state machine is that if you Program your state machine then you place your most probable states and the first states at the top of the machine And you expect that the less probable states you can place at the bottom of state machine And they will be executed later or maybe not prefetched to the cache at all However, gc doesn't have such knowledge about the state's probability and it Free to move states around the code and we observe that the less probable states Became at the top of the function while our the most crucial states Move it to the end of the function. That's not good. So if we use compiler barrier, we can force gc to keep the states as way they are and just Addition of the compiler barrier to state definition improves performance for four persons Next we consider stronger optimization compiler optimization and surprisingly at stronger optimization where we see lower performance and If we look more carefully at the what was the Optimization switches just we find that some of them Actually makes the performance worse and if we carefully choose the right switches, then we can get the more performance the green the green switches are auto-vectalization Options you can by default Switch tone on a higher layer of optimization and unfortunately not all of them Can be done and all the code for example in our hp state machine auto vectorization doesn't Do anything at all. So they are simply no optimization On the compilation phase this bit outdated gc tutorial actually Gc can optimize and vectorize much more sophisticated loops than this depicted on the slide But in our case it just doesn't work at all Next thing Intimpesta and nginx we do the same trick So if we have Some request methods Same get of two four characters Then instead of matching the characters one by one we just Compile We just Make the integer pointer a compare two integers. However, since we have no Have no idea which data came to the hp parser and we have no idea about its alignment We can hit CPU pinarty on misaligned word reading and this another Mica by benchmark and now we see that misaligned integer access Can worse performance in two times And it's simply it can be simply fixed if we just made First check with the address aligned to four bytes if yes, then we Treated as integer pointer. If not that we read characters by characters very straightforward However, if we try to employ the optimization technique in real life parser, we might Find that Performance results are becoming even worse. What's the problem? In the benchmark To measure the speed of which Memory read carefully we had to keep Benchmark code in one compilation unit and the function which Defends memory in another compilation unit. So now compiler cannot Cannot make the tricks about reading the memory and have to execute exactly that code Which it doesn't doesn't know that doesn't have knowledge about However, in real life parser, we don't need to keep such lightweight function as parity and just inline them into the parser and now the Gcc have knowledge about optimization of the matching routings and In that case we Can probably generate just different code that's not code. There's nothing special about the code. There are not more Significantly more branch misses. There are no more cache misses The code from a profile point of view just looks About just just as the another version. The thing is that assembly code generated by gc is Much more different and gc is just confused by more complex Statements in if branches and generates less optimal code. You can find more discussion about the topic in our issue on github So we end up with state machines and now we come to hps rings and Let's first observe why HP strings are important in hp parsers First of all, this is a usual URI. This is my Almost my URI from booking com when I tried to book a hotel in pasadena And if you check a lot of options on booking com, you easily get the URI of almost one kilobyte And on the right we have a state machine of URI parsing from nginx And surprisingly we see that This very long switch statement parsing character by character URI and we just saw that the case Switch based Automatons aren't so quick So if you have a slow logic from one side and you have a very efficient data to Make To give a hard time for the logic. So why don't not to use the data in your DDoS? DDoS attack so this If if you want to DDoS is not wise to use a simple index request instead of a simple request. You can just use large URI to To make The parser work hard and the large URI also efficiently load your backend logic. So you can just efficiently Nail the target server by DDoS with more complex URIs and the second thing is that in TPS we Search for injection attacks in URIs. So the second Example is injection in URIs. It's also not so long but still Also not so small URI which we can must carefully analyze and to Be sure that The problem is really important. We can just put a lot of more or less real-life requests to the Server again this access lock is switched off and we can easily see that both of the nginx Parse loaders are on the top and in this hp flute we Use very complex requests and now the profile isn't flat. So we have a very clear bottleneck in the hp processing What we can do with hp strings? First of all hp strings are very different In comparison with common c-strings. They do not end with Zero byte instead they end with various Dillimeters it could be one byte delimiters two byte delimiters and even one delimiter CRLF can be two byte or it can be one byte left only and second Think that we must carefully check A load alphabets in LFC a load strings for security purpose And the standard stale spn function provided by glipc Does very poor job on scanning the accepted Characters and the second interesting function, which is good to optimize is stale comparison And in in just in many places in modern hp passes We see special cases when you need to compare ingress string with some statically defined String in the parser. So the second string should not be converted in case And some hp passes we observed that Switch statements are used just as we just saw for nginx gliparser And we know that switch driving fine state machines aren't fast at all Um, let's make a little study about what the people do in this area. So Uh, nginx provides the most straightforward logic just character by character Processing it does very careful analyzing what which characters are lowered and not and pico parser library hpm From h to all seven Can Eat by 16 characters at once and otherwise them for against a load alphabets Cloudflare made extension for pico hp parser. So now they can eat by 32 Characters at once and They used avx true extension for that This example high psem starai Instruction works. It might be hard to read The functions on the slide This In tree hgc in tree index. So I select with both the most important pieces of the code. So first of all, we place our allowed alphabet in static Static memory we define it as cons. We provide alignment For the most efficient reference to the memory and the bad thing about PCI PCM see a starai instructions that It can treat by eight Ranges or 16 characters at once And if you have a range for example as the second range And in the range we have only one character. Then we have to spend the whole range to Declare only one character range So that's not comfortable. That doesn't suit all the alphabets in rfc So Cloudflare used a more simple approach. It's not so Precise as pico parser approach, but it's very straightforward. We just Check two bounds of the Allowed variable and also add a tab as one more Allowed character. The code is straightforward Simple and fast. This is a comparison of the two parsers. We see that cloudflare version Two times faster on large data, but Still somewhat slower on smaller data. This is because AVX Instruction set is more heavyweight than PCM see a starai I used the upper limit as one 0.25 Of kilobytes because this is the highest This is the largest Data chunk which can come to our zero-copy Processor this upper limit for is a net frame So if we compare the The string processors with stlspn, we see that stlspn is incredibly slow. So it's not good to use it And now we compare with our processor Our string matching is faster than Than both the Than both the algorithms on large and small strings and has very Very minor increase of performance in With the growing of the increased data. So The algorithm works in several stages. So the first stage is a function prologue when we process the small strings So we do exactly the same as nginx. We define the static table uri for load characters and we run very simple dummy Switch statement over four characters not more the limit for the Four characters is exactly the last statement When we have massive branching So if we add more characters, then we will slow down the last statement. And also we use static branch prediction likely statement to make the Court as Close to the entry point of the function as possible. This is because small strings are more sensitive to branch mis predictions The main Main loop body is This one we have free course of basically the same logic And we since avx Instruction set works with 32 bytes. We actually start from 128 bytes of processing The thing is Inside the function we execute the same code in parallel in four data streams This this allows the processor to use the data quickly and Execute code in parallel So after the main loop we have a Function epilogue this just very close to prologue. Now we since we Finish with 16 byte strings. We have to process a String with tails less than 16 bytes Just the same technique And here we go. This is the main Main description of the algorithm. We use a skid table representation, which depicts it on the Figure this representation uses eight columns 16 Rows and we start from loading Static table What we did here in the your IBM Table we We describe a ski Lines A ski table line by line So the first line is encoded as b8 We We read the ask here first ask here ask here low in In reverse order. So we start from p little p big At sign and zero and the third Row is Small q r big b big and two. So it's essentially f h Um Number so this how we encode the ask him lines And now we execute shuffle of the ask him Constant table in accordance to Ingress data. So if we have data s pr Then on the first position we Uh have Characters from the first line of a ski tables. So we good. We leave the Bitmask on the same position zero's position However arm is from third line of a ski tables and We need to move uh or shuffle the bit mask for field for field A ski line to the second position where our our character Is placed. So essentially now we just arrange a ski Lines and we say that uh at the first position of this link We have a character from the first line of the ski tables as the second position We have a character from the third Line of a ski tables. It's that's straightforward Next we want to do with the same with a ski table columns and arrange them in accordance with Ingress data to get the A ski columns. We need to do a bit more operation. We need to shift Characters for form bits left and split them. However in Our evx and sc extension. We have a shift only for two bytes. So we need to add Additional and instruction to clear the Move it Less significant bits. So by the end and Left shift we have from our PRR symbols. We have 0707 and it means that In first and the second Position we have a characters from the very last column of a ski tables. This is seven and we May do the shuffle and place the This column notification Define to the first and second places Now we the things is trivial We just perform and or over the two vectors and define which characters are lowered and which not Next we count zeros and return the how many bars Good for us the good thing about this approach is that um Since we used dynamic In previous slides we used statically define a tables of encoded Characters which allow it for our ski tables and now we can dynamically define In the table which characters are lowered for our ui which characters are lowered in our user agent or any different hp header and the resulting algorithm with dynamic Tables instead of static tables will work with very close speed So these are two examples of attacks. The first one is from black hat talk. We The injection uses At sign character is probably not the most Most common character in your eyes So if you don't need to handle the characters in your application, you can just Define this range for your eye and exclude the character the same you can do for user agent I was surprised that it's It's handy to launch attack in even user agent Could be static header, but nevertheless there are attacks in such headers and Dollar dollar sin is isn't a common character in user agent. So we can exclude it as well So we can define dynamic tables which characters are lowered and not So the last one is string comparison. This is very straightforward and very simple. We just Do Two things Three things the first thing is we convert Case of the characters in only one string. So in all the characters we compare In first vector we compare each character with a and that Constance and if the character is in from from the range then we convert the case We use a couple of tricks from hackers delight book to Reduce number of comparison in first two lines and also we had to subtract H zero from both the sides to Move from sign it sign it come from unsigned comparison to sign it So, however, the idea of the things are very very simple. Since we once we Determined which characters must be converted to different case. We just execute and operation with cast case Conversion constants and next we just do all with our original string. We get the string with Converted case and now we can compare it with the second string also not so much operations very simple And runs quickly It was mentioning that lips here also use I think this 2.18 or 2.19 version of lips here. I didn't check the later versions But that version of lips here Uses avx instruction set but In avx 2 we can get more speed and the results are on the slide The last topic so we end up with the parsers and the last topic which I want to discuss is FPU context switch the thing is that When we use Single instruction multiple data instructions in your programs or just compile arbitrary CIP program with highest level of optimization and compiler uses auto vectorization Then you use Registers from FPU unit. So the registers for single instruction multiple data lives in a different unit and The thing is that when you switch from user space to kernel and back from kernel to user space You don't spend time on saving and restoring the Single instruction multiple data registers. However, if you switch from one user space program to another you have to save and restore single instruction multiple data registers so Basically, kernel doesn't use Single instruction multiple data. However, if you need to Use the extension like crypto or some other Kernel code you can explicitly Save and restore State of FPU this is done by these two calls And since tempest works in Linux HP IP stack we care about the stuff and to Reduce the effect of context switch. We save Single instruction multiple data register when software queue begins and HP IP stack starts working and we restore the state when we Finish our job and let the user space to run further and just to Think just to see how how the Single instruction multiple data concept switches are built. There's another micro benchmark. It shows dramatic performance problems. However, as we just saw for Programmer alignment This Micro benchmarks aren't Do not show the problem for Full program. So if we are very bad at micro benchmark, it doesn't mean that you see the huge Penalties on context switch for your user space program. However, being in kernel is good that we Can eliminate the problem of FPU context switches As much as possible So I'm basically done this useful links. The first two links is my blog post about the topics which I discussed in this presentation Relatively old It's also good to read Cloudflare work and hackers delight. It's always good and useful So, uh, thank you Questions and we'll be Happy to see you on our github. So thank you Do we have any questions? Yeah, just one quick question. I don't really I haven't really understood you say that For this finish that machine Gc can sometimes we order most popular most important values with least important And you said that you eliminate this problem with just the memory barrier, but why memory barrier is just enough here? Um, one moment. You're about this slide, right? Yeah, um, look at the left we have Basically, this is a marker language. So so we have a state and the next each The state well Considered that we have a states processing Get header post header and the last state to process unlock Methods character by character. So when we enter the state we Since get and post request are the most frequent in Real-life traffic. We want them to have as Closer to entry point of the function as possible That's that's clear. However gcc rearranges If we use direct labels gc is free to rearrange the code as possible and Compiler barrier is Exactly the fin cut which prevents Compiler from rearranging code across the barrier So if we have a barrier we say to the compiler's precities that please leave the code or Both the barrier at the both the barrier and which is after the barrier It must be leave after the barrier a compiler is free to rearrange the code above the barrier But not cause Yeah, I actually actually I always have been using the barrier in different semantics But this one which we figured it out for state machines. It's very very interesting Just a question for me actually a memory barrier. There's a little bit costly. Is it is it possible to Use profile guide and optimizations to get gcc to recognize that the get and the post should be first sort of cheat This is not a memory barrier this compiler barrier Maybe I didn't get your question So you're right. It's a very different thing. It's not that expensive. I see a good point Yeah, the question Well, cool. Well, thank you very much. That was quite an awesome. Thank you guys. Thank you for coming That was great. Thank you guys very much And this ends the developer track for today Yeah, of course