Okay, this is mic one check check mic one Mike, can you hear Mike one? Okay, we're checking the lab Microphone on I don't know what channel we're on but how's it sound? I'm walking around the room a bit You might hear the music too because the next speaker said he didn't have any computer audio Testing one two three. Oh look, it's working Okay, I tend to ramble on a bit So I'm going to try to start on time in hopes of actually finishing somewhere close to on time Welcome to our developer track I'm your host I've also got the first talk as I'm filling in for another speaker. So I am sorry But we had a last-minute dropout. So I didn't think it was fair to bring anyone else in on the presentation This is working better. Yeah, you guys can definitely hear this better, right? All right, we shall start so welcome again to the developer track and Might as well dive into our first talk for the day, which is my talk It's called rules for the road for distributed compute And as I talked about in the abstract distributed computing is Increasingly pervasive almost every computer system now ends up being a distributed system what with the proliferation of SAS providers and containers and Network computing and just the internet itself So just to start this off as a fun exercise and encourage you all to engage and really do feel free to interrupt me with Questions throughout the presentation. I tend to work better in that kind of a context anyway But to start things off with a bit of audience participation I've got a quiz for people and anyone from Ticketmaster here and not allowed to answer The question is What was or when was Ticketmaster's first distributed system? Basically how long have we been doing distributing computing at Ticketmaster any guesses Yeah, the back there 1987 good guess you're off, but it's a good guess anyone else Get warmer get warmer 82 over here way in the back No, so actually that's that's pretty close that that in technically that's right depending on how you look at it Actually our first distributed system was January 23rd 1977 it was for UNMs Johnson's gym a show there by The Electric Light Orchestra That's that's how far back it goes And technically speaking actually this system was deployed in 76 earlier in the year But this was the first time that someone could buy tickets at multiple different locations and at every single location The same tickets were always available at the same time. So that was actually a real distributed system So we've been doing this stuff for a while and it was actually a very basic and simple distributed structure at that time was basically a bunch of Terminals that we're talking to a mainframe But it was none the less a shipping system Things have changed a little bit since then. This is what Ticketmaster looks like today We have a lot of virtual machines virtual servers physical servers lots of stuff in AWS lots of Kubernetes systems over 300 different products over Well, it's actually 887 I think is a little bit low But that's how many components software components we counted those are distinct components all of which are serving different functions So our distributed computing environment is now significantly more complex than it was then And this leads to our problem case The unhappy developer They're staring off into the darkness and it feels like they can do nothing right they can get nothing done and Everything is breaking around them and this is this is the problem that we want to solve And this is what our goal is And I gotta tell you this picture I searched on the creative column commons for happy developer to see what would come up and This was the closest thing I found to a realistic representation of a happy developer because we all know They're not smiling and laughing because software is frustrating, right? But here's someone who's productive. She's getting stuff done. She's typing away She's getting work done and she's feel like she's accomplishing something you can see it in her face She's very purposeful and focused if we could just get to that level that would be considered a win in my book so We looked at the various problems we have and we came up with some High-level systemic goals that we wanted to work towards that would get us to this promised land of the happy developer Working in a complex distributed environment So first before we go into what these specific Targets are I thought we should talk about the various enemies that you might encounter once you have a reasonably Complex distribute system and by complex we all know a distribute system like once there's five or more moving components It's already complex, right? You don't need 15,000 virtual servers before things start to get ugly really at two It starts to get ugly, but five is when it gets complex so The first problem the sad tale of uncoordinated caching This is where you know, you've got to distribute the system things aren't performing. Well, how do you solve your problems? Oh, I know I'll just add another cache and then you add another Cache and then you add another cache and all of these different caches are all they're all caching calls out to other services And they all have their own made-up in validation rules for when that cache expires This creates interesting circumstances where it's almost impossible to glean the truth. And in fact, it's not there is no one truth For example, you might have one experience that shows the state of the world Hey, you've got tickets for sale still for this event on your website But your mobile application when you look at it says nope sold out It's because they're each of the different channels are calling different services who have different cache expires and neither one of them Actually is telling the truth about the system. It's actually a third state. That's somewhere behind all those layers of caching And if you do it properly if you screw this up really right and I've never I've never actually seen it done But you can create a state where the system never goes back to the original source It's just one service calling another Service that still has a cache record from before calling another service that is pulling in that cache record, etc Etc. You can avoid actually ever taxing your databases. It really runs fast But it doesn't actually accomplish anything And and by the way every time you screw something up now you invalidate all those caches all your systems That's your repair process right you invalidate the caches everything works great again except nothing performs now And so now you're offline because you can't handle all the traffic you get at your doorstep So the other enemy we have is inscrutable state if you can distribute your State in your system across enough microservices you can make it impossible to know What is actually going on in your system at any point in time? You'll go and look in one place and you'll find one record and then you'll have to call it to another service to find the Other piece of that record and then you have to call it to yet another service to find another piece of record And finally now you've figured out that oh, it's Chris Smith And I know his age and I know his address But by the way we've already changed the address on Chris Smith and it's totally different values somewhere else in the system So this is incredibly frustrating for people who are trying to diagnose what's going on in the system And it also really makes innovation very difficult because you spend most of your time Trying to figure out how to access all the different components in the system in order to make anything happen And then we have state wars. Oh, this this is the best part If you can get multi master replication going and you don't know what you're doing You can easily create circumstances where multiple masters all disagree about what the truth is So I my name might be Chris Smith. It might be Christopher Smith I might have you know a two at the end of the email address Just getting just not really trying to make a joke that is true Or I might not and some poor guy in Chicago might get all my emails because that's what the system thinks is actually my email address This can get very difficult when you're doing transactional work And you want to ensure that you have properly recorded transactions everywhere if the system thinks two different people actually bought the product It's not gonna go well for you and they're cascading failure failures This is actually my personal favorite this one This is if you build your systems correctly you can make it so that the smallest most minute failure can take out the whole system and Then and then you can go home and rest right everybody can not know instead You're up all night your missing meals literally This is or how I learned to lose weight during business disruptions because you can't do anything else But try to fix the system. No, you we you have a really nasty problem distributed system People always think of well distributed systems are much more reliable And that's true if you design them correctly, but let's face it if you have one computer Then the only thing that can go wrong is that one computer crashing if you have thousands of computers Then you have a thousand different possible failure points Just on like strictly from an electrical problem Let's let's not forget all of the network protocol problems you could have All of the problems in coding logic and inconsistencies and state between the different systems But just from trying to get power to keep those machines on you have a thousand more ways to fail Okay, so that's that's the other one that is terrible And yes, the the question of trying to figure out why things are slow So you know that a particular service is responding slowly, but you have no idea why it looks like everything It's calling is taking an inordinate amount of time, but you don't know where that's coming from You start chasing it down those things are calling another thing those things are calling another thing It's spread across your entire network and when you trace it all down to the bottom It turns out that all of those different services you're talking to We're all calling out to other services which are in turn calling to this one poor service that got Smacked with a thousand requests just to answer your one question, right? And don't worry we can solve this we'll just put caching in front of it, right? Then we're back to oh shoot problem one all over again And this is the insanity that that goes on in your head as you're getting your system increasingly more and more complicated so we we were experiencing pretty significant reality of this at a pretty extreme level to the point where we sort of said wait we need to take a step back and reflect on how we're going to get out of here and part of that thought process was a reality reflection of reality that it's not enough to have a Target for where you want to get to and how you want everything to be perfect in your utopic Distribute computing environment But you need to have a plan for rolling it out and a process for rolling out that is practical That gets your results reasonably quickly and that can be employed Incrementally because obviously we're not going to change the software and 15,000 virtual servers overnight, right? This will be a journey that We don't expect to finish for years so I'm going to talk about the various pieces that we thought would be very key without requiring a rewrite of all these systems and And I'm going to present them in the order that we thought would make the most sense for rolling out And we've been moving through this process slowly, but surely over the last year and a half So the first one is distributed tracing How many here people here know what distributed tracing is or think they know? All right, I should definitely clarify then for everyone else So I don't lose y'all so you can think of distributed tracing as a smart form of logging in some sense or a smart form of performance Metrics the key problem that you have if you do logging with distributed systems is you get logs that are going Into a file somewhere from all these different systems And yes, it's very clear what's happening in each individual system But you can't tell what happened when a customer clicked on a button because though that generated logs in like 20 different machines And you don't know how to correlate them all together what distributed tracing Does is it provides a framework to tag? That information about what's going on your system with a context that is tied to that initial button click so that each Step of the way there's a new context created that's tied back to that original context and the system can trace The movement and the execution of code across all the distributed components of the system and pull it together for you So you can see everything that happened because of that one click and the nice the other nice part about it is you get Timings of all the different steps in between so you can see that oh my goodness We're losing like 50 milliseconds here And we were expecting to lose one millisecond because all we were doing was talking to another process there this idea and this approach was popularized by Google putting out a white paper called it was dapper was the the white paper it was about But it's increasingly become so popular in the industry that there is now an open standard for it called open tracing And that's what we've standardized on internally as our go-forward way to do tracing And the nice part about it is you can plug in lots of different systems into it. We're currently using Yeager which was out of the folks at uber which does a great job of Letting you query in introspect all the interactions between all your services But there's also a number of commercial providers out there Some of them are actually in the vendor pavilion if you go to talk to them and they're talking about solutions for tracing Now you'll know what it's about But the idea of this is that it gives us an Ability to have the insight that we used to have when you were dealing with a non-distributed system with a single computer And you were stepping through things with a debugger or looking at log lines in the log file And you could see the entire flow through the system easily this allows you to still have that even though the systems are distributed It also provides you with very good guidelines for when it comes time to refactor because one of the most Depressing challenges in a complex distributed system is when you want to replace one component You obviously need to make sure that none of your dependency none of the people who depend on you are going to be impacted by how you change it Well, that raises the first painful and obvious question who all is using me And there's there's the list of people that you intended to use you and you always have that in your mind You know the services that should be using you but in a more complex environment with lots of teams trying to execute very quickly The reality is if there's three people that you know that are supposed to be using you there's ten other people who figured out that was really useful to take advantage of the service you were using and You're gonna break them without them even being notified and that's that creates a very difficult environment where nobody can trust Anybody to provide services so once you have tracing though you can query the tracing engine You can literally see here's everybody who talked to your system over the last month, right? So that's step one and this is actually relatively easy to deploy because a number of web frameworks and application server frameworks actually already have Built-in modules and middleware that will hook you into distributed tracing systems So you can actually like plug this into your Apache server or into your spring environment and you Instantly get at least telemetry for everything going in and out of the framework So that's good. That's a nice start step two generic query dispatch doing composable queries so One of the things that naturally comes out of distributed systems is you end up building lots of APIs so that the different components in the system can talk to each other in clean and Established ways and you want that abstraction so that you can then drop in a new version of the service without Disrupting all the people who are calling you you just have this contract that you know you have to fulfill and if you fulfill it Everybody's happy the only problem is most of these APIs end up being technical debt They're not that APIs are a bad thing. They're actually a great thing But the important thing to realize is every time you are building out an API You're actually making it more difficult and for your organization to be agile to make changes because APIs have intrinsic assumptions about them about how you're doing business each day and That's great. If that assumption turns out to be true It actually is fantastic if it turns out to be true That's why we build APIs to make things simpler for those cases of things that we know we're gonna do every single day But what you tend to see happen in organizations is API is actually get built most for when you're building something new and you know what when you're building something new You still don't know how it's really gonna work with your business Whatever you thought was a good idea when you first did it six months later is now tech debt because you need to change how it's going to work So simple solution don't have API's just have a data model, right? And use a generic query engine to navigate that data model We kind of selected GraphQL for doing it on the front-end side for all the front-end clients And it's great because we can all you have to do is know how to navigate the data model and you can effectively make up your own Prease whenever you want your own API's for your use case, and it's very optimized for your particular use case There's a smart query engine that someone spent a lot of time in trying to figure out the most Optimal way to pull the data from these services for your particular use case Which is a lot better than trying to handcraft an API Especially if you've got lots of different potential customers and trying to guess all their different use cases and build a flexible enough API for it By the way, we've done this We actually have internally a number of API's that we built that are very flexible that do serve a very broad set of consumers And you know what they all look like?  They're already there We just built our own handcrafted version So it's much easier to use something off the shelf and then you and you don't slow yourself down Similarly on the back end. We've been working towards using Kafka streams and stream sets and more recently ksql Provide a standard querying layer for any any data that's moving in flight through streams Which we'll get to more than a little bit. So the one catch with all this is that in a true distributed system You know, you're gonna have schema changes and for a query language to make any sense of whatever is going on You're gonna need a consistent schema So you also need to deal with the challenge of schema versioning when someone changes their part of the data model You don't want to disrupt everybody else in the system. So we've Employed a few different technologies in order to help us with that probably the most obvious one is Schema registry for that Kafka has and and Avro the serialization framework that sort of automatically handles changes in schema versioning We also picked a number of database tools and technologies that are Very comfortable with the notion of a schema evolving in distributed fashion because that's the tough part, right? Schema doesn't change instantly You roll it out in one place and then it slowly percolates through the system and you have to handle all that transition without breaking anything and Then the the other catch about this as well You know, that's nice to have like a generic query engine and I'm creating data But like that may not always perform well the data may not be Organized and set up in a way that will be efficient for my particular use case Especially if I'm doing something totally new that no one anticipated before which is the goal here, right to make everything as agile as possible and the way that we address that is because What we'll do is we'll rematerialize the data for a particular app to get its performance. So we might have Customers organized by I don't know something crazy like last name, right? But now we got people logging in with an email address We need to look up on that But we'll just create another materialized view of all of our customers that is indexed on the email address So that we have very efficient look up for that case. That's a toy example, obviously But you can imagine the more complicated use cases So next piece composable functions. Oh, this is so important and and and so counterintuitive for a lot of people, especially if they're working in Working with microservices a lot and distribute computing The most reusable piece of code you can have is a simple composable function That's more useful than a microservice. That's more useful than a restful endpoint that answers a query It's it's it's the best way to package up your business logic It's better than a framework for for executing on your business logic and the simple reason is this It's the smallest piece of work. It's the least amount of work, right? You have left all a lot of other decisions for someone else to work with And to give you an example of how this can go sideways Let's say that you Have a e-commerce site like like we do and you want to be able to compute taxes first when someone is about to Buy it by your product You could have a function for that that you could plug in anywhere you want your system that computes taxes, right? It has all the logic for it and just it's a library You link it into your code you execute it or you could have a microservice because that would be you know The typical way to do it have a microservice out there that's deployed you query it you tell it Here's what I got tell me what the taxes are But if you were to design it the way things send it working you might go Hey, there might be a lot of stuff that I'm asking you to calculate the taxes on that be a lot to send in the request so why don't I just give you a reference to the shopping cart and then you'll go to the shopping cart and Query it for what it has in it its contents and then you'll calculate the tax and then you'll hand that back to the system Now you've got this interesting game going on that makes perfect sense in the one use case of someone checking out Right you put stuff in a shopping cart you then calculate the taxes for it you show them the page you move on that's great What if the the opposite is true though? What if you're like trying to take? all of the products that you have and all of the people who are looking for potentially might be looking for Your buy your products because you're doing a marketing campaign and you're thinking that everybody in the system is going to be very Price sensitive so you want to figure out in advance how much you're going to charge them for any particular ticket? Well now you need to build a shopping cart for every person for every product that you have That's probably going to a database somewhere because they don't want to lose the state of your shopping cart in the middle of a checkout Flow and then you're going to send out a million queries out to this one poor little restful service that's probably running on one machine because it doesn't have to do much work and It's going to have to answer these million queries as fast as it possibly can so that your system can now make a decision about Who do I send the next email to? Right, so if instead that function was pulled out of a microservice and just a library that you could link to you could Ask that question as you're scanning through the data in your in doing your marketing campaign trying to build up your marketing campaign So yeah, this is a weird idea if you just make your function As a library and make it available everybody you actually get something more useful I do you want to emphasize though the importance of composable The composable function basically the golden rule of a composable function is that for the same inputs It will always produce the same outputs That means you can't be a function that goes out and checks for the database somewhere It can't be something that has an internal counter in it that does some kind of odd thing You need to reorganize your code so that your business logic is always represented in an input Therefore this output kind of state and of course that makes a lot of sense right in a distributed computing Environment the toughest problem is managing state right if you can separate that logic from your business logic you get so much more Flexibility now your business logic can change Without having to figure out how to move data around let that problem be solved separately for the particular use case involved Yes bulkheads and circuit breakers So We were talking about frailty before right Most obvious thing to do in a distributed system to ensure that Everything goes down is assume that nothing will and write your code accordingly right, but you know reality is hey might lose Might lose a connection to our database Okay, well we could solve that by reconnecting as fast as we possibly can to the database that probably go well Right, and that'll work out great until the reason why you lost the connection database as the database is being overwhelmed By people trying to connect to it So you get into a pretty nasty mess So instead you want to have something that's more practical and you want to think through the process of what actually should I do? When I have a failure in the system should I rate limit how much I'm trying to connect to the database and just accept The fact that everything is going to slow down should I Continue to function without that particular system right maybe there's a way that you can operate like Classic example is you might have a Commerce site that is doing recommendations for additional products that someone might want to buy When that system goes down when the recommendation system goes down That's no reason not to render the page that shows someone what they already bought It just means you can't make recommendations for additional things for them to buy The odds are they're going to want to buy the thing that they already got in their shopping cart Why would you take out that whole system and not let them buy just because you can't make good recommendations for additional products for them? Right, so you need to plan your product around the possibility that these various services might go off Do and and have strategies for recovering when they do come back online. That's the other one That's always important is the oh that reconnect problems really bad. I solved it. I won't reconnect Well now if there's even a slight network glitch for a moment Your system goes offline until someone reboots it because it won't try to reconnect to the database ever There's some really great frameworks in this space one of my favorite ones is called resilience for Jay That lets you sort of declaratively talk about How you speak to each of your endpoints and what the strategy is that you want to employ to deal with failure And that's really the best way To tackle it because writing the code yourself means you'll tend to introduce a lot of bugs It's much simpler to just to clarify say hey, I want to have a retry mechanism By the way, I know it needs exponential back off And by and by the way, I should never be hitting it harder than like, you know 10,000 requests a second because that'll definitely kill it. It won't be ready for that And so you can just declare that and it will manage that process for you in a way that doesn't create new bugs The name of the Library was called resilience for Jay So there's a bunch of other libraries like that for and for a wide variety of languages The thing I like about resilience for Jay is it tends to be pretty lightweight a lot of Java frameworks Have a tendency to get very heavy resilience for Jay is pretty lightweight and focus on just trying to give you resilience And giving you a way to employ different strategies for tackling And Then here's here's the one that actually in a lot of ways I'd like to move to the front of the line but from a practical standpoint I knew I had to roll out after some of these other things are happening and delivering some important wins for us And this is my favorite topic because I a lot of my work is in data science And it's stream all the things and and it literally means exactly what I'm talking about every time There is a mutation of any kind in your system You want to publish that data out into a stream? We've been using Kafka as the tool for getting that done And the rationale here and in the cut the parts of Kafka that are really important for this is that it's a Pub sub model Which means that the consumers of the data are completely disconnected from the publishers the data That means you're publishing data But nothing that the downstream consumers do can break you in fact It's not even connected to you so even if someone does something stupid and deploys like 10 million new Servers that all try to consume the data that you're publishing that might cause some stress for Kafka Although it's very resilient to that case, but your system that's publishing the information they're getting to sees no additional load You push the data out once doesn't matter if nobody's listening to it or everybody's listening to it It's the same workload for you your system doesn't get stressed out when other people make changes But the but the key thing is and this is really the opposite of a lot of the design precepts that are Commonly found in microservices where the idea was every microservice should have its own database its own data store And that should be isolated and no one else should be able to access that data store You always have to go through the microservice and in principle There's a lot of good reasons for doing that But the downside is you start breaking up your data into a little tiny piece of spread throughout your system And there's no way to consolidate at all and get an understanding of what the heck's going on, right? But if you publish the data out, so it's available for other people now you have a lot more flexibility in your system Why tomorrow want to combine the information from three different? Systems to make something new that no one anticipated before It's very easy to just connect the three different streams that those systems are producing Join together the information and then execute on it Okay, instead of having to call out to these different services possibly through interfaces that never anticipated how I wanted to use the data And and and make sense of it all without knocking any of them over So you can see another common thing here is we're we're we're moving the data away from the business logic We're moving the state away from the business logic and so you can Decouple those two problems the distributed system and your business logic And Finally, and this is the hardest one that we've only started touching on just a little bit Is anyone familiar with the call me maybe website for distributing computing? Yeah, it's some great great blog posts if you're interested in distributing a lot. I highly recommend reading those blog posts But yeah, there's there's this Need we have these composable functions now right so we can connect them all together and rearrange business logic and do whatever we want But we also need our data structures to be composable so that we no longer have to worry about what happens when there is You know some server wakes up that went to sleep or went offline or crashed And it's got a bunch of transactions that it had processed like three days ago That finally get pushed back into the main system and you know if you've done if you've got a transaction processing system I'm sure your transaction records will still have full integrity You'll know that there were ten new records or ten new purchases that happened three days ago You'll get that right, but you don't want your metrics on your data warehouse to suddenly show that you got ten new sales three days later or to suddenly Freak out about the numbers that it had from three days ago You want everything to to just arrive at the consistency that it should have been in the first place Even though the data arrived out of order Or potentially even in reverse store So this is where this notion of conflict free replicated data types come from and And the win with them in the conflict free part is very important It means that you can have multiple copies of your data throughout your system without having inconsistencies in what the truth is it's always possible to figure out what the truth should be it just depends on Whether you've got all the data or not what truth you see but there is always a true an accurate truth at a given point in time and these are There's a whole bunch of different strategies for this So this is actually from that Jepsen.io is the site where where this information about CRDT's is up There's a diagram from there that sort of talks about the taxonomy of different approaches to achieving CRDT's depending on What outcomes you're looking for for your particular system and believe me? You got a lot of choices here because you're trying to achieve consistency Which means you have to have different levels of flexibility, but probably the simplest Model for CRDT to think about is one. That's actually commonly deployed in in a couple of eventual consistency databases Which is simple rule of last writer wins, right? So whoever has the newest timestamp, that's always the truth or the last thing that happened but it's important to think about what that means in a in a more sophisticated kind of context where if I have a record that shows up with an older timestamp That is actually just updating two fields Right and the most recent update was updating two fields as well But they're two different fields. I should be able to combine those two operations, right? And it's a very easy way to solve it Which is you look at the timestamps on each individual field for each time they get updated And you just say the most recent one always wins and then it creates this interesting Context where when you look at it, it actually looks like all of your rights for atomic They either happened or they didn't in every context in reality under the covers They didn't it was just all comparing timestamps and everything just happened to work out beautifully But this ensures that you can avoid having a completely Dispart system when you're dealing with that multi master replication problem that I was talking about There's always a way to resolve those multiple masters. You don't need a lock anywhere You don't need some arbiter about the truth You've got a set of rules in place that ensure that your systems never go down and at the same time Because you don't have a master anymore if there is a break in the network or someone Destroys an entire data center because I don't know what was the thing that happened to Delta They were testing EPS took out a whole data center That's that's not a problem for you you lose that data center for the moment All your other data centers know how to function with the data that they have and when the data from that data center comes back Online whatever you can recover It just magically blends into your reality without you having to write special case logic for handling that scenario and that's I Don't know about you guys, but I've built a lot of systems with a like single master where I have to deal with about 20 different failure cases and writing all of those branches and code and Testing them and making sure that they all work correctly is Way way harder than just having a simple eventual consistency model that ensures There's no conflict and I get to a truth at the end of the tunnel I'm down with Mongo. He says yeah up with Cassandra, right up with react down with Mongo. Absolutely Mongo's the framework that teaches you the importance of CRDT's So so the fun part about that though actually is is realistically if you build your own CRDT's Right, you can actually use systems like Mongo and not have a problem when there is a failure Because you've got eventual consistency in your own data models The only problem is you had to do a little bit of work to do that It's nice to use a framework that has that built in but you can realize that you can use this stuff with like an Oracle database and it will make handling replication lag so much easier Okay So there you go. That's the last and the hardest one And and all I can say now is like you know safe driving Enjoy the distributed computing route. You've now got a system that Will be resilient to failure that you can make changes and understand the consequences of them before you roll them out and Measure them after they're rolled out you can make decisions about How you want to reorganize your business without being Frustrated by the existing structures that are limiting how you want to want to go about your business and you're into a free and open distributed world So that's my talk if anyone has any questions. I'm happy to answer them now Thank you very much. I Think the dog had a question, but we're gonna skip over him, but you had a question over there. Yeah J E P S E N dot IO. I believe yeah Yes, so the question is You know, this seemed like a pretty Carefully thought out process for rolling it out Rolling out these kinds of changes in an existing environment But if you were building out a new environment, would you do would you roll things out in this order? Was it did I get that right? Yeah, so um, no actually yeah, I would do it differently if it was a completely from fresh environment I would actually approach it differently so I Would I would definitely first of all you would try to start with all these things honestly like if you're doing it from scratch It's a lot easier to just do all the right things So I would say that distributed tracing if I knew I was building a distributed system from this get go I would make sure that's there before I even start doing any work because it just makes all the work after that Possible actually, I'm not even gonna say easier. I'm gonna say possible because it's it's that bad when you don't have it The other thing that I would be tempted to do is the streaming part. Okay, right from the get go It can seem unnecessary when you build the first component, but as soon as you have a separate component That's in a different, you know different distributed context Things start to magically become easier when you work on a stream model and there's other benefits that come from the streaming model as well it allows you to Precompute things in advance so that at the time of request you do the least amount of work to resolve the request Which allows you to handle scale a lot better So there's other side benefits that come from it that make me want to make sure that I'm doing that earlier on the process and then and then Although bulkheads and circuit breakers are super important and a really really good idea The funny part is you can actually get to those later because when you're first building your first set of components You don't have enough components that egregious failure in the one component. It's gonna take down a whole system It's it's gonna be Terrible, but it's not gonna be the worst thing right your failure rates not gonna be high But once you start having you know 10 or 20 components now your SLA is even if each component has a 99% SLA You start doing the math on that and it's like now your uptime is like 89% Right and that's not good So you can wait on those until a little bit later and that might help you get out the door a little bit faster and then the and certainly CRDT's I would try very hard to put those in right from the get-go They are so much so they're hard to get your head around if you've never worked with them before Takes a little bit of time It's helps to have frameworks and libraries that do it for you But if you can put that in from the get-go it reframes how you think about Architecting your data model and your systems in a pretty significant way And so it's good if you can do that in the get-go instead of having to like sort of refactor all of that infrastructure at a later point Does that answer your question? Yes? So the question was you know are there cases where graphql What what are the cases where graphql makes sense does it make sense in the back end? and and also if I heard you correctly like a Lot of API's seem like they make pretty pretty decent sense by themselves And they're really independent and you don't see the benefit of gluing them all together through like graphql or query engine So first of all, I don't highly recommend it for back end a lot of back-end stuff It's possible to do and it works But there is a lot of overhead in translating everything into JSON and going through all the resolvers That you might might not be ideal a lot of back-end systems It's really important to be extremely efficient And that can be problematic And also a lot of the goodies that are in graphql are really designed to help you when you're going over a WAN as opposed to a nor traditional back-end environment where it's a LAN, okay? And so yeah, I wouldn't say that it's for everything on the other hand when you're building all those API's that are doing different things you tend to find that a whole bunch of them are like crud operations and And guess what? You know if you just build a data model use use something like graphql and really it's not about graphql specifically, it's about the idea of using a generic query engine You suddenly find that you're not Having to build a whole bunch of API's that are really just boilerplate. They're really not adding any value But they are creating risk right they're all ways that you can screw things up and also ways You can make it more difficult to change how you do things so Classic example is you might build a retrieve service that returns back this nice big object, right? That has all the information about I don't know some order that was put through your system, right? But there might be a use case where all you really want is to get the total amount of money spent from all the orders in The system and if you use that API to get that information back now You're pulling back a huge amount of data creating a huge amount of load on both systems Just to get what should be a very simple query answer and by the way sequel is seriously a legitimate solution for that There are sequel engines that can translate onto data models in multiple different systems You just create that way we chose graphql in particular because It fit well with our strategy with react on a lot of our front-end pieces And it was very it's very beautiful in that use case in that use case. I would really go strongly for that instead of Yeah, it's really great for that and and the nice the other nice thing about graphql is it's not purely a query based model It also has subscriptions in it so you can do the push based model So it can fit in well with that streaming part of the equation where you can say, okay Now I want to know the moment that a new product comes out in the marketplace that I know to do a whole bunch of Things in reaction to that but yeah, no, that's that's why we're not using it all in the back end We have things like ksql and in in stream sets and a few other tools that allows us allows it to generically work through the data You had a question back there so the question is how do you approach the Question of functional testing in a complex environment where you've got all these different pieces moving together So the short answer on that is very carefully No, so so if you take that approach of composable functions it actually helps a lot because you can test composable functions very easily independently and Also in aggregate together because that's the whole point that can be composed together And so you can test what happens when you put them all together in a chain that works very nicely Where it gets complicated is composable functions the whole thing is you don't have the state piece of the picture involved, right? And in reality all these systems there is state that needs to be managed and that makes testing more difficult, right? so The most important thing is to have a good framework for generating a state actually in the system consistently. I'm a big fan of Essentially using a declarative model to populate your system So you'll say something like I want to create ten orders that look like this with three different Users attached to them and I want special case where one of those orders we detected fraud, right? and then have the test framework generate essentially random data that is In compliance with that declarative statement, but otherwise completely unpredictable and and exercising all the corner cases That could exist your data you shove that into the system And then you look to see when I create the system now does it reflect the state I would expect it to be in one all of those things have just happened And it's really important to build a robust Integration testing framework that works in that fashion instead of the more traditional model of individual imperative Integration tests that are like just one particular functional piece of the system. They just don't catch enough of the problems Does that make sense? Did I do it by person-by-person basis? Is that what you said? Right. What happens when one service depends on ten others? So in an ideal world and if you're doing it the right way those ten other services should be Able to be spun up with basically a single button push right in a test environment And so the right model is actually yes when you're building your tests it should spin up those ten different services that you're talking to in your test environment and Run the test against the real services not some mock Because most of your integration challenges come from making bad assumptions about how those ten services behave and Getting a surprising outcome in the real world And so you want to test against how those systems actually behave And then you shut them down after the test of right, right? So that's the real idea. That's the ideal world We don't live in that ideal world And so what we end up doing is we have Qa environments and pre-prod environments where we have some guarantees about the Approximateness of those environments to production reality And those services are available for anyone to do testing against at any time I will say that we do have challenges sometimes with Multiple people independently wanting to run tests and plot pull out different pieces And that's where it's so much better if you can just spin them up on an as-needed basis for your dependencies Makes sense Sorry, there was a question over here. Yes Yes schema versioning. Yes So there's a couple of other so the question was what else to be used to manage schema versioning and the evolution of schemas So it's a good question. We're actually in the process of reviewing the tooling that we use for that and enhancing it We've been using a Couple of in-house tools along with some open-source solutions to get us to a good place there We're using a product called a secan ckan that Is a sort of data dictionary of sorts that provides us with insight about all the schemas that are deployed in our environment although We're still getting all of them integrated into it. So not all of them, but in principle a bunch of them And the idea is when a deploy happens it actually Part of the deploy process is to publish a message to a special Kafka topic that tells the data dictionary Here's the new version of the schema and push that Into a central registry where we can capture not just the fact that there's a new version But that there was a transition and what the previous version was you can go back through that history and manage it there's a couple of other tricks that we try to employ on the on the data science side like Representing everything as a feature vector so that we're sort of new fields just show up and it's not a problem but But there's a lot of sophisticated tools out there in the marketplace That do about half of the problem and the challenge is cobbling them all together To make something that makes sense in aggregate and I haven't seen a lot of good products that actually Handle all of it all at once seed you have to invest a bit of time and effort into it So case in point Cassandra which we're using in a couple of cases it actually handles schema versioning very well in distributed context You can make a version change online it ripples through all the clusters all the data centers that you have Cassandra deployed in a very reasonable fashion that doesn't break anything during the transition In between that's great. You've solved the problem for Cassandra But that's absolutely no help for anybody who was using some kind of tool to that was expecting a specific data structure in Cassandra at any particular point And say it was like pulling out all the records from a table and putting them into a spreadsheet somewhere, right? The tool is gonna be like wait You've now got another column that I didn't even know existed a few moments ago And could quite well break if it wasn't built to handle the reality of schema versioning But the biggest trick is tack on a schema version on every payload that you have moving around Because at least that gives you the possibility later on of figuring out that there's a different version and how to resolve it That's basically what the schema registry does right? It just tags everything with a schema version and then keeps a history of all the possible permutations Yes, another question there. Heather's smiling at me there at the back so We've gone through a number of different iterations on this Been around for 40 years and we've we've Attempted the attention organizations to try to impose governance and then recognize once you impose too much governance That you're creating too much friction and that no one can move forward very efficiently So then you compensate and go the other way and now you have Jungle of different technologies out there that you got on tangle and then you compensate for that by going back to the Crazy levels of governance and so forth and so forth So we've been through a few iterations of that I would say right now. We're probably closer to the jungle than we are to the tight governance model but we are taking an approach now that's fairly pragmatic of sort of recommending and guiding engineering teams as they go to solve a particular problem and encouraging them to sort of consult with Set centers of excellence that that can guide them on making the right decisions In the right places and one of the key things that we've been trying to do is ensure that there's someone who's the title that that we use as an architect involved in that conversation because There can be this dangerous tendency to think that you know what kind of problem you have Go to the expert at solving that kind of problem and then they solve it that kind of way and it's actually the wrong problem It's that they'll joke about like you know Don't go to a surgeon for asking for treatment about something you have unless you already know it requires surgery to handle it because the Surgeon will always tell you you need surgery for this, right? So actually I've seen this problem in a lot of different companies with Dba teams Dba teams are always the experts about databases and data People Developers love them because they can go to them with almost any problem and if it involves data magically it gets solved for them The problem is is that not all data problems in distributed environment are database problems some of them are distributed computing problems and if you go to a database team to address the distributed computing problem you Actually end up with a solution that doesn't really work because you haven't addressed the real problem the database will work great But the actual application will not And so that's where we try to have these Architecture experts who can help guide you maybe not to how to solve the problem But what kind of problem you have and who would be the expert to talk to to actually figure out how to solve that particular problem Does that make sense? Any other questions like we're done. All right, cool. Thank you all very much come back for the next talk we've got The next talks on zero MQ and we've got a graphical talk later if you want to get a little bit more feedback on that Thank you very much. Oh Okay, there we go. We have Welcome you can be my guinea pig while I make sure everything works Is it too loud when I when I got it turned on I suddenly go whoa the voice of God is coming down All right, good. It'll probably quiet down a little bit when there are people in here But I would hate to blow to blow people out especially if I you know get excited and Talking louder. All right, so that means we're basically ready to go. Oh Yeah, there we go So if you're inclined to play the home game you've got tons actually have more time than anybody else But it's entirely up to you. It should be self-contained without it. So yes Excellent is my pointer visible. Maybe is that visible to you if I use that? Okay All right. Well, then maybe I will I think I'm ready to go Looks to be working well. In fact, I now have the voice of God This thing is kind of loud. I think it'll be all right once people get in here, but I turned I was like So It looks like all is well though, okay, great. Thank you. What's that? Well, they might have to work at it It's you notice I'm gonna actually do it in there or at least I'll take it off at the last minute It doesn't it's not great to give a presentation in a coat But man the speaker's room is at least this cold and here I was hacking all morning polishing things and And I needed hot chocolate. Welcome to the North Pole I feel compelled to not give a talk in a jacket, but the rest of you should probably find your park is because it's quite I Guess the machine. I guess the heart the electronics will run cool But so I appreciate if you make it through the entire talk. I will appreciate your amazing dedication or you'll frozen to the chair I don't know which Yeah, I think it would be good to take the edge off that If it's any consolation I spent all morning polishing thing in the speaker's room, which is easily as cold So we have the speakers on ice today If you didn't notice there is a home game if you want to play along the slides that I'll be giving because I just pushed so I promise they're actually up to date and the example code is in the repo there and And for bonus points in the home game Tell me what my t-shirt is actually about There's a puzzle But afterwards Because I'm gonna try to try to stay on time. I think we have about two minutes Okay, great. Thank you although If we get much lower they'll freeze to their chairs and can't leave in the middle even if they're bored and I mean You know, it's it's win-win, right? for me But I have to give a more animated talk so I stay functional. All right. Well, it looks like basically it's time to start The there is a home game if you just came in there's a URL there if you feel like Cloning the repo and playing along feel free the slides are there to people I took me a while to realize that the best thing Do with the slides is to stick them in the repo so that you can pull them rather than me pushing them. So All right, let me start by time right here. Okay, so we're gonna talk about zero in queue Basically, what we're gonna do is we're just going to connect all the things. Okay So first of all the name is rather confusing It seems like it really should be a message sir a message queue or something Because after all the name is just like rabbit in queue clearly. It must be related. Yeah, it's not really I mean, I suppose you could use it to write such a thing, but it's not in any way a server Nor is it even a centralized message broker Even though I'm Maddox is also, you know heavily Or was heavily involved with a MQP and It's not actually for existing protocols. So you don't use zero MQ to say write your HTTP server You can bridge messages, but that's not really I mean, that's not what it's designed to do Okay, so what is it now? For those of you that are young enough that you grew up with social media I have tried to distill the description down quickly enough to match the modern attention span There we go Now for the rest of us There are a number of definitions floating around a reusable messaging layer is actually pretty concise It is asynchronous unless you choose for it not to be And it can look like a concurrency framework. That's kind of a tall order for it for something that's not totally specialized I'll play with that for a minute Now the documentation part of the reason for this talk Or at least what kind of informs my thinking is there's a guide that seems to be the Bible the guide is a pretty good Tutorial it's very thorough. It's a terrible reference Okay, because it's written as a tutorial and the author writes in a very friendly style And so you can't find them when you're looking you find them as you get to them also the example code is pretty Variable I honestly kind of despise the C++ the way they wrote it, but what I want to do then I Want to basically illustrate just how flexible and easy this is to use and especially get over a couple of conceptual Humps that I think would have made the guide a lot easier for me to read if I had had them I'm going to mainly focus on the socket types because that seems to be kind of the core of your thinking and maybe we'll look just briefly at the threading performance since they actually talk you can use it just for pure threads and Basically, it's built the structure. It's built around three key objects. Okay, so you have messages you have contact you have a context that you'll usually only have one and You have things they call sockets which look like Berkeley sockets a bit They've met the interface looks like it and that's only to confuse you it's sort of vaguely true if you squint but they're much different much higher level and It's so don't expect the semantics to necessarily be the same. Okay now We're gonna start with a simple warm-up exercise What we're gonna do is we're just going to have two threads talking over What amounts to a channel something like you'd have in go or rust? Basically because it's just really really really simple and they're actually gonna be synchronous just because again I'm really only illustrating how you get the library up and running So there's no need to actually write asynchronous code So first of all if you're playing the home game In the repo I should have it on the top of every slide for people who come in late. It never occurred to me That would have been a great idea Basically, there's a very stripped-down code in echo pair dot cc Now the funny thing about this is or at least when you look at it All you really need to know is the main function is going to function as a client meaning it initiates requests Okay, and it creates a server thread to respond to those requests. Okay. Now. How does this work? Well, this is the way everything works in zero MQ.  You're gonna create a single a context It's almost always going to be a single one for an it for each process And it's the only thread safe thing in the library You're not supposed to need thread safety for anything else because if you're passing them between threads You're doing it wrong, but the context has to be shared. So it is thread safe and re-entrant and all that stuff Basically after you have a context you use that to create sockets There are at least 11 different kinds of sockets and you have to specify a transport and What you share is as I said not the sockets you share the endpoints and then each thread connects to the endpoint It needs to talk anywhere else. So this is what this looks like in code. Can you see my pointer back there? Is this useful? You can see it. All right. Good. Okay. So again, this is echo pair dot c CC so everything they night in the C plus plus interface they put everything nicely in the Zmq Name space. So here's a context. I tell it how many IO threads. I want normally one They say you probably one per gigabyte per second of data. So one will be fine for us now You actually don't need any for this program because we're not using the network and so in fact it's direct, but Normally this is almost always be one. Okay, so then what do I do? Well, I create a socket with the context and I have to tell it what type it is So this case, it's a pair which is the least used socket, but it's actually the easiest to understand initially I need a an endpoint and the funny thing is even though this has nothing to do with the network How I specify the endpoint looks like a URL Why because it has a single unified interface in proc just means in process. It means it's inter thread only Okay, so we have to bind. Okay. That should feel familiar And then I create a server thread here You I'm not going to show all the code in anything because all I want to show you is the parts that matter And I'm passing it the context and the endpoint it should connect it. Okay now Here's the funny thing Semantically main is going to be the clients can initiate requests. Why did it bind? Okay, so this is the first place you realize. Oh, these really aren't Berkeley sockets They're only trying to confuse me First of all it actually doesn't matter near as much if you like using sockets This matters to you a great deal because you can't connect to something that's not there But in zero MQ you can okay because they just in the background remember we gave it an IO thread So one of the things it does is in the background if it just automatically reconnects if it's disconnected If it's not there yet, it'll keep looking and in the meantime just queue up your your messages So you can do almost everything in either order and no one cares Which is really really nice when you have a whole bunch of processes that are coming and going for any number of reasons And that's where we start to see the ease of use So the usual recommendation is the usual rule is it kind of looks like sockets But it's more of a of a principle than a binding rule You bind for things that stay in one place that have well known Sockets well know or well known ports well known addresses and you connect if you're the sort of if you're a client or something That can come and go The only difference is in proc is different in proc bind has to happen first and that's because It bind creates the message queues that they that they talk with okay, so pair sockets are also kind of special They're not like the other sockets in the system. They're they're They're almost like Berkeley sockets You can only have one on each end meaning they act like a Berkeley socket Yeah, they have some handy properties that make them basically a channel basically a message queue And really their only real purpose is to do exactly what we're doing here Okay, so what did we do in the child thread? Well, obviously we pass the we get a contacts We create our own socket we bind to the same end point Another rule in zero and cute not all sockets can connect to all other sockets The types have to be compatible pair sockets only match other pair sockets. That's as I said, they're kind of unusual So here is what the the thread looks like. Okay, so here we got we received the context I hope it's obvious if it's not obvious to you why it's a reference. You're probably not a C++ programmer and it's okay Here's the so what do we do? We just create a socket we connect and we're good to go. Okay looks kind of familiar There's a just I had to do a little slight Synchronization since I wanted to time this I didn't have to otherwise it'll wait until everything's up, but I Send a blank frame so that the in other words the the supposed server actually initiates a Frame so that the server knows okay now start your clock and start going okay So basically all we're going to do is we're going to sit here.  This is synchronous Which means you know it has its limitations But again, I really just want you to see how we how everything works every program is going to do basically the same thing Okay So here's what the server is doing obviously it sends first it sends, you know something important, right? And then it receives the answers Okay, so the real points are you've just seen the basic usage you see it's very simple It almost looks like sockets, but if anything it will just work where Berkeley sockets would punish you if you didn't do it right the by the way the s receive an s send functions are convenience functions. I just there's a Z helpers file in the guide You can download all the example code for the guide And it's also a nice way to see how to sort of make it easier to use the message objects aren't as nice as just using strengths and You use you often don't need them so now performance now This is a quite an old laptop, but still 80k message is a second not not super fast Okay, so just for fun and this is the only timing I'm going to do I wanted to see how it would compare with doing this with something that specialized for inter thread communication Okay, so first of all I wrote basically the world's most bare bones Q protected with a mutex lock and I got almost double the speed Okay, so all right. It was kind of slow But then if you do it right and you use somebody's Well-written cue I got about 25 times faster. Okay, so you might say well wait a minute This is supposed to you. I thought you liked this library. Okay. Well first of all I the timing is Probably only vaguely close but a factor of 25 should stand out from the noise and actually all the implementations there They're almost the same code. They're nearly the same length except writing a cue. So why would you use your own cue? Well, basically I set you up while I was just showing you how it works. I set you up That's probably that's a secondary job for zero MQ and it's probably the hardest one for them to do very well because Everything fits together. So everything has the same interface. Everything just works So I was really playing to its weak weaknesses. Okay, it's not actually intended for just using it It's not MPi and open MP are safe. We will not be running this on our supercomputers simulating the weather, right? Okay, but just think about that code for a minute. What if we wanted to have multiple clients? What if we wanted multiple workers? What if we wanted them running on different machines? Different processes. What if we wanted to do some pub sub now all of those would suddenly bloat the code up Enormously, okay, and would be not fun and it wouldn't reconnect when everything goes down Whereas as you'll see it's actually quite easy to do here. Okay So that's the problem with specialty tools and that's really what zero MQ is really designed to do Okay, to avoid all the code to do da da da da da da da. All right So let's look at something that's kind of real. Okay, so this is going to be a very simple load balancing server This is mostly in the guide. Although I've done some different things with it, but What I really want to show you so it's in load balancing server dot CC it is just what it says on the tin, okay? and What do I want? I want to have any number of clients. I want any number of work workers And I want load balancing that will actually accommodate slow and fast Clients and workers so I can't do round robin, right? Okay, so The clients are actually even simpler because they don't need to send ready because I'm not profiling it and and if you look at The example code I kind of print some things out so you can tell it's actually working Which means that the threads are slowed down to the speed of The console right so we obviously don't care about that yet Why am I okay? So the clients are all going to connect why because they're the ephemal pieces the server is the has the well-known the well-known address The only real difference is now the clients are using what's called a request socket Rather than a pair socket. Okay, so here's what it looks like remember. This is always the same context socket type Connect. Oh and those of you that are not c++ programmers. This is pronounced forever That is my prefer you can do while one if you really want to but that stands out and you go What in the heck is that and then you realize oh, I've got a I've got a potentially infinite loop here I hope that's what I wanted. All right, so what do I do? It's pretty simple really I can send things I can receive things and Let's see. Oh That's right. I actually gave you a Python example. So that's another thing, right? It's very easy to do things in different languages because the bindings all work similarly So let's do the same thing in Python and this is in load balancing client.py. Okay, I'm going to create a context The syntax is different, but it's exactly the same thing now. I'm going to create a request socket I'm going to connect it the endpoint is all the same and then since I can't do forever It's while true. I'm going to send something I'm going to receive something the only tricky part here is that you do need to send binary data, okay? Because it expects binary data and not utf8 Okay, so what about request sockets, okay? They're the simplest way to do To do excuse me to do request reply They're synchronous So you actually have to call them there like a state machine that flops back and forth, right? Request has to go first Then it has to receive something before it can send again and so on That makes them kind of fast and simple to use you don't have to do things Synchronously, but if you want to these things are very simple, okay? The server however isn't going to care So the server will be is going to be asynchronous and it will happily support asynchronous clients I just didn't write one for you And they use a different socket they use something in this case for a request the equivalent is something called a dealer socket Which has extra bells and whistles and comes with a five thousand five year or fifty thousand mile warranty Unlike the request socket, okay? So almost every socket type in zero and q comes in pairs Why are they in pairs because each one of them implements a specific pattern and the patterns are asymmetric Except for pair so ironically pairs the only one that doesn't come in pairs, right? so if you want a Request reply pattern, then you've got rec and rep for synchronous Work and dealer and router for sort of the full Monty, okay? You've got pub sub for pub sub you've got x pub and x sub which you only use if you're relaying really And you've got for you've got push-and-pull sockets for making pipelines right data splits out gather it in So these are really all the baked-in patterns Okay, and you'll see in a minute. You can do odd things with them, which is part of why they're useful But they're specialized right so these are bidirectional they obviously have to be and Router for example knows how to get messages back to the origin pub sub push-pull and obviously unit Unilateral or sorry unidirectional Pub sub has some kind has to have some kind of routing logic push-pull does not so again What you're going to do with zero IQ is you're going to think through the patterns and figure out all right Which Lego bricks do I assemble for what I actually want to do? Okay, and we'll see a little bit of that Okay, so what's the server going to do? Well the server can't use a reply socket because it better not be synchronous And it wants to connect to a lot of clients So it has a router socket, okay, and the great and now one thing that yes, it's coming up I'll show you well. This is where we start to see why this is a whole lot easier than doing this the hard way Okay, first of all, I don't have to select and pull even though. I'm going to connect to multiple endpoints Okay, I don't have to do except this just happens somewhere on a magically I Don't have to do different checks for things that don't have a file descriptor it just works Okay, so how does that look well alright, so of course I create a socket with a context I give it a type. What am I doing here? I'm just bind bind bind bind. I can just keep doing this I can bind to whatever I like Which is again, it's not really Berkeley sockets. It's just much nicer. Okay, and I'm binding to an in proc Endpoint so there isn't any There isn't any file descriptor just to pull on but it doesn't matter It's going to act as though there is it's all just going to work I this is IPC the way they implement IPC as this is a actually a unique socket and practice So the guide says it doesn't work a Unix domain socket So the guide says it doesn't work on Windows, but it's I believe Windows 10 might now have Unix doing does so this may well work But the guide says no because they wrote it back in the Windows whatever days single digits of some kind and then of course This is the real thing. I'm going to bind it to local host, but you can bind it to whatever you like Okay, so how do we route things? And this is where you start to see kind of the logic of using this for less trivial problems. Okay So, you know, we're doing all right.  Okay, so data comes in the review send a request data goes over it hits the The router socket on the other end and it wraps it in an envelope and the envelope knows the identity of who sent it Okay, so what you receive out of it is Not the original message, but the original message in an envelope that says this is this is where the reply is going and When you send outgoing messages on a router socket, it expects that envelope to be there And it will strip the envelope off and send it to the right one Because from it's secretly keeping a thread pool So it knows it knows where they went probably by which of the sockets in the pool came from but it will magically get there Okay, but you have to make sure if you do anything odd as we are going to do You will have to make sure that you give it the envelope. Okay, so This is where to know how we do envelopes. We have to know something about Zero MQ messages. All you really need to know is that they're Sequences of frames. I'm not crazy about that language because frames can mean lots of other things but what they mean is It is a sequence of chunks of binary data. They're just they're just binary strings okay, and That is all zero MQ has to know about messages, but it's enough to implement Envelope letter because all you have to do is just prepend extra frames with the routing data And they often like to use empty frames as delimiters, and I think they're like virtually free I think there is zero bite in the stream to be honest with you and Then you can manipulate the frames and make it do do what you want You could send it to a different a different origin if you were to put a different ID there because it is Transparent to you when you're working at this level. So this is how the routing works, right? Here's a request socket in a client. It has a client ID associated with it It sends a request when it hits the router socket what comes out is actually two more frames There's the client ID, and there's an empty frame delimiter, and then here's the original request So when it goes back, we have to make sure that this stuff is there Okay, and then here's a reply hits the router this gets stripped off the reply goes back to the same client ID Okay, so if you have that in your mind, you're halfway to kind of seeing what it feels like to work with this, okay? So now that was really easy. Okay, so how are we going to solve the load balancing problem? That's a little harder So what you would expect from what I've said if you've been listening you'd expect that the server is Going to send the requests onto the workers with a dealer socket, right? Because if a router is the equivalent of reply But with you know extra added features then the dealer must be the request and that's true But the problem is the dealer will just sit there and round rob it And that's not really what we want if we have workers that are you know sitting on another machine across the network and workers That are in threads so what we actually want the workers to do is to tell us every time they're ready Okay, now what's the problem with that? The problem with that is that means the workers have to send first okay So they can't use reply sockets because reply sockets can't send until they've gotten a request can they? So in reality the way to think about this is the workers are actually going to be making requests to the server for a job and their reply is Going to actually be Sorry, they're gonna make a request the reply that comes to them is going to be a new job so What the worker regard what is sort of operationally functionally a reply to the worker is the client's request? Okay, see how the language is getting tricky. This is why I said I wanted to get you over some humps The guide will make more sense if you already know this can happen, okay? We don't need an extra ready message because when it sends The the data is so what happens it said hey, I need some data come it gets it gets that So it got a reply from the server. So what can it do next it has to send another request so its request Functionally is going to take the reply and that's also going to tell the server now. I'm ready to go Okay, I do have a diagram coming which will make this a little clearer But the point is that the workers even though you would think that they are replying semantically They are going to use a request socket So we've inverted the sockets meaning in a sense and I'm gonna just say something about that There's a lesson for this is the socket types are named you would think they're semantic patterns Oh, I have a I need to send a reply. I will use a reply socket. That's not the case. They're actually What I want to say Where do I say this the point is that there's a difference between functionally and semantically I don't like this language I don't think it's that clear, but I couldn't find a better one what you do is you actually just think to yourself Functionally, what am I doing? Am I sending a message first? I'm a request I don't care what it means to some other part of the program if I'm sending if I'm only responding I'm a I'm a reply and so on so again the thing I want to keep clear is To use these patterns even though the type and the names kind of make you think that this is the semantic meaning It's really just the pattern of their behavior and you really just pick the behavior you want whether or not it means to you What it sounds like, okay? so The implementation is slightly hairy so in a moment We're going to see the diagram, but I'm just going to say what the diagram says so that you'll hopefully the diagram will make sense quicker So the server has to use a router again because remember the workers have a request socket So I need to reply to them plus I need to know who it is. That's ready. Don't I so actually router is perfect So what am I going to do? Well, I'm going to get the ID of the worker that just told me he's ready so I can store that I'm going to put it in a queue, okay and What I can do is Is is once I have something in the queue a worker that says he's ready When I get a request from the client, then I'll just pop it out of the queue and I'll stick the worker ID Remember I can wrap it up. I can send it to whoever I want Okay, and the router socket will strip that envelope off and then it will send it to that guy It doesn't matter that he's told me earlier because the router socket just going to do what router socket going to do, right? Okay And of course I'm going to keep the original client ID because I'm going to persist that because eventually I've got to get it back Okay, now the other thing is now. I do have two sockets. I have a front end the back end socket Okay, so even though I don't have to pull for you know a hundred clients on one socket I do have to pull for the two sockets, okay, and you can do that you can still pull You just don't need to pull on nearly as many things because so many of them can just attach on one Zero MQ socket, right? Okay, so here we go. This is the dreaded the dreaded data flow diagram So what's happening here? I get a request right what comes out of the of the router socket as we saw from the front end is there's a request But it's wrapped in this envelope with client ID and the delimiter So what I'm going to do is I'm going to pull a worker ID a ready worker ID out of my queue And I'm going to put another envelope so I'm wrapped put you know you guys all know the envelope Letter pattern and that's what's that is what's so great They gave you just enough structure to the message to do the most all the most important thing Everything else you use your own serialization libraries and anything you like this is all that they needed and that's Kind of the ethos of this library give you just enough to reuse without too much Okay, so now I've got something that has two envelopes and that's great because the router is good The back-end router is now going to strip off the first one. So I'm back to this aren't I but I'm back to this going to the Correct worker, okay, what's the worker going to do? Well, we still need to know what this is So he's just going to replace the request with the reply Send it back, right? So this gets to the router of course What happens well the router now wraps it up again because he always does so now we know who it was that just sent us the Reply, okay, and that's great because first of all I need the outer envelope to stick back in the queue So I know that he's ready for another job, right and then the rest of it is exactly what I need See how I have I now have the original envelope here But with the reply in it and now that the router the the other side the front-end router will now route that correctly Okay, so it's a little bit confusing But the fact is that's tremendously powerful because it's a whole lot easier than writing the routing code for yourself, isn't it? Okay, so how does that work? Well, so here's what a worker looks like that. This is the easier part What's he gonna do? Okay, this receive multi part here I did this is I just use the Python code for this one This is so you get all the frames at once instead of sucking it off frame by frame Because that's really easy. All I have to do is replace The final frame right I'm just gonna assign it here in this in the list or the yeah Python calls them list there are raised but they call them list and then I'm just gonna send it right back Because I'm done so the workers again are really simple and that's good because you only have to write the server once Right, but you may be writing a lot of clients and a lot of workers So the the complexity is in the right place just naturally. Okay, so what's the lesson here? Well, the point is Choose the right functional behaviors. We don't care what they say semantically we want to know what the behaviors are right and It's easy to just slice and dice mix and match shuffle together deal apart all of the message frames however, you like it and Just keep use frames for the pieces you need to slice and dice and everything is really simple. Okay now that is Is this time right? Okay. Yeah, we're all right. Okay now First of all, I do have some time here. So Let me go back to the big confusing diagram as I'm doing well on time Do we have any questions about the or any on anything really but but this is where I suspect questions may come up Is this making sense? Yeah Okay, so the question is is this something this pattern something I invented. It's not something I invented It is that I adapted something out of the guide because I thought this was an extremely nice way to illustrate sort of How you actually use the tools in practice. So there is a load balancing server there that I have kind of adapted But I think it's quite representative of Kind of the ethos in practice You don't worry about what the message frames look like you pick the sockets for the right behavior And then you just slice and dice the message frames until they do what you want If you're just relaying they even get you could it's trivially easy to pass them back and forth They even give you a function for that so you can write Proxies in a couple of lines, but normally you'll probably do something like this Anything else? Okay So here's our worker Okay now When I was doing this so believe it or not So the goal of this talk was to illustrate how easy and quick it is to get your messages where you need them to go And I got this fairly well polished up Believe it or not last night and I said you know what I need I want to show how quickly we can prototype things because that one was a little hairy So I actually kitted this up That quickly and that's kind of my point. It was just trivially for me to say Ah, let's just prototype something that has some interesting behavior. That's really easy. So Let's add a log server to this thing Okay Now what do I want? Okay? Well, first of all, I really just want to show you the other two major suck kinds of sockets Okay, so we have push Paul and pub sub How quickly we can just throw messages wherever we want them So there is and I did this in Python because we're prototyping. Why isn't that what slow languages are for? I'm only saying that to tease the Python programmers. Okay So I like Python, but yes, it's a slow language. I'm sorry So the code there's a log server and there's a log watcher So what we're gonna do is we're we're going to actually make it so the log server just routes the log messages and anybody You know anywhere you want you need to listen in you can do that and listen in on the ones you want Okay, now I will say this I only put the logging in on the Python worker and client and the reason was that if I log From the the threads which are also can run there at the same time It slows my C++ threads to the speed of Python and I just found that unbearable But that's a real I mean that that's not the fault of Zero impute that's the actual problem threads go much faster than the network, right? So you're going to have to figure out how to solve that and any in the library can't help you with that It could just put the messages where you want to go You have to figure out how to get them how to get threads to get their logging done without being unduly slow Okay, but nonetheless other than that and just my odd Dislike of seeing my C++ chug along at the speed of Python So how do we want the routing to do? Well, it's really simple But what I wanted to show you is the fact that why would I use why do I have different pairs of? Sockets here. Okay. Well what push and pull are good at is Sending out lots of chunks of data out to be run by you know many Many workers in parallel bringing and collating them back together. That's what they do Okay, so they're unidirectional you can't send anything back the other way because that's not what that pattern does So that's why I'm using this okay You might say well wouldn't pub sub be nice and the answer is no Because then every single pub sub is also unidirectional every single worker would be a publisher and The log server would have to subscribe to them, which is Pizarre, I mean you could do it but didn't make sense to me So instead the log server just sits there it binds to a pull socket and then all of the All of the anything that logs just connects to a push connects with a push socket to the same endpoint again We see that I've got there. There's there's probably a socket For every connection in this pool, but I don't care because the pool is hidden nicely away in the context and I never have to do another pull or Anything just for that reason, but on the back end I actually do want pub sub and why is that because I want anybody to be able to Listen, I don't want to have to know or care what they're doing with it I want them to be able to just pick the messages they want and If they're not there You know, I still want the logger to run so that is pub sub isn't it okay? so Let's take a look at how this looks in Python. So well, okay, so this is in the client now that I wrote in Python So he creates a second socket. It's a push socket, right? So what am I gonna do? Well This is the tricky.  Do I have enough data? No I'm in the middle of a frame Zero MQ frames or messages are all or nothing Okay, so if it can't deliver every Byte of every frame in the message you don't get it which if you think about it is what you want in Every if that's not what you want then Go grovel with TCP and suffer right okay? So that's what's gonna happen here, but I want to send two frames. Why do I want to send two frames? Okay, this is again this little bit where Python will do bad things to you if you don't send binary Why do I want that because the way that subscriptions are done is they're just pattern matched To see whether the message has your pattern as a prefix And that's actually kind of kind of brainless, but it's actually really nice for one thing a client Could subscribe to client Python And now he only gets messages that have that Channel name or he could subscribe to things that say client and he could also get Client dash CPP so you get a little you can actually kind of create a hierarchy of Channels if you want to which is really nice now the other thing you don't have to use two frames This could actually just be the beginning of this message. I don't want to do that because then I actually have to Disambiguate that I have frames so it's only going to pattern match on the first frame So now I'm guaranteed that it does not matter if this happens to begin with this because the pattern match will happen Exactly on one frame it that frame is entirely devoted to the pattern and that's it So it's actually really nice. Okay, and what's beautiful about it is that it's Does everything you wanted and the implementation is so simple and it means that you can just if you can your language can mess with strings Then you can do anything you want with it Okay, what does it look like in the watcher now? I didn't actually put the code for the log server because it's trivial for this simple log It's just grabbing them off of a Pull socket and it's broadcasting them right So what's a little more interesting is slightly more interesting is what the log watcher is doing So he's got to create a sub a sub socket now This is the this is a great way to waste 30 minutes of your life The first time you do this forget to call set sock opt Once again, we're do we're making it look like Berkeley sockets even though it really really really isn't so yes Everything we do every option is done with set sock opt And we have to subscribe and this is just a string. I'm using as the prefix right now The guide says and I believe I probably did it What you always do the first time you do this is you create a subscription socket You go ahead and you start receiving with it But you forgot to subscribe which means you're subscribed to nothing and then you wonder Where are my messages going and the answer is they're going right where they should straight to the bit bucket because you didn't subscribe to them On the other hand because it's just a prefix you can subscribe to the empty string channel And you get everything so it's exactly what you want Okay, so what am I gonna do? I'm just gonna sit here if I'm the log watcher all I have to do is is subscribe And then I just sit here. I just receive stuff and then I do what I want with it But I can have log watchers all over the place asking for different things And it just works and nicely the just if you're wondering The message is although you set the prefix you set the channel subscription in the subscriber That invisibly goes over the network to the publisher socket, which then doesn't actually send it to you So you don't have to have all the network traffic of having every subscriber get every message and then throwing them away So which is kind of nice. Okay? So What's the lesson here? Okay? Well, the lesson is that I prototyped that at the last minute just because I could and I actually thought it after I did it. I thought, you know, that was actually a better illustration Then it would have been if I had planned out, you know something a little more involved because That's what part of what I want from something like this there are many more Sophisticated ways to use this. Okay, there are ways to to work very hard on reliable Messaging and a lot of other things But the nice thing is to start by just getting the data to flow From all your parts so that all of your buddies can actually start writing clients and workers Publishers Subscribers and then later you can go back in and do that, but they're not twiddling their thumbs saying Where's the messaging layer? Right? Okay There was one other thing that was a little bit obnoxious and I want to mention it because so it turns out every socket I started building a table. I didn't give it to you because it wasn't finished But I started building a table of what are all the differences between all the socket types Because as a tutorial they get scattered all over for when you needed them in the tutorial Most of the sockets will block if their message queue is full So let's talk for just one second about these things so these sockets the guide describes them But he starts off by saying we took Berkeley sockets and exposed them to radiation from you know waste from a Russian nuclear power plant and then we gave them superpowers and they were bitten by a radioactive spider and the pros is a little Unsettling you if you're a techie you're going that's neat. What do they do? Okay, so all of the sockets have All of them will even have one message you most of them have a queue and in going an outgoing queue some of them Don't need it, but the point is that You can control the size of the queue with Get sock off But you will care about this once you start wanting to make things reliable because as I said my what will happen What happens when that queue is full the socket types have different behaviors most of them will block That's why I said if I send logging messages from my threads They slow down to the speed of Python. Of course. I was being mean they slowed down to the speed of the network Python is faster than the network. So I was just being mean just because I can But the point is a couple of them will just silently throw things away the de the router socket will throw things away Because it's a server. It's supposed to stay up The one disappointment I have with that is I actually want to be able to change this behavior with set sock up And I was going to show you how to do that and then I'm going like wait I don't see that option turns out doesn't seem to exist So if you don't want that behavior you might have to implement some flow control But as I said reliability will cost you some effort, but it will still be easier than doing it in raw sockets, right? Okay, but it's worth knowing that's why they slowed down They filled up their message queue and then they're sitting there blocked waiting for somebody to pull some of those log messages out So that they can continue on right okay, and as I said you need something fancier, but The important thing here is my point. That's why I called the talk connect all the things If you run the example code there are workers running in threads over in proc You can change it, but the Python client by default is going over local host It's going up through the through the the network stack the The Python worker is actually going through a unit socket and none of this you don't know you don't care It just works now Again for the social media generation who have already tuned me out because there were way too many words here I do want to summarize things in a more concise fashion I love you I just like to tease people who are addicted to social media and with that I'm going to end. Oh, actually the last slide is this I've not done this before but since I put the slides in the repo it occurred to me They would be nice to just have it all licensed you can do pretty much whatever you want So I'm sorry for putting legalese in the slides, but it is there That's the Debian guideline make sure the documentation is under the same license as the codes you can rip things out I don't think you're gonna want to rip things out of my slides, but it's whatever you like. So with that questions Be absolutely So save doesn't find to repeat questions. So so if you make a network-centric set of Processes that want to be monitored by the existing commercial monitoring Process software that uses sockets to attach it to If you have zero MQ can a regular socket attached to you said it's not a Berkeley socket But what happens when a regular socket client attaches to a zero MQ? The zero MQ endpoint What will happen is it will make no sense unless you have implemented the zero MQ protocol So that's why I said you don't use it primary. You don't really use it for existing protocols That's not its purpose. So the point is zero MQ has a well-defined wire protocol and You can certainly implement that But the what you would I think do in this case if you want to get first of all you can use any number of I mean So it clearly uses Berkeley sockets underneath each one of those Zero MQ sockets represents a whole pool of Berkeley sockets and it's so it's pulling on them night and day You just don't have to deal with it You can certainly also have your own Berkeley sockets You don't have to it doesn't require you to do all your networking that way if you're Interfacing with something else if you want to pass data between them, then you write a little bridge Which may not be very hard if you if they already gave you an SDK for whatever it is or I mean in other words If you can already just throw data on to a regular socket then all you have to do is You know use a zero MQ socket to pull message pull a zero Q messages from wherever you like in your zero MQ messaging Layer and then stuff that string onto the onto the socket So it won't help you, but it will also not hinder you in any way. Does that make sense? No, you can't do that because it will because no matter what you do You can send a raw a string, but it will send it using its own protocol For example, every string is counted so their account their accounts in that stream You would have to know where they are so you'd have to actually understand their protocol and you can do it But honestly, I would guess in almost every case it would be easier to write a bridge data streaming systems like It was kinesis and Kafka have this abstraction about atomically increasing numbers and ways for clients to Express that they process a certain message in a system and they come disconnect and reconnect and then Continue to process a certain point in a stream Okay, if you want to implement something like that in zero MQ is that an appropriate is zero MQ an appropriate technology for this kind of System where you want to be able to continue processing from a given point if you have some kind of outage in a worker So what you're saying is that the message is persist essentially It's a process goes down the message is persisted on disk somewhere or in something persistent storage And then you can bring it back. Yeah There's something persisted but the notion is that there's an ID that determines the last thing you process So when your thing comes back up, it knows where to start again Okay, so the answer is zero MQ isn't Going to try to help you with that in that it tries to do one thing Well, okay, it does a lot of things, but it all is related to having an in-memory message layer Okay, so it's not going to persist anything for you If you're doing but if you have to do enough of the other things that it does well It's probably it might be easier to write that that it is to do all the reconnection and all of the the other things it does But if that's not true, then it might not give you enough because it's not going to persist for you Yeah, okay, so What are the major advantages of zero MQ over a simple queue service as your service boss? Rabbit MQ IBM MQ Or most other message queues and can you configure it for FIFO? for oh FIFO, okay, so the well the advantage is it actually is in a different space. Okay It's a lower level tool and if you want for example It doesn't have it doesn't have any built-in notion of having a server Whereas like rabbit MQ is there's a server there with a persistent queue right so The equivalent to that is if you wanted that And you in zero rabbit and Q didn't do it for you you would write it Okay, in other words, it's a tool for writing those servers Okay, so what it's trying to do is it's trying to automate the really low-level code that almost everything that that touches a Berkeley socket already does right I mean if you want to be it's a pain to have to oh I my song somebody went down. I've got a reconnect Oh, I need to frame these things even though I'm going over TCP because in their infinite wisdom nobody gave us a Frame well, they did but no one implements it They didn't give us a nice framed reliable service, right? So what it's doing is it's trying to give you enough building blocks It's it's a it's a set of Legos the difference between a set of Legos and a nice metal toy car, right? It's not meant to give you a toy car. It's meant to say do you want a toy car or did you want a dump truck? Or did you want a spaceship? We have the tools for you if that does that kind of does that answer the question? Yeah, but the MQ suggests that it's a message queue not just the blocks of a message queue So that's that's kind of the nature of my question And the second part was is it easily configurable for first and first out last and first out like a sqs is Well, okay, so so don't let me forget the second one So yes, that's why I kind of I had the little what it isn't thing because with that MQ on there You unavoidably think that it's a competitor to rabbit just just don't forget about the zero at the right? Yeah, it was a hint that there's zero. Yeah, the zero is a hint that it's something different and the zero originally meant It's a zero zero late zero copy and I think now they they have lots of zeros because some of them get really excited about these things But none of them are here is a turnkey server Started up in a knit and then call it right as far as FIFO I'm not sure what you what you're saying But every one of those sockets will have one or both of the input and the output queued and that is first in first out Right, it's basically Sucking in network data off of normally TCP and then it's framing it for you So when you ask, hey, do I have a message it even if it's all there, but one bite you don't have to deal with that So the sockets are inherently FIFO. Okay, there's no out-of-band facility that I know of so Every message that you get will be in order. Is that what you were asking or would you ask to something more more sophisticated? Yeah, that's basically what I was asked about So there it's inherently that way then so my understanding was that the creator of zero MQ has moved on to nano message and Is there a reason this this talk is not nano message More industrial or there is a reason first of all if I recall right not only did you move on to nano message? But then he just moved on entirely and someone else forked it and now has it was either nano message to I think it now Might have another name. Okay, so there have been some forks and I looked at this because as With everybody who forks they claim to have improved something here's the problem most of the forks that have happened are now defunct Okay, and those that did happen they may be working now But I don't know how many years they've been working, but it's a lot less zero MQ doesn't seem to be going anywhere And also it's better documented. I mean like I said, I really want a really nice reference as well as the guide A lot of the other ones. I mean literally one of the one of them that look quite good The answer was mostly read the source code You know the idea because you know what zero and Q is now read the source code for what I did differently I'm gonna like Yeah So the reason is I didn't trust the other ones yet if I were deeply enough into it then I could make a decision But it might I this isn't going anywhere as far as I can tell that's that's the entire answer really If you go back two slides back There was a push and pull and then there is a Pub and sub right. How are the messages sent across this push and pull pub? How are they sent across? Yeah, do you mean how like like how how are they ordered? Oh, yeah, so what's basically happened is that the the pull socket is going to use I think it uses some kind of fair. So some kind of fair queuing algorithm So that it won't let for example a very fast pusher, you know entirely dominate the stream So it's distributing for you. I don't think that that's configurable So if you really needed something specialized That might be a problem for you It I didn't see anything that it wouldn't do it didn't do that I that I would I knew I'd want but it's going to sort of it's going to try to keep them all working without letting One dominate if that is that does that answer the question? Yeah, sort of okay I mean there is no buffering in any of these places, right? No, no there is there's the The push socket suppose the server goes down. Well, then he's filling up his outgoing queue, right? Well, there's buffering here because this is another socket so it's got an incoming buffer if your server is not pulling them off They're fast enough. He's got his own his own queue to fill up So that most of the most of the steps in this all have buffering on them Okay, and what it will do is it will push it as far as it can go and then it will start filling up the queue This is kind of It'll start filling up the queue that the way where the blockage is if that makes sense and While it doesn't persist them on disk as long as the program continues to run It'll sit there until it figure till it's able to deliver it if it can't So it tries to be robust, but it doesn't persist on itself Okay, can I assume that if I need a high throughput or low latency use case applications Zero on queues is a one possible. Okay, so they claim so I did not as you saw my my Benchmarking was Extremely minimal at best. I really just wanted to know how it competed with something that I knew was fast They claim that it is in fact high performance And that is likely to be true Certainly for the network things that it's really designed to do I would however Benchmark that especially if you already know what your goals are I would just write something and just stuff messages through it and see if it actually is doing what you want Because I don't have any numbers for it. They I can tell you what they claim But they also say that they use that the sockets were bitten by radioactive spider, so I I Just wonder if I can answer the question before about the difference between 0mq and rabid mq. Oh, yeah Okay, me. I don't don't ever think about them the same thing. I believe a rabid mq and a lot of them use AMqp as their actual protocol. Oh, it does. I would more relate that to what this Zero is doing. It's really just a library for using a network socket which provides some cool functionality and a cool pattern Otherwise, it really doesn't do much. I would also In my understanding I could be wrong, but in my understanding generally you don't want 0mq to do any type of queuing It's not its job. It's there to provide a buffer just as a Maybe a network queue on a switch. We do it buffers a little bit just so we can get traffic flowing but if you're queuing you've probably got a problem something backing up and then You should actually have a program like Kafka or something else in between there which would be handling your temporary storage of messages or something like that No, I agree with that. I mean when I say it says queuing that's equivalent That's roughly like saying a Berkeley socket does have a buffer inside there So that you're not hitting the network on every call to receive, right? But as you say if you if you're intentionally queuing Well, now I'm the voice of God again because I adjusted it Then it's not that's not what it's meant to do you can't get at the queues in this You can't do anything to them except set their their maximum size So if you need rabbit mq, then this is not doing that It could have been written to use this as its transport layer But that's yeah, I mean I totally agree. They're not they're not filling the same space at all And you made another point that I like oh if rabbit mq is using a Q&P it's I won't say much about that because I don't know much about it But I will say this I Maddox the company who backs this was I think they developed a mqp And eventually abandoned it because they said this leads to too much. It's too heavyweight We don't want to be required to have a central server So you don't have to you can write a central server with this But there are plenty of tools to make it completely distributed So in their mind it does compete with a mqp, but it's a very very different animal Okay, and According to them faster Hi, I have a question saying that I saw you use you Get multi I mean receive multi parts or something similar. Yeah in your code and send multi part and modify Using an index number. I'm not saying is there abstraction layer available for like Packing up like each message is each message or for example, and in another case for example, I'm using a C program and a Python program using unit Unix domain socket to Connect between them when this because the speed variance is really very huge So what would happen like like for them if I only receive something in Python. I just receive like six parts or nine parts I'm not sure I understand the question. Are you asking is this about primarily how to use the interface? How do you get off the frames? You said you yeah, but what if like the message pops up and it can't Is it possible that when I use it? I just get more frames than I want. No Never because the frames are always part of a message and you get complete messages messages are all or nothing So if the sender sends you nine frames, you will either get nine complete frames or nothing Yeah, but I mean like for example the minimal frame for an envelope is three, right? You could make your own envelopes you don't have to use the delimiter They do and they have reasons for doing it. You wouldn't technically I mean you're not required Do you use the frames any way you like if that is answering something? Yeah, but is it possible like using the way you demonstrate it that I will receive like two messages at the same time No, no, it's always again That's that is one of the better the best guarantees of it especially if you've done your own framing on TCP and you wrote the same code a bunch of times or you've said screw it and used a Serialization library it they always come in a bundle and It's not all or nothing on the frame level. It's all or nothing on the message level. So you will always get Okay, well, let me let me qualify that that's the way zero and Q works now the interface like the C++ interface You can pull the frames off one at a time And I don't like that and if I'd had a little more time I would have actually written the equivalent of the Python interface for C++ it'd be very easy and The nice thing about the Python one is you can just say give me all the frames doesn't matter How many are there you'll get the next message, right? That makes sense Yeah, can you discuss options for transport security for transport security? What I can mostly tell you is that they do have a security system now built in I have not tried to use it They did mention the person who wrote I think some a litter wrote some elliptic curve code for them So it may be well done. Okay, but I have not tried to use it. So I can't tell you much on the other hand You know there is in my experience Much of the time people do the all-purpose Thing and just send whatever they like over a SSH SSL tunnel So you always have that option, but it does supposedly now have a built-in security system Might look at salt actually uses your MQ for all of its command dispatch And it they those guys work really hard on getting some transport security On top of it and take a look at what they've done there They've had it audited pretty extensively as you can imagine no one likes having your infrastructure orchestration layer be a compromise so Last I looked was like two years ago or something and somebody had actually forked The zero MQ and created a sassel implementation for authentication, etc It does seem to attract forks so We're at the end of the hour. So I'm gonna End the questions even though this is going great if you guys want to bug Dustin after it's over that that's fine Absolutely, but let's thank him. Thanks. Very nice. Do you want? Thanks a lot. What's that? Yeah, yeah, yeah, I didn't know what you were pointing at because I was not thinking very well. I'm just running on coffee Hello, everyone. Welcome to the developer track At this point this afternoon. We're gonna be talking about lost in user land with will here and be presenting We're gonna get started real quick here. I'll let him take off and then we'll do Q&A at the end I'll just sit off in the corner there, but when Q&A comes Tell me to grab point in my direction to grab my all right well without any further ado Let's all find out about the joys of core utils Excellent. Thank you. My name is will Rose Kranz and thank you all for coming to find out how to get lost in user land So my name is will Rose Kranz and I work on the edgecast content delivery network at Verizon digital media Basically that is a very large network where if you have bits and you have clients we will send the bits to the clients We have over a hundred points of presence around the world. We serve terabits of traffic Zillions of connections a number that is always growing and not worth remembering exactly We serve lots and lots of sites that you're familiar with and lots and lots of video sites cat and gifts mainly and then All together it adds up to a non-trivial percentage of the total traffic on the internet so we are very concerned with performance and that's sort of a key part of my job and Basically the summary of all of that is cat gifts We do also serve edge cats net in addition to edge cast CDN net if you ever need like an Edge cat supply it will give you a random cat gift every time you go there So more specifically I am tech lead on the HTTP platform team We are responsible for building and releasing our core HTTP server software, which is called Sailfish Every time we release it. There's a really involved process of canary deployment where we're constantly looking into any possible performance regressions things like that so performance is a key part of my job and part of everybody else's job there and Generally speaking we have this very large distributed system lots of nodes around the world Running Linux so Linux performance is something I'm really interested in and we also have a software life-cycling community internally There's lots of different communities that people have started that are kind of separate from the management hierarchy One is the board game community some of them are more or less serious This one is called the long march forward and I created it with another colleague named Marcus Hildum And we do things like courting Coordinating OS upgrades on all of the systems that we have deployed in production And we are also really excited about end-of-lifing any sort of system or any code anytime somebody deletes a substantial feature or chunk of code or Shuts down systems that are no longer in use we award them with what we call the Darwin award Because we realized that management is always really excited about pushing This is the new thing that we're going to build and celebrating we built this thing And there's almost never a moment to really celebrate We destroyed this thing that has caused massive amount of maintenance overhead over the years and wasn't really that profitable and didn't help us that much So we kind of took an initiative to celebrate the thing that I think a lot of engineers are actually more excited about sometimes because the thing that has Been dragging them down and that thorn in the side that they've had for years And almost never gets kind of celebrated and the third bullet point is that I have very boring hobbies So I decided to write my own Unix user land because the Unix user land is this piece of legacy software that has been more or less Consistent in its user interface since the 1970s and I'm really interested in performance and I Have this theory that every large corporation has a directory that new guys are told not to run LS in I used to work in visual effects and we had internal pipeline utilities that would dump lots and lots of frames of animation Into a particular directory Previously I've worked in places where everything was on a really slow NFS server And it wasn't that many things but because it was coming over network It was really slow to run LS in and because I work at a content delivery network We do storage as a service for other people who decide how they want to organize their data So we can't be blamed for that directory and I can actually admit to it So I was really fascinated in why is LS so slow in these big directories that have millions of files in them And is there any low-hanging fruit in terms of performance? Is there something I can learn from kind of trying to implement my own LS and some other related utilities? So efficiency was something I was curious about How close could I get could I approach the efficiency of the LS and GNU core Utils? Can I maybe even exceed it? C++ is a language. I'm pretty comfortable with it's reasonably low level it's kind of like see which my next kernel is written in new core Utils is written in and All of that kind of dates to the late 80s or early 90s before C++ was as common as it is now So if you imagine a next-generation GNU project that had been started in the late 90s instead of the late 80s It probably would have been in C++. So I did that since it's native code pretty similar to regular core Utils I'm very lazy So I only worked on the easiest utilities to implement because as just kind of a hobby project on the weekends If you can write one in like a day, it gives you instant gratification, which makes it a really fun project So I created W Utils My name is will but I named it worst Utils because that helped me justify Not trying to match every feature in GNU And then I started doing a bunch of research on the history of some of these utilities and sort of fell down a rabbit hole that I found interesting so We have this concept of a Unix user land I need to define Unix and I need to define user land in order to explain what that is I'm using a very expansive definition of Unix in this context that includes things like Linux and Minix that don't share code with classic Unix Reasonable people will disagree with that definition of Unix But it turns out there is a lot of legacy even though the Linux kernel does not directly share source code with classic Unix The current version of Linux even if you build the Linux kernel 5 that just came out on a brand new system today There is legacy that dates back to really old Unix in terms of design influence and a lot of the supporting utilities user land Simple utilities are part of it mostly for historical reasons some simple languages like awkward part of it But languages like Python and Ruby are not because they're newer and they're bigger and much more complex you don't go to like awkpie to awkpip install modules for awk so The user land is fairly constrained and simple and then you have all of this other complex stuff like GUI's big languages the init system which obviously init systems are so Simple and boring that nobody would ever bother to like reinvent a new init system That's more complicated and significantly different different than the legacy 1970s version. I assume That's the best joke of this whole talk and it's all downhill from here And In it the code runs in user space It's not kernel mode, but it's part of the OS because a user user isn't directly interfacing it with it Things like cp and LS are the user experience or the user interface of a classic non GUI Unix system So that's what I was focused on but like I said, I'm super lazy and I wanted instant gratification So I just focused on easy stuff. I didn't write languages I didn't write a C compiler. There's people who are way better at that. So I wrote stuff like cat Which shouldn't be interesting, but it kind of turns out that it is So I started with false and true Which in C are numerically in that order rather than true and false because in C false is zero and true is one in shell True was zero and false is one The shell and Unix of course were created basically the exact same time But the shell predates C So they didn't agree with each other for various reasons that made sense to people at the time And this is a piece of legacy that is completely insane and we just accept I just really wanted to point it out because we get used to some of these quirks that a new person approaching the system Would be like but those who literally the exact opposite false is true and true is false and it's a very post-modern approach to truthiness So anytime that you want to implement your own version of utility check the documentation some of it's good some of it's terrible Can you true is documented as do nothing successfully? And then return a status code indicating success which is zero and then false is do nothing unsuccessfully pretty straightforward. So let's Evaluate the really early implementation of true This is the complete implementation of true They said that I shouldn't have a lot of code on the slides, but I think you can all grasp this It was a shell script that was zero bytes and because it had no commands in it None of those commands could fail and therefore it returned success which is true Which you really have to admire the simplicity you had to spin up the shell interpreter in order to do it So it's a little ram that was probably unnecessary But as an elegant solution you have to admire it then lawyers got involved. So by 1984 This is the implementation of true. This is the complete implementation of true circa 1984 It is all copyright text and some source control notes Fun fact, this is less than 512 bytes of copyright notice So on a 512 byte block file system, which was in use at the time and the file systems were still mostly implemented in like assembly So they didn't have really complicated systems for merging I notes together and stuff They got this metadata completely for free. It took no extra space. So on some level This was also a sort of admirable engineering accomplishment to include all this Duplicated data between all the scripts, but in some of them you just got it for free because it was still less than 512 bytes But new in the 80s they looked at classic Unix and they were like we can do better We can be more efficient spinning up an interpreter is insane So the GNU true the current true dates back to 1999 if you check it and get hub it's been through a surprising number of iterations and This is a 31 kilobyte binary, which is about a hundred times the size of the shell script that had 300 bytes and If you note in the comment True will occasionally return exit failure in the case where rights fail So GNU totally unprecedented new feature in true. It was a true that sometimes returns false Because they added dash dash help and if for some reason it failed to write the help output It would return false and and I was really impressed by this sort of new feature this amazing new approach So in in my version of true and false I was like I can take this to the next level and make a false that sometimes returns true Because if they get to take credit for failing at printing help for a utility that has no features Then I get to take credit for successfully printing the help for a utility that has no features So if you do dash dash help or dash dash version It will print out the help text and then I think rightfully claim success So having learned from you know that history I I Was inspired to create even more features and then down at the bottom There's if exit failure is true return exit failure So bin false makes a point of returning true Because see and she'll don't agree on the definitions and Then if you download the slide deck, I've included a bunch of links to things There's links to the original source and things like that that you can check out on your own time So a couple of other utilities obviously true and false pretty minimal I decided to write my own cat and I looked briefly at doing my own DD. I was really interested in the History there. I didn't actually release a DD with this talk because it turns out DD doesn't do anything like I thought I did So cat of course can catnate files and print it to standard output There is a famous page called useless use of cat where if you cat a file and then pipe it to something and only use one file One of your colleagues will during code review tell you you're an idiot because cat is only for concatenating multiple files Of course, they could have just made cat a shell built-in and then it wouldn't really matter and The less than greater than syntax for redirecting a single file wouldn't be more efficient But in 40 years nobody has ever bothered And then DD a lot of people learn this as the disk duplicator when I was first learning Linux and Unix That's what the utility was explained to me if you download an ISO image off the internet for like a CD ROM You can copy that onto a disk Take an image of a like a USB flash drive put it into a file things like that But if you look at the man page for DD it's convert and copy a file And in the man page for DD aside from the name DD convert is the very first word And that's an important nuance that I think a lot of people don't really notice So if you look at the whole man page Copy file converting and formatting according to rules and most of us don't really use DD for that If you scroll down in the ma