here we usually are pretty pro player and what I mean when I say that is I'm all for the players getting their bread all for players getting as much money as they possibly can because these franchises they want cheap play before as long as they can possibly get it they want to do everything that they can in their power not to pay these players but to get the ultimate production out of them and I'm all for players wanting to get their money and the reason that I'm like that is because I've been there before a lot of us have been there before a lot of us are there right now to where you may be working for a company and they try to get all this max production out of you but they don't want to give you a raise or they don't want to give you the money that you're actually worth and it can be one of the most frustrating things ever so that's why I'm always like a if this play if they need to hold out if they need to go somewhere else go get your contract go get your guaranteed money go get your signing but go get your bread Patrick Queen he did just that he went and got his bread he went and got his money we had all came to the realization that he was not going to resign with the Baltimore Ravens if he did we knew it was gonna be for a disrespectful disrespectfully low amount of money so I'm glad that he didn't because again I want these guys to get paid but where Patrick Queen where he threw a lot of people off is when he went to go play for the Steelers now I ain't got no problem with that if it's still is that's gonna pay you the most money and that's where you want to go no problem ain't got no problem with that now I did see a lot of fans disrespecting Patrick Queen color them all sorts of names and this and that and that part I don't agree with that y'all know that already I will never agree with disrespecting people like that and if you're gonna talk a little trash okay cool like oh yeah I hope you go over 17 I hope y'all lose every game that are the old stuff like okay no problem but to be calling my name and it ain't that serious and like they always say it is a business and I think fans oftentimes they forget that it's a business but while it is a business it can still get personal because we have personal feelings for these teams and fans develop personal feelings for these players and Patrick Queen it seems like he developed a lot of personal feelings for Ravens fans and he has some very strong words for Ravens fans that we're about to get into before we do it make sure you subscribe to the channel and turn notifications on and let's run them likes up baby it'll run them likes up even though I know you and a lot of y'all not gonna like what Patrick Queen had to say but let's run it up baby and let's get into it so with Patrick Queen he signed with the Pittsburgh Steelers he signed over the three-year 41 million dollar deal but he signed it with the enemy and that left a lot of Ravens fans upset because most of us especially me I had assumed like oh yeah he going with Mike McDonald to Seattle they got an opening that linebacker though yeah him and Gino stone they both gonna go over there but boy was I completely wrong and not even close we had heard rumors a little before free agency oh watch out for the Steelers for Patrick Queen and I just thought oh yeah that's just one of them little articles of they just talking okay he could possibly go to the Pittsburgh but he did he went to Pittsburgh and a lot of Ravens fans were upset about that and a lot of Ravens fans have been letting him have it on social media now Patrick Queen gift in a curse social media we always talk about because it gives you access to everybody but it's a gift in a curse because it gives you access to everybody and Patrick Queen has been the victim of so many social media moments because what Patrick Queen does and he's not afraid to let you know it because we would see it year after year after year Patrick Queen will search up his name he won't even be tagged in a lot of stuff he'll search up his name and he'll respond to stuff and it's like oh like it's tricky because yeah you don't want people just running that mouth about you any kind of way but at the same time you just letting people know that they could say whatever about you and you gonna see it and you gonna respond to it and he has done that for years with Ravens fans for years and your fans in general too and it could be a good thing you could be commending Patrick Queen oh Patrick Queen great job I appreciate that man or but too many times we've seen it where somebody's talking trash about Patrick Queen and he'll respond to it and I think that was one of the biggest mistakes of his Baltimore Ravens career just on social media alone but here recently Patrick Queen has been getting it from Ravens fans big time because they don't like the decision that he made they don't like how he made a decision I ain't gonna probably decision he made but then this is where I got lost though because let's let's leave you what Patrick Queen had to say toward Ravens fans he said honestly bro y'all can shut up now it's not even that deep for us players until game week and y'all can't understand the fact it wasn't up to me I gave y'all everything I had now and excuse me I gave you everything I had now when it's time for me to do what I need y'all heard about everything I say go touch some grass and feel the void in your life to one still showing love even with the switch this is not to y'all so Patrick Queen telling Ravens fans shut up it ain't even that deep and it wasn't even up to me as far as him leaving that part like okay I guess that's him saying like hey I really wanted to stay but they want to pay so I had to go I gave y'all everything so he's saying put it all out there on the field okay I could agree with that and now when it's time for me to do what I need to do as far as for his business for his family for his money that now y'all heard about everything that I say so go touch some grass so tell them Ravens fans get outside go live life and feel the void in your life to the one showing love even with the switch this is not to y'all so so he was obviously frustrated with a lot of Ravens fans that were talking that trash to him but see before this just a day or day or two before he tweeted that he was at the presser his introductory presser for the Pittsburgh Steelers introducing him as their new inside linebacker and he said I like being a villain I want to be the villain and like again I'm never for people disrespecting the player none of that stuff no I disagree with that wholeheartedly but you cannot get in front of the Pittsburgh Steelers media and really the NFL world you cannot get in front of them and say I want to be the villain I'm ready to be the villain and then send a message like this showing that you're getting upset over what people are saying villains love that villains love to upset people villains love to take people off villains love to set people off they love making people mad and irate and frustrated and all and they feel the villains love that they thrive off of that so for you to sit at sit there and say up yeah I want to be a villain I'm ready to be a bit and then go around until all Ravens fans y'all can shut up now y'all can shut up why how y'all hurt by everything that I say you're going in two completely opposite directions but hey you know what this might this might be just what we need to get that Raven Steelers rivalry back again cuz it's dead it's dead but this right here this may be the signing right here this may be Patrick Queen with his actions right this may be what gets Raven Steelers rivalry back on again and we need it we need it Ravens bingles it's fun a lot of trash talk back and forth there between two Ravens Browns it's cool it's a little trash talk here and there not as not nearly as much as Ravens and bingles though but Ravens and Steelers it's just falling off so much man so much it ain't been the same in a long time and that's for various reasons but still it has not been the same it just doesn't have that umph that it used to have somebody explained to me the other day it was in the comment section they said man from 2008 to 2012 just watching these teams battle it out just to get the chance to play the Patriots so Ravens and Steelers was all about they were just fighting with each other oh and it was crazy every game came down to the wire they were just battling it out like crazy but after that it fell off and then I wouldn't even say just a couple of years after that could remember 2014 remember that year oh that that rivalry took another turn that year especially come playoff time but um man I just I was just lost with Patrick Queen on this man I was so lost because it's like he wanted the title but then when he got the title that he said he wanted he's like oh no no no I don't want it anymore I don't want to be that villain but then get this get this because MLF football or ML football I think they they fell the same way because they said on Twitter update Patrick Queen can no longer handle being the villain it was just one day and Patrick Queen said I can and will I'm not withdrawing my statement thanks last time I'm saying something I promise you I can guarantee all y'all watching this video right now this will not be the last time that Patrick Queen says something I can guarantee you that watch because it's going to come up whether people are typing about Patrick Queen and because again he's he's already displayed this while he was on the Ravens I don't think just because you switched teams doesn't mean your personality changes I mean well actually you got a little more money now I mean he already had bread but now you got even more bread now but just because you switched teams you still the same person and Patrick Queen again search up his name and he'll respond to so much of that stuff so much of that stuff so much of the negative stuff and that lets people know like oh okay let's keep on putting it out there let's keep on saying it so we would PQ I was I was just a little disappointed with that but hey again at least he lighting the fire back for Ravens and Steelers so now when we watch these games they gonna mean a whole lot more


Well, one of the things I absolutely love about this program is its dynamic nature. The Masters in Marketing at the University of Edinburgh provides a perfect blend of theoretical knowledge and practical applications. The curriculum is designed to keep us on our toes, exploring the latest trends and strategies in the ever-evolving world of marketing. I think our program is a very cohesive program and it's really kind of prepping us to go out into the real world as, you know, the future marketers. It really gives you an insight on how to do research and how to apply research not only in an academic level but into an industry level. The amount of diversity in the program really is so rewarding because you get to learn so much about the people you're studying with, you get to learn more, you know, new perspectives.


Thank you very much guys. Thanks for coming the brief agenda. So I'll have a brief introduction We're gonna talk about the the punch on the title like basic configure option configuration options of you Izzy And why you should use them and then we're gonna talk about some other stuff Basically you Izzy provides worker management features above and beyond what most whiskey hosts do and I want to talk to you about them because they can Protect you in production against things you might not be expecting And I want to talk about additional features that you won't get from other whiskey hosts and then potentially have questions So I apologize for the click-baity title of like trying to like make some controversy about you Izzy I love the thing. It's great, but you do have to get some basics, right? So why are we here? We're here to be successful right and I mean something different for all of us some people here are language people Some people are data scientists.  Well, it'll make your organization more impactful My apologies so Yeah, so building a reliable performance system is important to be successful in whatever enterprise you are running So this is a talk about distributed systems architecture Specifically we're going to zoom all the way into the into the individual service level and you Izzy's a tool We'll be doing to do that. I bring this up because I often talk about different parts of the distributed system stack In 2017 I gave a talk at high Gotham about stateless systems and how stateless systems can make your system more testable and more performance And so this is you know the exact opposite of that We go all the way into the to the very smallest level when talk about each individual microservice and how it's hosted So why you Izgy? Well, it is very powerful the list of features is tremendous and it's very fast, but It was written in a different time back in 2008 when you Izgy was written People did things differently and a lot of the defaults represent that The a lot of there were a lot of quotes in here from unbit the developers of you Izgy Is anyone from unbit here? I was hoping they would be it was also scared. They would be all right So they're not thank goodness But they they have production customers people that pay them money and a lot of the decisions They make are about catering to those customers rightly so So I want to talk you about how to avoid those problems and then move on to the things that you was going to provide above and beyond other whiskey hosts Almost every item I'm going to present today at least in the first third Caused an issue for us at some point of varying difficulty and they might seem like things that are like wow How did that hit you? Well, I have you know a very large team of developers between 40 and 50 at any time Working on services hosted by you Izgy and those guys are productive to get a lot of work done They stumble across corners right so if you're about to start a project with a Whiskey project maybe with flask or Django or anything for that matter You should consider you Izgy as your host and you should use this as a reference Now you Izgy 2.1 is supposed to be released at any moment back in July 2017 The unbit developers posted to a mailing list that they have decided to fix all the bad defaults Especially for the Python plugin because little-known secret you Izgy hosts Java services Ruby services anything you can imagine really In the 2.1 branch the 2.1 branch has not been released as of June 2019 I know this because I check it every morning leading up to this talk word that they would The first reference to you Izgy 2.1 was in 2014 So like who knows when we'll get released maybe that will provide some respite to this prop these problems But as it is now you should use the contents of this talk as a basis if you're gonna start any Development of a whiskey service using you Izgy There's an official things to know doc that you Izgy provides that is very valuable I recommend you read it I'm not gonna say much more about that other than that and I reference it everyone so on throughout this talk I would also like to say that we have published the contents of this article of this talk in an article at tech at Bloomberg comm That an hour ago, so if you go to this website on your phone or on your laptop You will see in markdown form what we're gonna talk about today So in the presentation things will be abbreviated so there's not too much text on the screen But at the website it is like very very It's expanded on very greatly so you can kind of understand our reasoning for each of the items and like with specific examples So it's a much better reference than the slides are today All right, so let's get started. That's it. I can leave now Do this right now? I will explain each one of them But yeah, this is what you want to do. Let's just get right into it So the master process is what makes you Izgy special you want it on I mean, this is kind of obvious. We should just get past this point I want to bring it out though because you there are some circumstances where you want it off specifically when you're debugging if you want to test your Service and you turn the master process off and you run in a single process Which you have to with the master processes off then the PID that is created immediately after you start you Izgy Is where your code will be running so you can use tools like S trace which we talked we heard about earlier in the in the conference or You know any sort of command line profiling tools like valgrind for example And you won't have to deal with like all the forking options like follow fork child and all the stuff that you normally have To do when you're using a tool like valgrind to debug or to profile a python service So otherwise not that interesting Strict config parsing. Okay. So by default you Izgy let you put anything you want in the configuration file And the justification for that is that you can add Non-existent options to your config files as place holders customs options or app related configuration items But the truth is that most of us don't do that stuff when was the last time you ever passed through application specific configuration options in a Framework configuration file never you shouldn't do this So you should turn this off why because it happened to us somebody meant to type something In the config file to control behavior in a way and they fat fingered it They put a wrong character in they move the production the service works right? It's not like it didn't work It's just that that option wasn't set properly service moves of production And then something doesn't behave the way you want right this is an example of you Izgy having some default behavior that can Allow you to do something that you don't intend right so you should set this now If you do want to use this feature of passing through configuration options through the uizgy iini file Absolutely feel free to do it, but that's not the default right or at least it shouldn't be in my opinion The next option is vacuum. So this basically makes you as you clean up after itself Common example is if you run your service on a local unix socket Uh the without this option unix socket will not be deleted for you sounds harmless right no big deal Well in my environments we have multiple users operating in the same machine And if they start a development instance of their service the socket will get their username and they might not have set their Their mask properly so that all users in the systems can read and write those and delete those files So if another user tries to start up a service on that socket location They will get a failure because they don't have file permissions to delete that socket So another thing that you should just set by default There's no real reason to leave these temporary files around it's just one of those things you don't need If your application is special for some reason and you need it then feel free to Move this to false But by default it should just be there and if you have a large development team It might save someone cursing another one of your employees, uh, you know one day in the future Because that is what happens All right, so now we're going to enter a section where there's a common theme and that is uizgy default Default options changing the expected behavior of your code by default uizgy disables python threading Which therefore removes the gill the explanation for this these are all quotes by the way these gray background Sections quotes from the uizgy developers directly They do this because they think that it will improve performance I don't know if it does or not Maybe it does all the applications that I have that are of sufficient complexity end up using threads for something in the Background and so I had to turn this on right away I pointed out because while it's relatively easy to find the first time you come across this your instinct isn't to think Well, my service host must be messing up my threads your instinct is like ash I screwed something up in my threading code right you start by looking at your code And then like depending on how good your employee is at googling or yourself It can take anywhere from five minutes to a few hours to figure this out So just turn it on by default if you're super performance sensitive and you don't use threads You can opt to turn it on like intentionally, but it's not a very good default behavior Another one is this single interpreter parameter uizgy has a feature uizgy really has a lot of features This one allows you to run multiple interpreters in every worker process Uizgy is the pre-forking model So if you want to hold to run 60 requests concurrently it'll spin up 60 processes If you will allow multiple interpreters uizgy will allow you to run more than one web application in each process Basically just a way to condense your processes and keep your pids down I don't really think it's necessary in the modern environment people running containers Or in virtualization, but back when they wrote uizgy and they were working on shared hard shared physical hardware Keeping the number of pids down could end up being important So this usually has no side effects, but there are some third party c modules that do not work well in multiple interpreters They make some assumptions about like globals and c memory space, which isn't validated when you're multiple python interpreters I actually don't remember how this caught us I assume it must have otherwise we wouldn't have just found it randomly So sorry, I can't give you some background on how we stumbled onto this one And then the last one which is just frustrating is die-on-term. So by default uizgy Treats the sigterm signal as reload the application stack Maybe you have updated your code and you want to you know refresh what the server is hosting Sure, fine. That's a great feature. I think you should have it But why would you use sigterm? We all collectively agreed on what sigterm should do it should kill something And for some reason they decided that they wanted to do something else with it So I just don't get it But the first time you start using uizgy someone's going to try and literally type kill the pid number because they're You know for whatever reason they lost their terminal and the service kept running in the background And it's just not going to stop and they're going to be like what's going on And then if you're smart you'll do kill dash nine and get past it But someone might want to figure out what's going on and then you have to find this parameter so again It's just the common theme of uizgy Deciding to change behavior away from what we all expect programs to do to something else for their use case Which we no longer need all right So if you're starting a service in 2019 and you want to use uizgy as your host which you should just by the little things I'm saying Set this as a little as a default And this sorry, I get excited about this one. This is the worst to me. This is the most egregious configuration parameter I think I've ever seen in any project Without this parameter uizgy will start if it cannot load your application That means if you have syntax errors in your application or some other dependency of your Of your application cannot be loaded. It just starts anyway And guess what it does it serves 500 and 400s to everybody because that's the only thing it can do Now what's what's insidious about this is that a lot of systems will just check to make sure that the server is Awake right you have like a health checking system That's really basic or you have some sort of like you know a lot of these systems are pretty naive Your service starts it looks fine Then you start redirecting production traffic to it and everything goes boom The reason they did this is back in 2008 it used to be a common pattern to start uizgy with nothing loaded and then dynamically inject applications I can't imagine what the world was like back then although technically I was operating as an engineer professionally I don't remember doing it this way, but uizgy did it that way back then So nowadays you want to set need app equals true and what that'll do is if you try to start your application And there's a syntax error to fail right away. I don't know if you guys saw this talk today about the guy doing the um Look ma no HTTP But his whole demo relied upon him changing code and then having an auto loader restart his service every time his code had a His code was changed so that he could see the errors he made immediately If you were using uizgy the whole thing would be for not right because it would start up and it would look like you were Fun, so don't do it This one is uh is actually just an opinion. It's not actually anything that I can pin on the uizgy developers um uizgy is very verbose with logging Most of us if we're developing an application will have very concise but meaningful application specific logging Probably because we're feeding our logs into some you know grander distributed grep system Where we have to pay by byte and we don't want to necessarily spam the thing So I recommend disabling uizgy's uh default logging. Um I don't particularly like the format although if that's all arbitrary right We all have our own arbitrary arbitrarily preferred log formats But what I do think you should do and that's really what I wanted to talk about is you should enable the uizgy default logging For 400 and 500 error codes and why do I say I do that because it is very very difficult as an application developer To guarantee that you will always catch an exception or an error and log it properly It is far more likely that some percentage of the time you will intend it to capture all your errors But you won't because it will occur before your logging handler or some other situation that your logging handler can't anticipate So I recommend turning this on to make sure you always have some indication that some traffic was had failed Even if you also do that in your application logger two log lines aren't going to hurt you, but zero will All right, and that's kind of it. So if you stick to this You know and you're running a decent sized application team or you might in the future You can potentially avoid a lot of significant issues that will either waste developer time or cause potentially major production outages After this though, we're going to get a little bit more positive And again, I see for you guys taking pictures check out check out blue murk.com All this content will be there with much with basically all the things that I said written down instead of bare bones like the slides One of the things I want to talk to you about is worker management I feel passionately about this and most wizgy hosts don't even bother. They're just like you need 60 Sure, I'll give you 60 and that's kind of like they stop talking about workers at that point You wizgy gives you a real powerful wealth of features in this area The most dearest to my heart is worker recycling What this basically does is make sure your workers don't get too old and there's many metrics for that I have three different methods here max request basically tells you wizgy to restart your workers after a thousand requests That number is pretty small depending on your your situation At bloomberg. We're doing we're doing data services. So there's a lot of financial calculations that take a lot of time So a thousand requests actually might take a little while If it's been alive for more than an hour Or if the process has allocated more than two gigs of memory resident memory And that all of those go into effect after the request is finished So if you have a request that goes from let's say two gig from one gig to eight gigs in one request It will wait until it's finished and we'll clean it up after it reaches eight gigs I point that out because this is very valuable to help you prevent like a certain class of errors If you have a slow memory leak And you recycle your workers you might not ever figure it out Right if you have like a few bytes that are leaked everyone wants them on because an object somewhere That's not cleaned up and you turn on all these options here probably won't be a problem Now obviously you should find that slow memory leak anyway and fix it But that doesn't mean you should let it cause a problem for you in production So these features are great and you should turn them on front by default as far as I can tell There is really no cost to this Except for maybe if your application has a huge startup time These workers are forked off of the parent one anyway So that even isn't really a big problem, but I suppose if you had a huge amount of page tables that the forked time might be significant I don't know. I have never personally observed any issue with this whatsoever And then the worker reload mercy is just tells allows you to configure how long to wait before you like forcefully close your Workers if for some reason they're holding onto a resource that isn't gracefully giving up its control Then there's dynamic worker scaling so this this again is not strictly required This is a feature that I love because I happen to run multiple whizky services on the same physical host or in the same virtual machine And I also run multiple versions of the same service. Uh, if you see my talk about stateless services We run multiple versions of the same stateless services and then basically we probably request to all of them to compare Differences and responses between them So if you were to look at one of my machines, this is an obvious exaggeration. We don't run 500 processes, but uh, We do often run 96 or 128 And if you have you know 10 or 20 services with that many processes each Your top or your ps can be can be pretty Busy so dynamic worker scaling allows you to start up with a lower number of workers And then scale up and down depending on the load in the system and it is anticipatory Anticipatory, I don't know it anticipates how much traffic it expects based on how much traffic you're getting at the moment And generally leaves enough of a buffer that spikes won't Uh outpace it You can see in the bottom There's some parameters about the backlog that's feature in particular make sure that If it's not scaling up fast enough it will create emergency workers to make sure you can handle it And again as far as I can tell, uh, this has never been a problem for us. It scales fast enough It's not like a risk. It's just another no cost feature that you should use There's also it's pretty sophisticated. There's a lot of options a lot of different algorithms This is the configuration that I like But you should definitely if you're going to do this you should understand your application and check out the other other algorithms other than the busyness one that I've I've selected Okay hard timeouts This is a really powerful feature that I have not seen in other whizky hosts You whizky if you turn on harakiri will forcibly kill your worker process if it is stuck I point this out because this happens all the time and This can basically make that a non-issue I mean it's when it happens you should log it and alert on it and you should figure out why and fix it But we had another talk earlier today. I forget the gentleman's name But he was talking about um using asyncio and production and some of his war stories Granted he was talking about asynchronous cases and it's a little different But he explained a situation where he had a spike of traffic come in or Conversely one of his downstream dependencies was slowing down or reaching its limit and his service basically backed up with requests And he had thousands and thousands of requests going on in parallel and they could never catch up Even if traffic went down to zero it would take hours for everything to recover Right. Well, this is what this feature is for if a spike of traffic comes in and your system for some reason locks up Maybe not because of a deadlock, but because of just limited resources harakiri will clear all that after 60 seconds So when traffic does recover your system will too now granted you should understand why that Q situation occurred and you should expand capacity of the limited resources and make your system more reliable in that regard But there's no reason you have to get woke up in the middle of the night every time And that's what this is for right cya like protect yourself. Don't cause me to drown just because you made a mistake We make mistakes use the tools to protect you So you should use harakiri. We use 60 seconds. That's a long time for most people But we're doing like I said, uh, you know numerical calculations on uh mortgage backed securities another large financial instruments Some of our services go much higher than 60. Um, so the lower you can set this the better because that's basically your your um Period in which you can have a Q condition once that this period turns over you're all the stock requests will clear out This is an analog to this right So hard timeouts are kind of drastic because you're literally kill-9ing You are your workers, right? You really don't want to get to that This should have been in the first section, but I put it here so we could contrast it this Uiski configuration option allows worker processes to receive signals without it Uiski does not allow this to happen. I have no idea why I actually couldn't tell you I couldn't find the justification for this. Hence. There's no quote from them The reason that you want this is that you might want to receive a signal for example in the python signal module you can call signal dot alarm and tell the python interpreter to wake up this handler if you have uh After a certain amount of time So one example would be you set a hard time out at 60 seconds and you use the signal module to set a soft timeout at 59 seconds Oftentimes i'm serving requests where I have partial responses. I can return even if I can't return the full thing So in our system we use the signal module to wake up any Waiting process at 59 seconds gather with whatever data we have and return it to the client If that fails for some reason for example say we have an actual stuff process This will kick in Uh, sorry, Harakiri will kick in and kill the process to make sure we continue to operate properly All right, so these two things kind of go together But this is really a default you should just set this it's there's really no harm in it Like if no one sends your process signals, then fine But if you want to this is another one of those situations where it's like how good are you are googling to figure this out? Uh, this one is very important as well So we had a conversation uh, sorry a talk at this conference about observability about uh, you know tracing and logging and monitoring And how observability is more than just the three things that we that he talked about Which was logging metrics and tracing this is another area you can get observability to your software by default When you start a service host most of them give me this boring output That's just the command line that used to invoke the service which is literally useless practically I mean, I guess it tells you the python module that you loaded But that's really it you as he's aware of this and they give you an option called auto proc name That you can put in the config file that will do something like this It will tell you which process is the master which processes other workers and which which worker number it is This is still a little naive, right? What if you have more than one uizgi service in the same box you might end up with You know 10 uizgi masters and which one is the one you want So you can be a prefix you can say prop name prefix and give it a service name That's much better, but it's still not the best we can do there is an api you can invoke from within your code to specify dynamically what you want the prefix to be And so here's an example. This is contrived my I'm not actually hosting It doesn't have we actually do but it gets the example across where you can put context about what that process is doing So in this case you can specify the service the username and then the uri they are accessing What this helps you with is sometimes you have a problem where there's a lot of Slowness in your hosts and there's a lot of stuck processes and very quickly But it's running top or ps you can see that it's a pattern that it's all the same user or that it's all the Same uri or there's something common about that problem Which otherwise would perhaps take you a little while digging through trace or digging through logs You can kind of get a quick snapshot of it. So it's not a replacement for anything else It's just a straight free zero cost additional tool that you can add to your system Last section I want to talk to you about is additional uizgi features. So this is a controversial section Because if you use any of these features you have now lost compatibility with other wizgi hosts You can no longer switch back to g unicorn if you use these because you are now leveraging Uizgi specific functionality in your code that said it can still be worth doing I mean, honestly, we don't switch hosts that often once we get a system in production And these features can help you Solve or provide a solution to something much with much less complexity than you would be able to otherwise, right? So it's a trade-off between reducing complexity and having interchangeability of your wizgi hosts We end up using some of these because the reduction in complexity is just tremendous when you're when you're running a large system The first one I want to talk to you about is a cron and timer So that's literally what it sounds like running crons and timers So you can say do something every first of the month or do something every 20 minutes It does not let you do one time timers If you want to just say do something in 60 seconds, you should use the signal module Like why would they write something that already is in the standard library? I point this out as being useful because sometimes you want your periodic tasks to be synchronized with the software version The code that you're deploying right sometimes you have this problem where you want to put it a new service That has some some functionality. You want to run on a timer and you have to update your cron system separately And now you can deploy those things in an atomic manner If you're going forward This isn't usually too big of a problem because you can deploy the new features and then change the cron timer later But if you're going backwards actually could be dangerous because you've now moved your service forward There's some bug in it you want to roll back But now you have some cron job that expects that behavior to be there So this can help you deploy software in a more atomic fashion And also if you just don't like using cron or any of those things you can do it in the service itself Then you can also do it with a decorator Instead of with doing it at the global level, right? So that you can put a decorator around a function and you as you will call that periodically Next thing I want to talk to you about is locks. So this is actually very difficult to do If you don't have a like Framework supplied lock like this you have to try and get some code to run before the pre-forking portion of you Is the fork tell your processes and create a lock in global space and then share it around Or use an external lock and redis or something. It's actually a real pain I don't think you should write software that needs a lock. If you have to do this, you're probably messed up But if you do this is probably a better way to do it than all of the alternatives So i'm just making it making you aware that it's there even though like I would kind of feel dirty if I had to use it Then there's the cache system. So this is super cool. This is a cache That you as you provides so that all the workers can share information Uh, you could obviously use memcache or redis as well, right? Like of course you could but then you have to create a separate system to start those things and to monitor them And so it's kind of nice to have it all packed together We use this for rate limiting actually what we do is every time a request comes in we store in If you whizgy cache how many requests a given user is doing And then we like reject requests that they've reached some threshold You might say well, why don't you have ha proxy or engine x do that? Well, the reason is that we want to We want to Throttle people in a more sophisticated manner than one of those proxies can we actually want to look at the HTTP request in pretty Detailed ways in order to do that So This is a pretty powerful feature. It's got some gotchas For example keys or values that are too big for the cache will solidly fail to insert So you need to like kind of make sure you have a good test suite around it But once it's working it's bulletproof and it's much simpler than Starting up a remote or an external memcache or redis instance to get the same job done And then there's mules. So mules are worker processes that are not workers They're not servicing requesting your clients, but they get but they're there to do stuff We use this for example to aggregate metrics So all the worker processes will send the mules metrics and then the mules will offload those metrics to your metrics engine Right whether you use data dog or metric or metric tank or griffon or whatnot So you can put a little decorator on that and say target equals mule If you don't say target equals mule it'll run in the in just a random worker process or all worker processes So that's how you specify that and then there's ways to send data there as well Which I include in the longer form article that we posted on tech at bullenberg.com if you want to see that So that's it. I am hiring I am looking for an sre and just to qualify when I say an sre I mean a distributed systems engineer who doesn't get work from product people who want features built whose job it is to Optimize monitor and improve the performance of our system and its reliability for the sake of itself and not to get some Featured on for a business guy We are a pretty great place to work. Please talk to me. Um, I would love to tell you more about the role We have some people already doing this role. So you'd be joining a team. You wouldn't be bootstrapping an organization And but you'd have to come live in new york, which I know is terrible. Who wants to live in new york? That's it. I think we have a few minutes for questions if there aren't Yeah, we have a few minutes for questions again Microphones are there and they're please line up behind the microphone and uh ask away Hello, hi, uh, could you tell me please more about hierarchy option? Why should that I use it? Why should that? I just not set up like work Worker reload time So the worker process recycling Um, we'll I think we'll kick in I'm not a hundred percent sure about this But I think it will kick in and restart workers But generally your worker process Reloading Is going to have a long time interval. Uh, you might say an hour or thousand requests Right. So if you set it on request, well, it'll never reach that obviously If you set it on memory, it will never reach that either So it's only the time one that will catch it and then it's the question of do you want to wait an hour or 60 seconds That's really it to me. I want to wait 60 seconds if I wait for that hour to happen Many many bad things that happen. I mean this this actually happens to people You have a bug in your code in every worker deadlocks. It happens, right? So the hierarchy is a way to help prevent that whereas if you use just the normal worker process reloading It would the time interval be too long All right for them. Hi What about lazy apps loading so loading app and forking versus Forking and the loading apps after all have you been bitten by that or So we we prefer the post fork option. So we we'd want to load everything and then fork after You know, that's a great question. I don't think we have yet the doing it lazily makes the Process reloading take much longer because now you have to wait to load all of your application code, right? Which can take seconds sometimes So we have not yet But that's a great feature to be aware of if there's something about your application that you can't load it once and fork Trivially, you can enable the lazy apps option and that will prevent that from being a problem Usually that kind of stuff happens if you import something and it initialize something globally that's also like See extension and then you're doomed for lazy stuff Absolutely Thank you So thanks for the talk and I wanted to ask about your deployment environment because I noticed you have a huge number of workers And is it containerized that there are small machines or big machines with a lot of cpus? In our particular situation, we generally have dual socket linux servers physical servers Which somewhere around between 20 and 28 cores generally about half a terabyte to a quarter of a terabyte of ram Um, we have some virtual deployments and deployments on vm's But we don't really prefer them because we run a very high duty cycle in these machines We're doing like Monte Carlo simulations of mortgage-backed securities which is why I have so many workers and those Used like we have a cluster of 500 machines that we've run at 100 cpu for 20 hours a day And so if we were to go to a virtual environment, it would just cost more money Like we need more machines once you get the overhead there So we are capable of running in a containerized environment if we need to but for money's sake In efficiency's sake we run a physical hardware Thank you. Cool. One last question very quickly Do you have any recommendations or gotchas and so on for emperor mode or would any of your recommendations here change with that except for I think one option you mentioned To be totally honest, I don't understand what like emperor mode and zerg mode and these things are for I mean We we have a microservice environment where each service Can run independently and can be deployed independently. So I I honestly have a completely ignorant to all of that stuff Sorry, but we kind of almost recommend against using emperor mode at all. I don't know why you would use it Right. So I I haven't seen the use case for it. I suspect it's one of these back in the day That was the best that they could do You know now things have progressed All right folks, let's thank all our speaker again


God is the Lord of the breakthrough and we hope that you get your breakthrough today because we're so glad you're joining us on Hope Today where we love to spend this time to uplift and encourage you and we're so glad you're joining us today. I'm here with Tom and Amy and Tom, tell us about our guests that is coming up because we're going back into history talking about a very important person. I know. Well, you know, most of us have heard of Anne Frank, the Jewish teenager who wrote a diary is she and her family hid from the Nazis during World War II. You may have read it. I've talked to people who said, yeah, I read it in junior high, high school. Well, but have you ever wondered what Anne's faith was like? Well, Pastor and historian Johnny Teague is with us today to dig deep into Anne's life and to see what strength we can gain for this time, for this time in history through her example. I'm really looking forward to this conversation, guys. Yeah. And talking about going back in history, you know, I watched the Jesus Revolution movie this past week. And you know, it's not just watching a movie that this could have happened. This is actually a historical event that did take place that was a spiritual awakening and revolution and revival. And so I cannot even tell you what it did on the inside of me. I just felt, I just felt like do it again, Jesus. You are doing it. And I just, I just saw that this movie and this time and these revivals and these moves of God that are popping up all over the world that this is just part of it. It's like fanning the flame of revival. There's so much happening. And I know Sydney, you've mentioned so many things happening. I'd actually like to bring Johnny in real quick here before we go to the main interview. Johnny, you've studied the great awakenings. Tell me what your opinion of what God is doing right now. My goodness. We've shared of how so much of revival begins with prayer. It begins in the most unexpected places. We saw it with D.L. Moody. We saw it with George Whitfield. We saw it with Charles Spurgeon. But it really is organic. God just begins to move amongst the people and the neighborhoods and the streets and the homes. And all of a sudden the thing breaks out and it breaks out everywhere and it sweeps the nation. And that's what we're praying for. Amen. Well, that's a good point. And I know Sydney, you've shared so many times about what you've seen on social media of people that are coming to the Lord. Yeah, it's really exciting. So I know right now there's a lot of controversy with TikTok. So I know some people, I know there's a lot going on there. But I have been, I'm still on TikTok and I know the Lord is moving on there. But I have seen just in Mexico that they're incarnable. There was 1,000 people that gave their lives to Christ. Amen, I saw that even during one of the Jesus revolution movies, like afterwards people were on the ground. People were slain in the spirit. They were getting healed and delivered. So God is truly on the move. And I just want to share something quickly that I just saw God moving in a really special way. So I wasn't here yesterday because I acted to be in court because about a year and a half ago, I know many of you know that my car got stolen by a group of juveniles. So it's just been this whole ordeal that has been going on. But let me tell you yesterday that I just saw the hand of God in just a really special way. And so the whole time that God has just placed in my spirit, my husband, I like even the day that our car got stolen we prayed for the young man. And we just wanted an opportunity just to say like talk to them and just speak some heart to heart.  He was like praying underneath. What to say because it's just seeing, I think one thing that is beautiful that when you see within the juvenile system is that I just saw the importance of community and just really they're having their arms around this young man. And just so I shared like I forgive you and I believe you knew it was this whole move. And in that moment in that courtroom of just being able to share about a little bit about God and faith and just different things like even the judge has said it was a grace of God what was happening is it wasn't a courtroom anymore. It was a moment that we were lifting up this young man and he started sharing about his dreams and sharing about different things. And it was so powerful and moving. And afterwards we got out of court. I gave, we both gave each other a hug. I talked to his mom and we exchanged numbers. And I think what God was showing me in that moment is what Jesus does for us. You see a lot of times that we are in that place, right? What we have all fallen short that we have all have sinned. And what Jesus does and what God was showing is that he is the faithful witness that he will testify and he will speak of the things that we've done. But he says you are forgiven. And because of the blood and because of what the price he paid on Calvary that's what he does for us. So I had this whole revelation of just the amazing thing that Jesus does for us that is standing before the judge. But Jesus says to all of us you are forgiven. That everything that has happened all the things that the transgressions they are gone, they're in the past start your life again. And so with this young man just to be able to see that and just like his mom and just it was, it was just such a beautiful thing. And I just wanna say this is that wherever you are I know where you're seeing moves of God that are happening all around the nation but God wants to do something in you. There's something that you can do in a moment whether it's a courtroom the classroom in your family on the street in the grocery store are you going to be obedient to the Holy Spirit? Are you going to listen to him and say, you know what? In this moment I'm gonna be like Christ and I'm gonna show the love and light in the midst of the darkness. I'm telling you there is a generation that is out there. They are hungry and they are crying out for revival. They are crying out to touch and experience Jesus like never before. So are you gonna be the one today to go out and do it? So that's all I had to share but that's just been on my heart. And it's just like, yeah just really just a powerful moment. Yeah, powerful moment and grace filled. That's what I love about it. Grace filled. That's awesome. Well, as I said earlier, many of you are familiar with the story of Anne Frank and her diary that she kept during the Holocaust. Her diary gave the world a glimpse into what life was like for her and her family during those horrific years during the Nazi regime. And our next guest has spent years compiling research on the life of Anne Frank and he discovered some astonishing facts about her life and her faith. Dr. Johnny Teague is a historian, is a pastor, he's an author. And in his book, The Lost Diary of Anne Frank, he explores what happened after her diary suddenly ended on August 1st, 1944. Johnny, welcome to Hope Today. Thank you. It's already been a blessing to be with you. Thank you. Well, it's a blessing to have you. So for some who may not be familiar, tell us who was Anne Frank? Why does the world know about Anne Frank? Anne Frank is really the most famous, most touching story that came out of World War II in the Holocaust. Anne Frank was born in Germany to Jewish parents. And when Hitler took over in Germany, thankfully her dad had plans to support and to protect his family. They moved to Amsterdam. He was very successful businessman. And as Hitler's tentacles began to reach into Holland, he had already set up an annex, a place for the family to hide. And when his older daughter, Margo, was called up, they went into hiding. And they were hiding for two years. Right before they went into hiding, they gave Anne Frank a diary on her 13th birthday. And it's because of that diary, it's because of Anne Frank, that we know what a lot of Jewish people and a lot of people are victims of Adolf Hitler went through just to survive. And her story has touched the hearts of this world for generations. And the diary just ends abruptly, just ends. And what did you find in your research for your book? And I was mesmerized by what you've written here, writing, well, just explain the process you went through to research and write the lost diary of Anne Frank. Well, thank you. We see the diary in. And like anyone who's ever read that diary, we want to know what happens next. We want to follow Anne through the next chapters of her life, which were tragic, but we weren't able to. She wasn't able to take a pen and a paper or a diary with her. And so what I set out to do is to research what Anne went through from the day she was arrested. What happened? How did they find out where they were hiding? Where did they go after they were arrested? What was Westerbork like, which was like a transit camp? What was it like for her in Auschwitz and then Bergen-Belsen? So I traveled to all these places and interviewed eyewitnesses and survivors. This book follows the facts as best we can, but I write it in first person. I write it as if Anne continued with her diary because I want us to feel the emotions of it. And I shared with y'all my three trips to Auschwitz where I studied, and that trip to Auschwitz, that third trail, I would go into the Auschwitz camp. I would walk through very prayerfully, just really trying to grasp the weight of that place. And I got there when the thing opened on that third trip the first day and I followed everything that I could. I walked through it, I read everything. I would sit and contemplate. And I got to the very back of the camp where the crematorium was, and I sat there and I just prayed and reflected on Anne Frank and so many. Well, I'll be, I realized it was getting dark. I had been there all day. And I thought, man, I better get out of here. I better leave before they close. Was I headed back? I found that nobody was there. Everybody had left. I'd been so enamored by what was going on in that place, what Anne faced, that I'd completely lost track of time. So I hurried back to the front gate, that infamous gate where you'd enter. And I was locked in and no one was around. I pounded on the guard door of the tower, the guard tower, hoping that someone would be there. No one answered. It got dark. I began to walk the perimeter of Auschwitz trying to find a way out. I couldn't find a way out. My heart was beating. I was fretful. And I knew I wasn't in 1944. I wasn't there with Nazis, but the fear that I felt really kind of took me back. And so I circled around and went back to the guard tower. I saw light at the guard tower, I began to bang on that door again, praying, Lord, please let someone let me out. And by God's grace, I heard the heavy boots of a Polish guard walking down out of that tower. Obviously it was a place where he would stay the night and he didn't speak English. I didn't speak Polish, but he let me out. And it was really then that I thought, man, we have got to reflect what Anne Frank and millions faced like her. So that's really what really pushed me to write this book. Wow, what an experience, my goodness. And as you write this, and as I was reading this, you get drawn into what was going on and the horrific nature of it. But let me ask you, what do you speculate something in the book about her wrestling with God? And she says that so many people that were in there they turned their back on God, became atheists. What did you find out about Anne or speculate about Anne and her relationship to God? Well, the great thing is Otto and Edith Frank, her parents had a strong commitment to God. They were a Jewish family. They weren't as active in the synagogue as perhaps they wish they would have been. But Otto Frank and Edith prayed with their children every day. They discussed God's word every day. And Otto Frank even broached the subject of Jesus with his children. And he wanted, before they went into hiding, he had wanted to get them the New Testament so they could learn about Jesus. And so he set a foundation in Anne Frank's life.  And so that clearly carried on in the Westbork Auschwitz of Bergen-Belsen, eyewitnesses and survivors spoke a little bit about just, of course they didn't know Anne Frank would be anybody famous. They didn't realize she would impact our world. But just a brief grasp of some of them was her understanding that man is to pray, that man is sinful, that man brings harm, that God brings good, that God brings life, that God carries us through.  And I wanted that to be represented. Where others left God in the Holocaust, I believe firmly, Anne drew closer to his wings, trusting him not just with her daily bread, but trusting him to take her when life ended. Johnny, what happened in you as you put yourself in her shoes and you were finishing her diary and her story? What changed in you? That's such a great question. And honestly, I was transformed by the study of her life. A lot of things kind of came to the surface for me. One is she had these animosities towards her mom. They were petty. And once she got into the concentration camp, she started looking at people, not for their little negatives. She began to see the beauty of her mom and her dad and her sister. So I believe firmly one of the things that I've found in my own life, it's so easy to deal with the petty and forget to look at people for who they really are. And every one of us, God has placed himself, we are created in his image. And when we look at people, we can see his image. I think the other thing that kind of came to mind was that we treasure so many things and they're just stuff. And when someone like a Jewish person, the Holocaust, I mean, I'm sitting in my study right now, I've got awards, I got family pictures, I got all these things that I treasure. But when someone is yanked out of their house and thrown in a concentration camp, all they have is their being and their relationship with God. And it's really made me take things a lot less serious, folks in what really matters. And those are just two that really have impacted me. Johnny, we've seen anti-Semitism in various forms throughout history. Why does there seem to be a rise of anti-Semitism right now in the world? Well, I'll tell you what, we've left God as a nation. We see it in the headlines, we see it in the streets, we see it in our churches where we're compromising, where we're letting a public sentiment determine what's right or what's wrong. And because we've left God in a lot of ways, though we are seeing, I think we're seeing some revival. I think we're seeing what we've been praying for. But I think more than anything, when you leave God, then you leave God's people.  And the Jewish people are here to point us to God. And so really anti-Semitism to me is at a front against God. But because we can't hurt God, we seek to hurt those who belong to him, whether they be Jewish or Christian, there is this push against anyone who reminds us of God. And that's something that we've got to be very mindful of, which is one of the reasons I wrote the book. We've got to understand what this movement can do if it's not contested and if we don't return to the living God. You know, Johnny, what you're talking about anti-Semitism, you know, it hits home here in Pittsburgh. We have a very large thriving Jewish community. And several years ago, we had a synagogue shooting at the Tree of Life synagogue. And so what would you say, you know, from your experience even writing this and how we can be a bridge to our Jewish brothers and sisters during this time? Oh, I'll tell you why. That's such a great question, Sydney. And we as Christians, in particular, the Bible that we have was written by primarily Jewish people. The Savior that we bow before who has saved us and given us eternal life came through the Jewish people. And so what we've got to do is we've got to love them. We've got to stand for them. And we've got to constantly be praying for them. The Bible says we're to pray for the peace of Jerusalem. And I think it's just Jerusalem. It's also the Jewish people. I had a big, big visit with some church members the other day who were fighting amongst themselves. And I asked them, do you love them? Do you love that person? And they said, the right answer, yes, I love them. I said, well, you know what? They don't know you love them unless they see your kindness. And I think with the Jewish community, we must show our love through kindness. Johnny, just final thoughts, final takeaway. What, as the church of God, what can we take from this? You've already mentioned loving those that are the people of God. What other lesson, what maybe internal lesson can we take from this that we can walk the walk that God wants us during this time in history? Well, I think the Bible, I know the Bible says in the last days, people will be lovers of themselves. And there's such a big push in our community that with depression, with suicide on the rise, with all these things going on, that if we just love ourselves, everything will be good. And honestly, we've got to love God. We've got to put Him first, seek first the kingdom of heaven and everything else will be given to us. And if we love God and we love our neighbor as we love ourselves, we're able to see things change. But my prayer always for Johnny Teague is holiness. If I am holy, if I am set apart for God, if I am sold out for Him, every good thing and every blessing will come to me and I'll find a peace and I will find a unity with the people who live to the left or the right of me. The book is called The Lost Diary of Anne Frank. Dr. Johnny Teague, thank you so much for being with us today. Thank y'all for having me. Well, a powerful time and a powerful lesson. We're gonna have a quick break and we'll be back with a scripture and a time of ministry. When Laura called our 24-7 prayer line, she had so much fear that she didn't want to leave her house. She had lost her husband of 54 years, just six months earlier. Laura was flipping through TV stations when she came across Cornerstone Television. She felt compelled to call. One of our prayer partners talked, listened, and prayed with her for 45 minutes. At the end, Laura said how much the ministry had helped relieve her fear. Praise God for how He's using CTVN. When you give, you become part of what He's doing. This month, when you give, we'll send wild expectants as our way of saying thank you. This book will inspire you to live your life as God intended. To give and request your copy, visit us online at ctvn.org slash donate or call us at<PHONE_NUMBER>. Hope happens here. I don't know about you, but I loved hearing, you know, the story of Anne Frank and what her diary would look like. You know, if she had kept writing and I think it's so important that we remember, you know, what caused those moments in her life, the hate, the racism, the discrimination, and what do we do as the people of God today in 2023? How do we bring heaven to earth? How do we bring thy kingdom come? So we don't see that ever again. May we learn from history? Let's go to the word of God. And in this scripture, you know, growing up in a Jewish home, it's quite, it might be that Anne Frank read this very scripture Deuteronomy six, verse four and five, here is real. The Lord is our God. The Lord is one.  If we would just get back to the basics and we would just obey and honor the word of God, the teachings of the Bible, we literally could see such massive revival and repentance and the move of God and the love of God, such as the world has never known. You know, Sidney, it's so important that we don't just read the word and it not become a part of who we are and change who we are to represent him on earth. You know, one thing I love about this scripture when I was doing some like research and digging, this is actually called a Shema. So this is a very common prayer among the Jewish people. And the one thing I just love about this, it's not individualistic. And I think a lot of times in our Christianity, it's all about me, but it's about as a whole, as a community. And so what would it look like as us, as the ecclesia, as the body of Christ to do this is know that our Lord God is like, the Lord, we're one. We are one in him. See when we have that revelation that we're one in him, then all of these denominations, all these divisions, all those things, they come crumbling down. We celebrate that God has made us unique, but we don't let those things define us because we are identified and defined by him and him alone. And what would it look like as the body that we would love our God with all of our heart, that with all of our soul and with all of our strength. And I wholeheartedly agree, Amy, that we would see such a move and such a change. And that's why I truly believe that this move of God that we're seeing, that we're not seeing one person out in the front. That was the thing of the past. But now it's we're moving in unity. We're moving in solidarity because people need to see and to experience something different. I love that God says that he puts the solitary in families. And there's a lot of people that are on the outside, a lot of people that are hurting, that are broken and wounded, and they need a family. They need someone to come to. And we can be that in this moment in time. Tom, what are your thoughts? I think about that loving God with all your strength, that clause right there, that phrase. What is it like? Carried a refrigerator of steps. I don't know if you ladies have ever done that. Nope. I've done that. I've done that with somebody else on the other side. And it's all the strength you can imagine. I remember when we were moving and I was older than I had been when we moved in, right? And it was, and carrying, this time it was the washing machine of the steps and just carrying and thinking, I am straining everything I am of my joints and muscles and everything are fully 100% engaged because anything less than 100% wasn't gonna accomplish the goal. What's that mean for us? What's it mean for us to do something with all our strength?  And we do it with a little bit of our strength. Or we do it with some of our strength. Or we even do it with most of our strength. But do we do it with all of our strength? Are we gonna put every effort? If there was a pot of gold at the top of those steps and all I had to do was get that refrigerator up there, every effort would go into it. And when we have something here that is so much greater than any earthly prize, will we put today all our effort into it? I mean, that just really resonates with me, Amy. You know, today we went back in time and we talked about an era. We talked about a time. What bad happened, what good happened, how God did move, how evil was prevalent. And you know, there will be people that are writing about this time that we're living in. So it's our opportunity. It is our moment in time. What are we going to do about the move of God? What are we going to do about the evil on the earth? Maybe it's time as the believers that this is our call to action. This is the time to be courageous, to be very strong and very courageous and rise up and no matter where you're at in the workplace, in the church place, in the community, in your home that you are bringing revival. You're bringing the move of God to earth. You're bringing the presence of God.  God is moving now. He's pouring out a spirit now. So let's be vessels, Sidney, that God can use that He will move through us in this time when people are writing about this time, Sidney, our names are going to be in those stories. I'm just really excited. Are you excited?  There's so many things happening, but we know this is what's supposed to happen. And God has called us, you know, next week, I'm just really excited. It's like Purim, and that's when we celebrate, you know, Esther for such a time as this and all of the changing and things happening, you have been called for such a time as this. God has chosen you, He's chosen me for this time, this moment in history that we can make an impact, that we can clearly be the hope and the light of Jesus Christ. So let your light shine wherever He calls you to be, whatever that sphere is, because you know what? People need to know Jesus. People need to see Jesus because that's the hope that we have today. On tomorrow's Hope Today, learn how you can experience a more intimate relationship with God. International best-selling author, John Bevere, unpacks how a healthy fear of God can strengthen your faith and transform your life. That's tomorrow on Hope Today. Cornerstone Television wishes to thank all our faithful viewers, whose consistent prayers and financial support have made this program possible.


I'm sorry, but I have to say that the open source design thinking panel will not be happening since the presenter didn't arrive. If you're having trouble with that, sorry, it's not going to happen. What do we have here? Does it display? Yes, I think I might have that. I have not this one, but mine is the next one. Perfect. Is it your beginning? Hour or so? Perfect. I don't know. I'm sorry. I'm sorry. I'm sorry. I'm sorry. I'm sorry. I'm sorry. I'm sorry. I'm sorry. I'm sorry. I'm sorry. I'm sorry. I'm sorry. I'm sorry. I'm sorry. I'm sorry. I'm sorry. I'm sorry. I'm sorry. I'm sorry. I'm sorry. I'm sorry. I'm sorry. I'm sorry. I'm sorry. I'm sorry. I'm sorry. I'm sorry. I'm sorry. I want to remind you the sevens of the last show, that kind of thing, I think. But eight is really important. I think it's very, very important. I think it's really important. I think we'll follow that. But how to follow her? Because we know she's open and open. She's open and open. She has a good friend. She has a good friend. She has a good friend. She has a good friend. She has a good friend. She has a good friend. She has a good friend. You can't. Yeah, I just want to go. You have a little questions. What's the name of the show? So, what's your name? I don't know. Yeah, I think we're going to go all the way around to this stage. Yeah, I think we're going to go all the way around to this stage. Yeah, I think we're going to go all the way around to this stage. Yeah, I think we're going to go all the way around to this stage. Yeah, I think we're going to go all the way around to this stage. Yeah, I think we're going to go all the way around to this stage. Yeah, I think we're going to go all the way around to this stage. Yeah, it seems to work. Yeah, it seems to work. Yeah, it seems to work. Yeah, it seems to work. Yeah, it seems to work. Yeah, it seems to work. Yeah, it seems to work. Yeah, it seems to work. Yeah, it seems to work. Yeah, it seems to work. Yeah, it seems to work. Yeah, it seems to work. Yeah, it seems to work. Yeah, it seems to work.


Okay, so welcome everybody who made it here to this room. Also welcome everybody in the live stream. We're here to talk about communication between Linux host systems and FPGA-based subsystems. My name is Alex. I'm based in Munich, Germany. I'm an electrical engineer. I've been tackling problems with software for 20 plus years, and in the last at least 10 years I've been drifting more and more to embedded software and especially systems which rely on Linux and FPGA part. These days I'm wearing two hats. I work as an R&D engineer at Minerik. We make laser-based communication for aerospace, free space communication, pretty cool technology. But what I've been doing for a while longer is MPSI technologies. It's a startup with the goal to provide the embedded community with developer tools that replace repetitive and boring coding tasks by model-based source code generation. The tools are out there. In terms of adoption, there is still work to be done. Now on this slide, I will show you a couple of scenarios, which most of them I have come across, where you have communication between Linux host and the FPGA-based sub-system. The legend is common for all, so typically the purpose can either be you want to invoke a command from the host on the FPGA and get a reply, or you want to shovel data between the two systems, buffer transfers. In terms of bandwidth, you have different levels to it. Depending on the application, you will need more or less bandwidth. I will, you will immediately see this in the different scenarios, which I will be showing. So a very simple reason, but a very justified reason to talk between a host system and an FPGA-based sub-system, is you want to debug what is happening in the FPGA-based sub-system in a bit more comfortable way than with JTEC counting zeros and ones. But this is an application which doesn't require a lot of bandwidth, so you would probably pick a UART over USB connection. FPGAs are very good at reducing data, so you can have a source like an ADC that produces lots of data, and instead of bothering the Linux system with it, you can put the high-performance algorithm or the data crunching algorithm in the FPGA and just transport the result to the host system. Then, of course, there are also systems where you want a high amount of data to be transferred to the Linux host, or you need to have that. And then you would need a high bandwidth interface here, PCIe, and it's intentional that already the connected SRAM is shown on this graph, because typically you would go over the memory to talk to the host system, probably using DMA. A quite common variant of Linux host system and FPGA sub-system since around 10 years is FPGA SOC devices, which combine compute cores, which run Linux and FPGA fabric in one device. That simplifies things because then the interconnect is already given. It's typically, no, actually in all devices I know of is an ACI interconnect. But again, the principle remains the same, something which is quite tempting with such devices is also that you look at the FPGA and see, oh, it has so many free IOs, let's route some low-level interfaces through these pins. There is nothing wrong with that, and that's where I show these additional lines where you make connections available as drivers, but they are routed through the FPGA fabric. Finally, something quite popular recently is the so-called, well, the accelerator scenario, where the FPGA isn't even connected to external data source like in ADC or Ethernet, something which physically provides data, but where the FPGA is only used for acceleration. I will not talk about this here for two reasons. First of all, I know nothing about it, but second also, you are then talking about high-level synthesis or also the tightly integrated neural network framework where the vendors will provide you with something where you don't even need to think about that you have an FPGA there. Now, the reason why we've been looking a lot at the interface between Linux host and FPGA-based subsystem is in the context of our developer tools for model-based code generation. We have a demo project, which is this tabletop 3D laser scanner. You have a turntable and then a five megapixel camera and two line lasers controlled by some kind of embedded system. And we have lots of variants for that system. And so I list the ones which correspond to our scenario here. We have one where we have a Lattice device connected UART over USB to a laptop where obviously only control and result can be transferred through such low bandwidth. But then we also have a device where we choose a different architecture and then we pass high bandwidth camera data directly to the Linux and then we need PCIe. Another one is quite recent SOC, microchip polar fire SOC and then the usual contenders as well, Xilin sync. And also some other devices, which however don't fit in the pattern of this presentation. I'm showing a bit of software functionality for this project because I will be using this project throughout the presentation. So preview image acquisition, we are just doing binning. That's a very good algorithm to be run on FPGA and not on a Linux system. We have another one, this is a bit hard to see I guess. It's identifying the corners of a checkerboard. They're shown in blue, bit hard to see, sorry. And then for the 3D laser scanner operation, finally we have identification of line lasers in the image.  Each of these algorithms can either be performed on the Linux host or on the FPGA. So we have something to play with here and of course as you can imagine, this will greatly vary the load on the interconnect. The less data is processed on the CPU, the less bandwidth you need. So with that as an introduction, I would like to give you the outline of the things I want to be talking about. Similar to the network layers in this scenario, you can say you have on the bottom physical layer, then a hardware abstraction layer where it then does not become important anymore. What is the physical interface? Protocol layer and application layer. So I'm starting with the physical layer. These are some typical bandwidth results you can achieve without doing any tweaking. Of course you can get some high performance chips here and there which will probably get the numbers up. But without putting in a lot of effort, this is typical bandwidth. You can achieve in the communication between the two sides. So on the lower end you get UART already from most embedded, I mean actually all SOCs I have seen will have some sort of UART connectivity. UART over USB is particularly interesting for debugging so you don't have to worry about the long cable lengths. USB will handle that and then only the last piece which goes into your FPGA is UART. SPI similarly and then the last three ones are rather higher speed interconnects. So if you use AXI Lite, which is the interconnect you have in all FPGA SOCs, you get to 50 megabyte per second which is already quite something with 32-bit words and 100 megahertz clocks. AXI Lite always has a lot of handshake around each word which is transferred so it's not really optimized on that side but it's easy to deal with. PCIe is a typical solution if you have two different, maybe even PCBs talking to each other. You need differential pair routing and some control but you can get to 250 megabyte per second per lane in the PCIe one configuration. And finally, if you use the full power of the AXI interconnect here in an example, 64-bit time in the burst configuration then you can go to 700 megabits per second or higher. Next up, hardware abstraction. So from here I'm separating between the host side and the FPGA side. So to me, hardware abstraction on the host side which is Linux is well, what kind of driver do you have? And the good news is that really you can get very far with simple character device drivers. The transfers are controlled by the host, the Linux host and in the end you have open read write closed that's really handy. And this is then possible for all these three types of interfaces and I just listed some typical device files you would find on your system. If you have a board with a good board support package they should already be there. And then the AXI Lite category is also easily implementable like that. You can write a very simple driver. I guess it's a bit hard to read but if you get the address space of only four consecutive addresses you can make a very simple handshake where you notify the FPGA side, okay read transfer is starting, write transfer is starting then you send the actual data, you implement a timeout and in the end you can provide this as a character device driver. It's very simple, I'm mentioning this because the vendors will typically also provide you something and before you know it you would be vendor locked and here's maybe the moment to avoid that and just write the very simple driver can be cross vendor. I have very good experience with that. Then there is the advanced category in which I count the high bandwidth interfaces PCIe and AXI4 in the full bandwidth configuration which if you think about it typically will only make sense if you use DMA and interrupts because well Linux has a scheduler it decides when it has times and if you want to transfer huge chunks of data it's better to write them into the memory and then have the Linux part after an interrupt working on that data. The UIO already has features for that integrated with PCIe and there is also a helpful article it helped me to write a driver using this PCIe and what I also saw is tomorrow there will be a talk about debugging PCIe express so that's certainly interesting. In my case, I experimented with this I am a six host and the lattice as a client and fortunately there I didn't well there was not so much to debug. Now switching over to the FPGA side again for the simple protocols UART SPI and on the next slide also AXI Lite it's very simply implemented there is no need to use any vendor core for this you can easily write or download I will give you the sources, the references in the end very simple VHDL modules that handle UART RX, UART TX and SPI Slave and AXI Lite so you just have to react to some some of the handshake signals of the AXI Lite bus and that is really not difficult. Now for the more, for the PCIe PCIe is on the FPGA side a bit more advanced because it typically directly connects to the vendor specific gigabit transceivers so it cannot really be a fully cross vendor the good news is from the three vendors these blocks which provide the PCIe handshake are free of charge although not all of this is necessarily open source so this is what I found a couple of weeks ago so you can so you can at least work with this yeah. Now I want to show you the higher layers not one by one but by the example of first showing how a transfer from host to FPGA could look like or an interaction that's triggered by the host and then the way back afterwards. So starting here on the application layer then here I'm using the example of terminal which you can use to invoke commands yeah you type in a command basically on your host side and then we will go all the way until it is received and acknowledged on the FPGA. Here this is a functionality where I can in a small web user interface click my commands together and I pick the command a stepper motor move to so if you remember this reference application with the tabletop laser scanner it has a turntable that's driven by a stepper motor so a natural command for the module that controls the stepper motor in the FPGA is move to so it's visible there at command sequence also a bit slow so small it has two arguments you give an angle and t-step would basically be how fast should the motor turn. So from there yeah what has to happen next so you want to invoke this command and in the scenario I'm showing here there is a automatically generated library on the host side which translates this command into a bytecode that is then to be passed to the FPGA it is listed a bit here so there is just like the first section of the zoom in there are two t-axis where you first say okay where you tell the FPGA first okay I'm invoking a command this is the target and I'm sending also five bytes of parameter data this is then the second transmission and then you receive back an acknowledge if everything went well. Second command is step get info so that's the second block over there you would pull the angle which the FPGA knows because it's counting the steps it's put into the stepper motor until you have reached the target and that's basically it. So we have seen the host side the Linux side and now what happens on the FPGA side well you might know in FPGA is typically everything is hierarchical but that must not necessarily be the logical representation of the system. So in this case you would start from the left hand side there is somewhere the XI PSPL switch then you have your little FPGA VHDL module that decodes the XI transfer that's passed through an RXT DX block and in the module host interface the decoding of the byte string takes place and it will then know to give a handshake or a request to the stepper motor VHDL module so this is the endpoint so to say of our command invocation there is a request acknowledge signal and then below the parameters are passed. Yeah so that would be a very simple example of command transfers between host Linux host and FPGA and for the way back I have chosen instead something which requires higher bandwidth and that is preview image transmission. So the camera, so it's again left to right the camera supplies yeah, five megapixel images which in the end implies bandwidth of 150 megabyte per second but then you shrink them and you can transfer them to the host with an regular eggs highlight interface so there's some modules involved again so starting from CAM acquisition, ACQ the third from the left where the actual binning takes place there is a B buffer situation or ping pong buffer where always one buffer is being written to while the other is being read out to the host it goes again through the host interface that handles the handshake with the Linux host and then it's basically the return of what we had seen on the previous slide. This is definitely too small to read, sorry about that. So what the host does it basically pulls the buffer status that's again a command invocation if you will and then if it changes to A buffer full or B buffer full it will initiate the buffer transfer. And then, okay, then it goes all the way to a web-based user interface but that is definitely beyond the application layer which I mean by the little table that's shown there on the right upper corner. I can't deny that I'm a big fan of model-based source code generation so for these examples I have used our open source tool to generate from one single source of truth, one model file, all the glue code if you want between that was required from the place where we invoke the method and C++ to where it arrives and is being handled on the FPGA site in VHDL code and also for the way back this model-based approach is quite useful. So how it's done just very quickly so we have model input files for the modular structure of the FPGA site. So here it's shown in Xilinx Vivado how it then looks like this hierarchical structure and for some of those modules we say these modules are special modules which we named controller and you can attribute a command set to each of these controllers like the one we had seen step which had the command move to and get info. So once we tell this to the tool in a second model file we say well what are the commands and what are the invocation and return parameters and based on that description it is possible to generate all the code on the C++ site and on the VHDL site to make such transfers work commands and buffer transfers. Here I would also like to show three different concepts of how on the application layer like above the hardware abstraction layer you can possibly interact between Linux host and FPGA-based subsystem. So the one in the first column we had seen in the stepper motor example and also the buffer example. I call it simple. You are on the Linux side and you have blocking access to the FPGA resource for all requests for buffer transfers or commands. And in the FPGA you had seen there is one host interface module so it handles requests one at a time. An advantage of that is it requires a quite low FPGA footprint but I'd also indicated that you may have to pull the status like in our case how far has the stepper motor already been turning. You have to pull that all the time. That's maybe not ideal. Especially in view that FPGAs are meant, I mean they are ideal candidates for parallel processing so something like that is not really great. Second option is callback based so here the idea is you launch a command for example the move to command for the stepper motor and then you get replies to that from the FPGA side until the target is reached. This can be solved by having a command invocation and return buffers on the FPGA side so that requires a larger FPGA footprint but it is also much more powerful if you have many things going on at the same time. Something which you might have missed here is something which is typically done maybe some of you have worked with these cameras where for configuring them you have to write hundreds of register with a specific value in a specific order so that's something which I think if you have an FPGA and you have Linux on the other side I would not recommend although it's industry standard and good for bare metal but the problem is it is not really well defined what happens if you mistakenly write to registers in the wrong order. Yes, that brings me to my first conclusion, okay? Linux host to FPGA connectivity is not rocket science. Why is that so in my opinion so what helps? You have on the hardware side not so many different standards that are being used although there are several vendors out there which you could consider and these hardware interfaces they are both supported by the Linux kernel since very long and also on the FPGA side they are either very easy to implement or the IP as for PCIe is free of charge and in some cases not all also open source. So for this specific aspect also I think that model based generation of source code for this interaction is helpful, it simplifies life because in the end it even allows you to have an interface agnostic communication and you're also avoiding vendor lock in. So before I conclude finally I would like to stress all of the code which we have come up with throughout our experiments is open source. You will find it on the Git repositories. If you're looking for a specific interconnect and you cannot immediately find it, feel free to drop me a line and with that I thank you for your attention and I'm open for questions. Questions, I think the next sessions begin, one second, yeah 545 are the next sessions. Have a good day everyone. Thank you.