I've really had it with these people. Remember back when Trump took out that Iranian terrorist? And the media reacted by describing that terrorist as an Osteer Scholar? Well, here we go again. You think that they can't possibly expose and discredit themselves any further, but they always seem to find a way. After news came that former Japanese Prime Minister Shinzo Abe had been assassinated by a lunatic with a homemade shotgun. The corrupt Democrat state media immediately jumped at the chance to give us all the Communist Chinese party perspective and use it all for their own political benefit. A polarizing figure, he was a right-wing nationalist and conservative and a fierce supporter of Japan's military. He fought to amend the country's pacifist constitution in the face of the rising threat from China. Seriously, is the CCP employing these people? While in office, Abe met former President Donald Trump several times to reaffirm Japan's military and trade alliances with the United States. His political opinions were controversial. Why? It's almost as if they're implying that he deserved to be shot. A great man and ally has just been assassinated and they use it to demonize him in the exact same way they demonized Trump and the rest of the Democrats' opponents. And for that matter, opponents of the CCP. On that note, it does look increasingly like the Democrat party and the CCP are aligned. Not only by ideologies, but by a common enemy, us. But now, we have even more blatant examples with Joe Biden actually taking American petroleum stockpiles and selling them to his son's CCP business partners. Which is just straight up insane no matter how you look at it. Now I'm sure the media's gonna cover this story, right? Right? Back to the CCP-compromised CBS News hack. I seem to remember her from something in the recent past. This is not just a morning procession. It's a political message designed to demonstrate that all Iranians are united in anger and in outrage against the United States. I just say the sentence for President Trump.  Soleimani's daughter was also on hand. Her father's death, she said, would bring dark days to America. So this is all starting to make sense. Her friends are enemies of America and Donald Trump. Now as you might have guessed, CBS News was not alone in their attacks on Shinzo. NPR and the Associated Press got in on it too. For example, the so-called National Public Radio outlet described the former Japanese Prime Minister Shinzo Abe as a quote, divisive arch conservative. From an NPR article, former Japanese Prime Minister Shinzo Abe, an ultra nationalist who served in the post longer than anyone else before, stepping down in 2020, was shot and killed on Friday at a campaign rally. It's almost as if they're implying that he deserved to be shot. After all, they always use labels like right wing or nationalist as derogatory labels. But they do not ever use labels like left wing, socialist or communist as derogatory labels. Case in point, these same people had very glowing things to say about Fidel Castro and his family after he died. The 90-year-old was still a potent symbol of the revolution. What was the best thing he did for the Cuban people? I mean the education, health. Fidel Castro, because of what he did in education and healthcare, or hate him because of civil rights and human rights, there is no doubt that he is considered here a revolutionary hero, not only in Cuba, but in many places around the world.  Yeah, he's horrible. He did one good thing, he said. I don't think, by the way, that literacy program that he was talking about was a re-education program. Yes, it was. There's a big difference. I am introduced to Ramon Castro, and it's kind of jarring because even though he was Fidel's older brother, he looks a lot like him. As he's presented to me, he leans over and gives me a kiss on one cheek and says, this is from Raul, kisses me on the other cheek and said, this is from me, and then he kisses me on the forehead and says, this is from Fidel. It was kind of like getting the blessing of the Holy Trinity. Yes, you heard right. This publicly funded, far left propagandist from NPR actually described a kiss from Fidel Castro after he died as, quote, it was kind of like getting a blessing of the Holy Trinity. Lastly, I wanted to thank one of this channel's very generous viewers who goes by the name Drunk Batman. He sent me this amazing woodburn sign of my logo that he made himself. I absolutely love it, and I plan on putting it up somewhere here in the background. Thanks again, Drunk Batman. I'll see you in Fortnite. Okay. Alrighty, that's all I have for that one. As always, please drag and punch that like button, share, subscribe, then leave a comment to vent your frustrations. Thanks a lot. Have a great weekend.


So We have a we've got a lot of gear over here on the demo pad you might notice and we're about to to use some more of it so next up we're going to be hearing from Direct TV and the AT&T Entertainment Group ladies and gentlemen, please welcome Amit tank and Sunil Jatwani Direct TV now on open stack. How cool is that? Good morning, and thank you for having us today Within entertainment we are seeing some interesting changes in the industry We see a the way the video is being consumed has changed dramatically in the last few years and is expected to do so going forward Additionally because AT&T offers several delivery avenues to deliver that video to our end customers with satellite mobility broadband and over the top our Goal within within AT&T is also to make sure a consistent viewing experience across all of those delivery platforms we see cloud as vital in the overall delivery on our business objectives With cloud we get the speed and agility that the cloud offers and at the same time Using some cloud native principles. We have pretty much decouple product from the underlying infrastructure And that's something that's also the focus of our STN and NFE initiatives within the company What this enables us to do is? adapt quickly and also innovate with newer and personalized services of Course we have to do all that keeping reliability and Scalability at the heart of it so that we can replicate that traditional TV experience across all devices and all delivery channels Like I mentioned AT&T today offers video services across a variety of delivery channels with satellite fixed IP Wireless and over the top the challenge that we had to face within the cloud group was to come up with a unique and common entertainment centric cloud architecture and platform that can serve the unique needs of Different consumer segments if you will across each of these delivery methods The cloud that we envision and the cloud that we have built was really built around a lot of these design principles Now in the interest of time, I'm not going to cover each one of them But let's just talk about a couple that I think are very important in this context Using cloud native and agile framework. We were able to pretty much re architect and redevelop a lot of our entertainment workload into micro services base framework that we run on containers today Because that really allows us to manage our code and as well as rollout changes faster also Something that's at the heart of this conference We use a lot of open source including open stack as a foundation for our cloud platform Now to go over specific workloads that we are running on open stack I'll like to call upon Amit to walk us through the rest of the presentation Thank you, Neil We're interesting indeed So as we go through the life cycle of the content Many unique cases we have very unique needs the content ingest ingestion becomes a very interesting challenge when you have to intake feeds for thousands of channels and a lot number of video-on-demand titles as The content moves forward We go to process it so that we can build meaningful business applications around that Now the logical block of content processing is very interesting We chose to run that on open stack for very specific reasons the encode and transcode workloads Have very high CPU requirements. They are very very CPU bound The content storage due to sheer size have that a unique needs as well The on-premise model actually becomes very appealing there the live on the ward workflow have high throughput and very very demanding latency response and finally as they get delivered at the content delivery side, they have very edge-like characteristics So this is a very high-performance solution. That's being powered by open stack and We've leveraged the core principles of domain 2.0 architecture that AT&T has championed before and we've been able to build on top of that So let's look at what are some of the characteristics of the solution We're running on Mitaka Very stable code base and this reliability was the reason that gave us confidence to actually deliver Super Bowl earlier this year on this solution Heat templates are used to Deploy live new channels and our team loves it our teams leverages CI CD to deliver workload as well as applications While taking benefit of bare metal orchestration for Ubuntu nodes Kubernetes and Docker Containers are used in production at scale to deliver our microservices so that we can keep our workload very portable Why you may ask that's because we love our hybrid cloud We see a lot of value to derive from there and open stack as a platform really shines Well, when you have to make it integrate and work with other components So this is how the picture looks today What are some of the things that we are excited about future? Well, let's take a look Containers on bare metal is something that we are very excited about Function as a service when combined with serverless computing patterns is going to become very very potent at solving ephemeral use cases ephemeral compute use cases that come and go in burst and finally we definitely see Modern more usage of Kubernetes in workload as well as infrastructure orchestration so that we can make our hybrid clouds more seamless So having covered that I think we are ready to share with you how our application looks like so let's do that I Am going to switch to the so this is a regular app that's delivered to you in form of all the entire product and We are going to switch on to a channel that is going to allow us to There you go, let's see if we can switch on to the audience network Okay, we have a show here and that's going to run in for this thing. Let's see if we can see a title here. There you go Okay So now I want the audience to take a moment to think about this The frame that you are seeing on the screen is a high definition frame and just a second ago This frame was essentially a bunch of jumbo packets that are being encoded packaged switched Inside a VM running on OpenStack and seconds later here it is Being being blind for your consumption So very very powerful indeed in conclusion on Buf of all the teams at AT&T We really want to thank all the community members and all the contributors for making OpenStack a really really compelling experience Thank you so much


I'm pretty focused on skiing and I'm pretty driven to be out there every single cover day making as many amazing turns with good friends as possible. You're kind of chasing winter all the time and you're traveling and spending a lot of time on the road but I think my favorite place is definitely being home. I was born and raised in Lake Tahoe. It's a place where you can experience the outdoors to its fullest potential I think and it's been a really great training ground for me in my sport. I'm excited to go biking. It's beautiful. Biking started out as like a new way of exploring my home. I've lived here for most of my life and I was like wow this is like Tahoe's reinvented for me. Like my playground just got really expansive within I don't know a few hundred feet I'm on dirt and it's just like this endless playground. Mid-morning, riding lots of bikes, having lots of fun. I think mountain biking is like so similar to skiing actually. It's just like very reactive you're making these split second decisions and choosing your line and then the airtime feels really similar as well. It's also like pretty similar body position. The bike for me now is starting to feel a bit more like an extension of my body and that is just this level of awareness that I have with my bike as I do with my skis. I think I'm a better skier than I am a walker. Biking is kind of starting to feel that way. My style of biking is pretty similar to my style of skiing. I think I love going fast. There's all these fun little side hits and different ways you can read the trail.  I think I probably feel more adrenaline on my bike going fast than I do on my skis to be honest. Pretty fun park here. It's really nice to have this in your backyard. Timing is evident when you're in the mountains and I like to be able to ski as many laps as possible and so yeah that endurance that cardio based training having that foundation going in the ski season that I build from biking essentially I'm like just getting ready for the winter it's really transferable it's awesome. It felt like the right time to enter into my first syndrome the Transcascadia which is a mountain bike race up in the Pacific Northwest and it takes place over six days and it's about four to six thousand vert of climbing every single day. I definitely want to feel fit enough to drop in and not feel worked from the climbs.  I'm already noticing it first year on the bike just feeling more power and more strength and then for the downhill part what I'm really excited about is I bought a pass to the bike park at Northstar and Tahoe and basically the whole purpose of that was to get as much descent time as possible and riding the lift you're just making laps and getting you know ten times as much descending as you would if you just went out to ride. Biking's really been the one activity that I found to bring my muscles completely back after injuries and it's a good way for me to kind of stay mentally sharp too and like okay I can do this and build that confidence up which is really cool to get in the summertime. We did it! Mental health to me is equally as important if not more important than physical health if I'm not like totally feeling sound and right in my head then I can't perform on my skis. No matter the season I think being in nature helps me connect back to things that really matter to me in my life it's my playground it tests me I learn a lot of my like biggest life lessons from the mountains specifically but also from all these different elements that are all around us and a lot of ways nature has been very healing for me. Only the centerpiece of Lake Tahoe. It might be the bluest and the deepest lake and the most clear lake in the world. The lake itself is so refreshing I try to jump in it every day you can do all kinds of fun things you can surf behind a boat you could just jump in to wash the mountain bike off you it's just this place where you can kind of refresh your mind take a break a nice little pause check in and and just enjoy the beauty of it. It's kind of chilly. That feels so good to get in the water. So much fun. Good way to end the day. I think I really enjoyed taking the mental break after. I like it. Snow melts and eventually goes away and skiing's gone for a while and it forces you to take a step back. I really appreciate the seasons everything seems new and exciting even if you're skiing every single year it's like this refreshing I don't know this new beginning every year every time the snow falls it gets me excited. One thing that I've always told myself with skiing is that it's my favorite thing that I do in my entire life and it's always gonna be fun and remain fun being in the moment with your friends on the outdoors that's something that always brings me joy.


Sometimes it's okay to take a break and lay in bed all day. I do it literally all the time. And sometimes the guilt will creep in, oh, I should be doing that or I'm not being productive or I should be doing this. Oftentimes that type of mindset is because you value your self-worth based on your productivity. And I went through this for years and years and years, especially being in the sales industry. So you have to practice being free. You have to practice relaxation and you have to practice, most importantly, not feeling guilty about not being productive. Skip the gym, eat junk food late in bed every once in a while. As long as these things aren't becoming your daily habit, it's okay to step back and give yourself some rest and take a break from these rigorous routines that we put ourselves through all the time. And we need these days off from the gym, these days of eating junk food, these days of laying around and doing nothing, just as much as we need the productive and positive days. Like and subscribe for more content on how to improve your.


So Ephesians chapter 5 starting in verse 22 wives submit to your own husbands as to the Lord That's all I want to say Ladies I thought I'd make y'all laugh for a second because the moment women here submit to your husband You see him get seen a few y'all throw me some gangtons.  So wives submit to your own husbands as to the Lord verse 23 for the husband is the head of the wife Also, Christ is the head of the church just like Christ is he is the savior of the body Therefore just as the church is subject to Christ So let the wines be to their own husbands in everything verse 25 husbands love your wives Just as Christ also loved the church and gave himself for her. Let me pause right there When we start talking about the roles of a husband and wife How many you would agree that what we see in movies and hear in music and what our culture present is? Absolutely not consistent with the word of God. Would you raise your hand? Okay, so stop taking cues from the world That's the first thing If you're here today What we're about to talk about and hear the rest of the day you have to determine in your heart you have to determine Do you really want to follow God's plan? Because you can hear hear hear that's why I love what the scripture says don't be just to hear but do it You know and I'm telling y'all from the perspective Um raise your hand if you don't know my story just raise your hand real high if you've never heard about my story All right. Well, let me tell you. I'm not up here speaking From a place of perfection I Come from a crazy background my mom was married six times 14 schools 17 houses Really? Yeah, I seen craziness in the biggest way I'm the first person in three generations of my family to be married to never cheat and never get a divorce That's only the power of God We have gone through Extremely difficult times not only because we both came from different backgrounds. I'm from Louisiana I know it's hard to imagine I Know y'all thought I was a Yankee And my wife is from, California every bit of it and And I'll never forget You know when we first met You know it was that church And actually she wasn't a Christian yet I was a new Christian and I felt the call of evangelism and I want to evangelize And thank God I didn't the Lord touched my wife Saved her and then I wanted to disciple her and he said no you'll ruin it so He knew my motives wasn't exactly pure Because when I saw her Believe it or not. She was the exact Dream better than the dream when I'd become a Christian. I said well, I'm a Christian I want to obey the Bible and Can have sex so I want to get married I mean there's more to it, but come on. I'm gonna just be honest. I Don't want to sin against God if it burns. Yeah, you're married. All right, so So I was I was like, oh Lord, I really want to be man I want to share life and and I wanted love not just physical I wanted love and I remember that you know, you know, they'd say getting your prayer closet and you know My IQ of 36 I I went and got in a closet And I took a pen and a pad and I started writing down what I wanted because somebody had said in a Bible study Just really I mean be specific in your prayer life for what you want. So I was like, all right.  I Don't know. I just thought it'd be cool, you know and Sure enough when I met Eileen as she walked down I was I was actually already I was up front at a Praying for people and before service, you know, we're praying for people and then I see her walk in the church She's walking down.  I started looking all the other single guys Well after we got to know each other and and and all of that and then we you know It took her about a year to realize Mmm-hmm And we got sweet on each other Guess what she turned out to be my dark-haired green-eyed not only in the fitness. She was miss fitness USA and She didn't tell me any of this till way later. She's like, I have to tell you something. I was like, what? Because I come from a crazy background. It's like we've known each other. She goes. I just I just I Was I want to contest I said, but what?  I mean, I remember the first time You know, I met her family We're gonna go swim and they have a pole and it was up and it was So thinking hot so we're gonna take a dip in the pole and I never forget I was in the pole house guest house and And I remember seeing her walk her and sister come across the thing and she had a bathing suit on the first time I seen her in a bathing suit and she just very cute figure and I was like Yeah, do some push-ups And I literally start trying to pump it up guys She had a six pack more than me and I was But all of that and with the cherry on top she had a twin sister You guys I Just know this that God desires for you and I to have a relationship with our spouse That's incredible That's beyond even what you can imagine But oftentimes because of the darkness of this world our background in the influence of the demonic in spiritual warfare He keeps us from becoming all that we possibly can amen So the roles that God has for us You know, if you've married your spouse You're married.  I Just made that up But I think as we really fulfill the role that God has called us to be in our marriage Guess what your spouse starts to become more attractive. Amen yeah, and That's one of the things I loved about my wife even though we came from pretty different backgrounds and She had to grow in the Lord as I did I never forget We were sitting on a beach and we were in our courtship phase and she loved the Lord from the beginning I mean just an amazing God of Gal, but I never forget. She said this love Jesus Bible and it's just so amazing But there's a part where it said why submit to your own husband, that's weird It's exactly that this scripture right here now it whoa Lord us for you to work on But what we need to do and she didn't understand the time and let's talk about it a minute is really what does it mean? Submission well listen how quiet everybody got Gals are like, yeah The guys are don't get scared now Vic bring it man back it up Well first of all, let's go back to the beginning in Genesis and as a pastor referred to in Genesis chapter 2 18 that God brought he primarily the role Was for her to be a helper his helpmate and I tell you what I've had a privilege of studying and reading and learning from from everybody because I've had to In order for us to make it. I've had to have my mind Transformed and renewed so I can't follow my my family's past you understand so whether it's dr. James Dobson on focus on the family and the year she had and building the family and Relationships actually had to go to work for him to get it Yeah, I thought Lord. I'm so messed up you place me here in the ministry with them or Dennis Randy and family life or whatever grow and learn You guys have your mind transformed because of the word and Tell you I to know that God brought Adam Eve to make her his helpmate While all of us are called to be helpers to others the Bible places a special emphasis on the responsibility for wise Genesis tells us that God Realized it wasn't good for man to be alone and he decided to make a helper suitable for him Interesting to note that the Hebrew meaning of the word helper in this passage is found hereafter in the Bible By referring only to God as he helps us The fact that the same word is applied to a wife signifies that the woman has been given tremendous power for good and her husband's life see Ladies you are called for great influence in your husband's life if you look at it that way and It's not just you submitting to him Being lorded over by him.   I Mean I'll remember our first baby the babies is like And I just went I don't give them back to you right there And I said I'm gonna be there for you I think that but that little thing gotta get bigger Because I'm gonna break it gals you have the unbelievable ability To nurture you are more of a nurturer and than the husband does that make sense? What power what influence for good that you have? second thing is women your role is to respect your husband or ESBEC Respect your husband. What does that mean?  He ain't got a lot of wisdom You should see how he does that checkbook Well, then in your role of influence and helper complete her tell him honey Would you want me to balance the checkbook and kind of run that? Would that help you?  I'll just be the CFO And it's not in a manipulative way as I've heard one white say he may be the head, but I'm the neck the gals Sometimes your perception of a man because the way men have been conformed to be macho and you know, I got this Men have needs mean have needs We need to be listened to We need you to help us build our confidence in the person hood of our manhood. Am I right man?  He's so strong My wife still brings me jars to open right honey, I just can't get this some And yeah, I don't matter if I'm on the computer watching TV.  What's that? That about to break it open another victory.  My wife would back kick me Ah Your Wounded if you lose the last time you gave your husband just those eyes of My wife still makes me want to show up for her The first time we were visiting out of church and we're getting sweet She came to I have a background as a martial arts instructor and I remember she came to the karate school one time She just comes in and she's sitting down all cute And all of a sudden I became Bruce Lee. I was like What's all right there? It was sparring time. I was beating people everywhere It was just a kids class I was like, yeah Girls men want something to fight for they want something to conquer and and and and it's for you if you hold our heart and safety We just celebrated 25 years of marriage this year praise the Lord And we actually renewed our vows and when we did the only people we invited were our children We have five children three adults and two little ones and we let them preside over the vows We we said y'all y'all do this mom and dad will show up and they got flowers And they they took our rings from us and give it to us and then they each one of our children Told us vows that we were the take again They had their own I know When I kids will you promise to kiss mom good night every night? Yeah Like that one. I mean it was amazing and then each one of our children Told us what it meant to them that we've stayed together. Oh There were tears flowing because we've come through hard times We were sweeping separated twice in our marriage one time I loaded her up in the car with the kids and said No, I said babe you you need to go because I I suffered severe PTSD real acute and When you start taking your k-bar and stabbing in in the walls and riding cuss words and pulling on your rifle and start tweaking Sometimes that is the better road of valor to say honey They got me on medication, but we don't know what's going on going up to your mom and Her having to trust the Lord with me You guys And gals I'll tell you right now the whole idea of submission or allowing your husband to be head of the house And you come in that's being subject to him just like he is of the God isn't an issue of trust and Really so much of us based on how much you trust the Lord With the craziness that we've endured You have to just trust God and even allow your husband to make bad mistakes. I Mean we came out a hundred and twenty three visits to a trauma specialist from our brain I've been on debacle Debra King Prozac Zoloff with them. Yeah with me Yeah, so imagine my wife having to deal with that there were times she would look at me and say honey What you're thinking is not reality. I Go, yeah They're coming to get us Well, I need this right here. They're right, honey. It's not reality And sometimes she would describe me And kiss me and hold me and say this is reality Your love the influence of a woman you gotta trust the Lord Respect your husband and love them Tied us to four that it exhorts the ladies to love your husband and part of that love is physical union is intimacy And that tends to be one of the really most challenging areas in relationship and I have people tell me all the time, you know, so I mean, what's a healthy relationship, you know, if you're gonna Like, you know, how often I say five or seven times a week And I wait for them to kind of sputter I go for some people But you know, I think dr. Dobson was I said two to three times a week is a healthy physical relationship because intimacy is so much more than just that it's it's glue in your relationship It's trust it can be ministering it can it's healthy.  That's nonsense. It's sin. It's destructive Allow God to make your relationship how he wants and you may say all it's messed up We and had what trust God watch what he can do. Have you ever prayed before? Have you just held hands? I said Lord men You know how we like to be close, right? Amen What are y'all guys just say amen, come on Thank you. He said at the men's conference now your vibes here Right now Victor What do you think about me I got to be out in the foyer you afterwards all these women come up to me Gals, I will tell you something that my wife says that marriage retreats and women's deals She said when my husband travels without me, I make sure he is well-loved up Before he hits the room The enemy is use sex enough for destructive measures not only in your life and in your past So glorify God with it does that make sense and men be subject to God if you want to be close to your wife I have a thing. I typically say 99% of the time when I'm feeling desire. I want to be close to my wife I just always pray a little prayer Lord if it be your will and then I'm okay either way Most of the time now God has blessed us in unbelievable ways and I think it's because we kept our Relationship pure both of us coming from the world But when we met we said no we're gonna honor God this way and I tell kids that to this day When I speak in public high schools would say bring me in as an inspirational speaker I tell kids one of the best thing you can do for your sex life. It's to stay pure and wait till you're married Guess what they applaud they applaud kids and kids in this area I did a high school and they applauded because I tell them the outcome of it apart from Marriage can be very destructive Very destructive. Let's show the world Celebrate that part of our love in a way that so glorifies God people in the world go Wow You really are happy. I never get I told a group of young juvenile offenders locked up In this county and I said, oh you need to wait Fellas one guy said oh man this player got to share his love and I said no I hear you I hear you But I said that ain't the mark of a man Mark of a man is loved with one woman your whole life being faithful to her and pleasing her Come back to me in 25 years when you can say that and you'll know the truth and he was like That makes sense All right, so praise the Lord, let's let's see you men Men listen guys in essence God's called you to be a leader Some men are natural some are not Some men are strong and natural leaders Those that aren't how can they really leave their home if they're passive by nature? Paul says the same to everyone God has placed a husband in the position of responsibility Men you're the lead courageously reject passivity lead Leadership is not fun all the time and it's not about being popular. Say men When your house is in chaos when you got kids upside down when you got teenagers being Vessels of the wicked one. Okay. I don't know what else to say and maybe your wife You know got them biblical court and trying to control everybody and hopefully Men sometimes you just got to stand in the gap and say hey This is the way to cow eats the cabbage.  No, I'm not talking about that.   He's a he's a good young man He came and asked me permission to date my daughter There are still young nights out there Who willing to honor and do the right way, but we have to keep the bar high Do you understand men lead right and put the fear of God in them young men? I just simply said you know my background. Yes, sir Anybody here's my daughter They're gonna die and Be dismembered But not in that order He just tapped me said I completely understand. I said good. I hope it works out.  It's it's ridiculous How men are passive and don't lead it reminds me of the story of It was a sign up in heaven Men were lining up in this big sign said all those men who've been dominated by their wives stand here The line of men just stretched off for affinity And then it was a second sign that said all those who have never been dominated by their wives stand here And underneath that sign stood one lone man So one of the guys on the other one he couldn't believe you to walk over to him and he asked him Sitting hand. I just got to know What's a secret? How did you do it? I mean the other lines has millions of men and you're yet You're the only one standing in this line that said you weren't dominated by your wife The man looked around kind of puzzle and said I I'm not sure my wife just told me to stand here So men you got to reject passivity be a servant leader The key to be a great leader in your home is to be under subjection to God Almighty. Are you I'm asking you right now as a man Is your life under the Lordship of Jesus Christ? Do you know Jesus Christ have you ever surrendered your life to him as a man and said God? Forgive me of my sin. I Surrender all I am and hope to be to you Make me the husband you want me to be make me the dad you want me to be if you've never done that you need to today It's the best thing you can do So here you are with the responsibility to reject passivity be a leader love your wives unconditionally Because you know what let her know what her worth is She is God's gift to you And sometimes you may not feel that but it's still the truth Do you know when I don't feel a certain way how I combat it? I love my wife even more I just say it out loud. I love my wife best thing ever happened to me. Hey, hey Thank you God. Thank you God. Thank you God God I love my wife. I love my wife Just my wife to me she says honey. Thank you. I want to thank you from the bottom of my heart You've made me you've increased my prayer line you you've made me really want heaven so bad At times and I just said you're you're welcome What did she mean by that? Serve your wife lover and condition to let her know her value, you know for our 20th anniversary I did something that we we couldn't really afford, but I did it anyway sometime in you just got to get nuts 20th anniversary I went and bought 20 dozen roses red roses and had them in our house. Oh, he came downstairs and she was like And she said honey, you know, we're in ministry a lot of money.   Guys like I didn't even know she had a need Yeah One of the best ways you can serve your wife is to understand her needs and try to meet them Do you know what your top your wife's top three needs all right now seasons of life determine it Maybe she's struggling with little ones. Maybe she's wanting to be more of her grandmother Maybe she wants to play the piano Maybe she wants to go train with the Navy sail and do shooting Like we did last week. So, you know Last thing is another way serve your wife is to provide for her Man, I think providing for your wife women need to know that you care and want to provide They need to have that security of or at least my man is providing Also, they need to feel protected.  Wives is that true am I just making this up? Yeah, yeah, yeah, I mean why is want to feel like it's my man. I don't care if he's like this. It's my man I love that little guy. I'm not kidding It all I gotta tell you By a pistol listen They want to know you can provide they want to know you can protect when we were first married my wife was pregnant first baby and She had gone down we had a karate school in this little strip mowing She went down and she was trying to do some business at a of all things a copy place and the guy who owned it My wife said I don't really know any belittled her Don't belittle a pregnant woman, right and my wife at that time she just kind of stunned She was stunned because he was like can't you can and so she just walked out She came back and she was crying my wife. I was like, well, what's wrong? Is it the baby? What she said no this guy just really just belittled me and I said, oh my What does he look like And she told me I walked over there I remember walking into the business said hey, buddy. Can I talk to you? He was what do you need everybody's turn around? I said, I need to talk to you outside. I walked outside I said, do you remember that little pregnant guy that was in here? Yeah, you really belittled her. That's Disrespecting a gal, which is wrong and it's really wrong because my wife is really really wrong because she's pregnant Really really wrong because I'm a black belt and then he tried to posture Let me tell you what I said.  That's kind of a side of the salt Father you I'd put them in your pockets right now and he goes in anyway He apologized to my wife that'll work out good I remember Provision I remember we're first married and we've had lots of money in our life We've had good businesses. We lived in Hawaii. We know wealth and we know, you know God's provision of ministry of All right, Lord And you got God is the one that's faithful you seek him, right? But I don't forget we were praying for our second baby and we needed diapers and milk And I was waiting on a paycheck as a young man working at health club and I never forget I went over to my brother's house. This is down in Louisiana. I said man. I gotta get some money real fast Can I borrow your lawnmower? He and my brother knows me. He's now president of an oil company. He got rich and he said Yeah, more than the back gas. I'll pay you back for the gas He goes you don't have to I think that more and that can I went door-by-door in that neighborhood I said hey, I'm waiting on a check. I get in two weeks. Can I mow your lawn?  I think it was will Rogers That said our country needs cleaner minds and dirtier hands Hard work. Well last thing I must say and I'm out of here. It's been so good I'm enjoying the state is if you don't think that there's a real literal devil with demons that are signed to you and your wife You had better wake up Spiritual warfare against godly marriages is probably the biggest attack and risk ever And if you don't believe it's real guys wake up because you maybe want to protect your woman against them Let me tell you what you better protect her and your family against the demonic One time my wife was struggling and she was just being she was having a hard time the Lord showed me He showed me in a vision. She was being attacked by demonic forces And I felt helpless because I was like well make them appear so I'm ripped their eyes out Lord He said no you get on your knees and you enter seat for your wife right now That's where the battle is gentlemen. I'm gonna show this quick video That how many of you have not seen the film on our story raise your hand if you have not seen our film, okay? All right Well, we wrote a book and it was a film that was done on our life story And it is one of the most amazing love stories because what God can do But we did an opening scene For the film and it's called marriage is worth fighting for and I just want to walk you through it for one moment And you better believe you need a cleave to your husband and wife When you're married now, this is interesting Metaphorically, we believe these are demonic forces that would come against our marriage on floor. I'll get it First Peter 5h says be alert and sober mind Come on, let's play it. Give me your money and the lady to remember this one John 10, buddy, you can have the money, but Thank you That's not what I meant No, honey, I wasn't mean that end of the rod gatoria. Say no more. What do you think this is a joke?  What do you think I? Put the script up there again So do you want the lady don't say what she just did there right there You guys just know the warfare is real, but you stand in authority that Christ has given you you can have victory amen


All right. Hi, everyone. So first of all, I'm not a fan of Metal Gear Solid. I'm just a fan of cultural references Yeah, I've said I am Hinnick and I'm here to talk about last year About my last year because after I've wrapped up Python US which was in Portland Which is the Pacific Northwest of the US. I took advantage of being that far away from home and Took the classic road trip people just do usually in white Cabriolet's I went from San Francisco down to LA up to Las Vegas through the mountains back to San Francisco And a small detour to Hawaii Now this little tour plus Python took me more than five weeks Which means that I've been more than five weeks absent from work and with every photo I've posted People kept asking me how I do it and whether I still have a job Now it turns out I still do have a job last time I checked and And that job I'm responsible for almost 70 projects And it still worked out that well that I'm gonna even take seven weeks this year when I'm gonna Spend a month in Cape Town, South Africa. So let's see if that's gonna be another talk But it also turns out that the answer is a lot longer than 140 characters So that's how here at the airport of kawai with a seven dollar plastic cup of beer This talk was conceived and I'm here to tell you my story or more like the bits of it that fit into 45 minutes That I think are vital for my sleep I'm gonna touch on a lot of topics It's gonna range from DevOps see things like people are usually used for me, but I will also talk about General software engineering so there should be something in there for everyone and Each topic by itself is a talk by itself So as usual there will be a link at the end of a page with all the links for you to study If you want to know more about certain subtopic Now with that said let's get going Like every big change It all starts with your attitude you have to start prioritizing quality Which unfortunately means that you will spend a significant time on non features and That brings us directly to incentives Because nowadays everything is top priority if you ask your manager if quality is a priority They will probably tell you that this is the biggest priority ever However, how are the incentives if you don't all have time to write tests to instrument your systems It's kind of like a construction company that claims that safety is number one priority But forces their workers to buy their safety gear out of their own pocket and It's true that reliable systems do take time And if your only performance metric is to ship new features you may have a conflict of interest here However, the good news is that there is a solid business case to be made for rely up for spending time on quality and reliability and That is that the fewer Urgent interruptions you have because something is going up on fire The more you get to work on important stuff or you put it the other way around if you Build a lot of unreliable systems unreliable programs apps, whatever there is an inflection point where you just Stop being productive because you and your co-workers are busy fighting fires all day And you do not get to ship a new feature at all And we've seen it with a lot of startups that have a very fast trajectory and then like two years nothing happens Well, this is what happens And this is also how I ended with 70 projects because I'm forced because a very small company I cannot just tell other people to fix my crap. So I'm forced to build quality stuff and just accumulates This of course takes some long-term thinking Which is certainly not too common in our line of work and unfortunately that is the part I cannot help you with Yes But yeah, let's get more practical So the first thing I'm gonna defer to authority and no one less than Tony whore who gave us quicksert was quite accomplished with all things concurrent systems and What he says is the price of reliability is the pursuit of the utmost simplicity and if you prefer a Dutch sage Which at a Python conference is completely reasonable then dice grass said basically the same So you need simplicity Otherwise there will be no reliability and at this point it's important to stress that you must not conflate Simple and easy because easy solutions usually are not simple easy solutions usually a hack and Simple solutions like solutions that will serve you well in a long term. I usually not easy to find and I find a good way to approach simplicity is from the other side. So let's talk about complexity for a moment So complexity in software Is something I would say is the number number of concepts You have to keep in your mind when you are trying to reason about the behavior of a system of a program of a piece of something and Especially if you how many things you have to keep in mind when you're trying to reason what happens if you change a little corner of a system and Humans are naturally limited in the numbers of things we can a juggle both literally and figuratively in your mind and if you have too much things going on too many balls in the air you start dropping things and That's when normal accidents happen This term has been coined by Charles Perot in the wake of the three mile island incident, which was nuclear power plant incident in the United States and It's called normal Because they are inevitable in extremely complex and tightly coupled systems Now if your program is a contraption like this It's impossible to reason about changes What will happen if you play with this knob? it's impossible to tell and What breaks if the part with the knob breaks? It's I would say it's impossible to say but actually it's very easy to tell the whole thing is going down in flames That's the problem of having tightly coupled complex systems Now the irony is that if you try to make this thing safer You're adding more complexity What does it mean? The system is more likely to fail So you cannot fix a system or a program that looks like this. You can only rewrite it from scratch Now when you're talking about complexity We have to differentiate between two types. The first one is the essential complexity, which is a good complexity That's the complexity your customer or your boss is paying for you to solve It's inherent to the problem you're solving. The other one is accidental complexity This one comes from using wrong abstractions Having cumbersome deployment procedures that stop you from deploying or just using ancient and inadequate tools like Python 2. Yeah, I cannot drop this mic so so Obviously only in a perfect world you can work only on On essential problems, but still you should always keep in mind what problem I was solving here Is this essential or is this accidental? How is your ratio between those two? Now what a simple software and that of course is a talk by itself If not a conference by itself, but given we are in Italy. I really like the ravioli metaphor now Given that normal accidents happen in tightly coupled and coupled system tightly coupled and complex systems It follows that you should prefer to have simple objects that are self-contained and it have simple relationships Just like the ravioli Because if you look at it They are small. They are self-contained. Nothing is leaking out unless you overcook it and I heard in Italy There's jail time on overcooking pasta Yes, yes That's an adequate thing to do now in other words your objects your functions should do as little as possible But more importantly they should it should know as little as possible about other objects and other functions And this gives them a clear interface which you can program against and The few assumptions help you to have simple relationships because as many object oriented mentors say before Dependencies will kill you and by dependencies. I do not mean things to install from pipi. I Donald stuff is doing great work install as much as you want but dependencies between your objects and especially I really like to Not have bidirectional relationships, which are almost always bad I like to think of an object graph like an family tree if you have loops in your family tree It's a bad sign right, so Think about your object graph similarly Now all of this if you follow that gives you system components that are really easy to test because you have clear interfaces that you have to Either fake out or a mock or whatever you're doing and it's few dependencies. This is good Now bad designs staying on the pasta theme Um, it's the opposite it's back big classes Also known as God objects and a God object is an object that knows too much and it does too much And those are obviously very hard to test because just to instantiate such an object You need a dozen of other objects and you get yourself to fight accidental complexity really really fast The sad thing though is that God objects are pretty common in the Python ecosystem and the reason for that is That writing classes in Python is a bit annoying, but I will touch on that later Another mostly self-inflicted complexity is Basin your design around subclassing or as Corey puts it You will regret it and I'm not here to Hate on subclassing per se although it's I think it's well-known that I personally try to avoid it But subclassing is subject to a lot of misuse So if you think back what subclassing has been invented by its inventors for is specialization Not code reuse if you're using doing a subclassing for code reuse you're kind of Making the inventors of subclassing sad So don't do it and there's a bunch of rules around it like LISC of substitution principle the open-close principle All these things and if everyone followed those rules there wouldn't be any regret Corey is talking about here and it takes actually a lot of experience in design and Modeling to do subclassing really well now The thing is subclassing makes your software always more complex to understand. It's easier to write It's less typing no question there But it is harder to understand because you end up with namespace confusion So where is this attribute coming from can I name my attribute like this? Or is there some attribute up there in a inheritance tree that has the same name and Everything will break You have to understand the MRO your logic is not only distributed between in one level like one method calls other No, you have to traverse hierarchies to understand where your calls are going through so This is all not great, but it is about making trade-offs So I'm not saying you must never subclass, but you should be aware that that you are doing a trade-off here between easiness and complexity and More importantly, this is not where you should start so a subclassing may emerge as part of a design out of practical reasons, but it shouldn't be the thing you build everything on and So don't start projects by inventing hierarchies and As a reminder many new and modern languages that came up in the past years like go or us Doing just fine without having any subclassing at all and I'm gonna say that Swift only has Subclassing because Apple needed to ensure compatibility with objective C So Relatedly Meta classes very powerful Often overused there are a bit of pythons Monads because people tend to read into them then they find them interesting and then they've read a blog post Explaining everyone how great and simple it is, but it isn't You're again paying with complexity for mostly syntactic sugar So leave them to David Beasley. He will do something nice and depraved in his evil layer and Everything one is heavy. So just to be clear. I'm not talking about abstract base classes here Which are meta class and subclassing. It's just the way they are used So that's something different. Although I personally prefer the approach of the soap interface which using Which uses class decorators for it? But yeah, ABC's is not what I meant here Now I mentioned that writing classes is tedious in Python. So I decided to do something about that And I brought address. How many of you have heard of address? Okay It's getting better by the year so those whose arms stayed low are probably restless in your seats Dying to tell me about named apples, right? So turns out I do know about named apples and maybe I know too much about named apples And it's also such a thing that people discover them and it starts reading out that it's the best thing since sliced bread So first of all, that's a low bar because carbs are really bad for you But other than that named apples Have a history at least that's what I've been told by some ancient Python sages They're made if for the standard library if they return a tuple so they can attach a name to the tuple So it's more readable. It makes sense right named tuple tuple with name so And they're great for that But they are terrible as a class replacement Because then you end up with the tuple type in your inheritance tree, which means that you have very odd rules for equality You have accidental iteration. You can accidentally unpack your class which gives you very weird errors and Yeah bucks which Don't make any sense or you won't attach a method to a name tuple. Well, you have to subclass it You want to influence the initialization have fun implementing down during you? so but people do it because it's convenient and That's why I wrote errors because named apples if you ever wonder when to use them It's very simple. You have a tuple and you want to attach names to it You send a name tuple other than that write a class or use address which will write the class for you And it has many more goodies like validators converters default values including factories. So for the few poor Didn't know about it. Check it out and you may be asking you know Is this a serious project you should put into production and I'm so glad you asked. Yes, it is I have stickers get them Now moving on operational complexity Which is a complexity of running your infrastructure or parts of infrastructure and I think it's not controversial to say the distributed systems are hard, right? Now let's look at a very very simple one which many wouldn't even consider a distributed system We have a client that speaks to a content delivery network, which is not part of your infrastructure We have an application that has a work queue salary for example a database and a radius cache so far so standard Many run something like this I run something like that now the problem here is that every box You have there is a point of failure if any box goes down the whole thing is down, but wait, there's more Every arrow between those boxes is a point of failure to so in this case You come up with ten independent points of failures Which if you know something about probabilities they almost add up and The reason is simply that network partitions are a thing You may live in denial because it never happened to you before But it will hit you to eventually so it's better to accept your fate and learn from others They are kind of rare if you don't run at Like a huge scale, but especially if you run at scale rare occurrences are kind of common Now all I'm saying is think twice before you add more boxes and more links between those because each of them Introduces new exciting ways to fail and with that in mind. Let's talk about microservices Microservices is about splitting it up an application in many small web services There are many great reasons to do that But you end up with many more red boxes and many more red arrows between them So your big monolith that has certain annoying properties. We all know about ends up being a highly distributed system Which you may or may not be equipped to deal with well Because now now you have new decisions to make for example.   What message bus? And all the new problems that come with that like service discovery You cannot really run this without service discovery which service discovery Aggregate logging you cannot just look at the files anymore Tracing you shouldn't even think about microservices without having a distributed tracing In place which by the way, I heard there will be a talk later today about it. So you should check it out And of course all the other fallacies of distributed systems that we like to forget and that come back and bite our butts later Now what you really want at what you really need a boundaries You want to define and adhere with Between your Boundaries between your modules and your packages. That's what you really want You do not need a network between your classes Okay, and then you can have also separate teams working on separate parts if you have clear interfaces But once you have that it's very easy to graduate into Into microservices because the boundaries already there now you just put a network between them and there is a place for microservices if you need to scale out because as we know we have a thing called the gill in Python Which is kind of annoying or if you live in a heterogeneous environment like I do I need to interface PHP and pearls so Web services it is now in that way I need to stress that complexity is not the devil by itself But it's a price you pay to get things done So you should more consider it like a currency, but you have to be conscious of your budget Because the budget depends entirely on you on the time you can invest in running something that on the human resources You can throw at this problem And the price can be quite high. So if you take for example, Kubernetes Which is an amazing product? But it's super complex to run It's easy to set up.  I tend to act stupid Just today I forgot my wallet But you act stupid for good reasons too for example if you sleep deprived because you just Came over Atlantic and our jet lagged or the baby cried all night Or you're at the busy airport on your way to a conference and you to quickly deploy a hotfix or You're hustling really hard for to satisfy some well-meaning venture Capitalist that has the best intentions for you and your future Now I have opinions on that and my opinions are the same as the one of John Allspaugh the city of Etsy I don't believe in human errors I believe that if a human causes an audit outage It's a system that failed to them and if you remember the big S3 outage earlier this year There was a great post mortem written by AWS and you should really read it expect like any post mortem is worth reading but this one is also very good and It turns out that a human used a tool the wrong way and it took down half of the internet Now the post mortem doesn't focus on that it focuses on the tool and how to prevent this from happening again Because people make mistakes and it's your job to prevent them from doing that you need to protect them now What when building tools and API's you should always assume that the operator is dragged from the dentist Consoling a crying baby or is just sitting in a boring conference talk There are many reasons to be distracted from the thing you do and still need to do it so take in an account It's kind of like products for people with physical disabilities It always turns out that those products are better for everyone and That's true with this is just as well And if it takes just one click or one API call to lose all your data Someone will make that click and will make an API call Maybe they were just cleaning their phone and just hit it by accident But they will do it. There was other data was the story of a junior developer who kind of accidentally dropped the database of some startup and He got fired and the CTO was threatening him with legal action and everything and it turned out that setting up a development and development environment Consisted of running a bunch of Commands you had to copy and paste from some document and you have to replace passwords and usernames and If you have sharp edges like this, it's your fault if people cut themselves or cut your company So in this case they have they should have fired the CTO right away But yeah, you know what it is Anyhow part of that is how you handle input and it doesn't matter if it's The input is maliciously Broken or by mistake you still have to be careful about what you let in and I fundamentally disagree with postal law here I think you should be conservative in both what you send and what you receive and This by the way was neither malicious nor mistake a mistake. It was literally the only thing I liked about them Francisco now in validator I Like to think of it of a time bomb and I'm talking about things like an invalid date string for example If you have something like that you let it in your system It starts wandering through your system and the deeper it gets the more damage Will it cause and the sooner you catch it the better can be a response. So just for an example You let the user Give you on a date string if you catch it ideally in the browser you can you can tell the user Okay, this is wrong. You have to do it like this if it reaches your ORM You'll probably show you will get an integrity error and the user gets a 500 That's not good. Other edges are of course your thunder in it or command line parsing just so Validated edges Almost I think validation is not enough I think you should always strive for a simple canonical form of your data that the rest of your system can rely on Because your business your business coach would not know what even Jason is it should just take dicks or on a native Python data types or if you know the structure of the data it should actually use a class and I find this The decision between the classes and dictionaries is actually pretty simple if you don't ever iterate over the keys of The dictionary you probably should use a class because a class is much better at catching typos and other little accidents while developing it while Dictionaries make sort of work and just shadow some mistakes for you Now passing strings around in general is also Not great Because if a function takes a string it becomes a parser It might be a simple parser, but it's still a parser and parsers are not that simple And if they screw up, it's it can be a really bad So I don't like strings in API's at all So and if you need a symbol or something in Python 3.4 later We have enums and there's also a pi PI package that will give you the same classes and they are great Now I could keep talking and talking however given that complexity leads to normal accidents and computers kind of complex right and Distributed systems are kind of complex to the second It kind of follows that failure is inevitable Everything I told you before is about minimizing risks and you should do it, but eventually something will fail So if you work in computers Failures are part of your life where you like it or not so in practice your reliability will land on a spectrum somewhere between Twitter in 2007 which had 98 98 percent uptime, which means a downtime of six days and NASA in 1969 will land it on the moon Despite a human error on descent, but the software was robust enough to compensate for The stupid mistake of the astronaut and I think we can argue that astronauts generally are not stupid and yet they make mistakes Now the problem with NASA's Reliability is that you need an actual genius to write your software a genius that will just Invent software engineering on the site while writing the moon landing software no big deal So you might have to scale down your expectations unless you have miss Hamilton on staff in that case I would like to apply for an internship So failure is inevitable inevitable all you can do is minimize risks You can prepare for it to happen and then you have to deal with it Don't ask me how long it took me to get this done So So and I'm gonna give the rest of the talk away right now a long vacation is the result of good failure containment and Solid recovery and let's shift gears here and talk about that So you've embraced failure. How do you expect failure? So if you need a system or a program do anything Reliably you then you need to monitor it. It's that simple because for all you know It's down if you need to have confidence on it working. You have to monitor it and So you need to check for outages you need to instrument it You need to instrument your system and you need solid error reporting because a silent failure is terrible I've talked about these topics in the past two years my taste didn't change at all I still love Prometheus for instrumentation and monitoring and I still love century for error reporting full disclosure David Kramer may or may not have a video of me singing Beyoncé in a karaoke bar in Bilbao But both is open source Century has reasonably priced paid plans which you can scale in a scale or two. So There's no lock in either way Now your code, how do you expect failures in your code? Well, if it's local, it's simple You just try accept and let's that's it Remotely you're lucky if it's that simple because instant errors are rare and Fortunate like connection refuse or a 500 or something like that. We know immediately. Okay, something's going wrong the worst failure scenario is that nothing happens So whatever you do remotely you have to put a timeout on it and One missing timeout in a database driver was enough to ground an entire American airline once so always put timeouts on it now if You have a timeout What do you do you've just hit the timeout? Do you carry on like nothing happens? That gives you very slow very useless requests. It's a bad experience for users because they have to wait 30 seconds to get a 500 and it's giving more load to a system that might be overloaded and for that There's a circuit breaker pattern. How many of you have heard of it? Okay, I'm gonna make it really quick. It's very simple very useful so It's kind of a local proxy between you and a remote system and it on a normal state It's closed like a drawbridge. Okay, so the requests go to the remote API the results go back simple Now if something fails, it doesn't have to be a timeout but timeout is the best example for that and it fails More than once because in distributed systems. Let's face it things fail once all the time So if you have a certain amount of failures, it will Switch from closed to open and the requests have to wait like at a drawbridge at this point It starts ignoring the remote system and it's just giving arrows back to the client very very fast So after a certain time It will send out a probe one lucky request from the client to check whether the remote is still broken and if it succeeds great We are close the drawbridge. It's closed again Everything is like before if it fails. Well, it stays it stays open and we will try again later This is very simple very effective Now I've said that adding more components to a system is usually bad However, it's if it's the same component and if you hide it behind another component It is way more reliable than your crappy code Then it might become something good. So for example, if you take something like a proxy Which I'm not I don't say lightly is very good software. It's a piece of software. It never let me down So if you put multiple of yours and protect it with a with a proxy Your reliability is actually going up and in military. They have to saying that two is one one is none So meaning you should have at least two of everything and if something is really really important You should have three of them so you never even get into the situation that you have only one of something Now this principle made the internet almost Unbreakable unless everyone uses the same DNS provider or Or unless everyone uses the same storage system in the same region Including the monitoring panel of say storage service And this works at any level at the network. You should have more than one switch more than one uplink You should have more than one server You should have well more than one data center. That's like level up game. Not everyone needs that But you certainly should have more than one backup Because if this is all it takes for you to lose your data, you do not have backups and Also, if you do not test your backups, you also do not have backups ask kid lap. They have a story to tell Now if you want to be dispensable You must not be a knowledge silo So if people ask you something regularly Write it down if you have regular tasks and standard procedures. It cannot be automated Write them down and if you have something that needs to be done in case of emergency write it down Thinking that you will remember it when you need it is a very dangerous fallacy and it had been a staple of aviation security to have To have a checklist for everything like literally everything ask any pilot and Having a contingency contingency plan you can stick to if a shit is hitting hitting the fan is Priceless you can believe me and this can be also communication. It doesn't have to be something you do It can be something you tell someone like your social media team So they don't lie to your customers that everything is dandy and they should start to reboot their computers or whatever So finally how to deal with the failure in your application? Well, your database is gone. I bet service is returning invalid data or the infamous timeout. So first Failure containment don't make it worse Because cascading failures can happen and that means that everything just melts under you even completely underlight unrelated systems and In relation to that let's talk about something very simple with a very big impact and our retries Retrace are essential in distributed systems because if as you've seen before Transient errors are completely common. So you have to take them in account, but they're also very very dangerous because if you use them loosely You can dust yourself or even worse. You can dust a third party Which can mean that you land on a blacklist or you get even into? into legal problems so What do you do you back off? But how long do you back off if you just run into a deployment or a packet got lost or whatever one second is fine If a system is overloaded and you're just waiting for your schedule to provision more servers one minute Maybe adequate and if there's like a hardware failure Maybe a switch and some poor schmock has to go into the DC and replace hardware Well every five minutes just check if if he if he or she had done it and try again The trick is to do all of it you try with a very short period which you then raise exponentially and this is pretty good But there's still a problem because if all your services do it at once You still have a lot of load in the same moment And that's where a jitter comes in a jitter is just a random number that you add to the computed exponential bike off and It will space out the retries between your systems. So you don't get this thundering heard Okay, so this is intuitive. I think so but let's dig deeper. Let's say you have a policy of three retries per client click whatever and Let's assume that the back end is your responsibility and because you read the orange newspaper page You want microservices? So let's do it So at this point one request in the front end like or like a one click means nine hits to back and a in the worst case because three times three But we are not even close to microservices let's add one more layer and You're a dusting yourself again. This is called the combinatorial retry explosion and One user click means that a C gets hit with three times three times three twenty seven 27 requests now imagine if C was slightly flaky because because it was overloaded before Now it's toast And the simplest solution the one I use is that you retry only at the top like in the client but this only works if You know where your top is Which may or not be true and then you have to do something more complicated like per request retry budgets But then you have a state that has to wander around with your requests That's not that simple anymore And this is a good example for complexity through microservices because such a problem just doesn't exist in a monolith There's a lot more to be said back pressure have it have a way to signal to the up that you are overloaded Unbounded queues do not have them it's mathematically proven that a unbounded queue is worse than no queue at all do not use unbounded queues and There's a bunch of links on my page to that Now next one insight into failures something fails You want to know what happened and why did it happen? Because nothing is worse than a silently failing backup script again ask get lab In the best case you put something like century into it But at least something email if something doesn't work and this is true in code to and If you do something like this Especially in libraries. I have a lot of opinions about you because how am I supposed to debug this? I have no chance The same is true if you do something like this what I like to call a vanity exception like an exception That is special to your very special library very special application. The error detail is still lost Don't do that instead to exception chaining So this is the Python 3 syntax. This will attach the original exception in dunder cause and You can't disrespect what actually happened if you're using Python 2 you have my sympathy and There's a function called race underscore from in the 6th library So you can use it to but really if you do not know how to What to do about an exception just let it be Chances are that the user Knows how to cope with it better than you do deep in the guts of an application So the next one is a bit counterintuitive if your application is unfit for work Like the classic thing is that your connection pool turns invalid because this database server has to be restarted You could start adding complexity to reconnect the pool but only in one thread and serve some intermediate error to your user or you just add one line and More often than not, this is just fine And it also runs assist or so sys.exe. It also runs your ad exit Handlers, which means that you get to clean up properly And this is called crash only software and it's not something I just made up. There's solid science behind that There's a lot of papers which you can which you should read And there's a bit more nuance to it like micro reboots, but I found them a bit hard in So to sum it up feel fast feel loudly a crash is always better than a hang as a user I prefer a 500 over waiting for 30 seconds and get a timeout or then an error And depending on your audience you may would you can just present a stack trace to for example Redis takes it to the next level. They have if they have a crash They show you a stack trace But because most of the crashes are caused by faulty memory They will also run a fast memory test to tell you that your memory is flawed before you open a Buck on there. I think good up So once you have that you need to focus on recovery and this is where the MTTR reigns supreme The MTTR is the mean time to recovery and if you've accepted that failures happen and maybe started writing Crash-only software it becomes much less important that something goes down It becomes much more important that it comes up fast And that's why humans cannot be allowed for the restoring of service because if I pull the plug in your DC You and your interest starts getting up and a lot of systems need hand-holding Hand-holding to come up. How do you choose? No, this has to happen on its own And the prerequisite for that is that your app Does not expect what's there like the database you try to connect if it doesn't work you try again again again Maybe you just crash if your process manager allows you to add back offs, but I just use loops and back off Now what is the secret to a long undisturbed vacation? apparently nine dollar mites and plastic cups But also more practically what you can do about it build fault-tolerant systems That recover autonomously and throw your phone into the sea This is the promised link the QR code would bring you there to follow me on Twitter buy your domains from Vario media. Thank you very much