Fault tolerance on the cheap, how to make systems that probably will not fall over. Primarily because you're just trying to push things as far as possible, but we'll get to that. So, hello everyone. Thank you for coming to my talk. There's a very interesting talk on a brand new programming language that way, if any of you don't want to catch that on YouTube. So, I do things too and with computers. That's kind of my primary deal. In particular, I am a real-time network systems engineer.  So, I work on real-time systems. If you don't know what real-time systems are, those are things that have computation on a deadline. So, you know, you have three gradients of that. You have hard, firm and soft. Soft means that your computation goes past the deadline and it still has value. Hard means that things have caught fire. Firm is right there in the middle. Real-time systems are either fail-safe or they're fail-operational. So, your system, which is hopefully fault-tolerant, eventually does fail. Does it blow up or does it just stop? Guaranteed response, best effort. So, you know, do you get a response in every case or does the system try to give you a response? So, your aeronautics system, if you toggle the stick, you want it to be guaranteed. And then resource adequate, inadequate. In all cases, do you have enough resources to do your computation or not? Network systems, probably this is more familiar. You get messages out of order from other systems that you don't control. You have no legitimate concept of now. Now is a false thing in terms of computers. You have high latency transmission. That is variable latency transmission, which is even worse. You know, sometimes you can get a message from the U.S. West Coast, the U.S. East Coast, and a few milliseconds, and other times it takes about 10 minutes. Lossy transmission. So, you lose things in the transmission. So, you know, you don't know if the system on the far end is dead, or if it just hasn't been able to get a message routed to you in a certain amount of time. Which is even worse because when you say certain amount of time, you have to have a concept of now, which you don't actually have. So, the punk rock version of this is, here's a socket, here's an interrupt, go program a computer, and that's what I do. So, I work for a company called Adroll. It's actually an ad tech company, which is kind of a weird gig, I think, for that particular set. But what I work on is real-time bidding. So, this is the thing where when you have a web page load up, there's an ad there. That ad is not actually rendered with the page. It is rendered in a tenth of a second in an auction. So, the system I work on does around 40 million transactions a second in a tenth of a second globally worldwide without stopping. We use airline for that. And even though I use airline extensively and I am heavily influenced by it, this is the last time that airline will be mentioned. So, it's not a language-specific talk. So, we're here to talk about fault tolerance. For those of you that don't know your space history, this is the Apollo 13 command module, the one that popped an oxygen tank. Nobody died, and nobody died because the system was fault tolerant and had redundancy in it, even though they were terrifically uncomfortable for about five days. So, fault tolerance. Subcomponents fail, but the system does not. Not right away, anyhow. Eventually it will fail, so you have to have some sort of concept of coming back around and repairing it or shutting it down. So, what does it take if you set out to build a system that will never go offline or will never go offline for some definition of time? You have basically three options. So, your first option is perfection, and this is the space shuttle in case you don't know your space history. So, perfection has certain requirements to achieve, and it's the ideal case because you never develop any faults in the system that you control, which will be important later.  You can't be building, say, a rocket or a telephony exchange and have this thing that you've sort of carefully analyzed and then have some joker come in and just plug in a thing and go, no, no, no, trust me, it'll work fine. Because I guarantee it will not work fine. You have to have total understanding of the problem domain. So, the trick here is you can't just show up and go, yeah, you know, rocketry isn't so hard. I can figure this out while I'm working on it. You have to have a deep understanding of the thing you're making because what you have to do is build models. And this has been talked about a lot at this conference. You build a model that sort of matches reality, and then you make a program that fulfills the model's constraints. And you have to have specific explicit system goals. So, in a lot of business, you have sort of like the people on high and board meetings go, we need to make money. Let's do a vague thing, and then that comes to engineering, and it turns into a specific thing because we have these soulless, uncaring machines to convince to do a thing. But we don't necessarily know that we've bridged that gap, that we've been able to convince these soulless, uncaring machines to do this vague thing. But if you're going to achieve perfection, you have to have these explicit goals. And you have to have a well-known service lifetime. So, this is probably the most subtle point. You have to know that your contraption will function for a particular amount of time. There was a really fantastic bug recently in the Boeing where if you don't reboot the computer, it overflows an integer at a certain point. But the thing is, they just assumed that the service lifetime would be the people that originally implemented this. They said, well, we know about that overflow, but the computer in the aircraft will not run for more than a year continuously. And that assumption held true until airlines needed to squeeze more flight time in so there's not as much downtime for the computer in the airplane. So it overflows and it crashes, except they didn't crash because they fixed it. So, a really, really excellent example of this is in an article called They Write the Right Stuff. It's about the software group that wrote the onboard control for the space shuttle. So the space shuttle, if you don't know, had five computers, four computers voted on an uncertain action. So, you know, if the stick moves, the computers vote on what to do, and then the fifth computer is there to go, those computers are crazy if they're not able to reach consensus, and then it's the one that is the source of truth for the action. So they all vote and the really interesting stuff about the space shuttle computer is that the software tools that they had were garbage. It's just real-time assembler on a radiation-hardened system, no automated testing, nothing like that, no modern tooling that you would think of even in the late 70s. But what they did have was a very specific process to the point where if a bug gets introduced, they're able to track down the person that introduced the bug, the people that reviewed the bug, and the process that allowed the bug to get through review, and then they changed the process. They also don't allow that process to continue. So they changed the conditions that allowed faults to creep in.  Unfortunately, perfection is extremely expensive. The cost estimates for the space shuttle's control system were $100,000 per line of code. Now if you have $100,000 for every line of code, you can basically do anything you want, because you can hire the smartest people, you can spend as much time as you need, because if you reach the deadline for shipping, all you have to do is say, that thing is filled with liquid hydrogen and oxygen. It's not ready to go. Ideally, you would see the same thing in nuclear control plants, but you don't. It also intentionally stifles creativity. Creativity is the enemy of perfection, because if you have someone that goes, I have this great idea, blah, blah, blah, what if they didn't fully consider all of the cases? Now you might have someone in a system like this that has a great idea, and then they write up a memo that goes to their boss, that goes to their boss's boss, and in two years the process has changed to include their creativity, but that's an entirely different thing. So you have to be a certain kind of person to want to work in an environment that allows you to achieve perfection. And you have to design up front. So this is part of having total knowledge of the thing you're building. You have to sit down with everybody that's going to build this thing and talk through what's going to happen with it. So you can't sit down and go, yeah, we don't really know about this part, but that's fine.  And you have to have complete control of the system, but unfortunately, complete control is not complete. So, you know, I said perfection in terms of the space shuttle, but the space shuttle killed 20% of its crew that got launched. But it was not related to faults in the computer. First, it was a wobbly rubber ring. The second time it was a sheet of ice that hit a porcelain tile. Those things can't be controlled by the computer, but this thing that you've worked so hard to achieve perfection on still fails for reasons out of your control. And we'll sort of get into exactly what caused those things, not in terms of the space shuttle, but sort of foreshadowing. It was organizational failures that allowed those things to happen. So option two is hope for the best. This is probably what we're all more familiar with. So you've got, in hope for the best, you've got little up-for-up knowledge of the problem domain. You have this big idea, and you're gung-ho about it, so you just dive right in, and you'll figure it out as you go along. You have implicit or short-term system goals. So your goal might be, I just want this thing to print out Hello World. And then after that, I just want this thing to respond over the network. And on and on, you have just this little iterative approach, and you hope each time you make an iteration, you understand more and more about your problem domain, and you're not forgetting things. The thing that people like here is there's no money down. You just go at it. You just blast out, and then hopefully someone will slap down their credit card while you're doing this iteration exploration of this system. And you need ingenuity under pressure. You need ingenuity under pressure because things go sideways, because you forgot things, because you didn't fully understand what you were doing. So you need people who are capable of working through a crisis and resolving them. Sort of everyone knows Facebook's former motto, move fast and break things. This is kind of the poster child of hope for the best, except this is a little more positive and mine's a little more cynical. The problem of hoping for the best is the ignorance of the problem domain leads to long-term system issues. So consider all the time where we sort of bring our hands and go, oh, tech debt. Each one of those times where we bring our hands and go, oh, tech debt. It's either because we didn't have enough time to do something or we did not fully understand what we were building. That latter one is the more concerning one, at least in my mind. Is that really resolvable? Yes, and we'll get to that. Failures do propagate out to users. So in terms of perfection, we say that failures don't exist in the production system. In hope for the best, the thing just goes offline and you have to believe that that is acceptable in the problem domain that you're working in for the system just to not be available or not service requests or lose bits of information or solve a consensus problem and decide that half of your rights were actually garbage. The problem with no money down is actually you do have to put a lot of money down when you do hope for the best. Think about Facebook, which was really just, here's some PHP out into the world and now they're hiring top-of-the-line engineers to build extremely complicated research systems to wrangle the hot mess that they've put together. But they've got a lot of money now, whereas before they didn't. They don't have the federal government giving them $100,000 per line of code. And it's hard to change cultural values. So it's hard to take a group of people who are very, very talented at putting something together in a knowledge vacuum, under pressure with little information and little support, and then turn them into a group that can sit down and decide, you know, plan and do the right thing the first time. It's very difficult to make that transition and a lot of organizations don't make that. You know, I'm from the Bay Area, so land of startups, et cetera, et cetera. There are a lot of people who sort of hop from first stage startup to first stage startup and they do that because their particular skill set is being good at hoping for the best and then sort of second stage people like me come around and we go, oh, God.  The third option is embracing faults. So this is the Mir space station. And for those of you that don't know Mir, Mir occasionally caught fire. And the Russian space agency just went, no one died. And Mir kept flying until we crashed it into the atmosphere. But intentionally, we did that intentionally, just putting that out there.  So you have a pretty decent idea about what's in this thing that you're working with, but you don't truly know and you don't truly need to know. At a certain point, you can just assume a black box and work from there. Now, your black box might violate some of your assumptions, but that's a thing that you have to know going in and a thing you have to plan for. Partial understanding of the problem domain. You sit down to write a telephony exchange and you don't know everything about telephony and you certainly can't predict and say the late or the early 80s that cell phones are going to be a thing. But you have a really decent understanding of the fundamentals and you have a really decent understanding of how to build a system that allows you to adapt it while it's running. So you're able to fight your ignorance as time goes on and resolve your ignorance in the system, which will be reflected in the system because software is just this sort of cultural thing that we apply to unfeeling machines. And you kind of need explicit system goals, but not really. You need to know exactly what it is you want to build, kind of, but you're going to change your mind. And that's okay because there is no overwhelming process. There's not basically an infinite amount of money, so you need a little flexibility, but you're going to be stodgy about it. And you need to be able to spot a failure when you see one. You can't have a thing happen and then have people disagree about whether or not that's acceptable behavior or unacceptable behavior, which you'll actually see in Hope for the Best Systems a lot. And I'm sure you can think of examples in your own work if you work on Hope for the Best Systems. But in this case, you have to be able to point and say, that's wrong, that's right. And that's the primary difference of embracing faults from the other two systems, even though in perfection you have to be able to do that, but upfront. So Jim Gray wrote this really, really excellent paper and I'm sort of paraphrasing him here, but he says in the paper, what's the name of the paper, why do computers stop and what can be done about it, which is the best CS paper name I've ever read, fail fast, either do the right thing or stop. So, you know, you have this computer system and it enters this failure state and we sort of think as humans because we do a lot of error correction online all the time that we can convince our computers to correct themselves. The problem with that is you're taking a soulless machine that has gone into a failure state and you're assuming that it has not failed enough to pull it back itself back into a state where it understands what to do. Whereas Jim Gray's paper is just basically shut it down and start over from a well-known state. The sort of challenges here are faults are isolated but must be resolved in production. So, you will ship bugs, you will ship things that are wrong and you have to have the tools and you have to have the sort of cultural strength to be able to say, you know, that's wrong and it's got to be fixed. And I say cultural strength because usually when something is shipping and running mostly well in production, people outside of engineering will be super satisfied with it and they don't want it to change. So, you have to be able to fight the battles up. And you must carefully design for introspection. So, this sort of bleeds into the last point but you have to be able to not just look in the system and look at the system from the outside and say, are you or are you not doing the right thing? You have to be able to peek inside of it at a very low fundamental level and see what it's doing. So, some people hook up to buggers. My two favorite systems for this are the JVM which has extensive onboard metrics and the ability to sort of hook up a remote thing of a bob or airline. So, I guess I'm mentioning airline again which has a similar idea but you can actually connect to the running system and upgrade it and all that. And you have to do moderate design up front. So, this is a little more sticky. It's a little more expensive.  But if you do that, you have a relatively straightforward path and your organization has to have money for this. You pay a little up front now and later, later resolving the faults. But if you have money, you have things to protect. So, building something that might fail or might not, we don't know is really bad for protecting your core assets. And your core assets could be money or they could be people or they could be entire geographic areas depending on what you're building. So, in this talk, let's talk embracing faults. So, option one, perfection. We've sort of talked about at this conference embracing faults is actually a super set of that. And option two is insane. So, there are four conceptual stages to consider if you're going to go about building something that embraces fault as kind of a first-class citizen of the system. So, the first stage is at the component level. This is the most atomic tiniest system or tiniest level of the system. This is the thing where you can sit down and you can read in an hour and you know almost by inspection what it does even though you need tests and types and all sorts of nice things around it. And progress here that you have on fixing issues has an outsized impact later. So, anything that you can stop from failing at this point because this is the tiniest level because it's the tiniest level, everything will propagate outward. So, what you can do here will have the most impact on the system that you have. So, what can we do? Immutable data structures are brilliant. The idea of having this data structure that you can just sort of pass around without being concerned about well, what if another thread has this and I mutate it and does that? Just, you know, solve the problem at its root. Stifle your creativity and solve the problem at the root and then you don't have to worry about it. There was a really excellent talk earlier today about mathematics and sort of how mathematicians work and the point was made that there's a certain class of mathematician that is fascinated by building a thing an abstraction or a set of ideas that make previous problems trivial. Immutable data structures is one of those things that makes previous problems in systems, especially long-lived highly concurrent systems entirely trivial. They just don't exist. The other thing you can do is isolate side effects. So, if you have this component that will call out to say a database, you now have to consider all the failure cases of that database and that database has a lot of failure modes that you probably don't know about because it's a black box to you unless you happen to be a Postgres developer or something like that or you're calling out to a piece of brand new hardware or a rocket or, you know, whatever. So, if you can isolate side effects into these well-known spaces and mark them very heavily, Haskell is brilliant at this because you specifically have to mark things as being a thing that side effects. But you can do it with code comments or if you're working in assembly this sort of well-known warning just removing these things into their own spot is brilliant because once you do that you can consider all the areas that don't have side effects as this pure mathematical construct that you can apply all these brilliant tools that we have to them. Continuing on with that, compile time guarantees. So as much as you can have guaranteed about your system at the outset as possible is for the best. Atta is a really fantastic language that was cutting edge in the 80s and nobody uses primarily because it was rammed through in the aeronautics space but it's a really great language and I highly suggest you look at it. My favorite thing about it is that not only does it have an integer type it has much like crypto it has an integer type that has a width so if you target a very peculiar machine that say has a 7-bit integer all of the integers in your program can now be 7 bits and it will compile code to make 7-bit integers which is just such a simple little idea but it means now that when you compile your thing you have confidence unlike say C or something like that that the thing that the concept that you have in your mind reflects the reality on the machine. And the sort of end point here is why test when you can prove we sort of had all these talks about these amazing tools that let us make theorems about how these component parts of our system will work so that we can have not just confidence, not just like well did I consider all the cases or not but proof, honest to God proof that the thing behaves as it should. This also helps us suss out where we didn't think through the edge cases which is surprisingly common because we are just finnit biased people and we have this goal in mind of building a thing and we'll sort of gloss over the problems with it. Rubber rings can't waffle so launch it is that sort of thing. So everyone knows this song everyone in this room is aware of what this is this is functional programming so functional programming at a component level has an outsized impact on the eventual fault tolerance of your machine or your system it is the single best thing that you can do if you can swing it in your organization to achieve the sort of fault tolerance. Even though the shuttle system was written in assembler it was also written in a functional style. You know you have this entire swath of the program that is just pure that just takes whatever weird floating point system they had and they can test it in pure isolation they don't have to hook up all the impulse generators to the test cluster so even if you don't have a functional programming language you can do functional programming it will look goofy but you can do it so the conceptual stage up from component is machine. So this is where you start combining your components where you turn them into a single isolated thing so if you have say physical machine sitting in a rack or embedded in your heart or sitting right next to a fusion reactor that's what this is. So fault and components are exercised here so your components they don't do anything they just sit there and they look nice and you can feel pretty smug about them but they don't actually do anything yet they only do something when you combine them together and the faults and interactions are exercised here so this is the real challenge if you are able to model your components you can't necessarily always model the interaction of your components and the interaction of your components is really where the rubber hits the road this is actually how your system runs is the passing of data between these conceptual units so what can you do at the machine level you have these components that you trust or maybe you don't trust them if they have side effects and they are linked together in a way that you feel pretty confident about but not wholly confident about the infection here because we haven't got $100,000 per lines of code so the biggest thing you can do is supervise and restart so this implies that you've got a component and you are able to say if it has failed or not and that's why I said earlier you have to be able to look at the behavior and say failure not failure so your supervisor is a thing that automatically looks at all of your components in your entire system and goes failure not failure and going back to fail fast failure it kills it and starts it over again and if you've got components that exist in a graph it kills everything in the graph and sometimes if you have a system that's designed like this your graph that needs to be killed and restarted comprises the entire system and you bring the entire system down and you pull it back up so when you start designing systems like this you'll have to start tanking in terms of graphs of components which is a big conceptual hurdle especially in languages that don't have any tools to help you with this the other thing you can do is use addressable names so if you have a component and it's anonymous you have no way of referring to it you have no way of referring to it in your organization when you're sort of talking to other engineers you have no way of referring to it in your system your supervisor just sort of says anonymous component 100 failed in this way and I restarted it and then you're sort of sitting there going well how do I introspect on this thing because you've got all these fancy tools to reach into your system and look at what's going on because you don't have these names the other thing with an addressable name is you only have you have to upfront design your system to have a finite number of components or potentially infinite but finite number of things that you know about and you can have these things that are anonymous but they don't end up being critical components because you can't supervise them properly you can't deal with them properly speaking of which distinguish your critical components so you have some components that must operate that cannot fail or if they do fail they have to bring the entire system down because it's just too far gone and getting back to failure modes if you're going to do this make your system fail safe at this point if you've said things can fail and we might shut the entire system down you cannot ethically then have everything blow up unless you're a supervillain but if you distinguish your critical components you suddenly have this knowledge in your organization in your engineering team that some things don't actually matter if you're running if you're building a web service for instance the component that actually handles each individual web service unless you're taking money orders or something like that if it fails, so what just make a new one eventually when the user calls back if you're taking money now it's a different thing they're all critical and that requires a different level of engineering so you're able to say in the sort of embrace failures that these critical things needed an incredible amount of inspection before we flip things and these non-critical we can deal with the failures that they'll exhibit later because we'll know about them so a step up from the machine level is the cluster level so this is sort of the point you know if you think about the space shuttle you've got these five computers so these individual computers they now communicate so that's what this is this is the cluster of machines and this is especially common now that we've decided to start doing things over networks you know having a database that has all of its fault pairs and whatnot so at the cluster level what can you do? you need redundant components so in the example of the spatial computer you have four computers that are voting and then an entirely separate computer that is the source of truth that the four computers fail and you need redundant components because I should have actually said machines but whatever components of the cluster each one of your things has a mean time to failure they will eventually fail catastrophically but if you can sort of compute how long that will take you can then say well I'll take that probability and I'll just multiply it out by all the other probabilities of failure in the system and at no time except in these astronomically rare cases will everything fail at once and so I can address faults in real time you know I can have a human on a wheelie cart run out to a bunch of machines and plug a new machine in and we'll keep going so that's why having a well known service lifetime is very important you can't always have duplicates of things but if you can't have a duplicate of something then you have to do much much more extensive work and try to figure it out commiserate with this no single points of failure at no point should you deploy a system if you want it to be fault tolerant every component can fail so have no single components this can be expensive this can be very difficult to do you know we're doing a lot of research now into say consensus algorithms so there's a lot of distributed systems research that was born out of database research in the late 70s and 80s which is why you'll see like Chris Michael John the papers that he'll reference in his talks they're about transactions in terms of relational databases so the theory there was general was how do we write data in a way that it will not be lost in the face of say network partitions or computer failures and everyone here should intuitively know that if you have data that is important and you want to write it into a database don't have one machine running your database have two, have three have as many as your organization can support and you do that because if you have a single thing that is the source of truth that thing will eventually go away and your source of truth goes away so you need mean time to failure estimates and depending on the severity of your problem domain you don't necessarily have to sit down and do all the statistics you just have to have a rough idea of how long your different components will run this can even be you know in my own group we just sort of go yeah this you know S3 will fail every year or two which in terms of measurements we've actually seen it fail every 10 months but you know when we're planning the severity of the projects that we need to tackle will take into account the mean time to failure this also helps you decide capacity planning how many of these things do I actually need depending on failure and the other thing is instrumentation and monitoring you know not only do you need tools to peek into a single machine but you need tools to peek into the entire cluster and then you need another computer system that is able to read all of this information out which will be gigabytes a second I promise you eventually and decide you know this pattern looks bad this pattern is good and sort of alert the expert human to come do something to replace a component or you know shut everything down as the case may be this is in itself a very long term project and requires continual investment but it's incredibly important you can't build these things without it the last level is instrumentation so all of these previous levels have been entirely machine oriented and that's great but those are probably the least important and I say this because a finally built machine without a supporting organization is a disaster waiting to happen you know you've got humans that want to do the best work and they do the best work and then you've got another human that's like yeah well it makes money let's stop working on it over and over again throughout history these examples of things failing not because the mechanism itself was truly faulty although Chernobyl was a disaster waiting to happen no matter what but that's because the design process was pushed by an organization that needed to be whatever these other things like the space shuttle the Deepwater Horizon Magneto Gorsk, the Damascus incident the BART ATC the transit system you'll see a human sitting in the front that human is not actually driving the train there's actually a computer that drives the train and the human calls out the stops and has a big button on the front that says stop the train there are two interesting examples of the ATC performing in a way that was suboptimal and it performed suboptimally at first because it would occasionally blow through a station so each time it's supposed to stop at a station and let everyone get on and what it did was it blew through the station and then opened the doors because the doors are on a timer now the engineers that built this thing they knew it would happen and they went to the board of supervisors for the BART and they said this will happen and the board of supervisors for the BART were politicians and business people and their primary concern was having the increased commerce and having the sort of political clout added to them because of the successful creation of a transit system so they said shut up also you're fired these three engineers and then the BART blows through a station and opens its door and then it becomes this big hairy thing there's actually this really excellent paper in the IEEE about this very incident and the other one is an operator had a heart attack and he slumped forward in his chair and his head hit the stop button and the train was designed the ATC was designed to take it into the station and then stop right there which is kind of a bad assumption because the operator knows that there's a break in the track up ahead between two stations it'll still keep going but he slumps forward and he gets rescued so paramedics come they take him off the train and there's no alternate stop button there's no other switch so his head comes off the stop button and the train goes so you have this man that's had a heart attack and then he goes to the next station on the opposite side of the Transbay Tube by train and it's a 20-minute trip by ambulance this was also a known fault it was also not addressed because of the organization around the BART you know the New Orleans Levy you have the Army Corps of Engineers saying like hey this is going to break and then the federal government going you know New Orleans isn't a big city all of these you can look them up and I highly recommend it because they're all illustrative of what happens so if you're going to achieve these sort of fault-tolerant systems you have to correct the conditions that allowed the mistake as well as the mistake which has been a repeat theme here but at the component level but this happens even at the political level in an organization and this is the hardest one to deal with because you can be fired and we're not licensed so my mother is a nurse and occasionally she will quit jobs because she can say this will put a mark on my license and then the organization around her will go well you know okay yeah there's an alternate body that she's responsible to but as software engineers we don't have that we're not truly engineers in terms of licensing so we can't have this third body to appeal to which happened in the case of the BART engineers they had no third body to go that thing's crazy and that's a real problem even though licensing kind of sucks process is priceless so if you have a really excellent process you can overcome any wacky thing that comes along so this is mission control in the time of Apollo 13 and they put in round the clock hours but even in a successful Apollo mission they have this incredible and space shuttle mission Gemini Mercury whatever you want they have this incredible mission control where they have this process that they go through to achieve a mission you see it in regulatory requirement you see it in heart devices or not heart devices, medical devices where you have this regulatory requirement process stifles creativity it sucks, it hurts, it's expensive but it also decreases defects assuming that the regulatory body is not also in bed with the thing that's making money different issue so build flexible tools for experts so the only reason that the Apollo 13 crew survived were highly trained experts because Werner von Braun did not get his way and build a fully automated system that takes ignorant scientists to the moon and because of an accident in that they had two duplicate computers so they were able to go into the landing module and pilot the thing back home but they were able to do that because you had two groups of experts you had the astronauts who were able to adapt and fly the thing and take instructions from the group of experts on the ground to figure out power and build a new flight plan and how to scrub oxygen or carbon dioxide from oxygen when you don't have the right size containers and they were only able to do this because they had flexible tools now usually if you combine process and flexible tools the flexibility is totally wasted because you go through the happy path of things but the happy path is not always what you'll get separate your concerns so if you have a system that is the be all end all of everything it will eventually fail in a specific way and destroy everything else so this is London during the Blitz in behind you have firefighters who are fighting a fire and then you have the milkman the milkman represents a separate system of society he delivers milk to families firefighters represent a separate system of society they fight fires the milkman is not throwing milk on the fire now London is kind of an odd example but this is a catastrophic failure of civil society because of a war but you're able to maintain certain components of civil society because they're separated they're wholly separated build with failure in mind so this is my BART station the fantastic thing about my BART station is the BART cops do not give a shit about bike thieves so you will always always see tiny parts of bikes left behind sometimes sought in half this is because there is no covering the BART stations were not designed with thieves in mind so you just sort of lock it out in the open and hope for the best there's no security guard, nothing like that so it's something that's just hope for the best enjoy that but you can actually say in Chicago you can actually park your bike into a locker that you control and that's with failure in mind you can see it as kind of a cynical thing but in a non-cynical case going back to the shuttle computer you have four machines none of them will ever fail none of them ever failed within the flight time of the space shuttle and they even plan for that four cluster of machines to fail because you don't want these very important humans and the less important science to burn up in the atmosphere or drift off past the Lagrange point or even worse get stuck in a Lagrange point now at an organization level a Lagrange point if you don't know is the point between two gravitational bodies where you need no thrust to move out of it so it would be a permanent graveyard have resources that you're willing to sacrifice so this sort of goes goes back into it sort of goes back into have redundant components but at an organizational level you have things that are very critical and you have red shirts and the red shirts you're willing to let go this is sort of a jokey thing but you'll actually see this if you're deploying say a database cluster and you know that you need six, you need six for failure and you need six for load and your organization goes you know those are really expensive computers let's do three, we'll cut the difference you don't have any headroom and you will occasionally drop stuff on the floor so your organization has to be on board with having resources that you're just willing to sacrifice or in the example of the system that I work on we intentionally deploy in some cases over provisioned and we expect some of the individual machines to fail for reasons we can't even conceive of yet because we're over provisioned the system is viewed from the outside is able to continue to service traffic and function even while individual components are failing now if you're going to start designing these things you have to understand the context that you're working in so you need to study the accidents that have happened before and especially in software engineering not so much here in this conference because we're sort of weirdos in that we work in a very esoteric domain but you can get real comfortable in a sort of broad context software engineering tends to be very myopic this is the deep water horizon it is on fire and 12 people died exactly why it is on fire and 12 people died is written up in a very excellent report and it's a report that sees engineering failures, sees organizational failures that sees pressure to skip quality assurance to make increased profits and if you are able to sort of connect your work to historical failures you're able to identify things that are alike because all human enterprises no matter how esoteric they might seem at the time are basically all the same because we're all basically the same we've all basically got the same brains another really really important thing that's hard to get across but it's super important to get across is every system carries the potential for its own destruction so this is Charles idea of normal accidents so this is a book that I highly recommend it's called normal accidents living with high risk technology so peros idea is that every system that you build will have some fault in it and that fault will be catastrophic now you might not know that fault or you might know that fault like say you're building a boil water vision reactor like Chernobyl you happen to know that there's what's called a zero coefficient meaning that if you're no longer pumping energy into the thing to cool it it only does the temperature increase but the temperature increases and feeds back into itself so it increases very rapidly and then you pop and you pop and you spread radiation all over Ukraine now that failure is resident in the system and knowing that it's there you might actually decide oh oh oh too far boo what's going on there we go computers am I right um yeah you might actually decide that some things aren't worth building you might say that the failure that's resident in it is so horrendous that you just don't even want to bother you my own group you know we have two primary concerns of the system that we build one is catastrophic underspend meaning that we don't spend enough money it's an advertising thing the other that is more understandable is catastrophic overspend so if we've got an Etsy advertiser and they have 50 bucks and they're making quilts and they give us 50 bucks to advertise their quilts and we spend $100,000 for them um they owe us 50 bucks so we're just out the 900 you know etc so catastrophic overspend is potentially a business killer now we have in the past had uh systems pitched to us or additions to the system that would make catastrophic overspend more likely or inevitable and we've just sort of had to say no you know we will not build that like that makes no sense to build and you know in a business you have to say that makes no sense for the business but we're not licensed so the the last thing which you've already seen is understand networks so especially when you're building things that are distributed across some sort of network you must must understand the network you know uh when people get into it they think you know I open a socket back and forth and great it works great it doesn't it sucks it's terrible um and it's terrible because of fundamental limits in the universe and it's terrible because copper wire buried in the ground in the midwest gets severed by a backhoe um so there there are these sort of points these fallacies of distributed network uh uh thinking and I highly recommend uh a paper on it but you know the network is unreliable you will lose messages you know latency is non-zero so when you send something across the wire it will take time to get there and you will not be able to predict how long it will take to get there um backhoes bandwidth is finite you know you can only send so much stuff the network is insecure so people are listening I mean we're all aware of that now after the snow and revelations but people are listening and people always don't have your best your best hopes in mind when they're listening either um topology changes so if you're bouncing messages through a certain router this was more important in the earlier era when we had you know just a few things uh but now we sort of forget that we're actually bouncing messages through unreliable things that are run by unreliable people uh unreliable messages unreliable messages yeah uh there are many administrators and they have different goals in mind so they might hate HTTP in one part of the world and they'll throttle it uh we actually see that with totalitarian regimes all the time uh or Kansas um transport cost is non-zero so you know the time that you take to encode your message you have to include that in the design and system um and a lot of people forget that so especially when you're designing something for latency people will just assume that sending something across the network is basically free and it is not uh and the network is heterogeneous you know you have a certain version of the Linux kernel off in the distance that has a certain bug and it wrecks your day and you have no control over it um and that's you know when you have a perfection oriented system you can't have some guy come along and just like plug some random thing in and go you know trust me but that's not the world we live in you know we're not unless some of you are building rockets in which case I would like to talk to you um also if you use AGDA or ADDA uh you know when you exist on the internet you're living with the reality where people just show up that you don't know and plug random things into your system and you have to hope for the best or you have to embrace results and sort of deal with that uh by having certain components that you know are not critical fail when they get information that makes no sense and then not have that propagate up and kill everything um so that's my spiel and thank you all so much


What's up guys John Hammond here coming back at you with another video for the Natus wargame from over the wire So we just got the password for level 9 and we've got a script that will get the web page content for us so we don't have to use a web browser and This one looks like it's saying alright Natus 9. Here's a form. You can find words containing And inputs just regular text variable. It looks like with the variable name needle and It looks like it'll search for us the output is displayed in pre-formatted text and we could probably see that in the web browser If we really wanted to let's do that just so you get an idea of what this level looks like Natus 9 Natus 9 and You can see there isn't their option here to view source code. So we can check that out Index source at HTML. Yep. That was the name of the file And we can do that in our Sublime text stuff if we really wanted to we could decode that if we wanted to or we can just view it in the web browser So it'll handle that a little bit nicely. Okay, so looks like PHP code the Notes here with the variables with a dollar sign and all key is Apparently set to whatever we post to or whatever is requested once we communicate with the server And if key is not an empty string it will pass through What is that looks like a function a pass through I'll check that out and it's oh it looks like it might be running a Command here grep. So like we've kind of done in the command line with a bash shell grep Whatever we pass to it nice Key in dictionary dot text dictionary dot text must be A word that's trying to our dictionary that it's looking through. What is the PHP pass-through function? Is it literally just gonna run a command for us? Yes, it will totally just execute commands and display output Okay, so we could potentially have remote code execution here And we can totally get in the way of it because we're just passing in raw arguments to this thing So if we wanted to find words that say like Find words containing app Okay, all of these things in dictionary dot text Apple obviously, right? What's et cetera so? We can put more in here like How about a period So grep will use the regular expressions here and that will return a period from Period will match everything just like regular expressions will match any character so it returned the entire dictionary dot text file but Since we're supplying this as an argument that you can see went through without any Like quotes are surrounding it to specify that that is a single argument We could bleed into other arguments here and specify other files that we want to read or change up how this command works Let's take a look and I let's do that in our shell here or our Sublime text script Needle can be app to see if we get responses Let's run this we don't want to go to index source anymore I want to be posting it to the actual page and we're getting all of this output so can we Return anything with the period Huh Okay, it must be getting some strange characters out so let's go a let's just app Huh Okay, maybe some maybe some strange characters are in this dictionary that Python does not want to render for us. We can change the encoding here coding equals utf 8 That's the magic magic Python syntax to suddenly be okay with unit code characters. Will that work for us? No, okay Man, that's super annoying Let's do this with a browser then I suppose At least let's figure out our attack Let's say we can actually return anything and if we had a space in between this we can actually ask for another File that we want to look through I think right let's go to Natas pass and we want Natas 10, right? search Looks like it returns from the file that things matched in But the Natas password file Should get a result. It should have matched at some point. Let's check out. Let's control F for it Natas 9 Natas 10 no luck Huh was it in etc? Is it just going to display anything for us? That's not dictionary dot text Doesn't look like there's any other notion of Natas. No So maybe that didn't work. Okay. Well, we still have command execution, right? Can we do Period or like anything From nowhere Yep, nothing nothing from nowhere and Use a comment to end the end what we're in and the rest of the line here So it doesn't put in that dictionary dot text for us, but we use a semicolon to note a new command, right? Okay, cool. So we've got command injection just like that. So now we should just be able to cat Natas pass Natas 10 is that not the is that not the right location? Over the wire. Whoa what the heck over the wire dot log over the wild at org. What am I doing? Natas Etc Natas web pass.  I'm sorry. I didn't even realize that path guys I thought it was the same syntax as the other wargames. Well, hey, okay. That's done for us, right? There's the attack We can return anything And let's just get something that we know is In here Actually, no, we can just say anything and then we can still use a comment to not include it the Not include the dictionary dot text file Natas web pass Natas 10 and Use the hashtag here. So we won't return the Rest of the responses from the dictionary text file. We commented out the rest of that command here so now we've got our password loaded or displayed out to us and let's use our regular expressions here to carve it out Just like that we get our password and we are good to go. So Natas 9 is done. We can put this in the script for Natas 10 and Just like that. We are ready to move on So I hope you guys I hope that made sense a little bit of command injection because that that code was just being passed through To a real command like a bash shell and executing it and we can just kind of wrap around the arguments or do some unique things with it and we can load Content from other files because that's the way grep works and we can use other bash syntax stuff like comments or Other commands with the semicolon to do other interesting things. So Cool hack cool vulnerability there and that's a real thing command injection is for sure a real problem in web security stuff So, okay now they filter on certain certain characters, huh? Let's get started on that in the next video guys Thank you guys for watching. Hope you're enjoying these


QuickBooks Online 2022. Apply customer deposit credit to invoice. Get ready because it's go time with QuickBooks Online 2022. Here we are in our Gik Ray Guitars practice file. We set up with a 30 day free trial, holding down control, scrolling up just a bit to get to the 125% currently in the homepage. Otherwise known as the get things done page and the business view as compared to the accounting view. If you wanted to change to the accounting view, it is something you can do by going to the cog up top and the switch to the accounting view down below. We will be toggling back and forth between the two views either here or by jumping on over to the sample company file currently in the accounting view. Going back on over, we're gonna open up a few tabs up top, going to the tab, right clicking on it, duplicating the tab. Back to the tab to the left, right clicking on it, duplicating back to the tab to the left, right clicking on it and duplicating one more times as those are thinking, we're gonna be putting reports in them. Let's go and check out the sample company to see where the reports are located in the accounting view, which is down here. And it's indicated by the letters down here, R-E-P-O-R-T-S, which stand for or if read reports. And then in the other view, in the bookkeeping or in the business view, it's in the business overview area. And then the reports right there, closing up the hand boogie. We're gonna open up the balance sheet. Let's open up the balance sheet to start out with one of the faves of course, up top from 010122 to 123122, run it. And then we're gonna tab to the left, tab, tab to the right, tab to the right. And then we're gonna go to the business overview again and the reports again, closing up the hand boogie again, opening up the profit and loss this time with the year to date on the range change up top from 010122 to 123122, run it. Tab to the right, tab to the right, my darling. And then we're gonna go down to the business overview and we're gonna go to the reports and let's open up the profit and loss again so we can see one of them with a current month versus the year to date numbers. Let's start this one out at the beginning of 020122 to 123122 and run it. Now in prior presentations, we imagined a scenario where we had multiple people coming in wanting a particular guitar. We didn't have it on hand. We said, hey, we'll order that guitar for you. We will get it, we will prepare it, but we would like to receive a down payment from you first before we do so. If we go back to the first tab then when I hit the plus button, instead of making an invoice and then receiving the payments or instead of making a sales receipt when we get paid at the same point in time as when we do the work, we entered a received payment first, which we know typically, we know that the received payment will reduce the accounts receivable, but it has no invoice to reduce. Therefore, it made a negative receivable. We did that instead of making a positive liability, which would be the proper thing to do from a reporting purpose, but it works fairly well logistically as we discussed to make the negative receivable because the negative receivable will result in the system being able to make a credit for that particular customer, a credit meaning some, a credit balance or a credit on debits and credits because it's a receivable. It's on the credit side for that customer that we can then apply to the invoice when we later make the invoice. So it's not perfect for reporting purposes, but we can make an adjusting entry at the end of the period. And for just logistical purposes, it works quite well for us to then apply out the credit. So let's go back on, let's open up another report on the right-hand side, which is going to be our customer detail report. So we can see that activity. I'm gonna right-click on the tab up top and duplicate the tab again. Let's open up one more report over here so we can see exactly what the story is. What's the story, man? What's the story these days with Get Great Guitars? I'll tell you the story. Here we go. We're gonna go down. It's in who owes you. We're gonna be down here in the customer balance detail. Customer balance detail. And then I'm gonna change the dates to the custom date for 123122. Hold on a second. 123122, run it. Holding control, scrolling up just a bit. Now it only shows, well no, it shows the activity here. So we're looking for those items that have these negative balances like for Anderson guitars. It has a negative receivable that doesn't make sense from a reporting purposes because a receivable means an asset. And if it's a negative asset, it should be a liability. However, for reporting purposes, when I apply the invoice to this negative receivable, we will be okay at that point in time and logistically inside the accounting software, it's a lot easier to apply the payment, which will be the credit that will show on the invoice this way because we're in the accounts receivable, which is used to track the customer detailed information in the sub ledger. We can also see it if I went back to the first tab here and I went down to the get paid and paid area, which if you were in the accounting view, which would be in the sales area. And if we went in then to Mr. Anderson, Mr. Anderson, we see that we have this, let's close up the hand boogie so we can see it better. We've got this unapplied amount there, which is gonna be applied when we create the invoice, which is nice and clear in our bookkeeping system. So if they were to ask us if Mr. Anderson asked us, anybody can look this up and say, oh yeah, you got this unapplied amount that we can apply to a future invoice right there. So now we're gonna imagine we have that future invoice. One more look just to note, if you see this on like a flow chart, this is the desktop version flow chart, but we're just looking at it for demonstration purposes. If you see this amount, usually you create the invoice and then you're gonna get the receive payment, then we make the deposit. This time we recorded the receive payment first, that resulted in a negative receivable that we can then apply out to the invoice. Next time we'll look at a way that we can make the liability when we get the prepayment, although it has logistical problems, which I think is a little bit more complex from the bookkeeping side of things, even though from the financial reporting side of things, it would be more proper to have the positive liability other than the negative receivable. Again, if you have a negative receivable, you could fix that with an adjusting entry at the end of the period, which could work quite well too. Okay, so now we're gonna go back on over here and we're gonna say, let's make an invoice. Make an invoice and I'm gonna say, also just before I make that, just note that this 300, you might be asking, well, how would I know how much to charge for the deposit? So to do that, you might say, well, I might have a policy that I'm gonna collect like 10% of the deposit or something like that, and I could create the invoice and then actually figure out how much I want to get a prepayment on it for, or you might make an estimate. So just note that you might say, I'm gonna make an estimate if someone came in and said, I would like this custom guitar. And we say, hey, we would like to get a down payment. You might first make an estimate and say, okay, and this is what would be done before you got the deposit, right? I might say, okay, Mr. Anderson, Anderson, what kind of guitar do you want? And he's like, I want this really gaudy, bright gold guitar with sprinkled glitter all over or something like that. Like, man, I don't know if I even want to display that in the shop, but we'll order it for you, but we need a down payment. So then they could tell, then we can get the estimate down here and say it's gonna be a EPSH guitar. And he's gonna want two of those, he says. It's gonna be a taxable item. And then we can say it's an ELP. And so we could say that would be one of those items. So that would then come out to that. And then if I had the tax applied to it, I'm gonna adjust the tax to make this our generic 5%. I'm gonna change the tax to the generic five for the practice problem purposes, as has been our custom for the practice problem. Why? QuickBooks ask why they've got to know everything. Do you have to know everything? Every reason I do everything? QuickBooks, just do it, man. So that's gonna be the, there we have it. So then we might take this amount. We might say, okay, that's how much it's gonna be. I want 10% down or something like that. That's not what we did last time, but that's one way that you might calculate the down payment. And then you might say, okay, I'm gonna save this. I'm gonna save or send it. I'm just gonna save it and close it. And then when you come in, when I get the guitars ready, we already have the estimate ready for you and we made the prepayment in the past. So now we have our estimate here. I could use the estimate to create the invoice and I'll be able to then apply out the $300. So now we're saying, okay, we charged the 300. We know what they ordered. So I've got the estimate here and now we can create the invoice from the estimate and then look to apply that 300 to it. So now we got the guitars and we got our down payment. We can create the invoice here. I can say let's make the invoice for Mr. Anderson. And so it'll populate for us automatically from the estimate, for example. And then let's say this happened on two, they're gonna say this happened on 226, 226, that the actual sale, the sale happens down here and it pulls in the information from the estimate that we had. Notice it doesn't automatically apply out the credit here. So it's a little bit different here than what you will see on the desktop version.  Notice I don't see like the credit to apply out here. But what I'm gonna do is I'm gonna, I'll show you how to do this. I'm gonna save this and then it'll basically apply out the credit. So let's go down and say I gotta adjust the sales tax again. So I'm gonna go into this item and I'm gonna say I want the generic 5%. I thought I told you last time QuickBooks, it was in the estimate generic 5% because reasons, that's why, reasons QuickBooks. And then we'll save it. It's because it's a generic problem, that's why. So there we're gonna have, so there's gonna be our total amount. Now I'm gonna save it and then QuickBooks often will automatically apply the credit out and then I can open it back up again to have the credit. So we'll show you what I mean. I could say save it and close it. And then I'm in the detail here and I see their information. So now I'm gonna close up the hand boogie. And so now we've got the invoice and then the payment. So the payment happened first and then the invoice and it's got the partial payment here. So if I open this invoice back up, if I go back into the invoice, we can see that QuickBooks took the pre-payment and made a partial payment for it. So now we see the $300 down there. So that's how you can do it. QuickBooks will basically do it for you automatically, meaning it connects the credit out automatically, generally, but if it did not do that, then what you could do is I can close this back out and you could see down here, the payment now has been applied out. It's closed out the payment. If it didn't do that, you could go into the payment right here and then say, I'm gonna apply this payment down to the invoice, which is now being applied out. It did it automatically. And that's why you can go back into the invoice, open it back up, and then it does the calculation at the bottom down here for you after you apply it out. So notice it's kind of a two-step process, meaning you enter the invoice. You don't have any kind of automatic button that's gonna change things on the bottom down here, but if you close out the invoice, it will typically apply out automatically or you could then apply it out if it does not and then go back into the invoice and it should do the calculation. It'll keep on changing the sales tax, but I'm not gonna save the change. So just remember the sales tax is gonna be for our practice problem purposes, the generic five for reasons purposes, for practice purposes reasons. So there is that on it. So I'm just not gonna save the change every time they try to adjust it there, but that's gonna be the idea. And so then I'm gonna close this back out and we can then see what happened with the invoice and notice in here, because we'll do a method number two in a future presentation and you'll see kind of the difference between the data in here. The second method, remember, we'll make it a little bit better for reporting purposes in that we'll have the liability account, but the tracking in here looks perfect from an internal bookkeeping system because I can see exactly what happened. I saw that payment that I needed to apply out. I can see exactly what is happening when I look into an individual customers, basically records, the invoice is partially applied. So this screen looks perfect for me as an internal user to be dealing with customers typically. Okay, if I go then to the second tab, we entered basically an invoice which does exactly what you would expect it to do. We had an increase in the accounts receivable. If I go into the AR and hold down control and scroll down to the AR, we've got this accounts receivable for Anderson. There's the invoice. If I go into the invoice, remember it's gonna change that sales tax most likely again. So I gotta keep on adjusting it for practice purposes, but see it tries to recalculate it. I'm gonna adjust it again. Just don't save the change but I wanna talk about it here. So I'm gonna say 5% for other, confirm, okay. So there we have it. And notice it put it in there at this amount, including the sales tax. This bottom part right here, the $300. That's already been put in place. That's already in our books. We just put it at the bottom for reporting purposes for the customer. In other words, I'm not recording the invoice at 1,065, the amount that is still due. I'm recording it at the 1,365, the amount that we charged, including the sales tax. And then this $300 is already in our GL, which is gonna net out against this amount to result in them only owing 1,065. But the GL's being hit at this point with this invoice by the 1,365 is the point. Closing this back out. So there's the 1,365. If I go back up, then to my report, then go to the income statement and make sure it's fresh, a fresh statement. We can go into the sales item here and we're gonna hold control and scroll down and say the sales item are gonna be the line items that are on the invoice. Again, for what we charged on the invoice. So we're not decreasing it for the deposit that we got because we earned all of the income at the point in time we did the work at this point in time. So we're gonna go back up top and then the difference would of course be on the balance sheet back to the good old balance sheet and it would be in the sales tax payable that we owe that we had to keep changing to 5% going to sales tax payable. If you're in California, they got the department of tax and fee administration, blah, blah, blah, it would be in there. And then we've got the receivables going down. If I go into the inventory, the receivables are going down just as they would normally driven by the items for amounts that aren't actually on the invoice but the system knows about it because of the items. Going back up top again, back to our report. The other side in the cost of the goods that are sold, the expense account for the inventory we sold on the income statement. So it would be right in here in the cost of the goods that were sold, going back up top again, back to our report. We also know that the sub ledger and just note that this is our income statement for year to date. This is the income statement now for just the month of February. And so we have the same impact on income and cost of goods sold, but this is just the performance report for February versus the year to date. We also note that the sub ledger over here, if I go to the sub ledge, then, and I update it, let's run it again. So we have some fresh stuff we've got then. And I think it's only showing the balances here. So let's go up top and see if I can customize this report. I would like to see the detail if I may filter in it. And let's say we want to see not just, we want to see all of the stuff, all of it down here and run that so I can see my activity. So here's Mr. Anderson. And so now we've got that 300 payment. And so now we've got that 300 payment was applied out to the invoice here for the difference of that invoice on the opening balance on the right hand side, which we also saw by going back to the first report internally in the data for Mr. Anderson over here. So if I go back to the report to the right, so notice at this point in time, Mr. Anderson is good. We don't have a negative balance. So that problem of reporting unearned revenue versus a negative receivable isn't a problem after the thing is complete. It's only a problem if at the cutoff date, when you need to report the financial reportings externally, you have this basically negative balance here, which is understating the accounts receivable and you have not yet done the work or issued the invoice, at which point you might need to do an adjusting entry at the end of the period. Now, of course, we also could look at the sub ledger for the inventory, but I won't go into it now because we've seen that in the past, but it should tie out to our inventory account here. If I go back to the first tab, we see the detail for Mr. Anderson. We can also check out our detail by going to the activity, hitting the plus button. We can go into the business, no, it's in the bookkeeping area and the business view. If you were in this view, it would still be in the sales area. And we can look at the detail on the transactions, closing up the hamburger, and then we can go into the sales, all sales. And we can say, let's just search by say the invoices, drop down, drop it down and searching for the invoices. Here's all the invoices. And then I might say, well, let's just take a look at the invoices that are still open and outstanding with a further filter, and then we got the invoices outstanding. Here's the Mr. Anderson invoice, which was the total of the 1,365, which has been partially paid. The 1,065 we can, of course, and expect to receive the further payment at which point we'll have to receive payment form on the right. Okay, let's go back to, let's open up a trial balance and just see where we're standing at this point, hitting the hand boogie and going down to the reports. And we're gonna then close up the hand boogie, type in trial balance, trustee trial balance. That's what happened, I spelled it. I always spell it trail balance, trail balance, but it's trial balance. Let's range into the change in 010122 to 123122. Run it, here's where we stand. If your numbers are tying out to these numbers and you're following along, then that's good. If not, then try a date range change. Sometimes it's a date issue and we'll be doing a transaction detailed report at the end of the section, which is a great tool for diagnosing any differences.


We're back with of the press and a plus TV Africa that's on the breakfast. We have a zikonga. I took who joins the conversation this morning. Is it good to have you join us? Thanks for having me. It's a pleasure. Compliment of the season. Very same to you and everybody in the station. All right then let's quickly take a look at the Guardian newspaper this morning. NGF Moom as federal government shifts blame for high poverty to governor. So the blame game continues. Agba there are rather beautiful flyovers, sky scrapers, but leave fight against hunger. Sokoto has the highest number of poor people followed by Bayosa. Yahaya bello governors are governors committed to improving livelihood of rural dwellers. Rising inflation and falling wages increase poverty, social unrest, says ILO. These are the riders you find underneath the board caption and just again you have, I have fears about a possible outcome of 2023 polls says Jaga. Now what could he be saying? Cod dismisses suit seeking disqualification of Labour Party presidential candidate Pitao B. That's also on the Guardian. Or you tell us appointees left with official vehicles. Others are delicate reveals. And just before we move away from that 8th day, which is today, no end to infection by 2030 as 1.8 million Nigerians live with HIV. That's a lot. We'll move away from the Guardian. Let's quickly take a look at the punch newspaper talks about fuel scarcity. No planned price hike says federal government queues spread. But what's going on with, you know, the hike that we're experience experiencing at a time? There will not be fuel price hike during you tied period says government. NMDPRA assumes or assures motorist and current stock can last for 34 days. And that's because the NMPC says we have like 2 billion litres of petrol. Government cannot continue subsidy amid diversion. NMPC sources quoted to say these are all of the riders you find underneath the board caption. But let's move away from that. That's because we have more interesting headlines. Ina Gezap for elections in North East deploys beavers. And government pays Asu November salary with whole arrears. Federal government blames governors for the rising poverty and doh akwaibom defa. Don Bax F federal government policy on mother thong teaching. A delicate alleges or shun government vehicles missing issues ultimately. And though Hunter mistakes man for antelope and shoots him dead. What a sad incident and sad story. Asha Buhari, Twitter critic, remanded it in prison. I mean, that's, that's the conversation that's been going on for a very long time. We'll just quickly take a look at the daily trust newspaper election valence claim 1,525 lives in 25 years. Boko Haram hyper bandits biggest threat to 2023 polls. That's according to report. Politicians excess excesses polls ricks to elections. That's what analysts are saying. Politicians excesses polls ricks to the 2023 elections. Breakdown of security threats to 2023 general elections, but it's anything that can be done. Now, there's a pictorial representation to that where you have the map of Nigeria. And, you know, if you look at the map, it would be the nothing part of the country. Abandoned, not Western, not central. They have been operating for more than a decade, but the danger reach have increased during the presence, the present presidents tenure. That's what you find there. Now, this Boko Haram in the nut is, if you also look at that map, you know, to that side, you'll find desperate assurances by the government. Boko Haram remanded a serious threat around the Lichard Republic. Again, you find 55,430 people were killed by terrorist groups and criminal gangs operating across the country. And you find 3.2 million Nigerians have been displaced from their homes, mostly in the North. The data and statistics actually continue and beneath it, you find iPop, that's in the South East. The map also can show you that vividly. The group did not take up arms until after Boko Haram became president. Now, that's a lot. Life lost to election violence in Nigeria since independence in 1960. In 1999, data says it's 80 in 2300 and in 2007, 300. Now in 2011, 800 persons died in 2015, 100. Now in 2019, 145. Feels like, you know, the number increases every other time. Now in July 2022, 2.45 million were registered as internally displaced persons so much, you know, to think about as inch closer to the 2023 elections. Governors abounding rural areas and building needless flyovers and airports, that's what the federal government is saying. Troops recovered two female cops, rescued three victims in Kaduna Karazin. Heads 1,000 per litre as food prices saw. Only 33 rapists convicted for over 11,000 cases, that's what, you know, the federal government is quoted to say. And then quickly we'll just look at the nation. Ezekiel Nyaituk will be showing his thoughts in no time. Minister states to blame for the rising poverty and hardship. Governors building airports and flyover, investing in projects with no economic impact. That's what the federal government is saying. We could take on governor over derivation payment. And just against schools to adopt mother tongue as medium of instruction. INXS, PVC mandatory for voting in the 2023 January elections. I mean, what have we been doing before? Tunable for Chatham House, talks and security, economy, Abdul Razak takes 189.4 billion budget estimate to the national assembly. That's the much we can take this morning on the nation. Ezekiel Nyaituk, it's good to have you join us this morning, sharing your thoughts. Thanks for having me. All right. And let's start with the daily trust. Election violence claim 1,525 lives in 25 years. And that's the report. And I'm sure that you were following with all of other statistics. The fact that, you know, Boko Haram, among all the terrorists and groups are really a major threat for elections come 2023. How do you react? Okay, I think first and foremost, we need to draw a line between election violence and all these other crimes, banditry and the rest of them. Now, one is an enterprise that criminals have found very lucrative and they will do everything to continue, except there's an extremely strong deterrent. Now, these are criminals that are in the bushes. There is a second set of criminals that are in the executive offices. These are people who have seen politics and, you know, election as means to state capture and not a means to serve the people, on account of which they will kill, maim and do everything to get into power. These are two different sets of criminals. And at the end of the day, the people that they're receiving and are the poor Nigerians. And sometimes the time has come when these so-called poor Nigerians need to really sit down and know that the people in office are the people that should face the other criminals. But when these criminals are there for something else, then the other criminals will have a field day. The moment they have this understanding, they will now be careful into who they get into office in the first instance.  They need to understand that unless they come to support the good people, they will never have peace. And we actually spent quite some time, probably of two hours or there about, and they started to see things differently. And when I left, I left giving them nothing. Or next to nothing I would say. And they're happy. They prayed for me. They applauded. It's like, you know, your case is different. But really even the media needs to have a conversation on how we are going to tackle the criminals in the offices. Because they are the people that will tackle the criminals in the bushes. And for as long as we have state capture entrepreneurs coming in to suck our resources, these people are not interested in the matching order of what governance is all about. I've quoted this chapter and I'm probably getting identified with it. Chapter 2, section 14, subsection 2b of our constitution says that the security and welfare of the people shall be the primary purpose of government. Now, to what extent do we understand that? We look at the people that are at the forefront. To what extent do we think that Mr. Attico is somebody who understands about the security and welfare of the people. And he's primed in all ramifications to bring this about. Or Mr. Bola Ashiwaju, you know, is primed physically, mentally, spiritually, intellectually, emotionally, intelligence-wise to deliver this aspect of the constitution. Or Mr. Pitalbi. Or are we busy playing party, playing politics? And at the end of the day, we start to have another four years of letting the lives of the people get wasted. I think the time has come when we, the elites, or the allies, need to sit down and have a conversation with ourselves between these two criminals. The only people that can fight the criminals in the bush are these people that are in the offices. And once the criminals in the offices are the people in charge, they have no business with those in the bush because they are no better. They don't have any moral conscience to fight them.  You also go to the nuts. Of course, I talked about the North East. You have Boko Haram also majorly on that part. Should we be worried? And is there anything that can be done because we understand the significance of, you know, the 2023 elections and every other elections? We should not be worried. We should be proactive. Sitting down and worrying has never solved the problem, never. This is saying that people united are unstoppable. Let us give an enlightenment to the people. Let us give an agenda to the people and let the people rise up at this point in time and make sure that we have free, fair, credible elections. The people can demand it and they can have it. All this issue of, oh, we're worried. You see, let me tell you something we must know. There is faith and there is fear. Fear has never been a solution to any problem. The faith looks at a mountain. He knows it's a mountain and he says this mountain, be down, removed and cast into the sea and it is done and we move on.  If I ever thought of that, the last thing I would have done, a reasonable person like myself would have been to get into the governorship of Akwae Bomb State. When you know the actors on the field and the financial muscles that they must have, the most intelligent thing is to just respect yourself and leave the field for them. But there is something about the audacity of hope, of daring to dare. And let me tell you, times and seasons come up, but the people that should take advantage of such opportunities are too afraid. I dare to dare and I think the jury is out there. Let's see how it's going from becoming nothing in the race to becoming one of the top three in the race that is talked about in Akwae Bomb today. The other two have major issues. I'm the only one that has no EFCC, no ICPC, no court case, nothing, nothing. And I dare say that from nowhere, just the audacity of hope is about to rewrite history. And it can happen to you. Let's stop fearing and just dare to move on. So I totally understand where you're coming from from the perspective, but not worrying as in worrying, but worrying that it's an issue and then we're proactive just like you have mentioned. The government is acting to ensure that in all of this trouble spots, because this is a trouble areas and region, not to say that the entire country is not, you know, troubled, but to say that we have plans in place to ensure that this does not in any way affect, you know, or disenfranchise the people who affect the electoral process. However, this conversation will not end until we, you know, get to 2023. And of course, the elections are happening right there. But another one dominating some of the papers on the nation newspaper this morning, it talks about the state governors being responsible for the rising poverty in Nigeria. That's what the federal government is quoted to say. And specifically attributed to the minister of state for budget and national planning, Clement Agba, who said that, you know, during a briefing that, you know, this governor has paid more attention in having flyovers. Maybe there's a particular governor you're referring to, rather than thinking about construction roads in these rural communities and what have you. I'd like to share your thoughts on this. How do you react and respond to this? You see, this is one area that we really need to pay serious attention to. Number one is that we are the governments that believes in blame games, that are blaming everybody for everything, anything. Either the roads are bad or the people are doing this or the governor, that's on one side. But I'd like to be balanced. What they have said, I agree with them in principle. The details that they are giving is wrong. Now, if you live in a state like River State or Lagos State, where the traffic affects the economy, you can't spend all your life in traffic. You can't do anything. So you may need to, in constructing your traffic movement, build a lot of bridges or flyovers. There's absolutely nothing wrong in principles on that. But there is a point they have, which is why I came up with what they call social governance ideology. Now, the rural areas are being neglected and it is wrong. Let me blow your mind. I started saying in Akwaibom that I'm going to put 500,000 naira in. I'm going to invest, invest 500,000 naira in every village in Akwaibom state every month for four years. And people are like, how? How? I said, yes. Number one, they have agreed I would be wonderful with stimulating local economy. They asked me how? And I told them a little thing. Akwaibom has 2,226 villages. And if you put 500,000 in each village, it comes to about 1.1 billion a month. And guess what? That is about half, about half the security vote of the governor. That is about half the security vote of the governor, which means that without any appropriation whatsoever, I can put 500,000 naira in every village. And I showed them how that money is going to be spent, open up the rural roads so that the market woman will be able, the farmer can bring her goods to the house and market don't have to carry on her from there. We now have a kind of social security network where you can spend as much as 5,000 naira to all the very, very, very elderly women, the widows in the village, and then the maintenance of the women. When I feel people are like, I don't believe it, really? So this is possible? You understand me? We need to start to think of the rural dwellers as the people that matter because the constitution said that their security and their welfare is the primary purpose of government. We need to have to think in government, cerebral governance, where the rural area, the rural economy stimulated brings about peace. And peace is the number one thing for any attraction of investment. You want to go and attract foreign investment to where areas that you don't have stability and peace is not going to work. So we need to start to have people who are thinking as governors. And I want to challenge Nigerians. You hear these things. These people that have these ideas don't have the resources. This would have stolen the resources, do not care about these ideas. So the time has come when every Nigerian should stand up and support a candidate that makes sense. I know that that might be something to consider. I wish everyone should consider. But let's look at the kind of system that we practice as a country. It's the federal system of government. And of course, you have the three tiers of government. So you have the federal, the state, and the local government. The argument of the minister, Clement Agba, is that this governor should pay attention to the construction of roads in rural communities. I mean, why is the third tier of government not functioning as a third tier of government in Nigeria? What business does the state government have in terms of construction? Let's even look at the functions of the local government as it were, the reason that why they were created. Two things again, Nigerians need to be careful what they applaud. Now you come into a system, you lack confidence in yourself. So you want to do things that people will hail you. People will sit in 30 story building you've done. People will count the flyovers that you have done. People cannot count the number of people you've brought out of poverty. So it's not a priority to you. So we need to have a new set of people who start to define their channel by the poverty indices, what they met and what they've left behind. That should be the new world order in governance in Nigeria. People should think in terms of the lives they have built. People should be able to show that when they came into office, this was the unemployment rate when they left. This was the unemployment rate. This was the internally generated revenue and this was what they left. But that internally generated revenue must translate to the GDP that what they came in, those are the things that Nigerians should start to interrogate. When you do that, if the bridge that you build does not translate to a better life for the people, then you should be seen as a failure and not be applauded for building the bridge. When we have this mindset in governance as the media, putting that up as the basis, people will start to copy what I'm doing, which is thinking in terms of social governance, which is a bottom to top approach to governance and development with the primary objective of bringing the people out of poverty, which is investing in rural areas for their development and transformation as a priority, which is laying more emphasis on MSMEs as against the big and the big... But Ezeka and Yaituk, how can that be when the local government lacks autonomy? These are some of the conversations that we're talking about. Yes, I'll tell you two things. Number one is that the money of the local government is collected largely by the governors. I'm telling you this and this is just... We played the ostrich a lot of times.  Go and ask any local government chairman in a lot of the States.  And a lot of times, look, there's something that Madam Okonjo Iwala was doing and I hope we continue to do that, publish the amount of money that your local government gets every month. Publish that money and let the people of that local government ask the chairman, you've collected a hundred million this month, please show me where you spent the hundred million. Ezeka and Yaituk, in most cases... When it becomes really hot, they will now start to tell the truth that out of the hundred million, the governor just allowed me money to pay salaries and gave me additional five million for me to run the system. When that comes, they will start to ask the governor whether that is the right thing to do. Then maybe then it's really true that the state governor should be blamed for the underdevelopment or the poverty that the rural communities are faced with, especially with recent data and statistics. 100%, I agree. 100%. State governors have not been governors. They have not been what they ought to be. And I look forward to have a conversation with them, a direct dialogue. Because these people are... And they don't understand what... You see, people need to understand that most people in office don't understand what government is. They talk about experience. Experience. He has been a commissioner. He has been a... These guys are experienced in making all the property capital of the world. They have no understanding of what government is. They don't have. They don't have. And Nigerians need to come to terms with that. The conversation I've just given this morning, please sit how many of your people in government can understand what I'm talking about and that they are doing it. They don't. Their concept of government and governance is one word, state capture. That's it. Go and look at the houses they are building.  Your house is worth over 150 million. What's your salary? And people are not asking questions.  You are going around with the convoy of so many security aides. What are they securing you from a man who is earning less than a million a month? There's something we're not telling. And Nigerians need to wake up. And that time is now. Wherever you are, stand up and support a good candidate. There are many of them. There may be small parties. Support them. The electoral act is come to stay. It's going to one man, one vote. There's going to be a lot of surprises. Stand up and support a candidate that makes sense today. And Nigeria will be in for a change. And we need it so hard, so bad. Well, the punch talks about fuel scarcity. The federal government is saying that there's no planned price hike. That's what the government is quoted to say. Of course, the queues are already spreading. And we can't say that there's no planned price hike because, or hike, because people are buying petrol, 220 naira, 300 dereba depending on where you're getting from, the black market or from the filling stations. This conversation is almost endless. It's a lot that's going on. Fuel scarcity. Is it, what do you make of the situation and government's assurance? Yeah, you know, when there's a man that gives you cash, and you don't even trust if that cash is even original, in quotes. A friend of mine said something that his father told him. He said, may the man who is always trusted never lie against you. Let's flip it. Government has come to a point where they speak so much lies that the people don't even believe them. So much so that even when they are saying something serious, people are like, forget those guys, they're not serious. Because if you tell me as a government that you have enough reserve for 30 days, and that there's no cause for panic or alarm, and that there will be no price hike, and I believe you, then there will be no need for panic buying. But you see, what government is saying might actually be true, that they don't intend to hike the price and there's enough reserve, but people just don't trust them. They don't trust government. And that trust deficit is the paralysis of all elements of the things that we do in this country, because we don't believe government. And it's a shame, and it's sad. So as we are, people would rather want to err on the side of caution. Okay. And as a result, they are making life more difficult for themselves, because with the panic buying, the price is going up. It's just like when we're rushing to buy dollars that I tell you, don't be careful, that in January, the dollar is going to come down to 500. And during election, it's going to come back to less. And after election, it might start to come back again to about 500. Be careful how you panic buy. And be careful not to put fire in your house. Be careful. But is it a case of panic buying? Because in reality, there are some filling stations that are not even selling. They're not dispensing this product. They are short. And so it makes it very cumbersome for people to get product. And that's why you most times find a lot of queue, because if you have every other filling station selling, then there will be no need for the queue. It therefore means that the product is not available for whatever reason. I don't think it's panic buying. It's a combination of many things. And the Bible says wisdom is profitable to direct. At a time like this, maybe the time has come where you need to restructure your movements and make sure that you only go on the essentials for now. When things are like the way they are, you try to bet some people will necessarily need to buy. But I can assure you that nothing less than 30 percent of the people in the queue don't necessarily need to buy. And the answer is just one for me at the end of the day. Let's have a governance that is not about sentiment. A governance that makes sense. Look at the most competent, qualified person on the ballot to make your president the same for your government and the rest down line. Let us just wake up and know that the answer to all this is having a government that works, that is trusted, that is committed to the people. For me, it all comes down to that bottom line at all times. We need to go now. Thank you so much. I mean, so much for a country that's all producing, so much that we can't even refine our product. We have to export it to refine it and then import it back. Quite unfortunate. And then we can't even, you know, make this product available even when people are willing to pay at any price. Nigerians will buy definitely at any price, but then you can't find the product. And so the suffering continues to increase is so much, so much blame for the construction work that's going on in different parts of the state. I don't even see the connection because some philly stations are not even selling this product. But that's it. Thank you so much. You have a fantastic year. Thanks so much. Have a lovely Christmas ahead. Thank you. We take a quick breakdown and when we return, it's time for us to delve into our first major conversation right here. We'll talk about petrol and its scarcity, especially when it's the first period. What's the hope for Nigerians? Please stick with us.