I just realized I didn't pull out my notes, which is important since there aren't any slides. So we're going to talk about bugs, and the most important thing on this is where we start the conversation. So what's the definition of a bug? Because I'm talking here about bug zero. I'm going to make the claim that much like there's that whole inbox zero movement where e-mail it just becomes the thing you don't worry about because you don't have to deal with it. We're going to do the same sort of thing with bugs, only even further and we'll tell some stories about teams out there who just don't ever have, worry, think about bugs. It's just not a thing and how they got there. So the first thing we have to do if we're going to get there is get that simple definition. What's a bug and more importantly, what's zero? Because I find that while there's a lot of disagreement about bug, there's even more disagreement about the definition of the word zero. So let's start with the easy one, bug. What's a bug? A feature not guarded by tests. If a user loves it and it's all working, is it a bug? Something that doesn't work. Okay. Why is that a bug? Okay. So it got in the way of a user somewhat. Is that a bug if it has not yet gotten to a user? Is it a bug if a QA person sees it? Okay. Is it a bug if it has not yet gotten to QA person? Okay. Is it a bug if it has not yet gotten to source control? Yes. So there's some time at which a bug becomes a bug.  Right? Yeah. So there's some time when it becomes a bug. Okay. And then that one we said it was a bug because it got in the way of a user getting something done. What about a bug that doesn't get in the way of a user? Are there other bugs that we would consider bugs, but they don't actually get in the way of a user doing what they want to do? Such as? I saw an odd. Any other ideas? So what about if I'm running an ad campaign or something and I change the look of a button and I get half as many clicks on it as I did before? Is that a bug? I just cut my revenue in half. What was that? Yeah. So it depends on whether it's in the requirements. That's an interesting answer because that seems like blame shifting. Like, well, it's a bug if I screwed it up, but not if you screwed it up and didn't put it in the requirements. I mean, did it change the amount of revenue the company made whether it was in the requirements or not? And I don't mean to pick on you in particular. I mean to pick on that answer in particular because it's a common one. So the definition of bug that I use, and I'm always looking for a broader definition. So if you can broaden this definition, great, tell me how you would broaden it, and we'll use that one instead. So the definition that I use is anything that would frustrate, confuse, or annoy a human, and is possibly visible, potentially visible to a human other than the person who is currently actively writing it. So basically it's once it hits source control, then it could be viewed by some other human, that human potentially being me in my next commit. So if I do something and I think I had it working a particular way, and in the next commit I go, and then I'll go, wait, why is it bug? I just got confused why it was doing something. And I'm totally welcome to broaden that in any way that I possibly can, but my belief is anything that would get in the way of user doing their job would frustrate them. Anything that's going to drop the company's revenue by half, there is at least one human who will be frustrated by that, perhaps disappointed. So I'm using that definition of the word bug. So given with that definition of the word bug, and I'm talking about teams that have no bugs, what's the definition of the word zero? One minus one? So if you've got an additive ring, no. So, well, let's just start by sampling. So on your team, that you're on, oh okay, we don't want to talk about your team. So on a team of someone who works at another company that you know very, very well, that's totally not yours. I want to get an idea of for that team, I want you to give me a couple three numbers. About how many bugs do you have, does that team have in their bug database right now? Some total of anything that's not closed done. So if there won't fixes or open or I haven't gotten around to it yet or even triaged it or whatever, some total of that. Total number in your bug database. And then tell me the number of new bugs that you write per, let's say per month as a team. If you don't know the number that you write because it is much larger than the number you can detect, then say, don't know, but the number of additional ones we detect each month is. Because a lot of teams are in that particular circumstance where we write a lot and the number that we create is limited by how good detection we do and how many hours we spend on QA, right? That's normal, right? But I just want to get an idea of teams in the room. Where are you in terms of number of bugs in the bug database and number of new ones created per month? Or teams that you know? Anyone willing to volunteer some numbers? Yeah. 250 bugs in the database and about 50 a month, okay? Other numbers, about one new one a month, three in the database, okay? Other numbers? Anyone have more in their database than 250? All right, we got an honest person in the room. Yeah, anyone generate more per month than QA can find? Oh, we got a lot of honest people in the room on that one. Yeah, okay. All right, so interestingly when I asked this question, I find teams roughly categorize into three camps with an asterisk. So the first camp is how many bugs do you have in the database? No idea, lots. Because there's the number that are active in the database but then there's all those additional ones that were triaged or won't fixed in previous releases and we didn't roll forward and I don't really know how many because we haven't even looked at all those in 15 years, right? So there's that camp and then you ask them how many they generate per month and they're the ones that say more than QA finds. How many total bugs in the product? More than QA has found, don't know, okay? So that's the first camp. The next camp is I find there are a set of people that are able to give a pretty concrete answer like the one that you gave. We got 250 in the database, plus or minus eight, you know, whatever, and we generate about 50 per month and whatever and those numbers, they vary a little but they're often in about that range. We got a couple hundred open, maybe as low as 50, 75 open and never more than about 3, 400 open and we generate per month on the order of one, like it's a lot like one bug per developer per day-ish.   Number of bugs in the database is below five, often zero, the number of new ones is once a month, twice a month as a team, okay? And in that camp, they often answer the question differently. Number of bugs in the bug database, zero. Always zero, well, a sampling error. And in order to ask the question of how many bugs are in the database for that camp, often you have to ask how, during how many hours in a month is there a bug in the database? One, two, four, you know, those sorts of things. And then the asterisk is teams like that. There's a bunch of teams in that category and then there's teams like Hunter Technologies, which ask them the same sort of numbers and their definition of zero was, well, you know, there were a bunch and then we changed the way that we worked. And we started doing mobbing and we started doing some of the other practices. And the first week after that we had a bug. Then 18 months later there was one. Then we root cause that one so we didn't have a recurrence. We had a situation about a year and a half after that that could have been a bug but we'd fixed our practices so we noticed that and it didn't actually happen. But about two years after that there was one. Those are the definitions that I consider zero. Now, does Hunter have zero bugs? No, over six years they had three. Does your team have zero bugs? No, every month you've got one. But from a planning perspective, that team can pretty much treat it as zero. How do you plan an account? What is your triage process if you've got three bugs in the database and you create one per month? Yeah, the triage process is a fix it. Your management and oversight process. So when I talk about zero bugs, those are the groups that we're talking about and we're gonna talk about how people got there. So, Hunter Technologies I alluded to, they'll probably come up a few times and they're a common example for this because they went in relatively short order over the course of about a month from typical defect rates to just not. And that was when they introduced mobbing and in particular brought in, they didn't just mob with programmers, they brought in other people. So they were doing factory floor automation. I mentioned that defect that, oh, we would have had that one but we didn't. Well, they were doing factory floor automation and they had a bunch of tools and things that they were developing for. And they produced something like 50 products out of the one for control systems for different parts of the thing. And the second bug that they'd had, the one that was like a year and a half in, it was of the form of, the software does the right thing, it's just not actually usable in the physical plant because of actual user concerns. So they brought in, once they were to cause that, an assembly line worker and as part of the mob, they had one of the customers always on the team and he'd rotate through and he'd code all the roles that everyone did. And sure enough, about a year later after they, and they had a person in for a couple of week rotation and then get in some new blood. So a couple of years later, they had one where they're designing this thing and it's another sensor gauge display when something's in a control band and when it's going out of control so that people can do something with it. So they've worked it out and they know what they're doing and they're trying to figure out where to put it. So they put it up on this display. It's a lot like, it's for this one particular section of the system and so they put it on the screen for displaying that and they're going on with life. And then the guy pauses, I've got a problem guys. This we're putting on the thing that it's related to. Like yeah, that won't work. Well, that screen is for programming the machine and it's over on this side of the machine and the reason I need to look at this control band thing is because I need to stand there holding a lever and if it goes out of control, I need to throw this switch instantly and bring things to a grinding halt before the machine blows up or whatever it's going to do. And the problem is that that screen and this lever are nine feet apart and my arms aren't long enough. So I need you to put it on this other screen on a different machine that makes no damn sense. Yeah, that's true, but that's where the lever is. So that was a bug that absolutely it was a bug. It would have made the system not work and it would have frustrated the heck out of some user, but they didn't have because they had that person on their team. And those are the sorts of things that we need to do if we want to eliminate bugs. So in Mary's talk this morning, she talked about in the old world, we built systems to be defect free and in the new world scaling out, we're building systems to be fault tolerant. Okay, yeah. So when we say fault tolerant, what does that mean for these kinds of bugs? They're introduced by humans and developers and whatever else. How many of those do we have in a system that we're building to be fault tolerant? In the old world, we would try and build defect free and make sure we didn't have those and that was a big important thing, right? So what about when we're building fault tolerant? Thoughts? I heard two things at once. What was that? Fixed bugs faster. So it's focused on fixed bugs faster. Okay, what was you thinking? Yeah, if you're doing a fault tolerant system, yeah, how many bugs and what are the bugs like there? Okay, so high bug count? No, yeah, so this is a key mistake that a lot of people make.  No, that's sloppy, that's undisciplined, that's not scale out. That's just sloppy, right? So there's defect free versus fault tolerant versus unintentionally screwing up all the time, right?  A fault tolerant system has just as much discipline, just as much, if any defect is possibly detectable, the whole point is detect a defect early and recover from it early. So what's the earliest point we could possibly detect a defect? Is it live in production a quarter second after it happens or is it during development a quarter second after we created it or maybe a quarter second before we're gonna create it or the pair partner goes, uh, that thing you're gonna do, I do not think it does what you think it does. Yeah, so when we're building fault tolerant systems, that is not an excuse to be sloppy. In fact, if we're gonna build a fault tolerant system, the only way that it can really be fault tolerant is that when it goes into a state of fault, which happens rarely and it's hard to test and is usually not so well thought through, it needs to work exactly as the developer thought if it's going to recover when resources are constrained, the behavior of the world is not as we expect and it's still gonna survive, you cannot have any sloppiness in that system. So when we're talking about fault tolerant scale out systems, there's actually a higher bar for none of these sloppiness bugs. Another thing is I talked about a bug happens as soon as it would potentially be visible by a user or by a human, okay? So what does that mean in terms of testing? Is testing a component of getting to zero bugs? What role does testing play in getting towards zero bugs? Yeah, testing surfaces bugs that are already there. Therefore under my definition, it can't get me to zero, right? Because I've already got a bug. I just didn't know it yet. So testing has nothing to do with zero bugs except it's the outer control loop that tells you whether your actual process for getting to zero bugs is working or not. So we're gonna get to zero bugs and we need to have a mechanism that gets to zero defect development before tests are ever invoked. So I guarantee TDD is not part of the answer. I'm a big fan of TDD. It isn't part of the answer, okay? So there's a really interesting example here of, so what is part of the answer, okay? So if we're going to prevent bugs, that's the only way we can get to zero. We have to prevent ourselves screwing up, right? And a bug is an encoded developer mistake, right? So if we're going to prevent them, we have to know what are the causes of bugs, right? So how could we detect the causes of bugs? It's really two kinds of answers for how we could figure out causes of bugs that we can then go fix, prevent, whatever. Okay, so particular technique, pairing and tripling and the like can help us detect and notice mistakes, yeah? So that's an example. Another example, there are a couple of common practices that are built into extreme programming that are related to being able to source categories of defects. But there's basically two places that you can look for what are the causes of bugs, in your team or in the industry in general, right? So in your team, you've got a bunch of practices, retrospective and pairing being two of the most significant ones, right? For looking at why do we screw up? What is it for this team, right? And then industry in general, you can look at what are the things that screw up all the teams out there? And how could we learn before we even make the mistake the first time, right? Both of those are useful, okay? So Nancy Van Schodenburg has a really great example of really depending only on looking at their own team and how they screwed up. And so this was a paper that she published. Agile by the Numbers with Newbies, something like that, Agile Software Development by the Numbers with Newbies, something like that. Google will find it for you. In about 2006, 2007, something like that. And she had been working on this project. It was a control system for a combine, I think it was, hardware software combined project. And she'd been arguing at her company for several years that we need to do this Agile thing. And the powers that we had gotten annoyed with her, so they said, fine, all right, we'll give you a project that you can do Agile. But silently, we're gonna make sure it fails, right? So you can go ahead and do that. All right, so you're gonna do this combine and you've got limited scope, limited budget, you have to make it work. And here's your team. So you're building a combined hardware software thing. You've got a web developer, a DBA, a person straight out of school, you, who has done some embedded control work before, but you're supposed to be the manager, so you shouldn't code. And one person who has used C before. How many defects do you think this team had? Over the course of, I think it was about three years of active development and three years post-delivery that they examined, something like that. Two or three years after delivery, I've forgotten the exact numbers. How many defects do you think were found? So guesses, ballpark, 15, you'll only say that because I picked this as an example to demonstrate goodness, yeah. How many do you think a typical team in that circumstance would generate? This is 7,000. Yeah, something like that. This team, I've forgotten the exact number, it was like 53, 54, something like that. It was 47, thank you, 47. Yeah, and they know exactly how many it was and if you go look at that paper, they've got root cause information for every single one. And this is a team where, when you go look at some of those, one of the early bugs, I've forgotten which one, happened when they were caused it, the story that Nancy tells is, so we're a couple of months into, a month or two into development and a couple of the developers find this cool, interesting new language feature, global variables. They've never heard of these things before because they haven't used languages that have them. But they've been having this problem that they're passing information around from function call to function call in their program and the call stacks are getting, the argument lists are long and it's annoying and there's information that's used over here and they have to pass it all the way over there and wow, global variables will totally solve this problem. So they start just jamming all the information into a bunch of globals that anyone can access whenever they want to and building out this system. And what's Nancy's response to this? She's a manager, she's built a control system before, she knows what's going on. She has heard of global variables while her team hasn't, all right. What do you think she does? What would you do in a similar situation? Tell them holy crap, stop, yeah, yeah. So what would happen if you told them to stop? They won't understand why, they'll resist. They'll use them sometimes but not others, they'll, yeah. So what Nancy did was she said, cool, that's about it. They did their things, they did their thing and then from that moment on, she paid very, very, very close attention to every commit, everything that anyone was doing and the first time that she was able to detect, ah, they just introduced a bug. She then figured out what the test case was and surfaced the bug, all right. And poof it shows up and everyone, oh crap, we had a bug and she had built in from the get go. These things are optional, we don't need them, right. Let's not.  Root cause, she'd built the practice around, anytime there's a bug, all work stops. We root cause the bug, we find out what the total expanse of exposure to this, we're going to fix this symptom and any other symptoms because the bug is not the symptom, the symptom just surfaces it and we're gonna find this pattern and what pattern led to this error and we're gonna prevent the whole thing, right. And they do that and they go, global variables, this shared state and we can't do good reasoning over who's modifying it when and how and crap, this language feature is dangerous and the team says, yeah, wow, is there any time that it can be less dangerous and they thought about it, think about it and they're like, no, not really, right. We just need to not use this feature. She goes, yeah, seems like a good answer, right. That team never made another error related to global variables again, right. So Nancy in her role as management was focusing on making sure the team learned from every mistake they made and had the opportunities to make the mistakes in a safe way. And they were continually refining and inspecting and detecting and then when they made a type of mistake they never made it again and that's why they had 47 total bugs. We can do even better than that. I mean, first of all, we should do at least that good. So we should all go home and just stop writing bugs. Yeah, but we should start learning from stories like Nancy and like Hunter Technologies, right. And look at when we have mistakes, how can we surface them quickly, learn from them, prevent them entirely and go solve their root causes. The way that we can do better than that, I have a slide, don't worry, it's blank. The way we can do better is we can learn not just from ourselves but from industry, from what are common causes of failures. So, simple chart we're gonna fill out. We're gonna look for what are the reasons that bugs happen. So I mentioned earlier a bug is a mistake by a developer. Developers create like hand-crafted, artisanal, carefully selected errors, right. That's what these bugs are. Every bug is a one-off, magnificent creation. Now, do we honestly think that developers want to create the bugs? No, not at all, right. So what is causing them to? So, what are examples of some of the things that cause developers to make mistakes? Not understanding the problem. What do you mean by limited scope? I said limited understanding what they mean. Okay, I'm gonna call that the same as not understanding the problem. It's another aspect of the same thing, yep. I heard complexity and rushing. So complexity, what do you mean in terms of complexity because that has a number of different meanings. Okay, yeah, complex interactions where anything, I don't know what all's affected by what all, yep. Distraction, okay, and someone else said business. Actually, okay, other wrong assumptions, okay. Thank you, thank you keyboard. So by the way, these are ordered. The top line is the one that's the cause of most bugs. The second is the second most significant and third is the least most, or is the third of the top three. And these account for large, basically, most bugs. So they're all significant. But it is pretty common that all of the examples that people think of point to item number three. That people have trouble seeing the most common and the second most common cause of bugs. So there was an interesting paper that I read. It was in, I mean it was published in like the early 80s or something. That was analyzing, someone tried to analyze, they were comparing different programming languages and trying to identify, do we produce fewer bugs when we write in C than when we write in eta or assembly? And why? And as a side effect of that, they looked at what are the common bugs? So the number one most common cause of defects up until the mid 2000s, when it changed because there was a change in development tooling, not language, but the tooling that was commonly used by people. It's still fairly common today, but there are tools that you can use that absolutely prevent it. What do you think is the number one most cause of bugs in code that sets people up for error? What was that? Spelling. Actually spelling and poor names in general does appear it's a little further down on the list, but yeah, yeah, especially misleading names. When you call something log in and it doesn't actually do log in, it does something more interesting than that. So number one cause, no pointer, uninitialized data, that's actually further down. Compiler behavior change. Compiler behavior change? That doesn't even make the list. Number one, nobody ever guesses this. White space. Yep. If you have inconsistent indentation in a file, that will cause more developer errors than anything else. The best way to screw up a program is to screw with the characters that are not even interpreted by the language. Because the instant you do that, then any human reading the code and any compiler reading the code get very different results. And from then anything the human tries to do will be screwed up. So the tooling innovation that changed that and that made it significantly less is an IDE where you can auto format.  If, by the way, you are still programming in a text editor and your development environment does not have semantic awareness of your language, it only has syntactic awareness of your language, no, you are choosing, actively choosing to write bugs. Your choice. Yeah. Now a number of the other common ones, that first camp, here are a couple of the other items up there. So white space, names, misleading or poor names. What is another one? Long methods. Really, really, really long methods, like 26,000 lines long. Yeah, complex interaction on testable code. Presence or absence of tests doesn't seem to actually change much of the defects, but if I take, if you take code that was written in a TDD fashion and then remove all the tests and then modify it for a while, it still has fewer future defects. Doesn't matter that it was tested, it matters that there is something different about the shape of the code that results from it having been created, having been testable at some point. So anyone wanna guess what this first category is? Human error style, I actually heard the answer here somewhere. Readability, yeah.   That's a bug, it's a bug waiting to happen. In fact, by my definition, it's already a bug, it confused a human. But if you can't tell what a piece of code does even locally, there is no way you can safely modify it. So what's this next category? Architecture, design flaws, yeah. And it's actually a particular kind of design flaw. Mox is the big clue here, actually. So side effects, yes, side effects are an example of coupling, interactions, yes. These are context dependence of any sort. Context neutral code is very easy to reason about, whether it's because it's peer and side effect free, very easy to reason about, or if it's tell-don't-ask and it has no returns and it only sends messages, very easy to reason about, right? Asynchronous and interacts just via message passing, easy to reason about. It's a co-routine that executes in parallel with a number of other things and does non-local returns back and forth between not so easy to reason about, right? Even if I can read it locally, I can't reason about the damn thing, right? It uses threads, you know, just freely and mutexes occasionally to block. Not so easy to reason about, right? Yeah, so anything which is context dependent, even if I can read about it locally, then there are gonna be other aspects, you know, what happens when it accesses that other object and that can screw me up, okay? And the third one, what's this? So the first two were something about the code and they really are, and the things that most commonly can screw up developers and cause them to make mistakes are encoded in the code and it's actually your code that is biting you, which is nice because as a developer, the easiest thing for you to fix is your code, right? So the two most common causes of bugs are totally within the developer's control and they can just fix that for how C read by refactoring and other things I'm not talking about at this conference. But we can talk about later. This third one, are those code problems? Miscommunication, that's half of them, yep. Anyone? Experience, yeah. And there's an even broader category of that. Human, they're all human factors, yeah. And a little bit narrower than that. So some of these, you know, business wrong assumptions, not understanding the problem. Well, I guess all of these are miscommunication sort of things, but what about user doesn't know what he wants, but he sure knows it when he sees it. If the user can't tell you at the beginning what they want, but then they use it and they get really frustrated, is that a bug? Yeah, absolutely it's a bug. So what was the cause of that one? Yeah, well, it's not understanding the user, but it can be, it's actually a complete lack of information. Like nobody in the world knows the answer. Sometimes, you know, we're innovating, sometimes there's learning to be done. And if no one has done that learning yet, you have to do the learning somehow. If you do that learning by releasing your product out live and building it out in the product, then it's pretty expensive to learn. If you do that learning by doing a design sprint and figuring out what the right answer is, then it's a lot cheaper to learn and you don't have to write the bug. So, each of these has ways to detect and ways to mitigate or fix. So how do you detect a readability problem? Yeah, yes, and this by the way is one of the ones that everyone forgets because developers read code that's ugly and they go, well, my job's to understand this, good thing I'm a smart person. No, you just got a signal that you're most likely to make a bug.  Okay, well, I've got some work to do so that I can make this into a form that I can read it without being a smart person. So how do I do that work? I heard it mumbled. Refactor, yep. And in particular, what you see here is a lot of the core six refactorings and naming as a process. I'm giving those because you can Google them. I've got some blog entries and the like about that. But all of that is fundamentally about creating local understandability of the code.  Okay, how do you detect complex interactions and context dependence? Use it, and in particular what way? Yeah, so if I want to see where the code is context dependent, then I should try and use it in two different contexts. So one is integrated into the live system, what's the other one? In a test, and in particular, for it to be a different context, it has to be in a test that it's not connected to anything. Because I've seen what it looks like when it's connected to anything, so then not connected to anything. Great, what is the way that I take code and connect it to nothing? That's exactly the wrong answer. Yes, it is the answer everyone gives. Mox are not for that purpose. No, when used that way, what a Mox is allowing you to do is mask exactly this problem. It is telling you that you have code which cannot be verified outside of its context. And so we're going to make a fake version of its context that we promise we will always and forever make match the real context, except when we forget, and then we'll just have some false negatives in our testing, but that's okay, right? I don't like Mox. What Mox are telling me, when I have a test and I need to use Mox in order to verify this in some different context, that's telling me that my design has a context neutrality violation. It has a flaw. It has some really heinous dependency, like an object that I'm depending on an interface for. And it's got like a couple of methods on it, some of which have both arguments and return values. Like the huge heinous dependencies like that, which we should not have, right? And when you get a lot of sensitivity to this, it really starts changing your design. So this one is tests. Yeah, if I can't test it or I can't test it without use of Mox, I have a problem. And the fix here is refactoring again. And some people that I've worked with when I was in Microsoft have since gone and done a whole lot of stuff on dependency elimination. I mean, I've got some good blog entries. Like Brian Giesler has one there where he talks about with the dependency elimination principle, solid design becomes so. Because everything is single responsibility and that's fairly straightforward. And all the other stuff is to work around the fact that you're using a whole lot of inheritance and you just don't need that anymore once you eliminate dependencies. So the rest of solid vanishes. And the third thing, how do you detect miscommunications or learning to be done? How do you detect them? What was that? Retrospective and discuss. Yeah. Yeah. Honestly, this category, you're gonna make each mistake once. I don't know a way that you can detect it in advance because these fall often in the camp of you don't know what you don't know. Yep, you're gonna make those mistakes. So retrospective is your chance to, as soon as you find out, oh, I didn't know something I could have or should have known to do something about that. And how do you fix this? Who says refactoring is just for code, right? You make small local changes to your human systems, which are verifiable, which are rollbackable, which have all the properties of refactoring. So if you do these couple of things and you really look for those sorts of bugs, then go ahead and do the root cause analysis. You, too, can get to a very low defect rate. If you aren't already doing XP practices, typically teams do XP practices and just apply them blindly and they get to similar to the numbers that you're getting, right? Maybe not all the way there, but most of the way there. And then if you do these, you can drive it down to Hunter Technologies sort of numbers. So I encourage you to join the camp of Bug Zero. We believe that defects are optional.  We should talk about bugs in software about it as often as we talk about bugs in surgical procedures. That they're not a good thing. Or as often as we talk about fires that burn down entire cities, right? We could prevent all of those things. And if you're interested, please reach out to me. There's a Slack group that we occasionally chat on. And we'll just take this approach back to your team and fix your own bugs before you have them. Thank you. So I woefully ran over my end time, but there's a- We can take some questions. Anybody has any questions? I think all these categories and examples probably, there's a whole category missing which is the omission errors. So you haven't done it yet. Is there such a category? So, yeah, errors of omissions. So there's two possibilities, I believe, in something that I haven't done. Either there's something that I haven't done and nobody really expected it to be done, in which case no one's frustrated or disappointed. Or there's things that haven't been done and someone for some reason had an expectation that it would be. And those I would consider errors, consider bugs. And those I see as miscommunications because if I am setting up, like if I am implementing half a feature and I know I'm implementing half a feature, I should clearly communicate the half of the feature that's there and the half of the feature that's not there yet, right? And then people will go, oh, it doesn't do the thing for me yet. You say, yeah, we're gonna get to that. Actually, it's not next, but soon. It's one of our interesting items. Okay, right, people are okay with that. If, however, I mislead somebody, then they get partway through and they say, but it doesn't do, right, and now we have a bug. Yeah, because now we are doing this continuous delivering thing. And we are always at some point of our development. So does that mean that this kind of omission error is actually acceptable? No, I need to always be honest, right? The system does exactly what I say it does all the time. And so if I have implemented a half of the feature and it is useful to users, great, then I do want to roll that out and I wanna include that in my deployment. And I need to also have a way to communicate to those users that this does this part of the thing and it doesn't do other things, right? If I mislead users and they go, oh, finally, I've got sorting and I expect the entire system to everything that I can sort and I can define all the sorts however I want and whatever else, then they're gonna be very disappointed when all I've done is alphabetical sorting of a couple of columns, right? If however I say, we are starting to roll out sorting capabilities that seems like it was useful, we're doing alphabetical sorting in these areas because those are the most important ones to people, how does this work for you, and so on, then people go, okay, cool, you're headed down the right direction, please give me more. Do we have any other questions? If not, can we thank Alo for his very insightful speech? Thank you.


Hey, Tim Crow here with the Alabama Cooperative Extension. Just out in the landscape today, looking in some of our turf grasses, just to kind of see what weed issues we're dealing with right now. Hopefully everybody got a spring pre-emergent put out late February, first to March, and you're sitting in pretty good shape as far as our warm season turf grasses go. Most of what's kind of playing out right now is cool season weeds. It's germinated back in the fall, kind of early winter. And they're kind of playing out. The heat's going to drive those out, whether we do anything or not. So we've kind of turned our attention away from those weeds and towards the warm season weeds that are starting to pop up now. And our main target weed, most of the time here in the south for warm season turf grasses such as Bermuda grass or Zoysia or Centipede, we're usually targeting crabgrass, okay? Crabgrass usually germinates when soil temperatures get into that 55 to 60 degree range, which we are there. I'm seeing some crabgrass popping up in shrub beds, out in the turf grasses, all those different areas. So we have hit that soil range. And so if we didn't have a pre-emergent out, we're about to have a lot of crabgrass on our hands. Now crabgrass, it's a warm season annual weed. So what's popping up now is from seeds from last year that played out in early fall when we got a frost and it dropped those seeds. And they laid there in overwintered and now they're starting to germinate. And again, that's why pre-emergence are so important to help prevent situations from getting too out of hand in our warm season turf grasses. Now some good options for that. Pendameplin products, prediming products, dimension. All three are very good at controlling crabgrass. And they also have a big or a large broadleaf label, as far as weeds as they control this well. So good options for preventative control. If we do have some crabgrass like that I've got here, we've got certain seeds, small seedlings pop up. And this is about the size that we want to attack crabgrass, if we've got it in our landscape. Once it gets a little bit larger than that, it really gets hard to control from a herbicide standpoint. So some post-emergence that we can use if we do have some. Would be Celsius. It's made by Bear Chemicals. Very good product as far as controlling crabgrass. And then herbicides that contain quinchloride. Both of those are good post-emergent options if we have to spray some crabgrass out of our warm season turf grasses. Again, preventative controls, pendameplin, prediming, dimension, all three good pre-emergence. Quinchlorac products as well as Celsius are good post-emergent options. If you have any questions, you can also always contact your local extension office or you can visit our website at aces.edu.


A lot of good stacking options for tonight in daily fantasy baseball despite it being just a seven game slate there are probably six or so teams you could really justify stacking for tonight inherently if there are a lot of stacks in a seven game slate that probably means pitching is thin and that is very much the case or today in fact my third rate pitcher is not currently in the player pool so we're going to break down what that means who that is and how to play things for tonight in daily fantasy baseball by talking through those pitchers talking about stacks and getting you ready for Monday night welcome on into the solo shot that's right here on the fan dual podcast network and number fire dot com my name is Jim Saunas I am a senior writer and analyst for number fire here to break down Monday's seven game main slate with lock set for 640 p.m. Eastern for today and on this slate no weather notes a couple of retractable roofs are going to save us in certain spots so no games to worry about no massive temperature discrepancies bit cooler in San Diego but outside of that pretty green lit for this entire slate so we're good to play things straight and play the best plays we'll talk about who those are in just one second the first day reminder to make sure you're subscribed to the number fire daily fantasy podcast feed wherever you get your podcast we of course right here every weekday talking about mlvdfs pga podcast for the rbc canadian open is coming up tomorrow via myself brandy cadoula and of course usc for select events as well all right here in the same podcast feed so go search for the number fire daily fantasy podcast feed wherever you get your podcast hit subscribe and if you like what you hear leave us a five star rating as well it is almost time to crowd an nba champion and fan dual wants you to be part of the excitement because right now new customers can get a no sweat first bet up to two thousand five hundred dollars that's two thousand five hundred dollars back in bonus bets if your first bet doesn't win there's no better place to bet all the finals action than america's number one sportsbook fan dual official sports betting partner of the nba must be 21 plus and president select states first online real money wager only ten dollar deposit required refund issued is not withdrawal with bonus bets that expire in 14 days restrictions apply c-fold terms at fan dual dot com slash sportsbook fan dual is offering online sports waging in kansas under an agreement with kansas star casino llc gambling problem call 1 800 gambler or visit fan dual dot com slash rg in arizona call 1 800 next step protects next step to 5 3 3 4 2 in connecticut 1 8 8 7 8 9 7 7 7 7 or visit ccpg.org slash chat in indiana 1 800 9 with it in y oming in kansas 1 800 5 2 2 4 700 or in kansas ks gambling health dot com louisiana is 1 8 7 7 7 7 7 0 stop in massachusetts gambling helpline ma dot org or call 800 3 2 7 50 50 for 24 7 support in maryland md gambling health dot org in new york 1 8 7 7 8 open wire text open y and in west virginia go to 1 800 gambler dot net pitching preview for this monday main slate erinola checks in with the highest salary on fan dual his salary is 10 000 flat followed by blake's now at 95 we got braxton garrett at 9 000 with martin pares yohan oba though and branded belac as the others at eight thousand dollars were higher so as you can hear there it is definitely thin at the top for pitchers for today and i think the saving grace in this slate as an erinola has shown signs of life recently where his uh he was struggling to open this year i think it was potentially due to pitch clock stuff given how slow he pitched in the past and it would make sense that he would struggle but he's bounced back recently he's facing the tigers for tonight i think it's a good spot to feel good about him as our top ranked pitcher here it is a fantastic match up here to troy 74 w rc plus against righty's 103 iso both those are easily the worst on the main slate nola has made some tweaks to his repertoire since his slow start over his past seven starts he's using more sinkers and fewer forcing fastballs for a lot of pitchers that is not good for dfs because sinkers are low with pitches and that is also true for nola but for whatever reason it does work for him it's worked from in the past and it has helped him here as well because in that time with more sinkers he has a 3.88 skill interactive er a he has a 23 percent strikeout right which is not what he was last year but it is better than he was this first four so starts this year and specifically the past couple of events nola has been getting a lot more his strikeout totals the past three games have been 10 7 and 5 only two out of seven starts in the sample where the he's using more sinkers have come at home he's at home tonight he has had some fluky home road or some funky home road splits in the past that's a good thing i have nola projected for 6.2 strikeouts tonight which could be a bit light but i do feel good about it and that's the top mark on the slate as well so nola will be popular but i think he's worth it finally so i would say use nola deviate from there as far as your stacks go because he does make a lot of sense for tonight now typically i don't want to go to braxton garrett's because his pitch count is limited which does hurt his upside but there's aren't a lot of guys in the slates who have a true ceiling and i think that allows garrett to be in play for tonight pitch count definitely the biggest concern here for garrett because he has not gone longer than 88 pitches in a start yet this year which means he has gotten the quality start bonus on fan duel just once the entire year which absolutely hurts his appeal in a big way but despite that he's still getting strikeouts because he's been very efficient especially recently over the past six starts garrett has thrown more cutters instead of four seamers and the cutter has a 34 with rates according to baseball savants it's worked really well he has a 26 percent strikeout right in the six starts his skill interactive era is 3.66 which is the best number on the slate if we gave garrett more length he would be the top option above era nola for tonight but even with the low pitch count i still have garrett projected for 5.9 strikeouts that ranks third behind nola and blake snell but when you consider all the factors like you know opposing opposing matchup uh stuff like that i feel like it does point towards garrett being the better option garrett gets the royals you know not a fantastic match because they're pretty good against lefties and snell gets a high strikeout match with the cubs but they also have a lot of guys who can bang lefties in there so i prefer garrett personally if you want to go snell i get it but i think garrett is a solid play on this slate specifically for a salary of $9,000 now for that third slot i could talk snell because garrett technically is our value play for tonight but instead i kind of want to talk about a guy who's not currently in the player pool and i'm guessing he'll be added otherwise i wouldn't talk about him but that guy's andrew abbott he's making his debut against the brewers and assuming he is eventually added on fandall i think he's at least interesting for tonight abbott started this year back in double a he had a 64 percent strikeout right there so shockingly he got promoted went to triple a and across seven starts there his strike area was 35 percent with a 14.5 percent between strike rate that's awesome especially on this slate where there aren't a lot of strikeouts to be had abbott was not flawless he walked too many guys he led up a bit of hard contact and that means he could struggle at great american ballpark because he's also kind of a fly ball pitcher but the strikeouts are alluring on a slate where few guys can rack them up he's facing the brewers tonight they just got louisa reiss back it will be a domus homeward and a rehab game as well so he may be back here so when you look at the brewers numbers against lefties this year those are i would say misleading based on the actual righties they will have available to them in the very near future they're not going to be that bad but they should still strike out a lot which is kind of the appeal here and abbott has gone longer than 95 pitches in the minor so he actually has more length and more projected pitches in my numbers than braxton garret does so i like abbot enough to go there again assuming he's eventually added if not snell would be third for me but i really like to go abbot if i get the chance to do so so to me pitching ranks for tonight are going to be erinola one braxton garret two andrew avid three and then if i must i will go blake snell uh in the third slot if there is no avid added stacks as mentioned are pretty loaded and that begins the fillies and the fillies have not hit lefties so far this year at all and they're facing lefty and joey wence for tonight their wrc plus is 83 i think they might get a little kickstart here in a plus match but i do like the fillies for stacking in the situation wence has struggled quite a bit this year his eray is 7.28 across 11 starts he has a 5.62 expected eray in large part because he's just letting up way too much hard contact he's let up a 45 hard hit rate with a 40 fly ball rate he's let up just nine home runs uh which is lower a lower number than you think but he did have three home runs in one game so big games are very possible here in this matchup wence has let up five plus earned runs and five out of 11 starts and the hard contact issues may be getting worse because his hard hit rate allowed is 57 percent across his past four starts so the fillies haven't hit lefties well so far this year they did lose a key badnail bone who had denset lefties pretty well also so it's tough but i do think it's a great matchup so i'm gonna give the fillies the spin here rank them number one and be high on them for tonight now the boom thing is a bummer because i don't use boom ever against righties but i will definitely go there against lefties so not having him out there is a bummer but to account for boom going down they promoted drew ellis and ellis i think is either a double dong or at least at least one home run uh last night he had uh two run so yeah double dong for for drew ellis last night and if you remember ellis from college i think he went to louisville um pretty good hitter there so not a huge surprise has not done it in the in the majors yet but hit for power in triple a last year good power numbers in the minors this year i would guess it's probably going to bat eighth or so despite the double dong last night but i think if you need salary to get to the key filly bats uh and to get to aeronola i would say that ellis is a mighty fine play it's not just point chasing basically the two homer game last night but i think he's pretty solid across the board so drew ellis it's not point chasing i don't think to go there tonight semi he's batting probably eighth or so um if he bats higher than sick go all in but um eighth is still okay by me so far stacking against helio teyron has not gone well he's uh gone 11 innings across the two starts and has let up just one earned run so maybe dumb for me to keep on going here but i'm going to for today never been accused of not being dumb so we'll try one more time and stack the reds against teyron here teyron has faced the giants in the blue jays two pretty good teams against righties and he completely fooled both of them but it does seem a bit fluky because he's led up a 39 heart rate which is not super super low it's about average he has a 12 strikeout rate with a 5.6 swing and strike rate and a 53 five ball rate so it's expected the array is good at 3.41 but his skill interactive era is 5.51 i tend to believe skill interactive era more in such a small sample and this one says we can stack against him just based on what he's done in those two starts where he's had the great results so the reds are not great by any means they have an 89 wrc plus against righties with a 128 iso but the park very very nice and it is also the warmest park on the slate with the temperature 81 degrees so even though it hasn't worked out yet i'm going to give it one more try stack against teyron here and if it goes if it goes poorly again we can re-evaluate later on and decide if we want to keep on going down this road now we talked about madman claim when he first got called up and if you're looking at the results he's been good the power numbers have not been there he has a 135 iso against righties and just one steal so far but the results have been good and i don't think that's entirely something we should write off and i think there's room for the power to grow especially when he's playing at home mclean is starting to strike out a little bit less so i'm going to keep the faith in what he did in the lower levels you could see that iso see the lack of steals in the majors and not want to pay 34 to dollars for this guy but personally i'm pretty okay with it so mclean hasn't fully clicked yet but i still think is a guy we should feel okay using for tonight in dfs and i think keeping faith in what he did in the miners and buying into that is the right way to go for tonight for the third stack we're going to stack a week against adam wainwright which feels weird but he hasn't looked great since coming back he's up to five starts in his era is 6.15 and he faces a really tough rangers team for tonight so the rangers to me gonna be the third stack behind the fillies and the reds velocity has been an issue for wainwright his cutter is down 1.1 miles per hour from last year his curve balls down 0.8 miles per hour now last time out it was better and it led to more strikeouts for wainwright so that could be a sign that he's figured things out getting healthier and getting better but in that game with the increased velo he let up three barrels the barrel rate against him is currently 13 percent with a 37 heart rate so he's making mistakes and teams are connecting and taking advantage of those mistakes wainwright is a very smart pitcher he may figure that part out and the velo may rise as he gets further removed from the il as we saw in that last start but we haven't seen that lead to great contact suppression as of yet and the rangers are a very good team 118 wrc plus against righties 186 iso both those ranks second on the slate we could get burned if wainwright does turn things around but i think it's worth the gamble here given the rangers quality given the hard contact wainwright is let up given the barrels in that last start i think there is enough here to still feel high about the rangers from his stacking perspective for tonight and does help they just got mitch garver back as well from the il he was absolutely shredding last year before his injury and then came back look great homeward in his first game back with good last night too he's still probably gonna bat lower in the order probably seventh or so but similar to drew ellis i think that's okay without a plus spot in the order because we can feel good about him in terms of what he can do individually i think the offense overall should produce a lot of play and appearances so i still like mitch garver despite the fact he's batting lower in the order very okay with him at $2,800 within rangers stacks for tonight things to watch sticking inside that same game i think the other side there and the cardinals is pretty interesting for stacks too they're facing martin pares who hasn't been as great as he was last year letting up a lot more hard contact this year the cardinals have a 128 w rc plus against lefties living up to expectations in that regard so even with the roof likely closed gnarlington which does downgrade the park factor because it was it's warmer but should be closed that game in general is fun for hitting so full sides of cardinals and rangers pretty fun for tonight the pirates are facing jp sears and as mentioned i don't mind him it's you know more about the a's a's bullpen and it's not like sears is lights out either so you combine sears being an okay picture with a really bad bullpen behind him i think that does make the pirates pretty interesting for stacking very willing to give them a look and would put them behind the cardinals but still on board with him in general for tonight finally the asterisks are facing alex manoa who has had a really rough year so far he has had some more strikeouts recently but still not overwhelming and i think that i'm still skeptical that he's fully turned things around now he's facing the astros and the astros are improving as they get healthier but they've also got a lot of guys who steal and the jays have had a lot of trouble with limiting base runners so far this year so guys who run jeremy pania kyle tucker guys like that they might have more interest uh more appeal for tonight because they're facing the jays so i would say the astros are behind the pirates and the cardinals in terms of the secondary stacks but still a very good stack and one i'd be willing to go to despite hoping alex manoa does turn things around eventually let's finish it things up here with the dingard calls for today the boring one gotta go against wainwright given all the barrels he's allowed and go to at least garcia garcia has crushed righty so far this year just a phenomenal player and again from a dfs perspective if the home runs don't want to be in there you can pay out and get good bang for your buck out of garcia even without a home run so at least garcia has been in a Homer drought recently i think he snaps it for tonight gets back in that column so garcia is the boring home run call for today the fun one let's go spencer steer facing off against teyron teyron you do kind of want to skew towards the lefties and steer is not that but decent power playing great american ballpark puts the ball in the air so his salary is not low you're not like getting a fun home run call from a salary perspective if we're going for that i probably go garver or elis but i do like steer quite a bit so let's go with at least garcia and spencer steer as a home run calls but again garver and elis two fun value plays uh who i would be checking out for tonight if you want to save some salary that's all we got here for today on these solo shot again i think it's a fun slate given all the fun stacking options hopefully andrew abbott is added to the player pool eventually over on fandals we can have one more pitching option at our disposal we'll see how things play out there do not forget to subscribe to the number fire daily fantasy podcast feed to get these podcasts as they go live also check out the solo shot over on the fandal youtube page just subscribe to the fandal youtube page if you like what you hear there and leave us a thumbs up we appreciate all of you on youtube as always if you've got any questions for me i am on twitter at jim sonnis j i m s a n n e s you can also follow the fandal podcast network at fandal podcast want to thank you all for tuning in for today give that to you with your mlb dfs lineups for this monday night we'll talk to you once again tomorrow for a robust tuesday slate this has been the solo shot right here on the fandal podcast network


Hello everyone, this week we're going to be talking about confidence intervals. So just to give you an example right out of the gate, suppose you are trying to determine the mean rent of a two bedroom apartment in your town. You might look in the classified section of those paper, write down several rent values and average all of these rent values together. You would have then obtained what we call a point estimate of the true mean. And what we mean by that is the estimate or the mean that you've calculated does not reflect the actual mean of all apartments from the population in your town. You've only taken a sample of the rent values. So your mean, the average that you've calculated will be a little bit off or sometimes a lot off from the actual value of all of the apartments, all of the rent values in your city combined together. So there is a true mean and our mean that we've calculated will be slightly off. So we want to know how far off are we basically? How close are we to calculating the true mean? Because we want to know what the true mean is or what the probability of the true mean is from ours. We use these calculations a lot in pretty much all studies where we have some sort of survey or we're taking some sample of a population. We want to not only calculate, for example, the mean of the population that we've sampled, but also figure out how far, how confident are we that our value reflects the actual population. And that's really what we're trying to find out. How confident are we that we are correct? And if our confidence interval, we're talking about confidence intervals, if our confidence interval is very big, then that means that there's a huge range where values might lie, which means we're not very accurate, which means we can't be very confident that our guess is true. If our confidence interval is very small, then we have a shorter range in which we can say that our values fall within these particular ranges, and our value, hopefully, is closer to the truth. So today we're talking about confidence intervals, and we can basically use this to calculate how close our point estimates are, because we're always calculating point estimates from the true mean, whenever we can't sample every object in a population. How close are we to the true mean? What's the likelihood that we fall a certain distance from the true mean? So it's very similar to using distributions like we did before. We use sample data to make generalizations about an unknown population. So we sample data, which is a small portion of the population that we're looking into, and we're using this sample data to make some sort of generalization or guess about an unknown population. We want to know some attribute of this population. We call this inferential statistics. So really, so far, we've been doing a lot of descriptive statistics, just describing the data that we have, and now we're actually trying to make generalizations or assign attributes to a population that we don't necessarily know anything about the population. We have to infer what those attributes actually are. So the sample data help us to make an estimate of a population parameter. We want to estimate the population parameter. The point estimate is most likely not the exact value of the population parameter, but close to it. So just like we said before, whenever we're calculating the mean, if we calculate the mean of our sample, well, it's not going to be the exact mean of the entire population. It's just the mean of our sample. But it should be close, right? So if our sample is large enough, then it should reflect quite accurately the mean of the overall population. If our sample size is small, it will be further, it will possibly be further away from the true mean, right? So the point estimate is most likely not the exact value of the population parameter, but very close to it. And after calculating point estimates, we construct interval estimates called confidence intervals. So once we have our point estimates, once we've analyzed our sample and we've made some estimate, for example, the mean of the sample that we have, and we want to say how close is this mean from our sample to the overall population? How closely does it actually relate to each other? We can construct interval estimates called confidence intervals to say, we are confident that our point estimate is within a certain range of the true mean or the true value, the true parameter that we're actually trying to calculate. So first some terminology. We say margin of error, and that's our confidence interval, an amount usually very small that's allowed for a case of miscalculation or change of circumstances. So we have an interval from whatever value we've actually calculated. So let's say we have our point estimate, that's the mean, and we will accept a little bit of error. If we had a graph, it would be to the left and to the right, a very small amount of error that we allow for miscalculation, change of circumstances, or just kind of some unknowns in our measurements that we haven't thought of or something like that. So we allow a little bit of error into our calculation just because our calculation is not going to be perfect. There's going to be a little bit of error pretty much anytime you measure something. Our confidence level is the probability that the value of a parameter falls within a specified range of values. The probability that the value of a parameter falls within a specified range of values, and this should sound familiar because it's essentially what we were doing with normal distribution. So what is the probability that a value falls within a certain range of the mean in a normal distribution? And we can use the normal distribution there to calculate the probability that values will fall within a specified range. Now our confidence level is basically how confident we are that the value will be within a certain range. And normally for real studies, our confidence level is normally greater than 90%, usually something like 90, 95, or 99%, and most of the studies I've seen use 95, but it really depends on what area you're in and what you're measuring. The empirical rule in approximately 95% of the samples, the sample mean will be within two standard deviations of the population mean. Now this should also look familiar, it's essentially the normal distribution, right? So in approximately 95% of the samples, the sample mean will be within two standard deviations of the population mean. So here we go. Calculating confidence intervals, we can calculate confidence interval where the population standard deviation is known. So where we know the population standard deviation, which is not normal in most real studies, but if you do happen to know it, then we take the point estimate, which is the parameter that we're trying to calculate, where we have calculated, and it's our estimate that is hopefully close to the true value. We take our point estimate minus our error bound. And we take our point estimate plus our error bound. Then we get this interval. This should look familiar because we kind of did something like this whenever we took the mean of a sample, we took a mean of a sample and then calculated how many standard deviations it is. Well, in this case, we're saying, okay, we have our point estimate and we are subtracting the error bound and adding the error bound and then we get this range. So in this case on the screen here, we have x equals 10. So this is basically our mean and notice this is a normal distribution here, or it looks normal at least. And then we have this error bound. We've calculated our error bound based on the standard deviation. And then x minus our error bound equals 5 because our error bound is 5, so 10 minus 5 is 5, 10 plus 5 is 15. So this is our confidence interval whenever the population standard deviation is known. In this case, our confidence level is 90 or 90%, but 90% written as a decimal. So in this case, we have 90%. Now notice, if our confidence level was, for example, 95%, then it would be actually a little bit wider. So basically what we're saying here is that we are confident, we are 90% confident, we can say that the true value, the true parameter will fall somewhere within this range. So we have calculated our point estimate is 10, we've counted a point estimate of 10, we can say that the true value will fall somewhere within this range and we are 90% confident about that. Depending on what exactly we're measuring here, plus minus 5 may be really, really horrible or it may be very, very good depending on what we're calculating. So this is a calculating confidence interval where the population standard deviation is known. I won't talk much about that because we don't really see that in nature very much. We don't usually know our population standard deviation. Calculating the confidence interval where the population standard deviation is unknown and this is more like, well, can we say real life? So in this case, we write down the phenomenon that you would like to test. So write down the phenomenon you'd like to test. For example, the average weight of a male student in Hallam University. Now in this case, as soon as I wrote this, I already had problems with this statement because it's a little bit too vague for what I'm used to. In this case, the average weight of a male student in Hallam University. So I would have a couple questions about this statement. For example, how are we calculating weight? Is that with clothes on or shoes on or whatever? Or is it without clothes, without shoes? What is our standard for weight here? How are we weighing things? Is it just what people say they weigh or are we actually going to weigh them? And then I would say male student, which seems very straightforward, but are we actually doing, for example, a genetic test to figure out if they're male? Are we just taking people who claim that they're male? How do we actually find male students in this case? Like what's the standard for male here, really, or what are we basing it on? And student in Hallam University, I also have a problem with because are we taking any student that's in Hallam University right now, which could be potentially high school students as well as college students? Are we taking students who are not enrolled in Hallam, but are college students from another university? So this whole phenomenon that I would like to test is, in my opinion, too vague, but hopefully it works for this example. So basically my point with this is going back, and we will talk about hypothesis testing, but we're essentially making a statement about some population here. And we could be much, much more specific about the population itself. This in my mind is a little bit too vague of a population. So we write down the phenomenon we like to test. In this case, the average weight of a male student in Hallam University, and we select a sample from your chosen population. Now we need to also think about how we select these samples. A truly random sample would be best, but how do we do that random sample? If we stand, for example, in front of the student union, we are likely to get students from maybe certain majors or certain schools. If we stand in front of the library, for example, we're likely to get students from different majors or different kind of sample. We also want to think, in this case, the average weight of a male student in Hallam University, we also don't want to just stand in front of the gym, because if you work out, your muscle is much heavier, right? So very muscular people will weigh, on average, much more than people who don't work out. So where we stand and where we take our sample will really affect our results in this case. So select a sample from your chosen population and be very careful about how you select these samples. You might need to choose several different locations on campus. You might need to choose different times, things like that. Then we calculate the sample mean, right? So just like we did before, from our sample, we calculate the sample mean. In this case, we just add up all the weights and divide by the number of samples. So that's relatively straightforward. Once we have the mean, we need to calculate the sample standard deviation. Remember, we don't know the standard deviation already. But in this case, we assume, actually, we know that the weight should be more or less normal if we have a large enough sample. If we're talking about weights of students, it should be relatively normal. So we calculate the sample standard deviation. So we get the mean, which we've already calculated above. We find the variance, which is the squared differences from the mean. The squared differences from the mean. And we'll talk a little bit more about variance later. So we calculate the variance, and then we take the square root of the variance. And that gives us our sample standard deviation, right? So now we know the value of one standard deviation, right? And we can basically add and subtract that from our calculated mean to get all of our different standard deviations. So for one standard deviation, two standard, etc. So then we need to select a confidence level. So again, how confident do we want to be? What is our level of confidence in our study? Yeah, how confident do we want to be, basically? So we select the confidence level. One standard deviation, remember, is about 68% confident. Two standard deviations is 95% confident. Three standard deviations is 99.7% confident. So if we go back one, notice our confidence level is about 90%. And it looks like, I don't know if you can tell. But it looks like it's a little over one standard deviation, but not quite two standard deviations. Whenever I see this, I see something that's a little bit more than one standard deviation, but not quite two. Well, that completely makes sense in this case because two standard deviations is about 95%. So we had a little bit less than two standard deviations for a confidence level that's 95%. We have three standard deviations that's 99.7%. So in this case, if we're three standard deviations out, we can be very, or we can be quite confident that the actual, the true value, or our true parameter is going to be somewhere within three standard deviations. We can be 99.7% sure or percent confident that we're going to get the true value somewhere within there. If it's two standard deviations, we can be 95% confident that the true value is going to fall within two standard deviations of the mean or the calculated mean. And that's actually pretty high. Most studies that I've seen, like I said, use about two standard deviations are 95% confidence level. So for determining sample size, we want to know, for example, in a sample, well, lots of studies and lots of people ask the question, how do I know how many people to survey? Or how do I know how many samples I need to collect if I'm calculating some value? Well, we have to calculate, based on a couple different things, how many samples we should take. Now before I said kind of the hard and fast rule is you need at least 35 samples to be somewhat significant. We'll talk about significance later. But you need at least 35 samples. But that really depends on the population size. For example, if your population size is five and you get one sample, you know, that's one sample is not great in this case, but it's also not bad because you, as long as because you have a pretty good chance of representing the sample, if there's only five things in it, right? You can't get 35. But in the case of, well, okay, let's just talk about determining the sample size. So determining sample size, we need to know the population size. We need to have an estimate of how big the population size is. So we need to know basically how many, in the case of our prior example, how many males, how many male Hallam University students are there? And I think that's something close to like 5,000 or so. I think that's a little bit over, but we can say that our population size for males in Hallam are probably about 5,000, something like that. We need to know our margin of error, which is our confidence interval, which we talked about calculating before, the interval in which the true value will lie within some probability. And we also need to know our confidence level. So what confidence level do we want to calculate? Basically, with confidence level, the lower the value, the more lenient you are in collecting data. So let's say the lower the confidence that I accept, the larger the range of values end up being. And I'll show you an example of this, I think. Actually, you'll see an example of this whenever you use the calculator in a second. So for determining the sample size, we can use this formula here. And this formula says, basically, sample size is our z-score squared, z-score, and z-score corresponds to our desired confidence level. So if our confidence level, like I said, is normally about 95 or 99, I think the lowest I've actually seen in a real study was 90%. So our confidence level, or our z-score, corresponds to a particular confidence level that we've already chosen. So let's say we chose 95%. That means that our z-score is 1.96%. Again, we'll talk about z-scores a little bit more. It also describes them how to calculate a z-score in the book. But or we can just use this graph and it's just handy right now. So z here is our z-score. So we take z-score squared and then we take our standard deviation times 1 minus the standard deviation. And then e is our margin of error squared. And that gives us the top value of this formula. We take 1 plus z-score squared, standard deviation times 1 minus standard deviation divided by our margin of error squared times the population size. And that will give us our overall sample size. So essentially with population size, margin of error and confidence level, we can calculate how big a sample we need if we know the population size, given some confidence level. So there's also a handy little chart because like I said, this is used in almost every study that's looking at some population. So it's already been done for a confidence level 95% or 99%. You can get these charts online as well. They've already been calculated. We also, well, I'll show you something in a second. So whenever we're determining the sample size, just looking at this, we have our confidence level here is 95% and 99%. We have a margin of error, how much error we're willing to have, like I said, how much error we're accepting in our study. Now notice, and 5% or 2.5% or 1% are actually relatively common margins of error that we're willing to accept depending on what the study actually is. But look at confidence, let's do the strictest one, for example. So confidence level 99%. So we will be extremely certain that or we're extremely confident that the true value is within some range. And our margin of error is only 1%. If our population size is 100, then that means that we have to have a sample size of 99. And why is that? That's because we want to be absolutely sure that the values fall within our estimates and our margin of error is actually very, very low. So we have to get as much data as possible. In this case, we have to get 99 samples to make sure that our confidence level is 99% and our margin of error is 1%. Now, if we accept a 5% margin of error, then we only have to get 87 samples. I say only, we have to get 87 samples out of a population size of 100 if we want a confidence level of 99% and a margin of error of 5%. Now looking over, confidence level of 95% where we are relaxing a little bit how confident we are, which actually means that we're reducing the range basically reducing the range of values that we accept. We're relaxing a little bit how confident we are. And if we're looking at our margin of error of 1% with confidence level of 95% and our population size is 100, then we still need 99 samples. Because confidence level of 95 is still quite high and we basically accept almost no error in our measurement. If we accept not a 5% margin of error, then we only need 80 samples. So you can see how confidence level and margin of error kind of affect each other. Yeah, how the confidence level and margin of error kind of affect each other. If we have a population size of 100, then we need to take actually relatively a lot of samples. If our population size is 5000, we can get the same confidence and the same margin of error by taking fewer samples. So for example, in this case, we have 80%, we need to get 80% of the entire population to get a 5% margin of error if the population size is 100. But if we look at 500, the population size of 500, we only have to take 217 samples to get a confidence level of 95% and have a margin of error of 5%. So in this case, we're taking well over half for a population size 100 and less than half for a population size of 500. So and then if we keep going up, let's see, we keep going up. If the population gets very, very big, so for example, the population of 1 million, then we can still, we only need really 384, a sample of 384 to get a margin of error of 5% and a confidence level of 95%. This is why we can do phone interviews of what seems like a relatively low number of people whenever a population is very large. So for example, if somebody is doing a telephone interview, looking at people in Seoul, and we want to describe something about the people of Seoul, well, we can still get a relatively high confidence level and a relatively low margin of error. Even if we don't interview a lot of people, well, 384 is still going to be a very difficult amount of people to contact and actually have a reply to surveys or something like that. But we can still statistically be relatively confident of our answer even with somewhat kind of low numbers. Now, if we look at confidence level 99% margin of error of 5%, we actually have almost double, well, not quite double, but almost double the amount of samples that we need to have a confidence level that's very high. Now, if we look at our margin of error, you notice that basically, where are we? Yeah, our margin of error, if we want a very low margin of error, then we need a very large sample. And most surveys that I see that are sampling entire populations are not looking for a 1% margin of error, so they don't sample nearly close to, for example, in this case, 10,000 people for a population of 1 million. So just keep this in mind, there are these charts that can help you a little bit, but usually we're collecting around, for example, population size of 100, and a lot of people are getting 30 samples from a population size of 100. And if that's the only samples you're getting, well, either your margin of error is going to be very large or your confidence error is going to be very small. So if your confidence error is very small and your margin of error is also small, then basically that says that you're not very confident about your prediction. If your margin of error is very large, but your confidence error is also very large, then that says that you are accepting a huge range of error. You know that it's going to be somewhere in here. So basically, in that case, they wouldn't have any predictive power. What this confidence level and margin of error does is say, we're going to reduce our level of error to a specified, very relatively small range. And I'm very confident that my true value is going to fall within that range. If you start to increase that range, then you're not really making any predictions. You're just saying, well, our value is going to be somewhere within there and has no predictive power at that point. So real studies tend to use confidence levels about 95% and margins of error maximum 5%. So all of this is a little bit can be complicated, especially if we want to figure out, for example, sample size. So I'd like you to go look at this sample size calculator at SurveyMonkey.com. It's a very nice calculator.  So it will do all these calculations for you. So that's it. This is Confidence Intervals and we will be working with them more this week. So that's it. Thank you very much.


Inflation is still higher than we want it to be and it continues to affect us all. Companies have put up prices as their expenses have risen and workers are pushing for higher wages to cope with the cost of living. For the next three episodes we've got something special for you on offer. The ECB podcast summer school. In this mini-series we'll take a deep dive into the nature of the inflation that we're currently seeing. We'll take a closer look at the main way of fighting it, our key interest rates, and we'll unpack the role of banks in passing on the impact of our monetary policy to companies, to you and to me. So welcome to the ECB podcast summer school, helping you understand what's going on in the economy and central banking. My name is Katie Ranger. I'm very happy to have ECB chief economist and professor Philip Arlene here in the studio to break this all down for us. Philip welcome back to the podcast. It's a pleasure to be here. Now inflation has been above our target for almost two years. We know what's caused this, a complex combination of factors that include pandemic related supply chain bottlenecks, Russia's war in Ukraine and the related energy crisis, and a post-pandemic rebound in demand. We've spoken about this several times on this podcast. Now we get data on inflation every month. The latest set of figures showed the inflation rate falling to the lowest figure since the beginning of 2022, which is certainly good news. But all in all, it still seems uncertain to many people where prices are going to go. Philip what is making it trickier to understand how inflation will develop? Well I think you've answered your own question in your introduction. Just look back at these years. I joined the European Central Bank in summer 2019 and early 2020 we had to start the pandemic. And the pandemic I should emphasize really can be thought of as having two phases. One we all remember those horrible months of lockdown. So in those in that period if you like people were at home there was very little economic activity and in the initial months of the pandemic in fact there was downward pressure on inflation because the economy was so inactive. Then fantastically the vaccines were invented and really from 2021 onwards there was a big resurgence in the world economy. And that's a lot of dislocation. Put the world economy into sleep. Then reactivators. There's a lot of dislocation. A bit like what happened previously with World War One and World War Two. The dislocation of that did lead at the end of those episodes to an inflation research. You can't just flip the switch basically. Right. So that if you like is ongoing. Although the pandemic is behind us from a public health point of view. Realistically it does take time to settle down. And then as you mentioned especially here in Europe from February 2022 onwards after Russia's unjustified invasion of Ukraine that added a spectacular surge in energy prices. Now that has reversed the amazing efforts of everyone in Europe to cut down on use of gas, cut back on energy, led to a spectacular reversion in energy prices in recent months. But even that still takes time to work through the economy. So these are unusual episodes. Economics works by looking at the past and saying okay based on past patterns how quickly would inflation come back to 2%. And essentially the work of the ECB staff is to work out given these very unusual factors what is a reasonable timeline. And what we have in our projections is inflation to come down quite a lot later this year. But getting all the way back to our 2% targets it's essentially scheduled more or less for 2025. But again that assumes also that we deliver on our commitment. We deliver on our commitment in terms of making sure that interest rates will help that process. Okay so we're dealing with a very unusual situation here basically. Now usually policymakers look at what we call underlying inflation to understand how inflation will progress in the future and indeed fill it back on our last podcast together in June. We talked about how underlying inflation is all about trying to assess what will have a lasting impact on prices once the dust of any shocks like the ones we've been seeing settles. You essentially remove the volatile prices of things like energy and food to get a better idea of price developments. Now the ECB has just published a box as part of its economic bulletin that looks at how useful a measure underlying inflation is these days and listeners as always you'll be able to find the link in the show notes. What's your view on this? Is underlying inflation still a useful measurement in today's rather atypical environment? So I think the way you phrase it at the end is the critical issue. The concept of underlying inflation is permanently useful. We are always searching for as you say when the dust settles. When the dust settles what is the underlying pattern of inflation because that is really from a monetary policy point of view what we need to bring back to 2%. Because of the pandemic, because of bottlenecks, the unusual surge in demand in the last year for say for example for tourism, working out these measures is more difficult than normal. But again that's where the work is. The measures do have to be modified. That's for sure. To take account of this the very large change in energy prices and if you like the rotation in the pandemic where initially people could not go out but they could still order online goods to their home. So home gym if you're working from home office, new business equipment whereas more recently people are definitely have returned to tourism, entertainment and so on. So what I would say is conceptually we're very focused on underlying inflation but it's more work than normal to work out where that is. So what I would say is we have mixed messages and this is fundamentally why monetary policy at the moment is tricky. In one direction as we already talked about they're very rapid for the energy prices. We are confident we'll bring down costs across the economy. Every sector uses energy, lower energy prices and the easier supply chains now with the bottlenecks over will push down inflation over time. On the other hand what's happening this year is the high inflation we saw last year. I remember inflation was a 10% in the area last autumn. It's basically pushing up wages this year. And so what we say is that the domestic component of inflation coming from rising wages and also firms looking to rebuild profits is pushing up underlying inflation. So this force is working in opposite directions and this is why we're very data dependent. You know as we go into the autumn we're going to be hunting for clues looking at the incoming data essentially to see which of these forces is getting stronger which of these forces is getting weaker. Now you mentioned wages and profits there are two things that are affecting underlying inflation. I want to zoom in a little bit on those. What kind of things are pushing firms to put up their profits and what effect will that have on wages as well and perhaps also what how do people perceive these prices? At what point do they become unwilling to accept higher prices as well? What kind of considerations going to all that? So I think it's very interesting and the ECB in the last few years has developed if you like what's called a consumer expectation survey. So now we have a multi-country survey every month and it's super interesting because I think maybe a dimension of inflation that we all need to think about is when do people just accept it? Well prices are going up because there's inflation in general versus prices are going up because this firm is trying to exploit me. And in our recent survey we exactly asked this question and essentially so far people understand that the dominant reason inflation went up so quickly was rising costs especially energy costs. Next on their list was indeed profits and bottom of their list was indeed wages. And I think that was a kind of a good description of what happened last year. But that is changing over time. So now from that kind of cost structure point of view firms can no longer claim that energy prices are going up but they can now say well that's easing under the hand my pay bill is going up. Because people are asking for higher wages to recoup the cost. And that is true to some extent. But the other leg of profitability has basically been the level of demand. Now Europe has not had a general booming economy but what is true is at the level of a sector for example tourism. Last year after the pandemic there's a surge to book holidays. There's a surge to go out to restaurants. And at that point there was still supply was limited. Many hotels closed down. Many airlines reduced the number of planes that they flew. And so in that circumstance with a big increase in demand limited supply firms were definitely tempted to raise prices quite a bit. And we saw that in rising profits. So this year we do think that demand is normalizing. It's not back to normal. People this summer as we speak it's still a pretty strong tourism summer in Europe. By the way not just Europeans but clearly a lot of people from America especially interested in taking a holiday in Europe. So we don't think that the profit dynamic is over. But we are emphasizing this is something we need to see. Not all of the wage increases can be passed on to consumers. And so our assessment that inflation will come down does rely on a calculation that having had quite a lot of profitability last year that this year and especially going into next year firms will just have to live with lower profits. And listeners will be sure to put a link to that very interesting consumer expectations survey that Philip mentioned in the show notes. Now a lot of the factors that we've talked about that are affecting inflation at the moment actually lie beyond the control of a central bank. Things like energy prices and Russia's war in Ukraine. So my question is what can monetary policy do in this environment Philip. So the ECB has always been crystal clear that basically our promise to Europe is that we will make sure inflation comes back to our two percent target in the medium term. So we don't promise that every month inflation will be a two percent because as you say major shocks can happen. The net effect of the pandemic with the reopening phase was the drive up inflation. The net effect of the Russia's invasion of Ukraine was the drive up inflation. And there's been other global factors. And let me mention something that's quite topical now is also with the very hot summer in Europe. And all the talk about El Nino this autumn food prices are also being pushed up by by by weather. So inflation can go up and it can go down for global factors. However what we cannot tolerate and this is where the ECB comes into play is we cannot sit passively and accept that inflation will remain too high or too low. We have to act. And so it's not about where the inflation came from. So in my assessment the inflation mostly came from the factors you listed. But it's a job of the European Central Bank to make sure inflation doesn't stay high that it's guided back towards our two percent target in a timely manner. Well before we wrap this topic up we always ask our guests for a hot tip linked to what we've been discussing today so that our listeners can learn even more about it. Philip do you have something to share. So the phrase hot tip on such a somber topic such as inflation. Let me say that I do have a suggestion. So here in Frankfurt in the Historical Museum there's an exhibition about it. It's called an exhibition about hyperinflation. But in fact more generally it's a very interesting and very visually persuasive history of inflation going back many centuries and all the way to the present day. So for people here in Frankfurt or if you're planning a visit this exhibition runs until the 10th of September. But maybe online you can also find some resources at the Frankfurt Historical Museum. That's a great hot tip. I've been meaning to go and it's on my list of things to do over the summer. So thank you for that Philip. Well that brings us to the end of this episode. I want to thank ECB Chief Economist Philip Arlayne for digging deeper into this really important subject for us. Listeners be sure to check out the show notes for more on this topic. You've been listening to the ECB podcast Summer School with Katie Ranger. If you like what you've heard please subscribe and leave us a review. Until next time. Thanks for listening.