Perfect, perfect. Are you ready to take some action in this session? Or are you already tired? You're tired? But, but what? Okay, okay, that's very good. It depends on the action, right?  So, hi everybody, my name is Bernd Schiffer and my accent, well, I'm not from here, obviously. So, I came over from Australia and that's not an Australian accent. So, if you, well, we don't have any experience here right now. So, I can't, not really, you don't count. It's not, I do don't really count, okay? But that's a German accent. So, I'm living in Australia, I'm coming originally from Germany and my name, Bernd, is like, think of a bear, like bear with N-D at the end and then you have the correct pronunciation, okay? Bernd, doesn't make sense? V-E-R-N-D, yeah, yeah, that's right. Good, awesome. Now, I'm an athletic coach trainer, a consultant, and I'm freelancing in Melbourne at the moment. But actually, when I came over like 15 months ago, like over from Germany to Australia, that was a big move. And you know, you have all kinds of funny ideas like, well, do I want to do that athletic coaching thing for like longer, you know? Or should I do something completely different, okay? Because, well, it's not always fun, not always. Most of the time, we're not always, okay? Now, what I thought actually was, like, I could do that. I could open my very own cinema, you know? Like, I really like movies and I really like to have nice cinema, like the small ones, and I really, I could make a good cinema, like I really could create a good cinema, like a new business, you know? Now, here's the thing, could you please think of my cinema? I describe it and please take a seat in that cinema and watch the movie, okay? Now, I describe that cinema a little bit, okay? Now, it's very cozy. Actually, that's the name, cozy cinema, yeah? It could be a brand, but I'm just focusing on one cinema. Cozy cinema. It's cozy because it has like lounge sofa, yeah? So you can have a single sofa or like a double sofa with your partner, yeah? So, you're like, you really could relax and it has like that, those spectral leg rest, yeah? So, like, you could really relax on those things. You have a waiter coming before every movie and actually asking if you want something and then you can order that and during the movie he or she would come back. So, you can have a party during the movie, but still, you would be have like, whatever you want to drink, right? Or eat, eat, eat during the movie. The kinds of movies, like I'm a big fan of all the cool movies, yeah? So, for example, I would show Magic, only the first start, of course, yeah?  And for the ladies, maybe, I don't know, I'm not very, but I appreciate that. So, Titanic, maybe, yeah? I don't know, yeah? But, yeah, or, well, you guys, maybe, like a Bonifant movie, I'm not sure, yeah? So, I'm just checking. So, those were my kind of movie points, okay? So, what do you think of that cinema? I actually would like to have some feedback of you about your experience in that cinema, that experience you have right now, like when you think of that cinema, okay? And I want to, like, I read a very good book, that book here, it's called The Ultimate Question, by Fred Weichel, he's a US guy with German ancestry. So, that's why the name, yeah? I wrote that, I read that book, and this is actually what I want you to do right now. So, on a zero to 10 scale, zero to 10 scale, how likely is it that you would recommend my cinema to a friend or colleague of yours? So, you just were in that cinema. So, on that scale, with zero, it's very unlikely, yeah? So, and 10 is very likely that you would recommend this cozy cinema, yeah? And now, give me that number. Put that wave text, put that on a sticky note, yeah? And then answer, please, that second question. So, first question, just one number. Second question, why is that? Why did you give me exactly that number? You only have, you only have, like, limited space, deliberately, okay? So, just maybe one or two points, yeah? But why did you give me that number? Could you do that right now, please? Here are posters, there are pens, yeah? It's over now. Now, you're outside, and now... Now, if you've written down something, please put it on that wall over there. See that? Could you buy tickets? I don't know. Well, what was your experience with that? Just put it up here. Now, and when you're just at the front, could you please also sort it a little bit? So, put the tens more to the right, and the zeros more to the left, and some kind of continuum in between? That would be awesome. It's an eight? Ah, okay, great. Nice. Okay, thank you very much. Awesome, let's see, good service. Yes, I really wanna have a good service, that's very good. So, Colby and In-North, you're outside, cinemas only, is that? Yeah. Yes? Very nice, very nice. So, let's see, I think it will be good, but I'm not sure, yes, very nice. Oh, it's three-centers, ah, I really, is that, was that yours? Yeah, it was three-centers? Yeah, it was. Let's talk about that a little bit like Turner, like, let's keep that, not now, but I'll come back to you later, okay? So, and when you have a chance, maybe to see now, or maybe I should make some crawl or something, yeah, but, okay, okay, yeah, thank you, that's awesome. Okay, take a seat, thank you very much for that feedback so far. Now, this is the, you're in the middle of the so-called net promoter system, and Fred Reichel actually describes the question, the ultimate, the ultimate question, like this, on a zero to 10 scale, how likely is it that you would recommend us, or this product, or this service, or this rent, to a friend or colleague, and I also instruct the companies to ask at least one follow-up question, for example, what is the primary reason for your score, okay? That's very formal, I think, yeah, so if you just ask, so why is that? Why did you put that number, okay? Now, the reason of, because of that, like the net promoter system is, this is, is it, so, on that scale, you have different, well, categories, the one category here is the seventh and the eighth, yeah? We have the seventh, we have a lot of eighths over there, so we call them the neutral, yeah? And you can see them as the satisfied customer. It's pretty good if you have seven and eight, yeah? But they're satisfied, and satisfied people, if you ask satisfied people, so how was that, let that experience, let product whatever, they will answer you and will say, yes, it was pretty good, and I can recommend it, okay?  That would be, that would be this category, the promoter, the ninth and 10th, okay? This is actually what you want to have as a customer. You want to delight them, okay? So, like, really, wow, what an experience, how awesome was that, okay? And then, of course, we have, and you see that, yeah, the, the huge range, the left over here, which is actually, who's are the detractors? They would actually, they would, they would warn everyone without asking, without being asked, okay? So they would actually go to a party with, with, that's the case, like, and everyone would say, what, what, what's up, you're totally pissed, yeah, I'm totally pissed because of that cinema, that was lousy, it was too expensive, and, and the waiter, like, ah, yeah, yeah? Something like that, okay? Those are the detractors that did satisfy customers. Okay, they're very unusual, okay? Yeah? I don't know what to make out of that, so. Okay, okay? Okay, fair enough. Now you have those three categories, and I ask you to put them, like, a little bit in relation to each other, yeah? So you actually can see them here. So we have at least, does it do that every time? Ah, that hurts. We have two detractors, oh, I'm awake. We have, we have satisfied people here, those two comments, and we have, we have the promoters over there, okay? You clearly can identify them. Do you see that? All of you? Yeah? What we did, what you did actually for me, thank you very much for that, you categorized them, and that's the second step you take when you actually ask for feedback, okay? So first you ask for feedback, those two questions, you put them on, you can have them put them on sticky notes, you can use all kinds of tools out there where you can do that electronically if you want to do that. You can easily set something up with the survey formula, yeah? The form, something like that, so. And afterwards you receive that feedback, categorize it in the detractors, and the satisfied, and in the promoters, okay? Now, why do we want to do that? The goal is, with every business, the goal is to actually don't have any detractors, and actually transform the detractors into new tools, at least. And when you've done that, well, try to transform all the new tools into satisfy, it's a delighting customer. Not only satisfy the enlightened customer, that's the goal, okay? Now, why is that the goal?  You wanna have loyal customers, that's the idea.  So they provide value to feedback, and they cost less to serve and are less price sensitive. Meaning, less price sensitive, you can actually, like they don't mind higher fees, okay? Yeah, or a little higher price, doesn't make them, yeah? This is what Fred Weichen found out through research, so, and well, there's a huge correlation between loyal customers and the recommendation. If you recommend something, you have the most value out of that product. The value generation was at its peak at that point, okay? So, and the thing is, when you have loyal customers, when you have recommended, there's a clear correlation between the net promoters, like, here are less net promoters, here are more net promoters, yeah, so. And this is the growth of an organization, yeah, so. Meaning, our L is not so good as MSN or Earthlink. That was back in 2002, 2003, yeah, we found that out, yeah, so, pretty odd, but still, that article here is very good, yeah, so. Well, and that's actually what he found out. There's a huge correlation between the net promoter, what the net promoter system actually provides, yeah, so. And the healthiness of a business, doesn't make sense, yeah? So, what, on the X axis is the net promoter. So, how many promoters do you have in your business? Like 60%, for example, yeah? Well, I come back to that number later, yeah, but you can have actually negative promoters, meaning you have more detractors than promoters. But still, you can have that, okay?  Is that okay? Now, here's the thing, you don't want to have detractors. Detractors, you can have a lot of promoters, only one detractor, only one guy who's closing out these days, for example, via Twitter, well, that was a bad experience. I really hate that cinema, okay? Only one guy with 10,000 followers, and well, pretty much a lot of them are done, okay? Now, how can you fit our people with bad profit? I would give you a few examples of bad profit. So, number one, roaming fees. Oh, God, exactly. And it's getting better at the moment, but still, it's so expensive to have your mobile outside of the country where you're living in, and you have to pay roaming fees. And there's no good reason why I have to pay one euro for one text message. Technically, there's no reason for that, for that fee, okay, roaming fees. One example of bad profit. The other one, for example, gym subscription, you know that, like you desperately want to lose weight or want to get fitter or something, okay?  And you're caught in some kind of subscription net here, okay? Now, and Martin Paola told that at the beginning of the conference, the dark patterns of UI, where you actually, where you, for example, sign up, you want to have a newsletter, and it actually says, well, and you also sign up for these 10 promotional activities, yeah? So, in a very tiny font below that, yeah? And you didn't know about that, so dark patterns. Now, these are all examples of bad profit, and bad profit is you extract value so that you can make money out of it. You extract value from the customer, but the whole idea of agile is we want to build value, we want to create value for the customer, okay? So, whenever you extract value from the customer, you generate so-called bad profit, and you don't want to do that, because, well, the detractors. No, you have the question with where comes the minus one. So, NTS stands for Net Promoter System, but it also stands for Net Promoter Score, and that is, well, the score is very famous. Well, whenever you have something you can measure, it's very famous, and a warning out there, the score is not the system. The system is much more and much more important than the score, but still, where comes the score, okay? And this is actually the score, the NTScore, so, well, NTScore, okay? Well, don't take a photo yet, there's more, wait. Number one, you count the promoters, the neutrals, and the detractors, okay? Can you do that actually for this? Yes? Yes? Okay, perfect. So, wait, wait a sec, so, you count, you counted it? Yes, sir. Now, you have this, you subtract the detractors from the promoters. From the monthly, and then the detractors. Yes? Yeah? So, what is the number here? Two. Two, awesome, perfect. Now, number three, divide the result by the number of all votes. 13, so, two divided by 13 is what? Something with seven, eight, seven or eight, is that it? Yeah? Somehow, okay? So, now, multiply the result with 100. 31? 14 percent? No, that's 13 overall, yeah? So, I have, yeah, so, did you do that? 15 percent. 15 percent, thank you very much. So, and now I have the percent, and you got a number between minus 100 and plus 100, and that's the NP score, okay? We don't put the percentage in there, yes, so, it's just that you have, the NP score would be 15, okay? Exactly, exactly. If it were the other way around, you had four detractors and only two promoters, it would be minus 50, right? Yeah? But it can't be more than minus 100.  It can happen, but very rarely, okay? Now, that's the NP score. Now, in the real world, lots of companies using it for a lot of different ways, yeah, so, and I don't want to look at all of them, I'm interested how you can use that in an agile environment, so you know the at how many tests do, yeah, so, and well, for example, with the customer collaboration, it helps, it can help like a lot, like really, okay? You can actually find out if you have working software here, so the responding to change is actually very good, so, and the interactions are very good, for example, well, I would never think of like how the waiter would move through the cinema, yeah, so, yeah, just one example. So before you build the teacher, you could actually like describe it, so, and ask people those two questions, and get to think about it, okay? And the don't, it never underestimates the power of imagination of people, never, it's all, it's really all. So, number two, when it is responsive, yeah, directly after the feature is used, this is called the bottom-up approach, or also the transactional approach, meaning, for example, you're at the website and you just haven't, like, you used an online app or something, okay? You just accomplish a task, for example, yeah, so, and then suddenly, like, there's a small pop-up, and it just has those two questions, so how likely is it that you would recommend this feature to a friend or colleague of yours, so, and why is that? With a pretext key, so, you could do that, so, and, well, every one out of 10, well, maybe every one out of 100 would actually answer that, so, and you get feedback. Depending how many people are using your service, you get a lot of feedback in a very short amount of time, yeah, after every transaction, maybe, okay? So, and, or you could do it for a period of three, well, that works, periodically, while the features are used. So, for example, every month, or every three months, or every, whatever you wanna, like, how you're interested in that feedback, you ask all your customers, so, what was your overall experience? Like, how likely is it that you would recommend this product, this service, this, whatever it is you wanna speak feedback about, yeah, so, how likely is it that you would recommend that? Doesn't make sense, those three approaches, yeah? Now, I will give you an example, so, I'm, like, I told you, I'm an, I'm an, as a coach, I'm an, as a trainer, so, and every time I do my training, I use that, that thing here, so, and I just wanted, I didn't wanna bring, like, like, an IT example, I wanna bring, like, an example, I actually did in my business, okay, so. Now, this is what I do at the end of every training. I show a, a, a, a, a, a, flight like that, yeah, so, and actually, that's how likely is it you would recommend this training class to a friend or colleague, okay? Again, you see that, that scale here, so, and the question why after Remote, and I only read it out loud and the only thing I add usually is, well, given that you have a friend or colleague, okay, because I almost, when I don't do that, I always get at least one poster that's zero. I don't have friends who would be interested in that. And I say, ah, imaginary friends, virtual, OK? Yes, sir. OK, that's why I do that, sir. Afterwards, and again, I just provide posters and pens. Yeah, so, well, I do that all the time in my training. And I get the feedback here, OK? So you see all the numbers here? That was actually a very good training class. Yes, sir. And yeah, sometimes it's feedback, well, you're a good mate with an aide. Well, I think that's not what I'm looking for, OK? It's still OK, yeah? So, and sometimes I get really, really, really thorough feedback, see how much they write here, OK? They only have a minute to do that, OK? Still at the end, so they give me that feedback, OK? Now, when I have that feedback, I turn it into something like that, OK? So I have the NTS calculator here at the top. Well, that's the headline. And then I actually write down all the feedback, yes, sir. And create a chart like that, like an overview like that. And, well, and for example, there was one thing about that. The 59 minutes from was the best lot of practical fun and very eye-opening and so on, so. And the feedback for me is, wow, that game, I really should emphasize that. Like, maybe I can do more of that game, or it can improve it somehow, something like that, OK? So I'm not only looking for bad feedback, but also the good feedback actually helps me to preserve what I already have, yeah? Now, what I also do is I create an overview for my clients. So for example, if I do more than one training for a client, I can see how is the development over time, OK? This is actually a real world example here, yeah? That is the development over time of one of my training, OK? Now, and with all those sheets, I actually tend it, don't greet that all. It's just an overview of what I send to my clients, and they get a PDF, which looks like that, OK? So I have the overview of the one training class. I put it in relation to all the other training classes, and I actually explain or help them to understand all the numbers, OK? They love it, they really do. Not only I use it, but they, like my clients, use it too. So actually, it's a good idea to actually send, well, for example, ask you about your feedback after, well, maybe you come back, yeah? And I would actually ask you again, and if I can put that into relation, how you actually have a development in time over your experience, so I would bet you would find that at least interesting, that's for that one, yeah? Yes? Yes, yes, this one. Well, that one was just not so good on that one, yeah? It can happen, I'm not a robot. I do sometimes have my bad moments, and it's not really bad, it's just not so good as there, OK? Yeah? Great, now what I do internally, I create one larger table, where actually, well, that is one of the training classes I do, yeah, and, well, I set the number of scores 25, so I have 25 feedback at the beginning of last year, like May last year when I actually started working as a freelancer in Australia, so, and I keep them, I put every single score I get for that training class in one table here, and I look for patterns all the time. So for example, if one says, well, the 59 minutes one was good, I say, yes, OK, so one was actually pleased about that. If I get another one, that's a huge indicator for me that I'm up to something here, OK?  I really work with that feedback. Now, what I also do is, so these are my products, that's a made-up table, so, but still, there's the other overview, so, and then I have other products, other services I provide, so, and actually compare the NTS over the last three months. Now, the thing is, when I see something like that, so, I actually would take care of this one the most, because this is actually the one where I actually can delight my customers, yeah? Well, and I would take care of this, because this is compared to this, not very good. But I really should, I should really, like, have a disruptive change in there, yeah? So, this tells me actually, this overview, in terms of, so, where should I take care of? Which product should I take care of? OK? Just, again, my business, yeah, so, totally, the whole NTS, that system, totally independent of the domain, you can use it for all kinds of things, and I will give you more examples of what you can use. Yes? Yes? Well, I don't, yeah, yeah? It would show in the feedback, I'm pretty sure. So, I don't have that experience? Well, and that could be, yes. And that could mean, well, I do every time the same thing, and they get bored, and I should do something different, yeah? Right? That would be the feedback, and then I can react to that feedback. Sorry, yeah? I don't understand what you mean. I don't understand what you mean. Sorry, yeah? I don't understand what you mean. You can't understand what I mean? It's not going to show you the feedback. Yes? It's going to be bored. Yes? But the numbers show that. Usually, ah, OK, usually you have a huge correlation between the numbers and the feedback. If the numbers, if the numbers go down, I see it in the, in the qualitative feedback, too. So, I have a, like, my gut feeling is actually, before I actually calculate the numbers, I read all the feedback, the qualitative feedback, yeah? So, and I already have a gut feeling, like, ah, that was, that, that was not so good. So, and then I get a, like, an NTS of, I don't know, like, lower one, OK? And sometimes I'm, I'm, like, overwhelmed by the qualitative feedback, and I know all that's a high number, OK? So, it actually, it, but only over time I develop my gut feeling. One other. You don't know what you mean, just, yeah? That's the thing that's in the back, my creativity. Yeah? Yes? Yeah, no, I only ask those two questions. That's it. Because I know, when I ask 10 questions, the probability that I won't answer, yeah? So, it's a lot higher than just two questions. Those two questions are actually very easy to answer, OK? Yeah, yeah, yeah, yeah, yeah. And it sounds very similar to whether, it's still based on a recommendation, right? And it wasn't inspired by that, you say? Yeah, right, yeah. I don't know. What do you like about that? Yeah, yeah. I attended a training class, four sheets of A4 paper, like, with very tiny, like, and I was like, no, no, forget it. I just, I looked at the, like, how satisfied are you? And I just made my cross, and that's it, so. You're looking at it as a, yeah, yeah, yeah. It just, like, how many, like, Martin Foller mentioned that too in the keynote, like, Dartenstrasse on type, do you remember that? Like, be fair with what you collect from people. So, this is actually, like, you don't need more actually than that. But the thing is, well, one thing, I would actually ask to put, like, in my training, I ask, can you please put your initials on the postage if you are okay with me contacting you afterwards to have some follow-up discussion? So, for example, if you were doing that, yeah, so, I could actually call you, yeah, so, and actually talk to you about, like, well, please help me to understand your feedback. And I do that with at least all my disrespecters, yeah? Okay, now, here are a few other examples, yeah? I use that a lot for meetings, okay? Because even half an hour meetings, you can ask that question, how likely is it that you would recommend, well, what, my service may be, like, I call for that meeting as a product owner, like, I have a stake in there, yeah? So, how likely is it that you would recommend a meeting like that, yeah, so? You can do that for all the fun meetings, for example, okay? Or, when I'm a facilitator, it could actually ask, so, how likely is it that you would recommend my facilitation service to others? Okay, so, give me great feedback, okay? Usually, I do it with meetings which are, like, three hours or more, okay? Like, more workshoppy things, okay? That's one thing, the other thing is, there's something, actually, I didn't make that up, it's called an E-NPS, E stands for employee, so, the employee net promoter system, okay? You can easily do that on your own, so that's actually a trademark thing, so, but you can ask your colleague and peers how likely is it that you would recommend me as an employee, as a peer, yeah? So, to a friend or colleague of yours, yeah? So, the thing is, like, how happy are you working with me? Yeah, something like that, so. Now, and the follow up question, of course, where you get a lot of insight in terms of so, what actually do my colleagues see in me, yeah? When they interact with me? Again, very simple, very simple thing, so, and it works, like, and you can have the NTS just for me, that means, like, for the organization, it's more like, well, we could do an NTS program, yeah? And let everyone do that, but actually, if you're not, if you don't wanna roll off the big stone, like, you can ask everyone on your own, personally, okay? My wife did that at REA, actually, so she initiated that there, so, and she got lots of insight and very, very, like, deep discussions with her colleagues, yeah? And you can imagine, if you get feedback from your peers, so, and they give you, like, seven or something. Seven is almost like, whoa, there's the clip, you know? So, and I'm here on seven, like, whoa, why did you, can we talk about that? Can we grab a coffee, yeah? So, and I really would like to learn more about that, yeah? E&TS, now, here are a few more actions, just a little bit to inspire you about the actions, yeah? Yeah, just one is to add a little bit of feedback. Yeah, yeah, oh, the actions, get feedback from attendees about large meetings and workshops, you have that, you can do a retrospective with customer feedback as data, all the data you get, all the, when you have it in that youth list, use that in retrospectives, or in reviews, yeah, when you actually talk about the product, so, it can be useful in both meetings, do a backlog refinement session with customer feedback, okay? Get feedback from practitioners about learning sources, so, for example, in a book club, you could actually ask, like, how likely is it that you will recommend this book, yeah? Identify weak and strong features and products, if you compare the different NTS of the teachers, all the different products, yeah, so you can actually learn from other products, so what went very good at that product, okay? Then we have let every employee contact from most of the detractors, very good, in most organizations, the developers of that product are not in contact with any user of that product, this is a fantastic excuse, if the customers are okay with that, yep, so, to actually contact them and ask them about their experience, hey, you gave us a 10, you must be totally delighted, but what was it, I wanna learn more about that, okay? So, same with the detractors, I wanna learn what they can do better, yeah? And visualize company-wide to customer feedback and see everyone how to read it, yeah? The key word here is open books. We have, get feedback from colleagues about you and the employee, we have that with the NTS, and use the feedback from colleagues to grow, for example, with personal coaching, I did that, I received personal feedback in that format, so, and I actually asked my personal coach, well, can you work with me on this particular point, because I'm really not sure how to take that and how can I deal with that and what do you mean, yeah, so, yeah? It actually helps, so. Now, here are a few myths or truths, do you know that the former Pope, yeah, so, and someone made a nice reference to Star Wars here, yeah? So, well, it's up to you what is truth and what is myth, okay, but this is the game, I want you to show hands and I'll show you a few questions and you please show up hands, what do you think is myth or truth, I will ask you, okay? Now, first one, a product has an NTS of 80, therefore it's certainly market leader, because it's so, wow, so overwhelming, right? So, hands up if you think that's the myth, that's the majority of the night, and who thinks that's truth, right? Okay, so you're familiar with my answer? Actually, it's a myth, so, and I show you with only with one charge why that is actually a myth, because that's you, yeah? You have 80, you have 60, you have whatever for a very high NTS, but it doesn't matter because it's relative, NTS is always relative, you don't have like common ground, okay? Therefore, even if you have a 60 or an 80 or whatever, if your competitors are having a higher NTS, it doesn't matter, okay? So, again, don't focus on the score, the real value is in the qualitative feature. No, the next thing is, two independent trainers, yeah, like me, having an NTS of 20 and 60, the one with the 60 is the better trainer. Do you think that's the myth? Actually, I know it later. Who thinks that's the truth? Mm-hmm. Yeah, yeah. Sir, could be someone 80, yeah? What, why did you do that? Now, here's the thing, if I asked for feedback, I, as a trainer, at the end of the class, and I do it, for example, like, well, you know, like, my bonus is actually, it depends on what my feedback is, so a hint, the 9th and the 10th are very, very important to me, but I don't wanna influence you, it's totally up to you.  So, just take your time right now, okay? Give me feedback, please. If I do that, for example, yeah, so I actually can easily manipulate that score, easily, okay? I don't do that because I wanna, like I said, I just read out that slide, okay? So, because that MPS is just for me to improve, okay? I do it all the time the same, I try at least, yeah? But still, there's a problem with that. This is actually a myth. The reason for that is, well, you can't compare without a common ground, okay? Just because it looks alike doesn't mean those two trainers, okay? Those two trainers are different in their approach how they actually measure that, because they ask for it, okay, and can be different. If you wanna compare someone, you need, like, a neutral zone, like a neutral service provider, and there are dozens out there, like, consultancies usually, you can ask them, so I have these 10 people here, could you please ask them about the MPS? Yeah, so, and they ask them in a very neutral way, they give me the numbers, yeah, so, and I don't interact with that. If I do that with two independent trainers, I actually could compare the numbers. But just like that, no way, okay? But just be careful with comparing those numbers. Last one, ask the ultimate question, people in different regions and countries will answer differently. Do you think that's the myth? Hands up if you think that's the myth. Hands up if you think that's the truth. Nice, okay. It is actually the truth, okay? Now, the reason is, and there's a German saying, it is called andere Länder andere Witten. Yeah, so, that was German, by the way, so, and that means different countries and different behavior, okay, so. And it's totally different, like the MPS I got from here in, not here, in Australia, and it's totally different from what I got in Germany. So there's actually a drop in 10%, yes? Yes, definitely, so, yeah. Sorry? Well, cultures, yeah, but, now I would say it's regional, yeah? So I could compare, like, the MPS in, I gather, actually in Australia, yeah, so. But I couldn't compare them with, for example, Europe. Yeah, so that, yeah, yes, yes. There are interesting blog posts on the internet, like, we need a Dutch NPS system because the Dutch are always so critical, like, criticizing everything, so they say, that was awesome, I give you a seven. Yeah, so, mm, okay, so again, it's relative. Actually, all the car insurance companies in the US have a negative NPS, all of them. No one seems to like them, okay? They don't care, because even when you have an NPS of minus five, for example, you're at the top of the chain, okay? So, and that's the important thing for them, they're market leaders, yes? Fair enough, maybe that's the Indian NPS, yeah? Maybe? Nice, yes, yes, exactly. Very good point, very good point, yeah, thank you, yeah? I, so all the NPS I collected so far was in international environment, let's put it that way, so I actually compare them, yeah, so because it's both the international area, yeah? So, that might be, that might be actually the thing, yeah? So, and I haven't done that for non-advert group so far, so that would be interesting, how that actually would be put, okay? Now, last one, I forgot that one more, that's a good idea, give an employee a new bonus if she brings the product NPS up to whatever you set as a target. Do you think that's a myth? Do you think that's a truth? Okay, that's interesting. Now, here's the thing, it's actually a myth and let me explain that, that's actually the very, very, very ugly thing with NPS in general, okay? Now, there's the NPS score, score, yeah? And the NPS system, I mentioned that before. Now, I'm focusing on the system, not on the score. If you focus on the score, yeah, so it will cause cheating. You can easily, easily cheat that. I gave you that example how I could do that in my training class, okay? You can easily do that with like, yeah, almost all the time. And if you put money behind it, if you attach money to a metric in general, you get all kinds of very, very ugly results, okay? And I've seen that too, we tried that in, like I was in a consultancy before that, we experimented a lot with the NPS and we also did a little bit with that. It was a very, very bad idea, yeah? You get tabloids NPS score, yeah? Like scores, but you don't really grow with that anymore, okay? Now, my, this is another reason why only focusing on the scores is very dangerous. Now this is an app store, that's XKCD by the way, you know that maybe, yeah? That's a tornado guard, it's an app in the app store, okay? Now, and the place allowed alerts, alerts found when there is a tornado warning for your area. Okay, that's a handy app, yeah? So that's very nice, so the rating is four. Wow, and it has four reviews, so that's actually not bad, okay? So I might actually buy that. Now, let have a look at the qualitative feedback. Number one, five stars, good UI, many alert choices. Yeah, okay? So five stars, running great, no crashes. Well, low standard, but still, yeah, so. Oh, five stars, I like how you can set multiple locations, nice, awesome. One star, app did not warn me about tornado, okay? Be careful with the numbers, okay? Don't focus on the numbers, take them as a, well, a guiding system, okay? So again, you can express your gut feeling with that number, yeah, so. But focus on the qualitative feedback. Aim for a win, not a certain score, okay? Now, you knew that that one was coming, right? Like, on a zero to 10 scale, how likely is it that you would recommend this session to a friend or colleague of yours? And number two, why is that? Please, when you go outside, please take a poster, put it on, I don't know, on the door, yeah? When you go outside, put it on the door, so, and answer those two questions, because I don't get feedback, that's very interesting. We're on an agile conference, and they don't use a feedback system. Well, I'm happy to take care of that on my own, yeah? So, which I do at the moment, but this actually is the reason why I did that talk in the first place, where I wanted to do something like that. NTS is out there for over 10 years. It's not new, it's actually very mainstream, okay?  They build something, they ship it, and they wonder what will happen afterwards, and they are, hmm, okay, we don't get to this. This is a simple system, yeah? And I would be happy if you could give me feedback, really appreciate that. Yeah, sure, sure, sure, yeah? Yeah? Yeah? Yeah? Absolutely, that you are the strategist at one thing or another. Yes, sure enough, yeah?  Maybe? I don't know. But at the same time, just imagine, I'm getting a squad of heroes. Mm-hmm. I'm speaking because, you know, all the small of them are Achilles' brothers. Mm-hmm. So, when I say a squad of heroes is also not... Mm-hmm, yeah? Again, it's relative, right? Relative, because everybody is giving me a, what do you know does not have to impact your business? Yeah? Many people here in the world are giving 75% to 25%. Mm-hmm, yeah. So, what do you say? Yeah? What would you say 95%? Yeah? If it is 75%, then you say, thank you so much for all the support that you have. Yes, yeah, good. Thank you very much for the story. I would like to close it here. I would like to respect the time, yeah? So, please meet me, well, and please meet me afterwards, okay? We can talk about that, yeah? Thank you very much for coming here. One more thing. There, where is it? There, slide there, place, then shipper. I will cheat it also, and I put the slides up there, yeah? So, if you want to have that to the F, for example, yeah, for your own business, I'm happy to share. Yeah, just send me an email, so, and, well, don't forget to give me feedback. Thank you very much.


Thank you very much Adrian and thank you for inviting me here as well to Trieste. It's very nice to be here So it's Adrian said I'm working at the European Center So you know all about the European Center now from Frederick from last week, right? So I will keep it short in terms of introduction So I will speak about what we've been doing for the last four or five years on drought forecasting tomorrow we'll talk about flood forecasting as well, but so This is the just a basic outline of the talk today. I will talk a bit about In the season seasonal forecast and then I will concentrate on Speaking to you about how we do this drought forecasting and a little bit more technical terms a little bit about monitoring and forecasting and How we do the merging of our different products as well to create this and I will talk just a little bit on on finals on A short case study of is done on in the Horn of Africa There is a lot more case studies that I have If you're interested in this I'll give you a literature list at the end as well if you want to go to look more into details because I'm going to talk quite generally now about how we do drought forecasting and monitoring But as Andy said in the morning everything has to be Adapted to the local scale if you want to have something useful like indices or threshold something like that So we have some some studies showing how we worked on on local adaptation of the of the forecast as well Just a recap probably covered this last week as well So the European Center we are doing quite a large number of different forecast products So we have the high-resolution forecast on the 10-day lead time We have the ensemble forecast which is issued twice daily 51 members up to 15 days So they they're really interesting thing to you start using now And I guess is what this course is about as well. It's looking at going beyond these two weeks into the monthly 46 two months and then going on to the seasonal Horizon as well, so we have the monthly forecast now up to 46 days which going into this perfectly into this S2S region and then of course we had a seasonal forecast and Then we also have the reanalysis. Did you talk about that? Yes, but last week was your offer of that Yeah, so we I'm going to using this talk focusing on the seasonal forecast and the reanalysis products Tomorrow I will mention a bit more on on how we're starting to use also the monthly forecast So we've had quite a few projects lately looking and Applications of our forecasts and this is just a Summarized table of the forecast we have been using or that the these kind of problems we have been tackling so floods Obviously, that's one of our main things we're doing now in the application group at ECMWF. We're using High-resolution ensemble and the reanalysis forecast, but we're also looking into monthly and seasonal. So those should actually be added to it So we're looking also at fire forecasts droughts We had a project with agent on malaria. So we are also malaria forecasting The project is finished now, but they're still available. We're also looking at other things like wind and cyclone. So we have a large range of Problems that we're trying to tackle on these timescales so droughts This is quite important thing to try to to monitor and forecast the spin as you all know Dr. Scott calls a lot of human disasters in terms of actual people dying from it or Immunitarian crisis in Horn and Africa later just a few years back and the US drought had a Quite costly drought and the the one in California now is supposedly one of the worst in They say in in recorded history, but I'm not sure if that's really it's still ongoing. So we'll see how how How much that will be the impact from it? So there's a lot of different Things that happen when you have a drought and this was a difficult to Sometimes classify the severity of a drought event because it is really in space and time and The impact is also depending on how well a country can cope with the effects of a drought as well So there's a lot of social Sites of this as well. I will not talk about the actual impacts I would talk more of the hazard hazard forecasting of droughts, but that's another very interesting topic as well How to deal with the effects of the drought? So the seasonal forecast then It's a summary of the weather events during a given season and The question here as well if we can use the seasonal forecast to provide an outlook from the evolution of a drought because when you're in a drought you've already Hopefully good monitoring facilities so you can look you can see what's happening But then you want to know when will this drought end or possibly also when if you're in a good position when can you expect a drought to start as well and we have some skill in our seasonal forecasts and on the Left-hand side here. We see some correlation anomaly correlation For three months forecast issued in January over the US So you see that on average we have the the redder or yellow well if you have read you have a very good skill If you have blue we have no skill or so we have a good correlation between the seasonal forecast Issued in time so they have a good correlation in some parts of the world and the right-hand side We have over Africa here. You can see that we have a good correlation in East Africa so we can say that we have some Some idea that we can use our forecast for for these purposes and other parts We we have very little skills so this is an important issue as well as and it mentioned as well We have no when can the systems to be used be useful when can they give some information that it's actually skillful so monitoring is also very important because that's really gives us the the baseline where we are and There's a problem with monitoring often that we don't have enough good data and the data quality is We have to rely on maybe global products if you have local products Precipitation for example or temperature you you can use that but on a global scale We have to rely on on what's available and that's usually not a very very good quality So I will also show an example of how we can overcome this by using our own Weather forecast as a sort of proxy monitoring tool for forecast as well There's quite a lot of Systems as well already in place for for monitoring droughts There's some examples here We have a US drought monitor on the left to be have something called the African drought observatory on the right Which is run by JRC and they typically give this kind of products where you have an outlook or a monitoring product for for droughts and On the bottom here, we have a regional climate outlook for for Eastern Africa as well the Greater Horn Africa and I will come back to this case study as well where we did actually try to mimic the seasonal outlooks that did the That this Region climate outlook are doing as well So that's a good example where we can try to take our own global models and something useful on on the more of a local scale or a regional scale in this case So drought indices Maybe you've heard this already, so I just recap a bit on why we're using indices So the indices a drought is represents an anomaly in terms of the past climate and that can be the observations or models so we're trying to Make something that's easy to understand and comparable in space and time an anomaly is Not the same everywhere. So we're trying to Go away from the anomaly into more of a standardized format. So For example, if you talk about metrolocal drought, which is one that happens when you have a lack of rainfall We have something called the standardized precipitation index and I will explain to you in detail how we do that How many have heard of that by the way? So you're quite familiar with it already. Okay, it would be For you, so I haven't heard it, but I'll just explain a bit about how they works and You can also use something looking at drying spells on on the thresholds for example number of rainy days or Maybe precipitation amounts that are important for vegetation can also be used Now on we also have had radical drought Which means that you have a lot lack of water available for for river discharge and for for watering for irrigation And there's a lot of indices have been developed as well. You have the palma drought survey to index You have a standardized run of index soil moisture anomalies And then you can also look at actual cultural drought when you actually have not enough moisture for a vegetation. So there's a lot of different ways of Sort of standardizing droughts to looking at depending on what the problem you're looking at I will talk about the the SPI today Because we don't have time to cover all these things, but SPI is quite useful because you can use it for for many many purposes as well and it's recommended by WMO to use as sort of the standard drought index So it's quite known and many weather service and stakeholders already know about it So that's that's good things value until they explain it in detail It's only based on monthly precipitation, which also is quite important to stress here as well So we're not taking into consideration ever for transpiration. There are other indices. There will be developed later We should also look at sort of the the ever berateive Effect as well But the good thing is very quite simple to use if there precipitation you can calculate the SPI You you can look at different accumulation periods You can look at three months Six months nine months 12 months depending on what you're interested in as well So what you need is a monthly series of precipitation It can be local observations and or it can be modeled it can be a region average or you can go go into a point average as well or a point Observation as well. What we do need is long and homogeneous series. So at least 30 years of data we need to have And then of course you need to select your your accumulation time So you do sort of a time filter of the of the your time series It's just a submission of running it's running average and then you normalize this into a precipitation distribution So you transform your your anomalies into standard normal distribution With the subdivision of one so We do this standardization for each calendar month typically So all the January's are pulled together or the three months that ends in January for example So when I talk about a January SPI on a three-month basis, this means the precipitation of a December November December January which is issued in January That means that we are taking away the seasonal seasonality of the of the forecast as well or the the monitoring And this is important to remember that some you can have an SPI in a dry season but it doesn't really mean anything because Dry season is not the important season important season is a wet season So you have to also be careful on one when you're looking at it what where you have a good skill you want to capture that the Important months for your for your territory for your organism for your region So the the principle is that you we fit Distribution function you can use a non parametric function for this, but you we are using a gamma function so here you have your Your precipitation Okay, and the the red dots here is are your cumulative precipitation for your point or for your region The black line here is the fitted gamma distribution to those points and then for each As a top button, okay Okay, perfect. Thank you. So The first step is to fit again in a distribution to the to your observations And gamma is gamma distribution is a quite useful used for this and it's Probably the best one if you're looking at the parametric distributions You can also use your just empirical distribution function as well And then you look for each value You find a probability on the fitted the gamma distribution here so you find your your probability of an event happening and Then you look at the inverse normal function to go back and find The normal the normal distribution what that would be in the normal distribution So then you move from from a normal is into a normal distribution And from there we can categorize this From normal distribution this this indices into normed conditions or drought conditions Usually drought conditions are below minus one is one standard deviation, but you can choose basically which Treshold you want to use but minus one is typically and dry drought So as I said before there's no rule really what what accumulation period you want to use it really depends on your your Application if you're looking at the three months of the accumulation. We are looking more interest maybe in soil moisture It could be crop production in rain for the areas If we go into six and twelve months horizon We are maybe more interested in water reservoirs river discharge groundwater things like that So it depends on what your application is really and this is an example where we have looked at the the upper Niger Basin So here you have SPI for three months accumulation and 12 months accumulation and on this plot here I've just pictured the the correlation between the river discharge in September and different accumulation times And you can see that if you go to longer timescales the more red you have the more the better correlation is you have as well So if you want to look at river discharge, you should choose a Index with a higher timescale than just one or two or three months But you can also see here that we are if you're looking at July-August we have no No correlation with with river discharge that can happen that in this case because we have too much Inundated areas and we're not really looking at a hydrological model here. We're just looking at precipitation So here we have something else some other processes that are disturbing the discharge So they can't really capture that the correlation with rain at this time So you should also be careful that this is not the hydrological model This is not this is just looking at the precipitation itself. This is the entire basin. Yeah, but this is just an example This is not to really just showing that you have to be careful when you when you're Yeah, we can be basically we have no correlation between precipitation Accumulation over a longer time period and river discharge But this doesn't mean that we don't have a correlation between precipitation and discharge But those timescales are much smaller or much shorter than we can capture with it with a SPI in this case So this is also just to show that you have to look at You have to see where you have a good skill of your of your SPI use so I mentioned in the beginning here that we Don't always have good data. So I'm going to show some examples of how we can overcome that by what we call a probabilistic monitoring and The reason why we're doing this is that this is a picture of the data series called the GPCC Which is the global precipitation climatology Center and they are gathering precipitation data globally and creating a gridded outlook and or gridded product and here you can see are the number of Stations in this climatology going from 19 1980 up to 2010 and it goes back further as well But this is an example but as you can see here There's been a quite a dramatic drop in the number of stations reported because of many reasons either the stations are not maintained anymore Or they're not reported to be back in anymore. So when you're using a product like a global precipitation or Observational product you have to be very careful to see what actually goes into that product because it can be quite large Deviation and you don't have a homogeneous time series often And we see that when we compare with our own reanalysis product as well That they can be quite a bad correlation between certain climatology products or observational products So we have this huge uncertainty in what we think are the observations and this will affect our Product or our monitoring product or because if we try to build the climatology over 30 years, we will we will get into a problem here So the idea here was to try to see if we can use our own ensemble short-range forecast to generate a probabilistic monthly means of precipitation anomalies Because in our monthly forecast, we it's not perfect. It's not it's still a climb. It's still a model output It's not observations, but we are trying to fact focus on the uncertainty So using the EPS we automatically have uncertainty in the precipitation So here here's what we do normally if we try to do a monitoring Using this GPCC products. This is for a six-month period. So if we here's our our starting date and you would usually have four months of quite quality check data and then they will release sort of a short period of two months where they haven't really quality check the data so you'd have a mixture of two different data sources to create your your initial states and There will be quite uncertain So instead we are using we're taking the what we think is the best guess of the Previous climate, which is this monitoring of the GPCC and there we are concatenating that with our own EPS probabilistic in the probabilistic sense So we're using a short-range forecast with 51 ensemble members and The average over there of a particular amount to create 51 monthly means So we're trying to to shortcut this this uncertainty in the monitoring with our own on the forecast data instead For the GPCC in this case by using the observation the monitoring Yeah, okay. Sorry. That's a good point. So in for this case, we can use a handcast data set Did you talk about handcast last time as well? so for the handcast we have a 20 year handcast of the PS as well as we can use So to create a climatology we use the handcast of the operational product And the handcast is the same model as the operational but run 20 years back in time So we have sort of the climatology of the operational product But that's a very important point. So we need to have the the climatology of the operational product. Yes Absolutely not no, but we what we can do since we have the handcasts we can calibrate the model with the observations as well So we we we have looked at both using our era interim and then the seasonal forecast, but we do get a better Skill if we start off with the mixing of the observations with the forecast So you can what you can do is you can you can calibrate your your I think you will talk me about calibration as well this afternoon So you can you can look at the handcast and then calibrate the errors of the mean of the biases of the of the forecast using the observations Yeah, because What we're doing here is we're looking at a time in in the forecast DPS where we have a spread Which is high enough to capture the the uncertainty in the forecast. So we just taking the 48 The short-range forecast here Just to look at as a proxy for observations. So this is not really a forecast It's just like trying to capture the absurd uncertainty in the In the short in the monitoring part Yes, because the reason for doing this is that we we're still in the sort of monitoring phase We're not really going over to the the forecast phase yet So to capture the uncertainty in the monitoring we are using our own forecast as a proxy for that spread So the spread in the EPS should be similar to the spread in the observations as well or the uncertainty in the Yes, there's there's papers on this as well, which you can look for more details I've I don't have time to go into all the technical use here But we do get since we have the forecast we can look at Is this 18 years back in time and for this particular case we had five ensemble members So we have quite a lot of samples from the from the climatology to use. It's constant It's in this case. We have well this case. We have to use a 20 last year In this case because we only have the the or actually 18 years, but it's not now has been increased to 20 years But you should keep your reference period constant yes Otherwise you introduce if I well We are running a hind cast for the EPS So that's that's something done at our at the incident we have so every Thursday We run 20 years back in time with exactly same depression model and it's been now be extended to 46 days as well, so it's A good product to use. I don't know if you if for research purposes. I'm sure you can Try to use this as well Because it's something that that gives you a climatology back in time as well I said this is this is why we we use this probabilistic monitoring and this is an example from the Horn of Africa So we are comparing here now The spread from the EPS, which is the black line here And then we also had to do sort of an inflation of the actual spread as well. So we did just an variation inflation This is the gray one here the red one is the gpcc products and Yeah, if you compare them you can see that we are most of the time within the range of the of the gpcc It's not always the case we are but but in this case when we have this kind of probabilistic monitoring We can use that also in a probabilistic forecasting mode as well So we can combine the probabilistic monitoring with probabilistic forecasting So then you get into an example here from from from Africa where we can See if I have the Yes here. So here you have Here you have your monitoring period and then you combine that with a seasonal forecast. So you have to If you want to look at a 12 month of accumulation You have a mixture observation and forecast and if you have and the longer you go to Well, it depends on the lead time as well. So if you're looking at a 12-month lead time On a certain verification date You will have a mixture of observation and forecast depending on how long you are as well And here we are doing a bias correction of the seasonal forecast as well So we are looking at the the climatology of the feet of the Of the of the seasonal forecast to correct for the biases. Yes. So this We can you can either do quite simple bias correction just of the mean, but we also have done a Quantile matching of their of their forecast as well And that that does improve this field quite quite quite likely policies in the forecast Yeah No, this is not not here, but we have done that as well and I'll give you some examples of that as well I'm not gonna Because I didn't have time to go into that in detail here, but we have done different kinds of Bias corrections to look at how that can impact the results as well And here's here's what we look like in real time when you're doing a forecast so here's for the for a limpopo basin here in the South Africa and Here's just an example of a drought that developed in or in 91 to 92 the The gray here is the climatology or sorry that the the black lines here Is the actual climatology forecast just taking The pre historical observations using them as the forecast that is sort of like our benchmark forecast the blue one here is the seasonal forecast and the red one here is the verification of the This is a gpcp and the purple one is our own seasonal or reanalysis they are into him And you can see here just by looking at the drought of evaluation on this two Observational products that we're going from a fork drought the start sort of in October 91 and then deepened into February in March 92 and if we're using our our seasonal forecast, which is the blue one We would actually miss the uncertainty this case We wouldn't we would be not with a lead time of three to four months. We would not be able to capture this event With the climatology we would be in part with the climatology But moving closer to the event we can actually pick up some signal that the drought is emerging So we did we can't pick up the signal with a three monthly time And moving in forward into the time here We can see that we can actually capture the depth of the of the drought better than the climatology So we do have some skill here, but it's not perfect as well It's a lot of things that can be done to improve the forecast, but just an example of how you can use the the forecast for Drought monitoring and forecasting as well and I'll skip this one and you go to An example of a regional Project that we did in the Horn of Africa they produce before the rainy season they produce Consensus climate outlook for September to December and this is based on observations and a bit of model output as well So they can they Try to do a situation where they can say whether you're above normal or if you you expect to be near normal or below normal So they produce this kind of outlook maps and for our forecast we can actually See that we can We can do precipitation forecast for this period as well. So this is our forecast going into time and If we try to do sort of an SPI type of product We can create a similar kind of probabilities of being above normal or normal or below normal so we have defined here sort of same probability Quant or quantizers they have in the outlook form and in the September December precipitation In this in 2010 it was associated with the La Nina event. So we can actually say that We can't forecast the can this kind of events for the for this rainy season We're quite long lead time as well and here's another example where we try to forecast the March and May forecast in 2011 and for this particular event We didn't really do really well because we we can't really capture that the March to May forecast In our product we have a much better skill in the in the autumn rains in the summer rains or spring rains so for Horn of Africa for this case study we found that we we have a Quite good skill in October December season with a connection with El Nino the forecast for March to May season really doesn't have any skill at all. So We need to evaluate the monitoring in the forecasting for each each region really to say something about the usefulness of the forecast But also knowing that there's no skill is also quite useful because we can try to understand why what are we missing and To avoid that these are used just off the shelf. We need to know exactly when they are useful or not So just some fine remarks here that the forecasting is strongly depend on good quality monitoring in seasonal forecast products As a for monitoring local observations should always be used because we found that that's the really the best product if we use error interim you have to be very very careful to do proper validation of it and It's a short an example here. We can use the short range forecast to generate the probabilistic monitoring tool as well and We can you take advantage of this past forecast the handcast of the seasonal forecast So you can do a robust verification and you can also apply this bias correction method so for to tailor your application for a specific region and The as always the forecast is not better than of SPI is not better than the forecast of precipitation So we have to know whether our presentation forecast or are useful for a specific region as well and I'm not going into other fields like your surface temperature and soil moisture, but this can also be used of course as well In your in your applications, so I'm just focused on on precipitation for this specific case This this talk here. So This is going to be give to you as well after the lecture. This is a Some publications which from this is drawn as well and there's a lot more information on on the case studies There's one case study on them only important. We look at bias correction. We also have some papers we're looking at Seasonal focus in the climate change perspective as well and these papers are found in in hydrology in Earth system sciences. There's a special issue on droughts We should reason 2013. So if you look on Hess online, you can find all this information But I'll give you to you after the lecture as well. So That was a very brief overview of the drought. So welcome to questions


Hi guys, this is Christian from Swim2Fly.com. Today we're going to learn how to swim breaststroke in one lesson of three simple steps. The step number one to learn breaststroke will be to learn the hour breaststroke during five kicks. You have to glide on the surface of the water with your head under your arms kicking correctly. The step number two will be to learn the breaststroke with three kicks, means be able to glide during three kicks on the surface of the water and to pull with your both hands to take a breath after the three kicks gliding. The step number three would be to learn how to swim the gliding breaststroke, means being able to kick and glide with your head under water before pulling to take a breath and kick and glide as far as possible with your head under water before you can pull to breathe again on the surface. Alright my friend, slide, slide, go straight your muscle, push and blow, good.


Okay so now we're going to make our snowman cupcake. So just like before we took our fondant, rolled it out and cut our circle. Make sure you put a little icing on the bottom to get it to stick and get that top ready. The snowman is white so we have a white face that we're going to start with and then we're going to use this mold. So this is a sweater mold. Now these I think you may have to order online but you don't have to do this. I just want to show you what it would look like if you did decide to go a step further and use these or purchase these to use for your cupcakes. You most certainly can use it plain and not use a sweater mold and it's going to be super cute because I made one and it turned out to be pretty good. So to make the hat so you just take your fondant okay and then we're going to place it on the mold and the best way that I have learned to do it is to just take your rolling pin and roll it, press down really hard, give it one roll across like that and then start on the other side and go across pressing down really hard so that the imprint can get on your fondant. You're going to take it and peel it away and it makes a beautiful sweater mold. Isn't that amazing? It doesn't look like it when it's like this but when you put that fondant on there and press it it just looks really pretty. So you're going to make the hat for the snowman. You're going to take your circle cutter and cut out that design like that and let's just put this to the side for now. Now to get the hat going so you're going to take your cutter and cut that circle in half like that and then we're going to place it on top of the cupcake okay. All right before we go any further I want to go ahead and put the smiley face on our on our snowman. Use your tip as a way to put the indentation of the little smile it's already curved so basically all I'm going to do is take it press it and rock it back and forth like that to put a little smile on the cupcake. Now this part of it is also a little round tip to put a little dimple we can just take it and just not put the whole circle in but just kind of press the edge of that circle into the fondant to put a little smile on our on our snowman.


Hello, my name is Anders Reiniger. Some of you might know me, some don't. I'm working for SUSE, as you can see there, for, well, like in eternity, about nearly 20 years now. And I've been involved with Linux even longer than that. So the first Linux version I started off was, I guess, 1.05 or something back in them days, really that old. Anyway. And since then I've been active in various things regarding Linux. Recently I've been involved with storage and NVMe in particular. And this now is one of the, well, one of my pet projects, really, which finally came to life, namely the quest for large pages. So what is it? Why do we do it? So when you do IO, IO has inevitably been done in larger chunks called blocks. These blocks are currently limited by the hardware page size of the Linux kernel, which is typically 4K on an x86 box. And that is also, this page size limitation is also implicitly assumed by various drivers and subsystems. However, that's not the end of the world. Some systems and or applications actually might benefit from larger pages. There are certain databases, which really would like to talk in 16K increments, because that's how the database is organized internally. Also some hardware really would benefit if we could move to larger pages, larger sizes, larger block sizes, because then the overhead internally in the drives wouldn't be that heavy and the entire drive will become, well, more efficient and cheaper. So why do we even have that? I mean, couldn't we just say right this data go? Well, yeah, sort of. The problem is that you can't just do an atomic IO. There is no single assembled instruction do IO on these bytes. You always have to have several instructions setting up IO, transferring the data, getting back the results, and so on and so forth. This is obviously always increase the latency for each and every IO you do. So what you're trying to do is to minimize the amount of IO, meaning the actual number of IO, not so much the size of the IO, but the number of the IO. And then in the questions, right, how much IO is the best ratio? How much data should I transfer? What should I do here? There had been quite some bit of experimentation back in the early days. If anyone here is familiar with mainframes, mainframes still have the concept of variable block sizes for their drives, which are very interesting. So you have to decide on each and every IO you do, right, what might be the block size here? Interesting. And there had been a lot of experimentation going on and there had been researchers as Berkeley, of course, as usual, who eventually figured that, well, 512 gives you a good ratio between the amount of data you store, which typically tend to be very small, and the overhead you would incur by moving to larger blocks. So, but then again, this was 20 years ago. But we stuck with it and we still keep it that way for now. Okay, that's the IO. Now going to the page size. Why do I talk about page size and whatnot? So, the thing is that the CPU architecture has memory management unit. And that has some hardware assisted things, allowing you to tell whether a given memory area is dirty or not, i.e., where that need to be reread from disk to get the actual contents. So that's actually hardware assisted. And as it's hardware assisted, it can only operate in different sizes. So you can't just pick an arbitrary size, but the size which you can choose is actually given by the hardware. So say for x86, you have a choice of 4K to max, and I think 2 Geeks is the next increment which you can do. And that's it, nothing in between. For other architectures like PowerPC or ARM, you have a bit more flexibility, like there are still some power systems or even ARM systems out there which use a 16K page size. But this is just because the hardware supports it. You couldn't do that on an x86. And so, for Linux, we have this page size compile time setting, which tells the compiler and the entire source code. All right, the page size is now that value. So as we want to be compatible with other architectures, the common setting here really is 4K for basically throughout the board. I mean, it is certainly for SUSE. I guess it's the same for VEL, and so that's basically a common setting which we have. But still, page size typically means, right, that is the unit with which the memory management works. So now we have the memory management talking in page size increments, but as I already mentioned, well, we might need to refresh the data back from disk and into the memory and from memory on to disk. The thing which does that is called the page cache with greetings to Willi. And so that's the buffered IO because it's, well, we are not writing directly to this, but rather buffer the IO in the page cache, which is also working on memory pages. So guess what? Indeed, it's also working in 4K pages increments. And so there's the, when you work with the memory pages, there's a hardware setting.  That then triggers the IO to refresh the, basically to read in the page contents back into the memory management. And so this will typically be done on page size increments. You could transfer several pages, but then clearly all of these pages would have to be marked as dirty such that the hardware logic behind it can work. And so if we had units which are larger than pages, then we would need to treat it them as a single unit. And that is called a folio in Linux. And that is the, oh, I don't want to be there yet. So coming back to the folios later. So, and then we have a page just telling the block layer, right? Okay, I want to do IO. As already mentioned, we have two types. That's the buffered IO and the direct IO. For direct IO, it's trivial. That's basically user space thing. Transfer this data now. All right, and you do that. So nothing really you can do because user space already tells you how the IO should be laid out. For buffered IO, it's different because that's from the file system. And the file system typically just cares about the amount of data. It doesn't really care how you need to organize your data internally. It doesn't really care. And so to do that, there are actually several interfaces. How you could do IO. The primary one or the original one is called buffer hats. Then there's a successor or an underlying structure called the struct bio. And then there's a thing called IO map. Again, coming back to that. But in order to do so such that we can transfer larger blocks, we need to convert the page cache to folios. Now there it is. I always waited for this slide. I always, as soon as I heard this folio thing, I said, oh, I need to have a slide with the first folio. Right. I thought, okay. Anyway, he got it. Oh, brilliant. At least one. Very good. Okay. So folios. Folios are basically a common structure for a set of pages. Because as it so happens, we are not only having normal page. The management also knows of other types. There are things called compound pages, which essentially is an array of pages, and there are things called huge pages, which is an improvement from, well, several years ago. Initially, it was a separate file system. And then it got into something more flexible, which is called transparent huge pages, THP. Some of you might have heard of that one or seen it somewhere written in LWN. And each of these has their own peculiar work, which makes it quite odd, because all of them could be addressed via a struct page.  We got into some really funny issues there with when trying to transfer pages down to the drives. That's, say, the send page okay one, because that tells us, right, is this really a page or do I need to do something else? And so Matthew Wilcox invented a structure called folio, which is basically just an overarching type for all of these distinct things. So all of these various types can be addressed via a struct folio. The important bits in our case here is that the folio can be larger than a page, which is nice, because that is precisely what we need if we want to transfer larger blocks, because then we can identify one of these large blocks with a folio. That would work. And with that, we can do, in theory, large block IO. However, that would require us to, well, convert the page cache at the very least, but more likely even the memory management over to folios. That was first proposed by Matthew Wilcox in 2020, and had been a prominent talk for the Linux storage and file system conference ever since. And at all, if you could imagine, had lots and lots of controversial discussions at one point, someone even refusing to merge it at all, because why would you need that? Well, yeah, we do. And it's an ongoing work. So this is just a simple counter for the number of invocations of struct page and the number of instances of struct folio. As you can see, well, we have a long way to go. So this is ongoing work. We will get there eventually, but we are certainly not there yet. So, Buffered IO, what do we do? How do we do Buffered IO? So this now is, some of you might know, that's the diagram of the Linux storage slug, as you can see. And the really depressing thing is we are just dealing with this little grey rectangle in the upper right corner. So that's the area we are looking at. All of the remaining things are none of our concern. So, Bufferheads. Bufferheads are the original structure present since 0.01, meaning the very first instance of Linux. And that essentially is a representation of a sector, of a disk sector. It is 512 bytes, or assumption is 512 bytes. It's linked to a page and there's internal caching, the famous Buffer cache, which saves on doing IO for each and every access to the Bufferheads themselves. These are still in use by most file systems, actually. And there's also pseudo file system for the block device, which is also using Bufferheads. And the page cache itself was only implemented later, because Bufferheads did their own buffering. And just for the fun of it, this is the actual structure of the Bufferheads. And so, as you can see, yeah, it is quite something to store in. And the question really is, do we need all of that, or can't we have something simpler? And that's what ended up to become the BIO, or basic IO structure. This was invented by Jens Achspo back for 2.5, which is basically the basic IO structure for the device drivers themselves. So, this allows you to re-vectorize IO, meaning you can have an array of pages attached to a single BIO. And you can route and re-route the BIOS to devices as you see fit. DeviceNeper is the example, because that's doing precisely that, re-routing, re-formatting BIOS just to make whatever it wants to do, rate, or video, or something, you name it. And that's actually the primary structure for the block layer. And the Bufferheads are nowadays implemented on top of struct BIO. So, the Bufferheads will be converted into a BIO, and then that will be sending out the actual IO. And the BIOS themselves are used by quite some file systems, AFS, for example, others, such that those file systems won't be using Bufferheads, but BIOS directly. And then there's IOMAP, as I said, or Chris of Helwig, gone crazy. As he is want to do, he's not here, can we not record this? Sorry. He got fed up with all these various structures for IO and invented his own thing called IOMAP, which is, well, the modern interface, which thankfully already operates infolios. And thus, this basically just does away with all the intermediate structures. It just provides a way how a file system can tell the block layer how the IO should be mapped, and then it's up to the block layer to map it to lay it out correctly. Some file systems have already been converted, and that has obviously called us directly into the pagecache to enable it to work with the pagecache. Some files have already been converted, most notably XFS, ButlerFS, and SonarFS. Someone knows about that. So for these, clearly nothing needs to be done. But documentation of that? There must be some. Only it's hard to come by and not very accurate, because that interface keeps on changing. I mean, it is under active development, so every new release, you will find new features, which are not really that well documented. So what do we need to do to actually get to the point of doing large blocks? This is just for fun of it, just googled a large block, and it said, yeah, it's an area with more than 500 square meters. Not quite what we want. Anyway. So there is this long-standing trend in the storage community that buffer heads must die, because reasoning is that it's an ancient structure, and really it's a legacy interface, and everyone should be converted over to a struct bio or to IOMAP. And then there is this quote from Monday, I guess, or Friday. You can read it for yourself. I'm not sure whether I can read it without violating any code. And so maybe this direction is not what we should pursue, not if we want to make any want to have Linux to merge that code. So what can we do? So, yeah, conversion to folders is nice, but really only affects the page cache and the memory management substance. So we need to do more there. And the problem is that buffer hats actually assumes that I will be done in smaller instances or increments than the page size. We now have the opposite. We now have larger increments than the storage page. So several routes which you could take. One is convert to IOMAP, then we could obviously just switch it off, just don't use it and compile everything out which it does use it, or could try to update things. So convert to IOMAP. In an ideal world, that's what we would want to do, because IOMAP is a modern interface, and it's actually quite a nice interface. I mean, still Chris of Helvig, he does have nice ideas and workable ideas. It just, he also has complex ideas. So some files have already been converted. So in an ideal world, we would convert all files systems to it. So if someone has followed the discussions on the kernel summit mailing list for the next maintainer summit, there's a very long and lively discussion what we should be doing for legacy file systems, because while these are legacy file systems, and the problem with that is, well, there is not so much an active maintainer for it, because well, he has long gone if it had ever existed. Sorry about using the pronom he, I will be inserting the proper gender correct forms in the recording. So, and of course, the documentation of these file systems is hard to come by if it ever existed. Most of the legacy really, especially the old legacy file systems are actually reverse engineered by what they have seen on the disk themselves without any idea why it had been there. So how would you go about and convert these? That really is a hard thing to do. And of course, you would need to have a proper documentation for your own app to enable other developers who are not that familiar with IO map to actually do it. So, hmm, possibly not the way we could be or should be going. So, of course, we could also just say, all right, whenever there's a buffer head, just compile it out. There's a patch set from again, Chris of Helvic, which did exactly that. So you could just delete all of this because, and then trivially, both buffer heads won't be used anymore, and there's no discussions to be had. This patch actually went in in 6.5. So with 6.5, you can, well, you can actually switch it off. But if you compile out all file systems which use buffer heads, buffer heads will be switched off implicitly. So, hmm, that is a bit of an odd interface. But yeah, it is backwards compatible. It's just not only it's really hard to get there. And the interesting thing is that some files which actually are in common use like FATS or X3 will then no longer work because they won't be compiled in, because they haven't been converted. And, of course, there's always the possibility to update buffer heads. This was actually the direction suggested by Joseph Bacic at the last LFS in Vancouver. So, well, you could just convert the whole thing, convert to folios, and then see that it, that you get rid of the assumption that I would need to be smaller than the page size, but also could be larger than the page size. So, hmm, it could be quite trivial, because in an ideal world it's already coded that way, that everything works, or it could be a complete nightmare, that this assumption that you always would have I always just smaller than the page is basically implicitly coded all over the code and you would have to do a full audit of everything. So, originally, when I heard about that, I said, that is not the direction I would have wanted to go. And, of course, there's the Ketermkenzio, that buffer heads and so on. So, later that day, I've been sitting at the bar after having looked at the code and said, oh, God, that's a complete nightmare. How on earth is someone supposed to convert it and what are doing buffer heads at all? Why do we even have that? Complaining bitterly to my neighbour. I wanted to figure out later that this was actually Andrew Morton who said, well, back in the day when I wrote it was quite good and it still works, doesn't it? So, but thank God, not something which a good drink wouldn't fix afterwards. So, yes, but then again, it really made me reconsider maybe Joseph was right after all. But if you're updating buffer heads, you get into all these well-grubby details, which you really wouldn't have wanted to thought about earlier on. Like there's a void pointer attached to the struct page, which, well, it's a void pointer. And that points to the buffer heads if using buffer heads. If using IOMAP, it points to the IOMAP structure. So, hmm. And then figuring out that you're actually running in the page cache, meaning that this page is shared by everyone attaching this, accessing this very same page. It really makes a difference whether you're talking to buffer heads or IOMAP. And so, basically, it means that the page or the folder can either work with buffer heads or with IOMAP. And that is a problem for the block device because, well, as it so happens, surprisingly enough, every file system is running on top of a block device. And if that block device is using buffer heads, well, the file system better use buffer heads, too. Otherwise, you might get interesting results, like very nice color crashes. And so, the mix and match approach is something needs to be considered carefully. And the other one is that, well, UFI obviously requires a file system. So, if you want to switch off buffer heads, you wouldn't have a file system. So, booting UFI machines will be very tricky. And, of course, review will be hard, because you need to spot all the dependencies on the page size or the implicit dependencies on the page size, like an increment by one if you're in the right pages. So, and then again, why do we do that? Is it really worth it? Well, I think it is. But that's just me. So, the one thing which I know for a fact is that database really want to do larger IOs. So, that definitely would benefit from it. And the hope is that we're getting more efficient IOs, because in most cases, file systems already will be submitting larger IOs. I mean, butterflies is going through a lot of pain to ensure that always large IOs being sent. And XFS similarly. And, of course, we would make the driver vendors happy, because the drives they make can be more efficient or cheaper. So, is there anything which I did? Or is it just talk? Well, I had been happily coding away and basically had been finished my patch set as bonus one to do last week, when suddenly Louis Chambalain popped up and sent a patch set. Oh, here's a patch set converting everything to a large block. Thank you very much. You could have talked to me. So, as I said, I do not work for something. I have nothing to do with their work. And this what I'm presenting is entirely my work and no one else's work. And so, of course, I will be talking to Louis and his colleagues to join both our approaches to come up with a combined patch set. But yeah, isn't open source great. Just when you think you're done, someone else did the very same thing and faster than you. All right. That's the way it is. So, anyway, what did I do there? So, I figured that if we were to move the buffer heads off from assuming that it's attached to a page to assume it's attached to a folder or the underlying concept would still work, then the IO onto buffer heads would be still smaller than the attached unit, namely the folio. And we could keep all the accounting, buffer head accounting whatsoever in place because the overall rules won't change. And so, also, the page on this case, the folio would still have a pointer to the buffer head, a single pointer to the buffer head. And that would keep the number of changes to the buffer head or the page cached at a minimum. But then you have to adhere to some certain guidelines when converting. So, the problem is that you suddenly have different units which you carefully need to look at. Namely, the memory management, everything page-wise, everything the memory management is working on, it's still going in 4K or page size increments. The buffer cache will be working on folios, which will be on whichever size the folio had been allocated. And the buffer heads themselves also work on folios. So, that's fine. So, the buffer heads can do IO, but the buffer heads actually don't do IO. The buffer heads, as I just said, are sitting on top of the block layer on top of stroke bio. So, what about them? Well, and that's where it's getting ever so iffy, because the block layer is working on 512 bytes increments by block size, full stop. That's the logical block size for the block layer, and there literally is no way of changing that. That is built in each and everywhere, and no, you don't want to change that. Thank God, that is not what's doing IO, because IO is done on the lower level drivers. And the lower level drivers already merge adjacent pages or adjacent bytes together to a larger unit. So, if you feed them with a larger unit to start with, it will be enumerated in 512 bytes units, but the driver themselves will be reassembled into the original folio, or the data which is being pointed to by the folio. So, that means there's actually nothing to be done. It should just work. Just work. And it's not really the obvious way, but yeah, seems to be good for now, or we should work for now. So, and that's essentially what the final patch did. Of course, there one needs to order the page cache to allocate folios and to ensure that everything is really working on folios and really is incrementing by the size of the folio and not by page size. And then there's also we need to transfer the block limits, because really the driver which tells us which size we should be using in the page cache, so we need an interface for that. And that worked quite well, actually too well, because the first patch that I did also used NFS, and it turns out that NFS to be, well, more efficient, tries to transfer really large chunks like 128 megs, which surprisingly worked for a certain time. So, a copy worked for quite a long time until it finally went out of memory, which neatly proved that for large blocks leads to a higher memory fragmentation, if there was any proof, there it is. So, was that it? Well, sort of, because nice that we enabled the Linux kernel to talk to drives with large block sizes, but really there are no drives with large block sizes because no one can talk to them. So, what I did was update the BRD, the block RAM disk driver, to actually display or support large block sizes so that you have some test bit to test it out with. And that proved to be an easy test bit, so it actually worked quite well. And you could even, surprise, surprise, use this as a backing device for the NVMe target. And voila, even NVMe could speak with large block sizes. So, that was quite cool. But of course, there are still some testing to be needed, especially this splitting and merging in the block layer needs to be tested. And, yeah, it seems to work for my case, but well, what am I to say? And of course, yeah, what else? So, QMU would need to be updated, because for QMU it would be quite trivial just to support large block sizes. In theory, again, you would need to just need to modify the drivers to display or to announce large block sizes, just. I haven't looked at it, so, as usual, in theory everything's easy, but then looking at the code and knowing QMU it will probably split it all over the case, all over the code. So, I'm not sure. And, yeah, you would need to exercise the drivers with them and you could test with other subsystems. And, yeah, the other one is that I would need to unify the patch with that one from Samsung, but chances are that I'll be speaking to Louis next week and the hope is that we'll be able to merge both of them. And, of course, there are the usual reviews and follow-ups and everything. So, and then there's the issue of the memory fragmentation, which will be a real issue once we move to larger block sizes. I guess it should be okay if we're just talking about 16K, but still the problem is that memory management continues to run on pages. So, any internal allocation which would probably still be done on pages, which means that we will be having enhanced memory fragmentation there. We could maybe move away from that if we switch the entire system to just allocate with the block size, but that assumes that you have the block size. If you have several drives each with different block sizes, it doesn't really work again. One possible thing is, which might be worth doing anyway, is to update Slop, meaning the memory allocator, to run on higher orders for you than just the page size and obviously convert each and everyone who's using alloc page to just use Slop. That would get rid of the internal allocation and with that, I hope that it might be possible to reduce the memory fragmentation up to the point that you can do away with that altogether, because you will always allocate in folio size increments, but then you still haven't solved the problem. What do you do if you have several drives each with a different block size? So, for that, I really don't have a good idea. And of course, I'm open to other ideas if someone has ideas. And of course, I need more testers, because that's something which really needs to be tested. And well, and in case you're really bored, there's still the block layer, which as I said operates in far from 12-byte units. And so there really is no good way how we could how we could change that. Thank God the data itself is not stored in the bio, but rather in the attached structure in the vectorized zero, that's called the BioVec, which is basically that structure. And for that, it should be possible to, well, move the struct page to the, well, struct the folio and just use that or even have a union which would allow to either access it via the folio or the page, because really the first bits are the same, so you can have a direct, basically you can have a cast from a folio to a page and vice versa. And so that would be possible. I haven't looked at it. I haven't attempted it. It should be possible, but again, this is for someone really determined to do it, because that is no fun whatsoever. That is basically all over the place. I just checked the struct folio in the block layer is mentioned 10 times and the BioVec is mentioned about 4,000 times. So what should possibly go wrong? Yes. And with that, I'm actually done. Hey, it looks far better on the large screen than on my laptop. Okay, fine. And yeah, I did something. All right, good. Anyway, thank you very much for, and I'm open to questions. I'm still open to questions. All right, so you can always find me later on. I'll be sticking around a bit longer here or open the lobby and the hallway. Thank you very much for your patience.


The Navy's Office of Women's Policy in Washington helps shape policies around issues that affect female officers and enlisted personnel. Chief Jessica Myers explains how the office was formed. So our office actually was erected right after World War II when women were officially allowed to stay on active duty Navy and they didn't really know what to do with us. So they formed this office to kind of guide the direction of where women would serve. Today the office monitors gender trends including recruiting, retention, parenthood, classification and career development for the women that make up nearly 18% of the Navy's total force. You can read more about the Office of Women's Policy at npc.navy.mil. From the Defense Media Activity, I'm Petty Officer Andrew Johnson.