Hello everyone Really happy to see you all here on Friday on Europe item. It's really nice conference Do you like the conference so far? Thumbs up thumbs down. Yeah It's for me. It's my first conference outside of Bulgaria So I'm blown away by the organization by the venue by the city by the heat. No, I'm lying about the heat I'm Radoslav Georgiev. Yes, I know Eastern European names are hard to pronounce. So just call me Rado I'm Python and Django developer doing this every day and I'm also a founder of software company called hacksoft He's based in Sofia, Bulgaria. We are developing mainly with Python and Django Some Twitter GitHub whatever If you care one thing that's highly contextual for this peak is that we have an academy In Sofia where we do Python and Django courses something like nine in total so far Where we prepare people for their first job and we're quite good at this most of our students start working as Python and Django developers and I am part of the teaching team for almost all of the courses and This is highly contextual because the goal of this talk today is to share my experience and to be practical and helpful and I believe to have a lot of experience because when you're doing a course There's always and people are learning new things. There's always someone raising hand and saying nothing's working Can you take a look and you go there and you look at different people writing different kind of Python cold and At some point you start seeing the problems for two seconds in your cell you have a error here or you have an error here and While looking at a lot of cold I started seeing some patterns repeating patterns and this is actually The point of this talk to be practical to be helpful and to see different patterns in debugging not just tools I will mention tools in Python. Of course. This is your Python, but most of this will be patterns and Of course, this is by no means extensive Lecture or talk we need an entire week of debug con to cover everything about debugging It's a huge topic. So I won't be extensive. Don't hate me for not mentioning something that you are passionate about All right, I will talk about three different Things that can happen while developing any cold and for this talk Python First we will start we will start with there's an exception Everybody has seen stack traces. Everybody has seen exceptions. So we'll start with this Then we will continue at the point where nothing's happening. Nothing's working and things are starting to get frustrating and We will finish with we got our own cancer. This is maybe the worst of all. And of course, we will say some common things After all of this because there's always common things to say All right, let's start This is my beautiful boxes of frustration level When we have an exception and we start debugging it's usually not that frustrating and we'll understand why So this is level one of frustration The running example that I'm going to use because it's easy to use example here is that we have a File with some daytime intervals and call it in strings and we have a Python call that checks if there are overlapping intervals inside Not really. It doesn't really matter what I'm going to show you. This is just going to be the example and We have Yep, we have an exception And when we have an exception the first thing that we see we run the program There is an exception and we see a stack trace and stack traces are actually really really helpful This is this is a friend of the developer while developing because that traces they tell you what actually happened before you hit that exception I don't know if you have ever read stack traces front up to bottom. I love doing I love reading stack traces Unless they are like five pages of scroll then you just scroll them and don't read them at all But when people start coding they tend to not read stack traces. It's like boom exception Can you please help me something's happening? So stack traces are a friend because they contain a lot of valuable information. It's like on the bottom is the exception Valuer whatever and on the top. I don't know if you have read this But this most recent call last which means that the the first thing on the stack trace is the first function that was called and The last thing on the stack trace is the last function that was called or called executed before we hit that exception and I have some red arrows to point that To point that's the information. We have the module. We have the function name. We have the line number This is really helpful because you have an exception on line 45. You just go to line 45 You don't look for the function. You go to the line. So this is really helpful and Hmm Almost there and we have the exception a stack trace contains as I said the exception if there is an exception We can have stack traces without exceptions. This is also nice if you want to trace something The function method call path to that exception it tells us where where we went to and what functions we called in order to get the error and We can use this to actually the book what's happening and this is easy because it's all the information is there alright the first thing that people Do when they're debugging case to read the exception and I showed you an exception that fits in a slide but you will believe me that real-world exceptions especially if you're using frameworks are like five scrolls long and if you have a big Exception it's hard to read it because you have to scroll and most of the code is not on your side It's not your code. It's on the framework side if you're developing Django and something's wrong with the model You're going to have at least two scrolls of Django DB models PY Calls where something's happening and you get a value error at the end and this is not really helpful So my first advice when reading an exception is to find the line that separates your cold from the framework called Because most of the times it's not a frameworks problem Sometimes there are books but most of the times it's your code that actually broke And if you're debugging can reading Django DB models stuck trace is not really helpful because each time It's the same thing and then you get an error and if you have a long stock trace It can be your cold framework your cold framework the line that separates it is usually If you start from the bottom and traverse up is the first piece of code that's on your side It's usually the call that triggered the exception So this is like a tip for reading long stock traces. Just find this line for example here In exceptions PY line 10 in parse daytime This is the line that separates after this I'm going to in Python library. I'm not interested there I don't know what's happening and probably this is not the bug because it's Python library that's been tested and in production and so on and so on so if you find the line You will find quickly the piece of code that actually broke This is the line it's in some list comprehension I'm doing stupid things just parsing the entire line as a daytime format, which is not actually right so There's a golden rule if there is a bug and you can write a test for the book You should write a test for that bug because otherwise you're going to fix this and something else is going to break And I call this constant regression and I've seen this in many projects And is and this can make you crazy. It's like fixing things and breaking color things It's fun if it's a game, but if it's a production code, it's not fun at all so if you find a book write a test for it and That's what we're doing here. It's for the slides if someone that wants to look after this It's a standard test. I'm writing a test and the test is failing So we're fixing the code. We're writing a parse line function. That's that does the naive approach of parsing by splitting and taking the parts so good and We're running the code again And as you can see we started with an exception that Some kind of a bracket cannot be parsed and now we see some kind of a daytime cannot be parsed And if you have a trained eye, you'll see the space in front of the daytime and this is the problem But if you have untrained eye You can start getting frustrated. So When this starts to happening, there is an exception.  It's the bugger time It's probably the most popular tool in Python I won't go into tails for PDB and IPDB because there are really good talks in on your Python for this, but I prefer using IPDB since it drops me in a nicer shell than PDB and The thing that we can do is set a breakpoint. I call this breakpoint I'm not sure if it's the correct term But you just put this on a shortcut in your editor put IPDB.  It breaks builds Once you run the code All right, forgot the slate So PDB interactive Python the bugger with console interface and stops the world and lets you look around This is the nice thing about it. You ask the values of the locally and globally defined names and you usually find the bug this way or you start jumping on Breakpoints or you start continuing the execution of the program and helps you look closely at what's happening And it's the better print function everybody starts with print and then moves to IPDB So when we run this program, we are dropped in an IPDB shell I'm sure all of you have seen this or is going to see this and use it And if I ask the second value, I'm going to see that here is a nasty little space So we have a bug. We should not forget to add a test for it. We're adding a test We're fixing the function by splitting by comma and a space because this is the format of our file and we're happy tests are passing now If tests are passing does this mean that we have a guarantee that Running the program is going to be okay Yes, no, how do you feel about it? No, no, yeah tests are good first step But we need to run real world and of course running the program boom another exception index out the French and as you can see the Exceptions themselves are not really helpful. It's value error value error index out the French and this is by far the Least helpful exception for me index out the French. What? There's something really wrong with this and I don't know why so More debugging time. I drop a pity IPDB in this parse line because I know this is the only place I use indexes iron the code and this is something that happens to Junior developers that start to understand and use IPDB you usually drop by the IPDB in the loop where there's 100,000 items that's going to be loop and you start hearing can't enter can't enter can't enter in After 10 minutes can't enter can't enter Because you need to you need to get to the line. That's actually breaking and this is also frustrating and time-wasting It's good to do it once or twice or when there's no other option and you're not sure what's happening, but IPDB there are some nice context manager called launch IPDB on exception which if we wrap the function and Run the code it's going to drop us in IPDB shell at the line that calls the exception Saves a lot of time. It's actually pretty nice and if I ask the value of parts, I'm going to see empty string and It's one element array. Do you know why this is? For someone that if you have parsed a lot of files, you know that files tend to have a new line at the end That if you split is going to result in a single item array with empty string So this is pretty nasty first time you hit it with a buggy a second time. You just know it and Add a code for it in the list comprehension with trip if it's empty. We don't want it and All good. Yep fix fix test passings all good programs running tests are passing. We're happy and as you saw The approach was we started them with an exception We fixed this exception which gave us another exception We fixed this exception with which gave us another exception And this is usually the step you're going to the steps you're going to follow when you're debugging It's either you see the problem in general and you fix it or you just fix this small problem in order to get to the second problem in order to get to the third problem problem and if you have tests You can know when you're ready. So this is kind of everyday debugging. You don't try to fix everything at once You just follow the step trace and the exceptions and in the end if you're lucky enough, you're going to have a working problem for at least some time and There are some exception types that are quite How to say useful and helpful? Like improperly configured validation error permission denied. This is when the frameworks The folks guys and girls that are developing the frameworks are nice and want to help you and in Django There is like tons of exceptions like improperly configured You messed up something they will raise you an exception and actually tell you what to do You are missing the metamodel Field here on this class or you cannot do this or you need to do this which is really helpful And that's why reading the stack traces is good Sometimes it can be helpful. Sometimes it can be it can be index out of range error Which at least in Python is good because if you're writing in C++ you you're going to get segmentation fault, which is even harder to the back All right, and fix it should be here fixes here are easy. Just read it fix it. Oh good and To summarize the first part Stack traces are your friend if you learn how to read them. You're going to the book faster They can be huge so looking for the line that separates your call from the framework call It's really helpful because it will save you a lot of scrolling and Often the problem is not in the framework Sometimes it is and if you learn how to use IPDB in my github repo for your Python 2017 I've added some Materials for shortcuts for IPDB if you learn how to use IPDB you're going to be really good at the bugging like since Stack Trace jumping around so on and if you hit a book write a test for it, I cannot emphasize more on this and The more I develop software the more I realize that tests are really important and you should write tests constant regression Yeah, I Don't want no one to suffer from constant regression All right. This was the first part. I have 30 more minutes. That's good second part We learn how to battle with exceptions, but now nothing's happening. You run your program There's no exception. There's no output. Nothing. Sorry nothing's happening and this starts to get frustrating because You're in the dark You don't know what if you have an exception you know what happened if nothing's happening You don't know what's happening That's the case and you need to find some source of flights. This is a detective work You need to quickly find a way to break things. I'll go there in a minute You need to know what should happen and what is the expected behavior because Oftentimes in students. I see They're struggling with something that's working And they don't know that it's working they expect something else to happen and this is really bad the bugging something That's actually working. So you should have a good idea what should happen and what is the expected behavior of the code and of course if you're in this case Daught everything because there are some pretty nasty gotchas that we're going to cover that can make your hair white and If there is no exception The quick fix for this is to look for error locks. Sometimes we just open a web browser. It says 500 This is it. Just a number Probably not the winning lottery number and there's no exception. No, you don't see an exception So you need to look for error locks in order to get an exception and this is actually For me, this is the more the most important thing when I'm debugging something that's not nothing's happening or something's odd I want to break it on purpose. I start typing random syntax errors I start raising as exceptions left and right in order to get an error to get the problem to fail so I can fix it and If I can break it, it's good. But if I cannot break it Then I need to isolate the code. That's not working For example, you're parsing some day times in a language. That's not raising cars. For example PHP Instead of parsing day times in your entire big project. Just take this piece of code Create a new file. This is like my most given advice to everyone create a new file and test the tested there Is it the same? Is it this code that you're debugging? That's actually wrong or is it something else? open a new file and do things there and Sprinkle with the boogers and trace every step of execution You need to know the value of everything. You need to know what's happening in order to see why nothing's happening This kind of sounded smart This should be on slides. So yeah code isolation is really your friend because if we're working on a 50 line Python script The bugging is pretty easy. You have 15 50 lines. You can read them But if you're working in a 50k Django project, sometimes you need to isolate the code Fix first the code and then go back to the framework and integrate the code in the framework It's more easy to work on something small than the Integral part of everything with Django and frameworks And there are some common gotchas that actually happen in Python and the first thing is module hiding I was debugging why you cannot import daytime from date from daytime for about an hour When I realized that the students named his file daytime.py We were reading the Python called reinstalling Python Opening issues What's happening like I open a repo on my computer and import it. I open a rep on his computer. I Cannot import it and in the frustration. I kind of changed directories.  No stuck trace. Nothing's happening. It's really really bad If you're writing context managers, you can hide your exception really easy. I did this once There's a third argument. That's the exception. You need to check and Erase it It's okay. We're used to this kind of noise in Sofia alright, and other common gotchas is calling something by string and Making a typo and not having the code that's going to raise an exception if there's something wrong This is also really frustrating because there's nothing wrong. It's just the typo. It's just a string that you're calling a Classic thing you're debugging why this function is not working. Well in the first place we need to call this function it's like Writing a main method in a Python script and forgetting to add if name equals main called the called the main method really really classic classical thing and Editing the wrong file or running the wrong file. This is also debugging something that's working You're editing something at one place and running the same thing copy of it on another place There is no way for it to be working here breaking it on purpose works really well if you add random rubbish syntax error and the program still running you're most likely most likely here you're Editing your own file And of course debugging something that's working This is hard to catch because you need to know what means for your program to be working or not Sometimes you need just to know if this is okay and an example On Monday, I did a Django and salary training here on Europe. I turn I have an orange batch. It's really nice and I was some I was doing a live coding without preparation which always goals is planned and As you can see I have no registered tasks here in salary. It was working just a while ago. I Did something but I'm not asked and I don't know what happened Everything seems working when I start to panic and when I start to panic I do this thing called Binary search debugging which is dump and also really nice and it looks like this I Commented everything that I've added in user the user tasks. Puy. It's working. All right. So here's the problem I will get to the problem. I Uncommented the task and seller imports still working Uncommented the app imports that I met. I broke it commented a specific import working and Commented that import not working and I found the book and I don't know if this is an actual binary search But I like calling this binary search debugging because you're not putting your How to say your thoughts in the process. You're just commenting and checking. It's like good bisect pretty much doing what good bisect does and the generalized version of binary search debugging is Comment some parts of the code that you just added. So the program starts working and you're happy Uncomment a smaller portion of the code that you have just commented if if this is working then you can uncomment some more if It's not working then you go to step one but only for the code that you have just uncommented and things broke and by this you kind of Narrow down the Search area of in your in your project. So you know what's happening and you don't have to think about this You just need to comment and comment run comment and comment run until you have a small enough code to look at in order to find a problem and In the end you're going to find a breaking lines and I find this very very helpful because Sometimes you're tired frustrated. You don't want in something breaks that should not break You start doing this you find a problem quickly. You're happy. You go drink some beer at the beach or whatever you like to drink and a summary for nothing's happening Try to get an exception like this is the most important thing try to get an error so you can fix this error Isolate the code so you know that the thing you're working on is actually working and then try to integrate it in your project Because it's really bad to try to integrate something. That's not working in a project that actually can break the thing that you're integrating Yeah, make sure you know what needs to happen It's really important. Don't the book working things because you won't find a book Or if you find a book is going to be really strange and Try the binary search the bugging thing if The case is suitable for binary search the bugging where you just Comment and comment until you find a smaller portion of code where you can actually spot the bug or the typo All right, we know how to handle exceptions really nice. I love exceptions, you know the Famous thing that if you're cold runs from the first time it's really bad So try to get exceptions so you know that your cult is wrong Can you fixed it and now it's right if nothing's happening? There are several things that you can do but for me the worst thing of all is Everything's working There is no exception, but you get a wrong answer this is like the most frustrating thing because It's okay, but it's wrong. It's you and it's the algorithm that actually Needs debugging which is hard. This starts to get hard and you need to be focused and you need to be thinking a lot and When this happens, I usually for this five-step drip drip down I don't want this to sound click-baity or something like this, but it's just five steps that I Tend to do when I have a wrong answer and the five steps are like from Easy level to I should change my job level. So let's check them The first thing I always do is check tests because I usually test my thing with tests and If I get a wrong answer most likely my test is wrong and If you're doing Django, I don't know How many times it was the case that you need to refresh from DB your Model instance because you have a different state in your local test threat and this is like Fixing a lot of wrong answers first first check your tests write your tests good. Good tests are actually pretty hard to write I find this that you can write code actually pretty good with good abstraction writing good tests For me, it's even harder need to put more instruction more thinking And Not to make your tests need tests. Yeah, so check your tests If the test is correct Maybe you need validation. Maybe you've passed some wrong input Maybe you've passed something that your algorithm should not take in mind. So check this Is the value are the values that you're passing actually okay for you? Maybe you're passing wrong values then you need validation. This is good This is good because you add validation and your algorithm becomes more proof It's going to work more without breaking Now if you don't need validation and your tests are okay Then probably the next place you need to look at this at your algorithm, and I'm using an algorithm as a general Purpose called piece of code not by their search tree or something specific. That's called an algorithm I'm using this in the most general sense is your algorithm correct In order to catch correctness of algorithm You can either get a computer science degree and try to prove it formally Which is nice thing but takes a lot of time or you can put the buggers on each line of your algorithm and Put your headphones on or go somewhere else where you can focus and Start looking at values start looking at steps Is it from this step to this step? Do I have to do do I need those values or find the place where values get wrong? And this is quite hard to do and requires a lot of focus But for me, this is this is working really nice the burgers everywhere enter focus mode and start looking at values and most likely you will see where things get wrong Especially if there are for loops indexes or a sets whatever and this way you can fix your algorithm this is good but Things start to get worse if your algorithm is correct. Maybe your system design is not correct Maybe you designed this and forgot for a whole dimension of the problem It happens quite often it's like you do something for timelines in the future And suddenly you realize you need to do something for timelines in the past and you have zero code for this You don't have the models in place for this. You have nothing So here starting fresh and going to the white board is like the the way to go you need to Step away from the cold think again. Am I doing this right and maybe call it again or change the cold this is really really bad when it happens it happens from time to time and Probably the worst thing that can happen is if you don't understand the problem space well You have a wrong answer. For example, if I start doing data science If I get a wrong answer, I have no idea what to do There will be something about matrices and vectors and first I need to learn about matrices and vectors before I debug my matrices and vectors I need to learn about the problem space and This is bad when it happens because Someone else needs to fix your code or you need to do some quick and dirty hack And it's also good when it happens because it opens opportunities for new learning and this is always good and the Those were the five steps that you recap check tests check validation check algorithm trace every step Check system design. Are you actually solving the problem that you need to solve and finally Check your understanding of the entire problem space. Are you doing math and don't know math? for example and Finally some practical tips who I have ten more minutes so I can go through my last two slides five minutes each Alright some practical tips Explain your problem to someone. I'm sure you've heard of the term rubber duck debugging Sometimes there's no need to have a rubber duck. You have coworkers. You just you can use your coworkers as rubber ducks When someone asks me to help The first time I approach I always ask the question. What's happening? The the answer is usually it's not working and then I ask why it's not working and then the person starts Explaining to me why it's not working I'm not sure I listen carefully everything that he's explaining and suddenly the person is like, oh Yeah, yeah, wait a minute. I found my problem. Okay. I'll go down It sounds funny, but it actually helps a lot and if the person does not find the problem from his first explanation I ask again. Can you tell it again? And then I start listening so I can help But it really it really really really helps Explaining what you're doing to someone else Because then you're going to find if you have some small problem. You're going to find it If it's okay for you to talk to a rubber duck talk to rubber duck Whatever you find suitable Something I really like If you're frustrated and you have deadlines and you have a problem and you're you're not in your best Mood and shape to fix the book now, but it needs to be fixed You can practice the so-called power parallel debugging Which is ask a question of that car for all open a github issue ask a co-worker and continue working on the book it's like Give more more inputs to your to your problem because sometimes You're struggling with something and someone on stock or fall actually Answers the question and says you're doing this entirely wrong. Take a look at this and then you're and then you're ready or Sometimes someone on a github issue says I have the same problem Here is my coat and system and whatever you look at it and you find the problem that you're also having When a co-worker is thinking about it Also good more heads in the problem. So this can This is not actually nice because you're like firing random questions on stock or fall and github issues and I know people who are supporting open source They don't like this very much because it's like fix my problem at my work because I'm using your open source open source library so it's important when you do parallel debugging to Push back not push back but Contribute back to the community Answer questions on stock overfall if you see someone asking a question that you've been struggling with for a week Just sit down for 30 minutes and answer him. So other people can see it as Make poor request for documentation. I love doing this when I struggle with something That's not really nice nicely described in a tutorial documentation. I do it and Contribute back to the documentation. So other people won't waste time because I'm not wasting time when there is a good documentation and people before me have done this and Yeah answer to issues Help help other people because this is the way it can get better for everyone and waste Smaller amount of amount of time see bug fixes and One specific tip for stock overfall Read the small comments next to the answers because sometimes they Contain the better answer to the answer. They're like really eight pixel phones comments but they're also really good and Do a talk on a conference? It's nice to share experience seem to be helpful to other people and I guess this is it for me Thanks everyone if you have questions, we have like ten minutes. Yes, we have nine minutes twenty seconds for questions. All right Hi Great. Thank you very much. I just wanted to add to this parallel programming one crucial step Why we don't like it in open source and why how you can make it make us love you for doing this When you solve the bug go back to all those places where you asked about it Yes, and write the solution there and close your issue. Yes. Yes This does changes everything and we will absolutely love you for that.  Don't just leave it hanging there. It's it's I know it's bad Great talk. Thank you. I would like to know for example If you have a test that so many times works and it's so many times doesn't work What is your strategy of the best strategy to fix this? Yes, so that's actually a good question What do we do when we have this test that's working half the times not working together half the time so We had such a test a month ago The thing that we did is run the test a hundred times is like Write a bus script run the test go to go grab a lunch Go back collect the locks see where it's failing can start the bugging it. It was for us. It was a faker generating Names that were not compliant with our capitalized function and tests were failing can't run them and it's For me, it's like run it a lot See where it breaks run a lot the point that breaks and check values check what's happening Why why with this specific value? It's it's failing If you have tests that are failing the test may be failing because you have a different How to say system operating system when you're running come For example Jenkins. There were some really nice talk yesterday about docker rising py test like having the same Operating system on your machine in docker and the same thing running on Jenkins So you need to do this is like the nothing's happening. You need to do the detective work find what's happening can't fix it It's a nasty thing to happen Any other question? Have you considered using more visualized visualize the buggers like WDB or built-in ID the buggers like Python the bugger I Think this is like a personal taste For me is whatever works for you if you like more visual debugging. There are a lot of tools for this I'm a person who lives in the console. So I debug in the console It's a personal taste. Whatever works for you should be the tools that you're using and I can't I am coming from Java World where visual debuggers starting like next next next whole day the bugging I Something not related to debugging, but I was curious you mentioned something about the code Academy. Yeah in the Sofia Yeah Can you give us? I don't know a little idea of how long the courses and how long does it usually take people? All right So so the thing that we're doing in Sofia is called it's called hack Bulgaria and now it's being rebranded to hack software Academy for whatever reasons We do courses that are named programming one-on-one with Python, which is a four month long course we require some Programming skills before you enter and for like four months. We do Quite a lot of stuff. It's like three times a week for hours coding on place It's not like just lectures and it's working. It's intensive. It's working because it's intensive it and it's the way to learn for us Well, thank you everyone


Mae'n gweithio gyd yw gwaith i'ch gweithio'r cwmhwyl yn gweithio'r gweithio ar ôl 10, 9, 10 a 10. Mae'n gweithio'r gweithio'r gweithio. Mae'r unrhyw i gyda gymaint. Mae'r company wedi cymdeithasol gyda IQ, ymgyrchu llwyddiant, a'r llwyddiant cymdeithasol. Mae'n gweithio'r cybonetiadau yma, nid oedd 7 o 8 oed, oedd unrhyw o'r cymdeithio. Yn gyfer… mewn ei hunain yn y locysyllt yn leironion gyda diwrnod Cymunau Felly, felly ei sydd gennym ni'n problemon CSF, ond ar nghymru sy'n gallu gwylio, bywro fyddiau rhanau saluwsiau ar y berthynats ac chi'n fyrdd i gynnwys arlau ac mae gynnwys ar-raddeddio yma. Felly, корп blenderu gweldó â'r brif, gan bethau cymryd ystafell Felly, mae wedyn yn y cymryd y bryd rhanau yn y bryd ystafell yma ac ym nid yn chael, mae ei wnaeth y cymunau, Mae gen i. Felly, mae'r mawr pawb yw'n gwawb, mae'n fawr, a mae'n eich wneud nhw. Felly, maen nhw'n gwybod, gwneud e i fewn iawn. Dyma ar y pethau cyfanediol, mae'r cyfanediol maen nhw wedi gweithio arfer fel gen i'r тыan gen yn gallu'i rhaid ar gerfodd cynhyrchu eich cyfanediol neu mae'n gweithio arall, ond mae'n cael ei gofodd mewn gweithio. Mae'r mirwnt nil-ud ymddangos o berthynas i'r toray. Felly, mae'n gweithio'r toray o'r Fybwyr Fybed, Dewch wedi cael fel FfDV2 yn Lyfran, a mae'n rhaid i ffrwng o'r sculpt ac i irrydd lryd i phenoglau'rcuss. A bywch, mae'n rhaid i ffrwng o funud o'r sculpt o'r sorts, hefyd, fe'n mynd i ddim yn y cyfunetos eu cyfirlog. Felly yma yw'n mynd i'rMyrddiad hybridau hyn. ac mae'n mynd i'n mynd i'n mynd i'n mynd i'n mynd i'n mynd i'n mynd. I want users to not know that they're talking to multiple clusters.  I want the user to be completely ignorant of where that's running and I want to make it as seamless a kind of experience as possible, including stuff like failover, including stuff like how do we deal with faults, how do we deal with stuff like stateful services. And we're going to go through a little one. This talk actually came just from a chat with someone who was learning Kubernetes. And they were going, I've got a website. It's just a little website, just one. And it's just some static pages. And I want to know, how do I let people access it? That wasn't their voice, by the way. They've got a very lovely voice, not like mine anyway.  I've got a website. OK, you've got a website. And you want to expose it? Yeah. Have you got anyone that wants to use it? Yeah. I've got one person. It was them. They wanted to use their own website. That's fine. And it was going to be public. So they're like, OK. So we've got it. Don't jump too far ahead. I've got to expose it over the internet. Brilliant. Have you got a domain? Yeah, I've got a domain. It's called example.com. Because I can't tell you what it really was. And what address you want, I'm going to put it at mysite.example.com. Oh, brilliant. OK, so what do I need to do now? Well, you've given them an address. And how do people find out where to send their data, their packets? Where do they send that stuff? How do they know through the magic of DNS? And I never really realised how exciting DNS was. But it is. It really is. There was an old talk. Some of you may remember if you got great hair like me. Or not much hair like me. By a guy called Chris Boytart. I always say this.  I don't know if you know that, but I've said that a lot in my life. It's a great phrase. But DNS also has some amazing abilities that we can use. So learning about DNS, learning about something that is so old and the annals of time, learning about it, learning how to use it for your benefit rather than thinking about creating something new is really where we're going to go. And we're going to use some real key principles of DNS through some of the projects when we get towards the end. But anyway, DNS. So this is their IP address. So a client knows, they look up the address book of the internet or the address book of networking in DNS. There you go. That's the address. It knows where to send stuff. Cool. It's got that. User hits the website. Yes! This person hit their own website and they were very happy. The end. No. But then it got popular because they told their family about it. And so they had more users. And then they were worried about the scale. I mean they had four users. They needed to worry about the scale. So they deployed Kubernetes. Of course. They should have done it. It's just one website. Because everyone needs Kubernetes. Of course you don't all need Kubernetes. Don't always think that. Anyway. And so they deployed more pods. Brilliant! But I can only route stuff to one of these. I've got one IP address. So I need to put a load balancer in front of it. So from my four person website I need a load balancer. And through that I've changed the IP address. If you notice, subtle difference. <IP_ADDRESS>. <IP_ADDRESS>. Clever. And so we haven't changed the address. We've just changed where these packets are going. And this is the cool thing about DNS for a user. You know all this. And it seems really simple. But I like to appreciate the really basic things. It's really important to me to understand these underpinnings of how the whole internet works. Of how all networking works. We take for granted. And yet they're really important. So spend some time to appreciate them. Anyway. Please. Yeah. So they've got a load balancer. And they haven't done anything different. They've just done that. And the user doesn't know anything different. The user's totally seamless. Cool. But it got so popular. And also it wanted to make sure the reason why he deployed four pods with his static website on, or their static website on, was because they wanted redundancy as well. It wasn't just load. They wanted, like, if one died, they wanted people to still access their super... It was something like Fluffy Cats, I think, that kind of website.  Brilliant. And so we've got two IP addresses. So now what happens? I've got one name, two IP addresses. OK. What does that mean for a client? How do they know where to send it? Well, they can use either, right? Oh, yeah, yeah. Yeah. They can use either because the response back from DNS is saying, both of these are valid. You can use either of these. All right. I don't mind. It's cool. And then they went global. They still only had four users, but, you know, two of them were in Europe and two of them were in North America. That fits with the story I'm going to tell. It's like that's very, very clever what I've done. And so they duplicated everything, right? They put up another load. They brought up another Kubernetes cluster and they deployed their pods. And they basically duplicated everything. So now they have four IP addresses. Whoa, OK. So I've got no idea where anyone's going. Because they can go to any of these. They're all totally valid as a user to send stuff to. Right. Brilliant. Let's do that. If I can remember what window I'm going to. Oh, yeah. So I prefer live demos. I've had to record this because Wi-Fi is always terrible at these things. So, yeah, I'm typing. So I'm just doing curl. I'm just going to hit 10. I won't do that every time. So we're hitting an endpoint. I hope you can read that. OK. I hope it's big enough. Is it big enough at the back? You've got good eyes. I wouldn't be able to read it from up close. My eyes are old now. Anyway, so we've got this address, right? And you can see it says Envoy in there. Because I'm using Envoy Gateway. So I'm not actually using Ingress behind the scenes. I'm using the Gateway API, which you've heard loads about this week, right? I'm sure. Already. And you should. It's a brilliant project. So do investigate it. It doesn't make any real difference to what we're going to do here. I'm just saying, look at Gateway API as a replacement for Ingress. It still gives you Ingress. But anyway. So I'm just going to send 10 requests and look what happens. Boom. You know, as we thought. So what I've done is, again, super clever. The stuff I've deployed in Europe says, I'm in Europe. You get the restaurant. Anyway. So you can see round Robin. I've got four addresses. It's going to do the clients going to be sending to any one of those addresses. Round Robin doesn't necessarily mean round Robin. It like it has a chance. It's an equal chance over time. Right? So that's why you can see the order is not. Now I don't know if you noticed. I should have said before. Oh no, we'll see it in a minute. Oh, back to the slides. I'm very proud of that. By the way. Failure. Failure. So what happens then? I've got these four IP addresses and I've got one address to go to. God, better hurry up. I talk too much. Right. So what happens if I lose one of my sites? What's going to happen? Oh, by the way, I should have said this at the beginning. What I would do in your position, if I was sat there, is to pick holes in everything I'm doing. Right? So I hope you are. And I hope you're coming up with different ways of doing the same thing. Because this is just one way to do stuff. There are a million different ways. Anyway, we've lost USA. USA is gone. So we jump back. We jump back to a wonderful one. And what happens in the case of failure? So what we'll do is we'll scale down our EU deployment. So actually, I've done the opposite of the slides. And I've actually destroyed Europe, not destroyed America. I'm British. We like to destroy Europe. I don't. I don't. Anyway, bad joke. Anyway, so we're going to hit the same endpoint. And you know what's going to happen, right? Like, you can predict. Oh, look, some go through USA, which I didn't destroy. It runs through. And the other ones don't. We get 5.0.3s, gateway, time out, whatever it is. Oh, pause there. Oh, no. It's scaling it back up again. So by the way, this is in real time. This is the awesomeness of Kubernetes. I know you all know it. But like, when you're spinning stuff up like this, that wasn't like an Apple shortened display. That actually was in real time. When we get later on to the more fun stuff, we will get that. Then it's even more fun. I don't know. I can't explain. I get excited by the stuff. Right back to the slides. So we said we take out one of our zones. We don't have a clever way, right? Our users don't have access now to these services that are running there. And of course, there are things we could do. We've got multiple load balancers. We could scale horizontally. We've got more redundancy at our ingress level. Or we could, we've got more pods there anyway. But just bear with me. I'm trying to prove a very simple point. And then the other question is about stateful workloads. So there are reasons, requirements, legislation, of why some of our workloads can only run in certain regions. So there may be European laws around data can only be stored in European data centres. But I still want that data to be accessible from America. I just can't store it there. So what does that mean? What does that mean in a traditional ingress? So now what I've said is in Europe, I've deployed static and stateful. This has gone way beyond this person's website now, by the way. So forget them. This is now mine. So what happens in this case? So what I want to do again is a bit contrived. I want to hit this at slash stateful. There are other things you could do, of course. You could set up a DNS entry to point to these new services. But it doesn't fit with my story. Anyway, we're going to hit it under stateful. Oh, well, let's jump before we go there. Oh, giving it away. So what happens when we hit under this address? We hit a stateful workload that's only running in one click. You know what's going to happen. So you can see I've got slash stateful at the end of that. And, of course, what's going to happen is, as it's going to the place where it's deployed, I'm stateful when it goes to the place where it's not. I'm 404. Easy, just what you expected. And so what can I do? We can introduce Istio. So I've gone through the really basic stuff. I've just simple ingress. It goes out nice and easy. We all get that. And then, well, we can introduce Istio now. That's quite a leap sometimes. But we're going to go there. Don't be too scared of Istio. Who's familiar with Istio? Okay, fair number. It can be a bit painful. Learning curve can be a bit painful. And sometimes deployment and troubleshooting can be a bit painful.  A service mesh. I'm just using Istio just because. There's no reason. Right, so with Istio, what we've done is we've deployed the same pods. We've got the service. You know how Istio has these sidecars, these on-voice sidecars. We're not going to go through the architecture of Istio. Istio is the control plane for these on-voice sidecars. The data plane on-voice sidecars in every pod that has it injected. Or ambient mesh again. I'm going to gloss over that because I don't know much about it. And we're going to stick with the sidecar one. Okay, so we've deployed an ingress gateway. Again, this is using the gateway API, by the way. And you can see these things work the same. Just as before. There's no difference right now. So if I'm hitting either one of those IP addresses, which is now tied to the ingress gateways on here. That's the middle boxes. The blue ones on the edge. Exactly the same as before. It's being routed to the same cluster. And this is cool. This is great. But what's the benefit? Well, the benefit is it's two-fold, actually. So first of all, what happens with stateful workloads? So stateful workloads in Istio world in this way, well, you've got these pods, these services deployed in one of my regions. If I'm going to the ingress gateway in that cluster where it's deployed, just route straight to it. Simple, there's no difference. However, when we go to one way, it's not deployed. And this is where the first benefit comes. Woo! When you hit an ingress gateway in your other cluster, Istio is clever enough to have configured on-boy to route that traffic via MTLS from your ingress gateway, which is effectively just an on-boy proxy, route that across to the east-west gateway, another on-boy proxy to route that to the stateful service. So as a user now, it doesn't matter. I can hit any one of those lovely four IP addresses. It doesn't matter. It's transparent. Woo! There are other reasons why you wouldn't want to do this, but it's just to prove a point again. Now, what happens in the case of failure? This is like the second benefit that we talked about. One was what we do in failure, and the second one was how do we handle stateful things or services not deployed in that same cluster? So what happens if we lose a stateful service, a stateless service, sorry, in one of our clusters? Well, just like happened with the stateful service, boom! Immediately, dynamically configured, Istio will then route it over through the east-west gateway to our stateful service and the other one. So I've got failover between clusters. I've got transparency. The user doesn't know any of this is happening. It's great. Really cool. And there's loads of different stuff you can do in here as well. Remember, Istio gives you so much flexibility in that layer. For example, the initial configuration I had in here was to actually route to the local service in the cluster first. And then, if that's not available, then failover to the other cluster. So this is what you can do through destination rules, for example. So I have the flexibility to configure what my requirements are. My requirement for this demo was to be able to route it to a cluster where it works. Try locally to begin with and then route across to another one. That's what this demo was for. So we've sorted out stateful workloads. We've sorted out the ability to make that transparent to the user. And we've sorted out some failover. Yeah. So that's fine. But I was talking about in this one, taking out the stateless service. That's easy. The entry point for the user, that we've tried to, that ingress gateway, is still there. So from a user perspective, they don't know, because of that routing logic, what happens if we lose a whole cluster? Or we just lose the ingress gateway. What happens to the user at that point? Our services might be running fine, but our ingress gateway may have gone down. Istio can't really help now. Of course you can scale it, you can put redundancy in, but there's some catastrophic network failure. That's where KHB comes to rescue, but before we go there, we're going to jump back, because I did spend time doing these, and I think you should watch them. They may not be very good, but it's fun. I really enjoy it. If you want to know about the tooling I used for this, I found it recently, it's brilliant. Come on and ask me about that, as much as about any of the other stuff. Istio ingress global endpoint. It's just a slightly different address. I used a different DNS address, so I could show it in a demo. Istio rather than Envoy. Exactly the same as before in this case, because everything's up and running. We get that same round robin, although it's skewed towards Europe, that's for no reason. I did this in Europe, by the way. I was going to mention that. Let's see if we scale down the deployment. We're going to scale down our EU deployment, but what's going to happen here is that everything's going to work. Right? Because I recorded it before. But look at latency. I recorded this in Europe, by the way. Did you see that? It doesn't see much, but some of you work in areas where latency is important. When we go a little bit further on, you'll understand why some of the things we're going to do with our load balancing, with our global server load balancing, is because of the latency. That's just from the UK to Oregon. I think this was running in. So it's not far, really. But it still takes time. Latency is an issue, right? So we saw that. But it totally works for a user. The transparency for the user, they don't know that the service in Europe has gone down. They have a little bit more latency, but they'll survive, right? Maybe. Here. See the latency up between the European ones and then the USA ones. You'll be blown away by it. Do you see that? So that's recorded real time. So it does show a difference. I just found that interesting. I find too many things interesting. That's why I don't sleep very well. What about staple workloads? Same as we said, we've gone through the slide. So we know what's going to happen. Everything works. This is the same setup. Those four IP addresses to those four load balancers. Two in Europe, two in America. From a user's perspective, everything is brilliant. And so what the next bit is. Okay, yeah. So why don't we lose our region completely? So we'll scale down our Istio gateway. That ingress gateway. And then we'll hit it again. And we know what's going to happen. I love spoiling stories. Don't watch a film with me. And we can see that when it's trying to hit the European endpoint, there's a time out because our ingress gateway is gone. So from a user's perspective, it's fine when our services go down inside our clusters, but it's not okay when the entry point goes down. So we need to do something about that. Oh, yeah. That. Right back to the slide there. Okay, so K8GB. What is K8GB? Who's heard of K8GB? I don't even know if I'm saying it right. K8GB, KGB. Okay, so some of you. Who's heard of global server load balancing? More of you. Brilliant. Who uses global server load balancing? A few, but you know. Okay, so that's an interesting one. So everything I'm talking about here, by the way, some of you will be exposing services over the internet. Some of you will be on your own intranets, your own networks, and suddenly everything we're doing today applies equally. It does not matter. Anything we do can apply equally in your own environment to the internet as well. So K8GB. It's a cloud native Kubernetes global balancer. I think they should have put load in there, but that's what they said. So I'm just quoting them. We know what cloud native means. One of the key parts of that is just how responsive it is. One of the things about cloud native is to be responsive to change recovery. Right. We'll show that in a minute. Kubernetes is an interesting part of this. Why? Because it's everywhere. Everyone wants to run their stuff on Kubernetes.  There are proprietary ones. There are hardware ones. There are software ones. There are SaaS solutions. There are loads of different ones. The fact this runs on Kubernetes is a real benefit. I love the Kubernetes ecosystem, but it means we can run it ourselves really easily. We'll go through it. That's one of the big things. Not to be reliant on other services. If you can run stuff yourself and there's a low maintenance, low overhead to that, then awesome. So it does it via DNS load balancing. What does that mean? It's going to configure the DNS appropriately for the user. Depending on which strategies you want to employ. So as a user, I look stuff up. Remember what I said before? I look up a name. I guess my peer address is any one of those is valid. What we're doing with KHGB is we're going to look up the DNS name and only return IP addresses that are valid to the user. Those validity rules are becoming more rich, shall we say? There's a lot more rich that needs to happen there. To be honest right now, there's a few. We'll go through them in a minute. It's using DNS to load balance. We want to say load balance if that reader is directing the traffic to the endpoints that you want to balance. It's not doing anything clever connections-wise or anything like that, it's just at the DNS level. That is pretty good, we'll see. There's no single point of failure. This is another part of the cloud native. Single points of failure are bad. Scaling is making the MHA is a really important part. This scales across clusters, which we'll show again in a minute. There is no single point of failure. It uses Kubernetes native health checks. One of the ways that most other load balancers work is they will do PINGS or TCP or HTTP to check from outside of the cluster. That happens on a polling interval. It can be slow. It can be non-reactive. There are problems with it. It's not perfect. I think that's useful in conjunction with something like this. KHB is only based at the moment on Kubernetes native health checks. When I say Kubernetes native health checks, I mean probes. So, readiness probes really are the main ones. You know with readiness probes there can be checks inside your application. The cube checks every so often. If it fails those probes in a certain time out and a certain failure rate then your service is taken out of service. No traffic will get routed to it. KHB relies on those readiness checks. That makes it more responsive for one thing. We're running these in cluster. It makes it more granular because the checks you can make can be totally... It's up to you how you write your readiness probes. You can write them as granular as you want for your specific application rather than as coarse as... If I hit that, do I get a 200? This is the difference. You have the ability to make granular checks because what Kubernetes provides and KHB consumes. It really is. It is either one CRD or annotations. So, here we go. This is what we're going to go. One sec. Let me grab a drink. While you're staring. Wonder. At YAML. Wish there was something better than YAML for this, but anyway. So, we can see we've got CRD, there's the API group version. I love the Kubernetes API. It's just so great. You know what you're looking at. You know what spec means. I haven't got status in here. But you know what spec means. This is what you want. I want... Again, there's some work upstream going on. This uses ingress. It doesn't use gateway API yet. It will do, but not yet. So, you'll effectively declare your ingress. KHB creates that ingress object for you and manages that ingress object for you at this point. And there's some bits down the bottom around the strategy. So, the ingress part, declaration for your ingress, like what host do you want it stuff to come in? What should we recognize? Where do I want to root stuff? Path prefix roots, all these kind of stuff that you get with ingress, normal stuff. And then there's strategy. And down the bottom, you can see DNS, TTL, split brain and type. We'll talk about... Well, we'll talk about type and TTL. In a minute. TTL is such a funny one. There's a killer in it, especially over the internet. We'll talk about it now. TTL is a timeout, right? DNS is quite chatty and because it's such a key part of all networking, you don't have to use DNS, but we're humans, we want to use something we can refer to easily. Because it's such a key part, it needs to be protected. TTL is a way of having a hierarchy of resolvers. I send my request to my local resolver. I've actually got it running on my laptop. I'm using systemD.  That then roots it to other DNS servers which then roots it to other DNS servers. It's kind of a hierarchy. That TTL, that time to live is telling these DNS servers how long the authoritative source, the actual root source of this, how long it should be available for, till it expires. I can set this to 5. This is great if you're running in your own environment where you control all those caching resolvers along that chain. If you're going over the internet one thing you learn is you've got no control over TTL. It's a suggestion. You can't say I want 5 seconds because going to Google you'll have a minimum of 60 seconds. I can't remember the other one. I found the other day it was 300 seconds. Total freedom over that, over the internet. You do have to be aware of that. One thing to bear in mind. And type, round robin in this case. What we're going to do with the initial one is just to do the same thing as we had before. You can also do it through annotations on Ingress. Depends what you like. I much prefer using an API. I much prefer explicitness over annotations personally. It works. This is the secret source that's going on. There's a few components in here. Just to go through. These are all ones. The core DNS, the KHGB controller and the external DNS are all part of the KHGB deployment. This core DNS is not your cube DNS. It's not your in cluster DNS server. This is another one. There's a reason for that which we'll get to in a second. KHGB controller watches that GSLB resource. External DNS . We'll talk about that in one second. What I love about this is the ability of composing projects. Each of these projects has that own purpose. Tightly controlled, tight scope of that. Composing functionality from existing applications is a great thing. We don't need to build the world. Use what's there, especially in the CNCF ecosystem. KHGB controller watches the core DNS Ingress. It gets the address for that core DNS Ingress. It disposes core DNS as the authoritative DNS server for the zone, for the delegated zone. These may be some terms that I may have times gone. This is again the key part. DNS works like this from the very root. Delegation of DNS is a key part. KHGB controller read the ingress for the core DNS. What's the address? What's the IP address? External clients will come in to see. It writes some DNS endpoints. That's part of the external DNS API. It writes that to the Kubernetes API. External DNS is also another external DNS. You may already be using external DNS. For those of you that don't know what external DNS is, it's a way of using CRDs to tell external DNS to go and manipulate your actual DNS entries. It works with most DNS providers, cloud providers or your own or bind or whatever. This is another configuration. You may be running external DNS. This can also work alongside that. It writes these DNS endpoints to the API server and then external DNS reads those and it writes to your DNS provider some information. That information is the name servers, the addresses of these core DNS pods or these core DNS ingress. It writes those as the name servers for your delegated zone.  This goes all the way back to the .com domain or any of these TLDs. It goes back to that to know which DNS servers to delegate that request to. What we're doing here is setting up our own zone delegation inside our own core DNS. The second thing that KHB does is it reads healthy endpoints. We talked about that. It uses the readiness probes. It reads those and it writes more DNS endpoints to the API server. But with a different label or different annotation. External DNS ignores it, but core DNS with a custom plug-in doesn't ignore it. That core DNS plug-in that runs in core DNS reads these DNS endpoints and exposes it. User, look up stuff just like normal, totally transparent to the user and then looks up the address and that look up, the final look up, actually is delegated to the core DNS part of the KHB deployment. So all this change in your external DNS is setting up this delegated zone and is all done through these pieces. All you've done is set up your GSLB and some values when you've deployed KHB but that's right. Before we jump there back to my amazing demo. There's your GSLB. There's your endpoints. So there's a few things in here. This is the DNS endpoints for your zone delegation for the core DNS ingress. You can see this initial one. I want a point in my finger, but you can't see that and if I roll my mouse, can you see that? This is setting up name servers for your delegated zone, which in this case is that, and it uses some naming conventions in here so that when we have this no single point of failure both are deployment in the EU and a deployment of KHB in the US. They're both doing the same thing. They're trying to write to our DNS provider with these addresses and then they have one different one which is the local ingress endpoint. Each one of our KHB external DNS deployments updates the name servers for our delegated zone and the glue record which helps DNS to know where to root the requests to our delegated name servers. So we just show the same in the other one. So you've got different addresses, different targets at the bottom depending on which one we're in. We'll just skip this quickly. So these are the actual endpoints. These things get updated on the fly. Just showing that it's the same in both regions. And then loads of stuff here because I wanted to show you one thing. This is in round robin mode. We're going to currently have a stupidly long command to run it in an Amazon machine in Europe. And look round robin. Other load balancing strategies gloss over quickly but the one we're going to look at is GOIP because we have nearly run out of time but it's fine. He said I could overrun a little bit. So we just patch it. We change this value to GOIP. It uses MaxMind database format. And so if we're in Europe, now this is the first bit of cleverness from KHEB. We got there in the end, right at the end. If I'm in Europe, oh look I'm only sending stuff to Europe because of DNS load balancing. So I've chosen a strategy which says send clients to their closest endpoints. If you want to find out more about MaxMind GO database format it's really clever. It's not really but it is. And if I'm in the US then I get routed to the US, obviously. It's clever. And that's just that. That's just shining. So we're done. I could keep talking about it. Thank you. If you want to run through the demo it's at the bottom. There's the GitHub repo. You can see all the stuff in there. Does anyone have any questions? You've probably got a couple of minutes because I've talked too much. Do you have any questions? Yeah, you. Oh, it's a good one. It would be better to. Would it be easier to, is a different question? To set up your own anycast implementation is actually pretty tricky. For my demo. But there are other requirements around that. The clients have to understand anycast as well. Come and talk to me afterwards. Your perspective is from the... Oh sorry, hello. Where's it coming from? So your perspective is from the internet user to access your server. But what if my application is running inside of service mesh. So the application is coming to the inside of the cluster. Are you able to solve the same issue? Things are running inside the cluster. I don't think you need to. I think your service mesh will take care of that kind of for you. There may be plugins for your service mesh to do stuff like routing to local services. But this is all about your ingress. This is all about the external access to your servers. This is the layer that KHGB provides on top. So it's not able to router from the... Let's say your pagain is value. You're not able to router another cluster's pagain. Um... No. No. Come and have a chat. Let's see if you want. Anything else very quickly? It's been brilliant. Thank you very much. I've really enjoyed that. Take care. Enjoy the rest of your week.


When we're talking about skiing equipment, most people always think about their skis or the boots. One area people rarely focus on is the binding. Now the binding is the bit that actually connects you to your skis. If you've got a system ski, it's been pre-designed for you. Binding comes with a ski as a package and you have no choice in the matter. But as you start progressing and you might be looking at different types of skiing to do, we have four choices effectively. You have traditional alpine style like this one. You have a strong toe piece and a strong heel piece. Loads of safety features, that means if you do take a tumble, you release safely. But they're designed for the downhill. You have a lift up and then you have just about downhill performance. There is high ones in the market where you can put in a, like this one for marker, uses the sole ID system, where you can use a touring boot or touring style boot within this binding, but they are really all about the downhill. Then we move into the touring market. Now the touring market is split up into three separate components. First of all, we have the frame binding, which uses what looks like a more traditional alpine toe and heel. And it's great if you want to start looking further afield than just lift work. So this one allows you to have lift access and then do a small tour. You might have some hike and ride ski boots, which gives the brake at the back. So this one allows you to have a little bit of uphill use, the heel lifts.  These effectively come into two different categories, either the lightweight ones, which are dominantly plastic or the heavy weight ones, which have a normally magnesium or steel components, which tend to get stronger for the more free ride focus. There is a weight compromise with these, but performance and strength is key on the free ride side. As you move on from the first purchase of touring equipment, you can then move into pin certified ones. These are now the next level on from the frame. So you have an alpine style heel, but you have a pin up front. This one gives you a significant weight reduction on the frame binding, but without all the, but with still most of the performance. So you have a really strong heel unit that locks you well to the ski. So your performance is great going downhill. But as you put it into walk mode, you're rotating around the pins at the front. It gives you really, really good uphill performance as well. These have been a fairly recent addition to the binding market, perhaps the last four or five years and have really ignited the touring thought process for a lot of people because it gives you the same feeling of solidity you would have the normal alpine setup, but the freedom of use of a pin setup. Then we move into the pin market and pin binding. These are the lightest of all. They are great for multi day tours and they're great for producing a really light ski binding package. But you do have a site compromise on the downhill performance because we don't have the same grip into the back of the boot as you would normally get with a pin certified or an alpine style. These are all about the weight saving and the reduction of weight on the bottom of your foot. Now, that might seem a really complicated group of bindings to look at. My suggestion would be go to your local S Brigham store, pick up on the catalogue and speak to their staff who have been well trained on all of these products.


First and foremost, every command should have a diversity committee. Some may call it a multicultural committee and what it is is about diversity and inclusion and you know my favorite website www.npc.navy.mil and 21st century sailor, you know my one stop shopping and that's where you can find the diversity link. The other site that I urge sailors to look at and of course our diversity teams and our simios are already aware of that and that's the Deomi website. On there, the link that says special observances, when you click on that, that's your one stop shopping on how to find what observances are being held that month and there's a great guide on there, the guidance for organizing observances. It's the main publication for our diversity teams and it's a guide for a chain of command to look at as well for some checklist items and some recommendations for other groups, affinity groups that are recognized and tied with that observance. So the guidance for organizing observances is really the best checklist that our diversity groups will use and go in there and you know align by line how to put these observances together. But first and foremost you know what I want to talk about before we actually do this observance is why. So why do we have observances, remembrances, recognitions, celebrations? Well as I said at the opening of the show, it's about diversity and what we have to bring to the table as individual sailors from our diverse backgrounds and so we educate, we train, we learn from and today we're talking about diversity observances under the equal opportunity. But diversity also means you know our background as our ratings, as our sailors or our communities, aviation, surface, Navy medicine, submarines, special forces, CVs. What makes a great observance month and an event? Well first and foremost take strong command support. So for those that are in command roles out there you know strong command support is imperative. Quality leadership you know is the project officers or the planners in getting the different teams together very important. Planning planning planning and using the resources from the D'Yomi side or the affinity groups that they recommend or that they show on the site.  Committees have certain capabilities with other funding supports whether it's theater and lighting and so forth on the commands. And then you can talk with the chaplain's office, do some local newspaper posting. Your equal opportunity office is definitely critical and the JAG just for some legalities when you're in certain areas on bases and when you want to invite local citizens to come onto the base and do that kind of stuff it's important that you shoot some of that stuff through your JAG. Don't wait for things to happen to you, make them happen for you.


A Broadford family is single-handedly swelling the ranks of female firefighters, eight women including a grandmother, a CFA volunteers in the district and Joe Hall met them. It's fair to say in the Burgess family from Broadford, fighting fires is generational. Alright let's go girls, we've got a running grass fire. There's sisters Gabrielle, Jackie, Kylie and Dionne. We're all born and bred in Broadford so this was our way of giving back. Dionne's eldest daughter Tanika has just made the senior brigade, one of her younger daughters Lani is captain of the juniors and the other Shanay is a lieutenant. I want to follow on my mum and sister's footsteps. Then there's mum and Nan Pat who operates the radios. Kiln will control the bulk tanker ETA 20 minutes. It was how communities were touched by Black Saturday that inspired the Burgess women. We thought there's actually something that we could do here for ourselves, our families and our community. The sisters belong to four different brigades in the same district, all are highly skilled. The training that we receive is just world-class, second to none. That training was put to the test in a ferocious deliberately lit fire in Kilmore in February of last year, the maternal instinct coming out in Pat's radio communications. I do remember one of them was, have you seen Jackie at these? Jackie or us? I'm like mum, it's not the, you know. It's not the time of the place. Bring me when I'm not busy. As soon as the siren goes, they're on that truck. Yes, I think now, please God, let him come home. There's a lot of things that I've seen, the other 17-year-old girls and boys that don't actually get to experience or see. Of the 43,000 CFA volunteers across the state, 12,000 are women, a 30% increase in 15 years. In district 12, where the four women work, the number of females in operational firefighting roles has tripled since the year 2000. Little wonder then that all of the women say they're treated no differently by their male colleagues. Well, in my brigade, there's no real barriers.  It was a case of whatever the blokes can do, we can do it just as well. Do you think you can do as good a job as a blokes? We probably do it better because we're girls, we look better. The Burgess clan clearly love their work. They insist on having fun honing their skills, all the while trying to recruit other female volunteers, no matter what their age. I have a six-year-old who's claiming she's going to be the next captain. What can you say? So what would you say to other women thinking of joining? Come on in. Why not? Do it. Yeah, we can do it, you can do it. Joe Hall, 9 News.


This question makes a lot of sense since we talk a lot about money on this channel. If I talk a lot about money and entrepreneurship, it doesn't make sense that someone wants to know if I'm rich, right? Well the answer to that question is this, I'm rich according to my definition of wealth and that raises another big problem. For example, what is the definition of wealth and who is rich? I need to tell you a story to reveal what I think about this subject. You see when I was growing up as a boy in a poor household, I noticed that most people are slaves of money because they do what they hate several hours per day and still don't have any money. So I decided that I was going to try and be a free man. Then my definition of financial prosperity becomes do what you extremely love and earn a little more than enough to take care of your needs. I achieved this goal when I was around 30 years old. Now the problem you might have with me is to want to know how many millions is my net worth but don't bother yourself about that as it doesn't matter to me at all. What matters to me is to have more than enough money to take care of my needs and to do what I extremely love doing and by that standard I'm wealthy. Now that you know that I'm wealthy. How did I do it and what sacrifices did I make to achieve my dreams? That's what I'll be sharing with you in this video and before we proceed let me make it clear that I'm not speaking against the Bill Gates and Warren Buffets of this world. I'm not speaking against being a billionaire or even a trillionaire. I'm only telling you about my philosophy about money. So let's get into today's business. Books are boring. So are herbs. I'm a Nigerian and I grew up in a village. In the village where I grew up, whenever you're sick, the default medicine is a herb. I don't know which part of the world you are or whether you even know what herbs are but it's simply the mixture of various leaves that I believe to have medicinal value. Needless to say herbs are usually very bitter but whenever you're sick you have to take herbs so that you could recover from your malaria or any other illness. For most of us who are adults books are the perfect representative of the herbs because books are boring. For me the first sacrifice I made when trying to become wealthy is that I read a lot of books. Reading books has a way of changing the way you see the world and what you think is possible. Between the ages of 17 till 30 I read hundreds of books about entrepreneurship, money, relationships, psychology, marketing, public speaking and a lot more. Even today I read one book every week because my life depends on God and books. Growing up in a community where poverty is everywhere it was my reading habit that made me believe in the possibilities that I wasn't seeing around me. Herbs are bitter but they heal sicknesses. Books are boring but they change lives. Read books if you want to change your life. Yes it's not easy to read especially in the world where there are too many interesting things going on on social media but that's why I call it a sacrifice. Go against the norms. Immediately as I started reading the way I saw the world started changing and the way I thought started changing. I no longer believed in what most people believed and because of that I became the black sheep of my family. If you want to go far in life you must have the courage to go against what most people believe is right. For example, everybody in the community where I grew up believes that the best goal to pursue as a young person is to figure out how to get a job in a big company. I thought that the opposite is the right goal. Why not build a small company? Isn't it better to build a small company than to work for a big company? Walking against everybody is tough because we're humans and we need love and approval of others to remain alive but sometimes you just have to make the sacrifice of being lonely and that is what I did. I read to act. I know a few people who read books but only a few of them ever stand up to take action after they acquire knowledge. For these people, knowledge is to make their head bigger and I get it. Knowledge is sweet. It makes you feel good to know what most people don't know and it's even sweeter if you can find a few people to argue with to prove that you have deep insights but it's a waste of your time. For me, I learn not to know but to do. I don't know any single human who is as proactive as I am when it comes to getting out and trying things. Yes, I don't know anyone who feels as much as I did too but that's the sacrifice I'm talking about. If you want to play safe, you can do that by following the tested formula but I assure you that you'll never achieve anything extraordinary until you dare to take action. Would you fail? Absolutely yes. You'll make mistakes and fail but guess what? You'll become smarter and smarter until you become successful. Use people and let them use you. Some time ago, I read a book that your network determines your net worth and you better believe that.  If all your friends are poor, you'll be poor and I don't want to interpret this to mean monetary poverty instead I mean ambition poverty. If your friends don't think big, if they don't have a positive view about life, if they complain or make excuses, don't deceive yourself to believe that you can be their opposite because none of us can. Now, where the sacrifice lies is that you must be willing to say no to wrong and negative people and make further sacrifices to say yes to the right people because you need the right people to be used. I mean used because that's what it is.  This requires sacrifices. You'll have reasons to give up. Don't. I love to tell people that I fail in the business world for eight years because that's a crazy thing to imagine.  If this is not a sacrifice, I don't know what else is but that's what you have to do to achieve your dreams. I have no doubt you'll have reasons to give up. Life will maltreat you. Life will crush you. Life will frustrate you. Life will nearly disown you. Your ability to hang there and keep fighting is the only asset that will make you win.  Be a warrior and learn the rules of the game you're playing. Make a sacrifice to read and learn. Say no to the TV and social media if need be. Say no to useless meetings and parties if need be. And when you learn, you'll be courageous enough to take action. Yes, the next thing that will happen when you take action is that you'll make mistakes and fail. You can cry but never give up. Don't forget, life won't give you anything until it gets you to pay for it. If you like this video, you might like to join our new motivational channel. We call it Hunt TV. Look at the description box to join the Hunt TV. We love you.