Cool, so first of all, thank you all for coming out here today to spend your morning talking about failure, which is a light topic. So I think DHH sort of set you up and now I'm just gonna like knock you down. As was mentioned, I'm Jess Rutter and if you're the tweeting kind, that's my Twitter handle, so there we go. On to the show. All right, when I was 26 years old, I decided that I wanted to learn how to fly a plane and my friends and family were very skeptical. They had such words of wisdom as, you don't even know how to drive a car, you get motion sick, you're afraid of heights and all of this was 100% true, but I actually didn't see the relevance. You see, no one was going to clip my wings. So we're now six months in and I was on final approach for runway 21 in Santa Monica after a routine training flight. I eased back on the throttle, tilted the plane, the nose of the plane up a tiny bit, entered a flare and floated gracefully, more or less, down to the ground and suddenly the plane jerked to the side and my instructor was like, I've got the plane. I was like, oh, you've got the plane and he brought the plane to a stop on the runway and as our hearts calmed down, we got out and we looked at the damage. It was a flat tire. Now my heart rate finally was calm and with the plane safely stopped, a flat tire really isn't a big deal. You just have to tow it back to a maintenance garage and change out the tire. The runway was gonna be blocked for less than five minutes. So I was actually really surprised when my instructor said, hey, I'm gonna drop you back up at the classroom and then I'll come back to fill out the FAA paperwork. My heart rate jumped right back up. Whoa, whoa, whoa, whoa. It's just a flat tire, no one got hurt. Why does there have to be any paperwork? You see, in addition to not being the biggest fan of paperwork, I was also really worried that I had just gotten my instructor in a lot of trouble. But it turns out that the FAA collects details on all events, big or small, even a tiny tire blowout during a landing in my little four-seater plane. They wanna get as much data as possible so that they can work out patterns that can help them implement safer systems. They know that more data means they'll be able to draw better conclusions, but they also know that people really don't like paperwork or getting yelled at. So to make sure that pilots are willing to fill out these reports, they have a policy that if there were no injuries, nothing you did was illegal, and you fill out a report, there's not gonna be any punishment. Now think about the very different approach that we have to automobile accidents. When I was 12 years old, I was riding home from the Saturn dealership in a shiny brand new car. It was the first brand new car that my parents had ever purchased. We're sitting in a stoplight and suddenly we lurch forward. We'd been rear-ended. My dad got out to check on the other driver, an incredibly nervous 16-year-old boy. Now, the other driver was fine, everyone in our car was fine, and the only damage was a small puncture on the bumper from the other car's license plate. My dad looked it over and he said, well, look, I guess that's what bumpers are for.  No paperwork was filed, no data was gathered. In fact, there's not a single agency out there collecting data on car issues. It's usually handled by local agencies like the police, and they do not like it if you call them up to let them know about something as trivial as a flat tire. Heck, you can have an accident where two cars actually bump into each other, and as long as no one's injured and no one wants to make an insurance claim, this will never end up in any records anywhere. So these two different approaches have actually led to very different outcomes. I looked up the most recent stats available, which were for 2015. And for every one billion miles people in the US travel by car, 3.1 people die. And for every one billion miles people in the US travel by plane, there are only .05 deaths. Now, if you're like me, decimals, especially when you're talking about a fraction of a person can be a bit hard to wrap your mind about, so this is a bit easier. If you hold the miles traveled steady, 64 people die traveling in cars or every one person that dies traveling in a plane. Now, there is something very interesting going on here. We have two different approaches that lead to two very different outcomes, and the key difference is actually how each approach deals with failure. You see, it turns out that failure isn't something that you should avoid, it's a way to learn. Now, before we go much further, it's probably a good time to make sure we're all on the same page when we talk about failure. So what is failure? I think for some of us it might be that sinking feeling that you get in the pit of your stomach when something didn't go right and the person is yelling at you and they have the angry face and you're like, why did I even bother getting out of bed this morning? And I can absolutely relate to that. And as I was prepping for this talk and looking at what different people said, I found a lot of people that were like, oh, failure, that one's simple. It's the absence of success. And I was like, sweet, what's success? And they were like, psh, so easy. It's the absence of failure. Oh, not really helpful. But researchers, they actually have a very specific definition of failure. Failure to them is deviation from expected and desired results. And that's not bad. Now, I honestly think there's some truth in all three of these definitions. But that last one, deviation from expected and desired results, that's something that you can actually test and measure. So we're gonna stick with that one for now. Now, I couldn't actually find any definitive data on this. But I think that as developers, we have more results that deviate from our expectations than just about any other group of people. So you'd think that programming would be the perfect place to learn from failure. But one of the few places that I could actually find people routinely, most of the time, capitalizing on failure, was in video game development. And one of my favorite examples of this is with the game Space Invaders. Do you guys sort of know the game Space Invaders? So it's the old arcade game where you control a small cannon at the bottom that's firing at a descending row of aliens. And as you defeat more aliens, they speed up making them harder to shoot, right? No, that actually was not the game. That's not what it was supposed to be. The developer, Tomohiro Nishikato, he had planned for the aliens to remain at a constant speed the entire time. No matter how many aliens you destroyed, there would not be a speed increase until the next level. And he wrote the code to do exactly that. There was just one problem. He had designed the game for an ideal world. And I don't know how much you know about 1978, but 1978 was far from ideal. And he'd actually placed more characters on the screen than the processor could handle. And as a result, the aliens sort of chugged along at first and only reached their intended speed once enough of them had been destroyed. Now Nishikato had a few ways of dealing with this. He could shelve the project until processor speeds were fast enough. And that might seem silly, but maybe he had a vision and he was not gonna compromise. He could also have modified the game design, put fewer aliens on the screen so that it could run at the constant speed that he wanted. But instead of being rigid and insisting on his original vision, he decided to have people test it out. And they absolutely loved it. They got so excited as things sped up, they would actually project emotions on the aliens.  I'm taking them out and they're trying to speed up because they know that I am about to kick their butt. And it was so popular that he kept that in the game. And the failure was actually responsible for creating an entirely new game mechanic, the difficulty curve. So before this, games would always be the exact same difficulty for an entire level. And it wasn't until you got to the next level that things would actually get more difficult. After this, all bets were off. Things could get difficult at any point whenever the developer pleased. Now, I don't know if the developer here had read the studies, but he was actually capitalizing on a lesson that I see time and again in the research about failure. Failure is not something to hide from. It's something to learn from. In fact, it turns out that failure actually presents a greater learning opportunity because there's more information encoded in failure than in success. Think about it. What does success usually look like? A check mark, a thumbs up, a smile from a manager. And what did you actually learn? Well, there's research on this. Research actually shows that people and organizations that don't experience failure become rigid because the only feedback that they get tells them just keep doing the exact same thing you're doing. Don't make any changes because you're already winning, buddy. Don't change anything. Failure, on the other hand, looks a whole lot more like this. Okay, look at this. Look how much information there is available in this error message. If we read it closely, we can figure out exactly what went wrong. We know which line in the code has an issue, and if we have some experience with this particular error message, we probably know what that issue most likely is. And even if we've never seen it before, we're just a quick search away from pages and pages worth of information about this particular failure. Now that we've had an experience with an approach that didn't work, with a bit of effort, we could actually figure out how to write something that does work. Now, video game development actually has a long and honored history of grabbing hold of mistakes and wrestling them into successes. In fact, the concept of exploiting your failures to make your program better is so important it actually has a name. They call it the good-bad bug. Now, having space to learn from their failures, that actually came in very handy for a group of developers that were working on a street racing game in the 90s. So the concept for the game had players racing through city streets, and they were being chased by cop cars, and if the police caught up with you and pulled you over before you got to the finish line, you lost. There was just one problem. The developers got the code for the algorithm just a tiny bit wrong, and instead of law-abiding police officers trying to pull you over, you had these super aggressive cops trying to slam right into your car, and they would do it at full speed, no matter what you did. The beta testers, they actually had way more fun avoiding the cops than they'd ever had with the racing game. And as a result, the entire direction of the game was switched up, and the Grand Theft Auto series was born. So just, I wanted to think about that for a moment. The core concept of the best-selling video game franchise of all time in history ever would have been lost if the developers had panicked and tried to cover up the fact that they screwed up the algorithm. They made a mistake, but instead of freaking out, they thought, all right, let's see what happens. And they cashed in. Now, there are actually some larger programs where hundreds, if not thousands of hours of work have already been done by product leads and designers and business people before a developer ever gets to write their first line of code. And in game development, the work is encapsulated in a document called the game design document. Now, the GDD is considered a living document. However, it's actually a pretty big deal for changes to be made late in the game. It means that tech requirement pages need to be redone. Art pages need to be redone. Release states have to be pushed back. Budgets might be off. You get the picture. It's a big deal to change this. But that was actually the unhappy reality that the Silent Hill developers were facing. So they'd started out building the game to the GDD specs. But there was one tiny problem. Pop-in. You see the PlayStation's graphics card? It couldn't render all of the buildings and the textures in a scene. So as you walked forward, buildings would suddenly pop into existence and blank walls would magically have a texture. And as you can imagine, that sort of like, oh hi, trees suddenly, distracted people from the game. And a horror game is very dependent on atmosphere that sort of pulls the player into the game's universe. So this was kind of a game breaking issue. Now it would have been easy for every single person involved to start pointing fingers at everyone else. After all, everyone had sort of played their part. From the designers who put just one or two more buildings in the background to make it interesting, to the tech team that decided to make it for the PlayStation instead of the more powerful 3DO, to the business team that determined the release date. There was not a single individual anywhere along the line that had made an obviously bad call.  You see, the entire system had failed. But instead of running from the failure, the Konami team sidestepped it. They found a workaround. They filled the world with a very dense, eerie fog. And it turns out that fog is actually pretty lightweight for a graphics card to render. So now it obscures distant objects, which means you couldn't really see buildings and textures on the horizon popping in anymore. And as an amazing added bonus, it is really, really, really creepy. In fact, it was so creepy that this fog became a staple of the Silent Hill series long after graphics cards had caught up and become powerful enough that pop-in wasn't an issue anymore. So that's like another example of success being ripped from the jaws of defeat simply by embracing your failures. Now, these examples from the programming world actually helped to illustrate what was happening at our more high stakes example in aviation and automobile accidents. You see, the aviation system saved so many lives because accidents are treated like lessons we can learn from. Data is gathered and aggregated and patterns are identified. If an accident was caused by a pilot being tired, they never just stop right there. They look at pilot schedules and staff levels and flight readiness checklist to determine what contributed to that exhaustion. In contrast, who do we usually blame for road accidents? Yeah, the driver. Oh, she was reckless. That dude, he does not know how to drive. In other words, airplane accidents are always treated as failures of the system while car accidents are treated like failures of individuals. And with all that judgment going around, it's no wonder that people spend so much time trying to cover up their errors. I definitely stopped at that stop sign. That's the guy who went through it. I mean, they spend time covering it up rather than just acknowledging the failures and learning from them. Now, not everyone in this room is a pilot, but I actually think that we have a lot to learn from how aviation handles failure. If we're willing to use a system to track and learn from our failures as we write code, we're actually gonna be much better off. But that sort of begs the question, what should that system look like? Now in broad strokes, I think that there are three very important pieces that this system would need. And the first one is to avoid placing blame. We're gonna need to collect a lot of data. And then we're gonna have to abstract patterns. So step one, make sure that you understand that you are not the problem. Cool, that is much easier said than done, right? I mean, learning not to beat yourself up over a failure and mistakes, that's probably like a whole talk in and of itself or like a whole lifetime of self-discovery and work. But with aviation failures, one thing to note is that they never just stop at the top level of blame. So there was actually a case where the pilot made a critical error by dialing in the wrong destination in the flight computer and it caused a wreck. So on the cockpit recording, they could clearly hear the pilot yawn and say, I'm super excited to finally get a good night's sleep. Now it would have been very easy for the researchers to stop there and blame the pilot for being tired. But for them it wasn't enough to know that he was tired, they actually wanted to know why. So they verified that he had had a hotel during his layover. But that wasn't enough. So they verified that he had checked in. And then they looked at the records of every single time that door had opened and closed so they could establish the maximum amount of time that the pilot could possibly have slept. And even then, they didn't just say, okay, we've shown that the pilot could not have possibly had more than four hours of sleep total tonight. They looked at the three letter flight computer readout and they were like, wow, you know, now that we're thinking about it, that's an incredibly confusing interface if you're tired or distracted, which a cockpit is a pretty distracting place. Now they're always willing to point out where individuals have contributed to a failure. But they also wanna focus on what went wrong with the entire system. So if your takeaway from failure in code or anywhere else in life is anything like, I'm dumb, I just can't learn this, this probably just isn't my thing, you are absolutely missing out on all the best parts of failure. Now I understand not everyone is gonna be at the point where you can kind of quiet that inner critic yet, but if you just spend some time trying to ignore it and work the rest of the system, given enough time you're probably gonna find that the voice in your head starts to contribute helpful insights rather than just criticism. Now step two, document everything. Even things that seem small. Heck, I think you should document especially the things that seem small. So my flat tire on the runway in Santa Monica was a very small error. But just as we saw with the Silent Hill example, a lot of those small errors and missteps can start to roll up into a major problem. And catching those problems early on in course correcting is going to help you avoid major meltdowns. So how should we document things? I'm a big fan actually of paper documentation, but as long as you have some sort of record that you can refer back to, the form of documentation is really going to be up to you. You should definitely include details about what you were trying to do, the resources you were using, whether you were working alone or with other people, how tired or hungry you were, and obviously what the outcome was. Get specific when you're recording the outcomes.  Because that does not help. Trust me, I tried. Now look. The final step is to make use of all that data that you've been diligently gathering. Imagine how powerful that data is as you go through it and start extracting patterns for when you do your best work and when you do your worst work. Instead of vaguely remembering that you struggled the last few times you tried to learn how to manipulate hashes in Ruby, you can actually see that you were only frustrated two out of those three sessions and the difference between the one where you felt good and the other two is that you were well rested for that one. Or maybe you noticed that you learn more when you pair with someone or when you have music playing or when you've just eaten some amazing pineapple straight from Kona, Hawaii. On the flip side, you might discover that you don't learn well past 9 p.m. Or that you're more likely to be frustrated when you're learning something new if you have not snuggled with the puppy for at least 20 minutes prior to opening your computer. And that is a very good thing to know because it's a lot easier to identify the parts of the system that do and don't work for you when you actually have a paper trail and you're not guessing. And you're also gonna have a really nice log of all the concepts that you're struggling with which if anyone in here has ever said, oh, I'd love to write a blog post but I just don't have any idea what to write about, this log of all the things you're struggling with, that's your blog post source right there.  Sort of. The data got where I was sending it but it kind of ended up in the URL which was a bit weird. Cool. You actually have a very well-defined problem to research and it won't be too long at all after reading some form documentation that you realize you were using the get action on that form and get request, put the data in the URL. Post requests are the ones that keep it hidden in the request body. So now you're just gonna need 20 minutes of puppy cuddle time and you're ready to go fix that form. Now I've been focusing on how individuals can learn from failure today and the thing is this is also incredibly important for teams. So in the research on failure there's actually a pretty famous study that looks at patient outcomes at demographically similar hospitals and they found a very strange thing. At the hospitals that had nurse managers that were focused on creating a learning oriented culture instead of a blame oriented culture there were actually higher rates of error but they also had better patient outcomes and they were like that's weird. Here we have hospitals where people are encouraged to be open about mistakes and they make more mistakes but patients are better for it and so they dug in because that was not what they were expecting to find and what they found was that it was just that people were more willing to report their mistakes which meant that the hospitals could find what was causing the mistakes and correct them which meant that patients had better outcome.    And the underlining issues actually never get dealt with and if you show me a dev team that has a zero tolerance policy for mistakes I'll show you a dev team where engineers spend a good portion of their time covering up mistakes rather than writing good code.  Now look like everything else that you try the process that I'm proposing may not actually work perfectly for you the first time around. Thank you. At the risk of going a tiny bit too meta just figure out what about the process isn't working for you and see how you can adjust it. That's right. You can learn from the failures that you're learning from while you're trying to learn from your failures. And as you get more comfortable gleaning info from failures you're actually gonna find that every single bug is actually a feature as long as you can learn from it. Even if you end up deleting every single line of code and starting over again. Thank you. So that's a great question. So it's how to communicate that failure is something to learn from from the engineering side to the business side that might have a different perspective. And that's absolutely gonna be tough because if you are at a corporation that sees like a bug in production as the end of the world and you might lose your job because of it then that's gonna be a problem. And it's gonna be very hard for the engineers to be willing to fail publicly if they're gonna get punished even if it's not from their managers. So part of that is it's up to everyone to try to educate the other people on the team about how this is part of the engineering process. But that's obviously, your work is building things and like fixing bugs and maybe not fixing people. So it's unfortunate. I would say like if you're in management working hard to try to like help business people understand that is great. If not then buffering your team as much as you can from being punished for mistakes while allowing them the space. So maybe it's like fail internally, be open about it, have blameless postmortems but put a nice veneer on top for the C level people that need to think that everything's perfect and that's the best way to do things. Sure, so the question is about the difference between an early career developer making a mistake and being visible and a late career developer making a mistake and being visible. So I actually had a similar question where someone was like as a woman part of my career growth has been looking like I know 10 times more than the next person that was applying for the job and being open about failures could tank my career. What do you recommend? And I was like, I'm not an expert.  But I would say that I think it's great when late career developers are open about failures because that gives room for early career developers to be open about their failures. I'm not embarrassed by not knowing something so when I've been in code readings or anything like that and they're like, oh, are there any questions? I'm always more than happy to raise my hand and say, I didn't understand that thing because for me it's not embarrassing not to know something and so I kind of like am willing to take one for the team and afterwards people will come up and say, oh, thank you so much for asking. I thought I was the only one that didn't know. And so I think part of it is like if you're late in your career and you're comfortable with where you are and you're willing to show that it's okay to fail, then take that one for the team. If you feel secure in where you are in your career, take that one for the team because all of us are going to be much better engineers by being willing to fail and learn from that then we will be covering it up.  You could keep a private diary of what's gone wrong and how you've learned from it. You don't have to publish it publicly but I would say like don't let other people keep you from being the engineer that you wanna be and if the way that you get there is by trying stuff out, writing the code, breaking the code and then going back and fixing the code, don't be afraid to do that.  Like don't be afraid. So the question is about being so cool with failure that people no longer worry about making mistakes. So they're just like, whatever. So I mean one thing is I think most programmers want to build stuff that works. So it's gonna be very rare that you find someone that's like, I don't care that it's not working for people. Whatever, it's lunchtime. Like most of us have, we're like, my baby is broken. My code doesn't work. Like we put all that pressure on ourselves. In the research just at least in Western cultures, I can't speak for other cultures, but Western cultures like put such a stigma on failure that it's very rare that you would find someone that's gone too far the other way. And it's much more likely that you're gonna end up in a place where people are covering up mistakes and making things even worse because of that. I think if you do find yourself in a situation where, so sort of I think the tell if someone is getting a little too comfortable with mistakes as if they keep making the same one over and over again, and that's not what this is about. This isn't about being like, oh, stuff breaks, whatever. It's about saying like when stuff breaks, what can I learn to do it better next time? And so if you've gone too far on the like laissez-faire side of things where you're like, no, nothing's a problem.  You wanna like, let's come up with a way that we can make sure that doesn't happen anymore. And yeah, so you just course correct there. Yes? Yeah, I mean, so the question is sort of finding the balance between the extra time cost of documenting things versus the value that you get when it works out well and you get some value from it. And at the risk of this being a cop out, I'd say you're gonna kind of have to experiment and find what works best for you. I don't know if anyone in here has had the experience of like learning something, writing a blog post so you don't forget it or like writing some notes. And then six months later being very confused on a topic, Googling and finding your blog post that you wrote six months ago when you had the problem.  I know how to explain it to you. Here you go. And so like in those moments, that time you took to write that blog post, you either learned it better and were able to do it more fluently next time or you have this great resource written by you for you for the next time you're confused on it. And I think in those instances, people really see the value in like the documentation and the learning. Certainly if you spend like a couple months and you're like, I write a lot of stuff down and nothing's getting better for me, then yeah, scale it back and find a different thing that works better. Sure, so concrete steps that senior developers can take to help junior developers sort of document and learn from their failures. Yeah, I mean, it's probably gonna vary by individual and vary by team. I think certainly writing tests is great, like without getting too pedantic about it, having at least some test coverage, forcing you to like think about, especially as a junior, the system that you're about to architect, forcing you to have a safety net for when you make a change later and don't realize that it's gonna break things. Tests are amazing. I'm a big fan of having junior developers and senior developers alike, right? Blog posts and documentation, because it's like we get so much, especially, I mean in this room, I think most of us are Ruby and Rails developers, probably working with other open source libraries and other things, and like maybe we're not all at the point where we can sit down and write a gem or a framework that's going to like help tens of thousands or millions of people, but we can write a blog post that's like, I struggled with this, this is how I got around it, this is what really made it clear to me and that helps the person that's learning kind of solidify what they've learned and then is just like this beautiful gift for the rest of the community. I can't tell you how many times like people have been, so I work at a school where we teach developers and there've been plenty of times where students have Googled an issue and it's been resolved by a blog post that a student like two, three, four semesters earlier wrote when they were going through that same issue and so it's both a great way for juniors to learn as well as like a great gift to the community by people that can't necessarily give through like writing code yet. I mean that, oh sorry, the question is if you've experienced a situation where you thought covering up was a mistake was better than admitting it. I mean all the freaking time, because I don't like being yelled at, I like have such an irrational fear of authority which has zero to do with how my manager actually interacts with me, but every time he's like, oh do you have a minute to talk? I'm like, oh my God, this is it, I'm being fired and everything's the worst and he's just like, oh I wanted to tell you you did a great job on that feature launch and I'm like, so when is my last day? So, I mean the thing is that I've never actually found a situation where it would be better for me to have hidden it. I've found situations where I was really scared and the thing that I thought was going to be bad which is like a manager yelling at me actually did happen because I worked at a crappy place where that's how they treated small mistakes like a typo in a report that the client had said, oh we never look at that because there's just too much data in there and then my manager's like, you made a typo you don't care enough and I was like, yeah it's not the 18 hour work days, it's my level of caring. And so I think like there are situations where it feels like the better option is to hide the mistake, but I've never actually seen it where there's like long term value from hiding the mistake and I mean I may be privileged and lucky in that that it's like never like killed my career. I'm sure like being seen as someone who makes typos in reports certainly didn't help me at that ad agency, but I left there and learned to code and my life is better so screw those guys. Anyone else? Awesome, you guys have been fantastic, thank you so much. Thank you.


Hi everyone, Joe for jaskiescasebreaks.com. Happy Tuesday coming at you with a little absolute basketball. Five box, it's a half case, yeah, 10 boxes per case. Half case break, random team break number one. One spot gets you two teams in this break and all cards ship. So let's double you up. If you have a little rooftop next to your name, that means you won that spot in the filler for this absolute break. So thanks for taking a chance on the filler. Again, all 30 teams are in. Let's roll it, randomized names and teams, six and a four, 10 times for names and teams. One, two, three, four, and 10th and final time. After 10, we got Mike Tower right on top, pole position, and Kevin in the 30 spot. Six and a four, 10 times for the teams. One, two, three, four, five, six, seven, eight, nine, and 10th and final time. After 10, we got the Phoenix Suns down to the Denver Nuggets. Mike with the Suns, Eric with the Bucks, Faith with the Golden State Warriors.  So let's sort by column B by team, and feel free to trade. While you're thinking about trades, we're gonna pop open this case of absolute basketball, and we're gonna see which five boxes we're gonna do. Then we'll pause the video for the trade window. When we come back, we'll see if there's any trades, and then we'll have the break itself. All right, so there's five boxes on that, and you can see in the top camera too. Five boxes on that side right there, let's select a die, select that one right there. One, two, three for the left side, four, five, six, four the right side. Two, left side. So this side we will save for next time. I think we already have the next half in the store. Need a sharpie. So this is break number one, right? So we'll put right over Luca Doncic's arm there. Number two. So we know that's for random team two for next time. We should be able to run this back if you want to. Remember, one spot gets you two teams. All right, now let's flip back over here. We're gonna pause the video when we come back. We're gonna see if there's any trades, and then we'll have the break. Looks like wizards, calves up for trade. Looks like Houston, Portland up for trade. So maybe we'll have some trades when we come back. Stick around, B-R-B. All right, welcome back folks. There was some trade chatter, but in the end no deals were done. So here on a Tuesday the 19th, 2020, 2021, absolute basketball, five box, Ram team number one. Here's the final print out here. Thanks everybody who got into the action. Second half is in the store. Opening night of the NBA season. So if you're feeling the hoops, we got plenty on jaskyscasebreaks.com. So get in on it. Tis the season, the fall season. Spooky season. All right, here we go. Luke Kedachich on the box. That's what we're looking for here, per master box on average. I think there still should be the glass case hid. They still do that, right, for absolute? Steve, nice, got in on it. Love the prices. Love the prices. I don't like hearing that. That means our prices are too low. Should we raise the prices? Maybe we should. Out of 199, 23 out of 199, that's Kira Lewis Jr. for Mike and the Pelicans. We got Devin Booker. Remember all card ships, you'll be getting all of those. Kira Lewis Jr. And then we got Killian Hayes, rookie threads relic for Detroit. That's gonna be for Michael. And the autograph is Tyrell Terry. Couple pieces of the basketball there. That's 105 out of 199. That's gonna go to James and Dallas. And we got Derek Rose to 25, that is, for the Pistons. Andrew Wiggins, Devin Vassel. And we've got Tobias Harris, tools of the trade, dual relic for the Sixers. Philly, that's gonna be for Travis and the Sixers. And we've got Spud Webb, autograph, nice. Retired autograph, six out of 49 for Atlanta, Adam. Adam B. with the ATL. It's pretty strong. Rex, do I like those little Debbie's Christmas tree cakes?  All right, next box, redemption on the bottom. Have I pulled the glass insert before? I've pulled them in every single absolute case that I've done. Absolute football, absolute basketball. I think they're usually, they're one per case, generally. So yes, I have pulled many before. We've done many of these absolute breaks over the years. Got Jason Tatum to 99 for the Celtics. That'll be for Ryan. There's Udoka and Cole Anthony. Josh Akogi, established threads relic for the T-Wolves. That'll be for Dennis. And the redemption, that could be the glass. The glass is usually a redemption. We've got, no, this is tools of the trade, three swatch signatures, level two. Ooh, LA mellow ball. Charlotte Hornets, Brian Frank, got a spot in the filler, got randomized Charlotte, gets the lamello. Congrats, congrats. Very nice. All right, onwards. We've got Zeke Najee to 199 for Denver. That'll be for Kevin. Desmond Bain, Onyeka. And we've got Isaac Okoro, rookie threads relic for Cleveland. That's gonna go to Zachary. And back here is Moore Hornets. It's Caleb Martin, rookie autograph, 34 out of 49. So the Okoro, Cleveland, this is for you, Zachary, with the Cavs and Brian Frank. Wow. Bellinger. Wow. He won bomb to tie the game. In the bottom of the eighth, a belly bomb?  I literally was playing hotel California while I was setting up the shop today. And I'm like, I'm manifesting my home run. Oh my God. He's back. Bellinger's feeling good. All right, I guess the Dodgers aren't done. All right, Charlotte Hornets, Brian Frank. Two down, three more to go in this five box break. Second half is in the store. All right, so there's some life. Yeah, some life for the Dodgers, Carl. I've been wondering where this offense has been for a little bit. All right, well, now let's see if step one, tie the game, step two, try to win the game. Let's see if the Dodgers can finish it off. There's CJ Ellaby to 199. Ooh, Chris Taylor with a base hit. Portland, that's gonna go to Brad. Got Patrick Williams, pretty solid rookie for Brian and the Bulls. Tyrese Hallibur, another solid rookie here for the Kings, Michael Robinson. And a two out of five, Daniel Latouru. Nice piece of the basketball here for the clip show, Adam Bernstein. There you go, Adam. And Cole Anthony, triple relic and autograph to 199. And Carl's letting us know that that Lamello Redemption, it's out of 25, so it should be out of 25, Brian. That's awesome. If you end up redeeming it, Brian, definitely snap a picture or tag us on social media at Jasper's Breaks, we'd love to see it. We got Xavier Tillman to 25, Tyrese Halliburton again. Carl Anthony Towns for Minnesota, Dennis. And Jared Allen, piece of his jersey, established threads, relic for the Cavs. That will be for Zachary. This is for you. And back here is Robert Woodard. 15 out of 49, rookie autograph for Sacramento, Michael Robinson. See if we can find his teammate in here. Actually, we do have a, my Lakers are playing or hosting the Warriors a little bit later tonight. But we've got an early game here. We've got the Bucks. Are hosting the Brooklyn Nets. And early in the third quarter, looks like Bucks up 71-62. Did the ring ceremony and everything. Paul George 75 for Adam and the Clippers. Miles Turner. Kobe White.  Houston, Brad. And we've got Old Buck, Bob Dandridge. 13 out of 49. That'll be for Eric C. and Milwaukee. Isaac Acoro to 49. We've got Carl Anthony Towns. We've got Miles Turner, your two base cards. And your hits. We've got Obi Toppin, piece of his jersey, rookie threads for New York. They look pretty good last season. Hopefully they keep it up this year. That would be hashtag good for the hobby with all the youngsters on there. Michael Robinson with the Knicks. And Obi Toppin, there it is. Quad Relic and Autograph. 60 out of 199. Michael Robinson won that spot in a filler, got randomized, the Knicks gets rewarded with the Obi Toppin, Quad Relic and Auto. The fifth and final box. Good luck everybody. All right, there's Darius Garland for Cleveland. We've got Steph Curry using LA tonight and Jason Tatum. Two relics, the relic is Aaron Naismith, rookie threads for Boston, Ryan Kasem. And the Autograph is Devin Vassel. Six way relic, Autograph 105 out of 199. Goes to Kym and the San Antonio Spurs. And the final mini box. Thanks everybody for the fill. Second half is in the store already. Check it out, one time you won out of 199. John Morant, second year John Morant. For Memphis, that'll be for James. Your two base card. We've got Xavier Tillman, Vernon Kerry Jr. And your final two hits. It's Patrick Beverly, two color dual relic. Tools of the trade. Dual relic for Adam V and the Clip Show. And there's Obi Toppin again. Rookie Autograph, 15 out of 49. There you go, Michael Robinson. Strong finish for the Knicks in this break. And there you go, ladies and gentlemen. What a really nice break. Seven of hearts helping us out there. That was random team break number one. Again, the second half is in the store if you want to run this back. I'm Joe for jaskiescasebreaks.com and I'll break more hoops with you next time. Bye bye.


Good morning and welcome back to Tuesday Entrepreneurship. My name is Joy Muchacho. You are watching Why in the Morning on the Y254 channel. Remember you can watch us on channel 3076 on DSTV. And if you want to join us and to talk to us and also to ask us some questions and even share your opinions, you can do so on our social media pages on Facebook and Twitter. That is Y254 underscore channel. And then on Instagram, you can find us on Y254, actually that is Instagram Y254 underscore channel. And Facebook and Twitter are just Y254 channel.  I myself can be found on Joy underscore Muchacho on Twitter alone. And today we have a wonderful entrepreneur on set with us today. Her name is Sarah Nganga. She is the CEO and the founder of Prime Holiday Adventures. Please, let's welcome her. Thank you Joy for having me. Yes. Yes, I'm the founder of Prime Holiday Adventures, limited and I'm the head of reservations and I'm also actively involved. Wow, head of reservations and also actively involved. Yes. So you're Mambo Yorta inside that whole scene. Sure, sure. So that operation cannot happen without you, it seems. Yeah, I have to be there. I have to check on everything that is running smoothly. Yes. Okay. So maybe we can get into a little bit, talk about what Prime Holiday Adventures is all about. What is Prime Holiday Adventures? What is it about exactly? Yes, Prime Holiday Adventures is basically a travel company. We do our main destinations are Tanzania, Uganda, Zanzibar and Rwanda. We do safaris and we also do excursions, climbing excursions, Mount Kenya, Kilimanjaro. Wow. Basically, if you want to travel, you can always count on us. We can always count on you. Yes. That is interesting. Sure. There's something that I found quite interesting when I was learning about you when we're talking. You volunteered elsewhere first and then once you're volunteering there, you actually coordinated a trip for other people to come and it worked out perfectly, so impressively. And that's where you now got the interest to start. Can we talk about how that started off? You're doing the volunteering and then finally choosing to open this company. Sure. When I was at the college, when I was pursuing my bachelor's, so I did volunteer at one of the companies, travel companies, that is Steps Adventures. And when I was there, I was really active. I managed to plan a trip that was, it was so new to me, but I coordinated everything so well and the turn up was quite interesting. So I did it and it was successful. That pushed me too and the idea of Prime Holiday Adventures came about and I thought I can really do well when I have my own company. So after that hike, was it a hike? Yes, it was a hike at Old Doño sub-book. Old Doño sub-book. Yes. So you planned a hike and it turned out quite well. Then your interest to start off your company in tourism came. Sure. Okay. I thought a little bit about what you studied in college because as we were talking once again, you told me that you actually studied something completely different and then did a diploma in tourism after that. So what you started off with was what? I enrolled for bachelor's degree at the University of Nairobi, Amt Conflict and Peace Studies. So when I was a third year, that's when I got the volunteer when I was during my long holidays. Right. Amt Conflict, I still have passion about it. Actually, I'm a member of the Lendo Initiative, I'm a peace scholar and I feel I can also push it forward when it comes to peace and peace related issues. So the idea of having a travel company despite doing a different course from the University came about when I was an attache. So this pushed me forward because I needed something I can also do for myself and I can also build my own brand and push it forward as myself. So when I cleared before I even graduated from the University of Nairobi, I pursued a course in diploma in tourism travel. So I used to do part full time at the University of Nairobi and also part time at the college in town. So I used to do both of them. It was quite challenging. That sounds really challenging, that sounds quite heavy. So you're balancing two different colleges. You can imagine. Wow. You sound like quite an ambitious woman. I was.  This is why you're the founder of such a big wonderful tourism company. Yes. So I used to do both of them and I did the, I sat for the exam for the tourism travel and at the end of 2016, that's when I graduated from the University of Nairobi. Second class upper division, Bachelors of Arts in Crypto and Peace Studies. Wow. Nice. Good job. Wow. Okay. This is so interesting because sociology I can understand. You can, through sociology you can learn about the society and the community around you. Yes. And peace, conflict studies, that sounds so interesting. Why, why did you pick that to begin with? Where, which direction were you going when you picked that in university? Yes, you know Joy, peace is not just the absence of, the absence of conflict. Yes. But basically peace is something that everyone, for you to be here, you have some peace of mind and your environment. So all that I considered, I wanted to pursue in that direction whereby I can actively involved, be involved in peace building and reconstruction, resolutions towards the, I remember taking you back when, sorry, when I was, when I got into form one. Yes. So that it was beginning of 2007, 2008 post-election. So I passed so well but unfortunately I didn't go to that school where I was invited to. So it makes me, it made me feel bad about it. So when I was still at high school and I needed to do something that really would bring peace to the society. Yes. Yes. Wow. That's what pushed me to that direction. Right. So that's why you chose, why you chose that. Yes. That's why the two different interests came up. I'm sure. All right. So let's focus now a little bit more about prime, to prime holiday adventures. Yes.  Why prime? I know prime is, you know, something to describe top, the creme de la creme. It's something that describes, you know, the best of something. I'm sure. Is that the reason why you chose the name prime or is there a different reason? Definitely. I wanted to do something outstanding, something different and more interesting, this name came about when I was still in campus, when I was in the library and I would just write the mission, the vision of my company. So this is something that came about unfolds every time as we went by. Yes. I see. Okay. And when it comes to this particular business venture, I would say that it's mixed. I think that it's both men who do it and also females who do it. Am I wrong or right? You're right. Yes. And so how is it like when it comes to challenges and competition? When you walked in and you decided that you're interested in starting a tourism company, what were the first challenges you faced in the very beginning? Yes. I have been graduated at 2016 and I opened the company early 2017. Also it was quite challenging getting into the system with the vast knowledge in tourism, but a little bit of exposure. So I took the challenge really positively and decided to venture in it because I had passion for it. Remember, I always dream about it. Yes. I always, it was always my dream to have something that I can build up myself. So this pushed me and this company came about when I was just, during my first job as a hotel manager. Okay. It wasn't really my thing. So when I was just working, I just needed to get out of my comfort zone and start up something. Start up something. Yes. What do you mean get out of your comfort zone? Were you uncomfortable at first? Yes. I had not done anything to do with hospitality and I was in a hotel. So it wasn't really my thing. So when I was there, I wasn't really comfortable. I was just there because I was getting a pay and I wouldn't count on it for so many years to come. So I needed to start something. Okay. Yes. So when it comes to, you said that you started, actually was it in 2017, you said, 2016? Yes. Yes. And you said that some of the challenges were because you had to come out of your comfort zone, that you were not well versed when it came to the world of tourism. And that's why you had to go into a diploma and learn a little bit more about tourism before you could actually take the reins and start. So we discussed about some of the challenges you had when you started off. What about throughout your journey? You know, sometimes the ups and downs when it comes to business, lots of times. That's a definite, that's a given. What about throughout your journey? Now that you finally learned the ropes, what were you coming across that you found quite difficult to deal with? Yes. Something that was really challenging, it was of course the capital. I had not really saved much to do it. So I started with that little capital, but because I was very focused and it needed to work, so I would do anything I would travel up and down. I would use the little savings I had to make sure that I've done the registration of the company, I've got the licensing and all that. And all that, I see. And would you, are you, I don't know, when it comes to tourism companies, do they tend to work together sometimes? Yes. They do? Sure. Yes. So I started in tourism travel, a director of one of the companies held my hand and made sure I was getting on track. Why are you saying someone? Yes. You should be saying all their six names if they held your hand. Yes. He's Edward. Yes. Yes. The director of Kenya's tourism travel. Okay. Yes. And so that's a person that actually stood with you and showed you how to do things. Yes. That's interesting. And so what would you like to work with when it comes to, how can I say, helping people? Do people who are starting off in the tourism business reach out to you at any point? Yes. I think that's an ideal thing to do because actually when you do, it's interesting even helping somebody learn or get into the tourism field and that they can always counter me because somebody held my hand so I should do this more often. To people who are coming, especially the young entrepreneurs who are coming on board who really want to do this for themselves, I think it's not that hard. If you believe in yourself and you have the passion and the zeal, you can do it. You can do it? Yes. Okay. And you know when it comes to new entrepreneurs, maybe we can talk about some of the mistakes that they make when they're starting off in business and how you can advise them in ways, maybe three important points that you can advise in ways to start off business properly. Yes. If you want to start something, you must have a direction. Okay. Yes, and the destination. So if you have the directions and you know how to maneuver, you definitely get there. But if you want to just get there, but you don't have a direction on how to move the next step, it becomes challenging. Another thing is that you need to be patient also and let things unfold. Put more effort, but don't push, don't rush to do something, but take your time and it will definitely grow. So patience, having direction for your destination and having patience that will grow. That is great. That's great. And so thank you for that advice that you've given new entrepreneurs and I hope that they've been able to take that down. Once again, if you'd like to interact with us, do feel free to do so on our social media handles. Once again, that's Facebook, Twitter, Y254Channel, that's hashtag, Tuesday Entrepreneurship, hashtag Y in the morning, back to Ms. Nga. Yes. So I saw that you came with your sister and I could see you guys whispering together in the background. It seems like you're quite close. Yeah, we are quite close. Actually, she does journalism. She does journalism. Yeah, at St. Paul's University. All right. So this kind of motivation to her. Okay, that's great. I carried her along with me. That's great. Yes. So the reason I'm asking is because what is it like for your family? I'm sure they're supportive, I'm sure they're proud of you. And so what was it like when you first went to your parents and told them, you know, I know you didn't want to, you don't have to ask them for money. Sure, sure. But it's good to run ideas by our parents, whether or not we need their money. So you went, I'm sure you told them, so I want to do this thing. What was their reaction? I think I have a very good family. You seeing here, it shows that they support me a lot, especially my parents. So when this idea came about, they encouraged me a lot. Even right now, they just want to see me go somewhere. So they're really supporting me and I'm thankful to them. That's really beautiful. Sure. And your sister as well, that's great. So back to the business, what exactly is different about your holiday adventures? When we talked, you mentioned something to do with packages. What is so special about your packages? Yes, Joy, every time I'm doing this, I want to do it differently from other companies to stand out as well. So we package our packages differently, our travel packages. So understanding, I have knowledge in tourism and different people have different attitudes towards destinations. So different perceptions and mentally impressions. So you really want to understand this person who comes to you and then engages you that they want to travel to a certain destination. You need to understand their interest, what they want, what they're looking forward to, so that you can package it really nicely and deliver it to them so that they can really appreciate at the end of the day. Not just throwing packages, you really need to make sure it fits what they expect. Okay. So you put some thought into the packages. Sure, sure. Okay, can we play a scenario now me? Let's say I come to you and I'm interested in going to, I don't know, I want to go to Ambocelli and spend some time there. What would my package be like? Oh, so far... And I'm coming alone because I just want to rest. Yeah, alone? Yes. So you're basically a solo traveler? Solo traveler. That's what I'm known as? Yes, yes. Which I am, which I am. I am a solo traveler. I do travel a lot alone. Yes. So Alda, Alda, Alda, right now you're just in town and all that and it's a hassle.  Exactly. So basically I'll choose the best facility for you so that you can also get to enjoy when you're holiday. Okay. That's great. Thank you. And before I forget, we normally do facility inspections. So I travel a lot, engaging different hotels, yes, so that I can really understand what they are offering so that when I'm giving it to my guests, I'm really confident of what I'm doing. Yes, yes.  Yes. It's because you have a tourism company interested in bringing guests over, right? How do you... What are some of the things you look at in choosing a hotel for your guests, a hotel for prime holiday adventures? Yes. They should have some good grounds. Okay. Yes. They should have indoor games, facilities like swimming pool and spas and all that. And the rooms should be really nice and well done. Yes. All right. Clean, well done. Sure. Right. Nice design and all that. All right. And could we talk about some of the things that you're interested when it comes to prime holiday adventures? Because I know that you'd like for this to grow, is that right? How would you like for it to grow? To which extent? Yes. And in which aspect? Okay. Right now we have two branches. So I have this idea of getting new branches, opening up new branches so that I can also create employment and also encourage people to travel, change their mindset of traveling. But it's so interesting that most locals are now appreciating and really traveling a lot. Yes. Within the country? Yes, within the country. Wow. Beautiful. And that's good when the community gets to appreciate our resources, the help in maintaining it. Yes. I see. I'm so glad. Yes, because actually yesterday I was reading, was it yesterday or the day before that, I was reading an article, Kenya actually has been listed in Africa as the top tourism destination. Yes, sure. We are number one. I think it used to be South Africa. Yes. But I think in Jersey we took that car medal. Yes, sure. And it's great because first of all, people are coming from overseas to come and see Kenya. Not only that, Kenyans themselves are even interested in Kenya as it is. And maybe you can tell us what are some of the hotspots for locals and then you can tell me some of the hotspots for some of your international guests. Yes. What attracts more the international guests at the wildlife, Masaimara, Mount Kenya, that's international. Yes, basically we are heading to high season, late June, so we expect more tourists coming in Kenya. Right after the rainy season. But we're going to be very busy. And about the locals, they need day trips and excursions whereby they just go enjoy some place like the forest, we are there recently. So and the element, they just want to just go to a certain place that they get to enjoy and come back to their places, especially during and holidays. Okay, right. And you said that you do packages even for couples, is it? Yes. Honeymoons and everything. Sure. And solo travelers like myself. Yes. And also family packages. Yes. That's interesting. Can we maybe discuss what are some of the ideas that are in the kitchen cooking right now? What are some of the things you want to change, Kidogo, or to add a little bit when it comes to your particular company? Okay, but what we are really looking forward to is having travelers appreciate our packages and when they travel, they have what they were expecting is more of what they got there. So and we are also planning on giving them, we actually give them some hampers when they travel with us. They don't just travel and come back and forget about it, we appreciate them. And also they can also get a chance to negotiate like, Joe, you come to me, you really want to travel and you think that I want to go to Diani, but I don't know how it's, I don't know if I have the, they have saved enough to travel. So when you come to me, I'm able to engage you at that point and make you travel to a very nice destination and come back happy. Okay. Yes. Thank you for that.  What would it be like for you to balance? Do you think that you are ready for something like that? Are you looking forward to something like that? Or are you kind of focusing on yourself right now and on your business right now? Yes. I think in life you need to balance things. For everything to work, you need to check on everything. It makes sure that you're tending both to your family and also the business as well, because you need both of them actually. Yes. That's very true. And we don't have very much time left for our interview right now. And so what I'd like for us to focus on is our young people who are watching. Because we do have a lot of youth who are watching right now who are interested in business. But then, you know, kind of, I don't know how to say, where to begin. Let's say somebody doesn't have capital to start a business, for example. Maybe can you advise our young people ways in which they can first of all find some, get an idea and then get some capital and then finally get the project implemented. Yes. Oh, that's your camera, dear. Sure.  When you have yourself, you don't need a lot of capital to start up a business or to just get to somewhere, but you need yourself. That motivation is very important. So if you start the little savings that you have and you actually want to do something, definitely you'll do it. Yes. Thank you for that. Thank you and not least, finally, I'd like for you to market for this wonderful company because I do believe that the more people that are watching, the more you can be able to get some clients and everything. It is Tuesday entrepreneurship. We're not just, you know, here to listen to what you do for a living. We'd also like for you to prosper in what you're doing for a living. And so what are some of the things that you can tell our viewers when it comes to wanting to work with you, if viewers wanting to sit and booking a trip, for instance. Yeah. If you want to join us. I need your website, your social media handle, your phone number for the company. Yes. If you really want to join us, especially when you're an intern, we really appreciate you and we make you, we give you the skills that you need to make a step forward. So if you want to write to us, you can drop us an email at info at holydayadventure.co.k Please repeat that. Yes, info at holydayadventure.co.k. You can also check out our website, www.holydayadventure.co. You can also find us on our social media pages, Twitter at primeholiday17, Instagram primeholiday underscore adventures, and on Facebook primeholidayadventures. Okay. So those are the ways, all the ways in which they can reach you to book something. Sure, sure. That's great. We do have to wind up the show and pave the way for Hilda Wadili and the next interview. Thank you. Thank you so much for watching Tuesday Entrepreneurship. My name is Jerma Chachi. This has been Why In The Morning. Do stay tuned.


Hey everyone, this is me here for the laugh from Kevalabga, Afghanistan, and you are watching Trucker Josh Vlogs and YouTube. The highways and biways of Michigan, which is across the river from Davenport, Iowa. Somebody there needs some stuff, and I've got their stuff. Here we come. It's been raining quite a bit all day. They've had severe thunderstorm warnings in the area all morning so far. We've had some pretty good, pretty good booms. Made my day. So we don't have much further to go. We have about a five hour drive to go today. We're going to go to the Flying J and Davenport and stay there. And 5 a.m. tomorrow I've got to deliver this stuff in the box behind me to a place. And once I'm done delivering that stuff to that place, I got to go to a different place and pick up different stuff and bring it to another place. Guess I don't need my wipers anymore. So that is the extent of the excitement today. I'm glad you're here. Welcome. Don't forget to subscribe. Make new videos almost every day. It's been a gloomy day, not construction. It's been one of those days, not much to get excited about. But something that's good, there's always a silver lining at the edge of the cloud, you know, or however the saying goes. There's a silver lining. The scale is closed. So that's good news. 10 years ending and almost running the cones down. This Malibu right here from Michigan. We're going to go right past the south side of Chicago and out into, you just passed me in the exit line, didn't you? Past the social side and go to the western side of the province where it meets up with Iowa. And we'll actually stay in Iowa tonight at Davenport, like I said before. See you right there on the right? Big sign, big orange thing. The one good thing about construction around a scale is the scale closes down. See there's some good news in every day. Up for a bathroom break soon. There's supposed to be a rest area around here somewhere. Hope I can make it. I wasn't expecting this hold up in traffic. I haven't woken up yet in the sky. I don't think humans are supposed to be awake this early. Or we've got freight to deliver. We shall not be late. I think that's a commandment. It's a trucker commandment. Thou shalt not be late. Thou shalt be there yesterday. It's very hard to keep to that one, but it shall not be late. One is a little easier to maintain because everybody wants their freight yesterday. As long as you're there, when they're expecting you to be there and maybe a little early, I think you're okay. Really just going down the road here, let them take their pallets off, and then coming right back here and waiting for instructions. I'm pretty sure I'll have a reload. I'm in Davenport, Iowa right now. So I'm going across the river to here. I'm pretty sure I'll have a reload here in Davenport. It's a beautiful morning though, I mean. I don't think humans should be awake this early, but it is a beautiful morning. I went for a walk with diesel, short walk, but oh, it's just a beautiful temperature out there. No bugs. Like that is awesome. I wish we could go for a night or evening or morning walk in Manitoba without getting eaten alive by mosquitoes.  Those mosquitoes up there this year, they'll pick you up, they'll carry you away. Got my exact same spot back. It's like we never even left. That was a confusing, confusing place to deliver to. That was an old, a collection of old warehouses.  There was like 150 doors and six different buildings and none of them were marked or labeled. No signs anywhere, no idea where the receiving door is. I found out every building has a receiving department. None of them have an actual address. You just sort of got to bounce around until you find the right one. That was chaotic. But we got it done. We finally found the last building we checked. We found it. We got into the door on time and they unloaded us. Got back here to the Flying J and I'm going to go back in there, have a nap because even the load gods aren't at work yet. Even they're not up yet, so I shouldn't be up yet. So I'm going back to bed. And we'll figure out what we're going to do once we wake up. I'll leave the volume on on my computer here and when it goes ding, that means they're trying to get a hold of me because I got something to do. So I'll talk to you guys then. That'll be tomorrow's video. Don't forget to subscribe so you don't miss it. See what this trucker is going to be up to tomorrow.


Welcome, everyone, to the next talk of the session, which will be held by Michael McKean. And yeah, it's yours, Michael. All right, thanks. Yeah, my name is Michael McKean. I'm a software developer at Red Hat. I work on OpenShift Engineering. And I do a lot of stuff with cloud infrastructure. So we're going to talk today about testing Kubernetes at scale with Cluster API and KubeMark. So first, a little forecast here. We're going to talk about what Cluster API is, what KubeMark is. And then we'll talk about the KubeMark provider for Cluster API. I'll do a little demo of how this stuff works. And hopefully, that'll clarify what we're doing here. Then I'll show some helpful tools to make this process easier. We'll talk about some real-world use cases and then talk about what's next in the KubeMark Cluster API world. So first, we'll start off, what is Cluster API? And I don't know, put a one in chat if you're familiar with Cluster API or you've ever used it. And I'll just kind of keep going here, but I'm just kind of curious what, if anyone that's familiar with this project. But from its own description, Cluster API is a Kubernetes sub-project focused on providing declarative APIs and tooling to simplify provisioning, upgrading, and operating multiple Kubernetes clusters. Now, there's kind of a lot packed into that sentence, but let's just unpack it and see what it means. So if you're familiar with Kubernetes, you're probably familiar with this pattern where, as a user, I approach a Kubernetes cluster. I have a manifest that describes what I would like, maybe a pod. I give it to the cluster. The cluster does some stuff and it returns the pod to me, or it returns a result that it's done whatever I asked it to. Cluster API is very similar, but it provides a new type of primitive that people can use, which is a cluster. So now, I can go to what we call a management cluster, and I can ask it to create another Kubernetes cluster for me. And I use a declarative manifest to define the topology of that cluster. And then it will produce a cluster for me, and it will return back a kube config telling me how I can get into that cluster that it's created. And what's nice about Cluster API is that they've structured the project in such a way to where the back ends can be pluggable. So with one management cluster, I can spawn Kubernetes clusters on AWS, on Google Cloud, on Azure, on several other providers. And likewise, I can also manage those clusters. So I can create machines. I can delete machines. I can kind of change the topology of the cluster as needed. So really, Cluster API gives us the Kubernetes tools to start addressing infrastructure and clusters in the same way as we address containers. So what is kubemark? Kubemark, as described by the blog where it was announced, is a performance testing tool, which allows users to run experiments on emulated clusters. This one's a little more straightforward, but let's look at what that means. When I create a cluster with kubemark nodes, what I can do is use a real control plane, meaning a control plane that's actually backed by a normal kubelet and whatnot. And I can use that control plane to spawn what we call holo nodes. And the holo nodes are the kubemark nodes, and they run as pods or processes. They don't necessarily need to have their own machine available, so they're very lightweight, and they don't really do any action. They just say they've done action. So if you tell a holo node to make a pod, it'll just say, yep, I made that pod. Or attach this volume, it'll just say, yep, I'll attach that volume. It's not really doing anything on the back end, and it's got a Docker shim that kind of prevents it from really doing anything real, but it will say it's done it. And so this is kind of cool for testing scenarios, because I can go to a kubemark cluster and say, all right, give me a thousand pods, and it just immediately turns around and says, yes, I just created a thousand pods for you. And it's actually placing load on the API server, on the scheduler, so you can test the throughput of how these mechanisms of kubernetes are working. So a couple of notes here to take away from the holo nodes, they do not create containers, and they do not mount volumes, even though they say they will. And they do this by running a Docker shim that just kind of like responds to things. Yeah, like Makita, right. They do this by using a Docker shim that can just pretend like it's doing things. Now, if that Docker shim is unable to like understand an image spec or something, it might barf. So there are reasons that these kubemark nodes might throw an error, but usually they won't. Also, because kubemark is a process, a single host can run multiple kubemark processes. So, you know, you can use hardware to make it look like you've got hundreds of nodes, even though you really only have a few nodes that you're operating with. They also place a true load on the API server. So when you're trying to test throughput and you're trying to see interactions that happen in a live kubernetes cluster, this can be really helpful for you. And the other thing is that now they're able to advertise resource capacities. So a kubemark node can look like it has a GPU. It could look like it has accelerated networking. It could look like it has a hundred CPUs on it. You know, you can really make them look however you want to. And that really makes interesting possibilities when we talk about testing.  And in my mind, you know, peanut butter meets chocolate. And so for those of you who aren't familiar, this is the Reese's Peanut Butter Cup. They are quite delicious if you like peanut butter and chocolate. So what does that look like though when we deploy it? This is one methodology of deployment, especially when we're using testing. And this can be used on a single host and it requires the host to have Docker, perhaps in the future there'll be better support for Podman right now because of the way a cluster API creates its workload clusters. We haven't quite worked out all the details with how to make Podman do this. But for now, Docker is kind of the path forward here. And what we do is we use kind to create a cluster in Docker and then we load the cluster API management tools into that Docker cluster. Now from there, we can use that to spawn a new workload cluster just like we talked about before. And that workload cluster is actually running in something we call the Docker provider. And so previously and currently in cluster API testing, they use the Docker provider because what that can do is instead of having a cloud infrastructure or a public cloud like OpenStack that you could go to, it uses the local Docker as if it were infrastructure and can create new machines by spawning new containers in your local Docker. So what happens is we see in step one, the management cluster creates a Docker cluster which is the workload cluster. And then we'll tell the workload cluster to create Kubemark nodes. And those Kubemark nodes based on the way the provider is set up will just be spawned as pods in the management cluster. And I'm gonna walk through this and we'll see what this looks like in a live scenario here. So to kind of review this, the cluster API Kubemark configuration, it needs an actual management cluster. And for the purposes here, we're using kind, but that could be a real management cluster that exists on physical hardware that you have or something in the cloud. It doesn't necessarily have to be this kind solution. It also needs a real control plane for Kubemark because Kubemark actually requires at least one machine to be a real control plane. So you have to have at least one node and that's where the Docker workload cluster that we created comes in. But that doesn't need to be a Docker cluster. That could also be another real physical cluster or it could be something in the cloud. And then likewise, the Kubemark nodes become pods in the management cluster. And this is the way it's configured now, but in the future, we're gonna have some flexibility for where those pods could end up landing. So let's get into a little demo action here. And I realize I'm gonna have to unshare and reshare. So this might get a little tricky. I gotta make sure I grab the right window here. Yeah, exactly. Thank you, Peter. Okay, so we're looking at a VM here. And this is an Ubuntu VM that I've created because it's got really good support for Docker and kind of works the way I like it to. And I've got some Ansible scripts that set up this VM for me. So I'll share those later in case anyone wants to kind of replicate the same thing. I've also got a local directory of some script files here that I use for doing things. And so what I'm gonna do is I'm gonna start by kicking off my management cluster. And this will take probably 20 or 30 seconds to start. I've also got all these like little script files in another Git repo that I'll share at the end. So if anyone wants to replicate this entire process, you should have all these files available for you. So right now, what you can see what we're doing is we're using kind to spawn a cluster in Docker. And it's a 1.22.2 cluster because that's what I've just been testing with. I'm also gonna apply a couple other things here. This will put a local registry inside of my Docker cluster. So I've got a container running that's running a kind registry and I can put that inside. And this is a way for me to get images into that cluster. And now what I'm gonna do is start the cluster API tooling installation. And this is another script file. It's using this cluster CTL command which is from cluster API to initialize the various components that I need here. And so you'll see it's gonna put cert manager on and it's putting several different components of cluster API in. You can specify the core version, what bootstrap controller you'd like, what control plane controller. Most of these we don't need to change. And then finally the infrastructure controller and that's where you see we're installing Kubemark and Docker as our infrastructure controllers here. Okay, so at this point, I've got a cluster running and if I do, if I do a watch kudctl get pods minus a, what we should see is, okay, everything's running. I was waiting to make sure all the CAPI components are running here and they are. So the next thing I'm gonna do is start that initial Docker cluster. And I've got a manifest here. I'm not gonna go through the manifest because it's kind of long. Maybe if we have extra time, I can run into that. But it is, it's just a bunch of yam on it may not be that exciting to look at. So what we'll do is we'll create that control plane. And if I do a watch kudctl get machines here, we'll see that right now this is provisioning the machine. Now machines are a type that is created by cluster API. There are three kind of primary types that cluster API users wanna be aware of. And those are machines, machine sets and machine deployments. And you can think of machines kind of like a pod and a machine set kind of like a replica set. And then a machine deployment kind of like a deployment. So a machine deployment defines a way that you could deploy a set of machines. A machine set describes that set and then the machine is the individual machine itself. And so what's happening right now is this is just creating the Docker control plane and it's setting up things inside there. Usually this doesn't take more than like 30 or 40 seconds. So it seems like, you know, I might have tempted the demo gods too much here by asking for this. But eventually what's gonna happen is we should see a provider ID once it's created the once it's been able to create that node. But for some reason it's taking a really long time to do it. Now the nice thing about this setup and I'm gonna switch to a different window here just so we can look at some things. The nice thing about this setup is that because I'm using client kind it's really easy to tear down and rebuild it. So if this doesn't work I can try and tear it down and rebuild it really quickly and we'll see what happens here. Okay, this is taking longer than I would have expected. Although when you look at kind we can see that I do have that second cluster. So I think what I'm gonna try to do here is I'm gonna use this other script file to get a kube config for that cluster. If I can type it's still a little early here for me. Okay, so it looks like this other cluster has not created properly. So what we're gonna try to do here I'm gonna just tear down the whole thing and we'll see if I can put this tooling like really to the test here. So I'm really thrashing the machine hard here but what I've done is I've destroyed both clusters that I just created and I'm trying to recreate the management cluster now. And I guess I'll give this another minute or two but if this doesn't work, I've got a video of the demo so maybe we'll switch to the video we'll just see what happens. But you know, you never know, I could get lucky here. So we'll install the cluster API tooling again. Cert manager is usually what takes the most time here because it has to deploy it on the nodes and then make sure everything's running properly. Does anybody has questions or anything too? Feel free to drop them in chat. I'm looking at the chat now so I don't have any problems if people have questions or comments or whatever. So we can see it installing all the various CRDs and everything. Let's just make sure that everything is configured. Okay, everything is running. And let's try this again. So we'll do a kubesikio, I keep doing that. All right, now hopefully this time what we'll see is that it will actually make us the node. And if not, then I'm just gonna, I'm gonna switch to running my video and we won't get the nice interactive demo. So and sometimes this happens because there are networking issues like it can't, it has trouble creating the second container and then running it back to the first one. Normally, at this point, what I would do is start looking at the log files to see like why it's having problems creating it. But I just did this an hour ago and it worked. So maybe I've upset something with the VM here or whatever. It looks like we're having a similar problem. This is like really a bummer from demos now I guess. I'll try one more time. Okay, I'm going to abandon this and just get my video out here. Stop sharing. Okay, so now we're looking at the video version of this. So very similar, you can see that like, we start, I do that pretty much the same thing here where we're starting up the control plane. I'm gonna fast forward a little bit because I know we're running short on time here. See it kind of go through the rest of the stuff here. I was wondering maybe we could run this a little faster. We see the same stuff, it's trying to set up cert manager. Takes a long time to do it. Skip forward here. All right, so here you can see we've created that camcp and we're waiting for it to provision again. This is a step where we were just kind of stopped at. And what eventually happens is that we get that provider ID and then I'll move on to the next part of this. Let's see here if I can just skip. Okay, so it gets a provider ID and now we're just looking to see what clusters it's created. I'm gonna get the kube config for the second cluster. Let me see, I just saw Eric put a question here. Are there public services that run these management clusters? I don't think so, not yet. And next question was, are there public services that can run these management clusters that people use? I don't think there is a public service that creates these for people but they're pretty easy to install from the QuickStart manual and cluster API. I'll have a link to that at the end here. So what we see here that now is we're looking at the Docker cluster that I've created, the workload cluster. And I've gotten the pods to show that core DNS, then there's been no container networking started yet for that. And so what we need to do is we need to deploy a container network. And I use Calico in here just because it works fairly well with what we're doing. And what you'll see eventually is that these nodes become active. The containers get created and if we look at the nodes, they'll become ready. And I'm skipping kind of forward here. The next part of this that I'm gonna do is create the actual kubemark workload cluster. And so what we're looking at here is I'm doing a git nodes. You can see this kubemark MD0 is my machine deployment for the workload cluster. And we're just waiting to see the container network deployed to it. So now that it's ready, that container network has been deployed and we've got a kubemark node running. Now what I'm gonna do here is I'm gonna show kind of how the cluster is set up. So we're gonna watch the pods and I'm gonna create a workload. And this is just a standard workload I use. It's a little sleep routine. It requests two gigs of memory. So it's nice for this because when I use it with the autoscaler, I can use it to push out new nodes because only one will fit on a node if it doesn't have enough memory. And what I'm gonna do is I'm gonna scale the replicas up here. And what we should see is that we'll have a bunch of pods in pending because at this point, we only have one kubemark node set up in our workload. And what I'm gonna do at this point is I'm going to scale the machine deployment of the kubemark hollow nodes. And we're gonna see how quickly this turns around and the nodes get created and the pods get assigned to them. And it's gonna happen really fast, like much faster than you could ever expect in an actual Kubernetes cluster. So what I'm gonna do here is I'm gonna scale the replicas on this. And remember, this is the same operation as scaling the machines in my cluster. So I am growing my cluster right now. And even though it happens very quickly here, you could see how quickly all those provider IDs came up. And I'm just, I'll go back real quick here if I can catch it. You watch as the provider IDs come up and as soon as I've run the scaling, like those nodes are already created. There's no way this would happen. You saw even with the Docker provider, it took a while. Imagine how long it would take with AWS or GCP or something like that. But here you see these nodes are all not ready. They're going ready as the pod network's deployed and then the containers get deployed like right away and they're running. Now, if my demo had worked a little better, I would have gone into some auto scaler stuff and kind of showed you how that works. But I'm gonna cut things short now. I will make this video available if people wanna watch it. But I hope that kind of gives you a taste of why this is exciting in the testing space because of how rapidly these things happen. And like these nodes are real nodes. I can apply taints and labels to them. And I'll kind of talk about what we're doing with it here. So let me go back to the slide deck. I know we've got just a few minutes here, but if you've got more questions, please feel free to toss them in chat. We're kind of running at breakneck speed here. Okay, so there are some helpful tools that you can use kind as one of them. I've got some Ansible Playbooks that I've shared that will help you set up the virtual machines that you might wanna use. And I've also got those shell scripts. You'll find this presentation linked on the DevConf, the schedule page for this. So you can get links to all these things there. I wanted to talk about some real world use cases too for how people are using KubeMark. And some of these are kind of old, but currently the Kubernetes SIG scalability runs a set of KubeMark jobs in their SIPI. And this link here will show you that. And you can see they run these on every workload just to kind of see if they're making any regressions in that. And most of them are scaling kind of things where you're looking at large numbers of nodes and pods and whatnot. Several years ago, the Apache Unicorn Project, which is an alternative scheduler that works with Kubernetes, used KubeMark to do their scheduler throughput. And they were able to show how they could increase throughput through the API scheduler with that. Also, the Kubernetes blog, when they first released KubeMark, they showed how they could create a 2000 node cluster and launch 60,000 pods. And then my personal bugaboo is this last Red Hat OpenShift bug here. We're currently working through a problem with GPU deployments with the auto scaler. And I'm using KubeMark as a way to help me test that because I can test nodes that have GPUs without ever having to spawn them. And I can control how the nodes look like they have GPUs. And this is really important to us kind of diagnosing this bug. And I realize we're at the time here, but this is what's coming next for the KubeMark provider. We're gonna add continuous integration and some automatic image builds. We're also adding support for external cluster selection so you could choose where the KubeMark nodes land. And we're adding some scale from zero implementation to work with the auto scaler as well. And this will be the first implementation for cluster API. So hopefully it'll show people kind of what to do. So thank you everybody for bearing with me with all the technical problems and everything. These are some links that you can use to kind of follow up here and stay in touch with me. We got cluster API book, hollow mode, the cluster API KubeMark provider. And then a couple of things for me, those script files, the Ansible playbooks and then a blog post I wrote kind of describing how all this works. So yeah, thanks everybody. Thanks Michael. That was pretty, pretty ambitious demo. And I'm really sorry it didn't work out well. Unfortunately, we are on top of the time allocated. So if there are any more questions or if you want to ask Michael about something cool about this, please go to the work adventure thing we have set up. Thanks again, all the audience, thanks again, Mike. Thanks Peter, that was awesome.