So we were in that hustle phase, just grinding, trying to get it off the ground. Yes. And then we had our friends helping. We were doing events and expos. I remember buying like 5,000 wristbands from China that we were trying to use. Man, we did everything wrong. Marilla marketing. Merch. Merch, it's like that. T-shirts. So funny. We had this cartoon character made. We did a lot of like, you know, entrepreneur stuff and wasted a lot of time and money. Really early on. Ultimately, it failed and it led Kim into kind of a really dark place. But at that point in time, I had gotten a new job making like 150K. And so we were in a good space, but the business was failing and so she wanted something different, something more to do. And that's when we kind of developed the idea for Curl Mix. So after the Natural Hair Academy went belly up, we knew that the next business that we're going to start needed to, number one, make money on day one because we weren't going to do any more like ad companies, content, all this stuff, do a lot of work without getting paid for it right away. And then the second thing, it needed to be basically product-based, so it can be divorced from our time. So whether it was a product-ized service or a product, we wanted to make sure it was divorced from our time because we didn't want to have another repeat of that first failed business. So we tried to learn as much as possible from that failure. At what point, in terms of your app, did you know it failed? Was it just like nothing was happening? Or like what was the signal that this isn't working? We basically realized we were competing with Facebook and Instagram. And so we realized that women love to talk to, they love to showcase their hair, not talk to other women about their hair. So it's like, I don't want to connect with you about my problems that I'm having or this conditioner or whatever. I don't necessarily even want product recommendations. What I want to do is make my hair cute and take a picture and show you how cute my hair is. And so we couldn't get people to come back every day. And we had over 1,000 users or whatever on the social network. But it wasn't... That's pretty good. Yeah, I mean, if you think about it, so I think context is important here. So we're talking 2013, 2014. At that time, the App Store was a brand new thing, right? I mean, Blackberries were still, like they held market share still at that time. And so for you to have 1,000 users on any social media app, I mean, that's pretty good at that time. Oh, wow. Well, thanks. If you had told us that back then, we probably would have kept going. And we didn't make no money. And that was the other thing. And I was like, we don't... This is not a revenue generating business. I can't hire people or even feed myself with this. Someone just happened to be making a six-figure job or working on a six-figure job while I was doing it. And that's the only way that it allowed me to do it. But we also realized we're going to have to make content if we're coming back every day. I mean, I had to be something 365 days a year. And I was like, if I can't pay people to have to make the content, this is not a business. And this doesn't really work. And so I either need to do one, get more ad revenue so that I can pay people to make content. But even still, that's not a lot of money for a lot of work, you know, content for every one of these years. So I was just like, you know what? I need to come up with something else. And I just closed the business. And at that time, I had learned like photography, web design, like, you learn so much, you know? So it wasn't a complete wash. And I also made a ton of connections in the hair industry so I was producing content for professionals in the industry. So stylists and influencers or whatever. So really chemists, yeah. And so I was like, so I had a lot of education about the industry and connections at that point. And so at this point, you think, all right, cool, we need to create, you're solving for a lot of the things that you are trying to, like, you realize these things don't work according to this business. Let me go ahead and solve for them. And so now you're entering a product that people can use all year round. So it's not seasonal. And what was the first iteration of it? And how did you even come up with it? So we were watching, so funny enough, I was a DIYer back in there. Do it yourself first. I was making products for myself, for my hair. And I was buying raw materials from Whole Foods and I come at home and mixing them up. Just destroying my kitchen. From you, Dio. I'm the chef. I really love your relationship. You guys, the dichotomy is amazing. You can just flee from all over the place. You make stuff, you pay to go, just leave it out, it goes bad on the counter. Whole Foods is not cheap at the time, either. Whole Foods has never been cheap. You got $300 and you don't even like what you made. This is what I'm talking about. And so I was doing that and we saw an episode of Shark Tank where the lady was doing organic cookies, but she was putting everything in a pretty package, like ready for Crocker in a box and you could make it yourself and know where everything came from, knew it was organic and it was simple and a fail-proof recipe. Yes. So I was like, I wonder if anyone was doing this for hair because I would totally buy this for like 30 bucks. And then I looked online and I couldn't find anybody doing it and I was like, wait a minute, this is probably dumb. If this 2015, this box doesn't exist and there at the time there were like 1,000 subscription boxes, I was like, okay, I'm not gonna do it. And Tim was like, no, Kim, like, she's giving a shot. We launched it, I sent one box to my cousin and I was like, see, tell you this is dumb. Nobody wants this. And then he- You didn't launch it right. Yeah. Just try again. And so at this funny, at the same point in time that she decided to launch it, I just read, I believe it was Ryan Holidays, like Confessions of like a Media Manipulator. And I was like, you know, he has like a whole formula about how you might want to try it again. But you said Airbnb can relaunch seven times. Surely Chromebooks can relaunch twice. So I was like, fine, I'll try it. And I ended up reading like Robert Tildani's book, Influence. Influence. So having a win friend to influence people. It's like Six Principles. That was another one. And I applied those Six Principles to our website and I also applied those Six Principles to Pitching Journalists. So I looked up, how do you pitch a journalist? Found some templates and adjusted them. And then also went to BuzzSumo.com to figure out who the top journalists were in my category. So for DIY in the last six months. So found the top people, read a few of their pieces of article, use the template, pitched them with also like a custom intro about the work of theirs that I had read. And then we ended up getting Refinery29 on when it's time for our launch. So everybody who had told me no, that was a media company, I went back and traded up the chain and then say, hey, Refinery29 is coming to our launch but hey, for you to miss it. And this is like, Refinery29 was like a million followers at the point. So like now they're like really big in an act. And then they all said yes. So we ended up getting like seven media people to cover our launch. And we ended up selling like a hundred boxes on that first day. And that led to a year of revenue in 2017 for 130,000 and the following year 140,000. But we realized we weren't growing. And that is when we were like, okay, we're not growing, the subscription box company should grow. And we realized our box was novelty and not necessity. And then also a lot of the DIY companies that were boxes in the industry had started failing or going bankrupt. And I was like, there's not enough profit here. And I found out that the profit on our box is only 30% but we're competing with the hair industry that can sell a bottle for $20, $30 and their margins are 70, 80%. There's never, I'll never be able to catch up. And so we ended up realizing we needed to pivot. And that's how we landed on our flagship show which is our hero products and the thing that we do differently than everybody else. And you say we landed there, but Kim took, there's a lot there. It took us a while to get to that Black Sea Joe because we had tried making products on our own. We just couldn't get the Black Sea Joe right. But it was the number one seller, like by far. People loved it. We just couldn't figure it out cause it's really volatile recipe. It goes bad within like three days. It starts growing mold. And so we couldn't make it shelf stable like a cosmetic product needs to be. So my wife, she's seven months pregnant with our first baby at this time. She spent a month in the kitchen on her feet just grinding out recipes over and over again every day. It made like 50 batches. And so we could got something that was stable that we could scale and do more than 60 in a pot. We could do 600, you know, with the machine. And we launched it to our audience and we had sold hundreds of a matter of hours. And I'm like, oh my gosh, we've never launched anything that successfully before this has gotta be it. Yeah. So we did a beta, it was 60 bottles. We was like, okay, 60 bottles, cool. So that in a couple of minutes, let's do another 60. So that in about an hour, another 60. And then we're like, okay, crap, we actually have to make this now. So we ended up doing that beta launch or the flaxseed gel. And then we're like, you know what? This is actually what people want. This could be the next step in our business. So we decided at the top of 2018 to just quit the subscription box business. We had like six boxes lined up. We had all this inventory in our house.  And then we pivoted and... Is it still just you two at that time when you guys are doing this? We have some of our cousins and her little brother helping us out, but like, no official anything. Not even like official contractors, right? We're in our kitchen and our apartment. And then we, so in 2018, we've pivoted to a traditional e-commerce store. We started selling our flaxseed gel in various fragrances and various oils, right? And then we also turned our best selling boxes, so four of them, into full-size products. So they're just high-price products. We honestly go together, just high-price. And we ended up doing like a million in sales that year. So we went from 140 to like a million. Yeah, 10X. Yes. And then the year after that, there are a few things that happened. I can go back to it, but the year after that, which was last year, we ended up doing 5.5 million. So we got 5X to jump. So like, we've been kind of like, there's a cost would grow, right? There's so much you don't know. And then you have to find that out.  But now we have like over a staff of over 30 people and hopefully on our way, we're doing 2 million this year. So I'm excited. So in terms of, I mean, so many things there, but at some point, like one of the things that I learned moving from my first business to my next one was, I was not thinking big enough. Did that resonate with you? And you're like, yes, when we do this, even on the pivot, was that something that you guys were like, we need to think bigger? For me, it was reading a tremendous amount of books. And I was like, ugh, if I had only known this when I was that, you know. Yeah, I think the exact moment we realized, okay, this could be big, because we're 140, we thought, oh yeah, we're doing it. Like that's almost as much as a salary, right? Like we're really good. And then she went to a comp, that same point where we pivoted, Kim went to a conference called Game Plan from the traffic sales and profit community, which is led by Lamar Tyler. And I remember, I went there too with the baby, I was up in the hotel room, she was down at the conference and she came up at the end of one of the days and she was like, there was this lady there and she did $300,000 last year. Oh, could you imagine doing $300,000? And I was like, that's, that would be nuts. If you could do that, that would be Ruby on top of the world. It's a thousand hours a day, right?  Right, right. So do you think that you would have had any kind of the success that you're experiencing now with ChromeX if you hadn't had failed with your social media startup? No, we probably would have, we probably would have done our wills a lot longer. And then on, And wasted money on things. Wasted money on things that we didn't know didn't make money, but we did that. This time we called it like paying for our MBA, but it was a really cheap MBA. But you needed to, to learn those lessons and to be able to learn from them and take ChromeX from, you know, 140 to a million in a year, I mean, you kind of needed that stepping stone first and foremost to be able to vault your next platform to new and uncharted heights.  Data is king, and so that's why we won't do retail. We're on, we're staying online, because if I don't have your phone number, your email, then this is not useful to me.  I need to be able to continue to sell you over the lifetime with you as my customer. Hey everyone, thanks for checking out that clip. If you enjoyed it, be sure to hit the like button down below. And if you're interested in hearing the full episode, it's out right now on our YouTube channel. We've had a lot of great guests come on this show before and we've got a lot of great guests coming up in the future. So hit subscribe so that you don't miss a single episode. And one final note, we're always looking for new ideas and new companies to feature on the show. So if you know of someone or know of a company, write us a comment down below letting us know who they are and what they do. We'd be happy to have them on the show. Till then, I'll just be here waiting for your comments. So, see you later.


So while we're sitting here making content and covering news, and yes, there was actually some Mario 10 day news today about some new LEGO sets. We're gonna save that stuff and bundle it with news tomorrow because I'm fairly certain we're about to get some new Mario 10 news tomorrow, maybe some thousand year door and Luigi's Mansion 2 HD stuff if we're so fortunate. But what I'm actually one of talking about today is Tears of the Kingdom. And look, we haven't talked about Tears of the Kingdom in a long time. And some of this might come down to a question. I keep seeing pop up, whether it's on live streams, social media channels, other people's channels, this question that as I'm quoting one person on Twitter who just reminded me of this question, even though I've seen it thousands of times in the last few months, why has Nintendo abandoned Tears of the Kingdom? Now I find the question on its surface to already be a flawed question because I know that Nintendo has abandoned Tears of the Kingdom. Now look, you can look at this question in many different ways. Is it because they said they're not gonna bring Ultra Hand and other abilities forward to new Zelda's? Is it because they said the next big Zelda 3D adventure they're working on isn't going to be a sequel to Tears of the Kingdom, but rather a completely new direction for the series? Is it because Nintendo said the game is content complete and they're not gonna be making DLC and bare minimum, you think there should be a master mode and without them adding new content, man, oh man, it sure feels like they just sort of sent Tears of the Kingdom out and now they've moved on.  And I think it's born out of this strange phenomena of modern gaming that we're just not accustomed to. Tears of the Kingdom is a complete video game. I know that's a strange concept and I gotta admit, just sitting here right now, it feels weird even saying this out loud, but Tears of the Kingdom is a complete game. It doesn't have all this DLC coming with all these major patches of content. It doesn't have any new modes coming like a master mode. It doesn't have multiplayer stuff to keep us going forever and ever. And it doesn't have a ton of marketing even happening for it to this day, although there's a couple events here and there where they bring some of the statues around that's cool, but for the most part, Nintendo sent out a game that's a complete experience. And I feel like even with Nintendo games, this is strange. I mean, even when we look at the most successful Mario Party game on Switch, it had an online mode added to it years later. There doesn't seem to be anything being done with Tears of the Kingdom. They patched out some duplication glitches and did a few performance updates, but for the most part, they just completed a video game and put it out on the market and it's gone on to sell 20 million units, which is insane. That's a lot of sales, but I feel like we're so used to games getting constant updates, constant new content, even Elden Ring has new content coming, right? So we're so used to games getting new content and new updates and new this and new that that in the world that we live in here in 2024, it kind of feels strange that one of the largest releases of last year isn't even really talked about today, not by Nintendo and even among gamers, it's really only talked about in select communities. Tears of the Kingdom isn't getting these sort of legs at Breath of the Wild then and it's interesting because Breath of the Wild had DLC, right? It had bonus content, multiple different releases of content. It culminated in a game of the year victory and yeah, it seemingly had much longer legs than Tears of the Kingdom. Now, obviously we don't know what long haul sales will be and Tears of the Kingdom could get a re-released 4K edition that ends up coming to Switch 2 and that could end up extending sales even more, but it's not just about the sales. I think people feel weird and this question keeps coming up because a lot of us forgot what a complete video game really is. And now we're to the point that we think a game being complete at release is a negative and I find this mentality just personally to be very interesting because, look, I grew up where games came out and that was that. Now, yes, on PC we had expansion packs, but if you wanted to do new stuff in, say, let's say, Ocarina of Time, right? You had new ideas, you would have to release a whole new game in Majora's Mask. You couldn't just shoehorn it in with DLC or updates. Now on PC, they would do expansion packs that would almost be entirely new games and that are themselves because there was so much content in there, sometimes more content than in the original game's release. I find it fascinating that today, the way video games work and not just the free-to-play games in our multiplayer games, but even our single-player experiences, that all of them come out and there's always something else. I mean, even if you look at Mario Odyssey, which is probably the least updated Nintendo game in terms of, hey, we gave you a bunch of stuff after the game came out. It was a very content-complete game at launch. They still added that Luigi mode a little bit later, right? That little Luigi mode with the balloons and finding them in the city. That was an update added later. Here's where the kingdom is the first game Nintendo's released in a while where there isn't that expected update. In fact, they came right out and told us the game was content-complete. There is no DLC. There is no planned additions coming. This is it. We're moving on to the next game. And I think it feels strange for a couple of reasons. Not only because they added content to Breath of the Wild after release and they've done the Xenoblade and pretty much every IP they have. I think it's also feel strange, not just because of that, but because of the landscape we're in. Nintendo wants to quickly, as soon as they could, and they're already probably doing it, work on the next Zelda game. So it actually launches during the heart of the Nintendo Switch 2 generation. And I think they know that. They don't want it to be a Skyward Sword thing where they wait all the way to the end of the generation before getting an exclusive Zelda game out for the new platform. They want to get it out in the heart of the sales. And that is awesome and could be a massive boon for the system. But in order to do that, they had to decide, hey, whatever we wanted to put into this game, we're just gonna have to put into it before it comes out. And it's a very strange concept in today's world where at least you know, 100% complete game at launch. I think that what that feeling is coming from is something that, those that feel it anyways, cause I certainly don't feel this way, but those that do, maybe examine within yourself why you think it's strange and why you think Nintendo has abandoned a 20 plus million seller just because the game was content complete when it came out and there is no other plans for it. Is that really an abandoning of a game? Or is that, hey, instead of, you know, piece-mealing and DLCing and coming up with ideas later that we could shoehorn in, we actually released the game in the intended permanent state the game was supposed to be in. If they had never updated the game, let's say they never got rid of the easy duplication glitches. No, they just left it alone as it is on those launch cartridges. It would still be a content complete game with very few bugs and issues. It's such a interesting thing to think about. And as I think more and more and more deeply about this, I gotta even admit a little bit to me that even though to me I never thought about this until people started mentioning it, thinking back, it is a little weird. It is a little weird, but why do I have that weird feeling? Isn't this what we want? Don't we want content complete games? Or because of the era of the internet and social media that we're in, if things aren't constantly updated with new features, does it mean that the company doesn't care about it anymore? Is that our perception as consumers? I find it just to be interesting. Obviously, if there was planned DLC and everything else, we won't be talking about this when there's a 4K edition or something like that on Switch 2. We'll obviously go, oh, they must clearly care about it because they just gave us an upgraded version or if they release a payable patch or some sort of patch for Switch 2. All of us would go, they did not stop caring about the game. But I also think that it's a single player experience and we've been a bit spoiled with single player experiences where they seem to go on forever. Look at Elden Ring getting massive DLC, single player experience. We get spoiled by the single player experiences that seemingly never end. And because of that, it becomes to be our expectation for all of them when honestly, some games are best left as just a whole complete experience. When I got to the end of Tears of the Kingdom and I'm not gonna drop any spoilers here even though we're a well-placed spoiler territory with Tears of the Kingdom. But when I completed the game and the story and all of that, I didn't feel like there needed to be more. It really felt like we hit the end of whatever they were currently doing. And I think that is a magical place to be. And I think that makes it a very self-contained amazing experience that when you look at the broader spectrum is a little weird that that's just how it ends. There isn't anything new. There is no more modes coming or content or all that. It's strange. So every time I see this question pop up now, hey, why the heck did Nintendo abandon Tears of the Kingdom? That's not the right question. Nintendo didn't abandon the game. Why do we feel like a complete game, or a content complete game at launch is abandoned just because it doesn't get content piecemeal to it later? That is the question I have for you guys because I'm struggling with this as someone who grew up with content complete games. But I'm also struggling with it because I understand a little why it feels weird because no other game releases that way. And is that a bad thing? Is a content complete game a bad way to release a video game? Should we have wanted them to cut some things from the game to piecemeal to us later to make us feel like they still care? I find it just a fascinating conversation. Let me know what you think about this down in the comments below. I think that, well, I guess I did this whole video telling you what I think. So I got nothing else to add for now. Thank you so much for tuning in and I'll catch you in the next video.


That's me. I used to work for the Open Knowledge Foundation. I'm going to be talking to you about Annotator, which is, I'll describe what it is a bit later. It looks like that. Well, it doesn't actually, it's a bunch of JavaScript code and we'll get to that a bit later. And I used to work for the Open Knowledge Foundation who sponsored most of the work on Annotator and still look after the repository. I now work for the Government Digital Service where I am, as I've said, a maintainer of Her Majesty's photocopiers. That's not actually what I do, but if anybody wants to know more about what GDS does, I can talk about that later. I'm not representing them here. So what I actually wanted to talk about today has changed a bit. It turns out that on a 10-hour flight you have a little bit too long to worry about what you're going to talk about and whether it's going to be interesting. So I'm actually not going to demonstrate Annotator today for a couple of reasons. The first reason is that there are several projects you're going to see over the next couple of days, one of which is Hypothesis, which use Annotator at the core of, of, of their software. Annotator is primarily a JavaScript library and a set of plug-ins for doing annotation on the web, primarily textual annotations, but there are people who built image annotation plug-ins. I was chatting to somebody yesterday who's expressing an interest in some kind of video annotation, and it's secondarily a, a reference storage API. Annotator has a client-side component and it has a server-side component which is essentially a, a JSON API for storing annotations, and there is a reference implementation of that. So what Annotator serves to do I think is, is really be the, it doesn't aim to do everything with annotation. The Annotator user interface is, is, is beautiful. I didn't have anything to do with designing it because I can't design my way out of a paper bag, but it's not going to compete with Hypothesis. There are lots of other people doing interesting things with it, so that's not what I'm going to demonstrate to you today. What I thought I would do is three things. I'll give you a brief history of where Annotator came from because it's very much a pragmatic tool. It's not, it's, it's a long way away from Rob and Dan's world of annotating all the world's knowledge with a, with a perfectly interoperable system. It was very much a quick hack to solve a problem that we at the Open Knowledge Foundation had, which has subsequently evolved into something a little more interesting. I then want to go on to talk about a couple of people who've used Annotator and the interesting things they've done with it. And then the real meat of what I wanted to talk about are what I'm calling kind of Hilbert problems for annotation. And, and for those of you who aren't familiar with David Hilbert, this is one of the, the great mathematicians of the 20th century who in 1900 came up with a list of 23 huge, outstanding, important problems for pure mathematics. You'll be glad to know that because I am not one of the world's leading mathematicians, I'm, I'm not actually going to talk to you about problems for maths. I'm going to talk to you about problems for annotation. And you'll also be glad to know that I don't have 23 of them. I have only a few. They're really just, I just want to spark some conversations about this stuff. I'm not intending to present any solutions. So first of all, just a brief history of Annotator and where it came from. Annotator has a, I'm young, so forgive me, Annotator is quite old now. And it came out originally of a project called Open Shakespeare at the Open Knowledge Foundation, which is an effort to collect all of the free, as in Richard Storman, free editions of all of Shakespeare's works online for digital analysis and annotation. And at the time, we experimented with all kinds of different pieces of software. Some of you will have heard of some of these steps, was the annotation tool developed for commenting on the, the GPL version three commentary, the W3C's Annotator. And eventually we built something on top of a chap called Jeff, I can't remember his second name, Jeff, someone, somebody called Jeff wrote a thing called marginalia. And the first thing we wrote was a thing called Annotator with an E because we couldn't spell at that point. Annotator had a lot of server side help, the documents that it annotated had a lot of sort of pre-annotation done server side when they were rendered. It had, I think, I was trying to work out when I was writing these, I can't actually remember, it had a sentence or speech in terms of a play, speech level resolution of annotation. And I'll be honest, it wasn't very beautiful. It didn't have a particularly great interaction model. So we eventually worked out that we kind of wanted to build something a little better. So in the summer of 2009, we learned how to spell and we started a project called Annotator with an O. And the aim was to build something that was slightly more document agnostic. That is, it didn't need you to insert tags into the documents that you wanted to annotate. And up here in the top left of the screen is the first git commit to the annotated repository in which I describe it as a wee playground to see what can be done with a JavaScript annotation system. I'm not Scottish and I wasn't Scottish in 2009, so I'm not quite sure why it was a wee plea claimed. But anyway, it was incredibly buggy and it essentially remains still built around the DOM selections API. It did have some support for IE's text range model. Don't ever do anything with IE's text range model. It's horrible. So after a whole month we dropped support for IE text range. I built some very ugly user interface which you can see up here. And we very quickly got into serialization and deserialization of annotations on arbitrary web pages to things that were basically JSON documents. And the structure of those JSON documents was informed solely by what our needs were. I tried to kind of generalize some of it, but it's very pragmatic. It doesn't have any grand ideals. And I'm skipping out a whole load of stuff here. But throughout 2011 and 2012 an awful lot happened to annotate it. The OKF hired a chap called Aaron Carroll who actually knew what he was doing. He made the thing look half decent and have a half decent interaction model. It has some support for internationalization of the user interface, which for those of you who tried to do internationalization in JavaScript is actually a pretty big deal.  And in 2012 we did some fiddling on what was really the USB of Annotator, which is its server, its storage layer, can be used to annotate things anywhere on the web. And if you run a service, you have some kind of web application with user accounts, you can issue authentication tokens to your users that allow them to save annotations on your content to our or someone else's annotation storage system. So I will move on quickly from that. Deployments of Annotator. So there are lots of people who have picked up Annotator and done some interesting things with it, including people you're going to hear from today. So I'm going to move quickly through these. This is the MIT HyperStudio's Annotation Studio built on Annotator. This is a digital humanities project. This is an incredible guy who came and joined our mailing list and has just been, you know, he didn't speak very good English. His English has improved dramatically trying to talk about annotations and the annotation data models on our mailing list. And this is his sort of social open courseware port to Chinese Mandarin of Scott Chacon's pro-git book that has annotation support. This is our project. This is Annotate it. You log in, you get a bookmarklet, you can annotate anything on the web. If the page changes it breaks, sorry. You can go and have a look at that. This is a guy called Gary Tylman who's built an Annotator based system for marking up pharmacy students term papers, which I think is fab. And the assessors will annotate stuff and say, you know, you've got a problem. They automatically convert these things from PDF to HTML and annotate them and the students can then come and see what has been written on their work. And Dan has already spoken about hypothesis who are doing some really cool stuff, threaded annotations, fuzzy anchoring and so on. So to get to the real meat of this, and I will try and be relatively quick, I have a couple of things that I think are big, big issues in the annotation world. I know many of us have been thinking about these for a while. Dan has already mentioned some of them. He's stolen some of my thunder by fixing some of them. Well, having something that looks like a fix. So the first one, the first one is what I'm calling discovery. So the web is great at linking to things, or to be more precise it's great at outbound links. It's great at being on a page and being able to find out where that page links to. But finding out what links to a piece of content, or in our context what annotates a piece of content is a lot harder. And the web is, I think this is a problem. I know there's at least one person in the room who will be raising an eyebrow at this point. But Ted Nelson talked a lot in the early days of the web about the lack of two-way links on what he is raising his eyebrows and all. About the lack of two-way links on the web. And I think that this is a big problem because it doesn't, it means that the web is not a web. And it's a directed tree in which all links ultimately seem to lead to Wikipedia. And we have built something that is, if you like, the visitor distribution of the web is a power law with quite a large value of alpha. Wikipedia, Facebook, Twitter, get a lot of the inbound links on the web. And the inability to discover what annotates a piece of content is a contributing factor to that, I think. We need a web and not a directed tree. And I think this is something, the discovery of back references is something that we need as a community to solve if we want to try and reverse that process. The other thing, the next thing is about the annotation of documents, not the annotation of formats. Dan has already stolen my thunder on this interoperable PDF HTML annotation format. That is the point. If I annotate a journal article as a PDF, those annotations should appear in the HTML version. If I annotate a book on one page or on a Kindle, that should appear on another page or on a nook. And there are a couple of ways that we might solve this. One of them is by having some kind of canonical representation. One of them is by fuzzy anchoring, if you like, what I call content-addressed annotation, where you address an annotation by the content that it annotates. And this is linked, these are all linked, but this is linked to my next Hilbert problem, which is changing content. This is the big one, the ability to deal with the fact that the web changes. People move things around. And in general, annotating dynamic content is really hard. There is no, you can't even work out what the, even if you didn't know how to implement it, you can't work out what the ideal behavior would be. Because if the content changes enough, you can see that you probably don't actually want to keep annotating that content. The New York Times had a fabulous example where if the content changed too much, your annotation would just evaporate. So this might be solved by multiple people, fuzzy anchoring, but I think perhaps the biggest way of, the biggest and most likely way of solving this problem, which is my last point, Peter, is persistent reference. Or to put that another way, can we please put the internet, the whole internet, in Git? And for those of you who aren't overly familiar with Git, the important thing about Git is it is an append-only log of content objects. And at the moment, the internet isn't going to be able to use it, so I'm looking at a man over here who's trying to build some pieces of this. The fact that Tim says cool your eyes, don't change is not enough. The web is not cool enough. Your eyes do change. And the internet archive and Memento are trying to fix this, but I think that in some ways they're not ambitious enough. I want to be able to say, this is a thing that I annotated, and know that that reference will work. This is also part of the problem of the Wikipedia Facebook Twitter problem. If I know that those are the only sites that I can rely on to be around in three years' time, then that's what I'm going to link to. So that's it. I'm done.  If you want to know any more about Annotator, come and bug me, we have a GitHub repository, several GitHub repositories, a mailing list for people who are interested in Twitter, so you can contact me via any of those means or just come and poke me. Thank you very much.


That's really nice Let's do this thing. Hi, everybody. Hi, Steph. We don't have mics, so if you need us to speak louder, just kind of, you know, make a motion. I am Jen Krieger. I am the Chief Agile Architect at Red Hat, and here is my colleague Steph Walter. And I hack on a lot of open source projects and have probably worked on many of the things you guys have. You maybe got a patch for me. Maybe. Maybe just a little patch. So, hence today's talk. Yeah, a couple of months ago, Steph came to me and said, you know, I've been talking to all these people internally about this topic, and it seems very similar to the things that you have been talking to teams about, and wouldn't be kind of cool if we had a conversation. And I will preface this with actually our, one of the bosses in our organization said, you really should do a talk together. So I try to make the story a little bit better. So, right? Yeah, so really we're here today to talk about Agile and how to do self-sustaining Agile in your projects. So, throw out some words. What does Agile mean to you guys? Yeah, what else? The process, we'll keep adding process until the crying stops. Yes, okay, so the beatings will continue until morale improves? What else? Trendy. Plexible. Plexible, okay. Meetings. I feel like people from my teams are populated out in the room and are like giving me content to mock them on later. So this is really what Agile is about. Thank you Wikipedia for the definition and also all of the really long words up there. I wanted to bring your attention to some key words that are actually in this definition. The first being, it is a set of principles. So Agile seems to suggest that, or at least most of people's interpretation of it, is that it's a rigid set of rules of things you have to do in order to be faster. But in reality, it's actually just 12 principles that talks about a mindset shift that you want to achieve in order to actually get better results. And so it really doesn't say you must do this to be this. It says it's a problem that you have to try to. It's also about having requirements and solutions that evolve over time. And what that really talks about is that you're not supposed to go out with that big bang release. So the one, two, three, four year releases where you're about 40,000 lines of code and all of a sudden you're delivering that to your customers. Really what Agile is saying is just, hey, stop doing that and try to get shorter releases and more stuff out there so that you can actually find out what you were thinking was what your customer wanted is actually what they were asking for. And the final thing is that whole concept of self-organizing cross-functional team. That's a lot of buzzwords that just basically means, hey, that person who sits next to you or who is on your team across the world, you're supposed to work with that person. And so maybe you want to actually have a conversation with them to achieve results. It is not. And so a lot of times I hear, especially boss types, say something to the tune of, great, now we can do more with less. Anybody hear that before? Because I certainly hear it all the time. And so really what Agile programming refers to is the concept of doing the right things at the right time with the right people. And so it's not always about just putting a massive amount of code out there into production and seeing whether or not like a 5% of it might actually be what your customers asked for. It's about actually doing little swaths of things to make experiments to see whether or not that's actually what they needed. It's also not easy. And that is another thing that I hear people say a lot. It's just, oh, it's just a couple of meetings that you put into place and then all of a sudden you're going to go faster. And oftentimes teams only focus on that concept of meetings. And so they say, well, I've got to stand up and so therefore I must be doing Agile or I am grooming so I must be doing Agile. But the reality is that I actually am asking my teams to do both. I'm asking them to inspect their process in terms of how they're moving things around in systems to report to project managers or the boss type. But I'm also asking them to inspect their technical workflow. And so I want them to improve the way that they are delivering code or the way that they're integrating or the way that they're using their tools. I don't want them spending too much time over here or too much time over here. And it is certainly not Cat Boss telling you what to do at every point of the way. And so I've got several Cat Bosses in the room and what I tell them quite frequently is that, yeah, they might have an opinion about what work needs to be done but I never want to hear them say how it should be done. And so the very big distinction about that is I want feature A. That's great. You want feature A? I don't care if you want to use J Boss. I don't care if you want to use Cock Fit. I don't care if you want to use this test framework. Those are the things that I want the team to discover on their own. So thank you, Cat Boss, but be quiet. So the point of Agile is to be able to react to changes in the market in a sustainable way. And that sustainability is really something that teams often forget. And so we have this concept of like getting a release out the door and we're going to use Agile to get faster and we're going to push more out the door and we're going to push more out the door and we're going to push more out the door. And then what really happens to teams is that you have a huge core of people who kind of, I don't know, maybe get a little tired, right? Anybody tired? I'm tired. I'm tired all the time.  And they're the guys or gals who spend 80-hour work weeks and they're up all night and they're the ones who people go to when they say we've got this really bad bug and no one knows what to do about it. So we'll call engineer A and they're going to fix it for us. And that really isn't what Agile is about. It's about trying to figure out that correct balance of features being delivered to customers in a sustainable way. And so what you really need to know. I'm going to go over two points. The first one is focusing on your network of people. These are the things that I think you guys need to walk out of here understanding. And so what is focused on your network of people really mean? It is my theory that if you know the person that you're working with or you know that person you're going to toss your code over the wall to, it is more likely that you will be a better corporate citizen or a better work buddy if you didn't know them. And I have watched this happen for 20 years in my career where one engineer doesn't really know who's going to actually take that code and put it into production so they really don't care what the quality is. They really don't care what the result is. All they really care is telling their boss that the work is done. Anybody want to be honest and raise their hand? Yeah, I got one hand raised over there. At least one person out of 50 are honest.   So this is my favorite, you know, there is no I in team. And so a lot of times in open source communities this is what I see happen. An engineer decides they are going to do something.  They decide this is the right way to do it. And then they put the code somewhere and it goes to a community and then they are getting all this feedback, right? Is that something familiar? People say, yeah, I am not going to take this, I am going to take it. So, Agile just simply says, hey, you know that point where you have that idea? Maybe you want to open your mouth and use words and talk to the people around you and say, what do you think? This is the idea I have. What am I not thinking about? What did I miss? What is your idea? How can we collaborate together on this to make it better? And the point here is really how many have heard the quote, you are, you may be smart but you are not as smart as everybody in the room. That is the foundational point of this. And so I really like there is no I in team, but sometimes there is. And so what I am imploring all of you to do is to simply understand that your default operational perspective is going to be this. And so keep remembering that there are people around you who have thoughts, feelings, and try to be empathetic to what they are doing. And the second part, and the most important part of Agile, and so if you bump all those buzzwords off the screen and you think about the foundation of delivering smaller things faster, the feedback loop is the foundation of all of the methodologies in Agile. And really what it is talking about is a very simple concept. If you are working in a waterfall software development methodology, your feedback loop is likely going to be about a year. And so the feedback loop is the point of idea, creation, to the point that it is delivered into the hands of a customer. That is the loop. And so the point of Agile is to tighten those loops. And so Steph in a second, he is like hovering right here, he is like ready to go. He has had espresso and he is going to go for it. He is going to talk you through those feedback loops. And one of the most important feedback loops that, and in fact, Colin Walters, if you see him in the hallway, he will tell you the first question I had for him was about this feedback loop and when he said he didn't have it, I said, are you out of your mind? And so the bottom line here is that if you are not getting feedback on what it is you are doing and you are not constantly looking to inspect and adapt and shorten those feedback timelines, it is going to be really hard for you to understand what it is that you are doing is actually what you need to be doing to make your product successful. Okay, I am going to let you go now. No, that is totally awesome. And so what are the feedback loops? Think of them. There are lots and lots of them. You can think of things like retrospectives after a sprint. If you have done scrum, you can think of backlog, you know, during the backlog, you can think of sprint planning, you can think of sprints themselves and all these different things, various forms of process and meetings as feedback loops, where you figure out what you did last time and then you try and figure out how you should do it the different next time. Say it applies to handing something off to customers and seeing what sucked, what was great and adjusting to that. But let's arrange these in order of effect. Which ones get you the most mileage? And you are going to see a list like this, with CoderView right at the top. That is a feature of CoderView. You post something, you post some code, it sucks, it does something awesome, someone checks it, and you get better, better, better and then you merge it. That's a very, very short, small thing like you can get massive mileage out of that. I'm not even going to go into details there. I'm going to assume that you do this. But if you don't do your project, this is probably the biggest thing that you can change that will get you massive benefits out of Hangout. It beats everything else. The second one down the list is continuous integration. A lot of the documentation you see on AppHealth and the inspirational, raw, raw stuff assumes you do this. And when you're building a website or publishing stuff online directly, it's kind of easy to do this. It's almost as soon as it's part of your process. But for those of us building parts of an operating system, it's much harder. Yep. What's the first point about CoderView? And if you use 16-programmet or pair-programmet, do you still need to have CoderView? That's one way of doing CoderView is to pair-program together. That's a good point. And you can make that feedback loop between two people looking at the code very tight. And that's the goal there. So let's look at continuous integration. Because it's harder for us, for a lot of us, where we have to bring this into... Think about how we're going to do it and bring it into our projects, into our open-source projects, into our pieces of an operating system or products or different pieces that we're building. And then we'll look at continuous delivery. A lot of the other stuff we don't have time to look at today, so we're going to focus a lot around continuous integration and delivery for the remainder of this talk. If you look at the agile manifesto that Jen was referring to, 12 principles, 5 or 6, depending on how you read them, are related to delivering working software, testing, delivering. This is the meek of the thing. There's lots of other stuff in there. But this is almost the majority of those principles relate directly to this. That further underscores that these are the foundation, the fundamentals of agile. What is continuous integration? You see this thrown around a lot. It means a lot of things for a lot of people. But I'm going to give you a very rigorous definition. So, integration. Let's start from the back. Assemble everything together in your system, as it would be in production, or reasonably close to assimilate thereof, and then drive it like a user, test it like a user. You don't get to put your hand inside a puppet, play it from the inside, you get to drive it from the outside like a user would. What is continuous? Do that for every single change to the software. Every single change. The changes in your software are like a plug-in in physics. You can't divide any further down. Think about when it changes from this state to this state. There's nothing in between. Often this is a pull request. In the case of an operating system, it might be pushes to this git. These are changes where you go from this state of the software to the next one. Each of those changes need to be integrated and tested. Then you have continuous integration. What's continuous delivery? Continuous delivery is picking one of those changes that you integrated completely and saying, yeah, I'm going to deliver that one. Yeah, I'm going to deliver that one.  But you're able to deliver any one of those changes. That sounds kind of dangerous to me. I'll show you how it works. I'll show you in an action, and I'll show you how to make this work in your project. All right? And that's what we're going to do right now. The remainder of the talk is these two steps. In action, show you what's happening and how to make it happen. So start with what in the world does this look like in real life? In action. We're going to look at the cockpit project. We're going to use this as our example of continuous integration and delivery in action. What is the cockpit project? It's an admin interface for Linux. You can drive it in your web browser. You click around and you can do all sorts of stuff. I would love to talk to you for an hour about this, but unfortunately I cannot. You can drive containers. You can configure your networking. You can do troubleshooting, diagnosis, all sorts of stuff there.  So one of the interesting things about cockpit that's really relevant to this is that it talks to the system directly. It's a real Linux session in your web browser and the JavaScript interacts with the system directly. Here we're running a ping command directly out of JavaScript. This is supported. This is not... This is just part of the normal flow of how cockpit works. Each of those things are implemented in different ways like this. That was executing a command. Here we're going to execute... I call a debus API. So get that. From a browser, we're calling each part of the system directly. Here we are. We open the proxy to a debus object. This JavaScript has no idea about this API. It doesn't know about it natively. It doesn't have any intuition about it. It figures out what the API consists of and here we're calling a method on that API and we're changing the host name of the system. So there you go. And just to prove that it actually does it, we're going to do that from the command line. Again, I would love to talk about this. This is really cool and this is why I work on cockpit because this is amazing capability. But to summarize, cockpit is only the presentation layer. There's no mid-tier. We talk directly to the Linux system. In fact, we talk to so many different projects and so many different parts of the system. This is just a list of projects we have contributed, patches to and fixed. The list of things we talked to is about three times this long. If you want to know more about cockpit, there's a hack fest going on right now. Up, go outside, around the corner, upstairs in this building, room C236 at the end of the hall. If you're leaving this talk and walking out, I'm assuming that's where you're going. If you talk to like 100 different things all the time on all sorts of different Linux system, you can expect a disaster. You can expect those things to version at different rates and just to be an absolute combinatorial explosion. So how do we do this? It would not be possible without continuous integration. This is an example of something that is made possible by continuous integration. It lets you do amazing things you would not otherwise be able to do. We bring up 10,000 instances per day, VMs, real operating systems, per day to test the changes that go into a GitHub in the average day pull request that we create. When you open a pull request, you will see something like this. You can see test suite running on all sorts of different operating systems. There's Ubuntu, there's RHEL, there's different versions of Debian, CentOS, there's different browsers, Firefox, and so on. And each pull request will bring up about a thousand different instances in all those different ways. We found this way cheaper than finding these bugs later. And this happens on every single change. And we have built scaling recently. I didn't start with this. We built scaling so that you can bring up additional machines and it will just make the test go faster. And if some of those go down, it will just make the test go slower. This kind of stuff scales, it's distributed. This is really cool. I'd love to give a whole talk about this, but we're going to move on. To continuous delivery, we release 50 times a year weekly, like I said, pick an arbitrary change and that's the one we release. We sign a tag in GitHub and it becomes a release. What does that mean? Creates tar balls, uploads tar balls, does scratch builds, pushes to code you, does a Bode update, coper builds Debian packages, do two packages, Docker Hub containers, documentation uploaded, versioned for that release. I think someone's bringing in vagrant image creation. These all these things happen on my, like I said, sign a tag. That's the only action that the user does. I mean sorry, one of the project contributors did it. And then all that stuff happens. So obviously you can't do this like with thousands of people from all over the world working on this. You can't help you out. You need robots to do all those tasks. Otherwise it'd be extremely wasteful. And again, if you want to figure out how we do this, continuous integration delivery, actual implementation details, what I just showed you. Go to the HackFest. Go to the HackFest in room C236. Is that the live stream? Yeah, that is the live stream. And this is, yeah, it's very productive up there right now. Okay, so, the point of this talk is not to say, oh this is amazing. In the winter they came to down all the leaves and in the summer and springtime they put them all back up. It would take hundreds of people maintaining one tree. Instead, that's just not how nature works. That's not how it works. This is how it works. You have a seed. Put the seed in the ground. And it grows. Sometimes it's ugly. Sometimes it's pretty. But it grows by itself. And that's how this is, and CI will grow in your project. And here is the seed. I'm going to show this more than once. It's important. This is the seed. One, make the tests changeable by the people who make the changes. Make the tests in your project changeable by the people who made those changes. For every change you can change the test. And post rapid feedback back on each change. Post the results of running a test, even one test, back on each change. If you have one test, even just yum install the thing, start with that. Make that test changeable and post the results back. And everything else follows from that. If you miss either of these points, we're going to put a lot of effort into CI and it's going to be carving the tree by hand. Painting the leaves by hand. If you get this part right, the tree will grow. This is how we started with CI and all sorts of things I didn't imagine at the beginning grew out of it. So, in the corollary, we're talking about the changes. We're not talking about master. Once things are merged together in Git, testing that or testing it nightly. We'll come back to that in a second. But we're talking about the changes and that implies you're testing before you merge. You're testing a change on its own. CI lets you then identify. That's the change that broke it. Because the tests are testing all sorts of stuff. But CI grows from that. The fact that you're testing every change, the guy who made the change, and you allow him to change the tests.  And this is some of them. You'll use a test framework. We can all share test frameworks, if we want. But you might grow your own. A lot of people do. It's not a surprise. Some trees do different things than others. You can even change some of these pieces as you do grow. You'll probably move your packaging if you use spec files and RPMs. You'll probably get those upstream at some point. Why? Because you're integrating it as deployed in production. This will be part of the growth of your tree. Getting those spec files upstream. You'll start to gate. You'll start to say I'm not going to merge this until the test packs. That usually happens pretty early on in this growth of CI. But it happens after you start. You don't necessarily start with that. Scaling and out, distributing your stuff, tracking known issues that can't track down. It happens every thousand tests run. Something goes bump. You'll probably invent a way to track that. We can share knowledge on each of these things and look at what other projects have done, how they've solved them and the infrastructure to run your tests as well. You might grow into the infrastructure like CentOS-CI OpenShift online where you have this kind of stuff available to you. You might start with semaphore. There's lots of different options there, but this will grow as you do. You don't have to be scared of choosing the right one. I could talk about all these things a lot. I have opinions about these things. Sometimes strong opinions. Sometimes ugly opinions. But I'm not going to. And that further underscores the fact it's the seed that matters. These are secondary. It's that seed that matters. This one. Those two things make the test changeable by the people changing the software and provide feedback on every change. So what are the things that you can do to destroy your tree? Come at it with a flamethrower and just burn it to the ground. And Linus Torvald said hey, I'm going to take the build stuff out of the kernel source. You can give me a caller telling me what that thing built. Well, we pretty much forked Linus Torvald. I mean, everyone forked it right away. This wouldn't work. That's the kind of flamethrowers I'm talking about. Here's the short list. There's probably more. And it turns out one of mine finds your bullets. Nice. I wonder if I can help this because we're going to have problems with this. Just give me a second chance. Have you seen more? Yeah, I'll do that. So hide your tests. Don't make them available to the people making the changes.  You might use them in other forms of testing. You can use them in acceptance testing, regression testing, all sorts of other testing. But if you want CI to die, do that. Schedule your tests. Don't do them on every change. Schedule them lightly.  Sure, you can do that for other forms of testing. Acceptance testing, performance testing, lots of stuff like that. But it's not CI. Hide the results. Store the results in a cool, dry place. Make people look for them.  Test after you murder, not before.  And one last thing that somehow doesn't make it up there is force everyone to rewrite their existing tests to your beautiful new framework.  So, this is DevConf, not BullshitCon. So, let's actually look at some code. Here's an example for getting started. Some of you have already started. So, pardon us while we go through the basics, but I want to show you how simple it is to get started. And we even have an example repo ready for you. So if you want, get out of your laptops and we can actually do this. I'm not kidding. There's a repo called Cockpictuous on GitHub. You type on Google to find it. I'll also have the URL posted towards the end. There's an example directory in there. And there's two test suites. They're very simple. You can add your stuff there. And a setup file for semaphore. What this does is it brings in a fedora user line into semaphore. There's no excuses here.  Well, turn it up. It's a trivial to bring in another user line and integrate it the way your user would in production. Or a green multiple. And then run these test suites. You see this on every pull request. This is what the output looks like. And here in our case we're just proving that it is fedora. And then we have one drop here that fails. And that's posted on every change.  Let's do that. So here's Cockpitches. We'll come back to this repo a lot. This is the URL here. Here's the example directory. These are trivial. Pull. Prepare. Pull the image. We have 10 minutes. Mount the image when running. And then we run that this runs the test suite. And the test suites, again, are really simple shell scripts that you get to put whatever you want in there. Yum install. Start with that. This is so easy to get started but it matches that C's. You get results back on every change to develop. We made it and the changes are hackable by those people. So because we're running out of time we'll move on. If you already have a CI system that doesn't meet those rules you might probably are not posting the changes back to the guy who made change. There's an in. Nice. Cut off. In that same repo there's something called the SIG. And you can type your test results. Smaller. There we go. You can type your test results through the SIG and at the end put a JSON line and it'll actually update GitHub for you and link it to the URL that it ends up at. Again, this is in that same repository. Also if you're interested come to the hack test we can look at this more. What about delivery? Automating the delivery. That's not magic.  In this case we are going to use tar ball and patches to from a GitHub if we leave a command called release source we're putting in a directory called source we created a tar ball with the right name that uses make check I'm sorry make this check and create some patches on top of it. This is the current state of the GitHub checkout and it represents the thing that you want to release. Here we have release srpm that takes that and those sources and updates the spec file puts the git tag in the change log puts those patches in sets the version and revision number correctly all without human interaction here we have release codev which actually does another update to the spec file to make sure that the revision matches what gets checked out with git increment there and merge change logs together and commit it, push it make sure it works and so on all of the tasks that you would manually do just happen without any human interaction then release bodi makes a bodi update using that same git tag text which hopefully was descriptive that's not hard and makes an actual bodi update so that it goes out there's a bunch of scripts in this directory that give you an idea that releasing software, delivering the software is not a manual task you see a cobbler by the side of the road and you have a beautiful shoe he's polishing it, he's done such beautiful stitching on a beautiful leather and everything and he shows you this shoe and it's amazing, it's an amazing shoe you're so impressed and you're like where's the other shoe? he's like no, I made this shoe and I spent my whole lifetime using this shoe just so that's what a lot of people attitude towards packaging is whereas packaging should really be you can get it out and get it out regularly and that ends up being one of those feedback loops in agile you get it out to people who can then use it so all of this is in a repo called cockpituous and again, come to the hackfest let's review continuous integration assemble everything together as it would be in production and test it like a real user from the outside do that for every single change and here's your seed to get this started and from which the rest grows make the test changeable by else you make the software changes and post rapid feedback on each change alright fire away yes, how many tags does the repo have or tests does the repo have our repo our repo has git tags we have the numbered tags for each release and we're at around 130 of those plus we have for each rel release we also track branches for downstream releases with all the patches that are in them so we can run our ci against them as well so this in the case of an upstream project I imagine we currently have about 200 tags but that git panels that very well I saw once we haven't found the tags slow it down the tests though eventually as your tree grows you might have such a big body of tests that you start to optimize them and in the case of a project like fedora it is important to choose the right tests because hopefully they'll be eventually a forest of tests like I said rapid change rapid feedback is important and so at some point in your tree not a beginning these tests maybe not all of them yep so it's a question about the release source tool and how it makes those patches changes it turns out that parsing is well you can look at the implementation you can look at it after it will be fun and if people have been contributing back so we've already have some people using this the security guys are using it you're using some of this and so people are contributing back cases where it doesn't work for them and we're merging them bringing that tool to work better but yeah it is possible to completely automate editing a spec file yep so the question is how many lines of code versus how many lines of test code do we have in the cockpit project and it does look like about 30% 40% test code in many times when fixing a bug that gets reported from a real user we look at the bug the first question that people ask in the team is why didn't we find that before this becomes part of your system once you have CI and so it's a one-liner fix almost always for a bug yeah it's a one-liner fix and it's a 20-30 line integration test so when you're fixing bugs that ratio goes the other way in addition on time you do end up spending around 30% of your time writing tests at least that's what we found in our project because we've integrated so much stuff but that saves that allows a small team to do what you otherwise require a massive QE team to be able to do yep no this is question time actually I'm talking way too much go for it it's a good point you want the results as soon as possible ideally how much does it take to do this and how much does it take to do this for the request one so if we look at we have 13 test suites currently and we will depends if there's one pull request open like I said there's this distributor that's scalable and we have machines that come up and find tasks to do and they will paralyze all of this you'll start to get things simple things like feedback from semaphore quickly within about 5 minutes each of these test suites takes depending on how heavily loaded the machine is in our case takes about 10 to 15 minutes and multiple of these are done at once because if everyone goes right now and opens a pull request at once obviously that would take a long time to do the drain but if there's one or two or three pull requests that have been open recently the test drivers will actually share this load across them and you'll get feedback probably all the way across within about 30 or 40 minutes given the current amount we have about 5 or 6 test runners which are all some are dedicated machines some are open stack instances are all over and they like I said it's distributed and they choose to do the tests based on what needs to be done you get around 90 minutes to 2 hours of time of developer's attention before they start totally disregarding your CI I have a question for you how did you get the time to do this so all the features that you were required to put into cockpit cockpit lives and dies by its testing so one of the cool things about CI is that it makes you able to do things that you could not do otherwise whether that's fast paced software development or whether it's completely new software that could not exist without CI and cockpit is in that bucket it just absolutely cannot exist without CI and the whole architectural model of talking to those things directly and not inventing yet another management layer on top of every single rail system is only possible because of CI you should have continued what? right so the question is what is my opinion on what to test my opinion is first start very simple start with installing the thing making sure it comes up don't get too ambitious or you will never start to actually just plant that seed one or two tests you have to plant that seed then you start to expand your test coverage some of that comes naturally with the value of the tests and increasing their coverage and hating to find bugs later the developers under project and they want to write the tests earlier when you start to get more test coverage like that test things you start with testing things sometimes you need a login test the login test your command line tools if you have command line tools test and so on and there is I guess there is no one answer here but especially as you see bugs write a test for every bug you find and all you notice as a developer yourself is that the testing you do manually to check that your thing works you are like why am I doing this over and over again I will just write a test for it even for myself right now because I have to do it five times why should I do it five times yeah if we go to get a little bit support to code review do you have some rules like who can code review like something new the code review should be done with some more experienced people or something with some strategy so the question is about policy for who does code review the answer there is when you are starting any code review is better than no code review in fact if you are in this position code reviewing it yourself is better than no code review even just seeing it in patch format is you will find bugs that you would not otherwise have found they do their code review so then you have continuous delivery ship it and then you will come up with what feels right for your project to have some less experienced review it is absolutely not wrong that is fine back there the question is automate the merge and review the gating of the pull requests and that is one thing we haven't run on our free yet other projects have some projects decide to serialize things they always do their tests refacing that or merging that onto master then doing tests and then doing merge they have bots for this we have decided a little bit more privately there is tradeoffs here to do our tests against the current master if they agree someone can choose to merge them both of them are valid and both of them have tradeoffs and negatives so that is something that is not on our free yet it is part of what belongs on your CI yes so are we ready to test up the door or not we used to but no until the door or not I would suggest running your software as you would in production that so as you would assemble it as in production in our case that is why we have that list of various operating systems he has his hand up right there yeah I am curious about have you found any limitations on this basically we are getting a we are going to open this back as a master we are going to talk about the seat was planted at the beginning and it really worked out as well communication has actually suffered quite a bit and it seems like humans can't really deal with more than about 25 people I wonder if you have found any strategies for this cockpit is a relatively small project right now 6-7 people 2 pizza teams 3 pizza teams 4 pizza teams the foundation of agile is actually built in this premise that smaller teams are better than larger teams the amusing part about this is that the industry has this notion that you can scale this process and make it work and how many in this room think that is like a terrible idea I am going to cede the room because yeah it is hard it all boils down to that idea that you need to scale communication and typically in my experience I will be very clear engineers think that the best method of communication is via email how many agree thumbs up communication experts suggest that email is not the only method of communication surprise and human brains actually do not process information simply by seeing one email they actually need 3 different types actually so 3 different methods of communication in order to actually really digest what it is that you are saying and so text could be one of those methods voice is another method something visual is another method so at red hat here get this we send an email we have monitors all over the company that put the information up on the screen so you can see it where you are in the kitchen and we also do conference calls where people can dial in and listen to what it is that somebody wants to say and so relying is likely what the foundational problem is you are saying when I IRC to say that I want to talk about an email that just sent you you should also call me on the phone and also please what's at me telegram me, facebook message me actually sms message me and then send me an email and I might respond but poll request is another form yes yes so yeah do you also do you also do you also do you also do you also do you also do you also do you also do do you also do you also do you also you also you also you else you you don't you don't chill you know actually you So that would fall under performance testing so the question is do we have some way to detect when something is implemented extremely poorly or an SQL query for example is just wrong for the situation takes way too long and that would be what's called performance testing. I've found that performance testing is hard to bring into your CI because of the rapid feedback requirement and so that's not something that we've done automatically yet I would love to find a way to do that and I feel like that is missing from our tree. The question is if we decommission tests that are not appropriate anymore because some implementation has changed and that is definitely the case it's very the best way to meet direct the first part of your the requirement and the first part of the seed is to store the tests in the same repo if you can as the software itself that's the by far the best way and then in the same full request you update the tests for the feature that has significantly changed. Last question. How do you deal with the authentication because a lot of the things you have shown require authentication because they are supposed to be done by a real user. I mean building packages and coaching and stuff like that. The federal infrastructure team is trying to prevent bugs from doing this and you are developing a bug which is doing it too. So the question is how do we automate a lot of the continuous delivery. You are correct that the current mentality is out of the cobbler and if you see it even in changes that are rolled out in all of our encode for example incompatible with what happened last week everyone is forced like the cobbler to read the letter magazine and figure out which cows are going to be good for the next season and really stay up to date with all of this stuff. Automated. Which cows are good for the next season. The metaphors are a diamond. But the reality is that many of the tasks that you do can be performed automatically and they can be officially in those scripts. If you put your credentials in it right now Fedora has Kerberos on the location. Put your credentials in on the machine and if Fedora requires you to run that script by a person start running it that way. And put your credentials in at the top and run it that way. And you are there. You are logged in and it's running. You are not interacting with it at all. The last place I would suggest you interact with your CD is when creating your give back, signing up in time and everything else goes automatically from there. Thank you very much. Thanks everybody. Oh and go to the hack fest. Yeah. But only if you can type like this. Talk to the top of your CD projects on like the different stages. You can get the CD on your square. You can get the CD on your tracks. I had to try to do something like that. I have to say that most of the viewers might not be able to see it. You can see it. You can see it. You can see it. You can see it. You can see it. You can see it. You can see it. You can see it.


Planned Ignoring. Planned Ignoring is what will happen if you wear this shirt in New Zealand. Just take a look patiently. For those of you that don't understand, here's the anti-Trinations rugby. South Africa, New Zealand, Australia. Don't worry about Australia. Ah, shh, it's telling my Australian friends. Speaking of that, my Australian friends, y'all need to send me a wall up East Jersey. Because I got this one sent to me, and I got me an all-black jersey sent to me right here. Look at this, because the wardrobe would just keep over on the bed. I mean, the couch. So anyway, this is my all-black jersey, right? So, I need a wall up East Jersey. So, first person to send me a wall up East Jersey gets a shout out on camera. As a reinforcer, I guess. Anyway, no, so Planned Ignoring, if you engage in a particular behavior, it's kind of like a type of time out. Yay, spring box! And everybody goes whoop. And they turn their back on you. And they ignore you, and they don't speak to you again. For the rest of the game, I mean the match. So anyway, that would be an example of Planned Ignoring. You plan to ignore someone based on a response that they engage in.


Hello everyone, today we are going to see the programs using function call by value. Learning outcome, at the end of this session students will be able to write a C program using function call by value. Students must watch the following videos on the functions, here I specified the links of the videos. Now in this session we see the problem statements, write a C program to print all the perfect numbers in the given range using the function. Before seeing the actual code on the editor, first of all we see what is meant by perfect number. Perfect number means the sum of its positive divisor excluding the original number. If it is equal to the original number then we can say that the given number is a perfect number. Okay, here the 28 is the perfect number, the first of all we calculate the divisor of the 28. The 28 divisors are 1, 2, 4, 7, 14 and 28, but the sum of the original sum of the divisor excluding the original number. So here when we calculate the sum of the divisor, here we exclude the 28. So the sum of 1, 2, 4, 7 and 14 is equal to the 28. So here the sum and the original number is both are same. So that is why here the 28 is a perfect number. Suppose here the given range is there, the perfect numbers in the 1 to 100 ranges are 6 and 28. Okay, two perfect numbers in the given range, how we check the given numbers are the perfect number or not using function. So here we open the code block, first we create the new project, then click on the console application, select the language C, click on the next, give the project name, perfect, then click on the next button, then click on the finish. In the left hand side the project is created, then double click on the main file here. First of all we declare the two functions, so feature check perfect. First of all we write down the function for checking the perfect, here we pass one parameter of integer type to that function, then end with the semicolon, this is a function declaration. Afterward we write down one function declaration, perfect number. So here we pass the two parameter, starting limit and ending limit, integer ending limit to that function, here we declare the two functions, okay. Next in the main function we take two variables, starting limit and ending limit, okay. Here we display one message on the screen, enter the starting limit, enter the starting limit and that value is stored in the variable stl, so here we use the format pacifier per sendee and that value is stored in the stl, then end with the semicolon, next we display one more message on the screen, enter ending limit, ending limit, here we use the scanner function per sendee and that value is stored in the enl, next here we display one message on the screen, perfect numbers between per sendee, per sendee to per sendee r, that value is stored in the stl and endl and limit, okay. And here we call the function perfect number and here we pass the two parameter, perfect number, here we specify the perfect number and here we pass the starting limit and ending limit to that function, now after one we write down the function definition for check perfect, here we pass one parameter of type integer, here we check the actual perfect number, so here we take two variables i, i fault is a loop variable and we calculate the sum for storing the sum we take one more variable sum, so we here use the for loop, here i will be starting from 1, i is less than till n1 and then i++ every time, here we use the if decision statement, here if n1 mod by i equal to equal to 0, then we calculate the sum, so sum equal to sum plus i, here i means what we add the sum, after that after processing finally in some variable it stores sum of its positive divisors, then after for loop here we check if sum equal to equal to original number, if it is then here we return the value 1, otherwise we return the value 0, 0 to the function, this is the function to check the perfect number, then here we write one more function void perfect number, here we pass the two parameters integer, start limit and integer end limit, here we use the for loop for iterating from starting limit till ending limit, so here we write down stl means what starting limit to end limit, here we call the function check perfect and where we pass the starting limit to that function, here we display on the screen the divisor the perfect number devices, percent d and here we pass the stl end with the semicolon and here we increment the starting limit, after that we compile this program, so here the starting limit is 1 and ending limit is 100, so the perfect number between the 1 to 100 is 6 and 28, here we specify this slash t, the perfect number between 1 to 100 are 6 and 28, now after seeing this program, can you write a C program to check the given number is the amstrum number or not, pause this video and think about it by using function, so here first of all we write the written type of the function, here we specify the integer as written type of the function and we give the name to that function as a check amstrum and for that we pass one parameter of integer type, here we check or declare variables LD means last digit, sum for soaring the sum and number, here we initialize the sum to the 0, here after iterating original number it becomes 0, so that's why here initialize to the original number n1, here we use the value while num1 num is not equal to 0, we do the processing, so here first of all we calculate the last digit of the number, so that's why here number mod by 10, here we apply the mod operator and the given number last digit is stored in the LD, after that calculate the cube of the last digit and add to the sum and finally we remove that number from the original number, so that's why here we specify the division operator, so num equal to num divided by 10 and then then we here return the value n1 equal to equal to sum to the function change the number, now these are the references, thank you.


My friends, welcome back to my channel and a very special welcome if you're new here. My name is Jen. I'm a certified weight loss and nutrition coach. I follow weight watchers and I count macros and calories. Happy Monday, happy day after Christmas. Happy meal prep day. I have three simple recipes, but packed with protein to finish 2022 out strong. So if you're excited to give this video a big huge thumbs up, make sure you're subscribed and your bell notification is turned on because we do a meal prep every Monday and I upload five videos per week. Don't forget my recipe website. We'll be linked down in the description box where you will find all of today's recipes. Nutrition coaching for personalized macros and calories cannot recommend this enough. Get your macros and calories done so you're ready to hit your 2023 goals and I have one-on-one coaching if you would like to chat with me directly. Links and discounts to my favorite things and my Facebook group, come join us there. We love to have you are all down in that description box. So let's jump in to meal prep. For breakfast this week, I'm making sugar cookie overnight oats. I am really excited for these. They are packed with protein and they're giving me all the holiday, New Year's vibe. So let me show you what's in our recipe. First, you're going to need vanilla extract, sugar alternative of your choice. As always, I'm using Loconto, granulated. It is my all-time favorite. For me, it tastes the most like sugar, doesn't cause any digestive upset. Really, really delicious. I will link Loconto down below for you with 15% off-site-wide. You're going to need rolled oats, non-fat Greek yogurt, vanilla protein powder. This is a new one to my life. I actually haven't even opened it yet. I picked this up because as I mentioned in my goals video, I'm really cleaning up my diet in 2023 and that includes choosing really high-quality ingredient foods, including protein powder. So this is from the brand Ascent, which I love. I've always loved Ascent protein, but this is there, my seller, Kasein protein. It's a little bit slower digesting for sustainable recovery. I have vanilla bean, 25 grams of protein, really, really, really clean ingredients. I'm going to utilize this a lot post-workout as well, but I thought I would try it out in these overnight oats. I did buy it on Amazon, super affordable for how high-quality it is, so I'll link it down below for you. You're also going to need salt, almond extract, and then some mason jars. I'm going to make three days worth of overnight oats. I also forgot to mention that you'll need some milk of your choice. I'm going to use the low-fat Fairlife just for that extra punch of protein. To put together our overnight oats, we're going to add one half of a cup of non-fat Greek yogurt, two tablespoons of whatever protein powder you're using. And then whisk that together until combined. We're then going to add in half of a cup of rolled oats, splash of vanilla extract, and a splash of almond extract, two teaspoons of Lakonto Golden, pinch of salt. Pop your lid on that and shake it up really well. Here's what the sugar cookie overnight oats look like. Let's put together the other two jars. So here is the sugar cookie overnight oats. I did have four lunch this week. I'm making barbecue pineapple chicken. This is a super, super, super simple recipe, exactly what we need after the holidays, not to mention it is packed with protein. So let me show you what you'll need for the barbecue pineapple chicken. First, you're going to need chicken breasts, salt, and pepper. I will link my gravity fed salt and pepper grinders down below. You know how much we love these. You'll need some no sugar added barbecue sauce and a can of pineapple. You could do chunks, tidbits, crushed, whatever your preference.  We're going to add about a third of a cup of our no sugar added barbecue sauce right on top of our chicken. Go ahead and spread that out over the top. We're going to take our can of pineapple drained and add that right on top, and then drizzle the other third of a cup or so of barbecue sauce right on top. We're going to cover this with some saran wrap, throw it in the fridge for about 30 minutes. You can actually leave it in the fridge for up to about two hours. Our chicken is going into a 400 degree oven for about 40 minutes or until the chicken is cooked through. All right, barbecue pineapple chicken is out of the oven. It smells so good. I am so excited for this all week. I cut mine into about six pieces so I can have this for lunch each day. I'll pair it with some carbon, I'll pair it with some type of healthy carb, maybe some fruit, some veggies. I'll go ahead and put points, calories, macros here on the screen for you, but such an easy protein packed lunch. For a sweet treat this week, I'm making pecan pie bars. These would be a perfect thing to bring to a New Year's party and they're just a really good sweet treat for the week. So let me show you what you'll need. You're going to need some corn syrup. If you can pick up the dark corn syrup, that's even better. Light butter, four eggs, all-purpose flour. Again, sweetener alternatives. I have Lakonto granulated and Lakonto golden. Don't forget Lakonto is linked down below with 15% off-site-wide. Some salt, vanilla extract, and of course, some pecans. So we're going to add one and three-quarter cup all-purpose flour to a medium bowl as well as half of a cup of Lakonto. Mix that together. And then I have three-quarters cup cold butter. So we're going to just dice that into the flour. Chop it in there to make the mixture for the crust. You want your mixture to be a little bit on the chunky side. We're going to spray a 9x13 baking dish with some non-stick cooking spray. And then we're taking that flour mixture, putting it into the baking dish and then pressing it into place, forming a crust. Because I'm using an aluminum pan, this is going into a 325 degree oven for about 20 minutes. We want the crust to be browned. And while it's in the oven, we'll prepare the filling. For the filling, I'm adding two-thirds cup of Lakonto golden and then one-third cup of all-purpose flour. Make sure you mix those very well until they're fully combined. And we're going to add our four eggs. Give that a quick mix. We're adding in about a tablespoon of vanilla extract, salt, and the one cup of corn syrup. And then we want to stir that until fully combined. You're going to let this mixture stand for about 15 minutes while the crust finishes cooking. We're going to go ahead and add in our chopped pecans and then give it another stir. The crust is out of the oven. We're going to immediately add the pecan pie mixture to the top. Spread that out nice and evenly. And our bars are going back into the oven for about another 20 to 25 minutes or until the filling is cooked through. These absolutely gorgeous pecan pie bars are out of the oven. Again, these smell incredible.  Pan makes 24 bars, so you're getting a pretty good size bar. And compared to the points and calories in pecan pie, you can't beat it with these bars. So I'll put everything here on the screen for you and don't forget all of my recipes are on my website, which is linked in the description box. Thank you for joining me for this week's simple yet packed with protein meal prep. If you enjoyed today's video, give it a big huge thumbs up. It really helps out my channel and don't forget to subscribe and turn your bell on so you never miss a future video. Recipes are on my website, which is linked down in the description box along with nutrition coaching, links and discounts to my favorite things and come join our Facebook group. We'd love to have you. Thank you for watching friends. Happy Monday and I'll see you in the next one. Bye.