Mae'r gweithio gyfarfodd, oherwydd y gallwn ni fydd yna'r gweithio gyfarfodd. Rwy'n cael ei fod yn ddod. Gallwn ni wedi bod wedi'i wneud o gydag. Mae'r gweithio'r gweithio'r gweithio ar gyfer y cyfnodau mewn gwahanol, ac yn cyfrifio'r cyfrifio, ac yn cael ei wneud o gydag. Ac ynddo chi'n ei wneud o'r gweithio gydag. Rwy'n cael ei wneud o gyfrifio'r gweithio, oherwydd yng Nghymru. A'r gweithio'n hetoddyn nhw, If I was a marketing person yesterday that kind of annoyed me so I thought, hey, I would tell you that I am not a marketing person, I am a technologist I was a programmer, I am slightly power hungry as you would expect so I became a project manager, because I wanted all that power. Mae'r cyfaintrwaith. Mae dwi'n gweithio o DDSDm, mae'r f projectingau ynglyn yourselfaeth a'r cyfaintrwaith o'r DDSDm. Mae'r cyfaintrwaith o ddydig. Mae'r cyfaintrwaith o'r ddiwedd, oeddech chi'n gweithio. Ben ddiwedd, i'n ffordd, y cyfnod y connects. This is probably the least risque picture I could find for this presentation. They introduced me to Agile. It was the most amazing experience I could have gone through, right? Mainly because there was plenty of underwear right around the office and then a whole bunch of stickies on a big glass wall trying to explain what the website would deliver and when it would deliver it and whether it would ever deliver it. this was the new product that we wanted to deliver, actually, it was a new platform, the team I was working with wanted to deliver. So that's kind of a potted history of how I got into Agile, I think the point at it wanted to make was that I haven't been certified in any sense of the word. I've learnt from the job and I know that's slightly different from where we're coming from I know that London is an environment definitely values experience over qualifications But I do think it's really important to do. So I'm here to talk about hacking the sales process with Canban and Agile. I wanted to talk about it because I've been doing it for the last 6 and 7 months. It hasn't been a smooth process. So it's not been...if you're coming to this talk to find out how to do it for your own organisation, probably won't have the answer for you. I'm sorry. But I can tell you how not to do it and how not to do the obvious things. I'm also going to assume that you know nothing about my organisation because we started two years ago. We started in 2013. We're a group of software developers. We're a group of software developers that all met at Figleves as it happens. We're all a little bit older, so we're all in our sort of 30s and 40s, so we have a lot of experience. We can add up over 100 years of experience and good experience timelines. We are all slightly burnt out. That's probably the best way to describe us. We are all people who want to sit on the beach and programme. We're not a bunch of people who really want to be in an office, and so the guys on the pictures behind you, they are based in Barcelona.  We kind of run our sprints weekly with them all traversing all over the place. What we do, or why we exist, so someone yesterday was talking about, I think it was Mr Toad as he called himself. Toad was talking about having a vision. So we decided that we had a very clear vision of what we wanted to do. We wanted to make decision making software. When I was a kid, there was a very little choice, right? And we're now completely and utterly overwhelmed with choice.  You knew what that meant.  Now I have no idea. I have no idea if I'm going to get a paper boat mango juice, or a Coca-Cola, or a Pepsi leher, or whatever it is. I just don't not know with the wide choice that's out there which one to choose. I'm a bit useless at choice anyway. Not only that, we have technology that's kind of pushing this towards us. They're trying to make choices for us, right? My phone is constantly telling me that right now it's telling me that I'm two hours away from the Bangalore airport, and therefore about five hours from my home in Delhi. Continually telling me this. I don't want to go to Delhi. I really don't want to go to Delhi. It may not even be my diary it's telling me about as well. It's actually trying to tell me about my, in this case, my CEO's diary. And obviously what we are really interested in is this quote from Stephen Hawking's, which talks about could it all be the end of everything?  Might not. I don't know. We kind of don't believe that. But when it comes into technology, agile delivery, there are definitely things that we are doing. We're looking at reports, we're looking at KPIs, we're looking at numbers. We're kind of driven by all of those items. We're not really making decisions for ourselves. So what it means for us as an organisation, and I won't dwell too much on it, is that we focus purely on retail and purely on helping our clients make better decisions.  We have no idea what size we are in stores. That's compounded with online shopping. So we came up with our first product a couple of years ago, which is called What's My Size. And you put in your measurements, we tell you what size you are in each store. We went B2B 18 months ago, so we had to re-platform, and we actually deliver within product pages on e-com sites. We have a couple of other products all around iPhone and delivery, which I won't bore you with. But the point of that was to tell you that we built the whole thing in agile. When we started, we started from the ground up. We said, right, we're going to get together, we're going to do this, and how we're going to deliver it. And we decided we were going to do one-week sprints. Remember, my guys do not week to week. I don't know where they're going to be. I also don't know what country or what time zone, and also their availability. So we run one-week sprints, which they tell us on a Friday that they're available for the next week. We then do our, on a Monday morning, we do our retrospective and our sprint planning. And it allows things for this to happen. So one of my developers was in Barcelona meeting another developer and the team, fell off his bike, ended up in hospital. It allowed us to kind of adjust our sprint planning into the next week. So extend into weekends, deliver on time. But within people managing their own timelines, I think Zee talked about it yesterday where he was talking about not being linear in your processes, but kind of just extracting and working on the basis that you want to. I haven't used his terminology, I apologise. I've noticed that we haven't talked much about tools over the last two days. I don't know if it's interesting to you, but I will talk about ours because I'm in love with them. Going back to decision making and technology, telling me everything I need to know. These are the ones I'm in love with. We use a trailer backlog, obviously. You probably all do too. We use Skype to do our stand-ups. We know exactly when we should be there. We use Slack to message, which is probably the first thing I've been using recently. I'm going to have to admit it. We use PowerPoint, but that's because we have to talk to our board directors. Going back to my very first slide, one of the most exciting things about where I am at the moment is we're an all-women director team. So it's a really exciting place to work, but they still need PowerPoint slides. So I need to change that. We obviously use things like wiki and WordPress in big bucket. Okay, so the problem. We definitely had a sales problem. We had grown to 12 people as founders, and I know that a lot of you are from service industries here, so you've got much bigger organisations, much bigger teams, but as founding members of a company, you do your own sales. That's pretty much how that boils down to. My CEO was doing it. She invited me to come back six months ago and join her, and she had grown an amazing company. We've just closed 10% profitability in the first year. We have all our sales tasks effectively in her head, but as I'm being videoed, I'll repeat that and say, obviously, it was documented all over the place and it was just very low, et cetera. Not really true. All the tasks that this customer needed this item and we needed to close this contract on this date just wasn't documented anywhere in the process. The other thing to note was that last-minute requirements were completely killing our sprints. Obviously, we do our planning on a Monday, someone would go out and do a sale on Tuesday, everything needed to be changed, and that kind of, I don't know, cumflet ddiwedd lawerol ar gweithreoli ni i. Mae gennym i ymdain o'r taw, neu mi oedd y taw o'r taw o'r taw o'r gweithreoli. Fellywch yn ni oed i lle i ymddangos. Felly gweld hynny o fydda nhw, yn y rhaid i fynt o'r bach ac ydych chi ar sucif arall a fyddych chi'n wneud i'r newydd nifer o gyfer am yr awd. Felly byddwn i'n gwneud ymdangos. A ydych chi'n gweithio'n taw I was asked to sort of talk about how you influence change within an organisation. Evan talked about it this morning, how do you talk out to your marketing people, how do you talk out to your sales people etc etc. But actually remember I said I was power hungry. Life is about your personality of change. I think somebody yesterday talked about anthropology. You really have to understand the people within your organisation, the people who lead it, but also the people who are going to deliver against your thoughts, your ideas. Obviously India has gone through a huge amount of change with its new leadership. You can see that, you can feel it in the country. It feels so different from a year ago. Obviously more relevant to today, you can see change not happening. The England cricket team is nowhere to be seen. Obviously there's a lot to do with the fact that they haven't changed leadership at all within the last few months. They haven't made, they didn't make enough changes I don't think in order to get out of their group, but that's the case. My kind of leadership is definitely about, I said I'm power hungry, I've got my tantrums, I definitely do. If I'm saying that we're going to make a change, I'm co-founder of the company, I kind of get to say that. But that's not enough, right? It's not enough just to say hey I've got the job title, I actually have to take everyone with me, I have to go and talk to everybody. And what's really interesting for my journey was that I'd never done Kanban before, I'd never done it. So when I joined the organisation they started teaching me. They started teaching me and saying hey we've got this amazing process that you have to, you know, you put it out into the board like this, you take your tickets off like this, it's kind of, you know, we fit it into our scrum like this, we're writing a playbook, you should read it, you should understand it, it's like okay, I understand that. I still have a sales problem, I still don't know what sales are going to happen on one day. So I still want to CRM system, I'm kind of traditionalist, I want to CRM system. Right? And my criteria for a sales CRM system was price, usability and prettiness.  So definitely usability is really important to me. We looked at a bunch of things, we looked at Salesforce's most powerful system, if you're in enterprise level organisations, you'll be using it, I'm sure, or have interaction with it. We looked at Affinity Pool and Signal 37. My favourite was simply HQ, which kind of every time it went wrong, sent out some Nordic 404 errors, it's kind of there out. Does anyone have any questions on choosing a CRM system? No. Cool. I chose Bittrick. I chose Bittrick and why I really like it is because, as I had just said, I was being canonised. Is that a term? I'm not sure. Anyway, I was being indoctrinated and as I was being indoctrinated I was thinking, oh, this would be an ideal way to do your sales, right? You identify all your leads. You identify all of those leads that kind of might convert this month, this week, et cetera, et cetera. Then I'll actually pick out who those people are and actually contact them or get them to a seminar or do something with them this week and then actually close the deal or put a deal in front of them over the next few weeks. So being able to manage that in a process that's linear, this system allowed me to do. So, the next change we made to the whole organisation was that we made sure that everybody who did sales and was doing a sprint came to the same daily stand-up. And so they actually came in and they talked about what sales items they had. It got really frustrating. So this is the first thing where I had to learn really quickly. It took what was actually a 10-minute sprint. They go, what did you do? What are you doing today? What do you need from us? To being, oh my god, I've got this customer and they're being a pain and they want all these different functionality and they want it by next week and I can't close the deal. It was just like an angst ridden session. So we had to close it down and go, okay, that's not what we need you to do. What you need to do is retrain and actually say, right, we'll do these tasks and if it doesn't get done, we'll escalate it and we'll talk to somebody else. Obviously remember the problem with being power hungry and at the top of the list is that, you know, there isn't actually anyone to escalate it to so I have to go into a room and think about it somewhere. The other thing, so the next thing that you would think was quite obvious is to decide whether people within the sales function, i.e. me, the CEO, the other couple of people who do that would be coming to a planning meeting and actually sort of dictating what functionality and what tasks they want during the week. We looked at it. We looked at it really, really closely. We decided that those angst ridden conversations about are we going to make money, are we going to close the deal was just too depressing to have once a week in planning. What you want in planning is a much more upbeat conversation about what we're going to do and how we're going to deliver it. So what we have is something called the weekly sales call. It's got its old fashion terminology, which is fine, it's not agile, but it's actually a story writing session. It's a story where, so we do it about an hour before, I do it at 9 o'clock in a Monday morning UK time and then we go into planning and within that we spend our time kind of figuring out what stories are relevant, what things are actually functional stories for product development, what are stories around managing the client, what are just DevOps stories, kind of pulling those apart, explaining that to the salesperson who's leading the contact with the client and then directing them into the right board. And then it's up to them to then get to the right daily stand up and to get to the right sprint planning session if they feel it's important. I'm not sure if this picture does it justice, but it's kind of there. So, you know, we've got our roadmap, we have that very overlying roadmap that we talk to our board with. We have our sales meeting that's bringing up more stories. We obviously have our retrospectives that are more standard stories going, my server's gone down, there's not enough space, I need this kind of story onto the backlog, that kind of maintenance stuff that's coming through. And we're obviously managing that through the sort of Trello and Scrum process. Felly, an example. So, Z yesterday said, isn't it awesome to have marketing people in your sprints? It is. It is amazing having marketing people in your sprints. We had to relaunch our website. In fact, I wish I had a picture of our old website. It looks like the Indian train timetable website is pretty awful. But we wanted to convert that to being something a bit more glossy. The first time we did it, I wrote it in WordPress over a Saturday because I was a bit bored. But the second time we reiterated it, we put it through a sprint. And the reason that we put it through a sprint was that our sales team has said, it's pretty crappy. We need to fix this. We need prices on the website. We need to know what the product does. We need to have material about what's going on. And it's like, well, that's all like marketing stuff. That's all like literature that needs writing. That's all like pictures that need to be taken. My product, you know, my self-development team isn't going to know how to do that. So, what we did was we got Kerry, who's awesome. We got her aligned in the sprint. We actually got her to work with me to run it. We then got our scrum master and we put them together in one place. We decided what the website needed to look like, but we did that. This is a one-week sprint. So, in the first day, all we said is, this is all the stuff we need. On the second day, we simply said, right, okay, this is all the marketing material you're going to get by Thursday because we can't launch on a Friday because, you know, it might fall over. So, we're going to launch on Thursday. So, all the marketing material was delivered over the couple of days, all pulling it through the Kanban, Trello, Storyboards. We didn't do all of it. We forgot to do the QQs requirement, which is likely important by EU legislation. But we focused on the stuff that our sales team needed to meet their delivery to the customer and their sales pitch was on Friday. So, I was pretty excited about that. And it was really exciting to actually not have any technical people in a sprint for once. It was kind of fun. The only bit that we couldn't do was delivery out to live. So, we actually had to, the go live process actually needed some technical support. So, we're done, right? I'm on the beach. It's all calm. It's gorgeous. No. It's all about numbers at the end of the day. It's all great. We've hacked the sales process. We now know where the sales processes are. We know that my client A needs these three things before they're going to sign on Friday. We know we need these two things done by next Tuesday in order to get in front of them, et cetera, et cetera. But we're still not closing deals. It didn't solve the underlying problem, which is why I was brought in, which was that actually we need to close some deals. Now, that was problem one. So, hold on to that. We do something else at 50. I didn't implement this. My CEO who is brilliant, she came up with this. Has everyone seen this before? These are the 12 lists of employment, employment questions. Yes, one nod. If you haven't, there's a link going Google it. It's fascinating. What we did with it was we went right. At the end of the retrospective, we said how many things went right, how many things went wrong. We took the number from that. So, if it was like three items and five items, and we just took that number and read it out. And then we went right around the team and said, I don't know, my supervisor or someone work seems to care about me as a person. We asked these questions. What was really interesting to us was we were getting consistent feedback over about six weeks, a month, six weeks. The thing is that we weren't closing sales. We haven't been closing sales for a variety of reasons. The product may be what was a seminar yesterday or so. It might be one of the dogs. It might be one of the cows. We're not quite sure which quadrant it is. I might need to go back and do some homework. But we know from our own point of view, as individuals, that actually I, as an individual, is I'm not a salesperson. It's not what I do. Remember, I'm not a marketing person. I said that up front as well. But I'm not a salesperson. And I'm doing a job that actually I'm not enjoying. I'm taking tickets off a board where I'm going, actually, I don't want to phone that person up. I don't want to ask them what their requirement is because they're killing me. I don't really want to write a contract for them. But that's just me. There's going to be other people who want to do that, right? I skip over this slide, I think. We had to hire. We had to change. I had to find somebody who I could give these tasks to, who was really interested in making those things happen. They were as passionate as our products as I am. But they're more passionate about phoning the clients up and convincing them. And that was really important. So I think Evan, to call you out, talked this morning about the constant hiring, the agile hiring process, right? And that's exactly what we do at 50. We constantly have a group of people that we're continually talking to. And the salesperson that I picked, his name's Sam, he had been helping me from last August. I gave him a call in August and said, I'm back in town. Do you want to hang out? And since then, he once a month would introduce me to a new sale. Off his own back, I didn't even buy him a drink to make it happen. So you know he's a good guy. And then about two months ago, he said he was looking for a new role. And about two weeks ago, he joined us. We had already lined him up. We knew that he was the person that we wanted to work with should this day come. Kind of probably was going to come. And also, we knew that he was in for the long call because he was there and he was consistent all the way through. And we do that with developers. We do that with testers, et cetera, et cetera. We keep a roster of people who want to help with work with us, come in, do contract, freelance, take them on permanently. So, how much time do I have? I've gone really quickly. I apologize. You'll have loads of questions or you can get to the cricket. So this is two weeks. I'm like two weeks into my new exciting relinquishing power. Remember, power hungry. It's really hard to give all that power away to a new person coming in. They're not a fake product manager. They need to be a real product manager. I need to empower Sam to be able to talk to clients, make decisions, change processes, do what he needs to do to be successful. But that tension remains in the room. Is it the sale? Is it the client who needs to deliver? Or is it us that's trying to push our products forward? Is it our mission statement of helping decision makers? Or when the client says, actually, I just want to mine all your emails and spam them with my latest dress or something. My salesperson goes, yeah, that's exactly what I want. That's exactly what you should do. Buy it for us for millions. Do we let him sign the contract? It's hard to say. I do know if I'm talking to all of you guys over the last two days is that what I need to go home and do is do some serious training. I think I need to go back to the white board and say, right, Sam, you've come into lead the sales focus. We're about four or five people underneath you. They're all agile enabled. They know how to get their requirements out from your head, which is effectively what's going on, and your conversation over coffee and your promises that you may have made, and then actually put that into a process where he understands what the consequences of that are, put it into the process, and then deliver that out. I just wanted to say, watch the space because I think I need another six months to run the trial and see where we get to. I apologise. It was very, very short. Any questions? There's a good question. It changes per individuals. I can talk about them. Developers are thrilled. They're absolutely thrilled. They really want to process. They really want to know what we're doing. They really want to know what customers we're talking to, and when we're talking to them so that they can help us. Also, because they're opting into the next sprint, they can figure out that cool piece of functionality is going to come off the backlog and get delivered next week because that signature is going on that contract. They will put their hand up and change their holiday schedule, et cetera, et cetera. I'm interested. I think I can't remember whose talk it is, but there is definitely attention at the other extreme where you've got people going, look, I don't want to deal with 10 stakeholders in your team. I just want to say, yes, I'm going to sell this data model, or I'm just going to send this application, or I just want to change the pricing. That's actually what we did last week. We changed the pricing for all of our products.  You suddenly are being pushed into this environment where you're changing really, really fast, and that is causing some issues at board level, which says once a month ago you said you'd make this kind of profitability because you're selling at this kind of numbers, and then this month you're saying it's completely different. It's one client, higher margin, that kind of thing. Is that helpful? Good question. As we're quite small, most of us own the company. There's no reason to have a commission-based process because we don't actually have bonuses either to preempt that conversation. The idea is that hopefully we will be, once we exit or once we get to another level and we start paying shareholders, that would be our bonus and profitability. The model that we've got for our new sales people is on a commission basis as it happens, but what's interesting about that is that the functionality that they are over promising, I think was your question before, the kind of diminishing profitability out of their overselling is also coming from that product income. It's literally going right, okay, you're selling this, you're taking this much commission, but the product does this at the moment and you've oversold by this much functionality, that's going to come out of this project profitability, and then we're looking at each project profitability as such. Does that make sense? That's good. Mostly is the development process along with the sales process. So when I was selling, not much, but then I'm very honest and I'm very honest about where we are in that process, and this is a real tension. This goes back to one of the things I do not know how to solve. I'll put it back out to you. I'm a technologist. I want to sell what I actually have, the idea of just going in and lying and saying, hey, we'll do all these other things. Now, I'd be there. I've worked with salespeople who have done that. I think what we're moving towards, in fact, I had that conversation very last night when I got back that said, I'm quite comfortable with you saying that we could do this functionality, this functionality, and this functionality, as long as you say it's on the roadmap, as long as you say it's, you've put it onto the backlog and that we are going to deliver it at some point within the foreseeable future. Remember, if we looked at sugar, I remember looking at sugar. It didn't, for some reason, I couldn't get a licence. Is it possible, because I was in India at the time? I'm not sure, because I couldn't, because some of these licenses, it's open source. I can't remember why we didn't, it didn't make the final list. We certainly looked at it. I'll have to dig out my notes. I'm really sorry. Microsoft's interesting in that we definitely wanted a sort of, again, it probably is now open source, but at the time we kind of, I have my own personal objections to Microsoft, which I haven't got over. Anything else? It's a good question. So, we have adjusted retrospectives to fit in on a Friday afternoon if we need to. We don't fix it to a Friday afternoon in case the developer decides to, I don't know, one of them went off and did a marathon and then did the rest of their work on Sunday.  Sustainability is actually much better. I've done lots of different types of sprints, so three weeks at Iris that we were talking about earlier, we did continuous kind of three sprints, three week sprints cycles. It's just a bit slow. You know that momentum that you have, you start at the beginning, you're really excited for the first three or four days, you've got all these tasks, you know what you're doing, and then you get to the second week and you're like, oh, I've got another week of this, and my tester doesn't have anything to do because they'll be doing whatever they're doing. And then you get to the third week and it's like, whoa, I have to get this done. And then everyone does late nights. What we find with this is that you compress that, right?  And so from the sustainability point of view, I would advocate one-week sprints. I'm a big believer. Yes. I'm sorry, and I should have mentioned that in the talk, that I think is what makes it feasible. We talk as a team on a Monday morning, we get as many of the board in as we can so that there's some discussion about the product roadmap if there's a client who's gone off-piste and wants to do something completely different. We get all the sales team in who are talking to individual clients. So when we say sales, this is a nominal term, it's not a job description, so obviously, sorry, it's not a job title. So obviously anyone talking to a client is considered sales in this and they would come in. And I think, yes, having a week is quite useful. And then something you said earlier was that these couple of us are like lone hounds. We just want our client. There are a couple of clients where it's like, my client, I want to give it to you. It allows that for one week for those guys to go off, have their direct conversations with their clients, and then come back and report if they've made any progress. It does the other thing as well. We're in short products sales. We need to move on really quickly. These are retailers. They're paying about 1,000 UK pounds a month for this product. It's not like selling half a million pound tech project. So we need to know in a week or two weeks whether they're going to close. Now, obviously, we do services. If you look at the website, you'll realise we do services as well outside of product. Those sales cycles just do not fit into that one week model. You will be talking about requirements for two months and then say, right, we're going to do it agile and it's just on this bucket of money. Anymore? Cool. Thank you so much.


Welcome to this new Calix webinar. My name is Hanat Kavokant, I'm principal engineer at LightBan and today I will talk about how to build orchestration sagas with Calix workflow. So there are two types of sagas, choreographed sagas and orchestration sagas. In a previous webinar I covered choreographed sagas, so today I will talk about orchestration sagas. In both cases, a sagas is about implementing a longer running process in distributed systems. We are talking about a process that spends over more than one transaction. And as you know, a Calix entity lives in its own transaction boundary. When you want to mutate two entities, you send a command to one entity, you mutate it, that's one transaction and then you send another command to a second entity and that mutates this entity and that's yet another transaction. You cannot modify two entities in the same transaction. But in some, for some use case, I do need to modify more than one entity with one incoming request from the outside. So for choreography, we can use subscriptions and listen to advance and state change from my entities and then propagate the change. The other option that we have is to use a Calix workflows. And this, when we use Calix workflows, we can then implement an orchestration sagas. And it's about to build this process but having a central point. We can say that it's a coordinator or a conductor in an orchestra that is driving and managing the whole flow. So orchestration sagas is about implement a coordinator or a conductor in an orchestra that will be responsible to manage and drive the whole flow. You start, it sends commands to an entity, when that entity finishes processing this command, the workflow can move to the next step and send another command to another entity and so on. What's important to know that Calix workflow is guaranteed to run until it finishes and it saves the steps where it knows where it stopped. So if you are in the middle of a workflow and for some reason you shut down your service and break it back or there is some error that happens, the workflow knows where it was and it will restart from where it stopped. So just very high level, I will use, I have some diagrams to explain what are the main features in Calix workflow. And here let's imagine that I have an e-commerce application and I have something trigger the checkout of a shopping cart. So the input here is the request to checkout or a notification that the shopping cart has been checked out. So I built a workflow and this information gets into my workflow and initiates my workflow. And then the first thing that the workflow will say will do when it's initiated, it needs to do its first transition and it simply moves or indicates what is the first thing that needs to happen, what is the first step in that workflow. So the first transition here is, is a query payment. So Calix will move to the step, is a query payment that you define yourself and then a call will be made to some payment service to execute it. Now, if that call fails, it's not a problem, a workflow is designed to just retry it. So it will retry a few times, it depends, you can configure. If you say nothing, it will retry forever with some intervals to not overload your systems, but it will keep trying. You can also put a limit on that and say okay, I will retry maximum three times, let's say. But let's say that here we will just make a call, let's say it fails the first time and then it passed. Okay, I will retry and then it passes. And then I will transition, I will transition to initiate shipment that itself, this step also executed a call to the shipment service to initiate the shipment. And this call returns correctly successfully, then I transition to the end of my workflow. You may have arbitrary number of steps and combinations, you can move to one step, go back to your first step, doesn't matter much, you can decide how you want to compose them. As I was saying, you can, by default, a workflow once it started, it needs to run until the end. But you can also put a timeout, you can define a timeout, say okay, this workflow after one hour, if it's not completed yet, I need to tear it down. Or you can say that for every failure that I have in my workflow, for every step that fails, I want to retry that step at least three times and if it keeps failing, I will abort my workflow. You can have a timeout at the workflow level and retry strategy at the workflow level, but you can also have a step level. So you can define both timeouts and retry strategies at both levels. Now, one other important and I find it's quite interesting feature is that you may, you can also execute a step, it succeeds, but instead of transition to yet to the next step, you decide to pose your workflow. When you want to do that? Well, usually you want to do that when you do whatever you can do with the information you have and now you need to pose because you are waiting for some input from the outside. It can be a subscription, something that will trigger subscription that will come back to your workflow, and put it back into movement or it can be just that you pose and you expect that the user will come, connect to your user interface and fill some form, submit and continue the workflow. So the possibility here is to stop your workflow waiting for more input. That input can be from a user or it can be from your system, something that happens in your system that triggers a call back to your workflow and put it back into movement. And again, you pose, some requests come seen, you move to another step and then you transition to the end. As I was saying, I will use today the same use case that I implemented for choreography sagas. But let's recap, if you haven't seen this other webinar, I will explain here what is the use case anyway. So in event-sourced system, there is one major challenge. It's how can I have unique fields inside my model that are unique across all my entities for that specific model. So this problem is called set-based consistency validation. So let's imagine that the user here is an event-sourced entity and I have a unique ID that I can always use to reference to different models to different instance. And inside I have the name and I have an email. But I don't want the email to be reused. I want each user to have a unique email address. But I also want to be able to change that email address eventually. So I cannot use the email address as unique identifier. So the email address needs to be unique, but it's not my identifier because I want to be able to change. So if it was a traditional SQL relational database, I would have a table for the email. I would say, OK, this column here needs to be unique. But an event-sourced system, we don't have such a table. What we have is a journal with all the sequence of events that I can use to reconstruct my model here. So I cannot just go to this journal and scan all the payloads of all the events that I have to find out if someone is already using this email address. And as I was saying, the KELIX entity lives in its own transaction. So I cannot send a comment to an entity and say, OK, create this entity. But before you create it, make sure that all the other entities are not using this email address. I don't have the possibility to scan my whole system. So I need to have other means to do that. So the way to do that is one of the possible solutions for that is to create a barrier. So here we have this user. It will be implemented as an event-sourced entity. And we have another entity that I will call it a unique email entity. And the idea of this entity will be the email address. And because it's a value entity, a key value entity in KELIX, it's guaranteed that there is only one for a given email address and there is only one in the system. So it's kind of a barrier for me. So as I did on the choreographer saga webinar, what we'll do here is when I need to create a user, I will first create an email address. If it's already in use, then I cannot create that user. I have to change the commands of my request. But if it's not in use, I will create this unique email address for the first time and I will reserve for that user that I want to create. So I kind of create this record, this key value record, the unique email address, and then I move forward and I will create the entity. But as we said, those entities, they live in their own transaction. It can happen that I create the unique email address, but then I fail to create the user. When that happens, I need to have some compensating actions to free the email address because now the email address is reserved for this user that I was not able to create. So if I fail to create the user, I need to go back here and unreserve this email address. And if I create the user, I can go back here and say, okay, it's created, it's conformant, and it's in use by this other entity. And instead of using a choreography here where in the previous implementation, I was listening to events and to state change and coordinating the change by using subscriptions, here we'll create a workflow that will be our center point that we use to coordinate the whole. So visually it will be something like that. We have the workflow here. I will not have an application controller anymore. I will just hit the workflow directly from the outside and then I have my two entities. And when I initiate the workflow, the first transition for this workflow will be move to the reserve email step. So Kellex will execute move the workflow to that first step that is that we'll call the reserve email on the unique email entity, creating the entity if it doesn't exist yet and reserving the email or failing if it is already in use. So let's say that here's the sunny day scenario. I reserve the email, it returns, and then the workflow will say, okay, now that we've reserved the email address, let's create the user. Let's move to the next step, which will be, let's transition to the next step, which will be, which is create user. So now we do a call to the user, we create the entity and it returns and say, okay, good, I have, I did both calls and now I can confirm the email. I can go back to the unique email entity and confirm that it, that this entity is, this email address is effectively in use by the user that I just created. And when this call returns, I can finalize my workflow. So I have, I first transition to the first step, first step completes, I transition to the second step, it completes, I transition to my third step and then when the third one finish, I just transition to the end, to the end of my workflow. Of course that's the sunny day scenario, but things, errors happen. It can be that I have a validation error here or the user entity services down for some reason. So I reserve the email address and when I'm about to create the user, it fails. So in the demo that I do, that we're going to see today, I will put a retry strategy, a max retry set to three. So what will happen is first I create the user, it will fail and then I will simulate some errors, it will fail and then it will try again and then again with some interval between the retries and then it will retry for the third time. So we're going to see in the logs four tries, the first one and then the first attempt and then the three retries and then it will finally fail and the workflow will have another step, which is a failover step. After trying three times, we'll say, okay, that's enough. I don't want to try again because there are probably major issues here. So let's failover to yet another step here and then we unreserve email and then we unreserve the email as usual and then we can finish the workflow. So here the difference is that I have this workflow place where my workflow has a point for me where everything goes back to the workflow. The workflow can then make a decision. So the interactions between the parts are coordinated by the workflow and as the name says, it's an orchestration saga. So it's like the workflow here, it's acting like the conductor of an orchestra. It's the one that's managing driving all the process. Let's go to the code. So here the model is exactly the same as in the choreograph saga webinar. I did one small change here on the entity. I will show this right now. I put some random failure method here that I call. When I created the user, I will do some random failure because I want to simulate failures and I want you to see the workflow recovering from them. So what the condition here is, if the user ID that I'm choosing here is a negative number, I will randomly make it fail. So if it returns true, I will fail with this message. But if I send a positive ID or a string, because ID is a string, no, it's a string in Calyx. The unique ID from an entity is always a string. But if it's some text, if it contains some letters, it just succeeds. If it's a positive integer, it succeeds. But if it's a negative integer, I will eventually let it fail. So let's see the workflow. The first thing here that I want to show is that the workflow has a state. The user creation workflow has an ID that is exactly the same as the user ID, just for simplicity. It has a state, define it here. And the first time that you start it, it will send a post request to this method with the command to create the user. Then if the workflow was never created, if the state is null, or if it's in the positive state, and we're going to see that more in detail later, I will save the state. I will create the state, say, okay, I'm on that state, phase of my process, reserving an email. And let me show here what is the state. The state of a workflow is the user ID, the initial creation command, some status that can be reserving email, creating user, confirming email, finished, posted, or failed. And so that's the status, and eventually it may have an error message that I can inspect the state and see what is going on. So when I start it here, I will create, I will save the state, the state of my workflow, and then I will tell the Kellex engine here to say, and now it's time to transition to this phase, the reserve email phase step in which, and that's the input for that first step. I'm reserving the email here. That's the command on the unique email entity to reserve it. And then I reply to the caller, I send back the state of my workflow. And if the workflow is already created and it's not posted, I don't want to interfere indeed, I will just return the states as it is. I will not force any transition here. The workflow may be finished, the workflow may be running. If I send this request twice, I will just send back the current state of my workflow. Now, the definition of workflow is a method that you have to implement. Here's where you define all the steps of your workflow. And I will go to the end of this method where I want to show you this part. First, here you need to return the workflow definition. And the workflow definition, you build it by calling this API here and adding the steps. Those are the steps of our workflow. And we can imagine that... So the order of the steps here, you can add them the way you want. They don't need to be executed in that order. That doesn't matter. Because the goal here is that you need to declare the steps. And each step may transition to any other step. You can go back and forth between two steps a few times and then decide to finish your workflow. So the order is not important. But it's important that all the steps are defined in your workflow definition. So we have a reserve. The first step that we will execute here is the reserve email address. First is the battery. So at first I want to know if the email is available. If it succeeds, I will transition to the create user step. And this step is a special one because here I'm saying if it keeps failing, if it fails, I want to retry it three times, at most three times. And if after three retries it keeps failing, I want to unreserve the email. So here's the failover transition. Here I will go to this other step which actually unreserved revert the reserve email step. It basically unreserved the email. Now if the create user succeeds, then the transition, and we'll see in a while, the transition from create user is the confirmation of the email. So let's see first the reserve email, which is the first step here. It may be important to show here whenever you declare a transition, you have two possibilities. You can say transition to this step and there is no input, there is no new data that you need to pass to that step, or you have to pass the command or the input for the step. So here in the transition to I'm saying reserve email, I messed up something here, revert. I am reserving the email here, I'm transition to the reserve email and the input is this command here, this type. So let's go to the reserve email, this one. I have the name defined here because I want to use in the logs as well, so I just put on a variable and this call will receive as input the reserve email class. Then KX will take that and say, okay, I need to transition to that. I have already the input that you defined in the beginning of this file. Now when you say transition, KX will take that and it will execute that for you. And this is an async call. So it means that the lambda that you are passing here, this command is this type. We can see here, command is the reserve email class. So now KX in the background will execute this lambda passing the payload that you defined it and here we will use the component client to hit the value entity email, the unique email entity and reserve this email address. We will execute it immediately. So it's an async call. It's expecting a completion stage and if it succeeds, we will map it to result success. If it fails, we'll map it to result failure. We'll see now what we're going to do with those two here, but let's first just to recap what happens here on the reserve methods. On the reserve methods, here I have this value entity that works as a barrier for me to protect the email address and when I reserve it, so the email address has an address, has a status, can be not used, reserved or confirmed and it may have an owner. The initial state of this value entity is the address. It's not in use. It doesn't have an owner yet. When you reserve it with this command, you see it's already in use and it's already reserved for another owner. So if that's the case, I will emit this email is already reserved. So game over here. If you hit this method twice to reserve for the same owner, then it's already reserved for this owner. So I have nothing to do. I just say, okay, it's good. It's reserved for you. I don't have anything to do. Now, if it's not yet reserved, I will update the state and say, okay, this email address is now reserved for this owner here. Back to the workflow. So we do the call. If it's successful, we map to this one. Otherwise, we map to a failure. What I want to do is to capture the failure message because I want to put it in the state of my workflow. So what I do next is, so Kalex will do this call for you in the background. When it gets the results, that will be either result success or result failure. It will then call the end method. So for each step, that's how it works. You define a call that needs to be executed by Kalex by the runtime. Then we have the results. We come back to your step and say, okay, we got the results. What should I do next? And in this end-end method, you define what is the next thing to do. So the result comes back from that call. We pass it to you here. Now you can check. And what I'm doing here, okay, oh, it's a failure. So I do some logging. And I will return here an effect, workflow effect, in which I update my state saying, oh, it's paused. And here's the error message. During the demo, I explain why it's paused here. So the status is paused. So there was an error. They may have already resolved. That's why it failed. So I've just paused my workflow. I will not leave it, let it go to completion. I will put it in opposite state. It won't try again to reserve. It will just stop and wait for extra input. And then instead of transition to something else, I will just tell Klex, I want it to pause. So don't mixed up. Here, this pause is just the status inside my state because I want to be able to visualize it while here. Pause is us telling Klex, now it's time to pause this because I will bring more data eventually. Now, if you successfully reserve the email address, we move forward and we say, okay, now I will move to creating user phase and you return here the fact, no? So first update the states, then transition to create user step. Let's see the create user. So let's say that we transition to this step now. What will happen is, okay, I have... Let me go back here. When I transition to create user, I tell Klex what is the input of this next step, no? And the input here, it's something that I store it in my state, which is, let's have a look, is the initial creation command. I just put it on the state because later I want to use it. Now, and that's the moment that I want to find back this original command, the creation command. And now I move to the create user step and that's my input. Let's have a look on the implementation of create user step. I make a call. This time is not an async call. It's a call in Klex terms. It's a deferred call. I will use the client here and I will return a deferred call. That's what this call will return to me. No, that's not what I want to show. This return and deferred call that Klex will run in the background first. So what it does, well, it will talk to the event source entity identified by this ID, and we call the create user method on this entity. And here we come to this piece of code that I showed already. I showed already where we have this random failure, eventually dependent on the condition. We will force a failure here. If there is no name, it will also failure. Otherwise I will create the user. That's exactly the same that I did on the choreography one. Back here. Good. This call returns a don. So the output of this call is the input of the end-end. Once this returns, Klex come back to you, say, OK, I execute your step. Now what do you want me to do? And then you tell Klex, OK, that's the type. Here I don't know if you know this trick. I don't want to use, I don't need to use this type don. Here in my lambda, so like I have this trick, if you put two on the score, it's valid Java code. And it's just say, OK, on the score, I don't use it. It's just like a way to say, I'm not using this type here. Whatever. Here, update state. So I create the user. Now I update state. What is the next thing that I do? I will confirm the email address. And I transition to confirm. Here the difference is the transition, I'm not passing a type, an input type, because confirmation email does not have an input. So I can just say, now transition to confirm email. Let's see what happens here. OK, confirm email, yet another call. But this time I don't have an input type. It's just a Java supplier here. And inside it, I will call the value entity, the email that we reserved for it. And I will confirm. Let's have a look on the confirm method here that will be called in this step. If it's reserved, I will confirm it. And I'm done. If the status of my email address is something else, I don't care. I just don't want this method to fail because of that. And then I just return done. Here as confirmed. So I update the state of my value entity, taking the current state, mutate it to as confirmed, which confirms the status of this email address to confirm. OK, I confirm the email. Then Kalex, come back here. Again, here confirm, returns done. That's the output of my call. And then, therefore, that's the input of the end-end methods. Once again, I'm not using it here, so I may ignore and use the trick of using two underscores. But what I want to do here, it's OK. I will update my state. I say, I'm done. I'm finished. I reserve the email. I create the user. And I confirm that this email address is in use by that user. And then I can finish my workflow. So whenever I return here the fact, I will just show I can transition to something without an input. I can transition to a step with some input. We have seen both. I can transition to paused state. The workflow goes to a pause to a halt and wait for someone to provide more input or I can go to finished state. When I transition to end, what's the case here? It means that this workflow will not be as it could anymore. We reached the end of the workflow. OK. One more part is the... So we saw the reserve email. We saw the create user. We saw the user confirmation. We haven't seen yet the unreserved email. So the create user, if it fails, it will try a few times, three times, and then if it keeps failing, it will transition or fail over to this other step. Let's see what it is. So we got in this situation. We need now we were unable to create our user. We have an email that was reserved in the system. We failed to create the user. Now I want to free that email address that it can be eventually be used by someone else. So here again, the step defines a call and the call takes gold talk again to the value entity here, the unique email entity, and it calls the unreserved method. Let's have a look what it does. If this entity, if the unique email entity is in a reserved state and I call this method, I would just go back to the empty state, just reset its state. And otherwise, I would just ignore it. I just say, no, it's good. I'm not reserved, so you cannot unreserve me. And the empty state is back to not use it with that honor. If I'm on that state, I can reserve this email address for someone else. That's the goal here. Back here. No, one more. The call will unreserve the email address and when it returns again, it returns done. Again, I'm not using this type here, but I do update my state and I say, okay, I failed to create the user. So the status of this workflow is failed, game over, I'll not try again because I move this workflow to the finished state. I end the workflow. So here it's everything that I need to have to implement this use case, one single class with the steps and the coordination between them. The workflow here is managing and driving the whole process. Now, of course, we want to see it running. Okay, I hope it's there. So first, let me check like I did before. This email address is not in use. I will create user 001. It will also be the unique ID for this workflow. And here's the payload and it will reserve this email address. Let's see. Okay, that's the status of my workflow here. User ID this one. That's my initial command. It started in reserve in email address. That's the first step. There is no error message. But in the meantime, this workflow already completed. Okay, let's see what is happening here. I hit the workflow, start workflow, reserve email. It's here. And then it hits the unique email entity, reserve in the email address, back to the workflow step. Okay, reserve email. That's reserved. Workflow move to create user step. It creates the user. It hits the entity here, creating the user back to the workflow. Confirm the email address. Back to the unique email entity. Email is reserved. If I check the status of my workflow now, it is finished and my email address is reserved for this user and the user is also created. I have one here that will force a failure. I have a user without a name. And when I call it, let's see here the email address is not in use. I will do that. The email is now reserved. But if I check the workflow, it's creating the user. And I can see here that Klex is emitting some errors. There is an error here because it failed to create the user. The name is not filled, so we know that it will fail. Here again, here again, third retry. And now the workflow will fall back to the failover step, which is unreserved email address. If I go back here and check the status of this invalid user, invalid acme.com email address, it's not in use anymore. I have one more here, which is just for the fun. So as I said, when you create the user, this is the one that we just saw. I didn't put the user name, the user name, the full name for that user is empty, so it fails. But now we're going to see this one, the random failure. And as I was explaining, if the entity ID is a negative integer, I will randomly make it fail. Just for the fun. So here I will generate a request with a random number. The ID here, it's a random number from minus 100 to minus 1. That's the name. Then the email, because I will call it fill time, so I want to reserve each time a different email. I'm using a random UID here in the email name. And then we can later check here the workflow states. Let's see the first one. Okay, this one passed. This one passed. It's 68. First one passed. It's finished. Okay, let's try one more. Okay, this is failing. Okay, fail once. Okay, fail twice. Fail three times. Let's see it in fail once more. Okay, it passed because it's random. It failed three times. If you have one with four times, we're going to see it's which number was that? 18. We can check. It's finished. One more. Let's see one more here. Failing. I already failed it three times. Let's see if it will fail once more. Oh, it didn't. I want to see one unreserved email. Let's see this one. Oh, this one passed. Once more. Let's see if we got some chance. Ah, it passed. I want one more. Fail it, fail it. Third time. Oh, it passed. Anyway, so of course, if it fails once more, we are on the third case here. It will fail. After three times, it will give up. We could see it with the when I did this one. I will change the ID here. This one will keep failing because I don't have the name. I was hoping to have the random one to fail once. At least. But it didn't work out. Anyway, that's just for the fun. So back to the slides. So what are the main characteristics of workflow or orchestration saga? You can build it completely based on the workflow. You don't need to have subscriptions. You don't need to wait for the events to be propagated to your subscriptions to go back to your choreography. You have one single place where you can better control the flow of the calls that you want to execute. Once triggered, like choreography, it moves like a wave until completion. If not posed. I forgot to show this one. Let me go back here. I reserved this email address for John. That's why I had the pose one. I almost forgot to show this one. Now I will create another user and try to use the same email address. Let's see what happens. It says that it's reserving the email address but we know that this email address is already picked. So if I go here, it's saying oh, it failed to reserve the email address because it's already picked but it's posed. Why I put it in pose? Because if I complete this workflow and I'm using the same user ID I cannot restart it again. For that I need to have a workflow ID that is different than the user ID. But for my demo, I want to show this so I decide let's use the same user ID, make it simple, and then I can show the pose. What we'll do here, this workflow is not finished. It's not trying to reserve this email address anymore. It's just posed. If I go wrong, I cannot continue. So now I can come here and change the email address and I will send a request to the same workflow and as you see when I hit this method it will have here another payload. The command, I'm modifying the input here. So let's see. Now I'm reserving for undo and if I check here it's finished. So that's one of the case where you can use a pose. You can decide that something went wrong, you pose and you wait for someone to come in and change some data, provide more input so your workflow can move on. Just to show here what happened when we hit the start button, the start end points. If I don't have a state for this workflow it means that I really started for the first time but in that last case here it was in a pose state. So I accepted the request anyway and I updated the state and I transitioned back to reserve email so I kind of restart my workflow. That's one possible use of pose in workflows. So as I was saying, it triggers. Once triggered it moves like a wave until completion if not posed and that makes me remember that I wanted to show the pose transition. Once you can pose it and require request more input, receive more input from either some subscription for some other system or from a user. It's possible to inspect your workflow because it has states and that allows you to track where you are and that's the biggest difference compared to choreography where you don't have a clear way to track where is your workflow where is your choreography saga. In our orchestration saga because you have the central point the conductor of your orchestra in our case here the Killix workflow you have this point where you can go check the state and see how this is moving or if it's stuck because of some error if you need to provide some inputs if you need to unlock it or not or you can analyze the flow. I have this exercise here assuming that you have watched the other webinar about choreography and now we can talk about the compare both here. So what are the difference? In choreography we depend on events and state chains and subscriptions. We have our entities emitting events or emitting value entities propagating their state chains and we have subscriptions to react to them. On the other side we have a fixed place where we can coordinate everything. We have this conductor of our orchestra that is the workflow. Choreography moves like a wave until completion. I can say the same for the workflows but we can pause. Choreography it's important to watch out for head of line errors if you didn't watch the other my previous webinar I recommend you to watch it because head of line errors is an important thing you have to learn when you're dealing with subscriptions be aware of that and as I was saying in the other webinar it's hard to debug because you don't have a place to track the progress. Now back to workflows we can track the progress we can rely on events and subscriptions if we want we can pause the workflow and have an action subscribing to some events that hits back your workflow and put your workflow back into movement that's completely possible and as I was saying you can pause and wait for extra inputs you can also cancel your workflow it's running you can say every single point you can hit it with some endpoint and say whatever you are from now on you move to this other state and you can recover in case of failures you have this possibility to say okay I will try this thing I will try this step three times otherwise I will fail over to this other step where I have the chance to do some cleanup the source code for this workflow demo it's also on GitHub you can scan this code and download the code and play around with that and you can also join your Slack channel if you have questions and want to reach out otherwise this other QR code brings you to get started with Kaliq's page in our website that's all for this webinar I hope you enjoyed thank you very much


So, hi, my name is Dominic and today I want to show you how Cockpit works as a modular system. Right, Fedora is trying to modularize and that means different things, obviously. But I'm going to show you how Cockpit works with dealing with a bunch of different technologies and techs and APIs and how we survive as a project even though we're relatively small. So, our protagonist is Cockpit, I work on the Cockpit team for Red Hat and I'm only using as a case study to show you how we think projects like that can work. Mostly because I'm familiar with it, obviously, but it's always good to have a solid application when discussing abstract concepts. So instead of just talking about how it should be in general, I want to show you how we do it and then we can get into a dialogue of how that can apply to other different things. So as an intermission, I'm just going to start with a little demo right away, I think that makes sense. But we have, this is what you see when you log into a system, right, you collect your system with a web browser, you can just type in the username and this is the username from the system. You might have seen the log on the face already and as you can see you can log in. So what our goal is, is to make Linux and REL discoverable to a broader ad than audience and should make tasks in Linux more discoverable. And we can't look at all the features today, obviously, but we can see what the system can do. You can join in the main, you have system time, power options, you can look at services that are running, you can look at Docker containers, start Docker, it's not started, system logs, you can do storage, set of raids, logical volumes, networking, just a bunch of things. And all these little pieces are most of what you see are actually modules inside Pocket. So we have Shell, Pocket itself, and there are modules that perform a specific task. And each of those modules interface with different parts of the system. So we have modules working with modules basically. So that's why I think it's good to take away a few lessons from this. So back to the presentation, because in Pocket is a remote interface, a remote interactive admin interface. So what that means is, it runs in a browser, you can connect your system for anywhere, it's zero footprint, and you use it to configure a system. You don't configure Pocket, right? So if you log into your system via Pocket, and that's the system you see, Pocket doesn't have its own configuration. It doesn't have an extra state. That's just a system. Yeah, use a system user, and if you set up your system and remove the Pocket, it will work the same way it did before, although you can't log into the Pocket anymore. But it doesn't, I mean, some of you may know previous attempts to configure everything and control the whole stack. That's not what Pocket does. There are really great tools out there that configure a system. Some are usable via the CLI. Some have a diva interface or REST APIs. And what Pocket does is bring those things together. So what we do is we call system APIs directly from your browser. And I'm stating this again, just to drive the point home, you see it says clearly in the presentation tier. So you have really the lower level, the current level, you have LBMC groups. So you can say where the rubber meets the rubber, right? That's the name there. That's the nose level. Then you have the logic stuff that the sign was going to have. You have the system, the network manager, Kubernetes, all those useful tools. And then you have the presentation tier, Pocket, CLI, and other presentations you have. And those should work with each other and augment each other. And if possible, no logic should be inside Pocket. Because then it's not accessible by other means. And that means you're limited in how you work. And when you work, you want to achieve a specific goal. You don't want to work for a tool, but just use it to achieve something else. And to explain this a bit further, a bit more technical, we have the web interface on the client side. So that's a browser running somewhere on the operating system of your choice. We have the Pocket web service, that's PocketWS, that's network stuff, that interfaces with a browser. And that communicates with the Cockpit bridge on the system. And that bridge runs as the user that you log in as. So there's no demon that runs in the background, no other service. When you log in, that bridge spawns, it communicates, transits all the stuff. And when you log out, that goes away again. The web service can run in a container. And it doesn't even have to run around the systems. You can use one system as a gateway and connect to other machines. But that's more detail. And then the bridge forwards all the communication to the system APIs. And the bridge doesn't have to be aware of all those APIs. So that logic is actually on the client side. And the main means of communication, obviously, is SSH. You can't use SSH from the browser. It would be nice, but it would have bring its own problems. So what we have is a web socket. That's what we use. Our supporting cast, stuff we work with. This is by no means a complete list. And I'll spare you reading all of that. It's just, for example, Docker, atomic, all the CLI tools like sudo, all kinds of things that we interface with to make the web interface actually usable for the user and bring the stuff together. Some of the stuff we've contributed to, to make them to bring them together. Some of the other teams have helped shape their APIs so we can consume them. But the important thing is that we don't try to take their logic. We trust the other tool to do their job and bring them together to do what's necessary. So it's just a crazy amount of APIs to keep track of. So that's one of the big points here. And if we map this out, this is just a family tree so you don't have to read the text. If we would map out the dependencies, it would probably look something like this. We've never tried mapping it out because that would just be insane. So now I want to show you a specific example of how we interface via divas. Because divas is very important for us. It's asynchronous. And you can actually open a diva session in the browser. This is a short video. For example, we have the host name here. And if we open a diva session in the browser, we can actually connect. We can use JavaScript to connect to divas. So that object will have completion. For example, you can see, OK, I want to see the trailer release. You can use this to debunk. Actually, it's very nice. It makes divas accessible. Then we can say, OK, let's see set host name. We do that. And that interfaces with the divas process on the server. And you see right away, it changed. And if we go to the terminal and do the change in a different way, you can see it took effect. If we use a CLI to change the host name again, then the browser will receive the notification via divas. And it will change right away. So it's really interchangeable. Because in the end, when you want to do something on a system, you don't really want to care about how it's done as long as it's done properly. And what you really don't want is the second state that you have to keep synchronized. So the system is the state. And also, from personal experience, I can say that debugging or developing with divas is a lot more comfortable in this setting and trying it out on the CLI. So to give you a better picture of what happened in the background there is we have the web interface, the JavaScript, where we instantiated our little helper library that's part of the cockpit API that interface with the web socket, the web service. And that sends those commands to call a divas API with certain parameters and receive an answer. You can do that with a payment, for example. You can receive text back. You can say you set the host name. And those are translated from the bridge as the user you're logged in as to the system APIs. And this API of cockpit is stable, so to spawn processes, interface with divas, open files. It explicitly does not contain anything that's API specific, so it's a very generic and lightweight API. So what drives cockpit? We have, it's obviously based on the open source principles, on the agile principles. What that means, I'll go into a bit further later. We don't pull things in, we don't own the other things. We use them as they fit. We try to keep our work distributed, try to keep our work open, and deliver early, deliver often. And the most important thing is, with all the things that you saw, the APIs, we have to test a lot. So we can have lots of ideas and lots of good ideas, but if it doesn't work, then it's not really much used to us. So we have to test to make sure that everything works. So let's bring it all together. We have all the APIs with Docker API, we have different system APIs, user account management, disk management rates, network interfaces. And luckily for us, upstream APIs never change. They always stay stable. They're always durly tested to capture every single bug, so we never have to worry about that. All the distributions are the same. So we can just implement it once and we're happy, right? So what could possibly go wrong? Well, and trying out part of the globe in our faces, since we have a common tutorial explosion, if we say we have all these system APIs running on different operating systems, maybe even different versions, then you can't test it manually. It's just way too much. So it's safe to say that the project lives and dies by its testing. Testing is not always as fun as creating a new feature, but going back and having to fix features all the time isn't fun either, so testing is actually preferable. So due to the way it's set up, that we consume so many APIs, the testing is very vital to us as a project. And rather than paying the price for having this huge abstraction layer and trying to encompass everything, we think of every possible expansion of the future. What we do is we do a lot of testing, just to stay on top of things. And so right now we have about 10,000 testing instances a day. That means actual virtual machines with Fedora 24, 23, 22, Fedora testing being spun up, running a new pull request and being spun down again. And you can see the results in GitHub. So each pull request we have upstream gets really run hundreds of thousands of times before it's actually merged. But the key is that it is tested before it's merged because you couldn't do it manually. So it's very important that this is done upstream. Obviously we need robots for that because it's just way too much to do manually. So continuous integration is the key word here. If we want to release earlier release often and we have to make sure that the quality matches that. Open source has typically and historically take the hit exactly there. Say you have a certain cut for futures and then you start testing. And then it turns out you actually find bugs. You have to fix them. It takes longer to fix than the pushback release dates. So these cycles keep getting longer and longer. So in order to make this work you have to test before merging. You have to test continuously. This is what the tests look like typically on a pull request on GitHub for us. We test on real operating systems. We have Fedora, 24, Atomic. We have different browser tests with Chrome and Firefox. We have CentOS 7, RAL 7, RAL Atomic. This is not just unit testing. Unit tests are masqueraded as the Travis tests there. There are a couple of thousand unit tests that run also. But the integration tests are really the thing that catch the most issues that users will have because it opens an actual browser and interacts with cockpit. So here usually a pull request gets merged with all the tests to pass. So if a test fails, either the test is broken, which does happen, especially race conditions are our favorite, or the code broke something that wasn't supposed to break. And with all these operating systems it's really difficult to keep track of all the side effects of coding. For example, our master branch will also test against Fedora testing so we can see stuff that covers. We can see errors coming before they actually hit us in production. From an infrastructure point of view our virtual machines, they easily vert. You can run our test infrastructure from a developer machine. You can run it from a Docker container that you can deploy in some machine that we have. Or you can run it via Jenkins. It doesn't really matter. And this is reflected in our testing architecture. So we have GitHub which uses as the... we call it a single point of failure which does fail occasionally because that's where our source code lives, upstream. So that's where we have the status and the pull requests. We have fedorapeople.org where we store images, our virtual machines. Not every developer should have to create their own testing images of Fedora 23. And we have to make sure that we use the same testing images. We store logs, follow our tests on the servers.  You can run a test of your machine. You can run all the tests. You can run a single test if it's your choice. And we have verified machines that are publicly visible. They have...they check GitHub. They see if there's no pull requests to test those. And they can run multiple tests at the same time. Maybe they can scale. And we maybe...we have testing machines behind the firewall. It doesn't matter. As long as they push the results into the visible space, it doesn't matter where the tests are run. So this is a big important aspect. It doesn't matter to us where the tests are run as long as everyone can see the results. If you can't see test results, it didn't happen. So what we have there is we have... we look at testing tasks that have to be run like create a new testing image, for example. We try to create a new image every week for Dora, for example, to get new packages and see if the API has changed. We see the testing status on GitHub. We can pull images, push images. And we have, sadly, known issues like test load that we know that fail currently because of issues upstream from us. But, yeah, we keep track of those as well. So I want to show you a quick demo of how that testing works in practice. It wouldn't be productive day if I didn't open a pull request. Holding a talk is no excuse. Yes? We will see some of the best industries. Okay, yeah, I can... I think it's... It's not necessary right now about... The idea is that for us, we don't use a testing framework, right? So we have... I'll get more to that later, but it's like peek ahead. We have tests that use avocado. We have tests that use selenium. We have tests that use phantom.js and Python. We have tests that compile binaries. It doesn't really matter. The tests are really modular.  So I can... We can go into more detail on specific tests that you think are useful, but in general it shouldn't matter which language or which framework the tests are in. So... Yeah, so I looked at the readme for testing and noticed that there's some parts in there that aren't really necessary anymore. And... So we can commit my changes. We don't have a certain script anymore. So I committed this to my fork. Then we have a cockpit project. A local request. It's really big here. You should see our test first. Start to take this up. Travis picked this up right away. And... The other test runners, they periodically scan Github. And they should pick this up. So I can also do this locally. Since, like I said, it doesn't matter where tests run, right? For Github scan the D is just for dry run. So it doesn't change anything. And that should scan Github for new tasks. And that's exactly what the test runners, we call them, actually do. There's no separate... There are no separate scripts for infrastructure. So what you do as a developer is also the test machines do. And this shows we're making this a bit smaller. Yeah, so you can see that different... Let's go. Yeah. The different pull requests, this should be mine, I think. Should be tested for different operating systems and specific image versions. And also in a couple of days the images will be refreshed. For example, we'll have a new Fidora testing image in 2.5 days. That's like a 7 day task. This is not picking up for some reason. Our lab with the test runners is moving today. In the end, once that is updated it should look like the other pull requests. Like this one. And you'll see a list of operating systems. The tests that passed, tests that failed. And you can look at the results. For example, you have... This container is actually a container test. It creates a new Docker container. You can see that Fidora tests failed. You can look at the details if it opens. The important part is that the test results are publicly visible. Right? You can work with those. It doesn't matter where they run as long as you can see the results. And the connection is not really running. Luckily I have one open. This is an example of a test where a rebase failed. And the test runner says that it couldn't automatically rebase your pull request onto the master branch so please fix this. We can't merge it as it is. And in other cases you'll see a list of tests and the ones that failed and the ones that succeeded. But apparently not right now. So I'll just go back to the presentation. So the quick review, we have the different parts of the testing system. We have Github that we use. That choose what to run and they are very distributed. We have the test artifacts. We have the testing images. We have the containers. We have packages. We have a log sync. So places where you push logs. It's just a Python script. You drop in there and connect the SSH. And we have the different test suites. Those should actually be plural. Different sets of tests. Single tests or individual tests. So like I said, distributed is really the key here. You don't want to bind yourself to single testing framework. Because frameworks come and go, especially in the JavaScript world, things change very rapidly. Other languages. And sometimes you have different parts of a software that require different types of tests. Or maybe even have existing tests. That's the reason to convert working tests into a different framework just to have a different framework. That's usually wasted time. And by making the results public, making accessible, you really reduce the infrastructure load. Steward also scales a lot better.  So if you need more test runners, your Jenkins, provision them in the Ansible, it doesn't really matter. And just use them when you need to. So one could argue that one of the reasons that open source is successful is that it's distributed. So that's what makes it work. A lot of people collaborating to produce something greater than what they can do on their own. And tests are part of the software. So we can't hoard our integration testing. It's not an advantage you have. It's something you need to share. If you hoard it, you lose part of the advantage you have with it. It's the same mindset as you should hoard source code. Because it just doesn't benefit you in the long run. So in order to benefit from the tests, you need to be upstream and open source. So one could say, to sum it up, the framework is not king. You shouldn't write tests to a certain framework. That shouldn't be the goal. One project isn't better than another just because you use a different framework. They're tested better if they have better tests. If the coverage is better. The framework should be secondary. Instead, we can say the framework is king. This is based on the agile principle you'll find out in there. So we have feedback from the tests from users. That's the whole reason why we want to release early and often is to get that feedback. If we have long cycles, then it becomes more difficult as a developer to react, to changes, to wishes, to change your mission, so what you want is to really reduce those feedback loops down to a minimum. As an example of that, so if we work for Red Hat, we had the Red Hat QE testing, so we might be familiar with that, so what you do is you develop your software and it goes into testing and you get back a bunch of bugs and they say, please fix these. And maybe upstream, you've already gone three, four versions past that, so you have to go back, you have to do a lot of work. So what we did was we moved all those tests upstream. So for each pull request that we have, those tests are actually run. That's a lot easier for QE and a lot easier for us. The same with packaging. If you're going to package things differently than what you test on, something will break. Everything you don't test is going to break eventually. So what we do is we actually pack it in our tests, we have the test VMs, we package our software, we install them as those packages and test them as they would be if we had released that software. It also makes automating releases easier. She tests it all the time. And of course QE still runs tests and they have some very specific ones they also run, but the core of the tests are run all the time upstream. So one could argue that tests are like microservices. We have different tests for different software. When you're testing, you always test something that you change, you get something that didn't change. And when you have a test, you want to reuse that test. For example, cockpit tests could be used to test a new version of cockpit against a known version of Fedora. On the other hand, if you're working on changing the Docker API, for example, you might want to test, you could just use our tests with a fixed version of cockpit and run a new build of Fedora and see if the cockpit stuff breaks. You wouldn't have to write new tests for that, you could just reuse them. You don't have to care how they run, just run them and see if they break. And then at least you're aware of something breaking. So what our goal is is to make testing work more like the way open source already does. It's not a different thing. You want to have the open source, you want to apply the open source principles to testing. So to use a famous Lego blocks example, it doesn't have to be an unordered pile. If you know which parts to pick out, you can save yourselves a lot of work. There's no need to always start from scratch. So that's what I mean with the open source principle. You reuse things that make sense to reuse and only write new things when you have to. So like I said, testables from upstream can be invariance, downstream. It really, the terminology upstream and downstream becomes a bit muddy if you think about the feedback loops. What is upstream, downstream, sidestream, mainstream, I don't know. It's just, the key is to reuse work. So the same way you would reuse a library, you just reuse a test. And to reiterate what I said earlier, the real magic of continuous integration really happens before you merge. So that's when you want to catch the bugs before something gets merged. Then it's still easy to fix. It's still scoped, and then you can deal with them. The other aspect is continuous delivery. If you automate all the tests and do the packaging anyway, then it's not that big of a leap to say, to adhere to the principle of releasing early and often. And that also makes it easier to listen to your customers. It's one of the core ideas in open source. And one of the key factors in the success of Linux. So it's good to keep that in mind. If a user contributes something and you say, okay, this is merged, this will be pushed to Fedora next week, then it's a lot more fun for them than to say, please check out our next release in six months, and you'll see the results of what you did. So that actually helps to drive things forward. The agile manifesto, if you look at the first three principles in that manifesto, they're actually about delivering rapidly. There's a good reason for that. It helps to keep that loop and making sure that you stay close to what you're trying to do and think about what you did. So you want to reduce those cycles and do whatever it takes to make that happen. In Cockpit's case, we release weekly. So every end month is not nearly enough to do that. So delivering, of course depends on the project that you're working on. It could be a testable run of a package. It could be a complete release. It's what you decide it to be, right? And to make sure that the community grows, the source community, you need the feedback loop that's going to be short. So our conclusion is open source is distributed. I think it's difficult to argue against that. Distributive scales, we've noticed that with container and everything. And open source development is distributed. People all over the world contributing to projects make them work. Open source communication is also distributed. Different means of communication, different modes, different languages, different places. And integration testing should also be distributed. Since that's part of the development, part of the project. And action and delivery should also be distributed. So to sum it up, we have the code that's upstream. Discussion happens upstream where it's visible and transparent. Reviews are upstream. Integration testing is upstream. Packaging is upstream. Delivery is upstream. Open source is upstream. You might see a pattern here. I'd be lying if I said this was easy. And of course, it's a goal to work towards. And it needs constant care and attention. And with everything in open source, you don't have to do it on your own. If you don't hoard what you did, but share it, then you can share and maintain it together. It's not as much of a burden. So I'm at the end of my presentation and I'll open to questions. There was one question earlier about the actual testing scripts. Yes. I think I'll show you on Vitha if it's accessible. It's not right now. So no internet connection. I can show you source code. Is this big enough to read? So what we have is something that's not maximized here. So in our testing framework, we have different test suites. We have our core testing suite. It's called verify. We have these Python tests that work with phantom.js. So what we do is we boot up test machines. We simulate a browser that interacts with a JavaScript and clicks on things and checks to see what are the variables. So you fill out text fields, see if the variable is correct, wait for text, that kind of thing. We have known issues with known failures. For example, for door 24 there's a known issue with Stalin GitHub for mounting fails due to Nestling failure, for example. So we retract those. That's one more complex thing. We have a free IPA test that pulls up my free IPA instance and lets the machine join the domain and test that. So it brings up multiple virtual machines of infrastructure. We have open shift tests that bring up open shift instances. We have avocado tests that use avocado framework or Selenium browser tests, for example. We test that a login works with Firefox and Chrome with different versions. Those kind of things. And we have unit tests that compile binaries or just run shell scripts. Does this answer your question? So in the background we have a set of common scripts. So with the core of what this does we have scripts that run a virtual machine. It uses Livert and makes sure that the machine is up. In the background we make sure that the testing network is up and running. We have tools to download virtual machines. So right now we have the checksums for images in the GitHub re-repo. So each checkout has a specific image it tests against. So if you create a new image then you change that checksum and you make sure that everyone can share it. And tasks, specific scripts in the case of GitHub but the tests are pretty separate right now and I think we're on the verge of spilling those out from Cockpit. So if there is a project that says they can use our testing framework then we'd be happy to help make that happen. Recently I did some work to make our tests work on Jenkins. So Jenkins provisions a machine and that runs with tests. I can also if I run the test locally so if I'm here for example I can say I want to run the check accounts test for door 24 that will spawn the machine in the background run the test and they give me the result. So if I'm developing I can just do a single test. If I want to do them all I would usually push to GitHub so our infrastructure can do the testing. Should be done in a little bit. So what it does is it really spawns the machine starts Cockpit interfaces with that, really clicks on login and all that. Are there any other questions comments, tests passed, always good? Okay, thank you for your attention. And if you have questions don't hesitate to contact us on the RSE or visit our website and if you have something that you think should be in Cockpit then don't hesitate to talk to us. You can try it, we have some example modules, example plugins that you can work with and the hurdle is really not that high to get something running. And we have a lot of the scripts in place to work with the system so you can basically just start out with whatever you want. So the individual modules they don't have to be in the same JavaScript framework they don't even have to say have the same build system. So all they have to be is installed to the same target directory on the system. In this case for example you have different modules that are installed one part that deals with Docker one that's the basic shell the system so depending on your system you might have different modules here you can write a custom one and that will then be present when you log in. I see what modules are do you look for ADCs or also the others like we interface with SE Troubleshoot so everything that SE Troubleshoot can reward and you can also if the version is new enough then you can also fix issues so I think apparently I have no alerts right now but for example you can set the enforce policy to off a very controversial feature but sometimes it's very helpful and if you had an error then here you would be able to see the details you could see the logs and if there are potential fixes you can apply them and terminal to log in this is the real session for the system so you can you can create a ceiling denial in the terminal and be able to see any of the others we can do that afterwards if you want I don't know often how to cause one so I'll only cause them when I'm not intending to cause one that's a fairly recent feature the SE Troubleshoot team did a lot to work with us on that to make that happen but also works via debus mostly via debus API and like I said the modules they don't have to be JavaScript is a fast-moving thing so we have parts that are written in Angular parts that are in React some that use jQuery and each module can choose what they use if they use ES6 or not if they use webpack or just make it doesn't really matter as long as it interfaces with a stable cockpit API everything works yeah so thank you again