Roedd y gofyn yn ystod rhywun â'r hoffi yw'r hyfforddiol. Roedd yn bod ni'n gweithio'r hoffi'r hyfforddiol yn cael y dyfodol. Beth ddigonno'n cymdweithio, mae'n 5cio yma ddiogel thamol. Rwy'n meddwl hon i'n gweinio'n gofyn i fynd i gael. Llyfr oedd rydyn ni'n meddwl am Jim Stewart, fel ond yma'n meddwl, a'r ysgrifennidau efo'r rydyn ni wedi cyfnodiol o'r Si asko Gwyrraedd UK. ac ydych chi'n gweld, mae'r ysgrifennu yw'r ystafell yw'r gweld. Felly yma yw'r gweithio gyda'r Ysgrifennu Cyngorol Cymru, a'r cyfleoedd yw'r cyflog Cymru, cyfleoedd cyflog Cyngorol Cyngorol Cyngorol. Yn ei ddim yn gallu cyfleoedd cyfrifennu cyfrifennu, ac mae'r ddweud yn dweud yn cael digwyddol i'r gael gyffredig ymlaen i gael cyflog cyfrifennu cyfrifennu cyfrifennu cyfrifennu cyfrifennu. Diolch ar y gyrdih radish, pawtwch anodau r yeux a gablieng axeuniaid, sy'nативio anodau cyfacebyd, sy'n ddim sy'n rhywbeth gynсл pen porpassigol yn Paeddu cyfosibol i gwahleisaded mewn digid cherbyn, i chi'n ymddangos cyfosibol, y bydd bangos athwyllio yw buddi творogi yngног interpretr yn y cyfosibol spearau i gyddiwrs mae gwlededd yn f Dit Person gemul gy cé BOY I absolutely love the diversity of the work that I get to do and the different types of teams I get to work with, and I'm going to talk a little bit about that today. We work primarily with public sector and large international institutions, but we also work with large private sector organisations that operate at scale and have a strong sense of public purpose. One of the things we do is we publish these short booklets with articles from people in our network, and slightly embarrassingly I brought the French edition with me. So I have about ten copies of the French edition of the latest one of these. If anybody reads French and would like a book, please come and see me because I really don't want to carry them home. But when public digital talks about digital, most people start out assuming that what we're talking about is particularly technology, and technology is a really important part of what we're talking about and the ways that we use it and the ways that it's developing. But for us and in our work we take a much broader view, so we tend to rely on this definition, but when we're talking about digital we're talking about cultures, we're talking about processes, operating models and technologies that have emerged through what we refer to as the internet era. So kind of the last 30, 40 years of the development of many societies. Most importantly we're talking about how do we harness those things in order to respond to people's raised expectations of the services that they have, the organisations they interact with and so on. So all of our work is really about bringing these themes together, the culture, processes, operating models and technologies. And that's going to be a big part of the theme of what I'm talking about and how that comes together in the way we think about teams.  You might have been at Agile India. I think it was running about three years ago this time. Personally I was at home with what I think was COVID, but this was before the UK provided easy access to testing, so I've never had it confirmed that that was COVID, but I was off the back of quite a lot of travel, had a lot of what we now know were symptoms of it, and had just had to cancel a trip to Chicago in favour of hiding in the attic of my house and keeping away from my family, so it wasn't a great time. But I was pretty lucky compared to a lot of people around the world. One of the many things that we saw early in the pandemic was a significant spike in the number of people who lost their jobs, who needed some sort of unemployment assistance. This is a picture from the Washington Post of the first half of 2020 showing the massive increase in the number of people in the USA applying for different types of unemployment assistance. At the time I was working with the state of California, we'd helped set up their new digital services team who were doing a similar thing to what we'd done in the UK government, bringing together a lot of the information and services in a single website, and really orienting that around user needs rather than structure of government, but was watching very carefully as the California unemployment assistance system in common with many around the US and around the world collapsed under the pressure of all the people applying for services. It just wasn't designed for scale. It couldn't handle the sudden spike in people trying to access it. To their credit, the government of the state of California commissioned a report into what had happened, why they were struggling to scale this, and published the results of that. That's quite a bold move to not only admit internally that something's going wrong, but to go public and say yes, the design of this service and this system is not what it should be. We need to explore that, and we need to learn from it, and we're going to do it transparently. They commissioned Yolanda Richardson, who was running one of their big internal agencies that coordinates the state government, and Jen Pulcher, who's a technologist founded an organization called Code for America, and had worked in the US federal government to explore what had happened. The report they published is really interesting reading. It's not that long. It is reasonably technical about parts of their system, but tells quite a story of how a service like this unemployment assistance one evolves over time, and how people lose track of what the challenging parts of it are. They found, as they were looking, that the service could process about 2,400 claims a day, but through that spike they were hitting about 20,000. In terms of transactions through an application, through a database, those are tiny numbers, but huge amounts of what was going on there was pushed into manual processing. Identity verification, avoiding fraud, those sorts of things were all handled manually, and I can't remember the exact number, but it took something like 18 months to get one of those manual workers up to the point where they were really productive in the system. So scaling that part of it was incredibly hard. In order to make changes, you could apply some technology. You could shift parts of how the system worked, but you had to change your thinking about risk in order to do that. You had to change some of the underlying policies about how they made decisions within the system. It wasn't simply a case of scaling technology that would have solved their challenges. And as I was watching that, I was in my mind contrasting it with what was going on in the UK close to home for me. And this is a very similar kind of spike in demand for a service called Universal Credit, which is the UK's relatively new social security benefit programme. Universal Credit went through a very similar spike, but had none of those problems with processing claims. In the UK, as anybody who, I doubt many of you paid that much attention to UK domestic news headlines, the debate was all about how much money should people get when they apply for this, how long should they have to wait before the first payment, and those kinds of social policy conversations and important political discussions. Not, can we process people's claims? Can we get them on board with this service? That was a really important distinction, and not one that we could take for granted. Because in 2013, the initial attempt to roll out this particular service was in a very bad way. Over £400 million have been spent. You've been through five. The SROs are senior responsible owners. They're the most senior people responsible for a big programme of work in the UK government. Nobody had successfully used it, and so it was marked as black by one of the agencies that normally rates government projects in the UK from green, amber, red. Black means unimaginably bad. We need to stop this now. And so we had actually an opportunity and an opportunity that we were able to use to lead to that position where seven years later the service scaled pretty effortlessly. I say that. My colleagues who were working on it would not be happy with me saying effortlessly. They worked long hours. It's just from the outside, it all looked magically smooth. And one of the big contrasts was that in the first effort to build universal credit, they'd started with the simplest possible case. A single person, usually a man living alone, no children, and designed around that and then worried about how do we scale the database transactions, how do we get enough hardware in place to run this, how do we do all of that. Which failed as soon as it hit the reality that almost nobody's life is as simple as that. And that just because somebody applies as a single person doesn't mean that they'll be single by the time they're receiving the benefit. Just because they don't have children doesn't mean they'll never have children. The lives of the people using this are complex. I need to lean into that. My organisation in the central government helped set up a new team that was able to take a new approach to this. When we did that, we said we're going to start by leaning into that reality of human lives. Rather than taking the simplest case, we still need to start small somehow because you never want to start big on something this complicated. But we're going to pick a postal code area, an area with a few thousand people living in it, which means probably about 100 people who are going to be eligible for this service. And we're going to understand their lives and we're going to start them on a pilot version of this. And before we start encoding rules and technology, before we start building too much software, although we know we're going to have to eventually, because the only way to scale this is going to be to automate large parts of it, we're going to bring together a team who can not only understand what we might do with technology, who can not only design great user interfaces and run operations and do QA and all of that, we're going to bring together a team that includes people who normally provide face-to-face services to this population, that really know them and their lives, and we're going to bring together the policy people who have the ability to write and set some of the rules by which these decisions are made. And when somebody makes a claim through this service, when they register and we want to understand what they're eligible for, rather than have that software impose rules early, we're going to have a phone ring, and the phone is going to ring for somebody who can write the policy. And they're going to have to encounter the reality of the person who needs that claim and interpret the rules in real time for them with them on the phone. That's immediately going to build empathy. It's going to make sure that we've understood that, and it's going to put that understanding in the hands not just of people who can design software, but people who can work out what are the parameters within which we can shape this policy so that we can make all of the other parts of this experience as good as we possibly can. And we started by going end-to-end with the service as well.  A big part of universal credit is about trying to help people get back into work so you get a work coach, we allow for you to tell us over time about your change of circumstances, there's some evidence verification and some calculation. We went end-to-end, we put the whole of that together, not automated, but with a complete flow. And doing those things, understanding people's lives, building empathy, going end-to-end meant that we were very quickly able to understand what actually is going to be involved in running this service. So when it came to dealing with an unforeseen global crisis that led to masses of new traffic and demand for this service, the teams responsible who were working very closely together were used to working together, were used to changing regularly. They really understood where the likely bottlenecks were in the service and I've seen the text messages that members of that team were sending between each other going, oh right, so if we can't see people face-to-face, these are some of the changes we're going to have to make practically, operationally and these are some of the policy challenges we're going to have with that and they just understood that and they had that at their fingertips. And then with that they knew the levers that could be pulled. They knew where we got flexibility in the policy we've got, where do we need to go to the politicians to get some serious changes made, where does our technology scale seamlessly, where are we going to need new resources, where are we going to need to re-architect parts of our system, where are we going to have to take steps out of the journey short-term in order to make this work through this peak. And they understood the humans affected, they understood what proportion of their users relied on face-to-face channels, how many used the phone, how many were online and which ones could move and which ones were going to be challenged so that they could really think about the impact of any other decisions that they made. By having all of those things together and crucially by having this team that was really multidisciplinary, that understood the face-to-face, that understood the policy, that understood all of the issues in forward detection and was completely comfortable changing the technology, they were able to be prepared for the completely unforeseen. For me that's part of a wider trend that I'm seeing more and more with leading teams who are able to go beyond a lot of what we talked about when we talked about working multidisciplinary, where we sort of talk about DevOps, we really still mean the technology in that, we talk about DevSecOps, we talk about bringing design into that, they're thinking much more broadly and designing multi-channel services that meet people where they are. When the California team published that report that I mentioned, my colleague Anna here who had worked as a product manager on the Universal Credit Service wrote a blog post where she drew out seven reflections that she had from her experience working on one large-scale benefit system and applying them to what she was reading about California. Seven points here, you can all read the blog post and we'll share the slides afterwards so you can get the link. But there were three that really jumped out at me as part of the theme of what I want to explore today. The first one, real change isn't just about technology. I think we all kind of know this and we talk about it from time to time but it's in work like this where that really becomes real. But for some of the users of some of these services, online services aren't always simpler or quicker to use for everyone and this has really come home to me in my work over the last few years. I've moved from the UK where definitely not everybody can use online services but most people can to working much more globally. We do a lot of work in Africa, in Latin America, in various parts of Asia where the levels of internet penetration, the access to mobile, all of those things are quite different and you just can't make the same assumptions that you might in California. And that security must be embedded and not an afterthought. That was actually the theme of one of my talks last time. I was at Agile India and I'm not going to dwell on it too much but it's just one example of one of those sorts of disciplines that often we talk of a game as left to the side when we're building our teams and these be brought in right at the heart. And I sort of interpreted those things in a couple of contexts. Someone is just thinking about the way that growth of the internet has happened particularly since kind of the mid 90s we've seen a massive increase in use of the internet. People are much, much more familiar with using online services generally despite what I just said about not everybody can access them and it's just sort of saturated our society. And along the way, people like those in this room have seen the emergence or growth and helped with it and stewarded it and developed it of things like open source, web standards, mobile, cloud, DevOps, continuous delivery, countless different programming languages, styles and frameworks and of course all of the different parts of the Agile toolbox that we can draw on. What that means is that we often kind of make lives a bit more complicated for ourselves than we should and we're sort of pursuing the latest and greatest technology tools. Generally, if we want to make something with tech the barrier to entry for that and the ability to adapt and scale have both changed and reduced enormously. We're just much better at that than we used to be. We've learned an awful lot about it. We've learned lots of resources that we can draw on and once we do that and add some of the Agile and Lean practices that we're talking about this week that allow us to move faster and or find the right direction better we suddenly have all of this new capacity all this new ability to do things that we couldn't do before and we have choices about how we use that and one of the choices is that we develop previously unimaginable services and ways of running businesses and innovate in fantastic ways and that's really exciting and really important and we need to keep pushing things forward but when I think about what I sometimes call Agile dividend the things that we now have because of all of that development that we've done over the past few decades I'm also really conscious of some other opportunities that we have I don't know how many of you are familiar with a thinker called Clay Shirky he was quite prominent a few years ago in the talk about the development of communication technologies and what it was doing to society and he'd been very prominent at the early bit of Web 2.0 and he said that once technology gets boring the social effects get interesting what he kind of meant by boring was perhaps not changing quite so rapidly getting a bit more familiar and being everywhere so he was particularly talking about he studied Japan a lot he was studying the US a lot places where mobile adoption took off very quickly and was there and looking at the different ways that people worked and interacted as a result of it and that's often in the back of my mind when I think about what can we do with this new capacity we've got these new capabilities we can think socially we can bring more people to the table we can move away from thinking about purely online services and make a huge progress with that to really thinking about how can we harness all of this that we've got to shift how do we provide services for people where they are when they need them, how they need them much more efficiently or effectively because of the technology that we have on display and the ways of working as teams that we have so how do we move from making exciting fast moving but ultimately relatively exclusive services as technologists to bringing all of these people who have very different experiences of human reality together informed by technology able to harness technology but designing services for humans and it's easier said than done and I was talking to somebody last night about my experiences in the UK government and was quite rightly asked so did anything go wrong and a lot went wrong working at massive scale I was there for six years but a team that we established is still there worked on hundreds of services we took £4 billion out of the UK government's technology spending in three years and we transformed about 20 services in the first couple of years and you don't do that stuff without a lot of things going wrong most of them quite small but some of them quite big and we worked on one which was about agricultural subsidies and how do we support farmers where it went so wrong that there were parliamentary hearings my then boss was summoned in front of parliament to talk about why has this new service which is giving billions of pounds out to farmers to support agriculture not actually worked why has it been a lot of pain why are we very worried about whether farmers are going to get their money on time there are lots of reasons for that particular failure I'm not going to go into all of them here because it was a parliamentary hearing you can go and find the public records of that if you want but one of the crucial things on reflection for me with that is that we had come in too strongly as the people who knew what we could do with new technology who could build exciting new user interfaces and who went and talked to farmers about what they needed but hadn't taken the kind of operational and policy and political realities of what was going on sufficiently on board and I'm really pleased that more recently with our consultancy we're actually back working with that particular bit of the UK government working very differently bringing different teams together who can take that stuff seriously because we didn't take those things seriously and we didn't build the right team back then we weren't able to deliver the new service and that was a big lesson learned quite painfully so now I'm pretty fixated on how do we bring these people together how do we bring people who really understand those different aspects of reality into one team to work together that's easier said than done organisations generally are structured in ways that hold us back from the sort of collaboration that we need large silos, power bases, communication structures that don't really allow for people to say ok, here's a problem that we've got to solve who needs to be involved how do we bring them together how do we set up the right leadership structure and incentives I'm fixing that's an art, not a science there's no single solution to it and everything is about the context and it's about the people but I just wanted to share a few approaches that I've found really important in starting to crack that the first one is just being curious it's very easy when you're brought into a new situation you have to learn really fast to lean back on the things that you know well I've been brought in here because I have this expertise my job is to use that expertise to do the thing I'm good at that's a really natural response when you are under pressure in a new situation but if you lose curiosity about the circumstances you're in about why has this organisation got to the point that it's got to what's important for it what makes the people here tick then you're likely to hit a lot of friction and challenges and a missed part of the puzzle one of the organisations that we worked with in the UK government is the driving and vehicle licensing agency they're beautiful building in Swansea and South Wales they process driving licences and so on when we went to work there we found that the team we put together was really effective and partly because they had this instinct to go and find the people who ran the phone call centre and talk to them about what went on and understand what works and what doesn't where are people phoning up for issues that should never have occurred in the first place we used that as a way to understand what made the organisation tick what were the pressures on their users what was the case for change and who did they need to bring together to affect difference or this is Matt Edgar who's brilliant he is now working in the UK's national health service but he was part of that team working on the universal credit service and you can see the look on his face looking at this incredibly dry policy specification document you can tell how difficult that piece of work was going to be when you just saw the piles of documentation about the policy design specification that's never a good sign when you go into something but Matt somebody who gets enthusiastic about understanding what's underlying something like this and is really curious about what's at the core of this thing how can I understand it and how can I build connections with people applying that curiosity and that kind of determinism to just get under the skin of things has allowed him to be incredibly effective in again bringing people together from across disciplines so being curious is the starting point next one it's slightly harder to draw on examples for but giving something up this is really a kind of political manoeuvre but again often when you're in a large organisation there's a big pressure on you to prove your worth to prove that you are the person or your team and when we say your team in this case it's kind of the people who do what you do are the ones who can really make a difference but if you want to affect significant change often you need to step back from that so actually how can I be the person who builds a coalition how can I be the person who makes other people successful and that requires a real level of confidence sometimes it can be very challenging personally and politically but if you're able to start doing that then you can have a much deeper impact which is a bigger win for everybody overall like I said it's hard to draw on specific examples for this because so much of it is in subtle behind the scenes human communication and people just stepping back at the right moment or allowing themselves to kind of lose a battle because they know that it's in the greater good but one thing that came to mind as I was thinking about this particular lesson is some work that this wasn't me this was some of my colleagues did at a UK conglomerate called the Cooperative Group working on their funeral care business most of my examples have been public sector this is actually a for profit business they're cooperatively owned they took the leadership through a process of setting out a set of outcomes that they wanted from a transformation they were undertaking and they came up with these six almost everything focused on this first one giving people time back to spend with clients because in the funeral care business you're dealing with people at one of the most vulnerable points in their lives and the last thing you want is for your trained professionals who are good at that kind of care they're spending all of their time working out where the coffin is, where the car is and all of those sorts of logistics you want them to be helping the families that need help and that meant stepping back from thinking about how can we produce something that will look really good as a sort of this is how good our digital strategy is we've got this amazing new online service this great new app or whatever into how do we make those people who are dealing with families better able to do their jobs and how do we make them the heroes of this with that I just talked about outcomes and I'm not going to spend a lot of time on developing outcomes they're being really clear on the outcome of what you're trying to do is absolutely essential but it can be really common to develop a set of outcomes that are about the new thing that we want to create rather than the change that we want to see that brings everybody together and a few months ago I was working with an organisation which is another one that gives money in this case for research and development grants and they had undertaken a programme of work to replace the underlying technology platform that sort of processes claims allows people to evaluate is this a project that's worth investing in and then tracks it and they were having real trouble bringing the different parts of their organisation together around that to the degree that they needed and as I'm sure most of you are very familiar with there's a certain amount of kind of modelling the existing organisation that you can do when you're providing some new technology for something like this but generally you also need some change in the organisation around how do we think about things what do we standardise, where do we still allow divergence and so on and one of the things that we did with them was help them start to develop a service map this picture isn't actually their service map it's another one that I might come back to but to sort of say the service that we're offering together is something that we offer together the technology is just part of that but actually the other people who are deciding what sorts of funding do we give out how are we making decisions about that for dealing with the academics and the businesses that we provide for funding are also part of the service mapping that together rather than purely having a map of the technology platform underneath it allowed them to start talking about what are the points that we want to make really efficient what are the points where a human interaction will add more value started to bring them together to design that together but we found that there was another problem which was that they'd set out a great set of metrics when they wrote the business case for doing this piece of work but because one of those metrics was replacing the legacy technology platform that's all everybody focused on it was the one that the non-technical people were scared of and the technical people were heads down trying to do and it was the one that everybody could solely place the emphasis on this particular group of people for and nobody else needed to take responsibility for it so we pulled out some of those metrics from their business case and just worked with them on let's surface more of them let's rebalance that and start talking about the more aspirational things that we're trying to do about reduction in administration time particularly externally make it easier for people to apply for this funding because then they can focus on the R&D that they're meant to be doing not on how they fill in our forms and that then creates capacity to think more broadly about is this working how might we redirect our funding to be more effective than what we're doing we're bringing those metrics out and really working a lot with the leadership of those organisations to say you all need to stop just talking about one of your metrics the binary one of have we switched this thing off have we had to renew the contract or not start talking about these you can start to create an environment that brings your people together and then with that you needed an experimental kind of mindset to how you did parts of it and I've started using language of test and learn in these kinds of situations more than I use terms like agile and I've learned a lot of that from my colleague Lara who's very big on this working with large organisations where too often the term agile well we all know that it can apply very broadly and we talk at events like this about business agility and sort of shifts in business strategy that we can have as we become more agile and a lot of organisations have become so focused on how do we do the IT and a set of other baggage that again we'd all reject but which is hard to get over but sometimes bringing things back to something like test and learn that's built some new bridges that helps people think a little bit differently take some risks that they might not one of the places that we work is Madagascar and this approach of testing and learning was applied when they were thinking about how do they make their tax system more efficient and able to sort of move them from thinking about as we think about making our tax system more efficient how do we put big new pieces of infrastructure in place that will help us understand all of the data in the tax system that will equip the front line offices that collect taxes because a lot of folks in Madagascar are a very long way from paying tax online so let's find some but to an attitude of let's find something that helps us test how do we change people's attitudes paying tax can we collect things a bit more quickly and can we start to understand wider change that we might do as a result the start small thing that we talk about quite a lot over again but in this case they found let's just try to figure out have we got enough data about the people who we need to collect tax from that we just start sending them reminders through SMS text messages and they were able to do that you have to do some background work to kind of release that data and put it in the right place which helps you start to understand the architecture that you've got and where you might need flexibility but just doing that they saw a massive increase in the collection of unpaid taxes taxes that people were happy to pay once they knew they needed to and were told how to and that starts to instill let's start something small let's see what we learn from it let's see how that directs what we do next and in the universal credit service that I kind of started out with this was deeply ingrained and they really brought that to bear on how they thought about scaling what they were doing so rather than having any kind of big targets about the numbers of users the amounts of money paid out and so on they kept bringing what do we need to do next what do we need to learn next so some of the examples of those sorts of questions are there and answering the next question almost always involved going to a next level of scale we're working with exponentially more local offices that support people we have this number more claimants we have this much more money going through so that we've got more useful data to understand the fraud risk it drove scale but it drove scale based on what they needed to learn and putting it like this actually made a lot of the political conversations much simpler than if they'd started by saying we have to do this organically or we have to do this in a controlled way which would make a lot of the politicians think oh you're scared of scaling this thing saying what's the next thing we need to learn that's a conversation everybody can gather around so at each point that service decision to scale was made based on understanding what the next thing was to learn how do we need to explore that which parts of our operation are involved and leaning on the multidisciplinary team to say ok but if we're learning this we might have this side effect so actually what's the bigger thing that we're learning if we're learning this does this leave this group of people behind so how do we bring them with us if it brought everyone together and then my final learning on this is about I struggle slightly with the phrasing of this but working all the levels and starting where you can I was really lucky in my work in the UK government and have been in a number of other engagements very very high level support for what we were doing so the work we did in the UK and what we do say in Madagascar is sort of led by the top political leadership in those countries where a lot of our corporate work is driven at a kind of CEO chairman kind of level and that gives you a set of levers to make change which are incredibly powerful they're quite hard to wield because they can be you try and make a change at the very top of a very large organisation and the unforeseen consequences of that can be large and the resistance in the system can be quite big so you have to have to work it carefully but it's really exciting when you've got say President McCrew of Argentina saying I've got this vision we're going to organise around it I'm going to make that possible but whether you've got that or not you need to bring people together at a variety of levels and often there's much more opportunity to start by bringing people together at a more junior working level using the human connections that you have using the relationships and the conversations that you have with people day to day or using a crisis this is very rapidly this sort of thing has become very cliched but being ready to find the moment when you can say now I need some people from a few different teams to get together because we really need to understand this problem together and perhaps together being ready can be the most powerful thing that you've got which means being constantly on the alert hopefully we won't have another crisis quite like that one probably we will at some point but every organisation goes through moments of pain and crisis and challenge and knowing that you want to get people together having a sense of who might be up for it thinking about what that might do as an impact on what you're doing means that you're in a position to seize those moments when they come and also working together with people to start to kind of sketch out that when we think about what we're doing as an organisation how do the parts of this fit together when I first saw this kind of mapping I actually got quite nervous because I spent quite a lot of my life pulling people away from very top down enterprise and architectural approaches to driving transformation in organisations I was like oh no we're doing that again we're just doing it with a slightly different lens but what I've come to realise is actually the sketching together of the different components of operating the service that we've got where we think we might want to get to remains an incredibly powerful exercise you need to obviously keep it fresh say that this is going to change over time call out your assumptions within it but making sure that any of our pictures really have the humans they have as our users and customers the humans as our colleagues who are providing other parts of the service in the mix brings people together and then also seeking out the people who are already trying to effect change in different parts of an organisation so this is my colleague Amanda she works on project helping large organisations think quite differently about how they manage their data and bringing very multidisciplinary teams together around what we call data as a service but it's a kind of data stewardship set of activities of really understanding what's the valuable data in our organisation who are the people who depend on it how do we want to provide it to them and what's the lifespan of it so that we can manage it in different ways treat it as products and services not just abstract stuff in a database what's made her work particularly effective is being able to find the people who are already kind of on that journey but needed collaboration or needed top cover or needed some other kinds of support that we could bring and harness and most organisations of any scale have those people so those are kind of five things that I've found work for me in bringing together deeply multidisciplinary teams and organisations I feel enormously fortunate to have kind of lived through the past couple of decades of learning all that we've learned around technology but one of those big lessons for me has been that simply doing that without a sense of how do we bring more people into the conversation how do we make sure that the things that we create are inclusive rather than exclusive how do we build good conversations around what's now possible that doesn't throw away the learning of the past is vitally important to make sure that we capitalise on all that we've got but also a crucial part of making change stick in organisations because it's only through building coalitions and bringing people together that we can really embed different mentality approach and opportunities into our organisations thank you I think we have some time for a couple of questions hi I have a question so one of the things that really stuck to me was when you said that real change does not always happen because of technology but it's more to do with people so one of the things that I wanted to ask was that you worked with leaders especially the top management especially in the public sector which is quite complex from my experience in India and other markets as well so my question was that what have been your approach and strategy in getting them onboarded with this new way of working looking at the right matrices etc so curious about that yeah getting people on board to get started is one of the biggest challenges I've seen a few things at work one is capturing moments of change whether that comes from in a political setting it can be that there's been an election there's been a change in political leadership or just somebody's been recently promoted and they want to make an impact and you can kind of you usually need some kind of existing relationship where you're able to approach them but you can then say opportunity if you want to make that impact relatively quickly we can help you with that and if you can bring together people who've got a vision of what's possible particularly what's possible quickly with people who really understand the existing system so that you don't make unrealistic commitments that can be really effective so when we started our work in the UK government it was just after an election there'd been a change of the political party the first time in 13 years so we had a new group of government ministers coming in in that particular case their goal was to save money for government and we could definitely say we were the most expensive government in the world to run from a technology point of view and so they want to change that but we importantly had those of us like myself who came from the outside with a technology background and a different approach to doing things with well established civil servants who were able to say the likely challenges are how we approve spending how we do this kind of governance you need to go and get to know these security people really quickly and really well and build confidence in them so that sort of the being able to give some confidence to the political leader who was at a moment where they wanted change combined with understanding of the system kind of worked there sometimes about the sort of grassroots communication strategy so we've also seen in organisations where they don't have that top level support yet you'll often find top leaders are under huge pressure and feeling really embattled and they want some good news because most of life as a very senior leader is dealing with all the things that are going wrong and teams who've done something differently that's been successful even on a small scale who are good at talking about it can have a really big impact so we spend a lot of time helping people just to start blogs and do things like that where they can talk about what they've learned talk about what they've done in a way that's very easy to share and sometimes that's publicly sometimes it's internally there's sort of organisational dynamics to consider there but the sort of being able to talk about small quick wins and up quite rapidly and then there's something about sort of building coalitions so I already talked about like you've got the internal people who know the system and the outside people who know it but in a political environment you're going to have think tanks and you're going to have influential non-profits and in the US a lot of the change that happens in digital services is there, is initially funded by non-profits and I think there's some of that going on here as well and I'm doing a session with tomorrow to be able to talk much more about the Indian context for that Thank you, thank you so much I'll just repeat, if anybody reads French I have some books sorry I didn't bring the English versions another question Maybe one last question Thank you Please elaborate on give something up with an example like what do you mean by giving something up Thank you Sure, so Thank you It's that most organisations kind of set up the way that we provide incentives to people the way we give people opportunities for promotion and so on to be about kind of accumulating responsibility, accumulating power being able to say very personally this was my win and that naturally divides people and so in order to get a bigger success in order to achieve something deeper and more sustainable we often need to work against that way that our kind of organisations incentive structures work so that can be and the simple examples are kind of the stepping back we did this together but this person, this person over here they were the most important person in it or sometimes it's the really letting for me I've often found it much easier to help other people to do that test and learn thing to internalise it myself it's quite easy to say you've got to propose a solution to this and I don't agree with it so let's turn that into an experiment because I'm confident that the experimental proof that I'm right and turning that round on myself can be quite difficult because it has to bring quite a lot of humility and admitting that I may also be wrong but applying that and being really open to the solution I came up with is probably just as flawed as what anybody else came up with there are lots of those kinds of of um dropping the assumption that I'm the person who's going to solve this or taking the glory on your own from it which can be really important and it requires a bit more of a longer term mindset because it generally is good for everybody involved because if you get a more successful service over time that everybody benefits from that um but you might not get quite the same quick kind of I did this um and the rewards that come with it that you might ordinarily be incentivised for right thanks a lot Jims


Hey everybody, my phone says it's two, my computer just said it was two, and the clock in the hall says it's two, so we've achieved distributive consensus, and I think it's time to start this. My name's Bo Ingram, I'm here today to talk to you on entitled Plains, Raft, and Pods. It's going to be a tour of distributed systems within Kubernetes. So a little bit about me, like any millennial, I've got to list my Twitter handle first. I am Biluptuous on Twitter, Bo Ingram was taken, so I had to get fancy. That's my dog Ernie on the right, he is universally recognized as quote, the man, and I considered taking out all my slides and just showing pictures of him, but I figured you all would leave and they'd throw me out, and it probably wouldn't be good. So I am a platform engineer at Craftsy. We're based in Denver, Colorado, and we're an online education and e-commerce company for makers and crafters. We sell classes and topics like quilting, sewing, cooking, photography, and we've got supplies, we've got patterns, we've got kits to help you make projects, and all sorts of fun stuff. So I spend most of my time working on back-end features there, but I also do some of the operations-y tasks sometimes, as well as some of your traditional site reliability-ish tasks. I'm probably somebody's definition of DevOps, not mine, but somebody's. So I'm big on definitions, so let's start off with what's the distributed system? So in distributed systems for fun and profit, Makito Takata says distributed programming is the art of solving the same problem that you can solve on a single computer using multiple computers. Now, that's a little bit of a sarcastic kind of tongue-in-cheek definition, but a distributed systems problem is a problem that you have on one computer that needs to go to more computers, usually because it's gotten too big. We can say that a distributed system is several nodes, components, pieces, whatever, working together to accomplish the same goal. So it's KubeCon, but what's Kubernetes? We heard a great talk this morning about what are all the things Kubernetes can be. Well, for me, I like to go look at the documentation. The docs say that it's an open-source platform for automating deployment, scaling, and operations of application containers around clusters of hosts, providing container-centric infrastructure. What? I read that four times, and every time I practiced this talk, I had to run through that slide twice just to get the cadence of it right. It's a mouthful, but I like to think kind of back to what the number one definition was this morning, which is that Kubernetes is a flexible platform for running containerized apps. Not word-for-word what it was, but close enough, but Kubernetes provides capabilities greater than, hi, I'm a container. Please run me. It's got this really awesome feature set, right, it can handle rolling deploys, do load balancing, you can add health checking, all sorts of stuff. And they're great, and I think a lot of talks talk about how Kubernetes does these things, but I want to do something a little different tonight. I want to peek behind the curtain. I want to see how the components of Kubernetes work together to achieve these things. My question for us tonight, or today, is how does Kubernetes leverage distributed systems? So to start off with, Kubernetes has containerized apps. Well, what's a container? Container could be said as the usage of Linux C groups and namespaces to provide process isolation for an abstracted file system. C groups specify limits for how much memory and CPU a process can use. And a namespace can stop process from interfering with other processes. But like all definitions, that's great, and somebody has said it better undoubtedly. Amy Chen yesterday, she said a container is just a baby computer inside another computer. Way simpler than anything I ever said. A containerized app, though, something we might run in Kubernetes, could be something like an Nginx server or a little microservice. Or if you're feeling stateful, perhaps a reticence instance. So when you think of Kubernetes, right, and you come to Kubernetes for the first time, you hear containers, containers, you know, I want to join the container team. But in Kubernetes, the base unit of scheduling and the way Kubernetes thinks about things is with pods. So a pod is one or more containers that share a unique IP address in the cluster. You're guaranteed to have all containers running on the same host. A pod can share disk volumes. This is pretty helpful in many situations. For example, if you get an app that logs to a file, you could have a sidecar container sitting alongside it in the same pod that handles forwarding your logs off to a log aggregation service, like LogStash or Splunk. So the way we typically manage these pods is through a deployment. A deployment is a declarative way of managing pods. We can specify what containers we want to run and how many instances of those pods we want to have in our cluster. Under the hood, deployments use a thing called replica sets to ensure that the requested number of pods is equal to the number of pods running in the cluster for that deployment. So when we create a deployment, which contains the container images we want to run for our pods and how many pods we want to have, Kubernetes is going to handle creating a replica set. This replica set will ensure we have a desired number of pods. The replica set will create our pods and then eventually our created pods will be scheduled by another component and then run by yet another component. So going through this a little fast right now, I'm going to start at kind of at the top level. We're going to dive deeper and then dive deeper. So we talked. Kubernetes is made of distributed components. When I first got into Kubernetes, what I imagined is I would go to my computer, right? And I'd be like, okay, yum install Kubernetes. And there'd be a binary. And then I'd try to run it and it wouldn't be on my path. I'd put it on my path. I would point it at other nodes, sit back and relax. My monitor would glow. Angels would sing. And I'd bask in the glow of magical container orchestration. Except that's not the way it works. Kubernetes uses distributed systems to provide the aforementioned magical container orchestration. It's made of several components and each of them is worth their own deep dive. Each of them does their own pretty neat stuff. And each plays a crucial role in running and operating applications. So let's take a look at some of them now. We'll talk in more detail about each of them in a moment. But I want to introduce them to you before we get too far into things. So the four components on the slide here, SCD, the API server, the controllers and the scheduler, make up the control plane. These run on the master nodes of the cluster. So starting off, SCD. SCD is a distributed key value store that serves as the cluster's database. Now having access to SCD, read and write access, is equivalent to having read access in your Kubernetes cluster. So to control this, we stick in front of it a component called the API server. Other components in Kubernetes talk to SCD through this API server via good old REST calls. Next up, the controllers. So the controllers handle routine cluster tasks. They operate on what is known as a reconciliation loop pattern. The controllers will check your desired state, check your current state, diff the two, and then modify your current state so that it matches the desired state. There's a variety of different controllers, and we're going to explore a couple later on. So last up is the scheduler. As you might surmise from something that has things scheduled on it, there's a scheduler. The scheduler is in charge of scheduling unscheduled pods. On the nodes, Kubelet is the star of the show. Kubelet watches for nodes that have been assigned to its pod, and then runs them. Kubelet is constantly checking at CD, by extension the API server, as well as its local config for pods to run. It's constantly going, hey, at CD in the API server, you have any pods for me to run for me? Any pods? Any pods? Great. Any pods? Kub proxy on the nodes handles any networking things needed. It's in charge of maintaining port forwarding and managing networking rules. And since we're running containers, we need some sort of container runtime, whether it's Docker, Rocket, Cryo, what have you. So these are the components of Kubernetes. At CD is the data store. The API server provides access to it. The controllers handle running routine cluster tasks, and the scheduler schedules. Meanwhile, Kubelet is listening for pods to run on each node. Kub proxy is maintaining network rules, and there's a container runtime running containers. So whenever you're talking about the distributed components of Kubernetes, you've got to start with at CD. At CD explains so much about Kubernetes behavior. So as I mentioned earlier, at CD is a distributed key value store that serves as the database for our cluster. If you're familiar with console or zookeeper, it's pretty similar to those as well. So distributed implies that there's multiple nodes running, all working together to store data. So in order to operate, majority of the nodes in our at CD cluster need to be up, running, and healthy. So we refer to this minimum number of healthy nodes as a quorum. So what happens if there isn't a quorum? What does at CD do? To look at that, we're going to look at everyone's favorite distributed systems concept, the cap theorem. Yay, cap theorem. All right, the cap theorem. So it's impossible for a web service to provide all three of consistency, availability, and partition tolerance. So we're going to define consistency as always reading the most recently written value for a given key. We'll define availability as every request to a non-failing node receives a response. Partition tolerance will be defined as a system, our system can handle the dropping of messages between two nodes. So in the schema provided by the cap theorem, at CD is a consistent partition tolerance system. So imagine a five node cluster. If three of our nodes take a dive, the other two are going to stop responding to requests until the quorum has been restored and we have more than the minimum number of healthy nodes. So why does Kubernetes use at CD? Well, if your goal is to have a distributed system for running containerized applications, you need something that plays nice with clustered systems. So at CD's docs say that once again, big quote, at CD is designed for large scale distributed systems that never tolerate split brain behavior and are willing to sacrifice availability to achieve it. Split brain behavior is when you have multiple masters. Each one thinks they're in charge and they're each setting their own state. So you don't have distributed consensus and you're disagreeing on what the state is. Not great for Kubernetes. So you can see the cap theorem references in this quote though. At CD is going to sacrifice availability and consistency. Now I think there's an important clarification to make here, looking further ahead, right? So at CD is running as the Kubernetes data store and drops below the minimum number of healthy nodes and is forced to sacrifice availability. That doesn't mean that all the applications running on your cluster, your microservices, your engine X service, servers, your reticenses, they're not all suddenly unavailable. What it means is that Kubernetes, the platform is unavailable. Your ship is dead in the water and you're in a great state you're in, but your ship's not sinking. You're still taking traffic, but you can't schedule new pods, add new deployments, or do anything that's going to require a consensus in at CD. Your existing applications will still be running and taking traffic. So at CD does the typical CRUD operations and a couple other things on top of that via REST calls or a command line tool. But the simple interface hides the answers to some complex problems. How do the nodes agree on what the value is? We've got three nodes out there. I alluded to distributed consensus earlier saying all the clocks said it was two o'clock. But how does that happen? If not done smartly, things can go awry pretty quickly. So how does at CD agree on the value for a given key while still upholding its consistency guarantee? It's the raft algorithm. That's a really great logo by the way, I like that. So the raft algorithm is a method of achieving distributed consensus. Having multiple distributed servers agree on what the value is for a given key in a fault tolerant manner. Now there's a really trivial way to solve this. You can just say, you know what? The answer is always three. Everyone agrees on it. It doesn't matter the time, doesn't matter the date, doesn't matter where you are. Most formal papers though for distributed systems say you can't do that. I really think that it started with some exasperated person somewhere arguing with somebody. But let's take a look at a not raft system to illustrate the difficulties in distributed consensus. The raft, it's not at CD, it's been specifically designed to show how distributed consensus can be difficult. Don't go away and implement this and run it in production. Bad idea. So in our example system, we're going to have three nodes, A, B, and C. Each one holds a single value. Our nodes will receive requests from a client, they will write to disk, and then broadcast the new value out to all other nodes. So let's examine some not great scenarios. What happens if there's multiple updates to the value at the same time? So here we go we've got our three nodes, A, B, and C. Everyone agrees the value is X. Everyone's in agreement, things are happy, things are great. And then, oh no! Two clients come in, client one tells node A, hey the new value is Y. Client two tells node B, hey the new value is going to be Z. So A and B each write the new value to disk. A writes Y to disk, B writes Z to disk. Meanwhile, C doesn't know anything yet, C is in for trouble. So A broadcasts the new value Y and B broadcasts the new value Z. So we're going to assume that A's messages arrive before B is just via some arbitrary ordering. You can see here we've got problems. A and C think the value is Z, whereas B thinks the value is Y. We have not achieved distributed consensus. Not great. So let's take a look at another scenario. In this new scenario, same baseline. A, B and C are all in agreement that the value is X. Things are going great. Then, oh no! The cluster undergoes a network partition. C is unable to talk to A or B and A or B can't talk to C. All the messages to and from C are being dropped. Meanwhile, a client comes along not aware that the cluster maybe isn't in great shape and says hey, hey guess what? The new value is Y. So A broadcasts this out and says hey everybody, the new value is Y. B gets it successfully and is like okay, got you. Meanwhile, the messages to C are being dropped. So C doesn't find out. Meanwhile, time goes by and C recovers. But C hasn't gotten the messages because the messages were lost forever. And so A and B think the value is Y, whereas C thinks the value is X. Sad. So seemingly simple systems can fail in all sorts of fun ways when exposed to concurrent operations. In order for consensus to be achieved, we're going to require greater coordination between our nodes. We can't just fire and forget like our terrible example system. This is where raft comes into play. So raft is a consensus algorithm for managing a replicated log. A replicated log is a series of commands executed in order by a state machine. So we want each log to have the same commands so that the end of the day our log has the same commands in the same order and each machine will have the same state. So to do this, raft elects a leader. The leader is put in charge of managing the log, accepts entries from clients, replicates them to the other nodes, and then tells the nodes when it's safe to actually commit them. If a leader fails, a new one is elected in its place. So my mom always said to me, Bo, you can be a leader or you can be a follower. Generally I was in trouble that often included the statement, if all your friends jumped off a bridge, would you as well? But in raft, you have other options. You don't have to be just a leader or a follower. You can be a candidate as well. So nodes can be in three states. Followers just hang out and kind of respond to requests from leaders. Leaders like, hey, replicate this, and followers are like, all right, gotcha. So leaders handle all the rights and handle all the fun coordination between the followers. In our third state, candidate, candidate is the state used to elect a new leader. When a leader goes away, or disappears or whatever, candidate is the state used to elect a new leader. A node will declare itself as a candidate and say, hey, vote for me. So we talk about leaders. Part of dealing with ordered time stamps is dealing with time. Raft uses an incrementing integer time stamp called a term, which is tied to an election. Terms last until there's a new leader and there's one leader per term. Terms serve as a logical clock and aid in detecting obsolete info so that if a leader finds out that there's another leader out there with a higher term, the leader with a lower term will stand down and be like, okay, not in charge anymore. So the leader sends heartbeat messages out to the follower nodes saying, hey, still alive. Happens every fairly frequently. Just like, hey, still alive. Hey, still alive. So what happens if a follower doesn't get a heartbeat? Well, it immediately assumes the leader's dead. And that's a dark but effective assumption. Our follower will increment its term and declare itself as a candidate. That means it's election time. Our new candidate is going to send an RPC request out for votes to all of the other servers. So we heard yesterday morning about how HBO uses Kubernetes with Game of Thrones. Well, the saying in Game of Thrones was, you win or you die. In the game of Raft leadership elections, the stakes are lower. You win or you lose. Servers vote for the first candidate or votes assuming the candidate is at least as up to date as the voter. This means if the voter has entries that the candidate doesn't, the voter isn't going to vote for that candidate. Now, if the candidate gets a majority of the votes, it wins. If no one gets a majority, there's another round of elections and Raft uses randomized timeouts to make sure that eventually somebody will win, assuming your cluster is in an okay state. So after the leader gets a majority of its votes, it sends out a new set of messages to notify others of its victory and life in the cluster goes on. So in Raft, how do writes work? Well, writes all go to the leader. The leader gets every right. The leader then appends the command to its log locally. It then tells other servers via RPC to append it to their logs as well. Now, if a follower is behind, it's going to say, nope, I'm behind. Can't handle this right now. And it's like, okay. If there's a cluster behind, spoiler alert, nothing's going to happen. So once a majority have appended things to their nodes locally, so the leader has it appended and a majority of the followers have it appended, the leader commits it to its own log locally. Now, in subsequent messages to nodes, the leader is going to inform the other nodes, the followers, of the index of the last committed entry. So nodes are going to notice this. I'm like, okay, you've sent me everything up to 12 and you say 12 is committed and I haven't committed 9, 10, 11 and 12? Well, I'm going to commit 9, 10 and 11 and 12. So it's going to commit everything up to the last committed index. Now, this solves problems in our bad system from before. Our simultaneous write scenario is solved by the requirement that all writes go to the leader. The leader is going to receive each write and each write is a new entry with a new index. So if two nodes manage to somehow try to update a value simultaneously, each one is going to be a new write index. One will occur before the other. So that way we don't have the situation where, you know, two nodes think it's one value and one node thinks it's the other value. In our network partition scenario, followers will reject requests if they're behind and will eventually be caught up. Also, requests go to the leader and our dead node can't become leader because it's behind. The effect is minimized and also if we really care and we don't ever want this value to be stale, it's a consensus read. It's slower, but it makes sure that the value isn't the majority of nodes before it's returned. So we talked about the cap theorem a little while ago and how Raft and NCD is a consistent partition-tolerant system. Well, this is achieved through acquiring a majority of the nodes to act. Elections require a majority of the nodes to agree to elect a leader. All writes require a majority of the nodes to replicate the transaction in order for the transaction to be successful. If we lose more than half the nodes, they weren't able to do anything. So that was a brief overview of Raft. I do semi-deeply into it, but I didn't cover all the safety guarantees and some of the more formal things. I think the Raft paper, the formal paper is actually a really great read. I'd highly recommend checking it out. Also, for the visual-minded of you and really for everybody because this is a really great visualization, there's a visualization out there called the Secret Lives of Data which shows the way Raft clusters interact in various scenarios. It's definitely worth checking out. So moving on, controllers. Controllers are a loop that watch cluster state and make changes to ensure that we keep the desired state. So we're going to take a look at two controllers right now. The replica set controller and the deployment controller. The replica set controller is in charge of making sure that for any given pod spec or any kind of pod image thing, there's a given number of pods running at any time. And these actually suggest you don't interact with replica sets directly, so we'll instead use the deployment and the deployment controller to manage them. Now the replica set controller knows which pods to manage through labels. So we'll define labels on the pods we create and tell the replica set which labels it should look for so that it knows which pods to manage. Well, what's a label? Labels are just key value pairs and the replica set has a selector, another sort of key value pair to manage. Now the deployment controller. The deployment controller manages the whole deployment process of your app. So the deployment controller provides a declarative way of managing pods and then uses that to roll out your desired changes in a friendly way. So deployments will handle like rolling deploys, you can roll back your application if it's not going well. So we talked about the reconciliation loop a little while ago. It's how the deployment process is managed. Reconciliation loop is going to check and then modify the current state so that it matches the desired state. So if we're rolling out, say, a new version of the deployment, the deployment controller's reconciliation loop will change the desired state, either counts, of the old and new replica sets. You know, the old ones will go down, the new ones will go up. This will get picked up and then the replica set controller is going to handle tearing down old versions and spinning up new ones. That was a semi-brief overview of deployment controllers, but we're going to see them in action in a little bit. So the scheduler. The scheduler watches for unscheduled pods and assigns them to a given node. What it's looking for specifically are pods without a node name. It's constantly querying the API server and, by extension, looking for these unscheduled pods. One thing to note is that the scheduler isn't actually in charge of running pods. It's only in charge of assigning pods to a given node. So it does this by a scheduling algorithm. First off, it filters out nodes that aren't desired or just aren't a great fit. It then ranks the remaining nodes and picks the top-ranked node after ranking is complete. So step one, we're filtering against predicates. We're filtering out nodes that aren't desired or not a great fit. I think the simplest scheduling strategy is to try to assign a pod directly to a node. So you can do this via a node name specified in the pod spec. The host name predicate is in charge of the node name to the node name specified in the pod spec, if any. So it's going to exclude every node that doesn't match. And it'll let us schedule directly to a node assuming it passes the other predicates. There's some resource aware predicates later that we'll check that could prevent it from scheduling. So match node selector is kind of a variant of host name. Kubernetes lets you provide label selectors to associate resources using labels. This predicate checks whether the node selector matches the node. For example, let's imagine that you only want to schedule on high bandwidth nodes. You want the network power. You could specify a high bandwidth label, put it on your node, and then add a selector to your pod spec to only schedule on nodes with this label. So there's also the concept of affinity, which is a more expressive and flexible version of the node selector concept. You're not just limited to an exact match. You can do set matching and all sorts of fun things. Also, you can match not only on nodes, but also on nodes with affinity. Accordingly, there's a predicate that checks whether or not the pod plays nice with the affinity settings of the pods on that node. The inner pod affinity matches predicate. So taints are kind of the inverse of affinity. We can specify a taint that tells pods to avoid certain nodes. Pods won't schedule to nodes with taints unless the pod explicitly tolerates them. So if you didn't want to schedule to a node, for example, you could add a predicate to it, unless they specifically add a node schedule toleration. The pod tolerates node taints predicate, checks whether or not the pod has opted in to tolerating a given node's taints. So the schedule, like I mentioned, also checks the resources of each node. The pod fits host port predicate checks whether or not the host port, a hard coded port specified in the pod spec, is available for that node. If it's not available, we're going to filter out that host from being scheduled to nodes for this given pod. So pod fits resources is another resource I'll wear check. Pods can request a given amount of CPU in memory. This predicate checks whether the node is capable of satisfying that request. So check node memory pressure and check node disk pressure or some other predicates. They won't schedule on to nodes whose memory usage or disk usage is too high. Check node condition is a more extreme version of this. Check node condition prevents pods from being scheduled to nodes who are unavailable or network unavailable, who aren't ready according to Kubernetes, or are just playing out of disk space. Those are conditions that would make a node suboptimal for scheduling to. I didn't list them here, but there's also some volume checks. So if you're on a cloud provider, there's often a limit to how many volumes you can have attached and make sure that we're not using more than the allowed number of volumes, or maybe that there's no conflicts on volume claims. So after we filtered out our nodes, we have a set of nodes that are valid for scheduling. Some are probably better than others. Now, you could also be left with no nodes, in which case your pod is unschedulable. In that scenario, you're going to need to go back and modify the pod spec or maybe wait for some other nodes to come up or take some sort of action possibly in order to make your pod schedule again. But if you've got a set of nodes that you can possibly schedule to, it's time to rank. Ranking applies a series of priority functions that return a score. A higher score is more desirable. So functions are running against each node. They're all added up and the node with the highest score is the winner. Ties are broken randomly by picking a winner. So here's some of the ranking functions. So when you think of scheduling, right, you want to put your pod on maybe the least used node. So least requested priority helps out with this. It calculates how much CPU and memory equally weighted and added together would be left after scheduling our pod to that node. So this kind of helps with balancing resource usage across the cluster. Now, balanced resource allocation is a more focused version of this. It's attempting to prevent nodes from being largely weighted towards CPU usage or memory usage. It's specifically trying to avoid nodes with like 95% CPU, but only 5% memory. So it checks to see how the pod would affect the balance of resource usage on that given node. This function is going to favor nodes who would have CPU utilization closer to memory utilization after scheduling. So next up is selector spread priority. We lose a great chunk of the benefits of having multiple copies of an application. They're all sitting out there on the same node. That node goes down. You'll lose every copy of your app that's running on the node. There's a metaphor about eggs and baskets I think that's really helpful here. This function minimizes the amount of pods from the same service or replica set on the same node. Causing are currently being scheduled pods to favor nodes without its managed siblings. So the image locality priority function. Well, containers have images, and oftentimes we have to download them. This function prioritizes nodes that have already downloaded more of the pod's images. If you've got them all, you're going to get the highest score. If you've got none of them, you're going to get the lowest score. So node affinity priority and taint toleration priority. Node affinity priority favors nodes that match the pod's nodes affinity preferences. Taint toleration priority prioritizes nodes that have taints that are specifically tolerated by the pod. If you've gone out of your way to request some affinity or explicitly tolerate something, Kubernetes is going to notice that and say, you know what? You're going to get a little boost to schedule into this node. So what happens when we submit a deployment to Kubernetes? We've talked about all these components, but let's see how they work together. So I've got this deployment here. I talked together a really quick Hello World Go app the other day. It's going to be listed on port 8080. I've labeled it with the tag app Hello World. And I say I want three of you. And I've also specified a service. A service will select pods and provide access to them. If you see the spec selector app field, I've given it the tag app HelloCubeCon. And so the service is going to select that pods with that tag. We've also said, hey, I want you to forward port 80 over to port 8080.  So how are we going to actually submit this? I'm on TeamCubeCuttle, by the way, for pronunciation. There's differing opinions. I believe CubeCuttle is the best one. So we use CubeCuttle to get our app running in Kubernetes. It's how we interact with the cluster. We can view resources as well as modify them. It gives us a great view of what's going on. You can check the status of our pods, our deployments. But really importantly for this section, use it to construct a timeline of events. So what do we expect to happen here? Well, we're going to create a deployment, you know, CubeCuttle create-f deployment.yaml. Our deployment is going to create a replica set. A replica set will create three pods. Our scheduler will schedule those three pods. And then kubelet, which is looking for pods to run on each node, is going to run the scheduled pods. So what actually happens?  They have been formatted to fit my screen. And viewer discretion is advised. So starting off with, we can see here the involved object kind. So our deployment is the involved object, the one that I just created. Assuming I just gone kubectl create-f. The message is that we've scaled up the replica set. Okay, well I guess that means we created a replica set as well. So we created a replica set for our deployment and set it at three. You can see also the source of it. The deployment controller has done this. It's noticed via the API server and etcd. Hey, a new deployment has been created. I should check that out and do things accordingly. The reconciliation loop here. What's the current state? What's the desired state? The current state is there's not a replica set for this. The desired state is there should be a replica set for this. Let's create it and then scale it up. All right, now we can see the source component, the replica set controller is kicked in. It's created a pod. We can see the involved object at the top as well as the replica set. Our replica set has created a pod for us. We've got two more to create. Before we do that, the scheduler is kicked in. We can see the source component is the default scheduler. The involved object is a pod. The default scheduler is doing something to the pod. It has successfully assigned this pod in WC7K, looking at the end of it, to a given node in GKE. That's pretty great. Then hey, look, the replica set controller is back. It's created a pod. 03HFH. You can see the involved object is the replica set. It's created a pod for us. Is it going to be scheduled next? It is! Yay! So, the scheduler is scheduled our second pod, and it's assigned it to 33JG. What's the last one assigned to you? 33JG. So that's interesting. We talked about the various functions in ranking things. One of them was selector spread priority. That's going to favor spreading things out to other nodes. Well, because they're all added together and they're all weighted, selector spread priority actually lost out here. I had some other things running in my cluster and said, even though we desire to have things in multiple places, this node is such a great fit that we want everything to go here. Luckily for us, this is a very, very, very tiny web app. It's basically a hello world. Let's come back and create our third pod. 05KV9. And it's successfully been assigned by the scheduler to our node. The same node, 33JG. So this is something different. Kubelet's kicked in. Kubelet has been like, okay, I've got a pod I've got to deal with. Well, first, I'm going to pull my image. So it's pulling boeing.grome slash helloHTP latest from Docker Hub. And it's downloading that. Oh, the second pod is having its images download. It's downloading that single image. It's downloading the third image. Why is it downloading at the same time? That's a great question. I assume it's because I probably shouldn't freestyle on this, but I believe it's because it hasn't completed the download on any of them yet. Or it could be my image settings on the deployment. So we successfully pulled the image. Yay, it's downloaded. We can see that we downloaded the pod. Like the involved object is the pod. Kubelet is the source component that's handling this. We can also see there's a third field involved object as well that we haven't seen before. Field path. So it's saying, okay, I'm trying to find the containers in this helloCubeCon deployment. And that's how it got the image name that it's downloading. So after it successfully pulled the image, it creates a container. And then it starts a container. So Kubelet has pulled an image, successfully downloaded it, created a container, and then started a container. So I expect to see this cycle two more times because we have three pods we need to start up, each with a single container. So we successfully pulled the image. We've created the container for our next pod. And we've started it as well. Third pod, yep. We successfully pulled the image. We've created the container. And we've started the container. So we did it. Yay. So now that you've navigated to your browser, you would see helloCubeCon. Kelsey did almost the exact same thing yesterday morning. I just want to say I had my GitHub out there or my Docker Hub container out there three days ago. But either way, great minds think alike. I am honored to have the same thought. So what have we done today? So we've looked at the various components that make up Kubernetes. We've shown how Kubernetes handles distributed state. We dove into how we reconcile and schedule state and schedule pods. And then finally, we traced the deployment through the system. So we're out of time, but I learned a ton putting this talk together. And I hope you all learned something today as well. Thanks.


Green tea has been reported to have numerous health benefits, including reducing the risk of cancer, heart disease and liver disease, due to its K-chin content, particularly epigalocaticin-3-galate. Studies suggest that green tea K-chins may treat metabolic syndrome and reduce the risk of coronary disease, but more research is needed to confirm these effects and elucidate their mechanisms of action. This article was authored by Kutun Ramadasan, Tambi Priyati, Choco Sabu M, and others.