giving both of these talks independently is that they reinforce each other. So what I'm going to talk about is how we enable the practice of continuous delivery by using principles of evolutionary architecture. So I want to start with a goal. And the goal that we have is for business agility. And we all know that IT departments and the business don't always have the greatest of relationships. And part of that reason is because the business doesn't feel like they can get their features fast enough. As an aside, mostly that's because the business for the last many years has been driving IT departments to be as cost effective as possible, not realizing that agility and responsiveness does not necessarily come at the cheapest cost. So that's one of the problems. But when we talk about business agility, we have this virtuous cycle that we want to establish. And the first is we want a testable hypothesis. We want to be able to go out amongst our business folks and find that person with a great idea. But instead of having them say, I've got this wonderful idea, we want them to pose it as a hypothesis. A testable hypothesis that we can build a feature and build in whatever it takes to actually test whether that hypothesis is true, deploy it, see if it worked, and go from there. And so once we have the testable hypothesis, we want to go through our software development cycle, including using continuous design techniques, including thinking about experimental design. How are we going to design the experiment? Do we need a control group to understand what the results of this test actually are? And then we want to deliver it quickly and release it quickly, more importantly.  Very, very important, we want to measure the outcomes. The number of software development projects that we've done where they have this wonderful business case that very clearly lays out, we're going to increase sales and we're going to decrease costs, and they never go back and measure it. In part, that's because they have these grandiose figures and probably overlapping projects that could actually quote-unquote take credit for the same benefit. But we need to measure the outcome. And then we want to repeat this. And so the whole purpose is to be able to allow us to try things quickly, decide if they work, get them out safely as well as quickly, and then if they don't work, pull them back out again, if they do work, build on that. We want all of this based on evidence. So that's a wonderful hypothesis. Wouldn't that be glorious if that's what we could do? What does it really take? And there are several different factors that enter into being able to actually make this cycle work. You have to be prepared to change quickly yet safely. And it's not just you've got the systems in place to do that. And that's a whole lot of what we talk about in terms of continuous delivery. But your organization has to be ready for that. And your users have to be ready for that.  So everything needs to be prepared to make the change quickly and safely and accept the change quickly and safely. One thing that's absolutely critical is you need visibility. You need visibility into how well you're doing. You need visibility into how well the change is doing. So this is, again, this measurement, this use of actual data. Not my gut tells me this is working, but actual data is critical into both how you're progressing and how that feature is performing. And we want to be able to experiment often. I get involved in a lot of discussions around innovation. And very often people want to measure the success of an innovation program by the number of innovations that have succeeded or the percentage of innovations that have succeeded. And unfortunately, if you're never failing in this, you're not trying hard enough. You're not taking enough risks. You need to be able to experiment a lot so you can take risks on some of those things that you're not quite sure how it's going to work out. But that might be the winner. And if you're only going to try a few ideas, you're not going to take a risk on one that's a real long shot even though that might be the cream of the crop in terms of being able to alter the way your business works. We need to think about balancing predictability and opportunities. Lots of people would like to order the world. I can lay out a plan. And in five years, the world will look exactly like I planned it to be. There is a sense in which we do want to be able to predict what's happening. And part of what continuous delivery allows us to do is to have less fear of the unknown of deployments. However, we also want to be in a position to take advantage of opportunities. And so we need to be able to make those trade-offs. Is this the right time to go off and radically experiment with something new because of the opportunities that has come to pass? And finally, we want to be able to rely on evidence and not guesses. And this comes in in terms of measuring how successful a feature is. This also is going to come up again when we start talking about evolutionary architecture. And we want to make sure that we're making our architectural decisions and our design decisions and our feature decisions on the basis of as much evidence as we can practically collect in that period of time rather than relying on guesses. So how are we going to do that? Various techniques, continuous delivery. This is going to provide the structure that allows us to move things across to be safe, to be quick, but also to make sure that we can undo harm that we have done. Evolutionary architecture. This is how we have to think differently about the software that we build to allow this cycle to take place. I'm not sure there is any way of doing this without using Agile software development. I can't prove it, and being a scientist, I like to be able to prove things, but I have seen no evidence to indicate to me that you could do this with a waterfall method. Continuous design. I've been around Agile long enough that we've gone through the phase of, yes, Agile is good for developers, but not analysts, not testers, not architects, not DBAs. And now we're in the, now we need to convince the design community and the user experience community that you don't have to do this big up front either. So continuous design is an important part of this process. And then internal software quality, and this is kind of the underpinning. We need to make sure that the code itself is able to be changed. And we've learned an awful lot over the last several years on how we can actually ensure that that quality is maintained to the point that we can actually improve the software. I'm going to focus primarily on continuous delivery and evolutionary architecture and talk a little bit about internal software quality. And as I said before, there are entire workshops on this, so we'll go at a high level. So in talking about continuous delivery, I want to start with talking about the last mile of software development. Agile software techniques in general have been optimizing the process of getting software done. But done is not in production. It should be. I know people who say, I don't want to actually sign off a story until not only has it been deployed into production, but I have the data that says whether it's successful or not. There's a good argument for that. There's also a good argument to say maybe we need multiple definitions of done. But the last mile means we're not just interested in optimizing software development as it stands, but this whole notion of how do I know I'm ready to go live. And in calling this out as a separate stage, I am of course not implying that testing waits till the end or anything like that, but there are different aspects of how we approach testing that we want to consider in addition to the normal testing that we do during the software development lifecycle, and then of course deployment and into production. And if we're going to speed up the time from Mr. Marketing person coming up with a brilliant idea from having data in my live system that tells me whether or not that's a good idea, we can't just focus on any one of these. In many of the places that we come in, it's not unusual. After development says it's completely finished, including all of the regression testing and all of that, then it goes through an eight-week testing cycle. That means there is no way that cycle from idea to production can be any shorter than eight weeks. It doesn't matter how efficient anything else is. Okay, so our objectives are as follows. We want to be able to release to production on demand. That means when the business wants it, not when IT thinks it's all right, not when we can shut the systems down between 1 and 4 a.m., we want to be able to release when the business wants it, when there's something critical that has to go out. That's what we're aiming for. If we're going to make that happen, we need to automate pretty much everything. There's no way that a poor person coming in at 3.30 in the morning is going to be able to reliably type everything that's in a five-page deployment script and not make a mistake. They can try really hard. The script can be perfect, but we are human beings. These things should be automated. And I mean almost everything, builds, deployments, rollbacks, migrations, all of those things. That doesn't mean you don't have manual checks if that's what you want, but you want to automate as much of this process as possible because we all know if you want to do something a lot of times, if you want to do something quickly, and if you want to do something safely, you want to make it automated. Now, it's amazing the number of times I talk about continuous delivery, and I'm sure others in this room have talked about it, have the same thing where someone says, but I'm terrified of allowing something to flow automatically into production. The first thing I will say is it doesn't have to flow completely unimpeded by any human looking at it. You can always put in manual big red buttons if you want. But the other thing is just because you can doesn't mean you have to deploy all the time. But isn't it good to know that you can? Because if something critical comes up, you want to know that you've done everything you can to shorten that cycle so that you can get that fixed out quickly. I was talking to someone who's basically a hosting provider, and he said, why should I care about continuous delivery? I don't write the apps, I just host them for people. Cycle, okay, well what about a critical OS vulnerability that comes out, don't you want to know that those things can be rolled out quickly? So, what are the different aspects that we have to think about in continuous delivery and what do we have to do about it? And the first thing is environments. There should be no works of art in environments in your system unless there is an absolutely positively compelling reason to do it. If you've got 12 web servers, they ought to look the same. Unless somebody comes to you with a case that this situation is just so weird that we have to make it different. But what is absolutely critical is you want to make sure that you've got enough consistency in environments along your chain from development to production that you don't get surprises. It is incredibly difficult to debug a production issue when you don't have an environment that is similar enough to production that you can reproduce it. I've been in situations where there's completely inconsistent patching between even different production environments let alone the test environment. These environments should be standard. You should be able to configure and rebuild an environment in an automated fashion. You should know what all your environments look like and those basic parameters then allow you to more readily promote things as well as debug things into production. I'm talking about operating system, I'm talking about your systems tools, and one of the issues with this, and this is something that we very often discuss when we're in this, okay, what packages am I going to buy, you want to take a look at the package that you're dealing with and make sure that it's something that you can script, that you can configure automatically because a lot of these tools are not designed to be dealt with from a script. Fortunately, that's improving and we're getting very clever about how to convince tools that they're actually being driven by a human being even when they're being driven by a script. But this is something that should be added to every single checklist when you're doing a procurement decision. And I automate this. It's amazing how low that is in many people's checklists until something goes wrong when you're trying to deploy. Related this is configuration parameters. Hopefully in a crowd that's coming to an Agile software conference we don't have to say things like these are the sorts of things that you want to have checked into source control, but you do want to have these things checked into source control. And that's where the nothing to access comes in. Some people go completely overboard on what they want to be able to customize in their application. So you want to think carefully what are the critical things that I want to be able to configure at runtime. A lot of these are probably related to your environments. Clearly you're going to point at a different database with a different database password when you go from your test environment to your production environment. But you want to be able to manage these things appropriately. You want to make sure that you understand which configuration files go not only with which environment but with what version of the software and builds. Fundamentally this pipeline starts with the ability to rebuild your software automatically.  It's amazing how many people really don't know how to build something that corresponds to what they have into production. And this is again something that makes it incredibly difficult to debug. Because if you don't know exactly what's running in production, how in the world are you supposed to figure out how to do that? Now there's some interesting questions that come up and I've had two different clients assert just as strongly the following two positions. And the first is when you redeploy into production I want you to deploy everything. Because I know that this thing has been tested as a set.  Exactly the opposite.  Because I know everything else works. That's the safest thing you can do. Now I think actually this second point of view was probably right in a situation where you don't know what your environment looks like, you don't know what your configuration settings are, and you don't actually know everything that's running in that production system. And in that kind of situation, yes. The safest thing you can do is deploy the smallest thing possible because you've got the least chance of disrupting one of these little magic patches that somebody put in at four o'clock in the morning to make something work. If you know that your environments are solid, your configuration corresponds, you know what's there, and you've got an automated promotion process, then back to this first person's view. The safest thing you can do is deploy as a unit because you know that's what you've tested. You don't have to worry about those other things because they simply can't happen in your environment. And finally, testing. It's your safety net. We're talking about testing in this context. We're talking about an entire suite of things. Of course, we're talking about unit testing. Of course, we're talking about automated functional testing to verify the new features that you're deploying. Yes, we're talking about automated regression tests. You may, in fact, want to build some kind of automated smoke test to ensure that once you get into this environment that everything's actually wired up the way it should be. You're probably going to want to look at some form of technical testing and performance testing smoke test, depending on the nature of the changes that you're making. All of these things need to be automated. You want to think about these test suites in the same way you think about code. It's very easy to get this massive regression test suite that has one functional test for every feature that was ever deployed in the entire lifetime of your system. But just like you refactor code as new things come in, you want to be refactoring these test suites. You want to be thinking about what's in those test suites. So that that running of that test suite can be as efficient as possible. It's one stage in that pipe. It is one length of time that is standing between you and your feature getting into production. And finally, deployments. You want as much as possible for your deployment to be automated. And that doesn't mean if you've got some really paranoid CIO sitting there who says, I want to be able to hit the big red button before it actually goes into production. There's nothing counter to the principles that we're talking about that will allow you to have a manual step. What you want is to automate the things that can be automated. And if you've got a CIO who says no, I don't want anything to go into production unless I get a chance to look at something. I don't know what something he might want to look at. There's nothing counter to that. But you want to have every one of those stages automated and you want to have a process of promotion across all of the different environments whether it be an integration test environment a technical test environment some kind of staging environment or UAT or a security environment and all the way into production you want to use the same deployment process potentially configured to take advantage of or to take account of different numbers machines, etc. You want that all to be automated because you don't want to have some poor person having to sit there at 3 o'clock in the morning and type things in. So that's essentially in a whirlwind tour the kinds of things you have to start thinking about in continuous delivery. Now, I'm not saying this is easy. I'm not saying that if you start from a position where maybe you've got source control on a few of your projects you've got a set of tools that you don't necessarily know what to script the deployment of because the product vendor thought it was really cool to have this neat GUI interface for the administrator to work for it's not necessarily easy to get from there to this it is a path but if you follow that path what are the benefits and the most important thing is that your deployments are no longer stressful events they can be fast and you've done everything you can to reduce the risk that something's going to go wrong into production. Dramatic reduction in errors you won't have people pointing the UAT database to the production customer database things like that happen unfortunately quite importantly it enables experimentation because you can take risks with features without worrying that you're going to bring the entire system down you can experiment more freely much easier to debug production issues because you know what production looks like and I think most importantly at the end of a big deployment there's always the big party we did it but the party is not celebrating the features that went out the party is celebrating the fact that nothing catastrophic happened or if something catastrophic did happen we ought to be able to focus on the good stuff in terms of business value that's going out in our deployments not the fact that we survived another one with our jobs intact and that's a major benefit of continuous delivery done right okay so I've been talking a lot about operating systems and systems tools and environments what about the code and this is where evolutionary architecture comes in this is where we have to start thinking about from the perspective of my application from the perspective of my software development process what do I have to do differently to make it easier to support continuous delivery and so I want to start by talking about some of the principles of evolutionary architecture first one is the last responsible moment I like to call this the latest responsible moment but it was coined first as last responsible moment you want to both think about from the perspective of your architecture and your application design what it will mean to evolve it there's something called Postel's law that I'll talk about which has to do with how different parts of your application interact with each other you want to architect for testability you want to think about what do I have to do to make this code to make this system easier to test and then there are organizational aspects and that's where Conway's law comes in so let's start with the last responsible moment you want to delay all of the decisions that you're making until as long as you can but no longer you want to delay them to the last responsible moment now why would you do that well it seems obvious but the longer you wait the more you know the more information that you have to make a decision between A and B this also allows you to minimize the technical debt that comes from additional complexity that you don't need this is a different way of thinking about technical debt I first heard this in one of Neil's talks if you decide to put in some kind of heavyweight queuing system early because you think at some point you're going to need all of these features of this queuing system every single day of development that goes up to the point where you actually need that you've been paying a debt you've been dealing with a tool that's more complicated than you needed and in fact it's possible that you'll get to that stage where you thought you'd need it and you really didn't and if you made the decision back here you've accumulated all of that delay during that time as well as whatever you have to do now now that you realize that maybe that wasn't the smartest decision in the first place people often ask how do I know what the last responsible moment is and that's where you have to think about what are the critical factors that will be impacting my decisions and you prioritize your decisions on the basis of how they intersect with those particular issues I dealt with the system once and the person actually said I don't care about performance and it's like you don't care about performance that's interesting they at the time had a maximum of 80 that's 8-0 transactions a day maybe that was going to quadruple over the next couple of years they didn't care about performance what they cared about is every single one of those transactions was in the hundreds of millions if not billions or tens of billions of dollars he wanted to know absolutely positively that there was no way a transaction could get lost anywhere because every one of those transactions was so valuable so we sure we did a soak test to make sure the thing didn't fall down and there weren't memory leaks but we spent a lot of time thinking about ways transactions could get stuck and those were the technical tests that we did if we would have just gone in there thinking okay this is a normal this is a normal transactional system you know we should think about performance we would have been thinking about the wrong thing and so we put our effort in what are the design decisions that are going to affect the reliability of the message flow through the system you think about what your architectural drivers are what is it that's most critical and then you look at for every one of these decisions in that particular case does it affect my ability my reliability of message delivery absolutely okay well we need to focus on this one is it performance I'll worry about that later so effectively what you're doing is you are setting up the characteristics or the criteria for what constitutes a good architecture for your system now obviously there are things about you know good taste and all the separations and all those things people talk about in terms of architecture but in terms of the specifics in your case this is what is it about my problem that is most important how important is security how important is failover how important is scalability those vary drastically from organization to organization and this is how you decide what kind of experimentation you want to do to get resolution on what you really need for a particular architectural decision yes in that case what we did was we moved forward all of the things that had to do with the reliability of the message flow so in part that was what was our message transport layer how how did it handle persisting messages what kind of testing we wanted to do and what kind of capability the vendor could tell us on you know losing messages and what our testing showed us is that no there was in fact a chance for these things to get in the wrong order in a failover situation so we added a layer of sequence numbers on top of it because the vendor couldn't guarantee in the case of a failover it would come in the right order so on the basis of that we looked at all of those different decisions and worked on the things that impacted those most important drivers and didn't think about some others so we moved those decisions forward because those were the most critical to us the last responsible moment for each decision is different so the last responsible moment in that case for something that impacted performance was very late in the day because that wasn't one of our critical drivers so that was maybe okay now we're setting up the testing of this particular core that's the responsible moment for that when we still have time to fix it in case for whatever bizarre reason we couldn't get 80 in a day not likely so that was very close to the end the last responsible moment on the other hand for decisions having to do with that transport we're much earlier so there isn't one moment there is a moment for each decision yes can you speak up please I can't hear you okay so the question is does this apply in a particular phase and is there opportunity lost because you're delaying things your selection of the last responsible moment should take into account the opportunities so you're delaying it as long as it's responsible to delay it if there's an opportunity that you could take advantage of that is going to enter into your decision about whether it's responsible to continue to wait or to take the decision at that time as for the first this analysis should be going on throughout the entire process you're making choices at many different stages if there's something that you're talking about in an inception that you don't think you really have to nail down an inception than don't put in your options if there's something that you think you can delay until release 2 don't make a decision until you start release 2 so this notion this is related to the previous question it is not that there is a single last responsible moment each decision has this concept of a last responsible moment and that will differ depending on what the decision actually is and what the specifics of your application is and that's why you need to have these discussions of what are your technical requirements what are the things that are going to be the catastrophic style of failures where do you think the pressure points are going to be on your architecture and your design that's what determines your last responsible moment for each one of those individual decisions okay thank you that's part of why you need to design around it actually hold that thought for about three more slides okay and we'll come back to it so the next thing is if you are in a system where you anticipate there are going to be lots of changes and this is probably going to be the vast majority of your system you want to think about your ability to change that system now I am not talking about the magic silver bullet of architectural reuse because architectural reuse in general means some group of people sitting off isolated from the rest of the development team deciding how these business common objects across an entire enterprise will be reused and writing up all of those things with no connection to how they're actually used and no ability really to understand how different potential uses might relate to each other or in fact be incompatible with each other so the first thing is the sensible breakdown of functionality and sensible in this context is a combination of both what makes sense to you as a technologist and what makes sense to the business person and one of the things that we're finding in the business processes are changing and in the way applications are having to change that things that used to be different in a business context are now getting combined and so we need to make sure that we think about from the context of our functionality what makes sense in a business context so we can have these kinds of conversations with our business users so the sensible breakdown of functionality this doesn't mean everything needs to be possible grouping because obviously that gives you the greatest flexibility of being able to rearrange the pieces but that might be too small and in fact we've seen situations where people just took this to the extreme and had so many different micro components that they had a difficult time putting them back together you want to consider the data life cycle and ownership one of the things that we are finding in this transition of systems is the wonderful data silos and ownership of the data was in these massive applications and you had this strong correlation between the data and the system and what we're finding is that many of the applications now we need a little data from here and a little data from there and a little data from over there and so these questions of data ownership data life cycle data replication are becoming more and more critical so as you look at your application the boundaries around the functionality are but what are the acceptable uses of the data what are the proper ways to give different applications visibility to your data who owns it, how often does it change when should it go away all of those different questions appropriate coupling I don't say loose coupling I say appropriate coupling because we need to think about for these individual pieces that we've come up with and I'm trying to stay away from the the C word component because that's a very overloaded word so the different pieces of your application you have to consider what is the appropriate level of coupling if you've got two pieces that have to work together a lot in a very high throughput environment you're probably going to want to couple them a bit tighter than two things where you've got completely different interaction characteristics so for each one of these interactions between these various pieces what is the appropriate level of coupling if you go through that process of thinking about this that makes it much easier than to change if you've got these two things that you do have at a loose level of coupling it's going to be a whole lot easier to change it might be a whole lot more likely that those things are going to change if they're that disconnected if you have some business process and isn't such a tight loop the probability of that tight coupling affecting you is probably going to be less it might still happen none of this is going to ensure that you're never going to have to do more work than might have been possible what we're trying to do is ensure that you do the least amount probable not possible but probable lightweight tooling and documentation the more weight you can keep off a system the more flexible it is now developing for evolvability what I've been talking about so far has been the big picture the major pieces now let's talk about the code I mentioned internal software quality earlier there are a whole series of metrics that can help you understand just how difficult is it for you to change your code and there's two aspects to how difficult is to change the code and the first is the one that is very often forgotten which is how difficult is it to understand what's going on in the code because you can't change it even if you wrote it you can't change it unless you understand it and depending on how long ago it was that you wrote it you probably don't understand it anymore and then the second is how intertwined how difficult is it to change that particular piece of code and that's affected with things like what are the connections between things so we've developed some metrics internally that balance off method length class length cyclomatic complexity fan in and fan out architectural dependencies to basically give a measure we call it toxicity to give a measure of just how difficult it is to change a piece of code and these are things that you want to track over time now if you're starting with an existing system I very often get asked do you mean I have to make my entire system perfect? No you want to find where the hot spots are where the toxicity is the worst but you want to match that with things like what's your change log what files actually change talk with the business owners what aspects of the application are likely to change things that are more customer facing they tend to be more because our customers tend to want things to change when they see a new feature over here they immediately want it in our system if it's talking to a back end mainframe that is fairly static probably if it's a mess it's okay because you won't have to deal with that mess probably until the mainframe goes away and then you can probably throw the thing out anyway so you want to marry how likely it is something to change oh there's one other question I very often ask I try to go to the the QA folks and say what makes you the most nervous when the developers tell you they're giving you new functionality in this area and go to the developers and ask them what keeps you up at night when somebody puts when your name gets put against changing something what are the areas of code that worry you because there's a lot of institutional knowledge in our delivery teams that we need to take advantage of and then you've got those list of things that worry and generally speaking those things that people are worried about are also your hotspots and so that is probably going to bump things up a little because if they're actually going to answer that question to you with that module even though it may not change much it changes often enough to give them Harper and that means it probably ought to be paid attention to the next thing that's important about metrics is that it's not always the absolute number we get very asked very often what's the minimum you know what's the maximum level of toxicity you could have and you know like a good consultant well it depends what is important is watching the trend lines because this can tell you a lot this can tell you about pressure that's being put on your development team if the trend is going the wrong way it can give you an indication that there are other things going wrong in your development process it can perhaps tell you that a decision that you made is introducing a series of problems we had one example where a project manager was changed over and the metrics plummeted everything was getting worse it allowed a conversation happen okay new project manager what's happening with the dynamics of the team so that something can be corrected so in many cases it's not so much the absolute number as the trend line that you will see now reversibility this is the slide I wanted you to wait for reversibility is basically to the extent possible you can extend your last responsible moment if it's easy to change your mind and there are lots of things that we do to make it easy to change your mind we put in abstraction layers that are at a more functional level so that we can swap one queuing system for another more readily there are all kinds of things that we do when we develop code we put in symbolic variables lots of things that allow us to change our mind so whenever you're looking at these particular decisions is there a way that you can make it easy to change your mind and as long as that doesn't cost too much that's probably something that's worth doing if it's a decision that you're still not really sure is right you feel like you're taking a risk making the decision but you feel like you have to make it now it's probably worth paying a little bit more to make that decision reversible add in a layer of abstraction do something to make it easier to change your mind you don't want to do this always or you end up with this house of you know ability to reconfigure which is excessive all of these things come down to using good judgment I did have somebody once tell me that that that agile is basically either telling you to be hyper responsible or hyper careless and it's neither one of those we have a lot of responsibility to make particular decisions and this is one of them now post us law this is talking about when we think of these two different pieces that have to communicate with each other how should we set up the communication and there are all kinds of debates about we need a universal or ubiquitous business language for all of our messages that that flow on our enterprise busses there are all kinds of how do I design messages how do I design apis this is what this law is talking about and the first half of the law is be conservative in what you send the more you send out the more people are going to expect you to continue to send that out so one school of thought is whenever you put out a message about an object put out the entire object well you've just exposed a whole lot of information about your internal implementation that you probably don't want out there so be conservative in what you send but be liberal in what you receive if all you need to pull out of a message an object a database of whatever is a name and zip code don't parse the whole thing don't validate the whole thing pull out your name pull out your zip code and go on because that means people can add, subtract change everything else that you don't care about and you don't have to change a thing the only time you have to do anything is if they change something that you need you can't get away from that one I can't magically protect you from every change that anybody is ever going to make but we can be protected from the changes that shouldn't break our code if all I need is name and zip code my code should not break if they make a change in the customer coding I don't need it why should that break my code so be liberal in what you receive now unfortunately this does run counter to some some expectations about protections well if I'm getting a message I need to validate that the message is correct well why are the consumer of a particular piece of information why should you be validating that all of these other things that you know nothing about why should you be doing it it's an improper delegation of responsibility so be liberal in what you receive and that way the only time you have to change is if they change something that you need only validate what you need this should hold for any kind of information exchange you want to make sure that any time you're interacting whether it's a message any other kind of of inputs databases if you can get away with it every kind of information exchange you want to follow these policies and you also should probably think about what you have to do with respect to versions and such just in case some of these things do change because as I said we can't be protected against all changes architect for testability if there's one single point from the evolutionary architecture part of this talk that I want you to remember it's this one because what we've seen is that the more people think about testability the more people think how easy is it for me to verify how the system works the better architected the system is you very often find this in code development as well when you think about how testable is a piece of code you get the properties that you want out of a good system when you're focused on how easy it is to test the system this leads you to things like using your message infrastructure to send and receive messages not to put business logic in there because it's very difficult to test logic that's in places like this similar to stored procedures it's very difficult to test business logic that lives in stored procedures you want to think about how you componentize in terms of what functions are being delivered to the business if you think about behavior driven development if you think about acceptance test driven development all of these things are talking about what is the behavior of the system that is being delivered to the user if your architecture is designed around technical layers it's going to be more difficult to test these end to end functional scenarios so think about your architecture from the perspective of what makes sense in the way that you're going to test and the way you should be testing has to do with how the system is going to be used you want to be testing at multiple levels obviously one very helpful style of test I call it a contract test when you are interacting with another system particularly another system that's under development while you're under development you need a way of finding out when they're changing one of those things that you rely on now very often the role of a of a good enterprise architect is to know what are those assumptions that these different systems are making of each other so they can kind of keep an eye on making sure that doesn't break it's not a very scalable process so what we advocate here is you use tests if I'm working on a system and you're working on a system I give you a set of tests to run against your system that document the behavior that I'm expecting from you and this is how I am documenting my assumptions about your system and that way you know if you've changed something that's going to impact me and if you think about from the perspective of agile development what testing is all about it's one way to trigger a conversation when something fails you know you ought to go ask a question so if one of my tests breaks you know to come to me and say our functionality is now changing in such a way that your system has to change documenting the assumptions that are being made about the system so that we know when those assumptions have been broken at the earliest possible moment rather than waiting until we get to some kind of integration test environment which if we're doing continuous delivery won't be that long and of course this is another intersection of continuous delivery if we're going to have all of these different tests if we're going to think so consciously about testing that we're going to worry about it in our architecture phase at our design phase we want to have the right kind of build and test automation support to make this thing easy and then Conway's law although this is the technical side every once in a while we do have to think about organizations an organization's design reflects itself in the products that it builds there was a great story that Martin said once he was talking to someone who was responsible for building a compiler and they had collected this hotshot group of people that were located in four different labs and this guy was smart enough to say therefore I will have a four pass compiler because there's no way that he was going to be able to build anything else that didn't reflect the reality of the organization he had people sitting in four different places whether he wanted one or not he was going to end up with a four pass compiler so we had to figure out how to architect that four pass compiler very often you will see that if you have organizational silos that don't talk to each other very well that their modules don't interact very well and conversely if you have two organizational units that do work well together you don't see that same kind of organizational breakdown so you can see in a system some of the pathologies that exist in your organization silos often result from broken communication and this is why another one of the reasons if you go back to this architect for testability in these business sensible components if you start thinking about I'm going to have a front end team and I'm going to have a middleware team and I'm going to have a database team you've now set up that every single thing that you write is going to go across silos that's almost a recipe for failure because you've got so many broken communication channels you need to think about these things in such a way where you're optimizing the communication or your system is going to reflect the pathologies of your organization and so in summary you don't want your product to look like your organization you either have to change your organization or your product because they will reflect each other it's almost I think it's one of those fundamental laws of nature okay so evolutionary architecture one of the debates that I've had with Neil about this is he talks about emergent design and I talk about evolutionary architecture and so there's this question of course difference between design and architecture but I'm actually more interested in the difference between emergence and evolutionary and the way I think about this is that an emergent system is kind of self organizing and when we talk about emergent design you're looking at the way these things are changing on their own and deciding what you want to what you want to extract and impose in terms of order as a result of what you're seeing in terms of how things are changing when I talk about evolutionary architecture I want to go back to those architectural drivers that I was talking about one way to think about this is this is a fitness function of the architecture for your organization for that system and this is a way of knowing as my architecture changes as I add new applications as I change the aspects of my system am I improving the fitness if you will of my architecture and I'm going to characterize that in terms of those architectural drivers what is necessary in terms of scalability what is necessary in terms of security and every time I make one of these decisions every time I move my architecture in a particular way I need to check it against those drivers to ensure that the decisions that I'm making still reflect what I've said is the characteristics of the right architecture this is the role these central architecture groups should be playing is helping the development teams understand what are the characteristics that this system needs to meet in order for it to be an appropriate player within the broader enterprise estate delay your decisions as long as you can and that means you might have to to build things into your application to allow you to change this reversibility that we were talking about it doesn't always come for free where possible though if you can delay that decision you can have the information that you need to make the right decision one of the things I often find when I'm brought in to sort of mediate between a development team and a central architecture group what I often hear from the development team is they're picking these things that don't make sense and usually that's because there hasn't been the information flow of these are the things that you need these are the problems that we're facing and therefore these are the characteristics of the tool that we'd like you to select to solve our problem the architects are very often in a situation where they don't have that information so you want to delay decisions so that you can collect the right information that allows you to make the right decision it doesn't sound like rocket science but you'd be amazed at how there's such organizational pressure to pick the tool early but that very often is a mistake understand the various forms of technical debt some of this can come from tools some of this can come from the very conscious decision that we make to push something out the door because it's necessary to get it out the door take a look at the technical debt that you have with respect to continuous delivery where are those where are those systems that are relying on these bizarre configurations and what can we do about this where do we have holes in our automation process where do we have holes in our testing process technical debt takes many forms we often think of it as just emphasis on the maintain we talk a lot about making tests doing tests automating our tests but just like our code that needs to be maintained we need to think about what does our test suite in the large look like just like we think of not just what a particular method or a particular object looks like but we think about our application we need to think about our test suite in the large as well and maintain it not just create it okay it's not quite it I alluded to this earlier for several years boards and management committees have put pressure on IT organizations to lower their cost and how do you lower cost and lower risk you lock things down you make it hard and harder to change because if you don't change anything and it was working before chances are it's still going to work and so IT departments were pushed in this direction now all of a sudden the business wants all these new features and they want them fast and they're irritated at the IT departments because they can't change fast enough an organization that is good at stability is not going to be good at responsiveness it's a different mindset it's a different organization there are different metrics that you use to measure how good you are so these two organizations are incompatible so we need to think about how we are going to change the way we approach problems so that the part that needs to be responsive can be responsive now ironically continuous delivery applies to both the responsiveness side and the stability side because regardless of how hard you try even those things that you don't want to change very much well Microsoft is going to come out with a new operating system Oracle is going to come out with a new database system you are going to have to change these systems and what I find fascinating about continuous delivery is that both allows you to go fast because of the automation because on the focus of I'm going to optimize this time from feature to production but it also allows you to reduce risk because if you've got these things automated you know that each time you do this it's going to happen the same way and so whether you are a stable IT department where your job is really to just keep the lights on on things that have been running the same way they've been running for the last five years continuous delivery is just as useful because for those times you do have to change you have very low risk this is the way to increase stability is to reduce the risk when you do have to change things and yet continuous delivery also helps this other kind of organization by allowing you to move quickly by allowing you to focus on okay I'm going to intentionally destabilize this system but I'm going to provide this stable foundation so that I know when something goes wrong I can isolate it as quickly as possible so how do we achieve then business agility we architect for real adaptability we think about what can we do both to take advantage of those things that we can see but also make it easy to take advantage of opportunities that we haven't seen yet and this is where that preemptive reuse really breaks down the environment that our systems are operating in right now are changing so rapidly that no matter how good you are, no matter how well intentions you are, you are not going to be able to know what your customers are going to want nine months down the line let alone what kind of code is going to be required to deliver what your customers want nine months down the line so rather than trying to predict just like just like XP says embrace change architect your system to be adaptable and then keep them poised for change keep your keep your eye on technical debt try to take care of it so you're always in a position to be able to change the power of the cost and risk of experimentation continuous delivery this is a huge part of this you want to know that you can put something out and if things go wrong you can back it out you want to know that you can put things out quickly and not jeopardize the other aspects of your system you want to maximize the visibility and feedback so that your business understands what's going on you understand what the business priorities are you have a way of demonstrating the progress that you're making you have a way of supporting any arguments you might have about the risks that are being taken having all of this information available provides you the data and the arguments to have real discussions with your business to understand what their priorities are and importantly to help them understand the risks that you're worrying about we need both to lower the risk and increase the responsiveness and that's a lot of what we've been talking about here continuous delivery does this for us and evolutionary architecture does this for us the testing safety net making sure that you've got things properly encapsulated so changing something over here won't necessarily affect something over there looking at our message exchanges all of these things go towards lowering the risk and increasing the responsiveness of the system so my assertion is that by combining the techniques of evolutionary architecture including aspects of emergent design including everything we know about agile software development everything we know about unit testing all of the automated testing with what continuous delivery is brought in terms of how we ensure that we can in fact take an idea drive it through the process test whether it's doing what we thought it would do and react accordingly those two things combined put us in a position where we can start being responsive to what it is that the business wants us to achieve rather than just always running to keep up any questions I'd like to thank you by offering future goggles and stuff like that so I was talking at a different level there are lots of different approaches for trying to enable basically simultaneous development of different pieces of code so feature toggles for one example you're working on adding something you want to release so you want to keep it turned off for a little while there are many different approaches you've got some people will advocate feature branches if they're short enough lived maybe that's okay feature toggles are more useful when you've got functionality the cross is a broader range of a system so it's more difficult to control if you will what you include what you don't include to me those are less architectural decisions and more design decisions and that's why I was focusing more on the architectural level but there's there's still a lot of work going on within a particular organization what is the right approach to keeping a feature isolated enough so that you can determine when in fact it gets deployed doesn't it add to business agility when you selectively expose certain features to a demographic segment that way your ability to respond to change as a business definitely improves I feel that it's a part of business agility the ability to selectively roll things the ability to do A-B testing those are all definitely aspects of business agility there's no question as I said at the beginning each one of these topics could be an entire workshop and I'm trying to cover both of them in 90 minutes but one of the interesting things that I think is finally getting more attention is to start doing A-B testing how do you you effectively have to support both things somehow how do you support that how then do you ensure that you get the right kind of data testing so there's a whole realm of questions on okay I want to expose this feature to this population that feature to that population and here's my control group how do I manage all those things there's an entire range of issues that come around that agreed a quick question we had one over there first what do you think the disadvantages with the continuous delivery what do you perceive as the risks or the operational risks going with the continuous delivery apart from the infrastructure the operational risks of going with continuous delivery or the disadvantages personally I think there are a whole lot more operational risks not going with continuous delivery however the transition can be more of a problem as I said there are a lot of tools that are often used within operations departments that are not necessarily ready to be scripted so there's a fair amount of work to get those things scripted if you have different environments that are set up as these sort of works of art you've got to figure out well how much of that is necessary and how much of that was a systems person just having fun experimenting so you have to be able to move those applications to fit in those standardized environments so there's a lot of work that goes from moving towards that but as I said I think it actually reduces the operational risk once you get there because you know what you're dealing with on all of your different machines you have a mechanism by which the environments you know that this environment looks like this and therefore if something goes wrong you can figure it out because you know what's there so there are challenges organizational and technical in moving to this but I think you actually reduce your operational risk once you're there so sorry yes so but they're not necessarily your users if your hypothesis is something in a customer facing system yes you might have focus groups and such but your true measure of done is you thought that this was going to increase sales or decrease drop offs or whatever metric that you put in and you can only measure that once you go into production and so from the perspective of did the business get what it asked for you won't know that until it's actually in production yes you've got users and you've got acceptance test but the true test of whether or not this is complete is did it satisfy the initial hypothesis and I think that's the shift the mental shift that's made rather than the business cases we used to write and oh yeah we're going to be able to decrease our customer support staff by 75% and they never measure it so that's the distinction Rebecca you talked about feature branching and feature toggle and you said that we could do feature branching if the feature is it does not take long enough to build the feature what do you mean by long enough what would be the definition of that long enough um it depends um certainly personally I wouldn't want it to go outside the bounds of an iteration and I probably wouldn't want it to go too far outside the bounds of a day or two the problem that you get in feature branches are seductive because you have this ability to live in your own branch and make whatever assumptions you want to make about the world and the longer those things stay out there the more possibility you have for your view of the world to diverge from the real world and so feature branches are nice for experimentations and spikes you might want to isolate something but the longer something is isolated the more problem you have when you're trying to bring it back into the main line and the whole idea behind continuous integration was you are finding out as soon as possible what you did to break somebody else or what somebody else did to break you and if you're living on a feature branch you can be all happy because your build is passing but you don't know how your view of the world reflects anybody else's but do tools like which have distributed repository do they help in such cases? we have run a lot of distributed projects using a single code repository and using the the same rules and expectations around continuous integration you do sometimes have to think about it if your network connections are not very good but I've seen that unfortunately used more as an excuse to let people play by themselves rather than a legitimate cause for concern what we have sometimes done is you might use a local repository for a team and then those things get synced up at the end of the day to resolve those if it's easier from a distributed perspective but to me there's usually better ways of approaching the problems introduced by a distributed development team than letting the teams run independent of each other in particular if they're physically distributed they're likely not to be talking enough to really know what's going on between the two and that just increases the need to make sure that you're bringing those code bases together as soon as possible thank you I have a question so you talked about silos what I wanted to know was could you suggest some ways to basically break the silos so that your productivity is not impacted can you speak up a little bit could you with reference to silos could you talk about some ways one could break silos to increase productivity and secondly how do you handle cases where your product has a component which is research intensive and there is a black box and there's a team of scientists who are actually working in a silo mode to give you models like a data mining model then how do you kind of incorporate that silo into your regular productivity workflow where you have your web server team your database team all working in concert to integrate a research intensive component and apply agile so that you're able to still get productivity yeah that's what it seems so you're talking about basically you've got a research team that's going off and trying to discover something interesting in a piece of data and then once they find it how do they turn it over right so before you can go to production you need some solutions from a research team they need to get you some data mined patterns or something like that and nowadays many projects have some aspect which is kind of going into a hard science or a research mode so you have a bunch of people who are experts working on that and it's you can't apply this no silo rule in that case because these are people who know that domain they might be even PhDs in that area so how do you kind of incorporate that aspect also into this workflow the data scientists that we have working for us on our teams probably wouldn't say that just because they have a PhD doesn't mean they have to be in a silo by themselves and we have people who are doing exactly that they're the PhDs in mathematics and statistics and they're looking at these pieces of data and they're working very nicely with the delivery teams you have to have a conversation on what's the input into this model that they're building they might be building their model in R and then you have to figure out how are you going to reflect the actual implementation of that model in your production system so you have conversations about what's going to come in and what's going to come out and just like you've got we advocate the DBAs who know a lot about tuning databases are experts too and they ought to be working with a team you know the user experience people need to be working with a team there is a there has been this progression in Agile where it started with okay well it's okay for developers but none of the rest of the roles and then we brought QAs and BAs into the mix and you know the PMs we brought into the mix and DBAs are slowly starting to come into the mix and architects are slowly starting to come into the mix and user experience people are perhaps a little further behind the scientists are going to be the same but from the more general question of silos I very often hear that okay well we've got this pool of people with a specialized resource or a specialized skill and six teams need them so how do we handle that well you don't handle it by making a black box that is a silo and tossing effectively non-prioritized jobs into that box and having all of these delivery teams waiting for an unknown period of time for something to come back out of the box you know there are other ways of doing resource management than just using a silo thank you most of the multinational companies that we work with have a specific window where they deploy the application to production and it usually happens not more than twice a year so having said that how are we going to make sure that the features are getting deployed and we are checking the vulnerability of the application and getting the feedback from the customer when we know that the production deployment is going to happen only twice a year part of our mission in continuous delivery is getting more of those organizations who only deploy twice a year to deploy more often than that so and one part of that is very often they only want to deploy twice a year because it's too risky to deploy more often their internal test cycles take that long they can't deploy more often in fact we very often get called into organizations like that because they want to deploy more regularly they just don't know how to change so that's one that I think we are definitely trying to change the other thing I would think about that from the perspective of continuous delivery is I can't I can't emphasize enough the fact that we talk about this a lot in terms of this is how you can get things out fast but it's also important that this is how you can get things out and lower the risk and that's a message that a lot of even the the organizations that are not deploying as often like to know that when they do it they don't have to have this tense situation where they're not sure whether or not it's going to work and once you get that part in then you can start moving a little bit further out and what can we do around the internal testing cycles to make it that you can deploy three times a year instead of two and then four and then all of a sudden they start to really like getting things out more frequently and then it sort of steamrolls from there yes this was very interesting session I found that very interesting part was when you mentioned the cost control is towards the stability but the value creation happens and it is mainly due to responsiveness so there's a balance between both of them and you try to emphasize that with continuous delivery we can perhaps have a less risk and most responsiveness but do you suggest something else as well or is this the only thing that can enable that to happen or is it a number of factors that you have found in addition to the continuous delivery that will help reach that golden balance well I think the other way to reach the balance is you've got to separate out the problems there are aspects of an organization that it's like electricity you know you don't have a steering committee to decide which power company and how you manage your power and there are some aspects of the IT estate that really are utility functions and then you've got the things that really distinguish you from your competitors what is your business differentiator what aspects of your technology system and those you want to have stable enough so that they're not falling over all the time but responsive and then the one critical part of that is you have to have the ability for things to move across that barrier it used to be back a long time ago that the ability to communicate by email to a customer service representative was an incredible differentiator because not too many people did it you know and so email at that point and that email functionality was a critical part of the differentiation now everybody takes email for granted and people are now talking about things like oh we've actually got this little intelligent agent that can you know isn't really a person but you kind of think it is because it will you know take you through all these steps and hopefully be smart enough to send you off to a person when you're confusing it so you know some of these things can move across that barrier but fundamentally I do think we have to start thinking differently about those aspects of our business that differentiate and you can't manage those in the same way you want to manage your accounts receivable and your accounts payable because generally accepted counting principles haven't changed in a while SARBOX kind of you know set everything on fire for a little while but that's kind of calmed down so you want to think about those kinds of applications in a different way than you think about the ones where you actually are differentiating yourself from your competition very interesting point because most of the companies that is a trick question to be answered if we can reach to that answer it is a success thank you you're welcome and I think we've already run over time I'll hang around for a little while but I'll let you go for your break