Hello everyone! I'm Kieran. Hello Gary. I was in his presentation so I have to put Gary in my presentation. This is really special for me because two years ago at PHP UK in 2015 I did my first ever conference talk upstairs on the big track and it was the first time I'd ever talked in front of people. I've done quite a lot of talk since then but this is still a really special conference to me because I came to it for years before I ever did any speaking. If you did see Gary's keynote last night I can really attest to what he was saying that becoming more engaged in the community and getting more involved in open source not only is really personally rewarding but you get more money. Which is the main takeaway I had from that talk. So what I'm going to talk about is this thing, driving design through examples. It comes from a set of practices called Modelling by Example. This is a term coined by Constantine Kudrashov. It's a kind of set of different practices. I'm going to talk through what it is but it's not a really strict checkpoint thing. You can't say yes, you're doing Modelling by Example or not. It combines some concepts from BDD and some concepts from DDD. So I'll give a bit of a sort of primer in the relevant concepts from each and sort of show how it comes together in a particular workflow for building software. And show some code examples and stuff. But it's not really going to be focused on the tools that we're using. I forgot to say who I was. I work for a company called Invika who have a big stand out there and I lead maintainer of a project called PHP Spec. And I do a lot of coaching and programming and things for people. So let's start with BDD. Behaviour-driven development is quite well established now. A lot of teams think they're doing it but they're not. They're just writing weird tests. What's the point of BDD? It's that the behaviour of the system is what's going to drive your development. We have BDD acronyms in this industry but we're only going to cover a couple of them. So what does behaviour mean? It means we're going to start our process by talking about how the system should work. Which sounds obvious but lots of people don't do that. They start it by opening an editor and typing stuff. Maybe we're going to have a conversation with someone about what it's supposed to do before we get to that point. And it's got many focuses. It's about building things well. So having conversations and documenting how parts of your system should work, maybe in the form of executable tests, is going to help build quality into your application. You want to build the right thing. We also want to figure out... My unit tests are going to tell me I've built the thing that I thought I was building. I also need to figure out is it the thing I was supposed to be building. Which is sometimes different. And another part of BDD we're not going to focus on much is building things for the right reasons. So a lot of BDD conferences, people are talking about project goals and how do you know if your project succeeded to achieve the business metrics that we wanted and that kind of much higher level stuff. So maybe that's confusing. A good definition of BDD is Liz Keogh's definition. BDD is when you use examples in conversation to illustrate behaviour. Behaviour is this input leads to this output. And an example is a particular way of communicating. I'll stop filling with this. Why examples? Examples aren't actually that important to understand. What you really want is the person who's building the system has a mental model of the rules of the system that match the person who's paying them to do it or match the business needs that's going to be fulfilled. So really we kind of want to understand rules. Here are some rules. The problem with rules is they are broad and so it's hard to make sure the person you're talking to understands what you're saying. So we're going to work through an example that's a kind of loyalty scheme for frequent flying because it's a sort of pet hobby. We're going to start a budget airline that flies between London and Manchester. And the thing that's important for the people who are going to implement this system is important that they understand these rules and a business will often start talking to you in the format of trying to explain the rules to you. So the rules are every £1 I spend on flights I earn a point. 100 points can be redeemed for £10 off a future flight. It's a very simple scheme. And flights are taxed at 20%. Now these might be the rules that are explained to you, the development team which I include testers and all the different types of developers. Project managers maybe, get some credit. There's a whole load of other rules that are implicit to the business of flying that the customer aren't going to remember to tell you about. So there's a lot of assumptions they're making when they're conveying these rules and it's easy to read these rules and get a slightly wrong understanding. Can anyone see any ambiguities in here? You just shout one out, yeah, who? Yeah, that's a good point. So if it's £100 and I spend £120, how many points should I get? So what I've done there is I've asked the question with real numbers in and that's going to transform it into an example. What's the actual number of points I earn if flight costs £100? You heard one over there? So I'll repeat it for the thing. If I spend £1.49, how many points do I get? Do we have fractions of points? So even with very simple cases where you think, I'll just tell them the rules and they'll understand and they'll build the software correctly. I've got a context that I'm measuring these rules against. Like obviously in my industry, it's really obvious where the points get awarded for the tax because it's how everyone does it. And I'm not going to think to mention it as part of the conversation. Or obviously in our industry we know that things get rounded. Like I've been in the airline industry for 50 years and it's always been done this way.  So introducing examples here are what's going to reduce this ambiguity. Even for simple cases, you just say, give me an example when I'm flying and it costs £100. The kind of questions I came up with, when I redeem, when I've got a flight that I'm going to earn points on and I use some points to lower the cost, do I still earn points back on it? Maybe that's a good question to answer before the end of the project.  This tax thing, if I've got money off, is the tax related to what you're talking about. If I've reduced the cost by spending points, does that reduce the tax? Or is there some crazy tax law that we have to add to our list of rules? This kind of conversation is the bread and butter of behavioural development. Techniques for asking this kind of question. Making sure everyone understands the rules before they build the software. That's basically what BDD is about. It's not about figuring out what the user interface is going to be or writing the code up front and generating UML. It's just about making sure everyone understands this kind of behaviour of the system. What should the system do in these different cases? The examples become unambiguous. For the same set of rules, we can quickly, through a conversation, generate some examples. Let's say a flight from London to Manchester costs £50 if you pay cash. You'll pay an additional £10 tax and you'll earn £50 points. If you pay entirely with points, so you can throw as many points as you like onto it, you pay entirely with points that will cost £500 points and you'll have to pay the tax on the original price before the discount. You don't earn any more points because it's a reward flight of some sort. There's a scenario here. If I pay, just get some of the price off, this is what should happen. This becomes a lot more understandable. In this room, you can't read the bottom of the slides. I forgot. If you pay with 100 points, it will cost 100 points and you'll have to pay the rest of the cost and you pay the tax based on the original cost before you reduced it with points. You don't earn points at all on a reward flight. Often through these conversations, you discover a new rule. You can annotate that in your rules. If you spend any points on a flight, you don't earn any new points. These are harder to understand. That's quite important. It would be easier for everyone involved if someone could just explain the rules and then I immediately understand the rules and I go and build the software. In real life, there's normally some kind of ambiguity. You can think of examples maybe as an algorithm in one person's head and there's an algorithm that's been formed in the other person's head based on the chat you had and then you run some real numbers through it and if the same real numbers pop out for both people, we've got a high confidence we both understood what the hell we were talking about. If different numbers pop out, we probably need to talk a bit more. I can say, hey, why? Like you just did. Why am I not earning points here? What's the hidden rule we didn't talk about yet? That's BDD. At the end, everyone understands what they're meant to be doing. There's an amazing side effect. The examples are objectively testable. By generating these examples as a way for us to understand what we're supposed to be doing, we've also generated three test cases. Once the system's been implemented, I can validate that it works by setting up a flight that costs £50, trying to pay for it with cash, checking I get charged £10 tax, and checking that I get 50 points credited to my loyalty account. This is why BDD is so intimately connected with test automation. The conversation you have at the start leads you towards having nice test cases that cover all the important things the business cares about. You might have other tests later covering things the tester cares about that they thought of when they were testing, but it means you've got this kind of synchronisation point. We checked we both understood what we're supposed to be building, and we wrote it down, and we can use that to drive automated tests later. In BDD, this step of taking those cases and generating test cases, obviously there are tools to help with that. Let's look at Gherkin. Gherkin is a formal language for writing down examples. You do not have to use Gherkin. There are tools that will make it easier for you if you're using this format. The most important thing is to talk to people and deliberately introduce examples into the conversation. Say, can you give me an example of what would happen if I paid £100? Write that down. That's the most important thing. Writing it down in Gherkin format and using the tools that can automate Gherkin format will make your life easier, but it's just like a convenience step. You should only automate it once you're doing it correctly. Don't automate it when you're doing it badly. You just do bad stuff faster, more efficiently. Gherkin looks like this. You have a file called a feature file. You get a nice bit of free text where you can just stick stuff in there. Here's my understanding of the rules. Please see this page on the wiki. Here's the user story that made us build this feature. Here's a picture of my cat. Anything you want to stick in there helps someone who reads this file later. The formal bit is these scenarios. There's a keyword scenario, and then you say what scenario you're going to describe. These are the three we had before. Paying cash and earning points. Getting some money off the flight using points. What should happen? What should happen if I pay for the flight entirely with points?  The most important things are the when and the then. When this happens, this should happen. This input should lead to this output. We have a thing called given, which is like why is that input leading to that output? What happened in the past? What states the system in? That means when you do this, this is going to happen. You can focus just on the top scenario, but I wrote out the three scenarios. Given a flight costs £50, so there's this concept like already in the world, there's a flight being set up that costs £50. Someone went and typed it into the flight system. When I pay with cash, so that's the input, someone does something that indicates they're paying, push a button or a type of command or something, then I should pay £50 for the flight and I should pay £10 tax and I should get 50 points. We're not talking about a particular user interface. We're just writing down these business rules. We're writing down examples of the business rules we talked about. What's the system supposed to protect? I'll do the same for the other scenarios. We'll see in a minute you can use tools like Behat. There's a family of tools basically called Cucumber. Behat is the PHP tool that will help you write tests based on this format. You could just write this down as text and use PHP units and write tests following the outline and it'll be perfectly fine. Cucumber workflow is a bit more efficient if you're doing this gherkin thing a lot. Who writes these things? There's a concept called the three amigos. They're sometimes called three amigos sessions but don't let normal people hear you calling that than that because they think developers are weird enough anyway. You need someone who understands the business rules. In a scrum context, you might have someone called a product owner. There'll be someone, some subject matter expert who can take part in the conversation. You need someone who's good at validation, a tester. They're also really good at asking these, but what would happen if and finding an extra case where are the gaps in the rules? They're really good at identifying those things. You need someone who's going to be building the thing. Off alternatives. It's important that they understand the requirements, so it's important they're in the room. When do you do it? Before you start working on that feature, that's the last point at which you can do it, just before you start typing. Don't do it too long before you don't have this conversation about an entire 12-month programme of work and try and write out gherkin scenarios for everything. It's a massive waste of time because as anyone who's worked on a long project knows, you won't do half of that stuff and you'll actually build a load of other stuff no-one mentioned at the start. Not too long before, a few days before you're about to work on it.  If you're doing Scrum, for instance, you probably want to do this before your sprint planning. If someone's going to say how much work you're going to get done, you want to understand. You have to optimise it for whenever you've got access to the right people. Ideally, I've got all these business experts and testers and developers all sitting together in a room constantly trying to make the software better, but the reality of life might be different. You might have a business expert you get to talk to every two months for a half-hour meeting and you have to cram this conversation in. Sometimes you want to make these conversations very formal and say we're doing an example workshop. We're going to do an example workshop for the stuff we're doing in an sprint. You're invited. Please come. Once you've got a few examples, which are called scenarios as well, as part of the conversation, you drive, you know, what we've identified a few scenarios, what other scenarios can we think of? I think I've got this list of questions from Liz Keo. I get most things from Liz Keo. When would this outcome not be true? So you've got a scenario saying, I spend £100 and I get 50 points. You can start asking questions like, why I wouldn't earn the points if I spent £100? Having those real values in there is going to help. Oh, yeah, actually, if you're a member of staff, you won't earn the points. And suddenly you've discovered a whole new set of stories that you're going to have to build about staff discounts. What other outcomes are there? I earn the points, but also something else happens.  There's probably other things that happen.  And you kind of say, that's an easy test to pass. When I click the button, I should see congratulations you've purchased a holiday. That's easy to implement. You just echo some strings. There's probably other things are happening. Something gets reserved in another system. Does this implementation detail matter? It's natural for people to start talking about, and I can click, but just don't capture that in the scenario. There's better ways to talk to people about user interfaces, more graphical ways, maybe with a marker on a whiteboard and wireframes. Trying to capture UI detail in text is pretty much a waste of time. But remember, these things aren't contracts. It's a conversation you're having to try and understand what to build.  It's just a kind of living document. This is what we think is going to work. It's a shared understanding of a feature. It's a starting definition of done, so if we don't have another conversation between now and me finishing the work, and it does all these things, everyone's going to be happy. But you might have a better idea. The business might come up with a much better idea tomorrow, and we can revisit it, have another conversation. They give an objective indication of how to test a feature, so I touched on that.  So you can get a person to read the eight scenarios you generated for that feature and just check each one works manually testing it. Obviously, you'll want to automate it, because I think most of the people in the room are developers and are lazy, so we don't want to sit there clicking on things. Even testers don't want to sit there clicking on things. They want to be doing exploratory testing and thinking about the system. That's essentially BDD in 20 minutes. Let's talk about domain-driven design. DDD is a different DD. The hardest thing to remember is that the last D stands for design, not development. As long as you can remember that, you're one up on most people. Domain-driven design is a different set of practices. The point of domain-driven design is that the domain model, the business model, the business understanding drives the design of your system. BDD is about the behaviour of the system, is integrated into the workflow of the development of how we build the system. DDD is that the understanding of the business is central to the way we write the software. We try to understand the business first, perhaps, and then make the software reflect that understanding as much as possible. DDD is a very big field. Like BDD, there's no crisp boundary to it where you say this is in or out of DDD.  Eric Evans had a nice phrase from his DDD book. It tackles complexity by focusing the team's attention on the model of knowledge of the domain. It's something that a lot of development teams shy away from. They don't want to invest time in understanding the business. They want to ignore as many details of the business as possible to keep the domain model simple. The business tell you about all these complex things that we have to deal with. As developers, sometimes we say, I'll build you a form. That will save the stuff in the database.  My model is going to be really simple. I'm just saving some stuff. When you invest time in trying to understand the rules of a business, it really pays off in the software. The software suddenly does stuff for businesses. Instead of when you've got a very weak understanding of the business domain and you build a CRUD system, all of those rules are still there. They're just being enforced manually by people. They're looking at one field and typing it into another place in a different page, that kind of nonsense. You as a developer have to understand the business and build systems that make things easier for the people who use the systems. How do you do that? There's an important concept in DDD called ubiquitous language. It turns out all of these practices come down to communicating better with people. Computers can communicate with each other really well. People communicating with people is always the hard bit. A ubiquitous language is a shared way of talking about domain concepts. It means when I say a particular word to do with the business, everyone else in the team understands what that word means by our kind of telepathy. It's hard to achieve. It means we're all on the same page or using the same terms to talk about things in the software. It reduces the cost of translation when business and developers communicate. You sometimes get a situation where the business person says the invoice doesn't have any line items on it and the developer has to mentally translate that and say what you're saying is that the finance item list aggregate doesn't have any finance items attached to it because I've called it something different in the software to the thing that the real people use to refer to the same concept. That's what cost of translation means. Every time we're having a conversation we're having to map business concepts onto code concepts that are using a different vocabulary and different relationships between things. It's a small cost but it really adds up. We should try and establish and use terms that the business are going to understand. When there's a problem, a modelling problem in the code I can explain it to someone in the business and they'll understand what I'm talking about. That's the idea. I'm not sure what should happen when there are no line items on the invoice. How do we resolve that? The finance expert can tell you because they figured out how to do that in the 18th century and the modelling problem has actually been solved in the real world. You have two lists and you put it in both lists and we call it double ledger. You have to learn all this stuff but then the system is going to really map well onto the business. That's good. Modelling by example is a phrase coined by this guy Everett who works with me at Invika probably a year and a half ago I wanted him to call it BDDDD and he didn't go for it. It's a few things. A few things came together. We were doing BDD and he played a big part in embedding BDD practices into Invika and we were trying to work out how to be as effective as possible using behaviour driven development. We're having these conversations with customers to come up with examples of the system's behaviour. We're putting a lot of time and effort into having these conversations and we started to realise that this DDD concept of ubiquitous language really applied that the requirements conversations went a lot better when you're also trying to understand the complexity of the business. You're not having the conversations as a kind of requirements dump conversation just tell me what you want and I'll build it for you, leave me alone. You're really digging into the business and asking a lot of active questions. In his blog post which blew my mind when I read it here's the core proposition of modelling by example If you put enough effort into this example driven conversation it can also become a venue for doing your domain modelling live with the business you're going to be building the system for in a way they can understand because you're just talking. The best way to understand the domain model is by discussing examples. What's the domain model? Let's cover that briefly. It's a code and it's not a drawing. If it ever works on a project we have to generate UML diagrams for everything, it drives you insane. The domain model is this kind of fuzzy thing in the middle where it's the shared understanding the whole team, the developers, the testers the business of how the system operates. The code is one representation of the domain model your documentation is another representation of the domain model the tests are another representation of the domain model We start looking at the BDD scenario as a representation of your domain model and one of the tools we can use to build a better system and by doing some of our modelling up front iteratively feature by feature by doing some of our modelling by having conversations with the business the code gets easier because we've kind of worked out a lot of the modelling problems before we hit the code. We're going to write scenarios that deliberately capture ubiquitous language The scenarios I showed earlier were fine for doing a lot of BDD and a lot of automated testing afterwards but they weren't really focused on trying to understand how that business talks about things. The scenarios should illustrate real situations so as much as possible get real data in there real detail and then directly drive the way you write the code from the examples A lot of people will generate examples then write the code then use the examples to generate automated tests This is at this kind of feature level We're going to directly drive the code model in a way that's much more similar to test driven development in a much tighter loop That's just about it We now go on to the second half where we show you how to do that kind of thing I'm going to directly drive the code model Just at our interest who's using BeHat Who's using BeHat at a layer other than driving the user interface of your application Just Jacob and a few other people BeHat is one of this family of tools called Cucumber You write out in natural language I'm going to say English from now but you can use other languages You write out in English some examples of the behavior of the system and then you use this tool to drive your real application based on those outlines and the tool says yes or no to whether it works Sounds good and because most people interact with software through a user interface and in our case probably at least 80% of the projects we're talking about are web projects It's very natural to say well this is the way hopefully I've got some kind of layering in my application you may not might be using active record I've got some UI and I've got some sort of core bit if I'm really advanced the core bit might be split out into a few other layers like a domain layer and I've got some infrastructure like databases and queues and things that I have to talk to and because a user interacts through the UI the most natural way to use B-HAT is to use B-HAT to drive your user interface and that's okay Try not to have that user interface detail in your scenarios but this is what most people are doing There are problems though Someone shout out the problems with UI testing Anything else? Changes all the time Bloody front-end developers keep removing my CSS selectors Yeah and it's kind of fragile Things time out and then you run the build again and it's fine the second time around but it's a three hour build so the first time it timed out and then I restarted it and it passed the second time so that's how I spent my day I did a code change waited for it to fail restarted the build, passed felt productive, went home and that's just the nature of it when you're going through the UI you're going through a whole load of stuff you're testing that the web works you're testing that engine X works you're testing that databases retrieve records when you ask them to retrieve records there's a lot of stuff involved so the user interface probably iterates quicker than your business rules do at least for a lot of established businesses finance companies don't change their rules as often as they refresh their UI flow through the user interface and I don't have a slide about it but what's happening is that when we're testing the user interface assuming we've got these nice BDD scenarios they're capturing business cases when you test through the UI what you're doing is you're making sure your user interface supports those business cases in a nice way so there's a kind of pressure a very strong pressure that if your user interface doesn't actually support those actions the tests are going to fail but between the UI and the domain model there's another layer there's an API here there's things that your controller calls in the core of your application and there's nothing that's making sure that that's a good API that supports the business cases there's nothing that's making sure the logic is separated out between these layers so important practice in modelling by example is to test the domain first test the code first make something that works and then hook it up to a user interface that's an interesting concept don't build the whole application without a user interface for six months that would be crazy we're just saying when you're working on a feature get it working in PHP first and then we'll think about the user interface there's still a problem here if I'm testing my domain model with the real infrastructure similar problems it's going to be slow I'm testing it with a real MySQL thing it's going to be slower if I was just testing PHP it's going to be brittle I've worked places where we have an expensive license for the database because someone made a bad decision and the developers can't have a database especially when they're on the train so there's a lot of disadvantages to testing with infrastructure when you look at the kind when you're testing a bunch of business cases and you're using the real infrastructure sometimes you look at what the real database is doing and you're like I've got a thousand tests and in each case it's just doing find by ID I don't need to test a thousand times that find by ID works on this table I'm pretty confident that's going to work I want to test these interesting different business cases work so you can get rid of your infrastructure as well and just test your middle layer I'm going to test the database later but let's not worry about it immediately so in modeling by example you start by writing good scenarios and then you test the core of your application let's talk about the scenarios here's what I started with these are fine for testing it's easy to read them you understand what's happening in each case they're good for verification however when I come and build this system there's not a lot there to help me as a developer I'm going to have to come up with a domain model there isn't really a domain model there there's just inputs and outputs so this is what in modeling by example we're trying to add I want to add some domain modeling I'm going to add some detail and if the detail doesn't map with the business's understanding they're going to notice it because this is in English and they're going to tell me if I'm adding a complicated domain model in my code no one's ever going to look at that and they're not going to tell me that I'm wrong so one thing is to add realistic details there isn't a lot of detail for instance about which flight I'm taking and it just says a flight let's talk a bit about what flights are you can ask questions like sorry I'll just show you this slide a flight from London to Manchester costs £50 background is just like context that's going to apply to all of the scenarios and we're just going to focus on that first one because means it can have a nice big font when I fly from London to Manchester and I pay with cash then I should pay £50 for the flight and I should get £10 tax and I should get £50 points so I've added this detail that flights have a kind of identity that's from London to Manchester this isn't going to affect the automation really it's just deliberately adding these things so when someone who works for airlines reads it they think oh yeah flights to London to Manchester that's like the real world that I operate in I can get engaged in this conversation now the harder thing is to actively seek terms from the domain so it's quite easy to show this to someone and they say yep that's how it should work you have to then drill in and say is this really how you think about it I've made just in writing this I've made a lot of assumptions about the domain model a flight goes between two points and has a cost I don't really like start talking about databases or anything but you can imagine a flight has a cost is that true? are there changes in future where maybe there's multiple costs for the same flight do I want to design it in a way that's going to support those changes I can only what I can't mitigate the future too much but a good way of mitigating the future is for me to build the system in a way that matches your understanding the business thinks of course a flight is over here prices over here and they're completely different things I can come up with a new thing where we have 80 different prices for a particular flight that's going to break my domain model if I didn't match your understanding if my understanding was a flight had a price on it it might be does anyone have that situation where a new business requirement completely breaks your domain model yeah and you say that's going to take six months and we have to delete all the code and start again the business is saying what the hell are you talking about I just want two prices for one flight so you have to ask questions like what do you call this I've used this word flight do you even call it a flight in your industry is there a better word I could use I call it a flight because I look up in the sky and there's a plane and I say look a flight you might have an industry term for it points, paying cash, do you call it cash these might sound like stupid questions but tell me how you think about it is the cost attached to a flight is this thing called tax or is that just the word I used when I wrote it down what you're trying to do is understand how people in that business actually think about these systems you want it to be in a point where the changes they ask for that completely break your domain model are also completely changing their business and they're having to reprint all their marketing material and it's a big change for them as well and they're firing people and you want the easy stuff for the business to be easy stuff in the code you're good at listening you sometimes spot that the business person you're talking to started using a different word to the word they were using yesterday and you say hang on you've started calling it this thing is that the word we should have used from the first place so I might find out a bunch of things a price doesn't belong to a flight a price belongs to a fare and a fare is attached to a route and a route goes between two airports and the price isn't on the flight there's like a listing system where I have to go and ask the fare listing system I want to go from London to Manchester what's the best price today this is a simplification of how airlines actually work by the way I get quoted the cost when I buy the ticket that's an important insight if the price changes later I don't have to pay more because I bought my ticket already and I had a problem where historical data changes completely broke their domain model and all their old invoices had the wrong costs on them I know I have so this is the kind of thing that's going to help you so you might end up with this given a flight XX100 because it turns out they have a way of assigning unique IDs to flights I don't have to have an auto increment there's actually a thing and the flight flies on a route and a route is identified by two airports LHR and MAN it turns out there's a way of having a unique ID for an airport as well so again I don't have to have this crazy mysgrill number and the current listing system and the current listed fare for Heathrow to Manchester is £50 so the whole new concept we didn't talk about before that I just got to with conversation there's some sort of listing system that I have to consult and that's maybe a different system it's not just a table with the flight and the price when I'm issued a ticket so it's when I'm issued a ticket and they call it issuing they don't call it buying because you pay later and I pay £50 cash for the ticket the ticket should be completely paid and the ticket should be worth 50 loyalty points it's the same scenario there's just a lot more detail in there and someone who's reading that is going to understand a lot more about the business in terms of verifying it in a test it's not really added a lot to the outline of the test I'd write but because I'm going to use this to drive my code all of these new concepts I learnt about are going to be reflected in my code so let's write some code we do this first maybe just for one scenario or just one feature we're going to start with the first scenario the job of Behat is to take a scenario written in gherkin and some PHP code that you write and plug them together and it says when this line of the gherkin gets hit run this PHP code that the developer wrote and that's all it does it does it really well but that's kind of all it does so we configure a Behat thing this is just a Yamel file config with Behat this isn't a Behat tutorial so I'm going to go fast through it to give you a flavour of what it feels like to work this way so I say well there's a suite called core and there's a class called flight context and that's going to contain the code that you should execute and then I make a flight's context full of methods that look like this and the methods are annotated with a nice little pattern that says when you see a step in the gherkin that looks like this run this code so I run it I get this kind of output and it stops on this first step and it says you haven't really told me what to do for this first step so you now need to write some PHP so as we go through we just step by step going to write some code that tests whatever the step says and we're going to be driving the PHP directly so we have given steps this is existing context in those steps we're going to create the context with when steps which is like something happens we're going to do something and there are then steps where we check something we're going to check something in PHP so some interesting principles model values as value objects use objects for stuff this is a very good practice you'll find in a lot of domains lots of the logic is about how values are transformed or manipulated there should be objects for that rather than just throwing strings around so as I work through what I'm going to do is read the English step and try and make code that really reflects the terminology used in the English and reflects the concepts being used so I've renamed some of the placeholders here so when there's a step that's the colon is a placeholder when there's a step that's like given a flight with this number flies from this origin to this destination root I'm just reading that and I'm trying to reflect that as closely as possible in my PHP the same thing so there's a thing called a flight I'm going to instantiate a thing called flight there's a thing called flight number I'm going to make a value object that represents a flight number can't be created with invalid flight numbers that sounds handy and there's a thing called a root and the root gets constructed with two airports and the airports are constructed from the strings so loads of the concepts from the conversation have now been dumped into PHP some of this instantiation can be constructed using a BHAT feature called transformations this is just a quick way of doing this casting from strings to value objects I won't go too much into it but it basically simplifies my code and I get BHAT to do some of the value objects transformation so when I run BHAT I get an error because none of these things exist so this is very much like TDD I wrote the code that I want to exist and I'm thinking about that code's API and now I have to write the code and because I like unit tests what do I do first when I need an object where do I start do I just make an object write a test so I'm going to write a unit test for that object so I've got these BHAT tests and making sure the whole thing works together I'm still going to write unit tests for the individual objects and go faster I use PHP spec for example this thing called an airport it can be constructed from LHR and then when you call as code it should return LHR if you're not familiar with PHP spec don't worry about it it's just unit testing it can't be created with an invalid code so if it's constructed from this weird code it should throw an exception during instantiation very very simple test but just checking a few things that we want that class to display and this will compel me to write the airport class it can be constructed from a code it's got a regex for the codes it likes it can turn itself into a string again I should probably put return types on this it's the modern age now once I've done that I can run BHAT again and that step now passes so all of these objects that I I do it for all of the objects all of these concepts that I instantiated in that step now I'll instantiate correctly and I've got objects and I've actually discovered a lot about my domain I've got quite a few values modelled so I go on to the next step the currently listed fare for this route is this amount of money and this is interesting because I have to think am I building this fare listing service or is the one that exists already there's probably one that exists already so sometimes you find boundaries somewhere where you're going to talk to some other system and you just drop in an interface quite commonly it's a database or some sort of repository in this case there's a list of fares and I don't want to worry about it right now so I'm going to make an interface because I'm reading this text the current listed fare so there's something called a current listed fare from a route is this amount so I'm thinking about that terminology that came from a conversation so there's a fare list we're going to call that thing the fare list and I can list a fare into it I would have called it list but that's a key word in PHP so you can't always match exactly the thing you're just trying to get as close as you can for whatever language you're using so you can list a fare between two routes and then because I need an implementation of this remember we're not using real infrastructure I'll just make an object that does that really naively this is an object I'm just going to use in the tests to replace the real infrastructure but it's a fake fare list it just remembers the lists it means I can write my step the currently listed fare between this route is this fare I'm going to insert that into the fare list so I'm just going through this cycle step by step trying to reflect the concepts that are there we're on B-hat and that step now passes green that's complaining about the next one I haven't defined what to do in the next step so I am issued a ticket on a particular flight each step I'm doing a little bit of domain modelling I've done some of my modelling in the conversation there's a thing called a ticket and it's issued on a flight I've verified that with a human who understood the business so I've got some understanding of the domain model but on each step I'm now having to think how do I represent this in PHP what does this look like in PHP so that's a long way of saying you might disagree with some of my design decisions here and it's just a point it's just an example so I think a good way of representing a ticket being issued on a flight is to have a thing that issues tickets and I called it a ticket issuer maybe this is one of those points where I have to go and say, hey, what would you call that thing and I can change the step so once I find out what the business calls that thing I would try and capture that in the step and in the case I would try and capture that in the step and in the case but there's a ticket issuer that needs to know about the fare list and I say please issue me a ticket on this flight again this thing doesn't exist yet a ticket issuer doesn't exist and every time BHAP fails in this kind of fatal error way in fact every time BHAP fails I then try and write a unit test that fails for the same reason so this is a simple spec for a ticket issuer it can issue a ticket for a flight when I ask it to issue a ticket for a flight I should get back something that's a ticket so now it's going to catch it at a lower level of my testing it's going to catch it quicker and that makes me write some code makes me define a classical ticket issuer that issues a ticket on a flight and it just returns a certain just always returns a ticket that costs £100 so until I get to the then step in my scenario I don't have to actually have any correct logic I'm mostly modelling the relationships between things so when you get to the then of the scenario that stuff has to work so run BHAP again now it's complaining it doesn't know what £50 cash for the ticket is the earlier steps normally have more concepts in them so you're generating more objects in the early steps and by the end you're just sort of adding new methods and of course this is the first scenario this is the first scenario in this feature when we come to model the second feature, the second scenario most of these objects will be in place when I pay a certain amount for the ticket I'm just going to okay I wrote the ticket object a minute ago I'm going to add a pay method I call it pay because it says pay in the text I don't have to think as much about what to call it because I thought about what to call it when I was talking to the rest of the team and talking to the business obviously that doesn't work because pay doesn't exist so I make a unit test that says you can pay tickets that's a really bad unit test it's going to get better but for now I just I just want to be able to call this method without anything breaking and this will compel me to write this amazing method and then that step passes so you've seen a couple of times that the domain model is a bit anemic here but there's not a lot of logic that's because only when we get to the then step of the scenario will we have to actually implement a lot of logic and you need to hold yourself back a bit and not try and implement everything before you've got a failing test so we get to the then step the ticket should be completely paid so I have to tell BHAT what to do I say assert that ticket is completely paid returns true this fails because that doesn't exist yet so I write a unit test it covers the same case and also fails it's not completely paid at the start when you pay it it is completely paid and then I have to go and write some code to make that true and typically at this point you find you need to add some extra methods to the other objects for internal calls so the BHAT is interactive from the outside the kind of application API often the objects will then have to talk to each other a little bit so you'll add a few methods to those objects and in each case you're adding unit tests so I have to write some new unit tests for my fare object I have to add some logic to the fare object and then find some more unit tests I need to write for the fare change it and then we get to the point where it works in that everything executes but the logic doesn't work the logic doesn't work because tickets if you remember are hard coded a cost into the tickets so I write a unit test that fails for the same reason when I have a route between Heathrow and Manchester I have a flight on that route and I try and find the fare it's kind of duplicating it and then I write the logic inside my objects to actually pass the tests and then it's all passing the next step the ticket should be worth 50 loyalty points I then have to do the same cycle again the ticket should be worth 50 loyalty points I assert that the number of points it's worth is 50 and then I write a unit test that captures the same thing and another unit test and then I implement the logic in the object and then I've got everything passing so that's the workflow it's very structured which I like because left to my own devices I just read it or something so I like to know what I'm doing next it's also every step driven the code so you'll have seen that the API of the code is really close to the language in the text it's two steps to this chain the language in the text in the gherkin is really close to the way the business think that has to be true and then you have to have driven your objects very closely based on the gherkin and that means your domain model naturally maps the way the business thinks about things so where is the domain model this thing we generated is a domain model now it's something someone can read and they understand a lot of the business concepts on projects where we've put a lot of attention into this we find new members can be on board a lot quicker because we've got these things you read and it's just explaining how the business works so now that I know the objects work I might want to go back and check the user interface works but importantly I may have had 10 different scenarios explaining different pricing situations and different amounts of points and the question you can ask yourself next is do I need to test all of those again through the user interface if I know that the controller just calls that method and I know that the template just shows whatever the return value is if one of those tests works through the user interface probably all of them are going to work or if two are doing significantly different things in the way the UI interacts with the application layer you might need to test two of them but because we've run all of our scenarios through the business objects first and we know the business objects behave correctly I might get away with just running one of the scenarios through the real user interface with the real database the database might be doing fine by ID in each case the user interface might be clicking the same button each time and the only difference is the numbers that are popping up and this is the point at which I build the user interface as well of course so I had a second suite to be hat config I say this is only going to run the scenarios that are tagged with a particular tag called UI I pick one of the scenarios where I think if that one works I'm happy I'm happy all the other UI all the other scenarios will work through the UI and then I'll go through the process and just for that one scenario I'll write a different set of test automation that tests it through a browser by clicking on buttons and stuff so our attention is focused on use cases that helps you understand the core business domain that's one of the biggest advantages because you're getting a better understanding of the business it's going to encourage layered architecture because you're testing the core first the test suite gets a lot faster because the majority of scenarios are being checked through the objects and it's ensuring your objects have an API that corresponds to these use cases use it when it's really important to get a good domain model that's not all the time use it when you're likely to have to support future changes so like how long is this project going to be how many iterations of different types of businesses it's going to support and use it when you can have conversations with shareholders if you can't fix that but don't it doesn't work as well don't use it when it's not core to the business you don't need it for prototypes if you know they're going to die don't use it if this thing's small enough that you could throw it away and rewrite it when the business changes and it's a lot harder if you don't have access to business experts so I've gone over but I'm going to plug some things I organise a meet-up called BDD London on Monday we're doing a workshop on this subject it's at Skills Matter you can find it by just googling BDD London and it's free and Constantine and I are going to facilitate a workshop in improving scenarios JetBrains are getting me to do a webinar next Thursday about PHP spec so you saw a little bit of PHP spec there but we'll do a demo and stuff I worked for Invika and we launched a product called Continuous Pipe so come and talk to us about it and especially find the French guy with a beard Sam who did a lot of the coding and used a lot of these techniques to build Continuous Pipe so it might be interesting and please rate all the talks you see on joinedin it means a lot to the speakers I probably don't have time for questions definitely not oh okay I thought I was being ushered off okay wonderful talk just one very quick question let's imagine I work for a company where the scenario is someone outside of the dev team from accounts or somewhere else goes I want a website, I want catgis on it I want it by Monday how would you advise you move a business that works in that way towards BDD and DDD principles you might not need to if the catgis are achieving the business goals you don't need to have a strong domain model I would start by starting at the project goal level and questioning why we're doing this project so how will we know if this project is a success what we're trying to achieve by this project can you explain how the catgis make life better so walk backwards up to the rationale for the project and then once you get that kind of buy in we're going to measure it maybe we'll have some metrics and see if it worked then that gives you more freedom to suggest other ways to achieve the same change in the metrics if you're starting the conversation with a hard set of requirements it's a lot harder so you need to get more access to the strategic level why are we doing this maybe we can suggest different ways to achieve the same goals and then you'll be able to apply this better that's it that's it I'll be at the Amika stand I'm happy to answer questions


What's up everyone it's your girl Kenzie here from Layton Sports Guards, and I'm ripping a personal here for Steven A We've got a 2021-22 Panini impeccable basketball Hobby box. All right, best of luck Steven. Here we go And we got the bar card, let's go Last good luck we've got our base to 99 we've got Rudy go bear Joss got rookie to 75 Joshua primo for the spurs and Trey young that is a gold one of ten It's Trey way there First hit up for you. We've got rookie impressions auto to 99 Usman Garuba Usman Garuba Then we've got pelicans rookie auto to 99 impressions Trey Murphy the third Pelicans and we've got rockets. How about a Clyde the glide Drexler that is four out of 35 We'll throw back auto there for you Steven Clyde Drexler, and we've got a redemption Good luck, let's see what we got Stainless stars auto so it's gonna be a metal card card number four Heck, yeah way to end that pack we still have your metal bar here Good luck. Here we go. Oh, I see a rookie see a rookie through the little cut there You've got rookie draft three of ten Evan Mobley Another one Steven. What a box of impeccable Barnes and a Mobley on the fine silver draft 2021 that is sick Congrats on that one Steven Fantastic box of impeccable. We'll mag that one for you as well. Thanks as always for happening congrats on that box We'll get them right out to you


Good day, fellow investors. Welcome to the stock market news with a long-term fundamental twist. And we can forget about the trade wars and what we have to think about is what's going on on a basic macro level, politics, etc. And there are two conclusions. Higher taxes and higher inflation over the next years, decades. And that's something that I got from reading. As you know, if you have followed this channel, we are friends with the niche master's fund where Perienster, Dr. Perienster and Peter Barklin manage mostly their own money from a value investing value growth perspective. And they do that. And they also look at the macroeconomics. They have a newsletter that I'm subscribed to. And in this video I want to summarize the newsletter that touches really on those topics, taxes and inflation. Not so hot topics in the news, but extremely important for your investing and everything else. So let's discuss how what's going on in the environment will probably spur higher taxes in the future and higher inflation. So I'll quote a little bit Peter Barklin from his letter. He perfectly defines what MP3 is and he says that modern monetary theory briefly says that Western states can and should continue to print money as they have since 2008. However, instead of using the cash to buy government bonds and other securities, thus increasing government debt levels and driving up asset prices, they should spend them newly printed cash directly on things that enhance the public good, items such as combating climate change, reduction of inequality, etc. Further, instead of monetary easing, which since 2008 has caused inflation in the price of many types of financial assets, government should use dynamic taxation as in spending to regularly adjust fiscal policy to prevailing economic conditions, thus either cooling or stimulating the economy at large.  It has been tried many times before and in many places. Venezuela, Zimbabwe, Argentina, Cuba and the Weimar Republic springs to mind.  And here Peter goes to Ray Dalio's letter saying very insightfully how Ray Dalio is not for or against monetary policy free. He just says that it seems like it is inevitable that after monetary policy one, interest rates, monetary policy two, printing money, buying assets, financial assets, politicians and other policymakers will be tempted to simply print money and give it directly to consumers of the government. Because that seems the easiest way out. So they are not pro or con, it just seems that okay this will be done in the coming years or decade. And that's something we as investors have to really really dig into why it's going to happen and how we can adjust when it comes to investing. Peter also shows a few beautiful charts that I wish to share. So net worth of US households and non-profit organizations jumped more than double from the lows 10 years ago. So this is not because the economy got richer, because productivity is higher. This is just because of financial engineering and because the dollar lost half of its value. So forget about measured inflation. This is because currencies have lost their value, even if inflation doesn't show that. However, this chart shows how nothing much has changed, but the wealth, the perceived wealth has almost doubled. Another beautiful chart confirms the fact that the world isn't getting richer. It's just that there is more money. The number of millionaires in the world has doubled in the last 10 years. And you can see how of course Asia Pacific growing a little bit faster, but North America and Europe stagnating Europe has a much larger number of millionaires than it had 10 years ago. So a lot of financial engineering, a lot of money in the system. And the consequence is that economics aren't really improving. Poor people don't see their lives improving really really good, which means that there might be more monetary easing coming in the future and higher taxes on the other hand. If we go back to inflation, this is the dissection of how inflation has been measured over the last 20 years. And we can see that some things are much more expensive than the overall inflation over the last 20 years, which is at 56%. Hospital services, textbooks, college tuition, childcare, or all more expensive, more than double the inflation rate. Housing is on the inflation rate, food and beverage too, but cars are cheaper, household furnishings, clothing, cell phones, computers, toys, and TVs. Now tell me what's more important in a lifetime for you, housing, healthcare, education, or TV, cell phones, and computers? It's clear that okay inflation, especially the top inflation line is with financial assets, then it goes to other things that are in high demand. And then it's offset by things that lose and lose value where there is a market, there is a natural normal market. And given what's going on, given the distribution of wealth, what Delio has also been saying, there is high likelihood that when the next crisis comes, and that always comes, they will A, print more money, B, tax the richer the rich that got rich thanks to the monetary policy to money printing. So when it comes to that, we can expect inflation, more inflation and higher taxes because taxes are really at bottom slows today. So I will conclude this news video here with these insights that we have to prepare for higher taxes and higher inflation. Taxes are mostly a personal issue. You might move to a country with lower taxes or do something about it. See with your tax advisor and I'll prepare a video for Sunday where we discuss the concepts of investing with inflation for our retirement investment series. So thank you for watching. Just a note in the news for those subscribed to my stock market idea platform. I have published the new idea for this month. The link is in the description below. Thank you for watching and I'll see you in the next video.


Well, this one, that's for sure, okay? Okay. We'll foster, we'll talk about OpenStack. Hey, folks. Thanks for coming to the talk. So that's partially correct. This is a little bit about OpenStack, but it's also about a tool that we've started to develop along with using things like Form and Enansible to massively provision and automate infrastructure. So before I kind of get into how we do all of this and these methodologies that work for us, it'd be helpful for us to explain what I do so you can understand why we've gone the direction we've gone with our tooling. So there's not enough car analogies in open source. There's never been a bad car analogy to explain Linux or open source. It's never happened. So I'm going to give you a pretty terrible race car analogy rather than bore you with some multi-paragraph job description. So I work on the performance and scale team at Red Hat. I'm part of a two-man DevOps team who supports a large amount of internal high-performance infrastructure gear, servers, switches, storage. And our job is basically to, I lied. I said that I wasn't going to explain it that way. Let's go to the race car analogy. I think that works better. So think of high-performance computer servers as race cars. There's different sizes of different shapes, but they're always really quick and they're really fast. And think of high-performance networks as the race tracks in which these cars drive around in endless loops. They try not to crash and burn. And think about race car races, the actual races of the cars running on the tracks as performance and scale testing for various pieces of software. So using this very terrible analogy that I just made up, the race car drivers are basically the performance engineers who would test and vet things like OpenStack, OpenShift, REL, any sort of product that Red Hat or partners might use or upstream products, they would be the race car drivers. So who would not like to be a race car driver? That's a nice profession to glorify yourself in daydream about. And we are simply the pit crew. We're simply the engineers in the pit crew that make all the race tracks work and all the cars on the race tracks operate. And one of our goals is to schedule as many races as we can with all of the resources we have and be as efficient as we can with it. So we developed a tool called Quads, which stands for quick and dirty scheduler. It's not an installer. It doesn't do any provisioning. It's simply kind of a shim or a wedge that works with other ancillary tools to fit some gaps that we've run into in the scale and performance team. So it's important to just kind of talk about what it isn't before we can explain to you what it is and then how pieces like OpenStack can form and fit in. So Quads is not an installer. It's not a provisioning system. It bridges the gaps in several technology areas that for our specific use we find the need for some automation and some tooling. It can use form and is the provisioning, but it doesn't have to. You could backend it with Ironic. You could backend it with any sort of provisioning system that's out there. And it basically just helps us automate all the boring things that we don't want to do. And it helps us also auto-generate documentation in places that we wouldn't want to do it or that we would screw it up as humans. So again, this is just kind of going over what this does. And the whole center core piece of this is there is a YAML-driven scheduling mechanism. So all of the machines and the switches that we manage are part of a YAML structure. And there's metadata or key value pair information about each one of those nodes, like who is the current owner of the machine or what is this machine scheduled to do between this timeframe and that timeframe. So where is it used? Where do we use this quad system that we've put together? We have an internal environment in Red Hat called the Scale Lab. It's about 176 nodes. We're looking to scale up to 500 or 1,000 in the next year or two. And it's really meant for high-performance R&D testing of different products. This wouldn't be where you would go to get hardware if you were an engineer. This would be a place that you already had a use case vetted out.  This is an environment that requires a lot of spinning up and down of hardware, and it accommodates a lot of short-term use cases. So it's not a place where you would rent a server for a couple months, and you get to know it, and you pet it, you get friendly with it. Now, this is basically an Airbnb sort of a model that you only have a subset of servers for about four weeks, and then they're automatically spun down, and they're handed off to somebody else. And again, this is kind of the stuff that it does, how it's designed. But everything's based around a YAML schedule, that when you run quads, it defines the schedule, and you operate out of that. And we call to other tools. So we might call to some tools that go out to the Juniper switches and set a certain VLAN the right way.  So who uses the quads framework, and who takes advantage of this? Right now, just myself and my colleague, who unfortunately can't make it today, we're the only ones that operate it, and the tenants within the scale-ab environment, the back-end usage is completely transparent. The only thing that's facing to them is the documentation that's auto-generated, which we'll get into and how that works. There's another initiative called the Massachusetts Open Cloud. There's some schools, Harvard, Boston University. They have a similar endeavor called Hill. So we've been in talks with them to, there's kind of some overlap between the products, but there's one thing that our system does that might be useful to other people that are doing provisioning, and that's the scheduling aspect and the auto-generation of documentation. If it's not documented, it's not done, and that's unfortunately one of the things that falls through the cracks, and it's also a place where there's a lot of errors, because we're only human. Where Foreman comes in is, Foreman is just a provisioning arm. We don't actually do any provision, and we pass all that off to Foreman. And in a future feature request, we'll actually have Foreman views presented to the user. So if you request a set of 50 machines for three weeks, you'll get IRC notifications, you'll get a couple email notifications, and you'll receive your machines, and then you'll receive a Foreman login that's specific to you. And if you want to, you can log into Foreman and you can reprovision your machines at will if that's something you want to do, but otherwise, you don't worry about it. So what problems are we trying to solve here? There's about four or five main problems that we want to solve. The main one is server hugging. So who's familiar with server hugging? So yes, server hugging, that's right. Server hugging, we want no more server hugging. There's always a greater desire for hardware and resources than there are the actual hardware and resources to give to people, and I think that's a perpetual problem that will never go away. So the best thing that we can do is be as efficient as possible at how we divvy up those resources to other people so everyone can take a turn. So that's kind of the pinpoint of our goal here. This is different than a model where, say, you get allocated budget to buy a couple hundred servers for R&D, 10 or 20, and there's a Wiki page somewhere that has to edit. Someone gets loan hardware for a couple of months, two months turns into six months, six months turns into three years, and that's server hugging. It's your friend, it's your pet, but we want to get away with that model, we want to get away from it by being programmatic about it and being deathly efficient. This is your allocated timeframe. If you want to extend it, let us know, but when that's done, your machines are going to automatically spin down and you're going to get out. So we have clearly defined scheduling for servers, and the main benefit here, too, is with the YAML structure, if we know what development is going to be doing six months down the road or a year down the road, we can pre-schedule every one of these server allocations and network allocations ahead of time. So at 3 a.m., when people are hopefully sleeping, these machines are going to spin down and then spin back up for another purpose. So what are some other things that we're trying to tackle? Less human error comes with automation. So we want to give control over to the machines. Let's let the machines control everything because what's the worst thing that can happen, right? What could possibly go wrong? Siri, run quads, deep provision. Power. Mass, deep provision. Deep provision. Exit one. Exit one. Kill-9. There's still some bugs in the code. So we automate documentation so we don't have to worry about that piece. We have programmatic scheduling for all the provisioning, both on the network and the server side. And all the switch changes are done automatically. So I like Juniper switches, Cisco switches. I like networking just as much as the next guy, but I don't want to have a secondary job as a network engineer. So if it's repeatable, if it's something we can automate, then that's all part of it. What else do we solve besides the display going crazy here? I think the machines are taking that over too. We want to also maximize idle machine cycles. So in any sort of typical R&D environment, any sort of large-scale infrastructure, you're going to have machines that are running and they might not be doing something sometimes. They're Susan B. Scale Engineer probably works Monday through Friday. She's not going to be running tests over the weekend normally. So that hardware is still running in the data center. It's still using electricity. It's still costing money. It still has a carbon footprint. So we want to maximize these idle cycles of machines when they're not being used. And one of the ways that we can do this is again with this kind of programmatic scheduling that we can do in the future, that we can pick machines that aren't actively being used for workloads and spin them down and then spin them back up again to automatically participate in automated testing. And that's done with the YAML-based structure and it maximizes the efficiency of the hardware and basically we get all of our money. Execute! Oh, God! I don't know how that works. Yeah. So there's a couple other challenges that we want to solve. We want to be more like Airbnb and we want to be less like a hobo house where, you know, you have a wiki that someone edits and that's the authoritative source of everything and God help you if you do this on a spreadsheet somewhere. We want to be efficient. We want to have all the scheduling done ahead of time and we want to be as mechanical and surgical as possible. So we have clearly defined operating guidelines on the residency limit. So you can only have an allocation for up to four weeks. After those four weeks are done you need to put in another request. You need to get in a queue and it's going to depend on priority if you get a set of machines or not. So we've talked about the problems that we want to solve, why this set of tools exist and kind of our internal use case for it. So I want to dive a little bit into how it actually works on the back end. So I'm not going to win but this is basically the back end infrastructure at a very simplistic level. At the very top we have Milton from Office Space. Everyone's familiar with Milton with the red stapler? That's right. So we have Milton at the top of all this. He's kind of the user of the environment and we have the quads tool itself which does all the orchestration calls to other services to some provisioning back end. We choose Foreman but you could use something else. It will call to tools to go to the individual switches and reconfigure VLANs. It will go to the IPMI interfaces and if it says it's a new server it will create whatever users are supposed to exist on that IPMI. It will carve out raid disks. It will do all sorts of various stuff. And really the meat of this is the move host component which we'll get into in a little bit when all of this happens on a schedule. So when you have a set of machines and you're given a schedule to work on something, you get your own isolated area, you're guaranteed a certain level of performance.  So in these systems we take that change and what machines are actually doing what and we update the documentation automatically. So there's also a wiki update process that happens. We use the XMLRPC, Python library and WordPress and we basically scrape Formin. We pull all the information out of Formin plus the information that Quads knows about that machine metadata. Who owns the machines? Is there a change? Is there an RT ticket associated with that certain temporal assignment? All that is mung together and then automatically posted on documentation for people to see. And then the last thing that happens is we have a set of applications that are useful for us as kind of shepherds or the pit crew of this environment to see the health of the machine to see what's happening. So we have Ansible Playbooks that will automatically add Nagios checks for the core things we care about. You know, CPU, memory, temperature, fans, disks, things that would not affect performance if we checked, but things that we need to know about to make sure that the overall healthy environment is good. We also add in a Grafana component so that we can graph over time the purport bandwidth usage of each server. So we know how idle it is or we know how busy it is. And this will factor into how we determine which machines are eligible for automatically spinning up over the weekend if they're idle and doing and performing activities. So we talked a little bit about the automatic wiki generation that we want to automate the creation of documentation and not ever have to edit that ourselves. And we also want it to be 100% correct because humanness makes mistakes. And if your documentation is wrong, that can lead to some serious issues across the board. You could wipe out someone else's machines, you could make you difficult to do your job. So having correct up-to-date documentation is absolutely paramount. So we have a pretty simple system. We make foreman the authoritative source for all of the infrastructure, for DNS, for host entries, some of the host level and switch level information is in foreman. So we only have one authoritative source as the true source of knowledge. We have this ancillary metadata that's in the AML substructure. We combine that together and then we turn it into markdown. And then we take that markdown format and we use the Python API and we keep a page up-to-date. So I'm going to kind of go through some examples of what it looks like. So this would be the main page. If you have worked in an environment that has a lot of servers, you typically have some place where you would keep rack and server information. There's a lot of tools to do this. There's RackMonkey, some people use a traditional Wiki. You know, you might use Confluence, whatever works for you. But we chose WordPress because the API is pretty good and it's easy to spin up and it's pretty lightweight. So this is the front page of kind of what would be your racks documentation. So you have your typical information and it's not going to change that much. The serial numbers, the MAC addresses, those aren't going to really change. And by the way, I did not mark out the serial number here on these. And these servers are actually out of support so if you would like to pay for the warranty for these, the serial number is up on the screen. So no one's going to get a one up for seeing that. You have other stuff like basic things but the one thing I want to point out here, this dynamic, which is going to update or it's always correct, is the workload. We see different cloud assignments. Cloud is a generic name for an environment or a subset or grouping of servers that we give to somebody. So that's going to update depending on what that machine is doing. And we also have an owner. So Russell Bryant owns these machines currently or he did at the time of this screenshot. And over time that will update. And the last part is a graph and this is a little outdated because there will be a link here that will go to Grafana which will have collect D statistics on every 10 gig, 100 gig and 40 gig port for that machine so we can see over time what the bandwidth utilization looks like. So drilling down a little more if we were to click on the assignments part of the page, like up here, assignments. You would drill down into kind of an overview of what are all the machines in the environment doing right now. What do they have? And who was the owner and what is there a ticket request associated with the set of machines. So we have an audit trail but everything's kind of linked in one place. The last thing I want to point out here and this pertains to OpenStack is who's familiar with Triple O or Ironic or Director? So we could seriously we have a few bugs as well. So yeah so we auto generate the instack inf .json so you don't need to put that together yourself. That's one extra step you don't have to do when you get handed a set of machines to deploy OpenStack on. This will auto generate this for you. It will actually omit the information for the undercloud and we leave that reprovisionable as you see fit but we go in and we massage the pixie order and disable the pixie on the form interface because Ironic likes to do its own management of pixie, of interface ordering and things like that. So we simply just prep the environment and automate as much as possible so when a new OpenStack environment is presented to somebody say they're doing mass API response time testing or they want to test multi-cell performance there's no end to the number of tests that have been done in the lab but everything is taken care of them so that the only time they spin is actually testing the product. They don't have to do the installation they don't have to munge with a json file and mess with the out-of-band. We try to anticipate and do all of that for people that are using the environment. So if we are going to drill into one of those individual cloud assignments so we'll pick one here, let's say cloud O2 we could go into it and it would also tell us when did the assignment start how much time is left on it and what was the total duration and as people want to see more information we'll hack on this and get it to show more info but the combination of all those things provides a level of transparency to people that are using the hardware you can very easily see what is the total capacity, what are other people doing what is usage looking like and how long do I have on my machines we also auto-generate any faulty system so if you're familiar with Formin there's a concept called host-level parameters and it's basically a piece of metadata that you can set that you can then execute operations against so if we have hardware that's faulty and we don't want it to participate in the larger pool of machines we simply set broken state true if a machine has broken state true it's no longer available for allocation and it actually shows up under a faulty systems category that we can then go back through and either fix or replace something that's wrong with it but we don't waste any time saying oh you can't do your test because the machine's broken we'll just swap it out with another one and then we'll get it fixed when we have time the other thing here is the unassigned systems, anyone can look in and see if there's any free system cool so there's another component to the auto-generation of the documentation which is more of the visual quality so we also generate a calendar view so you could load this into your work calendar you could go to it we happen to just generate this in like a PHP calendar file and an ICS file but at any point in time you can see what's happening in the environment now what happened three months before what's gonna happen three months later so this helps with planning multiple groups when used a subset of hardware that if you know what your testing schedule is if you know what your development schedule is you can schedule all this ahead of time so the hard work and the lifting is already done for you but you can plan your schedule around what's gonna be available there's no guesswork another visualization that we generate is more of like a heat map it will break it down by days of the month and then we have a per month view that we can scale out as far as we want so we can tell what is the usage pattern in the lab look like with one glance we can say oh there's this big gap of five days we have these machines that aren't being used that's an excellent target for automated testing of something so again it's steps to move towards being efficient as possible with scheduling of a large amount of infrastructure so I'm gonna get into some of the commands that quads operates on I don't want to get into too much detail because this is all documented on the github page but right now it's CLI only we have some plans to make it more service-oriented architecture based and have proper push notifications and API and all that but as you can see there's a lot of bugs to work out as you saw earlier and it's quite deadly so we're gonna start slow and we're gonna move up but if you're getting started for the first time okay you would first define the environments that you want to work on so each we just call it a generic name for a cloud but it doesn't actually have to be a cloud it could be jboss related it could be any sort of product scale and performance test related but we call it a cloud so you would define your isolated areas and then you would have network VLANs that correspond to the requirements of each of those work groups and then you would give it a description field and all this is driven by a Python CLI that we have the next thing you would do is define a host and then you would associate that host with any one of these cloud definitions that you created and there's other little commands you can then list what hosts are currently managed by quads the next part and really kind of where it starts to get interesting is when you add schedules to hosts hosts can have an unlimited amount of schedules as far in the future as you think there will be humans on the planet so this is an example command of adding a new schedule to an existing host and we're going to associate it with a certain workload or a certain cloud then you can kind of list the schedule so what I wanted to show here was you know we have five schedules associated with this one machine we have a default environment that when it doesn't have work to do it will actually spin down and power off and its vlan config will be changed and it will move to the available pile that we saw earlier on the wiki right now there's no availability for three months out but if there was nothing to do for this machine even for a couple of hours it would completely power off and move to this other spot to not run to not use electricity to kind of be efficient there but this host here for example C08H21 it's got five schedules we can see the schedules it had before we can see it's currently on schedule five and we can see that schedule five is only about a day so it was some specific test that it was a part of and then the next available one is going to be schedule four which starts the 6th of February and ends the 27th of February so you kind of have this record keeping that's in the metadata for each one of these hosts and that's kind of the crux of how it's managed and then you have summary commands you can list what's currently there and then the actual heavy lifting the actual provisioning the actual going to switches interacting with them changing the configs of the ports that each server is connected to that quads does not do that quads is just a facilitator the move host commands there's a move host flag that you then tie your provisioning back into so we provide scripts to go currently to just talk to juniper switches we also have delforce 10 switches and we're going to be doing the same thing with Cisco but if there's not a vendor that we have supported it'll be added later and one of the things that we want to move to is to actually support my opinion the proper way to do it which would be using open daylight or sdn or something to use a plugin architecture to talk to these resources but for right now we have tooling that goes to the switches and does the changes and this all happens with the move hosts there's some other auditing tools that we ship with quads the most useful one is find available so find available you will feed it the number of servers that you need how many days you need them and then optionally limit to the type of hardware so if you needed 10 servers and you wanted you know a del xd which has a bunch of disks it's usually ideal for sef and you wanted it for 30 days then this tool would go and inspect the metadata of each one of the servers that quads manages and then give you the next available time frame that meet your requirements and then spit out a list of which machines those would be so if you're an operator if you're scheduling these future workloads for people this makes it very easy for you to know what are the target machines that will be available I can just simply enter them in the CLI they will go out and then later on when the time comes they'll just spin up and they'll be passed off to somebody and then again everything is done out of a common configuration file so we have a quad GML file that these are in no way exhaustive these are just a few options that I've kind of handpicked to show for illustration purposes but everything that we tried to be very good about this is that everything is variableized you know if you want to have email notifications that's simply something you can turn on and off IRC notifications if you have SAPI bot or some other IRC bot you can have notifications for events, pump through those and announce on channels things like that so again it's not exhaustive but it's all on the github you can check it out at your leisure so what's currently working right now so the automated system and switch provisioning that's kind of at the heat of it IRC email notifications and we currently use RT for tickets so if there's a need for another ticket system to be supported as long as it has an API or has some way to interact with it that's always a possibility the IP my provisioning is currently in place and the calendar visualization the wiki page stuff that we've shown and talked about that's also available and we just most recently have kind of a full CI sandbox we use Garrett for code review so every patch that would go in runs through basically a quad sandbox where it tries every command it feeds it false information to see if it catches an error and there's always improvements there but before we didn't have any CI and you guys saw what happened earlier what are we working on now what are some new things that might be going into quads that are in progress we'd like to have a web interface on here I'm a big fan of Flask I think it's lightweight it does a great job of kind of doing that but we want to have a Flask interface in front of it and to a certain extent we'd like to get to the point where we can let people provision and schedule their own hardware especially if it's not a shared pool this might be useful for hardware but they want to use this to manage it then they obviously need an easier way or multiple ways to manage it besides a CLI we want to be a little more modular so you have to keep in mind we're sysadmins devops people we are not software engineers by trade so you'll see a lot of this was written from more of an operations perspective and less of a this is the correct etiquette and this is the right style we'll get to that later we want it to work because it saves us time so some of the more constructive feedback we've gotten is that it needs to be more modular that we need to use more of a plugin system one of the bigger places for this is the SDN part instead of having an ad hoc tool for every switchfender that switchfender might actually change their syntax we have to maintain that we would want to leverage some of the other awesome work that's out there in the open stack community for SDN for managing switches specifically open daylight and some of the other initiatives there so that's at the forefront of improvement the form and view integration that I talked about will go in really soon and we want to have a little bit better ironic support as well but right now we do a lot of stuff to make it very easy so when people get past a subset of machines and a network for open stack they just go they run the triple O stuff and it works or optionally we even run the triple O quick start stuff we run the director stuff so they actually have a fully deployed cloud when they get handed the machines they don't even have to wait for introspection they don't have to wait for the installation any of that stuff one of the other things that we've been asked is to support LACP optionally for the switch provisioning 10 gig nicks or 240 gig nicks participate in kind of an LACP bond configuration but then the next tenant might not want that so we need to have flexibility in supporting more network architecture there's just a couple other things that we also want to improve but for the most part it's about six months old and there's only two of us that work on it so we're trucking along that's really basically it now's the time for questions and yes right it's a good question so the question was what is the storage solution where does storage fit into this because we only talked about the compute side and the network side we leave that completely up to the tenants for open stack we typically we have a class of machines that are ideal for Cef they're typically the larger super micro machines with a ton of disks and we will deploy a director based open stack deployment and then install Cef separately and then usually marry the two other occasions we'll use heat to actually deploy a Cef component along with the open stack deployment but we don't do any long term retention of any of the result data unless it's asked for we offer and we run a several node elastic search cluster and so a lot of the data that gets pulled out of the test is directly in search into elastic search and then that's that's moved elsewhere to another location and then that's also safer offline use so we try to capture everything in elastic search and we try to get out of the business of okay this is like VAR data and then I have to remember to back that up before my machines go away I'm gonna lose it so we try to push people to use things like elastic search to push their data out of their environment so if the machines do go away there's nothing that's lost because the results are achieved and stored in real time or it's close to real time as we can get so I did want to give a plug to a performance and scale tool that we also develop on called Browbeat which is it does performance and scale testing of open stack and part of Browbeat is a set up a full elastic search log sash Kibana or elastic search fluent D Kibana stack for you and then has a mechanism with an elastic search driver to push all of your results directly into elastic search and then save it for later if there's significant interest to add you know storage then we could do that but generally it's up to the user to do that and if they request it will obviously help them for like a big disparity in hardware like a wide swath not really I can see where that would be useful but for our use cases especially as it pertains to performance and scale we try to keep uniform specs of a lot of nodes so we might have different classes like this is a compute class or this is going to be heavy on disk or IO so we try to have uniformity across the hardware but we try to have lots of types of uniformity so if there was specific workloads like for ARM or you know something that was not as common and there was enough of demand for it we absolutely would rack that hardware get it in place and get it integrated but one thing we don't do because it's kind of skews test results is you don't generally want to deploy like a multi node application or a stack across machines that have drastically different hardware configuration because it tends to kind of skew the results that you would get back so because at the crux of this is you know managing an environment that's rapidly provisioned all the time this is efficient as possible but the end result of it is to glean performance and scale data out of a random amount of applications or workloads so you could take this concept the meat of this concept is in the automatic scheduling for future things in the generation of the documentation and you could use it to manage a completely different infrastructure it didn't have to be R&D infrastructure it could be you know a rapidly provisioned or growing production infrastructure as we've seen today I would get some of the bugs fixed first but does that answer your question awesome anybody else so it's a great question and I got to answer for it we it works like this but the general idea is that if you have one authoritative source for everything or maybe one or two sources of information that that is the one true place to get it then you can scrape that together and then use something that's API friendly media wiki will do it WordPress will do it there's a ton of other ones that do it we chose WordPress and we're basically I can do the code to do it it's pretty simple but you create a page and the page has an ID so in your API call with the data you basically say post this information but reuse this page ID so it's the same page that people are viewing but as the information changes if there's something to change it will simply just update the parts that have changed and then it saves you the trouble of oh wait you know Bob Smith has these servers this week and Susan doesn't have them anymore I forgot to update the wiki and then everyone that looks at is operating on bad information so it's a fairly simplistic workflow of how it works but but we've had some good success with it yes right well if it's removed from foreman it'll disappear or if it is in a state that we don't want to categorize it so there's the broken hardware if it gets that parameter set for broken state it will not show up and be eligible but it will then show up in the list of broken servers or we have an exclude list so there's certain machines that you are going to have in foreman or whatever your provisioning back in is that they're utility servers that you don't want them to participate but you want them to still be managed so we have an exclude list that we add and those are automatically removed but to answer your question no it's not as efficient as it should be it basically regenerates the entire page and then it copies it in the copy action takes a couple of seconds maybe a couple milliseconds maybe one second max is really quick but the actual regeneration of the current information takes about 20 or 30 minutes but it doesn't copy it in until it's actually got a new updated copy of the page and then it copies it but there's room for improvement everywhere so this is definitely not the model of optimal software development it's what works for us and what we want to share with other people alright I think we're 45 minutes man we're doing pretty good for time anybody know any good jokes anybody well if you have any questions about this let me know I'll be happy to talk to you until you're sick of talking to me and you can check out all the code on redhat.hub under redhat-performance and you can find me at hobo.house that's where I put some information like I have a blog post just on using the documentation generation part of this not using anything else so my hope is that there's elements of this tool or these set of tools that this might not be a drop in replacement for someone else but there's maybe ideas or elements about it that people can reuse and ideally more than that maybe people can contribute back to that's the idea anyway thank you for your time folks