Okay, although I'm not offended if there's still work emails I get it fired up. We're all privileged enough to be here, but work still needs to go on. Okay, we're going to get started. It's officially 1045. This talk for those that took the screen too far away is called Estimates, Expectations and Evolution. My name is Rick Minelius. I'm currently the Chief Product Officer at DRED. Again, as I told the people that got here early, first of all, thank you for coming early, that I've spent a lot of years as a sports coach, track coach, mentor, and I've actually started and run the training programs at the agencies that I've worked at and I really have studied how people learn and how to learn and how to get the best and most outcome to their time investment. You all are spending an hour of your time with me and I can't thank you enough. It means a lot to me and since you're investing time in me, I got to make sure that I'm delivered as much value as I can to you. So I've tried to condense a topic that I've worked on for probably over a decade in the Drupal space and I'm going to try to distill as much as I can in an hour and I really hope to deliver on that. But if this is just information that you absorb without actually apply it in your day to day, it'll just empty out of your RAM over time. So I don't necessarily recommend you take on the totality of what I go over, but I do recommend that you find ways to be thinking about during this presentation what specific actionable step could I take and a problem that I'm facing right now and then apply it and that one nugget will be worth this entire talk. And I actually have people who've come back from previous versions of this presentation, earlier versions that said I didn't maybe understand all the pieces but those two things really transformed my business and transformed how we work together. And the second is I really recommend that you share this information. Teaching is one of the best forms of learning because it forces you to both understand it and internalize it yourself, but it also allows you to own the material because you're going to put it in your own words. And you're also going to have now external accountability because when you share it with other people, they expect you to follow that as well. And then finally, I want to know what isn't clear. I've been doing this sort of internally within the agencies I've worked with, the places I've consulted at. And I'm always trying to make this better and more relevant to people. So there's a survey link tinyurl.com slash Nashville dash estimator. I have business cards up here. If you have any feedback, I would love to talk to you. And again, make this more applicable to everyone in this room. So that way, you found this a very valuable presentation to attend. So a couple quick shout outs, my company that's sponsored me being here and really allowed me to do this work, even though it's no longer my day to day. I'm no longer in the agency space. I work more on the product side. But with DDEV, we still have this philosophy of building websites, not only the technical components of it, but the actual workflow, the management of the whole process from pre-sale all the way to RFP, is a very complex system. And just like we're now trying to do on the tooling side, this presentation is really more on the workflow team management client expectations component. So that's done with a pitch. But how did I get here? I'm not a project manager by training. I've never taken a PNP certification. I've never gone to an Agile Scrum certified whatever. Even though I'm overly educated in some aspects of my life, I am completely uneducated in terms of formal classes and curriculum in this space. I come from an engineering background, where I was trained on how to solve problems in a critical way. And then I sort of got thrown into this space because I started to find out that it wasn't just the technology. It wasn't just like how to build the site, how to hire the best engineers, and how to deliver the best solution. But it was how to understand the client, how to have empathy for them, how to put pieces all together, how to bridge the gap between project managers, developers, stakeholders, business analysts, et cetera. So I've had kind of a windy road even to get to Drupal. But then once I got to Drupal, I moved more from pure engineering to pure operations and starting to work on everything from HR, training, onboarding, et cetera, et cetera. So if what I say seems to come from left field, it's probably because I wasn't, again, trained in the classical sense of project management. But I also have to acknowledge that while I may say things here that may seem like this is my own material or I have some unique spin on it, the reality is we all in this space stand on the shoulders of giants. Open source is all about contributing to and learning from the best of the best. And I can't thank the people on this screen enough. Seth Brown, COO of Lullabot, I met him seven years ago at Drupal Camp, Colorado, and his talk about using spreadsheets and getting high-low estimates from all of his developers and so forth was one of those transformative aha moments. Todd at Four Kitchens, again, Consultancy Scrum, huge kind of like light bulb moment. And Sydney McCourt, her book, still relevant today. I recommend it to any project manager that's dealing with projects over 30K. It is just a phenomenal resource in the community. And then there's people outside, off the island, everything from the goal by Goldrat, all the way down to Scott Birkin, formerly of Microsoft, and David Allen. So if you're a student in this space and you found this talk at any way valuable, I would absolutely recommend finding those authors, finding those people, finding old sessions, videos, et cetera, and study them because they have a lot of really good nuggets of information that I think take this to another level. And finally, this talk is about you. My job here is not to just go over what I've learned along the way. If it doesn't apply to you and if it doesn't make an impact, I have failed you. And I feel like in this space, there's a story, even this DrupalCon where one of the Lullabot individuals, Chris Keyboard Cowboy, who's been doing podcasts for years, still had waited years before he went to Jeffrey McGuire to ask him to join the pre-nout because he felt while he could contribute in code and contribute in podcasts, et cetera, but that he just wasn't good enough or couldn't, had to ask for permission to do something else in a different aspect of contribution. And so I want you to take this material, take what's useful, discard what's not for you, mash it up, make it your own. And next year or two, you be presented on this material. You take, stand on the shoulders of giants, everyone before you, make it your own, make it better. And so even if you don't need permission to use any of this, but I want you to explicitly call that out because it took me several years to get off the sidelines myself because I thought I'll never compare to Seth. I'll never compare to Todd because they've mastered this. They are the people that have mastered this process. It's not true. They are very good at what they do, but they want you to absorb this material and just like we review patches and re-roll them, this is like a PR to the community. So why specifically are you here? Why specifically am I here? Because I've, you know, it's great to join a company that's doing well and that already has everything together, all the processes, et cetera, but not all of us have the luxury of being, you know, land in a job or joining a company that has everything going smoothly, all the processes in place where you just plug in, join it and just crank away. A lot of us in here, we don't wanna necessarily admit it, but we've come from, you know, we've had some fights in the trenches. So I've had the pleasure and the experience of being hired in a turnaround situation, which is, you know, old company, great employees, you know, big brands, but you know, definitely have gone through some rough times. And so, you know, we look at problems as things to avoid, but sometimes the problem is the way. Sometimes that is the path forward to greatness. It is the potential to have to dig deep in yourself and as a team, figure things out and invent new processes, et cetera, that are even better than the ones that you were trained on and the ones that other people have been describing. So I've been working on, you know, larger projects, smaller projects, things where it was very, it would have been very easy to break process, break protocol and just cowboy mode everything. But a lot of what I ended up, I guess packaging and creating here was a result of those hard times and they not only helped us get back out of hard times, but they really helped us excel in good times. So that, again, I feel it was like a very unique experience that others may have in this room, but we don't like to talk about it because no one wants to talk about the failures. No one wants to talk about the 5x budget or the thing that was two years late or the fact that someone had to be let go because it's not just budgets, scope creep, it's like that caused the business and that caused cash flow issues. But it was through those experiences that we really, in my opinion, designed something amazing. And so the end goal of this talk is what I'm, I guess for lack of a word, coining the evolving estimator, and I really like the word evolving and force it to be in that because we are so indoctrinated to get to contracts and the estimate, this fixed moment in time of a number that happened six to nine months ago that we'll refer back to in talk 20, 30 times and we need to break that paradigm. We need to understand that there's a time component, there's a context component to that estimate and we need to have that in our minds from day one and set that expectation. And again, I'm very proud of the websites we've built. I'm proud of the teams I've hired. I'm proud of all those things. But like this work was probably my, the thing I'm most proud of in anything I've done in Drupal even with the PCI compliance paper. And so I'm just really honored to be able to share this with you and I hope it makes an impact to you as it did for me and the people that I've worked with. So I don't read this verbatim but I've had people go to previous versions of this and say that how it was very transformative to their business, specifically when they lost people in certain key sales or biz dev roles. I've actually used this to have clients sell themselves and upsells and budget increases. Said, oh, I see how I'm manipulating things and I see in real time how those are affecting the price and I not want those things. So can you move those into scope? And I realized as I do that the numbers improved to 20K or 30K more, cool. But I'm empowered to work with you to make that decision as opposed to it being dictated at them from the vendor and agency. And so I mean it was phenomenal. So those conversations were less adversarial and more collaborative and also they evolved because when you use a Google spreadsheet with revision history people can actually go back and self-check their own work where they were in time of how that estimate changed, which is pretty fun. But really more importantly you get to these three end goals which is you have to maximize or design a process that works for all stakeholders. So it has to work for the team. Developers are giving scopes and estimates that are realistic. It has to work for the business. We can't just try to define everything up front, spend 100 hours pre-RFP, getting everything tees crossed, eyes dotted, and then lose the business and cost the company a lot of money and all those chances. And you also have to protect the clients because if you have even the best developers and the best project managers and you can deliver the result but you can't do it in a way that actually fits their budget, their timeline, it's still a fail and it still can break the relationship. It still can result in a project failure. So this process in my mind and in my experience has been a best fit. It's not perfect, but it's been a best fit trying to make sure that we're doing right by all three of those stakeholders. So how do we get there? I'm talking about this and I don't want you to look at this like a checklist although you can get very mechanical about this process. I've trained people, junior PMs and sales representatives that were not in the website agency space that came from other industries. So I did try to, for the initial stages I did try to make it easy enough where you could literally fill out some formulas and some equations and get some gut shot reactions or gut shot estimates. But I feel like if you don't have the mindset and the concepts behind them, you won't be able to really work through every use case. So I want to spend a bit of time going through some concepts and some of the mindset things such that, again, you're empowered to use what's useful to you, discard what isn't, but when things aren't working for your specific agency or your specific clientele, you kind of have a sense of like how you can modify it to still achieve the same objective. So this is kind of like where we're going. I'm gonna stop at a later slide and actually go through this. I'm not a graphic designer. I wish I could make this look pretty, but I'm not. But in a sense, again, this is the journey from RFP to delivery and we need to figure out what estimation process is what level of detail, how do we break things apart and how do we re-bring them back together into the development process. So I'm gonna go over a couple of concepts, like things like lenses, things like the cost of customization, things like progressive enhancement that are gonna help understand each of these little funnels like how you go vertically, sorry, how do you go horizontally, how you integrate things vertically and how you kind of get to this map and we can go through it together. So whenever someone says the estimate, I mean it's like a four letter word to me. It just always grows all over everybody, the client, the individual. And you've definitely been in situations, I hope not too many times, but projects gone two years, three years or whatever and then people refer to the estimate back to the contract written in five hours, two years ago by someone that didn't necessarily even fully understand all the detail. And so I keep going back to we can't keep saying it as if it was that fixed thing. So in my mind, I define these things like an estimate was, again, I always say an estimate is a guess is a guess is a guess because it's never perfect, we never know all the information, but it's a guess regarding the level of effort of a task based on the current information at the current moment in time, period end of story. And expectation is a current belief of what will happen in the future based on perceptions of the past, perceptions. You could be using the same words in a conversation and people will come up with completely different perceptions of what was described. And that is important to keep reminding ourselves because we have had developers, project managers, I was crystal clear, I said X, well they didn't hear X. And we need to make sure that we have representations of like that's why we use comps, that's why we use wireframes. We're trying to look at things from different lenses which is a concept I'll go over. And then evolution, again, how the level of effort and perception evolved with time based on new information, new conversations and context. How many of you are familiar with the concept of user-centered design? A few, a few, okay. You're probably doing something similar but it's just a more formal way of describing it. So I use this a few times in the talk so I just wanted to make sure it was covered explicitly. But most people have the concept of a discovery period, a design period. Some people skip, they sort of merge definition in those two. And then there's that development period and whether you're doing agile, waterfall, et cetera. And then at some point there's a training, a migrating, deploy period. So I just wanted to make sure that those concepts were recognized. Okay, so these six things I feel like are just the critical components to understand how we get here. And I've spent a lot of time, again, working with individuals to provide examples and describe all these things. And I hope, again, each one of these will be useful and clear and I will refer back to them when I actually pull up the template and the estimator of how we use that concept to modify and to evolve that. So lenses. I used to use this analogy, because I just loved it, of the three blind men in the elephant. And it's this concept of, if each one of them went and put their hand on one part of an elephant, one had the tusk, one had the trunk, one had the tail, and you asked them to describe the hole. They would all fail because their lens into that experience is touching one aspect of something that's very complex. And I've seen designers, developers, we all have a different lens into a project. Some people, I'm a huge information architecture buff. I love breaking content types, fields, whatever. I mean, that's my jam, that's what I love doing. It makes designers crazy because I'm immediately on a build spec and I'm just off the races because that's what I love. And that's when I see a webpage, I'm really in my mind saying, okay, that's a view, that's a block, there's the list, yada, yada, yada. So we all look at things from the lens of both our experience, our skill set, and just where we spend most of our day. So let's try to not use a web example. Let's try to use other examples of how do we break things down into those subsystems.  There we go, whoops, all right. So there's many aspects of this. The nervous system, the skeletal system, the digestive, the muscle system, and they're not only independent, meaning you can study them individually, but they're interconnected. The respiratory system is what brings oxygen to the bloodstream. And the bloodstream is what transports the fuel to the muscles. And the muscles are what keep the skeleton up, and they're all interrelated. So we can look at things in each system, but then we also have to connect them and make sure that they work together. And the same thing with an RFP, like we get a website, and we get this five to 50 page document, and there's just bullets and maybe a screen grab, and it's like, hey, make this, but for my company. Here's the few things I want. And I use more of the traditional, like MVC, model view controller idea to kind of break things, there's the visuals. What, how does this look? There's the data model, like what's the actual database that powers this? Are we using tables on our side? Are we using Apache solar? Are we going to external systems? How are we structuring the thing together? And then even then, that's not enough. That's like knowing the bones and the muscles of a human body, but not knowing how they interact. The actual signals coming from the nervous system that are moving you and doing things. And that's where user stories are the if then, if this, then that. And each of those lenses has many, many ways of breaking that out. You can break the presentation out to, again, style tiles and mood boards and live prototypes in the browser and a whole host of things. And they only provide you one lens into the project. You have to look at it just like you would study the human body and all the subsystems. You have to look at the website and those individual components. And there's a lot of ways to look at it. We at one point try to inventory like, for a 500K project, how many different ways can we try to look at this and make sure that we find the detail and that we're looking at how they connect together. And this is just one level of it. Not only the project manager's side of like, okay, what's the just budget? What's the number? What's the phases? What's the timeline? But audit, security audits, your comps, style tiles, accessibility, compliance, HIPAAs, PCI, and those all have to be broken down and studied individually and then reconnected at the end. So the trite takeaway of a point of view is a view from a point and each stakeholder, you may be working with a client that's very marketing branding heavy and they don't care about those stupid details with the information architecture or they don't necessarily think through of like the actual user stories of how do I start? How do I go through a flow from start to finish? So it's up to us to sort of have all these tools to be able to look at the site from these lenses when appropriate. We don't have to do this for like a 10K project because this is totally overkill. But you find the number of lenses that makes sense based on the types of features that that client is bringing to the table. And as I stated before, the really important thing is that at the end of that process when you break it all into pieces, we have to figure out a way to bring it all back together to integrate it to the actual website we're building. So this is the concept of progressive enhancement for us. And I haven't found a better word of saying this and I know this means different things to, in terms of like fallbacks and so forth and so on. But to me, I love the quote, I guess the features are cheap, details are expensive. I freaking love Simply Test Me. It is fantastic, right?  Client's like, wow, you showed me that in a minute. So the rest of the site should take three hours, right? Yeah. And they're like, no, that'll take 100 hours minimum, up to thousands of hours.  I have an IBM DB2 connector to a thing that does real-time pricing based on clients and I need it to get inventory updated every 30 seconds. And they're like, huh, yeah, five minutes, no. You are way up, several orders of magnitude of complexity, right? And this is a problem that we face because humans are not really good about thinking in orders of magnitude. You're like, oh yeah, 20% more, double, is that twice as much? Is that three times? No, it's like 30 times as much. So that's not to say we can't get accurate. That's not to say that we can't get within that order of magnitude, but we have to use this concept of like, we see the RFP, we see this high level detail, but now you have to kind of zoom in and you really have to see, is it getting more complex or does it get less complex as you zoom in? So here's a picture, super fuzzy. Any idea what it is? You're amazing. When I see something like, and that's the beauty of this process. Once you've already seen one of those things, you're able, you have experience with what a mixing board is and you're able to discern patterns that others can't see. Because like when you get to this part, it's getting more clear and it looks almost like you could, someone who has more of maybe a experience with textiles or cloth might say, oh, this is maybe a pegboard.  There's not only each of these dials or each of these dials being different things and all of a sudden I went from, you know, is this just a textile of like a carpet? Or is this that much detail behind it? Next one, what's this? Telescope, yeah. So this looks like two suns, right? Or two stars, right? Well, it is from the Hubble Telescope, by the way. Let me go a little more resolution. Anyone know what it is yet? Pluto? And it's a nearest moon. So at that point, further enhancement doesn't really get as much more, right?  So you don't really need to spend more time getting there and as you get in closer, you don't actually get, you know, more detail. It's, you still just get the general shape of the heart and Pluto, et cetera, et cetera. But from a feature, you pretty much have it with some noise. And then this is when I love conspiracy, thirst, delight, you know, the surface on Mars, the face, right? You know, we had an older image and then as we had better resolution, we found out that there really wasn't a face at all. You know, we zoomed in and actually found less detail. Sometimes at lower resolutions, we may see things and say, aha, there's gonna be this thing and it's gonna have all the structure in it. But then when you're pretty up behind the curtain, that's actually not so bad, all right? But in every case, we had to get to some level of detail before we could say, is this gonna be a complex feature or is it gonna be an easy feature? The 10 to 100x, 10,000x cost of customization. I read a blog article about this. I really loved it, the RFP and the GI Joe line item. And I tried to do this in a humorous way, which was, you know, we do websites, right? We have no context of, I mean, some of us may, but I don't really have a context of, you know, the old toy industry and all the collectors and et cetera, et cetera.  And the idea is that small changes in like, you know, purchasing in a GI Joe action figure on Amazon may take you five minutes.  And then, oh, now I want to have it pixel perfect. Okay, now we got to go back to the manufacturing company because I got to get molds exactly for that, you know, height with the same smoothness and still the same, you know, skin tone and everything else. And, oh, now we're going to do an e-commerce site where everyone builds their own GI Joe and scales the size they want.  So the idea here is it's to try to show you that one feature could have literally so many orders of magnitude based in what order you live on depends on what level detail makes sense for the business, makes sense for the website. Is this just because you want it or because it's going to drive an ROI for the site and for the experience of their end users? But we sometimes focus on the details, like we feel like we're already at the right horizon. We're like, oh, I think what he wants is this, the small solution or the big solution. And a lot of what discovery is, is just figuring out are we on the right order of magnitude? And we do this again with e-commerce, right? It's like customer says, I have products to sell. Great. Do you have like hats that you're trying to sell one a month? Or do you have like full blown customized experience with a back end warehouse that you're doing real time inventory management? So the five minutes solution is, yeah, I can copy a PayPal button, drop it on the site. Boom, done, five minutes. Feature check, right? And then you have an hour, okay, maybe you have a few products but you don't really care about branding and you don't really care if they go off site. Okay, you can go Shopify, just turn it on, configure a few things, done, cool. You have an e-commerce, right? And then there's commerce kickstart. I love commerce kickstart.  Maybe you have to pull in a couple of their modules. Maybe you have to tweak a few things. And now you go to the full blown custom commerce experience. I've never really had a success at a custom triple commerce one for less than 100 hours of development time. I mean, it's just, that's where my experience shows me that's where I'm at. And then if I hear the word integration, I'm like, oh, cool, two to 300 hours, somewhere in that ballpark. I'm already going up the tiers, right? And then you have the Omnichaminal channel and I wanna, you know, a lot of customers that submit their own products and I want them to customize the colors and they have this graphic art logo and it's like, oh, okay, cool. Some of that might be off the shelf, right? There might be a module for that, quote unquote. But if there isn't a module for that, then you get back up to those price brackets. So how to blow a budget? We've never done this, right? In my experience, budgets are not blown by being off on one feature 10 or 20%. Or on all the features by 10, you may have a developer that's consistently above or below because they either classically overestimate or classically underestimate. And you can learn that and you can bias for that particular developer or that particular team, right? But, you know, being 20% of our budget sucks, but it's not lethal, right? You can get around that and the next time you can kind of start biasing and buffering for those types of things. But the reality is like, if you have that one feature that 10X is and you had 10 features, you've now doubled the budget, right? So the goal of early stages on the process of estimation is not to get all the granularity of that detail locked in from pre-contract or just after-contract. The goal is to figure out, are we on the right order of magnitude and what are the questions I can give to the client to assess what level they're gonna need so that way I know that we're on, you know, we're not gonna have that one that 2X, 3X is 5X is the whole damn thing because that one feature that I missed or that I didn't ask a clarifying question to see where I was at. So I had to spend a lot of time with the sales team on Smeltest. So API. I mean, again, if it's out of the box known, you know, the Salesforce integration module that I literally turn on and that is absolute, yeah, I could have confidence that's an hour, maybe some configuration, et cetera. But I've had people with documented APIs that their developers are changing daily and using us as testers.  Integrations, again, we now have ambitious Drupal, digital experiences and the Drupal sites are now a part of a greater ecosystem, right? So it's not just this website in isolation, it's that website in isolation with migrations and connected and then there's a marketing campaign and it's all, you know, it has to export data or has to bring in data from other places. So integrations, extensive requirements, if a client is good enough to you to give you a 50 page RFP, they've thought through a lot of things and they're probably even have more detail behind that because they went through that level of detail, there may be other stuff behind the curtain. Those that tend to put less requirements are either less thoughtful about it, which is a problem that you need to ascertain that information through discovery, but it also generally means more flexibility. It's like, yeah, I'd like that, but you know, they tend to be less rigid about those, but those that put very strict requirements tend to not be as flexible and they put them there for a reason in my experience.  PCI compliance, HIPAA compliance, all those things. When you start getting to standards, there tends to be a lot of care and a lot of focus and knowing like, do they just want it because they want it or are they gonna have like a third party audit scan and you're gonna have to be that and you're gonna have to keep that in mind and check the entire process. So I have an article on this. If you reach out to me, I'll give you the whole Smeltest article. It's pretty extensive. And also know your biases. You know, customers may overstate their needs. Again, you know, I don't need the PayPal button. I need the full blown experience. Okay, the 10 hats at $10 a month, you know, you're gonna have to sell this for 50 years to recoup that 100 hours custom development. Do you want to do that? No, okay, PayPal. But also developers. I mean, we, I love building new stuff, right? It's so fun. But it can also be super costly because it's like, oh, this is just this one tweak. No, that one tweak now put you in a different order of magnitude. Now it's a custom integration. Now it's this thing. So that developer that meant, well, didn't know where that trip point was. And all of a sudden that two hour task is now 20 hours, 30 hours, et cetera. And then vendors just may be overconfident. I mean, one of the things, I loved that the gentleman that was able to call the mixing board, right? Because have you go through those experiences of seeing the low resolution, the high resolution, you start getting this like, almost like artificial intelligence. You start being able to sense little things that I could see that pattern. And I know below that surface is that thing. And others just can't see it because they haven't actually gone through that particular use case before. So they just lack the visual cues that know where to get to that one feature. And that's where having multiple eyeballs on a project is very valuable, bringing your designer, bringing your developer, because they're all gonna look at it from different lenses and they're all gonna have different abilities to kind of see those visual cues and say, ah, that's a big thing underneath there. Let's go investigate that further. I put this just here for completeness. Hopefully most people are aware of the, just the interplay of features, budget timeline. It's fairly known, but it's just something to keep in mind as we're going through this budget template at the end, how to know that, how they're all interconnected and how they affect how we modify and go through the estimation process. Consultancy Scrum, I loved this slide that Todd presented. And it's all about, we get in this mindset of I gotta do it, I gotta adopt this methodology completely and totally through the whole process. And there's the waterfall approach, which is like, hey, let's not sign a contract we understand all the detail. Let's not start building everything, we cross T's, we dot I's and so forth. And we try to avoid risk by over planning. And that kind of works, right?  But then you get in situations like I was where a year and a half into wire framing and 214 signed and approved wire frames later, you start to develop, you start to design and the design changes the wire frames. What do you do? You can never avoid all the things that come up when people start seeing things farther down the pipe. So you want to find enough resolution, right? And we're just like, we'll just start building value, like we'll just start running sprints and we'll just get there, right? But the problem is there's choice points on the road that you may start painting yourself in the corner on a certain technology choice where my doing is multi-domain or am I using Apache slower or not?  So you're trying to avoid budget overhead by building rapidly, but ways could occur when you went down a completely wrong path that you could have seen if you got a better view up. So, and then Kanban, like once you've actually got through the project, now it's like per request and you pull through the system and it's very, but you sort of force the work into this very homogenized task-based system which doesn't work for a new project. So, consultancy scrum was amazing to me because it matches the process with a level of risk. Like what are you, what lens are you looking to, what level of progressive enhancement are you looking for in order to mitigate risk appropriately? So in waterfall or discovery, you're trying to do waterfall because you're trying to find those interconnections and making sure you're getting the right resolution. But once you have that, then if you're off 10 or 20% or then you could pivot like the individual features but you know the big systems and you know sort of their touch points and where you're going on those touch points but you have some wiggle room in between them. You find the right order of magnitude but now if they want them red or blue or you want to tweak it around or it's just a little bit of extra customization there, then you have that flexibility and that's where agile is perfect because you can get the exact acceptance criteria. Now if they try to then break that order of magnitude back, we did discovery to kind of know that we're not gonna go that level, we're gonna stay on this level. We'll deliver you a great feature at the 100 hour commerce level, not the full on e-commerce ERP with back end warehouse, like that was it there. And if we cross that, then we go back to that 10x or 100x cost. Make sense? I'll touch on it a little bit more on that final graph. So again, bringing it all together. So lenses, you know, websites are complex, period. They are just difficult things. We need to look at them from every angle, from every vantage point so we can evaluate individually and then how they're gonna play together. We don't wanna just look at the design only. We don't wanna look just at the information architecture. We don't wanna just look at, you know, all these just user stories. We need to basically say how do they all connect, right? And we need to use progressive enhancements. We need to look through each lens to the point where we feel comfortable that there's not gonna be scary levels of detail that crosses into these massive thresholds and massive spikes, which is the cost of custom. Drupal provides so much out of the box. Use more Drupal core, use more contrib with default settings or with configuration. But the second you get into custom code or other things, now you need more experienced developers. Now you have code that you have to maintain, that you own. We have constraints, you know, features, timeline, and budgets are interdependent. You can't move one without the other or others. And consultancy scrum. The process that fits risk with expectation. So how do we solve this puzzle? I mean, it's not. And I feel like I have a one solution or a solution to this problem and it may work for a lot of you, but I wanna at least walk you through the process. So the estimator for me was, I mean, really, when you boil it down, it's a Google sheet with revision history, used to create and evolve detail level of effort and dollars from project inception to launch. That's what it is. You could do this. You probably do it in a way or you have sort of an ad hoc or informal way of doing this. Back to sort of the first few slides when I mentioned being in a company where time was money and Baxware gets along and we had a lot of challenges, being able to rapidly assess RFPs sometimes in the order of 30 minutes and just come up with gut shots of, is this even like on the right order of magnitude and then a couple hours pre-sale, again, right order of magnitude. So rather than having nothing, I had to create something that allowed me to work all the way from that gut shot through something that could literally evolve a single document through the entire process and get to the end where I can use it for a post mortem with like actual recorded features and time spent versus prediction and so forth and so on. So again, we have the RFP on the left, right? And mostly gonna be bullet points with description of features, description of interaction, maybe a couple screenshots, right? And at the top we have, okay, sort of what level of estimate do we want? And at the bottom we have sort of the user center design stages, meaning like discovery, design, definition, definition again, meaning more like user stories, requirements, detailed builds back, et cetera. And then each of those sort of channels we're breaking things up into like the visuals, the presentation at the very top. We're breaking it down to like data models and information architecture in the middle and that business logic, what are the features categories? Okay, cool, what are the assumptions behind this? What are the questions behind this? Okay, cool, next level, what are the acceptance criteria? And we're trying to move those along in parallel because there are different lenses of it, but there are also different levels of detail that we're trying to assess at each stage of the process. And again, we don't want to spend so much time that we end up burning 50, 60% of the budget just planning, so we have to have an acceptable amount of risk at each step that we're saying we got an 80 to 90% one, we feel confident enough that when we go there's not gonna be a landmine at the next step, all right? So at the very top, when you get an RFP and you're in discovery it's like I started assessing things in level of features, meaning I don't need to get to full blown acceptance criteria, but I need to know have I properly identified everything? Is it 15 features, is it 30, can I go into the client and say I think I heard these 15 things? Am I missing something? Or is that still important? Is it really 10? Again, sitemap, have we caught all the old pages that you had, have we caught all the old pages that you anticipate having? Wireframes, here's what we think the different sites or pages are gonna look like in the sort of boxes that are gonna be there and in certain features that might be in those pages. Again, are we catching everything? Are we missing something? And then again, each step you just refine it along the way, again knowing that you're gonna converge it at the end, back to done. So again, I want you to have this sort of in mind as we're going through the template itself. So I'm gonna go through, again I said I was gonna, not to do this mechanically, but I am gonna go through the mechanics, so this would be a typical 30K to 300K budget. So if you're at like a million dollar account, you may have to spend a lot more time at the pre-sales process. If you're below 30, a lot of this could be overkill, but it does have a wide delta. But when I have the estimator template up and I had an RFP and said, okay, you have two hours. Tell me if this is worth going out to or not. So I had a set of questions at the top and my objective was just, do I have the feature label correct? Do I have a one to two cents description and start putting my questions? They say e-commerce, do they mean, just what kind of products might they have? Do they have a lot of variation product? So some questions to ask for discovery.  And because we know things vary by orders of magnitude, we don't just say, well, it can be four to six hours. No, it can be four to 40 hours. Because if they cross that threshold again, so we need to be very accurate and not just try to make the client happy by seeing a smaller delta, but saying, we don't know. This one feature could blow up. Let's work through it together and when I ask these questions, I'm gonna know if it's really up close to the 40 or if it comes back closer to the four. And again, get a sense of budget ranges.  This is not for us. And you can walk away from it or if it's too low. So the goal really is this, is it even possible in the sort of sweet spot that you're looking for to deliver a site of this value? And if you wanna be in advance, if you wanna get ahead of the game, if you wanna sort of like, if time or budget within the company is allowed, yeah, you can start splitting out the features and the phases and everything else, which I'll show you. Okay, I'm gonna start putting on the gas because I wanna get to the template in the next five minutes. Okay, so in discovery, you update features, you get the description a little bit tighter, you address initial assumptions, you may even draft out initial acceptance criteria, great. This is the key, this red, yellow, green, which I'll show you in the estimator, which is you want to highlight in the estimate what is mandatory, what is it gonna provide high ROI and what can you guarantee, quote unquote, within about an 80% of the budget? Like if a client has a $100,000 budget, you're like, I know based on what we just discussed that this 80%, about 80K worth of things, even if things go wrong, but based on those 10 to 20% deltas within the order of magnitude, I'm very confident we're gonna get those to you. And then you put things in yellow, which is like, these are high value items, we really wanna get to them and if we just crush it and everything goes smooth and there's no scope creep and there's no problems, we'll get those in as well. So let's work together to make sure we get as much of those yellows in there, right? And then red, here are the things you asked us for, we think they're massively intense, they probably don't provide you business value, we documented it, we heard you, but unless there's a significant increase in budget, these things are red flags for us and we potentially won't even take on the project because we know that it's gonna push us way, way above the budget. And then you work with a client to collaborate on this, literally meet with the client side by side. Let's walk through, did we miscategorize something in green when it should be yellow? It's not as important to you, so that way you get something else in yellow into green and I'll walk you through that. Design, you're really just kind of confirming previous assumptions, so if new features are surfaced when you start actually getting the wires in comps, you'll pull those into new line items. Again, you'll just meet with a client at the end of the design phase, say, okay, how do we do? Do we need to move things around, re-categorize things? Finally, definition phase, we do deep dive, I mean all the developers, technical project architect, solutions architect, we're going through fine-tooth comb, we're making sure again there's no land mines and again we lock in actual time and we start looking at new features differentiated and reviewing, et cetera, et cetera. Now you go into development, you go into agile, you start building, you go on your sprints and you can still, if new features keep coming up, keep popping them in the estimator, color them differently so you can differentiate them so you can show how new things were added and they changed and you can re-categorize if you look like you're going to run out of budget. And then finally get to your post mortem. So let's take a moment and just kind of look. Again, I am not a designer and I know this is probably not zoomed in enough but by the way, after this talk I will, if I have business cards reach out to me, I have an article which is going to go into depth about how to work through this mechanically as well as the actual template itself, the open source people will be able to download, you can go through the equations. So don't feel like you have to catch everything right here, I'll do my best to review it but back to that whole, again, RFP lands in your desk. Answer questions, what's your hour of the day of your company? How many discovery meetings you have? How many categories of things are being discussed? Is it super complex? Do you feel like you're going to have to bring multiple people in discovery or is one developer able to do the assessment or one business analyst or one solutions architect? How many hours do you think your meetings are going to be? Do you already get a sense of super complexity? Is this going to be a very design heavy, brandy and heavy one and those equations just drive different high-low features, right? So we have different categories, like here's when I think my discovery, my kickoff meetings and so forth are going to be, I want to get some initial scoping, design the wireframes based on the number of screens and so forth, again, wild ass guesses, just getting numbers on the page and those are things that are going to cost money and we're absolutely have to do them so they're in green. Okay, I start seeing things like sales force integration and e-commerce, cool, e-commerce is the whole point of this project, it was an e-commerce website, cool. So is this a 10 hour e-commerce or 100? I don't know, put it in the line item but it's definitely green because it's definitely mandatory because it's the whole project. So I green light it, I put a one, I see some other things in there, recurring billing, I don't know if they really need it but I'm going to put it in there again. Am I just turning on the recurring billing framework? Am I just, you know, or am I doing some more customization? Am I just going to offload it to Stripe, whatever and again, my goal is just capture, capture, capture, capture, capture, capture. This thing is pretty sophisticated in the sense that I have this one, two, three item so in real time you can get the high lows being calculated, the averages are being updated. So like as I'm saying, oh, you know, I'm working with my PM or working with my sales rep and I'm saying, well, that rating system is actually really important so I should put it as a one. It's all calculated in real time so I can see like I'm adjusting, I can move the colors around and the goal for me is like just rapidly changing this to see, you know, how can I get to that 50K budget that they're talking about? And then if I cross that budget, the worst case, so I'm looking at the high, so the high for the green, so everything I'm just saying is in phase guaranteed is already 17K over budget. Okay, I got to start slicing more. I got to start figuring a way to reduce risk and so those are the questions that I'm going to start putting into this column over here so that way when I start meeting the client, I'm like, okay, I think you mean this so are we closer to the two or 20 hours so you have to ask the right questions that get to that order of magnitude and once you get through the first phase, once you get through discovery, those things no longer become equations, they become the real numbers, they become what was actually spent on that project so then this number up here, the target budget as well as the current budget are updated in real time per phase and then once you get the descriptions and the acceptance criteria written out, you can then start recording your actual time spent over here. So if you use JIRA or whatever time logging thing you're using, you can just have links to those reports, update the reports and then you're going to actually see what you're working on, how good or bad you are on or off course and then you're going to get any updated numbers and at each sprint, you could literally go back to the client and say, look, all those things in green are actually on course or even better. Let's maybe change some things from yellow to green. Let's work with you to get more of those in there. Thank you for working with us and not scope creeping some things and sticking to this. So now we're able to get more for you or these things are adding new features or adding more material and now we need to take things in green, put them as number two is get them out of the project and move forward. And then sometimes you'll find things like, let me get over here, like back in fulfillment. I mean, again, are we integrating with something or do we have to have some back in ERP? Am I just turning on modules like, just to have stock inventory and order management or order fulfillment, shipping, et cetera, or do I have to integrate with Magento or have to integrate with something else? And again, that's where those questions come into play and if you later found out like, again, back to as you zoom in, you find that the client even didn't want it to begin with, you can actually change it as a number four, gray it out and then it removes itself from all the calculations. But here's something that's very interesting to me, which is you can see that if I really focused on what delivers value to the client, high value, mandatory parts of the process and things that I know have tight deltas and aren't spiking over to one, two, three hundred hours, I can deliver this project for 57, but those three features that were in red that have those very high gamuts could easily double the budget right there. And so you're trying to work with the client to say, I don't want to see that happen to you. I know that 50K that you have is you're all in budget. So we just literally can't go down that path without us both taking an immense amount of risk. And if we take on that one feature in red, we're gonna push out all those other features in green. You're literally gonna have that one custom feature and it's gonna absorb everything else. So again, it's a lot of working with the client, not against them in an adversarial way. Do you actually show them that? Yes, I work with them because, again, it's websites for a lot of people are like voodoo. They're like, why did that, why did that commerce kickstart thing take one minute? And why are you telling me like a slight variation of my theme is a hundred hour? Like what? Like they just can't see that. But if you say like, look, and here's another trick I've used, which is if there's one level that's a hundred hours and one that's 10 hours, I will actually put it as two line on us. And I'll green light the one, I was like, this is what I wanna deliver you because I know I can get it within your budget for 10 hours. But that sort of next level one is a hundred hours and I'll red line it and make it a hundred. And so if you really want it, we could do that. But you see like, I can deliver that feature for 10. You're asking me for a hundred, but we're in this, we just can't make this work, we're too far apart. And clients don't like that, but they appreciate and respect that because they know that you're not just trying to, you know, trying to burn them or you're not just trying to pad your budget. You're saying, absolutely. And every feature can be done in like a high medium low back to the whole PayPal button through full blown e-commerce, whatever. So here's some cool rules of thumb I've learned. I don't know why, but discovery 15% is like the magic number. If you're going too high, like 25% planning, you're too far in waterfall and you're not gonna, you've already sort of de-risked the project as much as you possibly can. In my experience, my opinion, super big enterprise projects, you know, one, two, three million, maybe different, but in that sweet spot I was talking about, this tends to work. The project management is typically 20% of the budget. Like it just happens. And here's what happens to a lot of projects, especially sales. Oh, there's a hundred hours of development. Okay, cool. We have 120 or a hundred hours of budget. Perfect. But they're not accounting for all those overheads. In fact, when you really get down to it, when you slice out all the sort of the de-risk in the beginning and the project in QAL in the end and the training and the deployment at the end, the actual time spent developing first is about 40% of the budget. And that's a good number to keep in mind when you are assessing something because we get so focused on how much time do I have to develop?  Cool. And we try to pack in a hundred hours of development into that thing, not accounting for those other buffers. So having a formula like this actually is good for accountability within the team because it's having a realistic assessment of how much project management time, QA, code reviews and all that's gonna take. And if you consciously slice it out, you actually have to then consciously remove it and say, I'm gonna move the PM time down to 5%. I'm gonna put it on the developer. Well, then you're gonna have less time developing or they're gonna spend more time essentially pseudo-PMing. So it was actually good for more internal stakeholders, especially sales and biz dev, to have those parametrizes like, no, this is what's necessary to do this. And the equations have numbers in them, so you can modify them as needed. I did have a skill factor for that. That's why I said it like in one of the equations you'll see is like, is this heavy branding or light branding? Yeah, that typically falls in, again, your mileage may vary 10 to 15%, I mean, it depends. Yeah, so, but these ones I feel are universal. Like they are just numbers that, if I see the discovery too low, I'm like, shoot, we're not spending enough time to like go through and look at things in the lenses. And this is comfortable because then developers who are like, I need to have realistic development things and sales and everyone else, sort of this number is kind of like where the best balance of those stakeholders lives. Good question, middle third of the budget, meaning post-discovery, pre-deployment. So those of the five stages, the design, definition, and development. So again, this talk is my pull request where I just need to review by community. I have worked with several people on this. I feel like it's legitimate. It works more for just one or two companies. And I feel like it does provide different ways of attacking the problem that others haven't attacked it in this exact way, but I'm not even in the agency space. I mean, I've moved on to Trud, I work on D-Dev, I'm more on the product side, but this talk was like my thesis. Like I really spent so many years in the trenches and I believe in this material so much that I wanted to make sure that I had one last shot to get it out to the community and I'm not there to take this further at this point, but I want to make sure it's done right and I want to make sure it's actually validated. So I would love feedback. I've had several people ask me if I could turn this into more of a deeper training course like mechanically walk through a couple RFPs and really actually work through filling out the template and so forth. I've run trainings internally for project managers to adopt this. So I would love your feedback. I have cards up front. I have a tiny URL, one tinyurl.com slash Nashville dash estimator. I plan on filling, finishing out the article with the template and if you sign up in any one of those formats or get in touch with me I will absolutely make sure you get a copy of those and I just love this stuff. I love coaching, mentoring, teaching, all this stuff. So I'm more than happy to in my limited but time to sit down with people because I mean this stuff really brings me joy because I've been the developer, I've been the project manager, I've been the count rep, I've been on all sides of this and when it's not done right it hurts. It hurts a lot. It hurts the team, they're burned out, you get these death marches, it hurts the client. You now all of a sudden are fighting instead of building. It hurts the business because you may lose enough money where you have to let someone go and that affects someone's life. I mean so this is a really important topic and having experienced the pain of not doing it right and having experienced the joy of when it feels like it's in such a good flow, I just love this topic. And so even though it's not my day to day anymore I'm more than happy to have conversations on this and I would love any and all feedback to make this better, make it more accessible, make it more understandable. So thank you all for investing your time to be with me. I hope that I gave you something actionable, something shareable and have sort of opened your mind to maybe a different way of looking at this in a way that you find valuable for you and your company. Thank you. Yeah, absolutely. How do you are targeting your budget to be?  You just have to find like what, you know, based on what you've seen, what you think your budget is. And yet, it's such that you wish they were just telling you what it was, but that's where you just have to take a shot in the dark and just go for it. You know, and here's another way you can frame it. You can say, I can better figure out what order of matters we can deliver on if I even know what range you're in. And because I don't wanna give you an estimate that's like two or three orders.


Yksi ensimmäinen asia, johon olemme käyttäneet multivallista dataa, on se, johon lopulta yksinkertaisuissa. Olemme myös kokeilla entri-klaista korrelatiosta, jotta nämä variaat olemme käyttäneet. Lopulta katsotaan esim. yksinkertaisuista variaatioita ja yksinkertaisuista variaatioita. Olemme täällä yksinkertaisuista dataa. Olemme 5 yksinkertaisuissa, ja yksinkertaisuissa yksinkertaisuissa. Mutta ajattelisiin suunnastamme yksinkertaisuvuus on koko 14 illsipuolelainen. Yksinkertaisuissa avilaisuudestaan on hyvä vualia, se on myös yksinkertaisuista. Yksinkertaisuissa ei ole silloin täydellisi, niin on sitä fundauksia veristä. Mutta se ei ole oikein yksinkertaisuista. diezotuja, yksinkertaisuiden profeetavuus seurataan veritus bunta. Varmista meidät alueet avilaisuja. Varmista mumsistolta on company-level-variations. These red mounts each present one company, and they are all working within an industry. And this blue area here represents the variation of the performance of all companies within that industry. So we can see that different companies vary. Their performance vary within company, but there are also variations between companies, so that this company here is consistently less profitable than this company here. So we have two levels, we have the within company level, and we have the between company level, which is also the within industry level. We can also add more levels. There is no limit on how many levels we can do, but let's go for an industry level. So we have these blue, five different industries, and the industries are different in their profitability. Some industries are highly profitable, others are not so. And we can see that the individual variation of the data here is a function of these three sources of variation. The between industry level, the between company level, and the year-to-year variation within companies. To understand our data and understand the phenomenon that the data represents, we typically need to decompose the variance, to somehow come up with percentages or some other statistics that quantify how much of the variation is here, and how much of the variation is here in our data. If our data is this small, we typically start with a graphical analysis. So we can just upload the data. This is 25 observations, five observations for each company, for five companies within one industry. We can see that there are some patterns. For example, this company, there is not much variation in performance. This company is less profitable than that company and so on. This kind of analysis works well when you have a small set of observations. If we have a large number of observations, but still a fairly manageable number of clusters, let's say up to 30 companies or 30 industries or whatever is our level to unit, we can use box plots. Box plots are graphical representations of variation of individual variables, and we can do box plots by groups. The idea of a box plot is that we first calculate for a variable, calculate the median, and the median gets this thick line here, that marks the median. Then we calculate the first quartile and the third quartile of the data. So quartile means that below this line lies 25% of our observations and above the line 75%. Median is half and half, and third quartile is 75% and 25%. We draw a box between the first quartile and the third quartile, and half of our data is within this box. Then we have these whiskers that indicate the minimum and maximum, and sometimes we also have outliers that the box plot algorithm identifies as circles. So why is this box plot presentation useful and how can we analyze the box plots? We can, first of all, start to understand the between and within variance by looking at the box plots. We can compare these medians, or we can do box plot with means, and we can check how much variation there is between these means or medians, and that is our between variation. We can also take a look at how high the boxes are, and that quantifies the within variance. And comparing these two dimensions tell us if the variation in this variable is more due to the differences between firms or is it just random variation or some other variation within firms. So is it a within firm or between variation that expands the data? We can quantify the level of variation between two levels also numerically by calculating the within variance and between variance. This is our data, and we start by calculating group means. So we take each of these companies and we calculate the mean of this. So these are the group means or cluster means for these five firms. And we check how much these means vary. The variation is quantified here with this statistic. And then we calculate how much these individual observations vary from the group mean. In practice, we do group mean centering. So we take each of these observations, we subtract the group mean, and that gives us the group mean centred values. Then we calculate how much the group mean centred data varies. And this is our between variation. This is our within variation. And this is our total variation, which is the sum of the between variation and the within variation. So the variation or variance is a statistic that depends on the scale. It will be useful to have a scale free way to explain on which level the data varies. And this is where the intra class correlation comes to play. So intra class correlation is simply calculated as variance between groups divided by the total variance. So it answers the question how much of the various data is attributed to the groups and how much is attributed to the variation within the groups. This is called ICC1 for reason that there are many other kinds of intra class correlations. So intra class correlation generally refers to correlation between observations. And because there are many, this is called the ICC1. There are like a few others, but this is the most important one that you need to understand when you work with multilevel data. Other intra class correlations are mostly about reliability of multiple raters, but ICC1 is this simple equation that simply quantifies various. When ICC1 is zero, then that indicates that there is no variance between groups. So the box plots are all on the same level here. There are no difference between means, and in this case the medias are close as well. And all variation is simply because there is variation within these groups. Then when intra class correlation is one, then that means there is no variance within clusters at all. All observations equal the level two means. So this firms profitability is always here, this firms always here and so on. So there is no within unit variation. Why do people calculate intra class correlation and how it is typically reported? The role of intra class correlation, the first role is to make a decision whether something needs to be done for the cluster. If all the observations within clusters are the same, then you can just pick one observation for each cluster and use those in regression analysis. And it doesn't really matter that you have the remaining observations because they don't provide you any more data. If ICC1 is zero, then there is no clustering in that variable. And if all your ICC1s are very low, then there is no meaningful clustering in your data and it's possibly safe to go without a multi-level modeling. There are exceptions to the rule, but generally when ICC1 is close to zero or when ICC1 is close to one, then a multi-level modeling may not be needed. But if it's somewhere between, like it's 50%, then you typically need to take levels into account in your analysis somehow. Let's take a look at the example of how ICC1 has been reported in published research. So this comes from HouseNet's paper and this is a good example because they first explain what the statistic is. So quite often people just report a statistic or report a number without explaining what ICC1 is. And this study provides this concise description. ICC1 values can be interpreted as the total amount of variance in the dependent variable that is attributable to between unit rather than within unit differences over time. So that explains what the statistics interpretation is and also if the values are high, then regression analysis could be inappropriate. And then you would have to do something else or, for example, use cluster over standard errors or multi-level modeling.  So giving this short introduction to your statistics is very useful for your readers because your readers may not be experts in using multi-level data. So make it easier for them.


up to navigate yourself. I mean, they're a pretty good team, you know. But if you make pitches, I think you can go out there, you know, and do a good job. What happened there with the pitch come in the third inning? I didn't work. I don't know what happened there. I was working good, and then, you know, I got loud, and they said, you know, after that, it didn't work because the next inning, we had to change it. Can you let me give me his? Yeah, I was surprised for you to see Tucker go in that situation. I was happy. Yeah, I mean, you know, I was trying to get a base there, but I was a little worried that when he was halfway, I got no chance that he could be safe. But, you know, I got to, I don't know how to score that, pick up or whatever, but good for me. Did you see him go, or did someone, did you hear someone? No, I saw him. I saw him when he was like halfway, but I know nobody's faster than the ball. What about having to move the pitch come thing and hold your hat and make the throw? There's a lot going on. I'm very good with my hands. Yeah, I mean, yeah, I think I have my, the ball in the other hand, so I was pushing it, and they wanted something to grab it, and throw it. It was a good throw there. Luis, there's obviously a good chance you guys see Houston in the playoffs later this year. What have you learned from facing their lineup? I mean, this is a good team that doesn't strike out that much.  So just, you know, attacking the song and getting those haters to, you know, put the ball in play.