Great. Thanks a lot, Brendan.  My name is Dan Olson. I wrote a book called the Lean Product Playbook. I'm actually going to be giving away a copy, like tweet to end of the raffle kind of thing, but it's basically a guide on how to achieve product market fit. I spent my whole career in product management and just wrote a book on how to achieve product market fit. So I'm going to share some advice from that with you in the time that we have. To share a little bit about my background, I started out with an engineering background. I've been coding since I was 10, and then I was an electrical engineering major. After college, I actually worked on submarine design. And after doing that for five years, I knew that I wanted to go to business school, so I came out to Silicon Valley to go to business school at Stanford. And that's where I learned about this awesome career called product management. I said, that's what I want to do. I had never done it before, so I asked people, where's the best place to do product management? And they said, into it. And so I was lucky to get a job and into it. I worked there. It was totally a great place to learn. Product management, marketing, UX design, development, it was great. And early in my product management career, I realized how important UX design is. So I made it a point to learn a lot about UX design. And also along the way, I made it a point to stay fresh and current on the web coding technologies as well. And after into it, I was a product leader at a few startups. And then basically I had the itch to do my own startup like a lot of people do. And I actually did that. I was CEO and co-founder of your version, which was a personalized news reader. We launched a TechCrunch disrupt. We won the People's Choice Award. So I consulted for a little while and then I did that for four years and then I went back to consulting. So I've been consulting and helping product teams for nine years now. And some of my clients, you can see here, are Box, Facebook, Microsoft, and Medallia. And I also run, I'm passionate about sharing best practices and advice. I run a big meetup group in Mountain View called Lean Product, Lean UX Meetup. And I've been doing it for almost four years now. We have over 6,000 members. And I host monthly speakers like the top product and UX experts that are there. So that's my Twitter handle. If you want to win a copy of the book, you just got to send a tweet with my Twitter handle in it. And I post all my slides on my website, dan-olson.com. And the book, you know, my book, it's a Lean Product Playbook. It's mainly a product management book, but it does have the word Lean in it. And Lean Startup is definitely, oh, hold on here. Lean Startup has definitely become very popular. You know, I've been giving this talk for a long time. It used to be called How to Be a Lean Product Ninja. That's why I have the Lego Ninja guy there. That's what led to the book being written. And I used to talk to audiences like this. I say, who's heard of Lean Startup? And a few hands would go up. And over time, more and more people would be like, why don't we just do it? Who's heard of Lean Startup? Raise your hand if you've heard of Lean Startup, right? If you haven't heard, you're like almost embarrassed to put your hand up because everyone else's hand is up, right? So you don't have to admit it if you haven't heard of it. But anyway, most people know about it. And if I had to say, hey, you know, what are the main points? If you had to summarize what the main points of Lean Startup are, what are they? And I think these bullet points here on the slide, it's about getting explicit about your hypotheses. When you're building a new product, you're basically making tons of decisions and assumptions along the way. And so it's about, again, explicit getting those out of your head down on paper so you can then figure out what's the fastest, easiest way to test your assumptions and see if you're right or not.  Like in the old days, you would have a team of engineers go off into a cave and code for 18 months and come out and launch what they just coded on the world, and then they'd find out nobody wanted what they just built. So it's about how do we make sure that we don't really invest a lot of resources until we have confidence that we're going in the right direction. The concept of MVP, minimum viable product, comes in that I'll be talking about a bit tonight. It's about getting out of the building and talking to customers and testing your ideas with them. And it's about learning and iterating hopefully quickly with the ultimate goal of achieving product market fit. Now as I list these bullet points, I see a lot of people kind of nodding along. Sounds pretty easy, right? But it's actually, it's not easy. At the high level, you understand the bullet points, but what I found is a lot of people when they get down and say, great, I just read the book, Lean Startup Book, or I just went to a talk, or I just read a blog post. When they go back to their team to try to do it, they run into challenges and obstacles at a tactical level. And the reason I share Lean Startup concepts is because in my mind there's a big overlap, you know, hold on, this is blocking the signal. It's a big overlap between Lean Startup and product management, right? And so I think Lean Startup helped product management out a lot because it gave us a lot of vocabulary and concepts, you know? So Lean Startup kind of took a lot of concepts and brought them together. Like MVP wasn't invented by Lean Startup, it was actually invented by someone else, but it made it popular, right? Product Market Fit, Pivot, all these concepts. So it gives a vocabulary and some common framework. So in my mind there's huge overlap between Lean Startup and product management, but you can also just think about my book as a product management book because that's really what my career's even in. And because I saw so many teams who got it at a high level and were motivated at a high level but didn't get it at the tactical level, were running into struggles, that's basically why I wrote the book, the Lean Product Playbook. And it's a playbook. And if you read the Amazon reviews, many of them comment on how, wow, this is the only book that actually tells you exactly like the steps to follow. It's very tactical like that. So it addresses that need of, well, what am I actually supposed to do right now? And as I said, I'm going to give away a copy. We have one right here. And if you want to know how to win a copy, you basically just tweet with that, Dan Olson. And I've got this cool tool that randomly draws someone that tweets with Dan Olson, right? And if there's a slide that you like, your photos are great. If there's a slide that you like, the framework or whatever, that's cool. Also, if you could add a hashtag, then other people that aren't in the room will also see it as well. So hashtag product management or hashtag Lean Startup, that'd be great. And basically, anytime you have a big movement like Lean Startup, there's a lot of buzzwords, like pivot, MVP, product market fit. Some of them are quite contentious. You can go online and find people having arguments about what an MVP is and what an MVP isn't. It's kind of funny to read those. Product market fit is a little different. People don't argue about what it means. They almost talk about it too simplistically. And they talk about it like some true or false condition of existence. So box, box succeeded because they had product market fit. Startup X, Startup X Sadly, they failed because they did not have product market fit. It's just like this yes or no, true, false condition. It's not really helpful if you're trying to actually achieve it with your product. And if you Google it, there's not a lot of advice out there. So that's why I wrote this book. And based on my engineering background, if I'm trying to accomplish something like product market fit, then I want to have a framework that I can use to define what it is and how to achieve it. So the key framework for the book is the product market fit pyramid. It's a model for product market fit. It has five layers. It is a pyramid. Let me walk you through it basically. The bottom two layers are the market layers. It all starts with the target customer. That's the foundation of the pyramid. And like a pyramid, each layer is meant to build on the layer beneath it basically, right? So it's like whose life are we trying to make better? Who are we trying to create value for? It's important to start with that because everything else changes. If you focus on one customer versus another, everything else is going to change. The next level of layer up is for that customer, what are their underserved needs, right? So we're going to be focusing on customer needs. And within the needs for that customer, we want to not address the ones that are already, they're happy with how they're being met today, but have some way of identifying which ones are being underserved or unmet. And we'll talk about that more. But basically those two together are market. It's a group of people that share a set of common needs. And if you look in a marketing or economic textbook, that's what I'll say the definition of a market is. So that's how I like to think about that. And these are basically hypotheses that you need to get right. The whole point is if you get all these hypotheses right enough, then you'll achieve product market fit. Your product, I like to represent with these three layers, starts with your value proposition. Your value proposition is okay, which customer benefits are we going to say that our product actually delivers on and how are we going to be better than the other products that are out there, right? That's where product strategy comes in. And it's hopefully it builds on the underserved needs, but it may or may not, depending on how you're doing. The next level up is a feature set, which is actually the functionality that conveys those benefits to the customer. And that's where the concept of MVP comes in. We won't want to overbuild or overscope before we're confident that we're going in the right direction. And then the final layer of the product section is user experience, right? That's what actually brings the functionality to life that you usually interact with to use the product, right? And so basically, you're making all kinds of decisions and assumptions up here. You can't really, these are in your control. These are not really in your control. You can pick which one you want to go after, but you can't like change people and say, oh, you have this need to know about things like that, right? So this is kind of, you're deciding on who you're going to go after. Here, you're making your decisions. And basically, product market fit is just, hey, how well do all the decisions we're making up here? How well do they resonate with the market that we're trying to address? That's product market fit. So when I had that framework, I realized, you know, as I said, this is just capturing the key hypotheses you need to get right. You need to get this right. If you get this right, this right, this right, that right, that right, and that right, then you'll have product market fit, right? Turning into a process where you can work through those key hypotheses and assumptions. And so the lean product process is what I'm going to be walking through today and sharing an example at the end. It's basically just starting at the bottom and working your way up, getting clear on what's our hypothesis about who our target customer is. What do we think their underserved needs are? Now, what's our value proposition going to be? Which needs are we going to actually deliver in our product and how are we going to be better than the competition? What should our functionality be? And that's a tough thing for MVP. You know, it's easy to put in too much. But what's the bare minimum that we can put in? Working through the user experience design. And then there's a sixth step, which is great. Once we have a user experience, whether it's a prototype, I'm a big fan of using prototypes, clickable, tappable prototypes to validate your ideas. And you'll see the case study I shared at the end. You can do that very effectively before you do any coding. Or if you're coding a live product, or you have a live product, you can do that. But then you close the loop, and the sixth step is to test with customers. And you close the loop, and that's where you see how you're doing with product market fit. So that's the lean product process I'm going to be walking you guys through today. And, again, it's a way to capture your key hypotheses. It's meant to be an iterative process. And how many people have heard of the build-measure-learn loop? So a fair number of you, right? How the build-measure-learn loop is it got people understanding that, hey, this is an iterative process. You build, you measure, you learn, and then you keep going and doing it. Some of the things, though, is it starts with build. And so some people misuse that, and they just build first. And as I just got done saying, you can actually do a lot of testing and validation before you do any building. So actually, I have a modified version that I prefer that starts with hypothesize. Because it starts out with what are all our hypotheses. Let's design a way to test those hypotheses. Let's test them, let's learn. And then we iterate and we modify our hypotheses and we go through that loop again. So it's a slightly modified version. So as we're going through the lean product process, we're iterating through this loop, right? Cool. So the first step of the process is determining your target customer. And by the way, I should say that you can use this process, obviously if you're building like a v1 product, you can use this process. You can also use it at the feature level. If you already have a product and you're adding a new feature, you can use it for the feature. You can also use it for a v2 product or a v3 product. You can also use it for a product that just you haven't used any of these techniques on. You're going to find ways to improve the product market fit. So it can be used in any of those contexts. But especially when you're working on a new v1 product, and it's like, okay, Dan's telling me I got to figure out who my target customer is. I just want to acknowledge that I know there's a lot of uncertainty. This is like the first starting with a blank piece of paper and it's like, who's my target customer? I don't really know. Because steps one and two are so related that as you get clear and talk to customers and understand what the needs are, you're going to go back and revise your hypotheses in step one. So don't worry about getting it perfectly right the first time. Just start out with some guests and we'll iterate from there basically. But I want to show why it's so important to get clear on the target customer. And what happens and what I've found a lot that product teams and companies do is they talk about products and customers and needs at a very high superficial level. It sounds good. In the meaning, it sounds good. But then you realize, oh, it's not really actionable. And to achieve programming for you, you really got to kind of get detailed. And the analogy you like to use is an onion. You got to appeal the onion, right? I remember I was judging a competition at an event. And basically there were these different startups and I just went up to one of the startup teams and I said, great, hey, who's your target customer? I was basically just going down this list with them to see what they were saying.  And at first I was like, okay, cool, millennials. That sounds good. But then I thought about it. I'm like, do you know how many millennials are out there? There's like millions of millennials out there. So it sounds like specific, but it's really not specific. That's what I mean about staying at the superficial level. Their product had to do with preparing food at home. So I know there were other ways that could have further refined and narrowed down their target market besides just saying millennials. That's too vague. They should have said millennials who like to cook at home. That would have been way better, right? It'd be like five layers into the onion at the first iteration, but you'll get there eventually. But let me show you why it's so important. Let's talk about a need that's kind of like at that millennials level. The transportation was 100 miles of my own. This is a need that a lot of us, probably everybody in this room has to get to work, to get to school, to get to the store, whatever it is, right? It's a common need. And if we just talked about the superficial level, we would only get so far. But the second you say, you know what, we're trying to meet this need for a certain type of customer, then you realize it, that that target customer really makes a big difference. That even at the high level, it's the same, but once you get more detailed, it's different. Let's talk about two different target customers, a soccer mom and a speed demon. They both have that need for transportation, right? They both have that need. But if I went and interviewed 20 soccer moms and said, hey, can you tell me what's important to you when it comes to transportation? They might bring up things like, well, you know, I'm like carrying all my children and all their friends and all their sports gear, so the car's just got to be big enough and stuff, and they might say things like, well, you know, I'm driving my children around, they're very important, so safety is really on my mind and I'm doing a lot of driving on the weekend so I'm thinking about how much money I'm spending on gas so it would be great if the car was fuel efficient. Those might be examples of things that we say. So it's not, instead of this, it's more like refining that, right? If we talked to 20 speed demons, they probably wouldn't bring up any of these things. They would bring up things like, well, it's important for me that the car goes really fast, it's important that the car looks cool and it's more important that I look cool when I'm driving down the street in my cool car, right? And you end up with very different products as a result. Both of these products help you with transportation within 100 miles of your home, but they've been optimized for that target market, right? And I love talking about the car and thinking about, I mean, sure, we have probably like 80, 90 people here. There's probably 50, 60 different types of cars you all drove here, right? There's so many different cars on the road. They've done a good job of tailoring the car to really meet the needs and preferences and in the book I get into how, you know, using personas is a great way to capture these, your assumptions, right? Personas are usually, sometimes they're not used or they're used in the UX design phase. I think you can use it up front. As I mentioned in the book, some people had a bad experience with personas. My advice is don't blame the tool. It's probably the people that used it.  They like long walks in the moonlight. That doesn't help you make product decisions, right?  Let's not use personas or bad. Let's not do that. So they're actually fine. What happens when you do like Facebook right, it has broad customers. So how do you do that? So you do cars, right? Cars do, too. You have to like define multiple segments, right? So the point is not that you have only one. Over time, you might have multiple segments, right? I mean, even within Facebook, yeah, you've got young people. You've got old people. You've got a lot of different stuff. So yeah, and in B2B, you might have two. So I didn't mention this, but like, you know, if you say you've got back to this pyramid, say you're Airbnb and you've got renters and you've got hosts. Well, guess what? You've got two pyramids because you've got two sets of tariff customers. If you're in B2B and you've got the buyer and you've got the user and you've got the admin, you've got three pyramids, right? The whole point is you need a pyramid for each one. So when you're in a broader offering like that, you need to get clear on who those are. One of the books that I referenced in my book is actually The Inmates Are Running the Asylum by Alan Cooper. So he's a big proponent of personas. And when I realized how important UX design was, I discovered and read his book. So he has a lot of advice when you have multiple personas on how to deal with that and focus on prioritizing them. So yeah, so if you have multiple personas or customers, you've got to get clear on who they are. So once we have a tentative hypothesis on who our tariff customers, then we move on to step two, which is, figure out what their underserved needs are. And when I talk about needs, a lot of teams don't talk about needs. They just talk about building features and things like that. And so an important concept here is what I call problem space versus solution space. Who's heard of problem space before here? Raise your hand. So a few people. So I've been talking about problem space since for a long, long time. And I'm excited to hear more and more people talking about it. But let me explain the concept you'll see I think why it's important. Problem space is basically a customer problem, a customer need or a customer benefit that your product should address. And sometimes people get, like, picky about, is it a need? Is it a pain point? Is it a benefit? It's all the same thing. It doesn't matter. It's like, what value is it trying to provide for the customer? Right? So a well-written product requirement would be in the problem space. A well-written agile user story, like, as a blank, I want a blank so I can blank. That would be in the problem space, right?  Well, let's keep going. The solution space in contrast is a specific implementation that's meant to address that need or requirement. So say I was like, you know what? I want to make it easier for people to share photos with their friends. That statement, make it easier for people to share photos with their friends, that's in the problem space, right? That's what I'm trying to make people's life better by providing that customer benefit. If right after that said, I know, yeah, last weekend I coded an iPhone app that does that, that app would be in the solution space. That's the difference between a customer need and an app. Or if I said, hey, my friend, you know, Bill, he's a designer and he did some mock-ups for an app that makes it easy to share photos with your friends, those mock-ups would be in the solution space, right? So that's the difference. This is like, what are we trying to have it do for customers? And this is the actual implementation. Yeah.  To me, that usually belies other issues if people are having turf wars. So I think you should be clear on who's driving what, right? The logo for my meet-up is a Venn diagram with product management, UX design, and dev. And I think that they should all be collaborating. When I worked it into it, one of our operating values was good ideas come from everywhere. So, yeah, if the designer has a mock-up and the PM says, oh, hey, I saw this cool dropdown on this other site, what do you think about that? That's not out of bounds. You know what I mean? There used to be more of those discussions back in the day about it shouldn't say the what and should just specify the why and that kind of stuff. I mean, in general, I think it's good. Conversely, you could be that PM who's like, I spent all night doing the sketch thing. Here you go, UX designer. This is how you need to build it. You want to stay away from that. You want it to be a collaborative effort and just be clear on who's driving the bus at that point, right? And in general, I think in the beginning of the process, the product management is more driving it, figuring out what's the business opportunity, what's the market opportunity, who are the customers, what are their needs. At some point, more like what we would call the discovery and product definition phase. At some point, you transition to the design phase and it doesn't mean the product manager goes to the involvement goes to zero or say it goes to zero. It just means the baton gets passed to the designer. If you have one, that's the other thing. You may not even have a design resource on your team, right? So, yeah. So I don't like when people draw, why are you doing that? It's usually a cultural indicator that there's a problem. There's not enough collaboration going on. Yeah. So that's the difference between these two. And the example I like to use to illustrate this is when NASA was sending astronauts into space, they knew that the pens, the ballpoint pens we used on Earth wouldn't work because they rely on gravity, right? And there's no gravity in space. So it wasn't NASA. I just want to be crystal clear. It wasn't NASA because there's actually, if you Google it, you'll end up on some urban legend site that says this is all baloney. It wasn't NASA. But one of their contractors, Fisher, this guy, Fisher, the head of the Fisher company, said, you know what? I think I can invent a pen that writes in space that doesn't need gravity. And he went off and he spent, he did it. NASA didn't ask him to. He decided to do it. And he spent a million dollars. And he actually invented a space pen. I have a space pen here. You got one too? All right. Cool. So, yeah. They tell me it writes in space. I haven't verified it myself. If anyone works at NASA down there and you want to send me up there to prove it, that would be awesome. That would be cool. Faced with the same challenge, the Russians, when they were sending their astronauts into space, gave them pencils. And you can actually get a Russian space pen. It's just a red pencil in a box, poking fun at, like, the whole space pen situation. Now, why do I bring this example up? I bring this example up not to make fun of it. It wasn't NASA again, but not to make fun of the situation, but to illustrate some points that come up. One is, obviously, if these are both equally good at meeting the problem, then the one that didn't take a million dollars in all that time and effort is a better solution. It's higher ROI. That's just the obvious takeaway. The second thing is, and I didn't say this, but many teams, they just jump straight into solution space without even realizing it. We live in the solution space. That's where we live. We live in the solution space. Like, human nature is to sink in the solution space. So it's like a learned behavior to say, you know what? Now, we need to separate solution from problem and get clear on what the problem is. It's a learned behavior. But even teams, then what that means is you just start coding or you just start designing without even thinking about who's this for and what needs is it going to meet for them. That's what going right into solution space means, right? And your odds of building a successive product go way down if you do that, right? But basically, even if you're like, okay, I get it. I get it. We're going to try to focus on the problem space. Even when you try to focus on the problem space, it's really easy for some solution space to kind of creep in. And I call it solution pollution, basically, right? So when the guy said, hey, I think I can invent a pen that writes in space. And someone said, okay, that's your requirement. Is a pen that writes in space? He had some solution pollution in his problem space, right? What was the pollution? Pen. Yeah, a pen is a solution. So, right? It would have been better if he just said away. Just be vague. Away to write in space. It would have been better than saying a pen that writes in space. But we can even do better than that. So in the Toyota Production System and Lean Startup is the five-wise technique where you say, well, why does that matter? Why is that important? It's like, okay, a way to write in space. Why do astronauts need a way to write in space? Why do they need a way to write in space? Take notes, yeah. Write down information, refer to it later. Do some calculations. There's a couple of things like that, right? That would be saying, hey, they need a way to write down information. Sorry. They need a way to capture information and refer to it later. Just stay away from it all, right? That would be an even better way of articulating the requirement. And then maybe it has nothing to do with writing it on. It's like some crazy Siri thing that we invent that you talk to it, you know? And it's like, hey, sorry, Dave, I can't do that, right? So anyway, so that's why I mentioned that, those examples, right? And now it turns out that this thing, there's other requirements that I didn't focus on. Thanks, nobody brought it up. Usually I have someone in the audience like, well, actually this thing burns in space and that's not good, right? It's true, yeah. So everyone uses space pens, but that's why I share that example. All right, let's talk about more of a software example. That's probably closer to what we're working on. So again, we have problem space on the left, solution space on the right. The idea is you want to map, you want to have clean, you want to start in the solution, in the problem space, and you want to map and work your way to the solution space and map and say, okay, cool, this feature idea is going to solve this problem and this one's going to solve this problem, right? So I worked it into it, as I mentioned, on Quick-In. Another one of our big products is TurboTax. Does anyone here use TurboTax? Yeah, exactly. Uncle Sam needs you to file and it's an easy way to file. So it's a product, it's a solution, so it's in the solution space. It competes with another product in the solution space, TaxCut, right? So for those of you that use TurboTax, what value does it do for you? What's the customer benefit that you get from it? Saving time. I need a lot of people here, yeah. Assurance. Assurance, okay. Simplicity, yeah. Error reduction. Error reduction, right. I know there's more TurboTax people in here. The IRS is right there. You better tell them you're using, I'm sorry. Speed, all right. Self-service. Self-service. One more. Maximize tax return. Save money. Save money. Save paper, saving the trees. All right, I love it, that's good. Huh? What's that? Be compliant. Be compliant. Follow the law, that's good, yeah. So see, that's the thing about the problem space. I'm the PM for TurboTax. I got to make sense of all that stuff you just said. How do I make sense of that? Forget about it, let's just code something and design something. It's way easier just designing code. That's the thing about problem space, right? People aren't computers. You're not going to give me the same ASCII string when I ask you why you use TurboTax. And why is that? Because one, some people care about saving money. Some people care about saving time. Some people care about assurance. They're talking about fundamentally different benefits. Even the people that are talking about saving time, someone might say it's speedy, someone might say it's this. They use different words even when they're talking about the same thing. So that's what we as product people need to make sense. And frankly, that's one of the biggest unique contributions that product managers can make. We're not there to design the product. We're not there to build the product. We're there to define who's this for and how is it supposed to create value for them. And that's what the problem space is all about. Now, we want to peel the onion, but it's helpful to at least have some overall umbrella statement to know what the context of, right? And if I had to put some overall umbrella statement over here, it'd be something like, hey, it helps me do my taxes. It helps me prepare my taxes, right? And all of the detailed things that you guys said would kind of live below that. And we want to make sense of that, right? The other person that really helped me appreciate this difference between problem space and solution space was the founder of Intuit, Scott Cook. He's a really smart guy. He'd be giving a talk to a group of product managers like this. One of his favorite things to do would be like, who's the biggest competitor to turbo tax? And we would all be like, eager beaver, raise our hands so he would pick us and he'd pick you. And he'd be like, it's tax cut. He's like, no, you're wrong. It's pen and paper because more Americans are doing their taxes with pen and paper than either of these software solutions back then, right? So that's the other thing about getting clear in the problem space is it helps you really understand what are the true competitors and substitutes that people are using to meet those needs that you're trying to address. And back to that pyramid, the bottom layer is the market. The top's the product, right? Those market needs don't change that quickly. Whereas technology waves on the solution space, on the product, they come and go rapidly, right? And the example I talk about in the book is the need, the customer desire to listen to music on the go. Start out with FM, the little FM transistor radios. And then we had Walkman, right? And then we had Discman.  And then we had iPods. And then we had these are phones now, right? So like five or six solution waves came and went. But that need to listen to music on the go didn't change, right? So that's the thing is these things don't change as much and these things change more. Now, as I mentioned, we want to peel the onion. We want to get clear on the problem space. And you guys brought up a lot of detailed things. I have some things here. Check my taxes. Someone said reduce errors, right? File my taxes for me. Instead of going and printing them out and having to go stand in line at the mail at the post office, I can just push a button. It can help you maximize your deductions and ask you questions and find ways to save you money. It can maybe analyze your return. These are just four examples. You guys brought up a lot of other ones, right? This is what you want to do. And this is actually the fun part, what I call divergent thinking. You want to, for that umbrella context, hey, what are all the different ways with your team brainstorm? We're not saying we're going to do any of these, but what are all the different ways we could make life better when it comes to doing their taxes, right? And over time, like, into it has some crazy stuff.  Hey, do you want that now? Anyone use that? And you know, you guys familiar with that? It's like, oh, you have a $2,000 refund. For a little fee, we'll give that to you now, right? All these things over time that they keep adding new and new ways to improve that value. So you want to peel the onion. There's the onion, just so we get the visual here. You want to not be it and peel it and peel it and peel it. And that's how you really add to create a lot of customer value in a G-part market fit. As I mentioned, it's messy. You're going to hear a lot of different things from people. And it's your job to make sense of it. And the first thing you'll find is that there are clusters that relate. They sound kind of the same. Like, someone said something about convenience here. Someone said something about speed here. Someone said simple. Those all kind of sound somewhat related. How do I make sense of that? So you may, again, like, say these three, these three benefits help me reduce my tax and reduce my outer check of my return. If we did the five, we talked to the users to interview them and asked them five bytes. Why is that valuable to you? Why is that valuable? Why do you like that? Why is that important, right? What we're doing is getting them to go from the very specific and get them to up level and get up to a higher level benefit. And it's like climbing a ladder. So we call it actually, it's called the benefit ladder, right? And you guys may all start out with these very detailed things, but if I did five bytes, we'd start to see that, wow, even though I had 10 different answers, there's really only like three or four ladders that they all kind of ladder up to, right? If I did this, these would probably ladder up to some overall benefit of empowerment or confidence. And the narrative we might hear from people might go, well, you know, before, I'm just not good at math and not good with computers and are not good, I'm done, the computer doesn't matter. I'm not good with math and I don't know anything about the tax law. Every year I have to file my tax returns. Those forms are so complicated. I have no idea what's going on. I'm bad with numbers. I'm probably making tons of mistakes. And then I use TurboTax and it's like completely different, right? It just holds my hand and walks me through, asks me some questions.  That might be what we would hear about that kind of a benefit. There could be a completely separate, distinct benefit ladder about saving time. It has nothing to do with empowerment or confidence and some of you guys mentioned save time. And then we could break it down and say, well, there's multiple ways that it saves you time, whether it's preparing or filing taxes, right? And there could be yet a third benefit ladder that's distinct about saving you money. And what we might find is that all or most of the features or most of the things that are in TurboTax eventually reach one of these three ladders. And maybe there's a fourth or fifth ladder, but these are just the three of the key ladders that are there, right? That is basically what I call a problem space definition and that's what you want to do before you start working on the solution space. That's your job as a part manager is to figure that out, right? And you'll notice this whole time I've been talking a lot about TurboTax without actually mentioning any features. I've been spending a lot of time talking about TurboTax in the problem space. And again, as I said, what you want to do is map, start in the problem space, and then say, okay, if you want to help people reduce your order, what can we do for them? And brainstorm the solution ideas from there. It just so happens that TurboTax has a feature decked directly one-for-one mapping with each of these problems. And when you have that clear one-for-one mapping and you have a well-named feature, there's a benefit, a side benefit that just by seeing the name of the feature most people can figure out how it's going to make their life easier or create value for them, right? And hopefully you can see that looking here. All right, so if you take my advice and as a product manager with your team, brainstorm what are all the different ways? What are all the different customer benefits we can do in the problem space to make people's life better within the context that we're operating? That's the divergent part where it's fun to brainstorm and create a lot of different options. The next thing is great, Dan. Now we have dozens of things that we can do for customers. We can't do them all. We have five developers. What do we do? So we need to prioritize now. Now it's time to converge and evaluate. Now it's time to bring in judgment and say, okay, which one of these do we think are underserved? Or are we taking it past that? Which ones are underserved that are going to create the biggest opportunity to create value? And the framework that I like using for this is importance versus satisfaction. Let me explain that framework. Basically, importance is for the need that we're talking about. How important is it to you, right? So if we just go back to TurboTax, saving time, saving money, and feeling confident. If I asked you to rate each of those on a scale of one to 10, you're going to give me different answers for each of those. That's what people will value. That's what we mean by importance. And you can think about it two ways. One is, as I just mentioned, like a survey where you survey thousands of people with a numerical scale of like one to 10, and you get statistical significance and you can come back and say, okay, this need is a five out of 10 and this one's a seven out of 10. That's kind of a quantitative way. You can also just think about it as a thinking tool. Low, medium, high. We have no information, just our guts. It's a team. What do we think? Low, medium, high. You can just do a little team vote and figure it out and you can do low, medium, high. So you don't get tripped up on, wow, how do I measure this very precisely? It can be used as a thinking tool. The other axis is how satisfied are people with how they're getting that need met today? So this is in the problem space and this is in the solution space. Again, we can say, hey, how are you doing your taxes today and on a scale of one to 10, how satisfied are you with that? People are low, medium, high with how that needs getting met today. So let's divide this up in a quadrant so we can talk about the different parts of the space. So in the bottom left, we have a user need that has low importance and people have a low level of satisfaction with the current alternatives. Over here, we have a low importance user need and high satisfaction with current alternatives. At the end of the day, neither of these quadrants are worth going after with your time. All other things have been equal. So people don't do this on purpose. They do this because they're not user-centric. They start in the solution space, they don't talk to users, they don't get clear about the hypotheses, they don't test their hypotheses, they go and they launch this and they think it's important, they launch it and customers go, no, we don't really like that product, it's just not important. In the upper right, we have a high importance of user needs so that's better than low importance but we also have high user satisfaction. So yeah, it's an important need but people are pretty happy with how it's getting met today. That's the definition of a competitive market. And I'm not saying you shouldn't go after those markets but when you go after those markets, especially if you have a smaller team and less resources, you better be crystal clear as how you're going to be better. And not just a little bit better, you hear people say you need to be 10x better, this is where you need to be 10x better. So I'm not saying you shouldn't do it, but you've got to be really confident in how you're going to be 10x better. The first product I think of, the space that I think of up here is Google Internet Search. If I said hey, how important is it to find the information you're looking for online, it'd be pretty important. But it's not like people walk around going, gosh, it's Google thing, I can't find anything online, it just doesn't work, right? You may have to modify your query once or twice or go to page two, but you'll find what you're looking for eventually, right? So that's what I think about there. And it's funny because there's actually, we're all laughing, but there was a startup in 2008 that tried to take Google on an Internet search and I talked about that in the book. All right, then the last quadrant up here in the upper left, we have a high importance of user needs, so that's good. And we have a low satisfaction with what's out there today. This is where opportunities lie, right? So this is where that's what underserved. The closer something is to this upper left corner, the more underserved it is. And these opportunities do exist. They probably don't exist forever because we live in a competitive environment, other companies are trying to find unmet needs and address them and innovate, right? So these windows can open and close, but if you don't do the right type of customer discovery and think about things this way and analyze it, you can find opportunities. The one that comes to mind for me is the ride sharing market, right? Especially in the Bay Area, I used to live in the city and if I said to you, let's just do it this way. How important is it to get to your flight at the airport on time? Pretty important, right? How important is it to get to your job interview on time? Pretty important. How is it important to get to your date on time? Whatever it is, pretty important, most of the time to get where you need to be on time, right? So high importance. And, you know, not every cab ride, but the city of San Francisco had a pretty bad, I used to call, you call a cab, hey, yeah, they'll be there in 20, 30 minutes. 40 minutes later, no word. Call back. Oh, yeah, I don't know what happened. We'll send you another one. You know, it's like, so maybe they show up, maybe they show up late, who knows, right? So if you just said, hey, overall, think about your last 10 cab rides that you took on a scale of one to 10, how satisfied were you? Obviously, they're good cabbies and good experiences. I'm not saying they're all bad, but you can imagine it being low. And we could peel the only one more layer instead of just asking for overall satisfaction. We could say, how satisfied were you with, you know, the punctuality of when they arrived? How satisfied were you with the cleanliness of the car? How polite was the driver? You know, how safe did you feel? How easy was it to pay at the end of your ride? You can imagine low scores on a lot of those dimensions, right? And so those companies have done a lot of things right to get to the large size that they have, but that's, I think, a fundamental reason, is that they were addressing an upper left quarter need. And what I like about this framework is it's meant to be a visual framework. So what I'm trying to say, if there's a certain product that's represented by that red dot, and we plot it with what we're hearing from people saying the importance of the need that it addresses and what we're hearing from people about how satisfied they are with how well it addresses that need, we plot it in that XY space, then basically what I'm trying to say is the area formed by where you plot it, that rectangle, is a proxy for how much customer value it creates. And if there's another product that addresses a higher importance need with a higher level of satisfaction, then it creates that much more value, right? And when you see this important satisfaction, this way you realize that the opportunity to create customer value is just how much area is there to the right of where the market is? Where's the best product in the market that's meeting that need? Are they here? Are they here? And the more room there is to the right, the more area there is to the right of where the best product is, that's where the biggest opportunity. That's why the upper left quadrant offers the biggest opportunity. And if you have an existing product, say your green dot is your existing product, you can create more customer value by just improving the satisfaction with which it addresses the need that you're trying to address, right? Now if this seems kind of conceptual, abstract, theoretical, let me show you some real data. And this is in the quantitative survey case where I was fortunate to have a lot of users that I could survey. We asked them to rate it numerically. So now instead of low, medium, high, actually for 13 of the key features in the product, we ask people to rate their importance and satisfaction. And now instead of low, medium, high, we have numbers going up to 100 and numbers going up to 100. Each of these dots is one of those 13 key features. Plotted with its values that we heard from our users. And the number next to it is just a satisfaction number so you can see it more easily. The first thing I saw as a product manager was this point up here. 100% importance, customers are telling us it's 100% importance and they're telling us that they're 98% satisfied. That's pretty good. So I was excited. My second thought was, okay, I have a limited dev resource. I'm not going to spend any of my resources on that because it's already doing pretty good. Let's go find other things that are going to create more value. We had a cloud of things here. We had this one down here. But this one is the one that's the most up into the left. That's the one that offers the biggest opportunity to create customer value. So we focused on that. And I developed this framework while I was into it. And then a few years after I left into it, I came across this great book by Tony Olwit called What Customers Want. And lo and behold, he had his own importance versus satisfaction framework. And so that made me excited because I'm like, wow, I came up with this thing and he came up with this thing. There must be something to this important satisfaction framework. And he actually gets really into defining it and getting quantitative in it. So if this idea of getting quantitative and rigorous about importance and satisfaction appeals to you, I would recommend checking out his book. He's also one of the co-creators of Jobs To Be Done. And he has a new second book called Jobs To Be Done just came out last year. I actually hosted him at my meetup when the book came out. The video is online. So I recommend checking that out. So we want to basically use importance versus satisfaction to identify which of the most underserved means that we want to focus on. Yeah. Just good market research. What's that? Yeah. Get a new survey writer. Yeah. I mean, it's basically like, you know, it's funny because as people want to get more and more customer centric and use either qualitative techniques of interviews or quantitative techniques of surveys, there's actually people that have PhDs in those things. And I was very fortunate into it that we had a PhD in market research on our team. So I learned that. So like everything else, there's best practices and mistakes to avoid. You don't need to get a PhD in it. There are certainly books. I have a whole chapter in my book on user testing and how to conduct user testing well. I'm not a big fan of surveys because they get misused. It's so easy to misuse. If you want to get a feature approved by management, I'll write the survey. I'll make people answer it in a way. I'll write the survey so that people answer it so it gets approved. You want to get it killed? All right. You can do that with the questions, right? So surveys are easy to misuse. But the good news is conducting in-person interviews, there's only a few tips and hacks to learn and things to avoid. You can actually get pretty good at it. And so I have some good advice in the book about that. So there are other books out there, too, that go really deep into how to conduct user research and things like that. Yeah. That is another dimension. That's another dimension. It's not the same as importance. It's an important thing to look at, too. And that kind of also, that will mainly also come in, it will come in also into your revenue model as well. So I know people that have a low frequency of usage and it can be tough. So now, if you're an entrepreneur deciding which market to get into, you can use it. If you're a PM at a company, guess what? There's an organic frequency of use. You can try to drive it and increase it a little bit. You know what I mean? Like if you're a TurboTax, how many times a year do you guys use TurboTax? Right? So that's why they're trying to find other ways.  So in general, yeah. So, but you're, I agree, and yes, frequency of use is another good metric to look at and can be a proxy. But it has less to do with the fundamental importance. Like how important is it to file your taxes? You're going to go to jail. I think I can get you, right? So even though it's once a year, it's pretty important. How important is getting married? Right? So frequency of use is not the same. Don't conflate the two. But within two things that are equal importance, I would pick the one with higher frequencies just because you're going to have more interaction with the customer and more ways to monitor. You know what I mean? More share of mind. It's more about that. It usually comes in more into play when you're trying to monetize it. There's more perceived value often. But all right, cool. Yeah. It doesn't have to be once a month. Yeah, it doesn't have to be one to one. A couple of things can happen. Let's go back. Let's talk about some anomalies. You want to have at least one. So what happens sometimes, if you go into a product where they were solution space focused and not customer focused, you find a couple of things. One is you may find a feature that nobody in the company can explain what it does for customers, and no one uses it. And it's like, why do we build it? Well, Dave, the engineer, thought it was cool, so he built it. So that's that orphan solution that you can get. That's one anomaly. The other is you can have an orphan problem where it's like, oh, jeez, we all agree. This is an important problem. But when we look at our product, there's nothing that actually addresses this problem. So that's where you get zero mapping. The reality is you may have multiple things that support. You may have two or three features. It's all about granularity too. I can break this feature into five chunks or I can all describe it as one clump. So it just depends. But yeah, it wasn't meant to be only one. It's like at least one. For the ones that we want to bite off, for the ones that are going to be on our MVP, for the ones that we just went through. So this doesn't talk about prioritization yet. It's just saying, hey, but after we run the prioritization and say these are the ones that are underserved that we want to have it, yes, you have to have a solution. Otherwise it doesn't provide the value. It doesn't solve the problem. That's exactly right. Yeah. All right. Cool. So once you've kind of gotten clear on our entire customer and what their underserved customer needs are, the next step is to say, OK, out of those underserved needs, which ones are we really going to say our product, we're going to kind of put our brand promise behind and say, well, our product does this for you. And here's how it does it better than the other competition. We don't want to launch a Me Too product, right? So that's your value proposition, basically. And the framework that I like to use to get clear on your value proposition is the K&O model. And so before I actually came out in Silicon Valley, I got a master's in industrial engineering where I studied lean manufacturing and quality. And I learned about the K&O model. So before I even got into software development stuff. But I think it's a great framework to think about your value problem. Let me walk through the framework. So on the horizontal axis, it's talking about how fully does the product or feature meet the need that we're talking about. From it doesn't meet it at all, like the product, the way we did it in the product doesn't meet the need at all, to, wow, the product fully meets that need. I'm fully set, right? And then the y-axis is based on how much the product meets the need, how much customer satisfaction or dissatisfaction does it create. That's what it's all about. Now, if this sounds a little complicated, don't worry. It's really simple because K&O model breaks it down into one of three types of benefits or features. The first is a performance benefit, right? The more the product meets that need, the more customer satisfaction is created. The less it meets the need, the less customer satisfaction there is. This is pretty straightforward, right? More is better, less is worse. If we were in the microprocessor business and our chip speed was 10% faster than the other guys, chip speed is like, you know, clock speed is a higher, is a performance benefit, faster is better, right? Let's go back to cars. Say I was shopping for a car and there were two cars and they were pretty similar. And it's similar features and things. And then I suddenly realized that one car had twice the fuel efficiency of the other car, right? All other things being equal. I'd pick the car that gets more miles per gallon because, you know, fuel efficiency for most people is a performance benefit, right? So that's performance. More is better. That's pretty straightforward. The second one is a must-have. Now, having a must-have doesn't actually make anybody happy. That's what this means. Say you've fully met the must-have. I'm not doing cartwheels or anything. But if you don't have it, I'm going to be unhappy. That's the definition of a must-have. It's not what your executive team says this is a must-have feature we need to have. This is the definition is that customers are telling you that if you have it, you better have it or I'm going to be unhappy. And if you have it, that's great. But that's not enough. I need more than just a must-have, right? A must-have. That's a must-have. So you can think about, like, a table stakes or cost of entry or checklist feature. Sticking with cars, say I went in, you know, I was buying a new car. I went to the showroom and I saw this car and I just loved the way that it looked, right? And then I read the little spec sheet on the window and I thought all the specs and features sounded great. And then I peeked inside and I realized, oh, it doesn't have any seatbelts. I wouldn't buy it because seatbelts, I'd be afraid of getting hurt or dying, right? So seatbelts are a must-have for a car. But if car A has five seatbelts and car B has 100 seatbelts, I don't say car B is 20 times better than car A, right? Once you have one seatbelt per person, you're checked off the list, right? The third category is delighters. So not having a delighter doesn't cause any problems because people don't expect it to be there. But having a delighter, right, the more it provides the delight, the more customers that it. We call this a wow feature, basically, right, or a delighter, right? Not today, but sticking with cars. When the first cars came out with GPS navigation, it was a delighter, right? Before that, people were getting lost or printing out their Google Maps or MapQuests on paper, you know, asking for directions, right? And then the first people who had GPS in their cars, they just put in the address where they're going and just changed how they got from point A to point B. But as we know, over time, Tom Tom and Garmin came out with their add-ons, and now we just all use our phones, right? So this is not a static picture. It evolves over time, right? So the needs and features migrate over time so that yesterday's delighters become today's performance features that become tomorrow's must-haves. And the pace with which that happens depends on the level of competition and innovation in your space. But basically what we want to do is use these three categories of must-have performance and delighter to categorize our needs and articulate how we're going to outperform and be better than the other products, right? So again, our value proposition is, hey, out of all those underserved benefits that we identified, which ones are we actually going to bite off and tackle and commit to solving and how are we going to be better? And so the way we do this is for our product category, you create a table and you list one per row each of the must-have benefits and one per row each of the performance benefits and one per row each of the delighter benefits, right? And I've kept it generic, so hopefully you can picture what that might be for your space, right? Again, if we were in the chip space, like, you know, clock speed would be a performance benefit, right? The next thing you want to do is create a column for each of your competitors, one column for each key competitor, and one for your product. That's the second step. The third step is you want to evaluate and score your competitors, right? And it could be, again, it could be like, hey, if it's like some numerical measure, like chip speed, you could put the numbers in there, but you could also just use low, medium, high, right? As long as you look across the row and say, oh, yeah, these guys are a lot better than these guys. These guys are a lot better. It can be low, medium, high, right? And so in this case, for example, both of our key competitors have the must-haves. Competitor A is the best of performance benefit one. Competitor B is the best of performance benefit two. They're both so-so performance benefit three and these guys have this delighter, right? This is the backdrop upon which we want to make an informed decision about our value prop, right? This is like the essence of project. Basically, and very few teams go through this exercise. They just start building. They go right into solution space and start building stuff. Or even if they think about the customer or benefits, they don't get clear on how they're going to be different. The number of stories you hear about some entrepreneur going to pitch a VC and is like, hey, I'm going to build some cool photo sharing out. The guy's like, hold on a second. Have you seen this before? And it's okay. And again, it's not saying just because someone else has done it, you can't do it, right? Yeah, you know, it's super easy. Human nature would be like, all right, we're going to be the best at everything. Hi, hi, hi, hi, hi, right? And I run workshops. I'm actually running a workshop tomorrow. And one of the cool things I like to do is see, okay, how many teams had one high? Nobody has one high. How many teams had two highs? And there's a couple teams. How many had three? And then four. Some teams had like four highs. There's no way you can be the best at everything, right? You just don't have the resources. It's like a clear positioning in the marketplace, right? If you're all over the place, right? And it's easy to say yes and hi, right? But one of my favorite definitions of the strategy is it means saying no to something. That's what being strategic is really about, is saying we're not doing this, right? So you got to say no somewhere. So given this backdrop, we might go with something like this as our value problem. We might say, of course we're going to have the must-have. We're going to, you know, deliver some performance benefit one, but we're not going to try to outperform competitor A. We're not, we're going to say no to performance benefit two. We're going to say no and not worry about that one. What we're going to try to do is be the best to performance benefit three. Maybe we've identified some market segment that really values that performance benefit and that's what we're focused on. Or maybe we've come up with some unique solutions-based ideas or technologies that lead us to believe that we're going to be able to deliver higher levels of satisfaction for that need, right? And then we have our own delider idea what matters the most is what's called your unique differentiators. It's the performance benefits where you're going to be the best, usually one, maybe two, and your unique deliders. That's what matters the most, right? And this is what we want to get clear on as we go into the next step of our MVP because if this is what our special sauce is and why we're going to be better and then we go into an MVP and it doesn't have either of these things in it, what the heck are we testing with our MVP? We're not testing must-haves. You're not going to win on must-haves, right? So that's what we want to get clear on. And just to kind of drive this point home a little bit, has anyone here heard or used Instagram before? Oh, cool. A few people, right? So what were there, not now, but when they first came out, you guys remember there were tons of photo sharing apps when they came out. So why did they win? I think it's because of their value prop. So what was their unique differentiators? Anyone have opinions? Filters is the first answer that I usually get. There's one of them, I believe. There's one other one. Yeah, just you. Huh? That's a second-order one. I think it's a good one, too. So square pictures meant you didn't have to worry about portrait or landscape. Your picture always looked good no matter what was going on. I'd throw that in there as another one. There's one other one that I'm thinking of, too. Posting directly for the phone. Yeah, the way I would say it is faster uploads, right? All of the other guys, you have to sit there and wait, and what they did, technically, they had a hack where they just started I think they had better compression, and they just started uploading it. They didn't wait for you to push the upload button. They just said, of course, they're going to upload.  Start doing it. So you gain like a second or so, and then you go, wow, it's a second faster than all the other ones. So it was a combination of just that UX hack and some legitimate tech hacks. Yeah, yeah, right. So faster uploading, filters, but let's just for simplicity, yeah. Okay, yeah, that's the five Y's. Like, why does that matter? Like, yeah, you look like a rock star with your friends, right? Right, so let's go back to this. So let's classify these things as either a must-have, a performance, or a delighter. Like, reduced upload times. Is it a must-have, a performance, or a delighter? Who thinks it's performance? Raise your hand. Okay, who thinks it's delighter? Raise your hand. Who thinks it's a must-have? Raise your hand. I can't be a must-have because the other people didn't have it. It's a performance. Any time it's faster, better, like a number, if you can measure it with a number, it's a performance benefit, right? These guys are 30% faster than the other. It takes five seconds on these guys' apps and one second on this app. That's a performance benefit, right? Yeah? Huh? It's just semantics of how you talk about it. If you have like a 10X of performance benefit, you can say, wow, it delights users, but if you go back to this diagram, right, it's just more is better or less is worse. It's not like if you don't. If you didn't upload at all, would they say, I'm good? But let's do this now. Let's do this. Filters, are they a must-have, performance, or delighter? Delighter. Delighter, yeah, right? That's why delighters are more like this yes-no binary, either have it or you don't. It's not like, oh, well, you could argue, hey, there's degrees of filters and how many filters do you have and quantify the quantity and quality. And then how about square pictures? Performance must have delighter. That's a tough one. I would say probably a delighter. It's kind of like a UX thing, like easier to use, maybe a delighter, I don't know. I mean, some people didn't like it. Like, why are you making my picture square? But anyway, so, but that combination of a performance benefit, it's like a one-two punch, a performance benefit and delighter, it comes up all the time. If you start to look at things through the K-no model lens and the value prop lens, you realize, wow, they won because they outperformed on this and they had this cool delighter. So that's a very common one-two punch that you see. Instagram is a good example of that. Okay, so once we get clear on our value prop, then we go to the next step, which is, okay, great, now what do we want? What feature set should we build? Now we're finally, the whole time, one, two, and three, we've been in the problem space. Now that we're clear on what's underserved, whose life we're trying to make better, what's underserved, and how we're going to be better than the competition in meeting those needs. Now we're going to say, great, now let's take those needs and now let's do the mapping to the solution space, right? And so, and again, we don't want to like overbuild before we realize, gosh, we didn't go in the right direction, right? So we want to take an MVP approach, which is just to scope it just enough to figure out if we're going in the right direction or not. So MVP is one of these terms, it's a buzzword, right? What does MVP mean to you guys? Yeah, literally it's minimum viable product. What does that mean to you, though? Okay, max value for minimum effort. Yes. Yeah, lowest cost product. Product sales front, right. So you guys are all giving similar answers, but there are some debates online about this kind of thing. So make money. Wow, you got to make money. She's all about the money over there. Ask it, all right, cool. Yeah, so people have different... Huh? If it wasn't any less, exactly right. Cool, yeah. So MVP, it's funny because there's two mistakes I see teams make with MVP. One is the whole point of an MVP is to avoid the old way of doing things where you like, you know, you bid off way more scope before you launched and tested and saw if you were right or not, right? So even though MVP is meant to like avoid overscoping your initial product, people still overscope their MVP. And it's because it's human nature. It's like, oh, yeah, well, let's just be safe and put that in. Someone might complain because it's missing, right? So it's really easy to just say, oh, yeah, put that in and put that in and put that in. What that does is it just delays the testing and it just takes more time to get to the MVP point, right? And then if you realize you're wrong about some other stuff, then it's just you wasted that time. So it's tough. And look, it's tough to have the discipline and say let's try it with this and not do it. That I see. And it's almost like you need someone else. I have people that read my book, like Dan, love your book, totally get MVP. But here's why my MVP needs to have A, B, C, D, E, and F in it, right? And then you get some objective third-party asking questions and going, do you really need that or not and you can kind of get. The second thing I see, the way I see people misuse MVP is usually from people that don't really understand it. I like to explain with this framework. So it's another pyramid. It's actually, has anyone here used Mailchimp before? Was it pretty easy to use? I remember when it came out, it was so much easier to use than all the other email platforms at that point in time. So Aaron Walters was the head of UX design for Mailchimp. So he knows a few things about UX design. And he has a book called Designing for Emotion where he has this framework. And the idea is we can talk about an evaluator product and talk about how functional is the product, how reliable is the product, how usable is the product, how delightful is the product. The idea is you kind of cut and color in how much you plan to accomplish at each phase. Right? And it's what I love, he's trying to elevate the discussion to delight, right? About 15 years ago people kind of had an awakening and realized, geez, wow, usability is really important, ease of use, like the usability answers can people use your product? Delight answers the question, do people want to use your product? How do they feel when you use your product? So that's what the conversation is going these days. And so obviously with our MVP, we're not going to build the whole pyramid. It's not going to color in the whole pyramid when we do our MVP, and we're not going to color in all the functionality, right? But people, the mistake I see people do is they use MVP as an excuse to do this. They build a subset of functionality. Okay, it's an MVP. We can't build all the functionality. Cool. But they say, you know what, it's okay if it's buggy, it's just an MVP. It's okay. We'll do the UX design later. It's okay if it's hard to use. It's just an MVP, right? How do you think those test when you test it? They don't test well. It doesn't matter if you've got some cool feature. People can't figure out how to use it or it doesn't work when they want to use it. You're not going to get credit for the functionality, right? So these things can get in the way of the value of the customer getting the value. So it's true that you only want to build a slice, right, with your MVP, but you want the slice to look more like this, right? So whatever subset of functionality you build, it's not going to be perfect, but it should be reliable enough and usable enough and delightful enough, right? Otherwise, going back to the value prop, what are we testing? We want to test to see if our unique differentiators resonate with people and people agree that it's better than what I'll say used, right? So a lot of times they see people, usually it's when they're trying a team's trying to do a cultural transition to Lean or Agile and they say, okay, let's do MVP and they do this and then they test it, it doesn't do well. They're like, oh, see, this whole thing doesn't work. Let's go back to Waterfall and do it the other way. It's like, well, they didn't do the MVP right, yeah. It seems like a story. Yeah, well, I mean, yeah, you could build this in an agile way. You could build that in an agile way and just, you know, if the team says, hey, bugs don't matter and UX doesn't matter, then they can do it, but yeah, so. All right, so those are the two mistakes and in the book I have a lot of advice on how to, like, map from problem to solution, how to break things down. They don't have time to get in today. All right, so once we get clear on that MVP feature step, the next step is to create a prototype. And this is where, you know, so going from step three to step four, we went from problem space to solution space. When we talk about our features, they're usually expressed as words, right, in JIRA tickets or in a Google Doc or in Confluence, they're just words, right, or PowerPoint. Now it's time to go from words to actual designs, right. We're in the solution space, but they're actually time to do designs. And we need to do designs for two reasons. One is just to figure out what the heck it's going to look like, but the other reason I'm a big component is to try prototypes so you can test it without doing any building or coding, right. And so we're going from words to designs, which means it's time to apply UX design. And the way I like to think about UX design is as an iceberg. Because the thing about an iceberg is there's a portion of it that's visible above the water, but there's a lot underneath the water that you can't see, right. When it comes to UX design, the part above the water that most people see and react to is visual design, right. You guys are looking at my slides taken in the fonts, the colors, the shapes, you're looking at me, we're all looking at each other, right. We've probably taken 90 plus percent of our information through our eyes, maybe a little bit through our ears when we're listening, right. But mainly through our eyes, right. But when you use a product that's easy to use and intuitive and you enjoy using, it's because the team has done a good job of these more foundational levels that aren't that obvious, right. And those foundational levels are, you know, it starts out with a conceptual design, like what's the fundamental concept for how we're going to use your interface here. The next level up is information architecture. That's how you organize and structure your application or product, right. If it's hard to find things, things aren't where you expect them to be, the labels don't make sense to you, then they've done a poor job there. If instead, things are where you expect it's just easy, you don't really think about how you're navigating and finding things, then they've done a good job. Interaction design is literally how the customer interacts with your product, anything they can click or tap on, forms, controls, navigation, that's it. Again, I don't have time to get into it, but I have a whole chapter on UX design because I think it's really important for product management to learn about it. I don't expect you to be world-class designers, but it's going to help you be a partner better with your design partners and, you know, give them feedback on the design, come up with ideas, so I think it's important. Now when we're going from words to live products, at some point you've got to go from words in JIRA or Google Doc or Confluence about what we're going to build in features, and we actually have to get the product. And we want to work our way through UX design. So there's a range of different design deliverables or artifacts that we can create, and what I'm going to share with you is a process that I recommend as a way to kind of work your way from words to live product going through UX design. And I like to categorize these deliverables by fidelity from low to high, which is like, okay, compared to the final product that we ship, how closely does the design artifact resemble it as far as like pixel perfection, so low would mean it's like a crude sketch, and up here it would be like, hey, it's pretty close to being like the live product. On interactivity, this is the question of how much can I interact with it compared to the live product? The design deliverable, right, our artifact. In the bottom left, low fidelity, low interactivity, we have the awesome hand sketch, whether it's on a piece of paper, on a whiteboard. This is like a good first step when you're going from words to sketches and designs for your team to work through and iterate, you know, on a whiteboard until you get to the point where you're like, yeah, that looks like a good initial design. The next level up in interactivity and fidelity is wireframes. In the old workflow, they would be static wireframes, but these days, the tools make it so easy to make clickable and tapable wireframes that if you're not doing clickable and tapable, just go learn how to do it. There's no other way. When I interview PMs and I'm like, have you done wireframes and they say no, it's just like it's not a good sign. It's like it's so easy. Has anyone heard of Balsamic? Yeah. So Balsamic is like great for product managers. Like you don't need to be a design. It's not sketch. It's not Photoshop. It just makes it really easy. It's deliberately low fidelity, right? And back to that iceberg, you know, I've seen this happen a lot where well, I'll come back to this point in a second, but basically clickable wireframes, tapable wireframes, you want to basically iterate until you are happy with that. And that actually is good enough to test with people. And I advise testing with people. The reason why is, yeah, it's not going to look like that. Back to our MVP discussion. Say you did have the courage to not put features X, Y, and Z in there. And you go and you test this. That's when the customer is going to go, are you crazy? Where's feature X? I can't use this MVP without feature X. That's where you'll learn it. Like, it'd be great to learn that sooner rather than later, right? They're probably not going to complain about why did you put feature B in there? I don't need feature B. They're not going to complain about extra stuff, but if anything critical is missing, they will tell you. This is also a great example of interaction design issues, right? The next layer up of fidelity is mockups. This is where now, okay, it is like sketch or Photoshop. A designer is like making, going through and figuring out the fonts and the colors and all that jazz. That's where we go to high fidelity. And again, the mockups are created in tools like Photoshop and Sketch. And then they export image files, right? But there are great applications that let you take those image files and actually create a clickable or mockups or prototype. Does anyone here use Envision or familiar with that? The cool thing about Envision is that as a non-designer, non-technical person, you can get a folder of screen designs or page designs and upload it to Envision.  And when they click that, then it's going to go to this screen. And you can create all these little hotspots that navigate around, right? And you can do every scenario or what we call the happy path, where you expect people to go down. So that's a very powerful way to get feedback without doing any coding, because now it looks enough like the final product and people can click around on it, right? The reason to start with wireframes, one of the reasons is back to the iceberg is people cannot help but fix it on vision. Unless you're all trained, just like teasing apart solution and problem space, teasing apart visual design from information architecture and interaction design, unless you're a trained designer, it's very hard. I remember this for a new product to the CEO and his first comment was, I don't like this color red that you guys are using. The first comment wasn't like, why did you pick these features? I don't think this feature should be here. This feature is missing. I don't like the layout. It was this red. People cannot help but fix it on visuals. So when you do wireframes, they're usually grayscale. You can't even argue. We're not even saying what the color is yet. You can't pick that fight yet, right? Let's just talk about what's on the screen. What's the layout? Is anything missing there? Are we using the right words for it? That kind of stuff. That's why wireframes are a great way to just not let people fixate on the visual design. They're done in grayscale. You don't worry about fonts. In fact, Balsamics default mode, the lines aren't even straight. They're like these squiggly, kind of sketchy lines to convey to you subconsciously. Like, hey, this is rough. It's a sketch, right? Definitely, this is a great place to test with users. I recommend talking to users in waves of 5 to 8, and then you pattern match the feedback that you get. You address those concerns, and then you test with another batch of 5 to 8. Once you get, you know, however many iterations that takes to get to the point where people aren't really raising any questions or concerns with what you're built, and they start saying, wow, this is actually pretty cool. I can see using it. Then you can proceed confidently to going to live product and investing the resources to code. And I do recommend that you test your live product because there's no, you know, there's no, like, database performance issues with a mock-up or compatibility issues with a mock-up. And sometimes when you put something into the pipeline, it doesn't come out exactly the way that you specified it. So just test that, and then you're good to go. To make it crystal clear, this is a wireframe. You can't argue with me about the red or purple because there is no red or purple. It's just gray. You can't argue with me about the font because we haven't picked the font yet. You can't argue with me about the pictures. You have to focus on what's on the screen, is that the right stuff on the screen, is that the best layout, that kind of stuff, right? You're in a balsamic. Here's a mock-up, higher fidelity of the exact same screen. That's the difference between a mock-up and a white one, right? Now you can say, yeah, I don't like, why did you pick this orange? I don't like it. You can do all that, right? But you want to start out and get alignment here. Now they did pick the fonts and colors and things like that, right? So that's the difference. All right, the final step, we have our awesome prototype to go test with customers, and it's time to go test with customers. Going back to our thing, now we've worked our way all the way through the pyramid. We've got our UX prototype, and now we're going to close the loop and test with customers, right? And it's important one of the key things I'll say here, I don't have time to get into it. I have a lot of tips on how to do this, but one is make sure you talk to the target customer you start out with.  And just, because it's like, no, it's not who the target customer, so you need to make sure that they fit in your target customer. All right, I'm going to close out with a quick case study here. People have said it's helpful to see the process, an example of it applied end to end. So this is from a product called marketingreport.com. My client was a startup CEO. They had an existing product that was in market, and he had an idea for a new product. And it was a very small team. It was me, the CEO, the VP of marketing, and a UI designer. And he wanted to see if there was a business opportunity here, and he had a very limited dev team, so he's the one that said, hey, we can't do any coding. I said, great, we're going to use prototypes, and we'll do this. The idea was a marketing report. Does anyone here get junk mail? No? Raise your hand if you get junk mail. You guys are crazy. So you get junk mail. Direct mail is a nice way of saying junk mail. Direct mail. The idea is like, provider product that would help people control it. Like, say I come home tonight and I go to my mailbox and there's a coupon for cat litter. Why did I get a coupon for cat litter? I got sent that coupon for cat litter because some database, marketing database in the cloud somewhere, it thinks Dan Olson has a cat. That's why. So that was the idea, is that you would kind of be able to understand that data and control that data. It's very analogous to the credit industry. Not today, but 25, 30 years ago, you'd apply for a loan or credit card.  And you would have no idea why. But over time, we had credit reports and credit scores, and you can go and say, oh actually that's wrong, you can fix it, things like that. That was the idea with the marketing, it was very analogous to the credit report. I said, okay, sounds good. And by the way, the target customer is like everybody who gets junk mail, so it's a broad consumer offering. That step one is target customer. I said great, let's talk about the customer benefits that we want to focus on. The best explanation of the benefit was basically learn why I received the junk mail I received. That was the benefit. The core benefit was that. Learn why I received the junk mail I received. That's the problem space. The top solution space idea was a marketing report, which would be like a credit report. It would have a profile, like all the data, oh you have a cat, you have kids, you're married, this kind of stuff. That they would be using to target offers to you. And then a marketing score, which was meant to be like a numerical score to you. And then one of the executives said, hey, in addition to that core idea, I'd like to also test the idea of money-saving offers. Maybe I get the cat coupon, I don't have a cat, but can I raise my hand and say, hey, I actually have a dog? It'd be great if you could send me dog-related coupons like for dog food. So can I opt in for relevant money-saving offers? That was, he wanted to test that. He had hypothesis that people might want to compare their shopping and spending habits and patterns with other people. Am I spending more on dog food than everybody else or not? Social networking was hard at the time. Social networking. Just pure solution space. Just put the feature in there, right? It wasn't even clear why. Okay, sure. The other executive didn't care about any of these things. He cared mainly about that main benefit, but then he also wanted to test the secondary idea of what if we just help people like reduce the overall junk mail that they get. And then we said back to someone else, hey, what you said earlier is like, hey, if we're like saving all this paper, maybe we can make some like green benefit save tree. And I got to this point and said, okay, is everyone's idea on the whiteboard? And they said, yes. I said, okay, now great. Now it's time to go to the next step, MVP. This is way too big for a single MVP. This is too much stuff. So what I did is actually created two MVPs. So this is where we basically took two feature sets and two sets of prototypes. So what I did is I took the core green stuff plus the yellow stuff, and we called that the marketing shield concept because it was like shielding you from junk mail. And because the green stuff was the most important, we took the green stuff plus the blue stuff and created a marketing saver concept because it was trying to save you money. So this MVP had these features. This MVP had these features. And we created two sets of prototypes. They look pretty similar. The look and feel was pretty similar. Here's an example. Next step is create your UX prototype. Here's an example of the main page. I'd call this medium fidelity. It has some colors. It has some logos. But we didn't obsess about making it look perfect like that, right? Medium fidelity enough to get feedback on. And then when you clicked in, this was the page you clicked in on. Each of the key features was like one of these boxes. It had like an overview. And then when you clicked on learn more, that's when you went to the full page that had everything about that feature. So that's how we avoided doing two completely different UI designs. We just mixed and matched those boxes. If you were in the saver concept, you got the saver boxes, the boxes for the saver features, or if you're in the shield, you got the boxes for the shield. So I moderated people through the clickable prototypes. What did we learn? That same diagram now color coded red, yellow, green. I should mention like throughout the interviews, there were tons of questions and concerns and comments, right? No one was really excited about this. So green doesn't mean like, oh, yeah, they were ready to buy this and go for it. Green meant we feel like we have a good handle on the questions and concerns that were raised and that we can address them and that behind there, there's enough potential to create customer value that we think it's worth, potentially worth pursuing. If we pursued it, there would be some customer value there. We did. You're right. We didn't go through that. Yeah, we basically, we could talk about it, but we could go through that. Yeah, we did. I skipped that. Good catch. Yeah, I did. Yeah. Yeah, and they can talk about that. I mean basically like, and we'll get to that in a sec as we evaluate it. So green meant that, you know, hey, that was basically what green meant. And yellow meant that there was kind of like low appeal, wasn't really exciting for people. And red meant like they were just confused or didn't like it at all. Right? So the number one thing is, is there any green? There's no guarantee that you're going to have any green, especially the first time you went. So luckily there was some green. We had some green here, some green here. The second thing is, was any of the green in the core idea that we were testing out? No. So it's like, thank God we tested something else. This happens all the time. You go out to the market and the customers with hypothesis that you think this is a big problem I'm going to solve for you. And you talk to them and they go, actually, I don't really care about that. But let me tell you about this other thing, right? Sometimes they guide you, sometimes you pick up on it, right? That's called basically like a pivot. Happens all the time. Does anyone here use Slack? Slack did not start out as a chat platform. Slack started out as a gaming company and they were working remotely and they developed the Slack tool so that they collaborate effectively and the game didn't really go anywhere and all of a sudden they said, hey, this is a great collaboration platform. So that's what it did. Flickr, same thing. It didn't start out as a photo site. So that happens all the time. So there's a lot of value just getting in-market and talking to customers with your initial hypotheses so that they can then guide you, hopefully, in a better direction. The second thing is, so there's no green in the core area, the other thing is the red. I'm just as proud and happy about the red, right? Do you guys know what a marking score is? No? I don't either. We didn't define what it was, but I know it would have taken, it would have required us to sign up for an expensive third-party data feed because we didn't have the data. We don't have any data on you. We have to sign that up. It would have taken a lot of engineering effort to design some algorithm and come up with some crazy score thing and write all that code. And it would have taken a lot of time and money to educate customers about what the heck a marking score is. But guess what? Low value, low importance. We don't have to waste any of those resources. So red is just as helpful as finding green. And you talk, you know, here people say pivot, we started here and we decided, are we going to pivot here? We're going to pivot here, right? And this is where it kind of came back more to the value prop and competitive analysis. We went here for a few reasons. One is the company had an existing product, as I mentioned, and this was more consistent with their brand. This, for money-saving offers, would have required a lot of time to do a bunch of biz-deb deals and partnerships so our time-to-market would have got pushed out. The other thing is that we're back to the competitive analysis. There are already people doing this. And back to the value prop, we couldn't think about how we were going to be better or different than anybody else here. So for those reasons, we pivoted to reduce sunk mail. And because we had done zero coding, we didn't shed a single-tier drop or get upset about just throwing away those old prototypes and starting from scratch with a blank sheet of paper, right? And so what we did is we said, great, I had like six pages of notes from the marketing shield concept interviews, and we just made sure that the new prototype addressed every single comment and concern that we heard, right? So we pivoted to junk mail freeze. Here's the new mock-up, right? So we did a quick report.com, completely gone, you know, because again, we didn't code. When you code, it's like, well, John worked really hard on that stuff. Can't we salvage that somehow? You get emotions attached to it. Also, as a technical person, it's like, well, they start building their data models and their APIs, and it starts to put constraints on what you can do. It's like, oh, we can't do that because we built the database this way. It's like, okay, well, so anyway. But it's just also like peeling in on you too. Like, why learn all kinds of things? I learned in talking to customers that not all junk mail is equally annoying.  And it's the financial-related junk mail. I didn't know that going in. People told me, like, oh, yeah, I get these cash advance checks and pre-approved credit card offers.  I'm like, well, why? Like, well, I live in a house. My mailbox isn't locked. I'm afraid somebody can go in there and grab those things and take money from my account or do identity theft or something. So once we learned that, we put that front and center. The second time around, when we'd test it with the second group of people, they would hone in right there and you could see them kind of nodding their head and like their temple kind of, their veins starting to go because they're thinking about it getting upset, right? Other things I didn't know about. You know, and before I jump into the prototypes, I always do like some amount of discovery and just ask some questions. Like, hey, tell me about junk, what you do with your junk mail. What's your mail?  Here's what I do. Every day I come home from work. I go to my poster. I go to my mailbox. I grab the sack of mail. I go straight to my shredder and I go shred, shred, shred, shred. Does anyone here do that? I didn't do that. So I didn't know, right? But these guys do this. I'm like, wow. How many minutes do you spend shredding? There's probably about five minutes a day. I'm like, the mail comes six days. That's 30 minutes a week. That's 1,500 minutes a year. There was a whole save time benefit that we didn't even have in our problem space map, right? So it's peeling the onion and they're finding some new benefit. So we put that right here. Hey, spend less time shredding mail. So then the next round of people saw that said, okay, that sounds cool.  We had like, you know, saved trees up here. Multiple people said, well, how many trees are you going to save? So we put 43 trees right there. So we just knew it didn't, right? So again, we just made sure it answered all the questions. And the second time around, the second time around, there were still some questions and comments. It wasn't perfect, but there were a lot fewer questions and the magnitude was a lot less. And both times I asked people, I didn't mention this, but we said, hey, would you pay 10 bucks a month for this service? We asked both times. First time, nobody had any interest in doing that. And it's always a little iffy if you ask me. If they don't have to actually pay you 10 bucks or bust out their credit card, what people say they're going to do and what they do, you've got to be careful with. But the second time around, what we heard pretty much everybody say is, hey, look, I need a 30 day trial. But if your product does what it says, I would gladly pay you 10 bucks a month. So it was like night and day from the previous round. And then the other cool thing that gave me confidence was after the test was done, we paid people for their time to come in and talk to us for an hour. Here's your check. Thank you so much, John, for coming in. Almost every single person in the second group was like, so is this product live now? Can I go use this thing? We're like, no, we're still working on it. It's not quite ready. They're like, can I give you my email? Will you contact me when it's live? No one was doing that the first time around. So that gave me even more evidence that we had kind of iterated and improved product market fit. So I was pretty excited about that. All right, so sum up the lean product process. You want to start out by getting clear on who your customer is. Use importance and satisfaction to identify which of their needs are underserved. Use the Kano model to get clear on how you're going to be better and different than the competition. You don't want to overbuild before you realize you're going in the wrong direction. So figuring out what your MVP feature set's going to be. Using UX designer creator prototypes, you can get out and test with customers. With the goal of iterating through that loop that hypothesized design test learn loop that I mentioned with the goal of improving product market fit. So that's the lean product process. I mentioned that I run a meetup group. Our next meetup is actually December 12th at Intuit and Mountain View. Our speaker is going to be Gib Biddle. He used to be the head of product at Netflix back in the day, a very talented product person. He's actually spoken to my meetup before. And his talk is going to be about how to translate product strategy and execution. A lot of companies struggle with product strategy. Or if they have a product strategy, they struggle with how to actually execute on it. So that's what he's going to be talking about. So if you can make it, it's like 20 bucks and we serve a meal. It's a pretty good crowd. We should get about 100, 150 people to come. It's an Intuit, so it's good. And then I'm actually, if anyone's interested, tomorrow in San Francisco, I'm running an all-day workshop. So if you like the content we talked about today and you happen to be free, basically go really deep with a lot of group exercises where we pretty much cover the whole book in a day. And I have a special 20% code for the product school folks. Anyone here in the room can use that. So if you're interested, take a picture or come talk to me if you have any questions about it. And I know we did some questions during the talk, but I'm happy to stick around for a little while and do more questions. Get your tweets in. If you want to chance to read the book, we'll do some questions. And then we'll break and I'll figure, see who got the book. And then I'm happy to stick around for some more questions. And then here's again my contact info. I put my slides. There are other videos of other talks that I've given at danolson.com. That's the Meetup URL. There's that. And I'm happy to connect with folks on LinkedIn. Just don't spam me if we connect, but I'm happy to connect. And with that, happy to answer any questions you guys have. Yeah. Yeah. Okay, cool. Nice. Nice. After you link in, you can unlink with somebody. Yeah. Yeah. The key, what most people do when they do a competitive analysis is they use features. If you notice in my table, I wasn't using features, I was using benefits. So even like compared to say pen and paper, you can do apples and oranges. That's the thing about the market, the needs. Again, remember the music, portable music example I said, right? The desire to listen to music on the go is the same even though there's different products, right? So TurboTax and pen and paper, you could say like, how long does it take to complete, right? And one is like two days and one is like eight hours, right? You can quantify it. You can say like, you know, how likely are you to make a mistake, right? With TurboTax, it's pretty unlikely with pen and paper by hand. It's pretty likely. So there are ways to articulate why TurboTax is better than pen and paper. There better be ways that, you know, obviously it's better because more and more people have been adopting it over time. So the key is when you do your value prop grid is to make the rows be benefits, not features, to be in the problem space, not solution space. And what happens all the time is you just see if somebody comes in or somebody comes in, it's like, they have feature X, we got to build feature X, right? That's being solution space centric, right? And the assumption is, well, feature X solves an important problem for customers and that other company that built that is so smart that they figured that all out and let's just copy them, but they probably didn't. They probably don't know why and you can probably build a better feature that meets that need better than they did. So just don't copy for the sake of copying. Understand what the underlying problem is and then you can probably come up with a better way. I do that all the time with my clients. We'll see someone, a competitor who's clearly trying to address the same need. They do it a certain way and there's some good things about how they do it but we find ways to make it even better. So, yeah. Talk about what? The shield and saver concept? Oh, yeah, all it was is just like I said is two MVPs instead of one. It's a little more complicated but basically we just... We took these benefits and these features and said, okay, with this shield MVP, MVP one, we're going to address those things and to test these other ideas, we're going to create a separate MVP, MVP two that has these things and these things. That's all. Just two sets of functionality and then we created two sets of prototypes from that and we got feedback on each of those. Yeah, that's right there. That's what we're testing. Those are one, two, three, four, five, six. This MVP was testing those six things. This MVP was testing one, two, three, four, five, six, seven. Those seven things. So they were both testing these four. It was in both of them. I mean, usually you just want to have one MVP. It was kind of complicated what they did. So we could have just said, you know what, let's just start out with pick one and do that but because we're doing prototypes we could do it all very quickly. Yeah. Yeah. Yeah? You're getting evidence to support, to give you confidence that what you're about to build has product market fit. Right? Yeah. And I didn't really, did I mention A-B testing at all as a tool? It's hot and sexy right now, right?  Because what I feel like when you're building a new product it's really about understanding the users in the problem space and it's not about A-B testing. The other talk that I give is, okay, once you do this and you launch your product and you want to see where you're at with product market fit and you want to optimize it, it's all about analytics and so there's two chapters in the books on that and I have another talk on that. And it actually turns out that there's really only one metric that tells you about product market fit. Conversion doesn't tell you.  It doesn't do anything and you don't come back. Yeah. Retention is the measure and if you look at my book and you look at the talk, I get very detailed and specific about retention rate and how to measure anyway, yeah. How many people do you talk to? These, I think the first wave was maybe 20 and I can't remember if it wasn't 20 per. It was like 22 total. Maybe like 10 or 11 per. Yeah, I recommend 5 to 8. Yeah, yeah, what happens is, and so yeah, that's, so what we're kind of getting at here is like the, wow, Dan, you only talk to 5 to 8 people. How do you know it's just the significant and then the other, you know, the people's like, you know, so that gets into it. But what happens is as you talk to 5, you know, 6, 7, 8, like your discovery rate of new issues goes down. Like you're just hearing more of the same thing, right? So that's the real measure. It's like, hey, as long as you keep hearing new problems and issues you didn't know about then keep interviewing, right? But what'll happen is you usually about 5 to 8, if you've done a good job. Now you may have noisy data because you didn't do a good job on having a cohesive target customer definition and you realize, wow, we actually have three different populations of target users so we're getting three different answers and you need to revise that.  And then what happens is you need to like remove those problems to get to the next layer of problems, right, right? And the funny thing about this, so I actually did this when I did my startup and I did like 80 one-on-one user tests while we were in private beta before we launched the TechCrunch and so when we launched the TechCrunch, they have like the judge panel on there, Marissa Meyer was one of the judges, she was at Google all the time, and she complimented her UI and it was because we had done all this user testing, right? But when I first did the first set of user tests, you know, there were usability issues and people weren't quite sure what to do and so they said, you know, so we learned those things. The funny thing about that is when we fix those issues and then we test it with the next set of users, they didn't know what the old UI looked like, so they didn't go, hey Dan, good job fixing those UI issues. They just didn't make the same complaints that the previous people did. So you get this weird progress where it's just the lack of the complaints you used to get represents progress and starting to get new compliments represents progress, so. Yeah, yes. Yeah, I mean basically, I talk about this in the introduction to the book, like to have a successful product, you know, you need to have what I'm calling product market fit, you need to have a big enough market. Say I do this and I'm like, wow, these people loved it, but there's 10 of them in the whole world. I have a market size problem then. I've achieved product market fit, but my market is small, right? The other thing is maybe people are only willing to pay a buck, unit economics, a buck for this product and it's like, gosh, it's going to cost me five to produce it, right? So you actually need not only product market fit, you need to basically have favorable unit economics. You need to have a big enough market size and you need to be able to like reach those people in a cost-effective way. So you need actually all four of those things. I'm not saying those other things aren't important. My book focuses on what is typically the biggest risk is achieving product, in my mind, product market fit. If you feel there's also a fifth risk, which is technical risk, potentially, right? And if that's the biggest risk, then don't even waste your time on this. Like just see if you can even do it technically, because if you can't do it technically, it doesn't matter. So what you need to do is figure out what's the biggest risk and assess that enough. And so what I recommend on the unit economics is just some kind of high-level spreadsheet back in the envelope. It's like, how much do we think we're going to make per user? How much do we think it's going to cost? And how many users do we need to get to for break-even kind of a thing? A simple analysis like that? Well, right is all relative. You're going to start out, it's going to be iterative. You're going to start out with some initial guesses, right? You don't know what people are going to pay you, but you can just say, hey, I think we can get nine bucks and it's going to cost us seven bucks and our fixed cost is this and our available cost is this. So what you can do is you can say, I think based on these numbers, we need a million users to break-even. And then you go to your marketing plan and say, okay, what's our marketing plan to get a million users? That's where it all starts to fall apart. I've helped with startups with this all the time, but like, oh my gosh, this isn't going to work out. So it's good to know that sooner rather than later, just like it's good to know the features that you picked or the UI design you did isn't going to work out.  My book helps you mitigate the product market fit risk, not the technical risk, not the business risk per se. Yes? Mm-hmm. No. No, it's also iterative. It starts out very like a sketch. It's just like a UX design. It starts out with, you know, a couple of salient attributes that you think matter. And as you talk to people, you'll learn more things and cross out ones that are wrong and add additional ones and be more clear and realize, oh my gosh, it turns out not all SMB people are the same. We need to create two now. There's this kind of SMB person, this kind of SMB. Yeah. So it's perfectly fine to them start out low fidelity and get more detailed as you peel the onion and learn more. Yeah. Yeah, in fact, people call that a proto persona. Just make it like acknowledge that it's okay for it to be minimalistic. Yeah. Yeah. Yeah, that, in my mind, that's what I was saying early on. It's like personas, it's easy for people to misuse them. And it's, you know, like enthusiastic designers to add all kinds of stuff and isn't based on evidence. It's pure conjecture and actually isn't even relevant to making product decisions. So I guess as you're building a persona, you should be saying like, okay, I'm about to add this attribute. You know, what evidence do I have to support this attribute or not or acknowledge you don't and say, hey, we have a high degree of uncertainties with purely hypothesis and secondly say, would this actually help us make it product decision or not, right? So you don't end up putting in their astrological sign because it doesn't really matter, right? Unless you're building a astrological product, I guess. Yeah. Yes. There's so many three-letter variants of MLP, minimal lovable product. Yeah.  It's like, I think that it's kind of like the persona thing. It's like people are pointing at MVP being misused and saying, that's dumb. This is better. It's just being misused in my mind. People don't really get it. So, or if they want to emphasize a certain attribute, like if they feel like their organization isn't doing enough delight, but usually I've found that people don't truly understand the essence of MVP and maybe some people in the org do, but not enough people do. And why would they, right? If you've never been an organization that's done MVP successfully, why would you know, oh, that's how we do it. And there's a lot of judgment involved. So, it's hard. So, no, because it's a misunderstanding. We're going to get there eventually. This is not the end of the race. The race is not over with your MVP. It's the start of the race. You want to make sure you're running in the right direction before you really start running fast. That's what it's all about. It's like directional. I would view it more as like you're picking a direction. Does that make sense? Like it's, no one's saying you're going to win in the marketplace with an MVP. And so, it's really about risk and uncertainty mitigation. That's all it's about. So, and some people, you know, there's a quote by Reid Hoffman. It's like, if you're not embarrassed by your beta when you launch it, then you waited too long. So, some people just have, they can't get over this perfectionist mentality if we can't launch it yet. It's not good enough yet. Yada, yada, yada. You know, I would just say a lot of successful companies.   That's why I showed that diagram. That's why I showed that pyramid that I showed, right? That's why I showed this pyramid. This pyramid. Right? That's why I showed this pyramid. Can't be junky. But some people try to bite off like half the pyramid with their MVP, right? But what you're saying is someone's saying, you know what, no, no, we need more features. No, no, it's got to be better. No, no, no, no. You can convince yourself that you got to build something before you can launch it, right? And it's tough. It's tough to figure out what's enough versus not enough, right? And some of the tools that you can use is like a private beta is a way that you can kind of get feedback without having it be out in the market so people go, oh my gosh, look at that product, right? The app stores actually make it a little, they used to be tough because you launch in the app store with the substandard product and you get reviews and those reviews never go away. On the app store, day one you get a review, it never goes away, right? So that's why there's like, well that's one way. There's also, they acquired test flight. There's a lot of ways to do it. So the whole point is there's a perception of risk that isn't truly there if you get creative about how to mitigate the risk of the perception of your product and truly it's all about just getting early feedback and coming up with creative ways to do it. So that's what I would say. Yeah. I think the number one thing is to illustrate the trade-offs. It's like, yeah, we can wait, but that's going to add three months. And that's three more months before we know whether we're going in the right direction or not. That's three more months that our competitors are going to be building features. So this is the easy thing. When the workshop tomorrow, I do this MVP exercise. And there's a feature that everybody, most people in the room want to be in the MVP. And the reality is it doesn't really need to be in the MVP, but everybody wants it to be in the MVP. So I go, who wants it in the MVP? Everybody raise their hand. And I'm like, great. Engineering said it's going to add four months to our schedule. Who wants it now? So if there's no trade-off, then why not? Sure. Give me all the pie and dessert in the world. There's no trade-off. There's no trade-off. So the key number one trick is to make the trade-offs transparent. So we can do that, but it's going to reduce our time to market by this. And by the way, by our engineering team working on that stuff, here's what's going to get delayed. The other stuff in the queue is going to get delayed. So that's the trade-off discussion. The other thing you can do is try to get out of that cycle and say, why don't we test the wireframe without it and see if anybody complains? You have a hypothesis that we need feature X in there. Why don't we test that hypothesis by not having feature X in this prototype and see if anybody complains? So instead of arguing about, no, you're wrong. I'm right. Elevate the discussion to, you have a hypothesis that X is true. I have a hypothesis that X is false. Let's try to create a way to test that in the cheapest, easiest way possible. So those are some ideas. Yes? Sure. Yeah. Yeah? Yeah. Yeah. It is harder. It can be harder. There's, again, it's kind of like getting creative and thinking out of the box. So like when Palm Pilot came out, they actually like made a physical dummy thing that they would use to see if they liked it, right? And that was level one prototype. Level two prototype was they had this little frame and they had paper and they would just, rather than doing any screen design, they had like a little paper and they're like, okay, you tap this and they would move the paper to the different screens. Those are some examples. 3D printing helps a lot. So with form factor stuff, 3D printing can help a lot. Other than that, it's hard. Like what my main advice is get really clear on the benefits. Like, you know, if you're going to outperform someone, why is your hardware going to be better than the other hardware? Get really clear and then ask customers, you know, it's not as good as showing them something that is better. But if you say, hey, if our router was 10% faster or whatever, like, you know, try to get at that.  Why not? There's no trade-off, right? So there are some techniques you can use to kind of like test different levels and see which one has the most sensitivity to that, you know. Sure. Kickstarter, sorry, you're not paying. Kickstarter is a great way for hardware, right? Then it goes back to the economics and the market size. You can see before you do anything if anybody even want this thing, right? Yeah. It was a nano. They hacked the nano for it, I think, right? Yeah, sure, yeah. Yeah. Sell the dream, and then fake it till you make it, you know. Yeah, well, I mean, they eventually did pretty good, right? Yeah. Who had a pebble watch here? Anybody? No, we aren't. Yeah. Yeah, so, yeah, Kickstarter is a great way to reduce the risk and make sure there's enough market demand for what you're going to do.