and I'm with Octane. Octane is a big supporter of the Entrepreneurs Forum here at CalIQD2 and also a big supporter of the Tech & Portal. I want to welcome you to today's event. I'd like to introduce Frank Cooper, who's over here on your left, and Patrick Laskowitz, who's here on your right. They're going to be talking about their vision right now of the kind of the end of classic startup marketing as we know it, and the beginning of a brand new age, hopefully. And I just wanted to give one quick pitch here for Octane that tonight is our regular free first Thursday networking event. We call it FIRSTY, and it's pretty popular. We get about 150 people that attend each month. And this year we're moving FIRSTY around every month, and we're doing it at the old Dubliner, which is in the New Tustin Marketplace that they've opened up over there. And we're doing it in conjunction with the Harvard Business School Alumni Association of Orange County. So a lot of the members of that association will be there tonight. It starts at 5.30, goes till about 7.30, and you are all very welcome to attend. So with that, I'd like to hand it over to Brandon and Patrick. Thank you guys very much. Thank you very much. So anybody here have heard of Lean Startups or Custom Development? How about the buzzword pivot? Anybody been reading about the word pivot? I actually throw that one out there because almost everybody has been aware of the buzzword pivot. But in this group, that's great though. I love to talk to people that haven't heard of this stuff before. So Patrick and I are super passionate about Lean Startups and Custom Development. So what we're going to do is we're going to talk to you a little bit about why we're passionate about it, what are Lean Startups, and then how that might affect the way you do market assessments. Do you have anything to add? Sure. Real quick, who here is actively doing a start-up? Anyone with one person here? Three, four, maybe five. Got it. The rest of you, maybe tell me a little bit about some of your backgrounds or what you guys are working on. I'm the sales and marketing director for a small company. Got it. Is that... I'm working with him as an independent sales rep, always looking for ways to tune up the wine process. Got it? Anyone back there, sir? I'm going to pick on you. I'm hospital, I'm an independent consultant, and I also teach at UCI Extension. Great. And the gentleman behind you? I'm Chris Perra, I'm a product developer at Lean Startups. Okay, I love a product development guy too. Good. So, can we just jump right into it? Yeah. So, the start-up ecosystem, the start-up environment is changing drastically, right? So, the amount of money it takes to develop a product now is extremely low. So, you have social media marketing that means that reaching a broader audience costs a lot less money. So, primarily if you're working in software, internet, wireless, hardware solutions for the IT as well. There's some markets, obviously, clean tech and some life sciences where it's still expensive to build a product. But even with those, the price of building a start-up company is drastically lower than it was even five, ten years ago. There's a ton of money out there for investment, and it's not just in the venture capitalists anymore. You have incubators, accelerators that are willing to start with very small investments to get people going. There's a lot of angels now in the micro-VCs. You've maybe heard about super angels. And so, there's money flowing to start-ups really like never before. This is also a global phenomenon, so it's not just a U.S. that there's these entrepreneurs coming out of the woodworks in order to build new products. It's happening in every country. Patrick and I, hopefully, have an opportunity to travel to Central or South America this summer, traveling to Canada. People that we see tweeting about lean start-ups and customer development are literally all over the world, so it's pretty exciting that way. Entrepreneurship is very social now, so it's not only in terms of your marketing, but it's also going to meet-ups. It's also the resources that are online. So the amount of mentoring that goes on, the collaboration between entrepreneurs is like it never was before. I used to go to networking meetings 20 years ago, and then it was really all about me. I wanted to tell you about me and give you my business card. And today, a lot of these meet-ups are completely flipped. It's like, tell me about you. How can I give you value? And it's really a great collaborative environment. And what it means is that today's entrepreneurs have a lot of peer support as well as a lot of mentoring. Even classes like this, right? So what that makes is today's entrepreneurs a lot more savvy than entrepreneurs have been in the past. So they're not more intelligent, but they're actually coming to the game with a lot more knowledge, with a lot more help, a lot more understanding, the ups and downs, the roller coaster ride of being an entrepreneur. So it really is sort of this new environment. And what we're seeing is that there's now hundreds of thousands of entrepreneurs. It just amazes me how many people are coming up with ideas and actually able to go launch their product because it doesn't cost that much. People graduate from college. People are actually in college assuming that they're going to be an entrepreneur. That they're not going to go work for a business for 10, 20 years and then become an entrepreneur straight out of college. So Patrick and I both participate in these lean machine, lean startup machine weekends. Has anybody heard of a startup weekend?  Voluntarily is what they do doing on their weekends and they build products over the weekend. And so the traditional method is you sit around and you build products that nobody cares about, right? Well, the lean startup approach to that is you actually take that same team of engineers and they get out of the building. And they do that in order to learn whether the products that they're going to build over the weekend, whether anybody might want them. Why I bring this up now though is that what's amazing to me is to see the number of quality ideas that are pitched at these startup weekends. You get 50, 60 people in a room, half of them stand up and pitch their ideas.  They're not all great ideas.  So to me it's just this phenomena of entrepreneurs. So because the costs have changed, because there's all of this money, the power in the relationship with investors versus founders is now shifting to the founder side. So it's no longer that you're going to have to stand in line to go meet a VC on the VC's grounds. You get to start building your product and prove your business model with low amounts of money, perhaps seed funding. And then the venture capitalists are going to come to you. My personal belief is that this is driving, VC's are going to need to change their model so that they're no longer just going for the big win, which is kind of their traditional strategy.  So a lot of the VC's that we're seeing, the micro VC's are really looking for the smaller wins too, which means the mergers and acquisitions that get founders a life-changing experience. They sell their company for 15, 20, 50 million dollars. That's a life-changing occurrence. So again, in my belief, if these startups are building their businesses in the new way and this new startup methodology, then there's this potential for our economy to develop what I call an innovation machine. So it's my belief that we're shifting away from services. Services are being outsourced now. So what is our comparative advantage going to be as an economy? And it is going to be our entrepreneurship. It is going to be entrepreneurs that are increasing productivity of individuals and businesses through the new products that they build. And so what we're having now with hundreds of thousands of entrepreneurs each targeting a market, carving out a niche and building products that are suited for that niche. I really believe we're creating this innovation machine. And I think the methodology to do it is through lean startups that we're going to say why. Real quickly, I just want to make clear that how we're positioned is radically different than what happened in the Silicon Valley, the Dot Com boom, 10 years ago, 11 years ago and now. Where it was all about the $100 million exit. It cost you $5 million in six months and 10 guys to build what would now take you one guy and some out of the box software essentially. So in 10 years I think we've had orders of magnitude change in terms of how to build scalable startups in terms of the knowledge needed, the knowledge transfer. Low cost has, again, orders of magnitude differences. So there's three popular philosophies that get a lot of play on Twitter and the blogosphere. This is the way I summarize. The first one is Be Like Steve. So instead of, you know, the Be Like Mike, you know, they say the ad campaign for Michael Jordan. And to me, the analogy was as a high school counselor actually goes and talks to these young student athletes and says, yeah, you know, really what you should do for your career is go be like Michael Jordan, right? So now we have investors out there. They're holding on to their dream of the $100 million win or whatever.  So that's strategy number one, be like Steve. Strategy number two is almost the opposite. It's the just do it faction, right? And so the just do it faction, my sports analogy for that is everybody could be an Olympic athlete if you just try hard enough, right? So it's all about execution. So in the startup world, what that means is all of you can be, you know, billionaires and if you're not, it's just because you're not working hard enough, okay? The third philosophy out there is the lean startup methodology. And so what is it about lean startups? Why lean startups? So here's the two big ideas around lean startups. The first is that a vast majority of startups fail not because they can't build a product, it's because they have no market. So if you think about that for a second, the technology is advanced enough now that most companies, most products that you want to build are pretty darn easy to build. You can build internet products, you can build software, you can build wireless iPhone apps. There's not technology risk. Where there is risk is in the market. Do people care about what you're building? So again, this is actually deceptively simple. So we often say this or I often reference this. And then people go, their eyes sort of glaze over and they go, oh yeah, of course. But it's actually not of course, if this is true, then we wouldn't have the dot-com boom again 10, 11 years ago. So the person that advanced this idea is a guy named Steve Blank. He wrote a book called The Four Steps to the Epiphany. If you're doing any sort of product development or startups, you should go out and read that book today. It's called The Four Steps to the Epiphany by Steve Blank. And this is a big idea. Again, deceptively simple, massive idea and really has huge now sort of sizing effects, ripple effects in the startup world. Also a guy named Clayton, oh there we go, Clayton Christensen. Great. Very similar. So the second idea you want to keep in mind is that your business model, you're building a startup, it's very likely that some major aspect, probably more than one aspect of your business model is going to change at some point. So most of the products that we know and love didn't start out the way we know them, right? So does anybody know how YouTube started with their, they started out as a dating site. PayPal started out as microprocessors on the Palm Pilot. You have any favorite projects? Flickr was, I think Flickr was like a chat application in a game. Who else? There's also, you can take stuff offline. For example, Ypro, the Indian IT services firm, multi-billion dollar firm, started out as a vegetable oil processing plant. Right now they're multi-billion dollars. The point is that due to market conditions, right, the companies and startups have to be involved. Well, even when Christensen wrote this in the early 90s, he referenced Intel, which started out in DRAM before they were into microprocessors. And Honda actually made their foray into the United States by essentially inventing off-road motorcycles when what they were trying to do was compete against BMW and Harley-Davidson for Street-Bike. And I think Facebook is the same story, right? They started out as like a dating site in the harbor with guys. Right. The more you learn about these companies, and this is why, do I have this slide coming up? The why pivot is such a big buzzword out in our world now is because everybody's talking about, oh, you've got to pivot, you've got to pivot, you've got to pivot. And of course you don't want to overdo this, but it really is now sort of this internet meme is all about pivoting. There's also sort of a conspiracy, which is sort of a strong word, but there's a conspiracy. When you see successful startups that made it big, the marketing people, the PR folks go back and rewrite history. They're like, oh, YouTube is huge, massive now. And this was what it was going to be from the very, very first day we started YouTube, right? And I'm not trying to pick on YouTube. I think it's a great company or any startup. You see it, once the success has been achieved, everyone goes back and rewrites history. It makes it look like it was, you know, preordained. And also then says that's the roadmap for the future, right?  Right. So Grant, when you use the word pivot, are you saying that when you start a startup, you pivot on something, but then you expand into a much broader scope? Is that the context here or I'm not getting this... Okay, well, yeah, I'm introducing it a little bit early. We talked about it a little bit more in a couple slides. So when we get to that, follow up with your question if I don't answer it, okay? So here's the basic tenets of the lean startup then, right? So you don't really know what the problem is or the solution. You have good guesses as to what they are. Your business model is going to change. So what you have to do is go through those changes fast enough until you find what works and you have to do that before you run out of money. So that's what the concept of the lean startup is. Real quick, actually, right? So what I like about these big ideas here, and I really think they are big ideas, is that this was always a fundamental truth in sort of the chaos of startup land, but the changes that people went through are always sort of driven by panic and fear. So we try to do, you know, pet food, you know, online pet food delivery. Oh, it's really not working. What are we going to do next, right? The pivot or whatever you want to call it, the pivot or iteration, was going to happen at some point because, you know, we have incomplete information about what the market wants and how we deliver that, et cetera, et cetera. It was going to happen anyway. This is actually embracing, my mind embracing sort of the reality of how startups actually work. So maybe an overly complicated slide, this is real quickly, basically, lean startups are about combining what's called agile development with customer development. Agile development is an engineering term for developing small amounts of codes, small amounts of features in a very short time and getting that out in front of the customers to validate that it works and that people want it combined with customer development, which is a similar principle to test your marketing. And it's an integrated approach since your, people out speaking to customers can bring that back into the engineers. The engineers can build it, so then you actually have product in front of customers and that's testing it and your solution team or your problem team that's out there figuring out whether they've got the right idea nailed with their customers takes that product out and so it's a continuous loop on both the market development side simultaneously with the product development side. I keep wanting to juxtapose this, I don't want it to be a straw man, but previous people often built stuff, six months in a cave coding an app. Not even six months, but two or three years. And then presenting it to a customer, like, what do you think? And then wondering why they didn't get traction. So the old style still being taught in a lot of places, you write a 40 page business plan, put in your requisite hockey stick revenues, then you go and you present that to your investors and if you get somebody that will actually invest on you, you've essentially signed a contract that you're going to deliver that these revenue targets that you pulled out of the air. So then you got your investment money based on this hockey stick and then you spend two years building the product towards the end of that process. You start hiring your sales and marketing people and you do this big launch, right? Big marketing launch, get the product out there. You maybe make a couple of opportunistic sales. You sell to the brother-in-law of one of your board of directors and you have this little bit of success and maybe you hit those really super low first quarter targets and then suddenly you start seeing in the second quarter, it's flatlining. In today's web world, we call this the tech crunch bump. You get your product out on tech crunch and you get tens of thousands of visitors and then as the months go, not only does the number of visitors dip, but the number of people that you originally got aren't no longer using your web service so your retention is really low. What Lean Startups tries to do is let's flip that.  Patrick and I like to say this is not... If you can just get these philosophical principles in terms of Lean Startups and customer development, you're good to go. You're advocating a step-by-step approach or a very rigid methodology. So basically, the philosophy here is question your own assumptions. So everything that you think you know about your startup are actually guesses. So you want to say what you think you know as forcefully as you can because you believe in yourself and all the rest and I agree with that, believe in yourself but be skeptical of your ideas. This is a big idea. The author of the forceps of Tiffany is called Get Out of the Building. What he means to say is I have a great idea for I want to sell cupcakes and I'm going to have these cool cupcakes. They're going to be jalapeno pumpkin flavor cupcakes. They're going to be amazing. Instead of me building the factory to build my jalapeno pumpkin cupcakes, why don't I go out and try to sell a few... You've been watching the cupcake board. It's actually a cake cupcake. That's why I keep using it. The point is why don't I try to do some validation before I actually scale. Again, sounds deceptively simple. If it were that simple or if we had understood this 10 years ago billions of dollars wouldn't have been wasted. So you guys maybe remember Webvan from the dot-com boom. They scaled to build home delivery online ordering for groceries and they actually didn't validate that idea. At that time they wanted to order groceries online. So Get Out of the Building and actually validate this idea. The last sort of steps is iterate on these ideas. So don't do it just once. I go to Brandt. I try to sell him some cupcakes. He goes, you know what, I don't like orange flavored ones. I like cherry flavored ones. I'll pay twice as much for those because I can sell them to my... I firmly believe that fruit and dessert should not be mixed. The point is that you want to iterate through these things. You want to iterate and cycle through these things. This Get Out of the Building is Steve Blank's shorthand phrase or get out to your client's site and figure out their reality. There's other sort of acronyms that say the same things. Nothing interesting happens in the office. I've heard that before. People have thrown that around. In the lean manufacturing philosophy there's actually a Japanese expression that means essentially the same thing.  So what does this mean in terms of market assessments? So traditionally there's talked about top-down approach and a bottom-up approach. So we'll talk about this real quickly. To me the first thing you want to ask yourself is what is your... what is the objective that you're trying to do with your market assessment? Is it because you're putting together one of these 40 page business plans and you need to come up with your hockey stick? Are you trying to raise money so you need to convince an investor about how much money you're going to make? I would say that the first reason why you should be doing a market assessment is to figure out whether you actually want to build your startup. And you should think about it in terms of if in the end nobody wants my product, how bummed am I going to be that I spent six months a year building the company, right? So the cupcake's top-down approach. Right, so this is pretty typical if you guys have ever seen a pitch deck big massive number $9 billion for cupcakes or for online food ordering or for online dog food ordering whatever it is, right? This is pretty typical I'm sort of what's the word I'm looking for? Absolutely. Anyway pretty typical, usually Gartner or Forster are one of these big research firms, right? You quote them all the time. You can buy reports, right? When the Gartner guys went out, did a bunch of research then you can quote them. Feels pretty good I think they're pretty skeptical about this I think they do a good sort of ballpark figure if it's a massive market or small but you know how do those guys arrive to that those conclusions right may not be suitable for your startup. So you know these numbers can be useful when you're demonstrating you have a big vision and it's a big vision inside of of a big market, right? So if you're trying to raise money investors will want to see what is sort of the dollar figure that represents the great wide world that you're actually trying to tackle but it's also a dangerous number, right? So it can be very self-delusional the number of decks that I've seen where they've had sort of a massive number up there, you know I'm not sure it's quite as much as a you know and then they'll say well you know conservatively we estimate that we're going to tackle one percent of that market, right? So one percent is a conservative number, you know but one percent of a bazillion you know that's actually still a big deal so it's pretty easy to get into a game of self-delusion there and you actually haven't demonstrated anything about being able to tackle that market so while I'm going to say I'm not going to say I will never do this I am going to say that you need to use that number wisely and you know I don't think that there's an investor out there that hasn't seen you know deck after deck that you know has been told about their you know several billion dollar market so be wary of the top down approach so then there's the bottoms up approach yeah so this bottom's up so this is clearly you know how you back into another equally large number hopefully right so here I chose red velvet coconut jalapeno cupcakes they're $40 a pop in my little market I think I can address in one year there's a million people that are going to buy and I think they're going to buy 14 times a year right X Y times Z right we get massive market right that's really pretty simple you can again this is a very simple example much more sophisticated depending on what you're selling what your product is I intentionally chose cupcakes to keep it simple I didn't want to you know you're running a freemium business and you're going to be disrupting you know salesforce.com you know CRM stuff it's going to be a different sort of model but essentially the model is the same I took this from David Clair's start up Viagra how to pitch a VC by the way so if you are interested in you know Billy meets the pitch deck for a VC go see Dave's stuff right so if you need to Google start up Viagra have a VC or just take that show you are out right there not not very sophisticated I don't think there's you know we can all do simple multiplication here in division so the the advantage to an approach like this is is that it actually exposes all of your assumptions right so it your assumption on how much money you're going to make you know for a cupcake or for a dozen about $42 for one that's Newport Beach prices yeah so an assumption would be the price there's actually maybe even a line in between here which is you know what is your reach you know based on your marketing how many eyeballs are going to even get on your offer then there's a conversion number there right how many people are actually going to buy out of those numbers so you end up exposing a bunch of assumptions and the reason why investors like that is because they're going to take all of your assumptions and they're going to you know slash them by you know 50 to 90% and try to figure out if there's actually any valid business in there right it's sort of like they're going to look at it best case scenario by maybe using your numbers and then worst case scenario by slashing them by 90% and trying to figure out you know how how bad off you are at that point so we definitely support the bottom up approach and not only because it exposes the assumptions but it actually gives you it gives you an idea of what you need to be working on right so you can actually start segmenting your market if you're thinking about it from a bottoms up approach and basically keep in mind that if you're going to get an investment or you're going to build a business you're going to have to do this sort of financial modeling anyway you're going to have to understand who are your customers who are going to be your early adopters the people that are the first ones that want to try out you know radical jalapeno based cupcakes and how you know you're going to be able to estimate what's the cost to reach them you know maybe there's a specific blogs for jalapeno lovers you know and so you can start estimating how much money it's going to cost for you to reach those people so it really is a it's a financially based bottoms up approach to figuring out what's the reasonable amount of money it's going to spend to get to those people so then what you really want to get to is to the point that you've got to here's what your cost of acquisition is and here once they become addicted to these cupcakes that these cupcakes this is my lifetime value they're going to continue to buy you have the the basics of your business model you're proposed your guest business model and that's what the investment wants to see but like I said before it gives you things to test right you can start testing those market segments so we're saying do customer development to validate the segments and your other assumptions and the customer development is you actually go out and you talk to these people right you can do surveys you can do landing pages online where you're measuring clicks through on on ad words and this is all just for testing right and there's a bunch of online tools that you can use ask your target market dot com bunch of ways available online now for you to start interacting with your customers but all of those in my mind need to be corroborated with individual interviews with speaking to people to see whether there's anybody passionate about this this solution that you're bringing into the market and it also helps you divide those segments right what is the profile or the demographic of this person what is their social market look like what who do they talk to who influences them that's what you kind of want to learn about your different segments so your numbers are just guesses until you validate your business model your your market assessment is more accurate with the more that you validate and guess what most investments these days do not come before proof so if you've got a bunch of if you've got several startups underneath your belt you've had a couple of exits then you'll probably be able to get some money based on a good idea if you've got the pitch together and the investors do believe it's a good idea but if you're a first time entrepreneur you're just not going to get money based on an idea anymore you have to start building your product you have to start proving your market you have to start proving part of your business model any questions by the way the question is this you know if you go back to the root or roots entrepreneurs are like some of them I know are like software engineers and they create this biggest mouse track but at least they know is the market side they always complain they don't know how to sell they don't know how to size the market and then you're trying to say you know what you've got this bottom up thing and it has to be a rigorous model so you can check their assumptions so that's where the disconnect is well it's one of the reasons why we love that lead machine weekend that we were talking about because we get teams of engineers that were hoping they were going to stay inside the building hacking on products all day and we make them get out and go talk to customers so it really is engineers will spend a lot of time trying to figure out and actually invent tools so that they don't actually ever have to talk to a customer so all of those tools that I was telling you about usability testing dot com they're actually going to take care of all that customer stuff for me and I'll be good to go so it is it's a hurdle to overcome the fact is that engineers can now do a lot of really good marketing without ever talking to a customer they can so it doesn't have to be this you know fluffy Madison Avenue marketing that engineers aren't drawn to it's actually could be process oriented it's similar to using a scientific process so that actually appeals to a lot of these engineers and you know if you give up that side of the business if you give up the sales and marketing side and you go and hire you know sort of an old school traditional sales and marketing you're going to be spending a lot of money without seeing results so maybe some of those engineers have to suffer that pain before they go well next time I'm going to do it I'm going to do it a little bit differently this is embracing the reality that we've all heard the phrase like build a better mousetrap in the world will be the path to your door right this is embracing the reality that's actually bullshit right for the most part right and engineers are we all we like solving problems often people solve problems that solve problems provide solutions for problems that don't exist this is what we're getting at so let me give you another example so Segway you guys remember Segway when that first was going to come out right was going to change the shape of cities and you know urban landscapes no one ever bothered to check if people wanted to spend you know three grand for this kind of dorky looking scooter thing right and five thousand right no one ever checked now they pivoted actually into like the security you know the mall cop sort of a touristy business but you know but my understanding at the time was aimed at square at the everyday consumer and it was going to replace the car that was right that was a hyper round that was the PR and hyper rounder that's how they were positioning it guess what people don't want to you know drop 5k for that you know a few people that do the point is that the technology was awesome right remember at the time it was like amazing you stepped on it bounced perfectly like mind blowing technology doesn't matter is what I'm getting at you got to go out get out of the building and like test these assumptions you can have the best mousetrap ever if it's not really solving a problem so the message is the engineer around here has to just get out and do it because I friend right yes that the hiding in the old school you know sales guy paying 200,000 bonuses etc have not worked at least the guys I worked with you hire them once you figured out what you value a lot of stuff right so once you go again I know exactly what I'm selling I know how to reach my customers I know which cost to acquire them I know the messaging they respond to I know their substitutes then you go hire that silver tongued you know 6-5 you know sales guy who's going on the golf course and he just goes and executes that's when you hire that guy and there are actually like in the B2B world you can hire what we call the renaissance sales guy and that's a sales guy that actually does understand how to go out and learn the market so it's not like to throw all sales and marketing people into bus I've been a marketing person for a long time and I guess it's the differentiation between number one old school versus new school you don't want the old school just grabs you know from a their toolkit and goes and executes on the X Y and Z that they've always done before and that's sort of a dangerous way to go if nobody has actually figured out what the right methods are and you can spend a lot of money and burn through all your resources by doing that so there are sales people there are marketing people that you can hire that will help you to figure out what that road map should be what the sales and marketing road map should be and then you turn that over to your turnkey marketers and your turnkey sales people now it doesn't get away from the need though for the founders who hold the core assumptions of their business that they're the ones that need to go and talk to the customers to validate what those core assumptions are is there really a problem that they're trying to solve is there the solution that you have in your mind going to solve that problem is it going to likely solve that problem or does that sound appealing to the customer who supposedly has that problem there's really no way around understanding honing in on the market signal understanding that pain so in one of these lean machine events that we did in San Francisco a couple weeks ago a team was building like a photo sharing service and we're all like oh another photo sharing I mean we need another photo sharing service and what they wanted to do was actually they were going to use some I guess face recognition technology or look at past photos that these people had shared and figure out what are the good photos right so what are the good the high quality photos of a family and they thought or whatever pictures that were taken and they thought that the pros might like this market because they're taking you know thousands and thousands of pictures so could you actually develop a technology that will pick the good pictures out of that out of that lot the pros hated the idea they actually love the process of going through and selecting those photographs what they found though where moms were eager for that they actually they felt like socially they felt like bad moms if they could not produce quality pictures of their kids and so the one of the guys that was presenting this idea at the end when we were doing the finals was saying when he felt the emotion from this woman talking about how involved they needed to be in choosing the pictures and that if she could have help with that and that she could have these great photographs that was really important to her and she was emotional to this guy when they were talking and it was and that was the point where this customer development he had sort of dove in deep enough to understand the emotional connection that's the painting that you're trying to uncover right in the B2B world it's usually money driven can you increase my ROI can you expand my market segments a lot of these things are very measurable but in the B2C world it's often an emotional thing that you're trying to connect with and that's what you're trying to do with customer development is trying to dig in deep enough to understand the need or that pain or that passion actually exists and also real quick riffing on what Brandt just said about I want to go take a step back about sales and marketing right there's a time in this customer development model which the cooling that Brandt and I obviously drink from there's a time for learning where the founders are learning there's a time for executing and so I actually have a friend up in LA he's got a pretty prominent well funded startup and I spoke to him a few weeks ago and I said oh how are sales going are sales guys awesome we just close like 10 big deals high fives all around you know rock and roll you know cash is coming in week later I talked to Mike oh how's that going goes you know I just figured something out we're still at this learning phase those 10 deals we close they're all different customers meaning different types of customers they're all for different amounts and we're all doing something slightly different we actually don't have anything scalable repeatable and the sales guy was doing what the sales guys pay to do go execute close these guys and the guys a great sales person right so sales guy my friend and CEO had have a meeting and say hey sales guy we're gonna put you on a leash because we don't know what you're selling right and this guy's just closing deals but what they're actually searching for is a repeatable scalable model you see I'm saying so this is not against sales or marketing this is there's a time for execution there's a time for learning and so it's you know context dependent and I think that's a nuance a lot of people miss about customer development does that happen all the time? Absolutely sales guys go then from CEO when you do this short you can do it and now it's a big headache for the software guys to change whatever that industry wants to do so how do you find that sweet spot between you know the soft launch the beta side testing gathering feedback fine tuning it to locking it down doing the formal launch scaling up? Yeah it's a great question and it's like one of the most difficult ones to answer right so I was afraid you would say so I was going to say I'm going to give you an irritatingly philosophical answer about this which I know you're looking for actionable stuff and like I wish I had like the exact you know if you get rid of your brain you know exactly you know what the answer is assume I know it which I don't there's a concept called product market fit I don't know if everyone's heard of this concept it's when you found the solution that nails the problem you know the price that people are responding to or the messaging you know how much cost to acquire a customer theoretically the long the long term value of that is higher that's when you go okay now let's go get some money and now we actually have a we built a machine where you put in a dollar and it spits out three that's when you scale theoretically again this is sort of philosophical sort of understanding of this and this is kind of what people talk about this is actually popularized by Mark Andresi pretty famous entrepreneur in Silicon Valley he actually wrote a blog post about this a few years ago and sort of explored these ideas so it's nebulous right the way I sort of describe it is you know you're there when you wake up and you have a whole new set of problems so before you're there your problems are you know where are my leads coming from you know how can I optimize my conversion funnel um we're not reaching it we're not acquiring enough people to our website you know how do I build how do I convert them now they're in our system they're not retaining my sales you know I sold ten but to ten different people and then when you've reached it suddenly your set of problems are I can't fix bugs fast enough I don't have enough customer support my servers are crashing um so it really is your customer is now leading you instead of you're leading your customer and demand overwhelms the capacity and there's different sort of also different sort of analogs as well is that what you call product market fit or product market misfit no that's product market fit if it's a misfit then you don't even have to let me give you an example these are great problems out these are problems that you can all solve with money this is the problem you would see right this is where you go you know server capacity it cost me ten million dollars a year the reason why is money is flowing in my pocket can I get good terms money solves these problems you're on a shared server before on an after you're on a dedicated server right and you have this this painful point where you know your stuff is crashing and all the rest and you have to go up and do other things but alternatively to follow on his comment not to pivot not to pivot too soon the valley of death sure it's true that's again it's you know it's one of the things that we actually talk about that uh we'll just hop around to some of these slides is that you know there's nothing deterministic about being startups in this methodology right it's not a paint by members approach and so this doesn't take away anything from the art of entrepreneurship and the question of when you when do you keep busted down that door because you really think that there's something there or uh no it's time to pivot that's a founder's decision right there's no methodology for making that decision and you asked about pivot earlier so the concept of pivot really is your business model you have several aspects of your business model you know you're trying it out something doesn't work but you have learned something so you keep one foot planted down in what you've learned and then you pivoting the other aspects of your business model and maybe you pivot multiple times and the second pivot is you're leaving the first where you first had your foot down so uh and that's different in my mind than iterating iterating is when you're actually testing a particular idea it could be messaging positioning landing pages you know product features so uh and those are just are sort of smaller changes that you're making to try to figure out what's the right thing and what works versus you know a business model changes really where the pivot comes in really quickly also don't worry about product market fit you also see sort of the negative so when twitter goes down when twitter goes out people go nuts right like people complain about the blogs the blogs when twitter goes down it's a big deal in front of mine he's got a start up his site went down he didn't know about it for eight hours not one of his customers no one complained about it right pretty clear he's nowhere near in the product market fit right and he we joked about this like there's no pain right uh another friend of mine a different guy their site went down for like five minutes they had CEOs of five of their customers call their CEO within seconds right so it kind of gives you some ways of triangulating around sort of this nebulous concept in in in terms of actual methodology and process uh I have several clients that are sort of mired in that area it's really where most startups probably die is searching for that product fit and actually you know maybe it doesn't ever happen right that's when you pivot um but in terms of process it's very customer driven so it's not that you're doing what the customer says which is actually this slide so customer development is not feature longer and it's not it's not asking what the customer wants and then going and building that which is another trap that entrepreneurs fall into fall into you want to use you know what's called the five why's right which I think is an agile approach to solving engineering problems inside your company when a customer says that they want something you you need to ask them why and you need to keep asking why do you understand why it is that they're asking for what is the pain that they're actually solving they don't necessarily know what's the best way to solve it they're going to give you a bunch of ideas but not until you understand what they're trying to solve can you maybe come up with the right solution and so what you're doing during this phase where you're trying to reach product market fit is you're hopefully iterating quickly on your product you're you're constantly talking to customers to try to hone in on what it is that they're trying to solve and what it is that they need and you're measuring their passion for the product right so there's a couple methods of measuring passion Sean Ellis sort of invented the the survey.io where it asks the question you know if this product went away how disappointed would you be and and you know people throw out their numbers like 40% 50% if you can get that number of people that say they would be very disappointed if your product was no longer available then that's an indication of passion and you're trying to grow that user passion trying to grow the fervor of that passion and you're trying to grow the number of people that actually have that passion and that's really what drives you I think towards product market fit if you're increasing the passion the number of people that are passionate to market signal that you're onto something and so you're constantly in conversations with your customers and really you know in this world of software internet and iPhone apps and all the rest your IP there is no IP anymore right it's so easy to copy this stuff you're not going to be able to get patents and protect your internet idea most likely even if you've got the patents your ability to litigate is very small your differentiation and your winning your winning combination is your relationship with your customers so if you're developing passion with your customers that's what's going to keep them your customers your interaction with them and the passion that you create is your differentiator and so establishing those relationships early on is what's going to hopefully you know tend you towards this product market fit why don't we go to the side of the folks so that they can follow and check out so Steve Blake he wrote the book called the forceps of the epiphany that's his twitter handle right there as well if you're at all interested in what we've been talking about he's sort of the progenitor of all this stuff I recommend reading his book forceps of the epiphany again immediately it's that dramatic effect on the way I view startups in the startup life one of his students has got an Eric Reese he took Steve's idea of customer development and melded it with agile software development and this is what he calls the lean startup he's got a great blog called startup lessons and learns Sean Ellis is a guy that Brandt mentioned probably the premier sort of startup marketer he's actually doing his own startup right now but his blog is pure gold for startup marketing he's Shaw is an entrepreneur up in the Silicon Valley a very data driven guy also part of the lean startup crew as it were his company is called Kissmetrics and then Rich Collins runs a Google group if you just google that you'll find out if you're interested in the things that Brandt and I are talking about we'll make this deck available too so you'll be able to get access to these Damon Clores should probably be up there as well and he's got some great his pirate slide on his pirate metrics if you're doing online but the marketing is really important he also writes about changes in the investment world that I think are right on I guess the last thing is so to pitch Brandt and myself so we wrote a book for all the entrepreneurs that have customer development it's a cheat sheet Steve's book is amazing you should go on and buy it but it's very thick, very dense really makes you earn the knowledge in it we did sort of a cliffhills version of it you can get it as a PDF at custed.com you can get it as a paperback on Amazon or Kindle and if you use the discount code there you can get 20% off if you buy it as a PDF and then our contact info here we like to discuss the book anything else that you'd like to discuss so thank you for having us any questions or anything we talked about throw them at us you have a question you mainly mentioned that customer is the major person you should consider when you are developing products but customer is only part of the stakeholder the vendor is the stakeholder the innovator is the stakeholder I'll give you an example when 3M invented they would have asked anyone do you know the post-it notes? no one would have a clue I don't need it but I can't do without it same thing with the post-it notes so I think we should also look at other stakeholders and not only the end user as the stakeholder it all depends on your business model we actually have a section in the book where we do what I call the customer development whiteboard and it's as if you were in a room with your management team and you draw your ecosystem what is your ecosystem how do we envision this product getting to market what is the value prop for each one of those stakeholders which ones are mission critical and so which ones should you focus on first and what are your assumptions around there and it lines up the things that you should do customer development to test so yeah it all depends we tend to talk a little bit generically about customer and product but there's all sorts of alternative business models out there that make those ecosystems very complex so you bring one up there's inter-market places where you've got it only works when all the entities are at the website there's network effect businesses so yeah I think that there's a way to use these methodologies for any of those business models but you have to be creative and you have to figure out who it is that you need to talk to in order to validate part of your business model so what are the riskiest elements of your business model back to your 3M posting example it's a great example I'm glad you brought it up so what Brandon and I are saying aren't that hey do you want, tell me what you want and then it'll come out fully formed from the customer's mouth oh I want a little square thing I want it to be kind of bright yellow I don't want to be able to tear them off and put them everywhere often that's not people don't think that way but as an entrepreneur you have the vision I'm guessing though the 3M story I know it's not in the news how they came about I'm guessing though what they probably did is they observed how people were taking notes so no one actually verbalized them this is what I want and I want it to be this cool and I'm gonna throw it up on my whiteboard but he probably observed or staples for example the guys actually went out and looked at how much office supplies people were purchasing at companies and they didn't ask for example like an admin or a secretary how much do you spend on supplies I don't know 100 bucks a month and they actually went out and went into the storeroom and actually started looking at it like wait a second this is actually $2,000 worth of supplies and they saw this mismatch on in terms of what customers are saying potential customers are saying and customers are actually doing massive difference often so you can still apply the same principles to that but again I just wanted to be clear no one who understands customer development thinks that the customer is going to spend exactly what they want to you they can talk about their problems and their pains and their passions and as the entrepreneur is sort of beholden to suss that out and create a solution


All right. We're already way over time, but it's always better if there is interest in questions. I'm happy to answer them. All right. So this is a much more technical talk about how we are compiling shaders in Radeon SI which is effectively our OpenGL driver in Mesa. I want to touch on two topics. The first one is NIR and the other one is something dynamic linking which I'll explain. So NIR, those of you who were here in the morning, maybe saw Alejandro's talk about SpearV you've heard the term. Here's an overview on, wow, the colors are really off here for some reason, but okay. Here's an overview of how shaders are compiled in our various open source drivers. I think it's worth trying to follow that diagram properly. This is today's defaults. So on the left most path, you start with GLSL. It's translated into this Mesa GLSL IR, intermediate representation. This is co-chaired between all Mesa drivers. Then it gets translated into something called TGSI, which is a gallium specific for those who know about gallium translation. Then within the Radeon SI driver, this TGSI gets translated into LLVM, intermediate representation. Some of that is shared with LLVM pipe, but in the end the IR looks quite different. Then this is passed to LLVM, which does the back-end compile and generates the actual binary that is then uploaded onto the GPU. Now to go a little bit further to the right, if you're running Vulkan with RadeV, you will start with SpearV. You will take this middle arrow to NIR. This is the translation that was first introduced with Intel's Vulkan driver, and which is now also used by RadeV. Then the NIR gets translated into LLVM IR, that looks like the LLVM IR that Radeon SI produces and gets translated into binary. On the right-most side, if you're running Vulkan with AMD's official driver, then the SpearV gets translated directly into something that I like to call LLPC IR. LLPC stands for the LLVM Pipeline Compiler that the driver has, and it's really just LLVM IR. But certain graphic specific functions, like buffer loads, image sampling, etc., are expressed via LLPC-specific function calls that act like intrinsics. They're not really LLVM intrinsics, but they act like this, and then still as part of that driver that gets lowered into proper LLVM IR, that doesn't have these special intrinsics anymore and gets translated into binaries. You see that the back-end part is shared between all the drivers, but the front-end has lots of different paths. The goal of transitioning to NIR is to have the picture look like this instead. So to have RAID and SI take the path of going from Gillesil IR to NIR, and then proceed like the Rad-V driver does, the NIR translation is of course already in place for Intel's OpenGL driver and also various Scallium drivers use it. Right. So have a bit of more sharing. So why do we do this? So one is reduce the code duplication. Another big reason you've heard about in Alejandro's talk this morning is to support the Spear-V features in OpenGL as well. It's just the least resistance part to do that. Also some future features that we want to implement, it's a bit more convenient to represent them in NIR than to add them to the TGSI path, like half loads and 16-bit integers and stuff like that. Also NIR is actually a representation that you can use to do code transforms on it. TGSI is really just a representation to transfer shaders and you can't easily transform it. With NIR you can do this which allows more opportunities for controlling how optimizations are done in the shader backend, maybe some hardware specific optimizations that may become what I mean by this, may become clear later on in the talk. So I don't want to talk much about how this transition is happening. It's actually very far along already. So thanks to a lot of hard work by some people, it's very close to feature parity with the default TGSI path. We've seen recently that maybe the performance isn't up to par yet. We need to work on that. But we're quite far along and if you have a recent Mesa master driver, you can test it out by setting this environment variable that is mentioned there. So definitely Kudos need to go out to Dave and Baz who wrote the initial translation from NIR to LVM as part of RADV. Then after I did the initial thing as part of my Svirvi experimentation, Timothy has done a lot of work to get us to close the feature parity. Samuel has also done some good work on the NIR backend. So thanks to all these people, we're getting very close. Maybe one question that comes up in this context is, well, what's the future of TGSI going to be, of this other old shader representation? I don't think it's really going to go away very quickly because there are various niche places that generate TGSI. I think there are some in the multimedia. I'm actually not sure right now. There are some helper libraries that generate TGSI shaders. There is the nine project, the D3D9 implementation for wine which is TGSI based. For all of these, the first thing that we can do is just keep the TGSI backend around longer. I mean, that's perfectly fine for now. The other part that we might at some point consider is to use this translation of TGSI to NIR which already exists and might help there. TGSI is also currently used as a shader transport for virtualization drivers. So both VMwares driver as well as the Virgil driver. So I don't know what their plans for the future are. There is now a binary encoding of NIR for the disk cache, which should be suitable to fill in that function of TGSI. So they might want to consider migrating at one point as well, but I really have no idea what their plans are if they have even thought about that. Given the lack of time, this was really everything I wanted to say about the NIR part of my talk. So if there are questions about that, maybe I'm bringing them up already now. Otherwise, I would just continue. Okay. So the second part, I wanted to talk about concerns of with dynamically linking shaders. So what do I mean by that and why do we want it? Or why could it be useful? This is really an aspirational talk where there isn't code written yet. It's kind of a goal to talk about and get feedback for. So LVM, what it gives back to the driver is a standard ELF object that contains the shader binary. It contains some GPU-specific data sections, but mostly it's a standard ELF object. Right now, what we're doing is we're just taking the code parts or the text section out of it, and we'll actually just paste together text sections of multiple shader parts. The goal would be that instead of this ad hoc pasting, we'll do a real linking step that can also take other sections into account and that can resolve relocations and all sorts of things like that. The main motivations for that are that doing so would allow us to really have a better treatment of read-only data. So right now, if you have a constant in your shader, like if it's just a scalar value, then it will become an immediate as part of the instruction stream like on x86. On x86, you have instructions with immediate constants and the same on our GPUs. But if you have a larger constant structure, so maybe some hard-coded lookup table that the shader uses, then in a normal program on a CPU, that would land in this read-only data section. But since we don't have that yet, the only ways we can deal with it are either to translate it into uniforms and treat them like uniforms that happen to be unchangeable by the program, or we generate code which has them all as immediate constants and build the table on the fly while the shader is running, and neither of these solutions are particularly great. So it would be good to have proper handling of these read-only data sections. The other aspect is that it would or should allow us to explicitly describe what the hell we're doing with LDS. I will explain what LDS is local data share. People who write compute shaders know about that. It's just called maybe differently. I'll explain the details of that. Maybe you're asking yourself right now, okay, why do we have multiple shader parts that we want to paste together in the first place? If you program OpenGL, you might be thinking of shader objects that are being linked together in a program, but that's not it. So in OpenGL, you can have, for example, multiple vertex program shader objects that are being linked together into a single program. But this linking happens long before we ever convert it to LLVMIR. So I'm not talking about this kind of linking. What I am talking about can maybe be illustrated with a small example. So this is one of the simplest possible pixel shaders, which I extracted by running this GLX Gears command. This is just the assembly that we output. I mean, no, I'm biased, of course, but I think of all the desktop GPUs, we have the nicest assembly. To make sense of it, I should maybe tell you a couple of things. So first of all, our ISA is honest about the fact that multiple threads are being run simultaneously within the same, what we call wave. So the program is running a wave, and each wave consists like a single instruction multiple data machine of 64 parallel threads. So there are scalar instructions like the very first one. It starts with an S like scalar, which is operating on a scalar value. So it copies a single 32-bit value from S9, which is the scalar register number nine, to the special register M0. Now, why it would do that is maybe a bit mysterious. S9 happens to be pre-initialized by the hardware. You can think of it as a shader ABI that's going on, and M0 is a special register that will be implicitly used by the next instructions. So the next instructions, they start with a V like vector. So they actually operate on up to 64 pixels at the same time, and they are interpolation instructions, which is made a bit of a misnomer because there's no mathematical interpolation going on. What they're doing is they're just taking a constant attribute value. So you see the P0 after 0.x, which basically tells take the x component of the zero index attribute and store its value in V0. Then that's that for x, x, y, z, v. It's a W. Then you have vector instructions that convert, packing 32-bit floating point numbers as half floats, by rounding to zero RTZ, taking values from the V0, so the first vector register, packing them together, storing the result in V0. Then one after that, taking values from V2 and V3, packing it into V1. Then finally, there is a special instruction called an export instruction, which will export this color data to the color buffer, which then writes it into the render target, possibly performing blending. Finally, a scalar instruction that says end program is the end of the shader. So there are these kind of, some of my animations are gone for some reason. Well, the main message here is, why are we doing this compaction to 16-bit floats? We're doing that because we have an 8-bit color buffer and exporting that way is faster, but that depends on what the color buffer happens to be. So the first part of the shader is really only dependent on the original input shader, so GLSL source or maybe some legacy OpenGL stuff. Whereas the bottom part, starting with the conversion instruction, is something that we can only generate once we know what the color target is going to be, because if the program wants to render into a 32-bit floating point buffer, then it would be incorrect to pack into a half-loads. We would lose precision. So this last part, we can only compile once we know how the shader in which context it's going to be used. This leads to a problem called stuttering where maybe you have a game running, and then an object appears on the screen where the program had previously compiled the shader, but now the shader is being used in a context that is not being prepared for, and it needs to be recompiled, and whoop, your scene stops for a moment. The way we solve, one way to solve it is a disk cache, of course, but that only works once the program has run, at least once. For the first run, what we can do is we compile this main shader part, and then initially, and then the prologue and epilogue, they are very short and they can be compiled quickly and then just paste it together. So that's why we combine different shader parts. There is another reason why we combine shader parts, and this has to do with how the shader stages work. So if you know OpenGL or Vulkan or anything like that, you should be familiar with the column on the left. It shows the shader stages in a graphics pipeline. It starts with a vertex shader, then if you want to use tessellation, you have optional tessellation control and evaluation shaders. If you want to use the geometry stage, you have an optional geometry shader, and in the end, you have a pixel shader that produces the pixel color values. Now in our hardware, the stages that we have actually look like the next column. So at the bottom, you see vertex shader and pixel shader, and above you see like geometry shader is familiar, but you see these other names that are kind of weird. And so what happens is that the API shader gets mapped to the proper hardware stage as illustrated by the next columns. If you have the simplest and standard case of just vertex and pixel shaders, then the vertex shader goes to the hardware vertex shader, the pixel shader goes to the hardware pixel shader, as it always does. But if you use the geometry shader, then the geometry shader goes to the hardware geometry stage, but the hardware vertex stage is after the hardware geometry stage, so we have to change the order. I mean, it ends up in the ES slot, and there is something called a copy shader, but the details are not important. A similar thing happens when tessellation is used, and in the most complex case where both tessellation and geometry shaders are used, you get the column on the right. And now what happened is that the hardware designer said, well, like in the tessellation case, between the vertex shader and the tessellation control shader, there isn't actually that much fixed function stuff going on there, so let's just combine these into a single hardware stage. A single hardware stage that first runs the vertex shader, and then runs the tessellation control shader, or the same in a second column with the vertex shader and geometry shader in one single hardware stage, one single program from the GPU perspective, you run both vertex and geometry shaders. So we have to paste also these together. Okay. So this leads to some interesting challenges. If you know a little bit about this stuff, you'll know that the vertex shader, it just outputs attributes for an individual vertex. While the geometry shader, it operates on one primitive at a time, so the input of the geometry shader will be an entire triangle with all the attributes of its vertices. That is the view that you have as the programmer. So if you think about how to translate that into this single instruction multiple data machine on a GPU when everything runs in a single shader, you first kind of work with vertex threads and vertex lanes where every lane is responsible for computing one vertex shader invocation for one vertex. And then in the second part, you will have physically the same lanes, but now they're operating logically on primitives, on geometry shader invocations. And you somehow need to transfer the data between them. And the way that this data happens is that the vertex shader part stores its output into the local data share, which is a small memory that is shared between all the waves within one work group. So typically we have up to four waves to process up to 256 vertices at the same time. And then the geometry shader part loads the inputs from there. Now, this kind of shows how the data is laid out in LDS, but what happens, the main problem here is that LVM does not know how we're using LDS. This means that we cannot use LDS for all sorts of things where it might be interesting to use it for like spilling. So spilling, I mean, it doesn't have such a big application because for every 256 kilobytes of vector memory, we have 64 kilobytes of LDS, but still maybe there are some cases where it could still help. We can't use it for dynamically indexed arrays where it sometimes might be helpful. It's difficult to use LDS for additional purposes even from the front end because the front end has to keep track of all the addresses manually and it just becomes complicated. It might inhibit LDS analysis in some cases, although LVM is generally very good at that. So the goal would be to somehow explicitly represent all the variables that we use in LDS, store them, represent them as an LDS segment in the ELF object that we get, and then when we merge together a vertex and geometry shader, they will have a shared variable which will be where they transfer the attributes, but maybe they have some other uses for LDS as well and we somehow use the linker to arrange those and calculate the right addresses. It's not entirely simple because if you look back, this thing here is kind of a two-dimensional array. So one index is the vertex number and the other index is the attribute number and component, and we don't really know either size when we compile at least the geometry shader part because we might have a vertex shader that produces attributes that are unused by the geometry shader and we don't actually know how many waves we're going to run simultaneously in advance, so there are some problems, but as a kind of a minimal demonstration that might already be useful for various things, it would be nice if at least we could represent additional LDS variables. Maybe we want to use it to store some dynamically indexed array that happens to be the same across all lanes, something like that. Maybe use it for spilling, maybe use it for something else and do that in the linking. So that would be the goal. Remember, I also mentioned read-only data linking. That part is fairly straightforward in comparison because it's just like on CPUs. We just need to think about what we want the ABI to look like. 64-bit pointers maybe, maybe we want to restrict ourselves to 32-bit address spaces for a bit of efficiency. There are some good choices to be made there. So of these two options, I think the second one is the better one but that goes into details. So yeah, just let me summarize the two points. So switching to nir and radian SI, we're going to do that and it's actually very far along already. And the other part is, well, aspirational. I want to explore this dynamic linking. I've explained to you the main purposes of it and what is involved. Interesting question in that is what kind of linker do we actually use? LOD, part of the LLVM project is kind of a natural choice because we already depend on LLVM although it does live in a different repository. It can be embedded as a library. It's designed like that. On the other hand, it's like a complete ahead of time linker and we only really need a dynamic linker. Well, this still needs to be explored and we'll see. Okay, with that, thank you for your attention.