I think I'm just going to keep going. Keep going. OK. I'm Zach Chandler. I work at Stanford University. My job title is Web Strategist. And I'm lucky enough to work on our own in-house Drupal team. And I consult with projects that go out to vendors. Produced many RFPs, hopefully not quite so terrible as some of the ones we might hear about today. But we've also started to look at alternatives to the process, so that's. Hi, everybody. Is this better? All right, I'll talk like this. So my name is Brian Scower, and I'm with Lullabot. I'm actually our sales manager. So I'm actually interfacing with most of the RFPs that come through our door. I'm kind of the first point of contact with them. And so I see a lot of RFPs, both good and bad. And I've been kind of exposed to the good ways of going about it as well as the bad ways. So we want to take just a moment here to recount some of our favorite RFP memories, because we've all been there. So who would like to go first? Brian? So my worst RFP memory was actually my first RFP. It was before my life in Drupal. It was the first RFP I'd ever done. I was a green salesperson. And it was a hosting RFP. And it was a really big name nonprofit that everybody probably would know. And it was probably like a 30 to 50 page RFP. And it came on my desk, and I was stoked. So I went through the whole process, ins and outs, answering every question possible. Hey, that's nice. And even got to the point where they invited us to come and present, flew down to the city where this big nonprofit is. And basically did several presentations in big boardrooms filled with lots and lots of people, worked out migration plans, spent countless hours of work on this. And ultimately, when the decision time came, the individual leading the project actually said to me, we're staying with our current vendor. Because if I do that, I'm probably not going to get fired. And yeah, that was a learning experience. So at Four Kitchens, we got an RFP from a very well-known Ivy League University in the Northeast. And we were very excited to do this work. And we got on our first phone call with them. And on this call, the two stakeholders who had put together the RFP said to each other, oh, it's finally nice to meet you. Good to meet you. Meeting each other for the first time. And then they proceeded to disagree on the scope of the project and what the project was about in front of us on this call. And the call became us just sitting there and listening while the two of them hashed out the scope of the project. And then we kind of ended the call with, all right, well thanks, everybody, for that hour. We'll follow up with you later. And we walked away. Because if these people are just meeting, and they're the stakeholders, and they're the ones evaluating the criteria for the proposal, and they're the ones that are supposed to understand the project, and they're hashing it out on their call with a potential vendor, bad sign they need to go back to the drawing board and rework what this project is all about. So how many people here are typically on the vendor side of RFPs, meaning you receive them and you respond to them? All right, how many people here are on the client side?  That's about what I expected. There seem to be more vendors than there are clients at times. So let's talk about why these things exist to begin with. There's the good reason. And this is the honest, wholesome, everybody can get behind this kind of reason of there's work to be done and somebody is looking for somebody to help them do it. That's what we all think as vendors going into an RFP that we think that that's the process that we're involved in. Are we in agreement? Excellent. Now all the bad reasons. Can I add one good reason before you continue? Please do. I think it also announces that it's going to be competitive. And it's a well understood format. So if you've had any assumptions, I think it presumes some of those. OK. So the good reasons and now the bad reasons. Sometimes a client may feel as though, well, this is how we've always done this. They may think that they are supposed to write an RFP because isn't that just how these things work? We heard in one case where we received this RFP that just seemed to be very formal. And there were a lot of bullet points. And some of them weren't really filled in. There were a lot of these sections. And they seemed to repeat themselves. And it turned out that this client had just googled for how to get proposals and downloaded a template and then filled it out and just thought that, well, this is how all RFPs look. And I'm just going to fill out all of these empty spaces. And then, of course, it's a standard requirement argument. And this usually comes from big organizations and governments and things like that. How many times have you come across something like this? Yeah, I think, well, there's sort of the government nonprofit thing where it's a requirement. And then within big organizations, purchasing departments sort of exercise an authority over the process. And a lot of times you can be working with the stakeholders and then separately have to go through their procurement purchasing process, which involves an RFP, which nobody likes. On our end, the RFP process was actually an improvement over what we had before, which was a referral by a water cooler. And so you trust the person that you're talking with, oh, who did you use? And it was always the same two names. And so we kind of ossified this process. And those companies got very, very comfortable. So I think as a waypoint to this RFP-less future, the RFP process was actually a helpful step for us. And in a best case scenario, it can actually be that solidifying of the project requirements and internal buy-in and that really necessary research pre-step to going out and getting a vendor for things. But oftentimes it's also just kind of a necessary evil to projects of a certain size, which is part of what we're saying maybe it doesn't have to be. But for current reality, for a lot of clients, that's still just the case. It's a requirement, and they have to do it. So related to the requirement is in some organizations, there are rules that say if a project exceeds or is likely to exceed a certain amount of money, then it has to go out to competitive bid. And you have to gather three or four or five or something numbers of bids, right? Now this is under the batteries in category because that means that in many cases, some of the vendors who are replying don't stand a chance. And they're doing this just to fulfill some kind of bureaucratic requirement. And it's really not entirely fair to the vendor. And they usually don't know. They think that they are doing this, that this is all above board, that they have a real honest shot at it. But sometimes a vendor has already really been chosen. But because this project is $50,000 or $100,000, they say, well, it's got to go out to competitive bid. We have to get two more people. They're just going to go through the process. We say no to them automatically. And you get it no matter what. So just to provide yet another counterpoint. So I work for an organization that does have such a rule. And even when it's fair and we carefully choose excellent vendors that all stand a chance of winning, this does create a bit of a problem that I'm still trying to figure out. And that is, what happens when the vendor that you admire and want to work with, you know you have a future with, what if they lose several times in a row just due to the circumstances of the way on the merits? So I really don't have an answer for that yet. So something to think about. And now the really bad reasons. And these are the cynical ones. Sometimes a vendor exists. And a client issues an RFP just to price this project to gain leverage over that vendor to make them lower their price. So in other words, you're working on a project. You have a good relationship with this client. And then somebody comes along within the organization and says, hey, are we sure that this is actually a good deal? Let's put it out to bid and see what we get. And then they get some much lower estimates usually because they'll pick the lowest ones and say, hey, this company X said they could do it for half the cost. And company Y said that they could do it for 75% of the cost. Can you come down on this a little bit? And it's creating this sort of false competition and placing pressure on the existing vendor to lower their price. This is. We can all share that pain. Yeah. And chances are we've all been a part of this and we didn't even necessarily know it. Sometimes, clients want free consulting. So they issue an RFP.  And then they say, hey, thanks, everybody, for all these great ideas. And then they take it in-house. And they use those ideas as blueprints. So there are some ways that we as vendors can maybe get around this. There's the idea of no spec work, right? Never, ever do spec work. And also push back pretty hard on that RFP process to say, well, here's generally what we're going to do. Here are generally the technologies we're going to use. We're confident that they're going to work. But we don't explain how exactly we're going to put the pieces together. And it's unfortunate because we really want to show off our knowledge, right? We want to show that we have a really good plan. We've thought about the project. We understand it. But having to hide some of this information because we're afraid of something like this happening to us is really unfortunate. How often does this happen to you and Zach? Have you ever been on the other end of that? So I think this is awful. And we have never done this, ever. But something I just picked up on what you mentioned with not giving a full blueprint and holding back, I think that there is a bit of a tension there with how do we then judge the way you would execute? Because especially if we're comparing three excellent choices, one might be more in tune with where we think the technology is going and our API design, for example. Something that's outside of Drupal. So if we withhold too many details, then we may not have the right grounds for choosing a winner. So again, it's very true. I have actually gotten a phone call that said, congrats. You won the RFP bid. And we're going to do it in-house. So yeah, completely that, Frank.  And we're not going to go with a vendor. Then we have risk aversion.  The stakes are really high. Your job's kind of riding on it. We need to make sure that this stuff goes off without a hitch. Go find a good vendor because we can't do it in-house. That person then is facing a tremendous amount of pressure. This is sort of an existential threat for them. So they put together some very onerous RFP or require a ton of detail and a ton of work from the vendor because it displaces that responsibility to the vendor. That person at the organization who feels that their job is threatened can always say, hey, they said they were going to do this and this. And as you can tell from these documents that I got from them, they put a lot of thought into it. And they did a lot of work. And it creates more of this defensive position that both the client and the vendor have to be in. Have you ever run across anything like this? I don't know if we have specifically because it's hard to tell what happens after what happens leading up to an RFP and then what happens after you win or lose it. Yeah, I think that, I mean, in general, everybody to a certain extent wants to CYA, right? And when I'm kind of evaluating RFPs when they're coming in, a lot of the ways that clues to that is if the questions are incredibly detailed to the point that they're not really related to the project at hand or the work at hand, then it sort of becomes, okay, maybe they're asking this level of detail in these sorts of questions about our financials and our background so that should anything go awry and the onus comes back on them as to, man, well, why did you guys end up with this vendor? Well, look, I mean, they have solid financials. They do this, this. And it's, you know, I think that a lot of times the way an RFP is structured, you can kind of pick up on some of that stuff, but I really don't see it quite so much, you know, I don't really see that as much on a regular basis or when I do, it's generally an RFP that we're not responding to. I think there's another way to read this slide as well and one that I've actually seen more often and it's not that the RFP is overly detailed, it's that the RFP is underly detailed and that can be a really good indication that you have a client that doesn't know where to start and we'll talk more about this later around, you know, clients who really don't have internal technical resources, but basically using an RFP process as a way to get their hands around a project, just get their mind around it in the beginning instead of doing the homework on their side and that can be another big red flag in the process of do they have accurate reasonable expectations around this thing? And finally, we have qualifying new vendors. So what I mean by this is clients, of course, want to develop a network of vendors. They want to have a whole list of people that they can contact when they have new work. They want to know what those people are about, the kinds of work they like to do, what their team makeup is, the types of projects they typically take on. This is a very valuable thing for a vendor to have, especially for a client to have, especially if they have a whole bunch of work stacked up. But sometimes they go about this in the wrong way and the wrong way is issuing an RFP just to basically gather resumes and to start to have this conversation without really intending to give this work to most of those vendors. And there's a much better way around this and this is something that Zach is doing at Stanford that we're gonna talk about in a little bit. But this is what we mean by qualifying new vendors. And I put this under the really bad cynical category because it's generating a ton of work for all these vendors who are responding to these RFPs that then are basically completely ignored because what the client will be reading is likely the boilerplate at the beginning and the team makeup and the types of work they'd like to do in their portfolio and things like that. So why should we even respond? As vendors, why should we respond? So we wanted to lead off with some really positive reasons and a best case scenario of how an RFP can actually be very good. So I may be kind of a mole on this panel. I don't completely hate RFPs because I've had some great experiences with them. I've been parts of teams that have gone out and won Sony.com, which and the version of it that won the Webby for Best homepage and a couple of other awards a couple of years ago, none such records, New York Stock Exchange.  It is possible, it's rare, but it is possible for a very well qualified team who knows what they're getting into and has really done their homework around this to kind of cinderella themselves into the big leagues to completely mix my metaphors. You can actually, you can get out there and that was in a couple of those projects where we're with a team of six. So very, very small companies going out winning a very big, big project that really ended up being a defining thing for the company and really a breakout project and it can go well. I will say that those RFPs were, those were not 40 to 60 hour RFP responses. Those you're really talking about upwards of 200 hours of research and 50 to 60 pages of writing and in some of those we went and delivered to the client things that they didn't know about their own business or their own competitive market and it was a completely different process than going through and wrote answering a form. But in some of these cases you, if you're an excellent fit and you have an internal advocate there, you really know what this is that you're getting into. It can work out. I would highly advise people against applying to the giant RFPs. And I, if you don't have that kind of prior relationship experience going after some of those big projects or a strong internal advocate. So I refer to this as practicing safe RFP. There are things that you can do to minimize your risks on this but we should also qualify a little bit here. There's a big difference between say a $50,000 RFP and a $4 million RFP. And both of those exist out there in the marketplace. And I've responded to both types and it's a completely different approach. So some of the things that you can do to evaluate whether or not you should be responding to this. I mean first off, in 99.9% of cases you shouldn't be cold responding to an RFP that's just posted out there or somewhere in the wild. It's just like bait looking for somebody to go, somebody has just gone fishing in that case and you can blow weeks of your firm's resources chasing after something that's just out there to get bids. So take a look at the RFP. As I mentioned before, taking a look is it doesn't have an appropriate complexity to the complexity of the project. Over under either way is a bad sign. If it's a $50,000 project and they want you to go spend 200 hours responding to this RFP, that's not a good sign. And likewise, if it's potentially a couple of million dollar projects and they just want a two page off the cuff response, that's also probably not a good sign that the client knows what they're getting into and can be a good client for you. It's a prior relationship or internal advocate. There are a lot of cases where even existing vendors have to, for these sort of internal requirements, have to keep resubmitting RFPs for every project. Oftentimes you get asked to write the RFP and then respond to it. So if you're a firm that doesn't have that prior relationship, you need to qualify if somebody else does. But if you do, then it can be a good thing for you. And the last point, and this was made by the founder of a great design firm I used to work for. And she referred, she made the excellent point of there's a lot of projects out there you can't afford to win. Especially the really big ones. The take into account your own ramp costs and lead times of payments, there are certain sizes of projects that can kill a small shop, even if you win them. Because the number of people that you have to pay before you're going to get paid can really, really affect your cash flows. So one of the points that she made was basically if you can't afford to pitch this well and lose it, you probably can't afford to win this project either. So it's something to keep in mind when you're evaluating what to go after. So here's another RFP positive story from Brian about the adjacent benefits of responding to RFPs. So yeah, I'm exposed to a lot of different RFPs and by and large, most of the time Lullabot doesn't choose to participate in RFPs unless there's a really, really good fit and a good match and some sort of benefit that we see into it. I totally agree with your message that unless, a lot of times, unless you have an advocate or you have some sort of in-road outside of the RFP itself, you're kind of the sucker in that game. What's the poker saying? If you don't know who the sucker is, you are the sucker. And I think a huge percentage of RFPs are structured such that one individual company is designed to win and it takes a lot to overcome that. There are great organizations like Stanford that do it the right way, but there are a lot of organizations that do it the bad way. So when RFPs come to us at Lullabot, really what I'm looking at is a lot of different factors and we kind of have a whole big matrix of different things that we're looking at to see if the project is the right fit for us. But one of the things that we consider that actually comes into play a lot of times with RFPs is are there any adjacent benefits to doing this RFP in the worst case scenario that, hey, we're gonna lose it. If we recognize that as a possibility and come to a point of comfort around that fact, then we can look at RFPs and say, should we still do it if we know that there's a good chance that we can lose? And this is very much an edge case, but there was a recent RFP that we did where we knew we had a really solid chance of not getting the business. But at the same time, we knew that it was with an organization that had a lot of projects and that we had a pretty good relationship with. And by making a good showing and understanding that, hey, we might not get this, but while we're making a good showing, we're essentially showcasing what Lullabot can do and showcasing the type of work that we want to get. In a way, it sort of serves to educate the client as to what we might want down the road. And so in that sort of circumstance, I see that as an adjacent benefit, really rare type of case, but that would be a case where I would want to make a showing on it and get the business, obviously, but if we don't, it's not the end of the world. So like I said, if you assume you're going to lose, and I think that's something that everybody should do in any sort of circumstance if you're doing an RFP, even if you do have the relationship and you know that your odds are really good, you know, just the mere fact that it's a bidding process and there's a selection criteria around it, assume that you're going to lose and then ask yourself, you know, is the investment and the time that I'm putting into going to be worth it in the circumstance? And another thing that we did, so, you know, another, this is the real, real edge case of RFPs, why they can be positive is you can use RFPs, they can be a good litmus test for new services and new products that you're bringing to market. You know, one of the things that Lullabot is putting some muscle behind, recently is our video service, which if you guys haven't checked it out, you should totally go to videola.tv and take a look. It's pretty awesome, video monetization. But, you know, interestingly enough, when we were doing that, we ran into a case where we had an opportunity to do an RFP and we knew that, you know, we had done market analysis and business analysis, but one thing, for all the negatives of RFPs, one of the things that they can do is they can give you a lot of questions that you've never thought of before and, you know, give you a little bit of insight into what your target client base is actually thinking and what's important to them. So in the case of, you know, this RFP for videola, it was a really, really helpful exercise and it kind of forced us in a non-threatening, low pressure type of scenario to, you know, encounter questions that are probably gonna come up as, you know, that service rolls out and reaches a critical mass. And for that, it was, you know, RFPs are, can be incredibly positive, but, you know, in general that's kind of the, that doesn't happen very often, so, yeah. So I guess the summary of that would be that sometimes RFPs can teach you about your strengths and weaknesses relative to some new market. Sure. Could you, well actually, let's do the questions at the end because we'll have to do them through the mic. So maybe you should not respond to RFPs. Here's a case study from Four Kitchens. We sat down and did the numbers and we took all of the projects that we won or lost that were RFP or not and we put them on a big board and we marked them. We were spending on average 30 to 40 hours per RFP response. That's an opportunity cost for us of about $5,000 to $7,000 per RFP that we write. We won only 22% of the RFP-driven proposals that we submitted, so that's our hit rate of 22%. But only 16% of the work that we do results from RFPs in general. So that means on the flip side, 84% of our work involves no RFPs whatsoever, just personal connections and word of mouth. So the things that we concluded were that RFPs have driven a small but not entirely insignificant amount of our business and it costs less for us to send a team of people to a potential client to meet with them in person rather than just immediately responding to an RFP. $5,000 to $7,000 opportunity cost against two plane tickets for two people, if you're sending two people, hotel expenses, some meals if you're covering that and opportunity cost of having people unavailable for those two to three days, less than responding to an RFP in paper format. So we started talking about this idea of not responding to RFPs in relatively closed circles here and there. We'd done a couple of small presentations on the subject where we explored some of these similar ideas about our numbers and how they compared to other companies and somehow the word got out that four kitchens doesn't respond to RFPs. And that word traveled very, very fast and we know of at least one case where a client simply ruled us out immediately because they had heard that four kitchens just doesn't respond to RFPs at all. So if you're going to adopt this kind of idea or if you're going to start talking about it publicly like we're doing right now, be aware that some people, unless you have a conversation with them about what you really mean when you say you don't like RFPs or your RFP averse or whatever language you want to use, make sure that you can communicate that thoroughly because there is that collateral damage of, oh, they just don't play this game. They're not involved in the game. Goodbye, we'll go with people who do. But you can try this at home, right? You should be able to calculate your hit rate. So list all of the RFPs you've ever replied to, one, and lost. That's your hit rate, right? How many people here have actually done this? Not bad, congratulations. Okay, how many people have not? You should do this, this is important. Then figure out how reliant you are on RFPs. So list all the projects you've ever won and then mark off the ones that actually went through the RFP and proposal process. How many people have done this? Not as many, how many people have not? Okay, a lot of people not answering, I understand. It's early in the morning. So here's how you do it. Calculating the cost of responding to an RFP. So we're gonna talk opportunity cost first.  Multiply that by your blended rate. That's your opportunity cost. That's how much you are sort of spending on writing this proposal. If you're doing in-person visits and other expenses, you need to make sure to add that in. And now you have the cost of responding to an RFP. Take this number and weigh it against the project's revenue. If that number is higher than the amount of money you may be making on this project, doesn't seem to really do you a lot of good, right? Also, compare the likelihood of your RFP win. And that's not only your historical hit rate, but also all the other factors that you need to figure out. And, Crystal, you have quite a few. So I'd like to turn it over to you, actually. Other things that you need to keep in mind that aren't just bottom line. So Crystal actually disagrees with me on this point. Okay, so it's not just, it's somewhat about the numbers. Obviously, that's important. And obviously, calculating what your numbers really are, that it's not against the total price that checks to the writing, but really what your profit is on those and having a good idea of that. But there are some big, I spoke to this a little bit before, but what is the likelihood of actually landing this thing? And that really gets back to it. We are kind of driving this home. Like finding out what their response market is for this. Like, were you asked to respond to this? Is there, do you have a good indication that you are viable for this project? Is it out to a big market? Is it a cattle call? Is it as, are you, as my former boss called it, cannon fodder for this RFP? You really want to avoid being cannon fodder. But other things, like, does this relate to a specific skill set, a specific value add that you guys have as a company? Maybe you specialize in video. Maybe you specialize in mapping. Maybe you prefer social action projects. Does, do any of those factor, are any of those factors involved in this, in this project? And, you know, looking at that, looking at it in a holistic way instead of just possibly getting, it's easy to look at, you know, possible budgets of these and get kind of dollar signs in your eyes. But, you know, these things can be very expensive, not only to bid on, but again, to actually execute. And it's really good to know what you're getting into before deciding to make that leap or not. Meanwhile, on the client side, Zack. I guess this is my section. So, I would say, like I did earlier, that RFPs have historically been useful for us. They did help us get to a slightly more rigorous process than we had before. And I think that is one of the key reasons why clients like them. It creates a sense of rigor around this process. And it's familiar to them. But there are other ways of creating a sense of rigor or real, genuine rigor around the selection process. It's just a lot more work intensive on the client side. But I think one of the things that you'll find is that the time that you spend in here ends up paying you back later. So first of all, do your homework, understand the shops that you're talking to, know who they are, know who's on their team, what are they good at, and what do they like to do? And then contact them personally and invite them to bid. So we don't do, or at least anything that I'm connected to, we don't broadcast RFPs anymore. And if my name is attached to anything that you get from Stanford, you know that you're already a finalist, essentially. So there's this question about the document. Now, the thing about an RFP that's problematic is that it starts with a document. It's okay to have a document, you just don't wanna start with one. So if it's not a specification, then what is it? So I've been working on this genre of document that I'm calling an expository sketch. And what an expository sketch does is it states the parameters of the project, it shows you that we understand it well, that we know what we want, and it also provides one plausible way of building it. It's not the only way. We're not telling you exactly what to do. It's not a specification where it's use these modules and this is exactly how we want it done. Now go make that. Because we know that we wanna be working with creative design professionals and we want your best ideas and we're not always gonna have the best ideas, maybe not even most of the time. So once the expository sketch goes out, we'll call for presentations from each vendor. Now in many cases, I'm already familiar with the vendor, but it's, so I have a client on the Stanford side and they don't know very much about Drupal or they certainly don't know who you are. So this is largely for their benefit to get a sense of your company to find out what it would be like to work with you and to see what kind of questions you have and how well you understand our project and how motivated are you to work with us. And so for those we offer to do those remotely or in person and I will say that the more personalized and more present you are, the more successful those tend to be for the vendor. So if you actually, like Todd was suggesting, you fly out, then we're never gonna say this but I guess I am saying it now. I have noticed the trend that those who show up in person tend to do better. And if you phone it in, it's quite a handicap that you have to make up for there. Then we'll have a Q and A period where we will aggregate questions that come in from the vendors about the project because inevitably because the expository sketch was written to be open-ended and interpreted, it's not a specification. There will be questions and refinements. So the process we're following is that any vendor who happens to contribute questions gets to see all the answers to all the questions. Once we anonymize them so you can't see who asked them. We want everybody to benefit from that in a very open source kind of way. Then, so a lot of work goes into the expository sketch that's kind of hard to make and it actually gets more laborious as we go down here. You wanna have a documented vendor review process. So we have a scorecard system where as a group, usually there's a group of stakeholders that is reviewing these vendors, not a committee necessarily. Lots of times it's a small team. And we make a graph of vendors across the top and we decide what are the selection criteria that we're gonna judge the worth of these proposals on and then we weight them. So there's a point system. And then line by line, by category, we'll say project management methodology and we'll give that 20 points. Then you score each vendor across and then you go down the line and sometimes we have seven or 10 or 15 criteria and the great thing about this is it allows you to give some weight to things that matter but not too much. Some intangibles, like what do we think it's gonna feel like to work with them? I mean, that might be worth five points whereas technical proficiency might be 50. So lots of times these are somewhere between 100 and 150 points and then what we do is we then break up, we tally our numbers and the great thing about this exercise is it can surprise you. It prevents you, any one person in this group that's deciding the outcome to have some arbitrary impression about one vendor in particular, they can actually score a lot higher than you thought they were going to and maybe higher than maybe the one you thought was gonna be in second actually has a higher total than the other one and these are not, it's not deterministic, these then start a conversation. We all come back in with our score sheets and we go around and we talk about them. It was essentially just a conversation starter but it allows us to have something to work from and then that goes into the next step which is writing up an executive summary. Now on the client side, this is something that the vendor doesn't ever really know about but usually at organizations like Stanford there is a stakeholder above the level of the group that's deciding on how to execute this project and they're gonna need to know how you decided this. Especially if you're maybe unseating an incumbent or something like that. If there's a vendor who has worked with your organization for a very long time and they might know the stakeholder that's above the level of people you're working with, it's really helpful to have to show a rigorous process so you can clearly state this is how we got there. So in an executive summary we'll, basically I document the entire process. This is who is on the decision making team, this is when we met, this is what the review process was like, these were the selection criteria, this is who we're recommending for this project and why. And those types of documents though they are, they require a lot of work to produce. I cannot tell you how valuable they are and how much they travel once you've done them.  And then I guess my final point would be that the statement of work should be collaborative and iterative with your vendor. You don't want to hand them the statement of work or this goes to contract and this gets stapled on because like I said, if we're really doing this in an expository way, there should be some back and forth and some good ideas should be exchanged. So I also wouldn't just accept whatever statement of work they give you. So review it carefully, make sure it's what you really want because if that does get stable to the contract then that's what you're gonna get. And finally, all of this stuff is really great but it results in a ton more work for the client. But it's gonna yield a better project in the long run. So I want to make sure that we leave some time for questions and answers. So for the advice portion, I'm gonna kind of go through this rapid fire and if you have anything you wanna add, if you could just pop up your hand, cool? So we sat down, we've been talking about this for a few days, we sat down and came up with some advice to each groups of people. So our first piece of advice to the client would be require speaking to the people who will actually be working on the project. Sales people are there as facilitators but they shouldn't be the mouthpieces of the vendor, meaning they should not be speaking for what the developers and the designers and everybody will be doing to the exclusion of their input. So always make sure when you are talking to clients that you actually get some of the people who are likely to be working on your project in that conversation. Because after all, those are the people you're gonna be working with, not the sales people, right? Please beware of low bids. They're almost always too good to be true. These are companies that maybe aren't really paying a lot of attention to the RFP or there are a lot of reasons good and bad why they might be charging a lot less but please beware of them. And in fact, it might just be a good idea to take the highest and the lowest bids and maybe set them slightly aside and talk to them about why it's so low or why it's so high because chances are they're simply misunderstanding something or it turns out that they engage in a lot of really weird like subcontracting practices and this is how they can get away with it so cheap and you need to know about that up front. If you're comfortable with that, great. If you're not, you should know. Please pre-select research and vet vendors. That's something that Zach at Stanford does all the time. He's constantly engaging in conversations with potential vendors to see if there's a good fit in the future. And don't do open bids, just don't do them. You will have to read a lot of proposals as a result. This is going to save you time, a client. You will get, here's an example. I did a version of this presentation with somebody from Happy Cog a few weeks ago and his horror story was he asked a client how many people had received this proposal and the client said 364. And his response to that question was how many did not? Do not create an exact specification in advance because the vendors will likely have a lot of input on there and they may think that what you're doing may not really be that great and that there's a better way to do it or there's a brand new emerging technology that they may not know about that they want to tell you about. So don't try to create a very detailed exact specification. Instead, talk about what you're trying to do not how you want it done necessarily. Be sure that you absolutely check references because word of mouth is great, but it's not always right. So do ask for references, do follow up with them. Please to the extent that you can be open about your budgets because the vendors will let you know whether or not this is feasible. So this is something that I know Zach is going to want to talk about. So let me quickly say something from the vendor perspective. From the vendor perspective, we want some idea of how much you think you're going to spend on this project not because we're going to say, oh, $200,000? This is going to cost you $200,000, right? That's not what we're doing. We want to make sure that you're not coming to us and saying, hey, we want to build a brand new search engine algorithm to compete with Google. We have $20,000, go, right? Not feasible. It's very valuable for the client to know that maybe their expectations are way off and the only way they're going to find out unless they have a conversation at the very beginning is to receive a ton of proposals and bids that are suddenly way off the mark and then they go, oh, we screwed this up. This is actually way bigger than we thought. Sorry, everybody. We'll have to reissue this completely or maybe not do it at all now that we know that it's going to cost more. They may cancel the project. You've wasted absolutely everybody's time at that point. So, Zach? So from our perspective, so I, from my personal perspective, I completely understand and trust the vendors that we work with that it's really a sourcing question. They can give you a $200,000 version of what you're asking for or they can give you a $80,000 version of what you're asking for. So, knowing what they have to work with has everything to do with what they pitch to you. Now, from the people that I work with that aren't as familiar with these companies and the work they've done, it can sound a lot like, how much does this cost? Well, how much do you have? So, I'm interested in this notion of how do we establish trust? Of course, if you've worked with them before, that's a heck of a lot easier. But one of the things that I like to ask firms that I'm interested in working with is what's the minimum bid that you'd be interested in? What's worth your time? And then I will do my best to make sure that you don't ever hear from us on something that's lower than that. Because from my perspective, one of the risks is having groups from my organization because I can't funnel all of this.  Stanford is seven schools, it's not one school. And hundreds of centers with their own budgets. So, one of the dangers that I'm trying to avoid is if they don't have an appropriate budget, but they don't know that, the work sounds really interesting. And there's a lot of back and forth, there's a lot of time burned on it. And in the end, we could have skipped all that had they just known that it wasn't appropriate in the first place. So I get to funnel some of the traffic from Stanford that way, but in terms of an industry-wide best practice, I don't really know how you go about fixing that. Yeah, a lot of this just comes down to trust, which is something that we harp on quite a bit here. Right, I was just gonna jump in and make a quick point also that a lot of times, the reason why it's good to be open about budget is because a lot of companies use RFPs as a way to set their budget. And that just ends up wasting so many people's times on both sides and if that's the case, being open about your budget in the near term allows every vendor that's participating to understand that it's a fully thought out project and it's not just a wasted budgetary exercise. So vendors, so there are more vendors in the room. Advice to you, ask the clients how many other vendors they're inviting to bid, right? If it's 10, okay, if it's 364, maybe not. Be open with them, the clients, and get them to be open with you about these details because this is a trust thing and when you actually start to work together, you're gonna have to trust each other then, so why not try the trust exercises now, right? So this is what I mean. You need to start asking some very blunt questions to the clients. Why were we invited to bid in particular? The answer to that question is going to say a lot. They might say, oh, we saw you on a list of vendors on Drupal.org. Okay, or we heard about you. I have colleagues in another space who did some work for them and this project is very similar and we think you might be a good fit. Oh, okay, well that sounds really good, actually. You should also ask how many vendors with existing relationships are under consideration. We have 20 vendors that we all really love and we're all pitching it to them as well. Okay, well, may not sound very good for you and if they're ever cagey about any of these questions, like what's your budget, why us, things like that, you can always ask, how did you find out about us? That's a very typical conversational kind of question. Hey, thanks for reaching out to us. How did you find out about us? We'd like to know, we're curious. That will help answer a lot of these questions. That will elicit all kinds of interesting feedback. Ask if another vendor has been hired to write an RFP or evaluation prior to distributing this because there's a very good chance that that vendor is going to get the job, not you and you should just know what you're up against. Ask about their selection process. What are the selection criteria? If it turns out that one of the things that they're heavily weighing is we need a local shop and you're virtual or you are somewhere else, you need to weigh that in whether or not you want to respond. And just like the client, require a phone, video, or in-person meeting and pay attention to how these stakeholders talk to each other. So in the example I gave earlier, we did this phone meeting and our evaluation was they haven't actually figured out the project yet because all they did was talk to each other and not us. So pay attention to what they're saying. Do the stakeholders know each other? Did they get along or are they fighting?  This is a really big one that a lot of us forget to do. Ask if the client has ever worked on a project like this before. If this is their first website project ever, they will have zero expectations in terms of time and complexity and cost and all of these very, very valuable factors. I said this before a couple years ago in a project management talk, but good projects do not happen in spite of bad clients. And there are bad clients out there, people who are, and usually not because they're evil people, but they're inexperienced or they just have unreasonable expectations or it just may be a company culture that's very counter to producing the kind of work that we do in this industry. But make sure that you like the client. You're going to be spending a lot of time with them if you get this job, but make sure that they're prepared to be a stakeholder on a project of this size before embarking on this. I can't stress that one enough. Here's another interesting one that we normally don't think about. Does the client actually respect what we do or we just bunch of code monkeys or techies or whatever? Hey, you just do whatever, you internet people, right? That means that they're not going to respect the work that you provide them and they're not going to respect the product. So are they actually approaching you with some degree of respect about the complexity and the creativity behind what we do? And you can always offer alternatives to the RFP proposal process. Everything's a negotiation, right? So you can offer up things as opposed to just delivering a proposal to them, which if you're just basing everything off of comparing proposals, it becomes an essay writing contest, right? It's a numbers game and essay writing. And what we do for a living is not crunch numbers and write essays. It's make websites. So you can offer to write an evaluation rather than a proposal. And there's a really great article about this approach in Smashing Magazine from I think beginning of this month, very recent. But there's a short little link and these slides are available for download on the session page as well. Suggest an RFI, a request for information as opposed to an RFP process. And the request for information is low commitment and allows you to exchange just a couple of small pieces of information or a phone call or a document that says, hey, we the vendor, this is what we're about, this is what we do, these are the kinds of things that interest us. And then the client says, this is vaguely the project, this is what we want the goals to be, here's who the players are. And if you both agree to continue forward, then you can go through with the RFP. And that's gonna save everybody a ton of time. This, I joke that this was the speed dating of RFPs, but it really allows you to establish if you are a cultural fit with that company, which is another often overlooked but really huge component of this process is, are you really gonna be able to work together from a philosophy standpoint of what both sides are really out to achieve with this? And so even just a quick meeting, phone call, in-person meeting before you go through and base this all on numbers, can really save everybody a lot of time. And if it goes really well, then you've got a leg up when you actually are submitting numbers. If it goes really well, then you've got, you've established that relationship, you've got a bit more of an advocate in that organization and it's gonna go better for you. So where do we go from here? Crystal, do you have any final thoughts you wanna add? I feel like we've been through a lot of things that we hashed out here, but just it's a process and there's a wide range of things that an RFP can be from the different sizes, but a lot of these alternatives and being open to suggesting these other alternatives of even charging for evaluations, answering with a request for information, the four kitchens model of, hey, can we just come out and spend some time with you guys and get before we go through this process? These kind of things can really get around a lot of the pain points of this as well as just the overall. Just as we, the design community has done a pretty good job of saying just say no to spec work. The development community really needs to get strong about just saying no to open bids, to open RFPs, open things out there, because this is a big investment, especially if you're a small shop. This is a very big investment of your time and put it out there wisely. The payoffs can be huge if this is done well, but the costs can also be very prohibitive. So just be careful out there. Brian? So I would say my closing thoughts is anyone who's making an RFP or responding to an RFP, well, I guess making an RFP. I would say RFPs are kind of a, they serve a very specific function, and I think that you should try to be as open as you're asking the respondents to be with your own processes, your own selection criteria, your own budget, your own stakeholders, and really get to know the people that you're asking to respond to those bids. Because at the end of the day, the RFP is kind of the criteria and selection process around how a decision is made, but you're gonna be working with somebody. And at Lullabot, I know for sure a huge, huge portion of what we evaluate when we're evaluating projects and clients to work with is if they're a culture fit and if we want to work with them, if they're good people that we enjoy being around and that challenge us in new and interesting ways, and those sorts of things don't come through an RFP unless you open yourself to that. And when you do that, I think you get a lot better bids in return and you end up with a really good result as opposed to just putting it out there for someone to bid. Zach? For clients, I would say do your homework, understand your own organization and its institutional preferences, and think of vendors as partners. And I would like to suggest that if we all stop responding to RFPs, they will stop writing them. But what we really want to do is maybe eliminate RFPs, but of course you can't eliminate the bid process. We want to create a responsible bid process. So we want to leave some time for Q&A, but I realize that we're pretty short of time, so if you need to leave, that's fine. We have some resources listed on the slides, on the website, so I'm just gonna skip past those. If you could leave us some feedback, that would be really great. Thank you very much and we'll have some questions immediately afterwards. So if you do have questions, just feel free to go up to that mic there so they can record it for the video. Hi. I think it is. Yeah. Hello, okay. I know you kind of mentioned that the RFP process, traditionally speaking, is more or less dead. Not necessarily in those words, but is it appropriate for clients to ask vendors for their recommended RFP format or their document template if one exists? How, is it appropriate to ask for, how do you want to receive an RFP? Yeah, I'm not sure I understood that. I didn't quite hear. Yeah, could it be possible to turn up the audience mic? I believe that the question was whether it's ever appropriate for a client to ask a vendor for their preferred response format. From my own experience, that would be a huge red flag to me as a vendor that would say this client hasn't done their homework. And so I'd love to hear what the other panelists have to say on that, but it would come off to me as really wishy washy and we don't know where to start with this. As a follow-up real quick, is there an approved, approved, loosely speaking, an approved format or a recommended format to create your RFP? Because then you have some that are 30 pages long and they have all kinds of project points or are there a selection of key points that this is what we need to know right now. Everything else is fluff. I've seen some long ones that are just awful. I can't imagine taking up 30 pages. And I think that just in the spirit of the panel I would say that there is no approved format that we have and we have used them in the past but we're very open to moving away from them so long as we can establish rigor around the process. Thanks everybody, this is very helpful. I run a small Drupal shop in Boston. We work for a lot of non-profits and many of them have really no in-house technology resources at all. So Zach, your idea about the expository sketch sounds terrific, but it seems to assume some technology knowledge on the part of the people writing it. Do you have anything that you would recommend for a small non-profit or like a small, even a small university's plan for writing something like this to get around the problem of not having very much technology knowledge? No. So in cases like that, yes we did, we actually talked about this a little bit. We talked about this a lot yesterday. The idea is if you don't have in-house technical resources to help guide your RFP or whatever the equivalent is process, you can always hire a company to do this for you, to do the discovery for you to write an evaluation. Yes, you can do that and a lot of companies do that for flat fees. So they won't necessarily be doing it time and materials by the hour or something like that. So you can always ask vendors like, hey, you guys are great. We have a project, can we pay you to help us write our RFP or evaluation? Awesome, thank you. Sure. Very informative. I came from a corporate background, so I've been the client. I became a vendor as a freelancer and now I'm kind of growing my business. I track all my time religiously and I have a couple observations. One strategy you can use is to track your time that you're putting into an RFP and work that into the quote. Sometimes it happens that it takes a long time if you are tracking your time. That can happen.  But sometimes you're dropping everything in a really critical moment with another project and that is another cost that you have to consider when you're doing this. Absolutely, good point. Thank you. A question for mainly Zach, I think. Most Drupal firms have now adopted some sort of agile development methodology, yet RFPs encourage more of a waterfall approach around a defined budget and scope. So I'm curious if you've looked back at the historicals around your projects around how much of the cost that the vendor originally bid came in, how accurate was it? And how much do you account for iterations and agile and things like that? Well, first I would say that we're totally open to agile processes. We have seen some budgets go way over that were a fixed bid. And recently I've gotten some access to our procurement records and I've seen a history where a project will start out and then go through some phase and there's this string of change orders. So I don't know if we're really even, I mean, going back years, I don't know if we were even really paying that much attention to the total cost of some of these. So that is with a more waterfall approach, I think that in a fixed bid, that is cost overruns is something that we have experienced recently. And with a more agile approach, I think one of the interesting options that we have at our disposal is time and materials based billing with a not to exceed amount. Because if you're doing agile right and you have a project backlog, you're gonna see these kinds of problems coming way before you get to that point. Agile takes a lot of trust and I've pitched in one actually several RFPs that started off very waterfall written and we've ended up actually winning them on an agile approach. It's a difficult thing to balance as far as how much you promise and how much you try to give them a roadmap for what the agile process is going to be like and setting expectation around those fees. It's possible, but more than anything, and I know Todd has a lot of experience with this as well, it's that client has to have a lot of faith in your team that they're going to be getting a good amount of value for every sprint, when it's not laid out explicitly with deliverables and an SOW of exactly what and when, that they're going to be getting very high value from very fast thinking, good, solid people to give you that kind of free reign that an agile approach really requires. Right, yeah, the agile and how it fits into this process is sort of a discussion unto itself and there's all kinds of ways that we get around it. Unfortunately, there's not nearly enough time to go into our process right now. So about five minutes over at the moment, if we could just do like last three questions real quick, that would be great. Sure, great panel by the way. All of you, Zach, I appreciate your transparency with your organization. My name's Darren, by the way, but I'm just curious to get your thoughts on building more trust in the RFP process at the time of which you're asking about budget. We encounter so much resistance at that point and without revealing too many trade secrets, I can imagine, just Brian, Crystal, Todd, any thoughts on that, I'd appreciate it. So I kind of take the approach of asking for rough orders of magnitude, understanding that there's a really natural and valid reason that people will not want to say, my budget is exactly this, but by doing rough orders of magnitude, 10,000, 100,000, you sort of can get a feel for what they're thinking without pushing them too hard on that topic. That's usually what I do. For my part, I get them to tell me and then without disclosing a number, I talk to the vendor and I let them know if this is appropriate or not, because I have a sense of what, you know, about the project itself and the vendor's requirements. Sure. I just try to get as much open honesty out there as soon as possible and if that counts against you, that's probably not a good sign for the client, but if you're willing to put that out there early on and be honest about it and not be trying to play games or, you know, get more than, you know, if you're not trying to play games with it, just be honest about where you're comfortable taking on that project. You can get things off to a good start that way with just some really upfront honesty. Thanks guys. Thank you. Yeah, hi, my name's Ian, it was a great presentation. I just started a new job. This is week three. We have no process. We're a client company dealing with vendors. So I have a specific question to the gentleman from Stanford. You said that when vendors ask questions, you share the answers with everybody.  They're doing all the research, asking all the good questions, but everybody's getting the benefit. Well, I think the fairness component, thank you. I think the fairness component comes in where if you didn't think to ask any questions, you don't get the benefit of that process.  We're still trying it out. And we happen to find that a lot of the questions were similar or were nuanced around the same sorts of ambiguities that might have been in our sketch, but were qualitatively similar so we didn't feel like we were, you know, giving an unfair advantage. One final comment. I'm going to be developing my own process. If anybody answers, isn't there a similar vote and will be interesting in collaborating on developing a client process? I'd love to talk to them. Great. So to speak very quickly about the Q&A thing, Zach and I did this and I found that process very useful because I don't think there was a vendor that sort of asked like one softball question and then anticipated getting a whole bunch of really great feedback from everybody else.  There's no such thing as, hey, I read all your RP. I've got just one tiny little question. No, you have a ton of questions. So a lot of the questions did overlap. A lot of the questions that other vendors asked were very good and there were things that we didn't necessarily catch. So the idea of if you participate in the Q&A, you benefit from the Q&A is actually, in my experience, has been tremendously fair. Hi guys, my name is Terrence and great presentation. Thank you. My business partner Dustin and I have a small shop in San Diego that we launched last fall and I believe that our view is that the RFP process is a good way to market ourselves for the time being. And so I'm curious, Todd, from your perspective, do you believe it would have been good for four kitchens to adopt the no RFP model early on or did you actually benefit it from it in the early days? Our first big jobs didn't involve RFPs. People saw work that we had done that was not client related work but kind of stuff that we did for fun. We find that marketing ourselves in terms of outside projects and open source involvement gets us the most business, not the RFPs process. But I do recognize that it's unfair to say to people who are getting started in the business, oh, RFPs are terrible, don't do them. How else are you supposed to get business, right? It really comes down to even though as a new company that perhaps isn't very well known or you're just getting started or something, even when you engage in the RFP process, somebody has to take a leap of faith on you because you don't have the portfolio, you don't have the experience, you may not have the reputation. So whether you do RFPs or not, it comes down to somebody taking a chance on you. You're gonna get a better ROI, especially in this community by getting out there and contributing back to the community and blogging about that, contributing modules, contributing themes, getting active in the camps and here, and that's a similar time, it can be a similar time and cost outlay and I think you've got a much better expected return from getting that stuff out there and you're helping the community at the same time. Thank you very much, everybody. Thanks, guys.


Everybody force and Jeff here. We got a 20. Let me disable this real quick 2023 was a football no huddle five bucks number 10 random teams can be just a sec guys and we will get started here The random and then we'll rip look. We'll do the random now Five and one six times names first one two three four five and six James D down to Peter K and six times on the teams One two three four five and six Detroit down to Seattle Really Lane that's interesting What's your best hit out of both? It's got to be from hobby. I guess James D lion Santino commanders mark D Niners Joe are Jags Keith be chiefs James D Falcons John S. Browns Marcos with the Cowboys James D Packers Hanno with the Bears Victor F. Giant Scott M. Dolphins Jamie with the Titans Hanno with the Bengals mark C Rams Taylor R. Bills Robert F. Texans Richard H. Saints Eric L has the Raiders Tom C. Bucks Andrew C. Panthers Wayne P. Vikings James D Broncos Heath G with the Ravens Mark D. Cardinals David F. Chargers Jerry H. Jets James D. Patriots Marcos with the Colts Cameron F Steelers Brandon S. Eagles Peter K. Seahawks results are posted here Killed it Good luck everybody. Let's see if we can find a black disco here. I think we pulled one last time we opened this This guy wants to dance, so we need a disco or turn off the lights Whoa Disco Devontae Adams Raiders I Think it's a good move for the Raiders Silver rookie is down for the Colts disco Purdy Montage There's a strad bass We go Houston When I saw what the Silver is out of prison and was egg or strad were selling for my jaw hit the floor last weekend We were at like I was looking up like in between like the commercials at Frogger's last week Look at up consoles like oh my god And then hooker there disco Roquan Smith for the Ravens Staggering manual for Ms. Will Anderson I've missed plenty purple Dan Marino dolphins out of 50 Purple Marino that guy God Diggs and digs and lamb carbon copy. I'm gonna roll a die for the carbon copies I don't think there's numbered carbon copies and no huddle, but there are or random instead But they'll all go one direction for the break one two three left four five six right Three I'll go left. I'll go left if there's a numbered one will random it instead Kincaid and Bennett's Dusty Sherman. That's pretty solid actually Disco rookie is Derek call for the Seahawks gold shimmer reach Orange sharp sugars under stock of a shot. Oh, that's pretty good. Actually that's a pretty good box I Was better than the first one you ripped That's pretty good Louis scene Vikings on the Scripps here was a Scripps auto silver is burns for Carolina Just go to a montage I know dude.  I forget you said me. I Don't know what I said, but I was like, yeah, look at this Or something like that. I brought it to your attention as soon as I saw the guy Yeah, I was just okay makers rams before even knew where he was sitting Before even knew where he was sitting and luckily you sat right about four feet right next to me There was like a small aisle. Yeah, just It's Ron Payne blue disco to 75 for the commanders folded him up I Met a mastery Roger. There's a leather space. Maybe we'll have to repeat it out tomorrow that lanes Yeah, I'm inviting the guy don't invite that dude, please I Don't want to like profile somebody But that guy the guy you said what those guys are up to no good trouble.  Yes Even for Florida. It was in like the low 40s like this dude's walking in in that like Cortes is talking to right Purple on a D. K. Ozama for the chiefs are in them talking the way they were talking you could tell their little bugs Silver MacLaurin for the commanders Travis Kelsey disco for the chiefs I Don't go out in the south to be the next UFC champ Dude they were not like they were like Very poorly like he did it himself and it was like to the day was like knee-length like past the knee-length George I don't know how to explain it. Let's go. Micah Parsons Cowboys. Just did that's pretty good man price young Jeans that he cut into George's that were like below his knees and they were like old and worn because you could tell there was like The hanging like loose parts total loser. I don't think he's about to go red He looked like he's about to go wrestle and alligator to be honest. Yeah Gold Garrett Wilson. That's pretty cool. Let me go jets Lamar Jackson moments in time. Oh wow razzle dazzle CJ Stroud short print Yeah, we are we are crushing it tonight Texan's crushing it right now We had him in this break. Yeah, it's a monster card Robert F with Houston That's pretty fire razzle dazzle SP Robert the old razzle dazzle right there Whoo Bingo is right you see Lane's shoes The shoes that Cortes got them Yeah, I think so the purple and black. Yes. Yeah, dude fire. They're so awesome. It's just bingo on the back Yeah, the cuffs. Oh, yeah They're sick. They're sick The game for him tomorrow, I feel like even if they lose my life doesn't even matter for his like value No The one they got that huge that was big so be John debut disco Larry Johnson chiefs, but the value can always go up I'll say that I don't think losing will hurt him tomorrow, but the value can always go up Dang Lane, that's crazy compared to what it would have been like three months ago. Yeah Even like a month and a half ago because the razzle dazzle as far as SPs go or not like the hottest ticket item Pink disco mac jones out of 20 the Pats disco montage tealaw Schoonmaker and Clayton soon If it weren't for the Jags everyone would be looking at the Eagles like that and they kind of are but the Jags really really did it will levis Disco waddle for the Dolphin Jalen Carter rookie blue rookie friends in strange out of 75 for the Jaguars Silver dalsfark Colts disco DK and AJ Brown Like how they let the Texans come back when that division is insane to me Disco or locker for the Bears Yeah, like I get Lawrence was hurt for a lot of it and like probably wouldn't have played if they had More games at that point, but holy cow Purple Jalen hurts national pride out of 49 To 50 Jalen hurts with Eagles disco JJ at us with Vikings Disco is Vince Young Titans DJ Turner touchdowns Yeah, we do a league here. You take my spot in the fantasy league. Do you take mine? So far it's her and more alliance. I don't know it was just go Russ and Kyler Isley to the apple in November With her she paid more attention than I did. Yeah Lane can have my second Mike. I think I have my team. We got two teams Blue Disco is Cortland Sutton for Denver out of 75. You're not running a next year. Did I told rich? I wasn't gonna do it. It's like you have to Like I don't wanna Disco met a mastery Lawrence. Thanks. I don't have I literally do not have time I'm doing one. I can't even keep up with the one that I've been in for like years at this point Like I really can't keep up. I'm the only person that's like not active in the group chat of that league Every Sunday, I'm here for like 14 hours. Disco rookie is Jake Moody for the Niners Skarotsky and Devon Hane Also worked every Monday night I get to watch a rich is the only one that wants to do it I See today was on the jets on the autograph Jets over burrow Randy Moss moments in time disco league in my pick-and-league. That's it See pick-and-zeezy. Yeah, yeah, especially if it's a weekly like payout type thing. That's that's when it gets like, okay Yeah, I'll do that. I'll get you in it. I don't think I'd be able to keep over that to be honest I'll be like Kyle. I just forget half the time Kyle would start purses like oh I forgot Yeah, you'll have it 10 a.m. on Sunday and be like I'll get to it But you've got to submit all of them by the start of the first game of the weekend, right? Yeah, as we're get tricky on this on the Saturday on the Sunday games their morning you can change them. Yeah Disco rookie Charlie Jones for the Bengals Have a Thursday night game in before pick your Sundays. Yeah All right, they can do like it. They can do like a five-person league Yeah Disco Jay-Jay CJ Stroud debut with the disco of Woodson for the Raiders jimmy or Gibbs and Anthony Richardson Yes, boy, Anthony Richardson At least you got a good QB this you got a good QB Purple brusky out of 50 for the Pats Nicobos Almost one one weekend to pick them, but I'll tell you what I am a Dodger fan Yeah, and have been for very long. I'm a Dodger fan. Yep. Just go Bryce young debut for the Panthers Dodgers tank down Listen that you see that over there go read that see that Good Sam Leporta rookie autograph for the Lions. There we go Detroit's I Met him he was Wilson Charbonnet Addison Yeah Great time it was a really nice guy He was Disco Munoz bangles and a hooker there Yeah, Kyle and that pick them league started out like up top and then I Don't know Berks is 75 Titans in the blue. She was way below me Just go rookies ring go for the Eagles One week you make a gold rookie for the Raiders jacorian Bennet But it's an jacorian Bennett gold silver mark Andrews disco introductions Quentin Johnston Is a juicer like a big card Juicers yet disco Barry Sanders Lions juicer is someone that uses for dancing drugs We did pull a shake earlier There's a razzle dazzle shroud right there out of an old box is huge Bijan Yeah, we had some big hits tonight though. We had some big cards blue to 75 welker for the Pats Lou B. What up? Oh Boy, Lou B's here. Lou B Hello This is Jeff They're up by like 10. Yeah, I like half-time. Yeah, if you want a one's a definitive earlier What's going on Blake? Let's see me to do they work here for a bit. Oh, it's friend of ours. Yeah Minor league pitcher Righty big nerd to disco Paul Krauss Paul and solid, you know, that's some size to him. It's also the biggest nerd. He is the biggest like the biggest nerd He saw Tigger Blue disco to 75 Munoz for the Bengals. Yeah, well, I forgot that I took a picture with King Louis that day Few So walk out like boom got to get a picture of him didn't have to wait in line or nothing just walked up. It was like, yep Security that was a good time disco a painter for the Saints Yeah, Vaughn was in here earlier not being too happy about it Team bonding Pink disco is Najee Harris at 20 for the Steelers if you silver musk rave for the Packers Klingon Just go Tyree kill I was gonna say it probably cost a few hundred bucks. Yeah, that pickle ricks easily probably a grand Yeah, oxen Just go take it to the bangles Honestly, if we auction that one didn't share you probably get the Auction pickle rick for charity, dude, let's do it. We should I'll auction my Dolphin chain blue disco is Ricky Moody for the Niners out of 75 with a trash list right on it on the back of it Ray flatbill a Lane hat Just go Willis for the Titans, I'm still trying to get my hat Lane said well, what do they say woe to I got the halo happy Silver Ray Lewis for the Ravens Just go Derek Henry Anderson there disco is Dallas Clark Yeah, no new era hats are not cheap Purple the white Stevenson out of 50 we're gonna send it to cars for kids No, I said one of the Boston hats I mean if you go to like lids.com or something there's way too many hats of every single team You just get lost in everything Ricky disco lot two for the Niners, but they're not cheap You line special hat Jesus all about the you line hats Silver genus for the Seattle disco introductions Addison He's I'm lost. I'm just rolling with it. Sean. Yeah redemption Sharpen a no huddle blue auto for the Seahawks Zach Sharpen a all right last box Here's all breaking out the Laporta auto Obviously the razzle dazzle has been big See if it's cars for kids lands in I think they'll take pickle Rick For a tax write-off Bronson talent, man. This is crazy This is a rookie McDonald for the Jets ain't no no, I know idea. I was more joking But I Support the only support Jewish kids silver Justin Herbert for the Chargers disco is Pat Peterson Sharpen a and Addison Sean I think Sean's done supporting kids that would be fair. Yeah, I'm done Cars for Sean is the new thing cars Yeah, we'll give you lanes. We'll give you lanes old cars. I think cars are kids will get that one back actually that one just go to it's Sean's looking for Corvette Corvettes for Sean Trying to get rid of the Lambo dodge Dodge Ram for Sean. Don't go trade it in cars for Sean. Donate it Yeah, should be solid Blake drifting in no time in it Go nice Aiden Hutchinson Second year for the line lines are really well here. That's the 10 7 to 10 This go McCaffrey I don't know what you're speaking in tongues or something. I'll definitely take your new car bow show This is a Roger from Jets taking Lane's Kart of Cleveland this year Yeah, but just look like the grizzles with all the stuff on the roof put a we could put a hitch on trailer Addison a guy DTR on the Browns rookie auto I'll tell rich. That's the plan Lane. We'll strap you know as soon as I mentioned something like that to rich She's like, all right, you guys can drive this silver can make up for the Panthers And then we all get volunteered disco Strange everybody else gets to fly. Yeah, we drive Drilled freaking Kentucky and Indiana. Come on man I'm only going to the national forties goes this year Disco is the hop for the Cardinals Jordan battle. I'll teach score tease how to do the breaker set up at the national. She can go do that Also, I'll send Jeff There you go. Let's go. I see my own bed for that whole week. Yeah, I'm down with that Why'd you take PTO and visit the national? You know if I took PTO for that There's no way I'd even get that approved off first of all blue disco is Dickerson for the Rams at a 75 This is your percent chance of that happening. Let's go. Dak.  Okay It was hot there That's right. We saw you there. Yeah, this is a rookie is Williams First Yeah, that was the purple Leonard at a 50 for the Colts Slover Marquis Brown Cardinals disco Josh Jacobs Disco is Tim Brown for the Raiders There's a fair show for Cleveland that sounds like a waste of space Bolton to 75 Sam would for sure be wearing a propeller hat. Yeah dunce that Yeah, oh, it's awesome All right, never cards of bolden Dickerson moody Munoz Welker Burks Sutton strange pain at Marino Usama Stevenson Hertz Shaquille Leonard Najee Harris Bruceki Mack Jones golds of Garrett Wilson jacquory Bennett's and In Hutchinson three golds in five boxes is really good Autos of Dorian Thompson Robinson Sam LaPorta, man lines is really well at the gold hutch to CJ Osama, Lewis scene Seahawks Redemption Zach Charbonnet and the razzle dazzle SP of CJ Straff for the Texans That'll do it. Thanks to get everybody Moving on to more football


From the SiliconANGLE Media office in Boston, Massachusetts, it's theCUBE. Now, here are your hosts. Dave Vellante and Stu Miniman. Cloud is dead. It's all going to the edge. Where is it? Hi everybody, this is Dave Vellante and I'm here with Stu Miniman. Stu, where does this come from, this narrative that the cloud is over? Well, Dave, you know, clouds had a good run, right? It's been over a decade. Amazon's dominance in the marketplace, but Peter Levine from Andreessen Horowitz did an article where he said, cloud is dead, the edge is killing the dead, the edge is killing the cloud, and really we're talking about IoT and IoT's huge opportunity. Wikibond, Dave, we've been tracking for many years. We did the original forecast for the industrial internet and obviously there's going to be lots more devices at the edge, so huge opportunity, huge growth, intelligence all over the place, but in our viewpoint, Dave, it doesn't mean that cloud goes away. We've been talking about distributed architectures now for a long time. The cloud is really at the core of this. Building services that surround the globe live in just hundreds of places for all of these companies. So it's nuanced and just as the cloud didn't overnight kill the data center and lots of discussion as to what lives in the data center, the edge does not kill the cloud and we're seeing some major transitions pull and push from some of these technologies and a lot of challenges and lots to dig into. So I read Peter Levine's piece. I thought it was very thought provoking and quite well done. And of course he's coming at that from the standpoint of a venture capitalist, right? Do I want to start, do I want to pour money into the trend that is now the mainstream or do I want to get ahead of it? So I think that's what that was all about. But here's my question, Stu, is in your opinion, will the activity that occurs at the edge, will it actually drive more demand from the cloud? So today we're seeing the infrastructures of service businesses growing at what, 35%, 40%? Sure, sure. Amazon's growing at the 35% to 40%. Google, Microsoft are growing double that right now. So but overall you're right. Yeah, okay. And then of course the enterprise players are flat if they're lucky. So my question is, will the edge actually be a tailwind for the cloud in your opinion? Yeah, so first on your comment there, from an investment standpoint, totally can understand why edge is green field opportunity, lots of different places that I can place bets and probably can win as opposed to. If I think that today I'm going to compete against the hyperscale cloud guys, they're pouring $10 billion a year into their infrastructure. They have huge massive deployment so the bar to entry is a lot higher. I'm sorry, the second piece was. So will the edge drive more demand for the cloud? Yeah, absolutely. I think it does Dave, because let's take something like autonomous vehicles, something that we talk about. I need intelligence at the edge. I can't wait for some instruction to go back to the cloud before my Tesla plows into an individual. I need to know that it's there. But the models themselves, really I've got all the compute in the cloud. This is where I'm going to train all of my models, but I need to be able to update and push those to the edge. If I think about a lot of the industrial applications, flying a plane, things need to happen locally, but all the anomalies and new things that we run into, there's certain pieces that need to be updated to the cloud. So it's kind of a multi-layered if we look at how much data will there be at the edge? Well, there's probably going to be more data at the edge than there will be in the central cloud, but how much activity, how much compute do I need? How much things do I need to actually work on? The cloud is probably going to be that central computer still, and it's not just a computer. It's, as I said, a distributed architecture. That's where, when we've looked at big data in the early days, Dave, when we can put those data likes in the cloud, I've got thousands or millions of compute cycles that I can throw at this at such a lower price and use that there, as opposed to at the edge, especially what kind of connectivity do I have? Am I isolated from those other pieces? If you go back to my premise of we're building distributed architectures, the edge is still very early. How do I make sure I secure that? Do I have the network? There's lots of things that I'm going to build in a tiny little component and have that be there. And there's lots of hardware innovation going on at that edge, too. Okay, so let's talk about how this plays out a little bit. And you're talking about a distributed model and it's really, to me, a distributed data model. The research analysts at Wikibon have envisioned this three tier data model where you've got data at the edge, which you may or may not persist. You've got some kind of consolidation or aggregation layer where it's kind of in between the edge and the deep data center and then you've got the cloud. Now that cloud can be an on-prem cloud or it could be the public cloud. So that data model, how do you see that playing out with regard to the adoption of a cloud, the morphing of cloud and the edge and the traditional data center? Yeah, we've been talking about intelligent devices at the edge for a couple of decades now. I mean, I remember I built a house in like 1999 and the smart home was already something that people were talking about then. Today, great, I've got my Nest, if I have, I probably have smart assistance. There's a lot of things, I love that. So I'm on Twitter today, somebody's talking like, oh wait, I'm waiting for my light bulbs to update their firmware from the latest push. So some of it's coming, but it's just this slow gradual adoption. So there's the consumer piece and then there's the business aspect. So we are still really, really early in some of these exciting edge uses. Talk about the enterprise. They're all working on their strategy for how devices and how they're going to work through IoT, but this is not something that's going to happen overnight. It's, they're figuring out their partnerships, they're figuring out where they work and that three tiered model that you talked about is my cloud provider, absolutely hugely important for how I do that and I really see it, Dave, not as an or, but it's an and. So I need to understand, right, where I collect my data, where certain aspects are going to live and the public cloud players are spending a lot of time working on that intelligence layer. And Stu, I should mention, so far we're talking about really the infrastructure as a service layer, it comprises database and middleware, we haven't really addressed the SaaS space and we're not going to go deep into that, but just to say, I'm going to look packaged software as we knew it is dead, right? SaaS is where all the action is, it's the highest growth area, it's the highest value area, so we'll cover that in another segment. So we're really talking about that, the stack up to the middleware and the database and obviously the infrastructure as a service. So when you think about the players here, let's start with AWS. You've been to, I think every AWS reinvent maybe with the exception of one, you've seen the evolution, I was just down in DC the other day and they have this chart on the wall, which is their releases, their functional releases by year, it's just overwhelming what they've done. So they're obviously a leader, I saw a recent Gartner Magic Quadrant, it looked like, I tweeted, it looked like Ronnie Turcott, looking back on Secretariat from the Belmont and whatever it was, 1978, I think it was, 31 lengths. I mean, massive domination in the infrastructure as a service space. What do you see going on? Yeah, so Dave, absolutely. Today the cloud is it's Amazon's market out there. Interestingly, if you say, okay, what's some of the biggest threats in the infrastructure as a service? Well, maybe China, Dave, Alibaba's one that you look at there, but huge opportunity for what's happened at the edge. If you talk about intelligence, if you talk about AI, talk about machine learning, Google's actually the company that most people will talk about, it can kind of have a leadership. Heck, I've even seen discussion that maybe we need antitrust to look at Google because they're going to lock things up. They have Android, they have Google Home, they have all these various pieces, but we know, Dave, they are far behind Amazon in the public cloud market and Amazon has done a lot, especially over the last two years, you're right, I've been to every Amazon read vet except for the first one and the last two years really seen a maturation of that growth, not just devices and partnerships there, but how do they bring their intelligence and push that out to the edge? So things like their serverless technology, which is Lambda, they have Lambda green grass that they can put to the edge. Serverless is pervading all of their solutions, they've got like the Aurora database. And serverless is profound, not just from the standpoint of application development, but just an entire new business model is emerging on top of serverless and Lambda really started all that, but carry on. And when you look in and you say, okay, what better use case than IoT for, well, I need infrastructure, but I only need it when I need it and I wanna call it for when it's there. So that kind of model where I should be able to build by the microsecond and only use what I need, that's something that Amazon is at the forefront, clear leadership position there and they should be able to plug in and if they can extend that out to the edge, starting new partnerships, like the VMware partnerships interesting, Red Hat's another partnership they have with OpenShift to be able to get that out to more environments and Amazon has a tremendous ecosystem out there and absolutely is on their radar as to how they push out the IoT. So we were at Google next last year, big push verbally anyway to the enterprise and they've been making some progress, they're hiring a lot of people out of formerly Cisco, EMC folks that understand the enterprise, but beyond sort of the AI and sort of data analytics, what kind of progress has Google made relative to the leader? So in general, enterprise infrastructure service, they haven't made as much progress as most of us watching would expect them to make. But Dave, you mentioned something, data. I mean, at the center of everything we're talking about is the data.  They're skating to where the puck is Dave and infrastructure services. Last decade's argument, if it's the data and the intelligence, Google's got just brilliant people, they're working at some of these amazing environments. You look at things like Google Spanner. This is distributed architecture. Say, how do I plug in all of these devices and help them work in a distributed gradual work? Well, heck, I'd be reading the white papers that Google's doing and understanding that they might be really well positioned in this 3D chess match that we're playing. Your eyes might bleed. I've read the Google Spanner. I was very excited about it and understood a little bit of it. Okay, let's talk about Microsoft. They're really of the big cloud guys. They're really the one that has a partnership strategy to do both on-prem and public cloud. What are your thoughts on that now that Azure Stack is starting to roll out with some key partners? Yeah, absolutely. It's the one that, Dave, if you use your analogy, looking back, it's like, well, the next one, it's gaining a little bit, gaining a little bit, but still far back there is Microsoft. Where Microsoft has done best, of course, is their portfolio of business applications that they have. They've really turned the green light on for enterprises to adopt SaaS with Office 365. Azure Stack, it's early days still, but companies, they use Microsoft, they trust Microsoft. Microsoft's done phenomenal working with developers over the last couple of years. Very prominent, like the Kubernetes shows that I've been attending recently. They've absolutely got to play for serverless that we were talking about. I'm not as up to speed as to where Microsoft sits for kind of the IoT edge discussions. You know, they're playing there. Absolutely, I mean, Microsoft's identity better than anyone, Active Directory is still the standard in enterprises today. So, I worry that Microsoft could be caught in the middle. If Google's making the play for what's next, Microsoft is still chasing a little bit what Amazon's already winning. Okay, and then we have enough time to really talk about China, you mentioned it before, Alibaba's legit 10 cent Baidu, obviously with their captive market in China, they're going to do a lot of business and they're going to move a lot of compute and storage and networking, but we maybe address that in another segment. I want to talk about the traditional enterprise players, Dell, EMC, IBM, HPE, Cisco, where do they stand? We talk a lot at Wikibon about true private cloud, the notion that you can't just stick all your data into the public cloud. Andy Jassy may disagree with that, but there are practical realities and you're certainly, when you talk to CIOs, they underscore that, but that notion of true private cloud hasn't allowed these companies to really grow. Now, of course, IBM and Oracle, I didn't mention Oracle, have a different strategy and Oracle strategy is even more different, but so let's sort of run through them. Let's take the arms dealers, Dell EMC, HPE, Cisco, maybe you've put Lenovo in there. What's their cloud strategy? Well, first of all, Dave, I think most of them, they went through a number of bumps along the road trying to figure out what their cloud strategy is. Most of them, especially, let's take the compute or server side of the business, they are suppliers to all the service providers trying to get into the hyperscalers. Most of them have, they all have some partnership with Microsoft, there's Azure Stack, and they're saying, okay, hey, if I want an HPE server in my own data center and in Azure, Microsoft's going to be happy to provide that for you, but David, it's not really competing against infrastructure as a service and the bigger question is, as that market has kind of flattened out and we kind of understand it, where is the opportunity for them in IoT? We saw, Dave, the last five years or so, can I have a consumer business and an enterprise business in the same? HPE tore those two apart. Michael Dell has kept them together. IBM spun off to Lenovo, everything that was on the more consumer side of the business. Where will they play or will companies like Google, like Apple, the ones that, Dave, they are spending huge amounts of money in chips. Look at Google and what they're doing with TPUs. Look at Apple, I believe it was, there was an Israeli company that they bought and they're making chips there. There's a different need at the edge and sure, a company like Dell can create that, but will they have the margin? Will they have the software? Will they have the ecosystem to be able to compete there? Cisco, I haven't seen on the compute side them going down that path, but I wasn't Cisco live and a big talk there. I really like the opening keynote and we had a sit down on theCUBE with the executive. It said, really, if I look out to like 2030, if Cisco's still successful and we're thinking about them, we don't think of them as a network company anymore. They are a software company and therefore, things like collaboration, things like how it's kind of a new version of networking that's not on ports and boxes, but really, as I think about my data, think about my privacy and security, Cisco absolutely has a play there. They've done some very large acquisitions in that space and they've got some deep expertise there. But again, Dell, HPE, Cisco, predominantly arms dealers, obviously don't have an HPE at one point, had a public cloud that pulled back. HPE's cloud play really is cloud technology partners that they acquired. That at least gives them a revenue stream into the cloud. Now, maybe it's- It's a consultancy. It's a consultancy. Now, maybe it's a one-way trip to the cloud, but I will say this about CTP. What it does is it gives HPE a footprint in that business and to the extent that they're a trusted service provider for companies trying to move into the cloud, they can maybe be in the cat bird seat for the on-prem business. But again, largely an arms dealer, it's going to be a lower margin business, certainly than IBM and Oracle, which have applications. They own their own public cloud with the Oracle public cloud and IBM cloud formerly software, which was a $2 billion acquisition several years ago. So those companies from a participation standpoint, even a tiny market share is compared to Amazon, Google, and Microsoft, they're at least in that cloud game and they're somewhat insulated from that disruption because of their software business and their large install base. Okay, I want to sort of end with the sort of where we started, the Peter Levine comment, cloud is dead, it's all going to the edge. I actually think the cloud era, it's kind of playing out as we, as many of us had expected over the last five years. You know what blew me away is Alexa. Who would have thought that Amazon would be a leader in this sort of natural language processing marketplace, right? You would have thought it would come from, so certainly Google with all the search capability. You would have thought Apple, but Siri compared to Alexa. So my point is Amazon is able to do that because it's got a data model. It's a data company. All these companies, including Apple, Google, Microsoft, Amazon, Facebook, the largest market cap companies in the world, they have data at the core. Data is foundational for those companies and that's why they are in such a good position to disrupt. So cloud, SaaS, mobile, social, big data, to me, Stu, these are kind of the last 10 years. The next 10 years are going to be about AI, you know, machine intelligence, deep learning, machine learning, cognitive. We're trying to even get the names right, but it starts with the data. So let me put forth the premise and get your commentary and tie it back into cloud. So the innovation in the next 10 years is going to come from data and to the extent that your data is not in silos, you're going to be in a much, much better position than if it is. Number two is your application of artificial intelligence. You know, whatever term you want to use, machine intelligence, et cetera. Data plus AI, plus I'll bring it back to cloud, cloud economics. If you don't have those cloud economics, then you're going to be at a disadvantage of innovation. So let's talk about what we mean by cloud economics. You're talking about the API economy, talking about global scale, always on, very importantly, something we've talked about for years, virtually zero marginal costs at volume, which you're never going to get on-prem. Because this creates a network effect. And the other thing it does from an innovation context, it attracts startups. Our startup saying, hey, I want to build on-prem. No, they're saying I want to build in the cloud. So it's data plus artificial intelligence plus cloud economics that's going to drive innovation in the next 10 years. What are your thoughts? Yeah, Dave, absolutely. Something I've been saying for the last couple of years. We watched kind of the customer flywheel that the public clouds have. Data is that next flywheel. So companies that can capture that. You mentioned Amazon and Alexa. One of the reasons that Amazon can basically sell that as a loss is lots of those people. They're all Amazon Prime customers and they're ordering more things from Amazon and they're getting so much data that drive all of those other services. Where is Amazon going to threaten in the future? Everywhere is basically what they see. The thing we didn't discuss there, Dave, I love your premise there is it's technology plus people. What's going to happen with jobs? You and I did the sessions with Andy McAfee and Eric Bronyolson. It's racing with the machine. Where is, we know that people plus machines always beat. So we spent the last five years talking about data scientists, the growth of developers and developers and the new king makers. So what are those new jobs? What are those new roles that are going to help build the solutions where people plus machine will win? And what is that kind of next generation of workforce going to look like? Well, I want to add to that, Stu. I'm glad you brought that up. So a friend of mine, David Michela, just is about to publish a new book called Seeing Digital. And in that book, I got an advanced copy. In there, he talks about companies that have data at their core and with human expertise around the data. But if you think about the vast majority of companies, it's human expertise and the data is kind of bolted on. And the data lives in silos. Those companies are in a much more vulnerable position in terms of being disrupted than the ones that have a data model that everybody has access to with human expertise around it. And so when you think about digital disruption, no industry is safe, in my opinion. And every industry has kind of its unique attributes, obviously publishing and books and music have disrupted very quickly. Insurance hasn't been disrupted. Banking hasn't been disrupted. A little blockchain is probably going to affect that. So again, coming back to this tail end premise is the next 10 years is going to be about that digital disruption. And it's real.  A cloud is obviously a key component, if not the key component of the underlying infrastructure with a lot of activity in terms of business models being built on top. All right, Stu, thank you for your perspectives. Thanks for covering this. We will be looking for this video, the outputs, the clips from that. Thanks for watching everybody. This is Dave Vellante with Stu Miniman. We'll see you next time.