We have a couple of minutes till 2.15, but I think we're gonna go ahead and get started if you guys don't mind. So welcome to the last session of DrupalCon. I'm kind of amazed that there's anyone in this room, actually, so thank you all for coming. We're talking today about the rescue project. Someone mentioned earlier that this is unfortunately an all too common scenario. So I hope that there's some lessons we've learned from a particular disaster that you might be able to apply in your own experience. So I just wanna get started by mentioning sort of who I am and where I'm from. So my name is Marcus Ianaltzi. I'm the principal and founder of Message Agency. We are a full service dev shop, design development strategy shop in Philadelphia. And we're also a B Corporation. How many people know what a B Corporation is? Okay, a few. We happen to, for those of you who know, we also happen to be a best for the world B Corporation. So we have a score that's in the top 10% of all B Corps. So we're super proud of that. And that's because we only focus on the nonprofit, higher ed foundation and government markets. So we can start ourselves a social enterprise. And this is a case study about a fairly large national nonprofit. And again, we have tried to scrub all identifying information because this is a particularly gnarly case. So some things I'll speak in general, generalities unfortunately, I'm sorry, but I have to do that. And it's really about a nonprofit organization, their budget to us is very precious. They're not coming by that money very easily. And it's really tough when a project has, they've already undertaken a project and it's been scuttled and they've sunk a lot of money into that, that sort of lost revenue for them really has an impact on what they can do. So there's a very heightened tension going into a rescue project for the vendor. And we're really talking a lot about how we negotiated that sort of those waters and how we turned the project around. So a little bit first about hindsight. So we all know that Drupal is super powerful, super flexible. Sometimes it's too flexible, especially for the uninitiated. And unfortunately what we've seen many times actually is folks who say, yeah, I can do Drupal. And they'll, there'll be a freelancer or a small shop and they wanna get into this space and they don't take the time to learn the tool. And so they bring to the platform a lot of assumptions and they don't bother to learn best practices and they really, you know, we all know hacking core and all of the other gnarly things that they do. Even for us though, for seasoned dev shops and we've been working with Drupal since four seven. So we've been working with it for quite a long time. You know, sometimes we have to take shortcuts, right? Sometimes the project is not a perfect project. Sometimes the client doesn't have the budget. They really need something that maybe the project wasn't managed as well as it could have been. And there are things you do that you might have done otherwise under different conditions. And we all kinda like to complain about how the other guy does it too. I mean, there's, you know, various degrees of bad and a shop who does really good work now maybe didn't do great work before and they got there. And, you know, so there's a lot of that that goes on. But, you know, the real sort of decision point here is as a firm or as a freelancer, you know, when one of these types of projects crosses your path, you know, what do you do about it? How do you evaluate whether or not you should take it? How do you think about what impact it's going to have on you and your firm? As well as whether or not it's really worth doing. So one of the things that, you know, that we didn't realize, especially with this project is the profound impact it was going to have on not only the current workload that we had but our potential to bring on new work.  These take a lot of care and feeding and it might actually limit your potential for bringing on other projects down the line. So that's something you don't always think about. So here's the scenario. You know, you come across this project, you're kind of excited. Maybe it was a project that your competitor did and you're excited to get your hands on it or maybe it's a project that, you know, you, you know, it's a great organization or you think it's something you want to add to your portfolio and you really want to do it because you want to be a superhero and you feel like, yeah, we can sweep in and we can save the day. You know, maybe there's a great budget too so that you're feeling like it's worth pursuing. For us, the way the scenario unfolded is we're very well known for doing Drupal and Salesforce integrations and actually back in 2007, we wrote the original Drupal Salesforce suite. So about 75% of what we do involves this. So folks come to us with these problems and they were migrating basically a custom CRM and it was built in ASP. It was terrible. It had been iterated on many, many times and so it was definitely, we knew once we started looking at that it was going to be very messy and it had a dot net front end but really the site was just a shell. I mean, it was just sort of spitting out queries to that really gnarly database. They had spent two years and we believe hundreds of thousands of dollars on both the Salesforce implementation and on a Drupal site that was built by a firm that the client never met. The firm was actually the Drupal firm. It was an overseas firm and that's no comment on overseas firms but the client actually never had a relationship really with that firm and wasn't involved in selecting them. It was the Salesforce, the shop that was customizing Salesforce that brought them on board and they were trying to take this, again, very gnarly database, put it in Salesforce and build a site around it. So the client was really unhappy with the result and I can't show you the site but I think if I did, you'd all be pretty horrified and they came to us saying, okay, we've got this, we've got Salesforce. I knew that would get a laugh and it's amazing how that percentage in particular is used and there's no rationale whatsoever for that value but that's what they said to us. It's 75% done and we kind of looked and we thought, okay and we saw the, there was a site, it was Drupal. There were a lot of views and some things that it was displaying. There was a Salesforce implementation. Data had been migrated but we weren't really quite sure where things were because there was no documentation and I'll talk about this a little later. So when we asked the client just in the initial conversation with them about whether or not we wanted to do the project, we said, hey, do you have any documentation? Do you have any kind of models, objects to find anything? And they said, well, we may have at one point. This was two years ago. So we were very, very skeptical and the first thing that we did, especially again, given the, how complex Drupal and Salesforce integrations can get and given the fact that this ran their entire business process and essentially what was happening, this project was going to have to replace wholesale their website and their CRM all at the same time. And that was a huge red flag and we thought there's a lot of risk there. So we were very, very skeptical. So there were other warning signs that we came across. Again, I mentioned, this was an all or nothing implementation. They had fired the Drupal developer and wisely so because it was pretty bad. They had an unclear relationship with their Salesforce consultant. And when I say unclear, they weren't being clear with us about what that relationship was. And it was very strange. And we had detected that there were some problems there, but they insisted that, yes, this consultant was going to continue on. They knew how Salesforce had been customized. They understood. Maybe if they didn't have the documentation, they had the institutional knowledge. And then they had this terrible lack of documentation. And finally, rescuing this project was actually deemed impossible by stakeholders on the board. And what we learned later and didn't know then the executive director's job was on the line for this project. He did not disclose that to us until much later. So there were many, many warning signs. There were a few good signs though. And it was enough for us to say, okay, this is very dangerous, but let's see if we can make this work. One was that the client did understand that there was going to be substantial ramp up and that they were going to have to invest in us, really doing a lot of digging and a lot of questioning and a lot of discovery. And they were willing to fund that. They also didn't insist that a rebuild was out of the question. And this was something that if they, we all decided as a team, if they said that was out of the question, we weren't going to take the project because we didn't want to be stuck in a contract where we were forced to work with what was there. They were also willing to rethink what we knew were very antiquated processes. They had been working again with this custom CRM written in ASP for over, I think at this point, 14 years. And they did disclose to us, as I mentioned, there were a number of iterations. So we knew that they had probably, their process probably followed how the system worked instead of the system working how their process should. And in fact, we found that it did. And they understood that and were ready, or seemingly ready to move that forward. So that's sort of the scenario. The things that at this point were really critical also for us to understand are what we kind of call the pinch points because you're having these conversations with folks, but there are constraints that you have to keep in mind. And this is some of the lessons we've learned that I hope you can take with you. The first is the pressures on the budget. So this is an obvious one. They've already spent and wasted a lot of money.  And when you actually might need more. So you actually may need more resources than the average project. And I'll also talk a little bit about why.  Your project managers are gonna have to deal with that. And the project owner at least, if there is an internal project owner will as well. The other is pressures on the schedule, another obvious one. The original projects is stalled. So they may say it's 75% done and they've told their board, well, it took two years to get to 75%. So it'll probably only take five or six months to finish out the project. So they've likely already set up expectations to the people that they're accountable to. And then they're in turn gonna hold you accountable. And you have to be very, very clear when you're negotiating about the schedule. And another is wounded trust. And I can't underscore this enough. How many folks have dealt, how many folks here are vendors, by the way, or are developers? Okay. And how many of you have dealt with a situation like this where you've, yeah. The client is wounded. It is a relationship, that metaphor. It extends to the sort of vendor-client relationship. It is a relationship. And once someone's been burned, they're really, really skeptical and they're suspicious.   People are gonna be afraid, not only that you're not going to deliver, but that you're trying to take advantage of them. And you have to be aware of that going into the project. And then the other is client fatigue. This isn't as obvious, but it's a really important dynamic as well. So the other thing that's going to affect this project is, you know, the client has been following this path for a long time, possibly. Hopefully they figured it out early enough and changed the relationship soon into that project. But sometimes it takes a while and they're going to be disappointed. There's been very little progress. And in some ways they're just, they're exhausted because it's not their, you know, especially with nonprofits, it's not their full-time job. They're not, you know, necessarily responsible for this, you know, and they're taking time away from other tasks that they do. And they're gonna have to start over. So they're gonna have to start over with a new developer. They're gonna have to start the clock over. There are all of these things that they are facing that they know. And, you know, they are really going to push you to work with an existing solution. So those are some of the things that you just have to keep in mind. So before you sign that contract, a few things that you should consider, especially with your staff. We took this project to our staff. And we usually do, we usually talk about projects because we have a really collaborative work environment and a very collaborative team. And this one, though, was definitely a big decision point. So some of the things that we talked about, technical debt and supportability. So the question is, you know, can you and will you support the solution after it's launched? If they don't expect you to support it, you know, maybe it's that reduces some risk for you. But if you are going to support it, and for us, we support 80% of the sites that we build. So we have a formal support arm. And, you know, it's why we try to build things right because we know we're going to inherit that at the other end and we're going to have our own technical debt to pay down. And, you know, while you might be able to fix what's broken and get something off the ground, maybe you don't have to rebuild. Maybe you can complete the project. You know, can you commit to supporting it in the long run? So you might be able to put the Band-Aid on it, but, you know, is it something you want to continue to support? The second is employee morale. You know, there's no, I guess, other real soul killer for developers or even design staff when you're working on something terrible that you didn't build yourself. It's bad enough if you're working on something terrible that you built, but if it's someone else's, it's, you know, it's pretty painful for people. The other is, can you take credit for this project? So for us, this was a project that we actually don't even list as a case study. It was the biggest project that we've ever undertaken and we don't even list it as a case study.  And also, again, if it's something you're building and you're using someone else's designs, you know, maybe you can finish out the development. Maybe it isn't that bad, but is it something you can put on your site? Is it worth it? The other is the state of the documentation. So in this case, there was none. Well, there was some. It was terrible. But what's the state of it? So it's one thing for the client to hand you a document, but is it current, does it reflect the current state of what was built? We all have been in that situation, I'm sure, even with our own projects where, you know, something's documented prior to build, maybe. And it gets built, there's implementation decisions, and, you know, that document doesn't get updated to reflect the build spec doesn't get updated. What, you know, where is the divergence? And can that client tell you where that divergence is, especially before you move ahead? Cause you, you know, probably aren't gonna have a lot of time or a chance even to really dig, you know, dig into what's been built at this point. The other part of this is about the real story of the project. So clients are not likely to disclose the truly bad bits. For example, the executive director's job on the line, he didn't wanna tell us that, but had, you know, we tried to have as frank conversations as we could up front and tried to get as much, you have to read between the lines sometimes, but we tried to get as much out of them as possible and engage them as much as we could to get the back story. They also might not even recognize their own role in creating the problem, because the problem might not have been the developer. The problem might have been the client. And this, this is another piece of, and I'll sort of describe to you how we mitigated some of this, but this is a piece that is, you know, we even see it sometimes with projects where we know we've done well by the client and, you know, they still don't wanna take responsibility for their sort of part in it. And then it becomes, you know, when it moves on it's like, oh, it was all the developer's fault. So we all kinda know that a little bit. And they're definitely gonna be blaming the vendor more than themselves. And then the other one is rescue projects are super messy. They are super detailed. There's so much to do. It's, you're not starting with your own patterns, right? You're starting with someone else's patterns and you need to have the project management talent to make it work. Your, you know, your Rockstar PM may not want to take this project, but they're the person who's probably best suited to deal with it. So those are the things that we considered. So the way that we decided to move ahead was to do a formal discovery engagement. How many of you folks actually do that routinely? Does anyone not aware of what a discovery engagement is or an audit? Okay, well, someone's raising their hand really, really high. You know, it's basically saying to the client, look, we can't give you a proposal because we don't know really what it is that we have to do. So we need you to enter into an agreement with us where we will investigate what you have. We're going to do an audit. We're gonna do a technical audit. We're going to look at the, you know, where is everything that's been built, right? Look at the documentation, scour everything, look at the code and really do an analysis. And it could, you know, for a regular project you do the discovery just to get to a budget and a scope and get the client to prioritize what it is that they want in that budget. But for a rescue project like this, it's really to evaluate how are we going to structure this relationship? Ken, is it really 75% done? You have your suspicions, but you need facts. So definitely look before you leap. The other part of this is don't do an unpaid audit. Insist that it's paid. I don't know, but don't do it, don't do it. And also ask for unfettered access. So if they're not going to, you know, to give you shallow access, if you're not going to get everything, then walk away. The other thing we also considered asking for was a question and answer, sort of Q and A back and forth with the previous vendor. That sometimes works if the relationship isn't too damaged. And, you know, you can sometimes get some interesting information from this. Their sales force consultant called me one day. This was after one of the first meetings we had. And he spilled his guts about this project. And in particular, he was very, very, he blamed their PM, I mean, it was pretty extreme, the language he was using. I was like, okay, this was a bad relationship. And it made us look at, we had some concerns about the PM on their side. I'll get to the rest of that story too. But, you know, it was enough to make us pause because we said, look, there's two sides to every story. And we got as much information out of him as we could, but we also realized there is no way we can move ahead with this project with this partner because their relationship was too damaged. And we only realized that by actually talking to the vendor. Make sure you do it and ask for it at least. Also, ask for the requirements that the client has documented, even if they were the original ones and or require the client let you develop requirements in user stories before you proceed. Either you're working from existing ones or they're going, if there aren't any that they will let you do that. That's, I mean, that's pretty straightforward. Also do a debrief with the senior leadership to determine what the problems were. Because one of the things, and I mentioned this down the line but it bears noting here, one of the things that has saved this relationship was that I had a direct role in this project, not as a project manager essentially, but I did all the relationship management for this. And as the principal that was a really, that was pretty time consuming for me, but I had a direct relationship with the executive director. And he and I would talk if there were hard questions or things that were coming up, we had a regularly scheduled meeting that was just the two of us where we could talk about what was happening with our staff and that helped actually keep the project afloat when it got very difficult. And the other part of this, and we're talking after the audit, you really do need to provide formal recommendations and that's really kind of the key to why this succeeded. I'll get to that in a minute. So what did we examine? They had project notes, artifacts, things were kind of a mess. They did have scopes of work and it wasn't until the later scopes of work actually that we noticed something, like that's where they started actually defining things later. There was no sort of foundational documents. They started using scopes of work to actually define what had been done and what needed to get done. There were no project really meeting notes, so we had no evidence of what happened during the process and most of the documents were really of little to no value. And as I mentioned before, the most valuable document was that third scope of work we looked at and that was the first time we saw a comprehensive set of requirements. So what we walked away from that understanding is that the client just kept getting scope after scope after scope from this vendor. There was never a process in place. There was never a plan. It was just sort of, I guess maybe the worst version of Agile ever because they just were sort of diving in, doing things, trying things, failing, getting change requests from the client and just continuing to build. So what did, when we looked under the hood after doing all of this, oh, sorry, we looked at sort of the Drupal code base, obviously in the database. Then we also looked at Salesforce and it was what did we find? And while this is an exaggeration, it's not much of one. This was the Salesforce, the entity relationship diagram in Salesforce. I don't know if you guys have ever looked at this tool in Salesforce, but it literally, I mean it was almost inscrutable. It was kind of crazy. So other Easter eggs that we found, there had been PM turnover. There was actually no continuity within the organization. There were two PMs within the organization who had worked on this project and there were multiple PMs on the developer side and four different development teams had worked on this. So I think at this point, so the current developer, there was 10,000 lines of custom code and the current developer had only worked on 50% of it. So there was absolutely no documentation. And the other thing we realized is that the vendor had used their existing custom database as the model. So all they did was reproduce the mess that was already existing in their custom database. They didn't stop to re-envision any of it or think about how that custom database would fit into Salesforce's conventions. There were in Drupal, 37 content types, 365 distinct Drupal fields, 730 SQL tables, obviously, performance was going to be a big issue. There were 135 fields on the user object. I mentioned the lines of custom code. There were 1301 lines of direct hacks to the Salesforce suite itself. So it wasn't that they were just hacking core. They were hacking the Salesforce suite module. And we realized what that 75% measure reflected. It was that 75% of the site map existed on the website. There were links to 75% of the planned site map. There at least was a site map. So definitely a big mess. So how do you turn something like this around? I think the biggest, the most important thing we did was spend a good deal of time preparing a very thorough report for the client. This was before we even signed a contract for implementation. So we tried to state things very, very plainly because we knew that the staff was very non-technical and we could give them, we gave them this, you know, the list and sort of the technical, we gave them the, we backed everything up with that assessment, a technical assessment, but we spent the time in writing to analyze what was there in very plain language. So we actually, it was a little forensic and because we kind of described in this report what we think went wrong with the project and we were able to actually get it pretty, pretty right. And what we did was present two different scenarios to the client because we wanted to respect the fact that their stakeholders felt this was impossible. They had sunk all that money in and they really would have wanted us to move forward. They would be lobbying for us to move forward with the existing site. So we gave them both scenarios. Here's a rebuild, here's a complete rebuild and what that might look like and here is what we'll have to do if we move ahead with this. And because we gave them scenarios that considered the entire life cycle of the project not just from when we would pick it up to launch, they realized it would cost more in the long run to keep what they had and that it would limit their ability to continue to evolve this site and what it did. And the good news is that they actually, they got it. The other thing that we did that helped us convince them was we spent a lot of time with their stakeholders or internal stakeholders during this discovery engagement. We sat with them for days listening to their needs and that in itself built a tremendous, not only did it give us the information we needed but it built a tremendous amount of trust and we had a lot of staff members doing that. It wasn't just one account manager or a PM and one other person. We had a lot, we threw a lot of staff at this project. And so it gave them the evidence that they needed as well to go to the board. And they could put this report in front of the board and say this is an independent analysis. It's very clear, it's about, even if the board members did not understand a word of the technical analysis, they could see that that work had been done and it was very thought out. So that's great. We've got a contract but now you have to deliver. And this is always the hard part. So as I mentioned earlier, the bar is really gonna be set high for your performance and in fact higher than probably on projects you just start off on a fresh start. So fatigue is set in, budgets are gonna be watched closely and then even the typical difficulties you run into, every project has its challenges. No project is perfect, there's always going to be challenges. It's how you deal with them. And the problem with this is even routine difficulties you might have are going to erode trust faster. It's going to have more of an impact because you're already operating from kind of negative trust here, right? So you can build it and then something happens and you can sort of drop you down rapidly. So you have to run the project with a tremendous amount of integrity. Goes without saying impeccable project management, right? So if you do not have the staff that you know can run a project, don't do this. Don't say well, maybe they'll figure it out, maybe they'll learn, maybe this will be a learning opportunity, it's way too risky. The other thing that was really important was stakeholder representation and stakeholder management. It turned, we are usually very good at doing this and making sure that all the stakeholders that need to be a part of a discovery are around the table. But it turned out that the project manager and this person will come back up, project manager on the client side and she'll come back up frequently about one of the key problems of the project is she told us that we had that full representation. She said she had gotten ambassadors from all of their major departments and we also trusted that she was going to include them in user acceptance testing and review. Unfortunately, she really didn't do that and we, I mean I'm trying to put this in a nice way but she kinda misled us a little bit and we found out that she didn't have the trust of her team and the people who needed, she actually was unable to act as a product owner on their side. She's really, folks didn't trust her. They didn't feel that she was working in their best interests and so some people who were critical to the success of that project actually weren't included or opted out because of her role and it was, yeah, that burned us a little bit towards the end as well. We also did a combination of Waterfall and Agile for this because we knew that they were kind of, they didn't understand Agile and we knew that they really wanted assurances so we took the time to do a lot of design and documentation first and walk them through everything and I know it's painful and we're all trying not to do that but it really was valuable in this case and a lot of times even with regular projects we find that some combination of Waterfall and Agile is really needed to manage the process well and in particular for us budgets well. In terms of transparency, we do this anyway. We're super transparent with our clients. They have, they know where we are on budget line items by an epic or by sort of a domain right or if it's a design budget, et cetera. They know exactly where we are at any given point but for this client and for these types of projects it's really important to be really forthcoming. You know, if you're a firm or a person that holds your cards close to the chest and you just say yes, yes, yes, this kind of project won't work because someone on the other side is going to, again, going to be suspicious and they want to see and we always feel like it's better to be really blunt and straightforward and talk about the problems than to pretend that they're not there. We also were forced to provide a formal monthly progress report to their board and that was a pain for me in particular because I was the one who had to do it and I was really super annoyed by it but I knew it was important and we had to give them an analysis of all of the hours that we used. It was really, really difficult and because it was monthly, the sort of continuity report to report was really important. They were scrutinizing everything. So because this was so difficult in many ways, we knew we had to focus on some quick wins. So while we needed to build as we went and we couldn't really be, you know, they weren't going to sort of get a site quickly and start prototyping, we made sure that we gave them something at every meeting to review and they saw progress always and we always made sure that there was a deliverable, no matter how small there was a work product. Also expectation management, that goes without saying. And again, being really, really blunt and honest and then continuous communication. So that was really what I mentioned earlier where I had that meeting with the executive director. And what happened, what was interesting about that, we knew we needed him to see our interests as aligned with him and because of that, we became more of a coach and a confidant and that really changed things. He was able to reach out to me and be frank in ways that I don't think he would have if we didn't have that open line of communication. So, you know, the kind of, the emotional intelligence here is really, really important. So how did it turn out? Well, sometimes mostly it felt like this. Yeah, it was a little hairy, but it was kind of a mixed bag. So I'll go through these quickly and we'll get to some questions. So the good, we launched a new Salesforce instance. We found a good partner and that's sort of an aside, but we actually had them fire their Salesforce consultant and we found a better consultant for them and that's when the project picked up traction. But we got a new Salesforce instance and website completed in roughly a year, even though it was deemed impossible. And again, that was despite firing the Salesforce vendor. We were able to make sure that there were some key stakeholders around the table, but in general, just talking to the staff, kind of re-energize them and it really raised their confidence that the project would ultimately benefit them. And that was really important because it's a huge lift to try to transform and completely transform a system at once. It literally was working one day, both their CRM and the website and the next day, it was a totally new system. And the transition was a little rough, but the things that we put in place helped to smooth out some of those edges. And then we also successfully negotiated a large enough budget and it handled all the contingencies we expected or we thought might happen and we launched with 7% of the budget left.  Now they also thought they were going to get a lot more for that budget than they did, but they got a very decent MVP, so it was helpful. For us, it was the largest project we'd ever tackled in terms of the complexity and the budget. So it was beneficial to us for those reasons. And then we did win a lot of trust and good faith and we're gonna have this client for many, many years. Whether we want them or not, we will probably have them for many years. One of the things we didn't consider, again, as I mentioned, that the PM for the client would leave out key stakeholders. And the other thing we didn't anticipate is the executive director kept promising his, making promises to his board to placate them, things like deadlines and without really talking to us or half listening or they were loosely, there were things we talked about that might be possible, but it wasn't down on paper and he would latch on to them and he was kind of part of the problem too there. And we were waiting a little too long to make sure that nothing was ever promised that we didn't have explicitly set out. So there were a few bumps in the road and we kind of met the challenge, but we didn't stop it early enough. And that kept happening and that really put stress on our development team. And it also meant that sometimes testing wasn't getting done as thoroughly as it needed to because we just had that pressure of the deadline that he had promised. Also, the other thing that was really troubling for us and the Salesforce consultant, we actually couldn't examine the structure of the source data from their custom software. We could only see what had been imported to Salesforce because it was kind of behind several firewalls and it was just, you couldn't get to it remotely. And no one, including the Salesforce consultant, knew how bad the state of their data was. So there was a complete redo of their Salesforce org and a complete re-migration of the data. And we had tried to mitigate that risk by having actually incremental migration. We were going to migrate key data points along the way and have the client validate that with the system. And because it was such an intricate integration, we kind of needed the real data for them to test anyway. It was gonna be really hard to use dummy data. But for a few reasons, that didn't happen as timely as it needed to. And it was just not adequately tested. So when the site launched, because so much of the functionality rested on good data and good valid status, valid statuses for particular for memberships and other things, the data was so bad, there was no way we could have anticipated that it would be missing. And the client and their PM did not take the steps they needed to scrub their own data ahead of time. So when it did launch, there was a challenge. Some users didn't have the right status, couldn't get access to certain things because those statuses in Salesforce were driving user experience in Drupal. And that was a big challenge for us in them. The ugly. So as I mentioned, the Salesforce vendor got fired and we didn't anticipate they would get fired six months into the project. So we kept trying to build, waiting for them and there was a lot of negotiation between the client and the vendor. And there were some questions about budgets and money that was left over and that actually stalled our progress. We had to start building to keep, to continue momentum, but we had to ultimately rebuild a lot of stuff because once we actually got a good Salesforce consultant on board and they took responsibility for redoing the org. I mean, we actually came up with a great structure together, but we had to do some rebuild. So that was a really bad part of the project. And this is one of the reasons that there really wasn't a lot left in the budget beyond the MVP. We couldn't give them everything they wanted. The client also fired the PM. Probably should have happened before the project even started. It might have been a really good thing, but they waited until after the project launched about three weeks exactly, after the project launched. And so that created other problems after the fact that we're still kind of dealing with. And then based on some of the problems with the partner, the schedule did have to slip. And that was a challenge for them. So just real quick, in terms of kind of our final thoughts, don't underestimate the amount of time you're gonna need for project management and time and budget for project management. At the end of this project, the number of hours that were billed on the project management line item equaled development and theming combined. So it was tremendous. And if we hadn't put that amount of time in, the project would not have succeeded. Also, performing well on relationship management was more important than the actual quality of the development to get the project to the finish line. Unless there is the potential for a future relationship, it might just might not be worth it. Might not be worth doing these types of projects. Also, each firm's gonna have their own set of metrics and considerations. For us, we actually almost didn't take this project because of the kind of nonprofit this was. It was a nonprofit that wasn't really in, I mean, looking back on it, not quite mission aligned for us. And for that reason alone, maybe we should have said no. We saw that really big budget and we were like, well, let's try. And it fueled our growth. I mean, we were actually able to add, even though it stressed us a little bit, we were able to add staff that we might not have been able to do otherwise. So we're still a little on the fence about, ultimately, whether it was a good thing. But it did have, there was cost there. And one of those is what I mentioned earlier. There were some projects that we may have been able to take on that we didn't because we knew we wouldn't have the capacity because we had to get this out the door. And then, I guess, I think that's about it. So thank you guys very much for listening to me. I'd love for us to just open it up for conversation or people comparing their own experiences. I'm on a rescue project. Thanks. Go ahead. He did not. In fact, I got a call from him today. So no, he did not. So far, anyway. Well, go ahead. Someone in the back and then you. No. I would not have taken this project. I think it, yes, we did grow. And yes, it did help us. But there were two other projects that I think would have been better for us to have taken. And we probably would have won. And there was one we didn't win because I couldn't spend time on, you know, cultivating that client. And it was one we would have wanted. So I think, yeah, would have made a different decision. Yes. This was so because we work for nonprofits, we're actually kind of liberal a lot with not charging for things like that. But for this client, we did. So that was all project management time. Yeah. And this gentleman in the blue shirt. Oh, oh, sorry. So there were many, there were many times in this project where we not only just had to tell a client, no, we had to explain to them that they weren't going to get what they wanted. Like that they just had expectations that they were going to launch with things that they just assumed either they had it before, but it was kind of a ridiculous scenario. Or it was going to take too much to build it. And what we had to do was always offer an alternative. And sometimes more than one alternative. When we'd have to say you can't have X, but you can't have Y and Z. Or there's not enough budget for that and this is why. And one of the, I think this is one of the reasons the PM got fired, but we spent over 100 hours just talking about how we might integrate, how we might actually deliver real online payments for them and how it would work across Drupal, Salesforce, and then they have something called Solomon, which is Microsoft Dynamics. But how that triangle would work. And the PM kept asking us for calls and we would explain everything and we documented things and we did, I did really kind of cheeky diagrams with clouds and drops just so she kind of would get the idea about how things were going to work and as plain as we could. And she wasted 100 hours just on that. And we were able to clearly display that. So it was being able to say, look, you may want this, you can't have it, you can have this, and this is why you don't have more time left. And being able to document that and point to documentation is really important. We had, so we don't have high turnover and we had one member of our team leave. And it was not related specifically to this project, but this was, yeah, this was part of it. And I think, yeah, I think you, yeah. And we, and our visual designer also left, but for different reasons. But I think that, yeah, I think there were, it was a tough year for us. We both contracted and grew. It was, yeah, we, yeah. There were a lot of happy hours and other things that we had to do to get through this. And we did have a little party after it launched. Yeah. Over there, sorry. You mean in terms of our internal processing or our processing vis-a-vis the client? Yeah, we had, and that is one thing I can underscore enough. I mean, we are, I have probably, I was probably the most frank ever in my career with this executive director and their team. You know, we were extremely straight up and we knew we had to be because we had to be super clear. And even when we were being super clear, he still was, you know, hearing what he wanted to hear, right? So we not only had an open dialogue internally about this and let people vent about it because they had to. When that PM was fired and we had to regroup with the client, we had an open conversation. And we said, look, these are the problems. The other benefit that we had was the Salesforce partner. You know, we're, we've worked with them many times and we have, we are very well aligned. So we were both able to articulate those concerns in a way that they heard and they got it. We didn't feel like we could walk away from this. Actually, either. But yeah, because these guys have the kind of lawyers that I think would have scared us. But wow, for, that's really tough. What's your sort of position in that, what would be your role in that scenario? And so it's tough. So the question was if you're part of an internal team and it's not a vendor-client relationship and you want to turn things around and you're kind of stuck in that role, how do you do it? And you're also only two or three weeks on the job. It's actually an advantage, honestly, that you're new because you're bringing a fresh perspective and I think most employers and most teams expect that from new folks or hopefully it's welcome. I would really recommend finding a person higher up who you can make the case to, but make a case. Don't just sort of say, hey, I've got some ideas. Actually take some time and formalize that proposal. You know, talk about how if you redesign this it's going to save, you know, it might cost a little bit of an investment now but look at what it's going to save for maintenance or we can easily add new features and it's gonna save hours. If you put, if you try to articulate the case in dollars and hours, you will likely get a better response and there are things that people probably haven't even thought of. Yeah. Go ahead. Answer all the questions before we get this call like, this is not a, hey, you can start in two weeks to get this thing done and say, we needed this thing done six months ago. We need you in today to get it fixed. We actually try to reinforce our value and the value add that we bring to a problem and we try to make the client understand why our process is important and then we also say, we're not gonna do this unless you let us do, you know, unless you sanction this and pay for this. We can't because we'd be irresponsible. We'd be irresponsible to you because we're gonna promise things that we can't keep and we're going to be irresponsible for our business because we're likely going to not structure the project well. There was also, yes, and then you. We didn't run into that just because we, that's how we operate and actually 50% of our staff come from nonprofits. So they understand why it's important to be transparent and sometimes it does not work to our advantage to be so transparent. I mean, you can imagine the scenarios where it doesn't but we'd rather err on that side and we're just, we're human and that's how we relate to our clients. Yes. Yes, that's you. Yes. Well, actually funny enough, I'm gonna take the second part of your question first. This launched four or five months ago and a month and a half after we launched this, another rescue project came across my desk. Now, the problem with that rescue project is that it was one, it was a project we had bid on two years prior and we were a runner up and we didn't get the contract. So there was a little bit of pride there and I was kind of a little bit of, and I told you so, but they, I actually brought it to the staff and it was a much more attractive project and we ended up bidding on it but what we did because we were so gun shy is we actually bid very high and we said, first of all, we said, we think this is gonna be a redesign and they kind of insisted that it wasn't and we said that we felt very, like we recommended that and we had a budget accordingly and so we did not get that, we didn't get that second bid and I talked to the developer who did recently and he actually convinced them to do a redesign but so I'm glad that they're doing it regardless of who's doing it but it made us very, very gun shy so we weren't as like, we weren't willing to be as competitive and as risky even though it would have been a great client for us and I forget the first part of your question so I can't, oh what was worse? Sometimes I think the technical challenges are more surmountable than the human ones because we can always find an approach or a solution, it's how you communicate that and being able to have the skill and kind of their delicate situations and so I think the technical ones are actually easier to solve, yeah. Yes.  No. You mean there's a... We don't have our budget now. So there's a resource that the client sort of wants you to work with and sort of insert into your process. How do you manage Jim? Oh God, yeah, that's a tough one. We've never had to deal with that. We have dealt with, when we work with universities usually there's an IT team on the university side whether it's like the, even if, say we're working with a small department that's part of a larger school, the school will likely have an IT department and a whole bunch of regulations that are annoying that we have to follow and so in that scenario, yes, we'll have to kind of interface with them and sort of deal with delivering something that they're ultimately gonna manage but we've never had to deal with someone kind of working with us alongside our team, only PMs. So yeah, I'm sorry, I don't have a good answer for that one. Yes, yes, so they're a partner. Yeah, they're a partner on the project but it's not like they would give us a web developer and say, here this guy's gonna theme what you guys build, right? So we've never had that. The Salesforce consultant, that was the other reason the project succeeded is we already knew a very, very strong partner there and I've partnered with them on many, many projects because it's too much for our firm to take all of that on at once and that is critical. I mean, there was a Salesforce consultant we worked with for years and again, we've been doing this since like 2007 and he kind of couldn't handle a project of this caliber so it was good that we had someone else to also bring on but being able, that relationship is very strong. If that relationship wasn't strong I don't think this would have worked. Go ahead. I think it's making sure that they can wrangle their staff to do what needs to be done and closing that loop and being as meticulous about the follow up in the open items and the things that we need because there are lots of PMs who can be pleasant and helpful but if they can't wrangle their team and there's critical information that we need that's gonna hold a process up that's when we start running into problems. There's a critical piece of information we can't continue working on a feature because or even the design of something because if we don't have that answer it could take us down the wrong path and so I think that's a really important piece and then also treating the vendor first of all like an expert. Sometimes clients don't do that and they don't listen to what we're saying and we're saying these things for a reason and yeah, we're gonna back it up if we need to back it up but you've gotta trust us and we're saying this to you because we want the project to work out well and we know that this particular request or this direction is gonna take us down the wrong path. That's definitely another one and also the projects that do work well for us is when the PM sees us as an ally and will tell us things that are frank and will say you know what there's this problem here and that problem there and I need to tell you guys about it because if you're in a meeting with this person and you bring this up or you don't present it in this way it's going to create tension and we don't care about the gossip we wanna know is there information that's going to help us run this better and that's gonna help you so rather than feeling like you have to be a gatekeeper it's like opening that gate a little bit wider and letting us kind of understand the dynamics. You'll actually, this lady first please. No. And I'm terrified that this is being recorded. I realize this is, yes. I know, I know. That's why I've tried to be as agnostic as I could be and then I'm sorry, your question. Yeah, go to that guy, yeah. That's exactly right and we were, you know, when we were smaller we also, you know, we're guilty of that because you don't wanna say no when you wanna please the client but we learned that lesson for sure. Yeah, yeah, you can't. That's a great point, it's a really great point. Anything else? Guys, thank you so very much for sticking around. It was a great conversation. I thought there wasn't gonna be anyone here so thanks and I hope you had a great conference.


You were involved in the early 80s in quite a bit of controversy with the band Judas Priest and the claim I think was that they had backwards messages in their rock music that led to some unfortunate outcomes Can you tell me a little bit about that? Yeah, was the late 80s early 90s actually I believe early 90s wasn't it? I thought it was 82 No, no, that was much much earlier. That was with a pastor from California Okay, so the trial with Judas Priest was actually I think in the early 1990s So what had happened? the general story is the two young men Two days before Christmas We're sitting in their basement of one of their houses drinking beer smoking marijuana and listening to Judas Priest and the all-afternoon and then at the end of the afternoon they picked up a shotgun Walked to a local church playground that had swings and tear daughters and that kind of thing proceeded to sit down on the swings or a bench and I can't remember the details now and Attempted to commit suicide. One of them succeeded. The other just blew his face off, but managed to survive In the hospital immediately after when the police asked him, why did you do this the response of the survivor was life sucks which is not a common response among Teenagers as they were But the parents then a few years later actually decided to sue but they didn't decide to sue the gun manufacturer They didn't decide to sue the the brewery that provided the beer or the place that sold it They didn't decide to go after the drug Dealers who had sold in the marijuana No, they decided to go after Judas Priest and CBS Records So that's the suit was lodged against about alleged Subliminal content of two kinds backward messages. So the messages that you hear the forward messages normal Language, but play backwards. It has a completely different meaning And some forward subliminals so low-level forward messages Okay, so the parents thought that there were messages in the music of Judas Priest That was the cause of their sons to commit suicide, right? It was the it was the straw that broke the camel's back So they may have been depressed for other reasons and drinking and talking marijuana's gonna be somewhat depressive as well And that this was had it not been for the backward messages in the music and the and the forward subliminals They their argument was they wouldn't do it now. That wasn't their idea. They came to this because of I suspect media exposure about other bands Aussie Osborne had been sued For the same kinds of things over in his case. It was all forward messages In a song called suicide solutions Which in fact was a Song against suicide with ironically enough it was sued for that and in his case They just threw it out of court because the the the free speech amendment of the American Constitution protect speech The judge you let in this current trial the Judas Priest trial let it go forward because he argued The the backward messages the ones that you would only hear in the forward direction, which would be innocuous rock commentary I Were messaged you couldn't protect yourself against because you weren't aware you were hearing them So therefore the 14th amendment of the American Constitution to apply that sounds like a reasonable Claim to me Assuming there are such messages as and assuming they affect you Right, so that was his his that's why the trial went forward Yeah, and because people were asking we'll give the president of Aussie. I was the president of Aussie Osborne Why are we going through this trial? Yeah, now my involvement then was after that So I received a phone call because of work. I'd done back in the early 80s. I received a phone call from the head lawyer for Judas Priest and CBS Records. It was actually a whole firm of lawyers In Reno, California, which is where the trial is going to take place It's about a year before the trial. I received this phone call Because of the work I'd done on backward messages and rock music with Don Reed a colleague of mine then at the University of Lethbridge that we done in response to this pastor from California coming to town and holding his big rallies about Backward messages and rock music that were leading young people Down the path of life such as sex and drug use And he claimed it was because of the backward messages in the rock music led to a big Rally at the end of his two days there where people showed up and broke records and things like this It was quite the cusp of it in in Lethbridge. I was a brand-new professor in 1982. I just arrived in Lethbridge And so I got a phone call back then From a local announcer of a rock station in Lethbridge calling me up after Pastor Gary Green won't had arrived Saying is there anything in psychology that we could use to speak about this alleged Backward messages and rock music or subliminal messages. They were calling them and I turned to my colleague Don Reed We both went not that we're aware of and that we did a little research We couldn't find anything on backward messages But you were initially skeptical of the claim that that backwards messages could influence or we were open to the idea that they could Moderately or I just minimally to be honest But what I did do is I then went to his presentations to see what he was doing It was leading so many people to be so upset about it because it was the front page of newspapers all the radio stations and so So I went to one of his actually one more more than one of his rallies to find out what he was doing And on the basis of what he was doing it seemed to be pretty clear That there really wasn't the effect that he thought it was going on But rather just some standard psychological phenomena that we were well aware of old old phenomenon psychology So was on that basis then that Don Reed and I said but To be fair we should run some research, right, but this is a claim nobody's investigated He's he was earnest wasn't like he was just con man or something. He was actually earnest on far as I know We still is actually believes this this is true and So we decide well fair fair enough But his claim and it's more subtle than most people think so when it really hit the press in the in the mid 80s People had the wrong idea of what in fact he was claiming So his concern is not that they're that Messages of being certain rock music because obviously the forward messages in many rock songs right our Promote drug yes, and the like so that was in his concern his concern was that as he rephrased it at one of his meetings I'm paraphrasing now is that well good Christian Teenagers with a forward message would hear it to understand and reject it Yeah But his concern was this as much for subtle and to be fair I thought that the guy deserves a shake because he's not just thinking very glibly about this stuff His concern was that because it was backwards and it's hard to register consciously If the message is still getting through the meaning of the backward version of the message was getting through Then they would be unaware of the source. They wouldn't be able to attribute it to the record Yeah But the thought popped into their head by some unconscious mechanism He never specified what that would be but if it did then they would have a message in the head with no attributable source Except themselves and so they couldn't protect themselves against it exactly so they would now be looking at thinking oh It's fun to smoke marijuana must be my idea Not that's because they would reject it He claimed if somebody just walked up and said hey, but it's fun to smoke marijuana All right, they can reject it because it's popped into their head Apparently without any source. It must be their idea or even a message from a high of power Okay, let's go there, but so that was his argument so that I Watched what he did and what he would do is he would walk in with sections of rock music That he must have spent hundreds of hours obtaining listening to all this rock music backwards and finding these passages and he's found quite a few So he's got quite a demonstration at the time you could actually write to him and he would send you full cassette tapes Of all the examples he dug up to that point and then more so what he would do is he would then come in he'd say okay This is this is for example Queen another one bites the dust and you play a passage of Queens another one bites the dust in the forward direction and sound perfectly fine to you Famous song and then he said oh here's the same section.   Yes, it sounds like that sort of but it's like visual illusions You're also aware at the same time. They didn't really like that. Yeah, right? So it's it's got those two levels going on at the same time, but it you could go Yeah, okay, I see what you mean, right? It does sound like it's fun to smoke marijuana And he's been to other examples like that, but that's his procedure and he just comes in for about an hour Says here's another passage. I want you to listen for them place it And by the time he's done after but an hour people are totally convinced all these backward message I heard them all when he asked me to hear them So every time he tells you what to listen for and then you hear the tape man here backwards So we thought okay, he's got a theory He's got a he's got an explanation of why he's concerned that if it did manage to get in without you being aware Of it's of the source that that in fact could be some concern So we thought and there's no research. So why do we actually see if you can influence people's behavior by the meaning of The backs or the the semantic content of the backward message when you're hearing the passage forward Yeah, so that the semantic content the words. Yeah Yeah, so they that there was something about their behavior is going to be influenced consistent with the meaning of the backward pass But they're not hearing the backward passage. That's the point. I can hear the forward direction now What we didn't do was use rock music Because we thought that just makes the task harder got all this other noise going on We in fact didn't study anything satanic All the headlines afterwards talked about us studying backward messages and rock music. We did nothing of the sort So what we did do is We took senses normal sentences that had various properties. We were interested in investigating the English senses and then I've heard them So the logic if you follow me for a moment So we played the message the the passage backwards because we know what it's so the forward meaning is not the backward meaning if you follow me Yeah, okay So in the forward direction, they're just hearing backward speech. Yep, which is yeah, yeah, right That's what it sounds like and then We asked for his questions. So I'll get back to you as pretty shortly So this is game we We started with some really recent questions So what can people do with backward speech and some of the questions we asked that came about because both Don and I don't read And I when we first started playing with these senses, but those are very speech like so when you turn Speech around English in particular what we were using you shouldn't have read it still sounds like speech But it sounds like a foreign language. You don't know Right now to us and because the Muppet show was on at the time To us it sounded like the Swedish chef on the Muppet shows Muppet show because neither was not Swedish That later came to By this When a global mail reporter completely misinterpreted what I was saying and said that vocation read claim that all rock music Would play backwards on Swedish. Yeah Anyway So we decided to explore that because it's speech like maybe there's a bunch of things that you already have the skills because you're most Of us are well skilled at language and speech that we could still do so the very first thing We asked is so we're playing them passages backwards And we just had some recorded by women and some recorded by men and we just asked them when they're hearing the backwards Is that a man or woman and They were virtually perfect in scoring that so they can easily tell the gender of the speaker It wasn't quite a hundred percent because one person meant to tick one box and tick the other box So otherwise we have a hundred percent So clearly you can tell gender that's not too surprising because women tend to have higher frequency voices than men and play backwards the frequencies remain So the next one we tried what we tried all sorts of things with a few well hit a few highlights We then wanted to see if they could For example determine what language the person was speaking. So we had a fluent bilingual in German French and English who We got all these passages and we had him translate them Because I'm barely unilingual and so what he would just translate the English sentence into it's equivalent in French It's equivalent in German and he he'd read them all in the normal fashion and then we turned them all backwards and So the meaning hasn't changed just the language you use is changing and we asked we told people what we done And we said now it's up to you to decide whether when you play a passage to you which of the three is it Yep Freddie given passage by chance alone you'd expect them to hit it correctly a third of the time and they were well They're up well over 60% so not as well as they could do with the gender judgment, but So remarkably quiet. We thought it was remarkable that they could do that Well in nailing what the language was even though they're hearing it backwards probably for the first time for many of these people They've never had anything backwards before and yet they could still tell the language We didn't explore the exact details as to why that's true but if you just think about some of the sounds in the language you can probably In your head figure out well that kind of guttural some backwards would sound listening so we're probably gonna Link to that paper, but can you can you summarize what the what there may be four or five things that people could and Couldn't pick up when they were backwards That's about what they could do. Yep Also, that's about as far as you could get and then we tried and those are Notice we're not even asking anything about the meaning despite we're just asking physical properties of the tape gotcha So people could detect mail from female. They could detect language in some way and is that it? Well, we tried some things we thought would work that were strictly physical like Question or declarative statement because we thought what people would realize that normally when you hear a question The end of the sentence goes up as in Valley speak And that therefore if you heard a sentence start by going down sweeping downward you should recognize It's probably a question. Yeah, couldn't do it. Can't do questions. It could even do that Then we tried again something that should be relatively simple starting to move toward meaning We took sentences and read them as they were or read them scrambled So one is meaningful the other's not the other's nonsense and simply asked We're not asking at all. What they're the meaning is just really tell us if one is meaningful or nonsense Yeah, and they couldn't do that. Okay, and so then we started to move into more direct Questions about the actual meaning. Can you get the meaning at all? And if we ask direct questions, so what would that mean if you heard it in the form of direction zip Right, even if we made it a two alternative first choice task. It either means this or this which of the two couldn't do it So we play the backward we play it so we knew what the forward meaning was we had ones that were matched So they had the same meaning, but we changed how we expressed it passive voice with versus active voice for example So they have exactly the same meaning, but their superficial structure changes They couldn't even tell that when we had to listen to the backwards So the bottom line is people aren't very good at picking up the the meaning or the words in Very simple messages when they're played backwards. No. Yeah, when they're when they're hearing them backward No, so you this is difficult to do. Don't tell me tell me what the bottom line is. They're hearing the the The forward sense backwards the meaning they're going for is the meaning the forward sense would have had in fact That's what that's what they're told if you heard this in the forward direction. What would it mean? Yeah, other kinds of magic desks But remember none of those are in fact what Greenwald was arguing He said it was a conscious process as conscious recollect the recollection of what was being heard or what was being played But unconscious and all our tasks are asking you to record. What does that mean? And He made no statements about that. He didn't say people were consciously in fact His concern would be had they been so he wouldn't be concerned if they could consciously experience it so We then went to indirect tests as they're called or tests where it's not necessary that the person be consciously aware of What they're hearing but they can still be expressed in their behavior And we did a series of tasks like that A varying sensitivity according to the literature for normal speech or normal tasks and They couldn't do any of them even indirect measures of the content of the messages didn't come through so for example simple things Such as trying to influence their spelling By pre-exposing them to particular meanings of Homophones so that we use homophone pairs that had a high frequency and low frequency interpretation That means one of them if I just said it to you that's the only spelling you think of so if I said read to you You don't think of marshes. Yeah, you think of books. Yeah, but so our EED is the low frequency spelling our EAD So high frequency spelling So we place we we actually copied it experiment that had been done with people suffering from a syndrome nose Corsacoff syndrome In which they've great difficulty learning anything new they're fine for the past but can't learn anything new and When they were exposed to Sentences like climbing a mountain is a remarkable feat With FE AT at the end. That's the low frequency interpretation They'd have that conversation with the whole bunch of homophones included and all those are non homophones as well They kind of disguised the task and then they'd be asked Then next day the experiment will come back the next day and say, okay I'm going to give you a memory test for words I talked to you about yesterday and of course that response was we talked to you yesterday So that's got a complete chance on the recognition test, but then she'd say well, I'm going to give you a spelling test not mentioning anything about the past and The spelling test would have some of the homophones have been biased towards the low frequency interpretation others that hadn't as controls and So what they should do is the ones that were biased towards a low frequency interpretation if the memory was there They should circle them they should select them and otherwise they should select a high frequency interpretation It's exactly what was found with people with Corsacoff Probably going to go through the whole the methodology of of your experiments, but what did you find? well, so we took that very basic idea and Repeated it with backward messages. Otherwise, it's identical. Okay, everything's the same except We just turned all the message all the all the sentences backwards first. That's it. And then give them a spelling test nothing Completely a chance they showed what they did is in fact for all homophones whether they were pre-exposed or not Backwards they chose the high frequency spelling as you would predict. Okay So even something as subtle as that which works with people who have no explicit memory for the task They still show this in their behavior people given backward messages in the exact same experiment exact same kind of paradigm Do not so people given backwards messages were where we're not influenced completely uninfluenced So if you to put that I don't want to put it in my words What what was the what was the bottom line of that experiment so people given we found no? A fact either conscious or unconscious direct or indirect on their behavior From backward messages in as far as we could tell none of the meaning of backward messages was getting through none Even in a very very subtle experiments. So given so you've created an experiment with Extreme amount of scientific rigor you've found very little evidence for the fact that backwards messages not okay You found no evidence for backwards messaging influencing people either consciously or unconsciously so presumably people Then just read your study and reacted positively to and the court case was thrown out. Is that what no we hadn't got to the court case yet so what happened is Don Reed and I presented because it had happened in our community We asked to do a public presentation at the public library for the community to find to explain what we'd found It was because a concern was still going on about this terrible thing about these insidious messages and rock music some state to banned Backward messages and rock music the government was considering Requiring stickers to be placed on all rock albums warning about backwards of liminal content You can imagine what happened at that point if you were a rock band wouldn't you want the sticker? What fortune your old boy could resist? Satanic messages. Oh boy So what happened after that point of course is that many rock bands started putting? Backward messages in there in their music and they're easy to tell unlike the ones where it's kind of a You know you barely get the message to work it out when you hear backwards They these were very clean ones. They'd put it start telling stories backwards So you hear it on the record is as the backward speech, but if you turned it around and played it was very clear And so Pink Floyd did it the Beatles did it many many bands started doing it Precisely because you'd sell more records if you could claim you had backward messages in there So the first effect of us presenting at the public library is it went over the wires? They had wires in those days called the wires And we got interviewed by everybody we were on every talk show you could imagine Right around North America and including Australia and the UK and so on for about a year It was just all everywhere. It was just constant and We eventually wrote it up both the research and our experience with it because virtually every Report we read after we'd go for an interview or a reporter would call us up Or are you an honest? We were on a talk show the report of it would always get it all wrong So We eventually published paper 1985 that explained our research and and what had happened with the media that they really couldn't understand these Very simple messages. So one of the biggest mistakes they would make is to say that we had found that You could have people Hear anything you wanted to just by suggesting it now In fact, we had conducted one of the major pieces of our research was a set of experiments precisely to show that that wasn't true That you can't make people hear or see whatever you want them to hear or see The the thing that they're hearing you're seeing has to at least be consistent in some way So for example, I could point to a cloud out there and and oh look, it's a bunny rabbit And it was just a round cloud you go. I'm sorry, but a bunny rabbit sleeping. I'm not sure what you're getting at, right? But you would have no question at all if it had two big protrusions at some point, right? Same thing has to happen with the with the backward messages. It has to at least be consistent so we designed a series of experiments where we actually took passages and Flip them around and then listened listened and listened much like Greenwald have done with his original demonstrations and Constructed little bits that fit certain passages of these of these particular tapes and what so we had two different Passages we had Jabberwocky because it makes no sense forward or backward Jabberwocky The was brilliant in the slide. They told us the guy and gimbled in the wave And on and then we also took the 23rd song because Greenwald had claimed that no religious Passages had backward messages And so the dawn read and I got together in good scientific fashion at his place because he had the better stereo I Recorded one of the passages and flipped it backwards and then don't read or quarter the other one to flip the Backwards so we can control for voice intonation and then we just listened to them very creatively now to help that process We used case of beer had to send out for more actually Eventually we managed to construct six Messages if you want for one about the six messages for say the 23rd song and another completely independent six for Jabberwocky And here's the basic experiment was then was this if we completely always played the whole passage to this was a class of introductory students in another class of Cognition students in the second-year cognition course and So we play always play the entire passage of either Jabberwocky or the 23rd song to them And then we'd say that we'd ask them Okay, I want you to listen for and we'd either choose something that we had we had constructed to be consistent with that passage Or we'd borrow a phrase from the other passage that wasn't consistent Okay, and we'd say listen forward and we give them the passage then they'd hear the whole 23rd song or Jabberwocky And we record how often they actually said they thought they heard it and basically the results were almost perfectly not exactly perfect But almost perfectly that if it was one we chosen what sorry one we constructed for that particular passage Then they would say they heard it if it was one that we borrowed from the alternative passage They would say they didn't hear it Indicating that it has to at least be consistent to the extent that Don Reed and I were able to construct semi meaningful phrases as I said the the Scientific beer meant we were sometimes more creative than probably was good. So some of the passages When you we just hear what we chose don't make a lot of sense, but they're consistent So one of the ones that we use in class all the time as a demonstration You saw a girl with a weasel in her mouth That's what Don Reed and I came up with and it does actually fit, but it doesn't make a lot of sense And and you won't hear that if we play the 23rd song backward Okay, you won't hear that particular passage, but you will hear it if we play the Jabberwockie backwards So that show that it's no we can't get you to hear just anything it has to at least be consistent with the passage Contrary to what most media reports would would Conclude that we had found So that the background published the paper in 85 Come the early 90s and we I get the phone call from the lawyers for Judas Priest and arena Nevada where the trial is going to Take place. This is a civil suit and the plaintiffs have filed for millions of dollars, by the way in damages And so she calls me up and says You're the only authorities on this nobody else's republished a paper on backer messages And they were claiming there were a number of backwards of limos in This particular album the stained-class album by Judas Priest and I said, oh, okay, it's that's fine I'm more than happy to serve as an expert for you I'll provide you all the research I can get for you and other kinds of things and she says well, we don't get thank you very much It's very nice, but we know we want you to come and testify for us and I said well There's a problem and she goes what what problem I said you I said you probably want my co-author To do this job. He's actually had experience in court as an expert witness not in this domain, but in eyewitness testimony And she said no, no, you're the first author. We have to have you and I said well I'll tell you what I want to agree to do all this and come to the trial if you bring my colleague with me She's fine. No big deal And so a year or almost a year She calls me in once twice a week. We talk about things are things going it's only says okay So, you know, you're ready to get thinking about your testimony. It's kind of stuff Well, as I said, I think you really want my colleague So here's what happens the day of we're flying down there for the first weekend meeting of actual face-to-face meeting Now the reason I was reticent and I'd actually told her is that I actually look like one of the band members Shaggy long gone here and The judge is known for having somewhat less than positive attitude toward People who look like me and she's all not problem. Don't worry about it so the day we were fly down to meet the team and Judas Priest Don Reed was leaving from think it was Vancouver and he got the earlier to Reno than I did from last bridge So I'm the last to arrive and I it's about five in the afternoon I go to the cabin big law firm and I go in there and there's a big meeting room There's big doors and open very very dramatic entry and I come walking in and there's this woman I've been talking to her for a year She's the head lawyers all the other lawyers that Judas Priest around the table There's dawn sitting there like walking in and she looks up and meet you because oh my god And then says now dawn when you're in the stand So I actually didn't testify to Judas Priest Wow Well, you wonder this is good. You had a good backup plan. Yeah, so but I did get to meet various characters in this long story of Subliminal messaging So Wilson Brian Key, for example was there to testify for the plaintiffs So I got to meet him and we spent some time talking. We had a lot of fun actually speaking not intentionally It's great funny. I used to be professor at Western Ontario University And then left in the 1970s because of all the books he'd been publishing on What happened in the trial? What was the outcome? Well, the first outcome was there was a pretrial Hearing we are dawn read presented our evidence about backward messages all we were going to comment on and The judge said well, okay, the conclusions. He's pretty clean through that all out So no longer did but backward message is playing a role. However, they had been one forward subliminal. They had alleged was on one of the songs Better by you better by you better than me on the stain class album That the engineers came in because they had the original night or 14 track Original multi-track tapes for this song and was able to isolate what this alleged forward subliminal was was if I'm remembering this correctly I may not be that it was a Passing just a passing light touch of a bass string that wasn't meant to be played at that moment And then fraction of a second later a slight click on the drum kit kind of backer noise happens all the time in the recording While the rest of the band is you know, making a lot of noise And so it gives a Which they interpreted is do it Not doing much for Nike stocks, I don't think but so the idea was that that The the plate is alleged that the boys were borderline I've drinking beer and talking about all that kind of stuff And they heard this message and they're now making the argument that Greenwald had made or similar to Greenwald's argument About background messages and that is is that because it's the top you can't really hear it. It's subliminal like very low volume on the Record that when it when they appreciated what they were hearing do it They thought it was their own thought and that's what led them to pick up the the rifle go across the street killed themselves So that that's what the case came down to So at that point I'm basically out of it. There's no longer about subliminal messages. It's about that one forward subliminal and After the testimony of the engineers about the where they found these noises on the tape The judge didn't dismiss the case because he thought the whole thing was nonsense He'd already remember ruled that it would be it would not be protected by the the free speech amendment He dismissed it because as he argued there was no evidence that Judas Priest and CBS Records had intended the message to be there Which is bit dangerous if you think about it so that left open a sort of precedent for this to not to be resolved Right, so that if somebody for example Put a low-level subliminal forward subliminal in and intended it to be there So I don't know how what the state of the the laws now with respect to that ruling But it was I thought it was an unusual somewhat scary ruling about that However, so Judas Priest and CBS Records in a sense won except the judge awarded all the costs against them Because of course the families had no money But they were suing so they won and lost at the same time But at least they they won in the technical sense that that Like Ozzy Osbourne. They were at least not convicted of anything or it didn't have to pay the lawsuit So that's the story Judas Priest That's amazing It seems that people are hearing things that aren't actually there. How is that possible? said Kind of a two-level answer to that one. The first one is that's all that ever happens Do you the world doesn't really look the way you think it looks? As you know Even when you call solid objects or just made up of molecules with big spaces between them So it doesn't really look like that. There are no colors in the world Okay, right color is a is a something you bring to the to the processing of of the information you receive So in some sense what you just said is always true. Okay, you're never really seeing what's out there So that's one level of explanation. So in fact Hearing things that most other people would argue or not there or seeing things that other people argue or not there is Not saying much. Okay, because that's always true What I think is meant is to try to dissociate I think what you're asking about from a straight hallucination When there actually is no input source that should lead to that conclusion about something being out there which is usually result of brain disease or or Probably do some chemicals as well where the brains Processing gets quite distorted and it's actually doing more than just trying to put together a reasonable construction It actually creates a whole cloth as as for example when people who have had their eyes are nucleated So they're completely blind, but claim they can see Okay, and you're saying oh, I'll drive, you know Interesting phenomenon one particular case the the individual Who they knew could not see because they had no eyes, but claimed he could They'd ask him so how you know how many fingers over holding up and they wouldn't even hold up in a figure Three What colors my tie yellow no tie on right? So there's there's that part of it So there there are people who suffer from various diseases that in fact lead them to really see things that aren't there in the sense that Another person standing right there with them. It's just there's nothing there. I think right those are hallucinations though So we're talking about in these particular cases where we could lead people to think that they heard I saw a girl with a weasel in her mouth something bit different We've given enough information Much like we do in the real world These aren't threshold phenomena that either is or isn't it's that Your perceptional systems are accumulating evidence And then you can also make use of all sorts of biases that you've developed over your lifetime To at some point say yeah, I'm confident enough to go with claiming I hear this or see this Mm-hmm at later turns out. Oh, it was just the way the blanket was folded Thought it was thought it was my dog in the bed, but it just turns out it was the plane was full But because I expected my dog there That's what I saw. So that would be not a lot not an hallucination An illusion of a sense of a type I guess it's the way to think about it. So that's all we're really doing and that's just Standard normal processing. So we're not there's nothing unusual happening here. It's just it's just what we're doing all the time Is that what they call paradigm? some yeah Yes Yes, okay, it's you got to be careful to separate it from hallucinations. It's the is the big issue Okay, I think I think the distinction is important Oliver Saxel ate his book he goes out of his way to convince people that there really is important to separate the two the two hallucination and Paradox yes, yes to that they're not don't equate one phenomenon with the other and then don't even think the mechanisms for one Are necessarily the same as the other as is often done where people try to blend the two the two issues So yes, that's that's that's what I'd argue, but I would argue that all perception Is paratolia that's what we're all we're doing. We're not really seeing the world as it is. We're we're just trying to create something that's Reasonably predictive of allowing us to act in the world It's just sometimes we're right in the sense that it matches up with what others would say with what others would say in the World and I agree with you versus Not or did not intend to do right so for example if you if you take a normal scene in a room like this and Pick up some color So let's say the color along that wall there and you might put the beige Although I'm sure my wife would have some subtle distinction. She detached that But it's all full of shadows So in fact even though you see that all is the same color if we actually went up and looked at it very very closely We'd seen that well, that's not beige at all. There's purple and blue over there And there's think of somebody who an artist when you paint the paint snow a good artist the one that's trying to represent the natural world as They see it There'll be no white where they painted snow It'll be blues and purples and yellows and so on. Yeah, you see it as white snow but more than that It is the impression a lot of people have is that all we have color receptors in the back of their eye And when reds in the world the red receptor indicates that when blues in the world Actually, I don't have blue receptors, but when greens in the world The green receptor tells us but that's not in fact. We see color at all That's not the mechanism underline color version except for under extreme circumstances If I put you in a completely dark room and give you a pure wavelength, then that's true Okay, you'll see red when I show you read that should that makes the the red receptors Go off Generally speaking the way you see color is in a color constancy sense So that we could all agree I'll walk into this room and go yeah, that's beige and yet if we measured it precisely We might find that there's actually no browns coming off that surface at all This was work that I didn't land it in the early 60s showing in fact, that's quite true Even if all of us agree now is that pareidolia? Because in fact there is no brown there And yet every single person who comes in says oh, yeah, definitely And that's part of the reason I argued earlier that there is no color in the world Yeah, color is something we bring to our interpretation of things And but it's not to say that it's not And why I say that is because evolutionarily speaking look at things like berries They go out of their way to make themselves a bright red to contrast with the green. So you'll eat them So it's some sense in the world too But it's more of an act that you bring to the world and I think that's true for for most perception It's a mistake to put it in the world As much as we all agree on our approach don't remember our point thing is that you get through the world without following in the holes Bumping into things and getting eaten by lions But that's it What what you what you perceive what what you construct to perceive? Just has to meet those criteria. It doesn't really matter that it actually matches the world in any other sense It just has to match the world so you can move your way through it and Which might be a good argument when you think about it yourself There's a good chance that maybe the way you perceive the world is nothing like how your dog does Or on the other hand you can tell a lot of the tasks we have to solve are much the same. So maybe it would be the same. Okay So following from that This course is about the science of everyday thinking there are people out there who want to make better decisions Think better and do better based on your research What in what what advice would you give them? I'm based on my research Aside from the obvious, you know exercise good diet. Get a good sleep. Is that what you're interested in? No, I don't know what my research says about that. We do know that when you put yourself under conditions of sleep deprivation ingesting certain chemicals and even common ones alcohol and tobacco and the like And put yourself in high stress situations So your limbic system is is going off that we tend to increase the rate with which we're going to produce these Errors. Yeah, right We're now become more likely to see blankets as dogs and shadows as things that might threaten us Makes good sense evolutionarily that you're in those situations to become much more Signal rules much more dangerous than it is But so part of it to make good decisions don't make them under conditions of high stress of lack of sleep under a high chemical arousal and those kinds of things to make a better decision about things So maybe my research is something like that. Okay, so if you're going around the world When you're making your way through the world be conscious of the fact that I'm trying to rephrase trying to rephrase what you said in In terms of the person in terms of what people can do so you touched on it in the end So in order to make better decisions be conscious of this which will allow you to do that Okay, I see what you're getting at. I guess what I was talking about is that is that it's all an adjustable Criterion that you use to to decide that there's something there. I have to act right So clearly it you're much better to false alarm to tigers In the bush next to you then not to miss it, okay, or North America here that bears And therefore if you're in a high stress situation, that's late at night, and you're all lost and so on It's that makes good sense that you're gonna see a lot of the things as bears Even though they're not and it's not that you just come to the decision by stroking your chin going Okay, I think that's a bear. No, you see a bear makes no question about that Even though in some sense, there's no bear there, but it's giving off enough Cues that are consistent with the interpretation that that's a bear much like our backward messages If they were consistent with the phrases that that Don and I made up lead you to that conclusion But it's not like it's not yes or no Under the law, I'm quite sure it with our backward messages experiments. Have we put people in highly stressed situations? They would hear through the wall. Yeah, right or if we put them in situations in which the cost went the other way If you get one of these wrong Right, you you will not complete the course Then they might be a lot less so if you if you flipped the payoffs around the Threshold might be different. Yeah, and make it so that no no you really have to control this stuff Don't make these kinds of errors. Then yeah, I suspect they would go.  Anyway, I mean they would come up with and It's and it's the way things actually work It's it's when you've accumulated when you when the system has accumulated enough evidence to go and has good reasons to do that It goes this and it makes it makes some really good sense. So in that sense is what I was getting at if you want to make Good decisions in that kind of rational sense, but it actually makes sense To go with it if you are in one of those situations is late at night. It's dark. I'm your lost Doesn't make good sense not to just assume there are no bears I mean seriously, right or when you're walking down the street don't have $20 bills hanging on your pockets to attract martyrs I mean just Even though you might do that So but because those mechanisms actually are life preserving and they've had been selected for I presume for for tens of thousands of hundreds of thousands of years That's why you're here and others aren't because you chose your parents well So that part of what is so if you're thinking about say Danny conan's recent book on thinking fast and slow There are certain situations see outlines where the the thinking fast really gets you into trouble But it also many many situations where it gets you out of trouble where thinking slow would not so it's there's not a simple panacea here My name is John. I think about knowing