We're going to get started. I'd like to welcome you. I'm Susan Collins, the Joan and Sanford Wildean here at the Gerald R. Ford School of Public Policy. And it's a great pleasure to have you here on behalf of the Ford School and our newly launched center for public policy and diverse societies. I'd like to extend an especially warm welcome to Scott Page, who is our speaker for this afternoon. He is the Leonid Hurwitz Collegiate Professor of Complex Systems, Political Science, and Economics here at the University of Michigan. He's also an external faculty member at the Santa Fe Institute. His research integrates the wide variety of disciplines, as you've just heard. And he is the author of The Difference, How the Power of Diversity Creates Better Groups, Firm, Schools, and Societies, which was published in 2007 by Princeton University. Scott has produced a very influential policy relevant set of research on the effects of diversity and variation in a number of complex systems, including economies, ecosystems, and political institutions. He teaches and consults around the world for academic, nonprofit, and corporate audiences. And we're really delighted to have him here with us to deliver a public lecture on his home campus. Today's lecture represents the Ford School's contribution to the University of Michigan's 2010 symposium in honor of the Reverend Martin Luther King, Jr. As you may know, the theme for this symposium this year is, quote, I am, was, and always will be, a catalyst for change. Our event today seems particularly well-suited to that theme, and we're here to explore the change potential of diversity, whether and how increasing the diversity of perspectives around the policymaking table can be a powerful catalyst for change. Today's event is just a second public event hosted by the Diversity Center, and I'd like to take just a moment to talk about the center. We opened our doors this fall as a first of its kind initiative designed to shed light on how public policy can most effectively navigate the opportunities and the challenges that arise as societies become increasingly diverse locally, nationally, and internationally. A number of academic institutions have explored issues related to diversity through lenses such as social science, education, business, and law. But the opening of our center here, the Center for Public Policy and Diverse Societies, enables the Ford School to be the first home to a university-based effort that's focused on the policy problems and issues associated with diversity, and we're very proud of that distinction. During this first year, we will continue to host distinguished speakers, and so we encourage you to visit our web page, and we will be circulating information about other events as they are planned. And with that, it is my great pleasure to welcome our speaker, Professor Scott Page. That's the work. It's great to be here. I'm glad people came out on such a cold day. What I want to do is I want to talk a little bit about some recent work I've been doing on diversity and complexity. So as Susan mentioned, three years ago I wrote a book called The Difference, which is about diversity. And at the same time, my co-author finished a book that he had been writing for a decade called Complexity. So now I'm really branching out and finishing a new project on diversity and complexity. So I'm way out there. So what I'm going to do today is talk about some of these ideas and how they relate to public policy. So the title of this talk borrows a quote from Emerson, which is that our knowledge is the amassed thought and experience of innumerable minds. And what I want to convince you of, hopefully, at the end of the day, is how is it that diverse groups of people are so smart and why this is relevant to policy? But to start out, what I want to do is sort of give a policy 101 view of things and sort of give a backdrop of how we typically think about diversity in the realm of politics, right? And so this is going to be sort of political science 101 diverse preferences. Then from there, I'm going to get into the stuff that really sort of animates my research. I'm going to talk about simple things, then difficult things, and then eventually complex things. And I'll talk about the role that diversity plays in trying to cope with difficult problems. And then I'll ponder a little bit about how diversity and complexity interplay with one another and how is it that diversity, at least diversity and how we think about the world, impacts our ability to make sense of complex environments. And then I'll close with some thoughts about the new iPad, Touch, and Detroit. Okay, so politics is usual. Here's how we, I'm not joking, by the way.  Why we think about diversity in the political realm is this. We think about sort of diversity in terms of identities. We break people into groups like whites, African-Americans, Latinos, or young people of American. We look at how they differ in their political views. So maybe this just gives you charts of how democratic or liberal they are, what percentage voted for Obama. And what we see is we see differences across these groups. Other things we might do is we might look across these groups and ask, are there differences in levels of political involvement? How many people vote, that sort of thing. And again, here, if you look at white, black, Asian, and Hispanic, look at voter turnout across those groups, you see huge differences. So these are the sort of things that traditional political science would study. So one of the things we do in traditional political science is we decide that a way to think about diversity is to think of ideological diversity. So we sort of put people in this one-dimensional array between sort of left wing and right wing. So here's our current Supreme Court justice. And this is easier to do than all of Congress because I had to paste 535 pictures. So I'm gonna do the Supreme Court. So here's our Supreme Court justice is arranged in terms of from most conservative to most liberal. According to recent work by Andrew Martin at Washington University. So what you can do is then you can basically figure out if there's nine people, the person who's gonna decide everything is the one in the middle, right? Who in this case happens to be Kennedy, right? And so what you can basically say is that the law of the land is set by Justice Kennedy. And you can do this by looking across their votes and seeing in fact most of the time he's what we'd call the pivotal voter. You can do this historically as well. So here's the Supreme Court from 1950 on. And each line is a different judge, right? And the black line here is sort of the median judge. And this is how conservative they are. You can see here if you look at the end, this black line is Justice Kennedy and that's filling it in. So what we think of when we think of diversity is we think that people have different ideological views. Some are left-wing, some are right-wing.  Now that's sort of the simple political science 101 view of things. If we get in the graduate courses it gets a little bit more complicated when we talk about some cycles of preferences and that sort of stuff. What I wanna do is I wanna take a completely different view of how to think about diversity in the political realm and I wanna think of it in the context of policy problems. So when I think of it in the context of when we've gotta think of something like coming up with a healthcare bill or figuring out whether or not we should bail out the banks or figuring out what we should do to help poor people, we're confronting a problem. And I wanna think about now people taking what Emerson called our innumerable minds or our diverse minds and how we apply those diverse minds to try and find solutions to those problems. Now when we think of a problem we can categorize how hard it is. It can be simple, it can be difficult, or it can be complex. But the point I'm really set to focus on here is that governments make policies and if you're making a policy one of the things you're really doing is you're trying to find a solution to a problem. So to start out I wanna give you a little bit of a history in terms of how we've thought scientifically about solving problems. And if you wanna think about that you start with this guy, Frederick Winslow Taylor. Now there's sort of two versions of Taylor. One is of this great scientist, the other is that he's a charlatan, right?  But Taylor got this idea of what is called scientific management and it is you could take a problem and you could quantify it in some way and this led to sort of an adage that if you could measure it you could manage it. And so one of the first problems that Taylor went after was to optimize the shovel, okay?  So what Taylor did is he created what we call, biologists would call a shovel landscape. So on this axis you've got the size of the shovel. So here the shovel head is incredibly small, it's otherwise known as a stick, okay? And at this end you've got a huge shovel. And on this axis what you've got is the efficiency of the shovel, okay? So the stick you can't shovel anything at all. And if I've got a shovel the size of Texas, I can't shovel anything at all. But as a shovel gets bigger and bigger and bigger I get more efficient but at some point it gets too heavy and I start hurting my back and I get less efficient. So if you plot this shovel landscape, what you get is a single peaked function and people who study difficult problems call this a Mount Fuji problem because it looks like Mount Fuji. It's incredibly easy to solve and you get efficient answers. So if you look at sort of what they call time study people and organizations or what they call scientific management or generally what you do is you look for problems like this and you quantify them, you solve them and then you have everybody do the optimal thing. Well problem is what happens when things become a little bit harder. These are what we call difficult problems and with difficult problems the landscape no longer looks like Mount Fuji, right? Now it's what we call a rugged landscape and by rugged landscape it means there's lots of little peaks, right? So you can't just sort of move along until you get to a peak and stop because if you do that you'd end up getting stuck at something that might not be optimal. So you've got to be more sophisticated. So what I'm going to talk about today is how diversity allows us to be more sophisticated. But things can be even worse in addition to being difficult things can be complex. So when you think about something being complex it means a landscape that dances, right? So you want to think of the landscape as not being fixed but it's moving over time. So if you look at something like the stock market you can see that this is constantly changing so it's a moving target. Now the idea of complex isn't just a metaphor it's actually sort of mathematically defined and there's a wonderful book by Stephen Wolfram called The New Kind of Science where basically it says like there's only four things the system can do. One thing it can do is it can be stable, right? It can just go to some sort of stable thing like an equilibrium. The second thing it can do which is the one on the upper right is it can just be periodic.  The third thing that a system can do is it can be just completely chaotic. So if you change one thing you can just get this sort of random spreading chaos. This is the butterfly flapping its wings, right? In Asian causing a tornado or a hurricane in the Americas. And then the fourth thing it can do which lies between two and three. So this is not only New Kind of Science but the New Kind of Math where four comes between two and three is that it can be complex. So it can be not quite periodic and not quite chaotic. So what is complexity then? When we talk about complexity what we mean is either one of two things or there's sort of two classes of definitions. One is we mean that it's what we call Bohr between order and randomness. So it's not ordered, it's not random but it sort of lies between those two things. Or the second way to think of it is that it's deep, that it's difficult to describe, it's difficult to evolve, it's difficult to engineer or it's difficult to predict. So something is complex if it's just sort of hard to deal with. So let me explain these two things in a little more detail. Before, if I have a sequence that just goes zero one, zero one, zero one, zero one, zero one that's completely ordered, there's nothing complex. If I have something that's just a random mess of zeros and ones, I just flip the coin here. That's not complex either, it's just random. But if I have something that kind of goes zero one, zero zero one one, zero zero one one, zero zero one one, zero one zero one, there's a pattern there but it's not a simple pattern. So this would be complex. Let's go back to the sort of the definition, the other definition which is deep, this one was harder to explain, right? Since it's harder to explain, it counts as complex. Now there's another way to see complex, let me give an example of complexity. Complexity is probably best seen, not heard, so I can get this thing up. I got a log in here. Carl, don't look. Uh-oh, I did that for Carl. I just thought there's a, everybody's gonna get their grades changed in about five minutes here, thanks. I didn't, so this is a street scene from India. So I want you to think of this as, just think of these two definitions. One is between order and randomness, right? And the other is that it's difficult to explain, describe, or predict, okay? So as you watch this, there's actually a dramatic conclusion to this, so we'll work our way through the whole thing. So first you see, you're gonna see these structures that start to emerge, right? So here's the structure sort of forming on the road and they're stuck and now they're gonna make in mass, they're gonna move across, right? So you can almost think of this as like one giant particle now just moving across this way and then blocking the path. And again, so what you wanna keep in mind is this isn't, clearly is not ordered, right? But it's also not random, right? There's structures and there's certain, in the short range, you can sort of predict the sort of things that are gonna happen. Like there you see this beautiful move by the scooters, right, coming across the bond. Now here's the example. Watch the white car in the upper right. Upper, see in the upper right? It's gonna go, it's going the wrong way down a one. I think it might be Carl, not sure. But watch, watch him, he's basically just like look, I know it's a one-way street, but it's complex enough that maybe nobody will notice if I just sort of drive down here wrong way that I'll sneak in with this other group, right? And then watch it, it doesn't just join the other group though, so it just takes a while but it's sort of fun that we've watched this con. That is like, there we go, that's a good shortcut. Okay, so that's sort of the case of a movie being worth a thousand words, different back. So that's what we think of as being complex. Now the other way to think of it is this deep notion, right? So when you think about deep to describe, there's a notion called minimal description length to come agoraf, which is just how many words does it take to describe it? So the more words it takes, the more complex it is. There's also a notion called thermodynamic depth, which is just said if you were gonna evolve this thing, how long would it take to evolve? There's something called logical depth, which says if you were gonna engineer it, if you're gonna build it, how long would it take? And then the last one, and this is one that'll be relevant for us, is due to Jim Crackfield and Cosme Chalizzi, he used to be a postdoc here at Michigan, which is how big of a machine would you need to reproduce the same pattern? So how hard is this thing to predict, okay? The reason I want these formal definitions is if we look at the challenges that the world faces from a public policy standpoint, almost all of them are not in some sort of metaphorical sense, but an actual mathematical sense complex. So here's a list of, I went to like five different websites to sort of list huge problems facing the world, and so this is sort of an intersection of those lists.  Peak oil is just a resource extraction problem, so even if it's just oil that's in the ground we gotta decide at what rate we're gonna take it out. So that's an engineering problem, right? But every one of these other things, climate change, epidemics, terrorism, economic fragility, all of these problems are complex in a classic sense, right? And so what we wanna do is we wanna understand how do we deal with these complex problems? All right, but to get there, before we can get to complexity, first you've gotta get to the question of sort of difficulty. So the first thing I wanna talk about is sort of how we use diversity to get around difficult problems. So a classic difficult problem, one that we talk about a lot within the academy, is putting people, so far only men, but putting men on the moon. Like this involves solving all sorts of different problems, and we used all sorts of different people to solve those problems. This is a nice sort of example, it's a good exemplar of sort of like how we can have diverse groups of people solve something, but it's sort of hard to quantify. So it's better to actually look at sort of newer things that have been over the net, where we can actually measure how diversity's been used to solve problems. So this is a guy named Timothy Gowers, who's a mathematician. He's a winner of the Fields Medal. Among mathematicians, you know, he's one of the 10 or 20 best mathematicians of the century, a brilliant person. Couple years ago, Gowers got a whole bunch of publicity, because he said, you know, say, look, I'm a pretty smart guy as mathematicians go, I've got a lot of fame, I've got the fields and all that sort of stuff, but I think we could actually do better mathematics if we did it in diverse teams. And so he said, I'm creating this project called the Polymath Project, where I'm gonna just pose, so let's just do one to start, I'm just gonna pose this really hard problem, and I'm gonna see if a group of us can solve it. So he said, here's this thing called the Hales-Jewitt theorem. Now the Hales-Jewitt theorem is interesting, because they actually had a proof of it, but nobody could understand the proof. What do I mean by that? The proof was so long, right, it required all these parts, that no one individual actually could have, over a lifetime, like read the whole proof and understood it. So the question is, can we come up with a shorter proof? The interesting thing about this theorem, and the reason why it's sort of a fun one is, it basically boils down to this, it boils down to sort of how many boxes do you have to remove to make tic-tac-toe impossible? Now if there's only three rows and three columns, right, that's pretty easy to figure out, but if you've got n rows and n columns and you've got h different players, some of it becomes a much harder problem, right? So that's the formal statement of the theorem, it goes something like, if you've got an n by n by n cube and you've got c colors, so c players, how many cells do you have to remove so that no one can possibly win? Okay, so in this case it's easy, right? You just remove three of them, and no one can win, but the difficult problem is really hard. So what happens if you post this thing, and a whole bunch of mathematicians sort of look at it, eventually 27 different mathematicians start playing with this in over 37 days. It only took 37 days for them to find a short workable proof of this theorem, right? And what's nice about this, you can go back and you can look at it and you can see how different people had sort of different representations of the problem, different people knew different tricks, and collectively there was a couple of something that even someone with brains as Gowers couldn't solve. So if you look at that particular example in detail, what you find is there are two things that drive this. There were differences in how people represented the problem which we're gonna call perspectives, and there's differences in the little tricks people use to try and solve the problem which we're formally gonna call heuristics. So let's look at the first of these first. Remember we had this sort of landscape idea. So suppose that I have this landscape of a problem, right? So here's why I encode things, and here's their value. There's gonna be what we call local peaks. So I'm gonna get stuck, let's say, possibly at A, B, or C, right? Well somebody else may encode this thing differently. So they might get stuck at D, B, A, E, or F, right? So you've got a different representation. So if the two of us work together, well then what's gonna happen is, well the only place is the two of us could get stuck are A and B. Now it doesn't mean that we're necessarily gonna get it right, right? We could both get stuck at B, but the important thing here is, right, if we think of me, I could have gotten stuck at C, right? And if we bring in somebody else who sees the problem differently, C isn't the place where they would get stuck. So there's this big advantage of having different representations because we have different peaks. So one of the things when you look at sort of this, the polymath project, and you read the thread, right, of people, sort of the ideas they had, what you see constantly is people saying, here's a different way to frame the problem. And by framing the problem a different way, they sort of created different sets of peaks. Now the second way we solve difficult problems is by using heuristics. So heuristics are just little rules of thumb we have for solving problems. So when you take an IQ test, they'll ask you questions like this, fill in the blank, one, two, three, five, blank, 13, and the answer here is eight, right? And you can do this, one plus two is three, two plus three is five, or you can subtract, 13 minus eight is five, eight minus five is three, three minus two is one, that sort of thing. When they give these IQ tests, they'll also ask you this one typically, right? And this is just to sort of show, can you sort of see how the rule works, right? And this one is just squares, right? Then they give you this one, which is a hard one. And I've got two good jokes about this one. One of my colleagues saw this and said, this is easy, these are the years that Boston won the pennant. Which is great. The other sort of nice joke on this one is I presented this at the World Bank and nobody got it, which is a bad sign. But then I was at a middle school in Detroit and two kids got it right away, which is very nice. So in the long run, my money's safe, it's my view. The answer to this one is the answer to the ultimate universal question, which if you read the Hitchhiker's Guide to the Galaxy is 42. And the answer to this one is 42. And I put this one up not because I want to say, here's a hard question, because I want to show, this is in some sense maybe the key to sort of economic growth, at least if you believe Leitzman. So how do we solve this one? Two minus one is one squared, six minus two is four, which is two squared, 42 minus six is 36, which is six squared. In 1806 minus 42 is 1764, which is 42 squared. Why does that matter? What was the first one, which was easy? Subtract. What was the second one, which was easy? Square. What was the third one, which is incredibly hard? Subtract and square, right? So what you get is this third problem, which is really hard, you just have to combine the first two heuristics. So people who study heuristic development talk about super-addictivity in heuristic space. So this is one of these fancy academic words, but if you think about it, what it means is if I've got one heuristic subtract and another heuristic square, I've also got for free a third heuristic subtract and square. Now, Weitzman, who's an economist at MIT, basically says, look, this is where growth comes from. So one of the theories of sort of economic growth, and he has what he calls a recombinant theory of growth, is suppose you have 20 tricks. Well, if I have 20 tricks, that means 190 pairs of tricks. It means 1140 triples of tricks and 4,845 quadruplets of tricks. So there's all these combinations of things, right? Once you've got a small handful. So I should turn the microphone over to my young colleague, John Holland at this point, right? Because I think actually in this very room, John had maybe not this exact same slide, but if you look at the combustion engine, this is just a combination of tricks. And it happens to be a combination of tricks where we had all of the parts long before we had the engine, right? And it just took someone figuring out how do we combine these different things, right? The piston, the little firing mechanism, right? In the jet to pump the gas in to create the combustion engine. If you look at the bike, right? We actually had the bike for 16 years before somebody thought, let's put a chain on the bike. We had gears and had the bike and it was just this 16 year gap. And it was probably the worst example of this or maybe the best example is we went, you know, by one estimate 2000 years before somebody figured out to combine the poached egg and the toaster, right? And this, by the way, sold over a million units in its first year at Target, right? There it is, right? This sort of combination of characteristics. Now, the examples are nice, but you can quantify this stuff. So this is a, there's a company called Matlab, MathWorks that makes Matlab. And every month they run a contest where what they do is they basically say, here's a programming contest. And what you have to do is you've got to sort of write code to solve a problem. So over time, this is the date on this axis. And over time, this fall off you're seeing is improvements in speed in the program. So faster is better. So if you look at all this code, each one of these little blue dots is someone sort of submitting some code, right? And that's the time it takes for the code to run.  And what you find is these little tiny improvements are people who know like one little trick, one little heuristic, right? How to write for loops or if loops or how to invert a matrix to have it do a calculation faster, something like that. It's all these little heuristics combining them until we do the sort of massive improvement. When you see the big jumps, those are people who had different perspectives. Those are people who had sort of a different way of representing the problem, right? And so it leads to a larger jump. Now, this actually gets this sort of idea you can construct a business model from, right? And the Obama administration's tried to do some of this stuff within the government. So there's a company actually called Seniv which basically does the following. If you have a problem in the pharmaceutical industry or in the chemical industry that you can't solve, you can just post it. You can just say, okay, here's our problem. We can't solve it and we'll give you $20,000, $50,000, $100,000, $3,000, whatever to solve this problem. Right now there's 150,000 people, actually it's the higher now who are signed up to be solvers on this thing. They've given over $3 million in awards and the success rate is hovering around 40%. Now, Kareem Lakhani who's at Harvard Business School has studied in depth which problems get solved and which ones don't get solved. And what's intriguing is the ones that get solved are the ones that are framed in sexual way that it's not clear what discipline you should use to solve it. So if they say, this is a chemical engineering problem, it typically doesn't get solved because that means it's already been a bunch of chemical engineers who couldn't solve it. But if you don't say what it is, then maybe some crazy x-ray crystallographer looks at it and they solve the problem. Now, an example of that is one of the problems that Procter & Gamble had was they were in trouble getting fluoride powder in tubes without making a mess. So they said, we can't put fluoride powder in tubes without making a mess. Some guy who's an electric guitarist on the side who knows a lot about electricity saw it and said, oh, that's easy. Charge the fluoride, give the opposite charge to the tube and it'll run inside. So he just emailed that back and he made $25,000 in 14 seconds. So what's great about this is that they then, I think, look for the nearest wall where they could bang their heads. What's interesting about this is that by exposing the problem to lots of diverse minds you get different ways of representing the problem. They didn't think of this as an electrical problem at all and you can solve it. So where I come into this story is I started thinking about this work on taking these ideas of diverse perspectives and heuristics from computer science and psychology and trying to think about how does this work in a problem-solving economy? So with Lu Hong, who's a colleague of mine at Loyola University, we did the following experiment.   So they had little landscapes and they got stuck on the little peaks and we ranked them by how smart they were.  We created one group of the best 20 agents and another group of funny random agents and we had these groups work collectively, right? Like as teams until they got stuck. Now the IQ view of this is that we had some sort of alpha group, right?  And so the alpha group were all people who did really well on their own. The diverse group included some people who maybe didn't do as well on their own. But what we found is that the diverse group almost always outperforms the group of the best by substantial margin. And when you say almost always here, we mean this in the mathematical sense so that means with probability one, right? So it's possible that they can't but given the limits we impose, it happens with probability one. And this was a paper we published a few years ago. Now the reason why this is true is if you think of people as having perspectives and heuristics as opposed to IQs, the group of really smart people all had very similar tools and the diverse group in here because I'm from Michigan, New Zealand, Wisconsin I'm gonna trash the state of Illinois like I always do. This group had people, some of whom had the right tools and others who had tools that maybe on their own weren't that worthwhile. So what is an example here? Let me clarify the second. What is an example? An example here is suppose you have people running the world's money supply, right? You're trying to figure out what they should do. Well what tools would you want them to have? You'd want them to have probably PhDs in economics. You'd probably want them to have knowledge of statistics. You'd probably want them to have knowledge of the world banking system, right? So those would be tools A, B, C and D. What would be tools I and L or E and Z? These might be people who understand network robustness. So if I took one of my friends in complex systems who studies network theory, mathematical network theory would I want them running the world's money supply? No. Would I want them running my checkbook? No. However, would I want them, if I've got 60 people running the money supply would I want to take the 60th economist out of the pool and add one person who knows something about networks? Probably, right? So that's why the diverse group is doing better. Now it's not, since this is like a mathematical theorem there's gonna be conditions that have to hold. So the conditions are, the first one is what we call the calculus condition. The problem solvers all have to be smart. We call this the calculus condition because when they write their landscape, so it's gotta be the case that they can find their local optima. So in a sense, assuming that they can take derivatives. The second condition is that we've gotta be drawing from a diverse set of people. If we draw from a very small set, then obviously you take the best ones. And the third one, this is interesting, when the problem's gotta be hard. If it's finding the optimal shovel, anybody can solve it. You don't need a diverse team, okay? Well, here's where things now get more difficult. So if you have a difficult, ironically, bad choice of words, if you have a difficult problem, it's fairly straightforward. You just wanna get diverse minds to it and you wanna just have them solve it. And this is sort of how most think tanks, most consulting companies, most universities work. We try and get diverse heads trying to solve problems. When you move to complex problems though, they become more difficult. And the reason they become more difficult is because of the fact you don't know the answer. So when you think about the healthcare policy, people are saying, why don't we do X? Why don't we do Y? Why don't we do Z? We have no idea what X or Y or Z is gonna turn out to do. Now in the old days, they had it easy. You could just go to Delphi and there would be this woman sitting on a three-legged stool. And in Delphi, there's this sort of, some sort of like thing coming out of the ground that makes you a little bit crazy. And so typically she would be sort of in an affected state. And she would say some crazy thing. And you'd bring a goat and sacrifice it first, by the way.  Well, we can't do that with healthcare policy and so it sort of stuck. But there's been lots of ways over time when we confront complex things. This is by this brilliant book called The Theory of Everything by David Arrell. These are the ways over time that we have tried to predict the future, right? So stars and planets rolling dice, smoke and fire, flights of birds, name of horses, guessing. And now we're in this new realm of things we're using called models, right? But and one wonders if at some point models won't be as funny as tea leaves in coffee grounds. But this is sort of where we are now, okay? But here's the rub. Individuals and their individual models aren't very good, right? As Bernacki found out, right? So there's this wonderful book by Phil Tetlock called Expert Political Judgment where he basically over a 20 year period coded thousands of predictions by experts. And what he found is that experts who take extreme ideological positions are just a little bit worse than random darts thrown at a dark board. So if someone's got a strong ideological opinion you're better off not listening to them, right? And just rolling some dice. If you take people, those are the people that he calls hedgehogs. People he calls foxes are people who are sort of much more diverse than they're thinking and they're just only these individually are only a little bit better, right? Than throwing darts at dark boards when you get the complex problem, okay? But here's the important thing now. If he looks at the individuals, they're not that good. But if he actually takes the collection of them, they don't do, they actually do reasonably well. They don't do incredibly well, but they do reasonably well. So when we have a problem that's complex and we don't know how things are gonna unfold, we're gonna have to have some way of making some sort of prediction or forecast of what the complex outcome is gonna be. So one thing people have advocated doing, and I'll talk a little about this, is turning to crowds, right? And asking is there some way that collections of people, right, collections of expert, teams of diverse people can predict the future. So there's a book written a few years ago by a friend of mine named Jim Sir Wiki called The Wisdom of Crowds. And he begins this book with a story of the 1906 West of England fat stock and poultry exhibition. 787 people guessed the weight of a steer. Average guess, 1187 pounds. Actual weight of the steer, 1186 pounds. They're off by a pound. Hey Sir Wiki's totally amazed by this, but I tease him because that's because he went to Yale and he lives in New York and he's never seen a cow, okay? I used to own nine cattle. It's not that hard to be within like 50 pounds. It's just like really big people. I mean they're like five times the size of people. You can be within 10 pounds on a person. You'd be within 50 pounds on a cow, okay? So 50 pounds isn't that amazing. A pound is amazing. But there's other examples of sort of wise crowds, right? And so if you look at the Iowa electronic markets which are used to sort of predict what a cloud comes. This is the last presidential election. The Iowa electronic market said that Obama should get 53.5% of the vote. He actually got 53.1% of the vote. And the final polls ran around 55%, right? So the Iowa electronic markets are sort of freakishly accurate. So one thing we can do is we can sort of say, while crowds are wise, let's just give things over to crowds, but that doesn't make much sense if you're a scientist. What you want to do is you want to kind of understand what is it that makes a crowd wise? Well, if you think about how people make predictions, what they do is we have, what psychologists will tell us and also what we do when we run regressions is we create variables or categories, right? So if I mentioned something to the United States, you might say, well, I can think of the United States in terms of the different time zones. You might also think of the United States in terms of sort of the traditional regions, right? How you frame a problem has huge implications for how you think about things and how you work within that realm. So let me describe the case of sort of two different companies and this will get into sort of just how important these interpretations are. Both of these companies serve liquid refreshments. One of them is coffee, the other one's beer. One of them divides their world like this, the other one divides their world like that. One of them is really good at what they call blocking and tackling, right? Whereas Paul Carant and Mussolini both once said making the trains run on time, right? The other one, oops, let's get this back. The other one is really good at meeting the preferences of their consumers, right? Well, it turns out this one is really good at making the trains run on time, right? Because everything's in the same time zone, everything's fine. But they're terrible at meeting the preferences of their consumers, why? Because people in Texas drink very different stuff, right? than people in Minnesota, especially this week, right? This other company is really good at meeting the preferences of their consumers, right? Because they've broken it down into sort of taste regions but they're terrible at making the trains run on time because they've got a couple different districts that sort of like cross time zones and all sorts of other stuff. So how we frame things has huge implications for sort of how things play out. Now what's interesting is we tend to think of these things as sort of being rational and model-based. But if you have a bunch of people, here's the interesting thing about policy. When you talk about a policy, we typically can't use data. There's no past data. If we get healthcare, we can sort of say, well, here's what's happened when we've sort of like, other countries have gotten a single player system. So we don't have lots and lots of past experiences with 50 different healthcare policies to say what's gonna happen. So we sort of have to like use our own intuition. So when people think about the world, what they do is they categorize things based on their own experiences. So if you ask a political scientist, why is one of my policy colleagues tell me about the state of West Virginia. This is how they would parse it. They would say there's three congressional districts and they would probably tell me really interesting things to them about those congressional districts. If I asked someone in food service, which I did, this is how I got this graph, about West Virginia, they would say, oh, here's the really interesting thing about West Virginia. You get a hot dog. There's a giant slaw region down here.  So depending on what you do, you're gonna parse the world in different ways. And as Borges said, sort of beautifully in this wonderful essay on the analytical language of John Wilkins, here's his categories of animals. Those that belong to the emperor, involved ones, those that are trained, mermaids, fabulous ones, others. All right, those that tremble as if they were mad, and those that have just broken a flower vase. So the thing is, there's lots of ways we can categorize the world just like there's lots of ways you can frame something, like a healthcare policy and these different ways we frame things, these different sets of life experiences, these different identities, give us these different categories. Why does that matter? How does this relate to this whole notion of sort of diverse and complex policies? Well, there's a book written in engineering called The Spherical Cow. And it says, suppose you're trying to predict not the weight of a cow, but how much leather you can get from a cow. Well, that's a difficult thing. If you ever try to take the integral of a cow, right? It's not easy, right? I think that's on Carl's take home final for his math class, but it's a very hard thing to do. So he says, look, instead of you should just imagine a spherical cow, we all know like the surface area of the sphere, and then you can make a reasonable prediction. And if you do that, right, you do get a reason why. But what I'm here to say is this, you do better if you also construct the gateway cow, right? So if you do a spherical cow and a gateway cow, you get two different predictions with two different ways of representing the problem and collectively you get a better prediction, right? So this plays out in policy. So after we had this sort of slight problem with financial markets in the fall, the IMF said, wow, you know, we should make sense of this. So they issued something called the World Global Stability Report. And in the World Global Stability Report, they have this wonderful language where they basically say, you can't understand this with just the spherical cow, you also need the gateway cow. They didn't quite set up. They actually said, you need four models. We can't explain this with one model, but we'll show it to you in four models. So one of their models is what they call quintile correlations where they said, you might think the way to look at how fragile the system would be to say, how correlated was this firm's profits with another firm's profits? But that's meaningless in a way. What really matters is how correlated were this firm's returns with this other firm's returns when this firm was doing really, really badly, right? Because you don't care in general if they're correlated. You care if they're correlated in their bottom fifth. They basically looked at their worst fifth of days and asked, are they correlated? And what you do when you see this is you see things that you see that like AIG, these numbers talk about how correlated they are. You see huge numbers between AIG and everybody else, right? And you see tiny numbers between Lehman Brothers and everybody else, which basically suggests Lehman had to die, AIG had to be saved, right? At least according to this model. Lehman Brothers probably doesn't like this model much, but right, that's this model. Another model they used is they said, look, we can construct a balance sheet domino model where for each country you just sort of look at how much money they've got in other countries and then you say, let's suppose we knock out country A and then let's ask us how many other countries fail, right? So instead of a firm level model, this is a country level model. And I'm sure, I was joking about this, I teach my undergraduate class, I'm sure they pressed the Dubai button sometime a few months ago, right? And saw nothing happens and they said, well, too bad, right? If it would have looked like this after they pressed the button, probably there'd have been all sorts of interventions, right? So the point is, even if you're trying to back cast and figure out what's going on, right? What you want to do is not just have one model, right? You want to have lots and lots of different models. So the work I've done on this is some work with Lu Hong again and what we found is if you construct the best model you can, given how you've parsed West Virginia. So if I look at like a food service person and Kim looks at like a political scientist, if we take the best models that we tend, then the more diverse are interpretations, the more diverse we've parsed the state, then the more negatively correlated our predictions are going to be, right? So if we want our predictions to be different, so one way to get our predictions to be different is for us to sort of parse the world differently. So why is diversity so important in trying to make sense of a complex world? Well, if we parse the world differently, if we're different in the categories we use, then our models are going to be different, our predictions are going to be different. Why does it matter that our predictions are going to be different? Because the following theorem holds, and this is a great theorem because three people got tenure for this theorem. An economist computer scientist and a statistician. This is before the internet, so no one knew anybody else had proven it. So it's all within like two years of one another. But this theorem basically shows the following, that the crowd air equals the average air minus the diversity of the predictions. So here's how far the crowd is off squared, here's how far an individual is off, just averaged across to individuals, and here's how diverse the individual's predictions are. Now you can write more elaborate versions of this, what's called the bias variance decomposition formula, but this is just sort of the simple version. So I've got something that's really complex, I don't know how it's going to play out, and I want to know, I've got people making predictions, how far off the crowd is, is going to depend on sort of how smart the people are, that makes sense, but also how different they are. And what we just saw, what Lou and I showed, is that a way to get differences is to have different categories, different interpretations, different ways of seeing the world. So we can play this out back with Sir with these cattle example, the crowd is only off by a pound, remember these are squared errors, the average person was off by like 56, 57 pounds, and the reason they happened to be right is because they were diverse.  It's not the case that everybody in the crowd just nailed it, it's the case that they happened just for whatever reason, have diverse ways of seeing it. So what is this meant for in practice? And what you see is you see some government, the US Army's in here, you see a bunch of companies here, a whole bunch of companies have started internal prediction markets, where they have people within the company make predictions rather than having just some sort of economist or market researcher decide what's gonna happen. So Best Buy for example, there's people in the business school here I'm doing some work with, Best Buy, Best Buy has, when they're buying a bunch of plasma TVs they're trying to figure out like, how many of these are we gonna sell in the next month? So what they typically would do is they'd have a market researcher come along and do some fancy analysis of like it's 46 inches large, it costs, this is the price point, here's how many we'll sell. They also have all their store managers just sort of make bets on how many they think they're gonna sell and they use that information as well. Now it turns out the store managers so far are a little bit better than the expert, okay? But it sort of varies across place and because there are different ways of thinking about it, the experts using some formal model and they're using intuition, it turns out actually having both of the two are better. Google's as interesting, they call this Profit, P-R-O-P-H-E-T. They had me in to talk to them and it was the first time all the people from Google who participate in this got to meet each other before it all been virtual and they gave the top 100 performers copies of my book and they gave the next 200 performers t-shirts and I just sit there and watch them trade my book for t-shirts. Not a happy thing. They were free. They just gave the, oh they were just, here take this, I don't read books, they're sort of the, yeah. Yeah. Was there money on the best stuff? So most of these, so here's what's interesting. So most of these companies and it varies across company. So places like Google, it's just a pride thing. Like there's a public ranking of where you are and there's a lot of cachet for being the high score. Other places give like trips to Mandalay Bay and things like that for the top three scores for some reason that seems to be a common place to send people, I'm not sure why. And then other places actually do have cash payments, but the cash payments tend to be very small. So the cost of running these things is really low, which is remarkable. One of my undergraduates did this project from the last summer which is fascinating. We wanted to ask, how does this play out over time? And so one of the few places where you can actually find people making predictions over time are the NBA draft and the NFL draft. I don't have necessarily a deep interest in the NBA draft, but it's one of these few cases where you have real experts who dig deep into information, who tell their stories, and who make numerical predictions. So here's a series of seven experts over six weeks making predictions on the NBA draft. And what you're seeing here in this first column is just their average error among these seven people. So they start out on average being off by 213, then some of them actually go up to 86, and then it sort of somewhat smoothly goes down to at the end, on average they're only off by about 70. But here's what's interesting. If you compute the diversity of their picks and the collective error, you see this really intriguing phenomena that all of a sudden in the last week, there's this total diversity collapse. Everybody starts looking at what everybody else is doing and just starts copying what seems to make the most sense. But the diversity collapse is so extreme that if you look at the crowd's performance, it's actually worse than it was in the previous two weeks. So one of the things that's really intriguing about this stuff, when you think about sort of the value of diversity, is you really need to sort of stick to your guns. Because even if somebody else has a better opinion, a model that's better than your model, collectively we're probably better off if you stick to your model. And so when I saw this, I immediately sent it to a bunch of different people and asked them what they thought of it, and they all said, most of them came back, these are government agencies and bunch of corporations, and they immediately thought, okay, we're gonna play with this. This is something that they all decided they were gonna play with in terms of looking at how often they should let people speak to one another, or just trying to also keep sequential data, because what you get here is you don't want people talking to each other too much, but clearly you want people talking to each other some, because collectivary was getting better and better and better, right? But at some point it fell apart. Okay. So let me close up with just some parting thoughts then. So the first one is, I think we tend to think of diversity in terms of this sort of red-blue thing, right? We think in terms of preference diversity. And even that I think is overblown, right? Because there's, political scientists makes this distinction, I use some language I brought from my wife on this, between instrumental preferences, which are preferences of the policies, and fundamental preferences, which are preferences over outcomes. So there's a lot of disagreement in preferences over policies. There's not that much disagreement in preferences over outcomes. And let me give you an example. I got a photo of all of the pro-crime, pro-terror, anti-growth, pro-unquality, anti-security, pro-pollution, anti-child, elected representative. There they are, right? So no one takes these positions, right? So there's almost complete agreement on crime, terror, growth, and equality, security, pollution, and the life of children, right? There's just tremendous differences in what we think the policies are, right? That will lead us to better outcomes. And there are times, right? And this is, I think, one of the high points in the last 30 years in American government. This is the 86 tax bill, unfortunately. There's no pictures of, even though Bradley and Reagan work closely on this, there are no pictures of Bradley and Reagan. And I met Bill Bradley a couple of months ago and I said, you know, I was trying to find a picture of you and Reagan for the 86 tax bill and he just laughed at me. It's like you're never gonna find a picture of me in there. And the, but anyways, this is the time when you basically have people on both sides of the hour realizing the tax code is a complete and utter mess, right? And that we should have fair taxes and they work together. But the way we wanna think about this, right, is we wanna think in terms of toolboxes, not into, we wanna think when we look at diverse groups of people, different identity groups, different interest groups, we wanna think of them not in terms of their different preferences, but we wanna think of them in the different sort of cognitive tools they have to bring to bear on policy problems. And here's sort of a funny thing. There seems to be a huge non-political advantage. So I've spent a lot of time over the last five years going and visiting government agencies and visiting corporations, visiting nonprofits, visiting universities. And the non-political organizations have a huge advantage in leveraging diversity because of the fact that they have a common goal, right? So if I go to Microsoft, if I go to Boeing, if I go to Google, if I go to Yahoo, if I go to Novartis, if I go to Genentech, they all stand there and they say, this is what we wanna do. They have a common mission. There's complete agreement, right, on what they want. And because there's complete agreement on what they want, they're really good at leveraging diversity, right? In the political realm, we don't see that. So one of the metaphors I've been playing with in the last couple of days is so the iPad came out today. And one of the great things about the iPad is not the iPad itself, right, is that Apple's got this new iPad, which is just like a big iPhone. It's just like they pumped it with air and made it bigger. I don't know what the big news is, it's just bigger. But what's interesting about this and about the iPhone is it basically created this landscape on which all sorts of people could create applications, right? So for approximately like $50,000, I mean basically three people working for three months can make an app. So that's probably like 50,000 bucks of labor, right, plus some computers, and you can make some app. And maybe that does something amazing that makes the world a better place. Maybe it doesn't, but even if it doesn't, you learned a lot in the process, right? But Apple's created a platform under which all these diverse talents can come in to try and make the world a better place, right? Well what's the equivalent palette for society? I think it's right down the road, right, Detroit. There's all this infrastructure there, right? There's power, there's water, there's streets, right? There's a huge airport nearby, and buildings are cheap, right? Land is incredibly cheap, right? We need to somehow think about how does government make Detroit, right? The equivalent of the iPad touch, right? So they can leverage all those things. So one of the things about Michigan that intrigues me is when I, I mean there's an undergrad, it was an undergrad here, you sort of worship Bo Schemeckler, right? And right before Bo died, Bo was giving a talk. Here my wife and I went, and Bo got up and he said the six words that Bo always says, which are the team, the team, the team, right? And this is sort of my the team, the team, the team slide. One of the things that, you know, so John Holland is sitting here, is one of the founders of complex systems. One of the things that's so amazing about complex systems, if you take something like the brain, right, and this is just a picture of a neuron, any individual neuron is a really simple thing, right? It can't do very much, right? It's got axons going in and dendrites coming out, and it just has this sort of sigmatical response function, it's a very simple thing. But if you have differentiated neurons, right, connected in the right way, they can create consciousness, cognition, personality, emotion, all these amazing things. The ramp up, if you had some sort of number of how much more impressive the brain is than the neuron, it would be huge, it'd be bigger than seven, my son Cooper likes to say it, right? We're not getting a similar ramp up in our government, in our organization. But if you say how much better our organizations than people, that number's probably less than seven, right? So I think the challenge, the opportunity is how do we do that? And I think we're trying to do some creative stuff. So this is my last couple slides here. DARPA did this fun thing a couple weeks ago, they hid 10 balloons, 10 big red balloons across the United States, and they said, so you can find them fastest, and whoever finds them fastest gets this prize, right? MIT won, here's where the balloons were, MIT won in less than nine hours. Now MIT didn't win because what's funny about this, people said, oh, MIT won, because those MIT people are so smart. What MIT did is they didn't do anything, they basically said, here's the deal, I can't remember how much money you got, it was like $10,000 or something. MIT said, if you find one of the balloons, you get one half of one tenth of the prize money. If you're the person who finds the person who finds the balloon, you get one quarter of one tenth. If you're the person who finds the person who finds the person who finds the balloon, you get one eighth of one tenth. So basically the idea was that either like, if you saw the balloons, say you found it, if not email somebody in other cities, say, hey, do you see a balloon? Because that was the way to get money. So what they did is they sort of just created this giant web that could solve problems. So I think that my final point here is, we really do need to sort of, one of the things that we learn, I think, from complex systems and thinking about diversity in complex systems is this amazing sense of wonder. And this amazing sense of possibility in terms of the ramp-ups that we can get from diverse people trying to solve problems. And this is why I think universities are really our great hope, because of the fact that they sort of bring together all these diverse people. But I think we have to keep in mind, my last political statement, that maybe we don't always want to go where everybody else has gone. And I tried to write John Holland on this, but I couldn't do it. Thank you very, very much. And I'm happy to answer questions that people have. Yeah, Tom. Yeah. Yeah, no, that's a great example, right? So the question was, he said he couldn't help thinking of Nate Silver's 538.com, right? And this is a website where he has all sorts of different people who make predictions. Terrifically accurate. Right, of who's gonna win which state, who's gonna win which congressional races and that sort of stuff. And then he figures out ways to average those, right, to make collective predictions. That's just a classic example of sort of combining models, right? There's a ton of research in computer science and what they call sort of ensemble learning theory where it's about sort of, how do you combine different models? So in the forecasting literature, there's sort of, you know, reason we need statistical papers about if I've got, which he's leveraging, right? Which I've got 20 predictive models and I know their accuracy and I know their correlation. How should I combine them? How should I weight them? Basis. Right, and also questions that have been framed in slightly different ways and that sort of stuff, right? Absolutely. Other question. Yeah. So in biology, I'm sure you're aware there's a problem of how much diversity in the population leads to a fast adaptation or finding a solution quickly. And often there's an intermediate diversity that is often what with a few, too much diversity you tend to lose the sense of an optimal situation. And what determines that is the structure of the landscape underlying the problem. So I'm curious like in public policy, how do you know, do you think there's an optimal level of diversity? How do you know the landscape looks like underneath different types of problem? Is there some problem that really just having a bunch of economics PhDs going to the same school is actually the best way to go? Right, so the question is how do we know how much diversity we should have as a function of sort of like how rugged the landscape is, right? And so this is a great question. And the answer is, right, clearly you want the amount of diversity to fit the problem. So for any, just like that, in some sense, if I could draw the equivalent of the shovel landscape where I had the amount of diversity on this axis and how good your solution's gonna be, but the thing is where that peak would be would differ depending on the problem. And this gets, I mean, so this is something that's puzzled me for a long time is we spend a lot of time when we talk about policy problems, talking about who's affected by them and the like. But we don't actually go through and do measures of how complex they are or how difficult they are. So if I said which is more complex or which is more difficult? Healthcare, welfare policy or tax policy. I mean, we don't have any metrics with which to think about those. And yet it would seem to me, right, this may be a naive view, but if you thought about from an organizational standpoint, what should the organizational design look like to solve those problems? That should be related in some way to the difficulty of the problem, right? But yet the way we analyze, the way it was trained in mechanism design as an economist, we focus instead entirely on the incentive problems and the information problems. We don't focus at all because we sort of assume people can optimize, which is sort of, I think, not necessarily a great assumption in those settings. So I'm with you, but I don't know how, it would be a large project on how one would get it underway, but it strikes me that it would be a very meaningful thing to figure out exactly how complicated is healthcare. So you've been hearing for months that healthcare is complex, healthcare is complicated, healthcare is difficult. But the thing is, how does that compare to tax policy? Right, how does that compare to welfare policy? We don't know, and if we had a better understanding, it seems like we could understand better how we'd go about solving it. So in engineering, if you think about, we're computer science, how do we solve the problem? The sophistication of the algorithm depends on the complexity of the problem, right? In an ecology, we know the diversity of the species that we need depends on sort of how rugged the landscape is and how fast the landscape is changing. Whereas the landscape is moving fast, we need a lot more diversity. So yeah, maybe we should get some biologists to move into public policy, right? Yeah, question. What are the differences or the role that cognitive diversity and cultural diversity play into each other? And if you're talking about the same thing or if you're different constructs. So the question is, how does cognitive diversity and cultural diversity play in the same, are they the same thing, how do they differ? So this is, in some senses, an empirical question, right, and it depends on, I think it depends on the particular problem domain. So there's a lot of, so in most problem domains, cultural diversity is just gonna play in them. If you ask people from different cultures how they make sense of things, have them put them in categories, you see big cultural differences, right? Particularly if it comes to anything involved in the natural world, right? So people from sort of Western countries, let me give a specific example. So people from Western countries, if you have them look at like a rainforest or a group of animals, they'll use sort of a Linnaean system where like they, here's the animals, here's the plants, right? Here's the trees, here's, you know, I think the fishes that live in water, that sort of stuff, right? If you take someone who lives in those cultures, they will say, or in those ecosystems, they will actually think of, they'll say, you know, here's this bird that eats the nuts off this tree, right? And here's the flowers that grow on the base of that tree. And if you say, well, why don't you classify them by animals, trees, and flowers? They'll say, well, that's the crazy person's way, right, of categorizing these things. So things in the natural world, we tend to, there's huge differences across culture in terms of how we do it, depending on just our familiarity with it. But in some respects, I think it's a very, it's, I think, just a really intriguing, open question about what causes differences in how we, you know, see how the world works. I mean, you know, my sort of, very, I would say, superficial reading of the psychology literature says that it's a mixture of sort of culture, your own sense of identity, right? You know, just sort of richer than culture. The stories you've read, the experiences you've had, the training you have, right? That all the different models and ideas you carry around in your head, you use some sort of weird case-based logic to sort of say, you know, this is the way I'll frame this. And you just get sort of massive differences in how people will frame things. For instance, I, this, in my undergraduate class, I had them predict a bunch of things, like how many chairs are on the Starbucks on Washington, to like, what's the tallest building in Brazil? And what's interesting is the tallest building in Brazil, the average guess was really close to correct. But some of the predictions were really low and some of the predictions were really high. And the people who had it as really low, they had tended to have visited some other country in South America and just said, look, they just don't build big buildings down there. And the people who had it as really high, they were like, it got the Olympics, Olympics only goes to big cities, sort of thing, and they had that, because I had them write explanations for why they predicted what they did. And so it was very clear there that there were differences in sort of life experience that translated into different ways of, you know, different categories in which, you know, they thought of the country of Brazil and they thought of Rio, the city of Rio, and that led to these different predictions. So I think it's a really intriguing question. Yeah, use it. I think there are two, it's very good talk, but I think the two pieces are missing from your talk. One is in terms of reward, some reward is collective, but is averaged out, some can be spread to the whole group at no cost. Technology is a ladder, for income will be the forward, because you have to share. So for sports, for example, basketball, diversity will lower the average, but if you play, what kind of golf, then you pick the winner, then diversity is good for you, because you have more chances to win. So there are two types of reward systems. One is average out, the other is you only pick the winner. That is true in the ladder case, then size matters a lot, and that's basically Jared Diamond's argument. The larger the population, the more likely there's invention, because invention gets replicated at low cost across large populations, then you have more innovation, more technology, more economical growth. So when it's a relationship in diversity and size, where the density matters, the other is different for different outcomes, actually, there are different types of rewards. When it's averaged out, the other is you pick the winner for the group. I think the solution, for example, you pick the winner rather average out, average score. So when it's zero sum, the other is actually pick the winner, everybody wins. So one of my former colleagues, Jojo McCloskey, who's an economic historian, has thought a lot about this first question, and she argues less empirically, more just in historical perspective. If you look at where great innovations have come, it's come from places that have been both dense, but also centers of trade. So if you tell stories of Athens, Rome, Amsterdam, London, United States, it's not only that you've got a lot of density, you've also got all these different cultures coming in and all these different ideas sometimes through people, and sometimes actually just through the artifacts. As things came back from China, to Rome and stuff, into Italy, that those have embedded knowledge that hasn't been able to be unpacked. So I think it's not only density, but it's also sort of exposure to lots of different sets of ideas. I mean, you're absolutely right in terms of this payoff stuff, one of the things I think that makes public policy so much more difficult than other domains is that we don't get to experiment with lots of different health care plans. And there's also this difference between you don't have, one of the things that people who study this stuff, the collective problem solving stuff, there's this notion of an oracle. So let me take the example of automobile design, because Jeremy's here. If I'm thinking about the aerodynamics of a car, like you can go into Ford and you can draw on a computer, like you can just change the picture of the roof a little bit.  So you've got an oracle, a perfect oracle. So what that means is each one of us in this room could go and start designing cars to try and come up with one that's aerodynamic and we could pick the winner. So we could totally leverage the diversity. But now let's take the question of designing the dashboard. Well, now there's no oracle. Now if you design a dashboard, we can't press a button and have it come back and say, that dashboard's really cool.  It's incredibly expensive to evaluate it. And so therefore, you can't have 10,000 people and then just pick the winner. So it's not just a matter of the density, it's also a matter of having some sort of, oracle in which you can do things. That's why the computer programming thing works so well, right? Because you can just press a button and it comes back and it tells you exactly how fast the computer program runs, right? So one of the big constraints on in open source programming is just, the real geniuses in open source programming are people who are able to take this giant open source problem, whatever the problem is, break it into components and have those components have oracle-like properties to them so that you know when you've solved the component correctly. And so if you, Brian Arthur and Paul David have spent a lot of time looking at this, when you look at the ones that have been successful, the open source projects that have been successful, they've been ones in which the subcomponents have had well-defined oracles. So that way you can just sort of choose the best and you can also very quickly sort of see when they're improvements. On your second point, some of the stuff I'm working on now gets to your second point in terms of what are the payoffs? So one of the really interesting things is different payoff structures, different incentive structures can either sort of encourage diversity or drive it out. So if you look at the amount that these different investment, I have a graph on this, but on this talk where different investment houses were sort of, how much they were leveraging their assets, Morgan Stanley was way below the other companies, but they weren't making as much money. And so right before the crash, they started leveraging at the same level as everybody else and they imploded. And the reason why is they started copying other people.  Now one thing about markets that's really intriguing though, is in a lot of markets, if you're right and nobody else is wrong, you get a huge payoff. So markets create this sort of weird incentive then, and this goes back to Hayek for a lot of cognitive diversity because if you can be contrarian and be right, there's a new book that just came out called The Greatest Steal Ever Made, the guy that made $15 billion leveraging against the housing market. So if you're right when nobody else was wrong, you can make $15 billion. And one can even tell a story that one reason democracies work well in market economies is democracies are actually sort of free riding of all the cognitive diversity that the markets are creating. Whereas a democracy and a sort of more totalitarian, I mean in a system that has a common religion and not a very diverse economy, there's not as much cognitive diversity in the pool. And so when you're asked to evaluate policy prescriptions or to think about policy, you don't have this sort of population of diverse thinkers that you can leverage. So I think it's really interesting thing about how do the incentive structures right in terms of pay affect how diverse the thinkers are gonna be, right? And because you could have too much diversity, you could have too little, and it gets back to this other question, sort of depends on the problem and that's why it's complex, right? Should I, it's not even, yeah. So you started out talking about sort of the old politics of old political science, which was about aggregating preferences and media and voter theory and left to right. And then you went into the new, which is more about assuming we agree on a goal, how do we find the best solution to it? Which it sounds like this second one is sort of the economist paradigm. Let's agree on some measure of output and then we find the production possibility for our two years and the best way to get to it. But it seems to then sidestep the question of how do we agree on what the outcome is? So the first paradigm, it seems like they're about different problems. One is about how do we agree on what we want? And the second is assuming we agree on what we want, how do we get to it best? Right, no, so your point is absolutely well taken. So the question was, I mean, one could say the first thing I was talking about was sort of standard political science, which is about differences and preferences. And you're saying the second part was saying it's sort of like economics but solve the problem, right? But the difference here is what I'm saying is that typically when economists are talking about solving a problem, they haven't talked much about diversity, right? And so what I'm saying is when we go to this, and when economists actually, if you look at most of the economics literature, right, it's about sort of how do we get people sort of to put forth enough effort? So we talk about like hidden action and hidden information, we talk about aggregation issues and that sort of stuff. But we don't, economists typically haven't looked at how do diverse groups of people solve hard problems? That's been more sort of within the realm of industrial ecology, engineering, that sort of, you know, psychology, group behavior, that sort of stuff. So what I'm saying is that one of the things that I'm trying to do when I think about, you know, public policy type questions is to say a lot of these things are really hard, right? And when we think about how you solve hard problems, you've really only got two approaches, right? One is to have super sophisticated people who can somehow solve them, right? Or it's to have diverse groups of people who can sort of somehow collectively get to a solution. And so the point I was trying to make is that when we think about, if you say diversity in politics almost anywhere, people are gonna think of that in terms of diverse preferences and diverse wants and people wanting different slices of the pie. And what I was trying to say is there's a, what I wanted to focus on was something different, which is that all those different sets of experiences, those different ways of thinking, those different sort of, even goals, because goals lead to how we frame things, right? Our lever on which we can stand to possibly find better solutions to some of these problems, provided we can overcome all these sort of other issues. But you're absolutely right in your dichotomy. Yeah, yeah. So with your Detroit slide, so. Yeah. So I buy the view we get all these collective diversity problem solving for Detroit, okay? So now you've got this great range of solutions. Right. So then the political science question is. How do you decide among them? No, no, assume you even have some mechanism by which to know. How do you actually put them into place given that they're winners and losers? Who, even though this is the optimal problem. I mean, so Google can do it because the two guys who run it can say, great, we're gonna go forward. So you get great ideas about Detroit, some of which involve shrinking the footprint of the city, others involve changing the racial diversity of the population. Right. It's sort of like in the using, assuming you do all the diversity in the problem solving. Right. Then how do you get the policies implemented? So there's, I think there's. Right, I think there's, your question was once you've, if you've quote unquote solved the problem, how do you get the movement? I actually would argue there's sort of three stages. One is that you've got this sort of stage where you have people who sort of are in some sense formulating policies and ideas. But then there's a second stage in which you somehow need, I think probably again a diverse group of people then to figure out which one of these then, which ones of these actually make sense. Right, because the thing is any one person's, you know, somebody's gonna have a view about let's shrink the size of the city or let's create enterprise zones or let's, you know, create new transportation sector, something because the people who form what tend to be advocates, you're not necessarily gonna get an objective view of how it's gonna play out. So suppose you get diverse people then that say, okay, this is the one that we think is gonna work best. I think the implementation thing is, I mean, it's certainly outside my area of expertise, right, from the outside everyone's. But I think that there's probably, I mean, I think there's an argument you made and this gets back to the sort of the red state, blue state slide. I think that we do need to change our political culture where we sort of understand that if we had a whole bunch of people look at this and if we had a whole bunch of people sort of make it a reasonable appraisal of this and we've decided this is the right one and even if mine didn't win, that we should go forward with it. Right, so one of the things like when you look at those slides and sort of the prediction slides, right, and I talk about this a lot in the context of my book is that if you go to a meeting and everybody agrees, there's only two possibilities. One is that it was, there's no reason to have the meeting or you just probably made a bad decision, right? But if you go to a meeting and everybody disagrees, right, provided you agree on the ends, which is to make you drive a better place, odds are the decision was a good one even if it's not your decision, right? So I think in terms of the implementation, especially in a place like Detroit, it's just a matter of put it go will and it's a matter of people willing to sort of give those resources, right? And I don't know, and again, how you muster that put it go will, I think it's a real, I think it's a hard question, I think it's a challenge that Dave Bing gets up every morning and asks himself, right? But I don't know. If I knew, I'd call it. Yeah, see you. Just follow up on the same discussion but to take it in the deep of it. What I was thinking about with earlier change was how the problem with these very complex problems, poverty, terrorism, unlike the examples that you were giving us was well-defined what people are trying to accomplish and therefore you can worry about whether you have this oracle or not to see how well they're doing to solve it. But if you have a complex issue where people don't necessarily agree on what they're trying to solve, how do you think through whether or how much diversity is helpful to define the problem before you try to solve it? Okay, so that's a hard, again, I should have easier questions, hard question. If I think of this sort of from the perspective of my training as a sort of a mathematical economist, it's sort of hard to think about if the problem isn't defined, how do you solve the problem? Because the problem isn't defined, right? But there's, so the question of problem definition, right? I mean, to what extent is it useful to have a diverse group of people even define the problem? You can think of categorizing that as something jokingly called the problem of problems, right? Which is itself a problem, right? Like how do we define what the problem is? And so then you could say, okay, the problem of problems is just a problem so then we can solve it. But that turns out to be a cheat because if you sort of do the math on that a little bit what you realize is that the dimensionality of the problem of problems is much larger, right, than any particular problem would be. And so and once that dimensionality gets large, given communication problems, you're at best gonna get sort of a random strike through the path. And this is why, and I'm not trying to punt on this, but this is why when you look at sort of the literature on sort of brainstorming and problem creation that there's just, there really isn't much, there's business professors who don't wanna correct me but in my reading of this stuff, I guess I'll review on this stuff three months ago. I guess really isn't anything that has strong empirical support that this always works or here's a good way to do it. Instead, there's sort of like 40 theories of brainstorming and 30 theories of problem development. And I think the reason why there isn't a good science behind it is it's an a combinatoric problem. The set of possible problems is so large and the set of possible ways to think about framing these things is so large that there's no way to sort of have a systematic approach to search through them and so it ends up being very path-dependent and arbitrary in a way. But that doesn't mean we should throw up our hands. It does mean if we go back to the sort of open source thing, I think it makes sense to ask, how do we decompose these things into problems that are possibly doable or how do we find things that possibly are measurable? But at the same time, I have concerns about that because of the fact that if we, sometimes the things that are most important are that like people's life satisfaction are very hard to measure and so they end up focusing on economic things. So I think it's just hard. Yeah. Okay, yeah. How do you see people's expectations and the whole rate of change affecting some of this? I'm thinking of things like companies have done just in time engineering and just in time delivery of goods. And if you had three people and they each said, today I need my heart transplant paid for. One says I need my flight to Amsterdam to be safe. And the third one said, I need my fourth grade kid to have a high quality classroom. They're not antagonistic to each other. They all agreed on the outcome like your picture that said all those Congress persons. But it seems like they each needed their thing today but we don't look at policy like, how do we get all those in one policy? We always keep saying, oh, there's the terror policy, there's the education policy and there's the health policy. But they have expectations about it being today because they see all these other things get delivered just in time. So how does your models address that or look at how that change over time makes a difference? So I think, you know, in one sense they don't, right? I mean, I think that that, I think it's a separate question. I think that the, in this case, maybe the Eugia's question before I think the nature of these political problems, there's fundamental difference in the nature of these problems than some of these standard business problems. So somebody wants a cup of coffee and get a cup of coffee and has to do with the fact that these are, a lot of things you're describing what we call as economist, you know, non-rival goods in the sense that they're, you know, there's something that I've got to create for everyone and so because I've got to create for everyone it's a huge undertaking. And, you know, you're sort of the only rejoiner of that in some sense is an anecdote that I was talking to someone from Toyota and they were saying they showed a concept car at the International Auto Show and three weeks later they saw China was making toy versions of the car, right? So you can make the toy version of something, you can make an individual product for someone like that really, really quickly but an actual car from soaps to nuts takes several years. I mean, so the thing is the policies you're talking about are things that are going to take, you know, decades to get undone, right? And so even if you get the right policy that doesn't mean you're going to do it fast. And if anything, having more diverse groups of people chime in on something may slow things down, right? But you're less likely to probably have make big mistakes. Yeah, last question. I'm sorry, I think some of the motivation there is some of the where modern and kind of thought is going to come from. So I'm wondering what you see as some of the differences between what you're putting forth as harboring diversity, solve problems and how that's different than just pushing for kind of a market approach. It seems like the highest view of coordinating knowledge was in a similar spirit to this but then the stock market crashing it's like control, it's a control market. So yeah, so I think there's a similar day between, there's a lot of similarities between people who study complex system stuff and some of the old Austrian economists in terms of how they think about, you know, diverse things aggregating into something that's better. But that doesn't mean that we don't have to want to continue to think about, I think that I would probably be a little bit less laws I fair in the sense that when I look at what happened in the most recent stock market crash, it seems to me that that was to some extent sort of a breakdown and diversity. You had all these people who had the same model in their head, right? You had a few people who didn't, but those people who didn't were basically people who were buying puts, right? And we're gonna make a ton of money if it failed but they weren't stabilizing the system, right? And so what happened is you had all these what we call positive feedbacks where Morgan Stanley, we all started copying other people by leveraging more because everybody else was leveraging and ended up with sort of a common model. Basically everybody believed that you couldn't have the entire real estate market collapse because it had never collapsed in the past, right? In the past when you had the 87 collapse, the New York market collapse but the West Coast market and Chicago market were pretty much okay, 2000, you had a collapse of the San Francisco real estate market a little bit, right? Not entirely, the West Coast collapsed a little, the New York market was fine. So it was just a sense that the real estate markets were more regional, right? But what they failed to recognize is that the Federal Reserve System, right? Which is a series of regional banks, that used to be way more regional and now it's basically just one big bank and they should have sort of realized that we're probably all one big economy now and the whole thing could go. So there really was sort of a common model in their heads in a breakdown diversity that caused this thing to collapse. And so one of the things I'm working on now I think it's really interesting is how do you construct economic and political and social institutions in such a way that in some sense you encourage the right levels of dissent and also that you sort of foster diversity, right? Not so much of it that we can't be productive, but enough so that you create a reasonably stable system. And it's not where you could say, well, let's just look to ecology, right? But if you look to ecology, right, we've had mass extinctions. Or if you did Doug Irwin's book, right, we've had mass extinctions and a lot of people would argue that it's not the case that everyone knows mass extinctions is because a huge meteor hit, right, that sometimes it can just be internal dynamics of the system can be such that WAMO, right, you just lose a whole bunch of different species. So there's a question of, do we just sort of sit back and let stuff happen or do we actually try and think in careful ways about how we can construct these institutions so that we maintain enough diversity so we don't have these big large events? All right, thank you very, very much. It's a lot of fun. I would like to invite you to continue the conversation for bit informally. There are some requests that's right outside of the audit for me. And thank you very much. Okay, Adam. Jump.


And only good morning. Good morning. Yesterday, Sandro Tonali had his ban confirmed out for 10 months for betting irregularities, breaching the regulation. I just wanted to ask you, I know it's a bit tricky, Bournemouth's got a sponsor behind Daffa Bet, but how do you feel about the relationship between betting and football? I think when I played, I was a player. Betting was not so developed. You know, it's not like today, probably that we find it in every day, but it's obvious that if you are part of the game, you cannot be involved in nothing like this. For players, have they got too much time on their hands?  Is there a danger that...? No, I don't think we can generalise everything. And I think each person has his issues, has his problems, his whatever, addictions. And then we have to take it separately. I don't feel, from what I see now that I've been coaching, it's a big thing in football players. I don't feel it, but it's true that there are sometimes these kinds of cases, but I don't feel it is something general. Now, on to your team. Last weekend, we saw Alex Scott make his debut. How did he come through after that? How is he feeling this week? I think he finished tired, obviously, but with good sensations. I think he's a hungry player. He's young, but when you talk to him, he understands the game, he understands the pressure, he understands everything, and I think he's willing to help us. He's willing to help us, and he's not only focusing in, okay, I'm coming back from a long injury, the debut in the Premier League, no. He is aware also about the situation of the team, and he wants to help here. And have you got any team news, any worries, concerns going into tomorrow's game? Yeah, probably from past week, we are going to lose Neto. He has been injured this week. What's the extent of his injury? Yeah, it was an ankle injury two, three days ago, and he will be out, I think, at least, until the international break for sure. And if everything goes well, probably after the break, he can be again with the team, yeah. And any other concerns? No, nothing, no other injuries. There are still a couple of players with normal issues, nox, but nothing important that we will take them out from the game tomorrow. Now, last week you said it has to be now to start picking up victories after the defeats of Wolves. Does that put pressure that this is really a must-win game for your side? Yes, I think so, I think so. I think there is pressure. I think we are facing a team that is in a similar situation as ours, also some similarities, because they are also a young team. Probably they are conceding goals in moments where they deserve more, they are playing well, but then they are struggling a little bit there. And we want to beat them, and we want to pass them in their standings. And it's a very, very important game for us, yes. With the nature of the Premier League, there's always talk about managers' future when the clubs are not picking up the points. Is that something you understand, why there's questions maybe surrounding your future? I understand absolutely the questions, the situation, what it means to be part of these leagues. But I'm not focusing on this, you know, because the same as the team, if you ask me for my personal situation, if we win, it will be a little bit better, if we lose, it will be worse. But at the end, I have to focus on what is going to happen on the pitch. And we understand the owner, it's going to be an attendance tomorrow's game. Have you spoken to him about the current situation, what he said to you? Yeah, Bill came this week, he's going to be watching the game, I think he watched the last game against Spurs here also, and we have been talking about everything in general. He's a little bit about football, a little bit about other things, about a lot of things, and I think it's good for me to spend time also with him and to everyone, and I think it's good also he's with us in these moments. And he's given you assurances about your position here? The problem is not that I don't feel back, no, this is not the problem. If this doesn't work or whatever, it's going to be on me, on the players, especially the ones that perform and are the responsible of the results, it's going to be like this. Thank you. And only good to see you again. You retain the support of the owner, you've touched on it there, how much does it mean that you have that support, particularly heading into this must-win afternoon, as you've called it? No, I've said it since the beginning, I have no complaints there. I think I'm not focusing on that side. Since day one here, I've been very happy with the players I have, obviously with Hudson, probably some injuries that have been important, but it's something that I'm sure is happening in every team. If you ask a company tomorrow, they will tell you the same. No, I'm not concerned about this. Did it take a few days to pick people up after the Wolves' defeat last week? What was Sunday, what was Monday like for yourself? No, I think especially when we lose the game like this, it hurts more because when you play, you lose a game, I think the worst we've done probably against Everton, you lose three-nil, they deserve better than you, you say, okay, we've just been not good enough and they've won and that's it. But the other day we felt that we were close, we were close to winning the game after the first half and we were ourselves, we talked about this, we've made especially two big mistakes that took us from the game. And it hurts more when you lose a game like this. How do you feel you can best protect your players from the pressure, from the current situation you find yourself in ahead of what is a vital weekend here? I think they have the experience from past season, most of the players have experienced past season something similar and they were able to turn the situations to get the points, to get the wins, but we have to, now it's not the moment to talk too much. The only thing that will solve this is go there and win games. You know, it's okay, I would try to answer everything you want to tell me, but I know that the only answer that especially Varmouth supporters wants is to go there tomorrow and win the game. You touched on Alex Scott already coming into the side after his injury last weekend and where Tyler Adams injured and Lewis Kier suspended. You need a lot from him already in that midfield, don't you? Have you got some difficult decisions to make on how you can make that midfield work this week? Yes, probably. You asked me for Alex Scott. It's not an ideal situation for a player to come after an injury, the debut in the Premier League, and to be demanded so much, probably, but it's the situations we have and he's the first one that wants to help and we will need him the same as we will need other players, especially in the midfield, yes. Vincent Company is a big presence in our gang. Huge name in the Premier League, gone into Berlin overhauled so much about them in the last few years. They've become a side that wants to dominate possession, don't they? And do you expect them to come here to want to do that this weekend? Absolutely, I think they are a possession-based team. They've got results with this way of playing. They have been trying to do it. I almost told the games they played also this season. For sure they have been adjusting and probably not playing the same way as last season, like we are doing week to week, but you cannot change all the philosophy of the team from one week to the other. We speak a lot about philosophies and approaches and managers' ideals in the game. It gets to a point, does it really matter how that three-pointer is picked up? For you and Bergen? We don't do it. We don't play a certain way just for the philosophy or for the win for two-in-games. You know, it's because we think it's the best way of the closest way for us to win games. Maybe we are wrong, maybe there is... But I think we have to make adjustments, we have to change things. It's true that whenever you don't win games, you cannot continue doing exactly the same. We have to make adjustments, we have to make changes, but I don't feel like neither Barley, nor us will go much, much to the opposite side, let's say.