So good afternoon everyone. Welcome to New America. My name is Lauren Ellen McCann and I am a civic innovation fellow at the Open Technology Institute at New America. New America is a nonpartisan think tank and civic enterprise dedicated to the renewal of American politics, prosperity and purpose in the digital age. Our experts work on a wide range of social, political and economic issues ranging from telecom policy to national security to work and family. At the Open Technology Institute, or OTI as you might hear it referred to, we bring together technologists, practitioners and policy experts to work in a variety of pressing technological concerns and considerations, such as the creation of affordable, equitable, safe and ubiquitous access to communication technologies across the US and the globe. Today we are gathered to discuss the ways in which technology can, should and is impacting local governance. Over the past year here at OTI, our fledgling civic innovation program has been examining the intersection between bureaucratic innovation and technology, with a particular eye to American municipalities, cities, towns, you name it. It's on this local level that we see the great theater of digital governance play out in sharp relief. Where tech driven experimentations in service delivery and government efficiency and genuine constituent engagement can be felt and not just seen. Our investment in digital and this whole idea of innovation in general, as part of the future of cities and local governance, stems from a few sources. The realization of the dramatic population shifts into metropolitan areas, the tightening of budgets paired with the increase in constituent dissatisfaction, the demand perceived a real for certainty and assurance in our policymaking and the absolutely real craving and necessity to transform the relationship we the people have with our governments. From social media to civic hacking, we see the use of digital tools all over the world, supporting literal and figurative revolutions. And so we assume that these mechanisms will have the transformative impact we seek when we take them into city hall. But there are a lot of dragons on the map when it comes to integrating technology into our governing process. We're in a whole new world. And for advocates and public servants alike, this rising tide of new tools, foreign concepts, and rapid production is absolutely overwhelming. Making critical judgments about which technologies to use and when difficult. Crafting a culture that enables bureaucratic creativity, I know it sounds crazy, and experimentation is necessary to face the challenges, not just of 2014 and the years beyond. We know that. And we all want city hall and town hall, for that matter, to do better. And not just better, but to be more equitable. We know that the story we've told ourselves about how our democracy works no longer keeps up with the reality of how we govern. But in our desire to make the system work better for everybody, we must also be critical in the approach we take to making it so. Exchuring that our experiments are responsive to context and to the expressed needs of the communities they're designed to serve. Even as we advocate for innovation, therefore, we have to look around us at how these communities that are engaging on the front lines are intaking this information and these new technologies, these data, learning from both their flaws and their successes. We need to attend to the real-time case studies that we're experiencing. And I believe we also need to keep our broader vision in mind. Asking not just whether the adoption of new tools and data-driven processes makes the governments that we live in today better, faster, stronger. But whether or not they positively impact our core democratic relationship, the power structures and hierarchies that we live in tomorrow. So with that, you know, very light-touch overview, We're going to go into a closer examination of this idea of digital governance. Focusing in on today and the past few years and in particular data-driven governance in American municipalities. In light of a new book, The Responsive City, Engaging Communities Through Data Smart Governance. We're joined by an excellent panel, including two special guests and two house favorites going down the line. We have Alan Davidson, Vice President of Technology Policy and Strategy at New America and the Director of the Open Technology Institute. Susan Crawford, Visiting Professor in Intellectual Property at Harvard Law. Professor of the Benjamin N. Cardozo School of Law and co-author of The Responsive City. Steven Goldsmith, Director of the Innovations Program at the Harvard Kennedy School and co-author of The Responsive City. And Dr. Holly Rossengillman, Civic Innovation Fellow at OTI, a favorite of mine. And former White House OpenGov and Innovation Advisor. Couple of housekeeping notes before we get started. Today's event is being live-streamed and will be made available on New America's website, newamerica.org, afterwards. And for those of you both in the room and watching online, we encourage you to interact with us during this panel using your fingers. Please tweet with us using the hashtag The Responsive City. And now I ask you to join me in welcoming our panel. Thank you. Thank you. Thank you, Lauren Ellen. And it is a real pleasure to welcome our two guests, Steven Goldsmith and Susan Crawford. I'll just say a short editorial note. I've had a real, it's been a real pleasure of mine to be able to work with Susan for a number of years on a lot of different issues. And I remember when she came, when we had a discussion at least over three years ago about the fact that she was going to teach this class at the Kennedy School and they're just going to partner with the mayor's office in Boston and they were going to get Kennedy School students to do these technology projects in the field. And I just like, I remember thinking like, or maybe even saying, Susan, like you were just at the White House and you've been, you've been, you've worked on internet governance around the world on the ICANN board. Isn't this a little kind of like small ball for you? Boo, right? I have to say all you have to do is read some of this book to get a sense of the optimism and opportunity that is coming up from the cities and localities. And I'm sure that's going to be a part of the theme of this. But Susan had it right. That's where the real action is. So a quick note on our run of program here today. We're going to turn to Steven and Susan to give us an overview of their book. Holly and I will do some Q&A in a sort of fireside chat type style. And then we will leave a lot of time for your questions. So please arm yourselves. And we look forward to hearing your thoughts in the audience as well. And we will try, we will wrap about a quarter of, a quarter of two. So with that, let me turn it over to Steven and Susan. I'm going to start with a few remarks. It's your call. Whatever makes you comfortable. I started my career as a district attorney and speaking to people whose faces you can't see always bothered me. So I'm going to stand. Nice to be with you. I'll answer the questions in the fireside chat, but let me just make a few remarks. And having been in local government for 30 years, I found your comments highly demeaning about Susan having, having descended into city government from the White House, right? Yeah, so let me just make a few comments about the, the book is kind of a lead up to the, to the rest of the conversation. And maybe the way particularly here in this particular audience, the way to think about this is that I'm going to approach this, the following kind of construct, right, that progressive government was producing regressive ends in city government, city hall cities. And let me explain the kind of the setup and then a couple of comments about how I think the technology revolution could make a difference. So, you know, we had this, you know, I was active in this area as deputy mayor of New York. And if you think about New York, you know, you think about Tammany Hall and you think about corrupt government and abusive discretion on the part of mayors and top officials and the way that we, with progressive government eliminated much of the corruption and abusive discretion was with a set of accountability and rule based measures that basically said, if we make sure that nobody has any discretion, they will not abuse it, right? So if you were good, we made it very difficult for you to be good. And if we made it, if you were bad, we made it very difficult for you to be bad, right? So now you go forward a century with a very well-developed bureaucratic and not necessarily even in the pejorative sense, hierarchical, command and control, job classification, traditional HR, union contracts layered on top of it, methodology and people, people meaning public employees are pinned in these little boxes. So that's that's one issue. And so against that, right, we have this complexity of current society, the issues that were mentioned in the introduction about the resources available in cities really not being sufficient to meet the problems that are are at hand and how do we how do we resolve that? And then you have everybody knows about, I mean, we're introduced and you're told that tweet about us during our comments. So that's, I guess that's, we believe in this kind of sort of engagement except when we're involved in the victims of it, right? But, you know, if you, we're all familiar with these technology tools, but if you look at them together, it's a, it's a breathtaking opportunity. Even in the three years since I left New York City about how much has changed. If you put the cloud computing and ubiquity of mobile tools and data mining tools that break through kind of legacy systems and, and the, the mining and include and, and use of social media and the like, these tools together provide a very substantial chance for change if we can change the way government works to accommodate it. And just three quick stories and then Susan will, I'm sure correct me or elaborate as the case may be. So in a story that many of you may know, but I think sets up the, sets up the first point here, you know, I'm working in City Hall and we get news that a family died in a fire of a building that burned down that had been illegally converted, you know, a building that had too many families living in it. And you're knowing that the, the posts and the news were going to call, you know, I called the fire department and said, have you ever received a complaint on that building before? And they said, sure, we got a couple of complaints, but we have 30,000 complaints a year and we work our way through them. And I called the buildings department. I got almost exactly the same answer.  I was trying to set up a data analytics center and said the government was a little bit early and the tools were a little bit blunt and the expense was a little bit high. And in the back of the room was a person that Susan writes about in the book in particular, a guy named Mike Flowers. And so, you know, Flowers is one of these kind of, he looks harmless, but he's kind of must be a secret agent for someone.  And he goes in and over the next 60 days, he gets a 911 reports, 311 reports, code enforcement reports, mortgage foreclosure reports, tax delinquency reports and comes back and says, I've looked at the data and we can increase the predictability by a thousand times of which of those buildings are actually going to burn down. Right, so now we assemble a joint kind of team of fire and buildings that go to those 300 sites that are likely to burn down and they take care of the problem, right? They either move the families and require the sprinklers to be installed or whatever the case may be. So this is point one, which is right that a government that's entirely reactive and measures this performance on how well it responds to a problem as contrasted to how it uses predictive analytics to solve the problem misses the point of how we can be much more effective in kind of the new digital age, right? So that's point one. Point two, a related one, is that, well, we have to give employees discretion. So the story I can now tell, which I was not supposed to tell when I was in New York City. So I went around to do these neighborhood meetings like every week because I wasn't from New York City and I was trying to learn the city and I'd go out to the places in Queens or Bronx or whatever. And the small restaurant guys would complain about the enforcement mechanism of the city in terms of restaurants, right? And one of these guys was saying, I got this hugely expensive ticket because my cheese was too warm when the inspector came in. I got like three of these cheese complaints in like one week. So I went back and said to the health commissioner, I think that cheese had to be 42 degrees and was 45 and was pasteurized to a difference in the make sort of thing. And he goes, I said, maybe your rule's right, maybe it's wrong, but really do we have to give these infractions? And he said, look, we have a very honest set of restaurant inspectors. And if we start allowing them to use their discretion about when they should write the ticket and when they won't, then we're going to introduce these abuses and you'll regret it. It was a very professional response, right? So if you think about that, well, that makes a little bit of sense, but then it doesn't. Because now in the digital world, we know from the handheld tools, we know how long the inspector was in the restaurant. We know which restaurant he was in. We know how many minutes he was there. We know how many infractions he wrote. We know whether he's an outlier in too many fractions, what type of fractions, no infractions, what his productivity is, what the complaints are. We know everything about him or her as an inspector. So why don't we change this tension that says we have to choose between accountability and discretion, which is how we've operated for the last 100 years in cities, and say we can inject discretion on the part of the public workforce, make that work more satisfying, more responsive to the citizen, and hold the worker accountable at the same time. So point two is these systems, these wonderful data systems will not work in the absence of discretion, and we don't have to grant discretion absent accountability. A couple other really quick points, and then Susan tells better stories than I do anyway. So the other, the next problem we have is that government's vertical, right? There's a parks department, and a sanitation department, and a street department, and a planning department, right? But people do not live in vertical departments. They live in a neighborhood. And the big insights are across these activity levels, if you're in a neighborhood. And so if we're gonna solve problems, then we need to be able to mine data across those verticals. That's how we solve issues. So I spent a long time, actually about 20 years trying to work with child welfare. And the life of the average child welfare worker, the case worker in America, is one really tough life, right? She's not very well trained. She's not very well paid. She's put out into very difficult circumstances and asked to make an instantaneously decision, life or death, often, with either inadequate information or a lot of information buried in kind of huge folders that she can't get to during the interview anyway. So now we say, and this is the case in Indiana, which, thanks to Andy Casey, is kind of a leader in this area. So now she has a tablet here. She has a tablet. And they have the information in the tablet. But then you go, well, so having a lot of data in the tablet's insufficient. If we want her to exercise her discretion, why don't we help her exercise her discretion? So why don't we mine the data and suggest to her decision support systems that allow and improve her instinctive evaluations of that family, of that child? Was the kid in school the day before? How about the siblings? Were there other complaints? How about the family? And then you say, well, how about the manager who can then evaluate the case worker about how does she make decisions? Or how about the providers who we spend hundreds of millions and billions in the country every year trying to provide services to that family? Which providers are good with which types of kids? And now suddenly, across all of these verticals, we're able to provide decision support tools that will allow us to use our dollars more effectively and buried in kind of our book, I think, as a sub-theme, which is that Republicans and Democrats, liberals and conservatives, can disagree about how much money we should spend on a various activity. But our position in the book is we ought to at least spend it well. And so repurposing funds that are not used well because we now can make a difference is equally important. Next point. All of this is unimportant, however, if we don't get the definition of a public value right. And that is to say that the goal of our book is not to use data to come up with the definition of performance that says running faster in place is our definition of performance, right? Because across every level of government, we have officials thinking they're doing a good job. But all they're doing is producing more activities, not producing more outcomes or outputs. And the data allows us to distinguish the difference. Now, so Linda Gibbs, a good friend of many of ours who sat next to me in City Hall, before she was deputy mayor in the New York City of Social Services, was the head of homelessness. I tell this story quickly, but she went into her office one day when she was in the homeless and said, what's our definition of success? And the people around her said when we provide high quality housing for those who are homeless. And she said, no, no, no, that's the definition of failure, the definition of success is when we prevent homelessness, right? We have to provide high quality shelter when we aren't successful, but the definition is reducing homelessness, not providing more shelter for those who are homeless as a result of some other intervention which could take place. And then over the next four years, Linda built a massive effort to mine data in all those social service areas, put it together, reach learnings about what works and what doesn't work, and so the data movement will only produce quality in government if it produces value instead of more activities. And then last, and this is just obvious, and Susan's evaluated this nicely in Chicago, this takes leadership, every lawyer in city government or state government has a reason why their data, no matter how private it can be maintained and how anonymous it could be coded, that data can't be shared with this agency. Technically it's impossible, legally it's impossible. The agency CIOs say it can't be done, right? So everything happens inside these verticals and so if you really want to create a data-based enterprise, right? It uses predictive analytics, it uses social media, it takes a leader like Mayor Emanuel in Chicago and Mayor Bloomberg originally then in New York and now in Mayor de Blasio there as well but it takes a leader who says we're gonna integrate the data and the data mining and we're gonna make the agencies pay attention to performance and to performance is managed by data. So in conclusion, there's one problem with my thesis and it's that the definition of professionalism in government is that you become more professional as you become promoted inside your technical training area which means that you become technically proficient but narrower and narrower as you are promoted and then we introduce data analytics and predictive analytics and now the professionals are absolutely sure they know what's in everybody else's best interest, right? They really don't have to listen to anybody and we need to resist that and for those of us that have been either elected or appointed for a long period of time, the definition of civic participation up till now has been you decide what you wanna do as a professional bureaucrat in the positive sense. Then go to community meeting where you're yelled at for four hours and then you leave and do what you're gonna do anyway, right? That's the definition of community engagement. So Susan now will present you cases in better civic engagement and leadership. Susan Crawford. Thank you, that's great. Well, you can see why Stephen Goldsmith's really a mayor of mayors and is beloved by mayors across the country and gives lots of advice. So it's great to be at New America. I was in the lobby a little early and I heard the words corrosive and polarized twice. Just, and so, you know, we're talking there. They're almost all the time about federal policy and we have an obligation to make democracy work better and cities are the places where democracy is still working and can be made to be improved by the use of technology. So ultimately, this entire book is a kind of people magazine of heroes throughout city government who are doing their best against almost impossible odds to make life better for citizens to ensure that by doing small things, much better by delivering services more effectively, by giving lots of touch points with the government across the city, making the government more visible, you thicken the mesh of democratic engagement, which ultimately, I believe, is going to help us prove the case for democracy around the world. That authoritarian nations may be great at building high speed rail faster than we can and even putting fiber lines down all over the country but we've got the long, wrong advantage of engagement and autonomy and agency and all of these digital developments feed into that, that richness and the nonpartisan seeing 10 years ahead quality of New America is aimed at the right place when it's thinking about cities and their use of technology because it's hard to see that we're in a period of tumultuous transformative change when it comes to governance because so many people are doing it. There's a lot of rattling around. We've tried to capture that moment in the responsive city, a period of genuine interactivity and change. There's a confluence of events, some of which Steve has mentioned, three big things, more handsets in people's hands, more fiber with any luck, more wireless connectivity, so infrastructure getting better, cost of communication going down, that's the first big development. The second is that processing is getting a lot faster and our ability to understand data, to know what we know, which is really the goal of predictive analytics, to take historic information and use it to ascertain the best ways in which to focus city resources. That's just taking off now in cities. You know, Google and Facebook have been doing it for a while in Amazon and the rest of them, but cities are just learning how to do this and learning how to do it much less expensively than they had in the past, so that's the second big development. Using those analytics, we can piece together legacy systems, which used to be completely without interoperability, just sitting in silos, just dark. These analytics tools layer over formally hidden data and allow the city to know what it knows. And then the third big development is people. People who are finally comfortable with technology coming into city government, both at the leadership level, which is absolutely essential, but also serving in CIO roles, CTO roles, and generally working as a team across the country. I see tremendous collaboration among different cities when it comes to their use of technology. Another great advantage when it comes to the authoritarian nations in the world is that we do collaborate. We do have friends, and this is a great place for it. So those are the three big development. There are enormous obstacles. Steve's mentioned some of them. Having a clarity of purpose, the idea of what is the public good, we're trying to get at. We still have work to do to aim towards outcomes instead of outputs. The lack of an enterprise view for any city government is a real problem. Understanding that these turf battles, it's really tribal among agencies have to be leveled in order for work to get done, collaboration to be effective is important. And then there's tremendous inertia and upfront costs of investing in infrastructure and people are expensive. So the status quo often does well. So here are my three stories. Chicago is the center of a lot of these developments, the great city of big shoulders, led by a mayor who's fearless and put technology at the center of his administration, Mayor Emmanuel, he sees the city as a platform trying to use technology to improve the lives of citizens in many, many ways, real data leadership in many occasions, taking these transaction systems that are used to serve the purposes of government and layering over analytics. So for example, specific story, the city had to map data against a graphical interface, a map essentially of each block in order to get ready for enormous demonstrations that were coming to Chicago. They called that interface Windy Grid and they're able to see for any place in the city the history of interactions with that place. The 911 calls, the 311 calls, the problems with buildings, the problems with inspections. Suddenly the city can see itself in a way that was not possible before and Chicago is gonna take that same Windy Grid interface and make it available to many other cities on an open source spaces, making it vanishingly inexpensive for much smaller cities to see themselves, to visualize their wellbeing and where their services are being allocated. We don't make progress on anything until you see it. Also looking at a shared screen reduces emotional heat a lot of the time and keeps people imagining what's possible. So the Chicago Windy Grid interface and this idea of a smart data platform that will be available to many cities is tremendously exciting and we talk about the heroes who were responsible for that in the book. Chicago is also planning an entire new community on the grounds of the former U.S. Steel plant. So U.S. Steel huge, hundreds of acres just south of downtown Chicago, the plant hasn't been used for decades. It's just sitting there and they're going to redevelop it starting from the ground up with technology, with having the architects understand how traffic flows, environmental flows, air quality, noise, all of these things will be moving around the city and how people's lives could be richer using sensor data in this development is part of their urban planning from the beginning. This is really rare. People in China are accomplishing this but not so much in America. The idea of planning a city from the ground up using the capability of sensors to understand when buildings are crumbling, when air quality is lessening, when pollution is making people's lives less happy. I'm pretty excited about that. It's called the Lakesides Development and the University of Chicago, partnering with the architects and the city are all carrying this out together. Chicago is also a place of an ecosystem that MacArthur Foundation does a lot of funding there of civic technology, huge civic tacking community. They all talk to each other, a large nonprofit called the Smart Chicago Collaborative gets community members involved in beta testing technology, serves as a sort of interface between city hall and the rest of the city in a very active constructive engagement, the likes of which we haven't seen in any other city. So really take a look at Chicago for all of that. So Chicago is the first story. The second one is about community engagement and here I wanna tie together cities of different sizes. First June, small, tiny city in Spain, only 3,000 people. In June, every police officer has a Twitter handle on his uniform. In June, the mayor is constantly tweeting at the citizens and they're tweeting back about their reactions to city services. And with their Twitter handles, get this, citizens can make healthcare appointments, they can reserve meeting rooms in city hall for community meetings and then when they get there, they direct message the conference room door which politely swings open, right? Why not? Real engagement using a very lightweight social platform that we believe and MIT is gonna be studying this, but we think this is against thickening the civic mesh when there are more touch points for government, when there's more trust involved in those relationships, government can do big things. The things it needs to do to solve global warming, fix infrastructure, less inequality, all of these steps. So that's June. Boston, we profile in the book, has taken a similar step with its Citizens Connect application. The idea of using a handheld device to talk to the city about conditions on your street, street lights, other things going wrong and have a constant flow coming back from the city of how your process is moving, how your complaint is being addressed. Now this is all pretty primitive. Engagement should not be limited to complaining. There's clearly a bunch more planning that can be done collectively, but it's a start. And when people use Citizens Connect, they don't feel like they're complaining, they feel like they're helping. There's something about these very intimate handheld devices that gives people a sense of agency and involvement with government that is really rich. And in Boston, the mayor's leadership, Mayor Menino, for 20 years only cared about knowing his citizens and helping them. By the time he left office, he'd shaken hands with half of the people in Boston. He saw technology as a tool to thicken that engagement and that current mayor is continuing to carry that on. And then finally, 311 is a platform for engagement. We tell the story of New York City, which gets more 311 information than any other place in the world and has now built their 311 platform as a kind of skin on top of all of the databases run by the city. So that suddenly there's an opportunity for engagement across city understanding and visualization, again, of how neighborhoods are doing, how particular services are faring. All of this adds up to a visible virtual layer on top of existing invisible processes. So you see what government is doing to help you. You can complain or get involved when you're not happy with how things are going. And you own the city. The idea is we're all in this together and only then can democracy actually function better. So the book is tremendously optimistic. It's a very cheerful book. We're optimistic about the future of cities. We see cities as the places where things can actually get done. So if you're feeling stuck here in Washington, go work for your local city council or find a way into a city you care about because this idea of presence in each other's lives, engagement is now actually possible as the costs of computation and processing and visualization go way down. And as more and more of our students go into local government. So we look forward to talking about the book with you earlier. Thanks. Excellent. Excellent. Well, Holly, do you want to lead us off in our discussion? That was really excellent. And I'm excited. So we'll do a little bit of fireside chatting and then we're really excited to have a genuine discussion with all of you guys here. So I think it's a very interesting turn and I think that your point about, people are really watching what's happening in cities. And this is not necessarily a new idea. We have an old idea that cities can be a place for citizens to come together, but somehow it's forgotten about in this progressive reform, in the way we thought about progress in the 20th century. And I think what's happening right now is that these are examples that the whole international community can actually take note of. So there's sort of a boomerang effect happening whereby these cities and their sparks that are happening that you guys really document very well in this book, people are taking notice, whether it's the open government partnership or the recent report by Nesta and Bloomberg Full Anthropy is spotlighting this as a really a global phenomena. And so I think the big question for me, the big takeaway is how do we build the cities that we want given the infrastructure that we have? And so how do we take sort of big data and encapsulate small V democracy with bureaucrats who not all of them are sort of the spotlight people that you guys talk about? There's a lot of hardworking public servants in government and they may not want to be on track every day. If I were this new cheese inspector, I'm not sure I'd still want my job.  So that would be really not the sort of spirit of this. It's unfortunately a good question to start with. I don't think anybody's really doing this well, to tell you the truth. There are lots of interesting small examples like the Boston one that Susan mentioned. But the definition today I exaggerate here, so I'm not insulting mayors around the country, but the definition today of an engaged social media city is one that uses social media to tell their citizens how cool they are, right? So we're using massive amounts of, but it's outbound. So I think that the answer is we have to restructure government to listen differently. How does it curate the conversation? How does it mine the social media? We have some examples, you know, the grade DC dot gov effort of Mayor Gray where you tweet back or whatever you're experienced with a city servant and then he puts that into performance scorecard is a really good first step. What I think though, the real answer to your question is the reason nobody's done it is so that the excuse for lack of community engagement is the transaction costs of listening to a million people are really quite high if you're a public official. So now what we have to do is design a system where we can listen and secure information and curate that information and interpose it in the decision making process of which sidewalk, how is the park planned? And how is what rules get problemated? What problem do they solve? But not just fashion a process to communicate more effectively back to government but actually to socialize an answer in an iterative way around a problem. And very little of that is actually going on today. Yeah, I think that's fair. This is all so primitive. We suggest to the book that it will be possible for groups to have memory and presence and voice online in a way that's very difficult offline that your ability to just use a little tiny bit of your attention and participate in the community group online and for that group to persist and speak and work together is actually easier online but we're not seeing very much of that yet. We're waiting for that to happen. My own belief is when things are more visible or more used to a different language, almost of literacy of wielding images rather than text constantly, this will get easier to understand groups and for them to have a persistence online but we're not quite there yet. And maybe just another version of the question that's a little bit more explicitly about scale. It's striking in reading the book, the case studies of these heroic people who've managed against all odds to create and come up with really innovative solutions to problems. Then you think, okay, there are 294 cities, according to Wikipedia, with over 100,000 people in America, right? So there's 294 decent sized cities. There are 19,000 municipalities in just in the US, right? Multiply that around the world. So how do we take the best that's been learned from these people and start to think about how all the, instead of reinventing the wheel, how do we get that out to all the others? Maybe it's the windy grid example. I'm just curious what you think we do about that. Well, our thought was if massive numbers of people buy our book, it would change. There you go. Probably would be a slightly more substantive. Well said. Well said. Right. Okay, done. So real quickly, one of the goals of our work, right, is to catalyze a movement around these tools. So that's one. Two is we hope people are inspired by those heroes in the book. But third, I think is the following, right? So which of these technology tools can be provided as a shared service or in an open-gov platform so that with cloud computing and the like, we can lower the cost of entry for the average city? So that's one and that's promising. The part that's not quite so promising, and Susan's a little more expert at this than I am, is the insufficient numbers of individuals with talent. The average city, I mean if you take out Dallas, Houston, Los Angeles, Chicago, and New York, the average city is not gonna be able to afford a data scientist, right? And so I turn over to Susan to solution to that problem. Well, luckily we have two institutions with challenged business models, government and universities. So I think there is a tremendous role for universities to play in training these cross-trained people who understand both data and policy and law. We're not there. I don't know of a single policy school that's doing this well. Harvard Law School is interested in working on this, but it's very nascent. And then those students could have very brief fellowships inside local government and then come back out, kind of a, be a public servant for America program. And we also need to be able to secund people from private industry very easily into local government. All these things right now are too hard, but we need to lower the barriers. And what I'm really excited about is that I have so many students who want to get involved. Now that they understand there's a chance to have agency to get things done, not to be stuck with bills that never pass and policy questions that never change, but actually to work in a city and see the effect of their effort. How many people did you speak to at Code for America? Yeah, there were 900 at Code for America. It was a tremendous meeting. Thank you for reminding me of that. 120 cities, people, mostly in their 20s and 30s from all over America, who are just filled with interest in serving and are tremendously generous. And America at its best is not cynical. And this group, as a movement, is not cynical. They see the opportunity for growth and change. And if we get rid of some of these barriers, we'll really change small D democracy. I was just gonna say, I was struck in the book also, so you tell the story about the responsive employee, right? And the lead-off story is about Lolita Jackson in New York City, who happened to be a classmate of mine in high school, kind of unusual. And I was so excited and surprised and anybody who knew Lolita Jackson in high school would not be surprised that she was taking a leadership role in fixing the second F when you subway construction problem in New York at all. But the question of like, yes, so riffing on that, how do you not just attract, how do we not just get more Lolita Jackson's, but also it was a story of empowerment that she was given, not just that she was there, but she was also given the tools to go out and fix things. Right, well, I mean, I think almost the irony of this is that we've created these systems with, so I teach at the Kennedy School of Government, and so you think a vast majority of my students would want to go into government, but they're not sure they want to go into government because they want to accomplish something for their community. Now it's not, right in there, so they're saying if I go into government, I can't use my discretion, I don't mind having big student debt and no income, but the trade-off is I want to actually accomplish something for my city or state or federal government, right? And then I get like, you can't do this. So whether we are having shorter terms, like Susan said, kind of recruiting and passing through, and my view is we got to do more broadband, we got to give people more discretion, we got to have more tools to solve problems, and they will then, I mean, everybody from the trash collector to the Lolita Jackson's who's going to solve the problems of congestion around the construction site and second avenue subway will solve a problem if they're given the authority to solve a problem. And one of the interesting things I think in the book is that this app that Susan mentioned in Boston, now not only does the citizen get to complain, if you will, to City Hall about their pothole, but the worker sees the picture, sees the site, and soon will be able to communicate directly back to the constituents. So I'm thinking and advocating that more retail government, more direct experiences where people can solve the problem of another will build up the confidence of the beleaguered public servant and the confidence of the citizen and their responsiveness. That's right, because the ultimate problem is trust. We don't, efficiency is not the American problem. It really is participation, involvement, a sense of trust and all these interactions and the sense of the worker of being seen and understood and having a sense of agency, discretion, wide banded job responsibilities gives everybody an increased sense of trust for the process, I think. Yeah, I think that's right. And I think the point is very well taken that there are a lot of people, especially young people who wanna get involved and be public servants, but the current pipeline is very difficult whereas consultancies come in line up and have very easy ways to get in. It's much more complicated on the government sector. So I think I have two related questions. The first one is how do we ensure that we're building long-term capacity including with the people, as I aforementioned who have long-term public servants who might not already have those skills if we have sort of these bright, shiny, tooled up students entering public service, especially if they're gonna be rotating in and out which I think is increasingly where we're seeing sort of the job force going, what does that do for long-standing notions of the public sector? And the second question is sort of related to your, I think this is totally right. I think we need to empower citizens as problem solvers in governance. And so the question is how can we really do that and where do you guys see the potential for using these digital tools for online and offline opportunities for governance? You do question one, I'll do question two. I want you to start talking. I'll do question two because I've forgotten question one. So as a, this is like in the dark ages, but as a very young mayor, I went out to cut a ribbon for a new park that had actually been totally done by my predecessor but I was now elected so I got to cut the ribbon. I went out there to receive a claim, right? And people were really upset and they were mad because the park had built the basketball court on the wrong side next to where older citizens had their homes and tried to sleep and they didn't want to hear basketball at midnight. And there was nobody on the other side of the park so the park was just totally misplanned. So then I got the message, so then I raised some money to rebuild our parks and I went out to the first meeting and said we're gonna rebuild your park, what do you want to do? And they go, we're not park planners, what do you mean, what do we want to do, right? So the reason I answer the question that way is because we have to create intentional methods so that the professionals in government are informing the community about their options but listening to them in terms of their priorities. So in New York City, for example, with some funding from Knight and Case, let me make this short story quick, we had a combined overflow problem, we had to solve the problem, we came up with some money and one of the answers was, why don't we find the wettest parts of the city, this is a negotiation with EPA and create places where we have permeable surfaces, tree boxes and the like. So the professionals in the engineering department and the water department said, great, we'll just pick the places and we'll tell the community this is what we're gonna do. And I said, oh wait a minute, why don't we just put the places up on a map and ask the community to participate in determining what they want to do, where do they want swales, where they want retention ponds, where they want trees, where they like to help plant the trees. And so that led to an iterative socialized process that may be partially in answer to your question. Now the first question, which is basically about capacity. So Boston City Hall, anybody been there totally brutal building, right? Really anti-collaboration, made of cements, you can't move around there, who knows what's going on in there. And when Bill Oates, the CIO, now the CIO of Chief Information Officer of the state, when he arrived in Boston City Hall, fresh from private industry, the first cabinet level chief information officer in the history of Boston, he heard the sound of typewriters, 2006, right? This is not very long ago. And so you needed to make a basic investment in infrastructure, in getting some servers in there. And the city needed to make some basic investments in people too. Now this is, we have a problem in government of always taking the urgent over the important. And because Bill had some experience and knew what he was doing, he said to the, and he had the mayor's trust and the mayor had a long history with Boston. He was able to tell the mayor, we need to make these investments now, so that you can be better engaged with people in the future. And then you find, when you invest in the people in city hall, give them some training, let them go out for a while, just open the windows, they will be delighted to work with these new tools. They want to be treated with respect and dignity and learn new stuff. And they're, you can overlay what Boston had was, had something called the office of new urban mechanics in the mayor's office. Basically those guys job, it's just a nudge innovative practices and kind of cheer people up and say, consider collaborating. You know, that's what they're doing. And those investments in people, in new skills, in infrastructure, pay off hugely over the long run. And we think we're at a point now where you don't have to persuade a mayor that you need to do it. But in a time of scarce resources, people will say, oh no, I need to crouch down and just fix the most urgent problem and not look out ahead. And luckily the things we're talking about are getting cheaper and cheaper. So the upfront fear is lessening. So all that will increase capacity enormously. You touched on this a little on the book, but how do we start to also think about dealing with the potential negative consequences of some of these technical tools, right? So you talk about, for example, either in the Indiana case worker example or thinking about new sensing technologies that raise these interesting privacy and security problems. How do we think about those? Or how do we think about issues around social justice and equity, potential impact of some of these tools on vulnerable communities, people who don't have access to technology? What's the, I mean, it's not all unvarnished easy good, right? What's the right way to think about that? Well, the right way to think about is that the benefits will, in the long term, exceed the burdens, but right now our conversation is so primitive about these policy questions that we're nowhere near coming up the balance approaches we needed. So I'm lucky to be out talking about the responsive city at the same time that I'm teaching the law of surveillance. So there's this beautiful balance here. Big data, yes, but then what? And it's so apparent to me that we have two polls in America that there are people who are very tech happy and haven't thought about the policy consequences at all. And then there are people who are so worried that they want to lock everything down and never let any data be shared. And we know that neither one of these is the right answer. So we suggest in the book, and this is where the next generation comes in because my generation has not succeeded in doing this, that we need to have people, again, cross-trained in these roles who have thought through very seriously the consequences of data sharing that business rules are in place that sharply limit who has access to what for what purpose, that forensic capabilities are very advanced so that we can know who's having access to which data for what purpose, how it's being used. And if data is abused, that the consequences are severe. It becomes a recognized norm that you're not supposed to mess with data, you're not supposed to have access to. All of that is yet to be established. So we're definitely raising a flag that policy is integral to this conversation. The same thing with underprivileged communities. A lot of my work has been focused on making sure that everybody has fiber across the country. In order to have a truly responsive city, you'd need to have that infrastructure in place just like clean water, just like electricity to all parts of your city, not just the parts that are paying the most in rent. Just one minute on Susan's last sentence. So there's obviously a concern about the digital divide, but that concern presupposes that the current phone-based system, 20 million calls a year to New York 3-1-1, is a representative sampling of the community, right? And my experience is that people of power have more voice than people without, and they find their way today to influence City Hall. So we have thought about this, Susan's obviously an expert on broadbanding, but I've been thinking about it in terms of, smartphones, right, 70% adoption rates. And if government uses the power of the smartphone and is willing to listen, it can give voice to those who are underserved in communities, right? Because imagine if the person who received the workforce training voucher was intentionally asked to tweet back or SMS back, was this training session worthwhile? Was the instructor any good? Did it help you get a job, right? So why don't we ask people who we serve for feedback? You know, I had a chance a couple of years ago to talk to like 3,000 federal government procurement acquisition officers. And I said, have any of you ever thought about asking the people for whom you're purchasing this equipment, whether it works for them? Right, this is a different sense of power, but I mean, this is the average employee. I said, well, no, we don't, we don't, whether the- We don't wanna know. We don't know whether the garbage truck has a step too high or the- So I think if we look at, and we've been talking to the mayor of St. Paul about trying to use this for after-school programs to collect information. So I hope that we'll have virtual marches on City Hall that use these tools to give more balance. And in the end, you know, when New York and Chicago will finish rebuilding the 311 systems, as Susan explained, I think that folks who, particularly those who don't have English as a first language, will be more comfortable in responding in their native tongues and their own expressions about their needs and interests. Virtual marches on City Hall, I like it. Maybe one, do you wanna do one more question? And then we should- And then maybe we can go to the audience. So- Absolutely. We've gotten like that. I loved in the end when you talked about the need for iterative learning. So we really don't make those mistakes that have been made in the past. I would love to just get, you know, very quickly how you best think we could do this iterative learning, especially since we're in this primitive phase. Well, I think what's going on quite naturally right now is a lot of shared conversations across many City Halls in the country. What we need to do is make those more obvious, make them more interesting, make them more transparent and available to every city, not just the huge ones. This is actually happening organically as mayors all seem to talk to each other and they get jealous. What's the other guy got that I don't have? And want to figure out how to increase their own ability to act through the use of technology. So iterative just means it should be possible to fail every once in a while in governments. That's part of broadening discretion. And to have that understood as part of the innovative process, it should be possible to have merit in innovative thinking recognized as part of your pay package. There should be lots of ways to incentivize innovative work inside city and government agencies. It's just hard right now. We've got a microphone and just for folks on the webcast, it's great if you could use it. Question up here? Mike McDonald, I've been leading some salons on from the National Sustainable Security Infrastructure Initiative on governance in times of transition, looking at this time of a lot of social media. And the book's really interesting in the way you present it, I think it's very useful. But there is another phenomenon that's happening that really hasn't been spoken about so much here. When we start looking at crises, and there's a lot of big crises that are emerging now, government tends to not be very good. I mean, look at New York during Sandy. The monies that went out to the neighborhoods and these are neighborhoods of 100 to 200,000 people, Rockways and the Staten Island and so on, that got impacted. It was the horizontal networks, not the city that functioned. So for three weeks, there was nothing in the Rockways except for the churches and Occupy Sandy. And Occupy Sandy had 80,000 volunteers versus the Red Cross is 12,000. So there is another type of governance that's emerging that may not need city hall. And as we create distributed collectively intelligent grids with new energy systems that are now controlled by the neighborhood, not by these large centralized systems, things may be changing. Well, this is a wonderful observation. I'm embarrassed that we didn't emphasize it more in our presentation. So I don't know if you could all hear the question, but so one way that we have thought about a version of what you just mentioned is how do these tools enhance the co-production of public services? So we have a model that suggests that public goods should be produced virtually exclusively by a governmental organization as contrasted to what's the role of the network and the network can be an ad hoc network and it can be an intentional network and at the hub of the network is information and your point is just a wonderful one and it's absolutely accurate in the worst time of aftermath after Sandy. The resiliency effort emerged from a lot of the NGOs that were connected in these free form data efforts, but they had to have some interaction with city and federal and state officials as well. So that making my answer longer than your question. I think your question is well taken. I think the opportunities here are dramatic to produce civic infrastructure through redeveloping and reemerging the civic side of the co-production, but there is a role for government that the role of government will be to use the networks more effectively, I think. That was a great point. Thank you very much for your presentation. My name's David Wetzten. I have two basic questions and I think you all have addressed them in part, but I'd like to push a little further. Question number one, if we don't have a thoroughgoing national urban policy, where does all together wonderful things you describe fit in? I'm not saying that I'm proposing a top-down imposition of this and then more important to me, question number two, where is the formative process of civil engagement for people who are younger than undergrads? It seems like there's a, to use a broad brush, there's a failure to foster that. I'm an older person. I can remember the excitement. I'm from New York originally, by the way, of state politics and local politics, but I don't see comparable analogs going on right now with that. I don't have an answer. Do you have one? Well, this is akin to what's happening with computer science education across the country, that it should be part of literacy and yet almost no schools either offer it or they don't allow you to take it for graduation. It's a huge problem for the entire country and unless we fix that, we'll have a whole generation of kids who are not really able to function the 21st century. Same thing with civics, using technology. That should be part of your literacy as well, the excitement of understanding what your local government does, meeting people virtually or now with fiber, you can actually be with them without having to go downtown. There is an enormous opportunity for much younger people to have this engagement using these tools.  We need to connect these worlds. As to your first question, national urban policy, I'm all for national policy and I think the frameworks are extremely important but often national policy learns from cities. So what we're suggesting is that the cities are able to take actions which will then tip into national policy. It might not happen the other direction. Yeah, no, I agree. I was basically, we run these programs for mayors and chiefs of staff and I've been doing it for a long time and there's virtually no mayor in the room in the top 30 cities anymore who looks for national solutions or national policy. I'm not saying whether it's good or bad. I mean, it's basically saying this, you used to have these meetings and half the meeting would be about, well, what's HUD or the White House gonna do or what's transportation. Now it's not even a peep. So I mean, that's not to suggest that's white but if you look at these tools as self-help tools whether it's self-help for a community or self-help for a city, then to some extent they are more emergent because of this. Hi, my name's Isaac Rowland and I'm with an organization called Public Agenda. And I do a lot of work as a practitioner helping policymakers, institutional leaders do engagement. I see a lot more of what you described earlier the town hall where folks are yelling. I'm curious, just a really practical question. What are the arguments or cases that you've been able to make that have been most compelling to policymakers at a metropolitan level for pursuing meaningful engagement activities? Actually, I was ready to answer a question which I thought you were gonna ask, but you didn't. You can answer that as well. Which is how do you make the case for data and the way you make the case for data is you make the case for solving a problem that data is necessary for. You make the case for solving the problem. I don't think this, one of the earliest questions was a similar one. I don't think there's been a lot of effective change inside city or state government in the way it listens. And it's getting a little better at the top. I mean, this thing with Mayor Gray, it's not a huge thing, but the fact that he's encouraging people to score him, right? This is not a typical mayoral activity, right?  Oh yeah, Joe, I'm calling Joe. So I think the grading issue is helpful. And secondly, I do think the extent to which we move this to engagement brings trust, trust brings kind of a core glue of civic society that will produce more confidence in the future. And after all, in the end, the role of the mayor is to convince the people in his or her community that tomorrow will be better than yesterday and that can't be just done with government. So I think we can make the case that this is more than just noise and more than just distraction, but it hasn't been effectively, I think, mine to date, really. Let's just say, Holly, you've done a lot of work on participatory budgeting. It's mentioned a little bit in here. I mean, that's sort of relevant to the, I mean, I don't know if you feel like that helps make the case. Yeah, I mean, I think participatory budgeting, which you guys cite, is one compelling example for where we're sort of engaging citizens with real decision-making power on the local level. And I think the pitch to municipal leaders is, this is a way to engender trust in your constituency and citizens are craving it and they're responding really favorably. So I think they're sort of the two sides of the coin. You can get to better decision-making outcomes and you can also increase legitimacy in those decisions. Should we go way back? Lee Drepman, I'm a fellow here. So civic skills aren't distributed equally. Some people, probably a lot of people in this room have a lot of civic skills, a lot of know-how, a lot of tech know-how. A lot of people who are not here don't have civic skills, they're busy. So when you say the response of city, how do you make it responsive to everybody and not just those who have civic skills and technological know-how? Well, the great thing about interactive media is that you don't have to spend all evening going to a community meeting to register your disapproval or your interest or approval in a planning process. There you go. My visualization of this, how I imagine this, is that cities will have screens everywhere and they're a very accessible layer of data for everything. People are interested in their neighborhoods even if they're very busy and under stress in what's gonna happen to that sidewalk, what's gonna happen to the park, how's the city actually functioning and being able to, with a very light touch, using a device or a screen with which you feel kind of intimately close to feel a sense of engagement and agency, seems to me to open up new possibilities where engagement in the past would have been much more difficult. Now use a shard of your attention, a moment to indicate that's all you need to do and you can reach many more people that way than we could in the past. And two, I think maybe two additional points. One is that, and Susan referenced this with a Smart Chicago example, that effective intermediary, right, so everybody's not gonna equally digitally literate, but if there are intermediaries, I mean non-profits, community groups and others that can, through the use of these devices, kind of elevate the position of those who are members of their organization or in their community, that provides a solution. A second answer to your question is to the extent to which we are visualizing situations in a community. So let's assume that the law professor has solved the privacy issues and that the censored data really will sense what the air quality is in a particular neighborhood, right? Or any other important issues that may be, that may show that a community is less healthy than another community. Or maybe it's the geo-tagging of public services which will show which city neighborhoods are more neglected in response rates or investment. So we can begin to visualize the disequilibriums, right, and the inequities in the system. We have censored data which will, even if folks are not literate in a particular community, can look at congestion patterns and look at air quality patterns. Now, we gotta get government to listen, like the last couple of questions, but I think that we can use digital tools to mitigate the imbalance even if everybody in a community is not equally literate. So it's just another way to think about the answer. Some questions up here. Hi, my name is Tracy Hughes. I'm the director of the Office of Open Government for the district. I'm the first person to sit in the seat, and I'm finding a bit challenging because we're facing not only a mindset in the change of culture, but getting the city to really wrap its head around new innovative tools. And I'm very encouraged to see that we're making progress. But I'd like to ask you, in your research and in writing this book, have you came across any examples of a movement beyond tools to promote civic engagement to the use of tools to collaborate and cross-connect with our communities? It's a wonderful question. I hope you liked our story about DC government was in order to ingratiate ourselves with you. You didn't even know you were here. Right, that's true. Okay, so very quickly, I'll tell you a story that I was involved with. So to some extent, a little less now, but still somewhat. The open data movement is a check the box movement. My data's open. I don't care if it's inscrutable, you can't read it, it's really ugly, it's in all PDF files, it's just my data's open. So thanks, we're gonna move on to the next problem. So in open data movement, and this goes with the question about kind of the co-production of public services. So how does open data create answers? So, and I think maybe this illustration will help. So 311 reported to me when I was deputy mayor more or less, right? And we had these 20 million calls and we had open data, which means that people could kind of barely maybe figure out what was going on if they looked at our data. Then the city developed better visualization tools and then we got a couple recent, I think, Colombian NYU grads. And we had better software and we, and I told the agencies these folks are gonna go out and train every community board president about how to use the tools to find out when the reports came in, how fast were they resolved, what was happening in a particular neighborhood. They could sort our data on their own desktops and figure out what's happening in their community. And so the agency said, transportation, et cetera, look, we have enough people complaining to us today, you're not gonna really hire people to go out and tell them how to complain to us more often, right? So, but that's not what happened. They already knew how to complain. What they didn't see, they didn't have enough access to the open data to be able to find the solution, right? And you know, I remember one, and I've sitting there and she said, well, this is really interesting because I can tell you why you've had so many pedestrians hit or almost hit at this intersection because the walkway signs are mistimed and the senior citizens homes on one side of the street and the pharmacies on the other. This is not a normal intersection, right? And so fix it, right? So I think it takes, until it becomes part of the civic culture, it's gonna take an active NGO or an activist government to go out and show people how to use the tools to engage in a better way for problem solving. I think it's, we have a big app world that, you know, where people are developing apps, but many of the apps are designed to help people rectify problems that government won't do as contrasted to this kind of interactive area. Another example that may be useful to you is in Boston, there's actually a building set aside for this purpose. The district building is a place where civil servants come and talk to neighborhood groups about how to use tools to build the capacity of the neighborhood to work together. I help explain where the open data is and how they can use it. Very intentional set of programs and having a space for that makes a big difference. And it's in a district that the city is very interested in making into an innovation area, but it's specifically about civic stuff. Same thing, there's a story in the book about in the slums of Rio, a place that kids are building called the Digital Aguara, which will be fed by gigabit fiber, have a place where people can be looking at city data, but also participating in planning. Places still matter in the virtual world and having a kind of storefront capacity for the district might be an interesting way to break down some barriers in a pretty lightweight engagement for collaboration. Got a couple of, we probably have time for just a couple more questions. So, Faith, I'll leave it to you. Milton Kotler, I'm an old timer and wrote a book called Neighborhood Government back in 69 and helped drop the legislation DC for the advisory neighborhood commissions. Now, it's been 30 years. We have in our neighborhood and many neighborhoods in the city terrific intra-neighborhood networks and data. We have active list serves. We have active community meetings. And I wouldn't eschew attendance of a community meeting, people enjoy it. And it's a duty. It's a democratic duty to participate. It's not something to be eliminated by digitalization, but enhanced. So our neighborhood is very digitalized. We know more about our neighbor than the city of Washington, the government here knows about our neighbor. The question is power. And how can you talk about all this without addressing the question of how do you empower the ANCs to make more decisions on zoning, on amenities, on other prudential matters and education in their neighborhoods? It's a question of 30 years have passed with the ANCs and we still don't even have effective great weight, not to mention a decision deliberation. So I'd like you to address that. Parrot House. I think this, in graduation on your work and I've tried a little bit of municipal federalism over my time and found that the people are for it. The city consulers generally aren't. So I think this goes back to maybe a place where I began, which is the city has to change the way it operates in order for the neighborhood organization to have more power and be effective. For example, how much of the street budget or the sidewalk budget or the parks budget or real decisions over resources are really affected by the priority set in that process. And to the extent that government grants real authority to that process, you know, not over every single dollar but over the dollars are specific to that community, then that process will be more effective. And then as you know from your work and what we try to do is to the extent that the neighborhood organization, the community board or whatever the case may be, can be seen by the community as being effective, then it gets more credibility and it has more of its own kind of civic development. So I think it will work, but only work after government begins to make priority decisions, resource allocations that are consistent with the information they're receiving. Okay, probably we're almost at the end maybe a lightning round of the last two questions. Maybe we can take them together and then we can, yeah. This may not be so much a technology question, but since you're both from Harvard, I was curious ever since Henry Gates was arrested for entering his house, there was discussion about how a dominant institution like Harvard or MIT and Cambridge, the city folks would share what administrative power or something. I'm wondering if you could describe how if things have changed since then whether any of your ideas have, we're applying secondly since you're a mayor of Indianapolis, do you have any ideas in how the city of St. Louis and neighboring Ferguson could improve relations that are given your ideas or just in general? Well, we'll take the second question first and hope it's easier. Hi, this is the, I'm a grassler in Virginia Tech. So a veteran reporter for Baltimore's son, David Simon, I think he had a talk at Berkeley a while ago, and. Can you speak up a little bit? And the case was made there that as soon as you introduce in any department or even in the private sector a metric for evaluation of performance, you would have people, what he referred to as doing what's called juicing the stats essentially. This idea that you would see any metric of performance evaluation as you would devise mechanisms to basically evade that metric. And so the iterative process that you mentioned towards the end, the iterative learning process, how would that account for this sort of systemic corruption? Or is that simply? So let me put these two questions together both because we're out of time in because they're both hopelessly complicated. So we have these stat programs in government to measure things and whatever you measure, you're gonna get more of. And it's generally not really what you meant. I have like 50 examples of performance management where I sit on the performance and oh my God, that's not really what I meant, right? Like we're gonna measure how many potholes you fill.  You know what I'm saying? I mean, it's just human nature to game the system in kind of semi-legitimate ways. So if you said, I'm gonna measure, if you said 10 years ago, I'm gonna measure police officers on performance, you're gonna measure them on arrests, right? Good arrests, bad arrests, arrests with citizen complaints, arrests without citizen complaints. So I think what we see here, just to kind of bring this all together, is that is the power of the new data systems will allow you to look at outliers and look at data in all sorts of varieties of ways, which would say to your question about kind of what's the relationship between Ferguson and St. Louis and the folks in one community and the folks in another that you both have kind of a, the mining of the community social media, but you also have the ability to look at the data and keep iterating the performance measures. So surely the answer to your question, I don't even mean it this way, is not that we won't measure. The answer to your question is we'll continue every year to iterate the measurements so that we now look at police arrest divided by police, divided by convictions, divided by citizen complaints that were, you know, whatever. And we begin to get a, you know, and is the, is this officer making arrests disproportionately on ethnicity compared to other officers? So all sorts of ways to look at outliers. Then we have to change management to use it. So I think we're in a promising area, but barely scratch the surface. And we call the book deliberately the responsive city because cities are for people. And the idea is that we're not going to take a very completely mechanistic approach to city life, that this will be a back and forth interactive learning experience for everybody. So I hope we never lose sight of that. And I hope that you'll get a copy of the responsive city because it's right outside. Thank you. Well, that's a good final word. If you want to get excited about cities as the vanguard of technical information, read the book. Please join me in thanking our guests. Thank you. Thank you very much. Thank you very much. Nice to be able to have this. Thank you.