Welcome, everybody, to this talk. My name is Chelsea Barbis. I am from the Media Lab, which is one half of the equation for this AI Ethics and Governance Initiative that we are jointly pursuing with the Harvard-Burkman crew. And I am very happy today to have the opportunity to introduce Julia. Julia is coming to present some of her undergraduate research work, actually, which is phenomenal that she's able to do work like this. So recently graduated last year from Dartmouth College and is now working as a software engineer in Silicon Valley. Her background is both in computer science and women's sexuality and gender studies. Yeah, yeah. So kick ass, right? And so I think her work really kind of embodies the power of being able to combine disciplines like that and is helping to fuel a lot of the underlying assumptions around the ethics of using actuarial tools to improve decision making by probing at really some of the most fundamental questions of, do these predictions actually help us make better, more accurate decisions moving forward? So super excited to hear your talk. And then we'll have Q&A afterwards, just some points of housekeeping. So we are recording this. It's also being webcast live. So if you're watching online, you can participate by asking questions at the hashtag BKCHarvard. And then we're going to wait to have questions until the end. And then when we do, we're going to pass around the mic. So please speak into the mic so everybody who's watching via the recording can hear. OK, thanks, Julia. Thank you. Hi, is this, we're good on mic? All right. Thank you so much for having me. I'm going to be speaking about my research on the accuracy, fairness, and limits of predicting recidivism. With the rise of big data and the prevalence of technology in everything we do, we become very frequent subjects of algorithms. Tons of data is collected on all of us on a daily basis. And a lot of our daily experiences might be determined by algorithmic systems. A couple of examples. Google has algorithms to determine what search results to give you, which ads to show you. Spotify has personalized song recommendations. Amazon has algorithms to determine which products to show which user. And even banks and online lenders like Zest Finance use algorithms to determine who to give loans to. Recently, algorithms have crept their way into the criminal justice system. For example, the police force in Atlanta uses a predictive software called PredPole to predict where in the city a crime is most likely to occur at each hour of the day. In Chicago, the police force there actually has technology to make a list of the top 400 riskiest individuals in the city ranked by their likelihood of committing a homicide. Tools known as risk assessment instruments take into account a defendant's criminal history and demographic information to make predictions about how risky a defendant is. Recidivism prediction instruments are a type of risk prediction. And they are meant to predict recidivism, which is when a criminal defendant reoffends at some point in the future. The Compass Recidivism Risk Scale is a recidivism prediction instrument that is built by the for-profit company called Equivant, which was recently North Point, has been rebranded to Equivant. Compass is used in courts in many states in this country to predict recidivism. Specifically, the tool was built to predict a new misdemeanor or felony within two years of when the score is calculated. Compass is used in pretrial and sentencing and parole decisions. And since it was built in 1998, it has been used to assess over one million criminal defendants. In May of 2016, which is around little less than two years ago, we came across this study by ProPublica on the fairness of Compass. What ProPublica found was that although Compass has pretty much the same accuracy for white and black defendants, it makes very different types of mistakes on these two groups. It found that if you're black, you are almost twice as likely than a white defendant to be classified as high risk and then not re-offend. And the opposite is also true. White defendants were almost twice as likely to be classified as low risk and then actually go on to re-offend. So ProPublica released the analysis saying that Compass was biased towards black defendants because the tool was under predicting recidivism for white defendants and over predicting recidivism for black defendants. Tools like Compass are introduced in criminal justice because we assume that they're gonna be more accurate and more objective than the people who are tasked with making these types of predictions. And people hear words like big data and machine learning and they sound really impressive and people are usually pretty quick to assume that tools that use these methods are accurate and objective simply because they were built on a lot of data. And we started doing our own research into Compass and recidivism prediction algorithms in general after this ProPublica piece came out and noticed that there was this assumption underlying the whole conversation about algorithms in general which is that algorithmic prediction is inherently superior to human prediction just because it's built on data. But we actually couldn't find any research that proved that algorithmic recidivism prediction was definitively superior to human recidivism prediction. Compass's accuracy is around 65% which seems pretty low and this tool has very serious consequences. The decisions that it helps makes have very severe implications on a defendant's future on their life. So we wanted to confirm is Compass outperforming human predictions? For this study we had a data set of over 7,000 criminal defendants from Broward County, Florida. This is the same data set that ProPublica used in their study. This data set has the defendants' demographic information, their criminal history, the crime they most recently had committed and the Compass recidivism risk score that they were given at their pretrial assessment. It also had whether or not they actually recidivated over a two year period after they were assessed by Compass and this two year period did not include any time that that person may have spent incarcerated. To assess the human performance of recidivism we used Amazon Mechanical Turk which is an online crowdsourcing marketplace where people are paid to perform short tasks. The short task for our study was that participants would read a small paragraph about a criminal defendant and then were asked to predict do you think this person will commit a new crime within the next two years? These are the seven variables that were in each of those paragraphs about a criminal defendant. They had the defendant's gender, their age, the crime that they committed and then at the bottom kind of a small description of what that crime was. Whether or not the crime was classified as a misdemeanor or a felony, how many juvenile misdemeanors and felonies were on their record and then the number of prior crimes on their whole record. You should note that the participants did not see a defendant's race on making these predictions. Our study, or for the study we randomly selected a subset of 1,000 defendants from that larger data set of 7,000 and we picked this subset so that it was a representative sample of, sorry, so that Compass's performance on that subset was representative of how Compass performs on the larger data set of over 7,000 defendants. Our study had 400 participants and the participants were each assigned to see a random block of 50 defendants. They were paid $1 to complete the task and we wanted to incentivize them to pay attention and to do their best. People who were doing tasks on Mechanical Turk make more money, the more tasks they complete so we wanted to incentivize them to slow down, take their time, try their best and so we offered them a $5 bonus if their accuracy was greater than 65%. It's also important to remember that these people who were in our study were people getting paid to answer an online survey and we assume have little to no experience in the criminal justice system. Here are the results from this study. Here's the distribution of 400 participant accuracies. These participants achieved an average accuracy of 62.1% with a median accuracy of 64%. And statistically this was just barely less accurate than Compass's accuracy on this data set. So these individuals responding to the survey they did pretty well compared to Compass but statistically were just barely less accurate than Compass. There's a concept called the wisdom of a crowd which is that the judgment of a crowd of individuals can actually outperform the individual judgment of those people if those individual people aren't experts in what they're judging. And we wanted to know if there was any wisdom in the crowd of people that we had assembled for this study. For each, oh no, that looks awful. That's such a bummer. Okay, shoot. Okay, well this is a better graphic. I'm sorry, I don't know what happened with that. Each of the 1,000 defendants was seen by a small crowd of 20 participants. Each of those participants made a yes or no prediction on that defendant. Yes, this person will recidivate, no, this person will not recidivate. So if over 50% of the crowd predicted that someone would recidivate, we considered that a crowd prediction that this person would commit a new crime. And then if 50% or more predicted no that that was considered a crowd prediction that that individual would not recidivate. So by pooling together this crowd wisdom, the crowd was able to achieve an accuracy of 67%. And this is statistically indistinguishable from Compass. So this is pretty crazy that by just pooling together 20 responses from people who responded to this online survey, we were able to achieve the same accuracy that Compass achieves. There are a couple different ways that we can compare the performance of the human crowd and Compass. First, like I said, in terms of accuracy, performance was the same. We can also look at the AUCROC value, which you might be familiar with, is a pretty standard way to measure predictive validity in this type of prediction. And in terms of this AUCROC value, the crowd and Compass had the exact same performance. We can also assess accuracy in terms of sensitivity and bias, which is a signal detection measurement. And in terms of both sensitivity and bias, the performance was the same. We also compared how this crowd performance compared to Compass in terms of racial fairness. Before I go into fairness, I wanna define a couple key things you need to know to understand how we looked at fairness. So in this sort of prediction scheme, there are two things that are predicted. Either someone will recidivate or they will not recidivate. And then what actually happens? Either someone does go on to commit a new crime or they do not. So if somebody is predicted to recidivate and they actually go on to do so, this is called a true positive. If someone is predicted to not recidivate and then they don't recidivate, this is called a true negative. What's important to look at is when the predictions are wrong. There are two different types of errors that happen in this case. The first is called a true, sorry, a false positive. False positive error in this is when someone is predicted to recidivate but they do not. And in terms of recidivism prediction, this is a detrimental error from the point of view of a criminal defendant. A defendant who is given a false positive error was scored to be high risk even though they actually did not go on to recidivate. And so because these scores are used in pre-trial sentencing parole decisions, they were scored to be riskier than they actually were and could have faced harsher consequences because of that. The other type of error is a false negative error which is when the prediction is that someone will not recidivate and then they actually do. From the point of view of a defendant, this is, I guess, beneficial error to happen to you because you were scored to be less risky than you actually were. So first in terms of fairness, we looked at how the human crowd and compass did in terms of accuracy on these two groups. So the black defendant accuracies on the right, white defendant accuracies on the left. Pretty much the same, both human and compass had pretty much the same accuracy on both white and black defendants. However, if we look at those false positive, false negative errors, they made very different types of errors on these two groups.  And the opposite was also true.  This is the same type of bias that ProPublica initially found about the compass predictions. And this shows that in terms of this measurement of fairness, the compass predictions and the human predictions are indistinguishable. Compass isn't being more fair than these human predictions are. There are a lot of different ways to measure fairness in an algorithm and the debate about what the best way to look at fairness is actually still ongoing. There hasn't been consensus about which measurement is the best in order to establish that an algorithm is fair or not. We looked at three other standard forms of fairness that I'm not gonna go into for the sake of time, but in every single way that we compared the fairness of the human predictions to the fairness of the compass predictions, the predictions were the same. There was no category in which humans were more fair or compass was more fair. To summarize, these results suggest two things. First, compass is not more accurate than human prediction. These people responding to an online survey were able to achieve the same accuracy as compass. The second is that compass is not more objective than human prediction. In all of the ways that we looked at, compass's predictions offered no form of fairness or objectivity that the human predictions were lacking. So to wrap this up, we've shown that this commercial software that is used in many states in this country to predict recidivism is not more accurate and not more fair than random people responding to an online survey making the same predictions. One of the major concerns about bringing algorithms into the criminal justice system is that they're what are considered black boxes. A black box is a term to describe a machine that takes in a series of inputs and then produces an output from a series of unknown or unexplained steps. You don't know what's happening in the black box. Judges that see scores like compass don't always know how a defendant's identity and criminal history were used to create those scores. Equivont, the company that builds compass, does not share the inner workings of the compass algorithm and they actually fought in court for their right to keep the algorithm secret because it is proprietary or a trade secret. Because we found that these non-experts were as accurate as compass, we started to wonder about what was going on here. What is compass doing? Why is it not more accurate than these people? And we wanted to know what the inner workings of compass were because Equivont doesn't publish this information. We decided to try to reverse engineer it and see what was happening. So we built some of our own classifiers. We built the classifiers on the full dataset of over 7,000 defendants and these are the same seven features that all of our participants saw in those paragraphs that they used to make predictions. On this dataset, compass achieves an accuracy of 65.4%, which is slightly different than it did on that smaller dataset. We started with by training a powerful non-linear support vector machine classifier, which can learn non-linear separations and data. And shockingly, this produced an average accuracy of 65.2%. How we built these classifiers was over 1,000 iterations of 80-20 training testing splits. So this is the average accuracy over those 1,000 iterations. So this wasn't any better than compass, which was a little bit shockingly. We said, okay, we replicated it, but maybe it's simpler than this. Maybe it's even simpler than a non-linear approach. Maybe a linear approach can do the same thing. So then we built a simple linear classifier logistic regression using those seven features and again, we're able to achieve the same accuracy, 66.6%. So then our next question was, okay, what's the simplest thing that we can build, the absolute simplest thing we can build that still reaches this accuracy? Again, these were the seven features that we were training our classifiers on. So we trained our classifiers on every single subset of these seven features. And what we found is that a classifier that only looks at a defendant's age and the number of prior convictions on their record produces the same accuracy as all of those other classifiers. So these are, again, this is compass's accuracy, the non-linear support vector machine, logistic regression trained on all seven features, and then this is logistic regression trained on just age and number of priors. Given that these all reached an accuracy ceiling of around 66%, it's important to start thinking about whether or not any improvement on this accuracy is possible. To summarize all of this together, this research has three main findings. The first, compass is not more accurate than human prediction. The second, compass is not more objective than human prediction. And the third, no matter what's inside compass's black box algorithm, it's the same as its equivalent to a very simple classifier built on only two features. Couple of conclusions from this whole study. First, when we're talking about the use of compass in the courtroom, we have to ask ourselves, would we place these same very serious decisions in the hands of 20 people who we pay $1 to $6 to answer an online survey? Because what we showed is that these two approaches are functionally equivalent. The second is that we can't take any algorithm's accuracy or fairness for granted. This just shows that compass is equivalent to this small crowd of people, but I think what's important to take away is that we can't assume that any algorithmic approach is going to be super accurate or that it's going to be more fair than human predictions just because it was built on data. Finally, we need to hold all of these type of software accountable. We need to have systems in place that test this stuff, that make sure it's achieving high accuracies, that make sure it's fair, and that make sure it's actually performing as we expect it to before we let it loose on the criminal justice system. Currently, there are a lot of software like Compass that are in use already all over the country, and if they are gonna be in the courts, it's essential that judges know how accurate they are. If they're gonna still see them, it's important that they know that something is only 65% accurate, that it's not more accurate than normal people guessing instead of thinking and assuming that these scores are the magic solution, that they have perfect prediction. As we move into the discussion, I wanted to pose a couple questions for the group and I'm interested in hearing what people's thoughts are on these. The first, what is more important? Accuracy or transparency? If I have a tool that I can prove to you has an accuracy of say 99%, but I can't tell you what's in it. I can't tell you what features it uses. I can't tell you what weight anything is given, but I can prove that it's 99% accuracy. Are we happy with that? Do we want to use that in the criminal justice system? Or would we rather a tool that's 75% accurate? 70% accurate, but I can tell you exactly how it works. If I can tell you it uses these three things, these are the weights, this is how you make it, but it's only 70% accurate, is that better? The second, if we can't have perfect prediction, what's an error rate that we're comfortable with? Do we want 70% accuracy, 80% accuracy?  And then the third is who should be responsible for regulating and verifying this type of software? At what point in the system should there be intervention to make sure that it's performing as we expected to and who should be held accountable when it's not? Thank you. Professor. That was a great talk, thank you. And thanks for that research. My name is Kate Crockford, I work with the ACLU here in Boston. Really interested in what you've done here. I'm definitely gonna use it in my work at the state house here. I just wanted to point out to everybody in the room that one word that didn't come up in your presentation is justice, right? I mean, there's accuracy and there's fairness. But say we had a system that predicted someone's recidivism with 100% accuracy, right? I mean, I think this is sort of what you're getting out with your first question. Would we as a society think it's acceptable to cage someone on the basis of the possibility that they may commit a future crime? I think that sort of flips our justice system at least the way it's supposed to work on its head. So I just wanted to raise that issue and get your response, thanks. Yeah, absolutely. I think that that is a really good question to take into account no matter what the accuracy is. Is that at every point, we're making decisions based on what someone might do in the future. And I think a good idea to respond to that would be maybe we need to shift how we're thinking about why we're trying to predict this in the first place. Are we trying to predict this because we're trying to keep them locked up? There's been a lot of research that shows that length of a sentence doesn't help you not commit new crimes in the future. So if you're gonna keep someone in jail or in prison indefinitely because they might commit a new crime in the future, seems like maybe not the best approach. Maybe we need to be, if we know someone's 100% gonna commit a crime, how can we put resources into helping that person not do that thing and change the course of their life? That'd be my answer to that. Hi. Come over here. Hi. That was great and I loved that question as well. But outside of that, wouldn't we expect that Compass would have the same accuracy rates because it's training on human data anyway? And also what I got from the two features being the most predictive is that maybe those two things are the most predictive and we should just be using that. But another thing is that maybe we should be thinking about why prior convictions is something that is so predictive and it's clearly highly racialized as well. So is there a way to look at that causally? And also, even though it's so simple, maybe should we be using that because it makes sense? So a couple of questions there. The first one is that did we expect Compass to be as accurate as humans because it was trained on human data? So it was trained on recidivism data, not human prediction data, if that makes sense. So it was trained on these are individuals, these are, this is their demographic, this is their criminal history, and then this is whether or not they committed a new crime. Sorry. Yeah, also. Sorry, recidivism also includes like a police officer has to catch someone which is also racialized and then if someone reoffends but doesn't get caught versus if they went to prison and would have recidivated but didn't because they were in prison, how do you distinguish between those types of features? Yeah. And that's, you bring up a really good point which is that this shows, and this confirms the work of a lot of different researchers that prior crimes is one of the most indicative variables of future recidivism, but it's not perfectly predictive. And as you mentioned, correlated with race. And that's where we see those racially sorry, the racial bias in the false positive, false negative predictions is because the black individuals in this dataset and nationwide are more likely to have prior crimes and more prior crimes on their record. And so because that's one of the most predictive variables, they're more likely to be classified as high risk even if they won't actually recidivate. And if you are a white individual in this dataset and in the nation, you're more likely to have fewer prior crimes on your record, which means you're more likely to be classified as low risk even if you will actually commit a new crime in the future. So there's absolutely so many stages where human decisions, human biases are baked into that data and have produced the data that the classifiers are being trained on. The question for you about the crowd sourcing element of this. Yeah. Did you find a relationship between unanimity among people who were voting in the, so if all 20 people thought they would recidivate, did that predict a higher accuracy than just one person? I don't remember if we looked at that, but that is definitely something that is worth looking at. I don't think that there were many cases where everybody had the same answer, but I could be wrong, but that is something that I will look into. So that's a great question. Thank you. Like Kate, is there, Kate? Like Kate was saying and also when you were saying, we talk about race and we talk about racialized, but I like for us actually discuss the word racism because there is racism inherent not just in the people that you ask to code, whether or not you put the race of the criminal, they have somebody in their mind that is a criminal and it probably ain't white. And so the data also is biased because it's the collection of all these other racist infuse interactions and media portrayals. And so kind of grabbing from Sophia Noble's book, Algorithms of Oppression, and also a discussion that we've been having internally in Kathy's Ethical Tech group, do you as the researcher believe that algorithms themselves can be characterized as racist or do you believe that algorithms are somehow objective and are simply used in racist ways? So I think that the algorithms themselves unless there's somebody writing if black predict yes, unless that type of thing is happening, the algorithm itself isn't racist, but the algorithm can be trained on data that produces racist outcomes and that data can be result of racist policing, sentencing, et cetera. So the algorithm itself, the math isn't racist in its own way, but that doesn't mean that it can't produce racist outcomes. Just a very simple question. Is Compass aware of your results and have they said anything? Yes, they have. They released a press release right when the paper came out. They had the main things they said were first, we had reported that Compass uses 137 features in their prediction because that was the number we had available to us given the documents that they have made public and so they responded saying that Compass only uses six features in their predictions. And then the other thing that they said was that the study confirms that Compass is predictively valid because it has that AUCROC value of 0.71. It's important to understand that 0.71 is at the bottom of the threshold for satisfactory performance. Satisfactory performance starts at 0.7. So, and that doesn't negate the fact that the human crowd produced that same AUCROC value. And this is an important distinction and something I've tried to emphasize in the reporting of this article is we're not trying to celebrate human prediction. This shouldn't, the takeaway from this study shouldn't be oh, humans are great at this because neither of these are great at predicting. They're reaching accuracies around 65%. So it's important to remember that the takeaways that Compass isn't more accurate than humans. Not that humans are as accurate as Compass. So thank you so much for your presentation. I wanted to ask about your use of the word human prediction as the category because obviously in the criminal justice context, the relevant actor or consumer of the product is not just humans in general, it's judges. And so there are sort of two scenarios that one could imagine. One is that because judges are professionals and the sense that they see thousands of cases that maybe they might have somehow a better sense of what outcomes might be than a random selection of humans. But then there are other scenarios that you could imagine where the judges maybe are racist or maybe even in cases where judges are elected, they're not whereas your question was framed around trying to increase the folks' accuracy. Maybe judges are concerned about accuracy but maybe they're also concerned if they're elected in trying to avoid false negatives, because that could play very poorly. If someone that they let out on bail then goes and murders someone, whereas false positives are less likely to cause political repercussions. So in which case in that scenario you could imagine a situation where we might actually view judges as being worse than a random selection of humans. And so I was wondering how you, in terms of your research, obviously you weren't able to run this with a thousand judges, but I was wondering how you thought about that relationship between humans in general versus the actual people who would be using these tools or that these tools couldn't very replace judges. That's a great question. It's something that we thought a lot about and we are trying to figure out if we could possibly run a study with judges because it's a question we've gotten a lot and I think it's an important one to ask is what would judges' performance be on this same task? In terms of the use of the word human, the efforts of this we're trying to figure out what's the baseline? What are we working with? If we're trying to make an algorithm that's more accurate than people well how accurate are people? And we couldn't really find any information about what that human baseline was. And so obviously these tools are meant to assist judges' predictions and these aren't judges' predictions, but again it's important to remember that this is trying to show that this software isn't performing as well as we expect it to and the human baseline shouldn't be used as a measurement of what a judge's predictive ability should be, it should be used as a way to show that this software needs to be held to a higher standard and there needs to be transparency in what its accuracy actually is. Hi, I am Kathy, a fellow computer scientist. I have a comment that will lead into my question. I actually respectfully disagree that algorithms aren't racist. I think we might not explicitly see if black than this, if white than that, but we can have if neighborhood than this or if income level than this and somehow sometimes those secondary features that are in the algorithms can inherently make the algorithm racist. So that leads into my question of what are your ways you think developers, for example, developers who made the compass tools or other developers who make many of these, many tools that are deployed across many governments and sell these tools together. How can do that they do better? How can they know that like a number of prior crime rates aren't actually, so how do they, what are your thoughts on getting developers and people who make these tools to do better? So to the first point, I think that even something that has a zip code or anything that is a proxy for race is a product of the data, not the algorithm. Maybe this is just kind of a distinction of what part of the algorithm you're talking about, but that is that the data in having zip code, in having prior crimes, you have a variable in there that is essentially putting race into the algorithm and that has racially biased outcomes. In terms of what kind of technologists can do in creating this, I think first is awareness. First is not operating under the assumption that what you're building is perfect, not under operating under the assumption that you're gonna be devoid of bias if you put even more data in. Even more data might not make this better and so I think it's asking the right questions. How can we mathematically try to level the playing field? How can we with an awareness of what the differences are in prior crime rates amongst these different demographics? If we know that that is what we're working with, can we try to tweak these numbers so that when you put someone in, they can't tell what the race of a person is even if that variable in the past was a proxy for race? So I think it's first just having an awareness of what the consequences could be and then trying to mathematically build to make that not happen. Hi, up here. So I'm wondering with the prior crimes and whether or not the data either compass or other algorithms that are out there are the one that you used. Whether or not you disaggregated what the prior crimes were because there are different recidivism rates if it's the same crime that the person is repeating over and over again or if it's various crimes over their life and so just for predictability purposes whether or not you think it's worth disaggregating data points that actually do present differences in the real world and if you tried to do that. So the data set that we had unfortunately didn't have every single crime that they had committed it only had that number of priors. So we had their most recent crime they had committed and the one that they if they recidivated the one that the new crime that they committed but we didn't have that history but I think it's important because if you're trying to predict whether someone will shoplift versus murder someone those are very different outcomes and so having possibly specific crime classifiers or trying to predict a specific kind of risk depending on what type of crime you think someone might commit. I think that's definitely a worthy question to ask and something to look into. I actually wanted to follow up on Kathy's question about race and algorithms and data and one thing that's occurred to me is that I agree with you that often it's bad data that causes these racist outcomes and algorithms but also what about and this is I guess specific to the context of machine learning which we're seeing underlies a lot of these the choice of what variables you're optimizing for, right? So the idea that preventing reoffense is the primary goal here is not an entirely objective when it's one that's laden with cultural values which are likely associated with race as well as income level and possibly even things like level of education and so something that was to say like if you did something like Compass but instead of using did the person reoffend in two years you said did the person subjectively see their quality of life as higher after five years, right? Then maybe you start saying you're optimizing for a different social value so I guess what is your thought about developers not just looking at the data but the choices they make about how to set up their kind of machine learning goals what's the responsibility there to look beyond kind of the status quo? Yeah, I think that the responsibility lies at every single step of the way which is why I'm grateful for the liberal arts education that I did get and the fact that I was a double major in computer science and women's gender and sexuality studies and this other major that was not related to computer science was the one that taught me to ask these questions and to be aware of how certain factors influence people differently based on their identities and so we need computer scientists in education at every level to be taking classes that make them think about the consequences of what they're building and they can't say oh I just made the thing they told me to and pass it on that's someone else's problem that's not my job to be asking the ethical questions I don't agree I think that someone who is making an algorithm that is going to affect other people should be responsible for the decisions and what they're optimizing for and then also at other steps of the way people should be verifying that what they built is actually optimizing for what they said that they were building for. I don't know where the mic is. You built several systems trying to mimic compass did you find that those systems all had the same racial bias as compass? Yeah and that's pretty much because the most predictive variable is prior crimes and that is correlated with race and so the racially biased outcome happens in pretty much everything. And you say neither compass nor the humans were explicitly given the race nor were they given like photographs or names or anything else that might try to derive race from. So basically despite attempting not to put that factor in it got there anyway. Exactly. Yeah the differences between false positives and false negatives by race were pretty revealing because there's been any comparable research to see if such differences exist when broken down by gender. They're definitely, oh I bet there has been. I'm not, I don't have the definitive answer for that but I'm sure that there has been. There's been a lot of research on gender differences also because there are very distinct gender differences in criminal behavior in general and so that's a whole different conversation as to whether or not there should be different prediction software for men and women in general because these two genders have very different criminal history but no I'm not sure if there has been work done. I bet that there has but I'm not positive. Hi, so you I think a couple of times and people in their questions a couple of times have mentioned how race became an item in the algorithm even though it wasn't explicitly stated and then if I remember correctly one thing you said was the hope or I'm paraphrasing this but the hope is that we could somehow take race out of the algorithms but I'm wondering also if there's a way to allow for race to exist in the algorithms or allow all of these factors that then put race into the algorithms and doing better work on realizing the bias in the outcomes and how we can then adjust for the bias in the outcomes. So knowing that race is gonna seep its way into these algorithms and then we see the higher rates of false positives for black people. Can we actually look at those results and then tell judges to know that there's a high rate of false positives so assume maybe that there's that bias in the information and use the information differently based on the race of the individual. Yeah, absolutely. There's also I think what's important is that there's awareness that even if you don't put race as a factor that doesn't mean that there can't be racially biased results and so maybe we should stop kidding ourselves by not putting race into the algorithm and maybe use it as a variable to try to even that playing field mathematically. I know that there are a lot of people working on this so there's hopefully will be research about how we can have an awareness of how someone's identity may have affected their entire life's history and if we can use that accordingly to make those predictions and then absolutely I think that in terms of judges there should be transparency in what the rates what the success rate, what's the false positive, false negative rates there should be transparency in what the performance of these tools are overall so that they can interpret those values as they should and interpret them accordingly. Hi, great presentation, really interesting research. I was wondering if you could tell us how your research or the compass tools in general came to have binary categories for race and also if you knew how racial categories were collected, whether it was self reported or whether an official in some capacity is the one that chose those categories. So the categories for race weren't binary in the data set nor in the study but they were the white group and the group sorry the group of white defendants and group of black defendants were the two most popular that's not the right word. Largest groups in the data set so we only looked at those two but I think that there were six categories of race in the data set and in our study. I'm not sure how those this was that was the data that came in from that county in Florida where the data is from. That's how the race classification happened. Was there a second part to your question? How it was collected but do you happen to know how the accuracy rates were for other groups? Not off the top of my head but I'm not sure. Thank you. Thanks for your presentation, it was great. You may have touched on this, apologize if I'm really asking the question but does mechanical Turk capture demographic data and was the sample that you used representative of a population of judges or the United States or some other population? So mechanical Turk doesn't capture demographic information but we asked so we asked for race, gender, age, education level there might have been more but that was it and our demographic was definitely not representative of the nation at all. There was a majority white demographic but we did have ages from 18 to over 74 and of all different education levels. So racially not as diverse as we wanted it to be with the respondents but we did have a good scale of other types of demographics for that. I was just curious if you could speak to the more general problem of using algorithms in the sense of using group level data to predict individual recidivism versus the more typical model in the past of having a judge make the individual determination. Yeah, that's a great question and something that there was a case in Wisconsin where an individual were given a sentence and that sentence included the compass score that they were given and that was one of their arguments was that they were using group level data to make a decision about an individual person. I think that that is just a question of whether or not we should be using these algorithms in general because you are using historical data about not that individual to make a prediction about that individual so that is just a basic question of whether or not these should be used in the criminal justice system in general. Actually I just have two questions. One is, is there an effort being made to educate judges about your research and then the other question is... Okay, answer the first one. I don't, there's not an official effort to educate judges or hoping that the media attention that this got brought this into public people's attention. We have gotten some emails from judges about it and one state, oh I don't remember which it was, there's a state that was undergoing a kind of committee review where they had something up for vote to mandate the use of one of these types of tools and so they reached out to us asking for an opinion. So I'm not sure if there's an official, as far as I know there's no official effort to have this aware or to make judges aware of this but it's also specific to what tool they're using and that's why it's important that there's transparency about any software that's in the criminal justice system so that when a judge sees a score it knows where it came from and it knows this is only 60% accurate or this is 80% accurate or this is an awareness of where that score came from and how reliable it is. And I just remembered my second question which is are litigators using your research to try to get defendants to make it recognize that maybe that's not the best predictor? Not that I know of. I just have a, sorry Sebastian Diaz from the Berkman Klein. I just have a question about the data specifically. Is the data, is the score that Compass gives a binary data or is it on a scale? Good question. So Compass's score is actually range from one to 10. There were a lot of details that I didn't get to go into about the study that for the sake of time but yeah, Compass gives scores from one to 10. One to four is low risk, five to seven is medium risk and eight to 10 is high risk. The decision to make it a binary cutoff was to be able to compare it directly to the human predictions and we wanted the human prediction to be as simple as possible and not have people be giving scores from one to 10 but it was based off of the practitioner's guide for Compass says that scores above a four should be considered an indication that that person will recidivate. So the scores aren't binary but they have given, sorry, the scores aren't binary but they have kind of a binary interpretation of how. And so this is a follow up. Since they're not binary, did your group metrics match the scores that were given by Compass? Did our group metrics. So when you had one to 20, I imagine if you divide that to two you would have one to 10. So how we did, yeah, so when we did translate the crowd votes into a score from one to 10 so that we could calculate those AUC, ROC values. So the crowd was just what percentage of people guessed yes that was the score that like binned into one to 10 that was the score that we were assigning people based on the crowd votes. Did that match Compass? Oh yes, it matched Compass's, oh no in binary ways. It didn't completely match Compass in the binary prediction. The crowd predictions and Compass's prediction matched in I think 669 of the cases, of the 1,000 cases. It's somewhere around that number. So when it was binary the crowd predictions and Compass's predictions were in agreement about 70% of the time. Hi, thank you. There's a lot of ways to reduce recidivism by social work, education, job opportunities. Is Compass blind to those programs? I'm not sure. I think that, well actually so they do in the assessment for Compass an individual has to answer those 137 questions and a lot of those are what are called needs assessment. So they only use those six variables which probably are very similar to the six variables we used, I'm not sure but they probably are. And a lot of those other questions give the court and anyone involved in creating a plan for them an idea of different places where they need support. So I'm not sure. I know that Compass, they do provide those questions so that the judge knows a little bit more about that defendant but I'm not sure if they are. The Equivont, the company that makes Compass has a lot of different programs and software available so they might be doing more in that sphere but I'm not sure. I just thought I'd follow up to that and say there's a great paper called Zombie Predictions by the guys at Upturn which actually talk about this specifically one of the weaknesses being that risk assessments right now don't incorporate learnings around risk reduction strategies. So you're likely to get zombie predictions, things that are based on older data that existed prior to some sort of risk reduction intervention happening and therefore would skew the predictive capacity of the results and so I thought maybe as a question of that how would you like to see people incorporating I guess risk reduction as a strategy into these tools? Are there ways we could change risk assessments to be more intentional about thinking about risk as this thing that changes and that we could actually reduce as opposed to just asking the binary question of lock up or release? I think maybe that's related to this question about what are we optimizing for? What are we trying to predict? What are we optimizing for? What kinds of errors do we prefer over others? And so maybe we can be optimizing forward if they're going into this program, how did that play out two to three years later and that maybe we're not saying play out in terms of recidivism but play out in terms of do they have a job two to three years later? And so what are we trying to predict and like what are those features we're putting in? I think could be a good way to start. Hi, my name is Sarah. I'm working at MIT on AI for Public Interest projects. I'm curious, you'd mentioned earlier different sort of methods for measuring fairness, algorithmic fairness. I was wondering if you could say a bit more about that, whether you've done research and what the trade-offs might be in utilizing different measures as we in a way audit the public sector's use of algorithms. That's a great question. So one of the, what came out of the ProPublica article about Compass, so ProPublica said these false positive and false negative errors, there's a discrepancy in terms of race, this is biased. And North Point, sorry, at the time North Point now, Equivant responded saying that their tool was well calibrated. So what calibration is, is any recidivism risk score should mean the same thing regardless of race. So Compass's scores are range from one to 10 and a score of three should mean the same thing no matter if you're white or black. So what they responded saying is that the scores are actually well calibrated, which is that at every score, your chance of recidivism is the same regardless of race. And they are accurate in saying that their scores are well calibrated. And this sparked this debate over how can this be true? How can both of these seemingly valid ways to measure fairness be contradicting with each other? And what people figured out is that under certain circumstances, satisfying these two types of fairness requirements is mathematically impossible. So if the recidivism rate in two different groups is different, you can't have a tool that is both well calibrated and it has the same false positive, false negative errors. That's kind of a math thing that we'll take a longer to explain, but basically what's important is that certain fairness requirements are mathematically impossible to simultaneously satisfy under certain situations, which is kind of crazy because we want both of those things. We want a score to mean the same thing no matter if you're white or black.  So both of these measurements are ethically valid to want in an algorithm, and if the recidivism rate isn't the same, we can't get both of those. And so this is why the debate about how to measure fairness is so important because there's so many different fairness measurements and fairness constraints that we want in these algorithms that are totally valid to desire and we have to figure out what we're optimizing for, what is important in a specific context, and maybe what is the most important fairness measurement with these scores. There are a couple other fairness things, but that's the main one that we also looked at, yeah.


Well, it's an enormous pleasure for me to be here and I particularly like also speaking to audiences where you have legal scholars because that is forever an engagement in the kind of work that I do and forever the need to discover how far does my argument go if I factor in notions of law which inevitably contain delimitations not limitations the delimiting of an argument so I also really look forward to your questions to your comments because I learned that way I'm by now a nice old little professor who keeps discovering I'm so happy that I'm still living because there are so many things that are changing you know it's like a little revolution continuously and as I travel around the world believe me there are many unsettlements that are taking place in many parts of the world so let me while you're ready look all of that and and what I want to talk about is precisely that the rise of extractive logics we tend to think of the extractive as in mining you extract once you extract the characteristic of an extractive logic in my in my reading is that once you extract you don't care what happens in the site where you have extracted it's an absolute effect whereas in other fields you want wherever you take something from to grow and to expand because you will need more of it and when you when you think of I'm particularly also thinking about extractive logics in our economy it in high finance is really algorithmic math you know it has nothing to do with traditional economics and it really functions as an extractive sector and I will be talking a bit about that so it is a very radical and extreme component of our current reality that I want to talk about with you and I repeat it is very extreme but it's happening and in a way it is spreading and some of the research I've been doing most most recently again confirms that this now has to do with the buying up by major financial firms of modest housing complexes not just little houses housing complexes all across the world not in China they haven't gotten into China yet these are mostly Western entities so when I talk about the rise of extractive logics I'm talking about something that is very real that is happening even though it is still confined only to certain settings you know it's not happening everywhere I'm sure it's not happening here in your town so who knows we have been surprised you know we have a whole team doing research across the world now let me start with this notion that we make sounds good they're good things that we made but we also made this the RLC you recognize it in 20 years we managed to destroy one of the largest internal seas that we have you'll have to stand back and say wow it's an accomplishment of sorts we did that we also did this you know these capabilities the questions that this raises in my mind is if we could do this what else could we do that is profoundly transformative so over hovering over the negatives for me is also the notion of recovering a capability that now is a negative how do we make it into a positive now this is something I just brought that in because I thought maybe some of you would would see the point here but this notion of key codes and the capacity to aggregate the capacity to jump up how we can with existing capabilities not just take it in the negative vector but also take it in a positive vector so I'm doing this with my son who is a crazy artist unlike his mother who is not crazy that's me by the way the mother because you know it's really interesting to see how artists are dealing with some of these issues let me start with raising a question to get at the the properties the particularities of our current epoch I don't know if I can do you hear me this way or do I need the mic I maybe need the mic so what is the steam engine of our epoch if you want that which can make a new ordering without changing everything because that is one of the crucial modalities of course of transformations that which shapes what is in and what is out and we have had such I argue that today it's high finance and high finance has nothing to do with economics I think this is a crucial issue and maybe there are some of you legal scholars who will disagree with me and let's have a discussion let me use as an example this this is a fairly known element and it's just one little element this is credit default swaps outstanding 1919 billion into a one sixty two trillion just a few years later question invitation think of anything that has this capacity of growth and this which I describe is only 10% of the value in play of finance so just how I'm not I lose the plot oh you know what this is it attached to moving here yeah okay it's okay now this is another element that I want to put on the table and then I'll return in a more generic way to some of these capabilities footnote we tend to think of the term capability as a positive capability sounds positive but of course there are capabilities that are very dangerous there are negatives that are marked by extractive logics so here but now returning to this so dark pools in finance I'm just I don't know how many of you heard about this but it's a growing condition in high finance in the financial world and see here we have Europe and the United States as you can see you know a bunch of these dark pools the term dark pools is not my term it's a term of the head of the central bank at a given moment in time about ten years ago when this sort of begins to get going now United States very number one likes being number one and Europe number two there are other countries that are also engaging this I don't know how many of you I repeat heard of dark pools in finance but they are private networks owned by private firms that are today the main instrumentalities in the world of high finance it's not the the the publicly known and recognized like the central bank or the whatever other institutions it's private networks number two there are very long lists of firms are talking business firms waiting to be accepted in one of these private pools number three and these are the words by the head of the central bank we do not know what happens in these dark pools and he coined the term dark pools for them so what we see we the citizens we see the stock market moment when it comes to high finance and a lot of this stuff will also have a moment in the stock market stock market is a public entity in principle anybody has access to it I mean you have to go through a few parameters so to say radically different from these dark pools in finance I mentioned this to just emphasize a capability a capability that is really quite extraordinary quite private we know very little the head of the central bank also knows very little and there are long waiting lists of entities the owners of these black dark pools are private actors as you can see it's also entered in Europe and I would love to know if there is somebody here who is working on what's happening in Europe because it's not so easy to get the data let's put it that way why am I having oh yeah right okay we discovered that that's ah it doesn't work either there we go so so finance as capability in both direct and indirect pathways so here I want to just give you a few examples this is what became a crisis in the United States it was considered a crisis generated and this is very important by fairly low income people who and I quote thought they could afford to buy a house put whatever money they had signed whatever contracts got a house and five or ten years later they all went bankrupt 14 million house 40 point 14.5 million households went bankrupt that's a very large number certainly if you think of a country like the Netherlands that would mean that we were like almost all country is bankrupt 14.5 million households could be you know you multiply the fact of all the people involved in a given household etc it affected a lot of people in a big country like the United States in a country like the United States which is still living a bit sometimes in the Wild West it doesn't need to know what all is happening all over it many people never found out that this happened now I want you to think what does it mean to expel 14 million households from their home because that's what happened basically I mean it took years we're talking a period of about nine years but it's a vast number of people and it was invisible every household could have been ten people five people three people we don't have a visual on it because that's another modality you can engage in massive operations that never reveal that they're massive because you know it happens over years it happens in many different districts it happens so many people never really in the United States never really found out that you had this little history going on and here are just some of the figures for closures for those I mean you are some of you at least you know are familiar with these guys and so this is also includes repetitions have because it's not just single but we are talking millions and that's another thing what is it about our current systematicity that a huge number can sort of navigate the situation as if it never happened many people in the United States never found out that this happened that all these people were expelled from their homes in 14 million households again it could be 20 million people 30 million people okay now here is something something else that is also connected well this is the this is Europe in Europe but we they have stopped counting I've been trying it if any of you knows after 209 it's very difficult to get access to these figures but Europe went through something similar on a much lower scale and so here are some of those figures and as you see you know and and this is European countries with either highest or lowest number of foreclosures and I don't know why we can't get data after 209 but I would really like to know if there is anybody who knows why but again it gives you sort of a sense and and really the list goes on and on I just don't have it now you understand the language of foreclosures right I know it's an English term but other element I'm just putting elements here on the table ratio of household credit to personal disposable income right so you okay now I want to point out so we all know that in 2000 the figure the dates here matter 2000 to 2005 2000 is when Eastern Europe especially I'm especially focusing here on Eastern Europe when Eastern Europe sort of you know enters basically the Western style economic system and certainly in finance and loans and all of that whether modest so I just want to so let's just go to this title ratio of household credit to personal disposable income credit credit is a very special word credit suggests that you have it you can spend it it's a bit yours well and listen to what happens check Republic ratio of household credit to personal disposable credit is dead by the way right it is dead for the household it's credit for is peculiar language so on this is not my language in other words I'm sort of also engaging in a critique a bit of this language but so here check Republic 2008.5 by 205 and these years are crucial here 27% are Hungary 11% household credit debt 39% so yeah so it just goes on now at the same time the United States was already way off the charts look at this 104 105 Germany on their hand admirable stability and 70 is very reasonable 70 70 70 70 70 is my name and so you look at these different countries you know I mean most of these countries down here have been rather reasonable now when I see these type of data and this is just one microcosm but it is a microcosm that is why there are mostly a certain type of country I immediately have a few questions and so I'm having a bit of trouble here there you go and so I want to know who owns that debt that you know those contracts that modest households bought in this case in Eastern Europe etc. Now here we see share of foreign currency denominated household credit and to a 5% household etc. So Hungary, Romania etc. are some of the countries where you had a lot of this debt functioning right and oh you know what I have a slight missing here I'm really sorry well let me I'll elaborate on it so these columns that you see here indicate foreign ownership of this household debt Germany owns a lot of this debt you know there are a few countries that own a lot of this debt all the Germanic countries in this particular group that I was looking at in Eastern Europe Germany Austria and Switzerland owned that debt of those modest households are in those other countries that to me is significant because you know if the local bank in your neighborhood owns your debt that is something totally different you know that's a little neighborhood effect and you pay your interest rates then the bank you know etc etc can develop more business and whatever you can imagine all the nice local bank these modest households became part of a grabbing operation is to put it a bit brutally but it was in a way a grabbing operation that is happening right now in many parts of the world including in in Asia etc not in China China totally different where you have using the use of modest localized households where international banks move in and take out they and if you have a million of those even if they're low payments it accumulates but on top of that there is an extractive mode it's not like your local bank that wants to enable you and wants to enable your sons and daughters know this is extractive and now it's in the housing question I've I recommend to all of you who might be interested to film called push not a good title but anyhow push tracks one of the major corporations that we think of as a financial firm based in the United States which in many parts of the world including in Asia is buying up huge housing complexes for poor people for modest income people and redoing that now they're making money of this and the result of doing all that upgrading is a doubling of rents you know the story right and so those the poor the lower income people who used to live there are out this is something now I don't want to mention the the corporation that is doing this this is just one we are tracking and now we have all the documentation etc it will enter the public domain pretty soon parts of it already half so this is one huge firm a global firm that does high finance and they're doing this this is this the this is you know buying up all these modest for all these big buildings with modest properties this is new this has a bit of it has always happened but on this scale that you have major financial firms buying up housing modest housing complexes all over is new and and it raises a whole bunch of questions that I don't want to dwell on now but that are worthwhile sort of thinking about in short extractive sectors an extractive sector is a particular type of sector can extract even from modest households so there was a time when a modest household was a modest household was a modest household today you actually need to check it out that very modest household is actually likely to be part of a huge field of materialities that the financial system has generated is interested in and has then generated and with algorithmic mathematics as I already said you transform a million little houses into something that is no longer we see the house the building by algorithmic mathematics you have a field of material assets and that is what the high-income investment sector wants they do not want derivatives derivatives are for you and me they're out for the high it used to be you know derivatives have been around for a long time they were a successful now what they want is asset-backed securities how do you generate asset-backed securities you need some materialities in there all those the acquisitions of all those very modest houses we're talking millions now has to do not with the little house it has to do with asset-backed securities so you have a real asset I don't know if you're following what I'm saying but this is pretty disastrous if a modest building can be transformed in something that can produce high value as an asset as an asset-backed security not good not good now here's another extractive mode so I think of these all as logics that are centered in extraction in not generating not building not taking a risk in extracting this is something that I'm assuming some of you have heard about which is really a very significant buying of fancy buildings in major cities the full list is a hundred hundred cities in the world which are sort of the top you know the liverers if you want or objects of desire and so this is just one year it happens to be a very significant year because it is when it first really takes off and the figures so that we're talking 2013 to 2014 a sort of a year now what you're seeing here is first in New York in the metro just in one year 55 trillion bought in-house in-house billions and that represent ten per nine ten per ten point nine percent is growth over the prior year London 47 Tokyo and then it goes on so you have a hundred now you have Hong Kong this minus does not mean that this is a minus this is a positive but compared to the prior year of acquisitions this is lower now I am working with a hundred and it is quite interesting I don't know if I'm if I'm communicating what this is about but this is about just buying property of all sorts and I will develop what comes next right but it is buying buying buying and this is just looking at foreign buyers foreign investors here at London is the queen of the domain and then New York there is much less so but again the list you know and again the list of the hundred is quite interesting now I must tell you that in London now just as an example London is an extreme all sort the the Katari Royals own more of central London than the Queen of England which I sort of find sympathetic almost you know what I mean sort of how neat she really good number two you can't see that it's not like the buildings are telling us now we belong to the Katari Royals so again how we are losing traction in terms of what the material world tells us because I think there were earlier times when the material world really told us a lot I think we're moving into a time when between algorithmic mathematics and I don't know what all the certain elements of our current internationalism like anybody can buy anywhere we are losing the material is losing the capacity if you want to tell us what is actually going on okay the total value of these types of acquisitions which is a limited number you understand it's a minority of buildings anywhere in the in the 100 top cities by acquisition this is just acquisitions of existing buildings not new construction that is all so you have mid 2014 to mid 2015 it was 600 billion mid 2015 to mid 2016 over a trillion so there's a lot of money in play sort of circulating as if it doesn't know what to do with itself by these housing some are empty some are not many are empty though I want to emphasize that I'll come back to that point okay yes so the top 100 cities by property investment in 2016 account for 10% of the world's population 30% of the world's GDP 76% of property investment now I put this these are not my figures and I put this in this type of separate modality because I think it's investment that's a language that they are using I think it's something else an investment we associate with certain types of product productive modes something this is different one result by the way our empty buildings in Manhattan London Paris we have a lot of empty buildings I alluded to it earlier already and people think ooh poor investors something went wrong and the question is did something go wrong or is an empty building actually better than an occupied it's a real question I'll come back to it too so here just a collective figures worldwide real estate assets these are figures that come from savels and savels is really everywhere in the world it's a it's a it's a money-making corporation it's not a research operation so I was in Pnom Penh trying to get some data having a hard time savels they had the data I needed I was a bit in shock because I you know it's not my mode to work with the with commercial operations but anyhow world but real estate assets comprise nearly 60% of the value of all global assets that's an impressive number and that takes a lot of people by surprise including equities bonds and gold so this is sort of a total figure of all these buildings that are in play if you want again the figures come from savels now one one question that I as a social scientist as a theorist and ask what do I see what do my eyes tell me about that reality in the making so one way of putting it is what does this all look like this is civilized London central London here is the Timbs and here are these buildings mostly they're old buildings so a journalist a German journalist they're always so serious the German journalist said after we had talked he said let's go to the place which you're talking about and I don't usually do that by the way you know I just sit there do the and so I found myself in this very touristic area in central London there is no and and I heard the tourists it's top full of tourists that's why I never go there I don't know I'm a tourist across myself too but I'm mad and so I heard them say look at these beautiful British buildings these are all owned and they are beautiful British buildings I think that's an ugly but anyhow most of them are very beautiful they're all owned by one Chinese company now I don't hold this against the Chinese by the way I want to emphasize that very very clearly I just find it interesting and it's an invitation to think what am I actually seeing what what see and and I'm dealing both with something quite abstract which is what is the capacity of the material today in today's world to have speech there was a time I think when the material had quite a bit of speech I'm not so sure today and this is one of those examples that you see all these nice British looking things and none of them is owned by a Brit so so you know there is something to me very interesting about that besides the fact cities on the ground you know at ground level which is the business is involved and all of that totally different just on pass on London property purchased by overseas companies all of these companies I have the full list are abroad none of them has a name that we recognize they all have you know there's a certain kind of formula that gets used for some of these firms now that's a lot of buying I don't know if you can see that that's just a larger London area this is something that is repeating itself I am tracking this in many different parts of the world I am not sure where it all goes I think that there are different utility functions involved so different projects if you want but there it is and it's probably growing as we speak okay baby come on now this is another thing that you see in Manhattan and you see it in London and you also see it a bit in Paris or more difficult to establish the New York has had a mode of building as you can see that sort of stands out if you go to Paris you rarely would see the Queen standing out there like that I mean the Queens at the buildings I don't mean any real Queen and so you see in Manhattan Manhattan is a little bit of land it's not huge you know Manhattan is not big you see these towers that are actually mostly empty and by now these are all dark why am I having such trouble that so 57% of condos are owned by shell companies you know this is a familiar bit I'm just going to show one more right and this is the Plaza these are all very fancy buildings and the point I'm I want to make here is that gradually less and less people are living there because they don't like living there because it's empty and in fact the buildings now they want them all out because the building can function as a physical element that can feed a different type of instrument having people is a bother having people is a drag having people means that you have to pay for this and for that leave the building empty it's the best business deal that is a bit alarming and now I told you we just did this huge project tracking across many countries in the world there is a film is out it's an amazing film called a repeat push that name but anyhow how much of this is happening in different parts of the world that they're buying up the buildings and leaving them empty and an empty building can produce more better types of income than an occupied building why because it's not the building it's what I started out with by algorithmic math you have a pile of assets doesn't matter what is the toilet the wall doesn't matter it's not a visual thing and there you build asset backed securities which is what the high income sector wants they don't want derivatives I repeat myself derivatives are for you and me they're tired they don't deliver much they have been overused so I am that good for us the high investment circuit asset backed securities that is what they want how do you produce an asset backed security you need a lot of materialities to produce the asset part of it and then you bring in you know all the other instrumentalities that make it a financial instrument you know all right so one I don't quite end here though this was also meant as an ending you know when I ask myself as I go around the world and believe me most countries and most continents are not like this continent here what are the formats of the future when it comes to this kind of stuff you know of the how people will live how what is the meaning of cities etc and thinking of our larger world where most people are poor and are getting expelled from their land because this is a thing that is happening so with those kinds of elements I spoke a bit in shorthand but I'm sure you could follow so this is one juxtaposition by the way there is no part in the world where you can stand and see this I just want to alert you to that because I very often get questions can you tell me where that was both are real though the one up there is a mix this is real but it's not like there is a place where you can stand and then also see that other but this is the juxtaposition that in many ways captures the future in most parts of the world except mean Europe is really different there and the United States a bit the Americas because it's so huge you know but this is you know this very dense not fully regulated but not criminal either you know the the this side of throne and then the towers and if there is no land we're building on water now we might ask why does all of this matter you know so what you could just say so there they do it that way they're here there are a few crooks involved in the whole thing but but for me the city is a critical space for those without power in today's world where most land we may look at a map and say oh but there's so much empty land it's all owned there are 30 governments that have been buying land all over outside their own land and there are over 150 corporations that for a whole variety of reasons and all kinds of different economic sectors have been buying land there is almost no open land there is a bit you know in certain parts of Africa and certain parts of Asia but basically we're out of land as we are also out of sand for building you have heard that part right I love adding that I just thought about it it has nothing to do with my lecture here but you know so so the city so why does the city matter the city as a space where those without power get to make a history and I think that that is very important and when you look at I repeat what I started at destroyed you know over the centuries what has survived remember that cities have outlived very powerful actors governments kings queens you name it the city is still there after all these regimes are gone and what has especially survived are the neighborhoods of the poor which in a way were untouchables in one version or another in Europe we didn't call it untouchables but they were basically and the the big palaces or the big you know public monuments and all of that what has gone are the areas of the middle classes the broad middle classes so but this notion that the neighborhoods of the poor and the sort of the central part of the representational side of the city has outlived across centuries all kinds of forms of power kings queens you know whatever is to me also an interesting datum secondly the capacity of urban space and I'm thinking especially now of big cities cities that are a bit messy that cannot be fully controlled so in the US we have a few of those but around the world we have a lot of those so the capacity of the city to make us all whether rich or poor into urban subjects even if only for a moment of the day now I think in here in your beautiful city though I haven't quite gotten to know it clearly but I've got to know very nice parts of it but maybe this all you know is sort of I think the Netherlands is special Europe is very special Europe is different really from most of the areas that I am talking about so I just want to make it and have this notion that everybody like in the subway in New York where you have I wouldn't say the super rich but many very high-income people you just take the subway because the only way to get quickly to your home or whatever it might be and the poor and the homeless and those who haven't been able to wash themselves for a month all of it is together in our subways our subways are also falling apart I don't know if anybody has recently been there it's a scandal huh public domain doesn't get any money but anyhow there are moments in the city where this is possible this absolute intersection of all subjects and this matters one instrument in supporting the notion of a civic subject you know is that that there are moments and there are conditionalities that make us all into urban subjects sort of the notion of a civic subject beyond wealth and beyond poverty and not to do with poverty the city and and so another language that I like to use I've written all all text about this the notion of the city as a hacker a hacker you know what I mean by that that you unsettle hacking is unsettling that you unsettle the basic meaning the traditional meaning and a city is an extraordinary place not a tiny totally controlled city but a big city that cannot be fully managed you know where you are really it sort of hacks subjects that think themselves all powerful and it hacks the poor because the poor suddenly have also a kind of presence huh this is a bit romantic I admit it doesn't work with with legal scholars but and so this is on like the typical trends for the post-World War two patterns because then you had a after World War two there is a lot of quite civilized building and and sort of a sense of the civic and you know it's a very interesting period actually after World War two that happens certainly in the West and and now I feel that after 1988 for me 1980s is the break when we globalized deregulate and privatize a lot in many countries extreme case is of course the the the Americas but Europe also begins to do that then you enter a different period and so I think one way of marking that difference is this very they have always existed logics of extraction have always existed that we really see an ascendance of logics of extraction so traditional banking coming back to that initial example traditional banking is commerce it sells something for a price money well we all need it we can get better whatever a better education better I don't know what but when you extract that's it there is no it's up and that's what high finances it's extractive so a lot of the stuff that I've been talking to you about actually is very much linked to extractive whereas we tend to think of housing the housing question as something we all need it a city needs it you know there are deep histories around the question of housing and I I really think this is a change again I urge you to see this film called push which looks at what I have been describing here on pass on a bit in in many different countries including countries in Asia where the housing question is no longer the housing question it's a different question it's really a different now very quickly I I just wanted to show these this is a particularly crucial moment these are just big trends I'm sure that many of you are already familiar with it but I just wanted to put that also on the map this is one way of capturing a lot of what I talked about these these these are the logics in play corporate profits corporate profits have long existed here is the difference look what happens here goes up the crisis famous crisis to await 2010 it lasted about three minutes for them and then higher than before for most of the people in the US it was a crisis I don't know if everywhere but in many parts right and this is even better corporate assets the crisis is that little little wrinkle there you see that where it sort of goes down a bit it goes up so we have two worlds in our big cities we have two worlds one that really can have a crisis in the other one that can feed off a crisis which is a rather different proposition this I don't know if people can understand the graph but anyhow it's 1917 to two or five but it really goes on I'm sorry it's kind of cut off there income share of 10% top earners so you have a high eye there you know and then it becomes this nice lower level curve for the majority are middle-class households that are doing very nicely etc and 1987 1999 it goes up again in other words concentration of wealth and of advantage right so we have this period in our West now this is the United States United States always a bit extreme but this is the same curve you see in many of our countries certainly in the UK and in Argentina and and well you have a period here is you know this is like after World War two you have several decades where you really have a sense of the growth of middle-class more opportunities and blah blah blah and many people still think that this is now no that really begins to change in the 1980s and you have again a concentration rather than this sort of more distributed notion I don't know if this graph I'm hoping that it that it is clear I want to end with something that I like to think of as unstable meanings a lot of what I've been talking about is marked by a certain kind of instability of meaning now legal scholars are very good at detecting where is the stability of a concept of us so I like sort of this notion of an unstable meaning now given all of these negatives we have no growing rapidly you can all read that etc this is the question who are we the citizens in this setting you know there's enough negative and the United States is certainly an extreme case so I wanted to show you this map and this map is in the public domain how many of you have seen this any I see somebody there you have seen it oh no you were maybe saying hello okay in the United States almost nobody has seen this map this is astounding actually and people often think that I'm showing something that is secret you know these are your other numbers all buildings that are tracking all of us if you were there for two days or have a day you're in that system if my old grandma went to have a cup of coffee in the village in New York she's in there we're all in there the entities are a vast number of institutions that are gathering data on all of us gathering data gathering data huge buildings are being built also in the middle of the country to house all the information now and it's in the public domain because the Washington Post with enormous courage and they have to pay a big fine because they were said we're going to put it and you find it in the Washington Post 2010 here's the information it has a lot of data I'm just showing you this one map to me it is very important to realize how few people in the United States know that there is this capacity to track all of us there is an ironic point that I make that not everybody agrees with me they have gathered so much data that when they really could use it they can't they get lost you realize I mean all those buildings are continuously gathering data so you know it's like the curve yeah good good good good and then you have so much forget it so they often what happens is they track the person in the field you know on the street or and then they try to find more information in this enormous collection of data all electronic no gathering and I you know they don't tell us really very much I don't know but it doesn't always work now if I may I'm just going to run through this so I'm thanking Jody Hvergen and Snowden you know snow right so this is what we know and I emphasize what we know which is very limited but this we know that the NSA the National Security Agency can do so far because by now you know who knows where they are so when you read it goes on and on and on and on and some of it is so ridiculous which is a really like I don't know if I can see one of them here but I can spy it can spy on you as law firms representing foreign countries and trade negotiations okay that is sort of a meaningful thing but I want there is another one that they have here that tells you I mean some of the stuff is just so preposterous it's like a system that has gone out of control like they can they can they can enter for instance the United Nations discussions you know which I close doors but my god if it's top secret you know it's any up so I just want to end with this now in Europe you know you also have this right started in 2015 right and it breaches recent European Court of Justice you know you know more about this than I do and that's it thank you very much


Hi guys, my name is Ben and I'm from Brisbane, Australia. For the past five years, I have been living in Chiang Mai, Thailand, working with different organisations teaching building skills and basic life skills. The one thing that I've noticed while I've been here is the high demand for good quality English teachers. I've always wanted to be a teacher and this is the place where I thought I should start. I went online and found the ITTT website in which I signed up for a 120 hour online TEFL course. I have a lot of work here in Thailand so the online course worked well for me. I mean I was able to study at any time whenever I was free. The curriculum and the video lessons were very good, they were very clear and I had lots of examples so you knew exactly how everything was done and how to effectively teach a certain subject. While I've been teaching different things here, all the students always come up to me and ask, what does this mean, what does that mean in English, how do I do this, how do I say that and I always feel like an idiot because I have no idea how to explain it to them. This TEFL course has helped me a lot to understand the basic English principles, the rules and just learn how to be able to teach people in an effective way. If you want to be an English teacher then this is definitely the course for you. This will get you from where you are now to where you want to be, teaching in schools, teaching as tutoring or whatever it is that you may want to do. It has given me the ability to be able to teach effectively, to be able to understand what each aspect of English does and what it means and just helped me to have the confidence to be able to teach people.