Welcome to the latest edition of the MIT Sloan Expert Series. I'm your host, Rebecca Knight. Our topic today is racial bias in the sharing economy, how Uber and Lyft are failing black passengers and what to do about it. Here to talk about that is Chris Knittle. He is a professor of applied economics here at MIT Sloan. And he's also the co-author of a study that shows how Uber and Lyft drivers discriminate based on a passenger's skin color. Thanks so much for joining us. Oh, it's great to be here. Before we begin, I want to remind our viewers that we will be taking your questions live on social media. Please use the hashtag MIT Sloan Expert to pose your questions on Twitter. Chris, let's get started. So there is a lot of research that shows how difficult it is to hail a cab, particularly for black people. Uber and Lyft were supposed to represent a more egalitarian travel option, but you didn't find that. That's right. So what we found in two experiments that we ran, one in Seattle and one in Boston, is that Uber and Lyft drivers were discriminating based on race. And we've already seen, actually, some racial evidence of racial discrimination in the sharing economy, not just with ride sharing apps. Sure. So there's evidence for Airbnb. And what's interesting about Airbnb, actually, is the discrimination is two-sided. So not only do white renters of properties not want to rent to black renties, but white renters do not want to stay at a home of a black homeowner.  Partly. So I was an optimist. We went into this, at least I went into this, hoping that we wouldn't find discrimination. But one thing that has helped, or at least put, shined a more positive light, is that there are ways that we can do better in this sector. You've talked about this study, which you undertook with colleagues from the University of Washington in Stanford, shows the power of the experiment. Can you talk a little bit about what you mean by that? Sure. So what we did was actually run two randomized control trials. So just like you would test whether a blood pressure medication works, you would have a control group that wouldn't get the medication and a treatment group that would, we did the same thing where we sent out in Seattle, both black and white RAs that hailed Uber and Lyft rides. And we randomized whether or not it was a black RA calling the ride or a white RA at that particular time. And they all drove the same exact route at the same exact times of the day. So what did you find? Let's talk about first what you found in Seattle. Sure. So in Seattle, we measured how long it took for a ride to be accepted, and also how long it took for, once it was accepted, for the driver to show up and pick up the passenger. And what we found is if you were a black research assistant that inhaling an Uber ride, it took 30% longer for a ride to be accepted and also 30% longer for the driver to show up and pick you up. 30% seems substantial. Well, so for the time it takes to accept a ride, we're talking seconds. But for the time it takes for a passenger to actually be picked up, it's over a minute longer. And I'll mention also for Lyft, we found a 30% increase in the amount of time it took to be accepted. But there was no statistically significant impact on how long it took for the driver to actually show up. So the thing about the minute difference, I mean, that can be material, particularly if you're trying to catch a cab, an Uber, or a Lyft for a job interview or to get to the airport. Yeah, this is introspection. But I always seem to be late. So even a minute can be very costly. I hear you. I hear you. So why do you think there was the difference between Lyft and Uber? So what's interesting, and we learned this while we were doing the experiment, a Lyft driver sees the name of the passenger before they accept the ride, whereas an Uber driver only sees the name after they've accepted. So in order for an Uber driver to discriminate, they have to first accept the ride and then see the name and then cancel, whereas a Lyft driver can just pass it up right away. So it turns out because of that, the Lyft platform is more easily capable of handling discrimination because it pushes it to another driver faster than the Uber platform. I want to come back to that. But I want to say, also, that difference caused you to change the way you did the experiment in Boston. So in Boston, a couple of differences. One is that we sent out RAs with two cell phones, actually. So each RA had an Uber and Lyft account under a stereotypically white-sounding name, and then also an Uber and Lyft account under a stereotypically black-sounding name. So that was one difference. And then also what we measured in Boston that we didn't measure in Seattle is cancellations. So an Uber driver accepts the ride and then cancels on the RA. Let's go back to the stereotypically black-sounding name versus white-sounding name.  How did you determine what those names are? So we relied on another published paper that actually looked at birth records from the 1970s in Boston. And the birth records tell you not only the name, but also the race of the baby. So they found names that actually 100% of the time were African-American or 100% of the time were not African-American. So we relied on those names. And the names were? So you could imagine Jamal, for example, compared to Jerry. All right. OK, Ayesha and Allison. Sure. Yeah. So what was your headline finding in Boston?  And what about Lyft? So for Lyft, there is no cancellation effect. And that's not because there's no discrimination. It's just that they don't have to accept and then cancel the ride. They can just pass up the ride completely. So it's actually a nice little experiment within the experiment. We shouldn't find an effect of names on cancellations for Lyft. And in fact, we don't. And also, the driver network is much thicker in Boston than in Seattle. So in Boston, although we found this cancellation effect, we didn't find that it has a measurable impact on how long you wait. And this is somewhat speculation. But we speculate that that's because the driver network is so much more dense in Boston that although you were canceled upon, there's so many other drivers nearby that it doesn't lead to a longer wait time. How do you think what you found compares to hailing traditional cabs? We started our conversation talking about the vast body of research that shows how difficult it is for black people to hail cabs. Yeah, we are quick to point out that we are not at all saying that Uber and Lyft are worse than the traditional status quo system. And we want to definitely make that clear. In fact, in Seattle, we had our same research assistant stand at the busiest corners and hail cabs. And what we found there is if you were a black research assistant, the first cab passed you 80% of the time. But if you were a white research assistant, it only passed you 20% of the time. So just like the previous literature has found, we found discrimination with the status quo system as well. You've talked to the companies about your findings. What has the response been? So that's been actually heartening. So both companies reached out to us very quickly. And we've had continued conversations with them. And we're actually trying to design follow-up studies to minimize the amount of discrimination that's occurring for both Uber and Lyft. But those are off the record. So we're not talking specifics. But what I can say is that the companies understand this research and they definitely want to do better. In fact, the companies both have issued statements about this. The first one is from Lyft. We are extremely proud of the positive impact Lyft has had on communities of color. Because of Lyft, people living in underserved areas, which taxis have historically neglected, are now able to access convenient affordable rides. And we provide this service while maintaining an inclusive and welcoming community and do not tolerate any form of discrimination. Uber has also responded. Ride-sharing apps are changing a transportation status quo that has been unequal for generations, making it easier and more affordable for people to get around. Discrimination has no place in society and no place on Uber. We believe Uber is helping reduce transportation inequities across the board, but studies like this one are helpful in thinking about how we can do even more. So let's talk about solutions to this. What do you and your colleagues who undertook this research suggest? So we've been brainstorming. We don't know for sure if we have the silver bullet. But a few things could change. For example, you could imagine Uber and Lyft getting rid of names completely. We realize that that has a trade-off in the sense that it's nice to know the name of the driver. Sure, you can strike up a conversation. It makes it more social, it makes it more personal, more peer-to-peer, if you will. But it would eliminate the type of discrimination that we uncovered. Another potential change is to delay when you give the name to the driver, so that the driver has to commit more to the ride than he or she previously had to. And that may increase the cost of discrimination. So that would be changing the software or? Right, so you could imagine now, like I said, with Lyft that you see the name right away. Maybe you wait until they're 30 seconds away from the passenger before you give them the name. What about the dawn of the age of autonomous vehicles? Might that have an impact? We already know that Uber is experimenting with driverless cars in Pittsburgh and Arizona. That would obviously solve it. So that would take the human element out of things. And it's important to point out that these are the drivers that are deciding to discriminate. So if provided you didn't write the autonomous vehicle software to discriminate, you would know for sure that that car is not going to discriminate. What about a driver education campaign? Do you think that would make a difference? I'm reminded of an essay written by Doug Glanville, who is an ESPN commentator and former pro ball player. He writes on talking about his experience being denied service by an Uber driver. The driver had concluded I was a threat, either because I was dangerous myself or because I would direct him to a bad neighborhood or give him a lower tip. Either way, given the circumstances, it was hard to attribute his refusal to anything other than my race. Shortly after we walked away, I saw the driver assisting another passenger who was white. Well, we all hope that information helps and eliminates discrimination. It's certainly possible that Uber and Lyft could have a full information campaign, where they show the tip rates for different ethnicities. They show the bad ride probabilities for different ethnicities. And my hope is that once the drivers learn that there aren't differences across ethnicities, that the drivers would internalize that and stop discriminating. Policy. Senator Al Franken has weighed in on this urging Uber and Lyft to address your research. Do you think that there could be policies too? Does government have a role to play? Potentially. But what I'll say, again, is that Uber and Lyft, I think, have all the incentive in the world to fix this and that they seem to be taking active steps to fixing this. So what could policy makers do? They can, obviously, it's already outlawed. They could come down and maybe potentially find the companies if there's more evidence of discrimination. But I would at least allow the companies some time to internalize this research and respond to it and see how effective they can be. Many think tanks and government advocacy groups have weighed into the MIT Sloan Expert Series recently sat down with Ava Malona of the Massachusetts Immigrant and Refugee Coalition. She will talk about this research in the context of immigration. Let's roll that clip. We're an advocacy organization and we represent the interest of foreign born and our mission is to promote and enhance immigrant and refugee integration. So anecdotally, yes, I would say that the research and given the impressive sample of the research really leads to such belief that, you know, discrimination is still out there and there is a lot that needs to be done across sectors to really address these issues. We are really privileged to live in such fantastic Commonwealth with, you know, the right leadership and all sectors together to really making our Commonwealth a welcoming place and I do want to highlight the fantastic role of our Attorney General for standing up for our values. But, you know, Massachusetts is one state and it could be an example but the concern is nationwide given a very divisive campaign and also not just the campaign but also what is currently happening at the national level that the current administration is really rejecting this welcoming effort and the values of our country as a country who welcomes immigrants. All sectors needs to be involved in an effort to really making our society a better one for everyone and it's going to take political leadership to really set the right tone, set the right message and really look into the integration and the welcoming of the newcomers as an investment in our future of our nation. Uber and Lyft have an opportunity here to provide leadership and come up with promotion of policies and that integrate the newcomers or that are welcoming to the newcomers, provide education and training and train their people and as troubling as this, as a result of this research are, we like to believe that this is the attitude of the drivers but not really what the corporate represents so we see an opportunity for the corporate to really step in and work and promote policies of integration, policies of improvement and betterment for the whole society and provide an example. Let me say thank you to Professor Kanido for his leadership and MIT for always being a leader and looking into these issues but if we can go deeper into A, the size, B, the geography but also looking into a wider range of all communities that are represented in looking into the Latino community, looking into the Arab communities and other parts of the nation and the more rigorous, more deep and the larger size of research will be very helpful in terms of promoting better policies and integration for everybody who truly is America to be their home. That was Eva Malona of the Massachusetts Immigrant in Refugee Advocacy Coalition. Chris, are you confident this problem can in fact be remedied? I think we can do better for sure and I would say we need more studies like what we just performed to see how widespread it is. We only study two cities. We also haven't looked at all how the driver's race impacts the discrimination. Now we're gonna turn to you. Questions from our viewers. Questions have already been coming in this morning and some overnight. Lots of great ones. Please use the hashtag MIT Sloan Expert to pose your question. The first one comes from Justin Wang who is a Sloan and MIT Sloan MBA student. He asks, what policies can sharing economy startups implement to reduce racial bias? Well, I would say the first thing that is to be aware of this. I think Uber and Lyft and Airbnb potentially were caught off guard with the amount of discrimination that was taking place. So the research that we've performed in the research on Airbnb gives new startups a head start on designing their platforms. Just knowing that this is an issue. Knowing it's an issue and potentially designing their platforms to think of ways to limit the amount of discrimination. Another question, did you look at gender bias? Do you have any indication that drivers discriminate based on gender? So we did look at gender bias. The experiments weren't set up to necessarily nail that. But one thing that we found, for example, in Boston is that there is some evidence that women drivers were taken on longer trips. Again, both the male and the female RAs are going from the same point A to the same point B. That was a controlled part of the study. That was a controlled part of the experiment. And we found evidence that women passengers were taking on longer trips. And in fact, one of our RAs commented that she remembers going through the same intersection three times before she finally said something to the driver. And you think, so you didn't necessarily study this as part of it, but do you have any speculation, conjecture about why this was happening? Well, there's two potential motives. One is a financial motive that by taking the passenger on a longer drive, they potentially get higher fare. But I've heard anecdotal evidence that a more social motive might also be at play. For example, I have a colleague here at Sloan who's told me that she's been asked out on dates three times while taking Uber and Lyft rides. So drivers taking the opportunity to flirt a little bit. Sure. Another question, can you comment on the delete Uber, the hashtag delete Uber campaign? This, of course, is about the backlash against Uber, responding that it was intending to profit from President Trump's executive order, the banning immigrants and refugees from certain countries from entering the United States. Uber maintains that its intentions were misunderstood, but it didn't stop the hashtag delete Uber campaign. Yeah, I haven't followed that super closely, but to me it seems like Uber is getting a bit of a bad rap. One potential reason why they allowed Uber drivers to continue working is that maybe they wanted to bring protesters to the airports to protest. So from that perspective, actually having Uber and Lyft still in business would be beneficial. Another question, did your study take into account the race of the drivers themselves? So we actually were not allowed to. So anytime you do a randomized control trial in the field like this, you have to go through a campus committee that approves or disproves the research. And they were worried that if we collected information on the driver that potentially Uber and Lyft could go back into their records and find the drivers that discriminate and then have penalties assigned to those drivers. So it just wouldn't be allowed to? At least in this first phase. Yeah, they didn't want us to collect those data. Last question, we have time for one more. Why aren't there more experiments in the field of applied economics like this one? That's a good question. That's a great question. And in fact, I think many of us are trying to push experiments as much as possible. My other line of research is actually in energy and climate change research. And we've been designing a bunch of experiments to look at how information impacts consumers' choices in terms of what cars to buy, how it impacts their use of electricity at home. And experiments randomized control trials actually started in developmental economics, where MIT has actually pioneered their use. And again, it's the best way to actually test, the most rigorous way to test whether intervention actually has an effect because you have both the control group and the treatment group. So why aren't they done more often? Well, it's tough. Often you need to find a third party. For example, we didn't need a third party in the sense that we could just send RAs out with Uber and Lyft. But if we wanted to do anything with the drivers, for example, an information campaign, or if we wanted to change the platform at all, we would have needed Uber and Lyft to partner with us. And that can sometimes be difficult to do. And also experiments, let's be honest, are pretty expensive. Expensive because you obviously weren't partnered with Uber and Lyft for this one. Right, but we had research assistants take 1,500 Uber and Lyft rides. So we had to pay for each of those rides, and we also had to give them an hourly rate for their time. Well, Chris Connell, thank you so much. This has been great talking to you, and you've given us a lot to think about. It's been fun. Thanks for having me. And thank you for joining us on this edition of the MIT Sloan Expert Series. We hope to see you again soon.


Yn gyhoedd Cahoosh i'r cyfrant, ac rydyn ni'n gwybod y prymt, rydyn ni'n gweithio'r piwyd, ac rydyn ni'n gwneud yw'r nid, rydyn ni'n gwneud am gyfoedd. Rydyn ni'n gwneud am gyhoedd, ac rydyn ni'n gwneud. Rydyn ni'n gwneud am gyd, Gwaith am ymddangos ni? Oedden nhw'n ddiddorol. Yn ymddangos ni? It. I. I. Y'r ysgweithiaeth u'r ysgweithiaeth. I. Mae hyn yn ychydig, ddiddorol y promp. Mae hyn yn ymddangos ni. Mae hyn yn ymddangos ni. Yn ymddangos ni. Mae hyn yn ymddangos ni. Mae hyn yn ymddangos ni. Are there anyone still trying to log on? Or is everyone who wants to be on on? Are you okay, Cezaun? As you can see this is very IT Wi-Fi dependent so it might knock off a couple of lights there, Cezaun, would you mind? Now you have an option. When you're doing this with 400 people it takes a while for them to log on. You might not wait. You might move on or somebody might come in later your lesson. You have an option to keep the pin, the game pin, visible the whole way through. So somebody who's late to it can log in or if the Wi-Fi, if the access throws them out again they can get back in again. Okay? Yeah? Sorry. That's all right. Happy to go? Everyone on? Yeah? Okay. So what I thought we'd do is we'll do a short little cahoot about a subject right close to my heart. And I want to assess your knowledge of this subject. So I'm using this as if I was perhaps going to teach around this topic down the line. And I just want to source out where you're at in terms of your knowledge, your prior knowledge first. And it also gives me an opportunity or gives you an opportunity to see where your own knowledge and learning is in comparison to everyone else. Because part of the problem in the big class and the nanny class, if people are afraid to ask a question or say so, it's because they're afraid to be wrong. And they don't realise there's probably 40 more in the room also thinking the same thing, you know, where you can see that. So this formative assessment for me as a teacher, but it's formative assessment or knowledge acquisition for the students as well because they can see, gosh, I'm not the only one who taught that. Okay. So we'll get going. Seven questions. Are you ready? Now this is timed about 20 seconds for each one. So the clock ticks down. It's coming out. It should be coming up in your smart device if you're logged in. Is it coming up? Okay, let's see how we're doing. Oh, Paul, you're top of the class at the moment. So meatloaf won a Grammy for best solo rock performance for which of the following songs. Bat out of hell, two out of three ain't bad, dead ringer for love or I'd do anything for love. So you're looking at the answer on the screen and clicking the symbol on your phone. Oh, now, wow. Okay, so obviously when I'm planning teaching my module on meatloaf, I'll be doing a lot of work around his songs, the awards he's won. Look at the assumptions that are being made there by most of my class. Are you going to sing? Nope. Anna, you're leaving the posse there. Which of the following characters did meatloaf play in the Rocky Horror Picture Show? And you'll hear, I'll turn it down now, Pip, but I just want you to hear. You can hear the countdown noise. Now, do you see it stopped at three seconds? It stopped at three seconds because you'd all answered. It didn't go to the full 20 because you were finished. Paul, passing out Anna again. In two out of three ain't bad, what were the two things? To want you in love. What are the two things? Sorry. Very good. I'd do anything for love, but I won't do that. Won't do what. Say I love you, lie or cheat. He actually says a rude or thing. Then, as far as I'm supposed to hear. It's just no one. I get it. It could be our wife after you've hit it. You're waiting for all the other responses to come in, and if all the responses are in, then you'll see the answer. Yes, or if the time's out, and then you'll see which one you got right. Some of those questions you answered very quickly, so we didn't even count down 20 seconds because you would all answer. Now, when you have a group of 400, you go to 20 seconds. You can time it for longer. Depending on the question, you can give them a minute, you can give them five seconds, but for reading, and I always read the questions as well as put them up for people who may not be quick enough at the reading end of things. That can put off other people though. Six out of seven. Which of the following wrote most of Meatlo's songs? Meatlo himself, Jim Steinman, Chris Christofferson or Bert Bacharach. Very good. Lisa, fastest finger on the button. And Meatlof is the greatest singer of all time, the greatest performer of all time, or the greatest artist of all time. You're all right. So, let's see overall. Lisa, if you come out on top, top scorer. So overall.  It's a lovely photo of Meatlof. Well done. OK. You can then ask for feedback and results. So you can ask the students to rate what they've just done, the learning, whether they feel they've learned from it, whether they're recommended to others and so on. OK. And that can, you can download your results later. The cahoote will store results. I think it's the last three times you've run the same game. It'll store the results. And you can see which questions overall were answered correctly and not. It also, I always use it with nicknames. I ask the students to put in there. Because the whole reason I'm using it is because they don't want to speak because they're afraid to be wrong and be identified or whatever. And you couldn't rely on it to use for attendance or something like that because somebody can be logging in as someone else. You've no way of doing it. And it just gives them a safe space to answer questions that you've given them or whatever. Sorry. I don't know where I am now. Quick. You know, it depends how long. Sorry guys. It takes. Sorry. Let me just find my little thing here because I'll go through that with you now. Yes. It depends on what you're using it for and what the quiz is. I use that there at the beginning of a session to ask you a few quick questions which might help me then decide on how I'm teaching a particular topic or whatever. And you know, it depends on what you're asking them the length of time it takes to put together. You type in your questions. You type in your range of choices. You can then drag in images which I did for most of them. You can connect to YouTube and you can take a piece of film clip and you can decide how much of that clip so you can edit the number of seconds that you are showing like I did with the meatloaf song there. You can edit how much of that you're going to show and then decide how much more time following completion of that YouTube video that you will allow them to answer the question. Okay. So I can't give you an answer in terms of how long it takes but I find it quite efficient and once you've made it, you've made it. You previous class. Yes. So I've used it in a few different ways. I suppose just to give you, this was the reason I'm not the most technologically advanced person but in the large class, though that be a group, we had done the postage ideas in terms of gathering information about attitudes and beliefs. We had done the co-teaching in terms of getting their voices around particular threshold concepts. This is a third approach that we're using in those taught modules to get the voice of the student and to formatively assess what they understand. In order to do this, I got some money thanks to the Teaching and Learning Committee and Anita and I went to the guy who created it who is a third level lecturer in Norway. He lectures in computer programming so he has a background in computer-y stuff and he developed this because he has 2,000 students in his first year module divided into four classes, three classes of 600 and one of 200. And he devised it for the exact same reason that I'm using it, it's for formative assessment and to get the student voice. And it's now supported by, as far as I can work out, by a company called We Are Human in London and they support this particular programme. It's free, it's free to use, any of us can access it. The largest class, one of the things I wanted was a programme that would be easy to use with a large group. Some of the programmes are limited by size. You can only use it with 50 or 60 or whatever and I needed it to be big enough to carry three or four hundred students easily on it. And this does that. The largest group I think, I'm not 100% sure, I think it's a thousand. It's the largest group that I've done a cahoot but I could be wrong on that. There's a huge amount of information on just Google get cahoot and log in, anyone can log in and set up an account. There's lots of information on that website and also lots of videos on YouTube by people who are using it for second and third level. There are frequently asked questions and I discovered last Sunday when I was preparing this there is the support includes, now it's not live but you can email a query and they come back to you like on Sunday they were back to me within a couple of hours on a Sunday evening I was amazed with my query and I'd never used that facility before. So it's just very well supported, it works well, the students love it, they absolutely love it. I used it for the first time with my first year be ads on Tuesday and it just lit up the room, they loved it and I had a little prize as well, the other thing if you go buying a prize the first time you need to have your prize every time.  It's just another way to get the voice of the student I wouldn't overuse any of these, I wouldn't say co-teaching is great, it works, let's do it and all our sessions, that's overkill. Likewise for cahoot, it is just a tool to get the voice of the student, I wouldn't overuse it and if lots of other people start picking it up and using it across our BN programme I might reduce my usage and move into something else because you want to show, again going back to what we were saying earlier about explicitly modelling for our student teachers a range of approaches to access learning or to enable learning and not just rely on one great, it works, let's keep using it, it becomes boring then and it loses the edge that it has. I wanted also to get feedback about what they understood in live time, in the classroom and so on there's other ways you can get feedback on what they understand but it's not on the spot in that minute in that 15 minutes that you're standing up in front of them and I wanted them to see where the things they understood how that matched with what other people understood that they could contextualise their own understanding and actually feel a bit safer around the things they didn't know or didn't know. Nine of you got the same wrong answer in one of those cahoots so there's a real safety in seeing that and understanding actually, we haven't learned this so how would we know it? and it's fun, okay? You're not standing there talking at them it's interactive and they can and they love it they really enjoy using this. So to set it up, somebody's asked me how to set it up you go to getcahoot.com and you register so you need your username and a password I think your email address will be fine as a username and then you set up your own password. There are three modes, okay? You have quiz mode which I've just used with you and with quiz mode you're at tributed points and cahoot seems to work it out on speed and accuracy now when you have, I had a student come up to me last term he said I'm really worried I came 252nd and I said no you didn't cahoot can't give a tie so it makes the decision so 50 people get an answer right in the same speed it just makes the decision it sieves them out into some sort of an order so that was something I hadn't even thought about and you know so now I'm clear don't be worrying if you get 200 and something that does not mean you're pitching in at 200 and something in the class and you can set the timing so I set 22nd timings there you can set them for longer if you wish your question to be longer so every, when you type in your question and your answers you are limited in the number of characters and it's a fair limitation it's a good limitation in terms of what people can read if you wish your question or your scenario to be longer you need to do it on a PowerPoint save it as a PDF and drag it in as a picture but you need to give the students much more time because they have more to read okay so you need to do your setting for a question like that in term maybe a minute because you want to incorporate how long it will take them to read it as well as to answer it, okay and I, as I said, I read the questions as well as I read them allowed to help people who maybe might not be as quick at reading or different number of students with dyslexia and that sort of thing in the classes I've also used survey mode survey mode it's very similar to be honest but there's just no, it's not a competition it's not a game so they, using survey mode you might use that to survey opinion attitudes, beliefs I might not use my postage thing anymore and use cahoote instead to get those the only problem with that is that I then need to list the attitudes I'm not getting it from them I'm making the choices and that's the only thing whereas when you ask an open question that people can write anything down to on a postage I get more important feedback on that okay Can you retrieve the data? Can you retrieve the data from the survey subsequently or is it just in the moment? No, you can retrieve it subsequently and I think my understanding with cahoote is that for each, so I've just done a game with you it will hold the results for the most recent for the last three times I played that game so your cohort won, I might do it with another group and then another group and then when I go to group four your data will disappear because you were group one relatively recently but they know what's there for all your results they're constantly, it's a very they're still working on it and there's new things coming in on cahoote all the time but it's easy to It allows you to download it in an Excel spreadsheet or indeed put it into your Google Drive as an Excel spreadsheet as well go straight to it Sorry So it allows you to save all your results and it saves them either as an Excel file or straight into your Google Drive and then you have it forever more then Yeah, Pip My question is about equity Do you find any issues with students not having access to devices so they can't participate in cahoote in class? Yeah, I asked because sometimes I can see I have three or four hundred in front of me and there's only 200 logged on and it's not just about equity some people may not have their device with them or have our device and what I say to them at the beginning is if you don't have a device or if you don't have access, pair up with the person beside you and between you make a decision on what you think the answer is The bigger issue though is electricity So for example there are three ways that I've used cahoote I've used them as I did with you just now which was quick quiz to see where you're at and now I'm going to teach you about meatloaf and so that's I'm gathering your prior knowledge in order to think about what I need to focus on and so on rather than assuming you know everything or know nothing or you use it at the end to as I did with my first years on Tuesday so I asked them seven or eight questions I had done two 50 minute sessions on general learning disabilities mild, moderate, spear and profound learning disabilities and at the end we had a quick cahoote on a couple of key concepts out of that to see what they had retained and so that they could see to do it at the end or somebody mentioned you could do the general learning disability at the top of the next session just to remind them and then move on to the next topic and that's another way another really really good way to use cahoote is instead of your PowerPoint so cahoote is your teaching tool or your visual and you survey mode is quite good for that or quiz mode you can use whatever mode you want and you put up a statement whether it's a question or that has right or wrong answers or whether it's a question that has no right or wrong answers it's values, opinions and beliefs and then you move to teach and work around that so it's the whole session cahoote is carrying your instead of the PowerPoint you're getting their minute by minute opinions on these key concepts that helps you teach that idea loads them didn't log on for that or if they logged on they fell out of it again one of the students after it I said why do they not they know where to plug in their phones they didn't want to run out of battery it was pure technical practical things and she said you're much better off more people will log on if you do the quiz quickly so it's just there's nowhere to plug in with nothing under our desks where they can plug in and you're on for an hour and they're starting to lose connectivity especially on your phone where you're linked to wi-fi which eats your battery so it's never been thought of it so they didn't log on and it's a pity and maybe it's something that organisations like this need to think about because actually it's probably the best way to use cahoote whether you use whatever mode you use I've never used discussion mode but I have looked at it and to be honest I find very little I can't see any difference between discussion and survey mode I just can't see what the difference is myself it can be used with loads of different groups big groups, small groups you know and for android devices they can download an app and that's a relatively, that's not a new thing that app has only appeared in the last while they are developing to the best of my knowledge don't know where it's at they're developing one for apple devices it's not yet ready it's quite simple to use on the internet anyway I suppose I sat in on a session with Anne-Marie and it's amazing the difference like even if they had it available as a shortcut on their phone or device that would speed things up because it does take them a while to... it takes them a while to... I'm prompted there to download it as an app were you prompted? ok so that's actually new since last time that's new so they're working on it constantly or you can log in what we did was the long way of logging in but it's reliable as well but it does take 400 people a while to click, click, click type and get in to it in terms of using it I told you the modes that I used in terms of assessment for learning and it's it's really really useful on the spot particularly if you use it at the top of a lesson like we did just now because immediately it focuses me on what the students know and what they don't know or what they believe depending on what you're using or whatever it's not that I'm going to be teaching something different that I would have planned but you might focus I might divide my time in that session a wee bit differently or I might draw be more explicit in the attention that I would draw to certain ideas if I'm clear they had to, you know what they didn't get that like there was one last week on Tuesday a question that I asked and I'd say 55% got the answer wrong and I had in my head I had been very explicit about how I had taught that and I have a feeling I know one of the reasons that that they have this perception but now I'm able to go back to that highlight it again explain it because I know that they don't know and we can make those links for them so I have found it very useful like that it lends itself to certain things you're teaching and certain classes it doesn't necessarily you know as I said like co-teaching I wouldn't be using this everywhere but it's a decision about where I'm going to use this and it's my third at the moment this is my third approach to formative assessment in those big classes and I'm hoping to, the limitations of this first of all it takes a while for them to log on they get used to it so by the time you're doing it the third time they know how to do it they're popping on very quickly the ranking of the respondents and the quiz power and connectivity sometimes I think you get bounced out out there earlier so sometimes like if we're all hanging from the the wifi at the same time they're all logging on to this I don't know why but somebody gets thrown out of it again and that's why you have settings when you set up your cahoote it's really important there's a question that's and you answer yes or no to them one of them is do you want to show the game pin the whole way through the answer to that is absolutely yes you do want to show it because you want them to be if somebody comes late, if somebody gets bounced out like Suzanne did if somebody's thought they'd forgotten their phone and then they suddenly find it on question three you want them to be able to log in even if it's halfway through so that they can take part so always answer, well my opinion always answer yes to that if you're going to bring a muffin the first week bring a muffin every week for the winner for someone like me I do not have a background in technology and I'm quite nervous of hitting buttons in case things explode and all of that sort of thing there was no substitute for going to that university and seeing 600 students do that and I found places nearer to me that were using it were groups of 40 and 50 I said no I want to see how this works with this big group and it meant that when I did it for the first time I was nervous because I'm not technology and me aren't the best buddies but not as nervous and I wouldn't have been experimenting on my students the way I would have been if I had not seen that in action in a similar setting to mine it was invaluable to have done that so anyone who wants to come and see it used here in a big class you're more than welcome there is no substitute and you sit up at the back of the room and you're looking over student soldiers and you can see what they're doing you can hear what they're saying even though in my case it was in Norwegian so I was extrapolating what they might have been saying but I could see the engagement sitting up at the back and as I said I used to share a device and work together I tend to read the questions for them because I'm afraid that there will be people in the room for whom reading is an issue and you need to just support them we've had we'll be talking about spiralling assessment in a minute but we have one group of students who presented to their peers in fourth year and they used CAHOOT as part of their presentation so they had taken it from what they'd seen and they used it and you can do survey during quiz mode so say like my last question was is Meatloaf the best singer the best performer there's no right or wrong answer there they're all right so give everyone a mark that's an opinion that you're asking for or a belief but you can do it in quiz mode rather than hopping from one to the other if you so wish but in that if there's no right or wrong answer everyone gets a mark so all answers are right it's the way I would do that as opposed to all answers are wrong and you have to say in fact you can't have all answers wrong you must that CAHOOT will not allow you to move on to the next question when you're setting it up until you attribute a correct answer okay so this is just one tool and it's just a tool in and of itself CAHOOT doesn't do anything and you need to think about that and you might need to try it at the top of a topic or at the end of a topic or you know how you might use it the biggest limitation is at the moment it's a one way Q&A I ask the question they give me the answer so my next thing now for my large classes is to find something that will turn the questions back ways without electronically 400 people shouting at me so I want something that will do what CAHOOT does but that students during my session can log in and ask a question in the same way like what happened here today is that as we move through somebody put their hand up now I don't know what they're going to ask and I don't know their name but I want a programme that will allow students to put their hand up without writing reams up here distracting from what's going on but that I can hop in and ask random people questions and possibly anonymously as well so that they feel safe and asking the question and CAHOOT at the moment doesn't do that so that's the next thing I'm going to investigate and I want to include that in some of my sessions again last term I only used this last semester for the first time I was not explicit at all about how I was using this for me but this term I am I'm telling the students I'm using this as formative assessment you will hear Zita talking about assessment and how to assess formatively in the moment that's what I'm using this for and I want you to see yourselves what you understand and how that sits and fits with everybody else and I also find just try one thing I was all gung ho I had about a list of about 15 different ed tech great ideas and I was about to do you know what no I'll do one of these last semester this semester I've got to continue with this and refine it and I might try one more this semester and we get that right and then I'll move on to the next one and that's what for me that's what I need technology to do on my cloud I don't want to use it just to be saying I'm using technology I only want to use it to replicate the small class characteristics into the big classroom that I can replicate that and let students be safe in answering or asking questions at the same time without 400 people shouting at me albeit in typing mode so that's where I'm going to go with it from here, no research done on this this is 100% practice based and a very early practice and I've only started so and there may be people in fact if you go on to YouTube you'll find lots of people who are using this and lots of different settings and some of the videos are really really good and very very informative so are there any other questions around that just grab the mic comment really I was teaching the same group of students on Tuesday just after Anne-Marie had finished and I have been trying to engage them using formative assessment over the last couple of weeks and I felt that I had flipped it to the point where they were now sick of me engaging them and I needed to get some feedback so I asked them to give me a star and a wish in relation to what I've been doing to say they blew my mind with all the negatives I have to say I needed a very strong gin and tonic afterwards which I have to say is important for two reasons one because I think we need to be aware that we are all teaching potentially the same group of students so the balance and the kinds of engagements we are requiring at different times of the day may need to be considered we need to talk to each other the second thing is I think that there is a backwash effect from their experience of the teacher-student relationship and they can take so much formative assessment and then they fall back and say I want you to teach me and I want you to leave me alone so I really got that back in the formative feedback that I got from them but a part of that and I said this to Annemarie earlier on just to reinforce what you're doing a number of the students said please try Cahoot so not a huge number but a number of them did having watched, it's really helpful Annemarie to be able to sit here and to get feedback from you I would launch in two things strike me and one is a positive which I think responds to what you were saying there which is that we often say that if students really know something then they can set the assessments so the potential for students to assume your role and to identify certain questions for their classmates might be an interesting way of using Cahoot the second one which leaves me a little bit concerned is the lack of diagnostic information so even I'm thinking of simple things I use thumbs and I use talk partners and I use a variety of things with the 400 students and frequently if I ask them a question Cahoot isn't terribly dissimilar to thumbs which is very quick and doesn't require any technical intervention and you can follow that very quickly by inviting students to have a talk partner to talk to themselves for a couple of minutes and then to give you a muddy part on a post-it which you can communicate very quickly and they will do that because it's anonymous but I'm just thinking that and this is my question to you is there a diagnostic element to this that I'm not a diagnostic element in terms of the results of what you've answered so you can get some great quiz results so you can download the results so I'll show you why I was emailing Cahoot on Sunday because I couldn't work out how to do it even going on YouTube I couldn't work out so let's say this is question one can you all see that now? that might be a wee bit small so I can see here the percentage who got this question correct is 41% I can see the total number of students who answered it and the average speed and I can see the spread so I gave them four choices one of which was correct so I can see that 41% got it right but actually nearly 70 there 70 people miss didn't 70 out of about 140, 150 more didn't get that so now I know is that what you mean? I see that information that begs more questions than answers but it tells me that they've got things wrong but I don't know if it's because they didn't think on it or they responded quickly and I'm not getting that feedback in the moment so in terms of me using that to direct my teaching in the moment which is the key issue in relation to formative feedback it doesn't give me that information that are required there and actually it's useful as a starting point I think and it may trigger something then like talk partners or something I think it would be useful from that point of view but I think it's a serious limitation of the tool that it doesn't give the kind of information that it would that it's sufficiently informative Do you mean why they think this? No but even afterwards I'm reading because that information is interesting but it doesn't tell me anything about the way they're thinking is that they've got something wrong You don't know why they're thinking or it doesn't also it doesn't also challenge them to stop and to critically review their responses and maybe then say well here's where I'm I'm stuck I would argue with that in favour of what Anne-Marie was saying earlier on and indeed the conversation on something with Anna it is just a tool and what it's doing is it's giving you the student voice it's how you use it as a tool and you use it once you leave with pedagogy you don't ask them like 10 questions in a row you ask them one question you then talk about it, talk about why this answer is right or why this answer is wrong and then go on with your teaching and maybe 10 minutes later ask another question you don't have the entire class full of those questions once right after the other you give them time to reflect on it I would also say from a teaching perspective you would use the teach-per-share think-per-share method where you get them to have a chat with one another first to think about it and then to maybe iron out one another's understanding and then answer back again and then you get much better two things happen, you get much better responses from the class but also what you get and it was something we were saying earlier on you don't get students getting distracted with their phones and I'm going to go on Facebook here because if I'm doing it with Anita if I go on Facebook well I'm meant to be talking with Anita about that I'm distracting Anita too so they're nearly self-policing one another in that respect but it comes back to the fact and you touch on it right on it is a tool to help you with your teaching it's not the panacea the perfect answer I'm just thinking is that they've got to use the technology so there's that time when they've got to connect and they may be flipped out or whatever else I'm just wondering what are the advantages of this apart from the record over and above for example asking students to do thumbs in response to as an alternative to white cards when you've got a series of multiple choice questions and following that with the think pair share because you've got to visual immediately and then potentially you can say talk to the person beside you or that person up there has something to they're here and that's the correct answer talk to them and see if you can talk with each other there's a transparency in these kind of methods even though they seem terribly simple there's a transparency factor there are people who want to talk I agree to a point but I think that the tool and engaging technology in the class it's what the students are using in day to day life they're using their phones you've got a degree of engagement that you don't see with just putting up your hands there's an excitement with using a new kind of a tool and something I said to Emery earlier this cuts across all kinds of students as regards engagement and participation I've used it with adult learners Emery's used it here with your student teachers and my son is using it in secondary school and in all of those there's a huge degree of excitement and acceptance of it so you're getting an engagement an interest at least in getting involved it's only part of the discussion it's only part of the learning but they're interested in kicking off that I suppose just while you're passing the mic over one of the ways as I said earlier I think probably the best way to use Cahoot is to support your entire session and in that way you do get to go in a little bit further into those to work out why we think this if you carry it through because of the feedback mind you only from a small number of students but because of the feedback from a small number and also the fact that they logged out again I was down to maybe 100 out of 300 or 400 on my Cahoot because it was too long but if you have it on for 50 minutes and if you're using it or if you're using it like that is to explore because now you're getting what they understand now you can go back and say I wonder why you understand this have a chat with a person beside you write it down on a piece of paper or say it to me or whatever and you absolutely could do that if you carry the Cahoot through the session there's the practical stuff in the session itself just to kind of maybe respond to the question there that I think is happening is that you get that kind of feedback on the screen where you get out of say 300, 150 yen for this 250 yen for that or whatever the students are also getting the feedback on their phone I use this in a classroom setting of 20, 30 students you can immediately address the concern if you say right I would have thought all students should have got that right and I can see here 8 out of 10 what were the two thinking you could open up a discussion forum there now I'm not saying that you could necessarily do it with 600 students in front of you I use this actively in the classroom and there and then on the spot you can address the issue and get a bit of richness and depth and conversation maybe that would address the issue that you have you're asking there one of the things I'd like to add is that I also have used it in the flip the classroom where you get the students themselves to go away, you set it as a piece of homework they must log on, set up their own account and make their own quiz and one of the benefits of that to actually come up with three wrong answers which is typically what they're doing in related to the question or the topic and one right answer is actually quite challenging and quite difficult so that in itself is a whole area of learning that's opening up there for them to come in on the day and be the teachers is incredible so if you've typically groups of three, four of them standing up there they have to come up there, they have to log on they have to set up the quiz, they need to address their audience all the skills that they're learning there is incredible so I have found that the benefits of that alone is amazing and from a language point of view one of the uses that I have used when I'm setting up the quiz the questions myself is that the language that you use for particular covering content knowledge how you can actually word a question in one way and then another different way even though it's covering the same content knowledge is amazing the response that you get the difference in terms of answers I've found it's gone from 100% correct to 2 out of 10 getting it right just because I changed the language of the question so there's a lot of ways that the tool, how you use it is huge it's just growing all the time That's a learning curve, you work these things out as you like I've used this in total four times three times with my fourth years and once with my first years so that's and it takes a while to learn your way and to feel your way through it OK Thanks, look I'm going to pass over to Anna and Fiona I'm going to pass over to Anna and Fiona I'm going to pass over to Anna and Fiona Lad, would you please eat all the pastry so get up and help yourself to tea and coffees as we go OK