Can everybody hear me okay? Great. Well thank you so much for being here early on a Friday morning and I think we'll have some more folks straggling in as we go along and make our way through today's conversation. I'm really excited about today's conversation. First of all, I'm Luke Hardik. I'm a fellow here at New America and the International Security Program. And you just couldn't pick a more timely topic for today's conversation which is a new Mercy Corps and Yale University study something that's really confounded us for the past 16 years and beyond which is how can economic interventions and aid be channeled to prevent violent extremism and political violence more generally? Of course this question has been central to our struggles in Iraq and Afghanistan over the past 15 years and more. We've pumped billions of dollars in those countries. We've flooded the places with civilian experts. People like General Petraeus have even come up with clever approaches like doing public opinion polling to get a feel for what's driving instability. And yet the results there, I think, for being honest, have been uneven at best. And so as the United States recommits and re-ups its commitment to Afghanistan, these are the kinds of questions I'm sure policymakers are struggling with today. It's also been central to counter-terrorism thinking over the past few years, particularly through the leadership of Sarah Sewell and she was at the State Department, the agenda preventing violent extremism, that's to say, channeling humanitarian aid and development assistance to the communities most at risk for violent extremism has been a big part of how counter-terrorism policymakers have thought about things. Of course this is also central to our work on fragile states and I think maybe most relevant here, some of the humanitarian work that Mercy Corps engages in and supports every day. And according to the World Bank, violence and violent extremism and violent conflict are now the leading cause of displacement worldwide driving 80% of humanitarian needs. And I think even more troubling, it's the same conflicts going back to 1991 that are driving the overwhelming majority of those displaced persons of refugees. And so we couldn't pick a more important time to try to figure out how do we actually get at the root of preventing this kind of violence and the negative effects that come from it. And yet with all this there's a big question that we haven't satisfactorily answered which is what actually works? And so fortunately Mercy Corps and Yale are here today to help us with that. There's of course, Mercy Corps needs no introduction with this crowd but a long experience of working in conflict zones. And it's one of the recurring cases of them presenting research here with New Americans so we're really glad to have their partnership and thinking through these important issues. And today we wanna talk about some of the incredible work that they've done in a study they're rolling out yesterday I think was the first day I rolled out on economic interventions in Kandahar specifically and the lessons learned for policymakers who are hoping to utilize these interventions to prevent violence. So let's introduce our guests. First of all, I have with me on my left Rebecca Wolfe. She was one of the lead researchers from Mercy Corps for the study. She's the director of evidence and influence on matters related to peace and conflict for Mercy Corps. And she has also been for the past year a fellow at Yale University's program for refugees, forced migration and humanitarian action and the political violence field lab. Next to her is Jason Lyle who was the principal investigator on this report from the Yale side of the collaboration. He's an associate professor in Yale's department of political science and director of the political violence field lab at Yale University. He also has a pretty extensive background as a practitioner and doing field work. He worked with USAID in Afghanistan from 2012 to 2015 on measuring the impact of stabilization initiatives. And responding to the study they've done is John Dempsey and I feel like JD needs no introduction to anybody who's closely followed Afghanistan policy for the last several years but I'll give him one anyways, which is he's been working on Afghanistan since 2003. He lived in the country from 03 to 09. He opened the USIP office in Afghanistan back then. And then in 09 he shifted over to working with the US government, working first for Ambassador Holbrook in the office of the special representative for Afghanistan and Pakistan and then for Ambassador Holbrook's successors following his tragic passing. So John, somebody who's really seen this at multiple levels, he's now a fellow here at New America and he's working on a study looking at causes of extremism in eight provinces of Afghanistan for the Open Society Foundation. So couldn't ask for a better crowd to discuss these issues. First of all, in case it's not apparent, all of this information is on the record so feel free to utilize the information here today. And I thought what we would do in terms of setting a quick agenda for us is we'll ask Rebecca and Jason to talk through their research. We'll give John a chance to reply to the findings they present to us. And then we'll allow them to converse a bit on this topic and we'll be sure we open it up for questions and leave a substantial amount of time for that at the end. So without further ado, let me turn it to Rebecca and Jason. Okay. That sounded very loud for a second. So thank you, Luke. We're excited to talk about this research today. We've been working on this project since 2015. So it's exciting that the results are getting out there. And so just to kind of frame the conversation, as Luke mentioned, Mercy Corps has done a number of events with New America presenting evidence of what works. And so we've had this stream of research basically which first started about why young people participate in these violent movements. And actually part of that research was a study in Helmand where we did a vocational training program funded by the UK government and it was under a stabilization frame. And so while the program was focused mostly on the economic outcomes of the program, we knew the Brits wanted to see, did it have results on broader issues? And we surprised many people to find it while it was actually successful economically, which TVET programs aren't often successful, particularly in a place like Helmand, which was a relief. It had no effects on support for political violence. And that was a big surprise, especially around this time, I had been writing many proposals with the theory of change, that if we improve employment outcomes for young people, we will reduce violence. And so this started a stream of research from Mercy Corps to look at like, so what are the reasons? And then we had a report come out in 2015 that talked about the governance issues really, about grievances and people's anger was really driving people's desire to join these groups. And then we got the question, okay, so now you know why young people join these movements, what can we do about it? And so since 2015, we've been really focused at answering that question. And so this study is trying to help further, like what can we do in these fragile environments to stem the tide of people wanting to join these movements? And so we have been working with Yale and Princeton on this study in Kandahar. It's a randomized control trial and we'll talk more about it in a second. It's a BPRM program. But then we decided to ask a similar question. So what I also wanna say, what's interesting here is we're replicating studies. And so we often see in aid environments, a lot of null results are a finding here and a finding there. So one of the things we wanted to do was look at replication. But we also added another intervention to try to tease apart, I think in vocational training. We're never sure, is it about the economic outcomes or does vocational training do something else for young people? Is it the soft skills that are involved? Is it that we're bringing young people together? And so we added a cash arm to try to disentangle some of that. And so I think we'll talk a bit more about how we actually did this in Kandahar and then some of the findings. So do we wanna put up the slides? Yeah, okay. Or, yeah. The click, okay.  I'll move. Okay. Reorient. And so what I would add to this, I think one of the, what this study also adds to the literature on these questions, one, we're doing it in an active conflict zone rather than a post-conflict. I think a lot of the research we've seen there on this issue has not looked at it at participation conflict with ongoing conflict. We also are using sensitive measures. Like people often don't wanna talk about their involvement in violence. And we can talk more about how we measured it during the Q and A, but we're using measures that get rid of social desirability. And for at least, particularly for Mercy Corps, I think one of the things we often have wanted to do in our work is push boundaries on measuring participation and support for violence. A lot of peace building work tends to avoid asking those tough questions. And we have always said, if we can't actually show that these programs work on those issues, then there'll be questions about the effectiveness. And so they will seem in a sense too soft. And then, as I said, we wanna really look at, there's all these theories about economic outcomes or beliefs in the world. I said, conventional wisdoms that economic outcomes affect political ones. And I think we wanna really understand like how does that happen? And so this study is trying to help us understand that. And so, and Jay can definitely talk more about this because he was on the ground in terms of the randomization. But we did, so we did a full randomized trial. So when I talked about our Helmand work earlier, that was a quasi-experimental match design. We weren't able to randomize at that point. So one of the things we wanted to do and we thought this was a great opportunity was doing a fully randomized control trial. And so we did, we had a vocational training and then actually Jay and I presented this research design in a workshop in LA back in 2015. And everyone was like, you have to add a cash arm to be able to disentangle cash from the cash people get from employment from the vocational training. And so fortunately Jay was able to find funding to add a cash arm to the study. So that was not part of the BPRM program, just to be clear. And then we looked at the combined effects. And so people were either in vocational training for three months or six months. The cash was a one time cash transfer of $75 which is worth about three months of salary in this area of Kandahar. And that was through M-Pesa. They were told it was from a foreign donor. It wasn't, it was vague in terms of who was delivering the cash. In a sense it was just a windfall on their phone. It was, they were not told it was part of the program whatsoever. And then we did, looked at follow-ups. So we first did a collected data right after the program implementation and then up to between six to nine months later depending which T-Vet treatment you were in. So a little confusing, Jay, do you have anything to add? No, I'll wait. Okay, Nicole? Yeah, after the training. So, which is an important piece of it. And we can talk, I think there's some really interesting, we have hypotheses about given their use of the cash how that potentially benefited the T-Vet but I'm giving you a highlight role already. And yeah, people feel free to ask questions. Fortunately, I see a lot of familiar faces in the room. So the population, just so you know it was largely displaced, 51% were displaced. It was obviously a PRM program. It's gonna work with displaced populations. That basically 36% female, high levels of poverty. So these were at-risk populations. Okay, so this is in some ways again for me quite cool. So economic livelihood finds, we get replication of what we got in Helmand. So basically what you see right after the program, two weeks after, nothing's happening economically which isn't shocking, given it takes people time to utilize their training. What we do find is post-program. So that's at the six to nine month point. The T-Vet training does improve economic outcomes. So T-Vet alone, just like we got in Helmand has some economic benefits. This was quite surprising for us in a place like Kandahar. It was the drawdown of the troops. And so the economy was contracting at the point that we were actually seeing economic benefits to this population. We find in, so the blue triangle is the cash only. So the red circle, sorry, is the T-Vet. Only the blue triangle is cash only. And then the line is the interaction. And what the interaction says is basically there's no additional benefit of the cash beyond training. So it's not that having both decreases it, it's just that there's no additional benefit of it. And so again, we basically get the same finding from Helmand. So it was a nice replication. Combat and support. So then the question is for us what was really driving this study was what does this do for support for the Taliban or the government? And what we find immediately after the cash swing support towards the government compared to the Taliban. And so again, if you look at those blue triangles on that first end line, unfortunately, we find six to nine months later there's a backlash effect. And so if you see here, basically it ticks up again of support towards the Taliban compared to the government. Again, we get a replication of the vocational training Helmand. It does nothing. T-Vet alone does nothing for support for violence. What is the most for me really interesting of this is the combined treatment. And so we see post-programs six to nine months later combined cash and T-Vet has a 17% reduction in support to the Taliban. And so basically we're seeing some that there's some combination of aid interventions that are signaling that yeah, it's switching support towards the government. And so one of the questions then is why is this happening? And so we're noticing it's not, it can't just be about the economic effects because we know from the previous slide the economic effects, if it was about economic effects we would see the same results for the T-Vet only and we're not seeing that. And so we looked at basically are there perceptions of government? So is the cash and the vocational training doing anything in terms of how people perceive the government? And that seems to be what's happening. And so what we notice is we're actually breaking, Jay has broken down the national government performance and local government performance and they're finding that on certain types of performance significant results. But what we found on government responsiveness, again in this combined treatment, people see the government as being more responsive to their needs. And so what we think is happening is that this combination of cash and vocational training is signaling to the local population that the government understands their needs. If we only gave cash it's a short term benefit and it doesn't do anything in the long term. If we just give vocational training the benefits don't come till later and so people are frustrated. But if we in a sense provide a support for people in the short term to deal with their day to day as well as then give them some type of ability to meet their needs long term that signals people understand that the government is actually dealing with what they need. And what was interesting we did some qualitative work about how people use the cash and they are using it for basic needs. And so what we would have loved to be able to show which we can't but other potential for follow up work is does the cash allow people to use their time differently in those first three months that they don't have to worry about basic needs and can utilize the vocational training in a different way and potentially have longer term economic benefits than in just that alone. Yeah. Just to emphasize on the slide it really is a story about the local government. That was the one that was being seen as responsible for delivering the program on the ground as well as the cash. And so if you open up this local government performance national government performance there's actually an index of about 10 different services or goods that the governments could be providing. And the thing that's really driving that on the local one is the job opportunities and employment. There seems to be a perception among the individuals on the ground in the program that the local government is seen as bringing jobs or some kind of employment opportunities which is exactly where you'd expect this type of argument or credit capture to be holding because the people are actually seeing this program coming in. Some of the other things we asked about infrastructure, water, electricity there's no movement on that. So people are actually sort of disentangling to different pieces of the government of services they're providing and they're focusing in on unemployment which is what exactly you'd expect with this kind of. Yeah. This is something that's a little bit odd in the data. So the vocational training was delivered by Mercy Corps but it was in government run schools. And so the government would show up at certain events. But it's interesting that just the TVET alone isn't signaling it. And the cash is not, it's not, it's from a foreign donor. What we've noticed since these findings came is that there's a number of other studies that show how local populations credit their government for in a sense bringing aid to that place even if it's from a foreign donor. And so there's a study in Bangladesh, there's a study in DRC and then just within the last day there was a paper on cash transfers in Tanzania that showed increased trust in government. And so there seems to be, there is a way of delivering aid in a way that local government can capture that credit. But I think, and this would be, I hope part of the conversation is how can we do that? Because we've seen in a lot of the hearts and minds approaches that's the goal and yet we've gotten it wrong. But it seems like we're starting to learn that maybe there are ways we could do it better. I would just add to you on the, it's a great question of whether they know it's the government provided. And here the agency that kept coming back or the director kept coming back was the Ministry of Labor Affairs and Martyrs and Disabled. That local entity was the one that was getting flagged in a lot of the qualitative interviews and it's in part because their officials were inside the program. They were being seen as being there. The certificate when they graduate is given as a government document. And so there was this visible presence on the ground. Yeah, Nicole. If it was a three month like fixed modular come in, get this training, get a certificate, kind of bounded program, okay. Just because there was three month and a six month. So there's three different durations, but yes. And did they have choice? Was there any choice that was offered? Yes, so there was, in terms of what type of technical training. So I mean, yes, there was the gender differences. And then a lot of people did solar. There was cell phone repair. There's nearly 40, yeah. Nearly 40 different vocations that they could use. Okay. Could you track the effects of people after they see, in Afghanistan or if they've traveled abroad with these skills? We don't have that data. There was some attrition, but we don't know why they are treated. Yes, sorry, I'm from totally. So I don't, I'm trying to use this so I don't have to ask, but I don't see the demographics of how it's broken down by age, whether people, I'm particularly curious about if these are single men or if it's in family. But of course, you mentioned number of women and that's also something I'm curious about. But like, yeah, so can I get the demographics if they're not in here? Do you have something? Yeah, absolutely. I mean, I could speak about it now, but they're certainly available, both on the Mercy Corps report and as well as the academic one. I mean, these are young men predominantly to about 21 years of age is the mean. They're mostly Pashto. And they are split on the marriage. So there's about 50, 50 in the marriage. So we're in there maybe a little bit shading towards the marriage being not married. 36% of the sample were female. It's a 20, just under 2,600 in the entire sample. And about half, just over half had been displaced. And when we mean by displaced, I mean it pushed out of your original home of residence or your location. The two top reasons for displacement were US airstrikes and suicide bombing. Those are the two most frequently cited in there. Almost the entire sample, as it runs 75 to 80% of the sample had not received any income in the month prior to the beginning of the program. And the education levels are fairly low as you would imagine in this environment. Does that help? Yeah. I'm wondering if, given we've started conversation, I'll put up the implications. I've already started to talk about them, but why don't we hear from JD and then we can open it up? Yeah. I can sit back on the stool again. I'm gonna come back up. All right, John, the floor is yours. Okay, thanks very much. And I'm happy to be here. This is a very timely event. And I was able to get an advanced copy of the report that's coming out and had a look through it. And I encourage everyone to read through it because it's really fascinating and important given where we are today in Afghanistan and the caveat. I haven't lived in Afghanistan for almost eight years. So I've been viewing and working on Afghanistan issues but from a Washington perspective. And so I'm gonna try to go back to my old NGO days when I was in the country working with the IRC and the US Institute of Peace and remembering what it was like to be implementing certain tactical initiatives on the ground to see what could be replicated on a more macro level scale. But then when I joined the State Department, it was looking at it almost from the opposite perspective where we're fighting an insurgency and what can we do and how can we use development assistance to achieve our broader security and stabilization aims in the country and it won't be a surprise to anyone here that those often were in tension with each other. But I see people here like Stan Byers and others who have a lot of experience on these issues. So I'll keep my remarks as brief as possible so that we have a lot of time for Q&A. And one of the things that struck me about this from the get go was that the goal of this project and it says it right up at the front of the paper was to improve economic outcomes and it was an economic intervention to see how people's livelihoods could be improved and that's it. An indirect benefit of that and that's something that they were studying was do projects like that have any spilloff effects on improving or affecting people's affinity towards political violence in the country? But that wasn't the goal of the project itself was to look at political violence whereas for those of us in the State Department and the Pentagon and elsewhere almost every development initiative we looked at through how is this going to affect violence? How is this going to affect stability? Which is important obviously and when you're fighting a war you're looking at it through that lens and you'll hear the generals who are commanding say we need to have development projects that are assisting our overall effort. We have to combine our civilian and our military efforts to achieve objectives that need to be measurable and need to be visible within the next say 12 to 18 months. That's the timeframe we're working on if we don't achieve things in that timeframe we've lost. And I think it's important to look back to 2001 and where we are today and just see what great strides Afghanistan has made in the development sphere regardless of the security environment. Everyone knows that security in Afghanistan by and large has gotten worse in the last few years that the Taliban have made inroads and there's this constant question and this time particularly is to like why are we losing? Who lost Afghanistan? What's happened? And I think it's important to look at the indicators of development and change in that country over time to sort of remind yourself that our intervention in Afghanistan has not all been for bad and that healthcare indicators and maternal mortality rates, life expectancy, the number of children going to school, the number of Afghans with access to mobile technology and the media, the free media that's grown up, the government institutions, however flawed that have been built. The GDP growth in the country since 2001 there are a lot of good news stories in Afghanistan. And so we shouldn't always be clouded by simply the fact that the Taliban are making inroads and there were high profile suicide attacks and things like that. That said, one question I would have, I guess that we can talk about during the discussion is over the course of the six to nine months that you were providing the vocational training, there are a lot of other things that could be going on in the country and in the area where the beneficiaries live that could be suicide attacks that could be a change in governor, local governor that could be other things that give them reason to want to join armed opposition groups or to sympathize with them and so how are you able to sort of carve that out from the results of your study and how are you able to attribute the work that you were doing on vocational training and plus the cash infusions to see whether the impact of that was actually the cause of their support or lack of support for armed opposition groups in the country and before you answer that, I guess like the last point I would like to make is the cause of conflict in Afghanistan, there are many causes of conflict obviously. A number of studies and lessons learned studies have been done over the years looking at our economic interventions, our development assistance programs and often it comes back to the fact that people have political grievances that are going unresolved yet it seems like the US government and other donors response to the problems in Afghanistan has largely been socioeconomic. Let's help people find jobs, let's improve people's livelihoods and therefore they'll see that because of our intervention, their lives are improving, we're at and even if we deliver these assistance through the government, through local governments or whoever, that doesn't necessarily address the main grievances people have, which is often corruption, the fact that we're delivering assistance through people like power brokers who people view distastefully rather than as working in their interest and so the lack of good governance, the lack of rule of law and an effective justice sector and corruption I think are often at the heart of people's grievances whereas if you're giving them vocational training and you're helping them to improve their financial livelihoods that may have tactical benefits on the short term but broadly and strategically I'm not sure it's going to move the needle so much and I guess finally, finally I would say back to the question of who gets credit for these types of programs and this is something I've seen the pendulum shift back and forth over the years and I remember when I first got to Afghanistan, every USA bag of wheat or school built or road built had to say from the American people a gift from the American people because we're here to win hearts and minds and to show you that we're intervening to help you, we're the good guys and yet it was also a magnet for attacks because the Taliban would say well that school was built by the Americans, let's go after it to see this is clearly an occupation and they're trying to deceive us through their propaganda. When the administration changed to the Obama administration I remember the State Department of Public Diplomacy people saying we've got to give the Afghans credit for this, the Afghans need the credit, we have to take our name off of it, this has to be, we're trying to build the legitimacy of their government, not us and so it was all about giving the Afghan government credit and then a few years later it started to be well why are they so opposed to us? Don't they realize what we're doing for them? And so it was this constant cycle of who's gonna take credit for what and on a program like this I know it's a very small scale but one of the impacts that you cited in the report was people's improved perceptions of the government and I wonder one, do people actually improve their perception of the government because they got $75 through M-Pesa that's a one-time infusion that they think probably came from a donor anyway or if it didn't it came from Mercy Corps or it came from somebody and that doesn't address their overall larger grievances in the conflict and I'll leave it there. Great, well thanks John that's a really helpful response so let's throw that back to Rebecca and Jason and if I can encapsulate both of those questions I'll bundle the second two into one which is how did you account for that range of other factors that might have been going on during this period and I think the bigger, more macro question is it seems like a lot of your indicators here are kind of indirect ways that people see the government or assume that the money came from the government otherwise giving credit but John's raising these really interesting questions of how do you think about these types of programs to actually improve the way the government does business and certainly it's more than what your program can handle on its own but it's, I think, a really timely question. Start? Yeah, I'll take the first one first I guess and go from there so this is where Mercy Corps gets a lot of kudos for letting us do a randomized control trial because this is actually gonna safeguard us a little bit from the concerns of the things that are happening afterwards and that that is a concern in this environment we would have loved to have done longer end line the way to year, wait longer but given the conditions on the ground the longer we waited the more people were gonna lose through attrition, through movement, through things like that or other events could happen and so we decided sort of pragmatically the longest we could hold it was about that nine month mark and then we really had to get in there for these reasons that you raised I'm a little less worried about sort of events intervening and driving the results only in so far is to do that to get these different effects they would have to be differentially affecting each of the treatment groups and that's clearly not happening here right so everyone's being exposed to that same shock so that's what the randomization gives us that power to kind of shield us from those things doesn't shield you from attrition and that's a legitimate concern but sort of these intervening shocks in between is gonna be distributed across the groups equally and so as long as people are not being targeted because of their treatment condition because they had cash or because they had just got the TV and nothing else then we should be robust to this concern I would have answered it the exact same way I was too worried for this one Yeah, on I think the question and is that how much of this credit capture argument like how much can we extrapolate that especially given it was a small relatively small program we are, we have some indication that we're getting, the government is getting credit for these things but it's probably not definitive in that I do, so I do think there are concerns plus how do we scale it up? It's a nine month program like it's the effects were nine months later so much there are so many other grievances and in terms of what the government isn't doing so for example, we know that one of the main drivers of young people to these movements is protection protecting their families, protecting their communities and if the government isn't providing security then we could still see these effects so I do acknowledge that we're only seeing government improved performance in a one area that's important to people's lives but it doesn't capture all of it and so I do think you do have these in a sense short windows of time and so people are seeing some government improvement if they don't see other areas of improvement in say another say six months to a year you probably would have some type of backlash effect you have to in a sense build on to these positive trajectories and so that's I think where we often fall short is that our interventions for one I think are tend to be either short term or long term we don't pair them very well but we also don't know how to layer on different result like different programs so you continue that positive cycle. Was there any thought given to channeling the cash transfers through the local government? Yes there was and in one of two ways one is just on the announcement when it went out on the SMS messages saying that you've won this and that a foreign donor has provided one of the things we really wanted to do was experiment and manipulate the credit being assigned so maybe half the people would have got one that said foreign donors the other half would have said local government and then we could have actually directly tested it unfortunately that didn't make it through our institution review board because that was considered deception because the government was not actually providing it and so in order to make this work we would actually have a formal partnership with the government to do that and in this environment the thing that worked really nicely with the cash transfers we know exactly where it went we put the money in we know exactly where it ends up all of it gets to the location if we have concerns about corruption or being sideline on the way to the people then you worry about it being siphoned off or not getting the full magnitude of the shock and so we decided on this particular one not to partner and we couldn't even experiment and manipulate it but there's nothing to say with a longer lead time that we couldn't actually get that formal partnership and have the government use M-Pays or something like that I thought one of the most interesting things when you look at the data and the outcomes six, nine months after was that from a pure economic development standpoint it seemed like that the cohort that received only the vocational training actually did the best and when you had some sort of a psychosocial well-being I think it was measures of self-advocacy is how it marked down and they also did better and it sort of suggests maybe there's this kind of pull yourself up by the bootstraps type of thing they've done well they feel good about doing that and those measures remarkably lower for those who also receive the cash transfer is this and yet we get this positive uptake in terms of less support for violent extremism more support for local government I mean are we creating a situation where stabilization goals and development goals kind of run in tension to each other how do you think about that? So I guess one thing to clarify it's not that the people who just got vocational training did better economically than those who got both it's just that the cash had no additional benefit on the economic outcome so they did basically equally economically so just to clarify that point the self-efficacy findings are a bit odd although there's a that but again it's this is it's the interaction effect it's not that basically in terms of self-efficacy they for some reason cash did deplete that ability to feel like you had control over your life I don't have a good explanation for that this is a very unsatisfactory explanation I'll say for various reasons a number of the measures on self-efficacy had to be removed so I do question I don't want to put too much into that result because it wasn't the full scale and so I have I would yeah and I didn't feel comfortable not including those results in the paper but I also don't want to spend too much time talking about them because I don't think they're particularly robust but to your I think the more general question in terms of I do think there are times perhaps that stability stabilization and development goals are in conflict I mean one of the it was interesting finding from in Iraq from Jake Shapiro who did the event previously in terms of what they noticed in more stable environment you had less economic interaction because people couldn't trade because but they actually had more stability because the military was there controlling things and so you do see those goals conflicting at time I think that idea is are there ways of creating a more stable environment but then finding ways of allowing development actors to come in quickly so that people can start addressing these longer term grievances I think what happens most of the time is that we get a stable environment and then the development side takes so long to ramp up that we actually lose all of the in a sense to build the state the security that we've gotten because there's this big a window of opportunity is lost I just say I worry a little bit about grafting the vocational training and the cash together if they're not designed to be working in conjunction with one another and so in the way I think you rolled out is the cash is kind of its own arm and it's not necessarily reinforcing the vocations in fact that's probably what we're trying to figure out is how they work together and you know I'm a little bit worried that the cash is inducing habits that may be undermining the vocational training and so that they're not actually working in the same sort of together but that there isn't on the economic side that there is a cash is maybe undercutting some of the vocational and the way I think the cash is much more of a stabilization kind of cauterize an issue or a wound really quickly whereas the more the TVV vocational is more of a development kind of a program and I just worry that the cash is potentially if it's not done correctly undercutting the vocational and so I think one of the takeaways is how to think about using the treatment arms not as separate treatment arms but actually their integration so that they're reinforcing each other because you could very you could tell almost everyone when they got the money withdrew it in one large chunk and had spent it very very quickly and so you know thinking well maybe they're going to use cash to start a business or they're going to build on their TVV skills no they were paying rent they were buying food and then you look way down into spending priorities a little bit on education but a handful of people invested in the business that was it and so we're thinking cash is a sort of engine to generate economic prosperity and to build on the TVV vocational that's not happening this is consumable kind of goods and it's gone within a month and so that may be the way that the cash was set up and maybe shouldn't be a one-time shock it should be a maybe a monthly stipend and that may then reinforce the vocational training but I think we need to think a lot more about how the two pieces of the programs fit together I mean just to add to that one of the things I would love to do is having the cash also part earlier in the vocational training and so we know that like the scarcity research that basically having those economic strains in your life make it harder for you to absorb information and actually be successful in learning new skills and so one of the things we don't think enough about when we're doing training programs are those stresses in people's lives to make it impossible or very difficult for them to learn new things and so if we had given the cash earlier that may have again had more interaction effects or synergistic effects for the cash and vocational training I thought that the purpose of the cash was actually to give people a cushion so that they'd have sort of the ability to go into these vocational training programs without worrying that they should be working and actually earning money to pay their rent or buy food or whatever at the time but we weren't able to do it at this point this was really we did the cash more in trying to disentangle some of the potential effects because there's a lot of programs on employment programs that are looking at stability outcomes but they're not able to disentangle are the employment programs successful because of the economic benefits or some alternative benefit and so we did it this way but I think now based on these findings and then there was actually a study in Liberia that had very similar results with a very different population and so how you use cash and training for a larger benefit I think it's something we need to think about more and how do you use cash almost as this booster for various reasons John I wonder if you'd put back on your policymaker hat for a minute and say if you're receiving this kind of information from the SRAP office how do these kinds of findings inform how you think about policymaking and about the efforts that we should be making on the ground I mean I guess it would be less about how it would inform and more a question that we would have to follow up in terms of and Kandahar is a great example but just given how complex the local demographics is in terms of tribes and sub tribes and clans and etc etc and one of the things that we always grappled with looking at development programs in Afghanistan was how to distribute things relatively evenly across the population so that we didn't engender resentment among those who weren't beneficiaries of these types of programs and didn't give people perverse incentives to act in ways where they'd be able to benefit from development programs looking at it from a macro scale I mean I can recall people in Hazarajat in Central Afghanistan saying look we're at peace you know we're not fighting we aren't insurgents and we're on your side and yet you're giving 90% of your development dollars to these inherently pugnacious Pashtuns in Helmand and Kandahar who have centuries old rivalries that you're never going to solve or understand you're throwing your money away give it to us we'll show you how to build a model village we'll show you how we can make these things work and then you know we can be the models of stability that you're looking for and if you take that down to a micro level like in Kandahar again you can exacerbate I think through development projects tensions between communities if you're only targeting particular parts of the population and so I guess coming back to this project like how were you able to or did you consider the impact of the selection of beneficiaries and how that might have a broader ripple effect across the communities both in terms of just their feeling towards the project itself but also when you're measuring people's willingness to support political violence you were looking at the beneficiaries of your project and their affinity towards political violence or distaste for it but if you look more broadly was did the project itself potentially have any impact on other people outside of the projects perception of the government or armed opposition groups so I guess two ways of answering that I mean that's always a challenge in development projects in terms of there's there's never enough money for everyone to receive it and so how do you make sure it's done in a transparent way in terms of who was enrolled in the project that was done in close consultation with communities and so that did try to address any potential adverse effects I would say what what although people knew they would be getting some of these benefits later on we do have that control group where and that's what it's compared to so it and we don't see we don't see like compare we see benefits of the program control but it's I'm trying to see how to answer this if I think if the control group was moving more negatively compare then we could say that it was potentially having these adverse effects but that's the way the data is directed I don't think that's what's happening recognize I did not explain that very well I don't know if Jay if you haven't I would just say that one of the things we did look at is when everybody was in the program part of that intake data was their tribe and so what we do in back and broke it down by the different sub tribe the tribes tribal groupings in Kandahar to see if there was a differential effects across the different subgroupings and we haven't seen much I mean there's a little bit of difference but nothing that stands out as being one group particularly receptive or particularly hostile to the program and that may be just a function of the intake of how they how they selected individuals to come into it but I mean I agree this is the this is exactly the right kind of questions for smarter programming if you're worried about different tribal structures then build that into the design let's make sure we draw individuals from each of those groups and then make sure they're all represented and then run the evaluations and at the end we can then break it apart and say well these are the different groupings and these are the different effects we see and I think too often a lot of these programs get running without any sense of the sort of local landscape and then you sort of just get an average effect reported well averages can hide a lot and if you haven't thought about the front end about the different groupings or whatever then then you're in a lot of trouble and then that average is usually meaningless I'm going to ask one final question and open up the floor and which is to say I thought one of the more enjoyable things about reading this paper was the pre extensive you know literature review and paper trail you had on interventions and what had worked in different places and so this is obviously you were careful with your findings to scope it around around Kandahar and this particular intervention but I'm curious if how this study as well as your broader review of work in this space what kind of themes emerge are the themes that you're finding here ones that would resonate in Liberia or wherever else or are there things that you think are kind of localized to Afghanistan again knowing that you probably couldn't back this up in an academic paper I'm asking you a riff so I think one what really did surprise us is that how much our findings track with this paper in Liberia that I mentioned earlier and so that was with a very different population it was with street kids street young men in Monrovia focused on crime it was cognitive behavioral therapy cash or both and again where they get the long term effects is in that combined treatment and so and that was a year out again was where they saw it and again the cash dissipated very quickly in that program it was a higher amount so it was more tied towards being used for economic development versus ours was more of a windfall but it basically for most of these youth because of where they lived it got stolen within a couple of months and yet again this is what I think is happening we don't have the data to quite say it is that that cash gives that cushion to and what they found is that cash gave a cushion to in a sense adopt new behaviors and so by the time the cash went away they had learned they started to see themselves differently these young men they started behaving differently and so that's why those behaviors lasted longer than if you just got the cognitive behavioral therapy but had no cushion you had to go back to crime to support yourself and I think we're seeing potentially something similar here and so where I do want us to start thinking more about in we spend a lot of time trying to do training programs for young people and I don't think we actually understand the constraints around their lives about why they're not effective and I think if we better understand those constraints we could help make them much more powerful and I think these two papers are starting to see that way I also think again with this paper that just came out in Tanzania and some of these other results is that I think there are ways of delivering aid that local governments can gain more credit and I think we haven't done that well in the past I think especially in places where they're in more kinetic environments that when we're trying to do trying to build local government credit while we're also having a military campaign we're having two different signals crossing each other and that is causing an unintentional ineffectiveness and so how can we better align those objectives? Just very quickly I would say the fact that we're seeing credit capture stories in other places is very reassuring because I think that wasn't what our initial hypothesis going into the study was going to be but we've seen it in Bangladesh and the Philippines now in Tanzania so that reassures me the social scientist in me says well we've got to be careful with generalizability here we don't want to push this too far and I would say that there are things that maybe limit how far this is going to travel just given how kinetic the situation is right I mean most of the studies that are being done that we're comparing to either peace time or there's no violence or the war has been over for a dozen years or more and so there is something different about being in a violent context that I think is going to limit to where it'll apply it'll apply to other violent settings but we just don't have many of those studies yet the other thing I would say is you have to sort of think of where we're entering the war right we are coming into the war in 2015-16 right this war has been going on a very long time now attitudes are pretty fixed towards the Taliban and towards the government and so in some senses it's actually remarkable that we see any movement at all just given how long the war has been going on and how durable and resilient you think these attitudes towards the combatants are at this point I mean there's probably very few fence sitters in Kandahar trying to pick a side right I mean their attitudes are pretty foreign by this point so the way I look at it this is a very tough case because of the violence but it's also a very tough case because of how long the violence has been going on for and how difficult it would be to change people's minds about that this is a low information environment about the government and about the quality of the government and so a lot of places we're comparing it to don't look like that so the fact that we see credit capture in here is really encouraging but I think that there are going to be things in here about the violence about the stickiness of attitudes that may not sort of apply to other places great thank you for that well I want to open it up to the floor I believe David's got a microphone so speak in the mic so anybody who's watching us remotely can hear us and if you would please identify yourself before you ask your question thanks let's go around the front first hi Jessica Baumgartner with Alliance for Peace Building I had a question you spoke a little bit towards this but you know you indicated this wasn't really a violence reduction program but more explicitly economic to begin with but I was wondering if you could speak to the difficulties or not of applying rigorous evaluation approaches in an active conflict so what type of conflict sensitivity issues did you run into or even ethical implications and then how can we reflect on this when we're starting to develop more of an evidence base within peace building programs especially as you were indicating some of the ethnic considerations or who's receiving these versus who's not and those kind of key rigorous aspects of an RCT so behind you Sarah has heard me talk about this quite a bit so I would say one I think as I said earlier in these contexts there's always people who get things and don't get things and so if there are ways we can do it that's transparent and for learning I think that is we should be doing that I would say in most cases the way implementers select beneficiaries is actually not transparent and I think doing it through randomization is actually much more transparent than in other ways we do it I will say I think where it does become bit more difficult in an active conflict zone if you're giving resources is that and people are quite desperate how long can you delay those resources for the randomization and so in a program like this waiting for some people to get treatment in a sense of the treatment the resources nine months is much harder than say a information program where it's while it is important for some of these piece building outcomes it's not more of the life-saving type of support and so I think those are some of the considerations I would make when doing a randomized control trial in an active conflict zone yeah it is I think the other piece that's hard and we as a result we over sample because we knew that this was possible in active conflict zones you have displacement rates and so you have a large probability that people will attread and so that's why we started with a sample of close to 3,000 in the end it was about 1,600 just on the ethical issues too I mean I agree with everything you said there's two concerns I always have going in as well which is to keep the people the beneficiaries safe and to keep my team safe and to keep the numerators who are going to do the interviews and think the survey work safe and so one of the things that we did here we haven't really mentioned but the questions on the sensitive issues are indirect so we're using these survey experimental techniques and they're designed to shield the confidentiality of the respondents in a way that a direct question is not and the direct questions and this is an endless frustration for me as somebody who does a lot of survey work in Afghanistan and other contexts like that on extraordinarily sensitive things and how much do you partake in the black market are you part of the Taliban do you support the government first of all the information you're going to get back is probably not that good but you're also putting your team in danger as well as that person who's responding and so we have to be more ethical about how we get the data and the way in which we ask the questions and so we use two different forms of indirect questioning techniques to lower both the social desirability bias but also the dangers to the numerators the teams and the beneficiaries themselves and I think anytime we're doing these kind of sensitive things or corruption or violence or just a host of these issues if you're asking direct questions you are putting people in danger and we need to stop that practice and so we're really trying to push a better way of doing that and Mr. Mercycourt's credit that let us do that because the typical evaluation is not done this way it's direct questions Yeah, no, no, no in the academic favor ad nauseam the entire appendix is basically our entire survey instrument and all the protocols so people can take this and do this and I'd say one of the kind of cool things we did with this was something called a randomized response technique and this has got a lot of traction in the field now we've seen it in the field in Nigeria we've seen it in Afghanistan the people responding really like it and the numerators do too so everyone sort of understands that their answers are being shielded that no one can back out their answer and find their identities and so we're finding we're getting much better answers to this so all of that's available on the websites There's a link to the academic paper in this one and we have a brief explanation of both those and I know a lot of people in the room have seen our Somalia results and so we use the random response technique in Somalia as well Hello, Stan Byers and currently a fellow here at New America in Afghanistan a couple questions one is what qualitative inputs did you get out of this either formally or informally from either the people that were part of the program or the numerators or government officials whoever in the work that we've done we've often found it's helpful to compare what you're getting qualitative to what you're getting quantitatively and then secondly what did indications do you get of how scalable this kind of approach might be in these kind of environments and I guess lastly I'd ask was there anything that was unexpected that came out of this that you didn't see coming that came out as you did the said okay there's some unintended consequences here positive or negative Well I can say that the dreaded what would you do different question and the last one in some ways and there's a lot of things that I learned coming out of this but maybe I can just start with that there were so I guess one thing I did not think through carefully enough was what support means actually in a gendered context so we have one-third of the sample are females and two-thirds of the male supporting an insurgency means different things and so when I look back at the questions we were using on the randomized responses saying what kinds of actions would you do to support the government what kinds of actions would you support the Taliban with if you're a woman you're going to answer it slightly differently there's different things you can and cannot do and if I had one thing to go back on I would actually want to have more of a I guess gender sensitive way of thinking about what support means in this particular context and so I do think many of the actions translate but not all of them and so when we were giving cash to women that's what they could do with it how that might inform support I thought was very different than what the men were doing just because the option range was so limited and so if I could go back and sort of unintended consequences I would go back and fix that and say we'd get a better measure of that as well that was one of the big kind of surprises I think also too just tracking people's movements was way harder than I thought and I knew it was going to be hard I mean I've done a lot of survey work on Afghanistan this population was very mobile the Kandahar does not have great cell phone coverage and so you know tracking person somebody off their SIM card is hard when they have four of them and so like trying to get people back in and just trying to get the attrition rate not lowering that finding out what they were spending their money on things like that were really really difficult so the qualitative work was crucial for that so your first question what do we have a qualitative we have focus groups male and female focus groups afterwards Mercy Corps also did interviews with key stakeholders in the program inside the government and then I have as a quality assurance kind of step that we take I have a company working for the Yale side that went back and debriefed all the enumerators and we got for best practices what was working what wasn't working both at the pilot session and then at both end lines and kind of go back and say at the baseline of the end line to go back and say what were you seeing in the field where did you feel threatened and things like that so we have all the qualitative debriefs on that other question so it's very deep contextually considering all these things which is what we always I'm sorry I'm Charlie Gillman from USAID I'm a Gender and Quality and Women's Empowerment Advisor and used to be a social scientist and so this I have a comment I hear an absence not just here but everywhere about basic operant conditioning principles where we're taking away something that was rewarding and then what's the response and there's a new building hypothesis that in the ups and downs of aid is a driver of violent extremism and so when we're looking at something long-term like this so like what is the response of disappearance of a windfall does that I just hear in my head some operant conditioning things explaining some of the spurry or some of the results that we can't really explain yet and I would offer to consider that and then my question is did you have power to compare any of the average from the sub-demographics like men versus women time out I mean the group was seems the group is large to me but maybe I saw the airside on some of the interactions and so did you get anything like that thank you I'll ask Jay to talk more because we also look at displaced and that's in the academic paper so we have in the appendices the gender differences and so the results are largely driven by men on most of the outcomes but yeah so the appendix has all the data for both the full sample and men and women and part of the design we blocked on gender we blocked on displaced versus not displaced and we blocked on the four different training centers so in case there was differential effects one of the training centers was a mixed gender facility so we want to make sure we were accounting for that as well so we have the power to do the subgroup analysis by the VTC by gender and by displaced versus not so we do that in the academic paper the boom and bust cycle of the aid this is what fascinates me and alarms me because it looks like the cash in this environment as a one-time shock is a really bad idea and to the point where it could be a driver of conflict people are clearly angry so we have a battery of questions about how you feel about the government and the people who got cash only report an uptick in the belief that it's legitimate to use violence against the state they're the only category that do that they're the only category that reports an uptick in disputes with the police they're the only category that reports disputes rising with neighbors as well there's something going on in there and the thing that makes me alarmed and it goes back to your scale-up question in some ways it's a small intervention though as an academic 10,000 people are going to go through this program that seems actually pretty big that's twice the size of Yale this is actually a fairly big program but the thing that worries me is not so much the scale above us but if you really think that the cash is getting this sort of blowback effect this cash looks a lot like our SERP programs at the lower levels of SERP spending we've dumped 2.8 billion dollars in Afghanistan alone in SERP spending if that SERP at the lower end some of the SERP is big projects but the small end SERP if that has anywhere near the same effect we're doing here there's a conflict driver right there and that may be undermining legitimacy of the government increasing support for Taliban so this is the part that I'm really worried about so I agree with your comment let's go to... sorry I just want to make sure that I understand this one the same input of $75 had a significant impact on people's willingness to commit violence against the government or to think that that's okay think that it's okay yeah so at exactly so at the end line you get a monster blow it's a small, it's about a 7 or 8 point percentage increase in the belief that it is legitimate to use violence against the state so this is at the 6 to 9 month point not the first end line, the second why do you think that is? I think they're angry I think that they got the money and that it's not coming again taxed on it or something that people are taking it from them I don't think it's a tax thing I think what's going on is that people's expectations were raised right so that the money comes in they have expectations that things are going to change and they correctly ascribe it to the government saying this is going to be a change agent and then it's gone and there's nothing following in and your life circumstances haven't got any better and I think at that point the worst thing was to raise the expectations and then fall short of that and now they're upset and so does that mean that we gave them 75 dollars that are out there now committing violent acts? I'm not sure that's true that's a pretty big leap but it does suggest that their behavior is at least considering it's more appropriate to do this it was more thinkable but the self reports are also showing more disputes too and it's only that category so there's clearly that an anger mechanism or frustration mechanism in there Hi I'm Nicole Golden I'm an adjunct professor at GW former state department and USAID consultant with the World Bank and various other hats formerly CSIS as well and just kind of picking on that last point and thank you for this I'm excited to really go through the whole paper and share with my class one of my sort of questions was to what extent did you look at or did the kind of demand side if you will the demand characteristics of the local economy factor in because on this point about the kind of give it and take it away on the cash transfer I think one of the things we often see in kind of employment side and these vocational training programs is very much a supply side intervention excuse me and it's like education and training to what and so I'm curious even though this was kind of a three month intervention especially in this environment where we know jobs excuse me jobs are scarce they're going into the informal economy how did that kind of factor in either into the design of the education program or to the possible impact so that was one question I understand why you couldn't do a longer term end line but I think that would be really fascinating just to address a lot of these questions but and then my other kind of question comment is again we've been talking about the sort of an economic intervention for political violence and while some of the violence may actually be you know could argue could be economic violence and to your kind of third implication of the multi-dimensional interventions I'd be curious if you thought it all about or if you were going to do this again how you might integrate a political slash civic engagement component into the intervention to really sort of test that cross sectoral dynamic because both of these interventions were economic sorry long questions and so to answer so your first question so we did labor market analysis and updated the training program based on those analyses and so at Mercy Corps we do these youth led market assessments and so the team was doing that as part of the development of the program and so probably because in this contracting environment that people were able to actually have economic outcomes I suspect that was a big reason for that success to your second question and I use multi-dimensional specifically and not multi-sectoral because I recognize that these were both economic interventions and a lot of people have tried to read this and say oh we should do multi-sectoral approaches I'm like this paper doesn't answer that I do think so in like our Somalia civic engagement piece is really important in terms of reducing support for political violence and we actually have a second version of that paper and we've expanded it to Putland and South Central and we actually have replicated the results from Somalia land and that will be out in probably early April and so that civic engagement component is also important I do I think this paper answered a different question I I don't I tend to worry about us not understanding how these different components work together and so what I would want to do rather than just adding it on but being able to add it on in a way that we can compare treatments because I think we are tending to throw this bucket of treatments on people and not understanding what's really happening and then we're getting these now cost-effectiveness questions and I think we need to better disentangle that Great, in the interest of time I think I'm going to bundle a couple questions so let me go hit these back to rows ending with Sean or we can take it that way too kind of the arc right here Sean Carberry with the Defense Department Inspector General's Office I'm one of the writers of the quarterly Afghanistan report and where you were going a little bit before talking about SERP and some of those issues that seems to me where this sort of points to is you look back to the PRTs and they built projects and they were trying to build up governance and transition support to the local government when they shut down projects disappeared local government wasn't capable of doing anything so really to me the question is doing something like this where you're trying to get people to trust the government but the government's not any better it's not providing jobs I'm curious about sort of the bigger picture you know thought process of is there a point to these kinds of programs if this other key dimension isn't changing Hi, I'm Tara Vasafi from Berkeley's Human Rights Center and I also work with the CTCV research firm here in DC and I've kind of started working with this collective of Afghan businesses or people from Afghan businesses and I'm just curious I know this isn't really maybe the main thrust of the report but I'm just curious to see if you had any points to take back to the Afghan business community if there are any. Thanks, I'm Tara Siegel and I'm a consultant with the World Bank I'm curious as to whether when you're measuring respondents in economic situations whether you were able to tease apart their actual economic status versus their perceived economic status because sometimes people's perceptions of their economic reality are a stronger driver of behavior than their actual economic reality. So the first question I mean I think that's valid like what can these development interventions do given all the difficulty in terms of building government capacity to take these over I guess at the same time I think we need to figure out how to better do that because I think our military approaches to stabilization by trying I think the military is great at getting us more stable environments in the short term I think the long term unless we can figure out how to do this then also what are we doing militarily because that's only a stop cap as well and I I mean this is gross extrapolation but I have really been thinking about this combination of short and long term and how to think about that at that more macro level as well to the question around business I I have to say I haven't thought about that and I don't probably have a good answer to that I apologize and then to your question I think Jay might have a better answer on like perceptions versus actual I know we most of our economic outcomes that we looked at we're trying to be precise so we would not it wouldn't just be these perceptual issues but I do think one of the issues in a lot what's driving probably behavior in is that relative deprivation versus actual deprivation just I'll go sort of reverse around that just we use a lot of Mercy Corps instrument for getting the income and economic changes and I was surprised at how many things Mercy Corps tracks so there were a lot of objective indicators live stock things like that property titles things like that but there was also a fair amount of perception in there too both in terms of where they thought their current status was but also where they thought they were going to be in six months where they thought they were going to be in a year's time like that and so it is possible to kind of separate those out the T-Vet did the best in terms of these economic indicators and where they tended to do the best was actually on the objective ones so you would ask people and say did you make more money this month or did you do last month they'd say yes things like that that may be perceptual but most of the things were did you have more stuff that's where the T-Vet seemed to actually be working so I think the effects we're seeing are objective but I absolutely agree the perceptions I think are crucial on this on the business community side I would say hire people the one thing as a social scientist coming a little bit outside of the project is this is often seen as an opportunity cost story but you're not really priming an opportunity cost story unless you're giving employment all we're doing is altering employability and so we're trying to operate in an environment that doesn't have a whole lot of jobs and so you're putting a ceiling on top of whatever effect you're going to see because these people can't get jobs and I think in this environment there's often this sort of like why isn't the government doing this well where's the business community too right and so why aren't they part of that story and it's interesting to say that because that you're working with the business community because right now and I've been in Afghanistan for a long time now and I haven't seen much of that business community on that side of it either everything seems to focus up on the government which is where I would end on your comment too I mean I agree if you can't show the government is doing something then I mean how do you get buy-in for the population and I think the one thing you can see in these findings the one sort of positive sign is that a lot of people that hadn't had any exposure to the government so we were asking in the qualitative questions like who do you think brought this to you and what do you think of these ministries and things like that people said I don't know what they did I thought they just worked for themselves and they were corrupt and said but now I understand what they do and so there's almost like this first time I was like wow actually maybe have a little bit of a function in government and I agree I don't think this works without some kind of government I don't think you have to have a fully functioning government though they get traction so I think it's sort of specifying what is it that this government this local government can do now can do well and build a reputation on that let's just start there we don't have a multi-spectrum government yet just do one thing build a reputation for it and then move to the next thing rather than trying to do a sort of government in a box and get the whole government up and running it's not going to work we're too late in the game now for that so we've got four or five minutes if there's another question or two we can take this one thing I just wanted to follow up on the perceptions there is in this we looked at so anger about finding a job and confidence in finding a job and that's reported in there great another question here and then did you have a follow up as well okay great we'll do that and we'll end on these two questions sorry we were just discussing what you mean by pairing short term and long term so I think those things mean specific things for the development community and so it was interesting because you say short term and you're really referencing cash transfers and so I wanted to just understand kind of using the terminology of short term and also long term you're talking about three month and six month interventions which again don't seem very long term but again this is very I'll explain just following up on the governance piece I was just curious you mentioned even just now sort of the ministries versus local governments I'm just curious how that was distinguished to what extent it was distinguished where you think those opportunities you know might be going forward especially when thinking about kind of more place based more place based interventions that might rely on local government first I'll take one and Jay will take the other so we mean short term benefit versus long term benefit and so yes the vocational training is only three to six months but the point of it is for a long term benefit the cash is more of a band aid and that's just for that immediate and so I think if we think about interventions and I'll use it from the youth development space more I think we tend to think about the longer term piece and we don't think about young people's immediate needs as we're designing these and so as a result they're not benefiting enough from these longer term interventions such as education because we're not dealing with their day to day and just thinking from the health sector I remember doing this work actually in the shanty towns of Lima this was on MDR multi drug resistant tuberculosis people can't take their drugs and be compliant if they don't have food they can't keep it down they have to understand those what is their day to day so they can be longer term compliant and so from that I had that insight and from these findings how do you actually help people absorb the intervention that we're giving them and we don't think about that enough I'll just say very quickly on the beneficiaries for them the cobbles seem very far away they can make a distinction between national government and local government they're very clear on that cobble seems very far away they're just a governor seems very far away all the effect seem to be local and could they name different ministries not many they weren't parsing out the credit to a particular one except for Mulsimid on the ground that was the one that people were referring to and saying this is the one that yes and that was partly because the facilities were branded as such right and so that may be part of the intervention itself and people are at least sophisticated enough observers to have levels of gradation and know where the credit's going to go and for us it's mostly resounding to the entity that they'd seen the most which could make sense but also that they had frequent interactions with a lot of the ministries were shells in Kandahar great well thank you all for just a fascinating discussion I hope the crowd will join me in thanking Rebecca and Jason for the incredible work thank you