Hey everyone! Welcome back to The Vintage Space. I'm Amy and today I thought we'd do something that might be a little bit fun and a lot cringe-worthy. Three years ago I gave a TEDx talk at the University of Rochester about the Mercury 13. I have since written this book, which means I've done a lot more research than I'd done before I gave that talk. So today I'm actually going to tear apart my own TEDx talk before someone on the internet does it for me. Let's start with some backstory as to how I ended up doing a TEDx talk on the so-called Mercury 13. Back in 2016 I was asked to sit on a panel about the role of women in space, and being that I'm the space flight historian, I was asked to speak specifically about the women who took the same tests as the Mercury astronauts in the 1960s. I read the books that existed about the women, namely these three, and from there drew my conclusions for this relatively small panel. All of these books tell more or less the same story. Thirteen intrepid women led by Jerry Cobb have the right stuff to fly in space, but they are the wrong sex. They are kept out of space because of systemic sexism. It is a feminist epic end of story. That narrative was my own jumping off point before I started the deep dive into the story that led to fighting for space. Of course, we're at the jumping off point when I'm giving this TEDx talk. Now, I don't remember exactly what I said in this TEDx talk. I just remember it being really awkward. I almost forgot to mention Jackie Cochran, and also it was kind of terrifying to have both a timer and the inability to have notes with me on stage. Nevertheless, I muddled through, and even though I haven't ever watched the tape of this talk, I hate watching myself on video, I know from the title alone of this video that I got it wrong. I know I got it wrong because I hadn't yet done all the research that led to fighting for space. So instead of just giving a rebuttal, I thought I would actually watch it live, capture my reactions, and then tell me, well, tell past me exactly how wrong I was. I'm not actually going to show you guys my reaction to the entire video because it's 17 minutes long and it's going to be largely quite boring. So if you want to pause and watch the old TEDx talk before I dive in and tear myself apart, it's going to be up here in the card. These are seven surviving women who qualified for space flight in 1962 but never got to fly because it was 1962. Well, I'm not even 17 seconds in. And past Amy, they did not qualify for space flight. They were not qualified for space flight. In fact, if you guys want to know all of the details about space flight qualifications in the Apollo era, I did a live stream about it as part of my virtual book tour. It is up here. It's long, but it's thorough and proves this old Amy wrong. Oh gosh, let's continue. Has anyone ever heard of the Mercury 13 before? Show of hands? That also wasn't what they were called. That was a name given to them to kind of make a catchy phrase for the media by a dateline producer in 1995. They were never actually, my new joke, they were never actually the sisterhood of the traveling space pants. They actually never all met. This idea that they were the Mercury 13, like the male astronauts were the Mercury 7, is purely a modern media construct. Now, the guy who helped select them was this guy, Dr. William Lovelace. He did all the physical testing. Why don't I just call him Randy? I don't think he's ever gone by William. He's always Randy Lovelace. That was a weird move past Amy. He thought maybe it would actually be better to have women in the cockpit. Yeah, so Randy Lovelace wasn't actually convinced that women were better sued to space flight. He was more curious how women would stack up against men because he had all these male data points from doing the testing on behalf of NASA and he had no female benchmarks. A lot of the female benchmarks he was using were from the Second World War or involved his good friend Jackie Cochran. The way I phrase it here, I also seem to imply that he initiated the program and that's not the case. There are two versions of how Jerry Cobb ended up doing the medical testing. The one she tells in her memoirs, which is also in the main narrative of Fighting for Space, is that she happened to meet Randy Lovelace and Don Flickinger on a beach one day. They got to talking about space flight and she effectively volunteered to do the testing if he wanted a female benchmark, which he did. The other version of the story, which Floyd Odlem introduces and comes into the epilogue of Fighting for Space, is that Jerry actually went to the Lovelace Clinic herself and asked to be given the tests. I wanted to keep it in the epilogue because not only did this come to light in 1972, so after the main action, it also kind of calls into question everything else Jerry did leading up to that point. So yeah, he was not thinking that women were better suited to space flight, although I will say it's catchy, but it's wrong. I will say that it was wrong. And women also made really good pilots. He knew this. Case in point, Jerry Cobb.  I really hate that I did that. To go from the segue of Randy Lovelace knew that women could be great pilots. Case in point, Jerry Cobb. When Jackie, oh, I'm annoyed with that to me right now. I've talked a lot about both of these women, both of these pilots. I'm not going to go through it all right now, but just as a quick benchmark, Jerry held three records and won one major aviation trophy. Jackie won like 20 major aviation trophies and had something like 70 records to her name when she died.  Jerry Cobb held in 1959 dozens of international flight records, distance, speed, altitude. She'd been awarded from international aeronautical federations. So she had, I'm not entirely wrong on this, although I am definitely overplaying Jerry. The problem is all of the comps for my book, when I was researching this initially, they all play up the Jerry side because it's a great story. She did have more accolades than just the one major trophy. The one she won was the Harman Trophy in 1972. She had won also the Golden Wings from the FAI, the Fédération Aeronautique Internationale. So she was winning other awards because she was setting records and she was one of the few women setting records in the late 1950s. But it's still the case. I'm overplaying how good she was right now, which annoys me. Knew me is mad at old me because the awards she was winning were relatively minor compared to what Jackie had to her name. But let's continue. In February of 1960, she starts going through the exact same medical or physical testing that all the Mercury male astronauts went through. And not only does she pass every test, she scores higher than the men, almost across the board. So this is an interesting point. It's not really super contentious. It's just when I was writing the book, I kind of came up with this realization, helped by kind of talking about it with good friends of mine that you don't really pass a medical test like you either perform well or you perform poorly, but it's not like there's a magical benchmark of passing, especially when you're dealing with something like qualifications for spaceflight from a medical standpoint. It's not like you have to hear a certain hurts to be hearingly approved for space. It's are you fit or not? So I just I don't like it's a verbiage and this is actually verbiage that I try very hard to stay away from. Again, Jerry says she passed the tests and there is evidence that says she scored as well as some of the men on some of these tests. But this makes it sound like she was this medical miracle when really like her blood pressure was fine. Her blood work was clean. You know, it's one of those things where the way she describes it in the way it's usually played up overstates. And for a woman who's flown every kind of plane you can have all over the world, the next level is literally space. Okay, so the problem here, and this is the distinction that I tried to hammer home so hard in the book.  And there's actually a prop plane, a jet and a spacecraft, such that the technology that we're currently dealing with in the narrative, that tells you where we are. Jerry had never flown a jet. That is the key distinction that I did not even think to make super clear at this point, because again, hadn't done all the research, we've said that a lot. So at the time, yeah, Jerry had flown a lot of different propeller planes, but she had never flown a jet. And that is the key difference. So he and Cobb worked together to recruit another 24 women to go through the exact same testing. Okay, so yes, they did, but he was also conferring with Jackie. The thing with all the other retellings of this story is they all underplayed Jackie's importance. It's not just that she was an ace pilot, it's not just that she sort of got involved and started to kind of take control of the program or try to, it's that she was one of Randy Lovelace's oldest and closest friends and that Floyd was bankrolling the clinic that did the testing and that Floyd was president of the board of directors of the Lovelace Foundation. So Randy Lovelace owed a lot to the Cochrane Odelums. This wasn't a relationship to gloss over and for some reason it took me a lot of digging. I only realized that Floyd was board of directors because I found a letter from Randy Lovelace from the foundation letterhead that lists Floyd as the president of the board of directors. It's one of those little details that is super important and somehow never in the story. So it's not the case that they work together, it's not the case that Jerry was the only one involved. Jackie was involved almost immediately. As soon as she started reading about it, she called Randy and Randy was like, yes, I want your help on this too. And in May 17th of 1961, he invites them to go through the second phase of testing, which is the psychological testing. No cop has done this as well. She passed. This is things like isolation chambers. Okay, so yeah, Jerry did psychological testing and again quote unquote passed, but she organized it herself and it had nothing to do with Randy Lovelace's program. When the Mercury astronaut candidates did their psychological testing, they did it at the Wright Patterson Air Force Base. The Air Force wanted nothing to do with this civilian woman who just kind of wanted to try her hand at psychological testing. So she found another doctor in Oklahoma City where she lived that could do an equivalent sensory deprivation testing. She actually felt so good about doing this that she had a life photographer and writer come with her when she redid the test just to have a public profile done of her mental fortitude. So yeah, it's not the case that she did everything amended. Although I am impressed at my past self remembering all the dates because I still like struggled to remember exactly when all of the things lined up. There's a lot of dates in this story. Also, the other thing I learned after this, and again, this only came out through letters. It's never explicitly written is that Jerry was supposed to go through all of this, this third phase of testing in Pensacola, the sort of simulation testing with the other women. But for whatever reason went down alone ahead of the other women and actually performed so poorly that this called the entire female testing program into question. May 17th, 1961. Well, there was a lot of other things happening in space around the time that these women were going through their testing. I forgot about Jackie Cochran. That's the worst. That's like literally the worst. Okay, I hate that. And I remember being like flustered because again, the typical retellings, I am bored with my own repeating of that statement right now. But the typical retelling of this story doesn't really get into Jackie, which is why when I started digging into her backstory and realizing not only how important an integral she is into the story, but how amazing her own story is. I'm like, why is she always the secondary or tertiary character that comes in at the 11th hour as the villain? She's not the villain. It's actually Jackie's story. So I actually remember how hard it was when I started getting into the story to remember exactly where Jackie fits in because like I said, she's just kind of plopped in when she's needed. So the typical narrative, it's so clean with like the history of the space program. We need astronauts. These are the men. What about women? Jerry, the other women, yadda yadda yadda, but then like plopping Jackie in is really awkward, which yeah. So I remember this moment and it like still makes me cringe because of course now I have like, I don't know, two dozen things from Jackie's makeup line. I have signed pictures, signed books, like we know I'm a Jackie fan. Sorry, Jackie. Jerry Cobb wants to be the first woman in space because she just wants to see a woman in space. She's qualified. Why can't she start to train? She's not qualified and she did not want to see a woman in space. She wanted to see her as the woman in space. Reading her letters to Jackie, to Randy Lovelace, to the other women, to NASA, to the president, you see the shift and almost the urgency in her tone. It goes from why not do a PR victory? Let's get women in space. Let's break down this barrier to I want to fly. Let me in an X-15. I want to go into technical space. It's not nearly this like feminist helping other women thing. It's not at all the case that she wanted to break barriers for her sex. It's almost like that was just the fallout of her being the first woman in space, which was her ultimate goal. Just being beaten in space, just had the failed Bay of Pigs invasion. He sees space as this thing that he can leverage into saving his reputation and really putting himself back, putting the U.S. back on par with the Soviet Union. Okay, I'm really pleased with past Amy for giving the historical context because that's the other thing that is always lacking in retellings of the quote-unquote Mercury 13. It takes the women in almost a vacuum and you cannot do that, especially with a story like this. The fact that NASA was going on and beginning like this is the time when it was starting to come up with the idea of the moon landing. This is when Apollo was more than a glimmer in Jim Webb's eye. This was a very critical time for NASA. You start to understand why one goal, getting men on the moon, figuring out how to beat the Soviet Union, or at least show technological superiority over the Soviets in space, weighed heavier than having women go up for the sake of a propaganda victory. NASA's a little bit more focused on this guy right here. On February 2nd, 1962, Don Glenn finally goes into orbit. I said the wrong date. I just heard that. I had to go back and listen to it four times. This is the 20th. That's a speco. Specos, damn it. Then Jackie Cochran takes the stand and says she goes to her experience with the wasps in the Second World War and says, so I trained all these women to fly in the Second World War and they were great. They had a better safety rating than all the men. But then they got married and left. So why would you waste— Okay, this is another thing that I'm super annoyed about because, yeah, Jackie did say that. And I think she made a pretty poor argument in this instance to look at the attrition rate of female pilots who drop out to get married and to start families. There's so much more context behind what Jackie said in that testimony. She was the only woman in the country who really knew how to work a male-dominant system, especially a system based around aviation and technology and now getting into space flight. She knew how to work within that system because it was something that was really hard when you were a woman. And what she was doing in this hearing was arguing that the women not be rushed into space flight, but instead organize amongst themselves to take on the system and to create something. She wanted a large-scale program that would gather all the data such that when NASA was ready, she could just hand the report and say, here, here's your data. I did all the work for you. Now go. She wanted that because she knew that there was no way NASA was going to interrupt what it was doing to fly a woman. So work within the system such that as soon as the system comes to you, you're ready. You're anticipating how to break into it. But the most incredible part of her story, I think, is the fact that she fought as hard as she did for something that she blew. I think my biggest problem with the way I ended this is I always feel like this is a really hard story to wrap up because there isn't a clean ending. It's not like Tereshkova flew in space and that's the end of the issue of women in space from the 1960s. It continued on. Jerry continued to fight. She continued to fight into the 90s when John Glenn went up in his second mission in a space shuttle. What bothers me about the way I tied this up and I still do actually like the kind of you can take inspiration from the fight because I think it's it's a great takeaway that a story that doesn't necessarily wrap up that doesn't have a good ending can still inspire and can still spur people on to do something great. I don't like that I made it Jerry's story and I will tell you guys that when I started writing Fighting for Space it was actually going to be a dual biography between Jerry and John Glenn. That was actually how I was initially thinking about it because I wanted to have John Glenn be the the NASA the male the political counterpoint but everything was going to be funneled kind of through his perspective use him as the vehicle to understand what NASA was doing to give Jerry's story context. It was only once I started really digging into Jackie that I realized how important her role was not only in the immediate issue of the women in space in the 60s but the kind of lead up to all of that to establishing what women could do in aviation establishing how women were treated as pilots. So I like the way I kind of wrap this up but I just think the general thesis is wrong. I still don't really know how to end the story because it doesn't have a clean ending. It's one of those things like I don't know how to make this a morality tale beyond again still you fight for what you believe in and you fight for what you believe is right. I just think that it should more be about how neither of them were right. People have asked me who's the villain who's the hero in the story is it it's Jackie versus Jerry they're kind of antagonists. I don't think one is the hero and one's the villain. I think they both kind of alternate playing hero and villain in the narrative and they're ultimately only hero or villain within their own and each other's narrative. Jackie is not villainous until she's seen through Jerry's eyes. Jerry is not a hero until you see her through the media's eyes. So it becomes this really interesting interplay of how much the media can play into something and create something out of nothing or bring a whole new layer into something that isn't inherently there. So I think if I could go back and redo this talk what would I do? A lot differently. I think the the biggest thing that I would like to change in this and why I wanted to do this kind of rebuttal to my past self is because I want to I want to make it clear that the story as far as I'm concerned now is actually Jackie's story that it starts with Jackie. It starts with Jackie being the one to break barriers, the one to forge connections, the one to save LBJ's life, to have Randy Lovelace operate on her and her husband Floyd and then Jerry enters into the story. She becomes Jackie's adversary because she's in the right place at the right time, not because she's necessarily any more phenomenal as a pilot than any of the other women her age in the country. It's really Jackie who's such a standout and Jerry ends up feeding into that story and that at the end of the day this whole messy issue of you know she said she said fighting to control this program that played out very much in the media, in newspaper, in magazines, in public speeches that all of this tells you that it is about the fight. It is about kind of taking inspiration as we look back at history, taking inspiration from what people were able to accomplish even if they never got to their final goal, but that everything has so many more layers and is so much more nuanced than it appears at first blush. So to never take anything necessarily at face value. So if I could go back to three years ago me who would definitely tell me all of that. So anyways guys that's kind of my take down of my own TEDx talk. I will freely admit that the research I did in writing Fighting for Space changed my opinion on a lot of things. A letter you know that letter from Floyd that's in the epilogue changed a lot of my thinking about the story because it introduced this question of Jerry's motives and how real her retelling is. You know I found letters between Jackie and other military brass that tells me maybe where the decision to cancel the women's testing at Pensacola came from. There's a lot more to the story and that's really what Fighting for Space became. It became tearing apart all the layers and all the nuances in this story because it's not straightforward. It's actually extremely messy and still really hard to sum up succinctly when you add in all the context that's necessary. So that's my new would be TEDx talk if I could redo the talk on this subject. So speaking of Fighting for Space it is of course available wherever books are sold right now. It's available as an e-book as well as an audiobook narrated by me and I promise I speak more slowly in the audiobook. I was told to use my pauses very regularly. If you'd like to follow me for updates on when new videos go up when I have live streams and anything else that I'm doing be sure to follow me across social media. Twitter is probably the best for quick updates. I am at Amy Shira title across the board and if this is your first time at the Vintage Space welcome and I hope you consider subscribing so you never miss an episode. Thank you guys so much for sitting through that with me and I'll see you next time.


The next topic is Sapir Vov Hypothesis with regard to the limitations and the possibilities that are there. The Sapir Vov Hypothesis concludes that our language determines how we experience the world we are living in and how we experience that experience as a whole. The language a person speaks affects his thoughts and perspectives on the world. Now, researchers that have been conducted so far could not provide a solid evidence whether to accept or refute this hypothesis. The researchers generally they do not have the solid or you can say the experimental data, enough data to refute or to accept the complete hypothesis. So general acceptance is that there is some truth that lies between the two. For example, no research says that language does not determine thought or otherwise such as that the thought has nothing to do with the language. So we cannot discredit one or accept one as we do not have till now the concrete and sound data which can enable us to accept one hypothesis and reject the other. But there is a general consensus that there is some truth that lies between both these versions. So the limitations are the lack of data, the experimental data and also the possibilities that yes if you agree that language does affect thought, then it can provide lot of opportunities for the linguists to look at the mental processes or cognitive processes and also kind of look at the cultural and social factor that are there. So the possibilities are there and limitations are also there. So now we will look at certain limitations as well as certain possibilities that the hypothesis offer. Raj Khatri in 2007 in one of his writings discussed Sapir Vof hypothesis in detail and especially the limitations and possibilities. According to him some sources indicate that the recent work in linguistics and cognitive science has supported the Sapir Vof hypothesis. Language in mind is an important aspect which is related to this hypothesis and this advances in the study of language and thought. Basically it's a collection of papers that gives three ideas on language and thought concepts. One is language as a lens, B is language as a toolkit and C is language as a category maker. Now language as a lens means the language we acquire influence how we see the world. Language as a toolkit means the language we acquire augments our capacity for higher order representation and reasoning. And C which is language as a category means the language we acquire influence where we make our category distinction. So the last point refers to basically the influences of the language that is being acquired that acquisition influences where we make our category distinction. Some studies conclude that the idea put forth by Daniel Chandler, George Grace, Lawrence, Chambl indicate the Sapir Vof hypothesis being significant but not being applicable to all situations. So the importance has been highlighted but its application is something which is questionable. A linguistic mystic, these are the notes from a linguistic mystics, explain why Sapir Vof is not conclusively proved and disproved. The mystic writes everybody views the world differently and uses their language accordingly. The language of the one from a teaching career is different from the language of the other who is from maybe some other domain or maybe photographic printing business. Similarly there might be individual cognitive and linguistic behavior with every person. The result is that there is no neutral sample of a given language. Culture is another important factor to be considered. People's views of the world might depend upon their cultural norms, beliefs and perceptions. Similarly people from the same given native language or dialect might share the same cultural background. So it is difficult to find out whether the given effect or consequences has resulted from linguistic or cultural perspectives or the perspectives from both. Such things might get the researcher into which came first, the example such as the chicken or the egg kind of debate. Another important point would be the issue of experiment itself. When one tries to study how people use language without biasing them, he uses language to explain the study and conduct the experiment. Then he might need a translator to pass on instructions which may bias the participants right. If there is some degree of linguistic relativity, it will likely be universal and thus the researcher will be influenced by it. In such a case, looking at the nature of these effects or consequences, a researcher who is studying this effect or consequences in another person might be like inmate studying the behavior of a fellow inmate. So these are the comments which have been written by Raj Khatri in one of his writings specially dealing with the Sapir-Vov hypothesis. Sapir and Vov state that language and thought are too closely related terms. Generally now the researchers come to the conclusion that Sapir-Vov hypothesis has some truth. Yet the extent of truth in the hypothesis is unsolved or yet to be solved.


All right, everyone, next up in our handstand prep progression, we're gonna take some of those scissor kick variations that we have been practicing over the last few days and we're gonna try to just shoot for doing an actual float and handstand. So I'm gonna preface this with, I'm not that good at this. So you might not see what a perfect one looks like here. So we're gonna describe it first. As I come in and I do that scissor kick, I lead one leg back this way and the other leg stays back this way. And I can kind of manipulate where they are until I find my balance point, okay? And I just bring one foot further away, that pulls me this way. And if that's too much, I can bring it back up and I can bring the other one this way. And if that's too much, I can kind of correct. And you just sit there and try to figure out your balance. It's kind of creepy when you do it the first time because you feel kind of, I don't know, everything's backwards, right? Gravity's pulling me the other way. So I will try a couple for you. Again, I do like starting this with the pike variations where I just kind of let a one foot come over and then I learn my balance point. But now we can use this to try to kick up into it a little more dynamically. Try not to hit any TVs. Let's just flip here. That's a little longer. It's wicked fun until it hurts, then it's not fun anymore. One more, it's pretty good. So I like to try that until it feels like my wrists and my shoulders just can't take it anymore. I'll notice them start to bend and then I'm just trying to muscle myself up. And it's not really the goal. That is the, what did we call this? The pike handstand scissor float. The idea is I'm floating in the air as long as I can, okay?


What's happening man besides black parking is back and ready to react and one of you sent me this Buzzfeed video.    I will try Not to giggle While watching this. Okay, I'm gonna try. All right. I have no promises So let's see If I can do it All right, here we go Is The memoir duty, right? What is it? I forget the exact duties the title on duty is something about One word does it do You take these nuts off under here You see that see I can do it. I'm scrum Goddamn it. The question is for all the Millions of magnet sights out there watching this. Did you giggle like a little school child? Grow up learn some image learn some maturity Release the immaturity Thanks for sending me that I probably been laughing if I have wasn't trying so Onto the next reaction video One million subscribers


Ko Elizabeth Pokoa, Pokoa ingoa te te noho oki te whananui a tara. I have the great privilege of introducing Jamie for our following. Jamie has recently joined Kanta Public to head up their evaluation practice in New Zealand. He has over 25 years experience in evaluation and impact assessment and specialising in international development projects for the British government. Today Jamie will reflect on his evaluation of the Newton Fund and how a theory of change can be, say, an advantage or also a disadvantage. But I'll let you talk about the Newton Fund and what reflections or theories of change. So to you, thank you, Jamie. Thanks Elizabeth. I think 25 years experience just means I'm old, right? Well good morning Australia, good afternoon New Zealand. Thank you very much for having me. Delighted to talk to you today. Have you got the slides? Absolutely. Hopefully you'll see those shortly. And we... There we go. So yes, I was just going to... Oh yeah. Perfect, thank you. So yes, as Elizabeth said, I was going to talk today mostly around the theory of change and largely based on my experiences on a project called the Newton Fund which was a largely UK funded programme that I worked on for a number of years before I made the move down to New Zealand, which was only a few months ago. So I'm hoping... Well it's a UK project, I think some of the things I'm going to talk about today are transferable in terms of some of the learning. It's less about the project itself, it's about how we use a theory of change and how others use a theory of change. So yeah, if you just click to the next slide. Just in general over here what I'm going to talk about today. I'm sorry, here we are. Ah, there we go, thank you. So yeah, I was just going to talk a little bit, just a bit of introduction about theories of change and why we as evaluators tend to love theories of change. But then I'll also start thinking about well, hang on, what's the broader purpose of these? And sometimes the fundamental thing I want to talk about today is how an evaluation focus can make a theory of change less useful perhaps. And I'm sort of building the whole discussion around basically a case study of an evaluation I worked on in the UK called the Newton Fund and I'll explain a bit more about what that fund was and what the evaluation focused on. So yeah, and this was, as I say, I'm new to New Zealand. I've spent the last 20 odd years doing evaluation in the UK and the last eight years doing international development evaluations. And this was one of these that was funded through overseas development assistance money, which had some additional challenges. But again, I'm not going to get into that. And I'm not really going to talk about the evaluation findings themselves. I have some questions at the end around that. But I'm really just talking around this notion around theory of change is a bit of a think piece as to how useful or not they might be. And I don't know, evaluation humour, I love this cartoon. It's one of my favourites. How do we think about things and how do sometimes our clients think about things in terms of how change happens? So yeah, if you click to the next slide, please. Oh, that's backwards. There we go. There we go. So yeah, this was just a little bit of history. Some of you may well be familiar with this. And theory of change does have its roots in evaluation, really. And it came about through some of these aspen roundtables back in the mid-90s. And it was Carol Weiss who gave the first definition, really, of what a theory of change was. I mean, I won't read this out. But again, essentially, her hypothesis was that if all your stakeholders are unclear about the process of change, about what you're trying to achieve through a programme, then it becomes very hard to then evaluate and understand what it has achieved. And that was the notion of these sort of mini steps that lead to a long-term goal. And this is a transform then into the theory of change that sets out these logical steps, the causal pathway, call it what you like in terms of how the various activities might ultimately lead to what we would understand now as outcomes and impacts. So that's kind of where it started. And I guess now we're all very familiar with theories of change and use them in our work. And if you click to the next slide, this was my kind of very loose summary of why evaluators now love theories of change. And fundamentally, if we don't know what it is that the funders and implementers of these projects or programmes are trying to achieve, or how they're trying to achieve that, then we're a bit lost. Are we going to evaluate the right things? And how are we going to do that? And we need them to understand the rationale behind the focus. So then to evaluate whether it makes a difference or not. And I originally worried that was what worked or not and there's a whole debate around that. But let's not get into that. But we need to understand that so that we can focus the evaluation really because sometimes you can't evaluate everything. You need to focus on what's important. And the theory of change helps us do that. And that's my final point there really because there is this theory to begin with. We can explore around very loose terminology but to try and understand why a theory that looks great on paper maybe doesn't work in practice or maybe there are some nuances to it that weren't considered in the theory or various underlying assumptions that whether articulated or not had a real impact on whether a programme was successful or not. So, a theory of change can be really useful. It can help guide and focus on evaluation. It helps us to decide what do we need to collect data on, where do we need to focus our efforts in terms of implementing an evaluation and what sort of tools we might use. And it can also help understand the context. And I think that's an important part as well. What is the wider context in which an intervention is taking place and what are some of the maybe the external factors that might influence or impact on whether a programme is ultimately successful or not. So, we love them. Sorry to click to the next slide. The question for today is, well, okay, we as evaluators love these, but how useful are they for the funders of these programmes, the implementers of the programmes so people are actually on the ground delivering them. And in theory, I guess they should be useful. Particularly in cases where funders haven't thought through this logic and that was very much the case in the example I'm going to talk about today. So, it did help sort of unpack that a bit and understand that process of change, those causal pathways. But there's also sometimes misunderstood and I've seen oftentimes theories of change or something that's presented as a theory of change. I wouldn't consider it to be that. It's a results framework, a strategic framework, a log frame, I don't know. We have all these terminology for different things, but to me they're not often theories of change even though my client, the funder, might think that it is and might not mean that to me as an evaluator. And the fundamental question for you is, do they use these theory of change? I think they're most often produced by evaluators or in the process of designing an evaluation. The funders might do that themselves. But do they use it? Do they reflect on it? Do they update it? Because my views, the use of change shouldn't be static things. You don't have a theory of change at the start of a programme and then not update it and refine it as time goes on because inevitably things change, projects change, the context changes. It should always be living documents, in my view, rather than fixed, rigid constructs that don't change from the beginning to an end of a programme. I guess what I'm going to talk about today is how do we make sure they're useful and I'm basing that around this example of the Newton Funds, which is a programme in the UK. Sorry, I can't see the chat, but I noticed Penny Hawkins popped up on the attendee list. Penny was actually part of the... What do we call it now? It was the Evaluation Steering Group or something. So it was a cross-governmental group in the UK that actually came together to advise on the design of this evaluation. So it's nice to connect to Penny, a small world. So you'll be familiar with some of this, having been involved in those early stages of the programme. So if you click the next slide, I'm just going to get some brief background on the Newton Fund because it's important context to where we ended up with this theory of change. So I led the design and implementation of this. Obviously, working with a team, I worked with a company called Tetra Tech at the time in the UK. It was a longitudinal evaluation. It was originally planned over five years. Interestingly, it was commissioned one year after implementation had already started. So lots of things were already underway before evaluation was commissioned. We ended up adopting a theory-based approach. I'm not going to go into why we did that or what the evaluation design was. I really just want to focus on the theory of change aspect for today, but there were some various reasons why we landed on a theory-based approach for the evaluation. And as ever, I guess as consultants, we've probably always accused of this. Ambitious scope and limited budget always a challenge for evaluators. But I think maybe you'll see, I guess complexity was one of the main challenges with this and I'll demonstrate that a bit further in the next couple of slides just as a bit of context. So fundamentally, what was the Newton Fund all about? Well, ultimately, it had this goal of supporting economic development and social welfare. And it was all about tackling global challenges. And this was all in the context of overseas development assistance. So it was very much that aid budget that was funding this programme and that placed certain limitations on how that money could be spent and also where to some extent. So the UK, we ran around £700 million to fund this programme. And it was intended that that would be co-fundate with various partner countries that they were going to work with to do this. And it was all about building research and innovation partnerships across the globe to tackle some of these development issues and to develop skills and talent in research and innovation. And it was structured around these three pillars of activity. People, research and translation. The people pillar was well, kind of, as his name suggests it was all about capacity building. It was focused on individual researchers of providing them with capacity building funding research projects. What they called mobility schemes so sort of enabling sort of transversal people so researchers from one country would come to the UK. UK researchers would go to another country in terms of build their skills, build their knowledge through international working. Research was the middle pillar I think it was the pillar that received the most funding and this was about international research collaborations and it was intended to deliver truly collaborative research between countries again focused around tackling these global development challenges. And then there was this third pillar which they called translation which was kind of okay well what happens next after the research how do you translate that into policy? So can we use this research to translate an influence policy around some of these development challenges or indeed it could be other things too about commercialisation of products or of ideas so it was that kind of think about it as kind of a pathway between okay you can develop good research skills you can develop the good research then and that research will hopefully lead to something rather than just being good research for the sake of research was kind of the theory behind it and if you just click to the next one it was complex put it that way so the fund actually in the end operated in 18 countries when it started there were 15 some were added later on and some dropped out for various reasons there were 7 UK delivery partners so all the UK research councils and a number of others so the academies and the British Council and the Met Office were all involved as delivery partners all with their own funding envelop to deliver activities under all 3 pillars some of them worked under all pillars some of them worked just under some and similarly some of them worked in some countries and some of them not so we had this really complex program of all these countries with many different things happening in each of them by different partners and well if you can another maps quite small but very different context in which all this work was happening so you can see there are some sort of big hitters there I suppose in terms of China, South Africa, Brazil where research infrastructures were quite well developed but then you had others where they were much less well developed so depending on the what would you call it the innovation ecosystem and how developed or advanced that was the activities varied to respond to that local context and in some places where it was far less developed, Kazakhstan was one that springs to mind that was in at the start where it didn't have that same research and innovation infrastructure as China would and that necessarily reflected the way funding was being channeled into activities so some focus much more on the translation pillar others focus much more on the people pillar reflecting where perhaps the most need was in terms of bringing on those innovation structures within each of those countries and yeah the best on the side again I'm just trying to overemphasise the complexity of all this for the number of projects, the number of organisations involved in terms of actually delivering a meaningful evaluation because we were contracted to deliver a fund level evaluation so we had to kind of capture the complexity of all of this in that evaluation work and the next slide please and I guess some additional challenges for us, the objectives were not very clearly defined at the outset so I've said there on paper it was about advancing research and innovation both in the UK and all of these partner countries involved there were no targets there were no country plans so there weren't specific plans of activity for each country and there was no monitoring system or at least no standardised monitoring system to capture all these activities so we had all the partners who had slightly different systems but no common understanding of metrics that we might use for an evaluation so that was a big challenges for us at the start and I guess the other thing that became clear quite early on was that actually there was sort of an unwritten objective around this which was about advancing UK foreign policy interests and sort of using this fund in a way to open doors to other things for the UK government in some of these countries which was again kind of an interesting one and it actually kind of explains you know some of the very early work kind of question will hang up why you'd be chosen some of these countries as partners because if you were looking at ones with you know the most advanced you know innovation system they're the ones which might offer the greatest leaps forward in terms of addressing development challenges they might not be the ones you would pick but there was some of this little driver behind some of those decisions to include some countries so again that was interesting as well I mean that wasn't something we were particularly explicitly trying to evaluate whether we ended up with some useful findings around the extent to which the fund did that but because I think because it was largely politically driven programme where we started was that this logic the theory of change didn't exist it wasn't fully thought through so they had this notion about learning science research and innovation across all these countries but the idea of an intervention logic was you know it wasn't there and that's where we started so if you click to the next slide very early task for us when we commenced this work was to actually develop the theory of change and we did as you can imagine was something this complex with this many partners we tried to make that as participatory as possible we got lucky at the start because there was a sort of a global gathering of all of the in-country teams in each of these countries that they assembled in London and we were able to get on the agenda for that and actually get those teams working together to think through what is this logic and we facilitated discussions to try and unpack what is it they were doing and how does this ultimately leads to change and on top of that obviously we consulted with our client base business energy and industrial strategy as it was at the time and the management team in the UK we also did some familiarisation visits to certain countries not solely for the purpose of the theory of change but to just get our head around this whole programme and what it was doing how it was structured but also that allowed us to capture the views of because it was a partnership programme we wanted to make sure we captured the views of the local funding partners the partners in other countries where we reflected that in this design of the theory of change so yes we had various workshops and activities to try and develop this theory of change to capture this incredibly complex programme for us to then be able to come up with a plan to evaluate it and hold your breath everyone if we click to the next slide this is where we landed as our first iteration of the theory of change which as you can see is super complex and I'm not going to talk you through this so you probably can't even read it but don't worry but as you can see what it did enable us to do to the bottom you've got the three pillars and we tried to capture the vast variety of different activities that were happening there and then follow that process through to decide what are the outputs of that and the outcomes the dark boxes in the middle we were able to agree these three core outcomes that everyone agreed that the programme was working towards and then I guess perhaps one of the most enlightening parts was that going from there to the ultimate objective right at the top that impact statement around the science and innovation partnerships being strengthened there was a huge leap between these outcomes and that impact so those sort of four layers of intermediate outcomes intermediate impacts calling what you want was quite a long road so that was quite enlightening in itself in terms of what does this theory of change mean for us and then it was useless what happened shown here actually were all the assumptions so underpinning all this there were like 27 odd assumptions about that were being made through all of these pathways about what needed to be in place for this to happen and we're able to articulate all those as well which I think was of some use to the client and understanding we'll actually hear some issues we need to get a handle on and things we need to be monitoring and we tried, given this was so complex and a bit of a jumble we did try to unpack it a bit and if you clip to the next slide just in terms of the visual representation we unpacked it by each pillar and this reverses it we've got activities on the top on this one working this way down to those fund level results to try and unpack it a bit and understand what we're doing we mapped all of those onto the diagram so that was quite very useful for us to work out what are all these pathways what is supposed to lead to what and when and some of these hadn't really been articulated before so that was quite a useful exercise in itself but I think our big issue and I think if you click to the next slide was that given all of this the evaluation team I've said loved it that was maybe a bit of an exaggeration but it was certainly a really useful evaluation tool so yes, it was super complex but it did really unpack and gave a representation of these clear logical steps all of the things that needed to happen to ultimately achieve the things that the fund wanted to achieve gave us a framework to then identify how can we try to measure and track these results pathways and therefore design so appropriate evaluation approaches and tools to be able to do that we identified those assumptions some of which were untested and we tried to then build that into the evaluation design and then we ultimately used it to create this detailed evaluation framework of how are we going to evaluate this program over this five-year period so in that sense it was super useful but it was very much designed for an evaluation purpose and as you can see I mean there was a narrative that went with all this of course it's not just about the diagram but even that it was really complex for delivery partners and the client together head around it what does all this mean, what do I do with this am I supposed to do anything with this or is it just something that the evaluators do and that's kind of where we ended up oh, that's an evaluation thing so we don't need to engage with that too much so they couldn't really see the benefit of it beyond evaluation which I think and that's kind of what led us to the next stage of work which was sort of recognising that but then thinking well how can we make this more useful and so if you click to the next slide as Simon on we sort of recognised this so we had this challenge of the funders in different countries trying to work out over an extended time frame you would have new people coming on board to the program and others dropping off because it was over this five-year time period so being able to engage with this and understanding what this programme was all about was quite tricky so we sort of collectively agreed ourselves and the client we called it a refresh of the theory of change to partly to reflect some changes that happened over that period but also with a much stronger focus on the usefulness of the theory of change beyond beyond our evaluation purposes so yeah to make it more useful you know as I say that first attempt was really an evaluation tool we didn't think enough about other audiences and how they might use it particularly those partners in other countries but also the delivery partners here in the UK who you know I think the programme is this type they tend to focus on what we need to deliver the outputs right and are not necessarily thinking day to day about well what's the what's going to say what's the point of those outputs but you know what I mean in terms of how do those outputs to this broader concept of change that we're trying to achieve so we went through a very similar process to update it again it was very participatory we tried to engage as many people in that process as we could and I think if you click to the next slide we should see where we ended up in terms of this new version and visually it's a lot cleaner and again I'll come onto that in a minute it's not all about fancy diagram but actually in terms of simplifying some of the content of that previous version I mean it's fundamentally not different in terms of the story it's telling the activities the pillars haven't changed the long term outcomes haven't really changed but we just represent them and sort of group them together in a much simpler way that try to emphasise how all of this comes together as a cohesive programme rather than just a set of individual boxes and again I'm not expecting you to read all this in detail reports were all published if you google around you'll be able to find those and you'll also see at the top we mapped all our evaluation questions against different elements of this that's the little circles at the top if you can see them we mapped our evaluation questions across all of this to demonstrate what are we going to answer what can we say about the achievements of this fund overall and this and again with the narrative that went with it provided a much clearer statement I think on the rationale of the programme I can't see it by the box at the end that currently my face is covering showed the ultimate impact of the programme but it was much clearer and what we found was that our partners found this a much more useful tool they could sort of engage with this in a much clearer way they could see where they fitted into this whole network of what was happening how all those things were ultimately supposed to lead to these different impacts so as a visual tool it was certainly much more useful to do that and it was used quite extensively and the feedback we had was really positive from these partners I can finally see why I'm doing this or how it's going to lead to these things that ultimately the fund was trying to achieve they might not be the things the partners were doing directly but they were all supposed to lead towards these shared outcomes these shared vision of success so yeah so what did we learn from all this sorry if you click to the next slide just to sort of sum up so yeah I guess we're started for me the use of change are of limited value unless they're actually used by the funders and implementers of these programmes in this case as I said none of this had really been thought through in much detail when we started so this was a really useful exercise in defining these realistic long-term goals and outcomes for this programme it was also useful in agreeing what are the necessary preconditions for success you know what needs to be in place testing those assumptions articulating those assumptions in a way that perhaps hadn't been done before which was useful for then the funders and members think about well actually we need to do something about this because you know if we don't do that you know if there isn't the funding in place if there's no diffusion of knowledge that happens after someone's been on an exchange programme none of these things happen so enable them to start thinking about that and perhaps tweaking some of their programmes to try and ensure that those things took place so I think that was a positive positive outcome and then if you click to the next one I guess perhaps the biggest one for me but actually it's the process of developing the series of change and this collaborative participatory way and then anything else more important than the fancy diagram I mean it looks great but you know so what but actually engaging those partners in that process and the interaction between the different partners when you've got a programme as complex of this you've got people doing very different things in very different countries but actually getting them together to share their experiences and their challenges but then finally to be able to come together to deliver the shared outcome was really really useful and not only you know but understanding where there were some shared experiences so another big challenge with this programme is with all these things happening in different places how do you actually capture the knowledge the learning that come through all of this so sort of through this process there was an opportunity to share that learning to engage more directly and you know overcome some of these shared challenges that previously they'd been sort of thinking about in isolation so it was useful it helped us as evaluators to challenge our thinking you know how do we go about better linking the work of these programmes to the outcomes and to help people to see these routes to impact and how long it might take I mean this fund it was always envisaged that these impacts would be further down the line you know ten years down the line even though the evaluation work would have stopped long before then but this was able to sort of set that out much more clearly and understanding what are the things that need to happen after this fund was ended for any of these impacts to be visible and fundamentally for us as evaluators we've been able to explain clearly to our client, to our funders, to the partners you know what are the benefits for the UK and for the partner countries of all of these different activities so I think I've sort of rattled through some of that quick to the next slide which is my final slide I just wanted to acknowledge that obviously this wasn't just me it was a huge team effort from the team at Tetra Tech and I particularly wanted to thank Ifa Murray she really led all this work on this redesign of the theory of change but more than that she was the one sort of driving this need for the theory of change to be more useful so a really big thanks to Ifa Murray for all of that work so yes I've rattled through that I hope it was interesting it was just some reflection from that work I'd done over the past few years on that project I'm very happy to take any questions or comments from anyone I can't see the chat I'm going to see if I can pull that up now Thank you so much Jamie for this extremely illuminating presentation on the complexity of a multi-program like multi-budget global fund since we don't have any questions in the chat please everyone feel free to pop your questions in the chat or turn your hand up I'm going to ask you the first question Jamie just an indication of how difficult was it to streamline the programs of the various countries and what challenges did you face when it came to agreeing on the outcomes as well as the methodology of demonstrating value for money or cost-effectiveness since it is a budget support evaluation I'm just wondering in the different countries in the different country context how difficult was it to get agreement on the methodology Gosh there are a few questions in there so agreeing the outcomes actually on reflection I don't think that was so difficult I think the client was pretty clear in their heads anyway if they hadn't written it down if you break it down to the most simple level if we were funding research what should the outcome of that be I think the process of agreeing those core outcomes wasn't so difficult what was more difficult was understanding on the pathways of how you actually get there and then as I hinted beyond outcomes how do we get to impacts that was a much more challenging part around how that process would work given that once you're at that level it's much more outside your control there are all these other external factors and influences that impact your ability to do that sorry I'm just going to question the chat which is related as well the challenge of agreeing clear and correct outcomes it was a challenge but it was interesting actually getting everyone in the room because in a way the funder was dictating some of this so this is what we want this to achieve so getting all these country partners on board was interesting it was a challenge in some respects because of the co-funding aspect and this being this overseas development assistance money there are only certain things you're allowed to spend that money on and some of the challenges to face were some of the funding partners wanted to do other things they weren't eligible for this UK government spend and that created some challenges because they wanted to do things that just weren't eligible for funding and that created some challenges with those partners in terms of trying to agree what a success looked like for them because they wanted things that were slightly different to what the UK was able to support put it that way so again the collaborative aspect of that was critical because I think if we hadn't been able to get all these people in the rooms that we did this sort of various exercises in I'm not sure we would have got there particularly for that final version it was really important there was a shared vision of success for all of this and I think we did get there in the end but it did require spending that time that energy to actually get people in the room and talk through this together and actually understand it and ask their questions they wanted about these pathways so that was Sabine's question I've sort of segued into that but the question about value for money that was really challenging and actually towards the end of it it was the client that wanted to implement a new rubric for how they wanted to measure the value for the money of this programme and it just took such a long time and also as I mentioned the lack of monitoring data was a real problem for us we didn't have any of that data even some basic concepts like cost per output or whatever there was no standard monitoring right from the get go we're saying please, please, please can we have some standard monitoring and actually collect this data so we couldn't do some of that so the value for money was a real challenge in the end and I think our evaluation report doesn't talk much to that because it kind of went off and became its own thing this fund sort of became not merged but there was another similar fund called the Global Challenges Research Fund that they kind of then put together on the same management and wanted to introduce a new value for money framework for both of them and our evaluation didn't ultimately get into a detailed value for money assessment because it was so complicated and because we're lacking a lot of information to do that so just to answer all your questions Thank you, Jamie Absolutely, there's heaps more in the chat Todd, Todd Would you like to unmute yourself Todd and ask a question Yes, can you hear me Yes I found the mute button I appreciate the mammoth effort that you've unpacked there and getting so many different perspectives and experiences to a shared understanding and I'm just interested I suppose in terms of how to move from a simple linear model to something that it kind of reflects the reality on the ground and sometimes programs do work nice and simply and as we expect but I suppose my experience is oftentimes I don't so how do we then reflect that with the stakeholders of the reality of moving on from a simple linear model I wonder if you've got any reflections on that Yeah, I think a theory of change, the fancy diagram is always going to be an oversimplification of any program but they can be quite useful in starting that discussion and I think that's where the question is because we started with the complexity and then came back to the simple better so the lesson we can switch that around can we at least get this basic logic a basic understanding of this before we go into the complexity and then having that collaborative process to unpack that that's where those sort of exercise doing that collaborative of these workshops these sessions, some shared learning but all of that would be useful the other thing that was sort of missing here I think I mentioned at the start was the lack of a clear because the model was different in each country and at one point we were suggesting shouldn't you really have a nasty theory of change for each country because the pathways were very different in different countries and that would have been super useful as an exercise even developing that strategy for a country would have helped those working in that country to understand what are we trying to do here it would have simplified some of that funding level complexity into what are we doing in Mexico to be able to focus a bit more on what affects me rather than because some of these fund level concept for abstract at times for some people engaged in it so kind of boiling it down to that level even a very simplified version of what success looks like what we're trying to achieve in each country would have been a real benefit to this program and I think that would have helped because then if you take it down to those manageable chunks it becomes much easier then to get into that messy stuff and to see issues here rather than trying to do it at fund level I'm not sure if that really answers your question but it's just my reflections on the newton fund and how that might have gone better in this case Thank you Jamie I think you have already answered Hadil's question as well I was wondering what you think is more appropriate and useful for a theory of change of detail or a high level one Oh that's a good question Yes, more useful for who right? That was going to where it started this whole thing I think maybe the sort of simplified one can be really useful for maybe someone who's not in the evaluation world but is just trying to get their head around what's this all about, what are we trying to achieve and how are they thinking about getting there I think that simple representation can be really useful for that I think as evaluators we probably do need the more in depth the more complicated one Maybe it doesn't need to be one that's shared widely with partners and others but as evaluators we need to understand that and the funders need to understand that so I think it's horses for courses I don't think there's ever going to be a definitive answer to that one Rula is asking about navigating the unwritten objectives Rula, feel free to unmute yourself and elaborate if you wish I think it was probably an intense process to navigate that and also because it's a lot of different context from the different partners that James worked with so I'm wondering the unwritten is currently written on the screen for us but how did you capture that and how was that helpful, did it benefit the project to sit that on the table define it, measure it see what can be done next if it's useful in another project or stage, I'm just fascinated because it's already complex with written objectives let go other unwritten objectives so just if you can speak to us a little bit about that please it was a tricky one because it was never really written all of this was being published and the UK government didn't want to be advertising the fact that there were some wider foreign policy goals behind this programme so saying to partners this is what we're trying to do and our evaluation plan was not the right thing to do so they were never really written down but in the way we designed some of our consultation, some of the case study work we did in the different countries we did try to tease out some of this about what were the wider benefits of this in terms of these opening doors and leveraging other funds and I think I'm in front of me right now but the theory of change hints at some of those things about leading to these broader collaborations so they were kind of framed and wording that was a lot more vague than I've put it on the screen today that allowed us to unpack a little bit in our case study work consultations with some of the high level government officials in these countries to unpack about what else did this lead to so we were able to unpack some of that but it wasn't done in a super structured way it was, well I'm not going to say anecdotal but there was a lot of evidence we tried to gather about to what extent did it actually lead to benefits beyond anything, the Newton Fund was on paper supposed to be achieving so they never really became written although we did talk to them in more vague terms in the Evaluation Report about how this had led to other collaborations other developments between the UK and these countries Victoria asks how often did you upgrade your theory of change given that things changed in five years Yeah, we only really did it twice in this case because it was quite a long time period and it was the nature of evaluation we were in, we did a lot of baseline work and then there was a period where we didn't do anything at all right, it was kind of the nature of evaluation where there were intense periods interim and final evaluation stage so in this case we only really updated it twice so this was the one we did at the beginning the first one I showed and the one we did well before we started the the end line evaluation I think it may have been useful to think about that a bit more earlier it would have been great if again I'm sort of doing my 2020 hindsight but if we'd had that much more simplified theory of change earlier I would have loved to have seen the client take more ownership of that and have them reflect an update on it and we could have worked with them to do that but for them to take more ownership over that to reflect the way the project was evolving and emerging and expanding but yeah for us if I would we only related that twice but I think again on reflection it would be useful to do that more often but as I say it's kind of feel the ownership of some of that needs to rest with the funders the clients of some of these programs thank you that is absolutely fascinating can I ask is the fund still going say that again sorry is the fund still active is it still being implemented to the same countries with the same partners oh gosh well they were about to I'm not sure where it's at now actually because we finished our work about a year or two back and there were various discussions at that point around what happens next and it was running alongside this global challenges research fund so I think it may still be going but it might be in a slightly different form now sorry I'm not haven't kept up to speed I've been too focused on moving to New Zealand lately to get up to speed on work I was just thinking thinking of the long term outcomes it'd be wonderful to do it before and after did we get it right as I was leaving the UK it was one of the things that was being talked about Bay's the client really wanted to do this sort of post implementation impact study but I'm not sure where they got to with that but there were all sorts of funding challenges in the UK so they were talking about that before I left whether they actually managed to secure the funding for that I don't know but there were certainly an interest in doing that and having that kind of okay a few years out where are we looking now because that you know our evaluation ended we weren't really to say say anything at all about impacts and even some of those sort of higher level outcomes we were only really able to talk about progress towards those outcomes so yeah the clients are the benefit in doing something further but it was sort of outside the scope of our contract at the time we have about 10 minutes feel free to unmute yourself and ask them questions we can have an open conversation we only have the people right now it's dropping in feel free to ask more questions from Jamie and make the most of 10 minutes we have from it's a vast amount of experience and knowledge as we are working Jamie I was just wondering did you have a huge evaluation team on your side how many people were you know in terms of partners So for us delivering the evaluation you mean? Yeah well huge I wouldn't say huge so yeah Tech Check in the UK has a reasonably big evaluation team there were about 20 of us at the time they were all working on this project we also partnered with some other firms to deliver some of the specialist work on some of the innovation work that we didn't feel particularly specialised then we brought them on board but then we then had to we ended up with kind of a case study design to various these countries to inform the evaluation and doing case studies of different things that were happening and we engaged there with teams of researchers in each of those countries to help gather that information and navigate those landscapes so yeah I suppose when I think about it it was quite a big team it didn't necessarily feel like it when I was sat there in London with two or three people around me but actually the whole thing bringing it all together was quite a big team so how does link people in each country more upskilled in terms of change and evaluation? Oh gosh that's a good question I need to go back and ask them I need to learn another evaluation I'd like to think so I think yeah certainly some of them could see what I was saying about making this more useful beyond just evaluation because I didn't understand why evaluation was useful to what they were doing and it was going to make that link where to actually being something more fundamental to the way this program was being delivered and monitored so yeah I like to think so I think that's probably the only way I can answer that question We have any more questions for Damie Patricia do you want to unmute yourself and elaborate? Yes okay hi so much that's a fascinating project so much hard work I actually work with a fund but it's much smaller scale than this and I have so many questions but one that you touched on that I think it's extremely important is the monitoring aspect of that and I was just wondering if you had attached to the outputs and outcomes indicators and if you did what was really the problem with monitoring I mean I think I can guess what you're going to say but given the multitude of people the resource what were the issues that and what do you think you would have in hindsight now pressed a bit more for them to do because obviously some of those metrics are what really we can rely on to check if we are achieving outcomes thanks so much James Yeah I mean that was something very early on when we discovered the resource because I was used to in the UK a lot of programmes using this ODA money are funded through what was at the time the Department for Industrial Development and they are very strict on you must monitor this very strong on the outcome frameworks and evaluation and it was interesting this programme was being influenced through a different department who didn't necessarily have that experience of implementing this ODA money so yeah we have very early on kind of mapped out partly building on that that you change okay well what are the indicators that you want to measure for each of these outputs and each of these outcomes then we made some recommendations early on on that I guess the challenge in doing that was because they had so many different delivery partners who were locked into contracts which were very much focused on the delivery and not the whole administration side of it which they would consider monitoring to be administration and there was a how do I describe it there was a an exchange of views on the extent to which you know they should be collecting the depth the extent of the monitoring data that we felt be useful for evaluation purposes but not just for evaluation for me there was a big accountability issue for the department here about you know how are you justifying the spend what are you actually achieving from it even simplest things like you know how many people are being supported or how many projects there wasn't you know a consistent way of doing that and you know we were trying to edge in towards can you have a centralized system or at least a common set of indicators that each of the partners would report on and they never really got there certainly where I was involved in it we ended up having to do a whole exercise of collating all the individual monitoring data from different people and then trying to bring all together and it ended up being really patchy you know we had some data from all partners on some things some data from a few partners on others so it became a really a real kind of patchwork so it helps in a sense but it certainly wasn't comprehensive so yeah it was it was super difficult and it was partly a kind of a contractual issue about the extent to which the partners were going to be expected to collect all this data Pini Hawkins Pini Hawkins I can see your message do you want to unmute I can do I just thought it was worth saying that and it particularly I guess points to the value for money and so value for who so at that point the UK Government and still now actually was primarily concerned about value to the UK and that really changed the direction of official development assistance so i chi for those of you who don't know is an independent commission for aid impact which is a kind of independent assessment organisation that looks at spending on UK aid money so there were lots of noises about the Newton Fund around the time and so an assessment was commissioned and Amber Red is not doesn't come up very often so I don't know if it's directly related but I know that Difford don't stop funding it unsurprisingly if you look at some of the findings in that report but I don't know whether did the evaluation come up I honestly can't remember it's too long ago but was that pointed to in the evaluation some of those issues were very much pointed to and there were a number beyond the issue of funding in the UK what are we evaluating here but there were also issues of how much it was really related to ODA eligibility particularly under that people strand there were funding lots of work well to develop the research skills which is perfectly eligible for ODA funding but some of the research topics that were funding them to do were very much not related to these global development challenges so there was a huge debate around that as well in terms of how this person's skills have been developed but they're actually not going to do with the global development challenges so that was a big issue from Aikai too and we highlighted that in the evaluation so yes and we did touch on some of these issues about where the funds were being sent most of all what we were asked to do was evaluate the funders at home and what is it achieving so it was kind of hinted at but we were trying to stay out of the politics of that and say here's what your fund achieved but there were certainly some very real issues around how this programme was actually aligned with ODA funding requirements put it that way probably worth saying that's become an even bigger discussion recently enough said on that I think yeah we have five minutes there's a question from Rula on ensuring cultural competency across teams yes so the way we did that was to work with essentially local evaluators in each country that we visited so often we'd kind of double hat we might have someone from the UK to go but also have a local evaluator on the ground who would kind of lead those conversations and provide that cultural understanding and be able to input and lead those discussions on the ground and so you know Tetra was a global company so we were able to sort of leverage the networks of contacts that we had so that each case study visit had a local evaluator at its core and that helped us to navigate some of those cultural activities and issues thank you so much Jamie for your time if there's no other questions I don't see any hands raising we might Yes Jamie thank you so much for sharing your knowledge and insights on this extremely complex newton fund project we have plenty of people applauding and thanking you I will do some googling myself and I will read some of those reports there's much coming to them I will try to navigate the complexities of international development on a massive scale thank you so much for your time