Hi everyone and thanks so much for joining us today. I'm not sure who's controlling the slides but we can go to the next one please. We just a quick overview of how today's session is going to go. I'm going to give a brief introduction to the program in which this work sits, and how it's connected some of the larger us aid endeavors that we work on. And then the group of researchers will be talking about, you know, why take assistance approach why this kind of complex understanding of, of the work is really important and necessary. And then we'll be building through two different kind of work streams that center on this work around complex aware theories of change, and also a deep dive into a program or project in USA Uganda that really works on monitoring systems. So this project sits within the higher education solutions network on home base, which is part of the new ddi bureaus innovation technology and research hub. And we're part of the research division under that and we were the previous inside the previous global development lab and he has been a longer term project we're nearing 10 years within USA, and it really works to bring together development researchers and universities to work on some of the large questions, research innovation and other other issues in this in this area. So we work together to bring missions and operating units, researchers and global academics, other donors and governments and research users and local organizations together. Next slide please, please. So, when we say bring them together what exactly do we mean. And so the question here really is, is, how do we take some of the information the research the evidence the data that our projects can generate, whether we generate that it through our project or it's already existing data and that's out there, and really bridge the gap towards its utilization so how do we connect that and make it approachable and understandable for people like yourselves decision makers within USA or other donor organizations, government, government policy makers, local organizations, local governments, all different kinds of players. So that's essentially the overall goal of what my team is trying to do. And we do this next slide please. So to do this, one of the programs that we do this with is as I said the higher education solutions network. This was started in 2011 and 12 and really to bring together universities and development in new and interesting ways to accelerate the creation testing and scaling of high impact technologies approaches and research. One project you'll see here we have a suite of projects from different universities, both in the United States and overseas. One project within this is from MIT, and it's the site project comprehensive initiative on technology evaluation. And this project initially started out with evaluating technologies. I mean, actual things. And we learned and discovered over the first couple of years that the approaches that they were bringing to this were really more applicable across a larger scale of things. It didn't have to just be solar lanterns or water filtration systems. But we could actually start looking at new, not just products and technologies but new methodologies and systems as a whole. And so now the project has evolved and grown into what parts of what you'll see today. And it's really working to understand how we evaluate and develop methods, constructing understand systems, and working on the things that we go forward. So with that, I promise to keep my remarks short. I will turn it over to our researchers to understand why we should talk about systems. I wanted to start off by answering exactly the question that Maggie just posed. Why do we care about systems, and specifically why do we care about how to monitor and evaluate systems oriented development projects. And the answer is basically that we should care about systems because they're important for development. This is a quote from a USA ID report in 2014, and it asserts that achieving and sustaining any development outcome depends on the contributions of interconnected actors and on their interconnectedness with the local system. So based on this, a lot of development projects are moving from a focus on more linear cause and effect relationships to understanding the complex dynamic systems like market systems, where there are multiple chains of effects that interact with one another, multiplier effects from the context feedback loop, and other similar dynamics. So there's a need to understand how the linear models that sometimes people are used to can be transferred to these more complex dynamic systems like market systems. So a good place to start is, you know, what what characterizes a complex system, and from complexity science and other areas, we know a few things about how change happens in complex systems. So change is often non linear. It involves feedback loops, for example, and change usually has multiple causes, and those causes interact with one another to create that change. And then importantly those causal mechanisms may operate in a really different part of the system or at a different level of the system than where we see the results. So for example of this, we need to figure out how to incorporate this complexity into activity design and monitoring and evaluation, and what we're trying to do is understand some ways to do that. But before we get into that let me give you a more concrete example from our work with USA Uganda. These are graphics directly from from the mission that described to feed the future value chain project. The idea was to support small holder farmers who are right here in the middle, not by intervening with the farmer but with the system around the farmer. So with the traders who buy from them with the input dealers who sell inputs fertilizer and seeds to them with the government and regulatory environment. All of these different places around the farmer with the goal of affecting farmer incomes. There will be six projects under this in this portfolio here that all attack different parts of this problem. And what I want you to recognize is what I just said there are multiple causal mechanisms at work, both close and distant from the small holder farmers, and there's a lot of nonlinear dynamics involved in in the change that's that's sought here. This talk is really two talks as Maggie said about how to plan and measure change in complex systems like the example that I just gave you, and they span this cycle from designing interventions to monitoring their progress evaluating their impact, and then adapting them as the system changes. The second project is about complexity aware theories of change, and Elizabeth will talk about on the design side here how to build complexity aware theories of change, and then how to test those theories of change with evaluation research. The second project is about system mapping and monitoring, and my colleagues and I will talk about how we map the system to identify what outcome pathways to monitor, and then monitor indicators on those pathways to identify enablers and barriers in the system. So what is useful to notice is that both of these approaches span that entire cycle but they emphasize different and complimentary parts of it. Without further ado, I'll turn it over to Elizabeth to talk about complexity aware theories of change. Thanks Erica. Yes, so as Erica mentioned I'll be speaking about the first of these two approaches, specifically sharing the results of the first phase of research research project called investigating inclusive systems innovation, which focuses on identifying how inclusive systems innovation processes work, particularly in the context of smallholder oriented agricultural systems which we know are complex. Next slide. So this research, the research informing this presentation has just recently been published this month by the Journal World Development in an open access article. And I think that article was included as a link in the invitation to the webinar today. So I'll talk a little bit more about it. So a lot of the details of the theory of change that I'll be presenting today are actually well described in the article. But I'm going to be kind of zooming up a little bit from the specifics to really talk about the value proposition of complexity aware theories of change, why they're important, and also kind of how they help us to navigate complexity. So I'm going to talk a little bit more on, yeah, kind of the, the, the place that these theories of change play in the kind of overall enterprise of doing complexity where Mel, how specifically I went about building one of these in the context of inclusive innovation, and then how they can be a useful tool for creating cross context learning and kind of portfolio level, Mel strategies. Next slide. Why do we need complexity aware theories of change as as Erica mentioned, you know, a theory of change articulates how and why a given set of interventions will lead to specific change. And it is often formulated in a very straightforward if then kind of linear logic, but behind that logic is a set of beliefs and assumptions about how change will occur. So in the context of interventions operating a complex systems, the theory of change needs to build on assumptions that are consistent with our growing knowledge about how change happens and complex systems. So instead of being complexity blind or complexity ignorant, it should be complexity consistent or complexity aware. So that's kind of what we're going for. Next. So, you know, returning to some of these attributes of complexity that Erica mentioned, and we know quite a bit from complexity science about how, both how complex systems work and how complicated complex interventions in those systems work. One of those attributes attributes like non linearity or multiple causality or emergent results are intention or conflicted that with how we typically think about and formulate theories of change in terms of a project and its activities being directly responsible for a chain of results in a linear sequence and I'm sure we've all seen many of these. So in the context of complex interventions, we need to find a way to build in correct assumptions about how the systems work and how change happens in those systems. And we need to do that in a way that that isn't overwhelming. Next slide. And that's, that's challenging, right, because complexity itself offers evaluation a set of difficult challenges, particularly around scoping complex systems are nested in each other. So, you know, which attributes of complexity are the most important to incorporate within our results frameworks and our mail plans, and what parts of these systems to map how close into an intervention or how far a field. Some elements need to be monitored and evaluated we likely can't monitor and evaluate them all. So how do we scope our complexity aware mel so that we aren't kind of awash in a sea of complexity. If those are if any of you are familiar with Ray Possen, he gives a nice tour of, you know, the field of monitoring evaluations attempts to kind of deal with complexity. And part of it's a critique, which is that oftentimes, you know, initiatives come away, crestfallen having bitten off more than they can chew. Right. And that has to do with, with kind of not knowing where to look in this complex world. Next slide. So that is really where complexity aware theories of change can offer value because the theory of change can really serve as a scoping and focusing mechanism. And how I did that in the study is I picked up on Ray Possen's critique and I followed his approach in using realist theory and realist evaluation approaches as a starting point. So, realist evaluation, for those of you who are familiar with it starts from a kind of long tradition of philosophy of science in terms of looking at how change actually happens. And it starts with several kind of core assertions, one of which is that results happen through causal mechanisms. So, so interventions don't directly cause results they trigger causal mechanisms at different levels ranging from individual reasoning and decision making up through kind of higher level processes, and the interaction between an intervention, its context and the mechanisms it triggers are really what lead to visible measurable outcomes. And that kind of core proposition is that even complex interventions, do you have a characteristic way of working a kind of modus operandi that can be investigated through evaluation research, and that over time and across studies evaluation researchers can create mid level, what Possen calls reusable conceptual platforms which really are theories of change to enable learning from one evaluation to the next so each evaluation isn't starting from scratch and kind of a wash in a sea of complexity. So that's that's precisely what I did in this research study. I looked at the existing evidence around inclusive innovation processes and how these processes work so multi stakeholder processes, taking place over, you know, five to 10 years, I'm aiming to create inclusive local systems innovation and agricultural systems. And I, and click Erica, thank you. There's yes, I focused on really the first piece of this three, three piece arc that realist evaluators recommend which is first build an initial program theory based on the existing evidence based on an evidence synthesis, prior to the program reviews theory, use that to create an initial focusing mechanism that tells you where to look in a complex intervention, then go test it through midterm evaluations and term evaluations or ex post research on on programs implemented by others to test the components of the theory so what's in the boxes, but also the causal relationships, and the key assumptions underlining the theory, and then use that empirical testing to come back in this in the third and refine the program theory, so that it becomes you know a better more accurate reusable conceptual platform for future use. In the case of inclusive innovation programs which is what I was focused on researching that initial program theory wasn't really there. So that's where I focused this first phase of the research. Next step. What I specifically did was I surveyed the existing literature, identifying a kind of suite of examples of inclusive innovation projects that all had certain core attributes they were all trying to do the same thing and kind of had a similar intervention that was very scattered across the world so you know case from from the East, East Coast of Africa from Madagascar a case from the Peruvian Andes, a case from the Philippines, different organizations different implementation approaches different details and activities. And what I did is across case synthesis of those three cases looking, you know first creating a theory of change for each of them based on the existing research on those cases and then going up a level to create a mid level theory that really abstracts across the three cases to understand how inclusive innovation was produced in each of those three cases. And I built that causal model formulated specifically as a realist informed mid level theory of change. So the paper presents the theory of change in two formulations, one a bit more abstract than the other this is the more abstract one and describes each of the components and quite a lot of details so I won't I won't do that today but what I will draw attention to in this model is the ways in which it's different from a conventional theory of change so here what you'll see is that this model really draws attention to the main conceptual categories of a realist theory of change on the left we have some of an example of some of the key activities that these implementing agencies engaged in so knowledge co production joint experimentation relationship brokering, and there's a longer list in the paper. And then below that there are the elements of the local context that those activities interacted with, including initial results from the activities themselves, which changed aspects of the local context. And what this model illustrates is that the interaction that the feedback loop the dynamic interaction between activities and the changing local context that resulted from those activities, triggered a set of four causal mechanisms, which are formulated in the paper at the level of mid level theory so social learning collective cognition social capital strengthening and consensus formation, none of those were processes that these inclusive innovation initiatives were directly facilitating themselves they were the emergent result of interactions between project level activities and changing contextual factors, and that the dynamic interaction between those three factors as realists predict so activities contextual factors and causal mechanisms jointly produced the intermediate outcomes that led to inclusive innovation. So if we go to the next, the next version of this model this is the more detailed version. Here we have a specification of the content of the theory so now looking beyond kind of its conceptual categories. We see what those specific activities and processes were. What were the initial results that they produced in column two. And these are the changed aspects of the local context for innovation. So things like constructive cross boundary dialogue, or the creation of new farmer groups or the strengthening of existing farmer groups, increased self confidence and collective efficacy. So, you can think of these as intermediate results of project activity, but they're also things that are changing the context within which those project activities are now continuing to operate over time. And once again this is another way of showing how that dynamic interaction triggers these causal mechanisms that are really causally responsible for the production of outcomes. Putting a theory of change in this kind of unconventional format does a couple of things. It enables us to start to understand how some of those complex dynamics play out so it allows us to not just look at changes a linear process but to start to identify feedback loops complex causality, and you can see some of that diagram tier how for example, you know, social capital strengthening and social learning need to interact together in order to produce collective cognition is not the result of just one or the other and we know that from existing social theory. So being able to kind of look, look beyond the specifics of a given project and bring in evidence from existing social theory and how these processes work to create a model that that starts to reflect some of those dynamics that we know complexity offers. Next slide. And really where this starts to become quite useful in practice is that these kind of mid level theories of change can help us to scope the male activities for complex interventions. So for an intervention, let's say spanning a couple of countries and you know three to five years. There are a lot of activity ground level activities, developing a theory of change focus just on the outputs of those ground level quite, quite massive and quite constraining. Here what we're able to do is we're able to focus directly on intermediate results which in that previous model we're in column two, rather than this long list of potentially changing outputs tied to implementation activities. We're also able to identify whether how and under what conditions project activities are triggering key causal mechanisms in column three. And so this this mid level of analysis enables us to be accountable to see whether or not a project is producing results, but to do so in a way that enables adaptive management so that the specific project activities and their outputs can change as needed. In order to produce those intermediate results that are really that the linchpins of the causal model right what we identify is that for example, it doesn't really matter if constructive cross boundary dialogue is happening through scenario design or, you know, participatory action research there are many ways of accomplishing that intermediate objective, and a project might need to adapt its strategy to a specific component of the local context. The important question is, is is whatever they're doing in that local context, leading to constructive cross boundary dialogue because that's really the thing that needs to happen so it lifts the gaze kind of one level up. And then finally, kind of at the portfolio level. This can help this this higher gaze can help us align mal frameworks across context specific projects. So once again this focus on intermediate results enables us to frame results in a way that have resonance across different kind of specific interventions that might need to be very tailored to, to local contextual conditions, and really start to frame things in a way that we can develop common indicators of, you know, concepts like constructive cross boundary dialogue or increased collective efficacy those kinds of intermediate results. So we can start to develop robust common indicators for causal mechanisms that are shared across projects. Allowing projects to come up with their own unique specific ways of triggering those that might have a lot to do with a very you know hyper local context. So these kind of frameworks provide an opportunity for cross site learning, and also for improving the conceptual framework from one project to the next so being able to develop a theory of change in one project that another project can pick up and test and refine that can then be used by a third to have stronger and more robust indicators. So at that point I will stop and kind of open things up for questions just on this first bit of the presentation and then there'll be a longer period for questions at the end. Thanks Elizabeth I there are a handful of questions. Let me just select a couple and then maybe we can get to the rest of them at the end. So the first question is from JP. When working on the theory of change did you separate was complex and what is complicated and so how did you do so and how did it affect how you develop a theory of change. That's a great question. Yes. So I was specifically in so the complicated part in part has to do with kind of the number of moving parts you know that the number of different things a project is trying to do the number of countries. It was trying to implement across kind of the inherent complicatedness rather than the system dynamics which have to do with complexity so I specifically wanted to scope the complicatedness of it and look at you know a narrow set of shared components so I would say the complicatedness is factored into the components of the model, the complexity is factored into the structure of the model. So, in components I identified nine common components now is that the full set no it's probably not, but it was it was it was nine activities that were repeating across these different projects that were present in all of them and I think subsequent research would need to really dig into, you know, can inclusive innovation happen with eight of those with six of those you know how big does that set need to be. But that's how I would distinguish the two is that kind of the the ingredients speak to the complicatedness of the intervention and the structure focused on feedback loops focused on system dynamics and focused on the emergence of mechanisms is really where the complexity is reflected. I hope that I hope that addresses the question. And we can take one more before we move on to the next section. There's several but I'll just do I'm doing them in order of when they were submitted so what it was this mid level theory of change developed as part of the activity design itself before the award was made after the award was made could you kind of give a little bit of information about the timing and logistics. Absolutely. Yeah, yeah so something that distinguishes this project from the one that you're about to hear about is that this project that I'm working on is a purely research project so there isn't the the the project itself is to conduct research on include on cases of inclusive systems innovation and through that empirical research to develop a better understanding of how inclusive systems innovation happens. So this first piece was actually the development of the initial theory of change to scope that case study research so what I'll be doing in the next year is conducting three or four X post empirical research studies of inclusive innovation processes, but now instead of looking at every aspect of those processes and how they played out over 10 or 15 years. I'm going to be using this theory of change to scope and focus that research so I'm going to be really looking at specific aspects of inclusive innovation process and using the empirical research to test the theory so it was the first year of the research project to develop essentially hypotheses that subsequent case study research can test what I what I identified in the existing process is that a lot of the research on inclusive systems innovation is very granular and based on learnings from specific cases and specific places, but there wasn't a lot of that one level up kind of mid level theory about how these processes happened in general. It was, it was a lot of kind of disparate data from disparate places that didn't really provide strong hypotheses for focusing subsequent research. Great, thanks. There are more questions but I think we should move on to the next part and we'll come back and answer them at the end of this time. So can we move on to Erica and Jared, you're up next thanks. Okay, great. So, I will talk next about a second buy into the same activity and this is called the market system monitoring activity. So this is the USA do you found a feed the future value chain project and Jared and I will talk about this today but there's a much larger team that has worked on this and I was also glad to see so many familiar names in the participants list from folks who have helped us develop these ideas along the way. So first I want to revisit the example that that we gave in the introduction about what it you know to give a concrete example of a systems development project from feed the future Uganda's value chain project. So the idea here is how do we support economic growth for smallholder farmers in Uganda. And as I said before, that's by intervening in the system around the farmers with input dealers who sell seeds and fertilizer to them the traders and processors who buy from them, the financing systems that support all of this and the enabling environment that makes it possible. But it's not as simple as I've drawn here. It's a much more complex system than that. These interventions where they're working with agro dealers or with traders and processors are pretty distant from the outcome that is being sought around farmer incomes. From a monitoring and evaluation perspective, if you just measure the main outcome, farmer income, it could take a really long time to see whether you're actually making a difference. And so the challenge for for monitoring evaluation and learning is how do you monitor an intervention like this that could take a long time and follow many nonlinear pathways to get to the change that's, that's being sought. And then how do you assess whether change is happening across this entire very broad system. And so this is the challenge that that we were brought in to help think about and implement. So we did that by leveraging techniques from systems engineering and specifically one approach called system dynamics, which was designed specifically to analyze complex systems like those that we've talked about today. And it focuses on how change results from the structure of the system. So I won't go into the details but there's several textbooks on these approaches, and we adapted some aspects of these techniques for development applications. So this is an overview of the approach that we've come up with. There are three main steps. The first is to map all the causal pathways in the system that lead to the key outcome. So it's kind of like a set of interacting results chains. The second step is to measure indicators along those pathways all over the system to monitor whether change is happening and where, and this can be based on existing data or data that's being collected for this purpose. And then the third step is to interpret that visualization to understand where changes succeeding and where it is stalled so in other words where are there barriers that we need to adapt to. And so I'll show you each of these steps with an example from an analysis that we did in Uganda specifically around agricultural financing. So the first step is to map the system, which is basically asking what enables what. So we want to get all of those causal pathways that influence the key outcome. And in this case in our example the key outcome is that farmers take out loans to improve farming practices. And if we work backwards what enables this one thing that enables farmers to take out loans is access to those loans. So access comes either from having a bank nearby, or from having access to a loan through mobile money. So this is one kind set of causal pathways, but farmer loans are enabled by a lot more than just access. So with all these different colored pathways in the map here, these include things like affordability supply information, etc. And so this final system map shows all the causal the all the influences on that key outcome. The second step is to measure the system and the question we want to ask here is, are the desired changes being achieved. So this element in this map represents something like a behavior by an actor or a condition of the system that might need to change in order to achieve this key outcome. And so this this map measures how widely adopted are those changes or how widely adopted are those behaviors or conditions. So if we take this as an example this says farmer has access to a mobile phone, and there's a survey that shows that more almost 80% of rural Ugandans have access to a mobile phone so this is colored green to indicate widespread adoption. We can also indicate moderate or more limited adoption. And the advantage of this kind of a visualization, it's simple, and it shows lots of diverse data from different sources because you're not going to have one source that can measure the entire system. And it shows them on a common scale. So it makes it easier to see where change is succeeding and where it is stalled across this entire system. The third step is now to interpret this measured map. So, we can use these colors to notice areas where changes are achieved or conditions are right for change that's where things are yellow and green, and red areas might indicate barriers to change or places where change has not happened or is stuck. So in this case for this pathway, our map shows that the key outcome is read few farmers are taking out loans to improve farming practices. This we would probably know from traditional monitoring and evaluation approaches, but now we can ask, why is this the case why aren't farmers taking out loans to improve farming practices. We can trace backwards through these pathways to find where the barriers are that might be preventing that. And in this case there's one clear barrier here that very few rural Ugandans have bank agents or bank branches nearby. So this is a big barrier to accessing loans. One of the big strengths of this method is that you can do the same thing not just for one pathway at a time but also to look at the entire system as a whole. Here we've zoomed out to show all the pathways. And if we look at this which looks a bit of a mess right now but if you spend time looking at each of the pathways, you'll see that this pathway here informal financing is mostly green and yellow, meaning that it's it's working pretty well. Informal financing is widely available, which means that most farmers do have access to some kind of loan. But the key outcome is still read they're still not getting it. So we looked at some of these other pathways and we found that this purple demand pathway down here is really the biggest problem. If you look at it, you can see that if you look at it in more detail, you'll see that it's farmers are not getting loans because they're not choosing to seek a loan. And it's partly due to things like a lack of trust in financial institutions lack of knowledge about loans and an unwillingness to take on risks. So, zooming back out to the overview of our approach the key here was to map these nonlinear causal pathways in that first step that influence the key outcome, and then choose key points along those pathways to measure, so that we can learn how to use propagating along those pathways as expected. And by doing this kind of analysis we learned that, you know, the biggest barrier that's discussed is access but in fact farmer demand for loans is at least as important as access. So we were able to learn gain new knowledge by looking at this sort of holistic view of the system and how it is or is not changing. And I'm going to turn it over to Jared to talk about how we adapted this to do this approach to do a more rapid assessment of the system during a crisis. I'm going to pick up the pace. It's just something we like to do in our humanitarian lab at MIT. In fact, we had been responding in different ways to coven and we were excited when the Uganda mission asked us to say well, can we use this complex map to help understand the complexity of a virus impacting agricultural markets so we leaned in and developed a set of steps that are not complicated, but that build on that complex mapping to add shocks that come from the virus to the system map. And then add knowledge from different sources open source data news articles reports and interviews from key informants, pull it together to analyze generate conclusions and the important thing is to iterate keep doing this over and over again as we as the crisis evolves. The key outcomes of this work is that we are able to curate the knowledge using the system structure of the map in a way that people can find it and collectively build intuition about how the crisis is affecting the markets they care about. And so this also helps the different stakeholders then to assess feasibility and appropriateness of interventions, and also identify what they need to continue monitoring and how long going forward so on the right you see the virus traders on the border and some data I'm going to link those really quickly with walking through how it looks in our system map next slide please. The big map the big complex map on the left, all the elements we had before there, the thing we added were the shocks the red boxes that are filled in, and then the effects. There's a color coding like what Erica mentioned before, but it's not reflecting adoption rather it's reflecting the likely impact of the virus on the different elements. We can do an outline around each of the elements to say how much information we have so that with a quick visual you can see what the impact is and where we have data where we don't have data in this emerging crisis. Next slide please. And we're going to zoom in on commodity distribution on the right there, and specifically one shock on the upper right which is international travel restrictions are imposed. The narrow there leads to an effect of traders facing the lay at borders this comes from wide reports in the news articles of 20 mile long lines of trucks waiting to cross the border as those restrictions are put in place. Click again, we see another effect of cross border trade being limited to cargo vehicles. This came from news articles but also a report that was on aggregate links put out by mercy core, talking about how exemptions for cargo were enabled with put in place to enable the flow of trade. And if a formal trade, but the informal trade, small and medium traders using public transportation, for example, were restricted in their ability to cross border for their trading. Next slide. We can see how this then gets put into we have a mapping software that allows us to put the facts that the links in the upper left, we can count how much data we have to support a particular condition. Click forward again. We can see how trace it to a key outcome here of collector trader earning a profit where we have more data if you see in the purple box in the bottom we have more data facts interviews. But that linkage between the particular shock and the outcome is what we're trying to highlight and the key point is in the middle there that yellow circle is that collectors and traders linked to international markets is a mixed bag, formally and informally. So the next slide we can see that played out in the data. So the top graph, both of the graphs in the right show the five year cross border trade as captured by the Bank of Uganda, the dark blue line captures the 2020 data through July. And the top is the formal trade you can see that as of May it had dropped significantly due to the virus but then it started rebounding quickly. Whereas on the bottom the informal cross border trade dropped in April and stayed low. And now we can see we see the data but we can now have a way of looking at the system understanding the complexity that leads to that kind of a situation. And we can think about how we can intervene to support the different market actors, especially the women who make up the preponderance of cross border traders informally. So that's just a quick example of how we could adapt a complex system map in a crisis. Okay, so just to summarize the pieces we talked about today that this was four years of work there's a lot more to discuss here but we tried to give you a flavor for for what these approaches can do. And I just wanted to bring it back around. We've shown you two examples right a detailed analysis of monitoring change in agricultural financing in Uganda and then an adaptation of that approach for monitoring the impact of the rapid crisis on that complex system. And so just briefly why was this useful for monitoring and evaluating. It helps to determine what to measure, although we didn't talk about that much today the system map can help you figure out what you need to measure and what you don't in that broad system. It helps to visualize diverse data on multiple parts of the complex system to find data gaps when more data are needed, and then to assess the system health both at a detailed level and on this high level across the entire system. I think that it helps projects to adapt to identify the barriers to an enablers of change and maybe alternative pathways to get around those barriers, and then also to build capacity to rapidly adapt in a crisis like COVID-19. And generally speaking it helps to combat linear thinking, get to the details, even though it's complex, engage stakeholders and find where their work intersects and identify leverage points and opportunities for investment. I just want to wrap up across, across both Elizabeth's talk and our talk with a couple of key takeaways across those two approaches that we've seen the complexity aware theories of change that Elizabeth talked about and the system mapping and monitoring that Jared and I talked about. So here are a couple of broad takeaways and I'm sure that you could that the audience here could come up with more. But the first one is that results frameworks too often focus on what was achieved but not how it was achieved. So we know if an outcome happens, but not how those causal pathways got us there or which ones are not working if that outcome didn't happen. So our approaches have both come up with this notion that indicators could be developed not just for the outcomes, but also for the causal pathways that get you there. And the second one here is that monitoring at the system level this big picture level is important for adapting interventions to emerging barriers or opportunities midstream. And so it's it seems like a good idea for projects to plan to monitor intermediate results at the system level across multiple activities or across a portfolio of activities, and maybe using some kind of project level monitoring evaluation and learning plan. So these are just a couple of takeaways and then we want to invite questions and discussion on both of the talks both Elizabeth sent and our talk that we just put together and I think I speak for the whole team when we say we're very open to feedback we're really bought into the idea that Maggie mentioned at the beginning of bringing research to bear on problems and learning how to do better research by engaging in practical problems and also learning how to do practice better by by building knowledge from research. So we're really eager to hear what you have to say about this. Thank you. I guess I'll just start going through the questions. The first question that's in front of my is How did you collect the data and what frequency these seem me for these seem to be more valuable and useful than standard USA required indicators. So that was for the second presentation. How did we collect the data and how often that's a good question and it differs depending on what analysis we're talking about. So we've used this approach to study I don't know six or seven different kinds of questions. But in the case of agricultural financing which which I talked about today, we actually didn't collect any of this data. We did this looking back at the last five years based on some different kinds of surveys so we use only publicly available data. We looked at there were two major surveys, national surveys that were done on financing and then we had a number of different data sources from different pieces. And one of the strengths of this approach is that you can learn a lot just by looking at publicly available data. In other cases we drew on the data that were collected by activities that were intervening in the system. So specific activities in Uganda, we're collecting their, their standard M&E data and we pulled those onto maps and put them on to kind of a common map across multiple activities to see how this works. One of the questions that this approach helps to think about is if you're going out to collect new data how often do you need to collect it, and it has to do with the frequency of change that's expected in the system. I'll add something quickly here to, you know, obviously the COVID situation we're using we're collecting data in real time and taking it from various sources. So, the idea of the map is to help structure lots of information coming in as people are trying to make sense of what's happening. And one of the point I'll mention is like Erica was mentioning, you can target where data collection might be needed. And we did at one point, there was some question about how farmers were engaging markets. So we were able to target some specific data collection to help answer some questions that was useful then across various activities in the portfolio, various different groups that we're trying to implement things. So it also helps you collect data that's useful for many different initiatives different awards that are out there. Thanks. The next question is also for Jared and Erica, which statistical software did you use to create the maps. I think I might have clicked that one to get rid of it sorry. I see it at the top so. Okay, great. I didn't mean to get rid of it. I used actually the software to create the maps is called Kumu. It's really useful in that you can interactively kind of highlight different parts of the map and see causal factors and create these pathways and look at them by themselves or color them differently. It's been really useful, but it doesn't do any of the analysis automatically right you kind of tell it what to do and it helps you to interact with a map. And then to do analysis of data we just used you know standard software like our. So that entirely answers your question but I think Kumu is the one that's unique and interesting for this purpose. And I put a couple of links in a response to a similar question in the answer section so. Okay. Okay, this next question is for Elizabeth then I'll give Jared and Erica break. It sounds like there's tension complementary complementarity between the existing research knowledge and the realist part, which is based on what is learned from implementation case study is that correct if so how is that negotiated. Hmm. This is a tough question because I'm not 100% sure if I'm understanding it correctly, but I will take I will take a crack based on how I understand it so the existing research and knowledge in the domain of inclusive systems innovation. So that's what the question is about definitely was not framed in realist terms. So, from that perspective, yes, I mean what I, what I did was integrate across fields so I looked at existing evidence in inclusive innovation and ag systems that had been framed mostly in the terms of site specific case studies, where the research that was being undertaken wasn't realist research and where the categories I was interested in weren't clearly, you know, drawn out by the researchers, but, and this gets to another question that someone asked about the quality of that evidence. And this is spelled out in the paper in more detail but one of the criteria that I used for selecting cases was that these existing cases had been sufficiently well documented in peer reviewed research by multiple researchers. And not just one research team. So these were typically cases that had been implemented over 10 or 15 years that are well known cases that have been written about by, you know, the donor agency by the implementers by external researchers coming to do evaluation so I wanted evidence that was triangulated because I was relying on secondary evidence due to COVID. And while that evidence was not presented in realist terms there was enough of it that I could understand using my own analytical categories yeah I could kind of extract the relevant information for the categories that I was interested in so I'm not sure if that's if that's quite what the question was addressing but I kind of overlaid a realist conceptual lens and realist theory onto an existing body of evidence that wasn't framed that way. Thanks. The next question will be for Jared and Erica from chip wouldn't it be possible to build generate a map and identify constraints and barriers through qualitative measures such as interviews is a big quantitative effort really necessary. I'm so glad you asked that question chip. We've just written a paper on this kind of. So yes, you can definitely generate a map with qualitative approaches like interviews. And this is how we generated these maps in the first place. So the kind of what are the causal pathways and how do they, how do they interact with one another we did that through interviews initially through and then through a series of workshops with stakeholders in Uganda where people edited the maps. The place where quantitative data are really important is in the measuring of that map. Because what we've just talked about in this paper is that intuition about these complex systems, it fails in the presence of feedback and delays and all of this complexity. And so while we could map the causal structure we wouldn't necessarily be able to pinpoint where the barriers are to the change along those pathways. By including some of these measured data points we can, we could start to figure out what is surprisingly not working, and how that affects the feedback structures and the causal pathways in the map. So you can get really far with this approach and in fact a lot of the work that we've done has not used this quantitative measurement. And we learned quite a lot using qualitative maps and interpreting them with groups of stakeholders who have different knowledge about the system. But you can go one step further if you can add data at key points along those causal pathways to see whether in fact the hypothesized causal mechanisms are are working as expected. And that's part of how you target your measurement within this broad system is is to find measurements that will either confirm or refute those hypotheses about how change is or should be happening in the system. I hope that I hope that answers your question but I'd love to talk more about this at some other point. Great thanks. I see that Elizabeth is answering one of the questions so I'm going to go to the next question. It says from Amy and it says curious if you all have experience using these methods to monitor and evaluate programming that involves media interventions via TV radio digital etc. And if so, what did you learn from that experience. I think the short answer is no we don't have specific experience with the questions are disappearing. We don't have specific experience with with those kinds of media. So I can't give you any sort of detail, but we think that these methods are not that specific to the exact context in which we've used them. So the same kinds of data about that you would normally be looking at. And for those kinds of interventions to understand their effectiveness can be put onto the map as one or several, you know, elements the circles that we showed, and then put into context of what else is happening in the system to look at how that changes interacting with others. I think Jared might have more to say. So related project we didn't present today but we've done some work in the past on, as mentioned at the beginning by Maggie, the technology evaluation the adoption of different innovative technologies, and how word of mouth and potential radio broadcasting can help increase adoption of new technologies. So there are ways to incorporate that kind of an aspect to affect behavior change in markets. So there's related work that has on that. Great thanks and then we've got one more question from Amanda I think it's related to what Chip asked earlier but I'll ask it to see if any examples of using system map developed in a lower tech way say flip charts or sticky notes involving local stakeholders as a basis for identifying key indicators to monitor. Yes, that we did do might even have a picture of it in this talk. We used the yeah so here's to sort of a picture of this. We did develop the system maps in this sort of semi low tech way so we had drawn them already in a you know the first thing we did was diagram things on a whiteboard with markers. And then we put them into into a you know on the computer but then we printed them out and we brought them to a workshop and we used sticky notes as you can see and erase markers to draw all over the map and ask participants to edit it. And we learned a lot of lessons about what what groups of people resonate with and what they don't resonate with so we learned how to start a map from scratch, and how to get people to back through all the causal pathways, but we also learned that it's a lot faster if you start with something even if it's not correct and then you ask people to correct it using sticky notes and crossing things out and adding new things. So we have I think it would be a lot longer conversation to share all those lessons learn, but we have done it it has worked. It is complex and it requires a few hours to get started you can't do it in 10 minutes but it has been the way that we've done all of almost all of this work. Including a fancy way to print the maps and then have a layer over the top to draw on it. We have some good tips on that. Great. And then the last question before we end is from set in complex situations when access and collection of data is limited how do you weigh constraints with efforts invested to using system map and areas to focus in, as well as potential biases that we inevitably run into with all these tradeoffs. I'll open that to all three of you because I can, I can take a shot at this. I think there might be an aspect of this question that I'm that I'm missing. So I'd be happy to, to learn more. But we one of the key things that we thought about in developing this approach was how can we use it to balance to better invest resources for M&E. So how can we use this to figure out which parts of this enormous map actually need to be measured. And how do we best allocate you know scarce measurement resources to two different parts of the system or to doing some things in more detail and others in less detail. And there's no simple answer to say well it's always better to do it one way. But to we use the map to say which are the most important places to measure based on how they affect the key outcome and how they interact with other things. So if we are expecting something to happen here but we see that there's this dangerous barrier over here, we would want to measure something on what we on that expected pathway and then something on this potential barrier pathway to make sure that we understand which one is operating more, but we wouldn't necessarily have to measure every single step along each of those. Right. And then there's this concept that folks have talked about in systems for a while of Sentinel indicators right so you place these sort of sentinels around in the different important potential pathways that could be affecting your key outcome so that you can flag if one of them is turning out differently than you expected. One of the ways in which the system map is, it's not the answer, but it facilitates the conversation that gets you to the answer right by looking at it and talking to people around it now you have something to point out and say we need to monitor here, we probably need to monitor something over here but not everything so it, it helps to focus on that now your question about biases I'd have to think about that a little bit more I don't have a ready made answer on that one. One of the biases is, one of the keys is getting a lot of different stakeholders involved, we had a workshop with around 170 people, including market actors themselves to provide input so you know getting outside the box thinking different donors different groups, engaging the country to help reveal some of the biases and build a better collective understanding of the system. I might just add in that workshops can be used for theory of change validation as well so there's a process called outcome evidencing that we're currently using in a Gates funded project in India where we're taking an initial complexity where theory of change and then we'll be doing six month after action reviews in workshops that look just like these. But using the theory of change as what's getting validated so what would we hypothesize seeing and then are we seeing that are we not and do we need to adjust the theory of change based on emergent outcomes that the project is observing. Great, thank you. I see there's still one or two questions left in the chat we will try to answer them after the session. But thank you to Erica Jared and Elizabeth for coming and presenting your work and also for Maggie for facilitating this. We will as always share recorded version of this on program net sometime in the next week or so. And then thank you everyone have a great day.


The president's budget proposal eliminates public media funding. Learn how you can help at lptv.org slash action. Well if you wanted to know more about marketing and management than the 18th annual student achievement conference held at Bemidji State was the place to be. More than 150 presentations were on display by undergraduate students. Presentations were underway at Hague Sauer Hall where students covered topics on psychology, economic costs, and supply and chain management. Conference gives students the opportunity to polish up their presentation skills so they'll be prepared for future employment. Faculty was also on hand to show support for the students. We need the students to be proud of their achievements and this is one way we bring students to present their knowledge and expertise projects and share it with the community. Presenting in itself is a good practice to have be able to speak in public and it also helps. It gives you industry experience for when you graduate and to become employable. Some students worked on their projects for at least a semester and participation in the conference wasn't required. The conference was also open to the public. If you've enjoyed this segment of Lakeland news please consider making a tax deductible contribution to Lakeland public television.


Hello, I'm from the Australian government. Are you ready for Amnesia Day? Amnesia Day is our national holiday. When we come together as one to forget our dodgy history with the help of our favourite memory inhibitor. Ah, tastes like a peaceful settlement. Amnesia Day! Sadly, Amnesia Day is under threat. Each year, more and more Aussies are refusing to celebrate on January 26th. What's causing this perverse behaviour? Reading a history book. Some of these factards are spending Amnesia Day actually remembering what it commemorates, which of course is the day the British arrived and started colonising the lands of the oldest continuing culture on Earth. The kangaroo! Amnesia Day! Worst of all, city councils are responding to their community's concerns by moving events and celebrations to another day. But don't worry, here at the Australian government we know how to gently bring everyone together in harmony.  Because Australia Day is all about honouring what makes us great, our freedom. It is now mandatory towards citizenship only on January 26th. So come on, Australia! Crack a tinny! Show our baby sheep! Stick a flag on your car! Or your knob! And crank up the hottest 100! Which has now been moved to... What the f- It is now mandatory to have fun only on January 26th! You know what?  We'll change the date! Then we can pretend that we've fixed everything and ignore the structural changes our First Nations actually want us to make. Nah, just kidding!  Like by appointing this dickhead to be Indigenous envoy. Here's how you can help save amnesia day from all these woke c- Don't make the day political. Actually, please do it. Gomo really needs to stoke a culture or before the election. It's all we've got left. Don't question tradition. That's the sign of a mature thinking nation which is able to grow. Mature thinking nation. Bad. Rat-ass-dun-conscious nation. Good. And if anyone questions tradition, tell them to get over it and that it's all in the past. Sure, you'll be saying the exact opposite when Anzac Day comes round in a few months, but by then you'll have forgotten all about it anyway. Happy amnesia day. Authorised by the Department for Partying like at 1788.


All right. Thank you guys for joining in and thanks for the people who are joining in virtually as well. So I'm Narendra and without much of an introduction, let's get started with the topic for today. So this is a story. This is my story as a new programmer in January 2020 to being a full-time open source contributor by March 2021. So I have been programming since I was a teenager, but I never thought about having to do programming as a career. So when I decided to actually be a software engineer, I had no idea how to start. I had no idea how to find jobs. I had no idea what should I learn. I had no idea about anything. So I was scrolling through YouTube and I found that contributing to open source is a good way to improve your skills. It is a good way to get yourself familiarized with actual software that is being used by people. So to contribute to open source, I need to know something. But so I started by pushing all of my source code on GitHub. I started pushing my toy applications, which I was using to learn stuff. I was pushing everything on GitHub. And eventually, a couple of those projects turned out to be somewhat moderately successful. But I was still unsure about how I can contribute to other open source projects, how I can contribute to these big open source projects that everybody is using. So I had no idea what to do. So that's when I found out about the Linux Foundation mentorship program. So I was thinking, which project should I contribute to? Do I have a skill to contribute to such open source projects? I was thinking along these lines. And that is when I found out about the Linux Foundation's mentorship program. So the LFX mentorship program is an open source mentorship program hosted by the Linux Foundation. And the program, what it does is it connects mentees to open source projects and with their mentors. So you can basically contribute to open source projects. And you will be assigned a mentor. And they will help you contribute to those open source projects. So I stumbled right into the LFX mentorship homepage. And it shows a list of projects that are participating in the program. So there was my answer to my first question, like, which project should I contribute to? So all these projects, they wanted people to contribute to it. They were very particular about what kind of stuff, what kind of contributions they wanted people to make. So yeah, I found this list quite helpful. So at that point of time, I was learning new stuff. I was learning about cloud native stuff. So I was looking through all these projects and seeing which project I can actually contribute to. And as you can see, it shows the required skills, what they're expecting, all sort of things. And then I applied to a couple of these projects. And I actually started contributing to making small contribution to some of these projects. So I was a mentee for the Mashary project, which is a cloud native computing foundation project. So I started by making really small contributions, small bug fixes. Then I eventually worked my way up to make bigger contributions. And after a couple of contributions, I was able to apply for the LFX mentorship program. And I got accepted as a mentee. So in March 2021, I was an LFX mentee for the Mashary project. And during my time as a mentee, like, I learned a lot of stuff. It wasn't always about learning how to code, but I learned a lot of other stuff as well. So I learned a lot of other stuff as well. I was speaking at conferences like UConn. I was talking about my work. I was representing the project in front of the whole community. So without this LFX mentorship program, I would have never gotten a chance to discover such projects and be part of such projects. And I was also giving talks at other conferences. So all of this happened after I was part of the LFX mentorship program and after I started contributing the project. And I made a lot of code contributions to the project as well. This helped me learn new skills, learn actual skills that are practical and that can be applied to building real world projects. And I was doing open source stuff for the entire day, like the Linux Foundation mentorship program provided me a stipend. So I didn't have to worry about finding a job. I can just contribute to open source projects, but it also was sustainable since I had some sort of an income. So I was a new programmer in January 2022, but 2020, but not anymore. So I was working in open source full-time. So right after the program, the company behind the project, the company who actually initiated the project before it was donated to the foundation, they hired me to work on their open source project full-time. So which was like the dream. So it was like a dream come true. To this day, when I work in open source, I can't believe people actually pay me to do that. I might, I would actually do this for free, but thank God there are people who are willing to pay me for it. So yeah, that's how I went from open source, noob to a full-time open source contributor. And open source is about paying it forward.   So people were helping me out. People didn't complain, they didn't get angry at me. They were helping me make better contributions. They were helping me a lot. So to give it back to the community, what I decided to do was to become a mentor. So in 2021, I returned back to the program as a mentor. And so I'm a mentor of two projects, two CNCF projects. So coming back to this program as a mentor was a way for me to help the community and give something back to the community. And by being a mentor, I also learned a lot, maybe more than what I learned when I was a mentor. So I learned a lot of leadership skills. I learned to work with a lot of people. By teaching them, by helping them, I was able to reinforce my knowledge a bit more. And this also gave me access to a lot more opportunities with this newly acquired skills. I was getting a lot of job opportunities and speaking opportunities, all sorts of things. I also gave talks about contributing to open source, helping people by sharing my experience as an open source contributor, my journey through open source contributions. I was also writing a lot of blogs and trying to help more students, more new contributors make contributions to open source. So programs like LFX actually help the LFX mentorship program actually help making the open source communities more accessible to people from around the world. Like it might seem like this is similar to any other internship program. It might seem similar to any sort of other internship programs that tech companies offer. But from people who are from the parts of the world where they don't have such opportunities, these programs can act like the catalyst. They can be the game changer when it comes to getting new opportunities. They can be the ladder that the people need to get access to such opportunities. And in the last couple of years, I'm seeing a lot more people from very, very different backgrounds being part of these programs. And these programs actually helping them to level up in their career, helping them to gain access to opportunities that they normally couldn't have. And it really encourages me to see that 82% of contributors, new contributors, feel really welcome within open source communities. But at the same time, the 18% who are feeling unwelcome are from disproportionately underrepresented groups. So there is definitely a bias. There is definitely a disproportionality between in this metrics. So there is still a lot for us to do, like programs like LFX could actually help settle this imbalance and make open source more inclusive, more welcome to people from all over the world regardless of where they are. But still 89% of people contributing to open source, they feel that they are making a positive impact in this world, which I think is what is most important. So people are contributing to open source and they are happy that they are doing it. And that's all I have for today. And if you have questions to me, you can ask them now or I'll be hanging out over here. And for the folks tuning in online, you can reach out to me on Twitter. Thank you. We have a question. I think they might, should work. It's a great journey for, it's okay. Oh, yeah. Yeah, you're right. So your journey is very good and very impressive. So what's your first programming language? Where are you started? That should go back to basic using QBasic. So print, what is your name, that sort of stuff. But like, no, basic language. Basic language. Then like, the actual language that I learned was probably C++. It must be GoLang. GoLang. Go, yeah. That's good. So you are a student enrolled in some programs? No, like now I work for the Apache API6 project, which I am there in the p-shirt today. So yeah, so I work for a company that actually is building some software on top of API6. But I work on the Apache API6 project. So this open source projects? My work is to work on open source projects. I try to be a mentor today, but like I stopped working with the CNCF projects, but like I plan to be a mentor for more open source projects like in the next years, yeah. Thank you. Yeah, like because, so I actually quit my, so let me repeat the question. So the question was, was being a mentee a full-time job for me? So I worked in renewable energy, solar panel, that sort of stuff. So I quit that job and I was looking to transition into software engineering. So I didn't have anything else to do. So I worked as an LFS mentee full-time. Yeah, that's a really good question. So LFX mentorship has two different schemes. You can work either in full-time or as part-time. So even if you are working professional or even if you are a student, you can like work 20 hours a week or maybe less depending on the project and like how you're interacting with the maintenance. And you can actually work for less and like you can complete the program within six months. So yeah, it is open for all. You don't have to be a student. You can even be a working professional and work part-time for this. Good, thank you. Thank you, yeah. As a team or is this hired as one mentee and pushing them all the stuff? So the program, so you can actually specify what project you want the mentees to work on. So the LFX, so when you list your project in the LFX portal, you are actually listing it as a project, as a sub-project. So maybe you want a new feature in your project and you can list it there. And the mentees who are participating, they will work on that particular feature. And the goal is to have the mentees working on the project even after their mentorship program. So that is to ensure sustainability. So these mentees will keep contributing to the open source project even after their mentorship period ends. And eventually they might transition to be maintenance of the project and they will help us to work the project in the future. Are there any questions online or? No. So thank you everyone for joining.


Over the weekend, NIO held its annual NIO day, so here's a quick recap of what was revealed during the event. NIO started by revealing its newest EV, the ET5, which uses AR technology instead of screens to allow drivers to keep their eyes on the road. The ET5 is currently priced at $51,250 and deliveries are expected to begin in China by September 2022. NIO would also reveal that deliveries for the ET7, which was revealed in January, are set to begin on March 28, 2022. Lastly, NIO would announce plans on expanding to over 25 countries and regions by 2025.