How are you guys doing? Oh, thank you. So cute. He's just leaning against me. Oh. Okay, that's enough of that. Oh, fine. Good. Tonight, a week from today, drum roll. That's great. Yes. October fest live cast in a few minutes. So I've been rounding things up for that. Everybody tune in. Stay on YouTube. They're so such a good information there. Oh, sorry. I'm on my phone because my camera decided to stop working today. Oh, no. But I have everything ready to go. So we have just we're counting down to wiki tree the day as Betsy and Greg mentioned. Here's the the help page. I know seven days, right? The registration link is in the chat. It's pinned to the top for y'all. And we have a winner.  The question was, do you have Italian ancestors? Congratulations. How appropriate. We were just talking to on on discord and he had just answered. So it was just really funny when Betsy picked what the random number would be that it ended up being frank when I just got through talking to him. So Frank, of course, you know, you just send an email and she'll get you taking care of for your prize. So I wonder if he is tuning in today. Looks like you got Kathy and Steven and joy. That's just funny. Okay. And we're we're getting close to Halloween too. So question. Wow. Ooh. Like the Salem witch trials, you mean. Yeah. Do you have any witches in your family tree? So that there would be other events in history where witches have been hunted and persecuted. Yeah. Yeah, that's true. Yeah. Right. Yep. So that's the question for today. All you have to do is comment on the video below and Twitter. So many different social media channels you can pick from, whichever is your favorite. And we'll do another drawing tomorrow. Sounds good. Yeah. And don't forget that you can answer no, I don't have any witches and it will still count. That's right. Very good point. Thank you for bringing that up. That's right. Yeah. All right. Well, thank you guys for joining me. Okay. See everybody tomorrow.


Hello everyone, good afternoon, good evening or good morning depending on where you're joining us from today. Welcome to Engineering for Change or E4C for short. Today we're pleased to bring you this month's installment of the E4C seminar series. This series was spearheaded by ASME's Engineering for Global Development Research Committee and its purpose is to intellectually develop the field of engineering for global development. We host a new research institution monthly to learn about their work advancing the United Nations Sustainable Development goals and more. Today's seminar is presented by Dr. Jeffrey Walters from the Civil Engineering Department at George Fox University. My name is Yanaranda and I'm the President of Engineering for Change and be one of the moderators of today's seminar. The seminar you're participating in will be archived on E4C site and on our YouTube channel. Both of those URLs are listed on the slide. Information in upcoming seminars is available on our site and E4C members will receive invitations to those seminars directly. If you have any questions, comments or recommendations for future topics and speakers please contact our team at research at engineeringforchange.org and if you're following us on Twitter today please join the conversation with our dedicated hashtag, hashtag E4C seminar series. Before we move on to our presenter I'd like to tell you a bit about E4C. E4C is a knowledge organization, digital platform and global community of more than 1 million engineers, designers, development practitioners and social scientists who are leveraging technology to solve quality of life challenges based by underserved communities. Some of these may include access to clean water and sanitation, sustainable energy, improved agriculture and more. We invite you to become a member. E4C membership is free and provides access to news and thought leaders, insights on hundreds of essential technologies in our solutions library, professional development resources and current opportunities such as jobs, funding falls, fellowships and more. Members also receive exclusive invitations to online and regional events and access to resources aligned to their interests. Please visit our website to learn more and sign up. Our research work cuts across geographies and sectors to deliver an ecosystem view of technology for good. Original research is conducted by engineering for change research fellows annually on behalf of our partners and sponsors and delivered as digestible reports with implementable insights. You are invited to check out our research page. You see the URL listed on the side to explore our field insights, research collaborations and review the state of engineering for global development, a compilation of academic programs and institutions offering training in this sector. If you have a research question or want to work with us on a research project as a fellow, please contact us at researchandengineeringforchange.org. And on that note, I'd like to share an overview of our current cohort who is actively doing ecosystem research in 2020 from around the globe demonstrated here through this map. Nearly 60% of them are women. They represent a variety of skills, including interdisciplinary contributions from aerospace, architecture, civil and environmental engineering, nutritional food science and software engineering from all around the globe. If you're interested in becoming a fellow, please explore the URL on this slide and please do feel free to apply. Now, we want to move on to a few critical housekeeping items. I know most of you around the globe are now familiar with how to use Zoom given current circumstances, but we are really aligned to hear from you and make sure that you are putting your questions and your comments into the right place. So please kindly click on the chat window and share with us where in the world you are joining us from today. We want to see where you're from. So I'm here today in Brooklyn, New York. I see folks from Oregon, from London, from Arizona and California, from Sweden, from Calgary in Canada, and more Brooklynites. Welcome, Texas and Switzerland, DC and Kenya, Nairobi, Denver, Karachi. Welcome. So fantastic. Welcome. We are so thrilled to see you here from Nigeria and Pennsylvania across the globe. So, so awesome. Just as a reminder, you can use the chat window to share remarks during the seminar. And if you have any technical questions, you are very welcome to send a private chat to the Engineering for Change admin. Just search in the participants. If you're listening and you have any troubles, try hitting stop and then start. And then please do use the Q&A functionality for any questions that you have for our presenter today. This will allow us to keep track of those questions. All right. Oh, and so thrilled to see one of our past contributing editors has also joined us as a listener. We're so thrilled to have you. Hey, Sean. All right. So with that, I am so honored to introduce our presenter and also our fearless lead in this as next. So Dr. Jeff Walters is an assistant professor of civil engineering at George Fox University. His research seeks to develop and refine decision support techniques written in complexity science and systems thinking to improve engineering practice and policy for sustainable rule and urban infrastructure system design and management in developing work contexts. His research has been applied within the sectors of WASH. For those of you who are not familiar with that, that's water and sanitation and hygiene, energy, food and engineering education. And the man who needs no introduction, but we'll get a brief one is our co-moderator, the co-chair of our Engineering Research Development Committee, Dr. Jesse Austin Breneman, who's an assistant professor of mechanical engineering at the University of Michigan with his beautiful bio, which you wish you can read at your leisure. So with that, I'm going to go ahead and stop sharing my screen and turn it over to you, Jeff, to share your presentation. Thank you for that wonderful introduction. By the way, this is this is this is really an honor for me. Thank you. Let's not start the problem. We have enough problems in our lives right now. Right. So again, it's it really is an honor to be presenting this research that and to say that it's my research is seems a little bit disingenuous because of how many folks that I've worked with on this work. And so the main goal of this presentation really is to engage in this discourse with you all. Hopefully get some more collaborators on more of this work moving forward. So this research really focuses on applying systems systems tools to improve upon the sustainability of service delivery, not just of water, but it could be it's issue agnostic. It's energy, it's food, it's any sort of structure. So let's get into it. So here's a quick overview, right? And really, I think the case is pretty easy to make for a need for systems thinking and rural water supply could be sanitation as well or energy food. There's a need to understand systems. And I'm going to highlight two approaches that I've been working on with collaborators that hopefully this is this stimulates an interesting conversation. Well, a stakeholder driven approach that is sort of getting stakeholders local stakeholders thinking and systems and extracting systems insights. And then an evaluation of stakeholder understanding, which is a bit of an abstract idea. And I look forward to diving into that a little bit. Do these approaches actually improve stakeholder understanding? And then briefly highlight where the blind spots are and areas for future research and collaboration with Paul that I hope gets us going on a good discussion. So the problem, okay, so you guys have probably seen this many times, but there's a lot of people that don't have access to clean water, right? Despite many efforts, there's still about 800 million people that don't have access to water. That's unfortunate, right? But I think the more unfortunate aspect really, and in my sharing my screen, are you guys all seeing this as well? So I need to get this out of here. No, no, we just see the problem. And I thought you were seeing my chat window. We are not seeing your chat window. Thanks. So yeah, the big issue here is that the interventions in the past have not been super successful. A large failure rate, especially in Sub-Saharan Africa, water points failing, gravity systems failing, etc. And so what comes as no surprise is that these failures are a result of a complex interaction of factors and how actors interact within that system. There's interlinking interactions of factors. There's dynamics that exist between those factors. And ultimately what this is is sort of complexity. There's a complexity that plays a complex adaptive system is what it's been called in the past. That makes it hard to plan for sustainable interventions, both in water as well as other types of development pursuits. So regarding water specifically, how do we plan and manage sustainable water services in light of these complexities, kind of learning to dance, so to speak, with the system? And so that's what my research thrust has been over the past eight years. Is it possible to model the system? And again, these systems are social systems. They're political systems. They're economic systems, environmental systems, technical systems, supply chain systems, all sorts of different systems coming together to affect sustainability. Is it possible to do that with local stakeholders in an accessible way, engaging the intimate understanding of local stakeholders in that process? And then is it possible to evaluate whether the approaches did what we hopefully did in getting stakeholders thinking in systems or at least changing their understanding for how different factors interact to impact systems outcomes? And again, so what I mean by system isn't necessarily just the water system itself, but instead sort of all of the sort of soft systems that surround the maintenance of that system. And then lastly, how does this impact? This is what we want to get at, right, service sustainability and high service performance. How does that change in understanding impact actions that impact service delivery? And really, the whole idea of this research is to automate this process to make it so that you can really go through this process of mapping, evaluating understanding, taking action, evaluating action, mapping, and so on and so forth. It's a cyclical process of adaptive learning and action. So that's kind of the high level view of my research. And I want to highlight, this is research that I'm standing on the shoulders of giants, both my colleagues as well as years and years and years of systems research in this area of complexity science. So I just wanted to highlight that. This is a really, if you guys have a chance to look at this at some point, just look at this up, the map of complexity science by Frank Kostlany. This is an interactive map. You can click on all these different tools that have been used over the past, geez, 80 years, 70 years. The tools that I am exploiting, so to speak, are those in system dynamics and in soft systems analysis. So soft systems analysis is really just getting folks together and talking about and diagramming, could be pictures, could be arrows, could be words, just getting folks kind of thinking in systems in a very, very abstract qualitative way. Now, system dynamics simulation and system dynamics modeling is largely focused on parameterizing these complex systems, actually putting quantitative variables into the model and using differential equations and seeing how things change and doing hypotheses in that way and testing hypotheses. My research focuses kind of at the nexus of these two approaches where the goal is to get local stakeholders with their intimate knowledge, understanding of the system, engaged in systems diagramming and systems modeling to make better decisions based on their understanding. So it's both kind of quantitative and qualitative to make decisions based on leverage points that are identified. So this is known as sort of structural analysis or group model building system dynamics modeling. And the idea is this is sort of a stakeholder-driven structural fact analysis for lack of a better word. And the goal of this approach is to develop a sort of a robust systems diagram or factor map that represents sort of stakeholders understanding of their local system. And then to then, once you have this map, pull out leverage points, pull out feedback mechanisms, influence between factors, centrality as a means to identify leverage points from a systems perspective. So that's kind of the goal of this. This is based on some previous work that I've done in Nicaragua. And for lack of a better word, we're just going to call this stakeholder, a stakeholder-driven structural factor analysis. And this approach that I'm continuing to develop with partners is quite simple. It involves brainstorming factors, getting stakeholders in a room, brainstorming factors that impact a particular outcome, let's just say service delivery of water. Condensing those factors into a digestible form, obviously, you can get a lot of factors, brainstorm, but the goal is to get between 10 and 15 factors to maintain some complexity and nuance, but not having to be unmanageable. And then mapping the interaction between those factors, you can see in that picture that's kind of showing a facilitator doing that. And then performing a systems analysis, and that's what I showed before, really sort of dissecting that systems diagram to pull out sort of leverage points and places to intervene in the system. And then sharing those results with stakeholders. So I wanted to run you through what this looks like really quick. So again, the goal is to get to this factor map that you can then dissect to pull out leverage points from a systems perspective. So first, identify the factors. So you basically ask the stakeholders, what do you think are the factors that in this case influence sustainable water services in your regional context? Where one of those factors is also the outcome, I'll explain why that is important in a second. Okay. And then you engage them in a process that's somewhat arduous. It takes between four to six hours, depending on the size of the matrix and the number of factors that are considered. But you fill out this table. And this is called a cross impact matrix.  And we've done this with without any access to electricity, right? You just draw this on the board, straw grid. And you get the stakeholders sort of considering every single interaction that could exist in the system between these factors. So let's talk about what this might look like. So one of these cells, let's just say community and operations and maintenance. So what that cell means, and you step the stakeholder through this process, you go and say, okay, based on the current conditions, how do you think community influences operations and maintenance? And this in and of itself is quite interesting. It's an arrow between community and operations and maintenance and influence between community and operations and maintenance. You say, how does that, what is the effect there? Is there an influence? Yes or no? If there isn't, great. There's no, there's no effect. If there is, then you say, okay, is there a positive or negative influence? In other words, if one increases, does the other increase one becomes better? And does the other become better and vice versa? And a negative would be kind of an an inverse or relationship was one goes up, the other goes down. This gets that dynamics. Okay, I mean, we can talk about that. We'll talk about that in a second. And then you ask, what is the strength of interconnection? Is it a weak, moderate or strong strength? And this allows us to really kind of add a little bit of granularity in terms of which factors are the most influential in the overall system. And what's really great about this process is the robust conversations that result from this, from each cell, each of these cells, to my chagrin at times, because it makes it take a lot longer, have had 10 minute long conversations about just one cell. Why does that, does that factor influence that fact? And the idea of doing this in a pairwise way is, is to extract local really rich information about a very complex system. We as people can't conceptualize all these interactions at once, but we can at least consider individually a pairwise interaction between one on the other. And this circles back around. And this would also have operations and maintenance on community as well. So what results is a, yeah, as an impact matrix filled with a bunch of numbers, which is interesting. But what it does is it allows you to sort of draw the factor map or the, or the model and then to pull out those sort of insights, those, those leverage points. So I'll, I'll step you through a couple of these leverage points. But first the factor map, you look at this thing, you go, this thing is, I can't get, I can't get much from this. And that's usually, and that's okay. A lot of times the main thing that we, that we get from this sort of factor map is to show the stakeholder group. Oh my gosh. Look at how interconnected these things are.  Oh my gosh. They're, this shows that this is, this is not just as a simple task here. It's not just one simple factor influencing service delivery, for example. But then we, we stepped the stakeholders through this process of looking at this idea of influence and dependence. A factor influence is how many factors in the strength of interaction a factor has on other factors, sort of the arrow leaving the factor and the dependence is sort of the influence from other factors on said factor. And if you map this in an influence map, it's possible to sort of identify leverage points. So let's, let's go through this. So what we see here is if you have a high influence and low dependence, that would mean that you would probably want to target those factors because they wouldn't feed, there wouldn't be any feedback on those, on those factors and influence this in a very controlled way. Let's look at this example of water service delivery. So what we see here is community and capacity building coordination, for example, would be a place to intervene in the system and really, really fortify, so to speak. Whereas we see in the upper right that policy has both high influence and high dependence, which would mean it's very important. It's, it would have a large influence on the rest of the system, but in and of itself circles, it circles back from all the other interactions back on policy. So there's a potential volatility that that is or an instability if policy isn't protected, this system could spiral out of control. And then in the lower, lower right hand, we see what we would expect where water service delivery and operations and maintenance is highly dependent on sort of the outcome. Whereas the lower left is generally low significance, which in this case would show the users are not super impactful on the system as a whole, which folks may disagree with. But that's just an example. Another way to look at this is as a network and to use sort of network theory centrality to pull out which factors are the most influential on the system, most important. This is using between the centrality or the bridging of the importance of factors ability to bridge other factors. And then looking at that and Sanke, those are the leverage points of the system. So we would see here that financing coordination based on this analysis are the leverage points. And then lastly, the analysis of feedback loops actually systematically pulling out feedback mechanisms, specifically feedback mechanisms that impact the outcome variables of sustainable water services in this case. And then systematically identifying which ones are the most important, and that has to do with a lot of, with normalizing the influences and sort of, there could be hundreds and hundreds of feedback, but the goal is to sort of identify 10 feedback services. An example here is water services continue to function well, operations and maintenance will be easier to do or done in a more effective way, which will make water, water services more sustainable. So that's an example of that reinforcing R3 loop there. So that's another analysis we can do. And again, the goal of this is to look at the system as the stakeholders have conceptualized. It isn't necessarily the real system, but it's a system that is conceptualized by those stakeholders and identify the leverage point based on the interaction between factors, not just saying finances is important, finances is important because it interacts with factor A, factor B, factor C, et cetera. And so doing through these different analyses, it is possible to hone in on really what are, where are the leverage points using these, using these analyses in concert. We might say in this case, let's focus on policy, community and regulations, because it has a high influence, low dependence, high centrality and a high amount of, and a high reference and feedback, or a high feedback strain. So this has been done in a number of different places between East Africa, Central America and South America. And, but it's largely been research that has been done with the help of funds from USAID through channeled through the sustainable wash systems learning partnership, which is working in East Africa and is, was a grant that was won by the University of Colorado, and I've called University of Colorado Boulder and I've been working with them over the past four years on this research. So thanks to those, those folks. So future research in this area stakeholder driven structural factor analysis is to do more of this, continue to, to refine the approach, evaluate impact, we'll talk about that more in a second, and really to, to refine the process. And what this requires necessarily is to develop a software. I'm sure some of you computer science folks are looking at this, and this could be automated. Well, the process of developing these diagrams as a structure, as a, the civil engineer or civil systems engineer, we do is in a very brute force way. And it takes, it can take hours upon hours upon hours to develop these models. And so we don't oftentimes get to share those models to the stakeholders of real time, which is so developing an icon based software that we can run stakeholders through this process. And it's an output real time analyses of the system so that we can discuss them with everybody in the room. That's where the system. Okay, so that was talking about sort of how to get system or how to get stakeholders thinking and systems and extracting systems insights to make better decisions. Now, the other part of my research is focused on, well, what is the impact of these interventions, not necessarily factor mapping, per se, but other systems approaches getting stakeholders thinking about the interaction of factors doesn't actually impact their understanding. So I'll move on to this, this part of the research here. So the goal here is to evaluate whether stakeholders that have engaged in some systems thinking approaches, whether it's talking about social network analysis doing fact talking about life cycle analysis, et cetera, engaging systems in thinking in stakeholders didn't actually impact their understanding of the interaction of factors. And so the unit of analysis for this research is to really say, we can extract their mental model, this comes from social science theory, system dynamics theory, essentially a mental model is the way in which a person conceptualizes the interaction of things that lead to a particular outcome. We all have mental models, whether it's what we're going to go to the grocery store and buy, the food we're going to order, who we're going to marry, what house we're going to buy, et cetera. And so if we can extract this mental model from people, it's possible to evaluate the shifts and understanding that a particular intervention, systems intervention had on that understanding. And so this has been researched, there's been largely with the help of a sustainable wash systems partnership in Ethiopia, Uganda, and Kenya with a number of local stakeholders, we've interviewed these folks. And so we're talking handpump mechanics, local government officials, and we have over 500 pages of text so far. And this is the goal is to interview these folks at three different time steps with the same questions, and to see if their understanding has shifted on the interaction of factors. So this has been done or will be done on three different time steps. So far we've looked at baseline and midline, that's between 2016 and 17, 2018 and 19, the end line interviews have yet to be done, they're going to be done in the next couple months. And the interviews that we asked these folks, in these interviews, we say, okay, what do you think are the main challenges impacting service delivery, water and sanitation? And what are some of the solutions that you would suggest to address these challenges? And in getting them talking, it's possible to sort of in transcribing what they say, extract their mental models. So if let's just say a person said, the sustainability of water services is driven by the ability of the water user community to operate and maintain the water system, that represents a causal statement. And so it's possible to in asking these very pointed questions, to say, okay, that represents a causal statement. So if we can, how do we can we extract that causal statement? And we see there's actually an approach called purpose of text analysis and system dynamics modeling, where you actually code these, these causal statements. So what we would see here is, okay, water user committee capacity causes an effect on operations and maintenance, causes an effect on service sustainability. In other words, you can pull out systematically cause and effect factors that then led insight into how the stakeholders understand how factors interact. And you do that at different time steps. It's possible to see changes in those cause and effect factors to see if there's been a shift in those interactions. And that can be done based on looking at the challenge and solution questions, seeing how those factors are different for those different questions, and also look at different stakeholder groups and see how different stakeholder groups have shifted their understanding over time, among other things. And what's interesting here is you do this enough times, and this takes a long time and I'll talk about that in a second, but you code these transcripts. Let's just say you get 20 causal statements from one interview. You do this with 30 stakeholders for a particular region and you get 600 interactions, right? You get a rich, rich representation of that stakeholder group's understanding of the system, which allows you to thereby evaluate understanding and how shifts and that understanding on factors has taken place. So this is a, a graphic that would show, and again, the goal, the crux here is trying to show how understanding on factor interactions has changed. But this is an example of how we're presenting this for a case in Kenya where the factors that were identified through the coding were operations and maintenance, service sustainability, finances, et cetera. And we can show here a change in the reference, the percent of reference to effect factors and cost factors. So operations and maintenance you can see here has decreased as both a cause and effect factor. Let's look at this some more. We see here the service sustainability. What this, what this chart is showing is if the solid line, it's an increase between the baseline interviews and the midline interviews and a dotted line or dash line indicates that there's been a decrease in reference. So we see service sustainability becomes more influential in the eyes of these stakeholders. Finances becomes more influential. We see coverage and access becomes more influential. It becomes more of a cause factor influencing the other factors. Conversely, we see external support becoming less influential and less affected. And we see here the political influence becomes also less, less, less influential. Again, this isn't saying that the system itself actually is changing in this way. It's showing that the stakeholder's understanding of the system has shifted. And like I said, you do this enough times and you get some really, we're getting some really, really, really interesting insights on how people's conceptualization of the system is changing. It's a very robust analysis based on both aggregated stakeholders as well as stakeholder groups that can be parsed into. And that's great, wonderful. But this takes forever. We have 500 pages of text that we've been coding. And this takes at least two hours per interview. And this is thousands of hours of work. And so the goal is to, in making this replicable in the future, is to automate this process, to be able to allow folks to be able to import transcribed interviews into a software, so to speak, that outputs those interactions to evaluate understanding or to potentially evaluate the system itself. And so what we're investigating is the use of natural language processing to automate that process, importing the sort of the transcribed interview and outputting the factor map and the analysis. So lastly, this idea of seeing if there was a connection between understanding and service performance, right? The holy grail here is we want to impact service delivery. That's what the outcome is, what we're interested in. And at this point, it's still a bit of a black box, largely because to identify and say, okay, whether a stakeholder groups understanding informed actions that led to improved service delivery or sustainability, it takes an enormous amount of time both to do as well as to see if it actually materialized. So sustainability implies that a service worked for a long period of time. So evaluating sustainability in and of itself requires time, and we haven't been able to fully see that yet. Even this USAID partnership, we've only had five years to look at service delivery and see shifts in delivery. And that's not necessarily a long enough time to see if there was actually the impact of this desire. And then this is even harder, attributing shifts in understanding to actions to service delivery. It's a quite spurious and difficult task to do that, but is an important task nonetheless. And so future work and steps forward is to really develop more case studies, more and more case studies, engaging with more stakeholder groups, more practitioners to map the system, get stakeholders thinking in systems, and to evaluate those efforts. And so what I'm doing with a number of colleagues is to develop a consultancy company that develops both the software to automate these processes of mapping and evaluating stakeholder understanding, but also to just get practitioners to have their stakeholders think in systems or getting local stakeholders to think in systems to decide having an impact on service delivery now. We would expect that there would be positive impacts. And it's sort of a separate arm of that consultancy and development of that software is to continue to do research on evaluating the impacts on stakeholder understanding and on service delivery and in refining the process of engaging stakeholders and systems. So that's really where it's at currently with the goal of again, automating the process of mapping, evaluating understanding and evaluating service delivery outcomes in a way that allows you to adapt to changes in the system. The system is always going to be changing. Understanding is always going to be changing, but to be able to be nimble and evaluate that in a very quick way will lead to this sort of adaptive learning and action that will lead to systems change, positive systems change. So thank you for listening. That was a lot of information to throw at you. Again, the goal was to give you enough information so that we can have a more robust conversation on how to collaborate or how to make things clear. And I want to highlight two of my collaborators, Dr. Nick Valcourt. He and I are starting this company that I referred to. And Dr. Walker Orr is a computer science faculty member at George Fox University. He's helping with the natural language processing, the machine learning aspects of this research. So thank you. I look forward to chatting more about this with you all. Thank you so much, Jeff. We're going to invite questions. So please do share your questions in the Q&A window. There's a number of them there already.  And perhaps I'll share my screen here. Oh, if Jeff, you wouldn't mind stopping sharing yours. All right. Over to you, Jesse. Yes. First of all, Jeff, thank you so much. This is the type of research I wish I was doing because it's amazing. So really great stuff. I really like the idea around systems level thinking. We have a lot of questions here around it. I have my own questions. So I'm just going to hijack everyone's questions, ask my own, because I'm so interested in it. But I'll start with theirs so I don't take all of the time. One of the things that I was wondering about and is asked here is what about situations? So you're talking about Uganda, Kenya, these are places in which people do speak English and we're part of the Commonwealth at one point. So English is a language. So perhaps there's less of this issue. But in places like you mentioned Nicaragua or in South America, I'm wondering about the translation in terms of, you know, you're creating these system models, but the language of the factors becomes quite important then, right? So I was wondering if you could speak on two things. So I'll just start first with just translating in different languages, right? So how do you deal with that situation as you come up with like trying to think about these, these ones, both for interviewing and for creating the stakeholder maps? Right. And so this is, this is great. So when it came to engaging folks in Nicaragua, I did some of this in Chile, I speak Spanish and I was, so I was able to engage with stakeholders in the local language. Obviously keeping it in the local language is really, is key because language has meaning, right? And the meaning of these different factors, while there may be similar factors in different contexts, the meaning of those factors is very context specific. The interactions are especially context specific. So when it came to Latin America, it was, it was, I was able to engage in that way, but when it, it was very much different when it came to engaging with folks in Ethiopia and in Amharic. We don't, none of the researchers knew that language. And so the way that we did that, and ideally the goal is to get local facilitators to do these factor mapping workshops and to engage with the stakeholders, first and foremost, but especially if it's a different language, you need to have different, you need to have local folks that, that know the language, that know the context, that know the stakeholders doing, actually being facilitators, that you don't need to have some gringo doing these, you know, these workshops. And while, yes, certainly there can be, we've sat back in the background and try to not be, not to, not to be a hindrance, because obviously always having someone different in the room makes things weird, and excuse the conversation, but to have someone in the back at least taking notes and then debriefing with sort of the facilitator afterwards. And again, one of the, one of the blind spots here with this, with this research so far has been the ability to, how do you actually engage the stakeholders afterwards and evaluating the system's outputs, because usually we've had to be like, okay, thank you for engaging in the factor mapping exercise. Now we've got to go back and just kind of process these diagrams and then hopefully present them back to you at some point. And in many cases, we haven't been able to present these insights back to people. And the main insight that folks sustainably get is, well, that system's really interconnected and, wow, one or two factors were really influential. But so that said, that the translation and sort of the engaging with the, with the stakeholders is absolutely paramount. And so you need to have a really, really good facilitators to engage with them. Because you're at each cell, each interaction has to have consensus. You want to get consensus with the stakeholders. And it takes a long, a long time. It takes a lot of rapport with the, with the stakeholders.  It's, it's, it's really, it's really sort of an art that the facilitator needs to know how to do. That's a really good question. And it's still something that we're working on. It's, it's how do you, how do you develop sort of a robust training for facilitators to be able to do these workshops? Yeah, thank you, Jeff. I think that's a really great point thinking about the quality of the facilitator and thinking about the team that you have, the impact on your ability to conduct it. It's not just the process that's important. It's like the team and the implementation of that process. And the details around that seem incredibly important based on what you're saying. And I agree with that. I think that's, that's a really great point. And I think it addresses many of these, these questions that are coming up around perhaps the quality of the interviewer, the style of the interviewer addressing variation as you're collecting this qualitative research data. You know, in those methods, you do have to think about your own positionality and perhaps the nationality of people on your team and their characteristics with respect to the community. And so, you know, I think what you're suggesting is perhaps having local facilitators or people that are really engaged with the stakeholders and have a connection with the stakeholders prior to your work may, may be addressing some of those issues already. So I think that's a, that's a really great point. We do have a couple other questions. I think one question that's important is to think about, you talk a lot about, okay, well, our goal is to get people to do systems thinking basically is what I heard. Right. So this is one process by which perhaps by going through this, people can be reflective about thinking, doing systems thinking themselves as stakeholders. Right. So one question that I would have or that, that came up here is what are some of the common roadblocks to people thinking in the systems way. So do people go through this? Do they improve on their systems thinking at the end? Or, you know, and if so, what are times where that worked better or worse interacting with some other thing that happened? Right. So whether cases where we did this and something else happened such that people really got it and started thinking about other issues in a systems way. Right. Because it can be quite difficult to try and shift these. So you're measuring these shifts and mental models. I'm wondering if you saw some cases where it worked better than others. Yeah, this is, and that really is the goal here is to be able to evaluate, first off, to be able to have solid shifts and a more complete understanding of the local system, but also to be able to evaluate that understanding. And so, yeah, right now, we're still looking at, first off, I don't know if anyone noticed this, but I didn't say anything about was there an improvement in understanding. Just the research at this point was to just look and see if folks were conceptualizing the interaction of factors differently. And whether that's an improvement is still, that's also an area for future research is, how do you know if their understanding has improved? What does that mean? But definitely, we've had, we've had some workshops that didn't go too well. And that was largely because we had people in the room that were kind of antagonizing the workshop participants, not antagonizing them directly, but sort of indirectly folks in the room that the stakeholders are like, I don't really want to say, you know, that the government is negatively influencing me. Right. And so in many cases, it's really a major hindrance is not having the right people in the room. The robustness and benefit of the system mapping exercises lives and dies by the ability to have the right people in the room. And also just, you know, we're seeing it in the United States, right? It's hard to change people's paradigms. This is just a one factor mapping exercise, right?  So a lot of these are, a lot of these, a lot of the understanding that exists is entrenched and will always persist, no matter if you engage them in systems thinking. And so, yeah, I would say that generally folks that don't, I suppose, don't have an open mind or are very much stuck on their ideas are more likely to not have that sort of shift in their thinking, whereas folks that are, were facilitated well in a workshop were with the right people and who kept sort of an open mind to, wow, there actually are these interactions. And wow, my understanding of that interaction wasn't correct in my mind, I guess. And now I've been convinced that I've talked this over with my fellow, you know, workers or colleagues or what, what not. This has changed my understanding. It really comes down to a lot of different factors that go into that. But well, this stakeholder understanding evaluation research is still underway and that'll be a very interesting point out is what approach had the most shift, what approaches had the largest shift in understanding, what seemed to be hindrance to those shifts, if there are any. Good question. So, so what you're saying is you want other people to engage with you on this research, people that were asking these questions in order to to help you try and answer those as a research community. That's what I'm hearing. I mean, I think absolutely to be to do more of these, to do more of these. This sounds really great. So I had, there was a question here and I think this is an important one that you just sort of touched on, but I want you to perhaps elaborate a little bit if you could. So this is a representation of the stakeholders understanding their mental model of the system and how it operates. So I have two questions. Number one, and people are asked that, you know, I'm going to use this classic for people are asking, right?  So, you know, I think this is a key thing when we think about doing development work is, you know, we're telling people as they do design, go do observation, go do interviews, go do talk to people, we're talking to all these people, you're creating, synthesizing a mental model in your mind, how that system works, right? And now you're creating an explicit representation of how it is in the stakeholders mind. Was there times where you yourself were very surprised or people in your team were surprised, the facilitator was surprised, like, oh, I believe the system to work this way. But the stakeholders all identified this completely different thing that I hadn't hadn't even considered or they thought this was important and I didn't think it was important for example. So can you, was there a time where you were sort of a counterintuitive result was surfaced by doing this technique? Absolutely. Absolutely. So the answer is yes. For the most part in the system's literature that I've seen this a number of times, these sort of modeling efforts tend to produce 90% of kind of outcomes that would confirm a suspicion or by the team, it's like, oh, okay, I kind of saw that coming. That made sense. That really just reinforces my idea. And about 10% of the time it's like, oh, my gosh, I did not see that. And this is a difficult, this is a difficult notion or difficult concept because in many cases, if a counterintuitive result comes out of the system's diagram, which you would hope for, right, that's like, oh, my gosh, like, now that really shed some light on an area on a blind spot at a place where we should be intervening, but we're not. It's difficult at times because a lot of times people see that, they'll see. So for example, there's a large shift necessarily in the sector and the water and sanitation sector towards more sort of utility approaches, sort of preventative maintenance approaches, sort of, I wouldn't say top down, but more not just having the community maintain the system, but engaging the private sector, engaging local governments and maintaining the system, which makes a lot of sense. But in some cases, because we're engaging local community members who have been engaging community-based management, a lot of times we're actually finding that the community, the water committee, is actually really, really, really important in the system in maintaining positive interactions between various components of the system, various factors. And so in many cases, we see that the local water committee and the willingness to pay and things of that sort are really important, but they kind of go against the grain of where existing sort of the paradigm is going with the sector and saying, water committees, the community-based management strategy is no longer as effective as we thought it was in the past. We need to have more of a top-down approach, which, again, is good. In many cases, folks will look at that, that particular outcome, which I say the community is really important, and say that doesn't seem to jive well with what we're seeing in the sector or the direction of the sector. So that would almost point to an error in the system diagram. In other words, this is not represent the real system, that's just what the stakeholders think. And so it's really hard to, there's been a number of times where, yes, it's been counterintuitive, and they influence the politics or the importance of community social dynamics or social cohesion has come up, where the discussion has been along with the lines of, wow, that's a really interesting place to focus. But that doesn't seem to make sense to me. And so really, the difficulty is seeing those counterintuitive things emerge and to not say, oh, well, that invalidates the model itself. Obviously, something's wrong here. And in actually saying, okay, wow, again, this is the open-mind idea, looking at it and going, oh, okay, well, then what does it mean to potentially target that area of the system? Or maybe we focus another mapping exercise around that area of the system to see if we can throw it down and understand it better. But again, to summarize, yeah, in many cases, we counterintuitive things do come up. And the issue is not looking at those things and saying, well, that invalidates the model. Yeah, that's a great point. I think I'm going to ask two questions here in response to that. And I think that was a great answer to the question. So hopefully, people were able to take away and answer the questions, several of the questions that were in the Q&A. So I'm going to touch on two things. Number one, I think what we're really talking about is that this is a representation of the stakeholders' understanding. And then, but as engineers, when we do this complexity system stuff, often we're drawing these connections from like a first principles perspective. So we assume that that phenomena is true, right? Like that it's a ground truth. And so when you say like, this doesn't invalidate the model, it's like, that's assuming that you're equating the stakeholders' paradigm for how the world works with how the world works in some objective sense that the system exists in the real world, right? And so I think one of the things that's interesting about this, and I want to ask a question that was in the Q&A, when you're teaching students about this and you want to change the way the engineers are thinking, how do you do you get across? Like we want people to think in the system's way, but when we're doing that, we have systems that involve people, which is what these are. How do we understand this difference? Like I would imagine that my students, if they're trained in like an engineering mindset where everything is like a physics, there's a right answer, right? Here, this is a representation that we want to use to be useful to create interventions which we think will affect the system which includes people, right? So we need to understand how those people view the system, right? And so I'm wondering about that tension between, okay, this is how the stakeholder group views the way the world works, their mental model versus perhaps the designer's mental model or the team's mental model researchers or and perhaps how like the phenomena, the actual impact of these different things, because you could do like a causal experiment for some of these factors, but not for others, right? It depends on the definition of factor. So I wonder if you could talk a little bit about that tension, both about teaching and then the tension even in practice between objective whatever you want to call it and these sort of mental models, which are more representations of how people believe the system is working. Yeah, and that's a really good question, Jesse. Thanks for bringing that up. And this has actually been something that I've been up against for the last eight years is folks saying, well, you know, that's just you're just pulling out the stakeholder's understanding of the system. That doesn't necessarily mean that is the system. And that's very, very true. And I don't want to use this as a cop out. But there's the kind of age old system saying all models are wrong. Some models are useful. And that is to say, no matter what model we create, some models are more accurate than others, but most models still don't quite get that reality. So what if for a model to be useful, that's the most that's kind of almost more important than being sort of valid. And it's kind of as car before the horse or whatever it's, if it's useful, and it and it actually produces. And again, we haven't been able to show this. This is this is the blind spot. Just glad you identified this is just does the model that the stakeholders identify, allow them to make decisions and to, to effect change that ultimately leads to a better outcome of service delivery. And if the better outcome takes place, you would ideally or hypothetically say, okay, well, that's because their understanding shifted in XY ways, and that the modeling exercise was in effect useful. And ultimately, the model that resulted from their work, the workshop was relatively represented of the real local model. But so so really that the proof is in the pudding, so to speak, it's really that the goal here with this sort of research is to sort of loosen our grip on this idea that all these models have to be fully validated. I would love for these models to be able to be valid and a very scientific sort of engineering sense, right? That's the goal. I would love that. And to actually start to be able to put real quantitative insights into these systems diagrams that develop system dynamics models to be able to do virtual sort of simulations and sensitivity analyses. This is where I would love for it to go.  Did it tend to move towards, and this is a bit hard, but it tend to move towards the understanding move towards a more, if the sector is going towards preventative maintenance, and that's been shown to work better, is the stakeholder understanding going towards tenants and aspects of preventative maintenance, in which case we might say, Oh, okay, well, then that seems like their understanding has improved or at least been changed through engaging in these systems activities. But really to be able to have a validated, a validated model will require more and more research. And that's, and I think that's going to be important now bill is sort of the legitimacy of these methods. Sure. So I think we have about five minutes left. So I'm going to be selfish and ask my question and then great questions. Thanks. Well, these are questions from the people and everybody else. Thank you. So this is my question. So it'll be worse for sure. So my question, I have so many questions, we're gonna have to have a phone call later about it. But my question really is about, can we connect this perhaps or your thoughts about the future direction of this type of work? Could we connect some of these things to sort of an implementation science perspective, right? Yes, you're just talking about some of this tension between like the real model, like, Oh, well, we understand that this service thing actually, you know, does this, you know, affects the sustainability of it. But everybody in the community believes that the government's input is actually the most important thing, right? And so the intervention, if you're saying, okay, well, we're going to try and implement this preventative maintenance structure, because we know through past experimentation, that this works better. You may have trouble getting adoption of that intervention or uptake of that intervention, if their mental model thinks that it's unimportant, right? And so I think that connecting this type of understanding and stakeholder analysis, stakeholder understanding analysis to this larger question of, well, we believe this other thing for some other reason, whether it's physics or prior research, actually has a better impact. Well, that discrepancy may prevent uptake or adoption. And perhaps we need to change the way we're facilitating the implementation or the way we're marketing the implementation to those stakeholders to that particular stakeholder group, or provide evidence in a different way such to think about their shift. And now you can measure their shift in their understanding also, right? So that seems really important. I would think that to get to my question, have you looked at creating stakeholder, these maps, these understanding mental model maps for different groups of stakeholders, and then measuring the differences between those maps? So that would be my questions. Maybe you could say, as a team, hey, stakeholder group one, stakeholder group two have very different mental models, and that means we need to change what we're doing. Absolutely. So this area of alignment. So yeah, this is something that we've looked at quite a bit. Very, very good question. And I think you touched on a lot of different things. So really quick, the implementation science aspect of this, I think is critical. And I think that's why I'm seeking to develop sort of a consultancy to kind of promote sort of thinking and doing more mapping, having more case studies, doing more evaluations, looking at different stakeholder groups, how they conceptualize the system differently. Because you're right, that's a really important outcome is seeing, well, these different stakeholders see the system very differently. And these stakeholders have their finger on this button. And these stakeholders still speak on this button. Maybe which system needs to be not shifted or what paradigm needs to be shifted? What sort of understanding is more relevant to the sort of the box or the boundary of the system? And if it's at the local community level, and they're seeing the system completely different from what the, let's just say the regional government is seeing, and they're kind of the ones that have their hands on the first strings, then there's obviously a problem, there's a disconnect there. And that's a place that there would show a place to intervene in the system, saying, okay, we need to engage these two stakeholder groups in a robust discussion about why these models are different. But yeah, ultimately, this is going to just require more and more case studies. And this idea of implementation science implementing, it's both an implementation of a factor mapping exercise as well as evaluating the implementation decisions those stakeholders took as a virtue of or because of that factor mapping exercise. So yeah, this implementation science is really the center of where this is going. And iterating, again, this is all an iterative process of continually going back and meeting with the same stakeholders over and over again, seeing how their understanding has changed, how the system has changed. So, thank you, Jeff, for all of those answers. We're now out of time. So I'm going to turn it over to you. I want to just recognize that there were several more than several questions in the Q&A, which I think were critical about representation, you were talking about who is in the room, while you're doing these things, who is the facilitator, who's on the design team. All of these things are incredibly important to building an understanding of the stakeholder group system as a whole. We didn't even touch on how do you identify who the stakeholders are, which I think is another one, you could create a mapping of the stakeholders in a very similar way, and what are their connections and that might be interesting also. So I just want to recognize for the people that I did not get to your questions, I apologize. We've run out of time was trying to synthesize as they came in. But Jeff's contact information is there. This whole forum is intended to build connections between everybody that's in the audience right now and the people that are presenting. So in discussions with Jeff before this talk, I know that he's happy to try and connect with people and discuss some of these questions in a more in-depth basis. And with that note, I want to thank everyone for joining personally, because this has exceeded beyond my wildest dreams the attendance things that I thought would happen for this type of seminar. So I'm really happy that you guys are all here and asking such great questions. I want to thank Jeff again for his great talk, and I'm going to turn it over to Yana to wrap it up. So thanks again. Thank you so much. Thank you, Jesse. And I whole heartedly echo the gratitude to Jeff. This has been so enlightening and we've already identified some parallels even in our own programs and how we actually train our fellows to conduct interviews. So a micro model of your system's approach. So with that, I do want to thank everybody for participating. We encourage you to join us for our next seminar, which is going to be next month with Dr. Erica Graloff, who's an assistant professor in the Department of Engineering Management and Systems Engineering at George Washington University. So the system's conversations continue, our system's perspectives, which we're so thrilled for. I do acknowledge also, as Jesse said, we have a lot of questions that went unanswered just because of time constraints. So if your question didn't get answered and you're very eager and you didn't catch Jeff's contact details, feel free to email us and we'll do our very best to convey them. We might also consider a follow-up article with Jeffrey compiling some of these questions and providing a synthesis to our attendees. With that, I wish you all a good afternoon, a good evening, or even a good morning, depending where you are right now or when you're listening to this recording. And we hope you'll join us for our next seminar. And thank you all. Goodbye, everyone.


Everybody for us to your rip in to one eight twenty two tops growing baseball hobby 12 bucks full case number 13 random teams She pick at the results right here. You can always find them late sports cards calm Let's get started everybody Halloween costumes big decision Big decision 12 silver packs. I'll do a silver pack after every box.    All the inserts in black are really cool Know the short prints to Fractor rookie Phillies Luke Williams orange refractor Shamanaya A's You know Oakland 8 of 25 combo spot orange Good start to this break now Ray Lewis Sam cry Prism Dodgers Cody Bellinger. Oh, he found out there was Ray Lewis Yeah, I'm really this isn't a lot of products. I'll back refractor Gil Cabrera Yeah, I did one case of black yesterday got refractor trade turner. So we pulled a spotlight signatures at 10 of DJ watt That's pretty sick looking Future stars Garcia. I was I was I was thinking about that Shane. Oh one thing I discovered you got prism Cleveland Jose Ramirez. I don't know how long they've been been in existence, but Cheetos have like a simply like white cheddar And they're really freaking refractor Jose. Yeah They're like simply supposed to like healthier Right, I don't know if that's really that true, but they're like white cheddar and they're great really Yeah, I love them. All right. We've got an orange again two oranges rookie Auto Nationals Mason Thompson Number two out of 25 Two oranges got a whole Grove going right now You go Nats Patrick Wisdom The white cheddar, but yeah anything white cheddar for like snacks I'm pretty much all in on but the best and the most addictive are the pop corners white cheddar Those are outrageous. I Don't buy them anymore because I'll eat all of them in one sitting refractor Jesse winker every time Yeah, healthy with quotes with quotation marks Cal Raleigh Paul Goldschmidt Been around a while. Yeah, I didn't I didn't know I don't see him all over the place But I mean like I said anything white cheddar is probably the best snack food Prism Adam Fraser Mariners got refractor Boba Shet Blue Jays J-Rod's a tough pole. What do we got here? We got a Rhymel top. Yeah, I'm a gent at a 399 Rockies White cheddar cheeto It's Matt Olson Yeah, J-Rod's tough hit new classic short on Alvarez refractor Dylan Carlson If you see a J-Rod you probably only see like one in the case. I would think Does not have base cards There's a wander base Or also only in the first box of 12 so Prism Carson Kelly refractor rookie Max Crannick Pirates Yeah, guys, you haven't if you've never tried in the pop corners white cheddar get those Josiah Gray Nationals rookie base auto gnats with both hits so far You go Denny J-Rod Superfactor all I get is a free trip to Disney Actually J-Rod Superfactor probably gets you three days at Disney J-Rod Superfactor you could probably afford a vacation Refractor Brian Reynolds got to get flights on your own though purple refractor J-Tero Muto Phillies 211 out of 250 Prism Adolis Garcia Rangers. Oh nice.  I just I don't know man refractor Adam Wayne, right? Prism Catel Marte Diamondbacks We'll see if the Phillies can win get refractor Charlie Blackman. We'll see Purple speckle rookie auto reds Alejo Lopez Numbered 162 of 399. I'm just gonna count the Astros out refractor rookie Sammy Long Giants Yankees gonna win the World Series eventually that's a that's a good prediction eventually is good Their fans probably still will boo him prism straight turner Dodgers. They'd be booing at the parade Champions Fractor rookie Royals Emmanuel Rivera. They're like the new Philly fans, but in like a bad way Like it's like flipped around like Philly sports are good New York sports Gavin sheets new classics Tatis rookie refractor Jaren Duran Red Sox Let's go Yankees no one said that in like two weeks y'all been booing Giants world giants are super bowlbound for sure prism Teosca Hernandez blue jays refractor Alex Bregman Five-card back with a Ryan Valle base rookie Otto Rockies Belied I was supposed to do one silver pack every I was doing at the end now Rolled through the boxes. So yeah, I sure as pitching is nasty Phillies little shakier I know Nola and Wheeler have been great. But after that, it's a little Less confident refractor Dylan Carlson wonder Franca rookie base card Yeah, 62 home runs not nearly enough red Speckle this is gonna be a short print.   She's a view card. Yeah, big card.   I don't know.  You've got to choose who's going on mission of space. You got to choose Louise Robert Sorry, only only some of us can ride test track guys. I'm sorry Prism Corey seager Rangers So I get a crazy Refractor Julio Urias Dodgers, I'm assuming you have some some savings account you can empty that you can you could probably take some more family members Blue refractor a rookie auto Joe Barlow Rangers Numbered 61 out of 150 Jaren Duran Refractor Bryce Harper not much up in the case. What are you talking about? There's a bunch of boxes left There's no like collation and there's no like guarantees and like Todd's chrome and we pulled two we pulled two oranges in the first box So think that there's nothing left this case is wild the way we've been pulling tonight, too Refractor Schwabber We're gonna we're gonna prove that statement wrong. I guarantee there's something else huge in this case now It just feels like things run together sometimes We go through so many cases.   Yeah, it's like one in 40 or 50 or something like that Grand Crawford Refractor Green wave rookie Diamondbacks drew Ellis nice one D-backs 12 out of 99 We'll do all those at the end. I was gonna do more after every box, but once I get just rolling I just keep going, you know Prism Alec Balm Phillies refractor Trevor Rogers Yeah, I don't know what it is. I'm saying maybe it's like whenever 40 or 50. I have I truly do not know I'm just bringing that number out of my out of my head Could be better could be worse refractor awesome meadows if it is better It's probably not much because we've opened quite a bit of them now future stars your dawn Prism rookie Josiah Gray Nationals refractor Robbie Grossman base rookie Otto Ivan Castillo Padres refractor Mitch Hannigar pinch striped Otani Prism Otani Guys Bowman Sterling is coming up next four box random team style break. Let's get it filled up There's part of the city Devers refractor Eloy 16 to go Breckos next what's Sean type? Breckos next Yeah, guys that breaks next it's got 16 to go almost halfway full Mullins refractor on these short print. That's a pretty cool one Hey, yo Cardinals rookie prison. I'm Andre Jackson Dodgers a lot of short prints so far We're four boxes and we pulled through short prints. It's this feel this feels like a good case you guys Rookie refractor Matt Manning Tigers not just because there was one monster card. We might get some more stuff, you know Point Park rookie base Otto Pirates I don't know why because I reached my left I'm not sure when statement club comes out anymore. Let's move so much There's a kershaw photo negative Dodgers refractor Evan Longoria. Yeah, that one's that one's been pushed back a lot But oh my god, dude shop if I just close me out of everything. I Might actually cry because it happens. It does happen every day like 458. It happens before five before 5 p.m. Every day purple speckle rookie Connor Wong Red Sox Number 36 out of 299 rookie prison Brandon Marsh angels refractor Jared Walsh You know him and as rookie refractor Philly's Luke Williams prison rookie Andre Jackson Dodgers refractor Salvador Perez Face rookie auto Mets Tyler McGill Let's go guys. Let's get Bowman Sterling rolling here. Let's get it filled up Breaking after this case pulling heat tonight, too Yes, we are hiring refractor Kevin Galsman blue jays always hiring the right people pinstriped one Soto prison with Dulles Garcia Rangers refractor Jose Brailleu 87 vado refractor Pavin Smith Not the wrong guy.  I think he was on he was not truthful that part. I Don't know how that slipped through But we'll have to do the lie detector test next time Eli Morgan rookie magenta speckle to 350 guardians refractor rookie Delacruz Marlins Half to 70% of your paychecks go back to the company blue wave rookie pirate Zach Thompson Most of us already there man, but he 375 Prism Hazel's Lazardo Marlins Sleep with the nearest bus stop. It's honestly not as close as you would you would hope Refractor Jack Flaherty public transportation in a pop. I don't think it exists actually Where is the closest bus I probably have hopped like downtown pop Think so yeah, everybody's applying now Randy was a random refractor rookie O'Neill cruise pirates Move in with Mickey. Oh, you do not want to know the rent for that Central Florida rent is expensive as it is Prism Dodgers Max Muncie, but I can guarantee you you'll be treated better here as an employee than at Disney 1000% refractor Alec Minoa. Yeah Well downtown area it when I say downtown, I mean that because that's what it's called Yeah, I would not personally call that a downtown rookie base auto Jake Berger White Sox Yes, there is a downtown Orlando. Oh, yeah refractor Max Fried. Sure. There's even like a downtown winter park Honestly, that's that's kind of you know poppin at least There's a lot more going yeah, it feels more like a down like a small downtown prism hungen Ryu blue jays Yeah, man Refractor John Gray if there wasn't soccer on every every Saturday morning. I probably go to the farmers market more Jovado, come on new classic Soto refractor Kershaw You'll commute from Port St. Lucie That's funny. Yeah, what a park is lit. What's Port St. Lucie two hours? More three prism rookie drew LSD backs less. What's up, Chris? To a pop. I don't know rookie refractor ever Cabrera Marlins What a park is fancy Yeah, what a park is price Rollins college old money Stewart Fairchild rookie Otto Diamondbacks Yeah, when you see places that like the Rockefellers used to own you're like, oh, yeah, this is old money area This is fancy under Franco 87 Florida's got pros and cons I keep telling rich so to move us out west More rich take us west Kyle Seeger magenta speckle here to 350 mariners refractor Brandon Woodruff Or does certain certainly not the best date? Certainly not got purple refractor Steven Strasburg Nationals is not the worst, but it is not the best 55 out of 250 Prism Tyler Stephenson Reds. Yeah west is where we need to go. That's where we need to go the best coast refractor Manny Machado Vegas, that's where we should go. That's exactly where we should go wonder Franco I'm in refractor Brandon Lau. Look at this guy. Let's do this to Jenner. You guys so excited Sign me up for that prison Willie Castro Tigers What kind of one be too bad, I guess refractor Ronald Acuna, Jr. Go to Vegas redemption Tyler Stevenson veteran auto blue parallel Reds You go Reds to 150 I for sure I'd for sure go to Vegas refractor Hunchen Ryu I would just like to be close to everything out west like driving distance to Southern California and You know quick flights to Denver San Fran Seattle Phoenix is it is a quick drive not quick, but you know, it's within a day prism branded belts Giants refractor Jorge Salera Braves. We should have a bidding war to where the next LSC is gonna expand What's up LSC West startup GM force I Don't want to do all that man. That's a lot of work refractor. Oh, yeah, I just want to break there. I Okay, if I'm standing for I'm not going to Tampa or South. I'm not going south. No way, dude Prism Anthony Rizzo. I've been to Tampa and I've been to Miami plenty of times to know that I don't want to go there refractor Patrick wisdom Also, not trying to be like rude, but South West Florida is in pretty rough shape right now South Southwest Florida, you know rookie auto Zach Pop Marlins plus they get hit by a big hurricane every like four or five years like a major hurricane Wander Franco 87 rookie LSC NC. I probably go to North Carolina I'd be about it At least you get seasons North Carolina refractor rookie Josh Lowe for the rays You know, I'd also be down to go to like The Philly area got awkward Josh Donaldson twins 191 of 199, although I don't think anyone else would want to go because like the winter Got Prism rookie Reed Detmer's angels I think I'd only be down because my brother's moving there both my brother-in-law's live there Fracture Lindor and it's it's just a nice area suburban Eastern Pennsylvania I'd be down probably wouldn't happen though. LA just risk of falling into the Pacific. Yeah, I wouldn't want to go to LA Um There's a refractor Logan Gilbert I'm not I'm not a big fan of like that thread of this a massive earthquake at some point. I'm not I'm not about that But being within like a day's drive would be nice. So like Vegas like Vegas a Prism Patrick Wisdom Cubs refractor Jonathan India Reds rookie auto helly at Ramos so you go Giants base auto there In the Washington Washington state Or the city of Washington. Oh, Washington state. Yep. I'm down. I'm in I'm in Refractor rookie Rwansi contraros pirates. I was gonna say DC is gonna take some convincing from me About a city with horrible traffic, that's the place right there Prism Yankees DJ LaMahoe It's like half touristy half like Also, I don't think it'd be the best place for a car shop to be fair refractor Verlander. Yeah Washington state. I'm down I'm in sign me up Phillies McCutcheon Magenta at a 399 police Robert refractor Wilson contraros Cal Raleigh Prism Tiasca Hernandez blue jays refractor Miguel Cabrera purple auto refractor Said that weirdly Joey Wendell Marlins 91 out of 250 Yeah, I know what you mean. There's a wander that I didn't miss guys got it right there though Sorry, I'm trying to read chat a lot going on in chat right now 13 to go on bowman sterling next everybody. Let's get that thing filled up Let's get it filled up refractor Anthony Rendon blue wave Tristan McKenzie guardians 49 of 75 Prism rhyme out top of your Rockies. Yeah, I mean with Julio. You're looking for a bit of a short print or an auto refractor Angel McCutcheon Petal Anzo New classics Turner refractors and her bow guards. I want to see a super factor too Luis Garcia future stars rookie prism Jake Myers Astros. I might have to go play point guard for the Magic Illegal Refractor Walker Bueller. I might have to our guy Cole is out. Sugs is hurt for a few more for a little bit more orange wave Rookie auto rangers, Yanni Hernandez One out of 25 nice hit Rangers. Does a pop you have an airport? Like a little Cessna one Like a like a dirt airport pretty much refractor Alec bone But Sanford has I've flown out of Sanford a couple times to have a legion there which Flies into like Allentown. We would do that. I've done that a couple times my wife prism Harrison Bader Cardinals refractor Rookie Tyler McGill Mets Luis Garcia Magenta the 399 Astros a pop gets a feel where planes are possible to land That's a really good way to put it actually you could probably not land a commercial liner there A little Cessna refractor Shane beaver Yeah, and fault is out too. Yeah, that's a point guard. Maybe it maybe it could be me I'm gonna go start doing some drills when I get home Prism Shaman, Ia is gotta get in shape got to work on the ball handling skills Change of direction drills all that good stuff, and I got to get warmed up Rookie I'm not rookie but base auto brand of woodruff Brewers Come on guys. Let's get the Bowman sterling four box are filled up three and two is after that as well It's got a lot of spots. I Give you guys like but everything that I have is closed down. Thanks Shopify Actually looks like it's all refreshing. Oh, no, not all of it Brutal, how's my shooting? It's all right Rookie refractor Joe Ryan twins when I played a you that's I was a shooter Hey, I gotta get a sleeve though Aqua Xander Bogart's Red Sox 126 of 199 I just need to practice for like a week or so, and I think I think I'd have it back Got prison Luis Castillo white socks refractor Kyle Lewis Mariners.  It comes back.   I used to be like all right at darts. I'm terrible now Mitch Annager blue ray wave Mariners auto there 104 of 150 I Feel okay, cuz I would play like I live downtown like the neighborhood bar. I would go to I would play all the time I was all right. I Would never say it was good, but I would I wouldn't some games a cricket with my friends refractor Joe Adele Arch is the best Prism John Lester Cardinals can't even see force Refractor Corey seeker, I'm not very good at pool Billiards as they say refractor Luis Garcia Like I'll make one shot every game that'll look really cool and everything. I just miss all the easy stuff Hard of the city Marifield. I Know I did not six Max Scherzer photo negative Mets Prism rookie Mets Tyler McGill Come on big hit actor Max Muncie We got a bunch of golfers LSE though. They think they're good Mets rookie auto Patrick Mazzica The same golf yeah, yeah, so I worried about me No, no way to compete with Donna no Donnie's like borderline pro God Gold wave rookie Jose Siri Astros Awesome gold wave number seven out of 50 Yo, Stroze for that Nice card there. We got prism Nico Horner Cubby's actor Goldschmidt Vidal Bruhan New classics a cunea refractor Burns Future stars Crona worth Prism Trevor story Rockies refractor rookie Orioles Zach Louther Blue Jays pin striped auto George Springer Number 12 out of 25 so pin striped orange nice hit blue Jays O'Neill Cruz. I was convicted. I saw your message like super late last night. I forgot to respond I was like passing out refractor rookie Zach pot Marlins Prism Blake Snell Padres To the Josiah gray rookie refractor Nationals Josh, what's up, man? New classics so Tony Ricky refractor White Sox Jake Burger. She pulling more super fractors last night, Sun Victor guys pull all super fractors And a wall of loans Prism Josh Hader Brewers 17 Fractor Soto What three and two? Three and twos at 17 guys Colton welcome rookie base out of Rockies. All right last one that we got some silver packs So we have 12 silver packs nine to go in the sterling which breaks next and And 17 to go on three and two which is after sterling. We also got luminaries breaks to fill up everybody. So Let's get some spots and refractor jazz Chisholm true gold Buster Posey is a sweet one for the Giants now 33 out of 50. We've had great color this case Nice Posey. Oh, that's pretty good though Nice sin got prison Mike Moustakas Reds K2 auto is cool.  Rookie refractor Nats Riley Adams.  Lane says ski balls fire Right in March Dude, we're gonna get we're gonna get thrown out of D&B when we're there refractor Tristan Mackenzie We're gonna get way too competitive Like betting money Which a D gens trust me on your cruise base filling Sam's ear up prison Marta mets Playing drinking games you refractor Joey Votto air hockey's money to yes, sir air hockey's a good one I'm probably not quite as good air hockey anymore Cutter Crawford base rookie auto Red Sox probably the last auto with a break unless there's one hide in those silverbacks Yeah, there's a few places like that in Florida that you can play for free Alrighty 12 silverbacks now Ali refractor John means refractor Pete Alonzo The light or the game right this up line in the jackpot spots go into Rookie extended base short print Padres CJ Abrams Grats Padres There's one of them right there It's been a great break great to break so far today actually Get that mag for you Padres Buxton refractor Gavin sheets Tyler Gilbert rookie refractor for the Diamondbacks. I thought you're some games were missing Trying what else is fun? I know it's older by NBA James. Obviously, you know rookie refractor Brian Dela Cruz Marlins I don't think I've ever been to a bus was honestly We cashed a refractor Trout refractor That's the most rigged game ever though. Yeah refractor rookie Zach Louther Orioles. That's the most rigged game of all time Rookie refractor Jake Myers Astros three more refractor Nelson Cruz smooth fish What's this? What is what do we have here? Oh? What is this I'm gonna do a quick one here Wow Purple first bowmen auto Louise Roberts 170 of 250 What a monster? out of nowhere for the White Sox Brian F What a break. Oh my goodness What a hit cool myself off with the box My goodness refractor Harrison Bader Yeah, what I was saying that that was the last auto we pulled the Crawford That's it. Oh, wait, there might be something in those silver bags. Yep. Yep Show no You know we pulled some absolute gas today out of tops chrome As you can see by that one of us has been chilling there full recap coming right now the force alarm went off Whoo, yeah the force alarm. I need like a siren button Best I got Okay, Otto's not Red Sox Cutter Crawford Rockies Colton Welker Mets Patrick Mazzica Brewers Brandon Woodruff Giants Elliott Ramos Marlon Zach Pott deep back steward Fairchild White Sox Jake Berger refractor Otto Mountcastle Orioles before 99 Mets Tyler McGill Pirates Roy Park Padres Ivanka Steele Nats Riley Thompson Rockies Ryan Vallee Jersey Otto Astros you're done Alvarez to 99 Josiah Gray Nats purple speckle Alejo Lopez Reds to 299 Rangers blue Joe Barlow to 75 or no, sorry 150. I'm sorry orange insert blue J's Otto the pinstripe there to 25 George springer Blue Ray wave Hannigar to 150 the Mariners Royals on hell's air put a 250 Joey Wendell Marlins the 250 orange wave Yanni Hernandez the 25 Rangers orange Mason Thompson the 25 Nationals Orange refractor a shaman I had a 25 Short print Nolan Aronato Cardinals Blue Redemption Tyler Stevenson Reds Gold wave rookie Jose Ciri Astros Really good case we had the extended base short print CJ Abrams Padres a first bowman purple Otto Louise Robert a 250 white Sox that one is out of 2018 was a chrome is he in 2018 bowman chrome? I believe Insane and red speckle Short print rookie wonder Franco that catching ball short prints numbered two out of five. Goodness gracious Monsters right there. Let's go. We're on the Sterling. Thanks everybody


Hey y'all what's up my name is Sonali welcome to my channel if you guys are new and if you are definitely hit that subscribe button and if you guys are already subscribed make sure your post notifications bell is turned on so you never miss a video of mine. Today we are doing a caraway pants review after using them for about 8 months or maybe even more than 8 months. I use these almost every single day so I want to show you guys the wear and tear of them and get into the specifics of why I even chose caraway in the first place. I'm going to give a little disclaimer though because I did a review after having my pants for about two months I think and they were a different color because I originally got the gray color and then the iconics collection came out which are the pants with the gold handles and caraway was so kind enough to get me this set so I was like oh I can't really say no to that they are just so stunning and so I got it in the like white color and then since I did that video they came in with so many new stuff I'm going to go through that later on in the video and show you guys the other products that they had. Even though caraway gifted me this pants set a while back these opinions are all my own so I'm definitely going to give you guys an honest review and let you guys know what I think about them after 8 months of use. I'm going to place chapters throughout the video so in case you want to skip to a specific part you can go right ahead but before you do that I do want to tell you guys that I do have a discount code it's 10% off I'll put my code on the screen and the link in the description box but 10% off sometimes it doesn't really sound like that much but it does shape off a good amount of money so definitely use it and save yourself some money the first time I ever saw anything about caraway was actually through an Instagram ad and at the time I was actually cooking with the always pan from our place and I really liked it when I first got it but then a year went by and the pan was just not how it was when I first got it the nonstick was barely there so I was like dang I wish I didn't have to buy new pans but I saw these caraway ones and I really want to check them out but I did get the bundle because another thing that really sold me on these pans is that they come with organizers if you get the bundle so these are actually magnetic organizers you can kind of customize them to how you want put them all together like that and they just stay really nicely organized in your cabinet I would drag going into my closet pants cabinet because it was just so chaotic in there and now it's very organized with this they also give you a little organizer which on their website I saw that you can put it on like your cabinet door unfortunately my cabinets are a little too small for it so I just put it like in my cabinet but it works just fine. Caraway is 100% non-toxic ceramic cookware creative teflon and other heavy metals it's nonstick ceramic aluminum core and stainless steel handles these pans are also oven safe up to 550 degrees so it's great if you have a recipe that requires you to go from the stove top into the oven you don't have to dry up any more pots or pans or anything like that out of all these pans I definitely use this one the most and they are naturally nonstick so what that means is that you can actually use less butter or less oil and your food is still gonna slide around my boyfriend's mom was actually over here the other day and I think I was just making some grilled onions and she was shocked at how easily they were just sliding around in the pan the best thing about these pans is that they are extremely easy to clean I will make like a curry or like I don't know anything sometimes it just needs a little rinse and it'll be spotless other times I might need a little bit more scrubbing but I'll just use the scrub daddy just to make sure it's not like scratchy with service. Caraway cookware is an environmentally friendly option because they release less CO2 emissions when produced compared to traditional non stick cookware I know you guys have been waiting for this and it is finally time to show you guys the wear and tear on my pans after eight months of using these as you can tell the inside of the pan looks brand spanking new if we turn it to side it also looks pretty new then when we turn it to the bottom that's where you're gonna see a little bit more wear and tear I didn't try to scrub it down just with this video so this is the actual wear and tear after eight months of use and honestly it's not that bad I feel like I've definitely seen worse bottoms of pans in my lifetime I will say that I do follow the instructions that it comes with really keep it in great condition I think that a lot of people's hesitations for even purchasing this cookware is their instructions on how to use them it says to use them on low and medium heat but nothing about that so I feel like I get the question a lot of how do I get my water to be boiling for pasta and I feel like it just takes a little bit longer but it definitely does get to that point because the ceramic can actually hold the heat more efficiently and if you do use it on higher heat you can wear away the non-stick coating so you definitely want to stay away from hiking so you don't ruin your pants another thing I do to keep them in great condition is I'll never put them in the dishwasher I'll always wash them by hand well like I said it is so easy to clean them by hand so it's not an issue at all I do want to show you guys the wear and tear of all the pans in the set so the first one I showed you guys was the fry pan this is the saute pan so the inside is again perfect condition side is beautiful and then this is the bottom here's the inside of the sauce pan the sides and the bottom last but not least we have the Dutch oven here's the inside of this one the sides and the bottom now it's time to show you guys all the new stuff that they came out with ever since I made my first care away video so as you can see right in front of me I have a lot of bakeware because I got the day where set bakeware set I believe comes with 11 pieces which is a lot but it also comes with its own organizer just like pretty much all the things that they have so we have a little bread loaf pan square pans circle pans got a muffin line a little casserole dish I don't think this is all of it I'm not really sure where the other pieces went but we have two cooking sheets and then oven is this a cookie tray I don't even really know what that is I would say I definitely use the sheet pans the most just because I'm not really on my baking grind right now even though I really want to be and I'm always telling myself I really want to like bake a cake because last summer I baked my friend a cake with like this disco ball theme I shocked myself because I didn't even know I can one really bake a cake and to decorate it that nice I'll put a picture right here I'm just like so proud of it but I haven't really been using all these so I definitely want to start doing that and even like make some muffins I feel like that would be fun just like the pans they come in so many different colors and I got the one in the cream color I believe and I think it's just so beautiful I wanted to save my favorites for last and that is the minis so yes these are the mini versions of the pans that I showed you originally although this one is again in that cream color instead of the iconic collection in the white color we have the sauce pan with a lid and then we have my little fry pan which I love to use for breakfast because I feel like I don't need to dirty up like a whole big pan so this is perfect for like cooking eggs or just something like a little bit smaller I wanted to show you guys a size reference especially with the fry pan and the actual big sauce pan here's the fry pan and then here is the mini so it is considerably smaller than the original fry pan here's the original sauce pan and then it's mini so after eight or so months of use I do not regret getting these because number one it just makes it so much easier to cook when you don't have to use a low grease to scrub them down after you all know that cleaning is the worst part about cooking so the fact that it makes it that much easier is amazing I also love the way that they look and they just pop in my kitchen like I don't mind just putting it out on the stove almost as decoration because they're just that stunning and I don't know if you guys have noticed but since the last caraway video I actually did a little apartment friendly DIY renovation in October I put up the stick-on backsplash the kitchen was originally like this dark gray but I really just have always wanted a white kitchen I had it in my last apartment but they didn't really have an all-white kitchen in this apartment complex so I decided to take matters into my own hands and make myself a white kitchen it was my first time with stick-on backsplash it's not perfect but the fact that it just like makes the pans pop even more you would think it would like blend in but all of my white appliances just pop even more and I just love how they look in my kitchen you're able to purchase the pans individually if you think you're going to use one more than the others or you can purchase the bundle which includes these four pans the pan organizer and the lid organizer for 395 and then if you guys use my codes and only 10 or go through my link that's in the description box you can get the whole pan set with the organizers for 355 so like I said 10% does not sound like that much but when it comes to check out it definitely saves you some money I love my pants so much but I actually got my mom a set for Christmas this past year and she was saying that she was looking into getting new pants I was like wow this is actually perfect timing so these could also be a great gift for any birthdays or holidays coming up or if your family member just moved into their first home I feel like this would be a really cute house for me get that right side of this caraway pan review if you guys have any other comments please leave them down below in the description box I want to know what color you guys are I know the site don't forget to use my code when you do check out and I will see you guys in my next video bye


When it comes to otologous stem cell transplantation during the COVID-19 pandemic the general recommendation was to postpone this and to add some more induction regimens. So after easing the lockdown I think the auto transplant procedures in France and many other countries started again and patients were able to receive the auto transplant procedure of course we cannot exclude that some patient are asking well why shouldn't we skip this auto transplant procedure because of it is known impact in terms of mucositis, pain, diarrhea and need for hospitalization and I think this is a fair question and I believe the answer is the sort of direct communication, direct dialogue between the physician and an individual patient. The general rule is about you know resuming and delivering these auto transplant that were delayed but on a case per case basis things can be discussed depending on the patient features and characteristics but also depending on the disease features and the response to the therapy especially the induction regimens.


Members of the Craft and Dry Goods Vendors Association are participating in a training seminar on COVID-19 protocols. We get the details in this report by Rajvaro Lawrence. The Ministry of Tourism, Investment, Creative Industries, Culture and Information in collaboration with the Department of Environmental Health is facilitating a three-day COVID-19 protocol training for vendors of the arcade and craft market. Tourism Officer Samantha Charles explained that upon completion, vendors will receive a certificate of COVID compliance from the Ministry of Tourism. This enables vendors the ability to leverage their status as safe operators, as well as increased marketability as they will be listed on the Ministry's website where their status will be featured as well. The intention of the training is to really bring to the attention of the vendors the health and safety guidelines that has been set aside for the operations. And those guidelines came about by a number of site visits to both establishments in the Ministry and the Ministry of Health to determine the example where proper exit is required, social distancing, etc. So following those site visits, a COVID action plan was developed for the vendors arcade and that will be replicated for the craft market. And the protocols were developed out of that COVID action plan. So we're moving that at the end of the week. We have as many vendors as possible trained to meet our subsequent sessions. We expect them to cover their respective vendors and that they understand what is the operations and what they should do and should want to do and how they engage the clients or visitors and workers when they come to the craft market and the vendors coming in. Approximately 90 vendors will be trained at the 3-day workshop. President of the St. Lucia Craft and Dry Goods Vendors Association, Peter Isaac, highlighted that the training will allow vendors to be able to once again benefit from the tourism sector, especially with the cruise season having recently commenced. This exercise here now is to make sure that the vendors understand what they're called, understand how to behave and their approach and their regulations and their rules, so that they could behave as accordingly to our discipline. So we are pleased that this is our meaningful opportunity because we're sponsoring the Ministry of Health and the Ministry of Tourism for the Vendors Association. And we are happy because we want the facilities that we have vendors there who are making out and living from the tourism industry to be a part of it, so to speak. If they come to the workshop and they are considered to be people who are trained and they are COVID-certified, then the facility with a few of them can be COVID-certified. That's the important thing for us because we think that we have been making a significant contribution to the development of tourism basically and that we would like to participate, we would like to be involved, especially with the cruise sector. The training is scheduled for the 2nd to the 4th of November 2021. From the Government Information Service, Rajvaro Lawrence reporting.