In this episode, you're going to learn what to do when you run up against the person holding the Excel sheet in your organization. You're going to learn how to show the impact of your work as a service designer. Here's the guest for this episode. Let the show begin. Hello, my name is Joyce. This is the service design show episode 125. Hi, I'm Mark and welcome to the service design show. On this show, we try to look what's beneath the surface of service design, what are the things that make a difference between success and failure, all to help you design services that make a positive impact on people and business. The guest in this episode is Joyce Yee. She's a professor of design and social innovation at the School of Design at the North Umbria University. If you've ever struggled to explain the value of your work, you're not alone. This is one of the biggest challenges in the service design community at this moment. Joyce is one of the people who has developed a framework that helps designers to show the impact of their work, and all while staying true to the emergent nature of design, where the fact is that value is created at times and in places and in forms which you didn't expect upfront. If you stick around till the end of the episode, you'll learn how this framework works and how you can use it in your work to show the value of design, even if you're currently in a context driven by short-term and quantifiable results. Now, while you're here, make sure to click that subscribe button if you haven't done so already and click the bell icon to be notified when new conversations like this come out. And that happens every two weeks.  Welcome to the show, Joyce. Hi Mark, how are you? I'm doing very well. It's a lovely day in the Netherlands, so I'm really looking forward to having this chat with you. For the people who don't know who you are, could you give a 30 second introduction? So my name is Joyce, I'm a professor of design and social innovation in the school design at Northumbria University. We're based up in the North East of England, in the UK, and I've been working as a design educator and researcher for the past 15 years, mainly looking into teaching, interaction design, service design and social innovation. And I think you have a special relationship with a guest that has been on the show not so long ago, Emma Jeffries. Can you share a little bit about that? Yeah, Emma and I go back a long way. We've been friends since she started doing her PhD at the school design. We got to know each other and basically ended up writing two books together. So yeah, she's been my kind of partner in crime in a lot of the work that we've done so far. The episode with Emma, we talked about empathy at work. We're going to discuss a different topic today, I think, just as interesting. Before we dive into the conversation, I want to do a 60 second rapid fire question round with you. The idea is that you answer these five questions as quickly as possible. Don't overthink them. Ready? Yep, ready. Question number one, what's always in your fridge? Coconut juice. Question number two, which book are you reading at this moment? I've started a lot of books, I haven't finished it, but at the moment I'm reading one on trying to read outdoor clues and sign. I'm a keen hiker, so it's interesting to understand how trees grow and read the landscape. I will add a link in the show notes to that book. Yes. Which superpower would you like to have? I would love to fly. I love to watch birds of prey, and I'm just amazed at how much they can just soar without any required energy, and they can just see the world very, very quickly. The next question is, what did you want to become when you were a kid? Definitely not a doctor, because I think I faint on the side of blood. I don't quite remember. I think actually very early on I wanted to be a designer very oddly enough, so maybe I've just been obsessed with it from young. Oh, good for you. Maybe quite sad. Well, I wouldn't say that. Finally, the final question is, when did you first hear about service design? Actually, when it really first came out, I think probably 10 years ago or more than that. I remember the design council had a promoter festival here, Design of the Times, 2007, actually more than 10 years ago. And they were bringing together, I guess, what you might call very early service design agencies and working with the community in an office. And I also have another good friend, Lauren Tan, who worked on the first book that Emma and I did. And she did probably one of the first PhD on that and looking to sort of design methodologies used in the projects that was part of the design of the time. So actually very early on, and I was working with people like Professor Robert Young, who was quite early on theorizing the topic area. So I was, yes, I was aware of it quite early on. Cool. Joyce, welcome to Touch Upon a Topic. We did a sort of preparation call, as I do with all my guests, and we went over a list of what might be interesting topics to discuss.  And I said, I think there's one topic that would really be interesting for our audience. We're going to talk about how to evaluate social impact, right? That's one of the areas that you're passionate about. I think there's a lot to explore. One of the most common questions I get is how do I sort of prove the value of my work? How do I measure the value of my work? How do I convince other people that it's actually working? So I think there's still a lot of mysteries that need to be unraveled. You've done a lot of research around this. So that's the topic for this episode. Before we come to dive into that, I'm really curious, how did you get on this journey? And how did you get interested in figuring out how to evaluate social impact? Yes, so I guess from a personal practice point of view, I've been involved in delivering some service design projects quite early on. So one of the projects that we were involved in involved turning a kind of traditional face-to-face work-based training program for a company in an office into completely an online digital service. And this was very early in 2010. And the project was hugely successful. By the end of the project, they managed to convert all their face-to-face learners onto this digital platform that not only enhanced the way they were delivering the program, but also helping a lot of cost efficiency in terms of not having to travel and for the advisors and the trainers to face-to-face and all that. So as well, I mean, they're still doing that. It wasn't replacing it, but it was much more efficient. And then they were able to kind of manage all the documentation online. So we knew the successful form of, I suppose, business part of you, cost savings, efficiency. But we also knew that it helped improve the learners experience. So we ended up wanting to do an evaluation on the social benefits of this service. And we at that time decided, OK, well, that's what are the kind of evaluation methods out there. And what's like, design is really hard to quantify in terms of the benefits and outcomes. So we ended up finding this method called social return on investment, SROI. So it's like return on investment, but with an S at the front, adding a social. So we thought, oh, that's perfect. It's an established way of evaluation. The business people will get it. They'll understand it. And we tried to use this methodology. I mean, it's quite an involved methodology. Actually, you have to be officially trained to use it. We almost kind of did it like guerrilla style, kind of learned about it and ended up trying, just trying it out as a test, really. And we got the organizational board and they were really happy to do this. But it ended up really, it just felt, it didn't really quite work out. I think it was because the fundamental way SROI works is that it gives a financial proxy to a, to an outcome. So trying to quantify some, you know, like cost savings of journeys, how much does it? Yes, you can save on fuel, but what about reduced pollution, better wellbeing, for example? It's very hard to, you can find some proxies, but they're also a lot overlapping proxies. So how do you avoid double counting in the kind of accounting speak? So that's the fundamental way in which we did it. We found it very challenging, very hard to really pinpoint exact benefits. And so we ended up just, we did it, but, and we, it did tell us certain things. But actually a lot of those, those, those insights really came from interviews and sort of long form narrative and understanding what has changed rather than trying to quantify that into a financial proxy. So that was the start. I kind of, you know, learned a lot from that and realized it just wasn't doing what we needed to do. And, and then how I ended up now focusing in much more in this space is because my research is really around working with designers who support social innovation initiatives, and mainly around the Asia Pacific region, because I'm originally Malaysian and I still have lots of links there and I'm really interested in the region in terms of how design is being used now, light in the West to, to deliver innovation. And so when we spoke to a lot of practitioners there, you know, just to find out what they do and what the practices are, you know, a lot of the challenges that they keep talking about is social impact and particularly around social innovation, you know, the main thing, the main outcome is about improving people's lives, giving social benefit. But of course they come, come across the same kind of issue that I personally have, you know, encountered, which is how do you mean flu evaluate social impact? Yeah, and even I think it's good to address that even though we're talking about social impact and maybe social innovation, a lot of the same challenges apply in the world of service design. So I think in a lot of cases it's almost interchangeable. One thing that I noticed from reading your stories and reading your research is that the word evaluating rather than something like measuring the impact. I think it's a subtle difference, but it makes, it makes, it makes a lot of difference, right? Can you elaborate on that a little bit like evaluating versus measuring? Yeah, actually we use the term evaluation because I guess in the parlance of the sector, it's quite common. So in international development programs or social programs, for example, so they often, you often hear the term monitoring and evaluation, M and E. So M and E in itself is a base in itself. So they often have an external partner that does an M and E on a program. And then it's not done externally. So M and E is a used term. We don't like the idea of monitoring. So we thought, okay, well, evaluation is actually what they know. So, and this is the term that is used by the practitioners. So we went with that. But I also think that even the term evaluation has a lot of baggage with it because it is linked to this idea of measurement. It is linked to this idea of monitoring. Making sure that the money that you've given to the communities is used properly and not being wasted away. So there is a lot of baggage relating to simply because historically that's the term and that's the function of trying to make sure that the money is because obviously a lot of these funding is either coming from government or private philanthropy and they have a board of directors and they want to know where the money goes and whether it is of good use or not. So that's why we've used evaluation. But we've also used it to change this idea of moving away from monitoring and assessment so much. Okay, I'm learning with each and every episode of the show that there are new areas that I know nothing about, which is awesome. One of the questions that I had was, what kind of limitations are there to existing ways of evaluating? Like why did you set out to find a better framework? So a lot of the MNE or monitoring evaluation models that are being used is very similar to the SROI that I mentioned which is pretty much set up as an assessment tool. It's also based on accounting models, so putting financial proxies to outcomes. What is a financial proxy? So for example, if one of the outcomes is to improve or increase the number of children going to school and finishing school. So you would then put a financial proxy to that. So if they were to be kept in school and then to leave, they would get better jobs and the jobs will give them 200% more than what they were earning if they didn't have school. So it was trying to kind of give those proxies to those outcomes. When actually what is more meaningful is that if they stay in school longer, yes, they might have an opportunity to get a better job, but it might mean that they have children at a later age they're not forced into forced marriages and they're not done. So all these are really hard, you can't quantify that. So I think that's also the challenge. So a lot of what we call the traditional or dominant way of evaluating. I wouldn't say all evaluations like that. There are a subset of evaluation methods that does take into account these longer term social outcomes. But the majority that is used and the one that is probably used because it's quite easy is a lot of its quantitative measuring number of people in the program, increased income, participation, all that sort of thing. So things that we know is easy to measure and it's easy to measure at the end of a program, but a lot of these evaluation doesn't often, unless you have a lot of budget, you can't go back six months, a year, two years after to look at the outcome. And we know actually social outcomes take a lot longer to develop over the, you know, some could be five years, some could be 10 years. So that's some of the challenges. And also the reason why I think we're very keen to change the conversation as well from, you know, what we hear from the practitioners is the challenges to evaluate and the stress as well, because they have been told by the funders they have to evaluate these and they may not have the capacity or the capability to do the kind of evaluation expected to be reporting at, you know, board level funding level. So it also puts pressure on the practitioners and they feel that sometimes the criteria is that it's being asked to be evaluated by the funders, which is set like miles away in some sort of boardroom or practitioners who do cross-national, cross-international programs, they want to do some cross-comparisons. So they set a baseline. But it doesn't quite work for the community, specific community. So they also find that the criteria they're being asked to evaluate doesn't quite match what they think is coming out of it. So they get frustrated that, well, why aren't they looking at this? Why aren't they looking at this? So the model isn't working quite well for the, especially in the way if you're using sort of a more design approach, you know, very emergent and you don't know what you're going to find out. You don't have an outcome. You know, you believe in the process, but you don't know what, you can estimate, you can hypothesize what outcomes, but it might not come out. So it doesn't fit the model, first of all. Secondly, it's not equitable. It's someone on the outside telling us what we've done well and what we haven't done well. And that's very disempowering for the practitioners and the communities. And it doesn't quite often capture what needs to be captured because they're probably measuring the wrong things. I so feel the pain. I've been in so many situations where this was the case and the challenge with services is that there are systems and systems are really hard to measure on an individual touch point level. You sort of have to observe the system and then it becomes really complex to measure all the side effects and side benefits. Like you said, measurements are set out, standards are set from outside. It sounds like a lot of challenges. And if we go back, what have you found to be some of the root causes? Is it because it's easy to measure people at the top are trained this way? What's preventing us from adopting a different way of understanding success? I think it's the same kind of challenge that a lot of designers face when they're trying to help an organisation be more people-centred or more design-led. There's this one way of doing business, which is very bottom-line and seeing processes and outcomes, but not seeing the human dimension of it. It's the same thing here. International development, certainly at top-down level, they just see it as programmes that need to be defined and has all these processes and methods in place. They don't quite think about that. I mean, its mindset is a way of thinking, I think, but also it's also very hard. It takes a lot of time and money to do it in that sense. For example, really evaluate and really look at the outcomes. You do need the resources to do that, but also the framing of this idea that we do need to kind of measure and know where the money is going. I guess what we're trying to do is not to replace this existing model because we understand there's a need for it, there's a need for it to be reported and a need for it to be accountable. Certainly with public money, you need some sort of accountability, but what we are trying to do is actually change the conversation from this language of measurement to using evaluation as learning. It feels like a missed opportunity because this idea of evaluation is something that designers do all the time. We don't call it evaluation, we just call it prototyping. What struck us was that actually what the practitioners were doing was they're doing this all the time. They're like kind of reflecting, they're co-designing, they're prototyping with the community, but then there's not calling it or framing its evaluation because it's not recognized by people who potentially are not involved at the kind of delivery level that is part of learning about the project and giving better outcomes. So that's kind of one of the reasons, one of the sort of challenges of this topic really. There is a gap between how we evaluate and learn versus the way people expect us to report success. Exactly. Have you found ways to sort of bridge the gap or bring these two worlds closer together because that seems like a really tough challenge. Yeah, we're doing it in a number of ways. So we obviously spoke to the, we ran a dedicated event through this network that I lead with my colleague Yoko Kama and RMIT. So we set up this network called the Designing Social Innovation Asia-Pacific. That's the up for short. And we gathered together the practitioners on a number of events in Asia-Pacific to A, understand how they evaluate their own work and learn from that and also to kind of have conversations about how they see the role of evaluation and what it does. And a lot of the work that we've then come from that kind of discussion. So we derived this series of principles and framing. And then we also felt that it was really important to talk to funders to analyze the gap. So the practitioners told us the challenges, but the funders, how would they respond? So we ended up having a workshop with the funders. And we publicized it as, are you interested in different ways of evaluating and obviously they self-selected. But the ones who came, they were ranging from quite large organization, international organization to all philanthropic organizations. And they were aware, they certainly aware that this is an issue that is a gap between how they normally do things and actually what happens in the crowd. And they're not aware, but they also find challenges, same challenges that we encountered is legacy. There's a set way that the directors want them to report. They don't have the money to look at these impact along the term. And some of them have the agency to change, like a smaller philanthropic organization that we've worked with. They're really keen and they work on what they call a trust-based relationship, which is basically they don't ask for project proposals. They work with partners long-term. So if the partners say they need some money to do a project and they say, right, let's have a conversation rather than submit a proposal and it gets evaluated. So they work in a very different way, but they can because they're smaller. Larger organizations have all these structures in place and ways of doing it. So we know there's a recognition, there's a challenge, but what we found out is that they didn't really know how to resolve it as well. So we're having ongoing conversations. And then also we've been starting to do specific targeted work with organizations. So I've been invited, for example, by a team in the Young Foundation in the UK. So the Young Foundation is one of the longest, I guess not-for-profit organizations working on delivering social innovation programs in the community since the 1950s. So they were invited to be part of this community-driving change program that's funded by the Tower Hamlets Council in London. And the program is really about trying to work with residents to help them understand things that can improve on their well-being and actually help them design interventions to deliver like, you know, service design type interventions into the community. And they're one of the many providers. So they work with other partners as well, but that team itself in the Young Foundation is really interested in using evaluation to help them learn about the program. So I then did a series of workshops and coaching with them and helped them just use an existing platform that they do and map onto what they always do anyway, like team stand-ups on a weekly basis, but get them to reflect more formally on this platform and note that down and has very specific questions about the project to use that reflection to learn and to refer back to on a weekly and monthly basis. Now, it may not sound like much. It sounds like, well, that's a really simple solution, but it's already tapping into what they're doing but making it a bit more thoughtful and a bit more kind of, you know, recorded. And that sort of helped them within a short period of time. They can sort of see the benefit of that so they can see what they reflect on, what action they needed to do on a short term but maybe perhaps longer term. And there's also, pleasingly, you know, the program manager said that, you know, he's able to go into these reflections and pull out some key unsanitary insights that he can put into the quarterly reporting that they have to do. So pleasingly, it's not just helping them learn about the project and improve as they go along. It's also feeding into the more formal type of reporting. So that's been a few ways in how we're trying to approach this huge challenge. So what I hear you saying is that one, maybe we as designers need to take more time to capture our learnings and formalize them and make them tangible. I think we like to think on our feet and really quickly incorporate our learnings into the next action we take. So I think we like to do things fast paced and formalizing and capturing things feels like it's slowing us down and we sort of overlook the benefits or the value of making things sensible. I think documentation is a big challenge in a service design for sure. There's so little documentation. There are so little known and well-documented case studies, for example. I'm curious, can you give a few more tangible examples of things that you've encountered that maybe service designers are already doing but not realizing that they could use those actions to use it as reporting? Yeah, so basically, what they're really doing on a kind of weekly basis, I guess, in terms of their own reflection. So one of the exercises we gave the practitioners in the workshop was to ask them to reflect on the way they evaluate the work, but on three different scales. So the scale of it being a personal scale. So how do you reflect on what you've done and what has worked and what hasn't worked? And then you have the team scale. So how do you as a team catch up on things and learn and work out, okay, we've done this, but this hasn't quite worked. So that's kind of the middle scale. And then the largest scale, which is, okay, evaluating with the stakeholders and all the other partners and the community. So that's the largest scale is the more formal evaluation that our practitioners are used to or used to be asked to do. But what was really interesting and what didn't surprise us was that there were lots of things that we're doing at the personal and team-based level. So things like on the personal level, keeping notes, keeping diaries, keeping sketches, keeping audio recording, after you're driving back from a workshop, you might just chat and say, this is my quick reflection. And then at the team level, if you work in an agile way, you'd have stand-ups, you have daily stand-ups, you have weekly stand-ups, and then you might have team meetings, you're not capturing everything at once, but you might have some notes that you make. You might have photographs, you might have team documentation, et cetera. So all these things are already part of a designer's process. What perhaps isn't so obvious is that how you use that to maybe assign some reflection time. So we use cold sort of reflection, action learning cycle. So you don't just describe what just happened. You try to understand why has that happened the way it did, how do I feel about it, and then actually what I want to change about it. And so it's very action-orientated, which fits design to the T anyway, but it's much more explicit. And when you have those little action points, you want to do this, this and that, that can be logged and taken into the team meeting. And so I've had this issue with so and so, and the community wasn't quite responding the way I wanted to, so I just tweaked the methods and the tools that I used and tried it again. So then you can sort of see and have a track of these action points as we go along. So just by having, and that's what I think hopefully is working with the Young Foundation team, the CDC team, because we were aware we didn't want to ask the team to do any more additional work that they already had to do. They're already time poor, but so the reflection, the diaries that we asked them to do is not long, which is just like four questions. And sometimes they were also interested in like, who do you speak to? How many people have you spoken to in the community? What were the reactions? So it was also a way to log the range of people they've spoken to and who they perhaps might have missed out on need to talk. So it was helpful for a number of things, revealing these things that is often very implicit in a practitioner, but not logged. And if you think about someone who then has the right report at the end of the project, they need to have access to these things to be able to kind of tell the story. But all the story they can tell is some service blueprints or personas, and then the, you know, the beautifully designed product services, it doesn't quite, it tells artifacts, it doesn't tell stories. And I think that's the important thing about understanding social impact. Sorry. What I've been hearing in your story and what I recognize for my own practice is that a lot of the value is in the process. The big mistake we tend to make is to sort of just only document the outcome. Like you said, the artifact. Well, a lot of the value is created in the process, in the learning. And there are, you need different ways to capture that. You do not capture that in a single drawing or single blueprint. And that's what I'm hearing you describe and capturing the value that's created throughout the story, throughout the process. Is that correct? Yeah, I think so. And it's not just for the benefit of capturing it and then obviously being able to celebrate the success with the funder, but it's more important to celebrate success with the community. It's also about understanding what the community value is. So you're working with the state, working with, sorry, service users, for example. So using this, if it's a service design project, what do they value? What do they value's impact for the work that you've done? The client may have a very specific outcome they want to achieve, but actually the value that has been given to someone who's just by making their life a lot easier to access tax credit, for example, it could be enormous, you know, but we just see that, oh, how many people have accessed the service, you know, 20% up from last year, great. So I think it's also about celebrating success for the community and also for the practitioners because it's not just about I think it's about changing conversation, knowing that what they're doing is evaluation and they're not asked to do something additional that they're not familiar with or they have to do, you know, they always tend to think, oh, I have to do evaluation. Oh, okay, I have to do an additional work, but actually this isn't additional, it's part of what they do and we do it all the time, and actually if you get them to think about impact right from the start of the project, it's really helpful because you're constantly trying to think, well, how would this impact on these criteria that I'm trying to aim for. So it's actually quite helpful to think in that way and to change the practice, to enrich the practice in this way. So when designers think, okay, it's additional work, I have to evaluate, I have to formalize it, there's probably a reason why this feels like a burden, is it why aren't they sort of feeling or experiencing the value of their efforts in capturing this? I think because it's often an add-on, right, so we don't we only have to think about the value of reporting, perhaps if we maybe are asked to do it by our client and we might think, okay, well, we need to put additional resources and all if they think evaluation, oh, we need to do some surveys, or we need to kind of test for these things. And so it does feel like it's an addition to what we already do when we're very much about you know, delivering or designing a service and you know, offering. So, but I think I go back to this thing about changing and framing not just for the funder's point of view but from the practitioners, this idea of evaluation, that it's not measurement it's learning and we do that all the time so if we can just find something in what we already do and maximize it, leverage on it, so for example the notes that you do and the the team meetings that you have and if you just have one agenda items around what have we learned that to the project that might help improve the outcome of the project and that little line could then be used to help build that story of the evaluations you see, I mean it's just small little things but surfacing that and then noting that down and whoever's then responsible for bringing the story together doesn't feel like they have to recreate that story, it's just there. And I think you touched upon a very important point here you have to do something with the things that you formalize so if there isn't a moment where you come together with a group and reflect on your learnings and adopt that back into your practice then it does feel like you're burdened because you're putting time and effort into documenting something that's just there for the history books rather than something that you actually get benefit from so that additional step of using it is I think the crucial key here. Yeah and I think that's why this thing about learning so using those notes that you made to say well that didn't go very well that meeting we presented the ideas and the stakeholder the users didn't really respond well why didn't they? Did they simply not like what we presented or did we present it in a different framing or were we completely off the mark so you use that to say okay what's the next step what we're gonna move forward with so it is very much reflective reflective practice it's action oriented you're not just pontificating saying well why didn't that work the world doesn't understand us but actually you need to then turn that into actions that you can test or try out a prototype and then we move on so it's very much about yeah it's not an additional thing it shouldn't feel like it but because of all these the history behind it the legacy and this idea of design always having to prove its value feels a bit like there's always a negative perception to it so it's used to be externally driven you're doing it for somebody else rather than for yourself and I also think if you look at the double diamond which I have some issues with like evaluation or the meta level like learning from your actual process it's not visualized in the way design processors are often represented and that sort of also gives misguides people like we're just moving forward we're just moving forward without incorporating evaluation or reflective thinking or critical thinking in our approach to actually move faster get more progress so it's also just a lack of how we have communicated about the design process so far most definitely I mean one of the things that I'm trying to imbue in our students at undergrad and master's level is how are you going to evaluate your work how do you know it's worked and they're like I'll just do a survey I might observe I'll get a few friends to test out the system and sometimes it's as much as they can do but I think so this idea of actually and also evaluation doesn't just happen at the end it happens at every stage right so but I mean it's also our fault we just never as a practice really talked about it and it's not the sexy end is it I mean we're like oh well and sometimes the ego gets in the way because you think it's going to work because I know it is going to work trust the process yeah that's what we say trust the process yeah that's what we say so I think I think it's partly the discipline practitioners need to also pick up on this as educators we are trying to pick up on it but it's not often that easy because students are still trying to learn how to be designers before they even want to evaluate right so but we evaluate as students we value them all the time so we want to emphasize that as well as changing the language and model that funders are using so it's it's kind of a many prong issue yeah I think and I'm not a trained designer by any means but from what I understood from a design field like graphic design design critique isn't sort of integrated part of the graphic design process or maybe it used to be and I don't hear almost any conversation about design critiques within a field like like service design I'm curious why we've never adopted a practice like that which I think would be extremely beneficial for everybody yeah that's interesting I guess because my background is graphic design and I moved into digital design and then interaction design, service design almost because I can see the benefit of applying design thinking across different ways of doing things yeah I don't really know actually I guess it does I mean probably still does happen at certainly education service design courses I know they when students present the ideas there is a level of critique there and discussion but whether that happens in practice I haven't really worked professionally as a service designer in an agency I've always done it in a kind of research academic context in which you're always trying to actually work out what you're doing is actually helpful or useful and there's an added incentive to also want to write about it because we need to publish so this idea of actually going back and reporting on what's worked to a certain extent we have added incentive but if you're a I mean I have worked as a professional designer and I know from that experience you don't have time you go from one project to another reflect and say oh how did that project go you know not even just reporting externally it's just within the team you're just on with it you don't have time to sit and reflect so I think it's also about the practitioners well talking about it talk about it with the client and even just saying that and planning it in in their costings just very simple things and I think it's going to be a really important part of it has to be mentioned because I think more and more these services you know you need designers to understand the business value the value of it to businesses so they need to be able to defend that so let's go back to that conversation because we sort of we need to still address it because at some point you are going to run into the person who's managing the excel sheet now I'm curious like have you found ways to open the conversation with people who have that kind of mindset like we're all about success will change it's emergent we don't know what will happen and then there's somebody who needs to manage an excel sheet like how do we successfully open a conversation with somebody who's in there in that position I think you try to sell it to them in a sense that ultimately if we adopt evaluation as learning it will lead to better outcomes and that could be better financial outcomes or better social outcomes or you know and they're linked so it isn't it's almost like you know why wouldn't you do it if it's and if you can say look it's not going to it's not going to there's an additional not a huge amount of additional time we're already tapping into what designers are already doing we're just formulating a little bit and if we can use for example these standard meetings and we record it or we do a quick recording at the end to say these four points what happened how we felt what we've learned what we're going to change right so and that we can take that forward to the next meeting and argue it from the point of is already there it's not like an additional thing we have to do we might have to enhance it to capture and be more mindful of of talking about it in our day to day practices and keeping an eye on that but ultimately it will then lead to better outcome and when we then say to our client we are able to see the value not just the value that you define but the value such as in this way you know it makes for a better story a better outcome reporting that you can feedback to the client really and the challenge still remains like if we are creating value that's not being appreciated like when it's not being captured in the excel sheet it might be value that we find valuable but if the people who are in positions who are in the decision making positions we still we will always keep this challenge right if we're saying okay but the community is happier and healthier but there isn't a square in the excel sheet to actually express that like that's I think the bridge we somehow need to get to show to educate or to co-create new ways of capturing value yeah I mean certainly I think the key is not to and I use this in my reflection you know not to dance to someone else's tune but to create your own tune and to invite others to join in right so I think if you advocate and if you can show and yeah some things just don't fit in the excel sheet and we should recognize that they are not just one way of doing things in this world and we know that if we do that it's led to what we have now which is kind of you know environmental disaster global pandemic falling economies so there's not one way and we know this one way of working doesn't work and this kind of so we have to kind of acknowledge that yes you can view the world through an excel sheet but it's limited and if you want to do other things with it and look at other kind of outcomes impact you need other ways of doing it and I think they can coexist I'm not saying we should replace one or the other I think the key is this idea that you can hold multiple views in your hand and not be like completely confused and you have a meltdown right but not many people can we take it for granted that I can you can see different multiple approaches to it I think it's about I wouldn't say replacing it's just bringing them closer or bringing it closer to each other and say they compliment and they support each other I still don't think the excel sheet should dominate and we need to do that I know it does that's the reality at some point in 99% of the organizations at some point there is the excel sheet that wants to have three that has a three month horizon or six month horizon and then that's the struggle we face one that the impact we create is hard to quantify and it's long term and those are two things that sort of clash really hard with the way impact is represented within at least profit organizations and maybe also not for profit yeah I it's you know we're limited to this construct which is yearly budgets reporting the length of a project which has a style and end point I mean I argue that the way designs set up so much focus on a project is problematic because a lot of the work that's done in social innovation is so long term and when you intervene you're just part of that at a certain point of timeline and then when you extract yourself they're still there so you need to be able to make sure that whatever you intervene you're then leaving behind capacity or tools and resilience so that they can carry on certainly the models in which we practice is set up doesn't quite fit but it's also because we're having to fit in with the way the world is set up the business world or even for nonprofits they have their own way of doing things it is a yearly cycle you've got a budget and you just spend it and then we don't know whether that's going to be progress I mean there's no I'm not offering a silver bullet here but I think what's actually important is to recognize there is a problem and to have that conversation and as practitioners to first of all think about it in terms of how they're evaluating their practice so that they can see the benefits for themselves then secondly how you communicate and convince the money man that it is a benefit I think there is hope because we're having these conversations people are talking to us they want us to support them they're reading out the results that we're offering so it chimes in them so they recognize the challenge they just don't know how to solve it like many of us because we're so tied into the systems we're in but they're taking even the work I'm doing with Young Foundation they're just taking small steps but that's better than nothing and they're seeing improvement at the kind of practice level where they can also see better reporting to the funders that was a surprising thing because the program manager is like oh yeah I can read through all the diaries and then I can really gain some insights rather than have at the end of the project have a long meeting and work out so what came out of that one key area here is that we also have to accept and be proud of that we express impact through stories through narratives is a valid way to show evidence and I think we are in a lot of cases intimidated by the quantitative data and sort of back down on our ways of showing impact and step one would be to agree with ourselves that we are to marginalize the ways of presenting impact that we have yeah definitely I think we need to obviously get smarter with understanding the data that comes out of quantitative stuff and but also call out is limitation you know not be afraid to call it is limitation and say well it doesn't tell us this it doesn't tell us that but we know like on the ground we speak to the community and we work with them we can see these changes and it feels great and I want to share it and how best to share and there are methods I'm not saying there isn't methods we're using we're using a method called developmental approach which is quite well established in Canada Australia and essentially it's about this idea of using evaluation as learning and it's an approach it's not like tool kit it's a method you can use anything you can combine them but it's about this idea of using it learning and it's iteratively it's a cycle so it marries really well with the design process you know and so there are already tools out there even just methods like stories of significant change so you ask the community what's the most significant change you collect that and it comes through as quotes or it comes with stories and I think we should really give prominence to those rather than you know the kind of excel data that comes out and go oh yeah yeah that's great I know it's easy to report on but it's a bit lazy on our part if we don't maximize the other side of the the kind of messiness of it and I think again you sort of mentioned something which is really important a way to present the value of the way we show impact is by showing the limitations of sort of the traditional ways so there are countless of examples where things were measured correctly and showed a positive ROI but in the end the effect was disastrous and I think that Trisha Wang who was recently on the show she talks about thick data versus big data like you can have all the big data you want but if that doesn't give you the insights to act upon it like it's useless so I think those kind of narratives we also need to develop and be smarter about showing the contrast and also showing like okay if you want to have big data then probably I'm not the right person if you want to have thick data and you want to have qualitative insights that will lead you to this I'm your man and I think we need more convincing and smarter examples there yeah most definitely I think you know data is as you say big data can be dumb data if it's interpreted the wrong way and it's all about the interpretation so but you do need data it has to be data driven but you need to be smart with how the data is interpreted and what importantly what data you're collecting as well you know sometimes it just spend a lot of money resource collecting the wrong data and it's just you know it just wasted resources and so I think we need definitely to be smart about that to call out but I think as designers you need to take ownership of it I think that's the key as well you know because you might maybe in traditional design education you're not really taught to think about impact or evaluate the work apart from some user testing and you're like okay but not really think actually how might this really impact and here we move on to this discussion around kind of the ethics of design and you know bigger bigger topical issues right and responsibility as a designer but I still don't think as a discipline we take we take ownership of these aspects as much as we just don't maybe consider as part of our our tools our toolkits you know we're good at idea generation and only recently we're like okay we can do some design research yeah we can do that and then we can use that to come up with ideas and we can prototype it and we can present it but then the evaluation bit is like oh that's for someone else to do you know or someone at the business side to do I don't want to be involved so I think it's also about taking ownership of this topic if you would have to if what do you hope is the one thing people will remember from the conversation we just had that to use evaluation as learning simple as that and to not be afraid of it and to not discount it and to take responsibility for it and that they're already doing it so it's nothing new they have to simply not simply but formalize it and think of it as evaluating the outcomes for the project itself yeah do it consciously and deliberately and formally that's already would be a huge huge win yeah I know you have a lot of material out there for people who want to dive deeper into this what are some good resources to follow up on I think to read up on this approach that we've referring to the developmental evaluation there's some easy quick guides that I can we can share with the links we've also written a number of reports and papers about the work that we're doing with practitioners and we introduce what we call the designing social impact evaluation framework and it's a series of I would say principles in how you might want to think about evaluation as learning and how you might do that in a more practical way yeah and we have videos of that we did of the of the gathering that we did with the practitioners about impact evaluation in which they share the methods and the way they're doing things and some of the challenges that they have so I think that would be quite rich resource to at least hear from the practitioners themselves and hopefully it chimes with a lot of your listeners I'll make sure to include all the links in the shout out so people can follow up on if people have ideas or questions about this topic is there a way they can reach out to you yeah feel free to just get in touch with me by LinkedIn or by email I don't know what you prefer to share so I'm happy to have conversations and just share ideas really awesome Joyce I think we've touched upon a super interesting topic it's definitely a topic that needs more maturity in the coming years so I hope this is a call to action to everybody who's listening to think about it work with it design it and I want to thank you for bringing this to the attention on this episode that's great Mark thank you for the opportunity to talk about this work to a wider audience because it's been mainly residing in the academic world and I think it's although we've been talking to practitioners I think it's really important especially for designers who are moving to space of social design and social innovation changing systems and processes and people we are making impact and we are making or delivering social impact we just need to make sure that it's the right type and that it fulfills it does deliver benefits to the community that you're trying to serve really so I am it's definitely a topic I'm passionate about and I would love to hear from any of your listeners about their experiences and anything else that we can discuss so thank you what's your biggest takeaway from this conversation with Joyce leave a comment down below and let us know if you made it all the way here you're apparently enjoying these conversations so make sure to click that subscribe button so you won't miss any new episode in the future thanks a lot for watching and I look forward to see you in the next episode


Hello friends, I'm Sanjay Gupta. I welcome you on Sanjay Gupta Tech School. In this video, I'm going to demonstrate certain cases while creating validation rule. So first is when record is created and edited. So validation will work in both the scenarios. Second, when record is created only so we want our validation rule to run when record is created only if it is edited that validation rule should not work. And third is it's vice versa. So if record is edited, then validation rule should fire. If record is created, then it should not fire. Right. So before starting, if you want to follow Salesforce training videos, you can search my name Sanjay Gupta on YouTube and you can follow my YouTube channel. So I created a validation rule on opportunity. So this validation rule is if an amount is not blank and description is blank, then this validation rule will fire and it will throw an error message that is please fill description. Right. So right now it will work in both the scenarios while creating opportunity record or dating record. So let me create new opportunity. So its name is test. Close date stays so I filled all these details. Now I'm filling the amount. So if amount is not blank and description is blank and I tried to save it. Sorry, this is not active right now. So that's why it won't work. So I'm activating this. Again, I'm creating new opportunity record amount, but description is blank. Now if I try to save it so you can see review the errors on this page. So it is showing please fill description like if you fill the description then only this record will be saved. Now I'm removing this amount and I'm saving this. So it is created because amount was blank. Now if I try to edit this record by filling amount and I try to save it. So again, this will throw an error message. So it is for description because amount is filled and description is blank. So this way it is working on both creation and updation. Now our second scenario is when record is created only so it should work while we are creating new record. So we need to update this validation rule. And I'm going to add a simple condition here that is is new. So this is added. So if record is new amount is not blank and description is blank then only it will fire right. So I'm saving this record. Sorry, I'm saving this validation rule. Now I'm refreshing it. So you will see on updation it will not fire amount will be filled and description will be blank still be will be able to edit and save the record filling the amount only description is blank. So you can see this record is saved and you can see description is blank. So on updation validation rule is not working. Now if I try to create new opportunity record and if I fill amount and leave description as blank. So you will see this is throwing an error message please fill description right. So this is covering the second scenario that is firing on creation only now we need to revert it for editing only. So I'm editing this validation rule. I'm just putting not with his new. So if record is not new, then this validation will fire so on updation it will fire on record creation it will not fire. So here, if I open this record. So here it is. So let's try to create new record. So I'm filling me close date and stays only. I'm populating amount as well description is blank still it will create this record right. So if I edit this record amount is filled but description is blank. And if I try to save this so it will throw an error right so this way you can use is new method. Whenever you want your validation rule to work for the record creation or record modification accordingly you can modify your validation rule. If you want your validation rule to work for both the cases then you can remove this is new method. So this way we can implement all these scenarios while implementing validation rule. I hope you understood whatever I demonstrated in this video. If you want to watch more salesforce training videos you can search my name Sanjay Kutna on YouTube and you can follow my YouTube channel. Thank you for watching this.


Hi, I am Professor Madhusudan Upadhyaya, a professor of anesthesiology, Kasturba Medical College, Mangalore. We all know that the women folk who deliver are doing one of the noblest jobs in life, procreation. And we also believe in anesthesia that every mother has to have a painless labor because she deserves the best. When she is procreating for the society, she also deserves the best. We have devised a lot of techniques to make the birth of a baby comfortable for the mother and pain-free for the mother. That is when we have devised a lot of techniques. We have drugs to be used and we have nerve blocks to be used and the gold standard today is the epidural analgesia. Everybody must be wondering what is this epidural analgesia? It's nothing but we push in a small tube into the back, into the epidural space which is just outside where the nerve roots are. And we push in local anesthetics or some narcotics, some additives to make the limbs and the abdomen or the tummy numb and the mother goes pain-free. The entire process of labor however long it is can be made pain-free by just keeping on pushing local anesthetics into the epidural space. You might ask me whether it has complications? It has but the complications are so rare when the mother is given the best of the facilities to enjoy the childbirth, the complications don't affect much. And each of the complications can be handled by a professional to a large extent without causing any harm to the mother or the baby who is going to see the earth. Therefore the advice basically is maybe the mythology myth says it's a myth basically saying that the mother has to suffer from pain. But mind you the pain the lady has to go through is akin to cutting a finger off someone. That pain is so severe and the childbirth is not so comfortable. Therefore we make it absolutely comfortable and the happy ending would be when you have a child on the mother's chest and the mother is pain-free and happy and contented.


Good afternoon. I will talk about evaluating neural network models within a formal validation framework. This is work done by my colleagues and me at the research center in Julech. I will start at the very beginning with models. So what are they and how can we learn from them? So as you certainly know, models are abstract simplified description of some system of interest and the key attribute is that they can generate testable predictions. And when these models do this very accurately, we might want to infer our knowledge of the model to the system. This is called inductive entrance and is good scientific practice. However, this doesn't mean that we actually uncovered some some truth about the system. It just means that we found a useful description. But how can we find out about useful descriptions and how can we decide which is the most useful? This is exactly where validation comes in. Validation is a process of quantifiably evaluating the prediction accuracy with respect to the system of interest. But let's define it in a more broader modeling context. This is a schematic you see about the modeling environment with three key components. On the top, you see the system of interest. I already mentioned and to give a very simple example what this could be. This could be, for example, a ripe apple tree with falling apples. And by careful analysis and some theoretical considerations, one might be able to construct the mathematical model of it as Newton did. You see where I'm going with it. The interesting part of this is a explicit separation between the mathematical model and an executable model. The executable model is a part which can run simulations and actually do the predictions. For Newton's laws, it would be, for example, the standard physics engine. And this, of course, this code needs to be checked for correctness by a process called verification and only a verified executable model can then be continued to be validated. The validation step directly compares the simulation outcome to observations of the system of interest. And when the two match, well, we are in luck, we found a very useful model. This rather straightforward workflow enables us to compare and evaluate models and therefore an indispensable part of simulation science. However, there are exceptions and additional aspects to consider, especially when the models become more complex. I will briefly mention three of such aspects first. Well, assuming you did the test and you got some result and what now? Well, the result of a validation test is called a score and the score tells you how much credibility you are allowed to place into your model. But any single score can never encapsulate all the features over the whole domain of interest of this model. Therefore, you need to employ many tests with different statistics and focusing on different features. So this also means that the score doesn't necessarily directly tell you if a test passes or fails. This highly depends on the intention of the model. So in the ideal case, the model knows a priori what the acceptable agreement, so the range in which the score should lie, actually is and can formulate this.  And in case you were interested in only the rate profile, you might be in luck because this matches quite nicely. However, if you were when you created the model rather focusing on the exact spike times, as for example, the latency of the first spike, you find a discrepancy and the resulting tests would fail. And there might be an additional test, for example, the regularity of the spike intervals, which uncovers another discrepancy, which is totally invisible to the first two tests. So you need many tests and you need to be aware of what you actually want to model. Secondly, in engineering, this is common practice to start at the very lowest level of simple building blocks, validate them and then build on top of that to validate the larger structures. However, this is not possible in neuroscience for several reasons. On the model side, we have all these different scales from ion channels, neurons, networks, up to behavior. And the relations between these are very complex and often unknown. And also in the data side, we don't have these kinds of multi-scale data available. Additionally, if you have, for example, a single cell model and validate it in some parameter regime, this might not necessarily be the parameter regime which the neural model occupies in the network model. Therefore, the validity of a network model is not implied in the validity of the neural model. And also the other round doesn't work. As, for example, Potience and Desmond showed that the large-scale network dynamics can be accurately modeled by a network model of simple leaky integrated and fiber neurons which are not very biologically detailed. And also beyond this conceptual framework, you might face additional considerations in practice when doing validation. For example, if you want to validate a model, you might lack an appropriate data set depending to that model to come to a strong conclusion. And these cases can be very helpful to validate not against data, but against another model which is more trusted. Or also validate against the same model, but in a different version or immutant of this model. This might be helpful to quantify the influence of little changes as, for example, the underlying neural model or the solver for the ordinary differential equations. One of these changes can also be the choice of the simulator engineer use. And I will come back to this precise example later on. But how could a software tool look like which actually reflects these different aspects and is versatile enough to adapt to the different challenges specific workflows can have? So the first thing you do when you develop a tool, you look out if there's already something like it out there. And in this case, indeed, there is a Python package PsyUnit which provides a general framework for validating scientific models. So our tool, NetworkUnit, for network level validation of neuroscientific models, fits directly on top of that and also uses the implemented electrophysiology analysis methods provided by the open source project elephant. So let me show you the structure of this tool. So the key components are the data and the model. The model is a class object which is able to run the simulation and if Newton would have been into coding, this is what he would have written. The data would be our measurements of the following apple. And the corresponding test could, for example, be a falling velocity test which tests the velocity of the apple after a given distance. The test is also a class object which is able to compute the velocity from the model simulation and then compute the corresponding score in comparison to the data. This score is not only the quantified result of the test but also this object contains all the information of the test and the model to help with the interpretation of the result and also to ensure its reproducibility. Besides reproducibility, another design object here was modularity. So aspects of the test like the type of score used, the statistic is not hard coded into the test but attached via a separate object. And also the parameters used are attached separately so that consequently there exists a more general test class which is agnostic about score type and parameters and can be reused for different variations of this test. This would be a general falling test here and scientists might be interested in having a more general test on and evaluating the velocity of an apple when it's thrown horizontally. So there can be a more general base test which handles all movement with arbitrary initial conditions and this is an effort of writing no calculation twice so that everything depends on the same calculations. For network neuroscience this could be for example and as it is implemented in network unit, one base test is handled all the calculation of correlations. A child test of this can then for example compare directly the distribution of correlation coefficients and another child test could instead use these correlation coefficients, build a graph, have further child tests which then test graph centrality measures and although these are very different tests they necessarily agree on the definition of correlations because they use the exact same code. Finally this framework also makes sure that the test actually makes sense and checks whether this model can produce the property in this case movement the test wants to evaluate. And indeed this framework proves versatile enough to also incorporate the practice of validating one model with another model simply by inheriting one dedicated test class we can use the same test to not test against data but against model and the test is actually formally equivalent. For our apple example this could be for example a relativistic model of motion and here we can identify the limitations of Newton's law without having to actually measure superfast apples. Okay let's jump directly to a real world application. So this is in the context of a reproducibility study of the polychronization model which was published in 2006 by Itzy Kiewicz and this model is interesting for at least two reasons. The first one is it produces a very rich network dynamics of spatial temporary organized and repeating spike patterns but the second reason is arguably more important. Itzy Kiewicz actually published also his code which made it possible to reproduce it. So first study focused on the exact reproduction of this model here using the engine nest. This was worked by Pauline Vidal and they discussed the model specifications which are necessary to actually do such an exact reproduction. And in parallel we worked on porting this model to the neuromorphic hardware system called Spinnaker and a neuromorphic hardware uses very different kinds of computation than a conventional computer and a conventional simulator so an exact reproduction is out of the question. So we necessarily needed to employ validation techniques to be able to somehow quantitatively compare this to the original simulation which was by the way written in some custom C code. So one study focused on the implementation details and the verification techniques and the second study on the validation techniques to compare this to the original study and I will show a few points of what we learned here. So we started off by a very naively porting this system as this model to the neuromorphic hardware system using all the default settings and you see in the very first line the spiking activity which we obviously had to improve on and we did this in two iterations. You see in two and three by tweaking the solver for the for the ordinary differential equations which improved a lot and by I already see that the third iteration looks way more similar to the original implementation simulation and see but it's very hard to see by I if this is a good reproduction or not. So over this early stage of the development of this implementation we use validation tests. You see again three rows for the three different iterations and three columns for three different tests focusing on the feature of firing rates regularity measured by the LV the local coefficient of variation. And the correlation coefficient. And this we quantified with an effect size to the distance between the two distributions. And here we could actually quantify that each iteration got more similar to the to the C simulation. However, this is still not a good agreement. So to to jump to the very end of the story we further dove into the model and approved upon the temporal resolution and the algorithm for the threshold detection algorithm and and the right here. Here we have a better fit of firing rates for their V also for the correlation coefficient and we introduced three more additional tests to have a more complete picture. The inter spike intervals the rate correlation and the eigenvalues of the rate correlation matrix. So at this stage we we can say okay we were able to qualitatively reproduce this model on this neuromorphic hardware and quantify the level of agreement by these effect sizes. However, this is not a strong statistical agreement. There are still discrepancies here most obvious in the in the higher firing rates and slightly higher correlations. And although these differences might be small they can have potentially a very large effect on the occurrence of spatial temporal patterns. So they should illustrate that all the upside validation testing can have even beyond the standard scenario of of comparing model and experimental data. So to take a take a step back I want I'll give some broader context where we are going with this. So in it was a network unit we mostly focused so far on measures quantifying characterizing the network activity based on pairwise and single measures of spiking activity. And we are now starting off also integrating measures corresponding to the spatial dynamics of of networks. Here for example as you see in an LFP recording of an implanted electrode array which shows wave-like behavior of the activity. Also I really have to mention that there are many other tools like it specialized for certain validation scenarios for as for example a neuron unit which handles the single cell validations. And they all and it's aimed to harmonize these efforts. Especially also as they should be integratable with the validation framework of the HPP which is which basically provides a searchable database of models and tests and scores. So to wrap this up three points you really should take away from this talk. First validation is important. Any simulation without quantification is just guesswork. And there are tools out there like network unit and many others to help with it so you can make use of them. And third if you run validation tests please use more than just one test. And even if it's not enough data you can use other models to evaluate your model. And with this I would like to thank you for your attention and all my wonderful collaborators. And if you want to dig deeper into this topic there's a poster on the simulator comparison study. And there's also a demo on SciUnit by the developer Richard Gerken. And what was this I would like to thank you again. Okay, thank you very much. Questions. Well I think there's no general recipe of how to tweak a model. You really have to dig deep inside and understand the model. And gladly we had collaborators who could very well understand the model and also the underlying structure of the neuromorphic hardware. And were able to find out the discrepancies where some dynamic in the model can't be represented in the hardware system. And there we then dig deeper to identify these mismatches. And for example found that the temporal resolution initially chosen by the model of the polychronization model doesn't work on this neuromorphic hardware. So we had to go to a final time resolution at least for the integration of the membrane potentials. So in this case you had a fully defined simulation as your reference. Yes.  So this is of course always a problem. Data is variable as we know. And a model fitting to one pair of data might not fit to another pair of data. So as I showed, we are able to compare two models. But we are also able to compare two data sets and actually quantify the variability within these data sets. And this can be helpful to define this acceptable agreement I mentioned where we could expect the model, how accurate we could expect the model to be. Thank you. The network unit, the tests that you have there now.  In single cell studies, I think people have been thinking a lot about what are the features of a spike train, what are the features of the membrane potential that they might want to look at and put into an optimization and put into some sort of validation of a model. And for network models, I think that's not, from the amount of time people are working on the science of that has not been super well established. And I know in the network unit you have seven or eight main things that you test, like you showed in the slides comparing the models. What else looking forward do you think people should be asking about when they're trying to validate network models beyond that? Like what are some future tests you could see in network unit that would be ways of looking into whether or not two models are similar, whether models are good representation of the system? Yep. Well, there are several aspects to this question. First, this highly depends on the intended use of the model, what you want to test, and what are the important features and which features you might want to neglect. As for example, in the example, if you only are interested in rates, you don't care for some spike regularity or latency. Another aspect is related to the variability in data is you probably want to identify features of the model which are fairly stable across data sets and don't vary too much. And these would be the features you want to focus on, which again might be different for different species or different kinds of data. So this certainly can be expanded upon those measures and probably should be. So one aspect of this is certainly graph measures, which can be employed independent of what the underlying weight measure is. It's correlation or functional connectivity or maybe causality. And it's certainly very useful to then explore the graph measures of these kinds of activity measures. I hope this answers the question. Okay, if there are no further questions, then let's thank Robin again.


Oh, so you're 12 and 13, I know. I don't even know. At least since I'm getting your career started, I look forward to seeing where you go.


Hey, this is Brett, and this 2019 Ford F-250 Crew Cab Short Box Larry D'Ultimate FX4 is stock number 14157Z. I am here at Summit Automotive in Fond du Lac, Wisconsin, you're new and used heavy-duty truck headquarters. This 2019 Ford F-250 has the 6.2-liter V8 gas engine, puts out 385 horsepower, it's paired up with the six-speed automatic transmission. This truck has been fully-safetyed and inspected by our service shop per the state of Wisconsin inspection process. It has a fresh oil and filter change, all the fluids have been checked and topped off, and this truck is 100% ready to go. I'm going to go all the way around in this video, inside, underneath, start it up and take a look under the hood to give it the most accurate representation of the truck possible. Magnetic gray metallic is the color, I showed all my videos in 4K, if you like the video you can subscribe to the YouTube channel, click the bell notifications to get updates on videos they do each and every day, as well as having access to one of the largest catalogs, a vehicle and heavy-duty truck walk-arounds on YouTube, over 8,000 videos in county. Comes with the 20-inch chromed alloy wheels, they're in really nice shape, I didn't see scuffs, I didn't see chipping on the chrome or anything like that, and it does come with Michelin LT 275-65R20 tires. They have right around half the tread left, maybe just a little bit under. Frame and underbody is an exceptionally clean condition. This is a two-owner, clean title history, clean car fax out of Illinois. Front fenders absolutely perfect, headlight lenses are nice and clear, it does come with the factory fog lights, the chrome tow hooks, and that front bumper and lower valence are in fantastic condition. You also get the chrome trimmed grille and the 360 camera on this one. The hood is in really nice condition, it also has the cab lights up top, which are nice. Passenger side front fender is in great shape, I didn't see any dents or dings on there. And the passenger side front wheel, no major scuffs or scrapes. As you go down this side, actually I want to show you the magnetic gray paint, it's got a lot of metal flake to it, really a good looking color, one of my favorite grays on the Fords. And as you go down this side of this 2019 Ford F-250, take note of how clean the body is, how reflective and mirror like that paint is. I take these HD videos, so if you're far away or even if you're close by and you just cannot make the trip down, but you're still interested in purchasing the vehicle, you can see the truck, hear the truck, and have confidence in what you're looking at before you even get here, so when you do get here, there's absolutely no surprises and you can make a smart and informed buying decision from wherever you're at. If this video helps you make a smart buying decision or just make the decision to come take a look at the truck, let your salesmen know that you saw the video that it was helpful and that Brett sent you, and then I can keep doing these videos each and every day. This one comes with the 20 inch chromed wheels in the back, very nice condition and these back tires actually have right about 50% of the trend left on them, so a little bit more tread in the back frame and the underbody is exceptionally clean as all the remaining factory exhaust and all the lower rockers and underbody looks really really good, comes with a nice set of factory chrome step bars, passenger side box, no dents or dings in there, you do get the FX4 decals on the side, chrome tipped exhaust, coming around to the back, the rear bumper, great shape as well, backup parking sensors, full towing package which includes the receiver Hitchfork and N7 pin wiring, and the tailgate is in excellent condition as well, no dents or dings on there, you get the power drop down tailgate, spring and bed liner, fifth wheel and goose neck prep package, you also get seven pin wiring back here and LED bed lights, this one also has the cargo cam up top and that spring and bed liner goes on to the tailgate and it does have the tailgate step assist, that shuts nice and solidly as we go down the driver's side just as clean as that passenger's side, no major dents or dings in the box and for full disclosure this back wheel just as nice as the rest, I can't stress how nice these wheels are, sometimes these chrome ones they'll get chipped from putting different tires on stuff like these are in really nice condition, the cab and doors on the driver's side just as nice as the passenger's side, no dents, no dings, you do get the power scope mirrors, they power telescope out, they power fold in, they have blind spot monitoring, built in directional signals and the 360 cameras on there, this one does have the door code entry, we do have the code for this truck and if you ever want to figure out what your code is for your 17 to 22 super duty in the upper right hand part of the screen you've got a video showing you how to find that, back seats are in excellent condition, no rips or tears, you do get the latch child safety system for any child car seats you may have, power sliding rear window with the built in rear defrost, these seats fold up like so you get a collapsible floor back here so you can lock that into place, it won't close because it's got that divider up in there but that's a really handy system to have and you get factory floor mats back here, carpeting is nice and clean, 12 volt power point, 110 volt 400 watt max plug-in and two usb hookups there, child safety locks on the back doors and the inside and bottoms of the doors all look really good, take a look at the vinn sticker real quick, no canadian trucks here and then your tire and loading information sticker, inside the lariat ultimate package gives you the black leather bucket seats in the front, no rips, no tears on these seats, smells very clean and size truck, both of these front seats are heated, power and cooled, factory floor mats up front as well, power pedals, power drop down tailgate, auto headlamps, fog lights, power telescopic and tilt steering wheel, there we go, so that's pretty cool, you get the led lights on the mirrors, memory driver seat, power windows locks mirrors and I always like showing the mirrors that they do power fold in and I always like showing both sides so that you know that both sides are working properly and they telescope out and in and out and in, hop inside, check out the miles, the radio and everything that this truck has to offer on the interior, you can see that this one has 41,433 miles on it, you do get the nice digital instrument cluster on there, very nice and clean, comes with the leather wrapped steering wheel, cruise controls and information center controls on the left, bluetooth and audio controls on the right and the steering wheel itself is in really nice clean condition and this is a heated steering wheel, you do get the six speed automatic transmission with the optional tap shift, this one has the sync eight inch radio, you get the factory navigation system on there, there is highway 41 so that is working nicely, you get AM, FM and Sirius XM radio capabilities as well as bluetooth capabilities on here and then you get your climate controls here, your apps, your different settings that you can do, this one has Apple CarPlay, Android Auto and then you can check out all your cameras there, there's your 360 cam as well as your cargo cam so those are all working nicely, stability control, downhill assist control and this one does have the B&O premium sound system, turned out four wheel drive, factory brake controller, more tactile volume tune and climate controls right there, you do get your two USB's down there, keyless entry with remote start and power drop down tailgate and then you get two cup holders, ambient lighting in the cup holders and then here's my magic trick, four cup holders just like that, 12 volt power point, 110 volt, 400 watt max plug-in, dual glove boxes and the passenger side format and seat are in excellent condition, once again it sounds very clean here, I don't think it's been smoked in, has the panoramic sunroof, second real fixed glass power sunroof in the front that's a nice option, there's your six auxiliary up-fit switches, your power side and rear window controls, sunroof and sunshade controls, map lights and your home light buttons for your garage door security systems and lighting systems, we'll start it up, take a look under the hood, starts right up, no check engine lights or anything like that, I would personally like to thank you for checking out the video today and hopefully from this HD video you've been able to verify the quality condition options and of course cleanliness of this truck all the way around inside and out, get those LED side lights, they are super bright and all the front lights are working nicely, cab lights working good too and under the hood we have the 6.2 liter v8 gas engine, engine bay is very clean, runs very smooth, once again this truck has been fully safety and inspected, it has a fresh oil and filter change, all the fluid tab and check pan topped off by our service shop and it is 100% ready to go, shocks are doing a nice job holding that hood up, there's the emissions sticker and this truck is 100% ready to go, I would highly recommend it for make quality and condition standpoint, I'd ship this one anywhere, California, Texas, New York, Florida, you name it, whoever's going to get this truck is going to love it, really nice condition, a lot nicely optioned out as well and just a really nice color and very clean of course and to see more pictures of this truck or one of our other 550 new and used cars, trucks, sdbs, minivans, wranglers, half tons, three quarter tons, one tons, you name it, we got to go to the website right there, summitauto.com, full pictures and descriptions of every single vehicle all at summitauto.com and if you'd like to check out all the videos that I do, you can go to youtube.com slash summitauto, click the bell notifications, get updates on the videos I do each and every day as well as having access to one of the largest catalogs of vehicle walk-arounds on YouTube, almost 9,000 videos in counting, in fact in a second you'll see a link to subscribe to my YouTube channel on the upper left, a link to all the f250 videos I've ever done in the upper right, link to this vehicle on our website on the lower left and a link to one of our latest YouTube videos in the lower right, click those, check us out, we're super excited to help you with this ultraclean 2019 Ford F250, Lariat Ultimate FX4 in magnetic gray metallic, thanks again for checking out the video, remember to like, subscribe, and share on the YouTube channel, I really appreciate it, thanks again.


M2. The TCP, IP and Ethernet ethers in this way are created only one. This provides an advantage because the computational cost is almost independent of the payload length, but it depends strongly of the number of crosses. Instead, on the right, we can see the behavior when the TSO is not present. In this case, the TCP layer segments the data, and each packet must cross all level of the natural stack. The TSO helps the sender. Instead, the RSC, received side coalescing or hardware error, helps the receiver. The RSC allows an IC to combine in hardware, incoming TCP, IP packets from the same connection into one large received segment before passing it on operating system. It reduces CPU use because the TCP, IP stack is executed only once for a set of received Ethernet frames. Basically, the RSC is the dual of the TSO. This diagram shows on the left the RSC, how RSC works. The RSC allows the aggregation, and then it's passed to the operating system. In this way, only the aggregate system will cross the natural stack. Why do we need a software implementation? There are several reasons. For example, if we have an old NIC that supports only EP version 4, the hardware upgrade is really difficult. Or if the card has a malfunction in the floating system, it cannot be used. Or if we don't have the hardware in the communication, for example, between virtual machines, we need a software mechanism to do a segmentation. In all these cases, it's useful to have a software solution to allow us to obtain the same benefit. Finally, the software is easier to extend than hardware to support new protocol or to fix the bed. OK, since FreeBSD 7.1, we have a software implementation of RSC called LRO, this software mechanism that requires to change each device driver to obtain the same advantage of the RSC. On the lab there is a hardware RSC, and on the light there is a software LRO. In the device driver, in the same interrupt context, the packets from the same connection are aggregated in one segment, and then it's passed to the network stack. At this point, the behavior is the same to the LRO. There is only one big segment that crosses the stack. Now, I will introduce a software mechanism that we have developed for the center side. It's called GSO, Generic Segmentation of Load. Basically, it's the software implementation of the TSO, but it supports not only TCP, but also UDP on AP version 4 and version 6. The GSO is available for FreeBSD current, 10 stable and 9 stable. To avoiding changing every device driver, we have the segmentation just before the device driver. We can see it in the next slide. On the left there is a GSO scheme diagram, and on the right there is without GSO support. Much of the advantage of the hardware TSO comes from crossing the network stack only once. The GSO does the same by doing the segmentation in software as late as possible. Ideally, this could be done within the device driver, but that would require modification of drivers. A more convenient, similar, effective approach is to do the segmentation just before the packet is passed to the driver. This diagram shows an example of TCP's data flow. Our modifications to other GSO support are referred to the network stack. In particular, we changed TCP output, IP output, header output, and we added GSO dispatch function to do the segmentation just before the driver. TCP output, sorry for... TCP output checks if GSO is enabled. There is some CCTV to enable the GSO or to change some parameters that we will discuss later. And after that checks if the packet length exceeds the M2. In this case, our set... TCP output sets GSO flag in the Mbuff packet header field called CSOM flag. We used 4 bit in this field. Both to indicate that GSO is required and to specify the type of segmentation. In this way, GSO dispatch will avoid to inspect the header to understand the type of segmentation. Then the packet is passed to IP output. If GSO is enabled and required, the IP output avoids to calculate the checks, both TCP and IP, because they will be calculated after segmentation on each new packet. And it avoids to do IP fragmentation. At last, Ether output, after building the Ethernet header, if GSO is required, we call GSO dispatch instead of if transmitted. In GSO dispatch, using the information contained in the CSOM flag field, the appropriate function is invoked to perform the segmentation. We used a simple array of functions to do this. For example, in the case of TCP on IP version 4, the function is GSO underscore IP4 underscore TCP. This function, similar to the others, for example, to ODP or TCP on IP version 6, performs three main operations. One calls the mseg function. This function returns mbuff queue that contains the new segment. After that, number two, fix the TCP and IP ethers in each new segment, because these ethers are simply copied from original packet into a new segment. And then some new segments to the device driver. These are mseg function. This is the mseg function from the original packet to mbuff. In the mseg function, there is some parameter. The m0 is the original packet. Header LAN is the first byte in the m0 that are copied in each new segment. And MSS is the maximum segment size. After the segmentation, we need to fix the TCP IP either. In this case, TCP on IP version 4. The red fields must be changed. In more detail, for IP header, total LAN will contain the new packet sites. Identification will be increased. And we will recalculate the checksum of the IP header. For the TCP header, the sequence number will refer the data contained in the segment. Some flags are only set in the first segment, for example, CWR, and other flags only in the last segment, push and fin. And we will recalculate the TCP checksum if the hardware is not capable. For TCP on IP version 6, the changes in the TCP header is the same. In the IP header, there is only payload length that contains the length of this segment. As we say, the GSO also supports UDP traffic. In this case, we need to perform IP fragmentation, but we delayed this just before to call the device driver, as we can see in the intergeal monolab. On the right, if GSO is not enabled, the fragmentation is done in an IP level, and each fragment must cross the Ethernet level that adds the same header to all fragments. The steps that are performed with the IP fragmentation are the same that we already described. I'm sag, fix the header, and send the packet. The only difference is the IP header changes, because in this case, we must do an IP fragmentation, and modify the appropriate fields, like fragment offset, more field, and we calculate the Ether checks. For IP version 6 to do IP fragmentation, we need to add an additional header, fragmentation header in red, that contains the fragment information. To manate the GSO parameter, we add some CIS CTL. There are two CIS CTL to disable-enable GSO or TCP or UDP communication. For example, net.inat.tcp.gso, enable or disable the GSO on all TCP communication. And there is other two for each interface to limit the GSO bar, and to disable-enable GSO on individual interface. Our code is available in these two repositories. The first contains the pages, utilities, and the PicoBSD images, and the brief description of GSO. And the second one contains the free BSD source code with GSO support. To compile the kernel with GSO support, you just need to add option GSO in kernel config. In this slide, we can see the changes to add GSO support in free BSD current. We add the two files, GSO.c and GSO.h in CSNET, and a very small piece of code in the network stack files, like IP, TCP, and UDP. At the end, I show you the results that we have obtained in the experiments. This is our testing environment. We use NetPath as a benchmark tool and 10 gigabit links between the machines. To see the effect of GSO or TSO, the receiver must have a lot of software or hardware enabled. In this slide, we see the result with TCP, EP4 traffic. On the horizontal axis, we have the CPU frequency and on the vertical axis, we have the throughput. The red curve is obtained with hardware TSO. The blue curve is obtained with GSO software and the red curve is without any segmentation of loading. We perform an experiment by scaling the frequency of the CPU because, as you can see from the graph, at high frequency, 10 gigabit links becomes the bottleneck. From the graph, we can see at 2 gigahertz the GSO saturates the link and from the table at 1 giga and a half, the GSO is almost twice fast without offloading. In this other slide, we can see the results with TCP on EP version 6. There is the same TSO on the red curve is TSO, blue curve is GSO and green curve is without segmentation. Also in this case, GSO allow us to have a double throughput compared with without offloading. With UDP traffic, we have a speed up about 20%. This is because the GSO in this case only prevents the crossing of header output. UDP output and IP output are crossed only once, both GSO and both GSO is enabled or not. For DP on EP version 6 is the same. Footwork will try to do more performance measurements, for example, by using multiple concurrent stream, we try to optimize critical path and add support to new protocol, for example, as CTP. Thank you. Do you have any question? Is it possible to make the GSO option enabling by default, automatic, I mean? So if depending on whether segmentation offloading is supported by the hardware or not to make it the... to have it enabled automatically or not enabled? If they have a TSO, the network stack uses an hardware TSO. Or if you disable the TSO with the config, for example, and enable the GSO with the CCTL. No, I mean, can you make it so that the system selects whether to enable GSO or not depending on the hardware support of segmentation offloading? Oh, yeah, if you enable GSO, for example, in this way, also both in the NIC and in the stack, if the NIC didn't have the TSO, it's automatic. Aha, okay. Okay, thank you. Have you tested this feature together with packet filters, PF, for to see how networks address translation? No, we didn't test. Hello. Question is how much extra load does it add to the CPU? We didn't have this result, but the load of the CPU is not very high. We think it's not very high. But you said currently it's single threaded. Hi, I was just wondering why is GSO so much faster than non offloading in lower frequencies than higher, like in 1.5 GHz. It was twice as fast, but on 2 GHz it wasn't that fast. Do you have any idea? Because the bottleneck for example at 2 GHz, the bottleneck is the link, is the 10 gigabit link. The link is saturated. Yeah, okay, thanks. These tests, what M2 was used on the network then? 1.500. Okay. Any other questions? Let's thank our speaker then. Thank you.


YouTube, Instagram, Discord, Reddit, Twitch, Twitter, everything is SFanTV. Hope you guys like the video. Like it if you like it, leave a comment, subscribe to the channel, all that stuff, turn on your notifications, all my stuff is SFanTV, follow my Twitter if you want, join the Discord, everything else. So, thank you guys for watching. I'll see you guys later. All right.   Ah.