not only about evaluation, so this is about the intersection between design and evaluation. And so what do I mean by design? I guess one thing to note is that we're not talking about designing and evaluation. I know that's obvious, but I haven't had my view, we've got a little bit confused. So we're not talking about designing evaluation, we're actually talking about designing a program or a service or even an organization. So it's how evaluators can help actually design the substantial thing. And so there's a really a big sort of increase in the amount of interest in design or social innovation in Australia and in the world and there's been more and more of it going on. So it's a really exciting time for us as evaluators to understand how we interface with this other discipline and how they come together. And what we've seen, what Matt and I have seen are three main ways that evaluators are working with designers. And one of them is simply that there's some fantastic tools in design and co-design and helping people to understand how to really work out what the what the user needs are and design things that delight the user. And a lot of our M&E systems are measurement and evaluation systems or M&E. They don't always delight users so we can learn use these fantastic tools to help build better evaluation systems. So that's one use. The second use though is that we get involved in helping designers, evaluators get involved in helping designers build things by bringing all our tools and our skills of evaluation earlier on up front into the design process. Some people call it developmental evaluation but it can just be seen as evaluators working alongside designers. And the third way we're seeing it is some evaluators are becoming designers, they're actually abandoning their profession and becoming designers. So that's happening too, maybe not as much, but they're the three ways. So some people are learning how to do design skills or doing both. And some people on this call I know will already be in that camp who already play both roles. But they're the three roles that we've been seeing and that's what this special interest group is all about. It's about the intersection of design and evaluation. So that's just a little bit about what we mean. Matt, would you like to tell everybody a little bit about the special interest group? Yeah, absolutely. And like all good evaluators, I have some slides. So the one thing I did just want to point people to is the learning spring hashtag there in the top left corner, so DE 2020. So if you're the sort of person that likes to live tweet commentary and various things by all means get involved. What I just want to quickly comment on before we dive into the provocations was a little bit more about the special interest group itself and some of the things that we kind of do day to day. So one of those things, welcome. There you go. There's the welcome that boy. So one of the things that we do is we try to bring lots of different groups together. So, you know, not just evaluators but also designers. So some of the people speaking today don't come from the sort of evaluation heavy background. So we like to bring people together to explore this intersection, which is I think potentially a bit of a unique aspect to a special interest group compared to maybe some others. We also hold events like this. So previously we've also been involved in the 2018 AES conference. We had a bit of a voice there. The other thing that we like to do is we like to maintain a space to share thoughts, musings, links to resources and things like that, which I'll speak to in a second. And the other thing that Jess and I particularly conscious of is making sure that we respond to the interests of those who attend. So wanting to be really responsive to your interests as members or attendees of these sorts of events as well. So while we talk about membership, membership is kind of a loose thing because, you know, you don't have to be a member to attend something like this. But we do maintain a LinkedIn group. So you can see the name of it there, AES Design and Evaluation SIG. And we also have an AES email for the SIG as well. So if you've got queries about anything, you can send us an email there. But in particular, what we're trying to do through this sprint this week is trying to sort of identify where there are those resources or tools or things that you might know of that maybe others don't. And so wanting to try to share those a bit more widely. So if you're already a member of the SIG, then you can post them in there straight away. If you're not, you can request to join and we'll add you in and you can post things. Or if you don't use LinkedIn, or you just want to let us know that something exists, you can just send it to us by email and we can share it with others as well. So we're trying to be really collaborative with this kind of effort. The other thing that I just want to point out is that at the end of the sprint, we're going to practice what we preach. And so we want to reflect on this experience and improve. So on Friday, we'll send around a survey to everyone that registered to basically understand what worked or didn't. And what do you want to see more of in 2021 as we move into a new year and new opportunities as well. So that's all I wanted to just comment on. Jess, I'll just hand back to you for the introductions. Thanks. And I just wanted to say, I'm sorry if I didn't say already that we are recording this session.  But currently, we won't be seeing everybody, but that's something to note. And we will make it available after this. So something to note. Okay. I think that's all the all the stuff done, Matt. We might go straight into the provocations. So there are three provocations. We've got three speakers, and they're each going to speak for about 10 minutes. So I'll just introduce the panel to you. And then without a further ado, we'll go straight into it. And I've got a little bit of a cheeky question for each panelist as well, just so we can get to know them a little bit. So first of all, I'd like to introduce Ingrid Burkett. So Ingrid is a fantastic social innovator that I've known for a while, very creative. She's the first time I ever saw anybody develop a theory of change with foam cubes. Do you remember that Ingrid? Yes. So three dimensional theory of change was my strongest memory of working with you, Ingrid. And Ingrid is a professor and co-director of the UNIS Center at Griffith University. She's a social designer, designing process, products and knowledge that deepen social impact and facilitate social innovation. She's also a bit of an artist. But my question for you, Ingrid, is when you were a kid, what did you tell people you wanted to be when you grow up? So thank you, Jess. It's lovely to be here. And I'm joining you from the land of Yggenbeur and Yagura up in Logan in Queensland. And to answer your question, I always wanted to be a private investigator or a detective. And my family is convinced that I actually achieved my goal because as a social designer, I get to look into lives and try and make connections and investigate things that maybe wouldn't be obvious to everyone immediately. So I think I've made it. Well done, Ingrid. And it's a bit like being an evaluator as well, isn't it? Being a private investigator. Yes, absolutely. Now I did have some slides, but it doesn't seem to want to let me share those slides. So do you want me to just go into my provocation or are you going to question me, Jess? Well, I'm just going to introduce Shani and Chris first. And then you've got time to work out how to share your slides. And maybe we'll make sure, Michelle, that we can share slides in a few minutes. Second panelist is the amazing Chris Vanstone, who's the Chief Innovation Officer at the Australian Centre for Social Innovation. And I've had the pleasure of working with Chris a few times. We've worked together as designer and evaluator a number of times. What can I tell you about Chris? Well, he designed biscuits at one point. That's not in your bio, but this is trying to find a fun fact about Chris. He designed biscuits at one time, but also was the founding member of both the UK Design Council's red unit and the participle whose experiments using design to tackle social policy problems are chronicled in a book called Radical Help by Hilary Cotter, amazing book if you want to look at that. Would you tell us a bit about you, Chris, but importantly, same question, when you were a kid, what did you used to tell people you wanted to be when you grew up? Well, apparently, I used to want to be a human doctor in the morning and an animal doctor in the afternoon, both careers to which I'd be brilliantly ill-suited. Which you think you would have done better at animals or humans? I think there are probably a few reliable claims from the animals. But thank you, Chris. I'm sorry I didn't make it to be what you set out to be, but I think you made a fantastic contribution. Okay, and last but certainly not least, Shani Rajendra, senior consultant from ClearEyes and Consulting, so works alongside myself. And Shani, what can I tell you about Shani? Well, Shani is born in Sri Lanka and raised cross-culturally and brings a keen awareness of power and cultural relativity to her design and evaluation. Been working both with design and evaluation in steps in both worlds. And also, on the side, you're the strategy leader at Moral Fairground as well. Is that right, Shani? Yeah. So, same question, you know it's coming. What did you used to tell people you wanted to be when you grew up? Thanks for having me. I'm sitting on the line of the Roanju people, the co-ordination, so I'd just like to take a moment to pay my respect to their elders past, present and emerging. What do I want to be when I was growing up? So, for the life of me, I couldn't remember this one, so I took it to my mother, the go-to source for all things childhood. According to my mom, I used to want to be a teacher. How interested in people's learning journeys and helping educate or draw up people's awareness to what I felt was important, as I'm guessing a six-year-old. So, there you go. Well, there you go. Well, you certainly been, I think you are a bit of a natural teacher, so you may be doing it anyway, what do you reckon? Yeah. Maybe. Okay. Well, thanks so much. That was a brief introduction to our speakers, and now we're going to give them a chance to talk for 10 minutes. And we've encouraged them to be provocative, so we'll see what happens. One thing to note is I really welcome you all to leave, put little chats in their chat function, comments, reactions as they talk, and that'll be a great way to participate as we go. But we're now going to try, Ingrid, to share the screen. Let's see if we can do it. And if not, just talk. And the floor is yours. You have full permission, Ingrid. There's no permissions taken away, so you should be able to do it. Yeah, it tells me I have to close down Zoom before I can do it. So, I might just talk. I have sent my slides to Jess and Matt, but they're quite big, so I'm not sure whether you'll get them. But I can just talk, and I can send people slides afterwards. I guess my provocation, and it's mostly a provocation to myself, as someone who's, I guess, increasingly spanning the design world and the evaluation world in the space of systems change. And so my provocation is to be more humble. And it's a work in progress, and there's two sub-questions. One is, how powerful are we really in changing systems? These are questions to myself and all my fellow designers. And then the second one is, is complexity actually more important than systems thinking? And I'm saying this to myself as someone who, before this job was actually employed as a director of systems innovation. So I'm questioning myself, my own design work in the space of systems. And I guess I've got to that position because I've increasingly seen that systems thinking is something that's into design in a really big way. And as I see it, so much of the systems thinking is plagued by what I think is actually too much certainty. So we've got things like map the system as if there was a singular system, or we could map it. Find the levers that we can pull in order to create that magical change, move the needle, shift the dial, all of these activities that are really plagued by, I think, far too much certainty. So as I've been traveling this world of social innovation for the last, I'm not going to actually admit how long, because that would give away how old I am. Most of the work that I've been involved in is in this complex systems realm, where actually learning through experiments is the only way forward. And it's critical. And outcomes, as far as this work goes, are really only evident in retrospect. So evaluation is actually retrospectively looking. But in the case of really complex systems, the retrospect needs to be actually quite a long way in the distance. And for a long time, I didn't quite understand the magnitude of that. And it was only when I really started digging around in complexity thinking and complex systems that it made sense. I'm just going to read you a quote that would have been on my slides. But it's from Dave Snowden, who's a very grumpy old man. But he does have some really interesting things to say about complex systems. And a quote that really stuck with me was when working with complexity, we can understand why things happen only in retrospect. Instructive patterns can emerge if leaders conduct experiments that are safe to fail. That's why instead of attempting to impose a course of action, leaders must patiently allow the path forward to reveal itself. They need to probe first, then sense, then respond. And to me, that has become a little bit of a mantra in the way that I approach systems change and the struggles that I continue to have with it. So one of the things that we do at the UNIS Center is to have a look at how we can achieve really big systems challenges.  So I'm up to the next slide. Thank you. So one of the things that we do at the UNIS Center is have a look at how we can approach really big challenges like the SDGs and set them up as missions with lots of cross sector stakeholders, lots of impact projects, lots of experiments that all have a theory of change for what we now call an impact map. And in doing this, I have an internal struggle because I know that we are aiming for systems level change, but that we're not going to be able to see that systems level change unless we experiment a lot at that bottom level, the impact project. So we need to be able to zoom in to that level at the project level, experiment really rapidly and learn and get a sense. And I'll come back to this because that sense making and the strength of that sense I think is critical if we're really going to make systems change. Thanks, Matt, on to the next slide. Just to give you a bit of a sense of what I mean when I say I'm working in complex areas. A lot of the work that we're doing here is related to employment and employment might sound like it's a quite a simple area, but actually it's part of multiple systems and a lot of the people that we work with are people who've been long term unemployed. And so I've faced in the last few years just a constant struggle that my funders and also the people that I work with, they're very much into solid outcomes and answers. And so oftentimes the outcome that I'm told that people want is a job. And yet I know from spending lots of time with job seekers that they've got all of these complex things that they need to navigate, especially if they've been out of work for a very long time. Okay, next slide please. So what often happens is that it pushes us to start to pursue transactional outcomes. So as a designer, I'm often faced with things like this, you know, coming into a designer program which is supposed to be, which is funded to be an employment program for people who've been disengaged for over five years. And the board member says to me, I want to see this program achieve real outcomes. And at the end of the day, it's only jobs that count. And so, you know, we're pushed into designing things that actually are going to lead to transactional outcomes, person plus job equals outcome. Next please. So what we know also from work, exploring what the actual outcomes are for people is that that turns them into cyclical, precariously employed people, underemployed people. So people cycle between unemployment, precarious employment and underemployment. And figures that are starting to come out from most employment programs now are saying that between 40 and 60% of long-term unemployed people, when they enter these services, end up churning back through them within a 12-month period. Next please. What we'd like to do is try and design for transformational outcomes. So what we know from the signals is that that means we need to focus on quality jobs with living wages and pathways to address other in outcomes. Next please. And we also know that this quality is actually good for everyone. So it's good for the employers as well, because spending time with employers leads us to realise that their costs of constantly having to re-recruit people when the people that they have employed leave actually costs them in productivity and also financial costs. Yes please, Matt. So in my attempt to try and become much more humble in what we're designing for and what we're evaluating, I've come to think about how might we aim for right outputs. And I've become actually much more fascinated by outputs that will then lead us to right size outcomes. So I've moved right away from thinking that I can design straightaway for outcomes and much more interested in how to create the right outputs. What I'm also interested in is how I actually end up convincing funders and boards and stakeholders that aiming for outputs and smaller outcomes like increasing employability rather than heading for a job is more likely to lead to quality overall in outcomes and it's actually more honest and I can be much more honest with the people I'm working with if I focus on things like employability rather than jobs. Next please. I have to tell you Ingrid your 10 minutes is up. Okay. I'll be one second. So even employability though can be really hard to measure. So this is pre-release a little figure of how we're working with a social enterprise that's focused on employing really long-term disadvantaged job seekers. And what we're doing here is mapping the enterprise to see where in the enterprise they actually can spend time and what kinds of employability skills and then those the people who are actually working in there have helped us to design technology that can track people so that we can see what's going on and last but not least the last slide is what that's leading me to think about is how can we look for signals rather than silver bullets. So having outputs at experimental level and then clustering those outputs from within and across the experiments to give a sense of what the signal strength is about how likely those clusters of outputs are to lead to positive outcomes. That's it. Thank you. Thank you so much Ingrid that's filled me with lots of thoughts. Thank you so much. So going back to the humble output and being more humble. Okay with that I'm going to go straight to Shani. Thanks Jess and thanks Ingrid for an absolutely brilliant provocation. So if you're based in the East Coast like me you've probably been sitting at your computer for about six hours now and for that reason I decided not to bring slides with me you're going to have to make do with my face. I will be talking about power today. So when I was invited to speak on this panel I had to have to admit I thought it was really hard to speak about trends and design and evaluation without acknowledging the COVID size elephant in the room. In a lot of ways COVID has both disrupted and catalyzed existing trends it's highlighted what is and is working in our social systems at this point in time. In particular it's shed a lot of light on social connectivity and community engagement especially in the public health and health promotion space. For those of you based in Melbourne like me or in Victoria or around Australia to be honest it's hard to forget the lockdown of public housing estates in North Melbourne and Flamington earlier this year. That intervention was put forth as a much needed response to COVID however its execution has been widely criticised from the provision of culturally inappropriate food to the absence of translated information offered to residents at the outset of the lockdown. It's safe to say that no matter where you sit on the spectrum of politics that public housing lockdown has not been recognised and will probably not be recognised as an example of a well-designed intervention. For me it's also raised this recurring issue of power in our practice, power in design but also power in evaluation. Power is and should continue to be a key consideration for every good evaluator and designer. We must always acknowledge the role we play when we bring an evaluation or a design process to a community. We must also acknowledge the implications of our processes for those who engage with them. So this time last year I spoke at a design conference about how designers might meaningfully share power in co-design. The presentation was a co-facilitation between Clear Horizon and Flamington Works, a community-led programme supporting jobs and small business growth for young people and women living in the Flamington housing community. We'd spent a year designing their women's co-design project together, a project that sought to address barriers to accessing employment for migrant women living in Flamington. Now a year later I find myself questioning the very content of that presentation specifically the ability for designers and evaluators to truly share power in our process and so I guess in that regard similar to Ingrid this is very much a provocation for myself. What's made me so cynical you might ask? Other than spending the last six months locked in my house like every other Victorian it would be another co-design project that I've recently embarked upon. This project is situated in the drug and alcohol space and involves co-designing of the historically marginalised and criminalised cohort. Unlike the Flamington Works project in which conversations about power and the notion of power sharing came quite naturally to us as a design team this new project has been less straightforward. While we address power from the outset so in our plan and process by acknowledging each of our roles either as process designers, content knowledge experts or technical experts and while we co-developed clear ways of working that held community at the centre of our process tensions with power have continued to emerge and have honestly had implications for the delivery of our design process. The staff differences between each of these projects has caused me to reflect both on how we as evaluators and designers approach power in the work that we do but also what does this mean for how we respond to and design for in the face of unprecedented social challenges like COVID where we're at right now. So to begin with power is relational it is not something that's owned or given and therefore it really cannot be shared. Instead power is developed through relationships and interactions relationships themselves being built on generations of history and cultural nuance. So relationships in and of themselves don't form or change overnight. Relational power also drones on pre-existing power currencies. So by power currencies I'm referring to the value that others place in the assets that you bring to the table. An example of such a power currency is lived experience. Lived experience can help to build productive and positive relationships by creating shared realities that we can work within. So when I talk about those shared realities they're not exclusive to design they're very much apparent in evaluation as well. If I go back to my example of Flemington works my lived experience as a first generation migrant and a woman of colour made my power as a process facilitator more palatable to the women that I was there to co-design with. Rather than sharing power they were gracious and accepting of the power that I held as a process facilitator in the very same way that I was accepting and celebrated the power that they held as our content knowledge experts in that particular project. However lived experience cannot be tokenised either you have it or you don't. And in the context of my most recent project my lack of lived experience in the drug and alcohol space did not thought well for my role as a process facilitator. I lack the same footholds to build the relationships that I was able to build in Flemington works project. So in this situation addressing power required a deconstruction of power in and of itself. It meant that I had to share some of my power currencies including to some extent my whole of my design process. Why might that be? Well there is power in process as a power that we all bring to the table as evaluated and designers irrespective of who we design with and where our values lie. What does this tell us then about addressing power? Well addressing power through relational mechanisms is only as good as the power currencies that you bring to the table and the power currencies that are held by others at that table. Relevant lived experience can make all the difference by creating the shared reality and neutralising all the various currencies that exist in each and every one of our banks. The second point I wanted to make was about how we actually address power in practice. So if power is relational and that indicates that we cannot share power what do we actually do when we say that we correct the power or that we share power in our evaluation and our design. Again I'm going to reflect internally and I'm going to look back at the Flemington Women's Project. In that particular project we invested our efforts into addressing power at two levels. So the first level was between Clay Horizons, Flemington Works and nine project support officers who were recruited from the community in the Flemington Housing Estate to deliver this project with us. The second level of power was between the project support officers and the broader Flemington community, recognising that nine individuals cannot represent the diverse wide-ranging views of a community rich in different ethnic and cultural backgrounds. So between Clay Horizon and Flemington Works, we co-developed practice principles to create a shared culture of working amongst ourselves and between us. We also created easy to use discovery tools so that the project support officers could conduct their own social research and replace traditional data synthesis and analysis mechanisms with visual methods and storytelling. Between the project support officers and the broader community, we embedded community feedback mechanisms so we had regular feedback loops to ensure that community voice could be fed back into a process along the entire process and make sure that we were better represented community in the work that we did. However, in doing so, in implying these methods that I've just discussed, what we in fact did was we framed power sharing through the lens of accessibility, improving opportunities at every turn for community to engage in a pre-existing human-centered design process. In reality, what we've done is substituted power with accessibility, where accessibility is just one aspect of power in all its rich complexity. Accessibility can improve the ability to engage in a process, but it does not guarantee the ability to exert influence on that same process. Where you have mechanisms in place to enable access to a process and to influence the process, such as having feedback mechanisms and practice principles like I described, a person can still only really influence the process as far as they understand the theory that underpins the work. Theory, which is often rooted in Western schools of thinking, and therefore are not universally understood. So by improving accessibility, we might be able to co-design an outcome, but the possibility of co-designing the process itself is limited at best. As such, accessibility has little to contribute to the dissemination of power. So thinking forward to a post-COVID future of rebuilding social connectivity and improving community engagement, to me personally, two things are very clear. The first is that power is relational. It cannot be shared, rather it's addressed through building meaningful relationships. And nothing builds relationships better than shared realities. To do so, you need genuine and relevant lived experience. The second is that as evaluators and as designers, we're at risk of conflating accessibility and power. They are far from the same thing. I just want to end this provocation by saying, like COVID, I'm just highlighting what is and working in our system. I was once told that the biggest barrier to change is the fear of defending and being told that you're doing a saying the wrong thing. To make change, we need to acknowledge that we're likely to continue saying and doing the wrong thing, even if we have the best of intentions. The only way forward really is to keep reflecting and adapting how we do what we do. Thank you so much, Charlie, that was really powerful. And I can feel the threads of humility coming through both presentations today, very reflective of your own practice. And it's hard to say the right thing too, after something like that. But I'm just going to keep going. I'm just going to keep going and invite Chris and then we're going to pull it all together. And I would say to Ingrid and Chris and Shani that you will have a chance to ask each other questions at the end of this as well. I will let you connect with each other. But Chris, over to you, Chris van Stone. Thank you, thank you. I thought I might spend the first few minutes of this reflecting on quietly on what Shani just shared with us. Thank you, Shani. Thank you, Ingrid. That's really rich. Coming to you here from Unceded Garnerland in Adelaide, I want to pay my respects to Garner Elders past, present and emerging. I work at an organisation called the Australian Centre for Social Innovation. And something we believe a lot in there is people-powered innovation. So really following on that perhaps less eloquently from what Shani's team of power. And we support a lot of different kinds of organisations to run participatory processes or projects or in some cases design 10-year long initiatives that are in some way, in many ways, participatory. So for example, we've recently been working with people who have just come out of prison on what used to be called a forensic mental health service. This is with Ballarat's community health. And creating a space for mental health professionals and people with that lived experience to come together and negotiate redesigns around that service, including changing its name from something that isn't forensic mental health service. Sometimes it's working, very often we're working with government. Sometimes that's around in more of a sort of commissioning space. So for example, we recently worked with families and people living with dementia to design a new living facility for people with severe dementia with the South Australian government and to design the specification for that. And we've recently been working with a number of towns that were affected in the summer bushfires to design a long-term initiative that would be community led. But also interestingly, the initiative itself was the whole governance over that initiative that's philanthropically funded is participatory. So it's participatory, it's a number of levels with a majority community board in that case, making a community member more of making decisions about the sort of shape and change of that. As Engrid was saying, they hopefully learn from those experiments over time and change the shape of that whole initiative year by year. So we did this because we think it can work really well, but as Shani's very well described, power is a big thing in this and this is sometimes hard. It can be hard and it's particularly clear when you're working with institutions where there are hierarchies and folks that used to making decisions and ministers or chief executives or folks like that are used to having final say or final tweak on something. So whilst we've had, you know, we're proud of lots of the work that we've done. We're also gutted about some pieces of work we've done that started off at one level in a very participatory way with a beautiful participatory way to go forward. But that was sort of ignored or if you were being generous, it was lost in reorganizations of government as it shifted between departments. And now they might be doing what they would have done anyway. And so I was wondering evaluators if you could help us with this. Because a lot of this power stuff is complex, as Shani was describing, and it is somewhat invisible or at least invisible to those who might be doing power over very often. But it is something that's happening in the moment, although sometimes that's behind closed doors, or sometimes that happens after the project is finished. And my simple provocation is, could we create ways of making this visible to all of us in real time? Could we get a little tap on our Apple Watch when we're overstepping the mark on power? Could we, and listening to Shani's thinking there, could we make it really explicit what aspects of power we might be sharing in a particular project or process? And what aspects we're not? And what might be the next step for us to learn how to grow into as a system, as an organization, as an individual? And could evaluators, acknowledging what Ingrid was saying about the need, you can only see what works a long time afterwards in this work, but could there be some real timish feedback on the power dynamic in particular projects? And could we really bring that to the fore so that we can reflect on what we're all doing or not doing and what we're helpfully doing and helpfully doing as we go through the work in the moment? That is it. I've held a challenger as Chris, I love it. So my head is buzzing and I'm sure people who are listening must be feeling the same with all of the things we've heard. I just like to thank the speakers for their time, but I'm also going to thank them properly at the end, but I'd really like to try and pull the threads together a bit, but I'm going to ask you to help me, everybody, everybody. So this is, we're going to do what's called a slam chat, right? What are you hearing? What common things are you hearing across the three presentations? If you could all type and I'll press together. Yeah, so don't press. Don't worry if you get it wrong. Just type something, chat slam. Yep. We're going to type something in. What commonalities do you hear across these three provocations? What do you hear? So tap, tap, tap, and then we're going to press together. I'm going to say some are coming through already. Go now then. Let's just go with the flow. Press go. Lots of, let's, oh wow, this is, yeah. So a lot about complexity, lift experience being talked about a lot in this, in this session. Power, power dynamics, complexity, relationships, difficulty of making judgment, power, power, complexity, power is complex. Ah, who said that? That brings it all together, doesn't it? Power is complex. Yeah, humility. Yeah, power is complex and we need to be, have, have, be humble about how we address it. And I love the idea of having a little watched. I've got that, you know that thing that people advertise at the moment where it stops you from slouching? I feel like maybe we need one of those like, you know, whenever we realize we're, we're abusing power. But now we're going to move into the final session here, which we've just got 10 minutes left for questions. But while you're thinking of your questions for the panel, I'm wondering whether the panel members have got any questions for each other. So I'll invite first of all the panel members to say if you wanted to ask any, each other anything while we collect questions from the floor. So if you're an audience member, feel free to type your, your question into the cat chat function.  Do you have a question for each other? I have a question for Shana, actually. Yeah, go. I wonder, Shana, if you might just say a little bit more about what you saw as some of the people, the aspects or dimensions of power, accessibility being just one of them, what might be some of the other ones on your list? Yeah, look, thanks, Chris. It's a really good question. And I think it's something I've been grappling with because as I pointed out a year ago, I thought power sharing, I thought very differently of power sharing is what I do now. So I think accessibility is important. And this might sound a bit close to accessibility, but I think information is important to, and you can argue that accessibility provides people with access to information. But I think that that isn't necessarily always the case, because you're coming, if you are designing accessibility, you're designing a pathway to enter into your particular process. You're, you know, somebody recently used the analogy of roads. You're creating a shorter path. You're digging a shortcut through the grass to get to that next destination, as opposed to actually building that trail together. So I think information can change a lot of things. And I noticed a comment from Victoria as well about where the lived experience needs to be a precondition for proper genuine power, sharing or whatever you want to call it. And I would argue that it's not, it's about putting everyone's cards on the table and truly understanding what power currencies are at play, and then the implications of those power currencies for a process, which ties into your comment around evaluators, what can you do with power? And it's about evaluating power. Understanding what everyone's bringing to the table, but then what that means moving forward in a process as well. It's acknowledging that we can't share power in the way that we'd like to, and that happens organically and intrinsically as a result of the way we interact with one another, as opposed to our intention entering into that certain space. I don't need to answer your question, Chris, but that's sort of where my thinking is at. Thank you. Thanks so much, Shani. And did you have a question for anybody in grid? Yeah, I'm going to go for Shani as well. I hope that's okay, Shani.  I wonder if you have thought about what you would do differently in the next co-design project that you get involved in, or has it really turned you off this idea of co-design? Because I think power is something that is often discussed in co-design. And yeah, it does turn some people away from the process. Yeah, great question, Ingrid. And I think I should start by pressing that by saying I'm very much one of the people that just described who's an evaluator slowly turning into a designer. So I'm trying to have the best of both worlds. But I'd start by saying I've definitely not been turned off of co-design or design as a result of power at play. I think for me, I am often concerned about buzzwords. My background was originally called for sustainability and sustainability more generally. And if that isn't a buzzword, I don't know what is. So I have genuine fears about co-design becoming that next buzzword. And while I'm not elitist about jargon or language in any sense of the word, I do recognize that there is a danger to framing both evaluation and design as a co-develop process if it is in fact part of the Patreon nature. So it's about calling a state of state as opposed to anything else. I definitely think that co-design is something we should and will continue to work towards. But it's about having that reflective practice of really embedded into what we do, both in that room and then a year later, and being able to look at it and work out what we need to learn from and what we need to develop. That's great. Thank you. And I think we need more evaluators in the design space. So bring it on, Shani. And I love the idea that we should have a go at looking at how power plays out as evaluators. Don't you as well that provocation that Chris has given us or invitation maybe even when we're evaluating a design? That's another way evaluators and designers can work together is where you're actually evaluating the design process, which is I think it's an underrated beastly process design in evaluation. And I know that it can bring a lot of, you can de-risk design a bit, which is governments really like the idea of this, that you've got an evaluator there to look at what's going on and reveal. But one of the things we should really be looking at is power and how we do that well is something maybe a beautiful idea from Chris. I wouldn't mind having a look at some of the questions that have come from the chat function. Matt, is there something you'd like to pull up or something that you'd like to ask? Yeah, there was one from Deborah just earlier to Ingrid just on the concept of outputs that you mentioned and focusing on outputs. If you could just speak to that a little bit more. Yeah, sure. So I think, I mean, I have a fairly traditional view of what outputs are about. They're easily measurable immediate consequences of what's happened. And I guess from a design perspective, what I've noticed is that we're not very ambitious with our outputs and the way that we structure outputs. And we could actually be more, I'm going to, more singier, which is not a word, but I'm German. So I'm allowed to make up words. So for example, if I think about, you know, that last piece of work that we were involved in, the outputs that government funders were wanting was how many people showed up to the employment service?   What does it actually tell us? So how can we get crunchier about those outputs? And one of the ways that we thought was, okay, well, if we knew where people spend time when they actually get to the service, and that's why we did that, you know, the mapping of where people actually spend time. And then we linked that to what are they actually doing? And what's their interpretation of how that builds capabilities that could lead to employability, that cluster of who's turning off how often, where are they turning up? And what are they doing? That cluster could give us strong signals about whether or not they are, you know, moving into greater levels of employability. And then if we add their own voice into that, to say, we know, what did you get out of today? And where you spent time, we've got like four outputs that together could give us a stronger signal about whether or not we're designing something that's leading towards employability. Your challenges are on what we evaluate because outputs, of course, are used as a measure of success. So the very measures of success need to be challenged in complex settings. Yes, absolutely. Yeah. We are really running out of time here. And I think it's been a fantastic start to the design and evaluation sprint. And I really, really would like to thank our panelists for giving their time today and their thoughts. So thank you so much to Ingrid, Chris and Shani. And maybe a little clap for everybody for giving them a time. And just so thank you so much. And before we dive off, we've just got a few words from Matt about the rest of the week and about the SIG. SIG, if you'd just like to stay on for a couple more minutes. And thank you, everybody, for participating today. Over to you, Matt. Great. Thanks, Jess. So just in terms of what's on for the rest of the week. So tomorrow we've got a session with a co-design facilitator, Christian Jewel, who's going to be running us through some of the processes that he uses to grapple with exactly what some people have been talking about. How do you practically deal with situations of power dynamics? And how do you engage people in that conversation? So he's going to step people through a bit of a process and a toolkit that he uses. So a bit more of a hands-on approach. On day three of our session, Jess, Chris and Shani, who you've all heard from today, are going to be running through a bit of a discussion around designers and evaluators working together for scale. So getting into that question of scaling and scalability. And then on day four, lucky last, I'll be running through some of these concepts that Ingrid's gone through about systems and systems thinking. But in particular, again, trying to sort of keep a balance between really insightful conversations and some tools or processes. So stepping through a bit of an exploration of mapping systems. But what that means actually from a people perspective rather than conceptually or conceptual mind mapping. So again, a bit of a practical application. And then at the end of the week, like I said, we'll have a survey that goes out. Basically, want to hear from you what you got out of it so that we know what we can do better for next time. And then ideas for next year as well. Yeah, I think besides that, I think we can call it quits. Oh, look at that. 30 seconds early. You can leave your, oh, yes, what I'll do, someone's just focus, I'll put the link to the SIG just in the chat, just while people are kind of starting to leave. But if you want to leave your comments or thoughts in the chat, you can. But otherwise, you will be free to go. Sorry, Kara, I'm specifically doing this for you. Come and join us for the rest of the week if you're not already registered and have a great afternoon. Bye bye everybody. Thanks to you very much. Thank you so much. Thank you.


So I know I'm starting a minute or two early, but I figure if you're hanging around this long to keep listening at app developer con Thank you Thanks very much for making this a really awesome and exciting day where I don't think I've been to a cube con day where most of the focus has been on not here's this cool thing that I built look how it works, but instead on here's how Other people will build on top of this cloud native stuff that we have to produce stuff that changes the world and so I'm gonna try to get through this so that y'all can get out of here and go see the friends you've been waiting to see or catch the lightning talks or other stuff like that First of all, I'd like to thank Diagrid and red hat for sponsoring this and making it possible to get all of you here We we can't do it without you, but we also couldn't have done it without them And uh So I was alluding to this earlier. So Software has this hierarchy of needs. You need some hardware to run it on For anything interesting these days, you need some way to connect to it Um, which probably means some networking or something like that Um, again these days you want to be able to do stuff across multiple computers It's stuff that only happens on one computer is mostly not interesting Anymore even in your car Um, so we need some kind of orchestration some way to make sure that stuff is running in the right places Then we need to figure out how do we actually get that software? We wrote to the place where it needs to be that's rollout and then on top of that We've got some application logic some stuff that you know, we wanted to actually have happened So, um That's our hierarchy of software needs Um, and we talk a lot here at kubecon As dev ops and it security practitioners and platform and infrastructure developers About how do we get that in those middle layers of the cake so we can get up to those top layers where applications get developed um And you'll notice that there's a little Top or on top of the cake too, which is all of the users of the software that we're hopefully building for um and If we think about it the actual stack of humans involved doesn't look Like this mountain it looks the other way So there's a few folks You know your cloud providers some of the folks in your itc and your data centers who are actually physically connecting That hardware working on the linux kernel and things like that more than that there's people building that next layer above it building those kubernetes and those ciliums and um The ui's that you use to get resources from your cloud provider and all of that stuff And then there's even more folks who are using that stuff Who are at your companies provisioning the it for you We talked about remote developer environments and how you know, maybe you need a kind of big kubernetes clusters So you could put a whole bunch of little tiny short live development environments on there So there's more of those folks Then there are building the platforms and infrastructure and that's good to be clear Um, and there's more app developers that they're supporting So this is fanning out the changes from the hardware are fanning out to the whole world um And layers that we don't talk about much here, but there was someone I was talking to earlier who's like I'm here, but I'm sorry. I'm not a techie. I'm a product manager and I'm like Product managers are techies if you are in this industry if you know what kubekan is You're a techie But there's a huge number of people Who aren't necessarily writing the application code, but who are doing that next level of making it useful And most of our value is how this software affects the rest of the world, which is why app developer con Seems like such an important thing to add to kubekan Because I don't care how cool your kubernetes operator is I care, you know, hey this hospital was able to roll out new Imaging software faster or protect patients privacy better because they adopted kubernetes and they used your tool to do it Those are the outcomes that we should be thinking about And it's too hard right now How do we get it so that everyone Just thinks of cloud native as the default um, a lot of people they're like I'd love to use this kubernetes stuff, but it's so hard to get a kubernetes And if you've been here a while you're like, I can totally get a kubernetes. It's so easy um Go pick a provider that you're not familiar with and You know try standing things up there It's too hard and it changes we keep making it better And that better Breaks all the stuff before so if you are a platform person here right now. I'm talking to you We need to be better about supporting those next layers those app developers Those business analysts those folks who are trying to figure out. Did we make enough sales? You know of our stuff that you know, everyone gets bonuses this quarter their it users as well um and students and hobbyists it needs to be easier for all those people Who don't come to kubecon and it needs to be easier than your local machine um So we'll be doing this again in europe and hopefully we will have a bunch of cool new things to make containers cloud native services microservices ephemeral deployments ci and all that stuff less painful And if you think you've got some content That you think everyone else should know how awesome this stuff is Um, cfps are open. Um, they're open for another month or so So get thinking put your thinking cap on if you think you can convince your company to Send you to europe to talk, you know, hey Here's your opportunity And You can scan the qr code for more information And If you're still here, thank you. Um, if you could either the bit.ly link or this We've got a short survey for questions, but it'll help us figure out what's on the agenda For that next app developer con tell us, you know, I really loved this talk or you know I'd love to have seen more stuff That was libraries and cool stuff that builds on top of all these cloud native technologies Whatever it is that floats your boat Um, and then just a couple more administrative things Um, we're here at kubecon. We're under the cncf code of conduct Be kind to each other And there is a co-located event reception in the foyer tonight.


Hi, my name is Vicente Uchurbaro, and I am a software engineer at the MariaDB Foundation. Today, I'm going to give you a introduction into two advanced querying features, which are window functions and commentable expressions. Both of them are coming in the new MariaDB 10.2. Okay, and I'm not going to give you a full description of what we support, but my idea is to try and show you what they are capable of and give you a few examples of how things work. So we're going to go through both of them, give you some examples, show you why things work a lot faster with commentable expressions and window functions, and I'm also going to give you a short detail on what we support. So I'm starting with CTEs. I'm starting with a simple example. Here we have a table of engineers. So this highlighted in red is the false commentable expression definition. We start with the keyword with, this is how you identify commentable expressions, then you have the commentable expression name, followed by the body, which defines what you will find in that table definition. Afterwards, we can use it as if the table were in our database. A similar solution can be done without CTEs using derived tables. The two queries are identical. The difference is that you have the table definition in line within the query. This gets complicated as you add more tables, more derived tables. If you look on the right, we have a nested view of how two derived tables would look like if they were to reference each other. However, with CTEs, we can define this one at a time. So here we first define engineers, and then we want to select all the engineers that are from Europe. So we define another table expression, EU engineers, which references the first table. You can see it here as engineers. This makes things a lot more maintainable, and I'm not sure if you have had the pleasure of working with 20 nested loops, but things get out of hand pretty quickly. So one use case example is say you are part of the sales division, and you want to see which products have performed better this year as opposed to last year. We can do this by defining first a table expression with all the products and all the sales for that product, grouped by ear, obviously, and then we can just join this table with itself. See here that we are referencing the common table twice, and this will give you all the products which have performed better than the previous year based on the amount of sales. Doing this without the common table expressions would either require you to use a view or to define the derived table twice, which is not scalable. So in summary, CTEs are defined with the with clause. That's how you're going to see them in the query. They are similar to derived tables, however, they provide more advantages. There are also extensions to allow recursive definitions. I'm not going to go into details on those. Obviously, they can provide you with cleaner code, and sometimes the optimizer can better understand what you actually want from the data, and thus optimize your queries to produce more efficient query plans. Now, the second feature pairs very well with common table expressions. We're going to see this shortly. Window functions are another advanced topic, and I've seen a lot of people not really fully understand how they work. So that's what I'm trying to show you here. Give you a brief understanding of how you should think about them, and how you should think about writing queries using window functions. So window functions are special because they behave like two different things. One, they act like similar aggregate functions, and that they compute their result based on multiple rows, like sum and average. But at the same time, they do not collapse your rows. So you get one result per each entry in the select statement. You can obviously identify them with the over clause, which comes right after the function name. Let's start with a simple example. So let's say we have this table, which represents a set of users. We have email, first name, last name, and we also specify what kind of a user it is. In this case, either admins or regular users. If you would like to create a primary key for this table, say you forgot to define one as an integer, you would want to get, say, an auto increment column. Well, you can get this by using the row number function. The function does what you'd expect. It sets a number that increases for each row in the select statement. Now, the problem here is that we have not defined an order for this window function. The order here that you see, one, two, three, four, five, is not deterministic. These rows can actually come up in this order, two, one, two, one, three, five, four. At the same time, you could get a reverse result. Now, the reason why this is is that window functions are computed after everything else except the order clause. So the window function can get these rows in any order unless we specify differently. That comes as the form of a parameter for the over clause. We tell them that the window function should receive the rows in this order, alphabetically by email. Now, a second question comes to mind. What if we want two separate sequences of numbers, one for regular users and one for admin users? This can be done using an extra parameter to the over clause called partition by. It does what you'd expect. It splits the data up based on the column defined in the partition statement. So here, our rows are split into users that are regular users and administrators. So you see we have two sequences, both starting from one, one for the admins and one for the regular users. Now, I also mentioned that window functions behave like aggregate functions and that they are computed over a set of rows. Let's look at this example. Say we have some sort of sensor that picks up data from time to time and it puts that data into a table. We have the time stamp and the value that the sensor recorded. If we order it by time and plot it, we might get some sort of graph that resembles this one. We don't like the data because it's noisy and we would like to smooth it out. We can do this by averaging across a set of values. So we want the point that's represented to be an average of, say, a couple of points before it and a couple of points after it. And we can do this using the average function. Now we've transformed the regular aggregate average that we would use in a regular SQL into a window function using the over clause. And now the function behaves slightly differently. We first need to order the data by time because that's how we want things to be averaged over. Then we need to tell it that we want it to average stuff with a few rows before and a few rows after. And we do this with another special syntax called a frame definition. Here we tell it to average three rows before it and three rows after the current value. And now this is an average of seven values. We can increase the window that the average is computed over by specifying different numbers. They do not have to be the same. They can even be references to other columns. So it's a bit more complicated, but this is in essence how you can define the set of rows that the window function is computed over. If I'm to go through a step-by-step how this frame moves, I've taken this, the previous example, have replaced average with some just so that I don't have to divide everything by a certain number. But the idea is the same. So on the left we average, we sum three rows, the previous, the current, and the next row. And the other one does it with five rows. In blue, I've highlighted which rows are used in the computation, and the red is the current row that we're computing. For the first one, we do not have any previous rows, so we just take the current row and the row after it. This gives you two plus five in the first column and two plus five plus four in the second one. As we move forward, the frame advances. Now we do have previous rows, so we add those as well, and we add another row as the next row has moved as well. If we move even more, the frame now starts to remove rows from the result and adds new ones as they come in. Notice that now even the second column starts to lose rows because we are advancing the window function even more. Computing this is actually pretty easy if you look at how values get added and removed. So for each new row that we get, we get to keep a set of rows, remove some, and then add some. This is the key to why window functions behave very well for certain data sets. Because each computation is constant time. We just remove one thing and add another. So what we call it is online computation, and it works for almost all window functions that we support. There are a few exceptions, but I'm not going to go into detail on that right now. This is how the end result would look like. So now I'm going to give you two examples from real life applications that I actually had to solve. Well, slightly condensed, but it's the same idea. So say you're in a bank and you have a lot of transactions from users. Now we have a timestamp for transaction. We have the customer ID and the amount that the transaction was. Either deposit some money or pay some money to a different account. And we want to get a balance statement for all the customers in the bank. So the balance statement can be computed with regular SQL by doing a subquery where we sum up all the transactions before it. Okay, so here we joined the table with itself. And by joining we add all the transactions that have the same customer ID as seen here, but have a timestamp that is lesser than the one that we are currently computing. So this is done with regular SQL. With window functions this becomes a lot easier to define because we just need to think about our computation. So what do we want to do? We want to sum up things. Then we want to split our data by customer because we want a separate balance statement for each customer. We order our data by the timestamp. And then finally we want to tell the sum function that it has to add all rows from the beginning up to the current row. So here this is where unbounded proceeding and current row syntax comes into play. Now why is this important? Well let's look at the performance of this implementation. So if you were to write this query with a regular SQL, on the left you have the number of rows used in this test and the time it took for the query to execute. With regular SQL, without using an index, the time increases quite quickly. And past a certain amount this transaction takes quite a long time to finish. If you do have an index, things improve, but compared to window functions the speedup is 10x. So we've managed to define a query which no longer uses self-join, so you don't have to do n-squared operations. Instead it scales up in linear time with the data set. So this is how you can get quite a huge improvement by just rewriting your queries using window functions. Another use case is using top n queries. When you want to get the top n people from a certain data set. You can do this both with a regular SQL by just counting them. The expression is a bit complicated but you can just count them. Unfortunately if you want to get a rank column for them you will have to define the query again. Which makes the whole query look like a mess. But with window functions we have a very interesting solution which is the rank function which does just this. First we just partitioned the data by department and then we rank it in order by salary decreasing for this example. Now I said we wanted the top five earners so we want to filter this. Well you cannot filter it by using a window function in the work clause because window functions are computed after the work clause is evaluated. So this is where CTEs come in. With CTEs you can wrap the window function query and then filter that one outside of the window function query. Let's look at the performance for this one. The reason I put dot dot dot there is because it never finishes. At least not until a battery dies on this laptop. So even with a very small data set getting the top n is very difficult with regular SQL. However it is instantaneous with window functions. So whenever you have these kinds of queries making use of window functions can greatly improve your application's performance. So in summary window functions do help you eliminate self joins like we've seen with both examples. At the same time they make queries more readable no more duplicating of sub queries everywhere.  Here's a set of functions we do support. This is almost an exhaustive list of what the standard suggests that we do. We do have a few exceptions but these are most of them. All right. Thank you very much for listening and if you have any questions. Yes. Yes. Okay. So the question is can you use can you have user defined functions used as window functions. And the question is not yet but we actually do have this planned and it will most likely come up in the next release. We have we actually have a Google Summer of Code student who's contributed a patch which allows user defined functions to be used as aggregate functions. And then getting aggregate functions into window functions is a trivial step. Yes. Okay. So I'm not quite sure I understand your statement. Sorry. Okay. What what what what I'm what I'm saying is that is that we have have user defined functions as SQL functions as procedures like you would define a. Yes. That's what I'm what I said that we have planned for. So we want you to allow to allow you to use regular SQL to define your own functions as window functions. All right. Well if there are no more questions. Thank you very much.


The National Election Petition Tribunal sitting in Maccordie, Benway State has nullified the election of the senator representing Benway-Nordis Senatorial District Emmanuel Ndende of the All Progressives Congress and declared Gabriel Soswan as the after People's Democratic Party as the winner. The three-man panel of justices presided over by Justice Ari Zick Ikeora while addressing while delivering judgment in a petition filed by Gabriel Soswan in Maccordie Justice Hill that Soswan has proved the allegations of overvoting during the election beyond reasonable doubt. Reacting to the judgements, the lead counsel to Senator Soswan, Barrister Donald Ketu, described it as sound, saying justice has come to stay and called on Ndende to embrace his opponent to serve Benway East people well. It is something that is no more when you come to court. A party is bound to win. One party is bound to lose. But the good thing about losing in a matter is that there are pellet divisions of the court appeal from the High Court to the Court Appeal to the Supreme Court. So it is not exhaustive. Yes, we've lost here but at least we still have an opportunity to also look at the judgment and express ourselves. Well, the next thing I was going to say, certainly as a lawyer, I am not satisfied with the judgment. I will advise my clients, let's look at it thoroughly and we'll take a final action to it. But I'm not satisfied. In this matter that was elected, votes that were not collected or were cancelled were not valid for the first respondent. So they were supposed to be collected in favour of the first respondent, the first petitioner in the matter. And because it is a first petitioner in the matter, Gabriel Tuwaz-Ustam, that was validly elected, he is supposed to have been returned by the third respondent, Ainek, as the person that was validly elected. So the court has ordered that in that regard, the certificate of return that was issued to the first respondent in this petition, Emmanuel Menzo Dende, should be cancelled and a fresh return certificate be issued to the first petitioner, Gabriel Tuwaz-Ustam. So the position of the tribunal is that the documents that were attended before the tribunal speaks for themselves. If so, look at them.


Climate change has led to increased awareness of energy efficiency which has resulted in new trends in electricity markets such as the obsolescence of the traditional electricity model and the gradual conversion of consumers to presumer profiles. To address this shift, the introduction of multiple distributed generation and storage resources with a strong involvement of renewable energies requires advanced metering or smart metering systems which must be able to manage and control these distributed resources. These systems have been developed using a variety of technologies and standards each with its own set of features. This paper reviews the key elements of a smart metering system including the technologies and standards used and provides an overview of the deployment of smart metering systems around the world. This article was authored by Noelia Uribe Perez, Luis Hernandez, David de la Vega and others.


Well, it's still early in the high school volleyball season, but for the Walker Hackensack Ackley-Wolves, they couldn't have asked for a much better start. 2-0 with wins over Sabika in Hill City, a very veteran group consisting of five seniors and six starters returning from last year, and they credit 2016 as a huge learning experience, but now they're a year older and ready to make some noise. We've been working together all summer, getting back in the gym, doing stuff with coach. We did a Summer League in Park Rapids. That went fairly well. We were just used to playing with each other. We didn't have to change anything in our rotations. We actually added one senior. Even the younger girls on the team have been with us that long, so we've been playing together for a long time, and it's great not to have anybody leave that. They also look to lean on their offensive weapons, including six-foot sisters Katie and Megan Benjamin to lead a very balanced group of attackers. We're pretty strong at the net. We have two middles that are sisters, and they're pretty tall. So yeah, we get a lot of touches on the ball.  It's been great. If you've enjoyed this segment of Lakeland News, please consider making a tax-deductible contribution to Lakeland Public Television.