So hopefully you're at the right session. This is Design and Evaluation on the Edge supported by the AES Design and Evaluation Special Interest Group. This is part four of a four part series on power, failure and washing machines. Before we start today, I'd like to acknowledge country and the traditional custodians and owners of the many Aboriginal lands we're meeting on today. I'm a more injury country here in Melbourne and I'd like to pay my respect to Elders past and present any Aboriginal and Torres Strait Islander colleagues with us on the call. You're very welcome and thank you for joining us today. My name is Nick Vogelpol for anyone who doesn't know me and we are going to be talking about design, learning and evaluation in multi-stakeholder collaborations today. So if that's not the session you thought you were at, this is a good time to leave, but if you are interested just by the title, please stay. We are at the end of today's session. We're going to be joined by my colleague, Jess Dart, who is going to take us through a bit of feedback and planning for the next kind of design and learning sprint.  If you haven't had an opportunity to catch up with the other sessions this week, they're all available to be watched online. My colleagues Matt, Joe and Shani have done some really great sessions on totally different topics, so do catch up with them if you get the opportunity too. Okay, so design, learning and evaluation in multi-stakeholder collaborations, as many words as we could fit into a title. I said my name is Nick Bogalpole and joining me today is Cameron Willis and we're from Day 4 projects. I'm just putting this up here so that if you are interested in some of this work that we do in partnerships, platforms, collaborations, alliances, please do reach out. We're always really interested to hear from people who are working in this space or practitioners or wherever you come from as we are really interested in this space ourselves. I'm just going to take us through a bit of an agenda for the day. There's way too much on there, so hopefully the recording will stand and we'll have an opportunity to go over some of these topics later. But Cam's going to take us through a kind of introduction, I suppose, to what we mean by multi-stakeholder platforms and partnerships. And then we were really taken by the topic of this kind of design and evaluation coming together and colliding. So we're going to talk a bit about the foundations of collaboration and how they meet and echo and collide with the foundations of design. I'm going to talk a bit about the role of design in multi-stakeholder platforms and partnerships and it'd be really nice to test some of those ideas with you on the call today. We're really excited. We've got our friend and colleague, Dr Melanie Pascott here, too. We're going to do a bit of a Q&A. So if you've got questions from Mel, or you like asking questions of panels, that will be good for you. And then we're going to try and do a bit of an exercise and get to know one another. I know lots of people have their cameras off and that's totally fine. When we get to that exercise bit, it might be nice to switch them on to talk to one another in some breakout rooms, but we'll do a quick session there. And as I said, Jess is going to take us through a kind of feedback and a looking forward session at the end of today. So do stick around for that if you're here. We'll finish with some practical tips to get started in this space and that will be the session. I'm going to hand over to Cam now. Thanks, Nick. And hello, everybody. Welcome to everybody that's joining us today on the call. As Nick said, my name's Cam and I work with Day 4 as well on all things. Multi-stakeholder platforms, partnerships and collaborations. And I suppose that word collaboration is kind of the overarching word that we have for all of those different words that get used within this space, be they partnerships, platforms, alliances, coalitions, collaboratives, all these different words that we find ourselves hearing and using within this space. And you'll have other words as well that you're familiar with things like interdisciplinary practices and cross-sectoral collaborations, intersectoral collaborations as well. So lots of words that fit under that umbrella of collaboration. And I'm sure for many people that are joining us today, are you evaluators or designers or researchers or consultants or policymakers and practitioners in all sorts of different ways. You have found yourselves or find yourselves working in collaborations all the time, working in partnerships. It's kind of become this way that we find ourselves working. And so I'm sure many of you will be familiar with being the trials and tribulations of working in partnership and working in collaboration. And as Nick said, we think there's a really interesting set of synergies and sweet spots that exist between the worlds of collaboration, between the world's designer, between the world's of learning and evaluation as well. So we hope that that's a shared interest with the folks on the call today. This definition from the partnering initiative is a really nice one around what to be mean by partnerships and platforms. And I won't read it out. Other than to note that really at the core, regardless of what the words have been used, at the core of collaboration is this commitment and belief in collaborative advantage that we can achieve more together by working together than what we can by working alone. And so these mechanisms of collaboration provide us with the tools to actually share our skills, our resources, our talents, our reach to different groups and communities in ways that will actually help us achieve a shared ambition or a shared goal. And it's not to say that we won't necessarily derive things for ourselves along the way. In fact, we have to be able to do that. These collaborations have to be able to serve our self interest. But there's something about the creation of something together, which is that collaborative advantage, which is why we're so interested in this way of working and why we think other people are as well. When we think about collaboration and a lot of the work that we do in collaboration can kind of fit within these four buckets of thoughts or lenses or areas of coming into collaboration. And when we're thinking about designing collaborations as well, these areas are particularly important. The first one is around the problem. So what is the reason why this collaboration exists? So why are we bringing a group of people together? What's a problem that we're seeking to solve? For many of the problems that we work in, they're complex problems.  Other notes and examples are things like reducing plastic in the ocean or tackling poverty or helping communities be resilient in the face of natural disasters and recovering from natural disasters. So those kinds of complex problems around which collaborations are really valuable for addressing. That's the first sort of bucket of area that we work in. The second thing that we might choose to look at is thinking about what's the type of collaboration that we have in front of us or what's the type of collaboration that we might need in order to advance towards solving that problem. Some of those words I mentioned, partnerships and alliances and coalitions, they fit within this. But there's other ways of thinking about types as well. And there's three that we often find ourselves looking at. The first type can be around the function. So what is the function of what this collaboration might do? Is this about a collaboration that's going to be investing in knowledge exchange? Is it about doing research? Is it about informing policy and practice? Is it about transforming systems? What's the function of what this collaboration is setting out to achieve? The second type might be about the structure. So who's in this collaboration? Is this a research, policy and practice collaboration? Is it a public, private, civil society collaboration? What's the structure? Who's in this that's going to help us advance our shared mission? And then finally, we can think about the type of collaboration in terms of its goals. So what is this trying to achieve? Is this collaboration about designing innovations? Is this collaboration about implementing innovations? Is it about learning and evaluation? About what's happening in the world and what's happening about innovations that we're creating together. So that's the types. Activities of collaboration is the third bucket that gets lots of attention. And if we are evaluators or designers or other people that are involved in collaborations, we might find ourselves working on shared projects together. They're very real expressions of what it means to work in collaboration with each other. There's also the, I suppose, invisible work of collaborations as well, the invisible work that happens within partnerships. And sometimes we don't even know that we're doing it. This is all of that coming together, convening spaces, holding spaces together, finding areas where we agree and disagree and resolve in conflict, building up a sense of where each other is going and what are our shared interests and our shared beliefs, et cetera. So all of that invisible activity that happens within collaborations as well. And the fourth bucket that we found ourselves looking, we find ourselves thinking about when designing collaborations is around the foundations of a collaboration. So what makes for a really strong and effective way of partnering? Other folks might call these critical success factors and these are things that are needed in order to be an effective partnership, things like trust and reciprocity, mutual accountability, shared vision and shared direction of where we're going. So those critical factors that underlie the success of what our collaboration might look like. And I suppose when we start thinking about and when we started thinking about those critical success factors and what makes for a really strong partnership, it struck us as being that it's so similar and there's so much overlap between that world and the world of what makes for good design. So the left-hand side, you can see some of the words that I've just mentioned around what makes for effective partnering. So really clear collaborative leadership, collaborative governance, mutual accountability and transparency, places where participation and equity and representation are lived values of the work of what a partnership is doing, where there is shared vision and shared objectives and the collective commitment to those objectives and trust and goodwill, etc. etc. The list goes on. So what makes for an effective partnership on the left-hand side? And this whole series of sessions that's been run this week has been about design in part and design and evaluation, that we're thinking about what makes for good design. Well, good design is inclusive. It's respectful, it's participative, it's iterative. There's a clear purpose where those people that are involved in design share that shared purpose and understand what that shared purpose is. Equality and reciprocity and trust and respect are values that are lived and breathed throughout the design process. So it really struck us as when we were thinking about the work that we do in collaboration and I'm sure your experiences of working in collaboration, the synergies that you find between that world and the world of design. And so how can we then think about maybe applying some of that design thinking, design concepts, design theory to the work that we might do when we're working in collaboration with each other as designers, evaluators and others. So what we find is that the concepts of design are really sort of woven through and threaded through all of the work and all the stages of what a collaboration might go through. The lifecycle I suppose of what a collaborative initiative might look like. So practically that might mean we might use some of our design thinking and design skills and design concepts when we're designing the partnership itself, when we might need to empathize with those folks that we're seeking to bring together. What are their needs? What are our collective needs? What are the problems that we're going to seek to work on together? We might need to get the insights from across all of those different perspectives. So design thinking at that early stage is hugely valuable. We might use design thinking and design principles when we're thinking about designing innovations themselves. So this newness when we're ideating and surfacing and bringing together spaces where people can do that collectively. New ways of tackling problems that are shared amongst us. We might be designing for implementation and this is where concepts around prototyping that are sort of so inherent within design thinking is so relevant to the work that might happen within a multi-stakeholder partnership or platform. How can we prototype within a partnership the kinds of solutions that we think are going to be most valuable to the problems we're seeking to address? And then finally designing for learning and evaluation. Again sort of that third bucket of area that we're interested in that evaluation component where we're testing the solutions that we've come up with and maybe using partnerships and platforms and collaborations as vehicles for agreeing on what actually do we want to learn about when we're implementing some of these solutions. How are we going to know if some of these things are effective? What are the evidence that what's the evidence that we each need in order to understand the effectiveness and effects of the interventions or innovations we're creating and putting out into the world? So for us design is really threaded through all of the work that we do in partnerships including in the work that we do as evaluators of partnerships. That might all sound a bit theoretical, a bit esoteric, a bit nebulous and so to take us from I suppose a very high level view of design and partnerships and evaluation. We thought it'd be really nice to have the conversation with Mel, so Nick's going to take us through that. Thanks, Cam. Yeah, so as Cam said, I'm going to introduce my colleague, Dr. Mel Perska. And Mel is a senior research fellow at the Australian National University and also at the Australian Prevention Partnership Centre. I'm going to start with a kind of couple of questions for Mel. If you think of questions that you'd like to ask Mel or to either of us actually, please do put them in the chat and we can keep the conversation going. Mel, this one's pre-prepared, so I'm going to read it out as it's written. But I wonder if you could start with telling us a bit about your experiences of evaluating partnerships or other collaborative entities. Thanks so much, Nick. And thanks, Cam, to thanks everyone and hello. Yeah, you know, I mean, this is a planted question. I have thought about it beforehand. And you know, when I think about evaluating evaluations that I've been involved in in the past, most of them have really just looked specifically at programs or interventions or policies, as opposed to looking at the partnership itself or the collaboration itself. And so it's actually been really new for me to think about design thinking and co-design and even bringing in a systems thinking lens to my work. So, yeah, I just wanted to share a brief example of an evaluation that I've recently completed on a past piece of work. So I completed a work, a piece of work that looked at the influence of nutrition-related inequities in Australia quite a few years ago. And that project had some pretty big claims in it. You know, we're going to make these big scale policy changes to the food system within three to five years. You know, we're going to have all these big changes happening within the system. And what happened was that didn't happen. And you know, I can be quite idealistic with my work and whatnot and sort of going, oh, you know, we're not making change in research or evaluation. You know, why are we seeing these changes? What is it about it? And so I have actually just recently in the last couple of years had an opportunity to come back and work on some projects that really incorporate learning and evaluation and reflective practices. So I used this particular project in particular to look back and say, well, what was it that prevented us from making change? What about the partnership and this collaborative effort hindered the process? And so I went through a very structured sort of process and I'm really happy to send the framework that I used to go through that. It's been published now. To reflect on who was in the room? Do we have the right collaborators in the first place? Do they even have a remit to make change? No, they didn't. You know, that was one of our big sort of learnings from that. We didn't necessarily have all of the right people in the room at the right level. And looking back, we had a big focus on understanding the system, but we didn't focus so much on change within the system. It was more describing and understanding. And so we really needed to bring in a really strong implementation focus. So it would have been really great to have brought in a theory of change. And I'm talking very abstractly here. So sorry about that. And I'm really happy to share the work. But I guess the point that I wanted to share was that in going backwards and engaging in this reflective process with some of my collaborators, there was about 15 involved in this process. And some took part in it, some didn't. It was interesting to see people's responses to going back and reflecting. Some people were very happy to look back and explore their blind spots and kind of look back and go, oh, we really didn't do those things that we set out to do and why. Other people felt very uncomfortable about reflecting upon a process and publishing on it. So we did publish our reflective process as well. Because I think when we come into different projects and whatnot, we might forget that learning lens. I certainly did. And that reflection lens. So yeah. It's interesting now, listening to you talking about this kind of the evaluation of it, I suppose, because on reflecting, you're doing lots of kind of learning cycles, continuous learning loops. And I suppose I'm interested in your take then, like on some of the contrary that we've been given today. Do you see that link between, I suppose, design-led approaches and evaluation, particularly in this work in the partnership example you're giving? Do you see any use for them? Yeah, absolutely. I think in particular that kind of co-design focus of bringing lots of people in and generating a shared understanding of a project, making sure everyone's on the same page. There's a lot of respect for different people at different levels of the hierarchy, I guess, and in different sectors being involved as well. And also this ability to iterate. I think that in going back and reflecting, if we had input within our project, reflection and learning cycles, we could have changed direction and had a greater impact with our work. And maybe it would have gone in a completely different direction to what we first intended, but it would have hopefully gone in a direction where it was feasible to create change, whether or not it was small or large. Yeah, yeah, definitely. And I was going to ask you the examples of where it's worked really well for you. I suppose in this looking back now, you had an opportunity to rethink about it. Were there elements of design that were embedded into the evaluation or into the partnership itself that went well from the beginning? Because it was a co-design process, that collaboration. And so there was a lot of focus on shared understanding, especially in terms of the method that we were using. Who would be doing what with those roles within the process? Yeah, I think a lot of it was kind of implicit. There was no explicit sort of goal around design thinking or some of those different factors that you mentioned. And I think that's quite common for a lot of the work that we're doing. I think we're unconsciously bringing in these kind of elements of design thinking into our work without really thinking about them in a lot of depth. And I wonder if, if we did, how it would really optimise the work that we're doing. Yeah, I think you're speaking to why this kind of this special intro script comes together. What happens if you start to think consciously about design and evaluation in the same space and coming at it from different angles? Thank you so much, Mel. I'm just going to ask anyone there if you have got a question you want to speak up. Do put the question out there. If you want to write it in the chat, I'll read it for you. Mel can read it too. Give you a moment to do that. Oh, Lucy. That's okay. You don't have to have questions. If you've got them and you want to save them up, just write them in the chat and we'll get to them as we go through. Thanks so much, Mel. I think as Cam said, there's something quite theoretical about these fields of how design or how partnerships or how evaluation comes together. And I particularly like a practical example of where these things come to life. So they think, oh, would I have done that too? Oh, was that co-design that thing that I did or whatever it is? So thank you for sharing that. I'm going to just bring my slides back up here. So, oh, look at that. I left an animation on. That was pretty snazzy for everyone too. Bill and I were talking about slides before, and that is why I don't usually have these animations on. But hopefully you'll enjoy that. I'm going to put you into some breakout rooms for a moment and try and have a conversation if you're open to it. The questions that are up there on the space, and we're interested if you're evaluating complex partnerships, approaches, collaborations, multidisciplinary things, alliances, pick the word. If you were involved, what would some of the benefits be of taking a design-led approach? And your answer might be no, none at all, and that's totally fine. I'm going to put you into some breakout rooms for about five, six, seven minutes. I'll decide. I'll let you know when we're coming back. I'd encourage you to bring your cameras on, to have a chat with your colleagues. This is one of our opportunities to talk to one another. I will ask anyone from the group to share back their thoughts at the end too. So if you're someone who feels like they want to share, do pay to share it with us as well. Opening the breakout rooms now. Hello. Welcome back, everyone. Thanks for giving that a go. I mean, we were just discussing. I mean, we've probably all been in a lot of breakout rooms over the last couple of years. So thank you for participating and getting to know one another. There's a really nice group of people here. So thanks for engaging. I wondered if anyone wanted to share their thoughts and reflections from that question there, what some of the benefits of using a design-led approach or design in your approach to evaluation might have been. If I can't see faces, but I'll just make sure I can. There we go. Please, do you just feel free to come off mute and let us know if you've got ideas or chuck them in the chat as well. No, I don't know. I did it for a clock on a Thursday. Oh, that was funny that we still got lots of smiles. That's good. Mel, from your group, I'm going to put you on the spot. Did anything come up? Yeah, I mean, one of the things that we're talking about is the necessity to be really clear up front when you're trying to implement one of these processes with a group, and that there's sometimes a lot of uncertainty about how things might unfold because of the iterative process and how there's so much work in the front end to establish those relationships in ways of working. But something that also came up in our group was this question of what's new about this in my group. There was someone who's been working in evaluation for a really long time and she shared that this is what she's been doing all along in her career. What's really new about design thinking or this emphasis on learning and evaluation as it relates to MSPs and collaborations? Yeah, I think probably everyone on the call would echo that in some way. There's different words for the same things, isn't there? And we kind of see them in different places. Of course, how you use them and the approaches have actually played out in different situations are different. But yeah, I mean, everything goes in and out of fashion and we have different kind of terms for it as well. Were there other reflections from anyone that would like to share? Maybe we'll just riff off that because we also talked about what is the difference, what is added in design, co-design approaches that wasn't in things like participatory action research or collaborative ways that have been around for a long time, some people have been using. But we were talking about, and I still think that prototyping is, when a new thing comes around it brings a lot of history with it, doesn't it? It brings a lot of what's been there before but it sometimes adds a new twist. And to me, and we were chatting about this and to the people in the group, we were talking about prototyping as being a new addition and rapid cycles of testing with community members. And that seems to feel different and present new challenges for evaluation. And that you have to sort of understand that if you're going to work as an evaluator with that and maybe it means that you need to take a different approach to evaluation and some of the answers to how you embed learning and evaluation into that are probably gained from understanding how you plan in chunks like agile planning, short cycle planning and reflection and rapid data collection and things like that. I think that's a really fantastic insight, Jason. And it is something that we see in partnerships as well. You know, the kind of way that partnerships are set up to trial things, they're kind of like hotbeds of experimentation, of piloting and of prototyping. And I think that's certainly something we recognize with people coming together, particularly from different sectors, you know, whether it be like kind of private or industry and then government and civil society sectors coming together. There is a need to prototype and I would add to that like the failed fast kind of a mentality around it too, where the need to kind of say, does this work before we make this of any scale or use or reach anywhere? And certainly that's like a real focus for those different groups coming together. And I think you're right about the evaluation and learning aspect of it, because it changes what you choose to evaluate and changes how you choose to evaluate that as well. I would agree that the prototyping is really important. I'm going to share my slides to wrap this up and then we'll move on to some forward planning. So just some tips to get started in this space and some of these things will be things that you've already seen before, or there'll be things that you already do, maybe some of them are new and then that will be counted as a tip. So I suppose the first is using design or design-led approaches to reconsider the what you choose to evaluate. And here's some suggestions of the kinds of things you might choose to evaluate. So Mel talked about this a little bit before, but how the partnership is going about understanding a problem. How did it do that? What was that process that it undertook? How that partnership is ideating and prototyping? And Jess has obviously just spoken about that. So it's not just what is the prototype, but what is the process of prototyping experimentation? How does that agree or arrive at new solutions? What's new about them? What's different about them? The results then of those experiments of testing of those prototyping sessions of those rapid cycles and of course the implementation of whatever the program, project, initiative, partnership might be interesting in. I think then, and then the second point then you can read it is, can change the way you go about evaluating things. So you can, it changes the way we involve stakeholders. And of course there's long histories of, as you say, participatory app, fashion research, of ethnographic kind of studies. So kind of being out of participant layer of practice-based, all of those words are connected and ring true. But design kind of gives us a way into perhaps reconsider the ways that we involve multi stakeholders from different disciplines in the development and implementation of evaluations. And it can shift the timing of when you decide to evaluate that kind of developmental evaluation. The being integrated, recognizing dynamic, emergent natures of what's happening between people. We don't always know where we're going. And as those things shift, we bring in different kinds of ways of evaluating and of checking that things kind of remove the formative summative world or lens. I think the thing, and I want to finish on this, is the thing we've found most useful. And you know, evaluators are always making a case for evaluation updates that researchers always want to do more research. But one of the things that we have found to be really true is that using these kind of approaches of empathizing, of thinking about what your users need, of prototyping, of testing solutions, they can be ways into continuously strengthening the partnership itself. So rather than just kind of, you know, doing that stakeholder map and spending all the time getting everyone together, it can really be about examining those core foundations of your partnership. Is there trust there? Is there reciprocity? Have you achieved shared agendas? Have you achieved mutual accountability? And if you haven't, why not? And how can you shift and learn to continuously develop some of those foundations? So I don't really encourage you, if you're not already doing this, to use design and to kind of think about strength in your partnership as well. That is a session for today. I'm going to leave it for one minute for questions I hand over to Jess and to any questions on any of the content from today. It was all clear and it's all recorded. And I would encourage you to look at all of the other recordings from this week as well. There's some totally different topics and some great insights from people presenting on other days if you weren't able to join. Thank you very much. Over to you, Jess. Thanks so much, Nick and Cam, for such an interesting provocative session. I've been maybe we're all thinking about it whilst we're being a little quiet because I'm going, oh, how could I think, how could I apply that? So yeah, very thought-provoking. Thank you so much for your contribution. Yeah, so we're wrapping up after this week, this learning sprint, this idea of learning about the intersection of design and evaluation, which is one of the emerging areas in evaluation. And I think runs this group volunteers run this group to get us thinking, but we're all really open to ideas about how to run this thing and open to new members as well. If anybody wants to get involved in the organizing committee, we've been doing it for, Matt and I have been leading it for a few years. And so we're really keen to get your ideas. So the idea of this very final session, let's see how many people are left. So we've got 28 people. Thank you for hanging in there to the end of the week. It's just to have a little bit of a think about, apply evaluation as always to the week. I would love to know how you found the week, what you liked, what we could do better, what you might do, what you might offer, and ideas for next year. So we're really up for what topics really picked your interest this week or other things that you're hearing in your field that you'd like to know more about or ideas for building our skill set. Because I mean, the reason why we set this sig up was because, well, I believe personally, quite strongly that this skill set of design thinking, even if some of it's recycled, and it's got some new bits is really gaining quite a lot of space. And if we want to maintain our legitimacy as a value as we, it really, really helps to get your head around it. And also, you know, design thinking seems to be offering a bit of promise to tackle really tricky problems in a way that other things perhaps haven't. So we want to be part of that. So this is a bit of like, how do we collectively skill up really? How do we skill up this sector so we don't become irrelevant? That's my fear is if we don't get on and really understand this, that our sector will maybe get left behind. So that's why it's so important to learn about this intersection. That's why I believe in it. So I'm really keen to know what you think we can do and what we should do next. So for those of you who are willing to stay on for the final stretch, we're going to do one final breakout group. Maybe we'll have let's see how many people we've got. We've got 24 about five groups, Michelle. And I have before we break out, I have a link to share with you. I have some Google slides so that we fill out in boxes. What I would really encourage you to do is to, oh no, let me get my link, is to look at what group number you're in and then go to the right slide. So at the top of the slides, I have shared, let me just find the chat function. I have shared the number of the group. So when you go into your group, look at what number it is. There's a click. So can you see the link there? Yep. Can somebody check that it's working? I'll just share my screen for a sec so you can see it. So can you see that? Okay. The link's working fine. Yeah, the link's working. And can you see my slides? Yeah. So the questions are, what did you like about the sprint? What could have been even better? And ideas for the future. And you'll see that at the top of each slide, there's a little black number that says group one, two, three, to only go to your group. Otherwise, you'll find some magic writing appearing and it'll get, you'll get very confused. So we're going to break into the group. So please go into the slide with the people that are in your group. We're going to have five groups. So it'll be one, two, three, four, five. And we're just going to do that for 10 minutes and then we're going to regroup. So 10 minutes to write it down and then we'll wrap up for the week. Okay. So if we could now go to our five groups, random assignment is fine. Just on this four group, four groups. Yeah, that's fine. Just looking at the dropout. So yeah, fair enough. Be too long. I just want to say thanks for your ideas. There's some great stuff there from book clubs. I really like the idea of a book club to a reading list, to thinking about the right balance of passive versus active ideas and like demonstrating technology and learning new tech such as using mirrorboards and things like that in sessions. Really great ideas. We're going to take them on board. And the only thing I would say, the only final thing is if any of you are interested in joining the committee, we're looking for new members, please reach out. And on this mirror board, which you still have, our contact details, send an email to myself or just design at as.asn.au. I'll put that in the link now. If you want to get involved or just let us know here, we would love to hear from you. And with that, I'd just like to wrap it up. And I'd just like to thank everybody for participating in the week. Thank the volunteers for presenting sessions. Thanks, Nick and Kam Sadeh, Shani and Joe and Matt, of course, will all put in their time to organize this from the committee. And thank Bill and Michelle and AES for hosting this forum. And it looks from the feedback that you've enjoyed it and you like things about it. There's always ways like designers always get better, iterate, improve. We've got some ideas, but I really like the book club. So maybe we'll do that. Maybe we'll have a quarterly reading and get together and discuss. I think we should just do that. So I'll bring that to the committee and try and bring some of this reflective space into the AES workshop. We'll put some stuff up. We encourage you to join us at the next committee meeting if you'd like or put your ideas on LinkedIn. If you've got an idea to present and you want a collaborator, it's always fun to present with somebody else. And that's it. That's the week. It's five o'clock.


Hello and welcome to the session. Let us understand the following problem today. If a is equal to 101, 012, 004, then show that determinant of 3a is equal to 27 into determinant of a. Before adding the solution, let us see the key idea. Consider the determinant of the square matrix a is equal to a i to a 3 into 3, that is determinant a is equal to a11, a12, a13, a21, a22, a23, a31, a32, a33. Therefore, the determinant of a is equal to a11 into a22, a33 minus a23 into a32 minus a12 multiplied by a21, a33 minus a31, a23 plus a13 multiplied by a21, a32 minus a22, a31. Now, let us write the solution. Given to us this determinant a is equal to 101, 012, 004. Now, solving this we get which is equal to 1 multiplied by 1 into 4 minus 2 into 0, then minus 0, 0 into 4 minus 0 into 2, then plus 1, 0 into 0 minus 1 into 0. So, we get which is equal to 1 into 4 minus 0 minus 0 plus 0 which is nothing but 4. Therefore, determinant of a is equal to 4. Now, determinant of 3a can be written as 303, 036, 0012. Now, let us find that which is equal to 3 into 3 into 12 minus 6 into 0 minus 0, 0 into 12 minus 0 into 6 plus 3, 0 into 0 minus 0 into 3 which is equal to 3 into 36 minus 0 plus 0 plus 0 which is equal to 3 into 36 which is equal to 108. We have 27 into determinant a is equal to 27 into 4 which is equal to 108. Hence, determinant of 3a is equal to 27 multiplied by determinant of a hence proved. I hope you understood the problem. Bye and have a nice day.


Okay, so let me start thanking the organizers for the invitation and you see the title and the outline of my talk is given here. I will talk mostly about our applications, development of post-DFT methods in the last couple of years. I will give a short introduction, then we'll talk about MP2 and RPA using the Gaussian and plane-based method, so that's the basic methods that we are using and then we'll go to recent developments and if I have time also to some outlook and challenges. So of course this was done by many gifted people. All of this is in collaboration, it was in collaboration with Yostfandi Wanderle and most of the work was done by three brilliant PhD students, Manuel Guidon, Mauro Delbein and Jan Wilhelm. What do we want to do? So we want to do liquids and solutions and we know that after many years of doing water and ions in water and similar systems, there are still some problems with GGADFTs and the question is can we go beyond GGADFT, can we use non-local correlation methods and improve for something like, for example, the density of water where we need a balanced description of hydrogen bonding and van der Waals forces. Another shortcoming that we recently found or it was given to our attention by some experiments at PNNL that, for example, the structure of the solvation shell of ions in water is not very accurately described by GGAs. Or for example, of course, well-known level alignment ions in solution at solid-liquid interfaces are problematic for GGAs. So, for example, if you look at the density of water, we all know if you do this for standard GGA functionals, you get a too low density. If you add now van der Waals corrections in some ad hoc manner, you get better densities but you get a widespread of densities. Depending on the functional you're using, typically are between 1 and 1.1 grams per cubic centimetre, but you can even have outliers like these mini-sorta functionals that are 20% of too dense functionals. So, is there a possibility to have here much better or more predictive values? Another point that as I already said is ions in water. For example, here this is potassium in water. If you do PBE D3 or also a hybrid functional PBE 0 D3, the peak of the radial distribution function is off by almost half an angstrom. So, could this be something that can be done with post-Hartifoke methods or post-density functional GGA methods? Another thing, energy levels in liquid water. We all know the band gap of water and that then is only described in these functionals also in hybrid functionals and this then affects also the levels of ions in water. So, to do this type of simulations, what we want to do is electronic structure calculations and we want to use non-local correlation because that includes the van der Waals correction and we hope to have van der Waals correction and for example hydrogen bonding on the same level so this should add to better accuracy. Of course, if you go to quantum chemistry, there are many methods doing this but we only want to do the most simple ones. For example, Mp2 or direct RPA may be in connection with a double hybrid system. Still, it's a challenge because we want to do system sizes that we can do sampling for liquids so at least 200 atoms in the order of 500 correlated atoms. We are using a Gaussian basis set or a localized basis set so that means for us the order of 4,000 basis functions because these methods are much more demanding on the basis than GGA functionals are so we are typically in the order of 4,000 more basis functions. On the other hand, we want to do liquids so of course we have periodic boundary conditions but the gamma point is hopefully enough so we concentrate on gamma point only calculations. Simulations are not enough. We have to do sampling. Sampling can be done either by molecular dynamics or Monte Carlo. These methods are expensive so you better do the best you can meaning multiple time step schemes or very accurate bias potentials so that you can get a long way without doing a calculation on the highest level. In all calculations showing this has been done. Then, still we need smooth energy surfaces. Monte Carlo will not work if you have a racked energy surface or accurate analytic forces if you want to do MD. So if we have these methods, they have to be analytic. We can cut corners but not all corners can be cut. We need in the order of 20,000 for water. This is now water at ambient temperature. We need in the order of 20,000 energy calculation. So this means 20,000 RPA calculations gives us one density of water. What does that mean? Nowadays, if you go to praise, you probably get easily from praise. CPU budget of one million no dollars. If you want to use this for this type of calculations and you want to spend it in three months, you don't want to wait too long, and you run on 512 nodes that means roughly one RPA or MP2 calculations in six minutes. So that's our goal. Now how do we do this? All of these calculations need at some point electrostatic integrals, two electron integral types of occupied times virtual orbitals interacting with occupied times virtual orbitals. Now we use a Gaussian auxiliary basis and an RI approach, resolution of identity or density fitting approach that these type of integrals are always calculated from this formula here where we have here three center type integrals and then here the inverse of the Coulomb matrix. This integral is calculated using the Gaussian and plane wave approach, meaning by Fourier transforms, and then these integrals can be calculated as a tensor contraction over these three index tensors where the three index tensors themselves are calculated using these GPW integrals that meaning we are calculating these integrals by putting these functions as these are our auxiliary basis on the grid doing an FFT, calculating the real space potential of this orbital and then integrating over the product of two Gaussian functions. This is the GPW approach. We are very good at doing this and then calculating this quantity, and this quantity then has to be transferred with your molecular orbitals to the BIAS. Once we have this, we can plug this scheme into the MP2 energy, you see you have these type of integrals, you calculate them using this tensor contraction, that is in fact the most expensive part. If you look at it, it's a contraction over the auxiliary basis that makes it an n to the fifth step, so we don't reduce the scaling, we just make it faster. So it's still an n to the fifth step for MP2. The RPA formula given in this form includes the Q matrix that is a matrix of the auxiliary basis and it has tensor contraction over the particle whole states and this makes it an n to the fourth step, so RPA calculations in this form can be done on n to the fourth steps. Now, applying these methods to our system, what we are doing is isobaric isothermal Monte Carlo simulations are liquid water, 64 molecules, 192 atoms. The basis set for the experts is a correlation-consisted triple-seat surveillance basis set, roughly three and a half thousand basis functions. The RI basis is 8700 basis functions. A little bit on scaling, these methods are very expensive but they are also very easily to, or rather easily to, parallelize, so parallelization up to several ten thousands of cores for this specific system is very efficient. Now, you do the scale, the sampling and in the end, you can look at how much CPU time I have burned to get my density of water. So you start with a GGA, you need roughly, we need roughly 100,000 CPU hours to get the density. Hybrid functions, depending on how you do it, is a factor of 3 to 20 in our code, more expensive to get the density and then RPA and MP2 are even more expensive, typically 50 to 100 times more than a GGA calculation. So that's what you have to invest to do this type of calculations. Now what do you get? Here is the Monte Carlo profile, the Monte Carlo Cycles profile of the density, MP2, some oscillations of course and fluctuations but it converges to a density of 1.02, that's 1.02 much, it's 1.02 grams per millilitres or not bad. RPA gives you a slightly different than lower density 0.99 grams per millilitre. So very good results, seems to be that really this non-local correlation and being doing everything on the same level is a way to get very good results for these systems. What about the sodium in water again? We get a very or rather different radial distribution function and the peak is now much closer to experiment again. Energy levels in liquid water, they are better, still not perfect, this is certainly something where we don't get away easily with just using MP2 and direct RPA. This was the standard implementation that has been done a couple of years ago, mostly by Maugo and in recent years we have made some progress on these codes. First, we have forces and stress tensors for MP2, both restricted and unrestricted. We have also implemented GW, once you have RPA, the step to GW is not that big. This has been done by Jan Wilhelm in the last couple of years and very recently implementations where we reduced the scaling of the RPA and GW from N to the fourth to N to the third. Now, some information how expensive are MP2 forces or RPA forces, these are now non-variational methods, so you really pay for forces. Typically, a factor of four. So the force is four times more expensive than the energy in these type of calculations. It's the same for close-cell and open-cell calculations. The additional factor going from close-cell to open-cell is roughly a factor of three. You have to do three independent energy terms instead of one in MP2 going from close to open-cell. With this, it was possible to calculate an MD trajectory of water and from the MD trajectory to calculate the infrared spectra of water. So what you hear see is the experimental and the MP2 infrared spectroscopy calculated from a velocity auto-correlation function on an MP2 MD trajectory. With some tricks, one of them is that it's scaled by 0.95 that all the quantum chemists will say, of course MP2 scaling factor for infrared frequency is 0.95 and we also use this. This seems to be a universal law because MP2 is not the final answer. There are still many types of correlation to be added. The GW code, just to show you, here the scaling with system size and here the parallel scaling with system size, if you look here in this double logarithmic plot, it's also n to the fourth. So GW in this implementation is n to the fourth like the underlying RPA calculation. Roughly the same timing. So these calculations with 20 quasi-particle energies took about, for a system of 64 waters, take about a couple of minutes on a machine of this size here, few thousand processes. Now last and newest development is a cubic scaling RPA. So here the idea is that we are using another resolution of identity that is no longer based on the Coulomb metric but now on the overlap metric. Integrals can be calculated through this contraction, tensor contraction. The advantage is now that these tensors here, these three index tensors are overlaps between three different atomic functions and so they are extremely sparse. Only when all three functions overlap, they are non-zero. So these are extremely sparse quantities here and we can make use of this sparsity. What you need is the three center overlaps, the two center overlaps, and you need here the integrals and I forgot to say, of course, whenever I write here an integral, in principle I mean a periodic local function. So it's an atomic function but it's periodic. Fully periodic means all these integrals, in fact these Coulomb integrals are in fact evolved integrals. And for this type of evolved integrals over Gaussian functions we now have a semi-analytic integration so we don't do this anymore with Fourier transforms, we directly calculate these integrals analytically. And this gives you a big speed up for this type of calculations, reducing the pre-factor again, not the scaling of the method. So we are using this sparsity of the three center overlap integrals and we need an additional trick. That is we are using also now a time, or we are going to the time domain and we are calculating the Q matrix in the direct IPA now in the time domain or with other words we are doing a Fourier transform. If you do a Fourier transform you can separate the energy part here and you can derive here a quasi-density matrix or in fact it's a Green's function that only depends on the occupied orbitals and one that only depends on the unoccupied orbitals. Now this is the final formula. If you go through it you make use of all the sparsity. This is cubic scaling. Cubic scaling because in the end you have to calculate the logarithm of this matrix. This matrix we don't assume it's sparse so you do a logarithm of a full matrix that's n cubed. That's where n cubed is coming in. Here is the comparison. The cubic scaling code compared to the linear scaling roughly 128 waters is the break-even point in the current implementation. The interesting thing is also that in fact the cubic part is not yet setting in and for all of these systems up to 1000 water molecules in fact the code is quadratically scaling. Even better the quadratically scaling part becomes linear scaling once this density matrix is also become sparse. The only cubic scaling part left working with this matrix is all the rest is in fact becoming linear scaling. The same can be done for GW so this is now a cubic scaling GW calculation. You already heard before of these nano ribbons so what Jan together with the people at EMPA he has calculated these nano ribbons the GW calculations and you can see here for these quasi 2D systems the scaling is even better and the break-even point is at roughly 100 carbons and it could go up to 1,700 nano ribbons with 1,700 carbon atoms. Almost finished. I also almost finished in time. A little bit of outlook and challenges that we have basis set convergence how to calculate properties for these type of functionals and can we do sustainable code development once we go in these type of complex systems. So basis set convergence I just wanted to say if you ask a quantum chemistry we'll tell you it's solved just use an F12 algorithm unfortunately nobody ever thought or I don't know of anybody of thinking of F12 algorithms for RPA and GW especially not once the tires are low scaling as the implementations we have. Another possibility would be maybe we should go for double hybrids with only long range not short range and then the basis set convergence is not a problem. The other thing is we need an additional basis set Gaussian basis sets are already something not that easy to handle especially if you come from a plane wave a community where you have one number and suddenly you have 1,000 different basis sets which one to pick and now comes additional RI basis sets so what do we do? We can automatically generate RI basis sets but they are typically two times bigger than the ones that we do by hand for the same accuracy but maybe it's nice to go this way anyway CPU time is always there and human time is very valuable. Properties as I said derivatives and everything we have non-variational wave functions it's hard to do and question is how do we do properties any property a response property is already a second derivative and so on so we have to do second order response it's something we have to think about also periodic boundary conditions so for example how do you calculate the dipole moment of mp2 in periodic boundary conditions Sustainable code development because I'm at the end I can just see everything means increased code complexity if you decrease the complexity of your algorithms you usually increase the complexity of your code that's something that is almost universal the same is true also for hardware the more sophisticated you want to go in your hardware usually your codes become more and more complex something that is difficult to be sustainable in the future I don't know I don't have a solution for this so with this at the end I'm happy to take questions thank you


A 2 year old girl presented with left sided facial weakness, an MRI was asked for to look for the brainstem and CP angle. Routine T2 and flare sequences did not show any parankymal abnormality, the brainstem was unremarkable and the CP angles were also unremarkable. Except there was a small rat case cleft cyst in the cellar which appeared hyper intense on T1 and hyper intense on T2 weighted sequences. We also acquired 3D gradient deco sequences to look for the 7th nerve. The 7th and 8th nerves accompany each other in the cistern as well as the internal auditory canal. These are seen as bills bars within the internal auditory canal wherein the facial nerve is located anterior superiorly. The cochlear nerve is located anteriorly and the superior and inferior vestibular nerves are located posteriorly and posteriorly respectively. In our case, it is difficult to visualize the facial nerve on axial flare sequences however on sagittal recon a thin linear structure is seen accompanying the right 8th nerve in the cistern segment thus confirming its presence and on the left side it is absent. CT cuts of the temporal bone were also acquired which show non visualization of left facial canal beyond the genome whereas the right facial canal is well visualized along its entire length thus confirming the diagnosis of congenital facial nerve aplasia. Congenital facial nerve aplasia is a rare condition where the patients present with signs of facial nerve paralysis. It can be unilateral or bilateral characterized by facial asymmetry, incomplete closure of eyelid, deviation of angle of mouth and loss of nasolabial furrow. It can be caused due to intrapartum trauma or can be developmental. In our case, there was no history of intrapartum trauma. Imaging findings The facial nerve itself is not visualized on CT however developmental abnormalities of the facial canal such as hypoplasia or aplasia can be seen provided thin sections of the temporal bone are obtained. On MRI it is essential to acquire 3D gradient echo sequences to image the 7th and 8th nerves within the cistern as well as the canal and as discussed earlier absence of facial nerve has to be demonstrated within the canalicular or cisternal segment.