All right we're recording now. Hi everyone and welcome to our cross-plane community day panel where we're going to talk about bringing the cloud to Kubernetes. So today we've got a panel assembled of folks who are experts in taking cloud APIs and modeling them as Kubernetes custom resources. I'm not sure what we'll talk about later. I'm not sure if there's a consistent name for what that problem or the software that does that is. That's an interesting thing in and of itself. So first I just want to introduce everyone that we have on the call here. We have Joyce Ma from the Google Config Connector team. We've got Matt Christopher from the Azure Service Operator team. Jay Pipes, not Pepez as he tried to trick me that it was before, from the ACK from the Amazon Controllers for Kubernetes team and my co-worker colleague from UpBound, Dan Mankham, a maintainer of the cross-plane project. So today we're going to sort of, I guess as I said before, think about how we could build a consistent KREM experience on top of the cloud APIs. But first I want to ask Matt, what should we call these things? Yeah, control is operators. Yeah, at least internally in my team, we talk about this as a cloud operator or an Azure operator for the cloud, something like that. We tend to prefer the operator nomenclature as opposed to something else like a plugin or add-on or something. But I think you're right that I haven't heard of industry accepted term for what this thing is. So Jay, you've used the controllers parlance but it used to be the AWS service operator, right? Was there any real thought in changing to controllers or was it just a nice acronym? Well, A, I'm terrible at naming things just generally. I'm like super flat and explicit about everything.  So it was AWS controllers for Kubernetes. Yeah, I actually like config connector as like the brand or like, because that's what it is, right? These are connector pieces between the Kubernetes API and resource model and the cloud provider's APIs. So it's like connecting the configuration that's stored in Kubernetes with the cloud. So I kind of like that. But yeah, operator was... These are more than just an operator, right? Like the connection of a CRD and a custom controller. They are specific types of controllers that provide that sort of integration bridge, right, between Kubernetes land and the cloud provider land. So I like the connector part of things. Yeah, me too, which brings me to... I think I'm a little hazy on the timeline, but I think config connector was one of the first projects out there doing this kind of thing. Joyce, do you have any background on what inspired the GCP team, the Google Cloud team, to start config connector? What was the background behind that project being bootstrapped? Yeah, exactly. So basically we've been heard from customers for years that they want to move to cloud and meanwhile they want a consistent environment like on-prem and in-cloud and they also prefer consistent user experience across APIs. So in that sense, imperative APIs is naturally limited providing this consistent user experience. And we started looking at this declarative world. And at that time, it was back in 2018, there were already a deployment manager in GCP that supported declarative configuration of the APIs. And also there were like Terraform Google provider. Those are good toolings, but both of them has its own shortcomings. And meanwhile, we see that there are already customers moving to the Kubernetes platform. So we think it's better to provide a more consistent user experience so they can manage their applications not only for cloud, but also for other native Kubernetes resources. So we decided that we want to build something on top of Kubernetes to manage GCP. Nice. Is that similar for you, Matt or Jay? Is that sort of the same kind of story? Yeah. I mean, very similar for us at Azure. We wanted a Kubernetes first approach, right? Even though there are other ways to do declarative APIs, if you're in the Kubernetes world, it feels kind of awkward to have to step out of that in order to interact with the cloud, especially because you're going to be deploying your core service in Kubernetes.  It's kind of awkward. So that was one of the big drivers for us at least. Yeah, it's exactly the same for us. I mean, we have had cloud formation, right, for well, what seems like forever, right? And there's also Terraform for another sort of declarative way of driving infrastructure changes. But yeah, just like Matt said, we had customers coming to us saying, hey, we prefer the Kubernetes API. We prefer the consistent Kubernetes resource model and API machinery. And especially our devs prefer that, right? So what we've struggled with really is that most of our customers, what we found, they have a lot of application development teams that prefer to use the Kubernetes model. But a lot of these customers are huge enterprises, right? And they still have these like central IT teams. And a lot of those central IT teams also like to work with cloud formation or they may use Terraform or other solutions. So it's sort of a mixed bag. And ACK, the AWS controllers for Kubernetes is primarily focused on that first set, right? The application developers that prefer to use the Kubernetes language. And we've had, yeah, some interesting discussions like how do you work in this hybrid environment where you've got a central IT team that's used to either imperative configuration changes or using cloud formation and things like that versus the app dev teams that are using and preferring the Kubernetes model. Right, right. So Dan, do you, do you, I was hesitant to, not hesitant really, but every time I asked someone from AWS, hey, why did this project start? I get the same answers because the customers wanted it sort of thing. And it's interesting. It was good to hear that. I didn't do that on purpose. I know, I know. It's kind of the best answer to the question though, to be honest, you know, it's much better than like we thought it'd be cool or whatever, which, which brings me to Dan, do you, do you know the sort of, have you done any archeological spelunking to, to find out some of the history of the cross-plane project? Do you feel like you could answer the question of like, where did cross-plane come from? Yeah, I'm, I know at least from, I was aware of the project when it was initially announced and, but wasn't yet with the company. So part of joining upbound and working full time on cross-plane was kind of like buying into the mission, I guess. So I can speak to that at least. And I know something that really motivated me to work on the project was this idea of kind of like a unified control plane, right, or a common way to address cloud providers. And it's grown to really more than just cloud providers, right, to address APIs. Kubernetes happening to be the vehicle for facilitating that. So I think, you know, a lot of the things are similar to the things that other folks on this panel have already said, but with the added layer of kind of like a common interface. And there were, at the beginning, there were some pieces of kind of the composability story, right, piecing these different things together and building higher level abstractions. But most of that has developed over time since the initial impetus. Yeah, my understanding is really mentioned the service broker stuff yet. As far as like componentization or whatever, composition, things like that. I don't know. I guess if I'm putting my archaeologist hat on, right, cross-plane and a number of other things started out sort of in the same time frame that the open service broker concept and community kind of started to become a thing. And I know that the service brokers kind of fallen out of favor and out of style, but I think it's important to bring up that topic as far as like our archaeological dig here. That's very true. I've got to admit I'm somewhat personally ignorant about the history of the sort of open service broker project initiative. I will say, for those of you who are recording this week ahead of time, but for those of you who are watching this conference now, I believe there will be a talk that relates cross-plane to the open service broker from Yarn at Accenture, I believe, if I recall correctly. So that'll be a cool one to keep an eye out for. But yes, that's a really good one to call out that they were definitely sort of forging the path with regards to sort of thinking about deploying all these apps to Kubernetes.  So just to provide a little bit more color and take this with a big grain of salt, because Dan and I were, I think, maybe two of the first people working on cross-plane who were founders of the project. So there was a period before it was open sourced when it was sort of an experimental thing that folks like Luke and Ilya and Jared and Versailles were working on. And I believe that it actually was an accidental thing where they were trying to go build something else and then they found it really useful to apply something a little bit like the persisted volume claim pattern to just databases and buckets of things like that. And then they built that to support the other thing that they were building and like, oh, hey, this is useful. We should open source this and get this out there sort of thing. I believe that's where Crossplane came from. So I also just going back to Dan and I don't want to talk about crossplane too much, but I do want to ask Dan, can you give a little context, Dan? Crossplane kind of approaches things a little bit differently than the other clouds, right? Or than the clouds, I should say. So the other projects represented all this call focus on one cloud sort of thing and delivering all the APIs of that cloud, whereas crossplane scope is a little broader, right? Yeah, that's exactly right. And there is a lot of crossover in terms of representing resources as Kubernetes objects. That being said, if you want to build a layer on top of that that allows you to interchange these resources as common building blocks, you have to have some interface, right, that all of them satisfy, right? So just like if you were doing object-oriented programming or something like that, if you want to be able to change things out, like for like, you need to be able to address them in the same way. So crossplane has a common runtime as well as some common embedded types that it puts into all of its objects. So I know Jay actually mentioned earlier today that it's kind of like the ACK types have kind of the top-level spec of a resource, you know, the fields that are actually going directly to AWS. Whereas with crossplane, we kind of have a container for those sorts of fields, and then the common fields on top of that. So just as an example of some things that would be common between resources, the way they reference each other, the way they produce connection secrets, the way they report, whether they're ready or healthy, those are all things that we need to be consistent with so that we can, you know, compose them into a higher-level abstraction and aggregate up some of those components. Yeah, I think another place where sort of crossplane approach gets interesting goes back to what we were talking about before with regards to potentially integrating with like legacy systems and things like that, because we opened up the, you know, there's only so much that's implemented so far, but, you know, we just released like a vSphere provider, for example. So you can mix and match your, you know, cloud of choice with maybe, and, you know, maybe your cloud of choice is vSphere, I don't want to paint vSphere, it's purely legacy stuff. But, you know, if you do have a different environment, sort of thing, let's say if you're moving from Azure to AWS or vice versa, you can kind of do both consistently or theoretically if you were using we don't have a ton of old school bare metal stuff. We have like an equinex provider, but I don't really think they're very old school. So, but if you were on bare metal, you could hypothetically have a crossplane provider that you can sort of mix and match there. Or you could imagine, like having a, you know, a provider that just goes and runs Terraform or CloudFormation for you if you wanted to. So this kind of gets back a little bit. I think one thing that crossplane has really focused on is thinking about the separation of concerns between sort of like platform teams doing part of the job and then sort of the consumers of sort of platform teams or the customers of platform teams doing another part of the job. Is that a dynamic choice that you see for conflict connector users? Who are the main people who you see using conflict connector? Is it sort of mostly SREs or mostly sort of SREs sort of just offer conflict connector as a service so that the average application developer, the person who writes deployment might also write a cloud SQL instance or something like that? Yeah. So we identify our customer, our target customers are like platform teams and platform administrators who are like designing, building and setting up their centralized infrastructure. And I think those are our main customers and they might create a general infrastructure for the entire organization or they might create infrastructure on demand for the application teams. So basically we consider those are our target customers. On the other hand, application teams can definitely use us but I guess it really depends on like what they, how they like design their infrastructure or how they design their own workflow but from our perspective like platform and means are our target customers. Makes sense. And I think I've seen, I know that Google has some examples on GitHub of building abstractions on top of cloud, sorry on top of conflict connector types, but Google typically seems to prefer to do those abstractions client side, right? I think I've seen like this tools like KBT, the way there's some examples of how you can have some use KBT to render out some tools. Is that sort of something that maybe would be more focused to application developers rather than platform teams? Yeah. So I think like CAPT is part of, is definitely like a new thing that is used to integrate with other toolings on the like Kubernetes ecosystem. The thing about like conflict connector is it's used to mainly manage the like manage the control plan of the resources. And for a lot of the application use cases, they might be focused on more on data plan. And so that's why we consider like managing the infrastructure is the main use case. For example, we have examples of creating resource hierarchy, which is basically create the organization, create the folders, and create different folders for different teams, set up I am permissions and set up networking connections. So those are actually something we find the conflict connector will be really useful, because it's providing the consistent template consistent to user experience and similar infrastructure can be set up for different teams with several configurations by variable supported by CAPT. So also, so the entire workflow can be simplified and like, yeah. So I want to sort of ask a simple and very into this question to Matt, because one whenever I think about managing what we in Crossland called high fidelity, or maybe a more common way to put it would be sort of the granular custom resources that map to a cloud API.  And in some cases, looking at UJ, the API for an S3 bucket is like 25 different API calls, because it is a branch. 19 separate API calls to update a bucket attribute. Anyway, go ahead. But so Matt, I always use it as an example here because Azure does something that I think is really good where when you create, as far as I'm aware, when you create postgres SQL server, for example, it will usually, it will not allow any traffic and then you have to go create some firewalls to either allow virtual debt or just an ID range to access it, which is, you know, secure by default, which is good. But then it changes the problem from just going to give me a database to give me a database and configure some network security and you know, potentially make a virtual network or something like that if you need to. So is this also something that the ASO uses are sort of pretty comfortable doing their sort of people who are infrastructure types who are using this? Yeah, so for the most part, we, you know, people who are using ASO are, as you say, exactly, you know, they understand what resources are being deployed and they sort of are like, if not platform administrators, at least they're bridging the gap between sort of a platform. They're like the platform admin for their team, right? So we have some customers where like they've set up ASO and they're using, you know, Kubernetes RBAC and stuff such that they're delegating platform administration to an individual team within their company and each team sort of does their own platform admin and then the ASO is maintained by the core team, but they don't concern themselves with what a particular team in their company is deploying. They just give them, you know, sort of carte blanche to deploy whatever they like, as long as it's, you know, in their namespace. And so yeah, the general case is that customers sort of understand how to configure these things. And one of the things that they like about sort of the operator pattern and ASO is, hey, like, I need to create like six of these things or whatever, as you say, like the Postgres server, a DB, some firewall rules, etc. And it's really nice in Kubernetes to be able to do that by just like applying, you know, six, six resources all at the same time. Like you don't have to worry about ordering. You don't have to worry about like, does this come before that? It's sort of just make it so. And then you wait a little bit, and then it's all there. And so that's one of like the big selling points, I think, of not just ASO, but sort of the general pattern that Kubernetes is espousing is like, hey, you don't have to concern yourself with, you know, Jay mentioned, there's these 19 APIs, but presumably like, you don't have to think about that when you're actually interacting at sort of the CRD level, which is a big win. Exactly. Change the spec and let the controller deal with all the ugliness behind the scenes, right? And which S3 update attribute API call it's going to make, you know, hide all of that implementation, that imperative implementation goop in the controller and allow the developer experience to just be one of just tell me your desired state. That's it. So Jay, speak, I'm not 100% sure about this, but I was reading a few weeks back now on the on the ACK GitHub repo, and was doing a bit of research myself to answer this, like, who is the target audience of ACK, and was I correct in thinking that ACK also has a bit of a bent toward letting people build higher order things on top of ACK stuff? Like, is there a thought there of like, hey, someone might want to build their own controller and pull in some ACK controllers, or did I today sort of not quite get that one? You mute it, Jay. Jay's experiencing technical difficulties. So let me while he figures that out, let me let me roll on back to back to Dan. So now sort of it's interesting to hear that sort of from from ASO and conflict connectors perspective, we're mostly seeing platform teams or SRE folks being the people sort of doing the end to end, spinning up the infrastructure sort of thing. Dan, Crossblade sort of has this approach of separation of concerns. Do you want to give me a bit of context around what Crossblade does there? Yeah, absolutely. And, and honestly, it's very compatible with what both Matt and Joyce have said so far in that the actual granular, as we call them, managed resources are things that we don't expect, you know, non-infrastructure admins to interact with, because they're generally pretty complex, right? And understanding how they work together requires a fair amount of background knowledge. And that really informs the composition model, which I was alluding to earlier. Another thing that is kind of a maybe the most controversial topic, I'll say, between Crossblade and some other projects is the cluster scoping of those managed resources, right? So all of these managed resources, the granular ones exist at the cluster scope, while most other projects, whether they're, you know, managing infrastructure or doing other are at the namespace scope, right, for isolation and RBAC privileges and that sort of thing. The reason why Crossblade takes that approach is because the composition model allows you to define that abstraction and expose it at the namespace. So then you're able to put the permission, permissioning, you're able to raise it to the level of abstraction, right? So you're able to say developers interact with this, you know, friendly interface or this friendly resource that has very simple fields that map to things that we want and enforce policy we want. And we can govern their ability to do that at the namespace scope. But then it renders out those resources at the cluster scope. And that's where the infrastructure admins do those kind of operations with, which config connector and ASO, you know, are providing them the ability to do as well and actually control those granular specific resources. Thanks, Dan. So we've still lost Jay. I'm just going to, oh, here he comes.  You didn't miss too much. Zoom crashed. Sorry. All right. Well, let's keep rolling on now to some of the sort of maybe more interesting to those of us on the call, at least technical stuff. Matt, what would you say is sort of the biggest challenge of building out something like ASO, either technically or logistically organizationally? Right. I mean, I would say that these two things sort of go hand in hand. It's consistency and scaling to the number of resources that there are in the cloud, right? And the more you try to scale sort of manually or by throwing more people at it or whatever, the harder it becomes to sort of maintain that consistency, as I'm sure everybody on the call is familiar with. And so it's this constant tension between like, how fast can we go? How many things can we support? But then are we giving a uniform experience? Are we giving, I think Dan mentioned earlier, like it's the same, I want to be able to look at the right properties and see state. I want to be able to do references the same and all that. And so I would say like, that's one of the biggest challenges is that constant tension between these two things. And of course, the team that we have working on ASO is just it's such a small fraction compared to the team in Azure that's building new things and shipping new APIs, that if you look at it from sort of a person hours perspective, there's just no way that like five or six of us could ever keep up with however many, you know, like thousands of developers, there are producing new APIs in Azure. And so we've got to come up with some way to do something about that disconnect. And that's one of the big challenges. Joyce, I know I've spoken to people in the Config Connector team and heard similar things. Do you agree that that's sort of the biggest challenge or is there something else that you'll find trickier? Yeah, I completely agree. Like consistency is always a big problem or big challenge, I would say, as it comes together with scaling, because the more resource you support, the more edge cases you were running to, and the more features you need to support to cover those edge cases. So it's really hard. The lucky part for us is we are built on top of declarative libraries, which they communicate with service teams directly. And we work closely with the declarative library team to make sure like they feature or the resources they support aligns with our customers requests. And then they can probably delegate their own request to the service team. So it's like there's a layer in between us and the APIs. So Dan, what's what's this like as a maintainer of crossplane who is, you know, we have almost the same problem, but another layer up I think, right? Yeah, that's exactly right. I was going to just respond with hard and next question. But it's definitely very challenging. All the things that have already been mentioned are applicable for us as well with the added part of we we are not at these companies, right? So some some internal things we don't have access to, of course. And, you know, we also well no one can right now but theoretically in a in a future time when we're not in a pandemic, be able to walk down the hall and knock on someone's door and ask them how an API works. That being said, we definitely had awesome partnership with the folks on this call right here, and being able to share some of the generation of kind of the manual steps for creating these resources. So specifically with AWS, ACK and provider AWS for crossplane currently share a code generation pipeline, which definitely helps accelerate both projects as you know, they have the context for, you know, understanding how these API is working intimate level, and we're able to contribute back based on our experiences as well. So that's definitely a factor. And then one thing we haven't touched on, which I definitely see as a as a open source maintainer as a big component is community, right? When you get more folks working on things, you know, whatever their background, it's a huge benefit for for driving progress. So especially since crossplane has joined CNCF and had its one dot a release and that sort of thing, we've seen a lot of folks just from, you know, all of the world, all different backgrounds, all different companies come together and say, you know, I'd like to learn how to do this because it it scratches an itch that I have. And having that kind of open community definitely supports that and and you know, hopefully those folks that come along and contribute can also grow in the community and take on larger leadership roles and take something away from contributing as well. Yeah, that's that's kind of one of our big approaches to scaling is scaling the community to scale with the amount of work of building building this stuff out. The whole consistency thing is really interesting to me. I think, you know, we talked about why people like to use Kubernetes to manage their infrastructure. And I think part of sort of what we touched on there was roughly just tooling consistency, they're probably already using Kubernetes to do other stuff to manage their applications. But I think part of it is because people like the Kubernetes resource model, the KRM, and I think people like it because it's a very consistent predictable API. And I think a lot of what we've all sort of seen, you know, cross-plane, we call it the XRM or the cross-plane resource model, where we're just, you know, a super set of that, we're just saying, Hey, what if we make this more consistent and more opinionated than people like, Oh, we like that. But then there's also an interesting thing where hypothetically you've now got, you know, when we're when we're when someone comes to write a new cross-plane provider, there's different layers at which you could ask them to be consistent. You could just say, Hey, you can go write this provider in Erlang for all we care, just have a consistent API. And there are trade-offs there, right, that you need a big enough community for that provider where everyone's going to be coming in learning Erlang potentially or like using a different tool set, building new libraries, all that kind of thing, but sure. And especially if that's open source community, we might want to have that flexibility. Or you can also, you know, take the approach that we've come closer to taking is like, we actually want you to build providers mostly in a specific way, using go, using these libraries, etc. etc. I'm guessing sort of, I know Joyce, you said that you've got sort of DCL, which, which I believe is like a go library ride. Jay at Amazon is everyone, do you think you're going to get everyone sort of pretty consistently building control is just using the ACK style sort of thing, do you ever have issues with people saying, No, I really want to go do this with Java or something like that? No, we don't really have any issues with that. I think it's our, our biggest challenges are primarily people based, right? I mean, ACK is a set of service controllers, right? The one for each of the service API, so one for S3, RDS, etc. And each of those services is backed by people at AWS, right? And those service teams, some have a very long history of doing things their own way. Some have a very short history and are like very comfortable working in the open source community. Some aren't. So yeah, it's been a challenge bridging across those different service teams for me personally. I mean, it's a challenge I welcome, but it's definitely been a challenge, right? Because I have to sort of explain how Kubernetes works in addition to, you know, like, hey, what are our needs from an ACK perspective? We generate the controller, right, for the, for the different resources in that service API. And then we work with the service team to build tests and that kind of thing. So yeah, the variety of knowledge about Kubernetes and container ecosystems within the service teams at AWS has been a challenge. I'll just put it that way. Yeah. So we're pretty much at times, but just to wrap us up, I want to, you know, we've been talking about technical challenges and things like that. I want to just do a quick round the room and I'm going to ask everyone just to give me a, you know, one or two sentence answer of what are you most excited about on your project at the moment or like what's most interesting to you if it's not a feature, a technical challenge or something. So start with Joyce. What's, what's excited you the most about Confit Connector at the moment or what you're working on? Oh yeah. So I think the most exciting part is like customer can actually use Confit Connector to manage Google, manage GCP resources, like native Kubernetes resources. It's like, like I see, I've seen a lot of, like customer calls about that. And it's also amazing that we can keep full fidelity of those unaligned GCP resources. So those are something I'm really excited about because that's exactly the goal of Confit Connector. And I'm so excited to see customer actually feel it. Yeah. Yeah. It's really exciting to see this sort of vision that we've all been working on, like becoming real and being used in production environments. How about you, Matt, similar answer or do you have a different take? Well, one of the things that we're sort of incubating in ASO right now is Azure has ARM and it has specs for all of the management plane operations are in OpenAPI. And so one of the things that we're working on is, you know, I mentioned the scaling challenges and the consistency challenges. And we're sort of pretty close to an alpha now where we're planning on generating, as sort of Jay mentioned, AWS is doing as well, the controllers for all of these management plane resources from these OpenAPI specs. And so I'm really excited to sort of hopefully be able to knock down one block on that challenge of, hey, consistency, scalability, we've got this thing that we can generate using OpenAPI. And actually, you know, like 90 or 80% of the Azure resources will just be able to code generate. And then, of course, we still have to solve testing and all of these other problems. But at least that's one big step toward a consistent experience across a large number of resources that doesn't involve tons and tons of manual effort and, you know, people massaging each and every one. We're sort of lucky in Azure because there's that uniform way to deploy through ARM and all the specs are there. So it's just a question of writing the code to turn it into Kubernetes resources. Nice. And I want to give a shout out one of the things that I'm most excited about in the novel space in general is Matt and his colleagues over at New Zealand put together a little proposal on how to do versioning of Kubernetes resources relative to cloud resources, which is really well thought out, really, really great read. I recommend it if you're a note about this stuff like me. Jay, how about you? Well, I think there's two things I'm most most excited about. From a user perspective of either crossplane users or users of ACK individual service controllers, I'm really excited about bringing a consistent way of interacting with some AWS service resources using the Kubernetes API machinery and Kubernetes resource model. I think that that consistency has been asked for for a long time by application developers who prefer Kubernetes. So I'm super excited about that and just the consistency in general. The other thing I'm really excited about is actually using ACK as a mechanism to bring the world of open source development to internal AWS service team engineers. Not only the Kubernetes ecosystem and sort of like how that works and CNCF projects like crossplane, but just generally getting some service team engineers at AWS out of their shell and using sort of like more upstream open development tooling and build tool chains and just getting them comfortable with contributing to open source projects. Like personally, very important to me. So yeah, that's what makes jazz is me up, gets me in the office each day. Yeah, I can totally empathize with that. I'm not there in any way bad at open source, but I spent my first five years, six years of my career in Google and there was a bit of a bit of a culture shock to leave Google and be using the open source equivalent of all of this internal tooling sort of thing. So I totally get exposing the average engineer to like more of how stuff's done in the outside world is always fun. Dan, and last but not least, what are you most excited about with crossplane? So this is I guess a bit of a deviation from the other answers. But one of the parts of crossplane that doesn't get focused on too much, but I'm particularly kind of partial to is the package manager, which kind of has some opinionated workflows for how you can install and manage and version CRDs and update packages and roll them back and forth. And in a proposed kind of extension to this in the future is the ability to support essentially functions. So right now we have providers which are basically controllers and CRDs. We have configurations which are just YAML objects that get installed into the cluster and then functions could really lower the barrier to entry for folks authoring some of this kind of specific functionality. So you could imagine day two operations or disaster recovery or migration. Those could all be things that could be packaged into small functions. And there's a lot of innovation happening in the function space and the Kubernetes ecosystem and also more and more folks are using to moving to things like Lambda and things like that. So providing that interface to interact with this kind of ecosystem that we've built of cloud provider resources as Kubernetes objects is something that I think could be a really powerful model. Yeah, super excited about that too. All right. Thanks everyone. I think we're pretty much at time. So I'm going to stop the recording now. Thank you all again.


Everybody for us here rip in 2021 NT football hobby 4 box case number 141 random divisions Let you check out the results right here. There's a link in chat for them as well You can always find these late in sports cards.com. All right Let's see a monster Yeah, yeah happy that was my knuckles Everybody heard it Cringing Sorry guys NFC South Michael Vick 19 of 25 AFC East fits magic 95 out of 99 getting just totally Short I forget that play man crazy no look pass AFC East patch card NFC South Calvin really two out of 49 Patch card AFC West Mike Williams 11 of 49 Dual relic Ravens AFC North Lamar Jackson Rashad Bateman 35 out of 99 AFC North rookie dual patch Rashad Bateman 32 of 49 nice Bateman booklet NFC East rookie jumbo prime signatures 68 of 99 Cadarius Tony Nice book there NFC East Prodigy patch rookie auto Tylen Wallace AFC North three out of 99 for color Ravens patch Solid star of the AFC North with some relics and this hit here AFC West a legend hit Dante Hall auto 15 of 25 Dude was fast faster than fast NFL gear AFC East rookie ball and patch auto 425 Mac Jones Mac oracle We need you back Mac We need you back. I'm out for you. Yes for colors Black, purple, gold and white Sick Mac silver NFL gear dual relic auto Kyle asked with AFC's congrats on that Tylen Smoochie Wallace. All right moving along moving along AFC North Mark Andrews 21 of 89 AFC South James Robinson 52 out of 99 Jersey card NFC East Antonio Gibson 28 of 99 AFC East Hall of Fame patch Jason Taylor 13 of 25 player Warren Taylor a AFC East rookie patch card 7 to 25 Elijah Moore AFC East again a Mac Jones just like that Elijah Moore 88 of 99 for colors NFC East cross over RPA. What a patch on that Kenneth Gainwell 23 of 99 That's like that's about a four color. That's like black gray green and white is about a four color Nice one NFC East beautiful passion that crossover FC North Nick Chubb auto 45 or 49 Four guys getting all these yards and he's got to play for the browns Wow, that's nice NFC North auto Barry Sanders 2 of 99 nice Barry Yeah, he should be back. I think next week. Maybe Rory. I think he's got a chance at least True RPA AFC East crushing it now Michael Carter 83 of 99 Nice true RPA. We'll get that mad. Dang man. All right Time for someone other than the AFC East to get a nice RPA or something Spread the wealth Sub liquid What up liquid? Yeah, they were selling off broken ankle season over but you never know man Who's the guy they thought towards ACL? It turns out. It was just like a bone bruise They're just overly cautious these days. They don't know. Oh, it was Wilson They thought Zach Wilson towards ACL the preseason and it was just like a Knee sprain or something. Yeah. I get it. One of you are really cautious. Everybody gets worried now. It happens so much so NFC East Alice Goddard 47 of 88 NFC West treasured Dwayne Asgridge rookie 89 99 scratch AFC North Jersey T. Higgins 8 of 99 dirty patch AFC West Austin Eckler 8 of 25 Three color with a grass stain. So four color, right? White got grass. You got yellow and blue Nice Eckler Jersey card AFC West this time. It's Justin Herbert 74 of 99 rookie Jersey NFC East Micah Parsons 75 out of 99 Yeah, there is redemption on the back could be a couple of people a few actually NFC East rookie Otto Gary Brightwell 3 out of 30 of 25 NFC West crossover RPA 2 2 at well 99 of 99 bookend Marshall Lynch Skittles patching pretty cool 8 color patch. Oh That's pretty cool. AFC West patch Otto Tim Brown one of 25 Tim Brown legend Definitely a top-ten receiver all time Maybe top five depending on who you ask So good Notre Dame legend to and redemption NFC South crossover RPA Kyle Pitts Nice hit NFC South They need to throw him of the ball. Oh, wow, that's weird. I Did yeah, good call now. Look at that Eckler. There's a Ram's logo on it It's a Chargers card. We all know it's Chargers. He only played for them. There's charge everywhere else That is so strange. Make sure you tag this. I've never seen that. I'm gonna photo of that AFC West I guess I Did I didn't notice that That's pretty wild says Chargers everywhere just has the wrong logo Herbert was fine. I guess it's just the Ecklers then Oh Good call keel. Good call Her card, I guess Yeah, I just wanted to show the back. I wasn't paying attention to the logo. I Don't even look at stuff like that. I'm just looking at like Like if there's damage basically like I'm checking corners and stuff like that Okay, moving on here. We've got a Darren Waller 65 and 99 AFC West printing plate AFC West Coral and Sutton one of one my guy They're actually having a really good season. He's like the one bright spot on the Broncos offense I thought the Padres putting up seven on Scherzer was surprising other than that. I don't think anything really surprised me too much Luis Castillo was fantastic. It's not super surprising, but he was amazing Debo and I uke 39 and 99 AFC West I think Scherzer was the only pitching performance that wasn't good. I mean, I know it was okay One of one Nike swoosh laundry tag Ronald Jones the second NFC south rojo patch Rojo You go NFC south, you know, it's better than one one of one two one of ones Laundry tag patch Dan Marino AFC East AFC East again Nice Dan Marino will mag that AFC North dual jersey Jamar chase Evan McPherson 46 of 99 two great rookies right there Yeah, three one of ones including the printing plate Crossover RPA Kellan mon three out of ten NFC North. It's a good box for parallels Check his next one out Midnight RPA NFC West Rondale more 17 of 20 Sick hit NFC West Rondale magged a Patch auto of Debo Samuel for the NFC West 75 out of 99 and on the back dude AFC East Absolutely crushed this break Congrats again, Kyle S. You got a prodigy patch RPA Jalen waddle 98 of 99 three colors on the patch Unbelievable AFC East man You need some shipping insurance Pretty good case pretty pretty good case Give me one second. We're gonna recap That's the end of group breaks tonight everybody. We've got person's coming up and they are full Alright, so we had a FC North dual jersey chasing McPherson NFC West Debo and I youk Eckler patch Chargers 25 game worn It does have a Rams logo, but it's obviously chargers. There you go Chargers AFC West an error card that noticed it before Micah Parsons NFC East Jersey card AFC West Herbert is he his is correct to 99 AFC North T Higgins to 99 AFC East Elijah Moore and to Mack Jones AFC he's also had a Jason Taylor NFC East Antonio Gibson AFC North Rashad Bateman and Jackson and Bateman NFC South Calvin Redley AFC West Mike Williams checking all the charge cards You had an RPA book Kaderious Tony NFC East 68 of 99 Debo it's a 99 patch auto NFC West crossover RP at a 10 Kellan Mond really nice Main plate maybe Maybe just a rookie Premier Jersey. Honestly, I might just say rookie on it. I don't know Three to ten nice one there for the NFC North Tim Brown AFC West patch auto to to at well crossover RPA AFC West NFC East Gary Brightwell auto NFC North Barry Sanders AFC North Nick Chubb crossover RPA gained off for the NFC East AFC North Tyler Nuala's prodigy patch auto AFC West Dante Hall auto to 25 So I think Eric are pretty cool for the most part Stuff like that. They're makes them really unique Honestly might add value to it NFC South Kyle Pitts crossover RPA redemption AFC East RPA Michael Carter NFC West Midnight RPA Ron Taylor more 17 of 20 One of one swoosh laundry tag Ronald Jones NFC South one of one laundry tag AFC East and merino The rest for the AFC East actually prodigy RPA Jalen waddle 98 of 99 And an NFL gear dual material auto Mack Jones for 25 on the silver parallel That is gonna do it. Thanks again everybody Personals coming up


So I'd like to welcome you all. We're about to have an interactive session with Sophia Erickson from the Department of Foreign Affairs and Trade and Dave Green from Clear Horizons, facilitated by myself Kim Vessely with Zoom support from Robert Grimshaw, as co-combiners of the Australian Evaluation Society's Queensland Regional Committee. I'd like to begin by acknowledging the traditional owners of the lands on which we meet today, the Yagura people and the Turbol people as the traditional custodians of Mi'anjin, the lands on which myself and Robert work, but also to the non-Indigenous representing the land on which Dave and Sophia work. And I pay my respects to elders past and present. I extend that respect to Aboriginal and Torres Strait Islander peoples here today. This session will explore the role of DFAT standards that they have played in efforts to improve the quality and use of monitoring, evaluation and learning in the Australian Development Program over the last decade. The session will explain how the standards came about, how they are used on a day-to-day basis and their strengths and limitations as a tool for evaluation capacity building. The Australian Government's new international development policy, performance and delivery framework, commits to continued improvements in monitoring, evaluation and learning practice, underpinned by the Department of Foreign Affairs and Trades, design and monitoring, evaluation and learning MEL standards. The standards set details and expectations for key products in the programme cycle, such as theories of change, MEL plans and systems and evaluation terms of reference, plans and reports. First, we welcome Sophia Ericsson, who is the Mel advisor in DFAT. Sophia has worked in international development for 30 years and for the last decade has focused on monitoring, evaluation and learning. She has worked for both Australia and Sweden's bilateral development agencies at headquarters and on postings including China and Papua New Guinea. Sophia was an integral part of the evaluation capacity building programme delivered to staff in Indonesia. This programme developed and piloted the design and MEL standards that are still used today. Most recently she was involved in the refresh of the standards and is now building up internal DFAT capacity to use them. Sophia is passionate about getting staff involved in the use of MEL. Next, we welcome Dave Green, who is a lead principal consultant at Clear Horizons. Dave has worked on Australia's aid programme in the Pacific and South-East Asia for over 15 years, across the public, not-for-profit and private sectors. And he has a deep understanding of DFAT approaches to aid, design and delivery. His experience spans investment design, monitoring and evaluation experience across diverse sectors including education, wash, health, governance, community-driven development, civil society strengthening, impact investing, gender-based violence and at the country portfolio level. He has developed strategies, MEL systems and performance reports as well as driven systematic efforts to institutionalise portfolio-wide MEL capacity. Dave is passionate about improving international development practice by providing credible and useful insights to policy makers and practitioners. Before we get started, just a note on housekeeping. You can post questions to the chat box. We will respond to intermittently at the end of each presentation. All other questions will be answered at the conclusion of the presentation during Q&A. Please stay muted as necessary unless you're asking a question. Please keep cameras off if not asking a question. And lastly, please advise us who you would like to respond to your question. This session will be recorded and uploaded to the AES YouTube channel in the coming weeks. Today's session will start with Sophia before moving to Dave. Over to you, Sophia. Thank you, Kim. Thank you, everyone, for participating in this session. Both Dave and I are really excited to talk about this subject as you might have gathered from the introduction. And first, I also would like to acknowledge the Ngunnawal people, Lusland, Dave and I are meeting you from and pay my respect to the elders past and present, as well as acknowledging any First Nations peoples in this presentation. So, Robert, if you could have the next slide, please. So I just wanted to set the scene a little bit because if you are involved in the development sphere, you will be aware that we now have a new international development policy launched a couple of weeks ago. And that policy makes a number of commitments around increasing monitoring and evaluation and learning across the development program. And it's very exciting because, of course, we have been doing this for a long time, but this policy is, I think, increasing the ambition on MEL. And I have a favorite quote already, effective monitoring evaluation and learning are critical to achieving results and ensuring our programs continue to innovate, improve and reflect best practice. So for us, it's very exciting to see words like that in the policy. Next slide, please. And these commitments, I think, are there because not just because we need more MEL in international development, but it is also in the context of the government wanting to increase evaluation skills and use across the Australian public service. So in this group, you will be aware of the 10 million over four years commitment to set up an Australian Center for Evaluation in Treasury. And this is also reflected in DEFED, of course. So I think we are very fortunate in the international development sphere because we have worked on this and focused, even though we can always get better. But there's also more interest across other areas of DEFED, like foreign policy, trade, and passports. Passports have been doing monitoring and evaluation for a long time. You might have seen during COVID when they were a bit hammered and in Senate estimates, they would have talked about how they monitored and dealt with those issues. We also have a lot of internal, there was an ODE Office of Development Effectiveness Evaluation published in 2018 that looked at contractors monitoring and evaluation systems and the findings there were that we can become a lot better. Next slide, please. So also just to set the scene a little bit is that when we look at performance reporting, it's different levels from the international development sphere. So the policy has a performance assessment framework. It's available online on DEFED's website, which has determined, I think it's 32 indicators that investments are reporting towards. We're also developing, it's starting now to develop, let me get the name right, Development Partnership Plans, which is sort of like a country strategy, what we want to achieve in each country and also regional programs. These will also have their specific performance frameworks. We do investment progress reporting, which is annual self-assessments, and all of these reportings rely on good mail systems at the investment level. I would say 90% of the information to feed into these overarching systems would be coming from individual mail systems. And I should also say that in DEFED, we as a DEFED investment manager, the role of us in terms of mail is to buy quality assure and use mail. It's contractors that deliver both mail systems and reporting. And also when we do independent evaluations, it's done by contracted in evaluation specialists. So I'm going to hand over to one of them, which is Dave, to talk a little bit about what happens at the investment level. Over to you, Dave. Thanks, Sophia. And hi, everyone. Yeah, it's just a quick note, I guess, for those who haven't worked in international development. And I'm still really getting my head around the difference between how this works domestically versus internationally for Australia and for Australian evaluators. But the bottom line is that most of the, as Sophia said, most of the mail expertise and the mail resources sit at the investment level. So at that country program level, it's really mostly just monitoring a set of performance indicators. It doesn't usually go much further beyond that. But at the investment level, as shown in the standards, there's a fairly expectation of a fairly comprehensive approach in cooperating, monitoring, evaluation and learning. And the resources and the expertise to do those things usually sit within a DFAT implementing partner. So that could be a contractor or it could be a multilateral partner or an NGO. And that team of mail people may sometimes be commissioning evaluation work, or they might be doing that themselves. But they'll also have a pretty heavy load of monitoring and learning and reporting work to do. And then, as Sophia said, there are independent evaluations that are commissioned by DFAT directly. But I think compared to domestically, I would say that those evaluations are generally shorter and have smaller budgets than you would see in some sectors domestically. So hopefully, I just give you a bit of a picture for those who are new to international development of how the space is kind of resourced. Back to you, Sophia. Thank you. And I also just wanted to point out that, as you see the last bullet point, because one of the things also in the new development policy is an increased focus on accountability and transparency. And of course, that also is good news for us that are interested in mail because the information stems from all our mail systems. So it's all exciting news. Next slide, please. So what we wanted to do also is to give you a little bit of history, because when we talk about building capabilities inside organizations, so this work, I think, started 15 years plus ago. And it was sort of within an Indonesia program. So it wasn't sort of at the corporate level of DFAT. And we wanted to see how we could build up the capability of the staff within that area. Because at that time, this was, I think, the largest program that Australia funded overseas. And what was needed to do this was definitely to have mandate from senior management that they were really interested in performance and performance data and gave space for people to build their capability. And then what is also needed is the right expertise. Because I mentioned earlier that all the mail expertise really sits outside of DFAT. So our role within DFAT is to identify by quality, sure, and use it. And in our case, we were really lucky because we had Dr. Sue Dawson, who had already worked with Osaid and DFAT and worked on evaluations and supporting mail systems and really investigated what was not working well and how that could be improved. I was extremely lucky because I came back from a posting and slotted into this severe, because it is really important to have someone inside the organization that can network and see what's going on. Use the political, economic sort of who is your with you and who needs to be nudged to work with you. And also to checking in on the mandate, so it's still there. And then the final thing that we definitely discovered is this systematic approach that needs to be there because a lot of organizations that I've been involved in, it's you do a lot of training on M&E, but you don't really see any changes in terms of capability. So in this case, when we talked about a systematic approach, it was in the early days, we didn't obviously have the standard. So it was working through templates and supporting people having help desks. It was coaching and training on the job. So more looking at what people actually needed to be trained in relationship to the job rather than two years before they were going to do an evaluation, for example. And in this work, it's over time, Sue Dawson looked at internationally, looked at standards. The Canadians were ahead on this and came up and wrote the standards that we have built on since then. And it was really appreciated by staff and also even though we didn't focus on the implementing partners, male people that worked within those organizations also really liked having the standards because it sort of will talk more about it later, but it sort of made it easier to have a framework that people could agree around. I think that is what needs to be said about history, but I could ask Dave because he was actually part of this history before he ventured out of DFAT. Is it anything you want to add here, Dave, that I've forgotten? I used to work for AusAIDS at the time and I was on the Timor-Leste program and so Timor along with some other countries, I think Vanuatu and maybe one or two others replicated or adapted this idea of an evaluation capacity building program that Indonesia was implementing to other country programs. So it was a real sort of diffusion of this idea across various at the program level within AusAIDS and it was multi-faceted. So in addition to what Sophia was saying, there was also usually an effort to build in-house capability to deliver some of the training, for example, that SUD Awesome was delivering at the time. So it was a pretty comprehensive approach. It was picked up to some degree by different country programs, but then organizationally AusAIDS adopted the standards, but not the ECB program. Obviously the standards were a critical component. Thanks Dave. That is a very good point. So next slide please. So the design and mail standards that we are now using, they were refreshed. So not new standards and then published in December 2022, which was quite timely because it was 10 years after they were first published. And I had actually been away for about seven years overseas. So coming back, so I left when AusAIDS existed and came back to DFAT. So I was extremely pleased to see that the standards were still there even though ECB was not implemented as it had been. With these standards that we have now, we have added some standards around concepts, which is a tool that DFAT is using to make a case for some programs that we want to implement in countries. And then that concept is made into a design. And we also have a standard on program logic. And in DFAT, we sort of talk about program logic, slash theory of change. We do not differentiate between these terms like others might. So the good thing for us to have a standard is because of, as all of you know, the terminology in mail can become quite difficult to, especially when you work overseas also. So at least we are trying to have a terminology that is consistent within DFAT and with our implementing partners that are contractors. But it's still a work in progress. We really emphasize that this is not a checklist. And I will come back to that a bit later in this presentation. And we also try to focus on that mail systems are user focused. Everything we talk about today, Dave and I are working progress because it requires a lot of behavior change on all parts. But it's no point creating mail systems that are too complex and complicated and then not very user friendly. So people don't use the information for decision making. When we refresh the standards, we had what we call a brains trust. So we invited some of our mail experts that work regularly with DFAT. I was lucky because I had just returned. So I could sort of provide, which is quite unusual, but I could pry the corporate history here. And we also had some of our local staff that had been involved in the previous years to participate. And that's then what became these refreshed standards. Next slide, please. So this is what it looks like. You can find these standards on the DFAT Internet. So the highlighted in yellow are the new standards. And then from five to 10 are refreshed from the previous versions. And the good thing for us now is that the standards actually cover the whole program cycle. And it has been also, it's a bit clearer in terms of mail, where what the expectations are at different stages. So that there are certain expectations for mail at the sign stage. And then at six months, we have, we expect a plan, which is basically a paper product. And then at 12 months, that plan has been operationalized, which we then call that it's become a system. And as you see also standard eight to 10 is focused on evaluations. So number eight, which is about terms of reference. That's very much an internal that DFAT needs to be in charge of writing the terms of reference. We know what questions we want to ask. And then nine and 10 is to be able to assure quality and discuss with our evaluators what the expectations are. Next slide, please. I just wanted to go through a little bit on. So as Dave said, we do not have right now, it might come on board again now when the new policy has a bigger focus again on mail. But we do not have a formal evaluation capacity building program. But what I'm trying to do in my humble little way is to try to replicate a little bit of what we did over 15 years ago. So what I emphasize to staff within DFAT is that the standards have been in place for a long time. So it's not something new. I know that our contractors have used them and mail experts have used them over all these years. Really emphasizing that it's not a checklist because this is a tool that we really want to encourage dialogue and discussions around the standards. It's also not something someone else does for you because what is could very easily and has happened is that a staff member gets a mail plan. They send it off to another expert to check the quality and then they get back the quality assurance. Then they send it off to the contractor and the contractor responds and then the mail plan doesn't really become more than a paper product. So it's really important.  I'm happiest when I get people calling me up. And the other thing that I do talk to people about is that to get good mail systems actually working and producing credible information that we make decisions on requires behavior change. It requires behavior change within DFAT with all our investment managers to really engage with the mail system and use the information and discuss it with partners. But it also will require change on our implementing partner side because if we haven't demanded in a consistent way what we want then it's very difficult for contractors to respond and not to be mean but sometimes they also are in old habits and respond in a way that they've always done which has not been the best mail systems in that we would have like to see. It is included in all contracts. So these are implementing partners that are contractors. They have in the contracts that they have to apply the standards and what is going to be a little bit new for people is that with the new policy in the performance framework we have something that we call tier three indicators which are indicators that assess ourselves and how we work. And we are going to have one of those which is that the mail plan is assessed against the standards and I'll go through a little bit what that might entail from my end. And we have said that we want to see mail plans at around six months and this is also to encourage that this doesn't slip away because in my all the years that I've worked on with in both Osaid and DFAT it's very easy that the mail plan does not get done and then it starts slipping and then you're halfway through the program before it's an agreed system that is actually in place. So this is also something that is in that behavior change box that we want to see this move in a more reasonable way than it has been up until now. And finally what I also really emphasize is that this having the standards and it's not a checklist it's really important to have a dialogue and create a partnership because without the implementers we have no information whatsoever since it's all done by our partners. So it's really really important to have a very good dialogue and create that trust with each other also because if you have that trust you can tell people when it's not going so well and actually learn from that. So I'll go through a little bit next slide and apologies because this is very small text but it's just to give you a little bit of an idea what an assessment of a mail plan that I do can look like. So here you sort of have the upper part of the standard standard five of the plan next slide please and then I've created this little tool for the plan so basically just added a box underneath each element which includes if a standard has been met partially met mostly met not met or not clear. And this might sound like a lot but what I discovered when I started pilot the standard is that to be able to create a dialogue you actually have to differentiate a little bit on what you are assessing because if you go hardcore and say that nothing is met which could happen then of course it will be very difficult to start a relationship and partnership so from my horizon it's better to have possibilities to differentiate a little bit broader on when things are mostly met or partially met for example. They're not clear is also very good for staff that are starting out on this because these standards should be possible to use by non-technical people and of course some of it is more technical than other parts of it but if you use not clear then you can have a dialogue and your implementing part can then explain to you what is happening and make it clearer and I also have many instances when I have to put not clear so it's a good thing to have there. Next slide please. So again apologies it's just to give you a little bit of a sense so this is sort of what it can look like when the assessment is done so hopefully you can see a little bit that you have partially met and met and then also the important thing here is that you have to articulate how it can be strengthened or improved because this is something in my experience in DFAT is that it's very easy to say I'm not happy but then not that easy to say how are the partner going to do better and so that we we are satisfied both of us so that's a little bit what it can look like and then next slide please and this is an ideal situation which as an advisor you rarely get to because you send off your advice and often don't hear back again but I use this this particular investment as a pilot so I talk to all the the partners that I was wanting to use it as a one of the first pilots and in this case so this is ideally what I would like to see for all future assessments is that the partner came back and said how they responded to the assessment and the reason why this is good is that there might be things here that cannot be responded to at the time the plan is submitted for good reasons and then you can agree between us and the implementing partner when that is going to be done and maybe in some cases hopefully unusually but maybe in some cases it can never be done but then that is agreed and documented this one of the problems we have when we come up to the end of the program and look at what's happened it's not been documented when things have been agreed to not meet standards for example so an ideal world you get a response and you can document and agree that it will be updated in the next annual plan because mail plans should be updated annually context changes program logics need to be adapted so this is a live document that investment managers should engage with so now I'm going to hand over to Dave to talk about strengths and opportunities in terms of using the plans yeah so if we just go to the next slide we've just put together some ideas here some sort of reflections on you know what what appears to be working well with the standards and areas where you know there are limitations or or risks this is really just a discussion starter that you know there's quite a few people in the call I think who have very practical experience with how these standards are being used so really interested to hear you know whether this resonates or what other views people have the obvious you know main strength is that they provide you know a level of shared understanding shared terminology which is a useful I think in international development where you can get a fair level of diversity of practice and you know as we all know evaluation is a very broad discipline too so there's a combination of those two sources of diversity but you know on a practical level when I or other clear horizon staff are working as the part of a mel team within and within an investment within an implementing partner you know often a new program are we starting bringing together a team that hasn't worked together before or coming from from different backgrounds and it can be useful to have the standards as a bit of a starting point to to build a bit of that shared understanding and consistency I think also at the sort of policy level you know like there's an interesting discussion going on now around impact evaluation and the role of RCTs in the mix of evaluation methods in the Australian government I think you know within DFAT it's really helpful for that dialogue to be situated within a the sort of the foundation really that these standards can provide because the standards are relatively agnostic around methods and just emphasize fitness for purpose so it's they provide a bit of a solid grounding I think that you can that can support those debates yeah they obviously have a key role to play in helping DFAT staff to get across their role and perform their role but that really you know that kind of strength is much more likely to manifest if these other sources of support to DFAT staff I think are in place that we talked about earlier like training or coaching or you know support from someone like Sophia internally to help them work through the process and you know just from the perspective of a an evaluator or a male practitioner it's the the standards empower you really to to build buy-in within say an implementing team or if you're part of an evaluation team to to get everyone on the same page about you know the importance of methodology in the evaluation process so it's sort of like a way to empower male advisors or evaluators as part of broader teams do you want to add anything on that slide Sophia or you're muted sorry a classic problem just to add that when I work with staff and contractors everyone really likes to have that framework that it's sort of it's not the opinions of people and especially because there is quite a staff churn on the DFAT side that you have some a framework that people can discuss from which seem to be very much liked by both sides over okay let's go to the next slide so this is yeah the flip side limitations and risks and again you know just questions here to get a conversation happening it's obviously quite a dense document so it can be overwhelming which can lead people unlike us who engage with that detail to disengage or for male to be seen as too heavy particularly if they are treated as a checklist and you know there's an interesting point here to be made about how accessible the standards are for staff for whom English is if English isn't your second language then this can be quite a dead dense and difficult document to get across it's actually interesting that this like say over the last five or ten years even there's been a lot of streamlining of of aid management processes within DFAT and yet these standards have survived that and in some ways have become more detailed which I think probably points to you know their value particularly to to implementing partners but do you have a perspective on that Sophia so you mean because of capability within yeah it's just interesting that a lot of other guidelines and processes have been simplified and streamlined over the last few years but the standards have have withstood that it's just an interesting reflection I guess well I think when we refresh the standards and then because of COVID it took a bit longer before they were actually published between having had that Brainstrust and people within DFAT including some of my bosses said oh there's so many standards so many numbers but if you make it to if you streamline it and and remove it I think sometimes it becomes more difficult actually because then you have to do a lot of elements within each yeah center so yeah sometimes a longer is in a weird way easier maybe it's maybe that's that you've been able to argue for that specificity yeah um yeah so the second question is just whether they are to or that they can encourage I guess compliance oriented approach and this is all comes down to how they're used but as I think Sophia flagged earlier there is a risk that the focus becomes too much on the document and not enough on how the document is used there's also a point here you know it's Sophia's mentioned a few times that it they shouldn't be seen as a checklist but I think that's still a work in progress and so you know you do come across situations where the application of the standards is a bit context blind in that there's an expectation that they should be treated like a checklist and I think that you know the taking a more tailored approach requires a certain level of confidence or experience with using the standards so that's you know it's probably not surprising yeah what about performance culture most of the standards are focused on the technical quality of mel processes there are some important exceptions but you know sometimes they get lost in the the focus on on the more technical aspects of mel and yeah it's probably also fair to say that they are heavily focused on the responsibilities of implementing partners but also working level DFAT staff and there's not much in there about the role of DFAT senior managers and yet we know that that's obviously a critical underpinning to whether mel actually you know has much impact these sort of themes were picked up quite well in the ODE evaluation of investment level M&E which Sophia mentioned earlier and that's available online and is worth a read the other critique that you hear sometimes is that they the standards kind of take the focus away from the main game which is strengthening mel systems within partner countries counterpart mel systems and the standards are actually there are a couple of standards in there that mention or that require the use of counterpart mel systems or alignment with counterpart mel systems but but yeah it's probably I think this this critique is still still fairly reasonable anything you want to add Sophia yeah I think the it is interesting with the strengthening counterpart mel systems because that is if you go back and look at the the standard for design you have to really look at or partner systems and see how they can be used and I think there are many sort of problems one of them is that because we are using Australian taxpayers money so we are accountable to Australians but we are working in a sovereign country which then there might be an example was when I was working on a program where we it was a multilateral organization that was responsible to help the government to build their systems but they were not very good at it and meanwhile we had to sit and sort of wait then and then you're accountable to your government and your people so it can become quite tense in terms of getting that data the other issue is that one of the things that comes up a lot is that oh we're going to have a mel system and then yeah let's just build a bit of capacity for the government partners as an afterthought so one of the things that I'm trying to emphasize is that if that is going to be part of our design it has to be resourced and also monitored so that it happens because it sort of comes like we're going to build capacity as we run along here kind of language and and that's also a risk I think yeah over okay do you want to take us through the next slide Sophia on how to get the most out of the standards thank you so I think one of the important things is that the standards and I am completely biased but the standards are good and it's a really useful tool but it will also require continuous training and coaching networking and sort of managing knowledge so that people investment managers know about them and and can use them as Dave said without treating it as a checklist and that that's obviously building that capability is a long-term endeavor and it is what I find is that in everything the more you know the more flexibility you dare to have so that's looking at the context and what's actually needed it sort of comes back to when you see you know investments with a massive amount of indicators because maybe something will stick rather than banking on three that are actually the three right ones so it's having that confidence and really the emphasizing again the dialogue and the partnership because four d-fat investment managers do not have to be M&E experts so it's a bit onerous on the implementing partners to make sure that the d-fat person actually can understand the mail system and and what's going on so that's important and then the the culture the performance culture is very important and when I do training now within d-fat we always have a discussion about the performance culture within d-fat and it varies a lot between different areas and it's no surprise to anyone that it comes down a lot to leadership and what they are how they allow performance culture and especially to bring up things that are not working well so we can really learn from that and the first time Penny Wong came to visit d-fat the thing obviously that stuck with me was that she said I'm very happy to learn from mistakes I don't know if if that's true but that's what she said so that was that's that's a good good scene for that the the other thing just to throw in there Sophia on on tailoring to context is the you know the need for culturally responsive or culturally competent approaches to mail and that's uh I guess that's that's a net let's I don't think the standards are incompatible with that but but they don't you know they probably could do more to to promote that and you know maybe that's something to think about going forward yeah and and I think that that comes back to even the debate that we're having right now with methods for impact evaluations because it's also how comfortable people are with methods they are not familiar with or they because from d-fat horizon if it's rigorous methods that gets credible information it's all good but it's sort of having the confidence and an understanding even by mail experts if if they can promote that and implement it so that is I agree that is definitely something that we need to think more about and I think also the other thing is you might have noticed if if you paid attention that we are now talking about mail in d-fat rather than M&E and if you're an M&E person like myself the L the learning is always front and center but I think words matter so it's really good that we are adding the L now so that it is really a lot more focus on on learning and sharing the learning from evaluations across d-fat and beyond so that I think is also really important I think next slide so where to from here so we we are very aware because you could say that Dave and I are working on on two sides here with the same tools and it's not a quick fix as I said I've been sort of involved with this for over 15 years and the really good and encouraging thing is that I can see improvement it has become a lot better than 15 years ago but of course it's still a long way to go it's sort of like two steps forward one back but that means that we are moving forward and the other thing is for me within d-fat as an organization is to really get staff to engage with mail and not see it as something someone else is doing on the side and sometimes when you talk to mail experts they also have that challenge within the contractor because in the standard now we are emphasizing that mail is the responsibility of senior management so to engage people to actually have good user-friendly mail systems is a key and then it is a challenge with capability which I'm defining defining as having the knowledge to do things and then capacity having the time to actually do it or the resources to do it and both those are I think within d-fat depending on the areas they are both weak but I think also that sometimes we think in d-fat that there is an abundance of capability and capacity outside of d-fat but that's not actually true we need more mail people in this world so I'm very happy to see all of you here on the screen and spread the word


CHAPTER XXII Before that well-nigh hopeless journey to the coast was half over, I became ill. So ill that anyone who looked on me might well have imagined that I had come to the end of my pilgrimage. That was what I feared. For days I remained sunk in the deepest despondence. Then, in a happy moment, I remembered how, after being bitten by the serpent, when death had seemed near and inevitable, I had madly rushed away through the forest in search of help, and wandered lost for hours in the storm and darkness, and in the end escaped death, probably by means of these frantic exertions. The recollections served to inspire me with a new desperate courage. Bitting goodbye to the Indian village where the fever had smitten me, I set out once more on that apparently hopeless adventure. Hopeless indeed it seemed to me in my weak condition. My legs trembled under me when I walked, while hot sun and pelting rain were like flame and stinging ice to my morbidly sensitive skin. For many days my sufferings were excessive, so that I often wished myself back in that milder purgatory of the forest, from which I had been so anxious to escape. When I tried to retrace my route on the map, there occurs a break here. A space on the chart where names of rivers and mountains call up no image to my mind. Though in a few cases they were names I seemed to have heard in a troubled dream. The impressions of nature received during that sick period are blurred, or else so collared and exaggerated by perpetual torturing anxiety, mixed with half-delirious night fancies, that I can only think of that country as an earthly inferno, where I fought against every imaginable obstacle, alternately sweating and freezing, toiling as no man has ever toiled before. Hot and cold, cold and hot, and no medium. Crystal waters, green shadows under cover-ture of broad moist leaves, and night with dewy fanning winds, these chilled but did not refresh me. A region in which there was no sweet and pleasant thing, where even the Ita palm and mountain glory and airy epiphyte, starring the woodland twilight with pendant blossoms, had lost all grace and beauty, where all brilliant colors in earth and heaven were like the unmitigated sun that blinded my sight and burnt my brain. Doubtless I met with help from the natives, otherwise I do not see how I could have continued my journey, yet in my dim mental picture of that period I see myself incessantly dogged by hostile savages. They flit like ghosts through the dark forest, they surround me and cut off all retreat, until I burst through them, escaping out of their very hands, to fly over some wide naked savanna, hearing their shrill pursuing yells behind me, and feeling the sting of their poisoned arrows in my flesh.  Not only was I pursued by phantom savages and pierced by phantom arrows, but the creations of the Indian imagination had now become as real to me as anything in nature. I was persecuted by that superhuman man-eating monster supposed to be the guardian of the forest. In dark, silent places he is lying in wait for me.   Ah, no, he cannot harm me! For every ravening beast, every cold-blooded venomous thing, and even the frightful Curipita, half-brute and half-devil that shared the forest with her, loved and worshipped Rima, and that mournful burden I carried, her ashes, was a talisman to save me. He has left me, the semi-human monster, uttering such wild, lamentable cries as he hurries away into the deeper darker woods that horror changes to grief, and I, too, lament Rima for the first time, a memory of all the mystic, unimaginable grace and loveliness and joy that had vanished, smites on my heart with such sudden, intense pain that I cast myself prone on the earth and weep tears that are like drops of blood. Where in the rude savage heart of Guyana was this region, where the natural obstacles and pain and hunger and thirst and everlasting weariness were terrible enough without the imaginary monsters and legions of fathoms that peopled it, I cannot say. Nor can I conjecture how far I strayed north or south from my course. I only know that marshes that were like sluffs of despond, and barren and wet savannas, were crossed, and forests that seemed infinite in extent and never to be got through, and scores of rivers that boiled round the sharp rocks, threatening to submerge or dash in pieces the frail bark canoe, black and frightful to look on as rivers in hell, and nameless mountain after mountain to be toiled round or toiled over. I may have seen real amma during that mentally clouded period. I vaguely remember a far extending gigantic wall of stone that seemed to bar all further progress, a rocky precipice rising to a stupendous height, seen by moonlight, with a huge sinuous rope of white mist suspended from its summit, as if the guardian Kamuti of the mountain had been a league-long spectral serpent which was now dropping its coils from the mighty stone table to frighten away the rash intruder. That spectral moonlight Kamuti was one of the many serpent fancies that troubled me. There was another, surpassing them all, which attended me many days. When the sun grew hot overhead and the way was over open savanna country, I would see something moving on the ground at my side, and always keeping abreast of me. A small snake, one or two feet long. No, not a small snake, but a sinuous mark in the pattern of a huge serpent's head, five or six yards long, always moving deliberately at my side. If a cloud came over the sun, or a fresh breeze sprang up, gradually the outline of that awful head would fade and the well-defined pattern would resolve itself into the motlings on the earth. But if the sun grew more and more hot and dazzling as the day progressed, then the tremendous Ophidian head would become increasingly real to my sight, with glistening scales and symmetrical markings, and I would walk carefully not to stumble against or touch it. And when I cast my eyes behind me I could see no end to its great coils extending across the savanna. Even looking back from the summit of a high hill I could see it stretching leagues and leagues away through forests and rivers, across wide plains, valleys and mountains, to lose itself at last in the infinite blue distance. How or when this monster left me, washed away by cold rains, perhaps, I do not know. Probably it only transformed itself into some new shape. Its long coils perhaps changing into those endless processions and multitudes of pale-faced people I seem to remember having encountered. In my devious wanderings I must have reached the shores of the undiscovered great white lake, and passed through the long shining streets of Manoa, the mysterious city and the wilderness. I see myself there, the wide thoroughfare filled from end to end, with people gaily dressed as if for some high festival, all drawing aside to let the wretched pilgrim pass, staring at his fever and famine-wasted figure in its strange rags with its strange burden. Anu Ahasuerus, cursed by inexpeable crime, yet sustained by a great purpose. But Ahasuerus prayed ever for death to come to him, and ran to meet it, while I fought against it with all my little strength. Only at intervals when the shadows seemed to lift and give me relief would I pray to death to spare me yet a little longer. But when the shadows darkened again and hope seemed almost quenched in utter gloom, then I would curse it and defy its power. Through it all I clung to the belief that my will would conquer, that it would enable me to keep off the great enemy from my worn and suffering body until their wished goal was reached, then only would I cease to fight and let death have its way. There would have been covered in this belief had it not been for that fevered imagination, which corrupted everything that touched me and gave it some new hateful character. For soon enough this conviction that the will would triumph grew to something monstrous, a parent of monstrous fancies. Worst of all, when I felt no actual pain, but only unutterable weariness of body and soul, when feet and legs were numb so that I knew not whether I trod on dry hot rock or in slime, was the fancy that I was already dead, so far as the body was concerned, had perhaps been dead for days, that only the uncockable will survive to compel the dead flesh to do its work. Whether it really was will, more potent than the bark of barks and wiser than the physicians, or merely the vismindic catrics with which nature helps our weakness, even when the will is suspended, that saved me, I cannot say, but it is certain that I gradually recovered health, physical and mental, and finally reached the coast comparatively well, although my mind was still in a gloomy, desponding state when I first walked into the streets of Georgetown in rags, half-starved and penniless. But even when well, long after the discovery that my flesh was not only alive but that it was of an exceedingly tough quality, the idea born during the darkest period of my pilgrimage, that die I must, persisted in my mind. I had lived through that which would have killed most men, lived only to accomplish the one remaining purpose of my life. Now it was accomplished. The sacred ashes brought so far, with such infinite labor, through so many and such great perils, were safe and would mix with mine at last. There was nothing more in life to make me love it, or keep me prisoner in its weary chains, the prospect of near death faded in time, love of life returned, and the earth had recovered its everlasting freshness and beauty, only that feeling about Rima's ashes did not fade or change, and is as strong now as it was then. Say that it is morbid, call it superstition if you like, but there it is, the most powerful motive I have known, always in all things to be taken into account, a philosophy of life to be made to fit it, or take it as a symbol since that may come to be one with the things symbolized. In those darkest days in the forest I had her as a visitor, a Rima of the mind, whose words when she spoke reflected my despair. Yet even then I was not entirely without hope. Heaven itself, she said, could not undo that which I had done, and she also said that if I forgave myself heaven would say no word, nor would she. That is my philosophy still, prayers, austerities, good works, may avail nothing, and there is no intercession, and outside of the soul there is no forgiveness in heaven or earth for sin. Nevertheless there is a way, which every soul can find out for itself, even the most rebellious, the most darkened with crime and tormented by remorse. In that way I have walked, and, self-forgiven and self-absolved, I know that if she were to return once more and appear to me, even here where her ashes are, I know that her divine eyes would no longer refuse to look into mine, since the sorrow which seemed eternal and would have slain me to see, would not now be in them. End of chapter. End of book. Thank you for listening.


The Murray Cod is a prize catch for recreational fishers, but it's very survival is threatened by many things, including overfishing. Hi, I'm Andrew Wittinghausen from Escape With ET and I'd like to tell you what the South Australian Government is doing to help this important fishery get back on its feet. The South Australian section of the River Murray is closed completely during the Murray Cod peak spawning season, which is from the 1st of August to the 31st of December. This is to reduce stress on the fish when it's captured and increase its chances of survival and spawning. Outside these dates, recreational fishers can enjoy the thrill of catching one of these iconic fish, but they must immediately release this fish back into the water. And when you catch a Murray Cod, you cannot lift it out of the water at all, simply remove your lure and let it go on its way. The absolute exception of these rills applies to the Chawilla floodplain and the Rallrall Creek near Renmark. These regions mimic the original flows of the River Murray and provide Murray Cod with their best chances of good spawning events and help the ongoing survival of juvenile Cod. No Murray Cod fishing is allowed in these areas at all. By following these easy rules, we Anglers are doing our bit to keep this magnificent species alive for future generations to enjoy.


The Appalachian Wireless, 30th anniversary holiday sale is going on now from now until the end of the year. Visit us in-store or reserve online for an unbelievable sale and give away. We know what you want for the holidays because we are you. We are Appalachian Wireless. Kentucky is receiving millions of dollars to save lives by reducing the number of drug overdoses. Bloomberg Philanthropies is donating $10 million to the state for a five-year program aimed at reducing overdoses. Bloomberg has pulled together an all-star team for this effort, including Johns Hopkins University, the CDC, the Pew Charitable Trust, and more. This public-private partnership has a five-year timeline and its goal is to save the lives of Kentuckians by increasing access to medications and expanding preventative services. Kentucky, we're only one of seven states invited to participate in this initiative, which is going to provide us with valuable resources, funding, and expertise. We'll be announcing more specifics about the partnership as we develop our plans with Bloomberg and other collaborators in the months ahead. While overdose deaths have been on the decline in recent years, numbers skyrocketed during the pandemic. Kentucky set a new record of nearly 2,000 overdose deaths in 2020, a 49 percent increase over 2019.


Okay. So I want to discuss with you why the traditional way of chasing women is not a good thing to be doing, just so for you why guys out there. So doing everything the traditional way sets you up as the provider. When I say provider, women will say, oh, I want someone to provide for me and so on, and that's not what I'm talking about in that traditional context, because that's true. I do believe that you should provide for your family and so on. But the way that most guys use provider is in that we give women things in order to get them to like us. We do things for them just in order for them to like us because we think that's the way we have to do it. And I'll tell you something that I forget out along the way is there's absolutely nothing sexy about the provider, the provider I'm talking about. Absolutely nothing sexy about it. We've all seen this scenario of the rich guy who has a ton of money, he's got a nice car, he meets some girl, he asks her out, he takes her to an expensive restaurant, and then at the end of the night, he spends all his money on it, at the end of the night gets a kiss on the cheek and he drops her off. Okay. Maybe some of you have done that. I've done it for sure. And then there's the guy, the other guy. He doesn't spend any money on the girl. He's not a great dresser. He doesn't seem to have many skills, but she's addicted to him and can't get enough of him. And this guy doesn't do anything that the other guy does. He's a complete opposite so he's not the provider. He's more of the friend with benefits type guy. He's more carefree. He's not trying to to push her into doing anything. And she's addicted to this guy, always trying to find out where he's at. You know, what is he doing? She's texting him, trying to meet him. Where is he? Is he at this certain club or this bar?  And sometimes this girl goes from the date with the rich guy and texts the friend with benefits guy on the way home, goes over to his house. So which guy do you want to be? You know, it sounds simple to me.  Okay?  I mean, the traditional way that we're taught, you know, especially when we're interacting with women, is, you know, we're out there and we're, we're, we're talking to them. We're, you know, we're kind of hovering over them and pressuring them. And you know, I go, I got to talk this girl into, you know, I got to get her number and I got to talk her into going out with me. And everything is about pressure. And everything is, is not, it's never letting up. Okay. And for those of you who say, well, you know, will women really do that? Well, you probably wouldn't know because you spend most of your time trying to talk women into something they already want to do, but you're trying to talk them into it. So in other words, women don't meet this cool guy very often. They meet the guy who's over pursuing. He's relentless. You know, I'm talking about in the creepy way, right? You know what I'm saying, right? We've all been there and done that. It's the guy who's desperate and needy and he wants something. He's trying to make something happen. And that girl's not going to be excited about being assertive. And what I find is that most women I talk to would love to be assertive. If number one, they knew that we could handle it, right, when they are assertive. And I don't mean aggressive. I mean assertive in a very healthy, fun way. And they don't meet that guy. They rarely meet that guy. They always meet the guy who's doing the same thing as every other guy is doing. And when you're doing the same thing that every other guy is doing, that's competition.  You know, I'm in a bar. What about all these other guys and all that kind of stuff? And I said, well, if you're doing everything that they're doing, then yeah, they're your competition. But if you're this other next level guy that I'm talking about, they're not your competition anymore. And so that's something to think about. Okay. So nothing sexy about the provider. Now, if you want to be the next level kind of provider, then the next level provider is a guy. And I know him. He's a legend. He's been around way back in the day when we all got the. And I'm telling you, I see these guys come up to me and they show me all these super long text message threads or instant message.  And I say, hey, so, so you guys got together? No, but we're, I think I'm going to talk about it soon. We're probably going to maybe get together. I'm like, really? This is, this is your problem. This is, this is over chasing women. This is the old way. And this is exactly why you're not getting together with her. I'm looking at all the stuff you're telling her. And it's the same thing every other guy is saying, right? It's how it's going to, you're trying to teaser. And you've gotten into it and you're doing it. Is it possible that women will just switch over immediately? Yeah, it is.  So we become too pushy, they're too passive. So they kind of wait around, right? They've been taught to wait around and the right guy will approach you or he won't. Right? Now some girls are assertive and I applaud them if they are, but very few. And so maybe in conversation with, you're already sort of like chasing the old way.  That's it.