Before we begin, I would like to acknowledge the traditional owners of the land from which we are all dialing in from. We pay respect to elders past, present and future and acknowledge the importance of Indigenous knowledge in the academy and the profession. As a community of evaluators, we are privileged to work and learn every day with Indigenous colleagues and partners. Now, throughout this presentation, we're wanting to encourage people to pop their comments and questions onto the chat box. And our colleague, and so, and John will be moderating these questions and we'll have a chance to discuss them at the end of the presentation. So, you know, feel free to be using the chat box, if there are any questions that are coming up along the way. So today we'll be discussing research on evaluation, and we'll be hearing from a panel of speakers which I have the privilege of introducing. So, firstly, we'll be hearing from a speaker who is a familiar face to many of us, Dr. Ruth Mastin, hailing from New Zealand and now a senior lecturer at the Assessment and Evaluation Research Centre here at the University of Melbourne. Ruth has a deep expertise, has deep expertise in the areas of public health and health services research, educational evaluation and health promotion in schools with a specialisation in programme evaluation. She's currently working on several evaluations of evaluates of educational initiatives in Victoria and the Northern Territory and the evaluation of the weak health local government partnerships with over 30 local councils in Victoria. She holds an honorary fellow position at the Centre for Adolescent Health at Murdoch Children's Research Institute. We'll then get to hear from three brilliant emerging evaluators who come from varied background, but are all keen scholars of evaluation and in fact are now completing your Masters of Evaluation with us. So, firstly, we'll hear from Cat Franks, who has a background in business operations, developing grant programmes, managing research centres and client service, and she has an experience working in government-funded organisations, non-government sectors, including academic research, consultancy and design, and she's currently working in evaluation at an agency in government. Then we get to hear from Hannah Morgan, who is a qualified social worker with experience in mental health, disability and LGBTIQ plus health. Most recently, she has worked in project management, coordinating national projects focused on LGBTIQ plus health and palliative care research, and she's currently working as an evaluator at the Black Dock Institute. This is Australia's only medical research institute investigating mental health across the lifespan. And last, but definitely not least, we'll hear from Stephanie White, who come to evaluation after years working in education research, policy and practice in the government and non-government sectors. She's led projects in early childhood education of original and Torres Strait Islander education, student engagement, school improvement, and most recently student wellbeing. Stephanie is a senior evaluation officer at the Victorian Department of Education. So without much further ado, please welcome Dr Ruth Aston. Thank you, Katina. And yes, I'm just thrilled to have the opportunity to present to our AES members, but also as part of this incredible team. Just the next slide please, Katina. So what we wanted to do was sort of to start with a little bit of context about the review project and before you hear from each of our speakers about their particular areas of interest as part of this review. So we are conducting this review because the last known published review of research on evaluation was published in the 2017 but included research that was essentially available up until the date of 2014. So it had been some time since that review was conducted. But what Coronetales, this is Chris Coron from the University of Western Michigan, what he found with his colleagues is that over that decade of 2005 to 2014, there were 257 research and evaluation studies published. And what Chris and his colleagues were most concerned about was the limited research available at that time on valuing and evaluation, ethics, and the use of evaluation. They and we reflected that since that review was conducted, there has been a growing interest in many areas of evaluation, including but not limited to participatory evaluation and co-design in the relationship between policy and program design and evaluation practice. So therefore we felt that synthesizing research on evaluation is really important to make sure that all available evidence is being collated and summarized and ideally appraised on a fairly regular basis. It also allows us to see where we are at in terms of the knowledge that we have about our field of evaluation and evaluation practice and importantly identify where there are gaps and what priorities we have for future research on evaluation. Let me slide please. So before we go any further, we just wanted to sort of clarify how we have referred to research on evaluation for the purposes of a systematic review. And in large part we've inherited this definition. So the definition you can see on the slide there was defined by Chris Pryne and his colleagues where research on evaluation is any purposeful systematic and empirical inquiry intended to create a strong evidence base and infrastructure for applied practice of evaluation. So at this point I want to note that while empirical inquiry was the focus of our review, we recognize that there are many other types of inquiry that are as important at adding to our evidence base and infrastructure for applied practice of evaluation. So just noting that it's a lot broader than just empirical inquiry, but that was the purpose of this review. Next slide please. Thank you. So holding that definition and the broad context in mind, the objectives of this systematic review were to replicate Corinne's et al's original review with some modifications focusing on all research on evaluation that's been published from 2015 to 2019. We had the purpose of understanding what research has been conducted, where they made gaps, typically within the gaps that Corinne identified, are they still gaps, are they existing? So that we can actually develop an agenda for future research on evaluation, what are our priorities, what do we need to be doing research on based on the practice sort of advancements on that profession. At this point it's worth acknowledging and certainly important to recognize that this review is being led by Dr. Wanzas, this is Dana Wanzas. She's an assistant professor at the University of Wisconsin-Stout, and although she was very keen to hear that we were presenting to you, she didn't quite want to get up at 3 a.m. to join us. So she's here in spirit, but we do acknowledge that this work has been led by her, and we at the University of Melbourne are collaborating with her on this project. And as part of that, our part of the pie, if you like, or our part of the work was to look at three evaluation journals, with a total of 782 journal articles being included in our portion of our review. So what did we do? We screened all the articles, so all the 782 articles that I mentioned before based on foreign systematic review procedures with some minor amendments. So we were looking at all 782 articles, they were all published in that publication period I mentioned, all in English. They were published in one of 14 journals on evaluation, including the Evaluation Journal in Western Asia. And to be included in the review, the articles needed to demonstrate that they were purposeful, systematic and empirical, and we're generating knowledge about evaluation, as I mentioned before. So essentially meeting that definition of research on evaluation. And they also needed to include enough information for us to code them as part of the review. While we were really interested in the systematic review procedures, we weren't sure how much interest they would hold for you all. So I might be brief here, and perhaps touch on a few key points. As you can see, there's a team of us, and research on evaluation is, we learned as we went through this process, although we had some foresight that this might occur. That there was quite a lot of essentially disagreements between us, whether something was, whether an article was actually meeting that definition of research on evaluation. So we undertook quite extensive calibration processes, and we found with the US team that they were doing the same things. That was a bit of a relief. But we did reach an acceptable level of consensus with all 782 articles on title and abstract as to whether they met the definition of ROV. And we were looking at articles in evaluation and program planning, the evaluation general of Australasia, as I mentioned, and new directions for evaluation. So once the articles had met that initial inclusion criteria, that list of ROV that I mentioned before, we then went through a coding process using mal-marked coding framework, which is essentially a tool to classify research on evaluation in terms of what the subject of the enquiry is, and what the mode of enquiry is, as you can see on the screen there. Again, touching on this, this step was again quite challenging to reach consensus on. So we had to lower our benchmark there, and we did reach 66%, which again was relatively similar to what the US team was able to reach. And you can see a little bit about our process there. But just to clarify, once an article met that initial inclusion criteria, i.e. was it could have been classified as research on evaluation, we then identified what was the subject of the enquiry and what was the mode of the enquiry. And the US team are doing the exact same thing with the remaining, the living genius. Okay, now I apologize if you might need to zoom in, but I'm going to, this is essentially the results of our review before I hand over to Kat. We're in the red box. So the first three journals there, Evaluation and Programme Planning, New Directions for Evaluation and EGA. And there's a couple of observations I wanted to offer before I hand over to the next speaker. So across the three journals that we reviewed, between 22 and 38%, so I mean there's a 782 bigger than we screened, between 38% of those actually met the inclusion criteria for ROE, we look at EGA. So it's one of three, but it's quite amazing. I was quite happy to see that the Australasian Journal is proportionately contributing quite a high degree of research on evaluation to the evidence base. Unsurprisingly, this is where we essentially reaffirmed the original reviews findings. The majority of ROE studies were about evaluation practice, so the activities of doing evaluation. A lot of it was about methods, synthesis procedures and so forth. This was closely followed by evaluation consequences and domain specific ROE, which could be things like looking at ideal evaluation capacity building programs for certain groups of students or it could be an organization specific evaluation framework, be a tool for a discipline and so forth. In terms of most of inquiry, again, the patterns that we see here are almost identical to what our own Utah review found. The overwhelming majority of ROE studies are descriptive, followed closely by development tools and models. So an evaluation that's often the development of rubrics, framework development, so articles that talk specifically about developing evaluation frameworks and the processes of doing that were formed that we had agreed as well. So if we come back to the 782 articles that we reviewed, 190 of them were identified as research on evaluation. That's approximately 25%. So of course, that's the border of all the studies that we reviewed were research. And in terms of EJA in particular, this is a huge increase since the previous review. So thinking about when the previous review ended in 2014 to where this review began in 2015 to 2019, choir and e-tiles reviewed found only six ROE studies and EJA over that decade period they covered, and we identified 13 in that five year period alone. So there is an exponential increase and some of the other journals represent a similar growth, which we think is really positive. And while our colleagues in the US are still continuing to screen their articles, you can see some numbers on the table above that reflect what they are finding. And on brief glance, it's a relatively similar trend based on the subjects of inquiry and loads of inquiries relatively consistent across the journals. And that's quite reaffirming again that what we found in the choir and e-tiles review that decade we're seeing again, but there's more research on valuations being generated relative to what there was in the previous review. So I'm now going to hand over to Cache who's going to share her findings on the use of evaluation. Great. Thanks, Ruth. So my research focuses on empirical ROE literature from 2005 to 2019 on the use of evaluation findings within government. I explored ROE studies through the evaluation use framework of instrumental conceptual and symbolic use for those unfamiliar with these terms. Instrumental use is the direct use of evaluation recommendations or findings to inform decision making for change for action or change. Conceptual use is when findings do not lead to action or change, but results in users having a better understanding about the programme of policy that improves their knowledge or attitude about the programme of policy. Symbolic use, also known as political use, is when findings are used to legitimise an existing position or supported decision already made, or when evaluation findings are used for political self-interest. Outside the scope of this review is process use. This is found when behavioural changes are seen within those directly involved in the evaluation as a result of being part of the process of evaluation. Research in this area is mainly focused on evaluation capacity building, which could have been a whole capstone project in itself. The second part of my research examined whether the literature addressed influencing factors of findings use and the alignment to the Johnson & Co framework shown here, which is based on an ROE literature review of evaluation use articles from 1986 to 2005. They found 41 articles, which builds on the Cousins and Leithwoods 1986 framework of 12 factors. This framework categorises factors under evaluation implementation, decision or policy setting and stakeholder involvement. This was the search methodology, which I won't go into detail. I will note that I incorporated the ROE articles from Corinne's review, which is why my timeframe is from 2005. After completing this process, my sample was 14 papers. I will also quickly note the limitations. The review was conducted within the parameters of ROE systematic review. As such, this doesn't include publications outside the evaluation journals and exclude studies from 2020 onwards. I have to continue building on this review and capture additional publications on this topic in the future. The next two slides provide a brief overview of the findings. The icon show whether a study investigated a findings use category and whether they address use factors. As you can see, a range of findings use was studied with many addressing instrumental, conceptual and symbolic use. I will go into other use categories such as strategic use, which is when high level decision makers use evaluation findings such as a decision in a spending review as an example. The settings for the studies varied from federal to state government departments and agencies to Parliament, legislative officers to the European Commission. The majority of findings use studies investigated instrumental use, the most common form of use at a programmatic level. The findings evidence of use that informed decision making to change a policy or program, which is positive. However, the studies were mostly conducted in European nations and North America. Influencing factors were examined in almost all of the articles with eight article studying influencing factors aligned to the Johnson and Co framework. The most dominant category study was evaluation implementation, followed by decision and policy setting category. There were two noble factors of evaluation policy and organizational capacity to use evaluation explored. Stakeholder involvement was the category lease exam. In an Australian study by Maloney, stakeholder involvement was a frequently mentioned influencing factor for use. We've been learning how examining stakeholder strategies used by internal legislative evaluators across two articles. Otherwise, it was generally unexplored category. What I took away from this was that the factors that enable the use of evaluation findings is complex, which I'm sure some of you will know. The most common factors explored include evaluation quality, findings, the information needs of the evaluation audience and the political climate. I'll briefly take you through one of the studies that addresses these factors. I'm a researcher, Leederman, taking a different approach. Calling for a move away from the findings, the most defining, the most important factor for use. This study looked at the context specific necessary conditions required for users of external evaluation findings to make a decision to change a program. A qualitative comparative analysis of 11 program project evaluations within the Swiss Agency for Development and Cooperation was performed. The relationship between context conditions, so the pressure for change, and the level of conflict, and the active conditions. So the novelty of value, the novelty value of findings and the evaluation quality were examined to test for hypothesis. Whether the evaluation actors as a conciliator and a waker or a referee or a trigger for a change decision. The 11 cases of scattered across the evaluation context and active conditions with instrumental use evident in six cases. So the Awakener. In a low pressure and low conflict environment, it is expected that an evaluation court can cause a change decision by awakening people. As long as the findings reveal something new and are of good quality. This assumption was generally accepted. The trigger. Where there is pressure for change and low conflict, it is assumed the evaluation triggers change if it is good quality with novelty less of an issue. This was evident. This was evidence that supported this assumption. The referee. In a high pressure environment where stakeholders are in conflict, neither evaluation quality nor novelty is necessarily for utility with findings accepted by only one set of stakeholders. The process was not accepted and revised to an endorser where evaluation quality and novelty are independent factors. That alone are not necessary. The conciliator during conflicts where there is a lack of awareness change substantial decisions are made when evaluation are a high quality and novel but this is rare. Change decisions were not made in two cases displaying these conditions as such the research was unable to test. Studies such as this increase my awareness about the particular context related conditions which are outside the control of an evaluator conducting an evaluation within a government setting. From the studies led in Canada by Borgus, these looked at the usability of evaluation reports to know whether the precursors for evaluation use report credibility and quality were present. In addition to examining whether program evaluation findings were strategically utilized and spending reviews within two federal government agencies. It also looked at the application of findings to ongoing program design and delivery improvements. It found that the evaluation reports were credible with evaluation quality demonstrated through clear evaluation questions, the integration of stakeholders through the evaluation process and sound methods used to produce the findings. The reports provided useful information with relevant appropriate and actionable recommendations which could lead to implementation and instrumental use of findings. While these factors should support utility the evaluation findings were not used to make decisions and spending reviews so no strategic use. This is likely due to the requirements of the Canadian 2009 policy and evaluation. With the evaluations not meeting the information needs of high level decision makers, the timeliness of the reports and a program level focus. However, this level of focus did result in mostly instrumental use of findings for program improvement with some instances of conceptual use. This study made me consider the different uses of evaluation findings with the government and how an evaluation policy can enable and also impede use. I was also unaware of Canada's evaluating system main government that includes the requirements to make evaluation reports publicly available along with management response to action plans that address evaluation recommendations. Canada suddenly looks to be more advanced in Australia in relation to the level of evaluations conducted and the use of findings and transparency. However, for all the evaluation reports published there were still very few are we studies in this area. So it's hard to say what evaluation use looks like in government agencies and departments in Australia through the are we in this sample. As I only found one Australian study published by Jade Maloney in 2017 and the evaluation journal of Australasia apologies if I missed any other Australian studies. But this article focuses on AES members perceptions on the levels of evaluation use, the factors affecting utility and how well evaluators overcome barriers to use in practice. Respondents in this study rated the non use of evaluation findings by government agencies in Australia as a considerable problem. However, many have participated in an evaluation resulted in the findings being used mostly for accountability purposes when asked to account their most recent case of evaluation use. So many of the franchise factors, especially agency leadership commitment and individual openness to evaluation and supply site factors. Mainly stakeholder involvement in establishing the real purpose of the evaluation and effective communication of findings were viewed as important factors to support use. We haven't already read this article I highly recommend. I'm raising I hope to build on this review and potentially expand beyond evaluation journals as I'm interested to know whether there is research out there that looks at federal state government policymakers and program managers perspectives on the use of findings within government agencies and departments. Or if anyone's interested in being part of a study that looks at this please reach out. Thank you. On our hand over to him. Thanks Kat. Give me one moment. Okay. I'm zooming in from Gadigal land and really pleased to be here to hopefully consolidate in 10 minutes. What was quite a journey of deep diving into research on evaluation that met the criteria for values inquiry. Next slide please. I wanted to focus on this area as you heard earlier in this presentation. This was one of the under researched areas in Corrin and colleagues review. Only 3.5% of papers on research on evaluation were coded as being values inquiry. And as a new evaluator I thought I'll be really interesting to see what I could learn from the various opinions of stakeholders and evaluators in my own professional practice. So first of all I wanted to define what we mean by values inquiry and using Mark's definition. Mark defines values inquiry as identifying value positions of stakeholders and public evaluation methods used to probe the values embedded in and related to a program. So essentially I was looking for research on evaluation that included the opinions, attitudes, beliefs and values of stakeholder groups and evaluators. Next slide please. So what values inquiry research on evaluation in was completed between 2015 to 19. Well I can say that there were 20 articles in total. Interestingly the articles that our team coded they seem to have I guess we coded more articles and I don't know if there was some differences and approach across the teams. But yeah I think it's also interesting to note there were four journals that didn't include any values inquiry research on evaluation. Next slide please. Thank you so much. So I completed a thematic analysis on the literature. And while values inquiry was diverse in subject matter, I did find and maybe it was my social work background, but there was an overarching repeating theme around power. And I found it really interesting because there was lots of opinions and perspectives around how power is distributed or not distributed. And the consequences this can have on evaluation framing processes outcomes and use. So I found the power was explored across different contexts, the cultural context, mostly around Australia and New Zealand evaluation and political context program evaluation but specifically participatory approaches. And as well as that there was articles that were about the evaluation profession. Next slide please. So I wanted to give you, I guess a short summary on some of what I found as well as the implications for practice. I know for myself I'm always thinking about so what does this mean for me in the field. I'll give you a little bit of a summary across those different areas. So for articles discussed evaluation and cultural context highlighting the ways in which evaluations mainly reflect Western understandings of the world and Western perspectives of merit and worth. So in terms of implications for practice there's a need to understand the benefits of practicing in a in a culturally appropriate way, being reflexive as an evaluator, having awareness of cultural bias and challenging theories that underpin program designs. And really thinking about our position in relation to the evaluation considering the history of colonization, acknowledging indigenous knowledge systems and thinking about how we're doing this in practice and if we are being culturally appropriate in the way that we work. Next slide please. Really interesting. I found this area and following on a little bit from Kat and some of the discussions raised. But there were discussions around the political context so multiple papers discussed the power that government and administrators have in influencing the way evaluations are conducted, how outcomes are determined and evaluation use. So Evans et al reported there was a high reliance on funding bodies in determining outcomes, sometimes less reliance on evidence based sources such as literature reviews, and a small survey of 63 Australian evaluators found that evaluators employed in medium and small organizations said funding bodies were influential in the choice of evaluation design. So it's this discussion around how evaluation can feel politicized and constrained, and that sometimes and I'm sure we've all experienced the idea of walking into an evaluation evaluation is predetermined outcomes. I think we can relate to that. So in terms of practice implications, Evans proposed that there were a range of practical actions that include adopting a values based agenda, asking stakeholders what outcomes are most important to them and taking action to defend evaluation of different outcomes that really go beyond the range of short term outcomes prioritized in fiscal and social policy. Next slide please. At the program level and I was really pleased to see this because I've done like quite a lot of co design in my project management work and I'm really interested in participatory evaluation and appreciate that there's always power struggles in those spaces. Well, there often is anyway. And so studies highlighted a range of different considerations, including how they can be different agendas between program implementers and those people who are engaged to work in a participatory way, and those who are essentially the people who are meant to benefit from the program. And it spoke about how there's often a political interest that influences how big I guess the extent to which participatory evaluation can happen. There was one single factor in experiment by chronic and Roman and this was so interesting because it really showed in this case that it can evaluation use is higher when you don't use a participatory approach when compared to when you use a participatory approach, but you do it in an inauthentic way. And I think that would ring true for most of us here who participated in participatory engagements and it hasn't felt like your contribution has really been taken into consideration. So I think this could be a really interesting area of future research. And next slide please. And a little bit separate but still connected to all of this is discussions around the evaluation profession. And mostly it was about the progression of the profession and getting evaluators views on how we should progress whether it's through certification credentialing looking at competencies. And while there wasn't a clear consensus on what could be pursued, there were concerns raised about the potential obstacles that could be created through the professionalization of evaluation field. And so discussions around for who would this benefit, why and in what conditions, who would be best served by decisions that might be made around professionalization. And I guess for me, it made me think about how values inquiry views and findings from the research are probably quite important when thinking about the progression of evaluation as a discipline. Next slide please. So the research to me has shown that there's a need to think about the way that power manifests across all levels of evaluation. And I think values inquiry research could have discussed in this thinking. And while these findings can't necessarily been neatly synthesized because there were diverse subject matter covered in the articles and articles that weren't included within this broader theme. I think that certainly there could be a need to interrogate some of these areas further with more research. And I think there could be a link between values inquiry research and how we might capture that and capture that information synthesize it and integrate it into conversations about the evaluation profession. Next slide please. So what was some of my key takeaways? They're still happening as I reflect on this topic. Certainly it was a challenge in synthesizing these articles because it was diverse content. But I think values inquiry can tell us useful information about how we do evaluation and who and who we want to be as a profession and where we want to go. It helps illuminate illuminating give voice to some of the complexity we face when working in the field. There was so many times I read these research articles and I felt really validated and seen. But essentially, this is not the only space we can find that validation and find this really important content that there's also not empirical research that provides a significant contribution to these discussions as well. And it was challenging for me sometimes to not include research papers that I thought would, you know, were making really important points around values and values inquiry in evaluation. And that's all from me. Thanks. I'll hand over to Steph. Thanks so much, Hannah. And I'll try not to rehash too many similar reflections over the next 10 minutes. So my nested literature review was on professional ethics in evaluation and I drew on three common definitions of ethics for this, the principles of morality, the rules for a profession and the study of ideal human behavior. As an emerging evaluator like Kat and Hannah I was really keen to explore how research on evaluation is building our understanding of evaluation ethics and what this means for practitioners. I think when when I've been thinking about evaluation and practicing this, it's like there's lots of components that on their own seem feel it fairly straightforward. But then when you bring it all together, there's the context stakeholders values politics, etc, etc. All of that messiness can sometimes lead to ethical dilemmas and, you know, despite an acknowledgement of the ethical dilemmas that evaluators face and that and evaluation ethics be my apologies. Ethics being considered important for the field of evaluation, the research on it over the years has been fairly patchy. So I'll briefly talk through the nature of my review the gaps and implications I've identified and finish off with a brief, hopefully not too rehashed reflection of patterns. Thanks, Karen. So my intention through the literature review was to focus on literature that really explicitly framed was really explicitly framed in terms of evaluation ethics. So it was quite, it was quite narrow in its scope. And I was looking at, yeah, I had three focuses through the through the review. I looked at how the study framed ethics in terms of those three relevant definitions I mentioned previously. I then looked at how the research sought to progress our understanding through contributing testing or generating new knowledge. And then using the AES guidelines as a proxy for the needs of practitioners, considered what the implications were for practitioners at different stages of the evaluation process. So what did I find? Thanks, Kat. Honestly, not, not much within the inclusion criteria I set for my review, but at the same time found quite a lot. So this is a map of the findings from the five articles that were identified. And so in terms of framing of evaluation ethics, there was some mention of this study of ideal human behavior and it generally referred to previous studies rather than being the focus of the paper itself. So ethics was generally framed in terms of the principles that underpin the rules of the profession. So for example, following mandated ethics review processes is important from the perspective of not doing harm to participants. In terms of how the research contributed to our understanding of evaluation ethics, the articles exclusively contributed to existing knowledge. So they built on previous research findings, which isn't, I mean, this is common in research, right? I suppose just in terms of pushing the limits of our understanding of evaluation ethics in terms of generating new knowledge and testing hypotheses and things like that was less of a feature. But in contributing to the existing knowledge base, they did so through exploring the ideas in local localised and contextualised ways. So one example of that was pleasure and colleagues took four previous studies about pressure on evaluators that were in different countries and then elevated that for a country comparison of pressure on evaluators. In terms of the needs of practitioners, again, there was some reference to the studies that, to previous studies that looked at the entry and contracting stage of evaluation, but they really sharply focused on the conduct and reporting stages of the evaluation. So for example, Nathan and colleagues looked at the development of a capacity to consent protocol for youth mental health evaluation, where parental consent was the norm in that jurisdiction, but was a challenge for their evaluation context. Interestingly though, these articles through the on the conduct and reporting focus did tend to sort of mention the entry contracting stage. This is something that we know through other literature that it's a really important stage in the evaluation. So really alluding to the opportunity for further research then. Looking at this little visual it's pretty clear that there's some gaps but it is only five articles. And as I said my literature review is quite narrow in scope so there were some articles that didn't meet my inclusion criteria that absolutely have implications for the ethical practice of evaluations. So, for example, where he pay Anna and McKay, I think Hannah also has this listed in her slides as well, have a paper on evaluative thinking took that talks about the ethical imperative to uphold various cultural ways of deliberating during an evaluation. I'm kind of heading into implications here but I think the other thing I found quite interesting was the recent writing that sort of falls outside of research on evaluation or even the literature that we reviewed that is calling on research on evaluation to do a bit more about shifting for professional ethics. So, one key writer is Thomas Schwant and Schwant and Gates in evaluating and valuing that's not going to work on the video. But evaluating and valuing on social research they call on our way to address topics such as the efficacy of deliberations in evaluation and other types of emerging practices in evaluation and there's another more recent publication I think practical wisdom for an ethical evaluation practice Schwant has a chapter in there where he and he calls on ROE to look at things like the practical wisdom as an as an organizing framework for our professional practice and you know perhaps ROE being able to look at enablers or barriers to that type of work in the field. Also you know as has been mentioned already there's you know current current work there's more you know these review this review ended in the period was 2019 was the last sort of literature we were looking at and just looking at the most recent issue of the Evaluation Journal of Australasia there's definitely some articles in there about ethics and even the book review Kylie Kingston's book review of a research agenda for evaluation that's even calling you know talking about professional ethics so certainly a lot a lot going on. Thanks Kat. So I probably covered quite a few of these implications already but I suppose in terms of professional ethics there was what I found through the literature review but there's also all of these other topics that are being written on that absolutely have implications for our practice and I think there was through reading the literature and taking that the lens of ethics to the papers there's a lot to learn with talking about methods unintended consequences evaluative thinking etc etc. But as I mentioned you know Schwant Schwant and Gates they're also calling on ROE to address some new topics as well. And as Hannah mentioned empirical inquiry is just one type of inquiry and I think you know ROE has its role to play but there are other types of inquiry that will be really well suited to complement ROE that are well suited to complement ROE in building the evidence base for professional practice, ethical professional practice in evaluation. On a more personal note this this process has really helped me this process of engaging with the literature has helped me to sit with my own ethics and sit with the messiness of evaluation. I think you know being able to see that experiences we have are reflected in others experiences through the literature. I mean it's certainly given me that opportunity to reflect and just I guess be cognizant of the values that I'm bringing to my practice and sometimes if I feel a bit of tension or there's sort of tension in the room or however it manifests being able to sit with that and unpack that. And just as there are many topics to still investigate with ethics in evaluation practice I have so many more questions I'm an emerging evaluator. And also really new to the research on evaluation literature so loads of questions but that is me ethics in evaluation practice. There's some stuff out there and there's still a lot to do. And thank you Katina. Thank you. Thank you. Thank you. Thank you Ruth, Kat, Hannah and Steph for the tour de force of making this challenging complex and extremely important topic. So interesting and relevant for us as practitioners. As many of you might have heard you know it's a the team spent considerable time reflecting discussing debating and trying to reach consensus on this topic of ROE and I think in reflecting on this body of work it seems to me that this research on evaluation should not only be the purview of the select few. I think if we as practitioners care about the profession and care about furthering the knowledge base on which we can draw on on this body of work then I think we need to pay careful attention to what gets added to this body of knowledge. Who is setting the agenda for example and you know what is getting prioritized and you know the other important question to us ourselves is you know are we willing to invest time into this into this important work. Now I promise that we will set some time for questions and answers and discussions but I just wanted to make this a call to action today. And we're very keen as a team and we're just a team of people who are interested in this topic came together and of course a part of that is fulfilling the capstone requirement but I think it's become so much more than that. For all of us we've said that you know we're very keen to continue to work together and we're wanting to make this a call for action to you as well today. And you know and we're wanting to invite more participation and more involvement from you and this is an opportunity I think as a community of practitioners can come together. So on the chat box I've put a very quick two minute form for you to stay in touch with us if you're interested to have more conversations. And we can think about three ways for us to continue to engage for you to engage with us so if you'd like to just be kept in the loop in terms of you know the future developments of this project where is it heading what are some of the findings. And you're wanting to just stay in touch that way then that's that's great. The other initiative that we're embarking on soon is around trying to get a sense a better sense in the renewal of the needs assessment. In the Australasian context around what are the topics that will be relevant for us as practitioners in this are we space and kind of the priorities that needs to be said so that's kind of a gender setting. So if you'd like to be a participant and for sure let us know and if even better if you want to be part of that in part I want to collaborate with us then you know be very very welcome and we'd be very keen to hear from you. So pop your name into the into the form and then we'll be happy to stay in touch. I think there are also opportunities, you know we see that the work in the collaboration with you know see a Wisconsin start as absolutely a strength to this collaboration and you know we are talking about opportunities in the future to kind of make this a resource for us as practitioners and open you know open access repository where people can access some of some of these works so. So yes, please you know we're wanting this conversation to be ongoing, and we're wanting to definitely have more conversations. So we have about eight minutes now for questions and answers and thank you again john for facilitating that and I might pass the time to you to feel the few questions for us. Great thank you Katina and thank you to our speakers. We do have a few questions on the chat room but before I get to those. I'll pose one question to all three speakers but not to answer it straight away, but to think about it and perhaps and towards the end. I suppose my question is, and in part, I know that I think it was. I think it's Hannah who said that she likes reflect on so what what does this mean step to get into the implications. So my question is, what are the, what are the immediate or implications for the as as a professional body. So I'll ask you in a few minutes to come back with if there was if this if there was one standard implication about the as as a professional organization, what might that be. But while you're thinking about that I'll now get into the chat room. And I know Hannah has already responded to this first question but I will ask it anyway. And that's from Karen which is that Karen said that the yes well the EGA had two special issues on the particular topic of our E, and the question is did you see and did you see similar concentration on the values topics in other journals and I know Hannah you responded and said, I think in short no but you want to elaborate on whether any differences across the journals that you saw in terms of the focus on values. Yeah, there was so with that question. There was the one special issue, but I didn't see other special issues. And in terms of the different journals and their focus points I haven't really done an analysis on how those different journals. You know whether the content they discussed was similar or different in those in that way so I probably have to have another look at that to answer that question with confidence. But yeah, thank you. So the second part of Karen's questions is similar lines but rather than comparing across the journals a little bit more the time horizon. If you did you notice any and you may not have been a specific area of study, but did you notice or have any sort of observations around whether there's been a shift across time in the focus or content regarding the journals on values. I didn't do an analysis on current at ours articles that they coded as values inquiry, and I didn't really look at the shifts over time. But when I went back and had a little scan of the time period we were covering there wasn't anything that stood out. So the articles were sort of across all time periods covering, you know different content so you might get something in 2015 talking about participatory approaches but then also in 2019 it comes up again. So I didn't see shifts in our time period but I think it would be really interesting to go back and have a look at that because I imagine, particularly with things like participatory evaluation. And the cultural context, those things I don't think would have perhaps come up as much in the previous review. Thank you. Another next question is well to Hannah and Steph, which is about the interchangeable to have have people use the terms around ethics and values interchangeably. And I'll add to that question. What's your take on those two, what differentiates those two terms, and where do they overlap. So first of all, have you, did you see in the literature, any confusion or interchangeability, but what's your both of your perspectives on the use of those two terms. I'd know who liked to go first perhaps Hannah, if you can go then I'll ask Steph to follow up. Yeah, such a great question. So rarely if ever did I see ethics talked about explicitly. But when I think about the themes that I've raised today that came out of that collection of research articles, I think there are many ethical questions and implications to some of the discussions that were had. So, yeah, it's a tricky one. Steph and I spoke about this throughout our process. And, yeah, I think that it was it was lacking, I think that drawing that line, particularly around I was thinking about the, the one study with the participatory situation where, you know, you can think about the harms that could be caused if we don't do it in a way that is authentic. And for me that brings up ethical questions. Steph, did you want to pick up on anything else. Thanks Hannah, I, yeah, can only echo that I will admit that I really naively came into this saying I said I think I said to someone, I don't want to think about values, I don't talk about value, I don't want to talk about ethics. And so, this is definitely, yeah, still a learning, a learning journey for me. I don't, I, yeah, I, I didn't read anything that alluded to the terms being used interchangeably but I think that the interrelationship between the two is, is really and I'm, yeah, still trying to unpick it myself. So to that second question, I'm not sure that I have a very, an answer that is articulate at all. That's okay. Katina, how much time do we have left because I know there's a few questions that might have a cherry pick. How much time do we have left. Right. Okay, I think the speakers are happy to stay on for another 10 minutes. But if we can take one more question, and then those who are happy to stay on could but we'll just wrap that up after the next, after the next question. Okay, not a problem. I might go to the question of gaps. I'm just scrolling back up now. There was a, we'll see it. Okay, this is from Santi Owen. Other than the, what other research gaps are given that you've all undertaken this review of the different aspects of ROE. What do you see as the main gaps that you would like to see address. I'd like to go first, but Kat or Seth or Hannah, from your perspective, what do you see to be the main gaps. Kat, do you want to go? Oh, sorry, I'm having technical. Sorry, apologies if you saw my screamer for I didn't realize apologies and we did that. Sorry, my bad. Yeah, I think for me, the tricky part was because we were looking at empirical research was we had to exclude a lot of articles that I think are really important and I think potentially for me in terms of the value space, some empirical research on values theory could be an area worth exploring further. Okay. And Steph and Kat, do you want to add to that? Yeah, I think I'm just looking at my, my Schwann and Gates book which has been, I've been reading over and over and I think that idea of a lot of the research on ethics thus far is kind of fit within a particular way of evaluating and I think there are those opportunities, you know, even though the research is patchy anyway, opportunities to sort of push our understanding a bit further and even, even elevate to the level of the profession rather than just the sort of individual professional ethics. That was my sort of takeaway from it here. Okay, thank you. So we don't run out of time because I know now we are out of time but because Tina said speakers happy to stay on for a little while longer. If there was from the worthy of done. Do you see any immediate implications for the AS should be doing as a body. Yes, yes, and just drawing attention to brands sort of reflections on that in the chat as well. I think one of the biggest challenges and why I was so grateful that that Steve, Kat and panel willing to do this as part of their study is, is getting funding to support work like this and I know there are many people on the call that are publishing our ways but for the reality I think for most of us practitioner academic or not, it's sort of an add on or I'll do it when I can and you can almost see that play out when we look at the patterns of publications. So, so that the big one, the other one that I would personally love but a yes to consider is to the way that the US team is really facilitated a lot of this is having a T get topical interest group or an hour language interest group as part of their association that is dedicated to research on evaluation. So that could be perhaps a more actionable implication that the association can consider. And maybe that's something we can all be a part of those of us who are on the call we were clearly interested but yeah just having a structure so that we can actually be talking to each other and even just sharing, not necessarily the load but you know the work of doing research in a more collaborative way I think that would be great. I think the annual conference to be an opportunity to shed some more light on this whole area. Yeah, absolutely I do. But I'm sure we yet everyone else is not into it. Yes, we're biased but yes we said definitely yes we have definitely see opportunities at the conferences there as well for sure. Yeah, so thank you everyone and thank you so much for coming along I think we've run over time as a testament of how rich a discussion we're having. I can already see people responding to the to the survey. Thanks John for so definitely guiding us and shepherding us through the Q&A. Thank you everyone for your time this evening. And it's been a pleasure to share our work with you and I think yeah like I said we're very keen to keep this as an ongoing discussion. The few of us are probably going to stay on for a bit more so if you'd like to have a chat with us, be very welcome to but otherwise have a great evening and thank you again for your time.


If Samba2Go has a flashing orange LED light that lasts for 5 seconds or more, its battery is low and needs to be charged. Plug the USB cable into the USB port. Plug the other end into the power adapter and then into a power supply. The orange LED will be illuminated during charging. When the green LED turns on, Samba2Go is fully charged. This takes around 2 hours.


It's Friday. Congratulations. Everybody's looking forward to a nice long weekend of working on assignment two. So today, we're going to talk about different ways to translate virtual addresses to physical addresses. So last time, we decided that this was a good idea. But we didn't know how we were going to do it. So today, we'll talk about multiple different approaches to doing this. We'll look at pros and cons of each. And we're driven by a couple of design goals, including, and most importantly, being able to do this fast, because this is going to happen a lot. Every time we use an address, remember, every address that a user program sees is now a virtual address. I never give them a physical address, so I have to translate every address, meaning that this has to be extremely efficient. All right, so there's one week left. Till assignment two is due. I can tell that at least some of you guys know that, because there are more people at office hours now, which is great. Sadly, there are no new members of the 8080 club to announce today. So maybe what happened is those three groups were the people that actually started when I told you to start the assignment. And so they finished, and the rest of you guys decided to wait until a week before it was due. And so I hope that you guys will get there by Friday. But we have plenty of office hours this week, and maybe we'll do something a little bit fun next Friday for the first big turn-in deadline, but we'll let you know. The midterm, as you guys know, is after break. The midterm will cover everything that we get through until the end of next week. So I'm planning, I have to see. I'm planning on lecturing through Friday next week. If we're a class ahead of schedule, maybe we won't have a lecture Friday. I know that sometimes people are traveling, but when you get back from break on Monday, we will do a review. The midterm will be on Wednesday. There are, at this point, multiple sample midterms with solutions up online from previous years. Those should give you a very good idea of what will be on the midterm and the format the midterm will be. Particularly last year is because we're pretty much on the exact same pace as we were last year as far as covering material. And the midterm is at the exact same point within the semester. Any questions about logistics? Hey. Okay, oh yeah, sorry. This was about no new members of the 8080 Club. The cat is upset. Please, make the cat happy. Do the assignment. All right, questions about virtual addresses before we go on and talk about how we're actually going to get this to work? Any questions about this whole new frightening realization that you had last class, which is that all the addresses that you've ever used have been fake? None of them were real. Yeah. Does this mess up caching? Does this mess up caching? Interesting. No. It does not mess up caching. Caching is typically done in a layer where it's allowed to deal with physical addresses. So the virtual to physical translation happens. The memory subsystem fetches memory. The cache locations are all done based on physical addresses. They have to be because at that level there's not necessarily even really a notion of a process. Remember my virtual address requires a process ID in order to make any sense. There's no need trying.  I need more information. And we'll talk today about the format that that extra information takes. That's a good question. Any other questions about virtual addresses? Yeah. Yeah. Are you going to do some live? Yeah, yeah. So the question was, why do we need extra hardware to help us translate addresses? And the reason is because software is too slow. It's just too slow. The only way that I'd be able to do this is imagine, again, if every memory instruction generated an exception that the kernel had to handle, that your computer would still be booting. It would still be booting if you started it a week ago. Memory's supposed to be fast. And today we'll look at, now some memory related instructions are going to generate exceptions that the kernel will need to handle. But the goal is to reduce that number to as few as possible. That's a good question. Any other questions? A little bit of review. So we talked about introducing a level of indirection. We talked about the fact that by forcing processes to translate a reference rather than allowing them to use some sort of lower level handle directly, this gave us a great deal of new control and it also gave us a bunch of new features, right? We talked about how we could stop a process from continuing to use a reference that we had given it. We can allow processes to share references that look different but might refer to the same underlying object. We can move the object that it's referring to because the fact that it has to translate this reference in order to gain access to it gives me a chance to undo whatever I might have done. This is something we'll talk about more next week. And finally I can also alter properties of the underlying object without the user's knowledge. So this was a good thing. The virtual apps, so we talked about as soon as we introduced address spaces we knew that there was something funny about addresses. And so we talked about the fact that now all addresses, the processes used are these virtual addresses and anything that a process uses that obeys the memory interface. So virtual addresses can actually point to things that will load, that can store data and will load data. Those things don't necessarily have to be bytes of memory. They can be bytes on disk. They can be bytes on another machine. They can be bytes that refer to a hardware port. So there are other things that I can now reuse the same interface for. And this is something that we'll come back to when we talk about files. Because the file interface is another interface that kernels have used and abused to refer to file-like objects that obey the file interface. All right, so we talked about how a virtual address points, a physical address points to memory. So when we talk about physical addresses we're talking about actual bytes of RAM. Whereas a virtual address just points to something that obeys the memory interface, something that acts like memory. And we talked about location, permanence, and protection. Whoa, slow down there. As new properties of virtual addresses, we could layer on top of virtual addresses because we've created this level of indirection. So here were the four places that we talked about where system calls that create virtual addresses. So the first one was exec. How does exec create virtual addresses? Yeah. Yeah, so it takes the L file and interprets the L file and the L file contains a blueprint that describes how this new address space is supposed to be created. Where are things supposed to go and what content's supposed to go at each virtual address, each range of virtual addresses within the address space. What about fork? So fork also created new virtual addresses. How did it do that? Yeah, just a copy of the parent, right? So these same virtual addresses, but with the new process identifier. So the process identifier virtual address pair that is required to uniquely refer to whatever the virtual address points to is now different. I've taken the same virtual addresses, those are identical, but there's a new process ID. And so I'm allowed to refer to different underlying objects. S break, these were two new system calls that we hadn't necessarily talked about. Or what does S break do? Yeah. What's that? Yeah, it allocates heap space. So it extends the process heap, it moves something that has been historically called the break point, which is the top of the process heap. This is used by Malik when Malik needs more memory. And finally we add mmap, which allows a process to create, to take a portion of its address space and ask the operating system to point it at a region in a file. And this is quite a clever thing to be able to do.  We talked about the MIPS memory map. You guys will come back to this when you do assignment three. So don't worry, see this again in recitation. Any other questions about virtual addresses before we go on? Okay, so remember in order to get this to work, so I've introduced this level of indirection and it's been clear up to this point that this is awesome. It's really powerful, it allows me to do all sorts of cool things. It allows me, gives me all these new capabilities. But how do we get this to work? And today we're gonna talk about how we can efficiently translate virtual addresses because keep in mind, now every address that a process uses is fake. Every address requires translation. And the goal here is to keep the kernel out of the loop. So somebody mentioned this in the early round of questions which I thought was great. Why do we need hardware to help us? It's just the kernel is way too slow. So remember the goal here was the kernel sets what? Remember this is one of our famous systems splits. Hardware provides what? Yep, all right, you have answered the puzzle. Should really get like a wheel of fortune type display for these sorts of questions. Yes, he has solved the puzzle. The goal here is to get the hardware to do the work for me because the hardware is gonna do it extremely quickly. But at the same time I need to make sure that the kernel retains control over how the translations are performed. So here's one approach at doing this. Here's one attempt. So when a process wants to use a virtual address, it in some way tells the kernel, here's the virtual address. Tell me what physical address this refers to so that I can use it. Can I do this? Is this something that I would be able to get away with? Who thinks yes? Okay, why? Yeah, but what's the problem here? What have I done? Remember there's something I didn't wanna do, yeah. Well that's bad, but let's pretend there was a way to do this quickly. Remember there was a rule that I wanted to follow here. I never wanted to give the program access to what? To physical memory, so this requires exposing a physical address to the process. And this is a no-no, I'm never gonna do this. The process will never know anything about physical memory, nothing. Every address has to be translated, all of them, right? I looked, it took me like 10 minutes to find this photo. I'm sorry, it's a little bit washed out, but yes, all your addresses are belong to the kernel. They are all translated. So here's another approach. The process says issues what it thinks is a write or read to 0x10,000. The MMU tells the kernel, I have no idea what to do. The process said to write to this address, I've never heard of this address before, what should I do? The kernel says, and this is when an exception is now generated. So here's where the kernel gets involved. The kernel is now going to tell the MMU, the underlined hardware, what the correct translation should be. It says, I know that for this particular process it's running right now, what it calls, memory address 0x10,000 actually points to whatever. That's because I've allocated the memory for it. And then the MMU says thank you and the story completes. This is essentially what we're going to make happen. Okay. Now the pattern of how this happens and how often it happens and how that has impacts on fragmentation and memory allocation is the thing that we're going to talk about. But this is essentially the workflow that goes on frequently. Now what part of this do I want not to happen? What would be ideal? So this is sort of the slow case. Yeah. Yeah, I don't want this, right? When the process tries to store to some address, I want the MMU to know what to do already. So I want to limit the number of times that the MMU has to ask the kernel for help. Because if this happens every time, then essentially I'm back to that world where I'm translating every single memory address and your computer is just too doggone slow to ever use. All right, so here's essentially how this works. I have this virtual address. The virtual address goes to the MMU, the MMU. And now at the MMU, let's say at the beginning of the time, the MMU knows nothing. Doesn't have any idea where these crazy virtual addresses that the process is using are supposed to be pointed to. So it asks the kernel, through an exception, it says, hey, this process is trying to use this address and I don't know what to do. So the kernel might say, here's the virtual to physical mapping that I've allocated for this particular process. But keep in mind, this is where that extra piece of information comes into play, which is what process is trying to use this virtual address. So once the kernel gets involved, the kernel knows what process is running on that CPU. And so it can add that extra piece of information that's required to make a virtual address into a physical address. I can't translate a virtual address by itself. I need the process ID. Once I have that, though, then I have enough information to translate it, assuming that the translation is valid. Now the next time, here's the nice thing. I'm running along and a couple of instructions later, I use 0x10,000. What happens now? Same program running. So a couple of instructions later, nothing has changed. Does the MMU need to ask the kernel again? No, it knows, right? So now I can just translate this immediately and this is what I want to have. This is what I want to be the common case, where the MMU knows what the translation is. It doesn't have to ask the kernel for help. Now here's something else that can happen. Later in my program, I try to use 0x20,000. The MMU asks the kernel for help. What else can the kernel do at this point? So the kernel can tell the MMU, oh yeah, I know this process. I allocated this translation for this process. It's allowed to use this bit of physical memory. What else can the kernel do, though? Yeah. Okay, so that's a great point. So I could tell the MMU a bunch of different allocations at this point. But what's it possible that's happening right now? What happens if the process hasn't actually been granted permission to use this virtual address? What can the kernel do at this point? It could destroy the process with a cartoon explosion, right? Or, more specifically, it can shut the process down, right? So this is how I enforce the allocations that I've granted. When the process asks to use a new translation, I'm allowed to look it up. If the translation is invalid, if the process is buggy, if it's trying to poke around and see if it can get access to somebody else's memory, the kernel has control, and the kernel can say goodbye.  And this is something that happens as well. So now let's talk about three different approaches. Oh, sorry, yeah. Yeah, so it's a great question. So on a multi-core machine, these translations are happening on a per-core basis. Now why is that the case? Why would I need a per-core, per-core MMU, yeah. Yeah, I've got a bunch of different processes running on different cores. So, and last I have some other way of telling the MMU the idea of the process, which in certain cases I might, depending on the hardware. But in general, I'm alluding translations for the currently running process. And so I want to be able to have different cores, have different translations. It's possible that if I have an eight-core system, every core has 0x10,000 translated to a different physical address, right? Because there are different processes running on all eight cores. Right? Mm-hmm. Yeah. So here's a great question. So let's say I got to here and it used it and let's say there was a contact switch. So what do I need to make sure I do every time the kernel switches between different threads and different processes? Yeah, I need to make sure that I reset the translations in the MMU. And I'm sure that you guys will make this mistake in assignment three. So in assignment three, you get to do this stuff. And one of the canonical mistakes in assignment three is forgetting to flush the MMU when you switch threads. And then it's really cool because the next thread is faster, but it's really kind of terrible because the next thread is referring to translations in the previous threads that are only valid for the previous thread. So that is not a good thing. But I guarantee a non-zero subset of people in this room will face this problem in assignment three. It's simple to solve. Yeah. Yeah, so OK, so this is a great question. So the address space has the same sort of potential size, right? But remember, going back to our ways of allocating, I'm going to regret this, ways of allocating virtual addresses, these calls have to be performed in order to create a translation for a particular virtual address. When I start running, like before I call exec, right, when there's a brand new address space, that address space has no translations that are valid inside of it. It's only through these system calls and also through the use of the stack that parts of the address space are loaded with valid translations. So in general, it's a great question. So in general, most of my address space is not mapped to anything. That's why, same thing with the bottom, right? We talked about why we don't set up valid translations at the bottom of the address space to catch no pointer exceptions, right? So that's a great question. Yeah, the address spaces are the same, but the parts of the address space that contain valid translations are going to be different in general between different processes. So great question. All right, I told you I regret this. Okay, so I did this. Okay, good. Any other questions? These are great questions, yeah. Yep? Uh-huh, mm-hmm. Okay, so the question is, how do I maintain all these mappings? Okay, welcome to the rest of the lecture, right? Like, I will, if I remember, you can ask me when we're done, but that's essentially what we're gonna talk about today. Or it's an aspect of what we're gonna talk about today. So clearly, doing this for every byte of memory, let's say I had a, you know, this is maybe the most obvious way of thinking about how to do this. I'll just have an array and the array will store for each byte in my address space the byte of physical memory that that translates to. And that's a terrible idea, because in general, I'm gonna have a lot of bytes in my address space, and so that table is gonna get really big. So one of the things that I wanna talk about today when we talk about multiple ways of translating virtual addresses to physical addresses is the space overhead of storing those translations. Because in general, we do not translate an arbitrary byte of virtual memory to an arbitrary byte of physical memory. We might want to do that, but that's not what we actually do. And one of the main reasons we don't do that is because you're right, most mappings would basically consume all of my physical memory. And this is sort of like scheduling. Any memory that I, the kernel has to allocate in order to do memory management is wasted memory. It's just overhead. So it's a great point. Let's talk about these approaches, and I think you'll see how this comes out. So in general, I'm not going to keep a huge mapping table that maps every virtual address to physical address. So let's talk about some other ways of accomplishing this. So the simplest is to keep, for each process, what's called a base and a bounds. So the base and bounds are both physical addresses. The base defines the bottom of the physical memory that I've allowed the process to use. And the bounds defines the size in physical memory that I've allocated for that process. And so for each one of these algorithms, we're going to talk about both, how do I check whether an address is valid? And second of all, how do I translate the address once I know it's valid? So here, what I do is I say, the virtual address is okay if the virtual address is smaller than the bound. Because the bound defines the amount of physical memory that I've allowed the process to use. To translate it, I take the virtual address, I add the base and I'm done. So this is very nice. Now don't worry, there's a picture. So here's how this works. So in physical memory, for this particular process, I've allocated it a chunk of physical memory. That chunk starts here and goes to here. So the chunk might start at some arbitrary physical address and it has some size, which is the bounds. You could also think of this as just top and bottom. But usually it's referred to as base and bounds. So the base is the bottom and base plus the bounds is the top of the physical memory region that I've allocated to this particular process. Remember the process doesn't see any of these addresses. What it sees is addresses like this. So when the process tries to translate this address, what happens? I just told the, I started at the beginning of time, the MMU had no information about what was going on. So it asked the kernel for help. All the kernel has to tell it for this process is two pieces of information. The first piece is the base in physical memory for this process. The second is the bounds, which defines how much memory this process has been allowed to use. After the MMU knows this, how do I translate this particular address? What does this address translate to for this process? Yeah, so base plus virtual address. Virtual address is OX 10,000, base is OX 40,600. And I've carefully chosen these examples to avoid any sort of carrying in hexadecimal, which of course everybody is terrible at. But anyway, so here you go. Pretty simple. Does this make sense? Very simple, simplest possible way of translating virtual to physical addresses. What about this address? I guess I lied, I didn't do that. This has an E in it, I'm sorry. Same thing though, right? So this one is now a little bit farther up into the address base that this one also translates. What about this address? What happens when the process tries to translate it? And so what happens? Boom, cartoon explosion, right? This is an invalid address. The operating system is gonna be notified. The MMU is gonna say there's an exception here. This process tried to translate an address that doesn't have access to and the operating system could do what it wants with that process at that point. So the nice things about base and bounds are that it's really, really simple. So on early computer systems, this was something that was extremely easy to implement. All the hardware needs to know are two numbers and it can implement this entire scheme. So the amount of information I have to store per process is pretty small and it's possible that I could even just store on the processor, as long as the processor has some idea of what the idea is, the kernel could just tell it this once and then I'd never have to tell it again. And so once it knows this one thing, it can translate all the addresses in the address space successfully. And it's also really fast to actually do the translation. So checking is just one comparison and the translation itself is just an addition. So if you think about just, you would implement this in hardware but still the complexity is roughly related to the arithmetic operations I would have to do. So the biggest problem with this is is this a good fit for our address space abstraction? Yeah, in general, so remember the address space was this idea that I had two GB or four GB or whatever of contiguous memory, pretend virtual memory and I could allocate things anywhere inside of that. And to implement this in a base and balance system, I would actually need to have two gigabytes of physical memory per process. Because the base, you know, so I'd have a base and then I have a balance two gigabytes up there and a lot of the memory in between would be wasted. So this is the other problem is that I have a lot of external fragmentation. So imagine this is my address space and imagine by some miracle I actually had four gigabytes of available RAM, contiguous available RAM on the system and I could put the base here and I could put the balance here and I might think, okay, I'm doing a great job but look at all this wasted memory. Actually I could put the base down here. I could even be a little bit smart, right? Look at and actually, you know, this diagram's not to scale. This would actually be a lot of virtual memory to have allocated. If you looked at, if you actually drew this diagram for bash to scale, you probably couldn't even see its code segment. It's so small. Just a tiny, tiny, tiny amount of data located in this huge massive address space. So in general, I can't use base and balance to implement the address space abstraction. It's just impossible. If I wanted to use base and balance or I'd have to give the process a much, much, much, much smaller address space and still suffer from all this external fragmentation. So in general, this does not work.  So we tried to keep it simple, but that was too simple. Base and balance, sounded awesome. Only two things, really simple to think about. Not going to work with the address space abstraction. So is there an obvious way to extend this idea to support the address space abstraction? Yeah. Yeah, so what if we give each process multiple bases and boundsings, right? So instead of one base and bounds, it's expected to cover the entire address space for each segment, each area within the address space that contains sort of a coherent stuff like the code, the stack, the heap. I allocate one base and one bounds and that base and bounds is designed to span that entire segment. So this is an idea that's called segmentation and we're getting closer and closer to explaining that mysterious error that you've been getting your entire life when you write C code. So we can assign every logical region, so we don't want to do this to too many regions because if we do it to too many regions, there's a lot of chances that we're gonna have to load new bases and boundsings into the MMU at runtime. But instead what we do is we say for each segment within the address space, I allocate a separate base and bounds, every one can be a separate size and I can also do things like I can protect each one differently. So I could say, for example, that the code segment was read-only and executable, meaning that sorry you can't write yourself modifying code, which is a real thing. I'm sad to admit, but it protects that segment from accidental modifications by buggy code. So here's how this works. Every segment, so I need one more piece of information, because before with base and bounds, it was sort of implicit that the start was zero. So here what I'm gonna do, every segment now has a start virtual, a start virtual address, a base physical address, so the start virtual address maps to the base physical address, and then a bound. So in order to check whether or not a virtual address has been allocated and is okay to translate, the MMU needs to do two things. The first thing is, well, really one thing, it just needs to find does a segment exist that contains this virtual address? And by that, of course, we mean that the virtual address is between the segment start and start plus bounds or start plus the offset. And to translate it, once I've located the segment that contains this virtual address, I take the virtual address, I subtract off the start of the segment, and I add that offset to the base physical address. So don't worry, again, there's a picture. So here I've got OX 10,000, the MMU knows nothing. So the first thing the kernel has to do is the kernel has to tell it something about a segment that contains OX 10,000. So the kernel could certainly tell it about more segments, but let's just imagine it just tells it about the segment that generated the exception. So the process is trying to translate this, the MMU is now given enough information to do it. So what this says is that there exists a segment within the address space that starts at OX 10,000, that maps to OX 43,000 in physical memory and has bounds OX 1,000. So where is this address going to translate once the MMU knows this? This is an easy one. It's gonna translate to OX 43,000, right? It's the bottom of a segment. So remember, the segment start address always translates directly to the base in physical memory. If I'm translating something that's inside a segment, I have to deal with a little bit of math, but the segment start address is the easiest one in the segment to translate. What about this address? Does the MMU know where this address is? No, because it's not inside the segment that I've defined. This segment is from OX 10,000 to OX 11,000. So this is outside there. The kernel might, but the kernel might say, oh, by the way, for some reason, this is dumb because this is a pretty small address to be able to translate, but I've also allocated a segment for this process that starts at OX 100 and has bounds OX 500. And the base address for this is OX 16,000 in physical memory. Where does this address translate? So he's saying up. That's cheating, right? It's above, right? No, how about a number? Yeah. 16,300. Does everybody see how we got that? I just want to point this out right now before we go any farther, which is that I am potentially one of the nicest people on earth. When I ask you guys about things like this on exams, it's always in base 10 math. So you can just thank me right now. You don't have to add or subtract hexadecimal numbers on an exam without a calculator. I'm serious. I'm potentially the nicest person on earth. I don't know anyone else who does that. Anyway, so we're doing this in class, it's text, whatever, but we tried to keep it simple. What about this? What about this address? This is a valid address, by the way. It's a valid virtual address. It happens to spell something funny, but it is potentially a valid address. But just for fun, what do you think's gonna happen when the process tries to translate dead beef? Yeah, the cartoon explosion. But again, this could have been a valid address. It's not just the kernel wasn't offended by the semantic meaning of this address. It's just not in a cycle. And here it is. Finally, the one thing that you thought you'd get out of this class.  That's why it's called this, because for a long time, operating systems used the idea of segmentation to translate virtual addresses. Now, we've moved on to something that's a little bit more modern, but the fact is that segmentation is still with us. And we still think in many ways about process virtual address spaces as being divided into segments. A segment for the code, a segment for the stack, a segment for the heap, a segment for the dynamically loaded libraries, even if we're gonna use a slightly better way of translating virtual to physical address. Okay, so this seems nice. This is clearly something that we can use to support virtual address spaces. We can implement address spaces using this translation approach. There's one problem with this though. There's a couple of problems actually. So let's talk about the things that are good about it. So it's still fairly simple for the hardware to perform these checks at runtime. Now, it's not quite as simple as base and bounds, because I might have multiple segments to look through, but the number of segments doesn't have to be very large, right? And the translation is very simple. Once I know the segment, I essentially am just doing an addition. Well, I guess I have to do a subtraction in addition to two operations. And it's also a nice thing that I can organize and protect the areas within the address space correctly. So I can assign, now that I've broken them up with this nice unit, I can assign different permissions or different protection to each segment. And it's also a better fit for address spaces because I don't have to allocate physical memory for those huge chunks of unallocated address space that are in between, for example, the code and the heap. I don't have to do that anymore. With base and bounds, I did. That's why it was such a terrible idea.  But what's the problem with this? Yeah. What's that? What's that? No, so that's the good thing. I do not have to give two GB of physical memory to each process. How much physical memory do I have to allocate to a process using segmentation? With base and bounds, I literally had to give it enough physical address, enough physical memory, the amount of physical memory that was equal to the size of the address space, the size of the virtual address space. How much do I have to give it in segments? Yeah. Yeah, essentially an amount of physical memory that's equal to the amount of virtual memory that it is able to translate. So, as soon as that it requests permission to translate some region of virtual addresses, I have to back that region of virtual addresses with physical memory. So, it's better than giving it two GB, but what's wrong with that? Yeah. Yeah, so there's one problem, which is I have these segments now that are these different sizes. So, what happens if I've got a segment that I've allocated physical memory for, and let's say it's the heap, and let's say that the process calls sbreak and requests some more heap. What potential problem can I have? Yeah. Yeah, so it might be already penned in in physical memory by two other segments. And so now, I either have to move it, which is terrible, because I have to copy all of that stuff into some other area of physical memory, just so that I can give it a little bit more heap. Or in other cases, I might have to fail due to external fragmentation, right? I've got these little, it might be that I have these little chunks of physical memory that are located between allocated segments, and I have enough memory to satisfy its request, but it's not contiguous. Could, now here's the thing. Could I move all the segments around to create more memory? Can I do that? What's that? Yeah, so given how we've set the system up now, can the OS defrag memory? Can I move segments around in order to create more space? Of course, right? This was the beauty of introducing the level of indirection. In order to move a segment, what do I have to do? I'm gonna move a segment from one place of physical memory to another place of physical memory. What's that? So I have to change the mapping. Is that sufficient? And copy all the content. So that's the ugly part, right? Changing the mapping is easy, right? That's just a couple of bytes that I need to fiddle with. It's the copying stuff around that's terrible, right? But I can do it. If I had to, if I had to have a system that worked this way, yeah, I can move stuff. I have to make sure that the process isn't translating that virtual address. I need to copy all the content somewhere else, then I update the mappings that I'm done. Talk about this more next week when we talk about swapping. So that's nice. But the real problem here, so this is a great problem about having this potential for external fragmentation, but the real problem is I still require the entire segment be contiguous in memory. I'm covering the entire code segment with a single chunk of physical memory, or even worse, I'm covering the entire heap with a single chunk of physical memory. And so if the process is using a little bit of its heap, all of the heap is still there. And so, oops, sorry. And so the, as somebody pointed out, the fact that the segments have to be contiguous means that I can have external fragmentation. I can also have internal fragmentation because I can have large parts of the segment that are not in use, right? So imagine that the process is allocated a bunch of memory on the heap, but it's not using most of it. It's possible I'd like to get that back so I can do something else with it, but I can't because the entire segment has to be in memory. And so I'm not quite there. I think the squirrel is actually gonna be able to grab this nut, right? It's hard to tell, you know, it's like, it's hard to tell which direction its little arms are going, but anyway. So you can decide what you want about how well the squirrel is doing. But we're not quite there, right? We'd like to do a little bit better. As far as how we translate virtual audiences, yeah. What's that? Yeah, so it's a good question. So the question is, stack and heap grow in different directions. And so it's possible, you're right, it's possible that I might have to allocate when I'm expanding a segment, I might have to actually either expand it up in memory if it's the heap and it's growing upwards or down. But all that means is that I can essentially collide with either the segment that's that way or that. The translation is the same. You can convince yourself the translation works, right? But you're right, I might have to move the, for the stack, what I'm gonna do is I'm actually gonna move the start address rather than the top, right? It's a good question. Okay, so let's go back to something that we wanted, right? At the beginning of the class, we said this was infeasible, but what we'd really like is to be able to quickly map from any virtual address to any physical address. This was the goal. This was the pie in the sky idea. And we said we couldn't do this because clearly this would take too much memory. But the other, and the other problem was this was going to be potentially slow because this would, so the other, so let's go back to, well, I'll just go forward here, I'll go backwards. The other problem is if the MMU only knows how to translate one virtual byte to one physical byte, then any time I use a new virtual byte of memory that it doesn't know how to translate, it has to ask the operating system for help. So that was one of the nice things about both base and bounds and segmentation is I'm limiting the amount of information that the operating system has to provide to the MMU. So before we talk about a better way to do this, I wanna introduce you to a new piece of hardware. So this is frequently what operating systems do. The only, and as software developers, the only time you should ask hardware people for help is when you're desperate, right? If there's no other way to do something, you've tried everything, you know, because we just don't wanna give them the idea that we need their help, right? But sometimes you do, and sometimes they can actually do a lot for you. So let's talk about a little piece of hardware, a nifty piece of hardware that allows us to do a much nicer form of virtual to physical translation.  We call those small fast things caches. And one way of thinking about your entire computer, actually, from the very, very top all the way down to the disk, is that essentially your computer is a series of caches, right? It's a Christmas tree of caches, from the top to the bottom. So here's one place where we're gonna use this trick. So we're gonna talk about something called the TLB, the TLB stands for translation look-aside buffer. No idea where look-aside came from, but it's a translation buffer. It allows the MMU to store translations and it also allows the MMU to search a number of different translations extremely quickly. And how this works is TLBs are a form of what's called content addressable memory. How many people have heard about content addressable memory? So, and you might think, what does it mean to be content addressable? So normally, when you address memory, you use a memory address. You tell memory where to look, and it returns the contents. Content addressable memory is backwards. You tell it the content that you're looking for, and it gives you the address. So it's almost like being able to search. And content addressable memory works extremely efficient. So here's an example. I've loaded these entries into my TLB, my content addressable memory. When I ask the MMU to look up this particular address, what happens is that essentially every entry gets searched at the same time. That's one way to think about it. There is a constant lookup. So rather than going one by one and saying, is this it, is this it, is this it, is this it, the TLB has hardware circuitry on it that allows it to look down every path at the same time. And that means that no matter how many entries are in the TLB, the lookup takes a constant time, which is awesome. That's pretty cool. So in this case, I'm gonna use the TLB to look up the entry that has the contents, OX800 and the TLB can tell me something about something else that's co-located with that. So in this case, it allows me to translate OX800 to OX306 or whatever. So you might be thinking, this sounds awesome. Why is it my entire computer memory, a content addressable memory? And the problem is that these content addressable memories are difficult to make really big. You can think, I mean, one problem is that essentially the hardware circuitry required to implement these scales with the square of the number of entries within the TLB. So it does not work. So it means that we have to keep the TLB or the content addressable memory small. I can't make this look up a billion entries all at once. That would be, that would cost an enormous amount. And probably you couldn't even build that. Okay, so now that we've done this new piece of hardware, let me sort of get to the end of today's lecture by sort of pointing out this tension that we've come to. So segments were too big. The problem with segments essentially was that they were too large. There's too much inside of a segment. It's really nice that segments, that there's not very many segments because there's not much I have to tell the MMU about the segment, about the entire address space really for it to be able to translate all the addresses. But the problem was that the segments were too big. Mapping individual bytes, on the other hand, bytes are too small. Okay, so here's the tension. My segments are too large. The problem is that there's unused memory inside those segments that I'd like to be able to deal with. Bites are too small. So the question I will leave you with for this weekend is, is there a middle ground? Of course there is, and we will talk about it on Monday. So enjoy your weekend, and good luck on assignment two. I'll see you on Monday.