Hello and welcome to this Lowy Institute live event. This is a series of events that form part of what we're calling the Long Distance Lowy Institute in which we communicate our content and analysis online while we're unable to do it in person. There's been great interest in this event. We've been very gratified to see so many people register for this event from around Australia and indeed from around the world. We have people dialing in from the United States, the UK, Israel, Singapore and elsewhere. I'd also like to mention that we have several Lowy Institute board members who are joining us on the call including our Chairman, Sir Frank Lowy, as well as Joanna Hewitt, Sir Angus Houston and Penny Wensley, a warm welcome to you all. My name is Michael Fully Love and I'm the Executive Director of the Lowy Institute. Joining me today are two long-serving and distinguished former Australian Foreign Ministers Julie Bishop and Gareth Evans. Julie served as Foreign Minister from 2013 to 2018 and she was perhaps best known for her resolve in dealing with China over the East China Sea and other issues and Russia over the shooting down of Flight MH17. Julie is now the Chancellor of the Australian National University, the first woman to hold that position in the university's history and she's joining us from her home in Perth. Gareth served as Foreign Minister from 1988 to 1996. He was known as an activist minister as well as an incorrigible optimist. I think that was Gareth's own phrase about himself. He later served as President and CEO of the Brussels-based Crisis Group from 2000 to 2009. Gareth was Julie's predecessor as Chancellor of ANU from 2010 to 2009 and Gareth joins us from his office in Melbourne. Before I go to our guests, some quick housekeeping. On the base of your screens you'll see a Q&A button where you can submit questions to the panellists. We'll be reviewing and moderating those questions throughout the proceedings. We've already received I think about 150 questions but please if you want to have a chance to ask a question, submit your question in the next few minutes and we'll gather them and put as many of them as we can to our guests later in the discussion. But first I have some questions for Gareth. So first of all Julie and Gareth, welcome both and thank you for joining us. My pleasure. Let me begin. Before I come to foreign policy let me ask you about Australia's response to COVID which looks pretty competent especially compared to other comparable countries. We've been through a period I guess in which our politics got smaller. Is it possible that our politics are getting larger again? Julie I might start with you. Thank you Michael and I'm delighted to be on this meeting tonight. I think the response of the Australian people has enhanced the government's ability to respond to this pandemic. Many people have taken personal responsibility in social distancing, staying at home, working from home and the like and I think that's greatly enhanced the government's ability to impose some pretty tough restrictions and limits on personal freedom. The initiative of a national cabinet has added to the effectiveness of our response to date. I think it's shown what COAG is capable of doing yet in this instance it really has worked. But also we pride ourselves on our competitive federalism and we've seen this at its best. Each state and territory has responded in its own way due to the different circumstances, different infection rates and the like and yet they've done it under a national umbrella. So I think the federal government and the state and territory leaders and their prime minister have all done a very good job in responding to an unprecedented pandemic. They have balanced the public health concerns with the need to keep the economy resilient. There are no easy answers to this. I don't think any leader could claim victory at this point. There is a long way to go and also the economic recovery will be difficult. But Australia has done as well, if not better than virtually any other country in the world and let's hope that that can continue and there has been a level of bipartisanship that I think has been a great relief to the Australian people recognising the sheer scale of this challenge. Well, I agree with Julie that the Australian response has really been exemplary and probably just as much more international recognition than we've so far received. We're right up there with Denmark and New Zealand and South Korea I think in the way. We've seen this combination of good behaviour from politicians who haven't been behaving like politicians at all, even though there's been some disagreement because the policy issues are obviously very difficult. We've seen a reliance on expertise, which has been rather diminished in public discourse in recent years, which is wholly salutary. We've seen a fantastic performance, of course, via the health-working sector, but we've also seen a terrific response from the community, the better angels with our collective community nature, I think have made themselves very clear through this exercise. Even the media has largely behaved itself. What I do hope is that the lessons we've learned from the virtues of cooperative federalism and cooperative nonpartisan politics can extend a little bit into the post-virus environment. If our parliament stops behaving like a third-rate, boardable house and starts more seriously debating policy issues, more seriously debating issues through committee reports and so on, then I think it will be a salutary change in our environment, whether that happens remains to be seen, but it's been a good exercise. All right, let me ask about one element of Australia's international response to the coronavirus, and that is Australia's call for an investigation into the origins of the pandemic in an effort to learn some lessons about it and prevent those pandemics from occurring in the future. Now, that's led to a split between Canberra and Beijing. So let me ask you about that, Gareth, you'd be in favour, wouldn't you, of an independent multilateral investigation into this epidemic? Absolutely, there's an overwhelming case for investigating, inquiring into the origins of the pandemic, its spread and that response to it in the handling of it, not just by China, but by every country, and there's equally a case for Australia exercising initiative on these global public goods issues in the way that we have done in the past. But I have to say, the inquiry thought bubble that was articulated by the government was very much an own goal and an exercise in diplomatic self-isolation as someone has put it. It was ill thought through operationally, it was ill prepared diplomatically and it played right into that stereotype of Australia as deputy sheriff to the US, because we immediately were seen as playing the US generated blame game. I think the way through this very briefly has been very well articulated to suggestion by Kevin Rudd publicly last week when he said there's obvious problems trying to get the Security Council or the World Health Organization itself or the Human Rights Council or individual countries doing this investigation. The best way to do it would be for the Secretary General of the UN to convene a high level panel of qualified scientists, including Chinese scientists, to do the lessons learned exercise that we need in relation to not just the origins of it, but the subsequent handling of it. Julie, two questions. First of all, on Garrett's point, was Australia right to go it alone when we did or should we have waited until we had more international support? And secondly, especially drawing on your experience with the international investigation into MH17, what are your thoughts on how the investigation should be shaped? Who should run it under? What auspices should it take place? Thanks, Mark. I think first I must say that I am surprised by China's response to the pandemic. China is a permanent member of the UN Security Council. China has a unique responsibility as a permanent member to maintain international peace and security. And without doubt, this pandemic is a threat to international security, given the disruption to economies and trade links around the world. And China should, in fact, be leading an inquiry into how this pandemic began, where it started, how it started, how it got into human populations and the like. It is regrettable that it's now descended into name calling and tensions and inflammatory rhetoric. And I think the goal of both Australia and China should be to reduce tensions. And I think it would be helpful for there to be some calm frank engagement between our senior officials at this point, because there's no doubt there needs to be an investigation. I would have hoped that China could have led it through the UN Security Council, but my experience with the Security Council is that because of the veto power of the permanent members, an issue like this can be defeated on the floor of the Security Council. And if China is not prepared to cooperate with an international investigation, an independent international investigation, without Security Council backing, there is no legal authority with China where that would enable other countries to go into China and collect evidence or interview witnesses and the like. So I think the analogy with weapons inspectors is inappropriate, because again, the weapons inspectors only go in by invitation. So they can be asked to leave if the sovereign nation doesn't want them there. So we need China's cooperation and support in order for there to be an international investigation. And I don't believe, given my experience in some of these international forums in the past, that we can coerce anyone into accepting an investigation that they don't want to cooperate with. So I think some very calm and considered diplomacy behind the scenes, not through the media, is required at this point, because it's in everyone's interests that we do learn lessons from this pandemic. Because one thing we know, this pandemic will come to an end, but there will be another at some time in the future. And the more we know about this one from a health, medical, scientific perspective, the better prepared we will be to respond in the future in a more effective way than nations have in relation to this pandemic. Let me stay with you, Julie. You talked about calm and considered diplomacy. Obviously, Australia is not going to back down from its quite reasonable request for an investigation. But at the same time, both sides, I think, have an interest in taking the heat out of the relationship. We're seeing, we're seeing what appears to be some economic retaliation from the Chinese in the last few days on Bali and now on meat exports. How would you recommend, how should the Australian government seek to find an off ramp to get the bilateral relationship back on track, even as we don't step down from our request for an independent investigation? Well, China regularly responds to criticism with threats of economic retaliation and nations can be placed in the freezer if China doesn't approve of the criticism or doesn't want to cooperate with a particular country. And that comes to official meetings as suspended or they don't take place. And we've seen other countries subjected to economic coercion by China, whether it be South Korea or Japan or even the UK at one point. China can also use economic retaliation just to do criticism in multilateral forums in the UN and Asia and East Asia Summit and the like. And I've certainly seen that. But what I think we need to do now is to behind the scenes, discuss with like minded countries. In fact, Gareth mentioned a number of countries who seem to have coped rather well in the COVID pandemic, whether it be South Korea or even Singapore at one point, New Zealand and others. And some of these countries could step up and show what global cooperation and collaboration can look like in the absence of global leadership. And I think that global leadership has gone missing in action in relation to this pandemic. So I would be talking to other countries, I would be then engaging with China at the officials level to talk about how we can actually do something for the global good. I'm sure that China agrees that there must be an exercise in lessons learned. And I would be fascinated to know whether if China, knowing what it does now, faced a pandemic again, what different steps it would take. Could it have taken actions that would have reduced the spread, particularly globally, because virtually every country in the world, apart from the Hanfield, have now reported cases of COVID-19. So I would be doing a lot of behind the scenes talking, getting countries who have a story to tell together, and sharing that information and encouraging China to take a leadership role. Gareth, what do you think about how Australia should seek to get the bilateral relationship with China back on track, given that we have a lot of interest engaged in it, as of course, to the Chinese? And more broadly, what are the principles that should guide Australia's foreign policy towards a country that is both our most important economic partner, but at the same time a superpower of 1.4 billion people that's run by a Leninist political party? How do we square that circle? We can't ever be a patsy for China's misbehaviour and be seen to be unduly sucking up for the sake of our own economic wellbeing when there are human rights violations, when there's military over assertiveness, as in the South China Sea, or when there are examples of shortfall and effective international behaviour, which obviously was the case, China's got a fair bit to answer for in terms of the way in which it handled the original stages, at least of the outbreak. But we're not going to get very far in playing the blame game and making charges and all the rest of it. I'd prefer, in the specific context of the virus, I'd prefer us to be talking about the absolute necessity for an international study of what went wrong and the lessons to be derived rather than an investigation on the premise that there's been some violation, which like a chemical weapons or shooting out of a plane, which needs investigation. It's a study that we all want. And it's very much in China's interest to be cooperative in that. As to the larger question of how we deal with China in particular, my mantra for a long time has always been that the way forward for Australia's relationship with China is for us to really focus on those areas of potential cooperation in what we call global public goods or regional public goods, which health pandemics is an obvious case, climates an obvious case, arms controls, another obvious case, refugees and population movements, international crime, trafficking terrorism. All those things are areas where we do have common global interests, where China has been by and large a pretty cooperative international player. And it takes the heat away from this ridiculous binary emphasis on security relationship on the one hand, the economic relationship on the other, and sort of steering a course between China and the United States. As always, when you're running into difficulty in any relationship, you try to find areas of common ground, constructive ground. And I really believe from very long experience of dealing with China, that there's plenty of scope for finding that common ground in this global public goods area. Gareth, let me ask you to go to the international level, you had a lot to do with international organisations when you were foreign minister and afterwards. How do you think the World Health Organisation and its Director General Dr Tedros in particular have performed during the pandemic? He's come under a lot of criticism from the US government, from Australia and other countries. How does his performance compare with that of his predecessor, for example, Grohal and Bruntland during the SARS crisis in 2003? Well, I'm a big fan of Group Bruntland and WHO generally has been an effective player, but any such organisations, always the prisoner of its member states, there are obvious sensitivities. And the way this particular issue has been handled, the WHO leadership obviously felt itself under huge pressure from China, didn't respond all that proficiently or that will, I think we had to acknowledge that. But to throw the baby up with the bathwater and say this is an organisation that we should run away from and not continue to thund, would be a grotesque overreaction. WHO's incredibly important organisation has been seriously under resourced for the job that we wanted to do. It has played a truly important role in multiple exercises of international health crisis in the past and it can in the future. And it's tremendously important on this as on many other fronts for us not to join the Trumpian cheer squad, talk about negative globalism, but to really focus on those international institutions of real worth and concentrate on building them up and not being too critical of where they've strayed in the baths, but focus on what we need to do to fix them in the future. Julie, let me ask you about international institutions. When you came to office, of course, Australia was a member of the Security Council and it formed a very unusual locus of your work as Foreign Minister when MH17 was shot out of the sky and Australia was holding the pen in the Council chamber and was able to lead the push for justice for our dead. What conclusions did you draw at the end of your time as Foreign Minister and from that period in particular about the importance of the Security Council and other international institutions for a country of our size? We often have almost a partisan debate in Australia about whether we're right or wrong to participate in these institutions. What conclusions did you draw from that period? I certainly drew the conclusion that Australia is best served by defending and upholding the international rules-based order. That framework of treaties and conventions underpinned by international law that was put in place after the Second World War to prevent a Third World War and Australia and many nations have been well served by that international rules-based order. It's not perfect by any means, but if it didn't exist today, you would want to create it. And so, of course, we use the presence on the Security Council to pursue our national interests in relation to MH17. I ensured that Australia stood for and was elected to the Human Rights Commission, the Human Rights Council. We became involved in a number of international organisations and carried on the work of previous governments because it is in our national interests to do so. I think the disturbing development that we've seen in recent times has been the almost decoupling of the United States and China and also the disregard by some nations of the international rules-based order or picking and choosing what parts of it are here to and other parts that they won't. And what I would like to see post COVID when we come through this global pandemic and we're trying to normalise our lives, that we continue to support an international rules-based order that does manage the way nations behave and towards each other. Just mentioning the WHO, I think what's been disappointing about the WHO is despite decades of planning and discussions and preparation for a pandemic, they were found wanting in terms of a global response because each nation has now gone back behind their own borders, even in the EU, to come up with a response. And I think many people were hoping that WHO would take a leadership role. I think its problem stems from the fact that it needs a much clearer mandate. Is it a coordinating agency that works with other nations to establish consistent frameworks in their health systems or is it a frontline service deliverer? And I think until such time as it is able to get a very clear mandate, it will continue to struggle to show global leadership. Gareth, let me take the discussion to the United States. I guess we can think of COVID in a way as a stress test that a lot of countries systems are being put under. And so far the United States is failing that stress test, I think. Do you agree with that? Do you agree that the US response has to COVID within its own borders has been unimpressive? And how does that affect your view of the United States as the global leader? Well, on any view, the United States response to COVID has been lamentable. This grotesque caricature of the presidency that we now have has let the country down, has let the world down, not just on the virus, but on multiple other issues. And I certainly fear for the future of the international system as a whole and cooperation internationally, if the Trump presidency were to continue. I think it's very important in terms of Australia's own foreign policy responses in the future. But we build our response around what I call four pillars, less America, more self-reliance, more Asia and more global engagement. I can spill that out in much more detail that we had time. I have my cart now. But certainly reliance on the United States for leadership, given the kind of leadership that we've seen over the last three years, is a complete bust. United States soft power is dead in the water. And it's going to be very, very hard to recover ground. My bigger concern, I think everybody's concern is what the whole future of the world order is going to be post-CODE. This is going to be a transformational event or not. One thing, one direction the world could take would be for an accelerated rush down that path of defensive nationalism, populist nationalism, protectionism, disruption of supply chains and all the rest of it. The other approach is to throw this to be a big wake-up call and the absolute necessity for cooperation, particularly around those global public goods issues that I mentioned before. Those things that Kofi Annan used to call problems without passports. But no country, however big or powerful, is capable of resolving on its own. It does require cooperative solutions. It is a chance that we will go down that particular course, as we did after the Second World War. But it is going to take the United States playing a committed leadership role, if that's to occur. And we're not just to rush off on the other direction. So a huge amount really does depend on this November election. If Joe Biden is elected, I think there's a chance of recovering ground, having a more decent global order. If Trump is re-elected, God help us all. Well, Julie, that's a nice point to bring you in on. You dealt with both the Obama administration and then the Trump administration. You dealt with both Secretary of State Tillerson and also Mike Pompeo. What was that transition like? How was that different from a sort of practitioner's point of view? And what would you say about Gareth's point of view that everything so much of the international order really turns on the result in November? Well, yes, I did deal with both administrations and got to know the respective secretaries and presidents quite well as Australian ministers tend to form those relationships with their US counterparts. They are very different administrations and they are very different times. I think that what has really come to the fore in recent times is the chronic partisanship in United States politics. And that has even affected the response to the pandemic. Yes, the president's response has been unorthodox, to say the least. And I think it would be preferable if he stuck to the medical and scientific advice rather than discussing some rather unproven and dangerous responses. But the president is dealing with a significant national crisis. There's a mixed pressure on him. His unorthodox approach is unsettling for many. And I think that we have seen both from China and from the United States a failure of global leadership. I mean, after the global financial crisis, of course, the United States chaired the G20 and and began the recovery, leading that recovery. I take Gareth's point about what's the world going to look like after COVID and after the US elections at the end of the year. I think that there are a lot of people talking about this is the death of globalisation. They predict that the end of globalisation, the walls will come up and trade restrictions will be put in place. Others say it's going to be business as usual on the globalisation of the flow of people and products and services and ideas that have given rise to tremendous economic growth and poverty reduction. It will continue as usual. Well, I don't think either scenario is realistic. Obviously, the economic forces of efficiency and economies of scale and competitive advantage will remain, but I think we're going to recalibrate the balance between full self-sufficiency and deep independence. I think we'll reassess the vulnerability of global supply chains, for example. In fact, the United States will encourage its allies to do that against China's interests. But I think nations will want to consider their own self-sufficiency. I think the biggest risk is going to be that of overcorrection, a swing back to the hyper globalisation of the 1990s or an undercorrection to allow nationalist protectionist order to emerge. And so let's hope that nations will advocate and measured and calibrated approach to the international rules-based order, which, as I said earlier, is not perfect, but needs to be in place with changes made, but needs to be in place so that we do see global leadership and a set of rules by which all nations can abide. Gareth, can I ask you about the impact of COVID on the climate change debate, which, I mean, climate change remains there. It may not be as urgent as a crisis as the pandemic, but it's probably every bit as important, perhaps even existential. You mentioned in your opening remarks that, at least in the Australian response, the attentiveness to expert opinion has been something that's pleased you. Is that a positive that we can take out of this, that perhaps expertise and the opinions of experts is rushing back? Or, but on the other hand, I guess that will be balanced by the desire of governments to generate as much economic activity as possible coming out of the pandemic.  Well, being an incorrigible optimist, I am inclined to believe that the impact is going to be positive, that there will be a much greater appreciation of the need to rely on expertise, less overt partisan posturing, I hope, on some of these issues, a recognition that on some of these big global issues, and it's not just climate change and not just pandemics, it's also the prospect of nuclear war. We are all in this together, and we've got to find cooperative solutions to this, and we can't just erect the barriers and look after ourselves and ignore the larger global responsibilities. So, I mean, it could go the other way. Climate could be a victim of a retreat into idiotic sort of building nationalism, I'm all right, Jack, bigger thy neighbour stuff. It could be, but I'm a bit more optimistic than that. I think that we are slowly growing up and the wake-up call that a crisis like the health pandemic has given us is probably going to be a big salary tree. I hope so, anyway. All right, let me ask one more question, and then I'm going to come to the audience questions, and let me put the question to you first, Julie. Let me ask you a question about international education given you're the Chancellor of the Australian National University. International education has become a big revenue source for Australian universities. It's become a big success story, actually, for Australian universities. But what do you say to those people who would charge that our universities have put too many of their eggs in one basket and they rely too much on overseas students for their budgetary revenue and in particular, perhaps, students from the People's Republic of China? Well, international education is a global market and I was Education Minister at one point in the Howard Government and I was acutely aware that Australia's attractiveness as a destination was a testament not only to the quality of our universities, but also our lifestyle, our standard of living and that there were many students around the world who felt that they would benefit from gaining an education and a qualification in Australia. And so significant benefits do flow to the source countries, as well, of course, to Australia. Not only do we get fees from these students and our universities rely to a certain extent, graduate or lesser, depending on the university on those fees, but it also adds to a rich experience on campus where you have students from all over the world mixing together at an Australian university. ANU has taken a range of steps to ensure that we can continue to engage with our international students. But about half of our international students are, in fact, close graduate students. And so we're doing online courses and distance learning and the like and keeping connected with our international students. So I'm hoping that there won't be a situation where revenues fall off a cliff because international students no longer engage with ANU. But as Gareth will be able to attest, because he was Chancellor at the time, ANU has limited the number of students, the size of our university. We just don't want to keep growing. We're diversifying the source countries. And we're also focusing on ensuring that we attract the best and brightest domestic students from across Australia. So I think ANU is as well placed as any university to ride this through. But I think that we will see a change in the composition of students, international students, in years to come, as maybe China will diversify its destination. Maybe South American countries, students then see Australia as a destination. But what we mustn't do is lose focus on the quality and standard of our international, of our education that attracts international students in the first place. Gareth, do you want to chime in? Yeah, look, I think international education has been a huge asset for Australia for all the reasons that Julie has mentioned, not least our export performance. But there has been probably an over dependence on that the university level, a number of universities perhaps become overexposed to international students. That's not really been the case with ANU. And I think we'll survive all this pretty well. But others are exposed. And too many universities, including all the GOA, the top universities, have become over-reliant on international student fees for cross-subsidizing research. And in the absence and the drying up of that market, whether for post-COVID reasons or some other larger political reasons, if that market does dry up, a lot of universities are going to be in real trouble in terms of how they're going to support high-level research in particular. And for some universities, it will be a real viability fundamental viability question. So I think this is going to have to be a wake-up call for the government to rethink its whole approach to the funding of the university sector and the higher education sector more generally. We're underdone in direct research funding. We've been over-reliant and international students. We've got to correct that. We're underdone in research and development generally as a country by OECD standards. And I think it is a responsibility to focus on that. We've been talking for ages about the possibility of the Chinese bubble bursting. And I don't think we should overreg the pudding about the Chinese ambassadors, you know, rather veiled threats to that effect. I don't think it's going to be as serious as that. But obviously, there are immediate implications for COVID. We've lost a good deal of this year's cohort, which is going to continue to hurt us all the university's income, higher education institution income for the next two, three years. There's going to be reduced mobility in the future. There's going to be increased uncertainty about the viability of travelling overseas, studying overseas. So all of these things are problems for us. The glory days, I think, are over. And it's time for the government to think harder about what its responsibilities are to the sector. All right. Thank you for that. Now, I want to go to some questions from the audience. We have collected, I think, about a couple of hundred questions. There's still an opportunity to ask a question if you'd like to. If you would like to do so, then please do it via the Q&A button. But let me put some questions to you. First of all, let me ask you a question from Ben Doherty, who is the foreign affairs correspondent for The Guardian. And I'm going to put this to Julie, because I think Gareth answered this question when I put it to him earlier. Did Australia rush to call for a COVID-19 inquiry? Should it have built a coalition of like-minded countries to push for the inquiry? Well, I believe that Australia needed to build support amongst nations who agreed with us that an inquiry was needed. Now, Gareth quibbled with an investigation rather than a study and there may well have been shortcomings in the diplomatic approach. But I think the whole world agrees that there needs to be a lessons learned exercise. Let me use some neutral language for lessons learned exercise and that in an ideal world, China would be part of it or indeed leading it. I don't think it's too late to encourage China to be involved in such an exercise. And I would be encouraging countries who have had a fairly positive response to COVID to date in terms of the infection rates and the ability to flatten the curve and the like. And for those countries to work together with China to see if there can be some parameters put in place for a study and exercise and inquiry and investigation, whatever you want to call it so that we can understand what went on, how it happened, how the virus got into human populations and what we would do differently if this were to occur again. So I think that there are a number of countries who have a lot to offer in that regard, New Zealand, South Korea, even Singapore, Sweden, others who have taken a different approach. Plus, we need to learn lessons from what happened in the EU and the fact that despite decades of cooperation, the EU nations were essentially went behind their borders to respond. So I think that there's a lot of lessons to be learned from different countries where whoever has experienced it and that this could be done with global cooperation. Now, I think we need to learn from others and do what we can to respond more effectively, more cooperatively, more collaboratively next time around because the one thing we do know from pandemics is that they do end but then another will occur at some point in the future. All right, the next question is from the the appropriately named Darcy French who was a graduate student at Sion's Poe in Paris. And Darcy asks, and I might put this to you, Gareth, what does COVID-19 mean for Australia's Pacific step-up and our relations with South Pacific states more broadly? Well, we've got a particular responsibility in the South Pacific states which we've under-exercised in the recent past. We've been playing catch-up because of a theory about greater Chinese influence and so on in the region. But if there's any area of the world in which we can make a difference and should make a difference because we're well enough resource to be able to make a fundamental difference to the performance of these countries and the survival of these countries and we've got a huge responsibility. So I'll keep the answer short because we've got a lot of questions out there. I think it's just again, a recognition that this is an area of acute Australian responsibility that there's a real positive role for us in the South Pacific and we've got to be a very, very visible player in that space. Julie, do you want to comment on the Pacific? I was just going to say quickly, I think that this pandemic has shown that there are many vulnerable nations around the world because this disease does not respect national borders, status or wealth. But interestingly, a number of countries that have not recorded any COVID cases include nations in the Pacific. Now, that might be for a range of reasons. They might have not the tracking or reporting or testing procedures in place. But one area where Australia must take a lead role in the Pacific is building their public health systems. I mean, I was greatly concerned when Ebola broke out in West Africa. What would happen if Ebola got into some of the South Pacific islands given the fragile state of their health systems? We need health security partnerships in our part of the world, particularly in the Pacific. And I think that even though a number of nations in the Pacific have not recorded cases, we should be very well aware of the fragility, vulnerability of their health systems and do all we can to build resilience and sustainability in health systems in our region. All right, I'll put the next question to Gareth. It's from Ben Bland, who's the director of the Southeast Asia program here at the Institute. And Ben says, what should we be doing to help our most important neighbour, Indonesia, a country with which you had a great deal to do as foreign minister, come out of this crisis sooner and in better shape? Well, again, I think it's a matter of being very generous with our aid and with our expertise, which is highly capable of making a difference in Indonesia as it has in the past. Indonesia has to be just a super high priority for Australia in the future. We didn't cover ourselves with any glory in terms of our role in bugging the Indonesian President's phone and a whole variety of other policy issues keep on erupting over the years to put that relationship at risk. But this is an occasion. And we did demonstrate in our response to the Aceh tsunami and other things that we are capable of stepping up. And when we do, it wins genuine gratitude, genuine affection. So I think rather than, you know, we're talking about billions of dollars of support, not just, you know, 20 million, 100 million, a desultry token amount of support, plus the expertise to go with it. Of course, it's got to be done in cooperation with the Indonesians. We can't just force this stuff upon them. They've got to want it and ask for it, but we should be extremely generous. This is probably, you know, our most important relationship to get right for the foreseeable future. Indonesia is going to be perhaps one of the four biggest economies in the world by 2050. It's been estimated as the biggest Islamic country in the world. It's a huge, huge, the important nature for us. So get this right and be cooperatively supportive, be warm and be empathetic to Indonesia's needs. All right. Gareth, while I've got you, I might put another question to you. This is from Brenda Nelson, who of course was a former cabinet minister and former head of the Australian War Memorial and now running Boeing Australia. And he puts this question. He says, Gareth, the world's attitude to China, especially in the United States seems as bad as it was after the Tiananmen Square killings of 1989. Yet we emerged from that. Are there lessons from Australia's response to Tiananmen that might be applied in the post COVID world? Well, it took time to get back to some degree of normality. I know it because I was part of the team addressing all that. But one of the things that we were able to do was work with China on a larger set of problems, including, for example, the Cambodia peace process and some where we had common cause to make. And I think my instinct on this is to focus in our relationship with China on issues where it's in China's manifest interest and in our interest in the world's interest to work together and where China will see us as not just interfering with the busy body or playing the, the, be sure of role for the United States or participating in the blame game or generally being a pain in the tail. But China sees us as the kind of constructive player that we are. None of this means, as I said before, that when China does overreach, over assert itself or misbehave in any extremely obvious way, as was the case in Tiananmen, we should be in any way backward in articulating our view of what they need to do to get things right and be very explicit about that. I've always been very strong on pushing back against overreach or over assertiveness in the South China Sea, for example, in a way that people might think it's a bit unlikely for a position for me to adopt. So it is a balanced exercise, but I wouldn't put Tiananmen and the response to COVID in the same category. I mean, Tiananmen was a grotesque and thuggish overreaction and the use of violence against protesting demonstrators for failure to respond effectively quickly as a different order of magnitude and moral turpitude. Don't let them in the same box. All right, I'm gonna put the next question to Julie Bishop. And it's from Latika Burke, who is an Australian foreign correspondent in London. And Julie Latika asks, based on China's threatened boycotts of Australia and now it's action on beef and barley, what advice would you give countries like the United Kingdom who are starting out their economic and security relationship a little behind Australia? Well, first I am hopeful that the issues regarding barley and beef are not directly tied to China's concerns about Australia raising the need for an independent investigation into the cause of this pandemic. However, the timing is rather unfortunate. And if it is indeed the case that China is seeking to punish us with economic retaliation because it disapproves of the way we raised the matter of an independent investigation, then I think we need to, in a very measured and calm way, decouple those issues and urge China to consider the case that we have in relation to barley, for example, where China is, I understand, accusing Australia of breaching anti-dumping provisions. And if Australia has a credible case, and I believe it does, to disprove it, then if China does impose tariffs, then we should take it to the WTO. And this is consistent with my longstanding theme that Australia benefits from the international rules-based order including most international institutions. So I would be encouraging China, as I did when we disagreed with China over its conduct in the South China Sea, that China should subject itself to the rules-based order, including UNCOS, the UN Convention on the Law of the Sea, to arbitrate disputes between nations. So whether it be maritime boundaries or whether it be a trade dispute, there are global bodies that have been set up to the very purpose of resolving those disputes. So in the case of the UK, I wouldn't dare to give any government advice. There's nothing more X than an X minister, but I think that the United Kingdom's commitment to the international rules-based order and encouraging China and all nations to take their disputes to the bodies and set up to that very purpose and providing support and funding for those bodies is absolutely vital and essential, as well as obviously building its bilateral ties. Now, the UK knows how difficult a relationship can be with China as well as any other nation. They've had complex and challenging times with China in the past, under David Cameron in particular, I recall it well, but as Gara said earlier, we build on the areas of commonality. We build on the areas where we can work with China. It is a different nation from ours in many ways. Our economic relationship continues to be strong and deep, but we have differences that we have to learn to manage, whether it be over conducting the South China Sea or human rights issues. I mean, I do recall at one point when human rights issues became quite a point of tension between us. We were able to form a ministerial level human rights dialogue with China back in 1997. That lasted for a good 20 years until China withdrew from it in about 2017. But we did have a forum, a bilateral forum, where we were able to discuss our differences without escalating into massive overt tension. Just quickly, while we're thinking about the United Kingdom, of course, you had a lot to do with Boris Johnson. He was foreign secretary when you were foreign minister. I remember you attended Boris Johnson's very amusing Loewe lecture a couple of years ago. How would you grade Boris's performance as prime minister and how did you feel about that remarkable period when Boris as the newly re-elected prime minister of Great Britain leading the British response to coronavirus was struck down by the virus itself? It was a remarkable thing, wasn't it? I was shocked, like most people around the world, to think that this virus could bring down one of the world's significant leaders. I mean, Britain is one of the largest economies, an important nation globally, and yet its leader was afflicted with COVID-19. It was just extraordinary and I'm quite simply shocked by it. Thank goodness he recovered and is now back at work because I think it would have been a terrible blow to British morale if their leader had succumbed to the virus in this way. But it also made it very personal and I think it has changed Boris's approach to COVID-19. It certainly made him say publicly how pretty it is of the frontline workers and he's essentially crediting the saving his life. But it was a shocking period for him and of course, he had a baby boy born shortly after, so the alternative scenario doesn't bear thinking about. But I'm so delighted for him that he seems to have recovered fully and that he's back in charge of leading the UK's response to it. I don't think any leader can claim to have had a perfect response. I think mistakes have been made by many and that's of course mainly because this is an unprecedented situation and I think it has exposed shortcomings in many nations but it's also exposed the strength of the NHS in the UK's case. But there's a long way to go and I think the economic impact is something we're yet to come to terms with. A global recession, how countries are going to come out of this and the impact of the restrictions that governments have imposed on businesses has yet to be fully appreciated. All right, Gareth, the next question is from you and it comes from Natalie Mobini. And Natalie asks, Australian academic Kylie Moore Gilbert has been imprisoned in Tehran for more than a year and reports of her situation are dire. How would you be handling this issue as foreign minister and what's the best way to make progress with a government like the Iranian government on human rights issues? Well, Kylie's case is extremely distressing and if I was in the government, I'd be certainly fully, fully engaged in trying to find some basis for a solution with the Iranian authorities. Who in my experience are doing pretty closely with them on the whole nuclear issue when I was running the International Crisis. Are not nearly as crazy or as obstreperously committed to human rights violations as the perception more widely would have it. There are internal divisions within the Iranian system between the hardliners and the more internationalist inclined. Australia's got itself into strife with Iran by again joining the United States cheer squad on too much of the pressure being put on Iran and with insufficient attention to necessity to restore that nuclear deal and all the rest of it. And we put ourselves in a very difficult position to make this, but it's not a matter of soft peddling on Iran's human rights violations. My general mantra on dealing with human rights violations in other countries is you do that which is productive. Of course, you don't hesitate to do that which is unproductive because it's sending a message and the international pressure does build up over time and make an impact, but you avoid doing that which is counterproductive and over its public diplomacy and over shouting on some of these issues can be very counterproductive for the very people you try to protect. So I'd be doing quiet diplomacy rather than noisy diplomacy in this instance and trying very, very hard to find some way through this. It's a very distressing situation. All right, the next question I'm gonna put to Julie and it's a question from Rebecca Hamilton about ODA. And Rebecca says that Australia's Development Corporation budget is set to fall this year with the government only promising indexation of the budget in 2022 to 2023. Given some research out of ANU that COVID could lead to up to half a billion people falling into poverty, do you think that's appropriate? I think that the response to COVID has put all our budget assumptions on hold. The government was quite determined to bring the Australian budget back to surface. Well, that is no longer going to happen. So I think tomorrow, oh, it's this evening, isn't it? That the government will be laying out the budget before coming years for the next four years, which will be vastly different. If the government is able to hold to the ODA commitment, then there is still a lot that can be done with four or almost $5 billion. If that's what it equates to my recollection is it's about $4 billion. If Australia is able to partner with the private sector, if Australia is able to partner with other nations, I work more closely with other developed countries in ensuring that we can direct aid to where it is most needed, being a lot can be achieved. But I think that this pandemic has turned on its head many assumptions. And I just hope that we are able to continue to devote sufficient funds, particularly to countries in our region where we have a particular responsibility and unique responsibility, particularly those in the Pacific, to help them reduce poverty and become economically sustainable. And that's why I hope that we focus our efforts. But so much more can be done. It's not the money that is the main issue. It's how effectively that money is spent and directed. And tackling issues like corruption in recipient countries is also an issue that has to be dealt with. And I think Australia has a good record in doing that, but we can always do more. Michael, I have to come in, because even though I don't want to draw any gaps between Julie and I who are in furious agreement about so many things on this occasion, her government did preside over a precipitate decline in Australia's aid commitment. We're now heading to something less than 0.2%, which is an all-time low to 0.2% of gross national income, where the international gold standard is 0.7. And when we were previously heading towards 0.5, and this is when the budget was in reasonably good shape, pre-COVID period and so on. I know from personal experience how hard it is to fight the Huns and Vizagoths of finance and treasury and so on. And all the budgetary purists who say that aid is not something we ought to be giving priority to. But this is a fundamental responsibility that comes with being any kind of a good international citizen. And Australia, really, if it's gonna hold our head, we're gonna hold our collective head up internationally as a good international citizen. We've got to do gigantically better on the aid front than we have in recent years and not use the budgetary trauma of COVID as an excuse for going further backwards. All right, we've only got time for two more questions. And I've actually got a question from my chairman that's just come in, Frank Lowy. And Frank asks this, and I'll give you both an opportunity to answer. It's back on the question of China and coronavirus. Frank Lowy asks, are we not naive to expect China to cooperate with an investigation into their behavior on the pandemic? Julie, why don't I start with you? Well, my expectation is that as things stand, China will not cooperate with an exercise that is seen as an investigation into China's handling of the pandemic, of the outbreak. There has been already much written and spoken about China's initial response, how they've tried to muzzle those who were drawing attention to the outbreak, the various theories about how it began, whether it came out of the wild animal wet markets or whether it came out of a lab in Wuhan or how it occurred. But I think that through quiet advocacy and persuasion, China can become part of an exercise in lessons learned and I wouldn't give up on the quiet diplomacy behind the scenes diplomacy, but I cannot see China agreeing to any investigation with the legal authority behind it of say the UN Security Council. I note that the Security Council's work agenda for the rest of the year does not include any resolution on coronavirus, even though it's a global threat to international security. So I can only assume that the Security Council will steer clear of this issue, whether it's through China's persuasion or otherwise, but it does have the veto. So I don't think we can be naive and assume that an investigation with the authority of the Security Council, the legal authority of the Security Council will be there. In the absence of that, the only other way to get China to cooperate so that we actually have access to China's information is through quiet diplomacy and persuasion. Frank is absolutely right that we would be utterly naive to think that China is going to cooperate in what is set up as an investigation into the genesis and initial handling of the outbreak of the academic. It's just not going to go there because it's been the subject of Trump's descriptions of it as the Chinese virus, the Pompeo's ideological blame game, and it's just not going to go down that particular path. But if you set this thing up, as Kevin Rudd for one has suggested, and I really think this is very sensible, and set up as a scientific inquiry, a study rather than a coercive investigation, what went wrong, not only in the original response to the first signs of the outbreak, but in the subsequent handling of it, when you also making part of that study, the handling of the reaction in countries like the United States and Britain and Western Europe and elsewhere in Asia, and not just China itself. If you set it up in that sort of way as a genuine lessons learned exercise, there's plenty of Chinese spokesmen who are now on the public record in the last week saying, of course, they've prepared to participate in some appropriate inquiry. And I don't think they're being totally dissimulating in that respect. I do think they acknowledge that we're all in this sort of stuff together and it is important to learn the lessons of what went wrong on this case because it's going to rise again. I'm going to take the chair's prerogative and ask one more question. It's a soft and cuddly question, but I hope you'll forgive me because it is quite rare to get two of Australia's most successful longer serving foreign ministers speaking at a single event. So let me ask you both. I might start with you, Gareth. What did you love about the job? What do you miss about the job of being foreign minister? And what didn't you love about being foreign minister? There's nothing I didn't love about being foreign minister, particularly at the time, which I describe once as, you know, bliss was at that dawn to be alive at the end of the Cold War period, the opening of the 90s, and to be foreign minister was the very heaven. Now, it's the best gig that I've ever had because you're in the room, you're dealing with big issues. Australia had some respect and some credibility and capacity to influence the movement on those issues. And you're also able to escape an awful lot of the domestic political circus. It's a great job, by far the best job I've ever had. And I think Julie probably enjoyed it just as much as I did, even though she had probably a less easy time of it internationally than I did when there was so much optimism about what was capable of being achieved. Julie. It was an incredible opportunity to help strengthen Australia's standing through building deeper relationships with countries and counterparts, advocating our position on different issues, arguing in defence of the rules-based order. I found it the most rewarding and satisfying and challenging position that I could have ever held. In fact, it was my secret ambition on entering Parliament back in 1998 to become Australia's foreign minister. And it took me 15 years to get there, but once I was there, I relished every moment of it. And it is indeed an honour to represent Australia as foreign minister. Because Australia is taken seriously as a significant nation, our views count, our perspectives matter. And I think that all foreign ministers feel a real obligation to use their time in the role to increase Australia's standing in the world and thus increase the security and stability of our nation. So it was an incredible experience and one I'll never forget. Best job, Michael. We both have had accepting Chancellor there in you. There you go. There you go. The Vice-Chancellor has just texted me to say thank you for that final remark. So let me close on that. Thank you, Julie and Gareth, for joining us for what I found was a really interesting conversation. Thank you, everybody, online for joining us for this Lowy Institute Live event. For the time being, we are live-streaming all Lowy Institute events, so please keep an eye on our website for the next event like this. Please also keep an eye open for our new podcasts. We have a terrific weekly podcast called COVIDcast and I have a new podcast called The Director's Chair in which I interview interesting people. We put out just yesterday, we put out the latest edition which has me talking to Enrico Leto, the former Italian Prime Minister. So ladies and gentlemen, we hope to see you all soon at Bly Street. In the meantime, from everyone at the Lowy Institute, thank you, Gareth and Julie. Thank you all for joining us today and stay safe and well. Good evening.


As Nigeria marks her 22 years of military handed over part of an elected civilian government in 1992, plus TV Africa's Jacinta Obuco went to the field to have a review of what June 12 means to a set of people popularly known as Millennials in Generation Z. 28 years ago, an estimated 14 million Nigerians, irrespective of ethnic, religious, class and regional affiliations in a period when religious acrimony and tension had reached its zenith, defied bad weather to elect their president with the hope of ending dark days of military detectorship. But today, how did the social-minded millennial understand the priorities of older generations? It's a day Nigeria was set free from the colonial masters. Democracy means freedom for the people, by the people and of the people. So I believe as young people we should be able to express ourselves. Democracy is tampered with, so it's just a normal day. I think every Nigerian remember that it's the day Nigeria gained independence from the colonial masters. June 12 is democracy day and it's the day we had a freedom to vote for who we want, whom we want at what particular time. Is this feedback something to worry about as regards the death of history among the Millenials? It's unfortunate that we have generations who does not understand what June 12 is all about. And who should we blame? We blame 30 sets of people, one, the governments. They didn't really emphasize it. They didn't put premium on it. They never saw the significant counts of June 12, so they never promoted it. That is why. Secondly, we should blame the original system as a non-value history. And this is also a policy issue, it's our government. You see that at a particular point history was banned in this country. Then the third level people are the so-called national orientation agencies. You want to ask me what are they doing, what are they orientating? Where is the national citizenship education? I'm not doing it. And on the rejection of the renaming of Union Lag after MKO, could it be that he was not really respected by both students and alumni? It is the coin, it is the coin of that name that was public, it was not public, it was that brand. So you put a malud, you know, malud in Yoruba means malu. And you know what students really like? They will say, ah, I am a malu, I am the student of malu. So those are the things those students will react against. And secondly, the academic field is slighted. And this is not the way to do it, let me also say this. All men of people using the name of an institution is bigger than personality. You see, even the name of an institution, I've seen that train, I think it's a bad train. Don't name an institution after anybody's. Because it's bigger than those people. It's bigger, people are still going to come and contribute. While the conversation continues on what constitutes history, individuals or people, the lesson here, according to social commentators, is that heroes and their exploits should not be discarded into the being of history. Just enter Obuco for Plus TV Africa.


Please make everyone sit. Welcome everybody. I am午 slightly.) I am a member of the USAWA committee. In the moment I'm going to provide today's acknowledgement country. As a committee we wanted to take a new approach for this part of our event. Felly, felly, mae gennym gael bod yn bwrt yn gweithio, Llywodraeth i ddodol o'ch cyffredinol i'r rhwng ac yn siaradau cymaint Passybl, ac mae'n mynd i bryd i gael'i llwg iawn i'w gymun.  Ac mae'r sefyllfa yn y rhan o'r bolluniau, I'm really proud to reflect this commitment by providing today's acknowledgement and by doing it in this way. As a personal reflection, I also acknowledge that I'm a Brit. A long time ago I came on holiday to Australia and I thoughtlessly climbed up Aes Rock and I called it Aes Rock but I've never heard of Uluru. Then I came to live here and I've lived here for nearly 20 years and I've since then certainly been on my own personal journey of understanding and recognition and it's been quite uncomfortable at times. I also work here at the commission and so I'm really proud to show you our beautiful acknowledgement to country. It's been written by our own Aboriginal and Torres Strait Islander staff members and it's quite new to us this is the first time I've used it. We acknowledge the first peoples of this nation, the traditional custodians of country, Aboriginal and Torres Strait Islander peoples. We value the unique and diverse cultures, languages, sacred knowledge systems and important connections to land, water and country. As we work to promote and protect the human rights of all Australians we walk with you on your journey for self-determination, for social justice, for respect and for recognition. We thank the elders both past and present for allowing us to continue to care and nurture this great land that we call home. Thank you and welcome. Thanks Jen, I think that was very touching and definitely very authentic so thank you and to someone who attended the evaluation conference as many of you would have this year in Sydney, one of the things that really struck me and was commented upon by many people was the really inclusive approach to Indigenous issues throughout the conference. Okay, so my name is Greg Masters, I'm also a member of the committee and have been responsible for doing some of the coordination of the free events and this is the last one for this year and can I encourage you to come along to the drinks that we'll follow at 5.30 this afternoon. We have had I think a really good year of free events on these Thursdays we've tried to put in practice the interactive request that people gave us to make them less talky and more involved and we'll be doing that today as well and one of the other suggestions we've taken on board is to, we're trialling filming today that does mean that people that are in the front here might get filmed peripherally so if any of you are uncomfortable with that you can get behind the camera line I'm saying that because we want to avoid the technicality of filling out forms and disclaimers and all that sort of stuff so we're doing it in a less formal way but if you are in any way uncomfortable it's going to be posted subject to the AES in Melbourne's approval on the AES website so it's not going to be broadcast more broadly than that. Just a quick overview of the program the committee has actually had a go at trying to program next year's activities so I'm going to go through these very quickly but to give you a tantalising glimpse of what's happening in February we'll be talking about evaluation in Buddhism which is a re-run or an extension of a paper that was very well received at the conference we'll be looking at a paper around complexity and how to evaluate complex programs in March April about evaluation capacity building and that will also, those three papers are all extensions of what was presented at the conference in May we're doing something I can't read oh I know what it is, it's the un-conference format you know, bring the evaluation challenges that you'll keep you awake at night ethical dilemmas in June indigenous evaluation issues in July economic evaluation in August the conference which you'll all be attending in Brisbane is on in September and we'll discuss in October some of the emerging trends from that conference in November we'll have, so 12 months from today we'll have evaluation of behaviour change so some of those topics need to be filled out but we've got an exciting program ahead and expect you all to turn up and all to become members because one of our goals is to have more members then you got it we're on the way if you're not a member I expect to see you become a member otherwise shame we're taking the trapping next time thank you very much thanks for the invitation to speak again so thanks for coming along to listen I'm talking today about a topic that popped up or didn't just pop up it was fairly prominent throughout the conference by the way I will make my slide presentation about it to see if I need to take notes or if I want to be thanked by this and it's a topic I've been working with and on over the past few years in a whole range of spaces and I thought it was time to give that interest from the conference to sort of talk about those issues and rubric and just to start with an example rubric has a couple of elements to it obviously it's got a scale that matches some change of progress it's an ordinal scale it's just different it's a progress in some sense that in itself is not a whole lot different between standard sort of a question for example what makes this a rubric is the addition of these statements that clearly describe the conditions that have to hold for something to give this range so it's the use of these clear descriptors for each point on the scale that make this a rubric so you've probably seen them just generally how many people have tried to construct rubrics in their regulation or other work and just by the sharp hand you can see how common they are coming in with my talk and the regular experiences and what you've learnt from them now I've used rubrics in a whole range of areas ranging from measuring symmetry maturity to behaviour change to assessing capability development capability training a whole number of different evaluation and evaluation like contexts and what I've noticed as I've tried to construct them myself with clients or reading the literature on what's out there is that there's some common underlying scale points that define the rubric scale but rarely do you see them all in one place so what I'm presenting to you is not something grand or invalible new it's just pulling together some common threads that appear across a range of rubrics in practice so before I get to that just some other examples just to illustrate the use people have arrived at rubrics through different disciplines in different practices different things some people might have heard of goal attainment scales which grew out of psychology and criminal justice and rehabilitation and mental health and they use these kinds of scales with descriptors attached to them for what each of these means similarly those of you who are calling for agency here the Australian Public Service Commission has rubrics for measuring capability development and there's a whole range of domains and some consistent underlying meaning to the points on this scale so there's just some other examples there's a whole heap of them that we can come up with so as I said what I want to draw out is are there some consistent scale points on these rubrics that you find that maybe provide us the basis for a universal scale you may notice in the title that because that might be something we can discuss in my stretching things to do here but we can have a discussion about that now obviously one of the common scale points you see is what I've called fully realised the outcome is there in its full form whatever that outcome is whatever that you want to see interestingly the one that rubric doesn't really have that and I find that's a big limitation when you rubrics have used if you were doing an evacuation is good good enough for the end of the year to be excellent to be good if nothing else I think for evacuation purposes a rubric that has the outcome defined in its fully realised state is a starting point to do the rest but it's also to have the utility making evacuation adjustments so you can think about what that would mean obviously at the other end of the sky my other common point is it's just not there it's absent it's kind of not happening whatever it is that's a fairly easy one to define interestingly you might think points that find the type of model of the scar but when I've worked with rubrics I occasionally find points that exist beyond those for example something might not be happening the outcome is not present not just because of benign reasons but because there's something actively hindering coming about so that cannot those kinds of things as oppositional forces can be either internal like a motivation like a tangibility or it can be external if I did that it would kill my career systems don't exist to support me doing that so those kinds of it's just not only not happening there are barriers to it feel free to jump in if you've got questions or comments or seen things like this as well at the other end of the scale something this I got me an insight when I was doing some work for the Commonwealth Department of Environment and Energy on measuring regulatory maturity when we were talking about where they want to be down the track they said in 10 years time we don't want to just be turning up to international conventions and learning what other countries are doing we want to be setting the stand for regulatory practice in this area so that I just want to realise the outcome they want to be leaving it or innovating they're not exactly the same things that you get a sense of they're doing something new or establishing best practice so they've got interesting points on the scale you're sometimes seeing that that could be very meaningful in some evaluation context in some context you may not actually want this but the manager is saying if we actually got to that we're wasting resources we're happy just getting here we don't want to get there we better use our resources so it's not always progress if we define it in its context but it's good to know and obviously in between is the meeting part of the scale which is you're on the way at the very least the common points on the rubrics that I've seen are started but really early days or we're making progress but there's still some work to do obviously you can expand between hearing into more points on the scale of that but they're the two I see most commonly as a minimum now what defines movement along that part of the scale usually I've found have usually resolved along two dimensions one is the breadth or the reach of the program or whatever it is you're doing if you want to raise capability among this group of people but we've only really got this far yet and we've still got to do it for this group you haven't reached your whole target audience yet the other dimension that can decide how far along this scale you've moved is quality so we've reached everyone but we've done it really badly in an evaluation context that can often be also determined if you've developed a nice outcomes hierarchy and outcomes chain say we need to raise awareness first and then with awareness we can increase knowledge and then after knowledge we'll get some skills happening so those points in your outcomes hierarchy or outcomes chain can define the points of the scale through which you move that's the other way of doing it that's just to give you an example and I really want to stop for any comments first just one thing about what are we describing or what are we rating the outcome right it works well but from opposed how are we describing the outcome when we say opposed it looks like it's something opposed to reaching the outcome rather than describing the outcome itself you're right it's not the negative it's not going in the wrong direction it's something that is an active barrier to it it is strictly speaking a mathematician it's not the same thing but for a valuation purpose it's often found that especially I'll talk about again from a decision making point of view knowing this is important the opposed also be a realisation that the outcome is not necessarily a good direction there are other directions that are better it could be this gives you a measure like any other measure what you then do with it how you interpret it how you judge when you place on it is up to you as a context but certainly gives you a start to thinking about that so just an example if you're interested in bringing about a certain kind of behaviour change I know we'll get to that in a year's time we're going to talk about this in more depth but just later you can sort of see the kind of statement you might come up with this is from a direct example that I've developed for sustainability what you might the terminology might use for the labels if you want to measure behaviour change if you've got any questions or comments on any of those they're likely to be different so over to you on your timings you have a good paper someone on your table turns it over what I'm watching to do we want to develop a rubric collectively here that's going to measure this aspect of evaluation, practice and culture I actually have done this for the Western Australian public sector commission before I did a capability building workshop I developed a rubric with them and we applied it and it allowed me to focus the kind of training and capability development I did on the basis of I want to see what you come up with each on your table so I'll take this one for example I want this table to come up with a statement for leading or innovating as their point on the scale for measuring this quality so each of you has a different point on the scale so start by talking among yourselves get to know each other write something down and then when you're ready to show to the rest of us go to this metameter link with this code and type it in because it will then come up on screen for us to be able to talk that out there you go we'll take 10 minutes to see how we go it's strange I hear today this is a straight for you it's fine it's fine it's fine it's fine it's fine it's fine it's fine it's fine it's fine it's fine it's fine it's fine it's fine it's fine ...dan yn y gallu'r anod. Ac ond, mae'n ddiwethaf ar rydyn ni gan o'n ddweud y gwyntaf gyda'r chweineidio cyfennol o'r ysgol. Yn y peth yn eu colliwn i gwbl swyddiad, yn yr anod o'r Llywunodau Rhywbeth, gofyn yn gweithio'r gŵr, oherwydd mae'n ddewisio ar gyfer y ddysgu staffol i'r maen nhw, yn y bwysig o llawer o gwylltiau'r model gweithl, oherwydd mae'n gweithlo'r modd bobl llai'r organ. If that was all happening you can go in and say, in my organisation, this statement describes the situation we can say that the outcome of whatever that outcome is using evaluation program design is cwydli will realise. A remove your understanding. Anyway, can you leave a comment or don't go to a pick it up? I will not hear that. I'm saying not a wrong statement because I got it right. However, other people are Conclude in if they were designing in that point on the scale we realized that. Woo. You would have whoo. Yes. And the discussion about the unit of analysis a point I was going to make maya that can be a very tricky one. You are talking about individual. dyfodol, fe fyddai gwahanol fel yw'r anhulu'n rwy'n rhan i ddweud, mewn i'r unig ysgol yw achos. Ymgyrchol maen nhw a'r wlad am gweithio canir ac ysgol, lle be oeddedd yw'r ysgol yn ni o'r unig, ac ar y pethau gwahanol. Ysgol o'r rhan i'r unig. Felly mae'n gweld ei alwch yn ddeg, ac mae'r dal yn oed yn dyma. Mae ydw i'n mynd i ddysgol am dyfodol am ddere anhulu'n ddere, a ydw i'n mynd i ddere o'r ysgol. Roedd yma eich… Ymgyrch yn ddullent y Moedd Grimiau i'r Ymwneud o'r bobl a'i ddweud. Yma eich 2bll�teriaeth. Gw pellwch yn ddim yn ddefnyddio gy prisoner i'w gw своod. Gw chillio'r amgylchedd i fath o'r mewn yn gwneud. Mae'r gwneud o'r amgylchedd i'r ond yn gallu gwneud. Mae'r gwneud o'r gwneud o'r ymgylchedd i'w gweld sy'n ddullion. Mae'r gwneud o'r wneud o'r ymwyaf yn dweud, ond rydw i'r ceisio'r ysgol iawn, rydw i'r bwysig gyrfa yllun? Dwy o'r ddweud o'r ysgol iawn, dwi'n gwybod y pethau ymlaen, ymlaen yw'r ysgol iawn,au'r gweithio'r ysgol iawn. Rydw i'n gwneud ymlaen arall. Rydw i'n mynd i'r rhwng o'r ffordd. Rydw i'n gweithio'r bwysig eich rheiniadau'n mynd i'r rhan. Rydw i'n mynd i'n gwneud ymlaen i'r rhaglen o'r rhan. The suggestion to it is that it is a collaborative process because it sometimes takes a bit of time to get things but I think it is a good idea to do something beyond what is minimally required for the outcome to save the success. Ion y cyfan o bryd wedi bod ychydig i'w ddweud yn ei wneud atgymu'r newydd? idinol o'r Cymru? Yn 3. Felly mae hynny ym yn arfermwyr. Mwneud o'r ffordd o'r proses cymaint i Cupysg Bryddi, mae'n llwyll dadle o'r rôl. ym Mhwyl S1rwm yng ngwybodaeth ym Mhwyl Rheidiw, newydd ym Ym Mhwyfgolwyr Mayfwrdd, gan hwyl iawn i'r ychydig i'r mhwyl yng Nghymru, astud am fawr o phae Maslaithaeth yn eu schylystaeth, yr Ym Mhwyl Rheidiw wedi gwneud allwy ni yn ym Mhwyl Rheidiw. Ym Mhwyl Rheidiw yr Ym Mhwyl Rheidiw, ac mae'n bellig yn eu hanechyd yn ei hytrifalu ym Mhwyl. De�iwch chi'n hyfrifwyr ar ysgilio'r���g. Mae'r prysyn, mae'n dderfynnu i ddweud, ac mae'n ddiwedd yn rhywbeth ffordd i'r lluniau hynny, ond mae'n cael ei ddechrau i'r rhannu. Felly efallai y rhai i ddychrau, roedden nhw'n gwell-nhynghwm, ond rydyn ni'n gweithio i'r yr hunrhyngau yn y bydd y rhannu. Rydyn ni'n dduld iddyn ni wedi eu sgw戴. Rydyn ni yn ymweld i sut ni wedi dduldol, mae yna'r rhannu i ddweud. and encouraging an inconsistent knowledge, culture and practice of evaluation. Evaluation is sometimes being used in program design, but maybe limited with lack of awareness, lack of resource or lack of formal process. So yeah, again, we've developed this one sort of down a little further and it's identified some of the areas which you need to think of if you're going to get through the gaps Any comments, thoughts, especially for those of you who developed it, or what was in your minds? Last one, making progress. There is an awareness of evaluation components, the benefits of evaluation and how they piece together. It's only awareness at this point, which is interesting. Some more all of the evaluation components are used in program design, but the use may be thought felly mae'r llwyfodd yma. Mae Fyfyrdd yn fawr, bydd yn rhan o'r perthynag, mae'r cymdeithasol yn ei ffordd, mae'r ymddangos yn eu ddweud yn annog. Mae'r ymddangos wedi'u gweld yn olygu'r ysgol. Ydw i'r ystod yn cyfrwng ysgol, mae'r bwysig yw sut ddwy'r hyfforddiad, mae'r bwysig yw ffordd, mae'r bwysig yw'r bwysig, a'r bwysig yw'r bwysig yw'r bwysig, it means however what is inconsistent tobleguorter is only done in half the number of few or sentence, tries to identify just where and how well and what is it… That's an exercise where I don't know what you think so I'm going to be working on the discussion now as exercising, get You to start with that so what are your thoughts or comments nor the actual details at all. Rwy'n fawr o'r sahen yng Nghymru. Rwy'n fawr o'r hyn o'r ysgol, o'r fawr ddweudio'r ysgol yng nghymfawr, yn y cerdd yma i'n ddechrau'r ystod yma. Ieithio'n gwneud y ddechrau'r ysgol yw i'n cymaint. Mae'r ysgol yn y gwerth o'r ddweudio'n ffraeg, o'r ddweudio'n ddweudio o'r ddweudio, o'r corwchio, a o'r ddweudio'r ddweudio o'r ddweudio'n eich ddweudio. Those should actually. When do you use one global scale for everything, or at what point do you break it up into multiple scales? Because I've seen some of that stuff that we've just talked about, even break it down and it'll have a separate routing for the quality of programming logic. Each thing you add in a statement might itself become the subject of an infra-judgment. Mae'r ailgo yma o'r ddaeth gyda myfyrdd o'r myfyrdd yma hynny'n i'r gwirio'r mynd, a'r meddwl i'r meddylidio drafodol i ddechrau sydd gyda myfyrdd yma, a wnaeth y gallai yma, a'r gwirio yma, a'r theib yn y byddodol i'r gofyn i wneud gyntaf? Mae'r pwysig hynny'n i'r myfyrdd yma e'n myfyrdd o'r gwirio. Mae fryf yn ôl eich cwrnod o wneud fynd a oedd i'r holl a ffyrdd o rhyw y'r gwirio, ychydig yn dweud o'r ffordd yn ymweld. Fy fyddai'n gweithio, ydych chi'n ddweud ystod ystod ystod ystod ystod, rydyn ni'n fyddai'n gweithio, a dyna'n ddweud, ond dyna'n ddweud. Yn y gallwch y ddweud, ydych chi'n ddweud ystod ystod ystod ystod ystod, ychydig yn ddweud ystod ystod ystod. Felly mae'r ddweud yn ddweud hynny, Mae'r ddifonwch yn ddifonwch, felly rydyn ni'n ddifonwch yn gyndeithiol i'r gwaith efo'r rhan o'r ffordd o'r cyflwng. Mae'r ddifonwch yn ddifonwch yn ddifonwch yn gwybod, mawr i'n ddifonwch â'r gwaith i'r ddifonwch. A'i ddifonwch, sy'n gweld cael ei fod yn goreithio'r ddifonwch, mae'r ddifonwch yn ddifonwch. Mae'r ddifonwch yn gwybod i'r wneud o'r ddifonwch yn gweithio'r ddifonwch. Ie ddod o'r ddweud o'r llwyddiad ymweld ychydig i ni i chi ddweud y dyfodol yn ymgrifwm, ac ydych chi'n rhai ddweud y ddweud o'r gwaith dim ond yn eu fathio. So, mae'r ddweud o'r ddweud i ddweud i chi'n rhai ddweud eich ddweud. Felly, rydyn ni'n rai'n gyfer y clywed? On that, do you ever get around that by offering another field that says other or explained why you chose that field? I got it! Gofyn, diwrnod! It just so happens! It wasn't a plan. I think there are limits to rubries however well you construct them. If you do construct them well, you can follow up and add really good intelligence and qualitative measures to them. If someone does tick a post, you can have a Q and Q that says, what is it about your situation that led you to that? felly i sut gwirionedd iawn y cysylltu, a müddio cyflwyddwr iawn i'r ddaluniaeth yn ei wneud. Oherwydd ta wedi bod angen amwylo i ysglwpio'r stafaelau, ond mae'n cael eu clwyddoedd y gwirionedd iawn i'w cysylltu'r ddaluniaeth. Mae'r ddaluniau gwirionedd iawn i'r gwyfn monitor. Roedd y bwysig yw'r gwaith y byddai'r syfydel, ond mae yw mewn ysgaf yn y cavell, Yn y gallwn, ydych chi'n gweithio'r sefydliadau? Yn y gallwn, y byddai'n amser i fynd i'r cyfrins o'r programau sy'n cael ei gwybod o'r gweithio. Yn y gallwn ymdweithio'r ymdweithio? Yn y gallwn y syniad, ymdweithio'r gweithio'r gweithio? Yn y gallwn y byddai'n gweithio? Yn y gallwn y byddai'n gweithio'r gweithio? Os ydych i'n dweud o'r cyffredin iawn o'r cyfnod Gwylig Llywodraeth, rwy'n cael ei wneudio cyhoeddfa am gweithio cyfnodau o'r cyffredin iawn, cyhoeddech chi'n dweud o'r cyffredin iawn o'r cyffredin iawn? Yn gyfnod i'r ffordd o'r cyffredin iawn o'r cyffredin iawn, mae'n dweud o'r ddiwedd y ddau cyffredin iawn o'r cyffredin iawn. I like having it all out on the table, but I've seen in some organisations the reluctance to give words to that, and so in one large New South Wales Government department there's a framework that's used for self-assessment across hundreds and hundreds of different sites. And it goes, delivering is the first category, and then excelling is the third category. And then I reckon they ran out of up in the middle ones called sustaining and growing. But there's nothing, there's no definition of what sits to the left in what delivery. So when those sites do self-assess, they can self-assess as this fourth category, which is known and trained as working towards delivering. But there's no descriptors for what that is, because there could be as many reasons for being working towards delivering as they are in different places. But I just wanted to share that as an example of, it's kind of half of full delivery, but that's leaps and bounds ahead of not having anything for self-assess. So it certainly, you know, I prefer having it all out on the table. But that's me, and I'm not the person doing the self-assessment. We know, when I did the WA public sector commission, they were already having problems with their diet of people, and they asked people about problems with that diet of system. It's kind of scratchous, or it's already free and more. So, yeah, there's sense of timidies around that point on the scale I want to think about, and change the language somehow, or assess it in some other way. But yeah, this is all just suggestions. Like I said, there's a question mark at the end of the title of this thing. I might not be completely universal. Some issues with using it, I know it takes a little bit different. They've just not come.  I find that it makes self-assessment a little easier and clearer because there's an underlying consistency for the scale points. If you go back to that very first one, if you actually look at this, the only difference between excellence and good is the word very strong. It's kind of shifted the subjectivity in about whether you call yourself good or excellent into the descriptor. The descriptor hasn't actually minimised the subjectivity in my opinion. Because these aren't really anchored in anything that people can easily grasp. Whereas I think this kind of scale is a lot more intuitive, it's a lot clearer about what it is. Oops, sorry. If you make self-assessment, there's still subjectivity to it. This is consistent with the cross-gins. You can also use it to allow other people to make an assessment. If you've got a field team saying all I would use is the same rubric because there's a consistency in the underlying scale. If I think it would produce a greater liability, a little more experts can use it to make performance or whatever it is. Or you can use some hybrid scoring between self-assessment and expert or other assessing. The other thing is you can use the rubric itself as a data collection instrument. Give it out, so please make yourself. Sometimes that can be hard or it can be confrontational as I mentioned. The other way of doing it is you can construct a series of survey questions. For example, they might just be simple yes, no questions. There are many things happening in your organisation. Depending on the sum of yeses and which knows, you then impute what that means for where they sit on the rubric. The people filling out giving you a survey question, you then amassat and synthesise it to determine where that sits. You can bring in other data. Some of the statements might say at least 50% of our programs you just get that from an administrative data, for example. So it doesn't all have to be assessment in that sense. So there's a range of approaches to actually determining where units of analysis sit on the scale which you can construct. Lastly, just some closing observations we might again open up for discussion. As I've said, I flagged it with a question mark, but I have been surprised myself by the range of applications. Again, maybe I'll become a Colton. I'm pushing something in there too. But I'll be interested in hearing from your particular context and the kind of outcomes and evaluations you work with, whether you think that could have helped you and that could have worked in that context on that. That would never work. Now, I can tell you a situation where I wouldn't use a rubric. It's where it's already well established measures for something. If I was measuring recidivism for prison rehabilitation, I'm not going to use a rubric. I won't tell me if I'm wrong with people working in the justice case. I think there's reasonably good reliability measures for that. Economic development. Well, there's some good reliability measures for that. So I think that's one area where I would push it too far. But I'd be interesting to hear from you where you think could be used to fill some measuring gaps. The units of analysis, which are sometimes different from the units of data collection. What do I mean by that? You might want to analyse government agencies and how well they're doing the evaluation. But the unit of data collection are the individuals working in that agency. How do you aggregate the information from the individual so you can say something about the unit? Now, these are ordinal scales and there's mathematical limitations with adding them and averaging and all of that. The 50% of people have to say something and all sorts of issues around that. One of the interesting things about the exercise we did where we had different tables giving different statements is it can expose different perceptions about what's going on and what even outcomes mean among key stakeholders. And that can actually be an interesting thing in itself. So, for the Victorian Department of Education, I got policy officers to write their practice and evaluation and I got senior managers to measure on the same rubric. And guess what? Anyone anticipate that as I see it?  The managers thought things were going well. The policy officers are not so much. But that allowed a conversation to be had with the managers about this is what you say, this is what the ground trips say. What does that mean, what are you going to do about it, if anything? But at least there was consistency about the base scale that allowed that comparison. And that's why I think having this common underlying consistency to the point on the scale that allows comparison. We'll talk about the idea if you have one rubric or multiple rubric, so that a global scale will break up into dimensions and then synthesise the information yourself separately. And lastly, I want to think, the great value I've got out of having these consistent underlying scales is allowing you to link a performance recording and decision making. This is something I've been with Duncan recently, because I was doing this stuff with the State of Victoria, and I developed the scale and Duncan and I said, well what could all of this mean for decision making if you've got this point on the scale? So these are the labels we've used, and sorry about the readings I've created. But for example, if something was absent, so this is capability building among local governments to improve environmental regulation and practice. You would raise awareness and articulate the case of change, introduce the practice into the old description of business process, provide work, examples and training. So what you do if you find a sort of breaking here or here or here, what then follows from it tends to be a little bit clearer. So it's linked to decision making is just that little bit tighter, and that's, I think, one of the really attractive ideas. So that's where I'll finish, and any discussion, comments, thoughts, rub them through. Just on that last one, were those suggestions that our decision making or possible responses were they developed collaboratively? No, it was back in the moment when I was one after this. Oh, is that not collaboratively? It tended to, Joe, when we showed it to them, when I showed it to them, they got really excited about this. They really liked this because it's more than just here are the measures, here are the measures and what you can possibly then do. Just curious, what are the most important things to consider when figuring out how many points on your group that you're going to have? Good question. Well, certainly if I was going to develop a group from monitoring purposes, I would probably extend the scar points. I'd have more between absolutely realised. I think if you're monitoring, tend to want sort of the other bit to take to find small changes over time. So I'd probably have a lot more points here. It really depends on context and how much fine-grained decision people can make when trying to store a group and how much the decision makers make the form of their decisions. So it's not going to get you the magic number, but I think this is probably the smallest set and you expand beyond that. I was almost tempted to put it through when I was here. There's often beginning, making huge progress in almost there and sort of half-way there, but it's not as common as these two. So I didn't put it in one universal scale. Can I ask you to follow my question back? The deep philosophical question about that you offer a neutral, a neutral decision? Don't get me started. I know the use of neutral midpoints. I know the grievance and satisfaction scales. I know them even more when they have neutral midpoints because I don't know what it means. I'm still not sure what it means to be not dissatisfied or dissatisfied. If someone could explain that state of being, it's not a first mark, it's been 20 years I've mounted it. I don't know what it means. I think what's missing really in those is you haven't got off the scale points. It's not applicable, but don't know. Certainly you would have those here as well. I'm going to comment about the middle point because I'm a real fan of middle points. I explained this to people all the time in psychometric terms. It's usually just a scale like satisfied to not satisfied in a five point scale and you've got a middle point or satisfied with your example. People treat that as psychometrically. They look at the psychometric, it's a scale, and they just use it on the scale. The problem comes in when people go and understand your questions. If you don't understand what your question means, they'll go to that, they'll default to it because they don't understand what you're asking. If you get a lot of people happening in that, then you know there's something wrong with your question. Yeah, but even with well worded questions, all the literature on those... All the literature on those scales and the classic, of course, is pulled by question announcers and attitude surveys. They do a lot of split-half experimental comparison. That nipon tends to capture people who are very different. Measurement theory, people who identify with a point on a scale should be consistent with the reasons they're doing it. That midpoint catches people who do it for different reasons. They do it highly because they know what they're unsure but also because they slightly agree but it's not really strong. Or slightly disagree, it's not strong enough that they're pushing the points on the scale.