Welcome back to theCUBE's ongoing coverage of VMworld 2021. My name is Dave Vellante. You know, I've been following VMware since the early days and one of the most interesting stories in the history of enterprise tech. One of the hallmarks of VMware over the course of its long history has been a strong number two leader. An individual who looked after operations or advanced corporate development and enhanced, if you will, expanded the eyes, the ears, the heart and the mind of the CEO. You know, at one point last decade, VMware actually had four co-presidents. Some of the most accomplished individuals in Silicon Valley have held this role and it's our pleasure to welcome in VMware's newest president, Sumit Dewan. Sumit, welcome back to theCUBE, good to see you. Thank you, Dave, great to be here. Okay, so you've been in this role for just over a hundred days after a 16 month stint as chief customer officer. So that's certainly a nice dovetail into your new role as president, give us an overview of your new role here at VMware. What are your priorities? What are the key areas of focus? SaaS, transformation, you got a lot going on. Share with us. Yeah, I think the main focus for me is to make sure our company's priorities are aligned with our customers. And in the first hundred days, my first objective was to spend as much time as possible with customers because it's a source of learning for us. It's clear speaking with customers what their challenges are and what we ought to be doing to assist them in addressing those challenges. So my responsibility, obviously, I've got all the operations part of the business which enable our customers to be successful starting from ensuring that we chart the right path for them in the sales organization all the way to making sure that they're successful with the adoption of our solutions with our services and support organizations. So spending time with the customers has been critical. And Dave, what I've learned is that customers are looking for VMware just like they have in the past be this trusted foundation for all of their innovation in the prior era, pre-cloud era through their data center and mobility technologies take that forward into the multi-cloud era which is where now is where they're building new applications taking their existing applications to the power of the cloud and across multiple clouds. And our objective is to make sure we keep providing that trusted foundation for them for the new multi-cloud era. And I'm excited about it. Yeah, me too. We're going to dig into that a little bit. So you're obviously spending time getting close to the customers of course remotely for the most part some of those big themes you mentioned but I'd like to sort of peel the onion on that maybe some of the challenges that your customers are facing in terms of actually bringing forth that multi-cloud vision and specifically what's your approach to solving those challenges? Yeah, so, you know, as we all know customers start out with this adopted started out adoption of the cloud. They started building some applications on the cloud. A lot of times these were the applications that were built which were customer facing. And there was this cloud first thinking at that point of time. But soon the customers have realized and now customers have realized that power of building new innovation doesn't just lie in one cloud because there are certain capabilities like AI and ML that maybe they get from a cloud like Google. There are certain capabilities that maybe storage and compute where maybe they prefer AWS productivity and identity maybe coming from Microsoft cloud. So the power comes in by adopting all of these services across cloud. It has lots of benefits. Innovation at the fastest possible speed for our customers. Secondly, it helps customers not necessarily risk locking in and helps manage their costs. But in this multi-cloud world, it can get very complex. Think about all the security, networking, developer experience, control. Now, this is where our customers need freedom and yet control to be able to have this multi-cloud environment managed and enabled for developer experience as best as possible. That's the problem we are committed to solving in our solution and we call that our cross-cloud services. I want to stay on this for a minute because I've been talking about multi-cloud, the subtraction layer. This is really your opportunity on the cube last year with John Furrier, he said the following, quote, multi-cloud doesn't mean you're running two different architectures on two different clouds. That's not multi-cloud. Multi-cloud means running a singular architecture on multiple clouds.  What you described is not trivial. It's a huge technical challenge. Can you talk about what VMware has to do to make that single architecture a reality? That's exactly the challenge, Dave, because you can adopt multiple clouds, but if you're doing so with different architectures, you're not getting the benefits of velocity, of building new applications fast. Security in a done in a unified fashion, operations at scale. To me, I would call that not a smart path to multiple cloud. The smart path to multiple cloud would be through a unified experience for developers. A control layer which helps you orchestrate your applications in a unified fashion for your operators. And security done in a unified or a consistent fashion so that you know that you have the right governance. That's what I consider the smart path to multi-cloud. Doing any other way would actually be not fruitful. And that's what customers have had to face with without a solution like VMware's. So we provide what I call the smart path to multi-cloud. All right, so don't hate me for this, but I want to push on this and get your point of view on record if I can, because it's an important topic. Now you've intimated that choosing a single cloud provider, it's problematic for customers. It limits the customer's flexibility and choice. And I want to unpack that a bit. And if I'm mischaracterizing your view, please correct me, but I want to understand why this is limiting. For example, if I go to AWS, I got access to primitives and APIs. I got a range of compute, storage, networking options, dozens of databases, open source tools. I get VMware cloud and AWS. So explain why this is a constraint for a customer. Yeah, it's a constraint for really three major reasons. Number one, different services are available across clouds that provide different capabilities. Sure, AWS provides very rich set of primitives. So does Azure, so does Google. And in certain cases, when you're dealing with data sovereignty requirements across different countries, so do those clouds. So the, but if you are really looking for the best possible solution for AI and ML, that may or may not sit in the cloud that you may have preferred for your compute and storage. If you're looking for identity solutions that integrate really well with the productivity applications that you have, that may not be the same cloud that you may have picked for AI and ML. You don't need to make compromises. In fact, developers don't want to make those compromises because by making those compromises, you're increasing your cost and lowering their customer experience. That's the power of leveraging innovation across cloud. Secondly, think about now, if you just build all your applications by services from one cloud and your entire business gets dependent on it. If there's risk, there's costs. And that's why customers are telling us that they have made a decision for multi-cloud. In fact, we did a recent study day. And in the recent study we found out that 73% of our customers are already running their applications on multi-cloud. If this is no longer something of a future, it's here today. They're just facing these challenges today with multi-cloud. Am I right that they're running applications on multiple clouds? But it's your job and your challenge now to be able to abstract the underlying complexity of those multiple clouds and make it appear as one. I'm assuming that's not fully happening today. Maybe that's an understatement, but that is your opportunity and your customer's opportunity. Is that a fair statement? That's exactly our mission. We are providing our customers that foundation so that they can enable multi-cloud and drive their own innovation agenda at the pace that they want to. We did that in the past for data center technologies or mobile devices. Remember, mobile devices come in different operating systems, different formats, and data center hardware from server storage and network has always come in different flavors. We have abstracted that complexity for our customers in the past to deliver innovation pre-cloud. We're bringing the same value proposition now in the world of multi-cloud. Obviously, the applications have changed. They're no longer traditional applications. Now they are more and more cloud native applications. So we have solutions for cloud native enterprise applications that continue to be the heartbeat of more, more, most customers. We have solutions for traditional enterprise applications and the new and emerging edge native applications because of just now people and workforce being anywhere. We have solutions for providing security and providing additional functions for edge native applications. So that's what we are bringing to our customers as a platform that abstracts this complexity of multi-clouds. So much to talk to you about because you're right. The applications are evolving. It's not just the standard SAP windows, et cetera. There's cloud native applications. There's data intensive applications. But I want to ask you, so in order for you to achieve that, you have to be able to exploit those primitives that we were talking about, whether it's AWS or Google, Azure, Alibaba, you've got to understand as engineers how to take advantage of whatever the cloud provider is offering and then hide that complexity from me, the customer, and then build that layer. And to do that and to accommodate all these new applications, not only do you have to have traditional, you have to have processor optionality, you got edge, you see ARM coming in. And if my understanding is that's a big part of what Project Monterey is all about is offering that optionality around different workloads. Can we dig into that a little bit? Yeah, so I think first of all, the people under appreciate or under estimate what it would be required for making sure that the complexity of all of these different cloud platforms is abstracted by VMware solution. So customers don't have to think about, what are the, what is the different storage or server or primitives that are needed on Azure versus AWS, all of that gets hidden from customers in a simplified fashion, so with our solution, okay? So, and yet at the same time, there's no compromise that customers have. They can still leverage all the native primitives and services that the different cloud providers are using seamlessly, okay? So that's very important. Now, in addition, what we are doing is we are continually making sure our platform can run the next generation of applications. We are continually innovating to do so. And that's where Project Monterey comes in. As customers build new applications, when they want to build those new applications and run emerging services that are highly sort of compute centric or network centric or are providing rich amount of data, this is where Project Monterey comes in. It enables our customers to A, take all of the traditional applications onto VMware cloud, run it on across any cloud, and then B, when they're trying to expand those capabilities into the applications, the Project Monterey enables them to do so by enabling new capabilities being powered in to the VMware Cloud Foundation. Yeah, so essentially you're building, what I would look at is a new type of cloud that comprises on-prem connections to public cloud, across public clouds, and then out to the edge. You've talked a lot about telco, the specialized needs of the telco. Clearly there's different processing requirements. You've talked about 5G. We might not always have connectivity out there. Developers need to be able to write code for that edge. So it's an entirely new world. You're essentially building out your own cloud. So you have to build in all that optionality, all the tools, and at the same time, just like the big cloud providers, you have to provide your own tooling, but also be open to providing other people's tooling. Am I getting that right? Yeah, I think you're right. In terms of the tooling part, Dave, what has happened is standards for controlling all of the infrastructure has become Kubernetes. Okay, so we have embraced that. In fact, the talent that has created the best Kubernetes at this point in time, we have it at VMware, okay? The most contributions that are being made in terms of that standard, the most interesting ones are coming from VMware. So in terms of Kubernetes, we have embraced it. And what we are seeing is tooling needs to be done in a way so that our customers can manage from infrastructure to their platform, all via code, all via standards like Kubernetes. And that's what we have embraced. While at the same time, this tooling is done in a fashion so that the entire VMware cloud and the entire VMware Tanzu platform can be controlled in a fashion that fits into customers' entire environment on how they manage it overall. Okay, so let's take that conversation to security. I don't know if you're familiar with the Uptiva chart. It's this mind-blowing, eye-bleeding chart with all the different security tools in there. And I've been watching the moves that you guys have been making, Carbon Black's an obvious one, what you're doing when end-user computing, and a number of other applications, creating a security cloud group within VMware. So that's a good example, but at the same time, customers are using all kinds of different toolings based on that chart. So are you saying it's the Kubernetes is the secret there, APIs that allow you to, if a customer wants to use Okta or CrowdStrike or whatever it is, you can incorporate that into the framework, or if they want to go all VMware, they can do that as well. Can you help us understand that? Yeah, I think our philosophy is that there are two components that are critical for making a solution help our customers take the smartest path to multi-cloud. Networking and security. So on security front, the philosophy is quite simple. These days when you're going out and buying a car, you're not getting buying the car and outfitting it with airbags and ABS and any other sort of safety features. Okay, why do we do that in the world of infrastructure and technology? It should just come as an either an option or a required component within the infrastructure itself. That's our philosophy. And so coming back to if customers say, they want to take an approach to multi-cloud, they want to make sure their developer experience, their DevOps capabilities and their infrastructure management capabilities are there across all types of three applications that I mentioned, the modern apps, the traditional enterprise apps and Edge native apps. Our approach is quite simple. Networking and security, firstly, is built in. It's integrated in. You're not installing agents. You're not managing security thing on top. You're not putting airbags into the car after the purchase. They come with the purchase. You can choose to activate them or not activate them based on your price sensitivity. Second, we have their consistent. Once you learn them how to do it for traditional enterprise applications, the same capabilities, the same security workbench, the same detection and response capabilities carry forward to cloud native applications and Edge native applications. That's the way we are thinking about for security for in our portfolio. Is the strategy submit to sort of be an end-to-end supplier of security? In other words, will you touch all parts of the stack?  Will you try to be best to breed across that chain or do you see yourselves picking spots? No, our focus is to pick the areas that we have focused on, which is to enable customers to run, build and run and secure those applications that I mentioned, you know, the cloud native applications, Edge native applications and enterprise applications. And our focus is to be able to secure those applications, A, in a consistent fashion and B, built into the infrastructure so it's not bolted on. So that's our focus and strategy and we still have great partnerships in the ecosystem for the rest of the portfolio, for the security technology to fit in with the rest of it. We just don't think that for the infrastructure that's running these critical business applications, you should have a requirement to build these applications, build this security on top of it. And that's sort of our commitment to our customers. Got it, that makes sense. I mean, you've got a pretty clear swim lane in your infrastructure space. There might be a little gray area there, but you'll let the ecosystem take care of that if it makes sense. So I guess I would say, I look back and first of all, VMware has had amazing engineering over the years. You're very well known for that. You just mentioned some of the best Kubernetes engineers on the planet. And of course, November is a big milestone for VMware with the spin. You now will become a completely independent company again. And that's a big deal in my mind because I think this is going to be expensive. I mean, to actually do this, these are big investments that you have to make. And I feel like you're finally going to get control of your own balance sheet so you can make these investments as you see fit. So that's got to be an exciting time for you. And because I think you're going to need that free cash flow to really drive this in addition to the other things that you're going to do with buybacks and stock options, et cetera. I think we are excited about this whole upcoming spin off from Dell. Dell will continue to be a very important partner of ours. In fact, we have coded and codified what we are doing with them on innovation as well as on sales and distribution perspective together. And I think to be candid just through that agreements that we have put in place with Dell, I think the partnership could even get stronger because we have 15 statements of work where we have defined new innovation projects with Dell, for example. But at the same time, like you mentioned, we get a little bit more flexibility to be able to chart our own course which is critical in the world of multi-cloud. We are able to, not that we were constrained on but customers still always asked us about how would you continue to sustain the partnerships with the cloud and hyperscalers? That's no longer a question in customers' eyes once you're independent. And secondly, it does give us flexibility on balance sheet to be able to make investments as needed to win the agenda that we have on multi-cloud without having to negotiate. Yeah, I think it's an awesome move, of course, because certainly since the Dell acquisition of EMC, your business has even grown more with those combined companies, so we've seen that. But I liken it to the coach who has a kid on the team and the coach is extra hard on the kid. That's kind of almost the way it had to be in that relationship because your posture with the ecosystem had to be, hey, we're an open ecosystem. And so now it's sometimes kind of weird and uncomfortable. Now it's clean, it's transparent. So I'm really looking forward to the innovation that you can create with Dell, of course, but with other parts of the ecosystem, which you always have, but I'm hoping the ecosystem now leans in even more. It's always had to because you've got half a million customers and you've got such a huge presence in the market, but I think now there's going to be a little more comfort level there. So I'm really excited for that summit. Great. Hey, so this was great conversation. I can't wait to have you back. I really appreciate your time and insights. Well, thank you so much, Dave, from our perspective at VMware, you know, as I started with customers, I'm going to end sort of this thing with customers as well. Always great times, great to spend time with customers. And we truly believe we have the best platform to give our customers the smartest path to multi-cloud. And I know the feedback so far has been great. It's always great spending time with theCUBE. Thank you for having me. It's our pleasure and we wish you the best. Thank you everybody for watching. This is Dave Vellante for theCUBE's continuous coverage of VMworld 2021. Keep it right there.


Hey guys, what is up welcome to my channel for today's video. We have another rankings video I'm going to be ranking all of my color pop palettes This is one that I've been putting up because I feel like I have so many but if you want to see what's the worst And then of course what comes out on top then just keep watching Hi guys, if you're new here, my name is Morgan. I am a product knowledge enthusiast I just love knowing anything and everything about all of the new makeup on the market and sharing my thoughts So with you guys and today's video is really exciting because it's actually in collaboration with my friend Lia Jenae And she loves color pop so I thought you know doesn't kind of a perfect mix here for a collab What I love her channel most for is her TJ Maxx videos Now I seem to have the worst TJ Maxx ever But she seems to have the best TJ Maxx's ever so if you're ever interested in TJ Maxx content She goes in multiple times week and shares with you what makeup is in stock at TJ Maxx I don't know why hers is so good, but I am so jealous. She also covers a lot of other content as well Like I said, she does other color pop content. She does makeup reviews. She does hauls She makes me want to go to Bath & Body Works all the time. She films a lot of Bath & Body Works hauls So if you want to be convinced to spend money or anything that's new in the market I definitely would recommend checking out her channel. Without further ado, let's get into the rankings So I own a total of 24 color pop palettes There was a couple that I did leave out because they've been long gone for the past few years And nobody's really interested in them because as you know color pop just comes out with a lot of palettes So it's kind of like I had to condense it Somehow so you probably won't even notice because they were palettes that nobody cares about And as far as the quality of color pop, let me just say that I love my color pop palettes I think they are well worth the money. Not all the palettes are as great as others Sometimes there can be a little bit of inconsistencies in the formulas, but I will say for the most part I haven't gotten a color pop palette that I didn't feel like was worth the money There's some palettes that I feel like are a little bit more worth and there's some where I'm like Okay, it was worth ten dollars, but I did spend ten dollars on it So yeah, that's kind of where we're at There really isn't a palette that I have tried so far that I dislike from color pop I'm a big fan of their products a big fan of their packaging and all of these I did purchase with my own money So let's get into it number 24 and you guys are gonna be like girl. What the heck? But it is the pretty Guardian Sailor Moon palette now if I were ranking this based on packaging This would rank probably in the top five honestly probably number one looking at it But I am a huge Sailor Moon fan of course growing up. I loved Sailor Moon, but the colors in here aren't my kind of colors They're very very pastel just very light. I enjoy the quality in here I don't think it's color pops greatest, but it's just not a color story that I've been so tempted to grab for so I've only used this like once or twice, but I definitely like to pull it out and look at it So while I do love the packaging I do not love the color combos that we have going on in there But again, that is just personal preference number 23 We have the very first palette to ever walk the earth from color pop and that is the yes, please palette I remember back in the day it was Okay, I remember back in the day it was heavily compared to the Natasha Denona It's in the sunrise or the sunset palette the very first warm palette that really got a lot of buzz from Natasha Denona And this is a great palette at this point this palette is probably Expired, but I still keep it. I used it all throughout college and I loved it But now as the years have gone on I'm just not into this kind of color story anymore It's a little bit more boring to me. I don't like yellow and oranges I mean I do but as you can see I just have other color stories that I admire more and want to use more So it's just a mixture of I'm bored of the color story And I'm bored of it because it's also one of the oldest color pop palettes that I own But it was very good back in the day, but I do think it's a little bit outdated at this point number 22 Again, this is another one where the packaging I still keep it in its box. It's incredible This is one of their newer palettes that I've purchased This is from their hello kitty and friends collection the snow much fun palettes And I have to admit I did love the couple of looks that I came up with But I don't love the quality on this I feel like the shimmers are a little bit lackluster and it just compared to other Quality that I've tried from color pop. This does not come up on top So quality is the reason why this is ranking at so low But even though the quality is not the best it was worth the money that I paid so it's still decent number 21 again, this is a newer palette. Huh interesting that the newer palettes have come at the bottom But this is the fade into hue palette and again This is another one where I like it and I can make it work And it was definitely worth the money you get so many colors in here But you do have to adjust it and adjust the way that you apply a makeup in order to make this work Because especially this bottom row I find the colors to be very very very powdery So you make a huge mess on your face So don't do your face makeup first, but I love love love love the color story on this since I'm equipped with how this Formula needs to be handled. I use it a certain way and I can make it work And I just love the options that I have here But again if we are ranking, you know, I'm looking at color story and quality This does rank pretty low, but I do still enjoy it number 20 We have the you're a cutie palette now this isn't sold anymore But I did buy it last year and again This is one of the palettes where I felt like the shimmers were a little bit lack luster But what I love about this palette is the color story and just it's so it's cute You're a cutie and I love the colors in this palette So even though again like I feel like it needs a glitter glue or a wet brush to apply those shimmers I feel like the look that I get every time is just so beautiful. So it's still a great palette for what I paid for Number 19, this is where I start to really enjoy all of the palettes that I'm talking about from this point on This is a really great purple palette. That's affordable. This is the it's my pleasure palette And you just get an interesting array of purple colors that you don't get in a lot of other purple palettes So I do highly recommend it There's more palettes that I've reached for more more palettes that I've been surprised by so I suppose That's why it's ranking on the lower end of my color pop products But I still really enjoy it and if you're into purple, I still do recommend it coming in at number 18 we have the uh-huh honey palette now. I haven't used this one very much That's why it's ranking lower but on the terms of originality I do not have another yellow palettes in my collection and I think the colors are really great The quality is really great I love how they put this brownish mustard shade in here because it does add the depth But it blends in perfectly with the other yellow shades, so I don't wear yellow too much So I haven't reached for this too much but as far as the color combos they put the color layout everything quality It's really really nice number 17. We have the at-forest site with raw beauty Christie I have a love-hate relationship with this palette. I'm not the biggest fan of the color story Just these grungy tones aren't tones that I tend to grab for very often And I did have some problems getting them to blend But then we get to the top two rows and I really did enjoy this palette a lot So it depends on the look that I'm doing how much I love this palette overall I do think Christie stayed true to herself. She created a beautiful palette for herself But for me as far as what I want to grab for I wish this had more shimmers and I just don't care for the grungy tones So that's when kind of my personal preference comes into play number 16. We have babygot peach It is just such an adorable palette I mean, I really like the variation in color here if you're staying in the peachy kind of color theme I like that you have this lighter shade. I like how you have an apricot But also like a deep almost rosy apricot and a bright orange as well I just feel like you can get a lot of different looks with this palette It really is a great one if you're into this color story and their nine pants are just so good number 15 We have the smoke show palette. This is actually renamed to blow and smoke. Don't know what that's us all about But this is the palette that I am wearing now I don't grab for it very very often But it really is such a statement in my collection if I want a really deep smoky eye Super cool toned gray black whatever I know I have this palette and the quality is quite good because these shades can go very very wrong Quality wise but color pop truly it did a really good job with this palette So if you're into these tones or you want to incorporate these tones into your collection This is a great one to go for and again, I created this look with it. It is such good quality. It's quite shocking Number 14 we have blush crush. I love this one because it's just soft feminine pretty colors It's just something really easy to grab for 2020 I fell in love with more pinky toned eyeshadows. And so I bought this in 2020. It's a really great palette It really fit my needs. Maybe I didn't buy this in 2020 I can't quite remember but it was either right at the end of 2019 I don't know. Anyways, that doesn't matter really beautiful really fit my needs for 2020 didn't grab for it a lot But I still love it number 13. We have adjust my luck This is ranking so high based on just plain originality I don't have a palette that has this array of colors the quality is decent Considering that these colors can be harder to make. I like how we have a little bit more brown leaning shades a little bit more yellow pastel again Whoever laid these colors out. I just feel like the process was really thought out and it's a great palette Unfortunately, it's no longer sold, but it was a really good one number 12 This is personal preference because I love these tones. This is lilac you a lot now It's my pleasure was that deeper purple one I prefer this one it simply because I like the tones better in this one. It's light purple. It's very very girly Just it's a good one again. I don't think it's color pops best best best quality But I just love the color story so much.  Oh my gosh. Now we're getting into like top tier color pop palettes the zodiac palette This was in collaboration with Kathleen lights. It is no longer available But what a thought out color story. I just feel like she covers so many bases with this color story The quality is really really good with this one. You have some bright colors. You have some purple tones You have some warm tones.  This was an amazing up palette Number 10 has to probably be one of my most used color pop palettes. This is California love this has traveled with me many times And I didn't think I would love it so much because I'm not as into warm tones And this palette has ended up pulling very very warm, but I don't I don't know if it's the packaging or what but I just grab for it It has every tone and shade and depth that I would need for traveling It has glitter texture all of it. It's just really really nice And even though I say I'm a cool tone lover I still like me some worn tones and this is just such a wearable palette and you'll see from now on for the Most part all of these palettes are a little bit more on the wearable side So I use them more. That's why they're ranking so high. They're just wearable, you know So I'm moving in at number and nine again another Kathleen lights collaboration This is the dream street palette again, unfortunately no longer available But again such a well-thought-out palette I feel like this one has a little bit more of unique colors Like you have this shade right here in this shade in this shade So that's why I like it better than the zodiac palette, but again neutral tones of warm tones Variations in colors and it's just a wonderfully thought-out palette So I really have enjoyed that one number eight This is just a completely wearable palette great for portability Haven't used it as much as I would like to but I just love the colors and the quality so much on this that it has To rank really high this one is less pink than blush crush which was the counter part to this So I like this one because it leans a little bit more neutral the colors look absolutely stunning And I just really love it number seven.  It's just a great range here Some of the colors do look similar to one another but the value in here just cannot be beat it can't number six Sweet talk palette. This is my number one for the longest time because I love the colors in here pinky peachy shades Lots of dimension in here. You got a couple glitters, which I'm not so into glitters anymore But back in the day I loved me at some glitters This is definitely one of my most used palettes from color pop everything about the story It's just perfect for spring and summer and I always find myself reaching for it that time of year number five I'm so happy to announce that this is back in stock. I thought they had gotten rid of it for good But this is the so jaded palette. This is another Kathleen lights collaboration and again You just get so many options. I always grab for this for holidays I play in this corner for Christmas and then you have some fun summer colors in here with Kathleen's palettes. She really doesn't stick to one color story She adds many but they all still work so cohesively together So this one is just a fun jewel toned palette And I do highly recommend that if you are interested in it to pick it up Because you get a lot of different formulas in here And it's a great formula to begin with and I'm excited to see this back in stock So that I can continue playing with it because the color the options It's just endless, you know number four is the Mulan palettes part of why it ranks so high might be because it's Mulan And I was Mulan for many years Halloween go growing up, but look at these I feel like color pop hasn't launched another palette like this as far as quality As far as textures again, I talk I'm big on variety I would like to see variety in a palette as far as textures and depth goes and I feel like I really get that It's a cohesive color story, but you have a lot of options and I have grabbed for this palette a ton from color pop It seems to be so high quality number three We have the double entendre this hands down is my most used palette from color pop it might not look like it But I swear it is a little bit warmer, but compared to California love It's more neutral which I think is why I grabbed for this one more a great travel palette once again It has everything that I personally feel like I need so I look at the color story And I'm not super in love with it But it's just so versatile and great for every day that it's ended up being my most used palette I don't believe they sell this anymore, which is unfortunate, but it's really great number two Going coconuts this one is a very popular one by color pop and for good reason So my shades might be a little bit mismatched because I dropped this one time and then I'm just kind of randomly put Them back in their spots, but the quality and the color story of this is great You get more of a warmer look you can get more of a cooler look I would say this leans right in the middle at more neutral, but again everything you need in a simple 9-pam palette I love this one and then finally my number one favorite Mmm, I don't think this will permanently be my number one favorite because it is just really untrained for what I Personally like right now. I think it's a phase that I'm in but the color pop stone cold fox I really love the quality on this one and right now I'm just so into cool-toed neutrals that this fits the bill of what I like right now So I do think easily as kind of my preferences changes that this can like drop down But right now January or February 2021 this is my number one favorite. It is beautiful. I believe they recently restocked it It is such a fabulous value the quality you're getting is really great And I just I love the options of colors here. So it's currently my number one. All right, you guys There you have it. Those are all of my color pop palettes ranked from number 24 all the way to number one Make sure you go check out Leah Jeniz video and subscribe to her channel. I will link that down below for you guys Thank you so much for watching and tuning in to another rankings videos Let me know what else you want to see for me. And yeah, I if you aren't subscribed to my channel already I hope you would consider taking the time to do so and I will see you guys in the next one. Bye guys. Have a good one


We can hear a lot about leadership without really understanding what it is Because there are a lot of different definitions and people look at it differently Some people look at leadership and think charisma that you have to have charisma to be a good leader So and certainly there are examples of charismatic leaders people who employ charisma to great effect in their leadership You know style and ability, but that's maybe too narrow an approach Some people think inspiration leaders have to inspire and they do inspire us and that's that's true, too There are lots of good leaders who do use inspiration and inspire people to great effect And and that's a big part of their leadership style yet other people though think of leadership as a management structure When they think of leadership, they think of their corporate structure and just you know What's the line of report? But I think we can all imagine that we've had Bosses that may have been in charge may have had that Authoritative power over us, but weren't necessarily great leaders, right? Not somebody would call leader as much as a boss really just a boss or a manager and not a leader So, you know, what is leadership? Where do we end up on all this and and how do we define leadership? Well? That's really the purpose of these next few videos And and the start of our discussion in leadership and we're gonna begin really with the foundations of leadership By defining what we mean by leadership and that will carry us through as a definition Through this series of videos on leadership and communication So let's start off by defining leadership Leadership is the ability of an individual or group to influence and guide others Toward the achievement of a goal That's the definition we're going to use for our purposes in our discussion of leadership here The ability of an individual or group to influence and guide others Toward the achievement of a goal So let's take a look at some of the individual components of this and dig a little deeper there first of all Leadership involves influencing and guiding note that we're not saying that influence that leadership involves You know bossing others around or using that authoritative power There's certainly a role for power in leadership and different ways that leaders can gain power But but leadership as opposed to something like management or that corporate structure Doesn't rely necessarily exclusively on authoritative power the ability to Reward or to to punish people it involves influencing and guiding so we're really just almost more nudging people or or Instructing them and helping them grow. We're serving alongside them or we're we're giving them the The ability to do these things on themselves or for themselves So we're influencing and guiding as leaders rather than you know, just using that authoritative power Secondly leadership involves others By definition if you are a leader then you should have followers or that's a part of leadership It's hard to lead anyone but yourself unless there are others who are willing to follow you so so leadership involves others and that Reminds us to be others focused or others minded That leadership involves other people and we need to be aware of that. It's not just about what the leader wants or needs or Feels like we should do there. There are others involved in this equation. So we need to to bear that in mind as well Leadership also is about the achievement of a goal You know when we play follow the leader as a kid That's that's okay as kids, but there's no real purpose there, right?  But but leadership in the real world involves the achievement of a goal that people are working together Toward something that's how a leader is leading people specifically to something and in the achievement of that goal So that's our very simple definition of leadership and what we're going to use here So as we continue our exploration into the idea of leadership, we're going to break it down into a couple of different broader Categories if you will so first we're going to start with a few videos on the fundamentals of leadership starting with this one And going into a couple of other areas as well defining leadership versus management And some other things so we'll start with the fundamentals of leadership Then we'll take a look at some leadership styles and we'll break those down a little bit the idea of You know that there are different styles for leaders. There's not one way to be a leader So we'll take a look at some of the the major leadership styles that have been identified and and categorized Then we'll look at some leadership communication strategies some things that influence communication as a leader things like power And and a few others here that we're going to take a look at but some of these different types of communication strategies and some major communication skills and premises that we need to keep in mind As leaders in order to lead people effectively And then finally part of our discussion will will center on performance and appraisal systems as a leader It's a big part of our responsibility to provide feedback On somebody's performance and to appraise how they're doing Again not necessarily to to boss them around or to chastise them but to to help them to improve We're working towards this goal and as leaders it's our responsibility to make sure everybody's performing In a way that's appropriate for the group and appropriate toward the achievement of that goal So that's going to be the focus of our discussion on leadership as we move through this series I hope it'll be helpful to you in Growing your knowledge of what leadership is and also your your skills and your abilities as a leader yourselves If you have questions about anything related to leadership, please feel free to email me I'd be happy to chat with you about that and in the meantime I hope you'll really start to dig in to this idea of leadership and the broader context of leadership And what we can do to become more effective leaders


So welcome to today's session on artificial intelligence and evaluation. We have a panel discussion. So thank you for joining us. It's being put on by the South Australian branch of the Australian Evaluation Society. So before we begin, I'd like to acknowledge the traditional owners of country throughout Australia and recognize a continuing connection to lands, waters and communities. I'd like to pay our respects to Aboriginal and Torres Strait Islander cultures and to elders past, present and also emerging. And I know that we're on at least several countries based on just the location of South Australian members who are here today. I'd also like to thank our Melbourne colleagues. Today's presentation was made possible by a conversation between myself and the Melbourne branch. Convener Ruth and she recommended one of our panel members, Christy, for today's presentation. So thank you to our Melbourne colleagues. If you're not a member and you're interested in learning more, then perhaps now's a good time to join the AES. You can grab the screen link there. It's pretty easy, www.aes.asn.au and you can find out all the details about becoming a member. So let's get into our discussion. I'm going to facilitate it. My name's Mark McKay. I convene the South Australian branch with many of the members who are from the committee who are here today. And AI has taken off with, I guess, a bang in the last year or so. It's been around a long time before that though. And we're seeing more and more headlines and the speed at which generative AI in particular is emerging is, I guess, interesting, challenging and leads to lots of opportunities. But we need to think about it, particularly as evaluators. I've got some of the headlines that have interested me. Algorithms are pushing AI-generated falsehoods at an alarming rate. How do we stop it? I don't know if anyone's used GPT. I'm sure you have, but if you asked it to generate references and then you check those references, often they'll be rubbish. We've got Google, we've got other platforms also launching into health and health is an area where we do sensitive data. So who's accessing that? So the systems learn. What happens to it when it does learn? How can we be sure the results are useful? We've got the crime side of things happening. So in the bottom left-hand corner of my screen, AI has been used to generate the voices of family members and then extort funds so that they're released. No family members were involved in the crime. Their voices were picked up and copied and it was made to be quite real and obviously people pay out. There's also the opportunity to send doppelgangers to university classes where if it's an online session, someone can be present and answer the lecturer and engage, but they're not there. So what happens? I guess also what happens if we do outsource our work to AI, where does our skill base go? So all sorts of things are happening. Even in the creative space, people are talking about the end of photography or how is it going to change? So lots and lots of interesting things are happening and I'm sure you've seen or heard radio talks, podcasts, seen it in the news, it's everywhere. So today we've got a panel of people to speak about it. I'd like to say thank you to them first and then let them off the hook to some extent. This is a new space. There's not gonna be a right or wrong answer. There's gonna be some opinions and I'm pretty sure I'm on side with the opinions that they have, but it should be interesting. So Ian Walton will be speaking first. Ian is a longtime colleague and friend of mine. He was a former data quality manager at Medicare Australia and has also been involved in the education of postgraduate students studying health care administration for many years. Nigel Bean is also a friend and colleague, former professor of mathematics and now working in the consulting space. Nigel has had a lot of experience using synthetic data, understands probability and also can talk about AI. So he'll have some interesting views as well. Christy is an associate director of ELT at Grovesner, sorry, performance and she has an interest, especially in ethics of AI as it pertains to evaluation. So let's get on to it. So the structure for the session, each panel member will have five minutes to talk about something on generative AI. We'll start with Ian, then move to Nigel and then to Christy. Then we're gonna put some questions to the panel, then open it up for question and answers. If you've got any comments, burning questions, if you want to write them in the chat and we'll get to them in section three, that would be great just so we don't lose time with interruptions.  So that gives you a warning, Ian, Nigel and Christy that you're running out of time. Okay, let's get underway. Let's have Ian talk.  Just stop sharing the screen. Thank you, Mark, start the clock now. Welcome everyone and thank you, Mark. I guess I have a strong bias here with regards to data quality. I regard it as the foundation stone for future AI. After 40 years of it working of just about every possible aspect in the health sector and working on two data warehouses, I have a very strong appreciation of the concept garbage and garbage out. So my proposal with regards to AI is to build the AI house on rock, not sand. What do we mean by data quality? Well, in my 40 years, I always use this formula, fitness for purpose. I want to use to tell my students is, would you expect a world-class swimmer to be able to hold their own in a boxing ring with a world-class boxer? And the answer is no. So therefore, when we look at data and we look at information, we should be looking at the fitness for purpose. Now, data emanates from a business system process that quite often involves policy, information technology, and frontline staff. It's a very simple model that I use to explain complex processes and to highlight the power of silos. So let me give you an example. Let's say the organization wants to create a field called data birth so they can register their clients. Policy says, I want a date of birthfield, please. And IT says, give me the specifications and I'll put a date of birthfield together. And they'll say, it's not a problem, we do it all the time, day, day, month, month, year, year, year, everything's fine. And the frontline staff will say, just give us the policy and the business rules associated with the registration process and everything's hunky-dory will produce you an outcome of client registration. What I did in one of the organizations I worked at was I brought the policy people, the IT people and the business operation people into the same room to look at all of the fields, one of which was data birth. Policy said, I don't know why you're reviewing this, we have a policy for date of birth, it's done. IT said, we don't know why you're reviewing this because everybody in the world in IT knows the date of birth is day, day, month, month, year, year, year. And the business ops people said, we don't have a problem either. So why are we reviewing it? And I said, well, here's a chart of the date of birth for the country. And there was a massive spike on the 1st of January and the 1st of July. So I asked the policy people, how could that be? And they said, well, there must have been a lot of people born on the 1st of July and 1st of January. I asked the IT people, don't ask us, we've got the field there, it's the business operations people that have filled it in. The business operations people said, oh, we can explain that easily. There are an awful lot of people who don't know their date of birth. So we just use the 1st of July or the 1st of January. Completely destroys the integrity of the process. Now, why do I give you that particular example? Well, that's a simple thing like date of birth. There are many, many, many others and you really need to know what you're dealing with when you're dealing with data from organizations. So to solve the problem of date of birth, we need a transparency of process. We brought all the people into the one room and looked at the system process to work out what was going on. We also identified that the data source must be identified along with the system processes. Now, when we moved to AI, the very first thing I'd be asking for is where did you source your data from? And what was the integrity rating for that data? I also strongly have learned over the years that we require technical, which is the IT view of the world, and business metadata. Now, metadata is information about data. In the example I gave you just now, the technical data would say date of birth is day, day, month, month, year, year, year. And if you looked at the business data, it'll say, and when we don't know, we'll use the 1st of July or the 1st of January. That's what I mean by this business metadata. How many people collect business data? Very, very, very few in all of the major organizations I've ever been associated with. So let's go back and look at the example I just gave you. We've got transparency and accountability as being the issues in these silos, policy, IT and business operations. What sort of problems do we hit with? Well, the same problems all silos have, the lack of communication between silos. It's not my problem, go and see the IT or the frontline operations people. We have the problem of what's in it for me. IT and policy may say, well, we're not interested in date of birth. There are other bigger fish to fry. You've got 30 seconds, Ian. We have no time and no resources. So what I'm saying is we need incentives and penalties with regards to this process. We need to educate people as to the value of data integrity.  And I'll just quickly finish with the Bee Gees, have a hit song once with How Deep Is Your Love, which includes the lyric, we're living in a world of fools breaking us down. Perhaps in relation to AI, we should rename the song, How Deep Is Your Trust? In closing, John Farnham got it right when he said, you're the voice I would add, use it or lose it. Thank you. Thanks, Ian. Okay, moving across to Nigel. Now Nigel, I know you wanna use a slide, so I'll just bring it up for you. Thank you, Mark. Okay. So thanks, Mark, for organizing this and welcome everyone. So I wanted to just start with a little bit of history of artificial intelligence. It's been around for a long time, but historically its main use was to support decision-making.  If you're training your decision-making algorithms on data that isn't worthwhile or isn't reliable, then you're gonna get unreliable decisions coming out of it. AI has a difficulty in that many of the methods do not provide any explainability of those decisions. And there are a number of areas where there are either regulatory or ethical reasons why you need to be able to explain decisions and that can be quite problematic for some of the AI methods. And obviously over the last, let's say 10 years, there's been a massive increase in the complexity in these decision-making algorithms.  That's exactly AI. The recent excitement over the last year or two has been in generative AI. And as Mark said, that's what we're gonna predominantly talk about today. With generative AI, there is no explainability. And that could be a problem. It's trained on the whole internet.  I'm a person who made my first post on LinkedIn a few weeks ago at Mark's request and have no other social media. So I think you'd know my opinion on that. But even with whatever your opinion is, if it's trained on the whole internet, does it, has it been trained on, does it know about the things that matter in the application that you're using it for? So for example, health data. How successful is Dr. Google? I'm not sure that it's a great way of running your medical life. How the generative AI works from a technical perspective is that it is able to predict the most likely next phrase. And I'm deliberately being nonspecific there. It could be word, it could be a collection of words. It can be a whole lot of different things depending on the actual algorithm. But when it's writing text, it is predicting what text should follow the previous text. That's fundamentally how it works. Now in its default mode, it will put the most likely next piece of text. But some of the algorithms or some of the interfaces allow you to increase the temperature. And what that does is then allows it to choose more randomly, but still from the more likely pieces of text to follow. And that's how it's working. It has a very impressive performance in terms of language and grammar. When you look at something that is written by AI, it's usually written pretty well. Certainly a lot better than previous generations of these sort of techniques. And a number of my friends say it writes better than I do. So why not use it to improve your writing? So it's very impressive in that area. But one of the points that I think is misunderstood because of, and at least partly because of the impressive ability to write is that there is actually no, and choose your favorite word here, intelligence, understanding, rationality. It is just predicting what words should follow what previous words. And my favorite example for that, which no doubt has been corrected by now was when a version of ChatGPT was asked the question, I have two jugs, a five liter jug and a two liter jug. Measure for me three liters of water. So a fairly simple solution, fill the five liter jug, use that to fill the two liter jug and you'll be left with three liters in your five liter jug. What ChatGPT came back was an 11 step algorithm and the final sentence was, you will now have three liters of water in your two liter jug. I find that a beautiful example of, or demonstration that there is no rationality. It's written out well, the algorithm all seems sensible, but it's willing to conclude that there's three liters of water in a two liter jug. So it doesn't understand those things. You got 30 seconds Nigel. Thank you. And then of course we've all heard about hallucinations. So I won't go into that. One other quick comment, Marks said I use a lot of synthetic data. What that's for is the data we collect is literally one instantiation of life. And if you're trying to test new methods, you want to be able to test them against other reasonable representations. And so for that we create new data sets, but we're very, very careful in replicating the key features or at least the key features we understand, but we know what we've replicated. If you use AI to create synthetic data, what's it based on? How do you know what features it's decided to keep and what features it hasn't or where it's created that data from? Thank you. Thanks Nigel. Okay, Christy over to you. Brilliant, thanks Mark. Thanks Ian, thanks Nigel. I just wanted to start with a bit of context as to how I'm approaching this session because that'll inform the answers and the responses that I give. So I'm not a techie. So I am terrible at digital adoption. You know, I don't know my SaaS from my XS all that sort of thing. But if you take one thing away from my segment here, I think that AI capability is something you can build. It's something I've built and it's something that's essential for us as humans in general, but as evaluators specifically. We really need to run alongside this technology and not let the horse get too far out of the gate the way we did about 10 or 15 years ago with cybersecurity. The other thing I think is really important to pick up here sort of reflecting on Ian and Nigel's points is artificial intelligence, AI is so broad. So I just wanna be really clear that AI does not equal chat GPT. So there are many other different solutions out there. So far we've spent a lot of time talking about chat GPT but there's the large language models, the LLMs. There's also the multimodal ones as well. Barbed by do co-pilot, so many others that I've forgotten. So the deficits of one solution may not be present in others because the technology is evolving so rapidly. I am a cynic. So like the other panelists, I tend to be quite conservative in nature and I tend to look for the risks everywhere. But I've deliberately decided tonight to try and be optimistic. So let's see how I go. I was thinking about the examples that Mark provided and sort of set the scene with. And I think it's easy for all of us to be doom and gloom on AI and I know I am, I'm just waiting for our robot overlords to come along. So I always say please and thank you to my laptop when I'm using it. But I think we've had several different industrial revolutions over time and the world hasn't ended. So the printing press was giving the access to language to the commoners. Oh no, they'll uprise and kill all the landlords. The personal computer was democratizing the internet and knowledge to everyone. Oh no, the world didn't end. I think the same here. There'll be some shifts. The rate of technological advance is happening at an unprecedented rate and also the impacts it can have are unprecedented. But hopefully the world won't end just yet. With my positive hat on, I think there's opportunities for us to harness this technology to answer some of the questions we've looked at for a long time. How do we have work-life balance as a human race? How do we advance science and knowledge to have unforeseen insights? How do we focus on work that we enjoy?  But I remember growing up with, you know, you had your cameras everywhere and you went down to the pharmacist and you got your 799 photos, you know, if you waited seven days for them. And Kodak had heard about the digital camera coming along and thought just a fad won't disrupt us. Where's Kodak now? So I just encourage us all to not be Kodak. We need to actively engage with this technology and build our capability to grapple with it, understand it and evaluate it right now before we become obsolete like the manual camera back in the day. I think also we as a species, we're developed for adaptation. So, you know, evolution has occurred to bring us a range of different advances for our species. The written word and our ability to pass knowledge on through generations being one. We saw with the advent of the personal computer that our human brains evolved at that point in time. So the human brain's working memory went from about seven plus or minus three bits of information down to about three or four plus or minus three bits of information. Because what happened was the way our brain functioned was we no longer needed to remember large tracks of poems or essays or religious stanzas, but instead we needed to know where to find that information. So our brain adapted in that space. I think our brain will adapt here. We're very resilient. I saw a video today sent around by one of our techies that showed what happens when chat GPT and large language models are implemented in an Android body. So if you want to look at something really cool, go to YouTube and look at chat GPT and figure and it'll come up with the test demos of this robot and the dexterity just displayed by the Android, its language processing, its response to the situation are unprecedented. And so there are unprecedented applications for this technology. So it will be used the horses out the gate lawsuits about how the, you know, training data was acquired unethically, not withstanding. We as a race aren't gonna turn the clock back on this. I also ponder to what extent does AI pose new risks or just extend existing risks that we've always had. So we as humans have always loved shortcuts. We've always loved rushing analysis under pressure. We've always loved taking credit for each other's work. For the person that's using chat GPT or similar to write their essays and analysis, isn't that similar to back in the day of the professors, you know, taking credit for their research assistance, original thinking, but publishing under the professor's name? About 20 seconds, Christy. Brilliant, thanks. And so from all of this, we need to really engage with this technology and that's what I urge every person on this seminar tonight to do because there's two perspectives we need to consider. How AI can help our practices evaluators and how do we evaluate AI-enabled programs and services? So my top four tips, upskill yourself now. This does apply to you. Number two, when you're planning evaluations, consider the type of technology that's being used, it's model, it's strengths and limitation because they're all different. Number three, look at the training data, the model design and how it was actually implemented. That's critical in this space. And number four, look at the transparency and the outcomes of the decisions being made. So if you follow those four cornerstones, that's how you can improve your practice as an evaluator in dealing with this brave new world. Thanks. Thanks, Christy. If you could post the name of the video in the chat, I think people would appreciate it. There's a few questions about it. I'm going to post some questions on a screen. So I'm gonna just share my screen again. So questions for the panel. I came up with some and they've had the advantage of having a little bit of time to think about them. So a number of people have touched on the source of the data. That's one question. What if the models are reinforcing existing societal biases and we use them in work? So either as evaluators or the people who are evaluating or organizations who are evaluating have used the models to determine things like who we should serve if we're evaluating some sort of social service program. What about the fact that generative AI can create documents, videos, photos, all sorts of artifacts? How do I know if that's given to me that the artifact is real? Is the evidence no longer useful? And there's a video you can check. It's using chat GPT. Yes, there's others as well with Excel. You can now pair the two together. You don't have to have any skills in Excel to generate the output. Just type in a line and it'll complete the report, give you the graphs, the tables, et cetera that you need. Yes, it's quick, it's cheap. It's easy to use and I could imagine as someone who worked in academia once that students would take to work like ducks take to water. I don't have to learn this technology and spend time learning it. I can just generate the answer to the question. Is that good? Are we lowering our skill levels and therefore our understanding of what's produced? And is there a framework we might need? Now, I'm not going to ask the... Okay, got it. The panel to answer all five questions, rather pick one. So, which panel member would like to kick off? Pick your favorite question and go for it. Christy? I will have a go. So, I will have a go at generating AI, lowering the bar for tools. So, full disclosure to the group, I was half concentrating on the chat. So, let's see how I go off the top of my head. So, something really cool about AI is it can democratize knowledge. So, provided you've got an internet connection, which is increasing across the developed world and in parts of the developing world, you can, as Mark was saying, do programming without having coding background. You can access all sorts of technology. It could help level the playing field between your cover letter and someone else's cover letter. So, it can create more equitable opportunities for all where your own capability can be demonstrated and can be assessed, rather than you being held back by your grasp of perhaps verbal or written language or cultural norms. So, I think that's one thing. The second thing is I have heard some applications that are being trialled in this space is to have customer-oriented sort of access to websites. So, that sounds really fancy. But the example that I heard was a government department is trialling, having all its information able to be searched through AI solution. So, let's talk about chatGPT as the main model people seem to be aware of. So, you can hop on and say, how do I get my phishing license? But really importantly, you can type it and get the answer back in your native language. So, Vietnamese or Afrikaans or Indonesian. So, information that may have been blocked from you for either your digital literacy or your grasp of a native language is now more accessible. So, I think there's really, really exciting application here. The flip side and here's my conservative side is it does increase the risk. So, we've now got an incredibly powerful tool freely available to millions and billions of people around the world who don't always understand concepts like hallucination and temperature and how to sort of make sure that sources are correct and the chatGPT trio that goes up to September 2021, 2021 for example. So, accessibility is at an all-time high but our educated use of the tools may not be matching that. So, that's the risk to be aware of. Thank you, Christy. Ian or Nigel? I'd like to build upon where Christy has gone. Well, I hope to build upon. I noticed if I go to question two, what if the models reinforce existing societal biases? Anybody who watched Q&A this week on the ABC heard them say, no, you'll use AI to get rid of the societal biases. In other words, you'll bias society with AI against its own biases. You'll educate it, so to speak. Good luck. I noticed today just before coming to this meeting that there's an article about the European government building a framework for AI and I'd like to come back to Christy's point. This, how many grains of sand are there on the beach? There are just so many different ways that this can be used. So, I've only got one criteria and my criteria is this. Let any AI application state its source. Provide transparency and accountability on the AI function that's performed and one of two things will happen. Everybody will get educated at the same time or AI will disappear quicker than it started. Because once you bring in accountability, the people who are making money and it's at the best of my knowledge, there's only four or five people in the world who are running this and providing the frameworks for it. They have total control at the moment and it keeps coming back to transparency and accountability for me and having a framework that provides incentives and penalties to get rid of the fake news if I can use that term. I also noticed that in the chat comments, somebody asked whether we are using AI at the moment. I would come back to Christy's point. AI is being used successfully in a number of areas but who's keeping tabs? Who's working out which is the good AI and which is the bad AI? Where's the framework? Where are the controls? Where are the penalties? If you let up the garden path and while the dollar runs this well, those people who control the structures will be more concerned about making a dollar than the example I gave earlier in the data quality about making sure they got it right. That's it. Thanks, Anne. Nigel? Yeah, I particularly like the Excel question about lowering the bar for using these tools. One of the things I think is really interesting in the chat GPT implementation of this, I've only seen a couple so I'm not universally aware of what's going on but the chat GPT version includes an explanation in its own writing of exactly what it's done. So when you ask it a question around an Excel spreadsheet or you ask it to create some table from some spreadsheet data or whatever, it explains exactly what it's done and show right down to the sort of logic that it's used. And then you can look in the Excel cells to actually see what calculations it's done. So in this case, and certainly in the hand of an expert user, it's very powerful and you have that ability to be able to check and make sure that it's doing what you want and has done it correctly. So I think that's really quite powerful. I've seen another implementation where it just produces the answers and I would be a lot more concerned about that. I think it's a really good question though about lowering the bar for access to tools. So much of, I guess, training and education is getting you to have enough of a background base in an area to eventually become an expert spurred in that area. And so lowering the bar is obviously a good thing from a societal point of view of enabling more people to contribute, more people to become involved and use these tools. The question is, well, the risk is around people using it without knowing what they're doing. And although GPT will do what you ask or the other programs will similarly do what you ask, whether you're asking the right questions, whether that's a legitimate conclusion, all those sort of things are potentially swept under the carpet through that. I agree with Ian's comments around societal biases. Given that we're training the data, training these models on generally a pretty much uniform what's out there basis. Now, they're so large, it's going to be very difficult to undo societal biases using these sort of tools. Could I quickly add to Nigel's point, Mark? We all accept that students are using these sorts of tools to finish essays and assignments, et cetera. And one of the things I used to do with my students when I was aware that this was occurring is I would call them into the office and say, well, look, can you explain the principles for me? And nine times out of 10, all I had to do from that point on was just hand them the tissue box. So at the end of the day, critical thinking is what could be lost here. And I don't even think critical thinking has actually got started properly in the universities and teaching organizations. So there could be a heavy swing to the universities in that regard to start up in the ante with regards to the critical thinking processes. And could I also finish by saying, I think AI, just like the atomic bomb, which was meant to create endless energy, can be used for a lot of wonderful things and a lot of very, very bad things. Thank you for those comments. Before I open it up, I think there's one other observation that I'd like to throw into the mix. And that's one of the things you drew my attention to Ian, and that was the post office software implementation problem in the UK where a lot of people lost money as small proprietary type business owners of the postal outlets because the software was giving incorrect answers. Yet it was all kind of swept under the mat. And it's ruined a lot of people's lives. So software has been around a long time. We use it all the time. We're using it now just to have a Zoom. However, if we don't pay attention to the problems, I think there can be pretty grave consequences. If you add to that the power of Twitter and the number of shows that have been on TV about the effects on young kids just using chat on Twitter, et cetera, and having suicides as a result of it, we really, really, really have to take this very, very seriously and get some framework and structure around it. And that's, sorry, just to add my two cents there as well. But that's where regulation and policy comes in. So the Commonwealth Department of Industry has been doing some great work in this space. Check out their rapid review and their interim guidance on the use of AI for Commonwealth government. New South Wales government also has a policy and framework out there. But the EU Act was just introduced last week, I think, which outright bans some use of AI. So there are frameworks in this space within this. Yeah, I think the rest of the world has to follow the EU and they tend to do it first. I don't think it's fully passed as yet. It's pretty close. But yeah, that will certainly have some implications for where AI goes. Can I make a quick comment, Mark? I noticed in the chat there's a bit of discussion around expert users. And this is really interesting. So my son is obviously of the generation that's most used to this. He's trained very seriously in maths and data science and so knows exactly how these things are working and understands them very well. He tells me about some of the advanced prompt engineering that is being used in using these tools. He's doing it within his business life. And there's some really creative, really interesting processes where you can use really good prompt engineering to manage much better what you get back from AI and really increase the quality of what it's doing. So expert users in that space, there's a lot of potential, I think, in that. Okay. Absolutely, and that's where I'd say, maybe even in my own view of the world, you'd scratch out the word expert and add in responsible because I don't know if any of us can call ourselves expert because the technology is just moving in such leaps and bounds. But I think a responsible user of this AI technology would understand the sort of ethical conundrums and challenges that can appear. So we've touched on selection bias or bias in the training data. We've talked about monitoring and assuring that the outcomes are actually fair and that decisions are traceable and transparent. I think the best way to do that is to have a good governance regime, evaluate the tech to make sure it's doing what it's expected to do and continue to upskill yourself and remain capable in the space. Thanks, Christy. I noticed some of the questions and in the space. So what I propose we do now is turn our attention to those, but also if someone has a particular view they would like to share beyond our panel members, please stick your hand up and I'll try and get to you. One of the questions that I think is interesting, I just went past it, I think it's by Caroline Henwood. Christy, is there a third perspective question? How are they using AI in evaluation and what does this mean for comparisons, synthesis across evaluations? And I guess, yes, one of the risks I see is people give us evidence and there was a piece in the conversation just this week about a long-term study that had been collecting qualitative data. Respondents are now using chat GPT or other tools to fill it in and they notice the quality of the responses has changed and is wrong. So that's dubious data, but what about us as evaluators using it to make the process easier for us or potentially greater returns if we're trying to maximize our income? What are your thoughts? I will start a bit tongue-in-cheek with the classic evaluator answer, which is it depends. So when you talk about use, what are you meaning by use? I would say using it to write your whole evaluation report is unethical, I wouldn't say that's ideal. I would say using it to help you get a sketch of a writing plan would be a wonderful use if people are unfamiliar with that, using it to get a summary sense of the literature on a particular topic or to understand who the prolific authors in one area could be to even sort of do summaries of swathes and tropes of stakeholder consultation information or survey information to do an initial sort of high-level analysis for you that would be an appropriate use. So I think it's about breaking down what we mean by use. So I think if we outsource our job entirely to the machines we might as well resign and go to McDonald's, right? So it's how do we work in tandem with the technology? So that's why I love the term AI enabled. So how do we enable our work through this technology? So for example, but there are perils. So for example, at Grovener we are looking at co-pilot and whether that's a solution for our business to help us speed up things like conducting reviews or synthesizing large amounts of data. But we hit a snag. So last week the person in charge of investigating it found out that this technology actually wants and takes access to your whole SharePoint. So not just specific limited databases you granted access to. So your data's out there forever. So we've actually just put a pause on that because we don't want to ever share anonymized or de-identified data or private data. So the sort of context within which you're using it depends if you're using the free version online, if you're using a private enterprise version and what you're using it for. But I do think it's an area where we need to understand more about what's happening in the field so that we can learn from it and share better practices. Thank you, Christy. Anne or Nigel, did you have any other insights that you might want to add to? Okay, no, that's okay. I noticed Bill Wallace from the society has posted a comment or a link to another learning opportunity, one that is coming up on April the 16th. There is another seminar on artificial intelligence 101 for evaluators. So if you're interested, that could be an opportunity to have a look. Are there any questions that people want to ask online? You know, using a voice, sharing their photo, just while I have a quick look. Can I just step in for a second, Marketsville Wallace, who's CEO of the AS. Thank you everybody for presenting today, really interesting and thank you for attending. I just want to also bring it to your attention that we're currently in the midst of organizing a festival that will happen during May, which will be dedicated to AI. So we'll be sitting around and say the date shortly for that. Thanks, Bill. John Pilla Mamello, I will get to you in one sec. I'll just start on John's question. So won't be a second. John asked the question, are there risks any different to a novice using Excel to undertake a statistical analysis without an understanding of the assumptions embedded in the different statistical methods? Nigel, you might have particular views on this one. Yeah, I was actually in the middle of trying to type an answer. Yes. So in many ways, the differences, oh, sorry, the risks aren't really any different. I mean, what you're talking about is a technology that has enabled people to be able to apply sophisticated methods very easily. And that's not a bad thing. I mean, if you go back to a 1960s statistics education, a significant component was how you calculated the variance because it was a really intensive calculation. And so part of the education, the training was how you could efficiently calculate the variance. You don't teach that in 2024. But what it meant was that for you to be able to use any of those tools in the 1960s or 50s, you had to have that education. Otherwise, it wasn't practical. So I think the risks are very similar in meaning. However, I think in scope, they're significantly greater because the sort of the entry barrier or entry threshold is lower. You don't even need to know basic Excel. You can just write and ask questions. And I think also the complexity and scope of the potential application is much greater as well. So I mean, I have no concept of how broadly this could apply way beyond anything I'd ever understand. So I think it's, yes, it's a very similar risk, but I think it's a more significant risk purely for those two reasons. Thanks, Nigel. And having seen students who feared numbers previously and Ian has seen the same sort of student as opposed perhaps to Nigel's who I hope would have liked numbers if they're in a maths course, I have no doubt that students will rush to this and adopt it very, very quickly. The biggest risk is understanding. So if it's easy to access, easy to do and get results, that's fine, except the people who then go out from training institutions into the workforce are getting results, but they don't understand them or how to derive them. And that I think is a problem, particularly when it just happens so much faster than what was done previously. It was easy enough to go buy an assignment, but you get potentially caught because of other techniques. Now it's going to be a lot harder, a lot, lot harder. Interestingly, just on that one mark, I was having a conversation with someone today who shared that her daughter was pulled up by her teacher for using AI in an assignment. And so we sort of talked a little bit about that example, and one of the reflections I had was, where should people use how to learn AI if not in school? So, okay, using it for an assignment isn't great, but isn't this the place we need to be teaching people how this technology works and how to operate it responsibly, as we've been talking about, so that when they do hit the workforce, they know all those implications and can navigate through it. So I think it's important to address it from the front, so that's all. Yeah, I agree, the challenge is they've already seen what's happening to student grades. If you're not adopting the use of AI, a good student will be surpassed by students who would in the past not have been as good as them, so the pressure is on for everyone to adopt AI, and I think that's disappointing, but it's a risk and we're going to have to prepare for it. I know, Mamella, you've had your hand raised for a while, so do you want to ask your question, please? Hi, am I in return? I think I am. Yeah, my question is follow up to Christie's response regarding the use of AI in evaluation, though it's directed to all the panelists. I'm interested to know if they have used AI in evaluation, how they used it, and what was the outcome? Well, I'm happy to jump in there. So surprisingly, even though I read about and talk about this topic a lot, I haven't actually used it that much, so I've got more of a theoretical sort of understanding and interest in the space. I'm also incredibly old school, so the first thing I teach new consultants that come into us is do your analysis, like with paper and pen, put the keyboard away. So I think it'll take a while for me to sort of move my Luddite-like tendencies into integrating it into my own work, despite knowing very well the opportunities and the need to do so. The thing that I'm sort of looking at for our business, though, is like that analogy with students that Mark set out, are we about to be on a race to the bottom? So if our competitors can undercut us by using AI technology to write their reports to do substantive data analysis and we continue to price for human effort, what does that mean? So that's sort of the risk we're looking at for the next five years, but now I haven't used anything yet, more because I love writing and I love creative writing and I love getting those used to the flow, but that's just me. Okay, if I could add to that, I would say that I haven't used it at all, but two things quickly come to mind. On the Q&A show on the ABC this week, the question was asked, what happens when you have rubbish information coming through AI and the arts are given by the panel? I've learned that people was, everybody will divert to trusted sources. Nobody ended up asking what the trusted sources were. And secondly, Mark came across an article just recently which I've been suspicious of for some time where because of time, money, gotta get your accreditation up, we've got a lot of university people publishing papers that have been where they've used AI to help them put the papers together. A further study has found that a lot of the research could not be replicated on review, which raises some serious questions about where research is going. So I know that is different to the question you've asked and the answer was I haven't used it, but the last part of the comment I'd like to make is Christy again mentioned how wide this field is so that when we talk AI, each one of us turns around to, oh, they mean, and we make an assumption which came out in John Pilda's question, I use the word assumption which I love. The bottom line here is what do we mean? So we could probably go back to Bill for instance, if we have enough time to say, what do you mean by what you're going to offer at your session in April, Bill, with regards to AI, so that the people turn around and say, oh, it's chat GP, I'm really interested in, or it's whatever. Sorry, Manolo, that's all I've got. Thank you, that was good. One other comment from me, which is not direct to evaluation because I don't do that, but I have used it around trying to write software and my general conclusion would be if it's something relatively standard, it's very good, but if you're trying to do something which is out of the box, then it can't have seen that before, right? And so it ends up pulling things together and it pulls things together that sometimes you actually can't go together. So it will never be a solution. It's not just you've got to tweak a few things, it will never be a solution. So I think it's another thing for people to be quite aware of is in terms of those uses. So the comment earlier about your cover letter, absolutely great for those sorts of things, but the more creative you are, the more you're on the edge of knowledge, the less and less reliable it will be. I was thinking to pick up on Ian's point as well over a few minutes and take the bill out of the hot seat. So the 16 April session, so that's me again as well as Gerard Atkinson of ARTD, Giant Gigantic Brain. So that's AI 101 for evaluators. So we'll be going through common terms and terminology and sort of theory and history. And then Gerard will actually talk through some of his applied work as well. So that's what that one is about. Thanks, Christy. I guess as a facilitator on the panel, I can say, yes, I have used ChatGBT with a client, but it wasn't for evaluation purposes. It was for something else. It was more of a innovation and research space client. And we got some very interesting results. If you weren't an expert in that space, you would have certainly been led up the garden path around the corner and probably through quite a few rabbit warrens as well before you realized it was giving you rubbish. So some of it was good, but some of it was, yeah, it was putting things together that you'd go, don't think so. And it would give us references for them and would hunt the reference down and you just wouldn't find it. So yeah, remember, it's probabilistic. So it was putting stuff together that looked really, really good, except it wasn't. So... We didn't get a titles with journals, with authors. They don't... Oh yeah, did it the whole lot. We're looking for these things and that don't exist. You guys aren't saying they made it up. Probabilistic model, it's creative, generative. So yeah, any last questions? Because I know we've got about a minute to go. Does anyone want to jump on? I wanted to just give a little bit of a plug to Christian Tremonga. So anyone that's interested, he's just what I've posted for him is a master's research project on the use of AI and evaluation. The details are in the chat if you're able to support your master's research, that would be amazing. Okay, thank you. And I did notice someone who was... Oh, it was you, Christy, who made reference to the CSIRO National AI Centre. Awesome resource. Checking out it as well. So first of all, I know we've reached the time to end. So thank you for everyone who's come along today and put questions in the chat and posted those. Especially, I'd like to thank our panel members, Ian, Nigel and Christy. We appreciate your time, your comments and your thinking before you join today's session before... So you could talk to us all. I wish you all a very good evening and look forward to seeing you at a future event.


Greetings friends survival doc here today. I'm going to plant four fruit trees Now if you haven't planted your fruit trees yet, what are you waiting for? It's November the 10th you plant during the dormant season. This is the time to be planting the fruit trees It takes generally two three years before you receive fruit from your fruit trees Our hyperinflation is right around the corner.  These are the trees that I'm planting today. I have a couple of small plum trees I'm going to plant a new apricot tree because my apricot tree just died from an insect a boar killed it and I'm going to be planting a Stanley prune plum tree the first one. I'm going to plant is a Stanley prune plum now I Recommend you plant if your space is limited you plant dwarf or semi dwarf trees That way they don't take up as much space and you can plant more trees All right dwarf trees will always be grafted and you can see here. This is where they grafted it This is the rootstock and this is the prune tree right here and it's grafted right here You want to make sure that you you plant the tree with the soil level about right here about two or three inches below This graph mark if you plant it too high what will happen is this part of the tree will start putting roots into the ground And you'll end up without a drawer dwarf and it'll grow to the Full-size tree, which is what this part wants to grow So you want the soil level about right here All right, you can notice that my four trees are in a bucket of water Before you plant you want to soak the roots in water for at least two to four hours up to about six or eight hours No longer than overnight Over here I have my hole as they say you want a Ten dollar hole for a five dollar tree you want to make sure that you dig your hole deep enough My hole is about two feet deep make sure it's wide enough To make so the soil be real loose and the roots can get off to a real good start What I have over here is the soil that I dug out of the hole now I'm going to mix that soil with what I have over here what I have here is rabbit manure directly from my rabbit hutches and then here I have a bag of topsoil and What I'm going to do is I'm going to mix some of this topsoil with the Soil that came out of the hole and with some of this rabbit manure make a good rich mixture and I'm going to put The topsoil in the bottom of the hole you want your best soil in the bottom Loosen it up really well so that their roots can get off the real good start You notice my soil is quite dry I'm gonna have to do a lot of watering and keep this Tree water for the next couple of weeks Now if you don't have rabbit manure you can also just mix a tree starter fertilizer in with your Use a little board or a stick to put across the ground to show me exactly where the soil level is going to be When I plant this tree Again, I want the soil level about right here. So I'm going to plant it to where the Where it comes up to right here with this Graph portion about two inches above the ground as you can see I've got about a foot and a half of hole to fill before even reach the roots and that's good Cuz I want to put this Really rich Topsoil and rabbit manure in the bottom mix it in with some of the Soil that I dug out of the hole and that way the ground to be nice and soft and rich Fertile and the roots can get off to a really good start as you can see I filled my hole almost full Now I'm gonna put my tree in you want to make sure that the roots are spread out Have as much room as possible to grow check my soil level which is going to be right here Okay, again. Here's the graph Want that two or three inches above the ground? So I'm actually going to need to pull this up to about right here and Later after I get this all in here. I can Check this again, and if I need to I can pull this up while it's still very loose pull this up It's critical that you get this two or three inches where this graph is That you get this two or three inches So that right there would be perfect Now what I'm going to do first Before I filled the hole in completely as I'm going to pour a lot of water in here and Just soak this hole really good and that will Get the air bubbles out Allow it to settle a little bit then I'll pack it down with my foot pull this up a little bit if I need to to get it at the right height and Then I'll finish filling in the soil again and Check the depth again and pull it up a little bit if I need to You know this I put my best top soil in the bottom of the hole and at the top. I'm filling in more with this So that I dug out of the hole And while the soil is still loose if you need to pull it up a little bit to make sure that that notches two or Three inches above the ground. That's fine I'm going to let this water soak in Then I'm going to finish filling the hole and Then I'm going to water it some more and over the next Few days and weeks I'm going to Need to water this several times because unless we get some rain Because soil is really dry today Okay, I have planted my tree on the side of a hill on a slope So I want to make sure that the tree grows up and down so I'll need to tilt it toward the sky if Necessary as the twig is bent so grows the tree As you notice, I haven't quite filled the hole yet That's because I'm going to water it some more a little later on after this is soaked through and I'll keep checking the Level to make sure that the graft is two or three inches above the ground Then I'll fill this in completely now in the wintertime rabbits like to chew on the bark of fruit trees, so I'll put wrap something around this to protect it and Just want to make sure you don't wrap it too tight and that you don't leave it on there too long because it will squeeze Tree also this little piece of plastic here that goes around the tree If you forget to take this off this can also squeeze the tree as it grows So I just took it off just hanging it on the twig Right here for now I'll also probably put a little fence around it too to keep people from bumping into or to keep deer from Knocking it over because at this point Trees very delicate has to be taken care of carefully Here's my next hole I'm going to plant a little Plum tree in here. You can see how big the hole is compared to my little tree and Again, that sucked so I can really loosen the soil up here put some good fertile Soil and fertilizer in here to give the plant a really good start now I dug my holes before I received my trays I ordered my trays from a Stark brothers nursery located in Louisiana, Missouri and When they're good they ship the trees when it's time to plant them no matter when you order of them and Before they when they ship them they send you an email and let you know They're shipped so that you can be prepared for them And so you really want your hose dug before you bring your trees home Because when you get your tree you want to be able to put it right in the ground immediately you just want unpack the tree soak it in water for a couple hours and Get it right in the ground and to do that you want to make sure that your hole is dug ideally before you receive your plant Okay for my plum tree if you'll notice it needs some pruning and When you first plant your trees you generally want to do some heavy pruning and I'll cut back all these branches here Because you want all the energy to go into establishing the root system at first. Here's the tag Here's the way to put it on there. You want to remove this tag so it doesn't choke Your tree as it grows You can just remove it and tie it on with a little piece of twine or I'll just usually put it on a little branch here for starters This is a Plum It's not grafted. So the depth is not as critical. I'll plant it About where they look like where it looks like they had the soil when they grill it Which will probably about about right here So to sit in the hole about like this says survival dock Reminding you to be prepared or be prepared to be fleeced