Okay, we're going to get started. I'd like to welcome everyone to today's CNCF webinar production get-ops in practice. I'm Jerry Fallon and I will be moderating today's webinar. We would like to welcome our presenter today, Rick Spencer, head of platform and inflex data. Just a few housekeeping items before we get started. During the webinar, you're not able to talk as an attendee. There is a Q&A box at the bottom of your screen, so please feel free to drop your questions in there and we'll get to as many as we can at the end. This is an official webinar of the CNCF and as such is subject to the CNCF Code of Conduct. Please do not add anything to the chat or questions that are in violation of the Code of Conduct. Please be respectful of your fellow participants and presenters. Please also note that the recording and slides will be posted later today to the CNCF webinar page at cncf.io slash webinar. And with that, I will hand it over to Rick for today's presentation. Thank you so much. I hope everyone can hear me okay. I guess I'll get some feedback. If you can't, I'm really happy to be here. I really appreciate the opportunity to talk about the way we use GitOps in our Kubernetes practice at InfluxDB. So first, to just provide some context, I thought it would be useful to just talk a little bit about Influx data. So what Influx data is at its heart is a time series database, although InfluxDB has grown to be a platform on top of that where people can write custom monitoring applications, custom IoT applications, people write custom finance applications. Anything where the time dimension is the most important dimension of your data, InfluxDB is a really good place to develop that application. So our special kind of database with a platform on top of it to make it easy to write applications for it. So I always put this slide here to stop to remind me to talk about open source. This is open valuing open source is one of our custom companies core values. We do have an open source version of InfluxDB. We maintain a lot of open source projects. The purpose of this talk, it's important because while many people use InfluxDB to monitor their Kubernetes installations or their applications running there, we do not see ourselves as a Kubernetes tooling company. So what we really don't want to end up with is maintaining a set of bespoke Kubernetes deployment tools. So we opt to try to find out like what's the most supported in the existing open source community. Pick what matches our use case the best. And then if we do need to write custom code, we prefer to do that code upstream within the projects that we're using. So I will touch on some of the open source projects that we use as we go. Okay, so the main product that I focus on in my role is called InfluxDB cloud. Informally we call it cloud two because there was a cloud one product cloud. InfluxDB cloud is a Kubernetes application. Obviously, since it's got a database at its heart, it's very stateful. So we talked to you a bit about what cloud two looks like in production. Okay, what and why this will build to why GitOps was a useful methodology for us. So cloud two is a fully SAS data platform. So it's like you sign up for an account. You don't have you don't host any servers yourself. We do like all the server maintenance hosting and everything. But the thing about a database is there's this notion for customers like that their data has gravity. And by this we don't mean that in like a marketing sense. It's almost more literal than figurative. There data has gravity in the sense that if you have like terabytes of data, it's very costly and slow to move it between regions within a cloud or between clouds, etc. They really want their data services to be as close as possible to where they're storing the data themselves. So they're generating data in Europe. They want to be able to, you know, operate on their data in Europe store it there operate it there they don't want to copy it to the US. They're generating their data in the US. They want to be able to store it and operate it close to where they're they're creating it there so Additionally, people like they may be generating services with a one cloud provider. So if they're you know generating in AWS or GCP or Azure they want to be able to store the data within those cloud providers. We are multi cloud and we're multi region right so we operate cloud to or influx DB cloud we are operating it in multiple regions on multiple clouds. Additionally, soon, we are going to start providing customers with private instances. And so, in addition to all the public instances that we have will have a collection of private instances that that will be available, you know to customers in the region of their choosing on the cloud provider they're choosing. So for this reason, very early in the project very early Kubernetes was chosen as the deployment as the operating platform because it does provide a cloud abstraction layer that really allows us to be multi cloud be multi region have like all these different instances, and have a same way to operate all those in production in mission critical way. So, Okay, so we decided to go for get off in order to maintain our continuous deployment through all these areas. And what I want to try to communicate here is what what we mean by get off so there's various definitions. I know that the term was coined. Very specifically by we've works but like, I don't think they would disagree with this what we say like we're doing get off or we set out to say we're going to do get off to say like, we know we will have achieved it. When these signifiers are true. So first, that developers get code into production by landing in the main branch right so once your code makes it through the gauntlet of code reviews and see I and gets merged into the main branch. Automation takes over and takes it from that get repository, all the way into production. So, whatever it takes to get into the main branch. That's all it takes for the developer to get that code out into production. And then for us also infrastructure is code should be delivered in the similar manner. Right so it's one thing to say I've changed the service I've changed the code in the service. And I've gotten that code through a pull request into the main branch. What about the infrastructure itself, right, like the Kubernetes infrastructure, as well as the architecture of the application. All of that should be delivered in a similar manner so if I will. For us will key signifier doing get ops is that in order to change the infrastructure of your application or your Kubernetes cluster itself that again is done via a code check in into a get repository. And finally, is the notion of testing and production. Right and what that means is that you have a facility in place where developers can get their code in the main branch. Even if it is not currently ready for users, it can still be in production, just not available to users and production or only selectively users selectively available to users in production. It's important because you want to, we want to keep up a pace where people are just constantly getting a code into the main branch and we never have big events where we're trying to like, you know, integrate a lot of code all at once. So these are the three signifiers that we set out for ourselves and I'll refer back to this a bit later in the discussion. So, now, I'm going to talk I'm going to show some diagrams about the way that that we visualize get up so before I do I want to say like, part of our company culture is really about like transparency and learning. So when you like see these diagrams, they will look like, you know, rainbows and unicorns and butterflies but there's actually a lot of blood sweat and tears that went to get into that state. There's a lot of complexity that's hidden by the simplicity of the diagrams. So, I'll try to touch on some of the things like a lot of times you'll go to a presentation like this, and you'll like leave thinking like wow like those engineers like they really like have it together that's amazing. But then like when you actually join the team you see like that it's actually a different story that it's like actually really hard to do what they're doing. I'm going to try as much as possible to give like a more realistic viewpoint of what doing get offs is like. Okay, so this as I said is our idealized diagram. And this is an idealized view of our overall environment and so I'll just walk you through it real quick. So imagine I change to service. Okay, so here's some code that I wrote. And I do a PR to the main branch here. And then our CI system. This is actually like way more complicated right because of course CI like you do a PR the CI system doesn't put it actually allow it, the PR to complete before the CI is completed. So what CI does is not just runs tests and integrates the code, but it also builds containers test the containers, and in the case that the containers pass everything and appear to be ready to go. The CI system will then update our configuration code, which happens to be JSON it and I'll talk about a bit more about later to say hey, the Kubernetes cluster, the application should now point to this new container for this service right and then that configuration gets updated automatically by the CI system. And then we have what we call waves of deployment. So the first thing is our go to text like hey, the configuration code for the staging environment has been updated. The configuration for the staging environment has been updated. Let me tell Kubernetes to apply that those changes. Once those changes are applied, we run some tests here. Right, so these are like automatically run by our workflow. If that works, then the Kubernetes configuration for the next wave is updated and it rolls out to what we call internal production informally internally we call it tools. The reason we call it internal production is because only we have access to it. But it's where we do all the monitoring for everything. So when any whenever there's any hiccup in tools, we treat it as a full blown production event. Right, because if like our, our internal production environment is is not operating optimally, then potentially our monitoring of the external productions is not optimal. So we, you know, that's why we use the term production in it because like we treat it fully as a production environment. Again, we run some smoke test here. If those pass, then we update the environments for all the external production external production. Are all the Kubernetes clusters and and instances of the application that end users use that customers use externally. Okay, so there's some like caveats and some more details that I want to cover in this part. So first, when we update the code, it's very like easy. It was easy to write, you know, tooling and a workflow that would do these waves. It's much harder with the configuration right because like the code. When we update the code, it says hey configuration. Can you please update this, you know, new update to the new hash of the container. But what if we're like changing part of the architecture like we're changing like we're scaling something out or, or whatever, we do check that in this code but we don't really these wave options. We actually have to do manually if we do it. Often we don't will just like just like change everything all at once. Usually, usually that's fine but if you have to be careful, we still don't have automated a way to like go through these waves if it's a con if there's no code change but it's just kind of. Okay, want to touch on the staging environment and I'll touch on this later as well. We still have a lot of conversations internally about staging. If you look at the literature, like what our thought process here is like okay, we're going to land on these small instances that are much cheaper to run. They are not mission critical, and we'll run tests on those, and therefore we will have like a way of getting confidence that the change that's being rolled out is free of flaws right and it makes makes a lot of sense. But if you look at the literature, the DevOps literature a lot of people will tell you like, you're just you're dreaming if you think staging is going to help you in that way. The staging will never be configured the same and it will never get the same load. So, we still have a lot of internal discussions about whether we should even have the staging environment or not. For the time being it's working for us more than I think it's hurting us so we're not feeling a lot of pressure to change, but it's something to keep in mind and I'll touch on it a little bit later also. So these, this is what we call this actually the automated feedback loop to set up a way that like we can deploy to a wave either here or here. Actually, we do it down here too, but since this is the end of the line. There's, there's no way to like it's good to know that there was a problem, but there's no way to block the next way since there is no next way. But right now this is a set of smoke tests. We're not using any of the monitoring and alerting that I'm talking about yet to be part of that automated feedback loop, but I have some slides later where we can go into a little bit more detail on that on those concepts. Okay, just in terms of like my comment about rainbows and unicorns and butterflies. Don't look at this. Okay, it's totally out of date, but this is really if you start to really unpack everything that's going on in that internalized view, you'll see a lot more components come into play, and a lot more complexity.  Okay, so when after we went through like all the effort to get to that state that I just just explained. We started to see some like really clear benefits. First was like, like logical we were just shocked about how powerful it was was just the amount of time that developers got back, because they were not working on release process. Right, like, like every day when like code was at a logical stopping point, they made a pull request. And that was it. So there was no effort to like batch up releases, babysit releases, the final release. There's no idea of pushing a release, you know just like developers can just like let it go and that gave them a lot, a lot more time to develop. Similarly, and maybe counterintuitively for some people, like our incidences, our rate of incidents in production went down, like, dramatically as soon as we started to automate the deployments through get ups. So we think that there's like two, like we went and we looked at like our incident RCAs and everything. The two main contributors to the decreasing incidents was first. There are just many fewer complicated releases like the changes. So, you know, rolling out changes like today we do changes like deploy like probably like three to five times on a typical day, maybe more on some days. And like the changes were like much smaller that were going out. And so their ability those changes ability to have a big impact was reduced. Most of us like very obvious what caused the problem. The problem went out so like smaller easier to diagnose problems, but also we did learn in one case. It's possible like since we have a computer program rolling out the changes to make errors on a much bigger scale. But it's much easier to recover from those errors. And in fact, there's just the reduction in errors caused by humans doing deployments was just a really dramatic improvement for our robustness and availability.  A computer program does things in the order that it was told so. So that was like, you know, just really those two things just to decrease in working on release process and the fewer incidents really immediately added a lot to just the quality of life for the developers on our team. Also, like, you know, one of our tasks is to add more regions. You know, the customer wants to work in region through since everything is automated. Everything is captured, all the configuration is his code, like all we have to do is just set up some files to define the region, and then our existing tooling automatically will roll out to that region and so, you know, it's like a single digit day process from beginning to end. Honestly, the thing that takes us the longest now in rolling out regions is just getting the, the machines from the cloud service providers, you know, another thing just in terms of everything is just because what's now what's in master or what's in the main branch is either in production, or on its way to production, just everything is easier just so much easier to reason about everything because if you want to know what's running anywhere, like, it's, it's just obvious right so there's no, there's no like dipping between the different environments, different between a production environment and the code. So, like that also has been like just an improvement to our robustness there. Okay. So now this part is a little bit more for leaders or managers right so I'm imagining like you're like wow, I want those I want those uniforms and I want those rainbows. So now this is just a little bit about like how how to get there from an engineering management point of view. Okay, so we broke it down and kind of like required elements, like this is a bit retrospective, like what was required, versus like what were some optional components. So obviously, you need a CI pipeline to build artifacts and to run all the tests and everything. If currently today, your deployment pipeline involves developers building containers themselves, like then you're like that's like a really first problem to fix like you just you'd if like your developers are building containers and then pushing them into production then you're simply not going to be able to rely on automation around to get repository to make sure that happens. Again, you need a way to represent your infrastructure as code. For the application itself. We use jsonnet for this as an aside we use Terraform for the Kubernetes clusters themselves but we use jsonnet and I'll show you a little bit about why we use jsonnet why it's beneficial for us, but the combination of jsonnet in cube CFG makes it really easy for us to represent the infrastructure as code in a multi cloud scenario. You need a way to deploy Kubernetes configuration right so once the configuration changes and get you need something that knows how to tell your Kubernetes cluster like make the application look like what's in get we use Argo for that. One thing that you like really need to do is make sure that your deployment pipeline is fast and this is actually an area where we right now have mixed success. If you're one of the things about like deploying to Kubernetes and I'll talk about this a little bit later also is like during deployments right there's containers coming and going and making those deployments smooth is a lot of work. Right so if you can imagine your services are the containers are being killed and you know the pods are being replaced by Kubernetes during the deployment. There can be availability issues during that time so faster it is the fewer those problems that that you might run into. Additionally if you think about just recovering right like so hey there's a problem like so for instance any problems that are UI tier we can roll back and within a matter of minutes we're back to the old UI assuming that you know it was a reasonably uncomplicated deployment because that deployment pipeline is fast we have a lot of confidence because it's really easy to recover. There's other areas of the product where the deployments are a bit more complicated. This may affect us more than it'll affect most people because we are a database or a stateful but managing those like state minimal to the deployments we do have some work to do to make that faster. But our deployments are fast enough. If your deployments are very slow you may find that GitOps is a bit problematic because for us one of the important signifiers is that you're keeping all your code in the main branch even if it's not ready for users you need some some way of feature flagging. Now we actually started feature flagging without any kind of service so when we started with it it was just totally manual. Currently we use the config cat service of a service like config cat or there's you know plenty of other ones like they offer a lot of value in terms of using feature flags like for product managers to be able to send you know specific users to try new features and things. And then of course most importantly you must must have production metrics and especially reliable alerts right so if like your developers are going to make a pull request into the main branch and then that's done it was accepted my job is done and turn their attention elsewhere. They cannot do this with confidence unless they know if there is a problem they will be alerted otherwise they will make a pull request and then stare at a screen until they're confident that the that there was no problem and making a screen that gives you that confidence is actually harder than it sounds because a lot more can go wrong than what you predict so reliable production metrics and alerts is really one of the core things that you need to build to build in and I'll talk about how we did that. So there's some components that we believe are optional. And some of them may be surprising that we consider optional. So for instance the automated feedback loop we actually switched to get ops and doing a continuous deployments without any automated feedback loop so there was no way for us to like stop a deployment. In between waves when we first started but we had so much when just by having an automated that like that. That it was worth it for us. Now we do have automated feedback loops and we do have deployment waves and we are. improving that process as we go. Another optional thing is like we call it the pipeline for a high level operations. An example of a high level operate operation is like running a set of tasks after you do a deployment. We use our workflow for that. Canary deployments. This is actually a next step for us being able to do canary deployments. A lot of people. There's different opinions in the literature about the value of canary deployments versus other kind of deployment methodologies. But we found that we got so much value from get ops without it that we actually deferred implementing it now we're starting to work on that. Another optional but really desirable piece something that really added a lot of value to us is this concept that we call external availability validation. So we ended up like saying like what is the user's view of availability right so we're developing we're getting all these metrics from inside the system and we can see like how many users are getting 503. Or you know 400 and that kind of thing. And but what we found was really valuable is actually running some processes from outside that just use the API and just measure like how many how successful they are. I'll talk about that in a bit but that was a really useful sort of rallying point. For the team so we call it external availability validation and then finally already touched on this concept of testing or or acceptance environments. We consider it optional like a lot of people in the literature consider it an absolute anti pattern so I won't take a stance on that but I always find it interesting conversation. Okay, so I wanted to touch a little bit on like I mentioned jsonnet and like why we like jsonnet so what is jsonnet so jsonnet is a super set of json and it's designed to allow you to overlay configurations on top of other configurations and this is very very useful for us because we have a base configuration for our application and then we can overlay specializations for the different cloud environments or even the different regions you know one region is utilized much more we may we may specialize the configuration allows us to maintain all of that configuration as code but in a very same way. And so just how that roughly works is if you see here like here's a this is a contrived example of base configuration and the main thing to look at here is it describes replicas and it says one replica etc. Then we say okay that's fine for the base configuration but we need to do something different in AWS. In AWS we need four replicas and furthermore we need to add some some limit requests here need some limits so okay so this then this tool called cube CFG then can take these two things and output yaml. And so you can see in this yaml this would be the yaml for the AWS prod which you can see has all of this content plus these overlaid different content here and in this way we can have sanity for maintaining all these different regions. So what happens is that the cube config tool will cause more yaml to be output will we will automatically check that output of yaml back into get and so that change in get is what tells Argo hey there's been a change to the configuration let me roll that configuration out to production. Okay, so there's other ways of doing this. There are other projects. This we've just found this to be just work really well for us conceptually as well as in practice. Okay, so I want to touch on something this may not be related to everybody, but for us it turned out what we call super smooth deployments was was super important. So for our users, it turned out they had a very very high expectations for availability their API calls, like as soon as it falls below five nines like they feel like I got a error in my API call this week I got one this week, like, you know that to them is too much. Right. So, but, as I mentioned before, like deployments themselves cause pods to restart. And so, or to get killed and be replaced right so frequent deployments cause frequent pod restarts that can cause. You know users to experience poor availability, or at least the perceived poor availability from their point of view because if you imagine you know, like if you make a request through the gateway front end. It goes to the compute here it goes to the storage here, and then all the way back and back out to the user and like pods can be restarted anywhere in that change during that that time period. Like any you know one call can, you know, anyone failure during an appointment there can cause the user to get back hey I was unavailable. You know, so we put in actually a lot of effort in our custom controllers I did I asked like a lot of trusted advisors, people who have been in the Kubernetes project from the beginning. Some did not answer me. Some, some answered me with just like yeah that's really hard, but we did get some really good advice, besides using the custom controller that our contrast, our custom controller code ensures pop proper ordering of everything. So we had to invest a lot in that to really make sure that like during the deployment process, you know the load balancer the gateway, the compute here all of that was shut down and read, you know, like the connections drain and everything, like an exactly the right order in the right amount of time before Istio gave up and just shut everything down by itself, etc. So a lot of custom code there, but one of the main things was, you know, like what I call like be one with the retries, right so if a pod gets restarted by a deployment, it will return of something in the 500 area like a 503, which means like the server was unavailable for that call. Can you please try again. Right. So we had do have a lot of code then to just like do a retry between the tears if you got that we have an extra challenge though, because if you're working with a database, many of the requests can have side effects. Right, so you can't just blindly retry something that has side effects. So there's many cases where we can't just retry for the solution but you know that's where customer control code comes in. Now, most people that I talked to, who are using Kubernetes, they simply do not have these high requirements for availability. So like they don't know most most of my friends in the in the who are using Kubernetes like don't really relate to like to this level of pressure to keep the deployment smooth. Okay, moving on. Okay, so now I want to talk a bit about how we do get metrics. So we gather all of the metrics for a few Bernidis cluster with Telegraph Telegraph. It's an agent for collecting metrics. It's got many, many input plugins and a few output plugins. Of course, the main output plug and being in flex DB. That's because this is an open source project. Telegraph is designed to be run as an agent to be run as a server. It's designed to be run on command it's designed to be run with long lasting connections. It's a very useful piece of software just for transparency in flex data the company we are the primary maintainers of it and it's within our it's in within our repositories but there's quite a big community around Telegraph and there are output plugins for many other targets, you know, like, I know, like Azure, AWS, a bunch of other companies have made plugins for it. So we use it. We like it because it's written entirely in Go. So we just get a single binary, which makes it easy for us to drop in as a sidecar into our pods. So we have like Telegraph there running as a sidecar streaming metrics back and we get a lot of metrics back like very, a lot of metrics. So one thing we get is application specific metrics, you know, like the different applications that we use off the shelf. They have their own metrics that they that they provide, but then we also write a lot of custom metrics like our services. Write metrics out Telegraph. We send those to Telegraph and Telegraph sends that back to our monitoring instance. And then there's Istio and other services that are running. Those are also all providing metrics of Telegraph. We have it in just different places in the cluster just streaming metrics back. So how do you use metrics. So there's a few areas where you like that are important to use metrics in a GitOps scenario. And the first is that you just need to get alerted on urgent issues. As I said, you want your developers to make a pull request, have the pull request succeed, and then be able to turn their attention somewhere, but get alerted and the unlikely case that something's gone wrong. Right. So alerts on urgent issues. That's the sort of the number one focus we found that that was the most important work that we did. But second most important, I would say we're learning on leading indicators, you know, we found like, hey, if you know this, these certain cues grow in length, or if the latency between these, you know, services starts to grow. That's an indicator of an upcoming problem. Right. And so like we have alerts that tell us, you know, it looks like something's going to go wrong. Of course, both of these, it's very dynamic, because like, when you like get these alerts, of course you go in and you solve the core problems, but then you need alerts for like, you know, the new potential sources of problems that can come up. We have also dashboards for all system health, we have dashboards for troubleshooting like if there is a problem to look at, and then we'll also make dashboards very, for very specific situations like if we're sending out a feature flag, we may want to say like, put a scope on that. And you know, make sure if we're sending traffic to it there's not not not an increase in errors or, you know, that it's performing as well as the old system, or that kind of thing. Really, we have found it's very important to focus on alerts rather than dashboards, dashboards lead to an anti pattern that I mentioned before, human staring at screens, it's like not what you want. You really want to be able to trust that the system will let you know if there's something to pay attention to. Nonetheless, I have some quickly to run through some alerts and dashboards to show. So some of the main alerts that we care about are Kubernetes health, like it's nice to know when the cloud service provider as it starts upgrading, you know, the worker node for us. We get alerted on all kinds of application health metrics, you know we get alerted on like dead man alerts are really important so we get alerted if we stop getting a signal right then it lets us know like hey you have not gotten in 400 error from this gateway in you know some period of time that seems very suspicious right so like you know that that implies that there's something mysterious going on if we're not if we don't hear anything from a different service and then I also mentioned the external running availability tests like so if we start to get like if three times in a row and API call fails we get paged right so we know like within three minutes like if we get three times in a row in any one instance we get paged so Okay, so these just some graphs I just want to touch on them real quick these are are some of the dashboards that we use. I just mentioned external availability. This shows like hey we've had a moderate number of deploys this week. During those deploys, we had no this is real data by the way, during those deploys we had no issues. We had no like, you know, no, we didn't see any, we didn't drop any API calls, but we did hear wake way after deploys. As you can see like, over like, you know the course of a week to two failed calls that's like, you know, we make the calls every minute, that brings us below our SLA here so that's why that's read. So, so this is a good indication of system health. Overall we say we're pretty, we're pretty healthy, but we do want to look at you central with what happened there why did you use that. You can ignore this over here this is just for our own internal masochism. This is the like availability metrics that are really important. So we do check every week we have like a recurring meeting where we just go in and review. And review our system health metrics where I expect them to be. Here's, here's an example we can see over the last hour in US West, there was some read availability problems we probably want to take a look. My guess is that we actually got alerted on this. You know this probably like fired an alert. Hey, that's a leading indicator there might be something wrong in US West, because of the lower than expected availability there. So this is a debugging flat debugging dashboard like if there is an event will often open this up like we're looking at US central over the last hour. We can see this doesn't look like there's any problems right know the query pods are crashing availability looks fine everywhere. All the, all the lines are flat which means like you know there's no, there's no rogue. We haven't deployed any any pods in there. Sorry. This is the same dashboard below and we've just learned like what like shapes look like healthy systems and not healthy systems so this all says that our storage here is actually really healthy. So this is really handy if there's a problem to go in and look is like there anything obviously wrong to help steer our attention to the right place. These are actual alerts. So like you can see like query request duration, if like users are making queries and those durations start to get too high, like how long they take we get alerts. And yeah, so just I just wanted to close off on quick outro with like some of the challenges that we face as leaders like bringing the team towards this get ups model. And the first was really just like a change of mindset from engineers from releases to continuous delivery. And this was a bit deep for us because like, like of everywhere I've worked, like I've never worked anywhere that it's smarter engineers and in flux data. A lot of the engineers have been working on databases for many, many years. But the like releasing a database that customers install on their infrastructure is very different than having a SAS service. Right. So like, there was a period of time when it was hard for people to let go of the notion like every two weeks you do a release, even though we were SAS, right and so that just took practice. I was based like really a learning by doing things.  Right. So like, if I'm going to break the contract between internal API is like how do I, like, how do I roll that out. Right. Because if you're just doing one monolithic release, you just rebuild the whole binary, the internal API exchange they all rebuilt that's no problem. But that doesn't work in a in a microservices architecture. So, coaching people through that. And then there's also just this like fear of breaking production. And like a lot like and that was just like no production will break. It's not about not breaking production. It's about knowing that you broke production, you know what to recover from breaking production and having practices that leads a fewer fewer, you know, incidents, but not being paralyzed by that fear, which causes you to batch up changes to do a release, a big release all at once which actually increases does not decrease your chance of breaking production. Okay, so next, we want to get some metrics back into the automated feedback loops as we discussed. We also have found that like our feedback loop per deploy is like after the deploy, we check if it's healthy. We need to check during the deploy, whether it's also staying healthy. So we want to do is say like if failability dips during the deploy, let's not deploy to the next wave automatically. Rather than if it's healthy after the deploy then go ahead. We are now in the process of putting our GitOps the same metrics and alerting like the GitOps itself, like how long does it take to do a deploy, how many deploys do developer, like, how long does it take to get out to get a release out. And then finally, we're starting to work on game days. Because we have reduced our level of incidents is so much that like, I'm a bit worried that we're getting out of practice of handling them. So like we want to make sure that we're like stay in practice for handling them. And then finally, as I mentioned, we're going to add canary deploy. Okay, I think I have a few minutes for questions. I'm not sure how the process goes down. Well, thank you for a wonderful presentation. We have about five minutes or so for questions. So if you have any last minute questions, please feel free to drop them into the Q&A box. We have one here right now. Can you please give an example of the exporters you have for the custom metrics? Also, can these exporters work with Alert Manager and Steam Alerts to systems like VictorOps? I'm really sorry. I don't have enough context for what they're asking to really answer that well. I don't know if they're asking about Telegraph itself or the code that we wrote. My guess is that if they're asking about whether Telegraph supports services like VictorOps and such, I suspect that it does, but I don't know for a fact. So just a matter of configuring the output plugin. And please follow up with me if I just completely whiffed your question and I'll be, I'll for sure track down the answer for you. Okay. Do we have anyone else with questions at all? Why do you need to add support for Canary if Argo already supports it by default? Yeah, that's a good question. That's like really related to our current state of using Istio. So because we're a stateful service, there's because, you know, like our storage tier is stateful, there's just some difficulties. There's just some more work that we have to do to express those as services that Istio understands so that we can use things like Argo or whatever other Kubernetes native tools for that. Okay. There's another question. I'm not sure what you mean by needing to check the health during deploy to ensure five nines availability. Doesn't rolling upgrade strategies in Kubernetes take care of that? Yeah, so we have not found that to be the case at the level of smoothness that we need. So we have found that during the ploys frequently somewhere in the chain, like either Istio or some other request or Kubernetes itself will, if we're not very careful, will restart a pod during a while we're servicing a request. Some of the requests can take up to two minutes, you know, because we can be crunching a lot of data. So in fact, you know, we can dip below five nines, but we've also had much more profound problems where like during the deploy, we just messed it up. Like, and so we, you know, at one point, like the code disconnected, like our code, we disconnected a service before we created a new service.  And so during that time, we just couldn't service requests. Kubernetes was doing exactly what we told it to. But since, you know, after the deploy, our test said, Oh, everything's fine. We want to head and then roll that out to our internal production environment. And, you know, so we had like a lack of availability during that deploy during that time. Right. So if we had the ability to say, Hey, like during the deploy, there's a lack of availability, then stop it. So I hope that answers the question, like super happy to talk more about it, but Oh, thank you for your presentation and answers. I understand that deployment ordering is critical for stateful applications and its components. You mentioned custom controllers for your application is the right path forward to ensure dependency and ordering for application components. Can you elaborate on that? Is a, is it a custom operator you have written that runs in the cluster or are you, or do you have Argo workflow taking care of the ordering? Like, so in the case that I'm mentioning to get to that super smooth deploy case, it's absolutely a custom controller that we wrote that's running in the cluster for us. We have a set of controllers actually that we wrote, but that are at different levels. But yeah, we just found we just had to get very fine grained control over the different parts. And the only way that like there's just no off shell off the shelf solution that actually worked up to the level of requirements that we had.  So yeah, absolutely. We had a right custom code and we continued right. Okay, well that just about wraps up all the time that we have for today. I want to thank Rick for a wonderful presentation. I said before the presentation and slides will be available later today on the CNCF webinar page at CNCF.io slash webinars. Thank you all for attending today's webinar take care stay safe and we will see you at the next CNCF webinar. Bye everybody.


How do the break-even costs associated with shale energy development affect its overall development? The overall economics of a shale energy project, whether it's an oil or gas well, are going to depend really on two pieces. You have those costs to drill, complete, and produce from the well, and a lot of those costs are fixed. And then you have the other side, which is the revenues that you're going to earn from selling oil or gas. The costs can be fairly substantial. A single unconventional oil or gas well is going to involve millions of dollars of upfront capital investment. But the exact cost is going to vary a lot by location because there are going to be differences in geology, in the depth of the geologic formation, in the topography of where you're drilling, in the distance between the location of the well and the market sale point. All of these things are going to influence the costs of drilling an oil or gas well. And though you're generally going to be talking in the many millions of dollars. On the revenue side, the payback for those costs incurred is ultimately going to depend on what happens in commodity markets for oil and gas. And on the natural gas side, those commodity markets are going to be influenced by things that are generally happening in North America. So things like weather, demand for electricity, things that are going to affect overall natural gas demand. North America is at this point a relatively closed market for natural gas. There aren't a lot of imports and exports. And so price fluctuations in North American natural gas markets tend to reflect things that are happening in the U.S. and Canada and Mexico. Oil on the other hand, oil markets are a totally different story because oil is really a global commodity. And so if you're drilling an unconventional oil well, your revenues from that well are going to be influenced by economic and fundamental and political things happening anywhere in the globe. And because oil is such an internationally traded commodity, if there is an increase in demand or an interruption in supply halfway across the world, that is going to be felt by an unconventional oil producer in the Bakken or the Permian or some very local place in North America.


So welcome back, I think people are still starting to join us and we will kick off in a minute or two. Okay, well I'm delighted to kick off the next session, so welcome back everyone, and we're going to be exploring the whole issue of achieving inclusive education using AI. I think inclusivity and inequalities are certainly one of the issues that the ethical framework really does try to address. I'm delighted that we've got Otterlandu Duruwodu joining us today to speak to this and Tunday is, he's a reader in education management and he's the Associate Dean for Diversity, Inclusion, the Faculty of Businesses and Law and Tunday is also a very active member of the old SIG, Anti-Racism SIG, so a very warm welcome Tunday, delighted to have you with us today and really looking forward to seeing how you're going to cover this really, really important topic of inclusivity and AI, so welcome. Thank you, thank you very much Natalie, thank you. Thank you very much everyone for joining us today. I think as Natalie said, yes I'm from Liverpool John Moores University and I'm the Associate Dean for Diversity and Inclusion and part of that is looking at how we ensure that us as a faculty and all LGME as an institution is very, very inclusive in what we do and AI presents a unique opportunity and we've been sort of looking at how to explore using AI as a means for achieving inclusivity. So what I'll do now is very quickly just share my presentation and then we can kick off, so just bear with me a second, I hope you can see the presentation, I'm hoping you can see the presentation. Thank you, we can see that perfectly. Yeah, oh brilliant, thank you, alright so my topic today is around achieving inclusive education using artificial intelligence and I used a statement, a disadvantage, where at doubt because I believe that when we talk about inclusion, one of the things that we need to address are the disadvantage risks that is involved and if you look at the picture in that PowerPoint slide, it's a picture of a young girl who's competing in the vertical jump competition and what you see first is that young lady's been sort of measured for their height and their reach and then once they've been measured, they use that height and reach as the baseline for measuring the actual jump performance. So the lady then jumps and then the actual jump height is then measured. Now what they've tried to do in this picture essentially is to identify what they consider to be a disadvantage risk and that is height and reach of the competitor because they believe that there are different competitors with different heights and different reach. Therefore if they are actually interested in measuring the jump performance of the competitor, then it's important to eliminate those disadvantage risks. And the same analogy is what I will sort of use when we're talking about inclusion within education. So what are those disadvantage risks and how do we level off those disadvantage risks in order to ensure that what we are actually performing, what we're actually measuring is the actual performance of the students. So there isn't any disadvantage and we ensure that there's equity and equality in what we are measuring because what we find right now in most high education institutions is what we do right now is we measure disadvantaged performance rather than just measuring the actual and the true performance of students. So we need to be able to address this issue. So the problem is how do we then you utilize AI because AI is an emerging tool that has been used in different fields to address what they consider to be important challenges. So in the medical field for example is being used to address you know in medical diagnosis it's been used in the physics world has been used in the creative art industry again to sort of enhance a creativity of art artists. I mean physics world is being used to solve difficult and hard problems. Now there's a perspective that AI is an important tool to address those important challenges. The question then is what do we see AI as a tool for addressing what sort of problems within high education. So AI really potentially can be a very useful levelling up tool but at the moment it seems that we're still not looking at it from that sort of perspective. We're still not viewing AI as a tool to address a specific unique challenge for us. But what I'm talking about today really is around how we use AI to address the unique challenge of inclusion and of course you just need to read literature and you need to just visit some of those discussion spaces and you would very quickly realise that yes we've got an inclusion challenge within high education and some symptoms of that challenge are things like a warding gap where there is a huge difference between the proportion of white students who achieve 2-1 and first class compared to non-white students who achieve the same degree outcome. There's a huge disparity in there and not just that another symptom is in what we consider to be the student experience for example the experience with regards to assessment or even with in terms of their employment outcomes. So there are gaps in existence. The question is how do we ask the university or as high education sector or even further education sector how do we address this issue. So inclusion is an important challenge and we need to be addressing that. Now currently within high education and further education what we currently see is that we've got different applications of AI. We've got chatbots, it's been used as chatbots, it's been used within assessment grading systems to ease the sort of marking load of academics and tutors. It's also been used as an adaptive learning systems. It's also been used in the way of analyzing huge volumes of data. But of course we currently have two classical problems with the current AI. We've got a design off problem and we also have a design for problem. So with regards to the design off challenge these are some of the things that we already know. We know that we always say we're careful. We tell our students to be careful when they're using generative AI because there are repeating biases in the output of those generative AI and that's because generative AI is replicating the bias because the AI algorithm is being trained by data sets that were generated and in those data sets we know that there are biases in them and if that is not addressed what AI will then do will be to replicate the existing biases and of course operators within this space within this market are looking at ways to address that repeating bias issue. There are of course ethical concerns and again you just have to listen to the news and you'll see what's happening with regards to AI regulation and AI ethics. Bloodlines of course between fact and fiction because as we as what itself entails generative AI means it comes through the interneting comes through the data set and then compiles what it believes to be the best answer to the question that you post to it and we know that it gets all sorts of data from all sorts of sources and you can't really tell whether what is presented is actual is actually fact or is actually fiction. Also we also have what we call the weird lens problem where most of the output you get from generative AI is based on weird lenses and weird of course is the acronym used for Western educated industrialized rich and democratized societies and if you if we use charge GPT as an example charge GPT-3 as of 2020 had I think around the 92% of all the data that is used in training GPT-3 92% of that came from English and the next language to that was French and that was around 1.8% and you can see a huge disparity so we see that it's very very English focused and if that's the case then where is that diversity of knowledge that we we hope to get with AI. So these are some of the design of challenges that we know currently exist and of course we know that there are still transparency issues you know where is the data coming from and how are they you know using those data you know hopefully not in a malicious way and hopefully I think if you listen to the news just a few days ago the EU eventually came up with with the regulation for AI and hopefully maybe that will start to get people get people to be a bit more sort of open and willing to use AI because the adoption rate worldwide right now for many businesses especially is around 50% and of course you can also look at the breakdown of you know by by country the willingness to adopt AI and also whether they trust AI or not and you can see the different spectrum where different countries exist within that within that line and you can see that for example the UK at the moment I think the willingness to accept AI still stands around the something percent and of course we still have a long way to go. So if we look at some of those design off challenges of course it's something that we need to be mindful of when we're using AI. We also have what I call the design for challenges okay and this really looks at what we're using AI for within higher education for for instance for example when we when we look at where we apply AI we mostly design it for the majority okay and if we look at the UK context that would be the white majority okay and it's important for us to understand it also in in relation to the minority part of of the student body. Also when we're never we're purposeful about how we use AI it's usually for a very limited number of protected characteristics and the one that we typically use it for is around disability and also neurodiversity and that's that's a good thing all right however we also need to be mindful that they are all that protected characteristics that also warrant our you know purposeful use of AI and that's some of the things that I would like to sort of focus on today. So often we ignore intersectionality we know that an individual all right might have several disadvantage risks which we need to be mindful of but the problem we currently have is that many of the interventions that we have always has that sort of single focus so it's high time we started to think about the intersectionality of these disadvantages and we'll touch on that a bit later on. Again there's still that perception within high education that you know inclusion means you know treating everybody equally and unfortunately it's still an issue because I spoke with some people just last year and you know some of the things that were coming out of that of those conversations was you know people still think that you know I treat everyone equally you know I can't you know I can't be seen to be doing anything extra now of course the problem is not the issue is not you know I'm doing anything extra the issue is being purposeful thinking about what are the needs of the various student demography within your within your institution and addressing those specific needs that's the key issue here all right and then the last problem that I think or the last challenge that I think we have is what we call is what I call the the academic limitations or the institutional limitations and what do I mean by that I mean that we've got time limitations as academics or as tutors or as educators and we have workload and limitations we've got subjectivity limitations whether you like it or not we are all very subjective in the way that we deal with things and sometimes those subjectivity might not be apparent to us so we've got things around unconscious bias as well. We also have issues around flexibility and we also have issues around lack of creativity and those are some of the issues that we currently face and that doesn't give us the space and the capacity to think about inclusion in a way that we can be a bit more purposeful and a bit more effective in how we deal with the issue because if you've got so much on your on your plate and you haven't got enough time and you you're not allowed that flexibility that is required in order to address inclusion issues then then that issue would remain an issue and it wouldn't be addressed and that is why it's important for us to think about how we lighten the loads in this particular area and AI presents a unique tool that can help us as academics or as tutors or as educators to reduce the burden of time the burden of workload by improving our workflow and also improving our productivity and hopefully with that we would have enough space and enough capacity to be able to be more intentional about how we use our roles to achieve inclusion all right in in our education so as an institution where are we okay can we actually achieve inclusion using AI so where are we as as an institution unfortunately I think we're still not where we ought to be and and this is a provocation and that is most of the policy around AI really is geared towards you know handling students use of AI for example chat GPT so it's either we've got a policy in place that says oh no you can't use generative AI at all or ones that says that oh yeah you can use it however you need to reference the fact that you have used it so there are different interactions with AI again it depends on the specific university but unfortunately again most institutions within the UK and of course around the world still have not issued any policy whatsoever on the use of generative AI and that needs to to improve the second status is around our current view of academics of tutors and of educators we've got this omniscient view of them where we expect them to deal with new technologies as they emerge to train themselves and to address the problem and because yes we do rely on educators but we tend not to think about the tool the challenges that educators are facing for example inclusion that we're talking about it sometimes you've got certain academics that are very active when it comes to inclusion but when you ask others you know it's not an issue for them they don't even think about inclusion all right we we had an interview on staff's perception of them of inclusion just a couple of years ago and some of the things that came out of that interview was that you know some staff don't even think about inclusion when they're designing their curriculum and in this day and age it's important for us to to think about what we're doing who we are teaching we need to know who our students are all right and then hopefully we can be a bit more sort of effective in that way therefore as as as institutions what we currently do is right now we're still very hesitant with regards to AI and that's that's that's understandable but we need to we need to do a bit more and we also need to help develop the capacity of the academic to be more purposeful in with regards to how they use AI and not just how they use AI but how they use AI for for inclusion purposes so as an institution we need to recognize that we've got limitations all right and we need to be able to overcome this limitation so AI can help us address some of those limitations that we that I talked about workload and productivity issues we also need to understand that it takes time and effort for an academic or an educator to be able to get to the point where they are sort of confident enough to be able to utilize AI all right in the learning spaces and of course institutions for example like UNESCO are coming up with framework that institutions can use to help develop that agency within educators and I think it's very important to sort of read up on that and follow through on that we also need to develop a multi purpose in other words we need to use AI that can address several challenges at once and then we also need to use a sort of multi narrow AI strategy so it's not just using for example chat GPT within your institution it's about how you use that collection of AI applications for example chat boards adaptive learning how you sort of use them in a sort of effective and sort of combined way and that's where we need to be as as an institution so if I also ask everyone here today that what is your AI use plan what will be your response will it be that okay right now we're just talking about policies on you know on how we help students to sort of them use generative AI or will I hear something a bit more purposeful a bit more sort of further down the line than that and saying that okay we we've adopted specific technologies using AI and we're training our staff in the use of this AI and so that they can hopefully pass that knowledge onto students and also help students to use AI in a safe and in a very engaging way so that's a question that we need to to address now if you look at the picture on the on the right hand side there and in order to develop an institutional AI strategy there are three things that I think are completely necessary the first one is we need to adopt a utility view of AI in other words AI as a tool okay how can we use AI to address some of those challenges that we've got secondly is we need to recognize the agency costs okay because we've got this reliance on academics but what can we do to help academics to be able to get to the place where we need them to be in order to help address or to further the AI and inclusion agenda within within our institution so do we need to have training in place do we need to pay for training all right and if they're going to go for training do we need to include factor that into their workload allocation for example do we need to make space for them and time for them to be able to to do that so those are some of the things that we need to sort of think about and when we're talking about AI there has to be a coordinate a coordinated view so it's not just about using just one type application or one using AI to solve one task it needs to be solving multiple tasks for us for example we use chat bot for example in engaging students we use generative AI for example in the learning spaces we use adaptive learning for example when students can sort of go away go home and then hopefully catch up with maybe the lesson of the day or help them to learn at their own pace so those are some of the things that we need to think about and it has to be coordinated and that's very very important so how do you develop an AI use plan first thing is you need to understand your context because when we're talking about AI for inclusion remember you are addressing the needs of people all right of humans and you need to know what those needs are so understand your context and then secondly would be to respond using AI so it's a matter of thinking how AI can help you to address some of these issues and that's very very important so if we look at understanding your context what that means essentially is you need to be able to collect and analyze the data that you've got okay so think about the demographic data that you've got so things around gender ethnicity disability religion sexual orientation area of deprivation and even those that don't come under what we call protected characteristics and if you look at OFS for example they've got they come up with this equality of opportunity risk register that can also be useful so I would say that it's important to sort of engage with those registers so understand how other forms of deprivation you know can be factored into what we are addressing as an institution but more importantly we need to understand the intersectionality of these disadvantage risks okay so understand that and then secondly look at your performance data so what we did also at the LGMU was that we looked at the performance of the various student characteristics and we also did a survey trying to understand the experiences of these student characteristics just to get a better sense of some of the challenges that they're facing and in order to then think about this the bespoke solution in order to sort of address some of those issues so it's very very important I would say this is one of the most important aspects of achieving inclusive education is to know where your students are coming from and understand what sort of disadvantage risks they've got going on and then hopefully once you've done that then you can then be a bit more proportional about how you use AI to respond to those operational issues for example how do you use AI in curriculum development how do you use AI in assessment delivery or assessment strategy how do you use AI in outside hours engagement and how do you use it in a sort of adaptive learning format and of course lastly you need to think about intersectionality there's a TASO report that came out recently looking at reviewing two different interventions around decolonizing the curriculum and they found limited evidence that the decolonization the curriculum interventions didn't have much success in addressing the awarding gap that I spoke about initially now there could be several reasons for that but that again it's a pointer towards trying to address a specific disadvantage risk and then and then not thinking about the intersectionality of all of this this advantage risk and how that impacts on the awarding gap and that is why being purposeful and intentional is very very important so what I'll do very quickly now is just go through each of this and talk about give you just an example of how you can achieve inclusion using the curriculum development and the other and boxes that we've got here again these are just examples the point here is for you to think about how you use it in all of this area all right in a very targeted way so for example in curriculum development you can use it to sort of pluralize perspectives on a particular topic for example if I was to I teach supply chain management for example and I also teach management theory so if I say okay I can tell my student to say to use generative AI to come up with you know five different aspects of management theory they do that and then I can then say all right how do we pluralize our knowledge of this topic I can then say all right use specific prompt okay that asks the generative AI to look at the perspectives from maybe for example an African point of view so when you start to do that you start to get concept around things like Ubuntu you know togetherness and things like that from that sort of African and cultural perspective and that just enriches the knowledge of our students okay so that's one way another example would be maybe to ask it to generate a reading list I remember the previous though in the previous session they talked about the use of a library and it's very very important when you ask students to do this it's important for you to ask them to verify and that's one additional thing that they can do you know to improve their learning to say this has been generated by generative AI are these lists actually in existence and they need to be able to go and then verify those those lists and then you can hopefully use those lists so that's just two simple examples of how you can use AI in curriculum development in an inclusive way the second one is around assessment delivery so formative assessment for example or low stake marking and using it as a sort of feedback mechanism for students and what you will find is that this also is very useful for example for international students who are not very familiar with for example some of the criteria that we've got or some of the language that we use in in the assessment brief that we give them so this hopefully would help them to learn more and familiarize themselves with those criteria so an example of this would be you could encourage the students to use the GAI to grade their own submission so you've given them a task grade that submission using the set of criteria that you're using in your module and hopefully once you've done that then they can then start to improve on the work because the GAI can then give them specific feedback and they can then use that feedback to sort of improve their work okay or you can get students to improve the quality of their response through prompt engineering okay and then you can then see how their responses get improved over time okay so that again can be very targeted and it can be very informative and educative in that way and then assessment for learning again you can provide students with a set of marking criteria and get them to mark sample responses so this are different ways of operationalizing that and get them to mark sample responses that you've marked already and you know what the answers are and then they can then interrogate you know the difference between the marks that you assigned them and the marks that and check GPT and hopefully through that they can then see where those improvements need to be made also you can also use it in outside hours engagement one thing we learned from the survey that we um from the survey that we did was that students typically when they've got issues with within their modules they typically don't go to the module leader they go to their peers and some actually also go to their family members in order to address some of those issues so what we're saying is in a very inclusive way and the reason why they don't typically go to many of those academic tutors it could be for fear that if they went and asked the question the tutor might think that they are stupid okay and that might leave an impression on them so therefore they avoid going all right it could just be again the language that we use we always say oh yes you are at the university or maybe you're a master's student now you need to be you know you need to be able to do things on your own and independently so again that also makes them a bit apprehensive about approaching their academic tutors to be able to answer the questions that they have so one way of taking that piece of information to them might be you know through chatbots all right and it could be as simple as just converting some of the FAQs to a conversational AI style and product and it can address things around assessment information it can also help you if you use it in a targeted way and to be a bit more transparent about your assessment criteria and what you're grading and help you to remove the hidden curriculum that we know exists in in in many assessments also it can also help the students to know more about the module all right module information also when we're talking about induction and I think this is a very big area because when students come to us we always give them a lot and lots of information so if we have a chatbot that can help them and to understand and digest the information we're passing to them I think that will be very very helpful it can also be used as a student self-evaluation in terms of writing so for example things like am i right or Grammarly and we know that language is a huge disadvantage risk for many students you can also use it in adaptive learning and I've put examples there for example Cogbook, McGraw-Hill, Connects, Khan Academic etc. Lastly think intersectionality so how do you use for example Chatbots, Genitive AI in a way that addresses multiple disadvantages or disadvantage risks for example example of those disadvantage risks will be English language proficiency so that could be an issue that affects for example the neurodiver students or international students or students whose first language is not English again it can help you can also look at how you address for example cultural gap and you can look at things around anxiety for example for example when you we know that students have higher level of anxiety when they attempt exam types of assessments so how do we use AI to move away from exams also in terms of diminished capacity for example we have students who are caregivers so which means that rather than have five days in a week for studies they've got diminished number of days compared to those who don't have care responsibilities so how do we help bridge those gaps so those are some of the disadvantages that we need to think about when we are thinking about using AI within our learning spaces so where do you go from here think about an intersection you want to prioritize so we're not just talking about one disadvantage risk we're talking about multiple disadvantage risks and how you can use AI to address all those disadvantage risks consider the challenges that we've been talking about and the limitations of the use of AI and that's very very also that's also very very important and of course we need to develop that instructor-student partnership approach prepare yourself engage with your institution diversity and inclusion plan you know participate in reciprocal mentoring trying to understand things around race and also around you know diversity utilize research that has already been been undertaken by different institutions there are several toolkits in existence trying utilize them and see how can I use that and marry that with my use of AI within the learning space and of course get involved with various staff networks and one that I would recommend is the anti-racism in learning technology and special interest group off of ALT and how can you get involved in that you can be involved as an ordinary member or you can report officer positions available at the moment or you can just show interest and to join as an officer and then we'll keep you in mind going forward or you can join us as a speaker we're always looking for speakers around around this so it's a space for a real change and to make a real impact so these are some of the things that hopefully once we prioritise them we can then start to look at achieving inclusive education so I'll stop sharing for now I think that's the end of my presentation um how how are we doing yeah that's great thank you so much Tindall that was that was a bit of a tour de force I think really really important points but also some really helpful practical examples so many thanks for that so we'll go into questions um and the first one is from Maria Walker who's saying can we trust AI not to pluralise perspectives from a stereotypical point of view that she's asked it to write a case study about a fictional African company and it had no depth there was no real representation in the case study so is is there almost you know something we've got to be aware of that we could ask using it in that way but actually there could be drawbacks at the same time yes yeah yeah absolutely right because um when you when you ask a genitive AI to sort of answer some of those questions sometimes you have to be we said you have to be careful because sometimes you you can't separate facts from fiction but what we are hoping will then happen with this student is that it then you know puts that sort of investigative mindset in into play and then they then go and then research some of those information that was sort of turned out by the genitive AI and that's that sort of extra step that you need to to take now what AI can do is that it can bring up different concepts but the student then need to go and then think about researching a bit more around that and the idea really is just to make sure that we're not just teaching them concept from one point of view it's about pluralizing you know the various perspective from that so there's no one perspective that is right or universal all right so it's getting the students to think about multiple perspectives sorry I can't hear you apologies sorry I muted because there's a bit of piss on the line yeah I was just saying yeah and that brings in all the critical thinking that we've been talking about in the earlier session so it's an opportunity then to reinforce developing that AI literacy isn't it yes there's a question there from Chris Rao can AI be anti-racist can AI be anti-racist I think that's where the design for comes into play I mean the design of challenge comes into play so if we are a bit more intentional about how we design AI we talked about the output coming from from this sort of weird weird that's an acronym weird lenses and when we think about those that are involved in the design of AI how diverse is that team all right so when we start to think about how we design AI in the first place and who's involved in the design of AI then we can start to use that sort of anti-racist lens in developing in developing it and I think that's really crucial unfortunately I think I might be putting my my foot into it now is when I think about some of these talks around ethics and I know that's very important and usually inclusion and anti-racism tend to be sort of subsumed under that for me I think there has to be a separation there you have to separate because anti-racism is an important component of achieving inclusion and I think I think a lot of people don't get that but it's important for us to be able to look at it from that perspective and and say that all right where are those inputs coming from how diverse are those inputs remember I mentioned that for example chat gpt and I'm not singling out chat gpt is this is just the only example that I could find 92% of the data set is coming from English 1.8% is from French 1.4% is from German the next language is 0.7% and you can then see all the other languages are less than 0% less than 1% so so if that's the case then it means that you know whatever the AI generates will be basically from from that perspective so where is that diversity of knowledge and I think that's very important so to address that question I think yes AI can be anti-racist but it has to be intentional question is I don't know whether it is at the moment yeah and you've got a couple of minutes left um Tindall I'm going to ask one question I want to ask you a final question if I can but I'll go with this one first um and it's essentially how do we reconcile the issues of AI and representation and its use it can feel like we are being complicit or supporting that would you say the key is to point out its shortcomings rather than boycotting it yes I think um and that's why at the moment we have to use AI very carefully and we need to it's important to understand the challenges that comes with AI or the challenges that AI itself represents and we've talked about that already but it's not enough to then say all right we then dismissed the use of AI completely um there are several surveys that have been conducted um and you can see that um it's been said that AI will become one of the competitive factors for many institutions in the next 30 years all right we can then say okay there are too many problems we close our eyes and we say no we're not getting involved but what we what what will tend to happen is then we then we get left behind and I think it's very unfair on students because AI represents a very very important opportunity for us to level up all right and if we're not sort of using that and thinking about how we can use that despite all the limitations that is got um I think that would be a significant missed opportunity and I don't think we as institutions should be doing that and a final reflection I noted that you know during COVID that some of those inequalities were addressed without pivots online and the blended learning approach and we saw attainment gaps narrow and we saw BAME groups particularly do a lot better and also disabled students and I remember during that time we did quite a detailed an EDI quality impact assessment and I'm just wondering whether is that something that you'd recommend that we should be doing with our communities maybe with that approach that Mary outlined different different you know professional services staff academics students together to really work through that so that we can begin to take that responsible approach to making sure that we adopt AI in an inclusive way yeah absolutely absolutely and I think I think the COVID period was a very good case or a case study for for for this because when COVID hit what we what we did basically we became a bit more flexible and so rather than having in-person exams we started having you know different formats for our exams the question is you know we changed the formats for exams and I know we're coming back because of AI as well we're thinking okay how do we prevent students from cheating but what COVID forced us to think about was how do we do things differently how do we accommodate many of these challenges that our students are facing but what we then find is that post-COVID you know some people want to go back to the way things that I've done and especially when AI sort of creativity hit the market and many of the responses were that oh no we had to go back to exams you know and you just have to look at the historical performance of students on exams you will see that as a clear gap okay based on ethnicity based on on on race and we can't go back to that so it's very important for us as institutions as colleagues to sort of come together and think all right how can we use AI just as we did with regards to COVID how can we use it you know in a very effective way to address some of the challenges that our students are known to face so it's important to know what those disadvantages are and how we can use AI to remedy them or level them off. Thank you again that's been a really insightful talk as people saying in the chat really appreciating hearing your perspective and the practical examples you've given so thank you very much and that is the end of the morning session we reconvene at 1350 and maybe just wrap up by saying do think about the old ethical framework we're really looking for case studies and I'm sure that the anti-racism learning technology SIG is also similarly looking for case studies you know and there is an old award around ethical practice so think about this as you're developing your approaches and your institutions and the kind of case studies that you could be sharing with the community with the broader community because we're all in this together we're all facing the same challenges we've all got that challenge of time and space we need our collective thinking don't we in collective perspectives so I'll leave that as a final thought but thank you again Tundi also to Helen and Mary our other speakers this morning and look forward to seeing you later on thank you thank you thank you for having me