I think you can hear me. Yeah. So good morning, everyone. My name is Kashyap Chamruti and I work as part of Rahats Cloud Engineering team. And I spend time focusing on integrating KVM-based virtualization components into OpenStack Nova. So I split my time between multiple communities. So what's the motivation for this talk? As most of us here now, since the beginning of this year, there have been a flurry of disclosures involving site channel attacks. And the disclosures kept coming in all the way until earlier this month. The most recent one being Portsmash, all the fancy names. So that impacts hyperthreading, processors with hyperthreading. And this is made choosing CPU models for your virtual machines a bit more cumbersome. So this talk aims to provide some sense of clarity, hopefully, about how do you go about arriving at an optimal virtual CPU configuration. So before we proceed, a quick note on what we won't cover in this talk. So we won't be touching any of the internals about site channel attacks. Meltdown and Spectre, or how to exploit them, or any of the detailed performance implications involving these. So for those, there have been several talks throughout this year, beginning from Fastem and kernel recipes, and last month in Edinburgh at KVM Forum and Open Source Summit. So I refer you to check out those talks if you're more interested in the gory stuff there. OK, a bit of a refresher. Most of you here might be aware of this, but for those who are not, a bit of a refresher on the KVM-based virtualization stack. So the bottom most layer is the Linux kernel with the KVM module sitting in there. And top of it is the QMI process, which sits along with your other processes on the host. And QMI has its disk images associated with it, and it interacts with the KVM via system calls. However, QMI scope is quite limited. So it has a per process view, only it knows about just one QMI instance. So that's where the LibWord project comes in, that provides a hypervisor agnostic API to manage multiple virtual machines and the lifecycle associated with that. And LibWord interacts with QMI via a protocol called QMP. It's a QMI monitor protocol.  And top of it is OpenStack Nova that has its own virtualization drivers, KVM, Zen, several of them. So it interacts with LibWordD and then launches virtual machines on your compute nodes. And then there's external tools like LibGest Office, which provide a set of neat utilities to be able to examine your disk images via LibGest Office's its own custom uplands. It's just a QMI process. So I should mention here that LibGest Office has some safety controls in place so that no two processes can write to the same disk image. OK, when you launch your Nova instance on compute nodes, most of the heavy lifting under the hood is done by QMI and KVM, assuming you're using word type as KVM or QMI in your Nova configuration. So QMI provides a lot of emulated devices. Some say too many of them. And it also has the guest memory M mapped into it. And it provides several kinds of SCSI controllers, network cards, graphic displays, and so on and so forth. KVM, the kernel module, it has two vendor-specific modules, Intel and AMD.  For example, one is to it safely executes the guest code directly on your hardware through the hardware extensions provided by processors from Intel or AMD. They all have virtualization extensions. They work similarly. So KVM handles the switching between guest and the host mode. And KVM itself does some emulation of devices in the kernel. Things like clock or a CPU ID instruction so that it doesn't have to do a heavy weight exit all the way to QMU. We'll see what that is in a second. So since QMU is just another process on your host, you can use your standard Linux tools like PS or task set, kill, et cetera, to manage or examine, inspect your QMU and novel-based QMU instances. So this is the classic guest execution loop involving KVM. So on your right mouse, you see QMU. It sets up self-system calls asking QMU to a KVM to create a vCPU, get the file descriptor back, and then it runs the iArch tool called KVM run. At that time, KVM will prepare to execute guest code directly on the hardware via so-called guest mode that is introduced by the virtualization extensions from Intel or AMD. And at that point, guest code is executing happily on the hardware until a point where the hardware cannot handle a certain device emulation. At that point, it asks, hey, can you please handle that for me, KVM or kernel? If KVM can handle that, for example, the CPU ID emulation, it will emulate in kernel and then, again, starts to execute guest code, prepares to execute guest code directly on the hardware. And that continues. However, if KVM cannot execute or emulate a particular device, then it asks QMU to handle that. And the loop continues. So QMU handles it, and the loop goes on. So going all the way to QMU is called the heavy weight exit. And if kernel itself can handle that, it's so-called lightweight exit. So that's the introduction. So let's see some interfaces that QMU and Libre provide to configure virtual machines. But before we go there, a bit of motivation why we should see them. So the default guest CPU models provided by Libre, actually not Libre, it's QMU, they're designed to work on any host CPU, but they're not really the ideal choice in your production environment. When I say work on any host CPU, that means that you don't have to do any compatibility checks that it will work on for live migration, things like that. But the default CPU models are really awful, because they lack several CPU instructions that are critical for initializing entropy for your guests, for example. Or more obscure flags like PCID that have become critical for performance and security, thanks to Meldown. Or the AES instruction set that is critical to have your TLS to be very performant. So thankfully, Nova doesn't use these things, because before we go there, before we see what Nova does. And the output you see there is from a virtual machine that is launched with the default QMU64 CPU model. And then if I traverse to the CISFS directory of dash CPU vulnerabilities there, if I graph for what all mitigations have in place, you still see that it's vulnerable to a variant of Spectre. So the point being that you don't want to use default CPU models absolutely at all if you care about little on performance and security. So always try to use an explicit CPU model if you know your CPUs, or use the default provided by Nova, which is the host model, and which we'll see in a bit what that exactly is. So what about the defaults of other architectures? We saw a default of x86. So on the AX64 ARM ecosystem, they don't, sensibly enough, provide a default CPU. But the default depends on what kind of machine type you configure. So think of machine type as a virtual chipset that provides certain devices. We'll get to the topic of machine type briefly in a later section. But for now, think of it as a virtual chipset. So there's different kinds of boards that ARM emulates. So depending on that, the default CPU is configured. And other architectures like S29DX and PPC have their own defaults based on whether you're using pure software emulation or hardware accelerated drivers like KVM. OK, the most simple interface for configuring these CPUs is the command line. And on QMU, if you don't do anything, if you don't provide a CPU option, then the default is QMU64. As we've seen, we don't want that. So we want to provide an explicit CPU model for that. The simple thing is to provide the CPU command line. And followed by that, a particular named model like IVBridge, Haswell, whatever, whatever, that are supported by the QMU binary on your host. So along with providing named specific CPU models, you can also control what CPU features you can expose to your virtual machine. So in this variant, you see the Skylake variant of a CPU, in this example. And followed by that, a specific set of features we are turning on and off. So if you want to find out what are the supported models on your host, you can refer to QMU's Help or Libert's command line tool Varsh CPU models. And followed by the architecture that you're running. So what about runtime interfaces? So QMU provides several runtime interfaces that Libert uses at its Daemon launch time so that it can cache those capabilities because the capabilities won't change for a given QMU version. So it makes sense to cache them at the start of Daemon start-up time. And then you don't have to do that again and again.  We won't delve into the details. That's out of scope. But as a quick example, here is one runtime interface to probe QMU for the details about CPU models that a given QMU binary on your host offers. So here's query CPU definitions. It provides all the models that QMU supports. And it thinks like, what are the features that are unavailable? Meaning, here you see an empty array for the unavailable features. What does that mean? It means the Vesemir CPU model will run without any modifications to it, meaning you don't have to disable or enable any features to run on a given host. However, if you see a list of features enumerated in the unavailable features section, then you have to disable them explicitly to be able to run that model on the host. And it also provides other information like whether the CPU model is migration safe. It means that QMU won't add additional CPU features or things like that so that it will break migration compatible. So in this part, let's see, what are the different kinds of CPU configuration modes? And how do you go about configuring specific CPU models and set of features? So some of you here may already know this. And I've talked to some operators who use these things. First, the host pass-through. As the name implies, it just passes through the host CPU's capabilities, CPU ID bits, actually, QMU, that's what it does behind the scenes, as it is to the guest. And that's provided by CPU-host command line on QMU. But that's taken care of for you when we use Nova's host pass-through config attribute. So this is just to show what's going on behind the scenes. However, the host pass-through is not really free of caveats. So the first thing is that neither QMU nor Libret will provide a predictable CPU for your virtual machine, for your guest. So you can't rely on having a predictable CPU there. The other thing is, if you have a mixed set of CPUs in your environment, then live migration with host pass-through is an absolute no-go. So host pass-through is quite performance. But if you have live migration as a strict requirement in your environment, like most scenarios, then it's not really recommended, assuming if you have mixed set of CPUs there. And when else can you use host pass-through? So if you're lucky enough to have a data center full of identical CPUs, then it makes sense to use host pass-through. But also, bear in mind that, along with identical CPUs, you need precise microcode, matching microcode version on your host, your source and destination host to be able to live migrate, and also identical kernel versions on your host, thanks to all the vulnerabilities that have been rolling in from the beginning of the year. So it's easy to miss that. Sometimes people may wonder, why do you even have to have microcode version matching precisely? Because you may think, if there is a microcode version 57 on source and 58 on destination host, if both of them provide identical CPU features, then it should be OK, right? No, because performance counters may differ between the versions of these two microcodes. So even though the CPU features are same, so even that can impact live migrate ability. So that's the thing that I learned from a colleague about when I was wondering, is microcode, should microcode version match as well? So there's that. See, the other kind of thing that QMU and Libre offer is the plain name-to-CPE models that we just saw, which are just vendor-specific models that are named after particular generations that Intel and AMD and others release.  The names-CPE models are a bit more flexible in live migration than hostmaster, because you can custom prepare a CPU model and the flags in your environment so that you can have live migration compatibility across your disparate set of hosts. QMU comes in with a set of built-in default CPU models and custom CPU models, specific named versions, and the set of CPU ID flags that it recognizes. And Libvert also exposes these via its own XML representation. So you can refer to those. And the third way is this thing called host model that is a Libvert abstraction. So it tackles a few problems. One is that it tries to provide maximum set of CPU features from the host CPU to your guest, and while retaining live migration capability. But it has some caveats. We'll see what the caveat is in a minute. And the third thing that host model provides is that assuming you have the right set of updates on your host, the microcode, kernel, et cetera, and so forth, it will add the critical CPU flags to mitigate your guests from vulnerabilities, some of the maildown spectra variant, like the spec CTRL flag is the example there, automatically when using host model. So that's a nice thing that it does. So host model tries to provide the best of both the host pass through and the named CPU models that we saw. And for better or worse, I think it's for the better. It's the default of Nova, at least Nova's Libvert driver. When I say default of Nova, I'm referring to Libvert KVM QM drivers. That's a simple example that you see from a Nova guest definition. How do you enable host model? You just mentioned the CPU mode as host model. And optionally, you can enable or disable a specific set of features. And Libvert will translate that into a certain suitable CPU model based on its own internal representation in XML. So what about the caveat that I mentioned about host model? So with line migration, when you're using host model and trying to do line migration, what happens there is the source guest CPU definition is transferred as it is to the target host. And once the guest is migrated, it will see the identical CPU that it saw on the source host, regardless of the capabilities that the target has, even though they are more capable. But when the guest call reboots, meaning you do an explicit stop followed by a start, it can pick up extra CPU features. Because if the processor is a newer version on the target host, then host model will pick up those features. Because recall that host model tries to provide maximum set of features from the host. So that will prevent you from migrating back to the original source host. So if line migration in both directions is an absolute requirement for you, then host model may not be the best option. So if that's not a requirement, then you can happily go with the host model. What about Nova? Nova provides configuration interfaces to drive all these things that we were talking about. So it provides a set of configuration attributes for the Libvert driver. So one is the CPU mode. It's straightforward. It can be any of the three modes. Host pass-through, the first one we talked about, or custom, meaning the name of the CPU models, or the host model, the Libvert invention that provides the best of both host model and provides the best of host pass-through and name CPU models. And you can explicitly provide CPU models via the CPU model config attribute and the extra features with the other one on the config attribute. So what are the possible values for that? You can refer to the help outputs of different commands out there, cumming, or the CPU map, XML file. And we've also written some documentation for Nova itself. So Nova has recommendations as to what you should use and which scenario. Not really comprehensive, but good enough to get started. So I refer to that for the Rocky release. I think that's where we introduced that. So this is a simple example from a compute node. So you just set these config attributes in the Libvert section of your node.conf saying, in this example, we're just seeing a custom named the CPU model. Hey, give me a fixed version of Ivy Bridge CPU and also enable these CPU flags to mitigate the guests from all the guests that are running on that compute node from CPU flaws. So lastly, what should we consider? What aspects do we want to consider when configuring CPU models and features? So the first thing is that if your scenario is like most any others, you have a heterogeneous set of environment with different kinds of CPUs in your data center. Here I'm just showing Intel as an example there. So you have Westmere, Broadwell, Nehalem, whatever, whatever. So how do you find a compatible CPU model among those variants? So this is where the Libvert comes in and does the heavy lifting for us. And these are the APIs that Nova uses to compare different CPU models and find a compatible model for your cloud environment with a given set of host CPUs. So the two APIs there, Comparis-CPU and Baseline-CPU, those are the two ones that Nova uses today. But recently, in 4.4 release a couple of months ago, I think, Libvert came up with a new variant of these APIs. So what is the reason for this? The older version of these APIs, Comparis-CPU and Baseline-CPU, they don't take into account what your host, the KVM and QMU, the hypervisor, is capable of. So these newer ones, hence the name Hypervisor-CPU in the API's name. That takes into account what your host hypervisor is actually capable of. So it is supposedly a bit more smarter. And I did some tests. But I want to see some more bug reports before we wire this up into Nova in a future release. So as a quick example, in this XML, I won't show a lot of XML. This is the only XML you'll see. In this example, you're seeing two CPU models in a single XML file that I call. So there's a hassle variant and there's a Skylake variant. In the hassle variant, we're asking to require a set of CPU features. And for the Skylake variant, we're asking to disable a set of CPU features. And how do you find what is the intersection there? So that's where the Liberty baseline hypervisor CPU will compute what is the intersecting set of features, what should be enabled explicitly, what should be disabled. So those are the set of features that are enabled and disabled after computing the baseline. So that's the intersecting set of features given our Skylake and hassle variants. So the point being that it's the so-called baseline CPU model that will allow line migration across your hassle and Skylake variants. So that's what Nova uses, the older version of this API. So I just wanted to show what's behind the scenes. Of course, you don't have to run this manually. So what's happening behind the scenes is sometimes instructive to see. So what else should we consider? So we briefly talked about machine types. So machine type, to refresh your game, it's a virtual chipset that provides default devices. And it has two main goals. So one is that a single CUMI binary will emulate different chipsets, for example, Intel's I440FX or more popularly called PC machine type. And the other one is a Q35, that's slightly more recent one. And the other thing, which is the most important one, is to provide a stable guest ABI. What does that mean? So it means the virtual hardware that is exposed to the guest remains identical, regardless if you upgrade your whole software, or hardware even changed that. The hardware that the guest sees remains same. So there's no change in that, or even if you line migrate. So that's a critical thing that machine types provide. Whenever there's a new CUMI release, it comes with a new machine type matching the version of that release. So here in this, you see the PC machine type. And it is alias to the latest released versioned PC machine type. And I don't know if you can see that, but it's released in 1996. So it's about 20-plus years old. So I first called it legacy, but legacy can trigger an allergic reaction in people's brains. So I used traditional. The other one is Q35. It's the so-called more recent one, but it's still quite old. If you can see the date again there, it's 2009. So nearly 10 years old. It has a set of features. I can't do justice to the machine types in one target resource. Several talks. There have been some talks at KBM Forum in the past. So the point being, the version machine types provide the so-called stable guest API to your virtual machines. So why talk about machine types? Why should we care about that? Because sometimes, machine types alter CPU features. Unfortunately, it doesn't happen very often, but sometimes it does. And changing machine types is really guest visible, meaning you can look at it as an analogy is that if you replace, rip your hardware, random hardware from your host, and plug in something else, it's equivalent to that, except in the virtual world. So machine types replacing changing machine types is guest visible. So when you upgrade QMU and using Libver, assuming you're using Libver Driver, you have to explicitly make us know about how to change the machine type. So the point of the stable guest API is that even though you upgrade your QMU, even though it's new enough and bringing new machine type version, Libver or Nova won't gratuitously update that version for you. You have to explicitly ask that so that to preserve the stable guest API. So once you ask Nova to change it, you need an explicit call reboot of the guest, and only then will it pick it up. So the main point to take home there is that before changing machine types, you might want to evaluate your guest workloads. And then, because the CPU features can differ in some cases, not all, but very few cases. So how do you go about updating to patched virtual CP models? We're talking about x86 here, obviously. So the first thing, I've talked to some operators who already did this, so I'm sure those who are here would know that. But the first step here is to update your microcode on your host kernels, on your host, and the host kernel as well, and the guest kernel. You can check the CISFS directory to see what mitigations you have in place after doing that. Once you're done with that, then you can update Libver and QMU, so they'll pick up the new CPU models. Followed by that, you tell Nova to please update me to the patched variant of whatever CP model that is relevant in your environment. The IBRs models are the ones that have the fixes for some of the Spectre or Meldon. I'm already drawing a blank, which one it is. So check those. And there's guidance over there. You see a taxi file, but you don't have to read that in QMU source. And there's a nice document written by my colleague Daniel Barangé. There's a rendered version of this, an HTML version on his blog. There's a reference at the end. So there's a bunch of details there. Before you upgrade, you might want to refer to that. So once you tell Nova to update to a relevant CP model, the last step is to you can't avoid cold rebooting the guest, no instances, so that they'll pick up the CPU ID bits, right CP ID bits. So that's the very high level problem. Maybe there are some steps in between that I'm missing, but I'm sure you'll figure out as you do them. But this is the core of things that need to keep in mind. What about CP flags? There are multiple of them. I'm not going to read them all out loud. But to mitigate some of Spectre and Meldon, there's several flags that you need in actually not a whole lot of them actually, just a few. Some of them are built into the fixed variant of CP models for QMU. And we'll pick them up as well so you can use them when you're using named CP models in your compute environment. Again, for that, refer to the blog post that was written by QMU upstream and the other one as well, the QMU CP model details that's in the QMU source itself. So what about the expectations from applications like Nova and expectations from whom? From projects like QMU and Libbird, which are doing the heavy lifting behind the scenes. So up until now, whenever there's a new CVE, QMU added a fixed CP model with a set of features in it. And Libbird tags along and adds a QL and CP model in its XML representation. But that's not really working out so well, given the flurry of CVEs that's been landing since January. So QMU and Libbird upstream have decided that they're not going to add any more new named CP models whenever CVE comes out. So instead, tools like Nova or Overt or whatever management applications, they should explicitly enable CPU features as required. So Nova today takes care of this already, so no problem for us. For other management applications that haven't provided a facility to supply granular CPU features, that's considered as broken. So there's a more interesting thread there on Libbird and QMU mailing list. So I just quoted that from there. So to kind of sum up, don't go with the default CP models and we don't even need to go when it is Nova because it has the same default.  But if your case is like anybody else's with a mixed set of CPUs, choose Nova's default. But if you know your host CPUs inside out, then you can figure out a custom baseline and some operators already do that. Just this morning I talked to somebody on the train. So they already compute a custom baseline for their own environment. People already do that. And before changing machine types, you want to evaluate workloads. This is not like super important one, but just important enough to merit a mention. Because not all machine types change CPU features but still deserves a mention there. And finally, you want to have like a systematic procedure to go through your updates across all the relevant components and then you can do the final step of using the patched CP models and flags. I think that's pretty much it. These are the references that I mentioned, some of them. I didn't update it with the KVM forum links but you can find them on the internet over there. So that's that. So if you have any questions or comments, well, welcome. I think there's a mic over there. I'm not sure it's working, but... So under one of the first slides, you showed that Chemo 64 is available for all KPUs. And actually it's not always true because Chemo 64 includes SVM instruction which is not supported on some Intel processors. That's a historical accident. And my question was like you showed that it's possible in NovaConfig to add extra capabilities. Is it possible as well to disable them in NovaConfig? That's a good question. So no, that's a to-do item on my plate. Currently you can only enable CP features but you can't disable them because the enabling is more pressing requirement from now so that with all the fixed CP models you have to turn on specific set of features. But yes, that's not there. In NovaConfig currently today you can't disable specific set of features. So that's a bug on that for me. So you can put me on the hook for that later. Yeah, okay, thanks. Hi, I'm Francisco. Just a quick note that this extra CPU flex... Can you please come closer? Oh, sorry. So I just want to note that this extra CPU flex stuff is just relatively new so you won't have that in Newton or something like that. I can't pass you. Maybe can you say if you can't hear me? Yeah, I can hear you but it's hard to follow. Go on. Oh, so the note I have is that the extra CPU flex NovaConfig option is something appeared in NoCata. So you won't have in Newton or Liberty or some other version. We back ported it all the way back actually. At least I back ported it three releases. I forget which versions. But if you're using any of the, at least rehab versions, so it goes all the way back to them six or seven. Okay, I'm using Ubuntu, so that's probably the problem I have. Yeah, the other question I have is related to these machine types. So that's, I have the impression that the machine types are practically, I mean, on Ubuntu 16 this is using something like a default, so this PC. Is there a way to configure that from Nova? Yes, and Nova has two ways to configure machine types. One is you can configure per compute node in a config attribute. So all the Nova instances launching on that particular compute node will be given that Q35 machine type or whatever version machine type of PC. The other way is per image template. So on glance there's a metadata property. So you can use a metadata property for a template image and then use the machine type for that. So, yes. Oh, make sense. Thank you. Yeah, thanks. Is there a way to configure QMU to use a specific default without having to pass them on the command line? Sorry, is there a way to what? Let's say I want to have a specific machine type. Is it possible to configure specific machine types in QMU? By default. By default? Without having to pass them on the command line. The default is PC. Yeah, so like, I have an environment where my command line is limited in size and I'm reaching that size. And like. QMU has several different kinds of configurations with machine types. It's kind of hard to hear in this room and maybe it's just me. I don't know, maybe I'm a bit deaf. But we can talk about that. If there are any more questions, comments. Okay, thank you for the time.


What's up everybody, Jason here for jazzykspace.com. We just sold out the last filler here to unlock the 26 box mixer guys. We're giving away three spots in that mixer with this 2022 Top Scrooge MLB Blaster plus Top Scrooge Platinum Blaster, two box break. Number one, so first remember, no vac commons will ship out of the break. We'll randomize some of the names to ensure the hits out. Then at the end we will take the names, re-randomize them that bottom originally and top three get into this mixer here guys. So here we go. Let's do the break first. Roll four and a six, 10 times. One, two, three, four, five, six, seven, eight, nine, ten, ten times. All right, let's get a new sheet. Six, ten, one, two, three, four, five, six, seven, eight, nine, ten. Rangers down in the Marlins. All right, again I go of course it is a blaster break but we're gonna have no trades guys. Sticks what you got. Here we go, here we go guys. See, Lance Lynn, Sepia, King Peter Lanzo. We got a heart of the city for Detroit. And a pink just so I agree. All right, we go guys. We got a Christian Gelich, number to 299. Marko Diplen. Ah, this is a nice blaster. Framinau Reiser, all the blasters like this. That's not a number though. Congratulations Stephen Kwan. Auto for the Guardians, nice. Walsh, Guardians autograph. That's Chris, then we have a Robin Roberts out of 100. Spencer Strider, Andrew Jones, Joe Morgan, Tim Reigns, and Nick Allen. Got an autograph though. Now a moment of truth, let's give away those last three mixture spots. Here we go, three of them. Dice roll, six and a three, nine times. Good luck, top three are in. One, two, three, four, five, six, seven, eight, and nine. Nine times, just missing out was Chris. Well you're in Chris, you're in. And you had the odds buddy. Top three buddy. There you go. So last three going to you, blaster number five I believe, right? Blaster number one, guess number one. All right, I appreciate you guys coming up next is the mixer. Thank you.


Thank you for attending in this presentation. My name is Fernando Angoli from the Irish University Association. I'm a student associate intern in the enhancing video teaching and learning in Irish universities project. So I've been making a presentation on student staff partnership, which is a case from our student interns where I am part of the internship. So first, I want to start by just giving you a brief overview of our internship model. So in the project, when the project started, we never had student interns and they were just staff working in the various universities, the partner universities. And sometimes in 2019 actually in November, there was a need of hiring a student associate intern with the core objective of the student intern bringing student voices to the project and being actively involved in the creation of digital resources to support online teaching and learning. With the success of involving this one intern, then there was this urge and this, you know, out of the importance and the value that the intern brought to the project. Then there was a need to expand the internship model for every partner university to have at least an intern working with a member of staff. So the internship is based on this structure whereby the student has to be currently enrolled and actively enrolled students, but not a student who has, you know, completed college and the students are working 15 hours per week, which is 60 hours per week flexibly per month, I mean 60 hours per month flexibly, because there are some months that are very heavy for students and there are some months that are light. So depending on, on how the college timetable looks like, there was that kind of flexibility and there was a payment of 13.91 euros per hour. And each university had at least one intern, there are some partner universities that had even more than one intern in their local university. And the interns work collaboratively with the project staff. And also we saw this urge where there was a growth of this collaboration between interns. So just giving you the structure in terms of how the model looked like. So at the Irish universities association, that's where the EDTL project is hosted. And now that this partner, there's the eight partner universities that are under, that the project is being implemented in and every university has a staff and has a student intern working with the staff collaboratively. So at the Irish universities association at the project, the higher level of the project we have a member of staff and also a student intern. So the internship model started with an intern at the higher level, then, you know, out of the work that this intern did back then in 2019, we saw it important to expand the scheme to the partner university. So as you can see in 2019 we started with one intern and the reason why I'm projecting this screen is that so that you can just see that where the project has come from and the kind of diversity and, you know, like the kind of interns that we had like this the first intern was a postgraduate student with a background of journalism so that they should just, it's not, you know, we didn't only hire interns from technical fields myself, I'm an anthropologist and I'm in the school of business. There was a diverse, you know, like a mix of undergraduate and postgraduate and, you know, students from law, medicine, business like myself and anthropology, digital marketing, data science. And also we had like 14 female students and seven male students so there was all this kind of diversity because all these different groups of students bring in different perspectives and three international students and 18 local students myself I'm an international student from Kenya. What are we trying to do actually because the purpose of the internship was actually to, for the students to create a digital teaching and learning resources that reasoned with a student perspective just from their perspective and to support the staff and to work collaboratively with the staff. At times in 2020 we decided okay fine, why can't we do an evaluation of the student internship scheme. And that's where this great idea of having a qualitative study to understand the student internship scheme came up and the core objective of just to understand the role of students as partners in digital teaching and learning. And specifically we were looking at documenting how students contribute when they are engaged as active partners in the digital teaching and learning space, and also just trying to understand the benefits. And, you know, like how students benefit when they're actively engaged in the student partnership space in digital and learning, and also as a bonus like we were trying to understand the value that the project added to our student intern. So why was this important or why is this important so the first thing is that this the project has been running like for quite a long time and we had a lot of informal lessons so it was important that these lessons are documented so that they can be referred to in the future. And just so that we can have, you know, future user centered approaches to teaching and learning, you know, because more often you know we design programs for students without involving them and this was very important. It's a program where there's a project where students are creating resources for themselves and they're being involved actively. So how can we transfer these skills to other projects and to other areas where we involve students. So I will invite two of my fellow interns, Katharina Katz and Schneid Muni, just to share their experience over to you, Schneid. Thank you very much for accepting to be part of this panel. So welcome to Alt 2022 on the screen. So I would like you to share with us your overall experience as a student intern in the project, just trying to tell us what are some of the things that you know you collaborated on with staff. How did you collaborate with the staff, like what worked, what did not work and what would be the barriers or what were some of the barriers of, you know, you as a student engaging with staff or what would be the like the barriers of engaging with staff and just what worked generally. Okay, yeah. So, I was student associate intern for the fire university associate enhancing teaching and learning project this year in Donut City University in Ireland. So the way it worked my collaboration with staff, I mainly worked with the teaching enhancements unit in DCU and I worked with Rob Lenny and Suzanne stone mainly. And so we would have a meeting every week and we would discuss kind of goals for the week. I'm looking at different initiatives that the teaching enhancements either already started or we're looking to start during the year. So some of the things that we worked on, we did a project called DCG 10. So this was some kind of informal learning sessions where students could come online in this space called gather, which had like a sort of pixelated fields. Well, some like sort of zoom video colleges and we would go through different things to look at like how to use Excel or how to make use like videos within like lectures and assigning content that sort of thing. Some of the other things we did. We did a weekly to tips in the students union newsletter for the virtual our virtual learning environment service loop or Moodle. Continue the digital project by creating an Instagram for it where we posted twightly tips and engage students through that. And we've also been working on our little skills hope repository site, which is called the skills.ie, which I based off a spreadsheet kind of produced by all the staff in the teaching enhancement unit. I got a WordPress like populated posts and it's aimed at both students and staff and give them resources on a different digital companies and fees that work for both students and staff and are aimed at different people. So what worked is, I think, stuff like creating the Instagram for the digital project really worth because that reached out to students where they were kind of on the Instagram is probably the most used social media for students stuff. We're able to engage with students really directly that way. And I've enjoyed the experience of being a student and working with stuff. Often I think one of the barriers can be that it can feel intimidating to give recommendations to staff. Because you kind of on using you don't have that much insight as a student, but you have more insight and you think and I think I was able to realize that I had so much more to learn about digital learning than I thought I did because sometimes you can think, you know everything already but it's actually more to learn. Sometimes what could be difficult is, especially when we were doing the digital sessions, we did both online and in person online. Like there would usually be I think about a max of about seven people, sometimes more than three or four who get a lot out of sessions and these people would usually be coming back every week and we like that same mature students and international students who I think got a lot out. But we kind of struggled to bring more students into it because I think sometimes it can be hard to convince students to extra curricularly like see things that influence their learning. So it kind of when we did, I found the online session were better than the in person sessions. It was easier to get people to catch up onto the online sessions. Which, yeah, it can be like difficult to kind of get students who think they already know what they're doing like like third or finally used to like engage in developing their digital skills. But I think it's a really important thing. I think as well as I mentioned one of the barriers can be students in nurse for giving recommendations as an interest sometimes I would find it difficult to kind of align my skills so I could work consistently on projects and I would have loved to have had more time to meet with other interns. So I think in general I really enjoyed working as an internet really felt like my opinion was valued by the stuff and I felt like I got to do a lot, learn a lot about building website, working on in from building a following there and I got to give some of the Digitown sessions. I got to lead some of them myself and present them with a really valuable experience, getting to explain digital learning concepts to other students because I felt like I learned more about them while I was teaching them to other people. So I think yeah, it was great. And I really enjoyed being a part of the enhancing digital team and learning project over this last year. Sorry, I think I was on mute. So thank you. So just to like to wrap up, and if you are to recommend or to give recommendations, what would be some of the recommendations that you will, you know, give out on, you know, the best ways of engaging with students as partners. You know, based on the experiences that you've learned from the project. Yeah, well, I think I guess some of the recommendations I would give it definitely kind of reach to students as early as possible in the year to see if they're interested in being involved in like initiatives like this. I know the enhancing digital teaching and learning initiative is sort of wrapping up now, but I don't think having more initiatives like this, either by the Irish university associations or by the individual universities will be a really good thing. Because like I said, I think people don't really know how much they have to learn. And I think people are enthusiastic to learn more about these things. And yeah, so I suppose kind of give bring up more like create more of these initiatives reach out to students early often and kind of keep checking back and keep recruiting more people. And yeah, just kind of, I think I think students are more trusted sometimes and people might think so kind of just don't be afraid to kind of keep developing those initiatives and give them opportunities to develop their skills when staff and students are working together. And I think students will gain a lot from that and so will staff. If that makes sense. Okay, okay, so thank you very much for your insights and yeah, thanks. Okay, thank you, Cathy for joining us in art 2022 virtually from. Ireland. Yeah, well, close to me. Okay, thank you. So Cathy, you, you are part of the like the first batch of the interns and who served like for a really long time in the project so can you tell us about your experience in the project mostly around them. What were your roles and then like, you know, how did things work out. How did you collaborate with the staff and how how was it like just generally in terms of what were the barriers and what are some of the things that facilitated the student staff partnership. Okay, so, um, yeah, so I'd love to talk a little bit about that. It's been, I think like two years now since I was the student intern, the student miss me new associate intern from the University of Minuth, and which was the role that I participated in the EDT L enhancing teacher teaching and learning project, which, you know, encompassed also universities beyond. And so I was kind of like the representative or one of the student representatives from Minuth University and with me, there were a few other interns as well from a new university. So we started off, I think we were two interns from Minuth University and then a third one joined us from a different kind of department or different educational backgrounds than the first two interns were. And I was also, you know, being the first intern, I came from, you know, anthropology, so I came very much from a social science kind of background. And the second intern who joined quite, you know, we've met the same time. She came from a business background. So there was throughout the whole, you know, year that we worked as a student interns part of the project. There were kind of, you know, and there was an insurance or kind of like a strive towards having students from different educational backgrounds as well. So we had, you know, at one point we had a fourth student intern also joining us and who was a little bit younger than us. So like I was a first graduate student, but the fourth student intern who joined was a I think first year IT student, IT and mathematics students are also very different than of course like social science and also different in business. And so as part of the student staff partnership at Minuth University, we were collaborating closely through the year with, especially the project leads and who was Morag was also from Minuth University and she was situated in, you know, kind of agile education. And so we kind of she was our first point of contact. So we had like weekly meetings with her. Of course, also when we had, you know, when there were off days or bank holidays or, you know, off times during the academic year, and we didn't have meetings. So it was really, really nice as a student to also kind of have some safety, some, you know, around working hours. And, you know, from the academic timetable that, you know, both students and stuff kind of adhere to us also as part of the project. And yeah, so we had weekly meetings. We had a Microsoft Teams page as well. And what I didn't say in the beginning is that the year that I was a student intern was during the, you know, COVID pandemic or the kind of like first year of the, and, you know, kind of working remotely. So we used a lot of digital tools, which I guess like made sense also in context of our project enhancing digital teaching and learning project. So we ourselves kind of used a lot of digital tools to collaborate such as, and we had like a WhatsApp group. We had like a Microsoft Teams group as I was mentioning the meetings also because we couldn't meet in person were online as well. So we actually only met in person the first time. I think after that year always like towards the end of the project that was the first time kind of where we had an event together in person at the university. And so yeah, most of the meetings were remotely also from Microsoft Teams. And we used a lot and the Microsoft Teams group because like that was kind of our platform or our shared space, where we kind of shared a lot of the work that we've been doing. And, you know, quick questions that we had. I was involved very much also in the Instagram team as part of, you know, Menuth University, where we did some campaigns together with the new students union, but also as part of the wider EDTL team. So we were kind of also a few other students from other universities, other student interns. We met also in addition to the weekly meetings with our, you know, local kind of project leads. We set up another Microsoft Teams group and another kind of reoccurring series of meetings with the Instagram team where we met with, you know, students who ever wanted to join as part of the EDTL team. And where we met and developed kind of content for the, you know, what we're going to post the next week on the EDTL channel, which didn't only concern, you know, Menuth University, but also to attention to, you know, a series kind of campaigns or support universities, such as, you know, like in court, for example, or, you know, other Irish universities who are our partners. Yeah, so that was kind of them, they were kind of the main things that I was involved in, in kind of working together with other student interns at Menuth. And as part of the Instagram team in Menuth and beyond Menuth, but also then there was the option, which I think was really great to have the option, but also not be kind of forced to join. Which were these kind of wider EDTL meetings from, you know, other project leads from the other university partners that were part of the EDTL project. And they were usually steered and kind of facilitated and hosted by Sharon, and, you know, was kind of like the main director of the project, or the main person responsible for the project. You know, I joined them a couple of times. And, you know, what we did there was updates, you know, other members from other universities, and what's what's going on in Menuth, what we've been doing, you know, as part of the Instagram team. So that was always nice to kind of also have a space there to kind of share our voice and come with questions and so on to like the wider team. And we were always reminded when we were sent out the meeting invitations for these wider meetings that, you know, we didn't we weren't expected to be part of this wider. So like each student intern was invited and very much welcomed this meeting said it was great for me to kind of learn about what's going on in these other universities as well right because it was really interesting then to kind of bring that back to our university or to the team and kind of like, and use that to create more contact content to steer our conversations. And so it was really a good opportunity to take part but also nice not to be forced to do that. Yeah, do you have any other questions? Maybe I'm out of town. Maybe just just a brief like 30 seconds like what out of your experience in the project like what will you recommend as the best way of involving students as partners, you know, trying to work with the with stuff like what will be a recommendation just just 30 seconds. Yeah, the importance of dialogue, for example, is something that's, you know, for me at the core, you know, making sure that students, whenever you have this weekly checking meetings or however you want to structure it, do have a space to also come with questions and and also feel safe to kind of express their ideas, you know, students might be a little bit, you know, still caught in the dynamic of looking up to, you know, staff partners. And so it's very, very helpful to be kind of encouraged to come, even with, you know, ideas that aren't fully developed and seek more and seek advice from staff as well, but also, you know, kind of be encouraged not to not to be, yeah, like perfect yet, but also not to not seek advice to be actively asked and reached out to as well. And to kind of come with, you know, what's there yet or what they feel like they would like to do. And it's dialogue is something that's at the core, I think, in partnership. Okay, thank you very much, Kathy, and we show the best over there in my notes. You too. Thank you. So thank you. Thank you, Kathy, and Schneid for that wonderful sharing of your experience. So we'll get back to our presentation and, as I said, we interviewed both staff and students and to understand this kind of experiences that they had partnering within the partnership of student and staff. And I'm just going to conclude my presentation by sharing the staff experiences. What did the staff think about the the student staff partnership and the internship model. So, generally, we found out that, you know, staff appreciated the value that this this internship brought to the project and there was greater emphasis on the authenticity that it brought this authenticity of the student voice and the student representation that the model brought into the into the into the project. And also they just say, you know, they thought that involving the students as partners in the project. It brought the authentic student voice in each and every project activity that we are doing, because you see even in the webinars and students who are being involved in presentation in workshop students are being involved. And right now, even in research, as a student, I am the one who is making this presentation and we are making this presentation, the three of us, myself, Schneid and Schneid myself and then and and Kathy. And that shows how a, you know, key it is like when you bring students you can hear directly from their voices rather than hearing from a third party. Then just also involving student and working directly with students brought some sort of a different perspective that will have never been captured by staff was most of the staff admitted that they were actually students like long time ago and the current students were experiencing different issues, compared to a student maybe was enrolled like maybe three years ago before the pandemic and, you know, so and just having a student brought some sort of convenience like you know you have somebody to help you, and you're partnering with directly without going through the complicated structure of the student union, which is like the most common available option for student staff a partnership. As you can see from the Boba team course, so these are three different stuff just talking sharing about the experiences that there's one stuff was who said like you know I delivered a workshop in collaboration with one of our interns, and I've done that workshop many times over the years but when I delivered it in conjunction with the student. It was completely different for the better because she was able to bring much more of an once perspective to the workshop. So just the different perspective that students bring to the project and involving student has really brought the authentic student voice to the project activities and has helped us to shape the project. So that is a member of staff, you know, saying, and the last quote there, the staff aren't getting inside inside a real inside into how students are experiencing things, and to be completely honest, sometimes when the students came together and discuss those experiences they allowed us to get that insight to hear them speaking to talk about the impact that was that was having on them. You know, it brought a different perspective, so sorry for the type of so just for us as students sharing with stuff, what we're experiencing, especially during the pandemic, some of us like me. I'm a student of, you know, of the pandemic because I started my PhD during the pandemic so the challenges that I was facing, you know, the kind of needs that we had as students at that time were just totally different from what maybe the students and of, you know, pre pandemic were having so it was important that, that, that, that you hear from, you know, students. So then the overall from this presentation from the experiences of the, you know, the students and the interns we asked like what would be the recommendation of, you know, students staff, a partnership what, you know, what could the students recommend and this is what came up so. So number one we found that there wasn't this a need for a structured model for involving student as partners in teaching and learning, and other than the student union, you see, just to explain the how and the what like the model at this, you know, the, this is the structure that this kind of internship like our internship part like you know, because the student union is a very complicated body where, you know, all these stuff are trying to collaborate with students to the student union but in this case we had a member of staff working with a student dedicated to them. Then it is very important also to compensate us students when you're involving us, you know, to cater for the time that we spend collaborating with staff because you know like some of us, we are not only students but we are also engaging in some part time job so when we come to engage you know to actively engage those 15 hours per week those 20 hours per week. We are giving our time that otherwise we could be doing other things out there, or even just you know, so it's important to compensate and recognize students effort by paying them that's what we did in the project paying the students the 13.91 Europe a hour. Then training for both students and staff is important to highlight possible areas of student staff because some of us did not know about areas of student staff partnership within our university until we joined this project. So there are lots of other opportunities for student staff collaboration that students are not aware of or staff are not aware of so there's need for, you know, sensitization and training and education. And also, there's always that fear of reprimanding, you know, like we feel like you know when you give your opinion, maybe you might be punished somehow you might be penalized. So it's important to sensitize students on ways and avenues of giving feedback without fearing the fear of being reprimanded. And just looking at my own experience and the experiences of the other like four international three international students you know the gender diversity that was there the kind of a level of education diversity that was there. So it's actually very important to have a diverse and inclusive be as diverse and as inclusive as possible, just trying to involve different types of students because different students come with different needs. Like an international students just moving to Ireland or moving to the UK or somewhere in France to start college. You know they come with different needs when it comes to you know what they what they need so we're in teaching and learning, you need to factor in the voices of such kind of students, you know, undergraduate students might have different needs from postgraduate students myself as a PhD PhD is a very lonely journey like you might not have networks as compared to undergraduate students. Then also and lastly but not least last but not least is a always involve those students who are currently enrolled because there are some internship models where students are hired post their graduation or their study period. Then that sometimes you do not capture that real time experience because you see a case study of the pandemic as student was a student in 2019. First, totally different challenges or had a total different experience from a student who started in 2020 during the pandemic like myself and many others and you know, just to conclude that kudos to all the students who started during the pandemic and they are still starting today so thank you very much for taking your time to listen in case you have any questions or any comments you can just DM me on Twitter at fkongoli and send an email to me, Fernando.ongoli.iua.ie. Thank you.