Idina's work with learning technologies helps to develop skilled data literate students who can change our world for the better. Teachers and students can develop and share coding skills with notable our Jupiter Notebook Service. Our DigiMap services deliver high quality mapping data for all stages of education. Heterological paradigms, so that sounds very inspiring indeed. If you are just coming in, please be welcome, find a seat and make yourselves at home. But first up, we are going to hear from Sarah and Neil, we have colleagues, and the title of the talk is Scaling Assessment with Adaptive Comparative Judgment. Also, I think in the program entitled From A Thousand Learners to A Thousand Markers. So with that theme, please welcome Sarah and Neil. Hi everyone, thanks for coming. So we're here today to talk about a project that we've been doing at Glasgow University. Unfortunately, Jeremy Singer, who's our lead academic, can't be with us today. So we're going to do our best to bring out our inner Jeremy. So we're going to talk about adaptive comparative judgment. This is a method of ranking artefacts like student essays, maybe, and making comparative judgments about what is better, what is worse, what you prefer, what you don't prefer, rather than making absolute judgments about this is an A, this is an E. So it's intuitively plausible as a marker. It's an intuitively plausible way of how we actually make judgments. And it removes the pretense that we're experts with objective standards when we're marking. Controversial. So what this does, what this system of grading does is it produces a fully ranked set of scripts from best to worst. And it allows for a separate consideration about where you put in the grade boundary. So you don't need to think when you're marking about this is an A, this is a first, this is a two one, this is a 60, this is a 61. You just think better worse, better worse. When you get to the end of the process, then you can decide where you're going to put your grade boundaries and whether you're marking to a curve or whether you're marking to rigid standards. So it accommodates all of these different ways of marking.  And for my point of view as a marker, it's a lot easier to mark this way. So it doesn't want me to have the next slide. It's not letting me go to the next slide. Help. See, as soon as he walks on to the stage, it does it. You've gone for the one after now, Neil. It's going on. Next one, please. That one, okay. So what it also does is it allows you to judge using a single implicit criteria. And again, what's better or what's worse, rather than trying to use complex, explicit sets of ILOs. It's much easier. And it can be used for, I always used to think it could just be used for questions that only had, that had subjective different answers. But actually you can use it for questions that have a single correct answer as well. All of this is stage scene setting. It will become a lot clearer once Neil starts talking, I hope. Okay. So distinctive benefits before Neil starts, it scales. You can use it for tiny sets of submissions like 20 or up to 10,000 potential for use in MOOCs. I think he's fantastic. And indeed, Jeremy's been using it in his MOOC. So it's compelling the naturalness. It is intuitive and plausible. It can be used with one marker. It can be used with sets of markers. So you can get your inter- and inter-rater reliability. It can be used for peer review, which is how we've been using it. So you crowd source, you get the students to do all of the judgments. And then as an academic, you come in and award the marks. So it saves a lot of time. It can be used to mark things that are very, very different from each other because you're just judging better worse rather than a set of ILOs. And you can also put in exemplars. So if you really, if you want to say, I want to see where my grade boundaries are, you put in the exemplar for the A, B, C, D, E, anything that was at the A or above would get the A so forth. And I think I'm handing over to you now. Oh, okay, I'll talk to you this time. Okay, so we've used the software at Glasgow. We're using it in the House School MOOC. We've used it to judge our conference submissions because when conference submissions come in, they're very, very different. But we were able to rank them to best fit to the conference and then decide which are the ones we were going to put in each of the streams. There was a major experiment done of this adaptive comparative judgment, which Pollitt talks about in 2012. And what Pollitt found was the expert markers who were highly skeptical initially of using this process. But by the end of it, they judged that this process was a better way of marking and it was a faster way of marking. So faster is always good. As academics, if we can do our marking faster, that's always great. But we also want to know that we're marking and keeping our academic standards. And it found that it kept the academic standards. Indeed, I think it outperforms regular ways of marking. This is it. Our implementation. We've Pollitt used his own implementation, or one that was at Cambridge. We've done our own very lightweight implementation. It's a simple LTI application, so it doesn't have to deal with user stuff. That's all done by the LMS. We use it with Moodle, or have used it with Moodle a bit in experiments. The study we're reporting on here, we used it with FutureLearn, which allows you to launch separate LTI tools. Our tool lets the submissions be text, PDF, a YouTube URL, a picture, things like that. In this instance, it was source code. And the students just paste in text and then a standard source code formatter makes it pretty printed and nicely colored to make it easier to read. You can also add students by the software allows staff to put in a set of things for students to review to make it a reviewing only exercise. That's our learning to judge from the student's viewpoint. Like other tools, Moodle Workshop, many of you will be familiar with, Aropa is a similar tool we use at Glasgow University. And I think a few other universities use. It's got a phase for submission and then a phase for review. We are thinking a bit about getting that more blurred in future. So the process, the algorithm, it's a round of sorting. So in each round, the people doing the marking or the grading, just look at two things at a time and decide which is better. And each artifact, each piece of work will be judged at least once and probably not much more than once, depending on random things about who turns, when people are looking at things. These rounds are put into three slightly different phases, which I'll talk more about in a minute. But they have slightly different scoring algorithms to improve the quality of the sorting. And this algorithm, it's a bit different from Pollitt's. Pollitt's algorithm is problematic because in his paper there's a typo and you can't quite work out what the algorithm is. So I developed my own base more on his description than his maths and used a simulation to refine the algorithm. I'll be showing you some outputs from that simulation. So this is sort of helping you think about it, but also understand the next few slides which show output from my simulation. So you start off with your artifacts, your pieces of student work. Here, six examples. They're just in a random order, numbered in order so you can see what they are, but this is just would be the order in which the students had submitted probably. To make things easier to view in my simulation, I color coded. Darker means more to the left. In this sorting, lighter should go more to the right. So in the simulation, these colors are effectively assigned to be the perfect score. And then a random number was added or removed to that so that it's got the slight error of reality. So in each round, each thing's compared, the one more to the right is picked and given a point. So and then a first sort. All the ones that got one point are on the right. All the ones that got zero points are on the left. So get on to the next round. And again, the better one or the one judge better gets an extra point. And now some have two points go further straight. Some have one point in the middle. Some have zero points. This sort of sorting over a long, long period would get it right. But that's not quite enough. But and so there is a sort of scoring algorithm put in there so that as the first two or three rounds are just done that simple way. But later on, it starts waiting the comparisons depending on how far out way in the sort are. In the second phase, it's waited to allow things to move quite fast because if randomly, a few really good or really bad bits of work had been put together at the very beginning in the first random order, one of them could have ended up very out of place. And then later, it goes so much more refined sorting things close together algorithm. So here you're seeing the first four rounds of a simulated one following one particular thing, number 43 highlighted in orange down as it sorts. And the red highlights are ones that has been compared against. And so its score at this level relates to where these are in the previous sort level. And with two more. This is then using the more refined algorithm. As you can see the sort of graciousness goes very smoothly almost from dark to light across showing that this is actually quite effective sorting mechanism. After about 18 rounds of sorting. It's near enough perfect. And this scales. This would work. However many artifacts were in there because it's comparing with a sample and using their positions. So here it is the same with this is just the middle third of a thing. All you can see of course is the color. But as you can see, it's quite smooth towards the bottom from being quite random towards the top. So that's with 600. I've experimented with up to 1000 on the little server I was experimenting on that's getting to its beginning to run a bit slow. So you can try this out. If you have your mobile device handy, you can log into our demo site. This is my first ever trial. I just keep running to show off. And I put up some pictures for people to start. It did show up some interesting things. These are pictures of wildlife, flowers, insects, birds. They're not very good pictures. They're just ones I found on my camera. But it was noticeable. Some people do sort looking at this type of artifact, which has got different categories. You could look at it. So some people would sort to make a Robin very good, even though it's not a great picture, but it's a nice bird. And other people would say spider goes right down, even though it's quite a good picture. And so there's did notice there's two artifacts. And that's my the two aspects at least to the way people judge. And this might mean you need a bit more guidance. But that's these pictures, maybe in academic work, it's going to be a bit better. So our case study functional programming in Haskell. This is interesting course in that it's run as the first half of an honors module, but also as a MOOC. So about 1000 people in this particular run, we're using it from around the world on future learn, and a class of about 80 honors students at the University of Glasgow. It's a as a programming language Haskell, it was developed at Glasgow. So that's a good thing for us. But it's from our students viewpoint, it's a slight paradigm shift in their way of programming. So it's a quite new language to them. Previously, they've programmed in very conventional languages like Python and Java. And then they do this Haskell, which is a functional programming language and quite different style in honors. They were given a problem specification to implement something which could be done in a writer page of code. Some guidelines as how to judge. So some criteria about, you know, look at readability, look at actually solving the problem. And then in the marking phase, this grading page, they were looked at their peers solutions to compare. So in this instance, we're using this as a peer tool. And then finally, at the end, they'll see their own ranking just as a what quartile it got into. We don't want to say to people, you are worse. But we do say you are in the bottom group. And I'm sure they will all have known this by having seen the others. And they also got given a sample solution. So this is a question, write a spellbook generator. It's the sort of thing that Haskell's a good language for doing. And they were given these instructions about how to write the good quality code. And then here's a sample solution. So you can see it's quite concise language. And this is so student comments. I'll hand back to sailor. So we did some evaluation. Of course, we did. We like to evaluate it. We sent out an online survey to all of the students, including the ones from the MOOC and our honor students. And we asked them a range of questions about what they thought about the ACJ software, because they're computing students, we wanted those sort of answers from them evaluation, and also what they thought about the process. So these are students who were probably fairly new to doing peer review anyway. And it was peer review using ACJs. We asked a range of questions. And what we got with a lot of students telling us that they liked the way that they got to see lots and lots and lots of different solutions. Because typically in peer review, you'd see two or three maybe. But with this, they could see a lot. And the ones that were really invested, of course, could just do more. We weren't limiting them. So it can be quite addictive. When we first use it for the conference, as me and Kathy Boval, we just got really, really addicted to actually doing the reviewing. So we did masses, because we just love seeing these photos. And we love seeing the abstracts and all of that. It was great. Anyway, again, here's one they said, they thought doing this helped them to think differently. Because they're having to think how to evaluate their peers code. Here, this is exactly what we wanted. We wanted to show students early on what their position was in the class without having a leaderboard. And they could see how well or how poorly they were doing for themselves. Instant feedback, early feedback, really important. And again, I love this one. I'd like to thank the course educators. Well, I think that's thanks to Jeremy, not to us, but we'll take it. And again, this, I thought this was really interesting, that as time went on, they could see that they got quicker at doing it. So yes, the process speeds up. And it's one more. So how could we use it some interesting stats? Well, we can get some interesting reports out of it if we wanted to write them. So we could set the software up to say who is the most deviant marker. So you know, sometimes you have a marker who's a bit problematic. Everybody else maybe is judging something to be good. And they're judging it to be really bad. Well, you might want to look at this marker and take them out. Or you might want to look at this marker and say, what is this marker seeing that everybody else is missing? So you can get that sort of information out of it. You can also see which submission was the most divisive, which was the Marmite submission, the one that some markers thought was brilliant. Some markers didn't think it was brilliant. Again, is it the case who's missing? What? Because, you know, I know myself sometimes I've marked something and I think it's maybe round about a C. And then somebody else comes in says, actually, do you know what? There's something really novel, really interesting. I would give that an A. These are the conversations that we have as markers. And these are conversations that we can see in the software. And again, we can see how converged the judgments are. Is it the case that everybody thinks that one is the best, that one is the worst, or is there a bit of a controversy? And so it's not the case that we just given a bit of ranking and we have to accept it. We can interrogate the data. So where next? Well, the software we have is still in development. It's still a pilot tool. We've been piloting it successfully for how many years now? Three, four? Before years? In small things. It's living software. Neil is the developer. Neil's a fantastic developer to work with because he understands what academics want. And he will work with academics to get a bit of software that works for them. So if there's a restriction in the software, very often Neil will work to get around that restriction. Obviously there are restrictions that you just can't do. You can't do magic. He can't give you a unicorn. But he has developed this bit of software in line with academics and working with Jeremy, who is a fantastic academic, who's very engaged in his teaching. And indeed, it's very engaging when he teaches. It's really, really useful because you get a bit of software that is suited for academics, but it's technologically robust as well. And I think at this stage, if you think it would be useful in your teaching, I think Neil is putting out a call to ask for a collaboration. The software, of course, is open source. It's on GitHub. Anybody can go and pick it up. But what we'd like is if people are picking it up, if they'd work with us. Because we're doing further research in this. We're still using it. We're trying to extend our pilots. And what we would really like is to work across the academic community to work with you to do a proper robust study. So scholarship, research, or people who just think, yes, I want this in my teaching. I don't want to do any scholarship. No research. Please just let me use it. All of these are fine. And I think that's it. Fantastic. There is a roving mic. So if you'd like to raise your hand and ask a question, we have had a few questions here as well. So one of them was around providing feedback to students. And the question was, if a submission is ranked at or near the bottom of the rankings, how does the process provide feedback so that the students know what needs improving or why they got a low score? So we'll start with that. But if you have a question in room, we'll come to you next. So there is no traditional feedback given. No thing. You should have done this. This bit was poor. I've been a student recently enough to remember that wasn't very helpful. What there is, is what David Nicol would consider to be internalizing of feedback. The students are seeing a range of work. There's a very good learning potential there. It probably needs to be studied more, but I think that's a, that is potentially a much more useful form of feedback. That's an interesting new solution. If you do have a question, can you just let us know who you are and where you're from? Thank you. Okay. Thank you. Steve Rowett from UCL. It's a bit like a binary sort algorithm, I guess. Yes. And by the end of that, a student is comparing two things that are probably very similar. They're near the bottom end, they're near the top end, they're in the middle. Does that make it very difficult because it's quite hard comparing two things that are similar? It probably does make the comparison harder at the end, and yet ideally we'd get students going through from the beginning to end, but you can't unless you've somehow got them timed into it. And since we're doing this fairly open, like they've logged in and they're on time, we can't do that. So that's something worth looking at and thinking about how to make it better from that viewpoint. But yeah, when doing the conference judging, which I took part in, yes, things get slightly harder at the end, but you also because of being through it, you get quick, you get good at it. If you're just jumping in at the end, yes, I can see there's an issue. But I think at that point, really at that point, you just say, well, we've got the sort, no more judging to be done, and if students needed, if it was a pair exercise, we could, we've talked about starting it again for students to go through the process. So we could have multiple sets. And then, and then we, fantastic. Then we could rank all of those against each other. Wow. What a, what an evaluation. One thing that if a student is seeing two very close together, the next pair they see will also be close together, but they won't be close to these two. They'll have been, it's quite different in this argument. Thank you. I think we've got time for two more questions. So we'll take one online and then if there's any more in the room, if you just hold your hand up, please, and we'll come to you for the last question. So there were a couple of questions around algorithms. And there was also a question around the ease of use and whether it's openly licensed. So maybe if we focus on ease of use and openly licensed. It's incredibly easy to use. I have been using Moodle Workshop for many, many years. I understand it's affordances. It's not the easiest. It's a lot easier to set up because there aren't as many settings in it. So from the point of view of a member of staff setting up is very easy. From the point of view of a student, it's really, really easy. They get two things on screen and they just have either push left or push right. That's that's it. In terms of licensing, I'm actually too licensed. There's a few bits of other open source in there, which are various different licenses, but they're all quite liberal open source funds. Great. Thank you. So time for our last question. Is there is anyone in the room or have you all posted them online? Otherwise, I think we'll finish with a question around the algorithm. And so we have a couple of questions. One is how sensitive is the algorithm? One is, is the algorithm just rules? And the last one, is it ethical to use an algorithm without checking the results or are you checking the results? And could you expand a little bit on that? So if we maybe round up on that, is that all right? The algorithm is just rules. Yes, that's what the algorithm is. It says, given this, we will award this number and use that in a sort. The checking, well, the simulation shows it works. Simulation shows very convincingly or with a lot of noise in it, which is interesting. That's where this algorithm is different from a standard computing binary source because there's it deals with noise. It deals with some of the judgments being different from others. Fantastic. Thank you very much.  I think there are some colleagues who Sarah and Neil who might be watching this on the live stream. So if you have been joining us online, a very warm welcome to you as well. If you could just put your hands together for our presenters. Thank you. Fantastic. And next up is Matt Kornock who is joining us from the National STEM Learning Centre and where Matt is the online CPD coordinator. And our second session in this half an hour is the enabling professional development by letting go of the pedagogical paradigms. And please put your hands together for Matt. Warm welcome. Thank you very much, now. My role at the National STEM Learning Centre is the online CPD coordinator and that means I'm responsible for the learning design of the online program and also the program manager. So I spend all my day thinking about how we can improve learning design for our online participants. And our participants, they are teachers. They are teachers in schools and colleges around the UK and of course our international audience because all of our online courses are available on the FutureLearn platform. So our program looks like this. We have over 20 courses. Most of them are free. There's a couple of paid ones for small cohorts. But most of them are free and they've been developed mainly over the last two years. We had five courses that we started with about five years ago and then I joined two and a half years ago and we've been rapidly upscaling the amount of online courses we have. So that we can create professional development pathways. And that I think is an incredibly important part of any online programs to have a pathway. So the professional development isn't just a one hit wonder. There's a journey that people can go through and we can support them through that. So the big numbers then, last year we had 50 course instances, 57,000 enrollments, 109,000 hours of professional development delivered. And that is actually quite an achievement when you can say to myself and my colleague Karen who is solely responsible for the online program, we bring out a lot of people to run the courses, to write the courses. But there's just the two of us that keep the lights on. So this session is a reflective session and it's a reflection on some of the ideas that have influenced how I view online learning designed for MOOCs. And I'm actually going to stop referring to MOOCs. I'm just going to say open online courses because I think the scale of a MOOC varies considerably. We could be looking at 100 people. We could be looking at 5,000 people. And that's the sort of range we get in our courses as well. And these ideas that have influenced me characterize the type of learning that's been designed for and typify how pedagogical approaches for face-to-face and online might not be able to be labeled in the same way for the open online course area. So I've been grappling with this dilemma that learning design is not necessarily the learning experience. It's an obvious point and as I stand today I think criteria has taken me that long to come to that conclusion. But this issue has started to pose very existential questions for me as a learning technologist about the role of learning design, the theoretical basis, learning outcomes-based activity. And as I explore the literature particularly around open online courses, I became increasingly skeptical about how particular pedagogies were being evidenced in open online education. So let's have a look at some of the theories that have influenced me, in particular around teacher professional development, Ana Lorela's recent paper in 2016, using MOOCs for teacher professional development, talks about co-learning. And she cites Avalos, a paper from Avalos in 2011, co-learning is about networking and interchanges among schools. It's strengthened through peer coaching, supportive collaboration, joint projects, teachers naturally talk to each other and therefore why aren't those conversations about education? Now within the STEM learning ecosphere, we have a network of partners across the country who do this on a face-to-face basis. One of the big things for me at the moment is to make sure that online and face-to-face are blended together and really draw upon the value of that. And that's where that network and interchanges among schools are taking place. But are those, are those, is that networking, are those interchanges actually taking place on online courses as well? We'll have a look at what that might be happening there. And Lorela's relates the idea of co-learning to communities of practice through MOOCs that have curated digital resources and orchestrated collaboration. And that idea of orchestration of collaboration that people are working together in an orchestra to come up with a great output, I think, is incredibly important and valuable. Now, Luiz, Monica Luiz in 2017 and her colleagues looked at a study in the Netherlands, not necessarily for online courses, but for teachers' professional development choices about how they are self-directed in their learning. And the range of aims that these teachers are trying to achieve, and the same applies to HE lecturers as well. So when I'm talking about teachers, you can also think about the colleagues if you're working in the university about who you're working with. Whether they're the newly qualified, their mid-career, or the very, very experienced, the types of things they're looking for are activities that help them to reflect on their practice to keep up to date, to experiment, technical things like managing the classroom, getting up to grips with the subject matter that they're teaching. And student care. So these are all traits of professional learning. And in the teaching sector in the UK, there's DFE, Department for Education Guidance, on what the professional's learning standards should be. And key to that is sustained and embedded practice, which is another reason why we can't just have one-hit-wonder workshops or anything like that. There has to be a journey, a longer period of development. And online is perfectly situated for that, because you get courses where you can try something out, put it back into your praxis, practice that week, get a bit of feedback on it from your mentor online, do a bit of reflection, try something out next week and do that across a whole term. You sustain that in a journey by linking these courses together. So these are the traits of professional learning, but how are professionals actually supported in adopting those traits? Our approach at STEM Learning is to have structure points of interaction in all of our online courses. We have the sharing of ideas as key to the learning activities. All of our content is evidence-based, whether it's academic research or whether it is through practice itself, the idea of action research, ongoing reflection, which we have reflection grids each week for our learners to complete and action planning to sustain the impact. What are they going to do? What are they going to take away from our course and what are they going to do next week, next month and next year? And this is what we do on future learning. We have that weekly structure. We break the structure down into sequences of steps, pages of the course that meet many learning outcomes, if you will. We have really high quality video. We go out to real schools, film their real classrooms, interview teachers, talk about what they're doing. We use quizzes, not as a mechanism for assessment, but as a mechanism for delivering content to challenge our learners' way of thinking. We have expert Q&As, asynchronous Q&As, learners post their questions, we get a response, sharing through Paddler for anything that can't be summarized in a short 500 character text box on future learning. So things like lesson plans, could be videos, we have some excellent videos of teachers and technicians demonstrating what they're doing with their classes and their practical work in science. We have our own high quality demonstrations from the National STEM Learning Center and we also have fundamental to all of this, of course, is the social learning pedagogy, those discussions, those constructed activities, those orchestrated collaborations. But we're trying to address here a range of learning needs. This is just from one course. These are the different learning needs that these teachers have. Want to know more about practical work in science, want to be able to meet the assessment criteria, help their students meet this assessment criteria, want to be able to evaluate their own practice, know how to evaluate their own practice, want to be able to relate their content to the real world so that students have that relevance of what they're being taught. And if they're new, there might be a non-specialist, an increasing proportion of the teaching workforce, the teaching subjects they don't have a degree in. It's not the same in academia. And that is a key area for us to support. And that all wraps up with confidence and confidence is the tricky one. We always aim for our courses to build confidence in our participants. But how do we measure that? How do we make that a learning outcome that we can actually measure? So I think there are some contradictions in open online course design. And that goes along the lines of trying to meet everyone's needs, but also having a very clear structure to a sequence of activity. All of our courses have a weekly structure. They're linear in that respect. But we have designed them specifically that you can, if you know how I think is the question I'm going to come to, you can choose the different activities that you want to meet your own development needs. But how do we do that as a design process? That's not a design process that I know of very well. And we've got ideas of personalization of our own where you can have these crazy maps and find your way through. But there's still a big challenge that was made by Kate Lindsay earlier in the session here about supporting our learners to be better learners, essentially. We've also got individual times. If you join before the course starts, you're more likely to complete because you might not be part of the cohort if you join later on that's already had the discussions. So how do we account for the openness of that timeline and how do we still build in that socialization? How do we make openness of access whilst also making sure that all of our learners have that self-efficacy to be able to learn with us? So what do we normally see in all the literature? Retention graphs. And these are for retention graphs from a selection of courses that we run. And we might see, okay, we know there's a dip in week one. Oh, that course is slightly different. The bottom course there, that I've just highlighted, is a five-week course which has a research story behind it, essentially, how we learn science with learning. And so we take people through a model and so they have that hook very much early on week one. The one at the top, Managing Behaviour through Learning. It's about practical approaches for managing behavior in the classroom. You could cherry pick if you wish. So there's naturally gonna be a drop-off. Three-week subject specialist course, a few bumps here, bit of a bump at the end when you've got a Q&A. What does that actually tell me about what my learners are doing, the decisions they're making? Not a lot. I don't think we can take much for retention curves. Now I was really grateful for the input from Seb Smiller and David Jennings who said, well, why don't you look at retention on a week-by-week basis? And actually this is more interesting because it tells me here that once we get people to week two, they're in, we're okay. And so the people who have committed to week two are gonna be the ones that are gonna complete more of the course. But again, I have to ask myself, do I really want them to do that? Do I want them to be able to complete the whole course? Do I need them to be able to complete the whole course? And that focus on retention is still driving a lot of the discussion. So again, we can look at the blobs on the graph. So open online courses, access measures that need to focus on outcomes. And I believe what you need to let go of retention because the outcomes are more important, I think. What a teacher is going to do with the course is more important. One of our learners is gonna do with the course is more important. And this is summed up beautifully by DeBur and colleagues in their 2014 paper that MOOCs open online courses, have loads of data, but it's the diversity of user intentions, their backgrounds, the unconstrained asynchronousity of their activities. What a wonderful phrase. I mean, that's just telling me that there is no way that I as a learning designer can control when and how and why a learner is engaging with my course. But there's still a quite a lot of faith in the data and it's a need to seek out patterns to explain the learning that's going on. And I'm questioning that quite a lot at the moment. I'd also sort of challenge perhaps that some of this has come from changes in the platform that we use. So FutureLearn has an upgrade model. So if you want a certificate, you pay, but if you want that certificate, you also have to mark the steps on a course as complete. So you have to go through and take a little box for our courses to say, yes, I've done that. And to the extent that some course designers will put this instruction at the bottom of every step that says, don't forget to complete because it ticks a metric somewhere down the line. For me, I tell my learners right at the start of the course, use the completion measure because that will help you keep track of where you are.  I want you to make the most of the course. But the platform and the process behind that is slightly different. Success then does not necessarily mean engagement in my view. And I want to move beyond the view that you must complete the course according to my design. We need to allow learners to be able to choose the steps that are appropriate for them. We shouldn't be chasing for correct responses to tests and quizzes. We should be thinking about, well, how do those tests and quiz responses help us understand our learners better? So non-engaging in a course, non-engagement on open online course does not mean a lack of design at success at all. We use the normal model of activity design. We have an activity in the middle that we're designing to lead towards an outcome. We think about how the relationship between learners, the content, the educate, and the cohort all mixed up in that different form of activity. We use the ABC design model. But for me, I would like to suggest that it's not necessarily that activity that's determining the outcome, but the context in which that activity is sitting that determines the outcome for that learner, which means you could have a whole host of unintended learning outcomes, which are more important to that learner than they are perhaps the learning designer, which makes the design challenge incredibly difficult. But we know that learning cannot solely exist online, and particularly in professional development, it sits within a practice context. So our learners should be supported to learn and develop, and we get the feedback at our interval course surveys that they have been able to take the course flexibly around work, they've enjoyed sharing, they've been able to try out new ideas, they've been able to rethink about their practice. And I love that quote because that came from a teacher who said, I've been teaching for 20 years, and I've had my practice challenged and changed through this open online course. So we can look at some of the data though to maybe think about, well, can we maybe infer something from the way that these types of learnings are taking place? And I've looked here at what I'm gonna call course wobble, which is the deviation of a learner from the intended trajectory of a course. So normally you go step one, two, three, four, five, six, seven, eight.  And I find that interesting because that really shows me our learners making decisions about missing steps. Our learners making decisions about, okay, this isn't relevant for me, I don't need it, or perhaps they're saying, oh, this might be interesting, this might be an unintended learning outcome. And we see a weekly spike, so learners jump to the first part of each week, and that's probably triggered by the weekly emails that the Future Learn Platform sends out. But in a three week course, those spikes aren't as pronounced, but we do see some patterns happening at the start and at the end of this particular course, again, due to the interventions we put in to do with synchronicity. And I don't mean synchronicity in terms of people being online at the same time, I mean, things that are stuck at a particular point in the calendar, like our Q&A with an educator. And that forces people to skip ahead. So some interesting ideas are about rhythms and learning routines, but most participants will take the course in a linear pattern. So if we're trying to encourage our learners to identify their own needs and use the course to meet their own learning needs, why are they taking the course in a linear pattern? Very few participants will also start elsewhere and then at the beginning, I had a look at four courses last night, that 10 to 13% of people will start the course anywhere else than at the very beginning. That's a very, very small percentage when we're thinking about people making their own decisions about what needs they're trying to address. Social learning, the platform is all about social learning, but it doesn't necessarily mean it has to be restricted to social learning activities. We know that maybe 20 to 30% of learners are commenting and commenting perhaps has a relationship to completion as it does responding to others, as does receiving a response to others. But of course, as Swinerton and Hotchkiss and Morris said in their paper on this exact issue, that completion and commenting are intrinsically connected. You cannot use measures like this to measure the success of a course. You cannot use measures like this to measure success of social learning because the more you complete, the more opportunity you have for commenting. So the dependent variables to begin with. Most comments are not replies, but does that matter? Anecdotally, the quality of the comments on our courses has improved over the last 18 months and our mentors have said this, but we may change this course to facilitate this. And the comments are becoming more thoughtful, they're longer, which the mentors delight in because it gives them a lot more rich data to play with to extract themes and ideas. And we're getting fewer of the, oh, that's a nice step or great video, getting fewer of those and more thoughtful comments, which is great. They're reflections on that teacher's practice and they're using the platform, not necessarily to talk to someone else, but as a way of capturing their development. So in terms of planning to engage with the professional development as well, the data shows us that if you, as I've said before, if you join the course in advance to the start date, you're more likely to complete. And I'm also looking now at whether there might be something to do with the time difference between steps. So if you engage in all the course in one day, how does that change your completion, perhaps of metrics that we can think of, performance on the course, compared to somebody takes over much longer period. And the outputs we have from our model, our approach and our challenging of sticking rigidly to a single pedagogy, as shown through the outcome surveys. And I have to admit now, these are obviously people who have completed the course or have completed enough of the course to get to the survey point. But we see here that 98% of our participants say there's been a positive impact on themselves, 80% on their students, 63% on their colleagues back in school, which when you consider most of our participants are probably taking these courses individually, with a lot of work we can do there about that blended model of embedding these courses as a structure for schools to use. So impact on their practice, but also improved understanding of the subject matter of the course, changing their practice. The course is relevant to them and it's a good use of their time. How have I met all those differing learning needs through a course that we've designed six month prior? These are questions I don't yet have the answer to. But we know that they like the structure, they like the discussion, they feel more confident. That's a feeling. How can we design learning for a feeling? Seeing the practical aspects and being feeling part of a community. So I'd like to conclude by saying that I think pedagogy should be open too. When I first started this role, I said I had to rethink everything I knew about online learning, social constructivism, communities of practice. They just don't work the same way in a course that is only three to five weeks long and is essentially got an anonymity cloak for our learners. The data shows by the absence of engagement as much as evidence learning paths that pedagogy is open. We shouldn't read too much in the data. We should think about the outcomes and the outputs from that learning experience. So to conclude, I'm very much steered by my own professional viewpoint. What I believe about education, I believe it's very much situated within a context. I believe that social learning is important because it helps you to relate ideas to your context. I believe that discussion is important. But they're my own prejudices about what I believe is right. I always say that our online courses are not online textbooks. Absolutely not. That is not the type of online learning I want to deliver. But how can I convey my prejudices about pedagogy to our learners? And I have to acknowledge that my way isn't the only way. Learners might be dipping in just for five minutes and being able to take something out of that. So I think where we're coming to then is a situation where, that's interesting I've been talking to a little on the slide, coming to a situation then where we really do need to think about as learning designers, letting go perhaps of our own prejudices, acknowledging that we have a professional viewpoint, conveying that to our learners, helping them to understand that, helping them to challenge not just their practice in the form of professional development, but also helping them to challenge what they think about online learning. And through that, hopefully, they'll be able to sustain their professional development as well. Thank you very much. Thanks, Matt. That was a really interesting talk. We have a roaming mic here as well. So please do either post your questions online or raise your hand if you wish to ask a question in person. Matt, one of the things I'm really curious about, I'm just going to use my chairs, to ask you to pass the question. Go on, chairs prerogative. What is kind of the next step in this development? You've talked a lot about what's been achieved. Where do you see this in like two to three years' time? Where would you like it to go? Within our own programme, we certainly need to do more in terms of course maps, which actually relates to the question there as well. And I know that the course maps approach has been explored by Granja Canola as well, in terms of helping learners think about the types of... The guidance they need about how to learn, but the types of activities that are available to them, what type of activity they want to be engaging, how they're going to be supported through reflection. And we put in our courses already extracts from Dino Royal Arts Conversational Framework as well to help them think about the types of comments they should be posting. So that was one of the changes we made recently over the last 18 months is to think, well, can we encourage our learners to be more thoughtful in their engagement? And they have embraced that. In terms of where online learning and professional development needs to go, it is definitely a blended model because we want to be able to equip our online learners with the capability to understand how to use the online courses to support their departmental needs.  How can you cherry pick what you need? So there's almost a level of metacognition that needs to wrap around all this as well. And I think the same with applying university context, where I have worked at university before quite extensively. So I know the sort of the challenges that you face. And I think that the most valuable part of working in a university in a learning town, our role is having those conversations with the academic staff. And you need to make time for that because you understand what they're trying to achieve through their learning. So you need to wrap your professional development around that particular context or enable them to make that those links. Thank you very much. Anybody raising their hand? No, we have some questions online. So how do you encourage to explore different pathways through the courses? Or do you encourage that? So that's what we're gonna be looking at at the moment with the course maps. But as I was putting them together with some colleagues, I was debating, well, should we do course maps based on activity? Should we say, okay, this is a reflection step. Here's a discussion step. Or should we say, is it a video? Is it a video classroom footage? What's more likely gonna be useful for those learners? So it's still something I need to unpick. And I think there's an interesting point there about encouragement of your learners as well, related to that. Because you want to encourage your learners to be able to work their way through a pathway and see the links between courses, but also buy into that idea of sustained practice as well. Yeah, so I presume that's the question on how do we encourage learners? So it comes to looking at the outcomes, it's sharing what people have achieved previously. But one of the key parts of our online courses is the mental facilitation. All of our courses have it. And they do a video diary in some of our courses where they pick up on some of the key points that that cohort has raised from the content. And that's been incredibly valuable for the learners to see the links between the content and their context. But we also encourage our learners through not necessarily reward mechanisms, because the reward is actually the changes in practice in the classroom. So that's how you encourage your learners by saying, and by learners we mean teachers and we mean academics in this context, is by saying, well, look what's changing in your practice, what's changing in your classroom. So having something that's small and easy win to begin with before you go into more depth as the course progresses, that's how we try and do it. Thank you very much. I think we have time for one more question if there is one, or maybe you've met everybody is sufficiently hungry that it's nearly time for lunch. So an opportunity now to ask any final questions. Oh, there's one more. So in your opinion or experience, what can't you teach through online learning? I think that's a nice way to finish before lunch. I think you can teach anything through online learning, but it is the activity that doesn't always work online. We had this wonderful debate when we first started on this journey at STEM Learning with my colleagues and the subject expert team. What works face to face? What doesn't work face to face? What works online? What doesn't work online? And so you come up with a whole plethora of different types of activity that will work in these different situations. But I think you can probably teach anything probably apart from, you know, I won't be very specific here, but there are some relationship building approaches that like human to human development skills that I think you can deliver something online, but you have to link it back to the face to face environment. You have to have that face to face practice opportunity. You can't build relationships with your students just by reading or completing activities in online course. You have to have that link between the two spaces. Thank you very much. And talking of links between people and in spaces, we hope that you will all join us for lunch downstairs now. We have, we're just five minutes ahead of time, but please put your hands together once more for Sarah and Neil and Matt. And also thank you for those who posted.


So I can eat everything in the game? Oh my god, yes! Hello and welcome to the original Manita. That's right, it is Hungry Shark World! Well, it really would be Hungry Shark Evolution. So, we have completed Manita. So what's next? What happens while we wait for Tripwire to create new DLC for Manita? We return to this. This game... Good English, that's fantastic. This has had a huge update. There is new sharks and there is a new map. Like, I can't even explain. This is the best update this game has ever had. I've had a quick try of it. So now there are three new sharks. There's the ancient Megalodon, the Alpha Zombie Shark and the Meltdown Shark. I love these designs. The Megalodon, if I can skip to it. Oh god, looks amazing. I love the jaw, I love the spike sticking out. It looks big, it looks bulky. It's so muscly, it's veins are popping out. The Alpha Zombie Shark has like a soul trapped inside, like a life force, keeping it alive. And the Meltdown Shark, because it's an atomic shark, has a literal nuke attached to its back, powering it. I love it! I love these designs. They're fantastic. I've got, oh yes, you guys, you did an amazing job. And not only do we have these new sharks, there's a new place to go with the sharks. And no, these are a new place. There's a new way to play the game. Oh, I can't wait, I can't wait. So in order to play as these sharks, all you need to do is get like a double exclamation mark, all your double exclamation mark sharks, maxed up, I think. We're gonna actually use this guy. We'll max them up because we have the coin. I've played this game enough in the past. I've got enough coin to max this up. I need to see it completely at its best.  Right, and best thing about this game, it's free. So while we're waiting for an update to come out for Manita, we'll play this. So we're gonna attach some sharks to it. We've got to hide the guard dog. I'm trying to think who's the best one. Trevor is really good for mine, so we need Trevor. Max and Super Steve are really good as well. Oh, no. Oh, fine, we'll get rid of the pappers. We'll get rid of the pappers. I mean, there's loads of pets. All of these, some of them are limited. There's also boosts that I've got equipped, but we don't need these for this. I think there is one more. It's saying that I've got one more equipped. There we go, we'll take that off. Also, we need to put on our force field shield, our lasers, the recycle cannon, and a jet pack. Yes. So what is this? It is the apocalypse. So now there's two modes. There's a regular mode where you can go around, look at the planet. It's a whole new redesign, it looks like it.  I can't praise this update enough. It's amazing, but there's a mode at the bottom which we can switch to Armageddon. Only thing is, it does cost coin. I'm wondering whether you actually get coin from the things that you eat. I didn't check that last time. And this is how you get pills. And in order to get pills, you need to survive a certain period of time and you can earn pills in the game. But that's not all. No. Oh my God, I'm so excited. This is so flipping good. Okay. Oh, look, we've got a special ability. So here we go. So already at the bottom, that's our gold rush. And also a new timer. So when we click over to the next, or we survive so long, we get a free upgrade. Now we can re-roll it to see what we get. 50% less boost. Okay, most of them are bad that I've noticed. Like smaller or food gets less. I think you can get good ones as well. And then you see at the bottom, the longer we survive, the more things we'll get. And this is the Pacific Island. So if you're like, you know this map at all, then you'll be all right. So the longer we survive, the more upgrades we get. And we get to re-roll. Oh, and come on, they're called mutations at the top left. What is it? What's it gonna be? Oh, 30% less health. Are you kidding me? Now we can re-roll it. However, we are getting an advert. I bought a pack to get this megalodon.  I don't have that time.   Oh, this game has everything. Planes, helicopters, drones, all that jazz. Anything you can think of, it's here. Oh, can I do a big? Oh, yes. I can do that. Can I just do that ability whenever? I can just chomp. Oh, that's awesome. Boom. That's so cool. Boom. Oh, yes. Oh, there we go. So we're in gold rush, so we should be okay. Boost drains health. Now that's not so good. Oh, so we can boost unlimited, but it also costs us health. Fantastic. So there does seem to be a variety of mutations but oh, there's a whale and we can eat big whales. This game's got everything. Oh God. Oh God, thank God I had my shield. So really the aim of the game for this is just to survive as long as possible. I think this is like a more deadly version of the map because my God, there's like mines everywhere. But force field shield is survived. So random boost, you know, oh right, so I don't get to control my boost. Oh, that's clever. Come on, boost. You're not gonna randomly boost me. Hello. There we go. Thank you. I'm wondering if that like that roar will destroy mines. Interesting. Bam. Oh God. Oh God, can I, oh God, I'm boosting. Oh, I think maybe it did or Trevor got that. So we need to survive for at least seven minutes. Oh geez. I can't control me boost. Oh God. Oh, there we go. That'll do. Oh, we got an H. Oh, we're tiny shark now. Oh, fantastic. Now, I don't know whether these all stack on top of each other. All we have to do is survive. Come on, can we get in the, come on. Let me get, let me get in the volcano. There it is. I've got random boost and tiny shark all equipped. This is fantastic. At least when I do that like big jaw attack. Oh yeah, there you go. There you go. I don't need to control my boost. Oh, that's a huge whale. Especially since I'm so small. Chisoo shark this name. Oh no. Oh, I can do this. I can do this. So we're getting close actually. Oh, here we go. Perlfish give more now. Oh, fantastic. I have all of the bad mutated. I'm tiny. I have boosting drains my health. Not only that, but I don't get to control when I boost so my health. I don't get to control when my health gets drained. Oh God, this isn't good. So if you want, you could like watch an advert to try and get better upgrades. Oh, there you go. There we go. Perlfish. So we don't have Gemfish anymore. And we all get in coin anyway. So that's good. Oh, Trevor. Oh my God. What? This is new. Oh no. Up is down and down is up. This isn't good. Oh no. I just have to survive. I'm so sorry guys. Make me sick.  I don't stand a chance. Oh, there you go. Oh, and I float up and step down. Would you believe? There you go. Seven active mutations. Geez. Oh, that was nuts. But we got 43 silver award as well as 15 pearls. So we could watch a movie or we can trade 25 Gems, four pearls. So I'm going to do that. Sod it. Give me more pearls. So we got 77 that time. Oh, and also you get to level up your shark, increasing its health and a size.  Let's roll the die. Let's see what we get. I just want to like see what the good ones are. That flip rotation. You get that early on. You're screwed. Three, two, one. What's it going to be? What's it going to be? Oh, come on. Tiny shark. Oh, I'd rather re-roll that. Should we re-roll it? That's who we get. Food heals that. That's not too bad. Okay, that's good. That's good. Okay, let's keep going. Let's keep going. I don't, I never play the Pacific Ocean. So like, I'm not, if this was Arabian Sea, I would be in my element, but this one's not like it. I don't really know if that's very good. Oh, no. I thought I would be able to eat those sharks. Oh, there's a gem fish, not a pill fish. Okay, next mutation. Oh, now we get tiny shark anyway. And I'm not watching an advert to re-roll it, because I make my money from adverts, not you, F-Girls. And people comment on my money this series that, wow, I'm getting loads of adverts for this, but in a good way. It seems like you're getting money, Beaver. Thank you. I mean, I've never like once sort of had like a Patreon or anything like that. I have just now, although, decided to... Because I mean, I don't need it. The money I make for my videos is fine. Like, I do, I don't need... That's why I take no sponsorships either. It's not that I don't... I wouldn't mind taking a sponsorship, but most of them are games that like, I wouldn't normally play. So I'm not gonna promote it? Like if Hungry Shark asked me, oh, if they pay me to promote something, or they want me to promote something, I would do it.  They're like, we could just release things and he'll cover them.  Ooh, that was like a show of cl- Oh, well, I completely destroyed that show of pamphlet.   Okay, what we're gonna get? What we're gonna get? Slower boost. Okay, this is all not too bad. And I'm not gonna reroll just yet. We're in another gold rush. Now I can hold this coin, although we're not getting apparently. That's good. Why is my gold not going up? Oh no, there it is, 600. Okay, not bad. Oh, we're in mega rush. This would normally be amazing, but it really isn't, because we're playing a pocalypto, whatever it is. We're playing a Mel Gibson movie. Woo! At least we can still boost. And it's not random boost. Oh God, what's it gonna be? In curry rotation?  Don't be random boost. What's it gonna be? What's it gonna be? Ah, stop it! Oh, I wanna reroll it, but you know what? Fine, fine, whatever. Whatever, whatever. So we got five actives. I think the most we got is seven. So can we survive over seven? You know what I'm gonna do is actually I'm gonna go back across myself this way.  Oh, the boost is saving me. The boost is saving me. Ah, this is gorgeous. Oh, go away jellyfish. Oh, there you go. Look at Trevor saving me. Oh, we got another mutation incoming. What's it gonna be? Please be something good. Human, human invasion? Humans are everywhere. What, in the water as well? That, I mean, that looks good. Oh God, does that mean divers? Oh, it means divers. Oh, that's not so good. That means harpoons everywhere. The pets help me out. I'm the same size as you. Oh, I'm small. Is it hikui? Is hikui in small Japanese or is that? I don't know. Might mean small person. I'm not sure. Oh God. We need to get over this. Oh my God. So instead of fish, there's like mainly humans. Oh, that's not good. Oh, here we go. The seventh one. What's it gonna be? Ah, no. Can I re-roll this one? I really don't want to re-roll this one. Oh, oh no. I'm stuck. I'm literally, I'm stuck with the inversion. I don't think, oh my God. I just get like submarines everywhere. Oh God, I'm being battered around so much. I can't do anything. Oh, and I don't have boost.  Oh, look at that. Isn't this great? No sound or anything. Did that work? Oh, it worked. Oh, it worked. Flea, flea away from these. Oh my God. Oh my God. Get away from the submarines. Oh, we got a G. Oh, that's good. Oh, this is nuts. No, I can't do this. I can't, I can't take these on. I watched all that advert for nothing.  Oh my God. So that's what they mean by humans everywhere. We were so close to the final mutation though. Silver tier. Damn. Okay, so we got 15 pills as well as any other pills we would have got extra. Oh, and we level up. Look at that. We're getting more. That's the thing. With money, you get so far and you just cap out. With this, you can upgrade and then also keep on growing and getting more health and stuff. And it just feels like there's more of a challenge. Is that a challenge? When you've got, like, I die. My shock can actually die.  Oh, whoa, I get times three for killing things with the... Ooh, that's really good for points. Okay, our first mutation. What's it gonna be? Small shock? Boost drain. Boost drain's health. You know what? I've got one free re-roll. I'm gonna wait till we get in vision or something because that's the one that does me over. So I'm gonna wait. I'm gonna wait a little bit. I'm gonna go left first as well this time instead of going right so that leaves me with enough time to come back over myself. So this is fine. This is fine. It doesn't matter about points in this one. It kinda doesn't matter about coin though. Okay, okay. I like this. This is really good. I can't wait to see the apocalyptic maps of the other ones as well, what they look like. Oh, food heals less.  I'll take that. That's not the worst one. It's the inverted one and the random boost that kills me. So I think I need to wait for those ones to come by. Oh, wait, is this the end of the map? No, it's not. Oh, it is. Okay, backwards this way. Oh, we need to do survive guys. Oh, see that as a helicopter taking them. They're beautiful. So if you are wanting a series with sharks, highly suggest checking out the hungry sharks. Mutation incoming, what's it gonna be? Oh, tiny shark. I'm okay with that. I'm okay. Look, it doesn't matter about the size of the shark. It's what you do with it. All right, God. Woo, look, Glowell has like destroyed this place. There's toxic chemicals everywhere. Oh, this is awesome. Woo, trying hero juice. Like a G fuel kind of thing. I want the pearl fish because then gem fish turn into pearl fish and we can get loads of pearls. If you get that early on, oh, you're sorted. Giant squid, not a problem. Mine's not a problem. Still jellyfish are. I don't know why jellyfish is still a problem. Okay, here we go. What's it gonna be? What's it gonna be? Random boost. Do we get the reroll on that? I'm gonna get, oh no. I'm gonna save for inversion. I'm gonna save for inversion. That's the one that really kills me. I mean, the boost is good for when I need to kill submarines and stuff and stuff like that. But other than that, I think I'm not too, I'm not too bad. Oh, look at all the gold. Gold, boys. Collect all the letters and oh, that was lucky. Yeah, I need boost to get through barriers. I forgot about that. Oh, there we go. We're into silver now. Come on, come on. We can do this. I can get gold.  Pollution? Junk is everywhere. We've got recycler, so that's not too bad. We haven't seen that one before either. Oh God, oh God. Oh, the boosting. Oh, the boosting. Oh, yep, there is pollution everywhere.  It automatically fires when I see human beings. Look at that. It's just raining my health. I need people. Oh God, I need fish. I need stuff. Oh, there we go, Gold Rush, thank God. Oh, however, though, when I'm boosting, I'm not really gaining much health. Oh, there we go, there we go. That's what we, look at all these shark infestation sharks everywhere. Oh, I'll take it. Yes, sharks give loads of health. Look at that guy. Doesn't stand a chance. That's way better than the human one. Oh, I'll take it. Yes, I'll take it. Oh, this is in a good place. Oh, blob fish. Blob fish are good. Will you eat the blob fish? Thank you. Stop being caught on stuff. Right, okay, next, oh, we're gonna do it. We're gonna do it. Guys, I feel it this time. I feel it. We're gonna survive income and mutation. What's it gonna be, what's it gonna be? You know what, I'm gonna spend this time just blasting through the world. Inverted controls, oh, we're re-rolling that. Oh, that's horrendous, eat anything. Eat anything but health drain is doubled. Oh, that's the worst time. That was the worst. Time for that to happen. So I can eat everything in the game? Oh my God, yes. Oh, but the health drain, the health drain is doubled. The health drain is doubled. I can go through the volcano now, though. Because there's loads of mines here that I'll be able to eat.  Come on, boost. Boost, you're bugging you. Come on, we can get to gold. We can get to gold. Boost, somebody boost. Yes, there it is, there it is, there it is. Oh, yes, I can eat everything. Oh my God, what, 50% less boost? Oh, well, that doesn't even matter because I have no control over that whatsoever. Oh, I can't get through these. Oh, I can't get through these. Okay, I'm guessing I'm gonna go over here until boost. There it is, did I boost? Think I boosted in, yes. Mega rush, oh, we're safe, guys. We're gonna do it. We're gonna get into gold. Oh, there is platinum, though. There is platinum. I can't reroll unless I wanna watch ads. So hopefully there's no inverted world because that would be the worst. Please don't invert my world. Oh, okay, that's fine, that's fine, that's fine. That's fine, I can do this, I can do this. I can get to gold tier or whatever it is. I can do that, that's easy. I mean, I'm pretty much eating everything. So I don't have anything to worry about. Oh my God, this is massive in my head. Oh, Lord, where am I going? Where am I going? I'm in gold rush, thankfully. I don't remember this. I'm gonna stay near the surface. I'm gonna stay near the surface because there's gonna be people. There's gonna be people, right? I mean, one would think, right?  No! Well, you know what? Oh, there was pearlfish. This, guys, this is fun. This is what I've been waiting for, something. 25 pearls, and we leveled up to level six. Oh, maybe even level seven. Oh, look at all the little bits of health we're getting. 505. What's war on 100, 100, 158 pearls. Right, so we're getting slowly better. So to get to the next one, to survive seven minutes, 30, we need, well, we need to survive seven minutes, so we need to get 75 pearls, but maybe we can get lucky. Okay, so the worst one to get, I feel, is either inverted controls or inverted world. They are the most horrendous to get. Good Lord. Food heals less, that's fine, that's not a problem. This is why I died right here. I didn't know where I was. Um, num, num, num, num, num, num, num, num, num, num. Eat it all, it's so satisfying. And I think that's what Manita took from Hungry Shark World was the fact that eating everything and surviving and just killing everything is the best thing in the world. It took its missions from Jaws Unleashed. It took its open worldness from GTA, but it just needed a little bit more. And hopefully they can fix that with a slower boost. Oh, do you think that's a problem? Oh, that's good. Okay, so we're getting some good free ones here. Oh, yes. Yum, yum, yum. If I can get slower boost and half boost and random boost, that's fine, that's fine. And what I really need to do, and Tiny Shark, Tiny Shark seems to be one you get straight away. I'm assuming getting inverted world is not quite or maybe even impossible to get when you're in the lower ranks, because that is super difficult. I like the idea that you get one free roll on the random mutations. That's awesome. Random boost? Yep, that's good. I'll take it. I want to eat everything. I want to eat everything but health rain double. That's fantastic. Oh my God, I love it. I love it so much. I mean, this is what the boosts were, right? And then EFGO were like, well, how can we make it better? However, we just have random, yes. And it doesn't cost them anything, but there's bad ones. Genius. Trevor, Trevor, Trevor, Trevor. Destroy the mine, please. Thank you. It did hurt me though. Ooh, what's that? Even though, what is that? Hold on. Oh, well, it's gone whatever it was. Right, so I can't control my boost. That's not too bad. Okay, Tiny Shark, I'll take it. So far, we've made it into Silver Rank, where we're almost at Silver Rank, and not a completely bad one. Not a game breaking one at the moment. Oh, my force field shield's wearing out. We're on Yeller. We're on Amber Warning. Oh, no. Ooh, we're on Orange Warning now. Ooh. So, oh, shall we visit Kempie Bass's grave? Oh. Poor Kempie. There we go, Silver achieved. So, we've only ever gotten one higher. And there's only one mutation here, it seems. Come on, please be a good one. Don't be inverted world. 50% less boost. That doesn't seem too bad. However, I am doing awful here. Oh, there we go. Blobfish was good. Blobfish give you low, and Trevor's helping me out loads here. Thank you, Trevor. Boost, yes. Okay, I don't know where we are, but I'm hoping this is a good place. Oh, this is good. This is good. This is where loads of stuff is. Oh, no, we do have another one. We do have another. Okay, gold, and still got my reroll. Still got my reroll. If I get inverted controls and reroll it into inverted world, I'll be so annoyed. I'm trying to destroy it. I can't, however. There we go. What's it got? Oh, boost drain's helped. That's a bad one, but it's not the worst. It is in game breaking. It's okay. It's okay. Oh, come on, we can do it. We can do it. Where is everything? Where is everything? What? I died, because there was nothing to eat. Okay, I'm gonna, you know what? I'm gonna spend gems, because I sold the adverts. You can take my gems, F-Gold. You're not getting me ad revenue. Oh, there is nothing. What is going on? Hello? Anything? Please? Thank you. Good Lord. It was like, you know what?  Okay, so what's this? I'm already anticipating a reroll. Pollution. Okay, that also isn't too bad. I guess we could have tried to reroll it into something good. There really isn't any good mutations. There doesn't appear to be any good mutations. Okay, well, we may as well go over here, because there might be stuff. Yep, there's the parrotfish. Ooh, more lovely jubbly. More parrotfish, more junk. We're in Gold Rush. Finally, some saving grace here. Get the health up.  Okay, so there's no humans. I don't think there's pollution everywhere. So another mutation. What's it gonna be? Shock infestation. Yes! Yes, I'll take it. Oh my God, yes. Please. There, there they are. There they are, fantastic. Oh my God, are we gonna do it? Are we gonna do it? Please tell me I can do it. Come on, sharks, where are you? Where are you, sharks? Come on, shark infestation. There it is. Come on, we can get it. We can get it. We can get gold. Give me gold. Give me gold.  Lots of things. Health draining quicker, because I'm boosting now. Health draining. And I'm not eating things. Incoming mutation. Please be something good. Please be something good. Oh my God, what is it? What is it? Pill rush. Is that good? Oh, I'm in the gold rush. So does that count if I get pills from this? Instead of, instead of thingy? Oh, please tell me I get pills instead of whatever you call it. Oh God, oh God, let's go this way. Come on. I have like, oh, I've got full health. This is good. Are we onto it? Are we onto the max? Can we do it? I just need to eat everything. Eat everything. That's what I need to happen right now. We're not doing bad guys. We're not doing bad at all. Begin by the thresher shark, which isn't very good because he doesn't let me. Once I'm being eaten, then I can't eat. Okay. Oh, eat anything. Yes! Yes, this is it. This is what I'm wanting. This is it. Oh yeah, and we've got pills. We've got 100% pills. Oh, that's crazy. Yes! Oh my God, yes, keep going. We're on mega rush. Oh my God. Yes. Oh, let's go. Let's go. We need anything. All we need to do is survive. That's it. All we need to do is survive. I mean, that is the name of the game, really, anyway. Oh, this is got incoming mutation. What's it going to be? Don't be inverted controls. Don't be anything like that. I'm going to re-roll you if you are. There it is. Thought so. OK, let me re-roll. Let me re-roll. I'm re-rolling that.   Oh, God, pull, pull fish. That's good, right? I mean, I'll take that. Is there anything up here, though? Is there anything I could eat? Oh, God. And I'm on to the mountain. I'm dead. Yep, there we go. Oh, that was so good, though. Oh, my God, we were halfway there. You know, no, I'm doing it. I'm saving myself. Oh, my God, this is going to cost so many gems. I just want to get them once. I just want to get to the end. Oh, Lord. We need anything. However, there is nothing. I just need to get past this. I need to get past the... Oh, inverted. Now we're dead. Now we're flipping dead. Oh, come on. It sort of inverts my controls for a while as well, which is a bit of a shame. Come on, boost. Boost, you buggy. Oh, my God, we're so close. Sonnet, whatever gems it costs, we're doing it. Inverted world. That's fine. That's fine. I will find a way out of here. Yes, here we go. Come on. Come on, we can get that. We've already got 100 odd, like, pearls. 172 pearls. What happens after this? We've got 12 active things. I want me blummin' em. I want me pearls. I've only got 175. I want more. Give me more. I think that's my total. Extinction survived. Oh, oh, that's it? Oh. Oh, my God, if I spend 55 gems, I can turn it into 200. There you go. There you go, 250. We've got 268 pearls earned. That is 1 tenth, I think, of what we need to unlock the next giant shark. Oh, that was great, though. And was that all of the mutations? Because I don't think it was. I want to know what there is. Oh, it's so awesome. Anyway, guys, I'm going to leave it there. But if you enjoy this video, also check out Hungry Shark. It's free on all app stores.  But I think there's more enjoyment in the game in grinding it and earning everything yourself. But if you enjoyed it and you'd like it until next time, I'll see you later. Oh, bye-bye.


If you have ever drawn out some water from a well through a bucket, you must have felt that the bucket full of water feels lighter when inside the well or inside the water. But when you pull it out of the water, it starts to feel heavier. Why does that happen? We can even experimentally verify this. Here I have a toy and I have attached it to a weighing scale, which can tell me the weight of the toy. Right now the reading is 0.16. But notice how the reading decreases when the toy goes inside the water. There you go, look at it, how it is decreasing. And the toy also starts to feel lighter and lighter.  So, here I have some water and here I have some, here I have a glass slab. This glass slab has a weight. Now, weight is a type of force which acts in the downward direction. Heavier objects have more weight, which means they will pull my hand in the downward direction with a greater force.  As soon as I do that, I feel that the glass slab has become lighter. So, what is happening here? Well, it turns out there is a new force, there is a new type of force that acts on any object inside water. And that force is called buoyant force. This force acts in the upward direction on the object. Now, as a result of this, the total force which acts in the downward direction reduces. So, now less force is pulling my hand in the downward direction and the glass slab feels a little lighter. So, buoyant force acts on objects when they are inside water. Even if the object is not completely inside, still buoyant force will act on the object. And the object will feel a little lighter. So, can you now explain why does the bucket full of water feel heavier outside water and lighter inside water? Try explaining it.


Okay Welcome to all of all of you We would like to tell you something about philis automation and what we accomplish with yahooke in open stack Let quickly introduce we have Felix here from schwarz IT. He's a cloud engineer. We have Jonas here, which is a Team lead of cloud development at cloud and heat and I'm Marvin. I'm a product owner at stack it And we would like to talk about what is decade What is the schwarz group then show you some use cases for what we are using open stack and the growth of our cloud platform and then we talk about yahooke To talk about the schwarz group first. It's one of the Companies which has multiple sub companies in it mostly famous for the retailer brands Lidl and kaufland We also have a recycling company called pre-zero and we also have a multiple production facilities with Bongellati, which are doing ice cream or we have like the emigre who is Refilling water bottles like for little for example and within this group Felix and I are part of the schwarz it which is like the let's say the the heart of all of this where all the IT solutions are made developed and operated and Within this schwarz IT we created a stack it which is a sub-print in the schwarz IT and There we created the Stack it cloud which we offer for for the dach market in Europe and Our target customers are mostly like these small medium enterprises in the dach market and For doing such Challenge we think about it's easier with a companion and there comes cloud and heat in place which helped us to developed yoke and You know, we'll tell something about yeah need just a few words cloud and heat is a small company which has started about ten years ago with the mission to provide a sustainable and holistic cloud stack built upon innovative water cooling and Converged infrastructure using open stack as the software stack, which is why we're here after all and we have recently also expanded in the into the managed Kubernetes Yeah market we have a partnership or rather joint venture with the sequel neck Which is the sickle stack which is a hardened version of open stack. They are also some people from that here and Yeah, we're supporting the schwarz in running their Open stack cluster and developing yoke Okay, then I would continue a little bit and talk about our use cases So on the one hand we're using the open stack environment that we have Inside the schwarz group a lot and on the other end a little bit for external customers The nature of providing a cloud service is that we definitely don't know all of our customers at all Because from an infrastructure perspective, they are all just VMs But we have two big users that I want to highlight a little bit We have a use case named app cloud based on cloud foundry where a lot of internal and extra Internal apps are running cloud foundry is basically allowing you to ship applications Using some predefined build packs. So you don't need to care about Docker files deployments or things like that you just push a code and System handles the rest this whole environment is Quite or consists of two environments in total around three thousand virtual machines Six and a half thousand vcp use and 20 terabytes of memory basically amounting to 25 to 35 percent of our whole environment depending on if you count instances of vcp use if we take a look at the other use case It's the second Kubernetes engine That we don't provide one big Kubernetes cluster, but we provide our users with individual Kubernetes clusters for each use case so you can go there and say I need a Kubernetes cluster now say what kind of workers you want and get it In total there. We are running 250 clusters with 2500 instances 10 song we see views and 25 terabytes of memory And with that they are definitely our largest shareholder if you count based on vcp use Over the last year. We had a lot of growth there We grew from 13,000 vcp use to 24,000 mostly based on the Schwarz Kubernetes engine and Also increased our memory Footprint that we gave to our users by 60 percent and yes You can see that our open-stack Prometheus exporter died in the middle of the graph And Now the question is how do we run all this? We do that using the oak and I'll hand over to you for that. Yeah for this part We want to start with a little bit of motivation because When you in these days come up with a new open-stack lifecycle management You need to have a really good reason for that and We hope we have that So we we made three core decisions here The first one was we wanted to run open-stack on Kubernetes So have Kubernetes as a layer below firstly because declarative things are really nice They allow you to define how your infrastructure is and then there's a thing which ensures that the infrastructure actually is how you want it to be and In addition to that we need this gives you some real good goodies to which are useful in the open-stack world for instance Having replicas of your API's and being able to scale this up easily That usually doesn't come with a let's say classical lifecycle management like Ansible or so It's not easy to say. Yeah, I want like 10 replicas of the keystone API running on this note or I want Five I don't know. That's generally not that easy. They are not that flexible So Kubernetes is much more flexible in this regard and you get the replicas you get networking and you also get Liveness probes, which is of course really important to run a reliable service to have things which are not working Recycled taking out of the rotation taking out of the load balancer So that API requests are not failing all the time or not failing because of the temporary outage or whatever As a bonus when you run stuff on Kubernetes It's easier to port it to other platforms and you get easier testing because all you need is a Kubernetes cluster and you Can get that for instance from a galina instance or from a cluster API provider You just can set it up and then you have that cluster for testing Yeah, which improves development drastically in our classical lifecycle management That was the major problem that you always had to get like physical hardware and have the installation procedure Which is really daunting and finally you also get observability from Kubernetes because you have all the status and it's continuously Checking the status of services. So we can monitor that more easily without having to write specific checks for each service So that is how we oak was born it is yet another open-stack lifecycle management on top of Kubernetes You might get the drift where the name comes from by now It is a Complete tool with the holistic background to cloud and heat has so in installing and upgrading from the bare metal on using open Open-stack ironic as the bare metal layer so to speak then installing on top of that Kubernetes using Ansible and then on top of that open stack How do we install open-stack? It's not with open-stack helm This is the second decision and Drawing back to the title feel as automation automation being the keyword here We took the idea from Kubernetes to have control loops which continuously check the state of the system because in an unintended System the breakage can only increase So we have these operators which are for each open-stack service one or more Which take the declared state of the service like I want a keystone here with three API replicas a database We just backed up regularly and so on and so forth And they check the state of the cluster against that So is the deployment using we are using of course communities deployments Is the deployment set to three replicas do we have the database is the database healthy are the users created in the database and so forth and Have checking this regularly and also based on watches to orchestrate the entire Kubernetes wallout and also to handle Kubernetes life-cycling like upgrades in this fashion Taking glance as an example if you know Kubernetes, you know what this is. This is a Custom resource where you where we define the glance deployment so to speak and because we have operators we have for each Yeah, open-stack service component a separate resource type Which is referred to by the kind and then you put in all the specification of the Which makes up a glance like how many replicas does my diet database have how many replicas does my API have The back end configuration the policies you want to inject your target release of obviously, which is yoga what else would it be and Definitely no Queens glasses in production. I assure you or no pike either or rocky or train. No none of those it's all yoga and And you can of course also inject arbitrary config to some extent I will get to that in the decision 3 And also self configuration and that is then fed into the operator which reads that and gets notified on updates So when you edit this it gets a watch from another an event from the Kubernetes API and then it ensures that your Specification is actually put in place in the cluster During all of this it goes through this fun graph, which I won't go into detail. Just it's just an illustration So for instance, we have the config generation step below here, which feeds which has some input So this is really a state machine so to speak or a dependency network rather So we have for instance the config thing down here, which of course depends on the keystone information It needs to know where keystone is to get keystone user credentials It needs to have its database to generate the database URLs which grants needs the user password and so on Which is it's just sent fat into a database sync job and to the into the meter depth loading of glance Which is also a separate job and only when that is done It will install or update the API deployment and then it will create the Kubernetes service and then from that we have the service monitors, which is the monitoring and so and so forth and the point of this is that this dependency network is Directed as a quick graph, which is nice because that means we can go through it in a defined order and have it Have this be our core of the reconcile loop and we don't have to write all of this by hand That would be crazy But instead we have a declarative approach within yahook itself in the code which allows us to write this in a more sensible manner and The loop then on every iteration checks where the things are which have divergences for instance When I change the configuration then this will mark as unready and the configuration will be applied And then in the next iteration it will note. Okay, the configuration is changed I need to run deep DB sync and then until it eventually reaches the API deployment That also means if the configuration is invalid it will never reach the API deployment For instance, or rather what means what does invalid mean? This isn't free Of course, we introduced Kubernetes in the stack. We introduced operators. That's a lot of moving parts So we have to eliminate complexity somewhere else. Otherwise, this gets completely unmanageable and configuration management is hard and For glance it may be rather easy because you generally only have like this glance config and then you're happy But for something like neutron or nova where you have might have no specific configuration where you might have to inject for instance On your gateway notes, you need to inject more VLANs, which are provisioned by L2 To make them accessible your provider networks. They all need to be configured, but they may be not specific So the question is how do you merge all of this together without being completely? Lost when you need to track down where is this incorrect configuration coming from and For that we use Q-Lang, which is a framework for yeah working with data And we use it for you working with Configuration and Q-Lang doesn't allow you to have two conflicting values. It's that easy So if you have two inputs to the same configuration and they say different things Then it will just error and then the operator will halt and tell you well That's not well that you need to fix that and I won't hand it down to any service before you do That's just one example of complexity elimination But this is a really nice one because I think this is one of the Bains of other configuration management tools and other lifecycle management tools that you have Multiple inputs to the same thing and if they differ well one of them wins But it's not clear where that come from it might be clear when you know all of the inputs Which will win but if you look at the output it's hard to track down So if you have an incorrect configuration that it's hard to track down This isn't the case here because only one can be valid and otherwise it will tell you there's a conflict Yeah, that was our road to your oak there's obviously a lot more to it in the end a Slot is only that long although. I think we do have some time left So if you have any spontaneous ideas you want to talk about Otherwise, we will have you have us guided by questions that works So maybe just a quick rundown from then at 20 2018 the stack it was started with the open stack endeavor then using a different platform I might probably not name 2020 support by cloud and heat was bought in and we have started supporting them with operating the platform and in mid of 2020 we had the initial idea of how that we should do yahoo and the concept and Nearly a year after it was open sourced and in September 2021 I think you pulled glance into production and that worked nicely and then yeah more stuff is pulled into production They hit 100 compute nodes deployed in parallel the other day, which was nice And yeah, it's who you are asking if it was nice, but yes It mostly worked there were some hitches along the road, but that's what you get when you run a thing first time Yeah We have a website for yahoo It's all open source on github.com and That's the main deal and we are now open for questions. I think yeah For the questions there would be a microphone Otherwise, I will call you out and then you can just raise your question I will or any of us will repeat it and then so that's on the screen to Yes, you were first I think Yeah, it's okay It does you're right about that so the question was How does the bare metal part of yahoo quirk essentially so especially day two and storage So I will maybe go about day first a little bit to get some context and then we will talk about day two So I already hinted at we are using OpenStack ironic obviously We have a small component called the metal controller which essentially ties together a net box OpenStack ironic and HashiCop Walt HashiCop Walt for the IPMI credentials and some onboarding of the nodes Net box to know which IP addresses they should get and OpenStack ironic for the obvious part actually getting the thing to do what it's supposed to do That then it gets a config drive which essentially Contains I think nowadays the full ansible. Yeah, the full ansible repository which deploys Kubernetes As well as some configure additional configuration like not the join token that's fetched from Walt But you get the drift and that is then run on the first run on the first boot using cloud in it and the run script Then it is joined into the Kubernetes cluster and from there on it is mostly Kubernetes things Node upgrades is currently not fully defined within the O project. I'm not sure what you are doing We are doing it with yeah kind of running ansible. Yeah on top of the notes. Okay, you do the same thing So the Ansible playbooks exist for Kubernetes and they can also be used to upgrade the nodes and also upgrade the Kubernetes clusters There's not much automation in that regard. That's something which is being worked on also to integrate And to communicate with the operators to let them know beforehand so that they can evacuate the workload when you're starting to Yeah, upgrade the node. So this is also maybe one key thing about you that the operators are Communicating about things like this. So if you change the layer to configuration for instance It will generally proactively try to evacuate the node first if compute the workload is running on it because it might be disruptive And it tries to avoid that Regarding storage right that was the second part of the question and Brooke obviously We are not tied to theft though. They are running this with net up Yeah, just obviously completely different, but the ansible playbooks for the communities class I also cover Installing Rook if you want it optionally otherwise you have to integrate differently You can run it next to it doesn't matter just as long as it's reachable And basically what you saw as a configuration for theft is basically in the root format So if you use Rook you can directly link to it and the format just yeah So it expects kind of the Rook format, but you can use different stuff to generate the same thing Yes, you were next I think no behind you. Sorry Sorry, I didn't catch it. Can you try louder? Yeah, that's easy. That's or it was a fun story to begin with The question was basically how we handle the conflict but potentially between Kubernetes networking and OpenStack networking We in our environment were using IP tables firewalls rules previously and try that out and that didn't end that well the solution there is basically to switch to the open-v switch firewall driver and Then basically avoid this whole kind of issue. Yeah, just don't use IP tables in OpenStack The question was about the format of the CRD's and how they are getting there the CRD's are basically handcrafted at that point in time The configuration validation and afterwards is going against the conflict definitely defined by also conflict Maybe to be more specific The TRD's are not completely handwritten that would again be a daunting task But they also generated via Q-lang it can also output JSON So we are effectively just putting snippets together of these here these so like we have the API section there We use it for all services obviously, so that's not always copy-pasted, but it's Composed together, so but yeah, so and yeah the other part is Oslo Which isn't in the CRD's that there the CRD's are actually freeform and the generations the Validation against Oslo is only happening in the operator So you can actually apply an invalid configuration for instance a list instead of a string But the operator will then reject it and tell you that doesn't work. You need to change it The question was whether a CNI is used or whether we are using host networking exclusively If CNI is used that works planetly actually our default is Calico, but we are not sold on that I think you can use any CNI which provides the basic stuff Yeah, but there's also the side effect that potentially weeks LAN ports might overlap between the CNI and what mutual does Then IP tables rules to do migration might rescue you, but they are horrible Do we have any further questions? Yes The question was whether the yoke open stack layer so to speak can be used standalone without all the stuff below that The answer is clearly yes. So for instance, we run our CI on a gardener cluster Others use well, okay, that's again Yoke Kubernetes. So yeah, that doesn't count But it isn't tied to the Kubernetes layer. So it just needs basic Kubernetes functionality We have a list of requirements in the documentation actually, but it isn't that steep So you need an ingress third manager and I think that's storage class And you need the storage class, but other than that any Kubernetes cluster will do So I actually running on k3s on my laptop that works as well. Although it drains the better record fast Anything else Okay, if there's anything and not nothing else then I would say thank you You can meet us here or in the yahoo channel on the OFTC ISC, right? That's OFTC. Yeah, and Thank you for your attention. I guess And also if you would like we have stickers here