I'm actually surprised so many of you showed up this early after yesterday's great party. So my name is Leon. And today we'll be talking a little bit about testing, monitoring, production. I'm really awful with titles, as you can see. So I'll look at this slide about me. This is me. I'm pretty sure that was right after I wrote my first Hello World. I've been doing this long enough that I hate pretty much every technology equally. And for many years, I've been working on MUTI where I get to get my hands dirty with all the really big systems, which is really cool. I will have the slides posted on SlideShare, maybe with different background color. But they'll be there. I'll tweet it out. And like Jason mentioned, spend a few moments and moments. You have speakers in there. I just want to say hello to everybody. I had a lot of great talks yesterday, right? There you go. You have a round of applause for some of the more first speakers.  All right, so let me, like, have a little bit.  I got the following. And before some people rise to stage, show them it's my business. I think testing is absolutely important. If you don't have some sort of a testing in your pipeline, then you're doing something really, really serious wrong. But I also think that testing is completely not enough. Show of hands and have at least one of those in your development. I'm going to use to wake up in the morning because of me. So testing can give you full services. You've got all my tests passing and pushing to production. That's it. It's the term of the system, right? You're testing. It's rarely happens like that. So there's a couple of problems with testing. First of all, of course, it's a data problem, right? This is a whole, it's working on a lot of damage. No matter how much you try, I had to start with people many a time. You cannot revoke the production. You're either going to have the one extra record that's going to break a camel's back, or you're going to have some weird traffic better than production. Or you're just not going to expect certain inputs that are going to come to production. By the way, I'm going to have a lot of who here knows who to pull from a lot, so I don't even know. As long as you know the last name in the world. How many of your applications have been tested to support this? Which, of course, brings me to the second problem, which is probably the biggest problem. There's going to be that next user who comes in and goes, oh, I wonder if they can make this even worse. So how many people here have played World of Warcraft as well? Come on, let me try. There's more of us here. So for those who don't know, World of Warcraft is a massive, most popular online game. It used to be one big mess, one big, most popular, and now it's probably still in the hits. It's had this again, right? You get to play for orgs, for dwarves, elves, wherever you want to be. And they're also known for a lot of really, really interesting bugs that they introduced. Those who play actually know. So anybody know what corrupted bug is? Yep. So to get the game interested, the developers keeps on coming up with a new inventive content with different mechanics. Can you hear me now? Oh, I'll try to speak a little louder. Is it better now? I'll take what I can. Thank you. So anyways, they tried to come up with more and more inventive ways and different game mechanics to keep the things interested. So with one page, they came up with a boss who would put a curse on a player in the group. And it would do a lot of damage, but the caveat there was if the players stood next to another player for a couple of seconds, the curse would jump to that player. So if you're not careful enough, basically you wipe the whole party. Problem is the developers did not account for users. And what does the first user do? It's try to exploit anything they can. So I believe day two of the boss being in production, one user got that curse and teleported to town. Mechanics of the curse were that the curse would jump to any player, either human or NPC. This is what major town looked like. Bigger problem with that is the curse would also jump to more people if somebody died. And since the town had low level creatures, it would basically occurs that it was spread across the whole realm. They had to do a roll and restart of the servers just to get rid of this bug because the servers were empty. You could not survive anywhere unless you were hidden deep in a forest somewhere. It was actually defined as an in-game plague because it was really the same effect. So short of that, of course, there is other factors why, now going back to our original topic, why testing is inefficient. I mean, there is always a lack of foresight. I mean, Y2K is probably the best example of people not thinking things through. There's always too many use cases that you can't or you don't check for. And I'll give you the last word of work of reference. So when they released the first big dungeon for high level players, about six months after they started receiving the bugs, the some players could not enter it. And to enter the dungeon, you had to jump through the window. And after that, you appeared in the dungeon. And some players were reporting that they don't fit through the window. But that's not all. So after some debugging and looking through it, it appeared that the only players who couldn't do that are female torrents. And torrents were a class of minotaurs, bulls, who were supposed to be wise, one with a nature. And female minotaurs were their counterparts, which were effectively walking, talking cows. And the reason that bug was not noticed for six months after it went to production is because nobody wanted to play a walking, talking cow. So nobody actually got it. And that's what millions and millions of people played. Got that specific combination of character high enough level to try it out for six months. And of course, there is a chance to assumption, right? How many of you had to develop something for a use case when they launched the production? And then business said, nope, we're doing something completely different. So to summarize a little bit, testing is great for no notes, right? When you're known for something, you can test for that particular case. And you're great. That's what testing was for. Testing is OK for known unknowns. Like, to a certain degree of certainty, you can plan for problems. But testing is really bad for unknown unknowns, right? You can't really test for what you have no idea. I mean, you can't test for user stupidity. Ha-ha, finally we get to the monitoring. So why do we monitor? I mean, in case you haven't been listening for the past 10 minutes, is because testing isn't enough. But seriously, we monitor for a whole lot of different reasons.  In anybody who tells you they write a buck-free code, you can hit them with a stick, because they're lying either to you or to themselves. Systems become more and more complex, and you want to make sure you keep track of every moving part. There is always external dependency worry, which people tend to skip. How many of you have dependency on a third party service, whether it's major one, like a full integration of Salesforce or just using Facebook Connect? And of course, it is all source of other reasons. But in reality, it can be summarized in one thing. We monitor because things change. And when things change, they usually change in production. So what do we monitor? What do we monitor to try to solve our problem of things breaking in production? Everything, right, would be the short answer. But more specifically, we monitor systems. We monitor databases. We monitor applications. We monitor integration points, yadda, yadda, yadda. I mean, how many of you monitor all of the things listed here? Oh, wow. We've got to have a talk. All right, so question, is it enough? Right, is what you monitor enough to keep you asleep at night? Or is it too much and you get way too much traffic and waste many alerts, and they don't let you sleep? So the question is, what is important? What do we want to monitor, really, that will help us keep the systems up and running and quickly react to them without overloading with information? What do I learn? So Twitter is a perfect example. Well, first of all, the notorious for breaking things. But a little while back, they had an interesting bug where you could go to the website or through the client. You could submit a tweet. It would say, great, your tweet is submitted. Everything is great. Site is up and running. And your tweets go into Dev know. Right, so from my operations perspective, servers are up, APIs are up, they're returning 200, everything is working. But in reality, it's not, right? So all the systems checks are fine, but the business is failing. By the way, that also goes for unit tests. If you rely too much on unit tests, it has the same problem. If all your unit is surpassing, it doesn't mean your stuff is working. But anyways, this is problem I favorite quote ever. It was told me years back by a CEO of the large company when we were talking about some of the technical stuff that we need to fix. And that's actually very true, right?  All they care is if the business is successful in business and making money effectively. So we monitor because things change, right? We talked about it. But changes affect business. And a lot of people, especially in attack groups, either don't understand it or they're not previewed to the larger picture. I always a huge fan of the top-down approach when you're in, right? You monitor the business and everything else is just used to support those particular metrics. So in order to do that, you need to understand the business, right? You need to understand what are you building, right? Your software is only there to support some business objectives. It's not there for the sake of using technology. You gotta define the baseline. You gotta understand what constitutes good versus bad. And it's not necessarily binary. It's not necessarily, oh, it's up or down. It's more of a threshold, right? How many registrations do we get in an hour or a minute? How much revenue is coming through the systems? What is the traffic patterns, right? And of course, you should be able to correlate the data because once you identify the problem with the business metrics, well, you still need to look at your system metrics to figure out what's wrong. So here's another example. And actually, I was too close to involved with this one. So one of the companies, they are actually pretty big. They have about 100 million users, send about a billion emails a month. They have 5,600 metrics. I think it's even more now. So they monitor everything starting from intense outtests on their servers all the way up to how many registrations they had in the past two minutes. So as everything, it all starts with a call, right? When the client picks up the phone and says, something is wrong with the website.  That's general how it starts. So something's wrong with the website. And I was like, great, would you like to elaborate? And the guy goes, well, we'll look at the numbers and our revenue is down, okay? So I was like, all right, let me see what he's seeing. So luckily, we're actually monitoring the revenue. So as you can see, there's clearly a dip. By the way, we fix it if you look at the second dip eventually. But you could see a dip in revenue, but it's not like it's zero, right? It's just lower than the average. And at that time, we didn't see the other spike. So I'm like, okay, so maybe you're not doing your job and not selling enough stuff. I'm like, what are you calling me for? I was like, but let me see, let me do my due diligence. So let's look at the actual traffic. So look at the traffic and it is lower than expected. So it's in line with the revenue they see.  That seems legit. I'm like, all right, but let's dig a little deeper, right? Let's look at low times. Maybe performance like something on a page, third party dependency, maybe something went wrong. Performance plummeted, people can check out.  Performance look just fine. So that's the first place where I could have just had to unscrew it. I mean, it's your problem, not mine. But let's have to dig a little deeper. So let's look at the database, right? Because you can send it, again, it's a Twitter problem, right? Something's happening, but it doesn't come back, right? Everything is fine with the database. Everything is fine with the systems. Everything looks normal. I mean, there's less people coming. There's less revenue, sounds legit. Luckily, we'll continue digging deeper and deeper. And finally, we looked at the email, the availability. And apparently, one of the major ESP providers, I forgot whether it was Yahoo or AL, whoever it was, accidentally put it on a blacklist. So all the marketing emails that were supposed to go to people in that domain, and that's a big chunk of it, bounced. So less emails got to their customers, less people got to the website, they made the less money. So that's the question I get often, it's like, great, you were monitoring all sorts of cool stuff, right? And you were able to troubleshoot it. So what if we didn't have email monitored, right? I mean, we would have probably still figured it out at some point, but it would be monitored after this. Instrumentation is never done. That's another thing.  But instrumentation is an evolving process, right?  If not alert. So we had an example with another system which has very similar systems, except that they had higher decline rates. All of a sudden, again, it starts with a phone call, and it goes in something wrong with our website. It's like, okay, what's wrong with your website? It goes, well, the decline rates are higher. I was like, okay, okay. And we must have spent hours and hours going through logs, going through database, going through rows, trying to figure out what's wrong. And nothing was wrong, right?  Like, everything seems to be normal. And the next day, we come in the morning, we continue troubleshooting, and the same client calls, it goes, oh, by the way, could you create a ticket to take off the American Express logo from our website? I was like, why would you wanna do that? He's like, oh, we stopped accepting MX. And worse, a wedding singer, again, this information could have been brought to my attention yesterday. But realistically, nobody would block the decline, monitor the decline rates from the beginning, right? I wouldn't even think about it. How would you, why would you monitor, like, decline rates for different types of credit cards? Like, that seems overboard, right? That goes to the sound white noise. But apparently, that's the actual case, so we put another monitor on it. So, to summarize, I'm all done. It is testing and monitoring, in no way I'm preaching to get rid of testing. Testing gets rid of almost obvious issues of the whole, oh, shit, I press on the button and everything blew up in my face. You absolutely must have testing.  Understand the business. This is actually one of the challenges that hopefully the whole DevOps movement is trying to solve, is get all the groups involved. And for technology people to understand the business that are supporting is actually gonna help us to support. Much like performance and security, monitoring is not a feature. I don't know how to be. You cannot build your whole application, go through a test, and then as an after-sourcer, say, hey, let me put some monitors on top of it. You will fail in that case. Make sure the monitoring and instrumentation of checks is part of your development process. You develop your feature, put a metric in it. It doesn't cost that much and it will save you as at some point. It's all about continuous improvement, right? You never can cover 100% of the cases. Same as with testing, same as with monitoring. You can cover 100% of the cases. So when you discover things like, you know, decline rates on a specific credit card type, put a check for it. That's it. Monitor everything but alert very selectively. You don't wanna alert on every single metric because, well, honestly, anybody who's on call is gonna start ignoring a lot of things and things are gonna start falling through the crack. I'm also really bad at conclusions. So that's it for me. I think we have a few minutes for questions. Any questions? The question is if I have a specific set of tools for collecting the metrics and monitoring it. I have my preferences. The graphs that you've seen, we use Torconus. We spun out that company so we kinda eat our own dog food. Realistically, there is a lot of monitoring solutions out there. Any solution that will let you collect random metrics, whether they're text or digits would work as long as you can correlate them later. Some do it better, some do it worse. There's also Nagios, which I wasn't gonna mention at all. But yeah, realistically, anything that lets you collect non-typical metrics, I guess, because system metrics you can get from anywhere. But the business metrics, like the arbitrary metrics, the registrations, the revenue, all the stuff that you wouldn't think about it, you want something that supports that. In second part, you wanna support the correlation. You wanna be able to graph multiple things on a graph to see, or at least side by side. Question is, have I seen effective ways to effectively pipe alerts into the Ticket Tracker or something similar? Yes and no. You have to be very careful with that. Again, because you're going on a volume, it's a lot of false positive, right? You need to hone your thresholds enough so it'll be effective. So for example, let's use the same example. You monitor on revenue, right? And you set a threshold. You assume any given minute, the revenue over the past roll in five minutes should not fall below X, right? Well, is that really true, right? Does it count for like two AM when most people are asleep, right? And if you set a real low bar still, like sometimes it can fall below. So when it does, is there a problem or is an anomaly that somebody should wake up, look at it and say, nah, everything is good. It's just slightly below it and go back to sleep. So it heavily depends on the business requirements, right? If you can identify the metric that you know for a fact once it goes outside the threshold or once it falls to zero, let's say, it's a problem that somebody need to look at it tomorrow, then, yeah, you can pipe it easily into a system because you do it the same way as you do with anything else, either with VictorOps or with PagerDuty, just pipe into alert and pipe the alert into your pipeline. Yeah, so question is, can I talk a little bit about responsibilities in organizations for monitoring? The answer is for collecting the metrics, everybody should be doing this. So developers, ops people, database people, even QA people, right? Anybody who is responsible for architecture, for code, for adding new features, should be adding new metrics in. That said, I also big supported that anybody should be able to put an alert in. However, no alert should be going in production undocumented. Like, we have in my company, all the developers have full access to monitor system. They can put as many alerts as they can, but if somebody gets to work up in the middle of the night and the only way for them to solve the problem is to call that developer, they're gonna be real pissed. Right, so if you're going down that model that any alert that goes into production should have a full documentation, full troubleshooting steps, what to do, what to try, what other metrics to look at to see if anything wrong, if everything else fails, or no, then you ask for it. But to answer your question, anybody really. Like, it depends on the business. Whoever is in control of application mostly, whoever is on call, should be able to put metrics. It's, I think that that question is more of a culture thing, right? It depends on the company. I'm not a fan of actually making somebody completely responsible for monitoring, right? It's the same thing as when they don't call, right? If you put responsibility on the people who are responsible for fixing it, they become much more diligent at it. Generally, if you had to put responsibility on somebody, I would say it's people who get woken up in the middle of the night. Because when they do and they realize there's not a monitor on something, they can go back to, if it's developers, they'll go back to ops, if it's up, they'll go back to developers and say, hey, I got this alert, I didn't have any information of this stuff. We need to instrument this, right? So here's the ticket, go instrument these things. I don't know if that answers your question, I hope it did, okay. Maybe one more question if there is one? Dashboards actually have very little to do with actual alerts, right? Dashboards, you want to show the information in consumable format. Alerts, when I say alerts, I mean, it's something that wakes people up in the middle of the night. So you can show as much, so the graph I was showing you, if you want to show 20 of those graphs with 10 things and if they're readable, an actual show available information, then why not? Right, as long as you don't alert on every one of those things individually because any one of them may not necessarily show a problem. Right, when you put an alert in, it's all about actionable alerts which is a whole separate conversation altogether. But you've got to ask yourself three questions, right? When you put in the, do I care if it's broken in two in the morning? Right, can I fix it in two in the morning? And can I fix it tomorrow? Right, and if that's any of those questions is, yeah, yeah, I can fix it tomorrow, then why would you alert on this? Right, you probably just want to send an email. Sure, all right. Yeah, I mean, the short answer is yes, right? Because it is iterative, right?  Because some of those things can be discovered during the testing, right? Or you want to pull it just back into monitors and just instrument the metrics, right? And since Jason is trying to kick me off the stage, we can talk about it offline. But yeah, there's a couple of things you can do with this. It depends on the model and what kind of organization and what kind of data you're collecting. Yeah, thank you. It could be something we can do as an open space, possibly, right? Yeah, absolutely. So let's bring that up. Okay, everybody put your hands together for Leon. Take it on, appreciate it.


Hey there friends, how's it going? My name is Kevin and today we're going to be looking at my subreddit to see what you really think about me. It's been a while since we looked at the subreddit, so... Let's see how we're doing over here. Let's go to the top today of Valentine's Day cards for everyone to use. Fantastic! To Kevin from Kevin. That's not the most depressing thing you've seen all day. I am sorry. Oh wait, there's more. There's four of these. Oh, there's a turg one. That's pretty romantic. I would let you out of the basement sometimes. While that's a lie, that's still pretty sweet. And then there's just defaults saying, Okay, that's just today. Let's go top this week. Let's see what you've been talking about this very week. Oh, this was a big thing. I would recommend for you to not keep the capo on your fretboard. I actually saw this. There was a warning just about the damage it can do to the guitar and the whole stream that night people were telling me to take it off. Kevin, someone on Reddit said not to keep the capo on your guitar because it ruined the strings. I know. That's why I have two on there now. Just a little tip I do is to put the capo on the top of the guitar head. That is a good idea. Thank you for the tip. Fix capo. Deadline. Six to eight weeks. Don't worry. I've made that note and in another six to eight weeks, it will be taken care of. Oh my God. F in the chat for calling me Kevin Clips. The poor guy. He was such a talented content creator and he just disappeared.  Oh my God, this meme. I love this meme. This came from the streams. I can't even remember what I was doing. The shadow makes it so much worse for some reason. What? What is this? I try to search for similar images. Like oh maybe we can see some of the other memes with this template and it brought up Julius Irving, an American basketball player. Like Google, I know you're kind of in control here of my channel and you can just delete me if you want but I don't see the resemblance. Oh my God, it's me. Wait no, it's me. Looking at me. Who is looking at me? There's so many me's and we're all wearing the same thing. We're like a cartoon character. Oh that is cool. I like that. Oh my God. The drawing. It's so nice and wholesome and everything. It's just a cool little drawing someone did and then I scroll down and then we have eat that meat and a drawing of me holding a body from among us on top of an altar. Just eating human remains. I'll show you the clip but I feel like you need a lot of, you know what? I don't think the context will help regardless. Roll the clip.  So I'm going to start now. If you could all chant eat that meat, that would be great. Okay.  Eat that meat. We have accomplished our goal. Oh God. Oh no. Oh God, I don't feel good. Alright, I'll be back later. Oh God, I don't feel good after that. Oh God, help me up. Oh, I'm dying. That's your screen saver. Really? That's what you want on your desktop. Ah, very good. The chaos mod. Yes, I like it. That is the exact smug look I picture on all of your faces when you're torturing me with that mod. Oh my God, what is this? This is from the AI dungeon video, is it? I'm bad. I want to scare you. Oh my God, let's do a little two picture. And a little owl in the background. You know, you're kind of attractive. Oh, the little zoom-ins. That is fantastic. Not what I pictured in my head, but I feel like I didn't even see the little bimsons and stuff. That is beautiful. For a minute, I thought Kevin was being stalked or asked by a mad woman named Jolie. I always do like weird cryptic titles for these music videos. Just a little joke, you know? But they always go wrong. I don't really think of what they could be. Like that, this is Long Overdue video where I sang Country Roads, like, posted it on the first of June, Pride Month, and then had a picture of me looking into the camera, just saying this is Long Overdue. Long Overdue. First of June, World Milk Day, as well, by the way. So, maybe I was just going to try milk. I'm not a big milk drinker. Alright, you know what? Let's move on to this month. We're moving our way up the ladder here. Eating soup while watching Kevin, and he'll never know.  I'm coming for you. Wait a second. Not only did they do it once, they did it twice. We have a traitor in our midst, I think. Oh, another fantastic community moment where I did a poll. It ended up getting over a quarter of a million votes, by the way, so thank you for voting in it. And it stayed at 50%. One of the most impressive things I have seen all year. Yeah, so it sums up the entire idea of getting the Twitch chat involved with my games. Let me walk into my parents' room. It's right here. I have to tell them I pissed the bed. Why is it so distorted? I guess because that's me as a kid. Oh, little Kevin's back. Oh no, I can already see what's coming up. I just see whitest thing on earth. Yeah, there I am. Even the face is like me. Oh my god, there's more of them. They called me Kevin's subreddit. This format. Kevin leaving the theater. Oh no, and it's Jesus. That's a good edit there. You got me out of the background and everything. I like how he's the same shadow though. Describe the gd5k. That's not one word me. No more Mr. Nice Jesus. I mean, that's not one word. Maybe if you say it really fast. No more name. No more nice Jesus. Oh, that's a cute little drawing. Even though I hate the person in it. This guy. He's fine. I'm Catholic. I gotta love him. Everyone makes fun of me for this. Everyone thinks my water bottle is big. Okay, it's not that. Okay, it's a vase. All right, but just get over it. It makes my hands look regular sized if I hold these things. Okay, see don't I look like a normal human now? Hmm, hydrated. Oh, that is a cool one. I like this drawing. This was a fan art of me standing. I was just demonstrating to people how good I am at standing now because I've been using my standing desk every now and again. And I just a lot of practice built up my skill, I guess, because I'm just getting really, really good at it. And I don't like to boast. I don't like to flex on people, but I just think I'm doing a good job. In fact, you know what? I'm usually holding onto something. I'm usually grabbing the desk. I've handles under here and a little seat belts to keep me stood up, but I'm ready for some more freestanding, unassisted standing. Okay. Oh, maybe a bit more practice. All right, that's fine. Practice makes perfect. And injuries. Oh my God, there's another version of me. How many are there? How many times am I standing back there just in that pose? In fact, I'll give you another one. There you go. Is that good? Is that a good one? I even got the funky pants on. Let's face it, I just don't wear pants anymore. It's what's the point. I'm just gonna be at home all day anyway.  No, no, no. Pregant Santa. I'm not falling for that one. I'm not gonna incriminate myself. Fun fact that you know that Kevin and Stuart Little were best friends in school. First of all, his body doesn't look very mouse-like.  And finally, I didn't have friends. Oh my God, I remember seeing this one. Hold on. All dreams have a meaning. I relate it to this a lot because my dreams are so fat. This is literally like the dreams I have. Wake up. Wake up. Wake up. Oh my God, I don't remember this part. Am I in a dream right now? Oh my God, this is terrifying. But yeah, that seems about right. I've had a lot of dreams similar to that. It explains a lot about me as a person, I think. There's so many of these for like streams and I don't know what I'm doing. I don't know the context. Stain glass Kevin and to prove we're not a cult at all. And that definitely proves it. There we go. That's my evidence in court to the eventual trial that we'll all be involved in. And you're all involved in it, by the way. That's into disclaimer when you subscribe to the channel. And if you haven't already, please subscribe. There's no hidden commitment or anything. Just subscribe. I forgot and I made these at some point. I mean the boys at 3AM looking for beans. Well, maybe you should forget it. Maybe we should all forget this. We can repress it together. Oh no, come on. Oh yeah, that's what that one was from. It was me calling nice bike. I remember now. I kind of forgot about that meme. Yes. Oh wait, hold on a second. Obsolving myself of sin by eating potatoes and watching Kevin. Okay, you are forgiven. That is all you need to do.  Oh my God, look at them. They're multiplying. There's so many. I shouldn't have given you that one earlier. That was a bad idea. I forgot about this. I'm using my picture on Tinder. You should use the one of me being a feckin Neanderthal over there.  I don't know if that was some sort of attack. Also, I'm not a jakey. I don't look like a jakey. Alright, you know what? It's time to unleash the beast. We're going all time this year. I hope this is like a 12 month period and not just like this calendar year. Because if so, that's basically a month anyway. Oh my God, the top post is from my post. Well, it's someone else's post. But of me, I don't want to surprise, it's my own subreddit. But I just don't really tweet all that much. So to see me up there, I feel very proud. I gave my mom a YouTube figure and she was not happy to find out the box hours. Me about to be hit by a car. That's true. That's true. Fellas, if your girl has a beautiful smile, pale skin and a good sense of humor, that's not your girl. That's 2.4 million subscriber YouTuber calling Kevin. You had to pick that one. Of all the pictures, you had to pick that one. Who is your favorite Avenger? Me. I love that the title is just a hawk guy. I mean, like some weird knockoff. Oh look, it's hawk guy and Captain Ireland. Say his name, you won't be laughing when he calls you Swiss. Pee pee poo poo man. Oh, can we fund this? Has it already been funded? We're probably too late. Four months ago, that's probably a production now. They probably cast Pee Pee Po Po. Who would play him though? I would have to imagine Vin Diesel. I always get sent this. There's always that jerk named Kevin that shows up in cartoons.  And then me. Why? There must be some nice Kevin characters surely. Characters named Kevin. Oh, there we go. See, we got Kevin the bird from up. Kevin the minion. Kevin McAllister. I get sent that a lot as well. The mom going, come on. Okay, maybe all cartoon Kevin's are annoying. I don't know. I'd have to see some more data on it. But it's not looking so promising so far. Kevin Swanson. Oh my God. My son. I can't believe I'm going on the Family Guy fandom wiki. But I just want to see if he's a good person or not. I'm here but now I'm realizing that I'm not really patient enough to actually read like three pages on Kevin Swanson from Family Guy. So I'll leave it up to you if you want to find out more. FECC call me Kevin. All my homies watch ZX. No regrets, ZX said. Really? Really? You think he's better than me? I can see why you like him so much. I just see the gameplay through all these pixels. No. You were there.  Sorry, not me. ZX no regrets. He's just such a talented content creator. I'm just jealous. Alright, don't listen to me. God, my center of history is just me looking up myself from like 10 years ago. Old Call of Duty and now Kevin Swanson. Oh my God, this is amazing. I didn't see this before. This is me playing board games with myself during the initial lockdown. Oh, even that sounds fun now. The lockdown has gone very old, very fast. Oh, that is lovely as well. I love that. That's very nice. Um, no. No, not this one. What is this? I think I've seen this image before. I've definitely seen this one. That's the drawing. When you can't play online games like you can recognize, DJ Kevin suffering from success. That is legit. Okay, there have been games that I cannot play anymore. It's tragic. It's really, really a big problem and you should feel sympathy for me. Why do all the bad things happen to me? I do enjoy pretending to be a soundborder. That is one benefit.  The only problem is, it's so rare that you'd actually be able to fool people with it. I do enjoy pretending to be a soundborder. That is one benefit.  The only problem is it's so rare that you'd actually be able to fool people. It's always hilarious. The only problem is you enjoy it. The only problem is you enjoy it. You enjoy it. You enjoy it. You enjoy it. Celebrities, we're all in the same boat. They're a boat. My boat. Look, if it made my bit life character happy, it's good enough for me. This is great.  They might just become caring pediatricians, nurturing teachers, doting uncles, loving fathers. That's very sweet. Thank you very much. That's very nice of you now. That's lovely.  Irish people will be named. It'll be like it's pronounced Kevin. Oh my God, should we do like a video just on Irish stuff again? It's always so fun. I've even got like an Irish dictionary. Let's learn a word, shall we? Okay. So please, lermus the video, please. Lermus the video. Leave a lermus if you have a chance. I literally just opened it and that was comment apparently. Lermus chocked. That's a new one to me. How do I say like and subscribe? You know, I don't think subscribe will be in here. I'll take a look. I'll subscribe, but it's really hard to pronounce for scrive. I think it is. Oh my God. Streamer is here. Sri Lone, I think, but no, let's not get distracted. Okay. So please. Lithid? Is it Lithid? I'm so bad at my native language. It's not like it's used to be fair. All right. Well, Lithid, August for scrive, Moshe Dole. That's like and subscribe, please. I hope. Oh my God. That is fabulous. I've never seen this before. I missed it somehow. That is, that is amazing. Good job, honey snail. So how is life going? Me? That could not be more accurate right now. That is more accurate than it ever has been for me. All right. Well, we're going to end it there. Let me know if you want to see more of this guys. Thank you so much for watching. I appreciate you. I hope to see you next time. Bye. Bye. Bye. Bye. Bye. Bye.


Another day's here, everybody. It's gonna be a good day in one second, one minute. Put any blame or point fingers, but they handed out to you in these weird little trays now, right? That's a new thing. And they had put the English muffin in there and a napkin. And they grabbed the napkin and it, yeah, whoops. Other than that, it's gonna be a good day. Today's agenda is Arberg. We gotta deliver some stuff up there, which is backwards. Usually I pick up stuff from there, but now we're delivering stuff there. And it's under a tarp. I just remember that. So no curtain sides today. Just rolling up tarps when we get there. Living Arberg here. It was a town that not a lot of people would hear about unless if you come through here and not a lot of people would come through here, it's sort of like an end of the road kind of town. And I don't mean that as an insult. It's actually a very nice town. It's just at the end of the Brooklyn. There's not much past it. Came here the first time, surprised me how nice it was.  Because it's called the Icelandic River, it must connect with Gimli or something in some way because I know Gimli Manitoba was settled by Icelandic people. So it sort of makes sense if that was the case. No, not too sure. The kind of town where you don't have to lock your doors or if you know anybody from. So I'm just getting the tarps off here, gonna roll them all up and we'll get this unloaded quickly after that. I really do prefer the covered trailers, but I guess you work with what you got sometimes. Maybe I should take like an online videography course. It's actually a really nice town. One of the gems of Manitoba, I would say, along with Steinbeck and Winkler. Morphin is just a few miles west or several miles west of Winkler. I gotta be careful with my directions because last time I went to Canora, I got my directions messed up. I meant to say east and I said west. I think you guys figured out what I meant. Parking lot in the back here. I drove past it the first time. Say it's just a drop and hook, but I'm gonna have to take that tarp and all that rug with me that I had that was used for that last load to Arbor. Oh, I see one of our van trailers there already. Yeah, this must be it. What's the grab trailer, 536. So I'll just put this one I have now in one of these empty holes. Who wanted me to pick up 537? Hold on a second here. I haven't written down 36. See a 537, see 537. Well, let's get parked in here first and then we'll slip it right in here. I don't see 536 anywhere though. Still not seeing 536 anywhere. I see 515, 511, 535, 523, 540, 525, 541. There's a whole bunch of our trailers here. I've got 522 here. And the only other one that's close to it is 537 here. I wonder if he meant to say 537, let's see when the safety is up here. Safety is up May, 2021. This must be it, yeah. This needs a safety by the end of this month, May, 2021. I'll have to, but no, 536 is not here. All right, that's what phones are for, right? We'll just make a little phone call. It was supposed to be 536. Another message? No, that was nothing. Interesting, okay. So I'll take another look around here, but this is where they keep all the trailers. And if 536 isn't here, then I'm just taking 537 back because it needs to be safety by the end of the month as well. So that's another one that needs some attention soon. Happen to it. Ha, till that time of year where I'm blasting the heat in the morning, I'm blasting the air conditioning in the afternoon on the way. Good to see you too. Good to see you too. Bain burgers. This is gonna be good. So last time, apparently I overdid them, cooked them too long. So this time, pay a little closer attention. It's my ender. We found out I'm actually the boss this weekend. Still no talking to me much. It's getting better though. Oh, she's wrecking you with pickles. Blast for me. Pickles are life. Right up there with coffee. Don't even compare them with coffee. Oh, I'm comparing them. That's not okay. You putting hot peppers on there too?  The nights don't have much of a palette for flavor. They like things very basic. Bread, meat and potatoes. That's life. And ranch dressing. I put these on chicken caesar salad. On chicken caesar? Those hot pepper rings, I always put them on chicken caesar salad, but you've been home so often lately. I didn't want to be judged. Oh, I would judge. I'm judging. I'm judging you. What do you guys think? Is that normal? I'm waiting to get in here so that I can get my burgers down too. But she's hogging the kitchen. Well, whenever I wait for you to get something done, everything's cold by the time you actually get to it. So I'm just being proactive this time. Fair enough. Let's scoop some salad onto my plate over here. Oops. Okay. Looks good. Got your salad. Almost got away with it. If there's room. You better be. What you hoping for, Diesel? Would you like some salad? No. Maybe some burger. That's the one. Don't tease him. Oh. You already had your supper. You guys just finished. Don't beg. Don't. Why don't you go to your bed and lay down? Okay. Yeah, that's your bed now. All right. I'm gonna make my bed right here. You're such a good listener. Such an avid listener. Oh look, you got the commander all whiny now. Yeah, they feed off each other. They ignore each other the rest of the time, but when it comes to mealtimes, they feed off each other. You got wiener over there watching very closely. But at a respectful distance. Yeah. Got Chevy who has no idea what's going on ever. Whoa, whoa, whoa, whoa. So we just had the strangest experience that's happened to us. Maybe you guys have had this happen before too, if you're on town water. I've never been on town water. As far back as I can remember, I've always had well water. We've always had our own well. But now that we live in town, the town supplies us with water, which is supposed to be clear. I'll attach a picture in here of what our water looked like just about a half hour ago. They called the emergency line of the water works at the town here. And apparently our fire department has been training or practicing tonight. And I guess whatever they're doing, tapped into some water lines that released a bunch of sediment or something into the pipes. Now half the town is getting brown water. So the water works department is doing an emergency flush of the whole system in this end of town at least. So that's gotta be a nightmare for them. But that's the important part right there. The key word is for them. Finally a problem. That's not our problem. We don't have to fix it. But anyways, it's time to put this all together and go to bed. So thanks for watching today, everybody. If you liked it, hit the thumbs up button. If you haven't subscribed and leave me a comment down below. Let me know what you think. Don't forget, hit the thumbs up before you leave. I'll see you later.


The next item of business is a statement by Shirley-Anne Somerville on OECD education next steps. The cabinet secretary will take questions at the end of her statement, and so there should be no interventions or interruptions, I call on cabinet secretary Shirley-Anne Somerville. Following publication of the OECD independent review of the implementation of curriculum for excellence and Professor Stobart's paper on options for Scotland's future approach to qualifications and assessments in the senior phase, I signalled my intention to consider a reform process underpinned by those findings and recommendations. Today I will provide the chamber with that update. It remains a key priority of this Government to ensure that our approaches to curriculum and assessment are fit for purpose, and so guarantee the best possible educational experience for children and young people, not least as we emerge from the pandemic. It is 10 years since curriculum for excellence was introduced, so it was appropriate to review its implementation. Furthermore, the exceptionally challenging circumstances created by the pandemic have led to innovation and creativity in the education system, which we can also learn lessons from. I am proud to note that the OECD found wide support for CFE and stated that Scotland's curriculum continues to be a bold and widely supportive initiative, and its design offers the flexibility that is needed to improve student learning further. Professor Stobart in his paper described CFE as a pioneering example of 21st century curriculum reform and highlighted that Scotland's curriculum continues to be viewed as internationally as an inspiring example equated with good curriculum practice. Those findings show that the Scottish education remains on the right track. There is no suggestion that any significant overhaul in terms of the curriculum is needed. Instead, the OECD report sets out recommendations for us to build on our current approach and ensure that our curriculum continues to inspire learners now and into the future. Every education system must be open to further improvement, and I have previously made clear my intention to work with all those in education to consider those recommendations and deliver reform required to improve outcomes. This work should be seen as a supportive process and as part of our continuous improvement to retain Scotland's pioneering status in this field. Crucially, that work must be informed by children and young people, and I am committed to respecting children's rights and ensuring that those voices can be heard clearly and genuinely informed in the direction of travel. That is why I announced the intention to establish a Children and Young People's Education Council and ensured that there are young persons representatives and children's rights experts on the Scottish Education Council. It is also important to gain the views of teachers as those closest and best places to take decisions about children and young people's education and health and wellbeing. Therefore, we will engage further with front-line teachers through the national teacher panel and will seek ways in which the role of the panel can be strengthened and expanded. We will also engage through other routes, including the professional associations. This engagement is essential if we are to build on the work that began in 2019 to enhance and develop an empowered education system. While that work was paused effectively during the pandemic, our commitment continues and we must build on the good work pre-pandemic that saw effective developments in teacher agency through the delivery of bespoke remote learning, community activity and response to the local needs of families and communities. I have already taken action on the OECD's recommendations about structural change and support for education. I announced the intention to replace the Scottish Qualifications Authority and consider a new specialist agency for both curriculum and assessment. We are also taking forward reform to education Scotland, including removing the function of inspection from the agency. I invited Professor Ken Muir to act as an independent adviser to the Scottish Government to consider and advise on the implementation of those reforms. Professor Muir is currently gathering views from stakeholders and will submit his recommendations to the Scottish Government in early 2022. That work will play a crucial role in delivering the foundational infrastructure to support the delivery of the curriculum and assessment process, as well as helping to inform the wider reform process. Today, I am pleased to publish a framework for implementation that sets out our approach to engagement and priorities and sequences the initial phases of this work. I chair the Scottish Education Council and, through this, have already started a dialogue on the process, but it is clear that, for this to be a truly co-produced approach to reform, wider discussions are needed and that is what this framework sets out to facilitate. The implementation framework sets out the initial priorities that we will be looking at. They are to reassess the vision of CFE, to agree a measurement and evaluation approach, to align assessment and qualifications, to clarify roles and responsibilities, and to increase curriculum development capacity. We will convene and facilitate dialogue to revisits and assess the 2019 refreshed narrative for Scotland's curriculum, which will be informed by user feedback, Professor Muir's consultation and learning from the pandemic. That work will support initial considerations for the development of a curriculum review cycle, as recommended by the OECD, and will underpin the work on the alignment of assessment and qualifications. We will explore the data and research that is required to inform an evaluation system and to support the systematic approach to curriculum review. Specifically, I will be asking the Curriculum and Assessment Board to set up a short-life subgroup to explore options for a sample-based survey looking across the capacities of curriculum for excellence. The subgroup will specifically consider the workload implications for staff of such a survey. That will be the first time that we will have information that looks across the four capacities in such a form, but that has to be done in a proportionate and in a manageable way. We remain committed to teacher professional judgment as the primary means of assessing progress in the broad general education, and we will consider how we can better support that and the achievement of CFE levels data in the future. National standardised assessments will continue to have an important role to play in that. The outcome of that work will be reflected in the national improvement framework from 2022 onwards. In addition, communications and stakeholder involvement strategies will be refreshed, and we will develop and agree next steps to further school empowerment with teachers, practitioners and other stakeholders. I will also continue to work with the Scottish Negotiating Committee for Teachers on the commitment to reduce class contact time for teachers by 90 minutes per week. In addition, we will work with local government to deliver the commitment to recruit to 3,500 additional teachers over this parliamentary term. Furthermore, we will evaluate current support and access for teachers and practitioners to professional development for curriculum design and development. The package underlines our commitment to the profession and our support for their on-going role in curriculum design, development and improvement. To enable a coherent and effective response to the OECD's recommendations, we will ensure that there are appropriate programme and government arrangements as recommended by the OECD, building on our current structures and putting the voice of teachers, practitioners, pupils, parents and carers at the heart of everything that we do. Alongside publishing a framework for engaging on the implementation of the recommendations of the OECD, I also want to set out my position on the future of assessment and qualifications. That is distinct from the on-going arrangements for 2022, on which a series of announcements detailing the approach and potential use of contingencies have already been made. Professor Stobart's working paper, Upper secondary education student assessment in Scotland, makes an important contribution to the debate about the future of qualifications and assessments in Scotland, a debate that has intensified in light of the experiences during the pandemic. Professor Stobart does not set out specific recommendations but suggests options based on best practice from around the world, which we must explore as we seek to find the best way of recognising our learners' achievements in the 21st century. Taking Professor Stobart's findings alongside the lessons learned through the pandemic, I believe creates a case for reform. The issue of assessment and qualifications generates strong and sometimes conflicting opinions. However, I am convinced that, given the experience and views expressed over the last two years, the time is right to signal that the Scottish Government supports reform of national qualifications and assessment. Just as work with the OECD and responding to recommendations, it will be vital when considering reform that we work with all stakeholders to, as far as possible, build a consensus on that issue. To that end, I can announce that the Scottish Government will concert on the purpose and principles that should underpin any reform of national qualifications and assessments. That will be the first step in a process that could be done with careful thought and consideration and recognise the importance of national qualifications to learners. I am pleased to announce that Professor Louise Hayward of Glasgow University has agreed to lead a reference group with its membership drawn principally from the curriculum and assessment board. That group will provide advice to Scottish ministers as to how agreed principles may be translated into a design for delivering assessment and qualifications, while ensuring that externally assessed examination will remain part of the new system. Professor Hayward will begin this work in the new year. While I expect this process to lead to reform of our approach to national assessment and qualifications, time must be taking to get this right. I am acutely aware of the impact of the pandemic and, therefore, a dialogue leading to a planned and achievable reform programme will be essential. As I stress throughout this statement, those reform processes will be collaborative and actions will be created with stakeholders wherever possible. I am also very mindful of our children's rights requirement and I hope that through this process and its outcomes that all children and young people can feel a sense of ownership for the changes that we will make. Those actions reaffirm our commitment to ensuring that every child and young person in Scotland can benefit fully from an education and have access to qualifications, which will see them realise their potential and step confidently into their future choices. I will now allow around 20 minutes for questions after which time we will move on to the next item of business. It would be helpful if those members who wish to ask a question were to press their request to speak buttons now. I call on Oliver Mundell. I thank the cabinet secretary for advance sight of our statement. Far from restoring standards in our education system and seeking to reverse the 14 years of damage that the SNP has done to our once proud educational traditions, the statement confirms that the only plans that the SNP has got is to double down on radical and ill-thought-out reforms to exams, which will end exams as we know them. On top of that, we see a continuation or the very denial that got us into this mess in the first place, which sees a Government glossing over the identified weaknesses in CFE, particularly in relation to knowledge. Can the cabinet secretary tell parents, pupils and teachers why they should trust the SNP to fix Scotland's schools when it is this Government who got us into this mess? I can genuinely say that I am still not quite sure that Oliver Mundell has actually read the OECD report that came out, despite it coming out many months ago now. I mentioned some of the quotes from the OECD in my statement about how it has endorsed curriculum for excellence, how it is the right approach for Scotland and that it does not see the need for fundamental reform of the process. I am not asking parents, young people and teachers to believe any political party. I think that we should look at the independent experts that we have brought in and the messages that we have given. I also do not think that Oliver Mundell read the statement that he got in advance or listened to what I have said, because I specifically said in the statement that we are not talking about ending exams. What we are talking about is having a discussion about the best way of us being able to look at what a learner achieves and be able to recognise that achievement. That type of discussion has, apart from the Conservatives, been largely welcomed among many stakeholders, including those who will not only get the qualifications, but those who will look at the qualifications in terms of other parts of the education sector and employment. I genuinely say to Oliver Mundell that it is not too late to drop the sound bite and the pressure release that you have no doubt already put out. I genuinely take part in a discussion about what the qualifications and assessment process will look like going forward. We are going to look as a Government about what the principles will be. I genuinely hope that the Scottish Conservatives will raise their game and take part in that process with me. We are now, in the cabinet secretary's own words, many months on from the announcement that the SQA is to be scrapped. Far from taking urgent action to begin these much needed reforms, the cabinet secretary is content to leave the SQA unfit in its current form, as the cabinet secretary agrees, presiding over our assessment process this year and potentially beyond. Education Scotland presiding over curriculum, development and inspections. Over five months ago, I laid a motion in this Parliament setting out a timetable of ambition for reform. It would have seen by now an independent inspector established by executive action, an interim body for assessment and curriculum, while the current consultation proceeds and negotiations under way regarding a new deal for teachers stemming from the OECD review. That would be real urgency and ambition for young people. Deputy Presiding Officer, how many more months the minister plans to talk about reform instead of implementing it? By what date she would expect a new assessment and curriculum organisation to be operational? By what date she expects to take forward proposals to reform qualifications? By what date she expects class contact time to be reduced for teachers? Or is this simply set to be another wasted parliamentary term for education in Scotland? I listened carefully to the proposals that Michael Marra put forward earlier on in the year about this. I listened carefully to it because I hope that between Scottish Labour and the Government we can have discussions about how that can be taken forward. Michael Marra will know that we have Professor Ken Muir undertaking that work. The work that he is taking and the way that he is undertaking that reform is crucial, so I appreciate some of the demands to move faster and to move quicker. However, I stressed in my statement and I will continue to do so, that I want to work collaboratively with stakeholders. It is very easy in Government to make statements about a way forward and then try to work out the detail or make statements on the way forward and then work out what stakeholders are thinking about. What we have within Professor Muir's work is a process that has been well respected by stakeholders, which has seen significant numbers of practitioners, of parents and now of young people taking part in that process. What Professor Muir will come back with in the end of January is a considered recommendations based on a very collaborative approach. That is the way that I want to work and that is the way that I have encouraged Professor Muir to work on it. I will, of course, look at that carefully and quickly once Professor Muir reports back and will make further arrangements and announcements to Parliament on that. In the meantime, of course, there is absolutely no doubt that 2022 assessments and qualifications will continue. The SQA will, of course, lead on that process and will do so very effectively, as I have no doubt the chamber would expect. I would like to ask the Scottish Government how it will ensure that teachers and staff are central to the process of reform. I will try to lay out some of the details of that in my statement. I am keen to look at the role of children and young people and the role of teachers and practitioners to ensure that they have a very central role in that reform process. One of the ways that we are doing that is through the national teachers panel, which I am looking to strengthen. I am also keen that Education Scotland will work directly with teachers and schools to build local, regional and national communities of practitioners to lead and to shape and design the reform work as it goes on. I also mentioned in my answer to Michael Marra about the work that Professor Muir has already undertaken within stakeholder engagement that has been exceptionally well attended by teachers. However, I would certainly welcome any further suggestions and proposals about how we can further strengthen the role of teachers and front-line practitioners on this issue from Mr Stewart or, indeed, other members across the chamber. Megan Gallacher, to be followed by Rona Mackay. Beatrice Pond, from the OECD Education Directorates, stated in June that the curriculum for excellence had too many owners, while lacking clarity about the responsibilities and that curriculum for excellence is just moving forward and has no structured approach to look forward, plan and communicate with a long-term perspective. Cabinet Secretary, the statement provided to Parliament today adds more levels of bureaucracy without clear direction. Why is so little progress being made since June? To give one example of what has been happening since then, the discussions that we have had within the Scottish Education Council on how we can take this forward has been a very important part of this. One example of the aspects about this is how we look at that refreshed narrative, which the framework sets out early timetables for movement on this. That is the first step to the review process that the OECD has recommended. The OECD recommended a way that we could implement this. It said that this was a long-term approach that was required. We have accepted the recommendations from the OECD and the suggestions about how we should implement this report, not just the actual recommendations per se within the report. That is exactly what we have been doing. I hope that some of the examples that I have given once again can reassure the member that we are making good progress on doing just that. To ask the Scottish Government what steps it is taking to ensure that any sample-based system of measurement and evaluation complements data that has already been collected and protects professional judgment of our teachers. It is important that we ensure that teacher professional judgment and the achievement of curriculum for excellence levels data, ASAL, remain primary means of assessing progress in broad general education. As I mentioned in my statement, there will be a short-life subgroup that will explore options for sample-based surveys to assess progress across the four capacities. That will include consideration about how we can provide better support for teacher professional judgment. The consideration of that short-life subgroup will sit alongside, of course, the upcoming consultation on proposed changes to the national improvement framework measures of progress. The Scottish Government has promised 3,500 more teachers by the end of this parliamentary term, but we know that the teaching profession is struggling, both with recruitment and retention. Can the cabinet secretary tell us specifically what the Scottish Government is doing to encourage new teachers into the profession, but perhaps more importantly encourage the existing teachers to stay? Myrnton, we have quick-filled a very important issue around how we can achieve that target of 3,500 extra teachers. We are, of course, undertaking work on recruitment, as the usual recruitment campaigns continue. We are working very, very closely with local authority colleagues to ensure that we are doing everything that we can at a national level to take away the impediments, for example, of temporary funding. That is why I was pleased to announce further baseline funding for local authority that will assist in the recruitment of permanent contracts that will be more attractive to some teachers to come into or stay in the teaching profession. Very importantly, I will remain committed to very close dialogue, as I have since becoming Cabinet Secretary for Education with the professional associations, to ensure that I am listening very closely to what they have to say, both on recruitment and on retention. Thank you very much. I will ask the cabinet secretary what steps she will take to reduce class contact time for teachers and giving them more time to plan their lessons. The Government is committed, as we said in our manifesto, to reduce class contact time for teachers by 90 minutes per week, with the aim of giving teachers that time to ensure that they have more time to develop curriculum and to ensure that they have the time that they need to effectively teach their class. Of course, the terms and conditions of services for teachers are not the responsibility solely of the Scottish Government. They are the responsibility, of course, of the Scottish negotiating committee for teachers. It will be for the SNCT to agree the implementation for this reduction in the class contact time. That work is on-going. Those discussions are on-going, and I look forward to seeing the progress that we can make on that. The cabinet secretary has been left to clear up John Swinney's blunder over national testing. Having scrapped the national survey just a few years ago, the Government is now bringing it back, this time with a tail between its legs. Why on earth is the cabinet secretary still clinging on to the national collection of local testing data, even though that leads to damaging school league tables? Can we be very clear about what we have said about the data proposals that we have, particularly for sample-based surveys? We will ask a subgroup of the curriculum and assessment board to explore sample-based surveys. That does not mean that they will bring back the SNLN. There are reasons for that. That was taken away and why we moved to a different approach. One of the reasons for that was, as I think I quoted to Willie Rennie in a previous committee experience, some recommendations from the OECD in 2015, which said that the SNLN did not give national agencies enough evidence, which is why the moves were made. I appreciate different views on standardised assessments from some parts of the chamber, but what we are seeing as we move forward is a greater use of those by-teachers themselves. A real understanding, for example, from staff feedback surveys in each of the last three years that reports are providing practitioners with valuable insights into the learning needs of individuals. I have explained the reasons why we have made the change, I have explained the reasons and the advantages of what we have with standardised assessments and the process that we now have in place to look at what we can do in the future to improve the data that we have. Cabinet Secretary, I warmly welcome the Children and Young People's Education Council. Can you tell us anything more about plans to ensure that students' legitimate and constructive views and assessment will continue to be valued and respected and provide reassurance that young people's voices will not be lost in the midst of expert public and political deliberations? It is very important as we move forward with education policy as a whole that we have a very strong voice for children and young people. That is why I took the decision to establish the Children and Young People's Education Council. That will have parity of esteem with the Scottish Education Council, and I will chair both of those. Those are two bodies in which children and young people will play an important part in, but it is very important that we look to encompass children and young people in all the work that we are doing within education. I appreciate that, as I said in my statement, there are differing views on how we can move forward in assessment and qualifications, but I hope that all of us, regardless of where we are with those views at the moment, will take a step back and listen to what children and young people want from their future. It is important that we all do that. I do not share the cabinet secretary's enthusiasm for standardised testing. I welcome the fact that the implementation plan includes a number of green manifesto commitments, including the reintroduction of a sample-based measurement system and a review of the role of indicators and measures with her view to reducing teacher work-code. How that review into indicators and measures will be taken forward, and by whom? I appreciate that we still have a difference of opinion on some issues in education, and robust discussions continue on those. I want to work carefully with all colleagues across the chamber, particularly my colleagues in the Scottish Greens, to ensure that we can have robust but constructive conversations on that. In the particular issues that Ross Greer mentioned in his question, what I would like to see is us working collaboratively. The Scottish Education Council will play an important part in our discussions. We will also ensure that whatever way we are determined to move forward with particular working groups, the voices of children and young people and practicing teachers will play an important part in that. Of course, I am happy to continue to have particular dialogue with Mr Greer on those issues, if he should so wish. As we have emerged from the pandemic, it is clear that far more must be done to guarantee the best possible education for Scotland's children. Today's statement rightly recognises the need to involve pupils and teachers. What engagement will the Scottish Government have with parents and carers as key partners in the education attainment? A very important point, which I hope that I can reassure Ms Goslaw that I share her opinion on the need for us to speak directly with parents and to hear them listen to their views. One of the examples that I could give was speaking to Professor Murgis yesterday on the work that he is undertaking on reform, in which he spoke about the number of parents that are coming forward to take part in the webinars that he has organised as part of the reform process that he is in. I can also point to the fact that parents representatives are members of the education recovery group as we look to Covid. However, particularly as we move forward with some of the packages that we have been speaking about today, they are represented on the Scottish Education Council as well. Posing on our young learners, can I ask the cabinet secretary that, prior to a newly reformed system of national qualifications and assessment coming into place, if it will assure the current cohort of learners that they can have absolute confidence in their hard-won qualifications and achievements? If that is indeed the last question, which is, of course, entirely up to you and not me, but if it is the last question, I think that we end on a very important point about reassuring young people, their parents, teachers and those who will look at the qualifications, although we are undertaking reform of qualifications and assessments. As we do so, we have national qualifications that remain highly regarded in their current form, which has been demonstrated by the credibility that has been attached to them over the past 100 years, including over the past two years, despite the efforts of the pandemic. The current cohort of learners can have every confidence in the value of their qualifications and achievements. Again, that is not necessarily just the view of the Scottish Government and the SQA, but I would appoint to, for example, a letter from many employers led by Sandy Begby, who recognised the value of this year's qualifications just as much as any other here and how every young person should be proud of the qualifications and their achievements at school and within education.