Hi, I'm Greg. Hopefully it's all working. I'm going to give a talk about Linux kernel security, the process of the Linux kernel security team mostly, how we do our work, how the security process even works and all that stuff. This is also based on talk I gave last week at the kernel recipes conference in Paris. So if you've seen that already, it's much the same material, hopefully different questions. And it's also based a little bit on a talk I gave for this very mentorship series a year ago, talking about how the kernels developed and some issues around security things there. Here I'm going to dive more into the specifics of how the security team works because it seems like a lot of people don't really understand. And first off, big disclaimer, this is all just me. My personal opinion, I've been part of this kernel security team since we started it in 2005. And I'll go into how it started later. And of course, nothing reflects the opinion of anyone else except me. And hopefully they agree with me. Usually they do. Sometimes they don't. But that's okay. We're all individuals. Big question is why talk about this now. The CRA is very, very important. It's coming up a European resident. It's coming in for Europe, be it a good version or a bad version. It's very, very relevant covers secured cover software and devices, which is a very good thing. Software and devices should be regulated just like paint and toys is regulated software and toys be regulated as well. Their goal of it was to carve out and say open source software is very important and rely on it in the EU here. So let's make sure it covers in the same regulations as close or software. That's good. I want a level playing field. However, the way it's written that really is going to be harmful for open source software so much so that as written and has originally reproved, it might mean that nobody can use open source software and devices which is going to be very good at all. There's two posts from Linux Foundation there that give you some background. Linux Weekly News has done a great article. Apache is posted about this. The Eclipse Foundation is posted about this. ML.org is posted about this. Lots and lots of companies post about this. The German automotive manufacturers said this is a bad idea and here's how to change it as well as the French telecoms. The German auto manufacturers agree on something that's very important. And to be fair, I'm creating laws is tough. We're trying to work through it, working with a bunch of really good people in Brussels to try and educate our representatives on how open source software works, but it also covers a lot of security issues. A lot of security things are caught out in there such as no devices can ship with any known vulnerabilities. My answer to that question is that is known by whom, known by me or known by you. If you as a company don't have to know about any vulnerabilities then you're not even going to look for them. There's lots of other things like porting problems and fixing them within 24 hours of reporting and other unrealistic things, but hopefully it'll work out. Another big reason is every month or so companies keep asking me to join the Linux Kernel security team. I'll go and talk about it why there is no companies on the Linux Kernel security team. That's not how we work at all. Along those lines, people keep asking for early security notices. They want to know the bugs that we know about ahead of time because we're going to fix a lot of open source projects have these. I'll talk about it later why this isn't going to work and why it's probably just illegal. You can't really have these things. And also going to hardware embargoes. We've had a lot of issues with hardware for the past couple years. Spectrum meltdown is called this out dealing with hardware is hard. The job of a kernel is to paper over all the bugs and hardware to make it look like a unified model, you know, care with CPU you have underneath you should just work. Unfortunately, hardware leading times and fixes are very, very long, which is actually going to tie into the CRA because the CRA directly affects processors because processors run microcode which is firmware which is covered under the CRA. So if you're going to have a very short time to fix a bug under the CRA, you're not going to be able to have any of these 15 12 month long embargoes that the hardware manufacturers seem to keep imposing upon us. That's going to be a big deal. Hopefully AMD and Intel know about this now and they're working to help resolve that but that's going to be a big issue with that. So please interrupt or ask questions until you can pick them out of the chat and talk to me. So real quick, here's the size of the kernel. As of the last release we did six of five, 36 and a half million lines of code. It's huge. It's very, very large. However, you do not run this all you run about five to 10% of this. Everybody runs a different five to 10% about that. So about 2 million lines of code, two and a half, three. Your phone runs around three and a half million. Your laptop runs about 2 million. Your server runs about one and a half million server runs less code than your phone by almost double. Everybody runs a different part of the kernel. So we don't really know who runs what and that'll come back later. So when you see this huge number all the time that 37 million lines of code, you're not using that. Again, you're only about using 2 million lines of code. And this is our rate of change nine changes an hour very, very fast very quick to keep on top of this we've been running this rate for the past five or six years keeps going up and down a little bit but we're averaging nine changes an hour for a very long period of time. In the years this is a huge rate of change this is a lot of a lot of people we have over 4,000 maybe 5000 different developers from 400 to 500 different companies every year, huge huge rate of change. And talk about this, the new release model is still comes up every once in a while as of 2004 we changed our development model to be, we do a new release every two to three months and all releases are stable. This is in contrast to the old. You have a stable tree and you have a development tree and then we switch them every three or two years. Again, every two or three months new stable release. That's all we have. People still are confused about this. And back in 2007, we got together at the kernel summit and we made this rule, we said we will not break user space because we want you to be able to always update your kernel to a newer one. And this is important for security because we want you to be able to take all our latest versions we will not break user space. This will just work. And of course we have a caveat on purpose. We're human. We make bugs we make mistakes. You tell us about it we'll fix it. So I get reports are saying hey this broke over here. Great we'll revert it and we'll move on and keep on going on purpose. That's the only really big rule we have in the kernel development process is we will not break user space on purpose. People get yelled at when they do it on purpose. We fix it and roll it back. I saw some report that somebody booted an old red hat. Five or something not real red hat five all user space on the latest kernel that we just released and everything just works so it's a good indication that we have kept this promise. Version numbers also of course mean nothing there's no odd and even and development are stable just we just increment them every two or three months. New version comes out I'll talk a little bit briefly about that and we move the bigger number. Every couple years when it just gets too big so don't think of like oh the five series or the six series or the four series they mean nothing it's just a new version after one after another just means this one was older than that one. And here's how we do releases as I've said many times Linus has released six dot four and then we do the development releases every first one is every two weeks and every week after that does a new release candidate. And then after two and a half to three months he does a new one and we go on this is how we do our development is how everything works. And at the same time, I'm taking the latest release from Linus and doing a stable release. And we take all the bug fixes that landed Linus's tree and into the stable tree as needed we pick them out we do we do AI we've been doing AI for about five years it's just pattern matching to find the bug fixes and throw them over there and put them in a stable tree they're tested really really well and go out shoe helps out a lot with the testing here on the call other people have a lot for the testing.  After a new release from Linus comes out I throw away a stable release and go on to the next one, you upgrade because you can always upgrade to a newer kernel, all your user space code will work. All is good. Um, the stable current rules are pretty much this. The unique thing was made must be in Linus's tree. That was unique I've seen other development and models where that doesn't happen and that causes a lot of regressions of time because you don't know where you need to back where it fixes and whatnot. Fix must be in Linus's tree less than 100 lines usually sometimes bigger. Copy them over there and away you go. These are the rules. Pretty simple. Um, one kernel a year I pick for a long term kernel. We maintain it for at least two years recently there's been some news that this is somehow changed the ruling hasn't changed we always say we'll do it for two years. And then we bump up and we'll say we'll do it longer if we talk if companies say it's okay. For some of these 414 and 419. They're running for six years it turns out nobody's really using them. Nobody uses the 414 kernel for six years. Nobody's picking them up. So we're not going to maintain these for six years anymore. Look on the current or website on the release page it'll show you how long things are going to be maintained for if you have questions about it just email me. I've seen lots of really odd speculation and strangeness, but I work really closely with the distros work really closely with the embedded developers. Chrome OS Android to pick up to pick a good window of how long to maintain stuff. Sasha helps me out a lot with the stable kernels and long term kernels so we do a really good job there is a great job there. I picked the last kernel of the year. It's just a time based kernel so it looks like 65 if you look at how when we do our releases, maybe 66 depending on. Oh sorry, it'll be 66 or maybe 67 whatever comes out by the end of the year looks like it's going to be close in January for 67. And as long as it works well 61 we had to make sure that worked really well. It did it didn't have a major bugs or that we knew of. So that was good to go if 66 looks bad and maybe we'll go to 67 things like that. Anyway, that's it. Nothing new here. This is, I don't know why this suddenly got in the news, but it did. But nothing has changed here. So we're ready to change. Thanks for the link. Longer kernels, of course, slow down because it's harder to apply bug fixes from the latest kernel to an older kernel. I'll call it a number of the ret-ledon's spectrum meltdown problems that we've had are not backported into the very, very old long term kernels because they just didn't fit. So 414 is missing a lot of those. You really wanted to use a newer kernel. Hence the number rate of changes. And this is kind of interesting. So like we're running 36. So we're running nine changes an hour in latest history, we're picking out about 36 changes a day so it's an order of magnitude smaller. These are the ones that are going to this history. And these are sorry, these are the ones that go into the stable tree and the long term kernels.  So it's a smaller subset, but they all have to be in latest history. And over time older kernels are actually harder to maintain the newer kernels you need a senior engineer on that. So it's more changes to and it's more work. It usually is just not worth the effort. A lot of problem with longer kernels is older kernels is the threat model has changed. The world has changed. We now treat USB devices as insecure. So newer kernels can handle this older kernels can't. So if you're trying to still run an older kernel in a modern day, maybe the threat model doesn't match up so well it's not a good idea. Always go the newest kernel you have the latest security features and a lot of latest other issues as well. And kernel releases. Like I said every release we do is considered stable, meaning you should always be able to update to it. We guarantee we will not break anything and we do not ever want to to fear upgrading so you should always be able to update kernel should just work. When you have other problems with out of tree code out of tree drivers that's on your own companies know about that they are paying their own developers to maintain that. And they need to pay them to drag them forward. You'll see a lot of SSC trees that have a lot of out of tree code. The Pixel 6 has a lot of out of tree code it has 300 out of tree drivers. They have successfully run every single RC release from Linus for the past two and a half three years with their tree. So you can drag it forward every single release works just fine upgrade works. So it can be done. It is not very difficult. So when people say oh we have too much code out of tree we can never update to a new kernel. I think that you just don't want to do the work. It is possible. It isn't that hard at all. We have a couple of questions in the in the chat. So, yeah, no older kernels might not be compatible with newer ones are compilable. Well, older kernels may not be compatible with newer compilers. So yes, so older kernels are harder to compile with newer versions of GCC and clang. That's very interesting as well. Darren said the core kernel team LTS duplicate work code. CIP is different. There are different projects. CIP piggybacks on top of the LTS work. Talk to the CIP people. I talked to them pretty frequently. So it's a different project. CIP is saying we're going to only support the kernel in this tiny little subset of these pretty much three million lines of code and we're going to maintain these based on these configurations. So they're taking a subset of the kernel maintaining longer. They're piggybacking and relying working on the work, building on the work that I do and Sasha does. So we work together, not duplicate effort. So how does the security team, all right, I'll talk more about security bug fixes later. So I'll talk about that. Oh, and actually, how do we, the security team identify bugs that only exist in the stable version. That's very, very rare. We do get bug reports saying this isn't a bug in an older kernel. The fact the odds of it not being a bug in the newer kernel is usually very slim. Sometimes it's because we fixed in a newer kernel, and we just backport the fix and away you go. That's the majority of it very, very rarely will we see that something is only applicable to an older kernel version, just by the way code works.  You'll see there is a bug. I'll call out there's a bug fixed for butterfs file systems on the stable mailing list saying hey this only applies to older ones. Butterfests maintainers. Can you please review this to make sure it's okay. We just ask the developers. And why Android not find it difficult with shorter LTS versions. No. Another EU fund rule is the EU rule about keeping devices alive and maintainable. So there's a new rule that came out I think it starts the beginning of next year that any new device that shipped must be supported for five years after the last device was shipped. It means that really a lifespan of software in these devices for seven years. So to do that, you couldn't even do a six year LTS. It just doesn't work. It doesn't work at all. So Android is going to have to update their kernel and their devices over the lifespan. And it's going to work. They agree. I was talking to Samsung, and Samsung was like we update 2000 packages every year and Samsung phones because of the new Android version. What's the kernel? It's just one more package. We'll do it. So new device devices will be updating the kernel, be it every year every two years talking to different companies to a new major version in their devices over the lifespan of the device because they know it needs to stay alive. It's just a good maintenance of devices. So Android is actually going to be fine. So and I'm working closely with Android team. So it's not an issue. Cool. And how'd I get involved in stable kernel I started it in the beginning of Christchurch. There's another talk all about stable in the kernel development process. I think I caught everything. Good. One last just came in. I'm learning about debugging on the kernel dummies.org. I'd say go there. And we have also a lot of webinars from various developers on the, this webinar site too. So there is several of them. Go ahead and check the check those out. Okay. Oh, and a lot of these questions are already answered. I gotta have a whole blog post about this and how to pick a good kernel what the different kernel versions are. And look at that. There's a whole bunch of information. If you have questions after reading that, please let me know. It should explain a lot about here and also talks a little bit about the security stuff, which I'll go into some more. But also here's something I want to real people to realize and a lot of people don't. The majority of the world does not run commercial distributions anymore. In the beginning Linux kernel, Linux distributions were a very big deal. Companies wanted support. They wanted to do it that way. And they grew and Red Hat and Sousa who I used to work for are great. And still are the most economical. There are paid oracle paid distributions that work very well. Turns out, the majority of the world realize that they can run community distros instead. They can run their stuff there on. Very large cloud providers very large ad providers all run debbie and they run their own kernels. based on kernel.org, so much so that over 80% of the world's servers at least are running not commercial systems. They're not running commercial Linux. They're not running a supported one. They're usually running Debian. Debian runs the world. And then even more so, Android. Android is in billions of devices. All the other Linux systems are a rounding error in there. They're all running kernel.org kernel with a little bit of Android on top of it, some patches that are all open source. Android developers work upstream very, very well. And it's all the latest LTS releases. Android requires the latest LTS releases in their devices. So these are not companies that are happening. It's the communities. Debian is developers, the kernel developer community, and it is not companies. So when hardware companies or other companies want to try and report problems or issues, they try and do like corporate NDAs or corporate companies talking. It just doesn't work. They do not achieve anything. And we, community, of course, cannot sign NDAs. It's just as simple as that. So you have to work with the community. Companies need to work with that and realize that. So again, 80% at least of the servers are out there are running Debian or kernel.org servers in the world. They're not running Red Hat. They're not running RHEL. They're not running Oracle. They're not running Rocky. They're running other things. Now, not to say there isn't a huge quantity that are running Red Hat, Suza, Rocky, all these other ones. There is, but it's just not the majority. So please always remember that. So let's talk about the Kernel security team.  And I'm going to talk about reactive security. We do not do proactive security. There's a whole other, the Lennox Harding Project run by Case Cook and other developers. Gustavo does a really good job. He's sponsored by Lennox Foundation. They do proactive security. They work in the kernel to add proactive security features, find whole classes of patterns of bugs that could happen and knock those away. They change the code to work so the compiler can prevent things from ever happening in the future. That's what they do. We just do reactive security. We are reacting to bugs that are reported to us. Again, two different groups. Oh, there's also the LSM, the security model people. They're doing security models like SE Lennox and App Armor and such like that, SMAC. Those are other types of things. This is just talk about reactive security. Again, other groups do the proactive. So this started back in 2005. Developer asked us, hey, where do we report bugs to? And they wanted to know what was going on. And we realized we didn't really have an official way to do this. So Chris Wright, who was working for OSDL at the time and now he's the CTO of Red Hat, got together with a bunch of us and came up with some rules and posted a patch and said, here's the security contact. And we started with the initial version of a couple of people and we went and so that was 2005. 2005 was an interesting year. It's the year the stable kernels started and it's the year they get started and it's the year that we had security team. 2005 was like the year we grew up. It's kind of interesting. One really good thing that Chris Wright and others, the lawyers that helped us and Lennox, from the very, very beginning, we said we are individuals and we are not a formal body and we cannot sign NDAs. And that has come back to save us so many times. It's a very, very important thing that people need to realize. We work and we say we will not disclose anything and we will not talk about anything and we have never broken that promise. A long time ago, we had a few emails that got leaked from some broken email systems but nothing ever leaked from us on purpose or accidentally, whatever it was about. So we did not do NDAs and we're not a formal body. We're there by individuals.  They were there on their own recognition. We're bunch of current developers. We cannot represent any companies. Sometimes there was somebody on there once that was representing a company or was trying to take the information that they learned there and told their company about it but that person's not on the list anymore. Alias, this isn't even a mailing list. It's just an alias. You send an email to there and it explodes out. It's not stored anywhere else. It's just an email explorer. And that's it.  We create it and then we work to fix it. All our job is you report a bug to us. If it isn't in one of our subsystems or areas, we will drag in the responsible developer. We'll work to fix as soon as possible and we will get it merged into Linux history and then we're done. And then we don't say anything else. So we move on to the next issue. We get about one or two reports, real reports a week. Some times more, sometimes less. We say we work to create a fix as soon as possible. Some of them take longer. One is taken, I know, six months. There was a networking randomization bug that took I think over a year, year and a half working with a researcher. We do try and get them done as soon as possible but we sometimes also things happen and some bugs are harder to fix than others. That's it. And here's the interesting thing. People ask, do we wanna join the security team? You're asked to join the security team if your subsystem has enough problems that we keep having to drag you in. So that's how the group has grown over time. We're like, hey, we keep copying you. Let's just drag you in now and that way we save the round trip up another email and you're on the list. You handle all the bugs that are associated with your subsystem. So it's not really a good idea to be added but people do a good job. So, and then every once in a while we'll go through and ask that for everybody. Still wants to be on the list or not and herds the list and keep things up today.  So it changes over the years but that's it, we're just a bunch of kernel developers. Our rules are we fix the issue as soon as possible and we do not, once we have a fix we do not want to ever embargo it. We don't ever want to sit on a fix that we know. Our rule says seven days. I think we did one for 14 days once after the fix happened just because we needed the vendor needed some time to roll it out to their fleet and they worked with us. They were the ones that reported it up to us so they knew about the issue ahead of time. And of course we do not do any kind of announcements. We don't tell anybody anything. If the reporter wants to say something afterwards that's up to them but we do not do anything on our own. And is working on a security question, paid work for most of us or private fund. Some of us might be able to get away with doing it on our company time but the majority of us are doing it as an individual and as our private. So private emails, personal emails and we do it on our own. So we are not able to do this as part of a job unless your manager or your company allows you to do this but again you do not represent a company. And that's it, we don't do any kind of announcements and that has caused people confusion over the years and people keep wondering why and people don't really understand the issues involved and so much so that in 2008, right after the team got started for a couple of years people were complaining that we were fixing bugs and not telling anybody what the bugs were. And Linus wrote back, here's the email and I'll have a link to the full thread in a minute. The problem with the kernel is bugs are just bugs. We don't know if their security bugs are not usually. Sometimes we do, most of the time we don't. So we don't want to ever explicitly mark a bug as a security bug because then that takes away from the bugs that we don't mark as security bugs. And Linus here calls us out saying people might think this is a trustworthy thing if we say this is or this is not, it is not true. We do not know how to classify bugs and Linus goes on and on all bugs are fixed all fixes are important. You should take them all. You made the guarantee that we're not gonna break user space so there's nothing wrong with taking all the fixes. So take them all. And here's the whole links for the whole thing. And you just want to take them all. And this has come back to be proven over and over again. Many, many years ago, there was a bug fixed in my code in the T2I layer that I wrote the bug originally and I fixed a couple of years later. I thought it was just a normal bug fix. Turns out it was able to be cause you could get root on all the rel systems in the world because they never had updated to it. I didn't realize it was a security fix and I mean, I think I know what a security fix is at times but I didn't, but it was a bug fix. And any bugs that the kernel layer have the potential to be a security issue because we're operating in a privileged environment and they could be exploited. And people have amazed us with what they've been able to exploit. We've had just an off by one count people can actually exploit, things like that. So people can exploit things that are very, very good. I'll keep on going. Here's that link to where to report bugs and things like that. I do also want to call out, remember we're going 36 patches fixed in the LTS kernel today. One company a couple of years ago said we were going to evaluate every single patch that goes into a stable kernel and declare whether it is or is not a security fix. I said, wonderful, that's great, please do that work. They came back to me in a year and they said there's no way they couldn't keep up. They took them too long. They didn't know how the use cases were everything. They didn't know how things ever worked for other people versus how they're, even in their system sometimes they couldn't figure it out. And so they said, we're just going to take them all. And that's the best thing to do, just take all the bugs. The Android kernel security team did a study and they proved that 90% of all their bugs, security bugs that were reported to them were already fixed in the stable kernel tree before they knew about it. So, and if you look at CVE, CVEs don't matter, but CVEs were showing up an average report date of a hundred days old. That's because Red Hat abuses the CVE system, but the way that bugs are fixed in the kernel, we fix them, push out the fix and we don't report anything. We need people to update them and then update their systems with them and then they have the fix before it's ever publicly known, be it by us or by somebody else or just by people figuring it out that that was a fix that this is a security issue. You should always take the bug fixes. And again, almost all bugs can be a security issue. I think there was some comments, questions, questions, questions. Yeah, there are a few questions here. Questions, okay, cool. It starts at, is Android considered? Is Android considered a digital and sorcery? Android is, I mean, all devices, everybody has a fork, right? And I have a fork for my systems, so patches that are applied on top of Linux's are a release base. I have a sys call that I use that nobody else has and upstream rejected it. Same thing with Android. There's a handful of patches. They give a report every year showing how many have been accepted, how many have been merged, how many are remaining. It's just a couple of patches on top of the latest tree and then they go. So it's not very different from the LTS version. Android tree is public. You can actually boot a device, an Android device without the Android specific patches, but there's some things that they add that a few user space interactions to some things that some Android tools like. So it's not very different from the LTS version at all. Again, go download it, you can see. Android patches not present in LTS Cursion. They are, Android common kernel is maintained a different way. That's up to Google, talk to them. It's just stuff that isn't, most of the stuff in there is upstream. Some stuff isn't, some stuff comes vendor, some stuff is backported, some stuff is an LTS is someone, it's up to them. And user space never breaks a new kernel, sometimes it breaks a third party kernel modules. We don't care about third party kernel modules. Literally, we don't know about them. We don't care about them. It's up to you what you do with that. That's out of tree, you're paying. It costs time and money to keep code out of the kernel. You're doing that for a reason that's on you. But again, I showed pixel 300 out of tree kernel modules update every single RC release is possible. Maybe I changes, API changes are very small, usually. Easy to keep up to date. You can do it if they want to do it. Luckily, Google's finally trying to move those 300 patches or modules upstream, but that's up to you. We do not have an internal kernel API that's stable. Oh, we have a document called stable API nonsense. Talks about why you don't want a stable API. You can have it and you can do that, where you go from there. We do change the kernel structures and APIs without notice, whether the code is the notice itself. You do it all the time. Again, out of tree drivers, we do not care. And that is the cost of out of tree drivers, just merge your code upstream. There's no reason not to get your code upstream. We will take drivers for if there's the one piece of hardware out there. You're maintaining code out of the tree for some reason, usually because you like it and you need it. Submit it upstream, we'll take it. If it's a driver, wonderful, we'll take it. I think I caught everything, did I, Shira? Yes, I think you have. I was wondering if, yes, I think there is one question I thought might not have been answered. Let me take a look. Okay, at the Linux foundation, there is a type of RFC where changes, failures, new implementations, et cetera are recorded. I, do you have any comment on that? Oh, I missed that one.  There's 4,000 kernel developers, 5,000 kernel developers last year from 500 different companies. The kernel development process is independent. The Linux foundation does help with kernel development by sponsoring some of us. Obstensibly, Oracle and Google and Intel have more kernel developers than us, so they provide more to support. There is no RFC changes. The only way we accept changes is if you have working code, you submit the code because it's working for you. That proves that it works and we accept it as part of our development process. We have whole talks on how the development process works. So there's no real RFC other than working code. Internal kernel ABI changes so often works like a security feature. No, there's no obscurity in the kernel. I mean, with all the source code is there. So we're not fixing by the fact that we changed so many bugs and we fixed so many bugs. Maybe you can say that, yes, we're sneaking in known security bugs along with the other bug fixes, but also realize some subsystems refuse to be part of the kernel security process. Infamously networking for decades did not want to be part of it. They wanted all their bugs reported publicly, would deal with them, get emerged quickly and pushed out the door. They didn't want to deal with anything privately. That way they could have more developers working on it and work better. So some subsystems still work this way. So it's not a matter of visible or not. All our fixes are visible. So just take them all. And kernel doesn't really have viruses. So there's no such thing. And risk five, kernel security policy that we don't care about processor types. We support 12, 15 different architectures. Risk five is just another one, no difference there at all. Okay, I think I cut them all, right? All right, that's the tip. Okay, so again, all bugs can be a security issue. So people always complain, well, I don't want to take them all because they could cause a problem. Yes, we're human, bugs happen. So here they'll say this, a fix for a known bug is better than the potential of a fix causing a future problem. A future problem is when the found will be fixed then. And we have documented proof of this. A very big cloud provider, edge cloud provider, thought they would just take a specific fix. They weren't going to take all the fixes. They thought, oh, this is the one fix I want to fix this security issue. That was unknown CBE had to do with a spectra issue, speculation issue. So they took that and then they went off and then they reported a bug to us and the kernel security team said, hey, the mitigation doesn't really work. And we said, yeah, we know because we fixed it three weeks ago. It didn't, because, you know, some of these fixes are fixes on fixes and fixes on fixes on fixes and 20, 40, 60 long fixes because it turns out that some things don't always work right when they hit the real world, get better testing, so we fixed them. And we will fix them. And so that's the thing. We test these as a unit. We don't test these as cherry picks. We test all the fixes together and then there's a problem with that. We'll add more fixes and do a next release and add more fixes and do a next release. So always take them all. And a potential you always want, your system is always more secure if you have known bugs fixed. Otherwise you're running a system that is known to be insecure. Otherwise that's always much more riskier than the potential of future problems. And future problems are always gonna be there. So test, test the system, test your devices, test the kernel. Again, you're running a different portion of the kernel than I am. So test that all the time. Because the most important thing is we don't know how you use the kernel. So a lot of these bug fixes and otherwise there's no way I can tell if it's a security issue or even a bug fix because I don't know your use case. And like I said before, I don't know what percentage of the kernel you use. I don't know what lines of code you use, what lines of codes you don't use. I don't know any of that. And I really, really don't want to know that. Open source, the license of open source, one of the rules is that we do not dictate use. You can't tell you how to use our software or how it can be used. So we can tell you the license of the copyright and things like that. But we can't tell you how to use it. So I mean, Linux is used from what? Satellites from helicopters on Mars, automatic cow milking machines, routers, all your Android phones billions in the world, servers, keyboard, my keyboard here, runs Linux, all sorts of crazy cameras, run Linux, different types of use cases for things. I don't want to know this and I don't need to know this. Just take all the fixes that we give you and you're good and you're happy. There's some cool tools out there. I know the Android team has a patch and a script that matches up all the, they know what code they build. They know what 10% they build and what they rely on.  The three of these happen to be things that we build. So let's just review them to make sure they look good. They review those three instead of the 50 and they're good, they're happy. It makes it easier in the way they go. Much, much easier. But they take the whole patch because they don't know what other people might be using the Android kernel trees on top of the way they go from there. So we don't need to know any of this. And Ben Hawks, as a security researcher, said this really, really well. I'd really recommend going reading this blog post because, and this is also, governments have a problem with this. People don't realize that a bug can be very, very serious on one deployment, somewhat important than others and no big deal and not even a bug at all in others. It all depends on what's being used, how it's being used, how the user space is considered around it, configured around it. It's just all is this, all at the same time. So when the EU says rules like all bugs must be fixed, all vulnerabilities must be fixed. Well, how about all vulnerabilities for your device or all vulnerabilities for how it's used in the network and how it's used in this configuration? Those are more and more realistic things. Vulnerability remediation is very, very hard. So please just take them all. And about the updates. Do I have one idea why some users opt to cherry pick over updating the version? They think they know better than they don't. It's sad to say, I've worked with many, many companies of very big phone manufacturers. I walk in and say, hey, stop cherry picking patches. Look, here's your latest kernel and your latest device. And here's all those security bugs you missed because they're cherry picking things. Just take them all. There's no reason for you not to take them all. There's no reason for you to think that you should only be able to pick certain ones out. I mean, a one line fix should be tested just as much as a thousand line fix. It just, your testing infrastructure should be the same. If you don't have a way to test and update your devices, do you have bigger problems than security issues? Fix that first. And then just take them all and away you go. There should be no reason we test them as a unit. We ship them as a unit. We support them as a unit. You should never try and just cherry pick. People think that you should, I don't know why they do that. It's just not a good idea. And it's very anti-good software engineering. How much is kernel CI helping us catch bugs early? Kernel CI is awesome with testing the infrastructure for all the stable kernels and they test and see how that works. Kernel CI and Lunaro is a lot of really good testing as well, sponsored by Google. And other test systems out there, Chrome OS, Shua does it and other people do this. Kernel CI does a lot of work to catch the trees as they're going into Linux's tree and Linux's tree is up. Kernel CI tests individual tree. Kernel CI does a lot of great work. I strongly recommend Kernel CI and everybody should help. Note, I'm on the Kernel CI board and I'm a member of it and we want Kernel CI to be the unified way that we do testing for the kernel. We can't handle too many different ways of doing testing. So Kernel CI is great. Architecture specific issues. Yeah, we have architecture specific issues. Lots of different ones. That just depends on the chip. But again, I think CPU is no different than a mouse. Some mice have bugs, some CPUs have bugs. Just some people happen to run more CPUs than others. Anyway, just a hardware, we fix them and we move on. And I'll talk about CVs in a second. And my thoughts on CIS killer. CIS killer is awesome. It's great tool. It's wonderful, but it sprays a lot of noise. And I wish that the companies that run the CIS killer would provide more resources to fix the bugs that they find. They are not doing that. They've been caught off for that. They know that, they don't feel like funding it. But CIS killer is a great way to get involved in the process. She was mentor program and I help her out a little bit but she does all the work there and all her mentees start with CIS killer bugs. They're great. Great place for her to start with bug fixing. Sometimes it's a very artificial but sometimes it's actually fine. It's good stuff. You want to learn more about security vulnerabilities. There's lots of resources out there. I'm not going to talk about different security vulnerabilities and things like that. Cool, I think I caught everything, right, Shua? Nice, now. One, there's a couple of them payment artists now. Yeah. Okay, so what's the tip you would give to a company that is using really outdated kernel, say 3.18 apps? Update, update to the new kernel. That's it. Just update. There's no reason why you shouldn't be able to update. If you have out-of-tree code, drag it forward and your 3.18 kernel is known to be insecure and vulnerable. So here's the thing. When we ship a kernel, it's known and good for everything that we know at this moment in time and the world at this time. But the world changes, vulnerabilities change, ideas change, how to exploit things, networking changes. If your device is hermetically sealed and not interacting with the world, great. From 3.18, you're fine. Stuff that's off on a moon or in Mars, wonderful. You're not interacting with the world. Stuff that is interacting with the world, you have to update it. Otherwise, it's known as secure. So just update it. And there's no reason you shouldn't. I'll call out Siemens, huge giant company. Tons of industrial devices. They support devices for 20 years. I think one of their large air conditioning systems. They said, there runs Linux, and they said we update the kernel every year for the lifetime of 20 years because it's part of maintenance. You fix mechanical parts, you fix electrical parts, you update the software, keep it up to date. You're keeping a living and breathing system up to date and secure with the changes that happen in its environment, be it physical, electrical or software, same thing. And then at the very end of the lifespan, maybe the last two years, then they'll stop updating the kernel to newer versions because they know it's on its way out. Same thing with the maintenance of the device and they're gonna replace it with a new system anyway. So if Siemens, giant, giant, huge industrial company knows how to do this very well, any embedded system company can do this. It's just not. Rocket science is just being a good developer. And that being said, Linux is in Rockets too. So, cool. All right, I got a little bit more. Maybe. Okay. Again, here's our rules. We fix all the known bugs to as possible. We get the releases out to users very quickly. And but this doesn't work for hardware bugs. And it doesn't work for hardware bugs right now because hardware bugs are slow. We get notified of bugs. I got notified a while ago saying, hey, we found a bug in our CPU. We're gonna fix it in 15 months. Oh my God, I don't really wanna know that. But just fix it faster. It takes a while for them to roll up microcode and things like that. And hardware's in companies, bugs. So, we have rules about this. So, we handle these separately. We have an encrypted restricted mailing list that we spin up for each individual project. We do not have NDAs. We have an implied NDA that we will not tell anybody. Everybody as on the list cannot tell anybody else outside the list, even for the same company. You can't test it on infrastructure that is not, that has access to other people. Very, very restricted. And we cross company, where you talk to different operating systems, we drag and zen, sometimes the BSDs. You can talk to Microsoft for Windows for some issues. So, we coordinate companies and we coordinate the operating systems. And sometimes we drag in the distros as well to get the releases out on the same day. For this, we will tolerate embargoes. That being said, this is a very, very difficult thing to do. For one point, a couple of months ago, we had five of these running at once. And it was caused a problem because one developer turned out was not on one of the lists. And he got mad when it became public and all these things were fixed in his area. The colonel, we thought he was on that list because he was on the four other lists for other four issues. This is a difficult process. Linus has talked about this. And I was, these hardware companies, we are gonna try and make these embargoes much, much smaller. These long, long embargoes just don't work. In the end, we do all the work to fix the bug in the last two weeks anyway. So this, maybe we just make it a two-week embargo. Just don't tell me about the bug. So until two weeks before you're gonna make a public. Things like that. So we're trying to work through this and try and figure out how to do this very well. It'll work out a little bit better. And here's the whole description of how this whole process works for the hardware embargo list. If your company wants to be involved with this because you make hardware like chips and you're not on this list already, on that link, it has a list of companies and representatives, talk to me and we can get you out of it as needed. And again, I wanna talk about this again because this gets very important. We do not do any kind of announcements. We don't pre-announce, we don't assign CVEs. We cannot assign CVEs. And we cannot assign CVEs on purpose. I think I talk about this later. No, I don't. I gave a whole talk about CVEs. I think a link will come up later about how CVEs are broken for the kernel. Mitra knows they're broken. If you really want CVE, just go ask Mitra for one. They'll give you one. They refuse to give them to me anymore. They just don't work for the kernel. Red Hat, as an engineering department, abuses the CVE process. They admit it because it allows their engineers to add patches through their engineering process easier. So CVEs for the kernel don't mean anything. Another fun example, CVEs can't reflect the fact that they took 50 pouches to fix one issue. Things like that, I have a whole talk about this. And there's no early announcement list for anything. And let me make this more and more clear. I view any early notice list that you have that is a leak and it should be considered public. Any early notice that you have where you think you're only notifying a small subset of your users is just giving people that information to the world sooner. Unless your project isn't used by anyone. If your project's not used by anyone, then it doesn't really matter that much. These early notice lists are very nice and they don't really useful or anything. But otherwise, these should be considered public. And the reason I'll say this publicly, why would your government allow this to exist if they weren't public already? If your government didn't already get the information from these early notice lists because your government runs Linux, your government runs open source. My government runs open source, the government where I live runs open source. France, all of France runs Linux and relies on Linux, Netherlands runs Linux, Germany, all these German train systems, all these things run Linux and rely on Linux. If you have an early notice list and the government is not on it, then it's, I mean, it's just really not useful. Look at who uses Linux the most. China, Korea, North Korea uses Linux a lot. All these major, major governmental groups use Linux. So the fact that we don't have an early notice list has actually saved us. When Spectrum Meltdown happened, the US government called many corporations to task interviews and had public hearing for some. I had to talk to the Senate subcommittee about this and they asked me why we, as kernel developers, Linux did not tell the US government about this problem at a time. And we said, because we told nobody, we don't tell anybody this, so we told, we didn't tell anybody and we just fixed the problem and pushed it out as soon as we were allowed to. And they said, wonderful, that's about the only answer you could have given us. And that goes to say that either we tell all our governments or we tell nobody. So the Linux kernel, the fact that we aren't a corporation, in fact, we're individuals, in fact, we don't have NDAs, in fact, we don't do early announcements, has proven that this is the way open source software that the world relies on has to run. So just take all our fixes, because again, we don't know if they're all security fixes or not, just take them all and away you go. So call that out. So along those lines, they happen at least once a week for things we know, mostly more times because lots of public bug fixes are really security fixes anyway. We don't do all our security fixes in private. Look on the mailing list, look up, see all the fixes that go into networking stack and to scheduler, all these different subsystems are happening. They are actually security fixes and they happen in public. So there's nothing new about that. It looks like any other big fixes, sometimes we don't know them until later. And again, we don't give any CVEs anything and we don't differentiate. It's a bug, a fix, a bug is a bug is a bug as I said many years ago and that's what we mean. Here's my talk about CVEs, why they're not working for the kernel, why they don't work and you guys can go see that link to find one and how do you go for that. It's a couple of questions. Oh, sorry, get a couple of questions. All right, I'll go to school back. Oh, not many. Good, automotive market, specific changes, Harvard security model be added or extended. My car runs Linux, just fine.  Automotive is not anything more special than anything else. Linux runs in rockets, runs in satellites. If you want to add special security modules for automotive, wonderful, great. Well, we'll take it. Cars aren't, cars actually are very low margin, but they're also very low quantity as far as devices that run Linux. There's not a lot of cars out there in the world compared to how many phones there are. I don't know, there's nothing different. OSV from Google GitHub, including one for the kernel. There is a group that I can't remember their name right now that we do interact with and we ask for identifiers. And I gave a talk, if you look at my talk, I talk about how we can identify bugs and security fixes in the kernel and how they're tracked over LTS releases and how they can be tracked into your device. We do work with them and we do pretty much get a new identifier for every single fix we have, but we have identifiers. Identifiers are unique for Git, so we just use the same identifier. And we're working with them to create these stuff. And that project is run by some people that used to work for Red Hat and a few other companies and they used to be CNAs and they know Mitra and they know how the whole system is currently broken. And I think, Shua, did I get everything? Yeah, I'll set. Cool, all right, my last talk. My last slide, I mean, sorry. So I will say this, I've been saying this for a few years. If you're not using the latest, I need to put the latest in there. I was called out on that on my last talk. Latest stable long-term kernel year system is insecure because it's full of known bugs. Let's call it out right there. So I will publicly say that the enterprise kernels for most companies are insecure because they are not taking all the bug fixes. Oracle, I'll call it out, is doing a very good job. Oracle takes the LTS kernels. They do very well all the stable kernels. SUSE does a lot of the cherry picking and most of the backporting and SUSE does a good job. Red Hat is kernels off in the weeds over there. Talk to Red Hat if you wanna get their kernels. Most people that run Red Hat can run their own kernels too as well. So yeah, if you're not running the latest one, I mean, I'll call it Android does a good job. Android, their kernels are maybe six months out of date but I'll take a six months out of date kernel then a three year out of date kernel any day. Trying to narrow that window. Some devices, some vendors are very good and there's some very like fair phone stays up to date very quickly. Other smaller phone vendors stay up to date very quickly. So you can do it. Sony, Sony does a great job of keeping their devices up to date with newer kernel versions all the time. So that's it. If your kernel is not using, if you're not using the latest one, it's known and secure and that's it. I'll save this off slide. All right, any questions? Oh, one last question. What takes the most time? So we have scripts to pick out the patches and identifying the patches is easy. We have tools that do this, although my scripts are public. We've been doing this for a very long time, applying it to old trees. I'm seeing if they apply, seeing if they're relevant. It's not really long. It's up to you. It depends on the developer. It depends on the patch on how long it takes, how much part takes most time. It's just, each patch is special and unique just like every other one. So there's no other logistic reasons. We're usually good.  I was traveling for the past two weeks for conferences. I need to catch up on the rest of them tomorrow and then we'll be up to date. So it's really not very fast. How do I deal with conflicts if the script fails? The LTS are very big. If it fails, you get an email that says, hey, this failed. Do you want to have this applied? People take those failed patches and either developer can update them and send me the updated ones or people I will call out. There's some very good developers that help us out and they'll fix them up and send us updates and do that as well. So people do this all the time and you get an email. If the patch that you say needs to go to a stable kernel doesn't apply and it fails, we will let you know. What tools do we find vulnerabilities in the kernel? That is a different talk. I think we had a whole talk show about from Gustavo and whatnot on how they find vulnerabilities and how they fix things. That's it. I'm doing reactive when you have to refine the problem. Other people find problems, otherwise. Yeah, we have a few talks, several on proactive, like you're saying, KMSAN, KSAN. So we have several talks on how to grow, proactively find things and fix them. Take a look at that static analysis, dynamic analysis and Sysbot. What Sysbot does is falls into fuzzing and dynamic bug finding. Yeah, go ahead and take a look at those.  If you're interested in doing that, there's tons of, I mean, we have just look at the Sysbot output and pick up bugs from there, do that. Finally, people do try and report Sysbot bugs to the security team. We almost do not accept those anymore because they're all been publicly found anyway. I just go and fix them. It's not an issue. Is being a developer mandatory to contribute?  That's it. There's whole talks on how to get involved in the current old development process. I think she was given a great one about that. We have tutorials online. There's processes for getting involved.   All right, Rust. Rust is cool. Rust is a nice language. We are doing fun things with it. That being said, Rust only solves problems where you have some type of reference counting or some type of variable going out of scope bugs. It does not solve logic bugs. If you look at the majority of our security bugs or majority of bugs that are going in the kernel, we have lots of logic bugs. Like oops, we didn't check for that or we didn't know to check for that or the range that we checked for that was the wrong range to check for. Rust won't solve that at all. That being said, Rust will take out a huge swatch of some types of bugs. And if you look at the work the case and Gustavo are doing, they're taking out a huge swatch of those same type of bugs in C already. We have ways to handle strings better. We have bounded check. We have bounded arrays. Lots of those things that Rust ostensibly solves for us is already solved in C. It'll be nice to see if somebody gets a Rust driver in the tree that actually does something. The problem with Rust in the kernel is people are like, we'll just do a driver. Drivers consume from all of the kernel. So you need to have bindings from Rust to pretty much all the kernel in order to get a driver to work properly. It'd be easier to write a poorer portion of the kernel in Rust today than it would be to write a driver. That being said, there are patches out there for lots of cool stuff on the Rust mailing list. People are working on bindings. I know the Android team has a binder written in Rust. There's some five driver patches in Rust on the mailing list right now that people are reviewing that actually look pretty good. You can write a nice tiny little five driver. Five drivers are pretty well safe, self-contained in Rust. Now that being, if it actually solves any bugs or not, it's actually really easy to write a five driver and see that isn't buggy. Yeah, this is interesting. Let's see how it works. So it might take a few classes of it, but people aren't standing still. They're fixing those types of classes in C to keep them us from ever having them happen. They're working with the compiler developers as well. And there's also open talks on here on this process about Rust. There's training for Rust and there's a whole bunch of cool stuff by the Rust developers in this presentation series. I go recommend watching those. Whereas the entire kernel move to Rust in the near future, that's near future is not very realistic. You can't just rewrite code. It doesn't work that way. Oh, and there's the, of course, the Apple, the RM driver is written in Rust. But if you look at when that driver was submitted for inclusion, it turns out there was bugs in it because it was making some assumptions about how the RM driver should work that were not correctly. So these were just pure logic bugs and logics with how it interacted with the C code that Rust, the language didn't solve but these were still problems in Rust, the driver. So that's a good example of how Rust doesn't solve logic bugs, but it does allow you to write nice drivers in Rust if you want to. Anyway, I do not know what Falco is between Kubernetes and the kernel, sorry. And I think I answered all the questions, right Shua? There is one looks like about, you might have answered this one.  It's a little bit different question about the modern. Yeah, I just want. Modern driver production. Again, you can write a driver today in Rust if you want to. This step interaction with the kernel, with the C code and the driver code, that's the interesting part. Whether you like, for Fi, you can write a driver based on the bindings that are being posted right now. Whether that's accepted or not, let's find some issues involved in how we do bindings. If you want to try, but I will say a DRM graphics driver has been written in Rust for the Apple hardware and works wonderfully. So you can go use that today. It isn't merged yet. Do all the bindings aren't all merged up but they're working on it. Developer did that, did a wonderful job. They're great developers. So there's a real world example of a modern driver in Rust today, but it's not merged yet. It'll get there. A shot, Alex, yeah, those developers. And everything in the kernels that are continuous development, that doesn't stop. I think it's, did I run out? No, I saw time, but anyway. Linux security versus commercial players like Q and X security. Q and X is a different type of operating system. It's a microkernel, it's slow, doesn't run on all the hardware we run, doesn't have the use cases that we have, it's just a different model. So Q and X is nice. If it works for your hardware and your use case, great. Just like SC4, SLE4. Great, wonderful types of things. Different types of security models. That's, again, the security model argument like for the LSM or the SC Linux or the Alparmar or things like that, different types of things. And there is not going to be a new version of the Linux device drivers book, as I say every time. A Rally doesn't want to do one, so that's up to a Rally. They all have the copyright, so we are not gonna do it. Sorry. When there's an API change, we must fix all the drivers in the kernel. The maintainer doesn't do the work. No, if you submit a patch to me that breaks the API without fixing all the internal users of the API, I can't take your fix. It's that simple. Making cross tree API changes is hard on purpose in a way. We do it all the time though, so it can be done. There's tools and procedures for doing it, but it's up to the person who wants the API change in the kernel to make the API change for everybody. That's one of the advantages of getting your code upstream. You get your code upstream, it's merged, any API changes that would break it on the build and hopefully as it runs are fixed by the developer who made the change that would have broken it. That's why you want to get your code upstream. You don't have to adopt staging for your API. I would disagree. Staging is actually pretty simple, but yeah, maybe you can ignore staging. Staging there for kernel developers to get involved in the kernel, it's full of lots of very old, crafty code, and it's a great place to get involved to learning how to fix up code in a very safe and self-contained way. We use it for mentor programs, for interim programs, and outreach uses it, and people can fix it up there. And I think I got it all. Russ and the kernel were replaced C in the future. This has already been answered, sorry. I don't think so. Again, look at the other presentations in the series that talk about Russ. They answered that question. There's nothing wrong with C. C actually works really well.  Should anything else I need to do or something? Yeah, I don't think so. And there are no more questions. I think all of the questions are answered. If you have a question, you can either put it in the Q&A or chat or raise your hand. I think we have, we don't have any questions. Could I get any of the Q&A channels? Linux Foundation do succession planning. That's plenty. We know what, we've talked about it internally. The kernel developers know what we're doing. Don't worry about it. It's not the Linux Foundation, it's a project. Projects, we have to let ourselves. Yeah, there seemed to be a lot of confusion about what Linux Foundation does and what is the difference between Linux and Linux Foundation? So it's a Linux project is a separate project. Happens to be under Linux Foundation's umbrella. Yeah, I mean, Linux Foundation is a, it holds the copy, it holds the, doesn't hold the copyright. It holds the trademark, actually a sub-corporation holds the trademark of Linux. And it acts as a help for us to do foundational issues. If the Linux Foundation wasn't there, we would have created something like the Linux Foundation to do this. All big open source projects need this, they need to be able to do marketing, they need to do conferences, do things like that. Linux Foundation helps out huge, it does our conferences for us, it does stuff like this, it sponsors three of our developers, but sensibly it's there as a trade organization for a legal place for companies who wish to contribute on projects together to talk in a way that's allowed by law. So to deal with anti-monopolistic rules, you have to be able to do this in public in a trade organization. That's why it has to be a trade organization. There's just the law in different countries. There's now the Linux Foundation EU, there's the Linux Foundation China, it's been Linux Foundation Japan for forever. Linux Foundation Korea, I think might not still be around, but that was around for a while. It's just a law, it's like the milk board, milk corporations have to talk to each other through that as well. So, Linux Foundation doesn't control development, doesn't tell us what to do. In fact, like our contracts say, I say all the time, I can't tell Linux Foundation what to do and they can't tell me what to do. So they just, we do our own work as far as it. And can me and Linus travel on the same plane? Since we live in different parts of the world, we used to live in the same city.  Oh, no, we've traveled on the same plane before. Yeah, it's not a big deal. It's not a big issue. Anyway. That is one about, oh, sorry. Oh, that is one about static. Is all kernel code statically tested? Yes. So yeah, so we have tools. Coconel is a wonderful tool by Julia Luol and company or her graduate students. She has done a great job. I think she's given a mentorship series, talk about it as well. She's ostensibly fixed more security bugs on the kernel than anybody else and preventing them from coming back in. So we have tons of good static analysis tool. Dan's... Carpenter. Smash. Smash, Dan's carpenter. Smash, smack is a security model. Smash is a great static analysis tool from Dan Carpenter. He does a lot of great work. We have a very good static analysis tool. We even run co-varity.  Co-varity is a little interesting, but that's a close source corporate tool. We do run that as well. You can fix those bugs if you want to. But yeah, we run a lot of static analysis tools on our code body all the time. And we run it on every patch that's ever submitted. So every patch is submitted. We have bots that go out and run all the Coconel tools and Dan runs his tools on them and we report bugs if there's anything wrong with it. We won't take the patch. So that's actually it. And all of them did webinars. So go check those out in this static analysis series that we have on the same site that you'll find Greg Stark as well. Great, yeah, they do great work. You should see that. If you're interested, go check their workout. Very important for us. Standard checklist of tools. Yeah, I wish we could run to make sure new series is not breaking something. Test your workload. So I don't know what your workload is. My, your workload is different from my workload. My workload involves my server or other things to that. So my phone, phones have a different workload. So test and exercise the code in a way that solves your problem. Linux is a tool to solve your problem. That's all it is. It's a utility for solving a problem that you need to solve somewhere else to do something else. So test it to see if it solves what you need. Let's check this. Kernel documentation is updated. Most of the time, all the time in the code itself we have self-documenting code in that we write the code comments that get pulled into documentation. The website's updated every RC release. The entry kernel is documented. I think Jonathan's given a whole talk in the mentorship series about kernel documentation, right, Shua? If not, I should.  I haven't yet. Yes. That would be good. So documentation is always up to date. How to envision an evolution of external security measures in coming years. Talk to the kernel, the people that are doing that. There's the Linux hardening project. They know they have a whole long list of things that they want to do to the kernel to make it better for handle hardening in different types of situations. Work with them. They have a whole list of cases, a great list for people to just start picking up a going from there. So that's the evolution right there. That's a good start. Otherwise, if you have other ideas, submit them and go for that. CVEs, again, don't matter for the kernel. So don't even think about that. Anybody can ask for a CVE, it doesn't matter. Again, if anybody takes about CVEs, mean nothing for Linux, doesn't matter at all. I think that's probably good enough. Thank you, everybody. Thanks for coming. And thanks, Shua, for running. This is a very, very good resource for everybody. Really appreciate it. Thank you. Thank you, Greg and Shua, for your time today and thank you, everyone, for joining us. As a reminder, this recording will be on the Linux Foundation's YouTube page later today and a copy of the presentation slides will be added to the Linux Foundation website. We hope you are able to join us for future mentorship sessions. Have a wonderful day.


Na kumatev siri na mtumont Kenya United under the ownership of a politician and people thought that things would change at the helm but still there is that complain of unpaid allowance and salaries from the playing unit. I think they picked their first point last weekend against Sharks and that was quite a surprise. This particular, we came today and talking of Viga United, they got beaten by TASCA last weekend. What's your expectation? Do you think it will be a walk in the park for Viga United against Mount Kenya United? I think considering that Mount Kenya haven't got a win yet if I'm not wrong in the season, it could be a tough match for Viga. Futbol, nama na Kenya, but worldwide is a very expensive venture. And some of these guys that come in, you know, you look at it from the outside, it looks very green, the grass looks very green and it looks easy. You look at the wage bill and you see this is manageable, but then you start looking at the niti grittis. You know, everyday training, the preparation. Do they need lunch? Do they need double sessions? Do the players need fare to go to where they are going to? Do we have sick players? Do they need medical attention? Which needs to be paid when you start looking at these logistics. Then you realize that it's not as easy as it might have looked before you got there. So these are some of the things I think that this club is struggling with because, you know, in this day and age and after all these years that players have been complaining about their welfare and, you know, not being paid and all these things not being taken care of, you would expect someone who comes into the playing field knows exactly what needs to be done for the players' welfare and to get the clubs to where they need to be. So they are struggling with that, yes, but when you get to the 90 minutes, I think it's going to be a very tough match for Vihiga United, not a walk in the park, but definitely not an easy match for Mount Kenyan United. Another former club of yours FC Leopard is playing, so here a football club, so here has been the Kenyan version of Southampton always getting raided, they are technical b**** getting, you know, tell them, Barnard Malala left for Bandara and has been doing spectacularly good at the docas, but he also left with some key players and Gurmaa Football Club last season raided them and signed several of the influential players. Do you think that has been such a setback to their title beat especially after getting promoted to top flight last season and they are playing FC Leopard which is also getting into the style after recording three maximum points last weekend against Rangers? Fonzoya, I think the way the team is set up, they are not set up so that they can win the Kenyan Premier League and I think they are doing a good job bringing through the talent and developing it and actually exposing it so that these players get better clubs in terms of payment, in terms of exposure. So when you go to play for Gurmaa here you are almost certain of playing in the continental competitions number one, number two, possibly getting into the Kenyan national team. So I think they are doing a good job even with losing off their players they don't seem to be risking relegation like the other teams, some of the other clubs have done. So I watched Fonzoya play against Matare United, they have a good team, they lost a lot of chances in there. I think that was their first to second match in the Kenyan Premier League, they lost a lot of chances, they have very good players. One thing that I like about Fonzoya is that they bring young players through. So in this match they had a player who was 18 years old. We had a player who had played for the Kenyan U20 team he was only 19 and this was his first season in the Kenyan Premier League. So when you bring through young players who have great ability and the talent I don't see a chance for you going wrong especially when you show them that you have belief and faith in them. And Matare United just like we spoke earlier playing against Gurumia Football Club and up he'll task for Francis Kimans his charges and Dennis Oliec likely to feature. I know these are fixer that everyone is looking forward and probably with the comeback of Oliec on board a lot of fans attend and expected the place will be crowded. Do you think the Slambos will get frightened by the signature of Oliec and even is present on the pitch as well? It's an exciting moment definitely there'll be some nervousness at the beginning it's normal but I think the boys have played together long enough to know how to handle themselves even when they face such a player and I look back when I played for Matare United back then most of the guys had not played in the Kenyan Premier League in the international matches but then we got to the Kenyan Premier League and played teams like Mumias who had seasoned players we played Gormaia who had seasoned players Taska with Ndege being one of the younger guys but they had players like Abubakar Yusuf John Lichuku they were very seasoned but none of us was afraid because you were prepared well we had worked together we had worked the journey together so when you get to such matches I think Matare United players they have been prepared well to face a player like Dennis it's just one they don't have 11 Dennis so it shouldn't be a problem for Matare United Harold Chemilil Sugar playing western steamer tomorrow I think those are one of that is one of the fixtures later for tomorrow Chemilil against western steamer western steamers we speak right now they have been in good form since getting promoted top flight sitting second 18 team log with 10 points and Chemililil Sugar has not been doing quite good despite being one of those experienced teams in top flight do you think steamer can continue with its good form against Chemililil especially now that they are playing at home of course they can because I happened to be in Kisumu when steamer played Taska football club and I could say in the beginning when they were starting the season for me I wasn't as that confident as I am with them right now because I didn't believe in the signings that they had but that much against Taska convinced me that there is something western steamer has that most of the other teams don't have they have unknown players and it's difficult to prepare against a team whose players you don't know at the time we go into a march we go into preparation knowing that Simon Mulama is going to be the midfield kingpin this is how he plays this is how he controls the ball in the second half he fizzles out because he maybe doesn't have enough energy it is easier to prepare against teams that they have interviews than you know but steamer has that niche of players who are unknown and it's going to be very difficult for the likes of Chemililil to prepare against them and that's when they come up with surprises they come up with the results that people do not expect like the fixture they had with Taska so I believe as they go into this fixture and having even won against the likes of Taska they will be having a lot of confidence the enthusiasm within the dressing room will be so high so I believe with that confidence going into that march I believe they expect to get a result if not a win Bandar is started on a high after beating Goma football club at the Mbaraki tomorrow afternoon they are playing Sony Sugar Sony as we speak sitting 4th while Bandari 2 places below on 6 Bernard Molala has been quite a revelation since he joined the Dawkas from Zoya football club what do you make of that particular tie I think looking at what Bernard Molala has achieved before and looking at where he is heading he is one of the few coaches have seen who are willing to take that risk and he has been tipped as one man probably should take charge of one of the national team even if it's under 20 I believe he is in the right direction but the problem I think we have in Kenyan football is we start we start giving individuals a lot of praise before to the benchmark which for me I believe it's very risky he has performed well he has performed very well we count eggs before they are hatched let me tell you something one of the few coaches William Molul and Bernard Molala they are in the right direction they are taking a lot of risks to the young individuals they are doing their own thing they are not recycling players they are doing something different from the rest of the coachings that will not give it up to them we are talking about the national team and yet the coaches we are talking about have not won any championship or they do not have the experience of coaching at the continental level it's going to be you are putting somebody into a lion's den then when he is slaughtered you start complaining again I think that time will come when the likes of Bernard Molala and William Molulia will be able to take the national team Okumi was not ready and his time will come what we need to do having seen the likes of Molala and William Molulia doing something good what are we doing now to get them to the right platform to get them to have the right mentality to get them to start winning titles for them to be ready for the national team what are we doing about it from the federation point of view or from the club point of view what are we doing about it because we have already seen they have potential are we assisting them are we giving them the right infrastructure for them to succeed are we giving them the right impeters for them to succeed moving forward tomorrow I think he has already prepared the team well he had a very good pre-season they went to Zanzibar you can see his mindset he takes the team away from the normal environment they are used to so that they can be able to concentrate during the pre-season and I can guarantee you Bandari is one of the title contenders for me title contenders going into this much for me I think they will win I don't think it's going to be a very big problem Bandari and Sony Sugar Sony Sugar have lots of problems on their backyard Bandari don't so I expect the players at Bandari regardless of the situation to turn up and win because everything their welfare is sorted out playing for Bandari you have all those allowances you have your salary, you have your medical you have everything sorted out for you I think I will be the devil's advocate and say based on what the coach achieved the second part of the season last year or last season coming into this season he has a lot of pressure at Bandari and to look at him he is only won twice and I think he has tied three matches he hasn't lost yet but he is not even the top four and this is a spot where everyone expected him to be in this time of the league five matches you finish with great momentum you go in, you start winning your matches first five, ten matches top three, top four and he is not there he is not winning much out of five his own twice it is a problem and it could be a lot of pressure for him which will then translate to pressure to the players because he will now start demanding more if you lose a chance then it's a problem when he came in there was a lot of freshness and you could lose a chance and he would clap and tell you it's alright it's not going to be happening this time around because now teams are in front of them Mathare United is up there Western Steamer just came the other day despite the fact that they have been in the league long enough and they are above them so I think because of this pressure to perform and replicate the same results like last year is going to cost Bandari this season and talking about coaching in such a risking decision Simon Pamzo Omolo was fired from poster rangers after a string of poor results and George Maena has taken charge in acting capacity to be assisted by FOMA long-serving international and I think one of their players Pascal they are playing kakamega homeboys this particular afternoon tomorrow afternoon and talking of homeboys it's a team that brags of all season and experienced players in the name of Alan Wanga and Moses Mudabati new signing the always there several of them having been a task FC do you think the woos of even the person who's taken over from Sami Omolo might continue tomorrow possibly and you know the exit of Pamzo you know everyone who's in the football cycles knows once technical director is brought into your team you know that's a message to the head coach you know and so it was not surprising that he left Pamzo was asked to step aside or was fired going into this match I think those are some of the things that they will be struggling with you know who's next now is it the players, is it the coaches if we don't win this match are we next on the firing line because once a coach is sent away because of performance trust me if you come next you are going to be judged by that if you start performing if you do not perform then the consequences consequences are the same playing against a team that has been together long enough I think it's one of the teams that has also signed a lot of players they've signed 13 three of them renewing their contracts including Moses Mudavadi and then all others new signings when I looked at their performance against FC Leopards in Buhungu Leopards that wasn't so good but also had not changed the starting line up so much we still had the Nabui Reyes and everyone in there I think it's going to be a very tough match for poster rangers they lost a coach, there's a big transition some of the players are struggling because of non-payment and no motivation in the team there's a lot of inside politics that go around the results that poster are posting today so if they put their heads together early enough they might be able to secure at least a point but if some of these things have not have been taken care of the fate is the same a lot of problems around the results that poster are posting today that is Simon Seve-Mulama for him but as we wind up on K-Premia review Harold the last clash probably we'll see Wulindi starts the military side playing again a zoo zoo since the patted ways I don't know whether they were raided to release Michael Madoia 2017 overall KPL of the year has it been such a problem for them in ways with him and Wulindi starts on the other side also struggling despite having been at him that boasted of positive results before I think every football club will never want to lose its best player you will always allow a player to live but deep inside your heart you do not want that kind particular player to live but eventually if a player decided he wants to live you don't have to let him go what you need to do as a football club and I think that is what zoo are doing is to try and find somebody else whom you can replace into that particular position losing the likes of Kip Kirui and Madoia it's not that easy there are contracts who are almost expiring by the time they were being released so there was no need of holding up because they wanted to live but I believe moving forward it also depends on the policy of the club I believe one of the principles of zoo FC is to nacha young talent and maybe release that particular talent so if the likes of Madoia have left if the likes of Kip Kirui have left the only thing that zoo can do is to find players who they can replace and fit into those particular positions and looking at the way they have started this season they have started much better than the way they have started in the previous two seasons so I believe there is something good going on for them and as they face Ulinzi this season Ulinzi also has been 50-50 you don't know whether they are title challengers you don't know if they are going to be a meat table team the results they are posting they are not that no consistency there is no consistency they are not that confidence the Ulinzi we know about is not the Ulinzi we are seeing at this particular moment but I believe as we start this season it is still too early to be judging most of these clubs some of them will be posting good performances some of them will not maybe because of a variety of reasons the fitness levels are not they are at the consistency is not there the camaraderie within the squad is also not there so it is too early but for that particular fixture for me I think both sides will fight for the points they will go at each other we all know Ulinzi is a physical side and we also know they might have young players but they can also apply the physicality into the game going to the match for me maybe a draw for me I don't think it's going to elicit a lot of fire power in front of goal and for that reason maybe another aspect in Kenyan football that yesterday elicited heated debate in one of the sports journalistic forums the introduction of banters in Kenyan football we've seen several clubs tweeting those tweets are sort of psychological and trying to attack the opponent Mother United for example tweeted and tagged former Gorma Dilan Kaa telling him whether he talked to his successor Hassan Okte with regards to what is supposed to expect at the slambos when the two teams lock horns tomorrow and I don't know whether the twitter handlers of these local clubs are emulating on what the foreign clubs are doing especially Borussia Dortmund and AC Roma are very good at bantering on twitter what's the impact of this is it for the positive of the game or it's a little bit pedestrian it depends all the purpose of social media in a football club if you start by that definition then maybe you'll be able to understand that football clubs need to have social media pages and to have the right content in these social media pages I will give a natural example as you are walking around the streets in town you will find commodities being displayed on windows a social media page of a football club is like a display it's like you are doing window shopping when you see something attractive on the other side of the window you'll be forced to enter the shop and maybe inquire about that particular commodity and eventually buy it so the same role that these displays do at the shops is the same role that social media accounts of sports organizations are supposed to play they are gateways into the football clubs if the content within the social media is all about bantering then we are getting it wrong banter is right once in a while it is good but professionally if you keep on every now and then your content is about bantering then for me I will not be attracted to whatever you have because that's all you got it's just banter Simon Seppemulama over to you in terms of this banter I think we need them because we need them we need them FC Leopards playing against Gormahia we need the banter we need the excitement we need to get charged so that you guys should know we are not beating us we are telling you we are winning against you so let's meet at the field but does it have the effect that is desired are we sure that everyone who is on twitter and seeing the banter will be the same guys that attend the stadium that could be the problem you and I and then anyone else who goes to the stadium had not seen this maybe he doesn't have even a cell phone that can access that so if the banter is just the right way and we go beyond the twitter and meet those guys who cannot even access the internet and let's have this banter but it doesn't have to be physical I think it's good for the game because I challenge you if you think you're going to beat us then I'll be mocking you after the match during the match we had those back in the day we had those with Gormaia and FC Leopards recently it has risen when they play you will see banter so we need them to get us charged up and excited about the match but it has to have the right ending it has to reach the correct people to have the right effect so that the main aim should be to get you guys to the stadium cheering up the boys and paying the get fees so that the boys can get their allowances right their salaries right and the motivation and everything that goes with it finally, overally, Kenyan football new year, new expectations the national team to the this year's African Cup of Nations after 15 years of waiting the last time they featured there was in 2004 mechanisms that should be put in place in terms of preparedness to the relevant stakeholders especially those handling the national team if you have to get to African continental shop and sparkle handlers of the national team I think just the most important thing is to give the coach the support he needs if he wants friendlies then he must friendlies if he needs more time with the players try to find a way of accommodating that in the busy schedule that everyone has so that he can have the time that he needs with the players because the few times that we were at the cup of nations we had very few players who played outside Kenya now this time around we have a lot of players who played outside of the country and maybe recently when Franciski Maazi and Mignya came together is when we can see at least 2-3 local players starting in the starting lineup so the challenge is when we had players that had not crossed the borders to players professionals we got to the cup of nations and at least we won a match now that we have guys that have crossed the borders what legacy is every one of you guys as a player leaving the stage with because I'm not sure if Victor Onyama will play another cup of nations or another world cup if we don't make it to the 2022 so what legacy are you leaving behind having been that player that is more experienced than your predecessors who has been more exposed and now has the big stage to carry the flag and the name of our country the biggest challenge is to the players Harold there has been inconsistency in terms of selection players who would represent the country during continental and even international assignments and you know that has also drew huge attention and a lot of criticism from football quarters getting into this showpiece which is later for a few months from now as we speak do you think Sebastian Minye the Frenchman should stick to the squad that has conquered in qualification beating Ethiopia and beating black stars of Ghana of course a majority of the players who have featured in the qualifiers of the Africa cup of nations I expect most of them to board the plane and turn up for Kenya in the Africa cup of nations but one of the reasons why we have inconsistency at the national team is because we don't have consistency at the leadership level okay today we have this coach tomorrow we have another coach so each coach has his own tactics he is going to look for his own players whom he believes will be able to fulfill whatever he wants in his tactical program secondly also we have some players who have had injuries of late you can see Kalaba has not been featured in because he has been having injury even David Checho Wino yes the players also themselves some of them are not having consistency at the club level they are not playing regularly so by the time they come into the national team you find the form level is not as where you expect so you find another player who has been having consistency at his club level will definitely perform better in training and hence will get a nod at the starting line up that's not to say there are times that a coach may overlook the form of a player and pick a player because of his experience for example if for example Victor Anyam has not been featuring regularly at Totenham Hotspur so by the time you call him to the national team you cannot call him and then put him on the bench there's no point of calling him and talking about the same Victor Anyama is such an excellent player extremely talented he's conquered at Celtic at Southampton now at Spass but of late not getting regular football club if the situation continues like this and a few months to Avcon and of course he happens to be called of which he will be called definitely he's the captain for the national team and he's done much of the country should he get guaranteed over sport in the national team despite the fact that he's not getting regular club level from my perspective it is debatable from my perspective he may get guarantee over call up but he may not get a guarantee of starting I'm saying that because Victor Anyama's presence in the dressing room is very important whether he's playing or not a lot of players who are young in the national team they look up to him they look up to him for guidance and experience might be key even at some point we might even Denis Oleg has already come back into Kenyan football at some point we may need him at the national team not to play but because of his experience because of his exposure he may be able to guide these players off the pitch so that they can get the right performances on the pitch for example if the coach gives you a higher treatment at half time maybe has a thing or two against you somebody like Denis Oleg or Victor Anyama can come and try and convince you at this particular moment you did wrong that's why the coach was so hard at you next time try and do something different or do 1, 2, 3, 4, 5 so that the coach cannot be you see such kind of he can be that brother who puts an arm on your shoulder and convince you and see the point that the coach plays is important to have these experienced players within the school they form will determine whether they play or not because you might be injured and I call into the national team you are definitely going to play so why am I calling into the national team so that you can play a certain role with these young fellas into the national team I think it's important for him I think lastly before you went up I think it's important for Victor to be there it's important that he gets the fitness that he needs because the country needs him to be in that field because he comes with the attitude and you know you talk to any football person yes the fitness yes the ability it's because you have the ability that's why you're playing even in the club that you're playing but then the right attitude wins you matches and wins you championships Victor has been exposed enough he knows the right attitude to put in we really need him when the stakes are high and when we are at that big stage so no doubt we need him on that field and I hope that in the African couple nations he'll be the one to get us at least to the quarter finals thanks gentlemen for coming through fantastic discussion quality conversation it has been on the touchline for this particular afternoon 5th of January 2019 we are just talking about sports and of course two robust gentlemen joining me to put things into perspective local headlines in the country is concerned Simon Sepemulama International played for FC and Madara United and hopefully we shall be looking forward to a scenario where FC Leipards as well would recall him to solve their midfilling tribulations and Harold Negi prominently featured under Jacob Ghostemule at TASC FC and maybe when TASC starts struggling under Robert de Bullis Matano he can also be recalled now that Dennis the menace has set the pace it's been the touchline we still continue we just taking a short break of course coming up next is how sports can spy economy well packaged featured by Dan by our very own Haligan Agade don't go away stay tuned continue talking to us at Waseke Maxwell at touchline 254 and at Y254 channel always a pleasure having on board stay tuned don't go away continue enjoying