Okay, so my name is Josh Boyer. I am theoretically the team lead of the Fedora kernel team. That really just kind of means I'm the contact point and I shield people from various nasty comments as much as possible. This is the traditional state of the Fedora kernel or Fedora kernel talk. It is not about like super cool technologies in the kernel. I'm not gonna tell you how RCU works. If you would like to know, read LWIN, excellent articles on LWIN. Today we are going to do a little bit of a release overview on how we operate the kernel in Fedora in terms of rolling out new versions and things like that. We're gonna talk a little bit about the team. It's changed over the last year or so. We're gonna have some focus areas that we're currently either working on or looking to get into in terms of what our team is doing in the kernel to make Fedora better. And then we're gonna talk about some future kernel developments. That includes like Fedora 23 and some of the new shiny things that showed up in the kernel over the past year. So release overview. We have one kernel pretty much. That's really two. But we have one kernel for every release, right? So Fedora 22 is on 4.1.5 is in testing. Yes, thank you. But right now, 4.1.4, 4.1.5 is in testing. So 4.1 is the major version. Fedora 21 is also on the same. Fedora 22, I already said that. Fedora 23 is on, we're in that weird branch state in Fedora where we just had alpha. But it's on like the 4.2 RC candidate kernels. So what we do is we kind of rebase as the new upstream release comes out and we roll it back in a graduated fashion, right? So when 4.2 comes out, Fedora 23 will get it. Rawhide is continuously rebased every day. Fedora 22 will get 4.2 probably around the 4.2.2-ish time frame, so a couple of weeks after release. And then Fedora 21 will be a week or two after that. And we do it that way so that the oldest, most stable release kind of isn't, it's not broken right away. And if there are major issues with the rebase, we tend to fix them before we roll out the next one. We do it for a few different reasons. Primarily we do it so that we stay close to upstream because the developers are either still working on it or actually they haven't heard that whole release out of their mind. And we do it to consolidate bugs, basically. Pretty simple. This next slide here is, it's an example of why I'm not a graphic designer. But it's a really terrible, not updated flow chart from the first time I did this presentation about how we do the rebases.  I'll post the slides on the schedule description afterwards. And you can marvel at my terrible skills. So right now, like I said, we're on 4.2 for Rawhide and Fedora 23. I didn't add it on this slide. That must have meant I started my slides before Alpha came out. Wow, I'm impressed with myself. Fedora 22 started with 4.0.4. The way we pick the kernel for the release is kind of, it's kind of throwing a dart and sticking it at the wall because it depends on upstream kernel cycle. It depends on the Fedora cycle and depends on how willing QA is to let us squeeze stuff in at the last minute or conversely, how willing we are to subject ourselves to the punishment that comes afterwards. So as you can see, Fedora 20, which just went into life, went from 3.11 all the way through the 4.0 kernel and it worked out pretty well. We had regressions as we go and we get those when we try to fix them up, isn't it? So any questions on how we handle the kernel in Fedora? It's pretty fast, but it's also the same way I've talked about for the past three years. Team stuff. So like I said, our team has changed right now Laura Abbott is our newest hire. Hi, Laura. Justin Forbes has been on the team almost as long as I have, right? Or did you join at the same time? Yeah, you were doing the bird stuff. Yeah, so Justin is four years, I think so. He's been on the team since the first block in a way. So at least three, and then myself, and then we have various architecture contributors. Peter Robinson, who is actually giving a talk on power and arm right now in a different room. So if you're interested in arm, that's probably where you wanna be because I'm not gonna talk about it. Is one of the only other people who actually has a consistent amount of commits in our Fedora kernel. He handles all the cool 32 bit board stuff for us. And we love him because that means we don't have to worry about it. We have Dan Horak, who does, he doesn't really do kernel stuff, but he handles S390 and he's more than happy to point out upstream patches or broken issues on S390. And then we have a very large contingent of Red Hat engineers behind us. And when I say that, I don't mean that it's like RHEL, right? RHEL has a literal arming of kernel engineers for various things. But we do get to leverage them when we ask them nicely to look at certain bugs. And we try not to do it too often because they're pretty much buried all the time in upstream and RHEL work. But when we do approach them, they're very helpful to have. So what do we do all day on the team? In the past, the next slide, the answer to this question has been about four or five slides of bug numbers, charts, graphs, showing all this other stuff. So today, there are no bugs. We don't do bugs. They're all fixed. So I'm not gonna talk about bugs. Sorry to disappoint. If you wanted to see updated graphs, I can point you at bugzilla and the queries I use to generate them and you can enjoy them all you want. There are 702 open. On any given day, there are around 700 bugs open across all the releases. So no more bugs. We're not talking about bugs. Let's talk about retrace, right? Yeah, what's retrace? Everybody know what that is? Good. Anybody not know what retrace is? Okay, retrace is, it's me being clever and saying we're not gonna talk about bugs, but really we're gonna talk about bugs. No, it's this infrastructure. So in Fedora, there is a automated bug reporting tool, ABRT. So if you get a kernel crash and it's not one that locks your machine up, say your wireless driver craps out and you get a backtrace, it'll collect that and send it to the ABRT servers and the retrace servers. So it collects all these reports in an automated fashion and then it points out how many of each instance there are. And if you go look at the retrace server, it's kind of frightening because you'll see like one specific backtrace from a wireless driver will have like 30,000 hits, which means 30,000 machines or instances has happened just for that one. And then if you look at all of them for the kernel combined, it's just as frightening as Bugzilla is, but we switched to it this year or in the process of doing that because Bugzilla is a mess. Bugzilla, there is humans involved, okay? So you get the automated report, you get the backtrace and you start poking at it and you realize there isn't enough information to even start looking at it and then you start asking people for more information. Sometimes they're really great. Justin just had a talk last session about how to get your bugs fixed, but sometimes they're not. Sometimes people are like, you know, my wireless doesn't work, just fix this now. And you ask them for information, they don't want to be participating. So it's really, really hard to get a broad overview of which bugs are hitting the most number of people by looking at Bugzilla because it's specific to an individual report. So retrace offers a high level data where we can look at it and say, okay, this i9-15 bug, you know, there's 30,000 machines hitting it or this wireless driver, there's 40,000 machines hitting it, whatever the case may be. And Laura has done some work since she joined us to help us cut down on bogus data and kind of help prioritize where we're gonna look. So we use that now. Problem areas we see, i9-15 right now. So in the last three kernels, so 4041 and what will be 42, they're doing this work in the i9-15 area and it's really for all kernel mode setting drivers where they're doing atomic mode setting.  It's all gonna be atomic now. In the past, it's kind of been this collage between X and the kernel and how it does it. It's complicated, but it hasn't been atomic. So sometimes you get things that are out of sync with each other and then you get bugs and it's nasty. So as kind of a byproduct of that or a side effect, the i9-15 drivers kind of destabilized a little bit as they work this in because it really is a complex problem. So we're seeing a lot of issues there. We're doing our best. Suspend Resume is a traditional one that we always have problems with. Right now they seem to be tied to the i9-15 problems, but they are also, you can have Suspend Resume problems for any number of reasons. Single driver not failing to go into Suspend is basically all it takes. Or you have bogus firmware. We really don't like those because you can't fix the firmware on their machines and people don't understand that sometimes. Wireless used to be like at the top of the list. As long as you're using the open source drivers, it's not that bad. That being said, the Intel wireless driver does tend to freak out a little bit these days, but it ebbs and flows. And then platform drivers is a typical one. So your function keys don't work. Like in fact, this laptop here, I can hit the brightness keys with the function keys and it doesn't work at all. I think Owen actually has the same laptop with the same problem. I have a patch to fix it and I send it and the i9-15 driver, people were like, okay, that's good, but you need to do it this way. And doing it that way requires rewriting like half of the ACPI i9-15 infrastructures so I'm not gonna do that, sorry. Fortunately, the little GNOME slider works just fine. And then Bluetooth. Bluetooth is usually an area where people suspend, they resume, and Bluetooth, that's not this nice back trace. Why? It depends on the chips that depends on the firmware. So these are the problems we see and we're using retrace to kind of help us prioritize. So what do we need to buy priorities? In terms of priorities, this is what we have. We focus on the most severe ones, obviously, first and that kind of goes counter to the number of people being impacted, but we do that to prevent, you know, CVEs from turning into major events. High retrace count reports, like I said, corruption issues, fortunately in the kernel, unless you're using butterFS, we haven't really had any corruption issues that are major as of late. I think there might have been one EXT4 bug in the past year that I could think of. Yeah. And there might have been one XFS issue where you mounted the file system in a certain way that the metadata didn't match and something minor happened. But, you know, corruption does happen. So we focus on those when they come in. And then general kernel crashes. Again, brightness, touchpad, they all kind of fall out after that. Sound is down towards the bottom of the list. It should be higher, but we just don't have the time to get to it, right? It's really frustrating when people buy a machine and they plug in their headphones and they can't hear anything. I understand that. I would be frustrated as well. We just don't have any expertise with the also subsystem. But our FS is on our list contrary to what everybody thinks. It's not that we hate it. It's just that we don't promote it by default. But it is tied to basically reporting to upstream saying, we're seeing this, are you guys seeing this? And they're really responsive when we do that, but they see a lot of issues. So they're also the hope one. And then down at the very, very bottom, after global warming on Mars, we have I686. We don't focus on Intel I686 bugs at all anymore. We sent out an email to the development list in February and said, hey, 32-bit Intel I686, if this is gonna survive, then the community needs to step up and do it. And everybody ignored us, basically. So it's just not a priority for our team. It's not that we don't have nostalgia for it and it's not that we don't even have machines. I have a 32-bit machine at home. But if it had an issue where it wouldn't boot, I would go buy a cheap 64-bit machine instead of fixing it. That's where we are today. If you really enjoy I686, please find me afterwards and I'll be happy to talk to you about it. Yeah, it's fine. There's nothing wrong with people telling us that it's broken. I have no issues with that. I would love to know that information because then I can say, look, this is what needs to get fixed. And it turns out that that issue was in the kernel for two months and nobody noticed. So, right, right. So, it's just, it's not something we focus on. It's not that we hate it. It's not that we want it to die. We just don't have the cycles to spend on it. That's all. Focus areas. So focus areas are, I mean, they're not a new concept, but it's something new that we're trying this year because staring at retraces, staring at bugs, we can do it, but it's enough to drag you crazy. There's always going to be bugs. We're not going to fix them all, unfortunately. So, what are some of the things we're looking at? We're looking at power management. This one came from Christian who conveniently left, but Matias is still here. So this one came up, not actually, it was fairly recently in terms of Fedora timeframes. What can we do to have better power management? Suspend, resume, obviously is one of them. If it doesn't work, then your power management stinks because if you fold your laptop closed and you stick it in your bag and it doesn't suspend, then when you get home, either it's overheated because it stayed in your bag and stayed on, or hibernation failed, it doesn't matter. So that's something we're not charging forward on, but it's something we're looking at right now. It's hard to figure out exactly what we want to do there. More automation. So we have a number of different projects. We already have the kernel test harness and infrastructure so that as a build comes out of Koji, which is our build system if you don't know what that is, Justin wrote this test harness and framework. It picks it up, it boots it, it runs a regression test. All in automated fashion has results on the web and it's actually very, very useful. Because before we had that, it was one person would build a kernel and we booted on our local machine or few machines and if it worked, everybody else gets it, right? So we're improving our test coverage through automation. The exploded Git tree that we had, we started at the end of after last flock, I think. So Fedora's kernel is packaged as a traditional RPM with like a tar ball and patches applied to it and stuff like that. And all the kernel developers we would talk to would say, this is terrible. How do I recreate your kernel? Because you guys are carrying patches. Which is absolutely true, but we don't carry a whole lot of patches I think. We have on any given day in raw hide somewhere between 50 and 80 patches and most of them are very, very minor. So I created an exploded Git tree that developers were used to using and that went fine until about a month and a half ago when we tried to switch to using that instead of like the traditional RPM setup as the canonical source and it worked, but it was really, really cumbersome to take that and go back into the Fedora package that everybody actually consumes. So we're gonna, I'm going to step back and look at automating the creation of the exploded tree instead of vice versa and then testing a cover with Justin. He's actually testing Linux Next. If you don't know what Linux Next is, does anybody not know what Linux Next is? Good, excellent. I see some confused looks, but it's basically what it sounds like. It's what will be the next version of Linux. So he's testing that, which we've never done before. We aren't gonna ship it because that would be a little too crazy. But, you know, testing it now means we find the bugs and we report them upstream while they're still developing them before they get into Linus Torval's tree. Which means you don't see them, which is excellent. And so I have on here upstream and it's kind of a depressing goal for a focus area in a number of ways, but our daily job is not developing kernel code. It's not even necessarily fixing kernel code at this point, but that's something we're trying to get to. Over the past year, I might have three commits in the upstream kernel and for a kernel maintainer, that's sad days.  So we're trying to have like a getting back to roots, going back upstream and making sure we're actually making an impact there as well. So that's one of the focus areas. And then at the bottom, I have our yearly plan. So I'm part of the, our whole team is part of the Fedora engineering team. It's run by Paul Freels. We try to be open and transparent what we're doing. We are some of the few people who are blessed to be working on Fedora, 100% of our time is what we get paid to do. And so kind of as a, you know, a thank you or encouraging participation part of that, what do we do? We put it on the wiki and there's our plan. So if you're interested in what we're looking at beyond what I have on the slides, you can certainly click on there. You can email us on the Fedora kernel list and ask questions and we're more than happy to answer them. Fedora 23 Outlook. All right. So Fedora 23 Alpha just came out. Yay! I think it shipped with 4.2 RC5, right? With like one extra patch to fix the I386 bug. We're probably gonna wind up releasing that with the 4.2 kernel because the development timeframe for 4.3, particularly now that we're getting into conference season for kernel developers is probably gonna lag a little bit and it'll just cut it too close to the end point. So 4.2, it'll be like a 4.2 stable release. So geez, by then it'll probably be like 4.2, 6 maybe if it lasts that long. So, but 4.3 is possible. Dependent. Some other changes that happened. So Fedora 22 came out with, was it Fedora 22 or Fedora 21 where we did kernel core and kernel modules? I think it was 22, 21. All right. So we don't tend to change the packaging of the kernel very much, but in Fedora 21 we split into kernel core which is like, it's not like a pure true core but it's a much smaller package basically usable for most VM machines and then the rest of them are in kernel modules. This time around, I was talking with Harald Horrier and his team and he said, why are we shipping stuff in slash boot? We should only be shipping in slash user slash lib, whatever. And I said, well Harald, you have to have a kernel in slash boot so Grubb can grab it and boot. You can't not ship in slash boot. So after about two weeks of trying to figure out what he was trying to do, really what he wanted was all the packages in Fedora should only install in slash user essentially so that you can go in and do like the equivalent of a factory reset and all the other directories go back to whatever they were before you installed Fedora. Makes sense. So we worked out a way so that we shipped the actual kernel part that is normally in slash boot in the RPM package itself. It's in user lib modules and then the kernel version string. And then when we install the RPM, there's some scriptlets to go and copy it over to slash boot. So it's not that we're not installing a slash boot, it's just that we do it in a weird way so you can do factory resets. And his team finds that useful. Whether that'll boil out into something that you see in Fedora, I don't know. Probably not. Somebody asked me of the day if we'd see it in containers and I'm a kernel guy and I was like, oh yeah, maybe and then I realized you don't need a kernel in containers. So probably not, but I don't know. Harold and his team come up with some crazy stuff so maybe you'll have some use out of that. Best case you just don't even notice. Minor tweaks. So just moving some modules around inside the content of kernel core and kernel modules. Nothing really big. And since the question always comes up, we still do not recommend Plutter FS as the default file system for your machine. Every year we have a conversation in some form. I think we asked on the desktop list again this year somebody did about whether we want it for workstation. So I emailed Joseph. Joseph, how do you say his last name? Do you know? Anyway, Joseph is one of the main Plutter FS developers. He used to work for Red Hat. He moved on to Fusion I.O., which was like this flash-based data center type storage company and then he's now at Facebook. Yeah. So I emailed him and said, hey, this question's gonna come up. You're the primary developer. What do you recommend? And he said, not yet. And so what they're doing at Facebook is they're rolling it out kind of behind the scenes on their storage areas and trying to scale it to 40 terabytes across one file system, which is awesome. And I've seen the commits go in for that. There's fixing a lot of corner cases still, some data corruption bugs at scale, right? So I emailed him and I said, Joseph, that's awesome, but nobody in Fedora is gonna have a 40 terabyte laptop. I'm sorry, but that just doesn't, it's not the same kind of workload that we see. So he said, they are doing other things that will produce similar workloads to what you would normally do in a laptop, but they're not there yet. Once they do it in Facebook at scale, they'll shake out all the bugs and then it'll be maybe time. So another six months, which has typically been the answer ever he released for the past three or four releases. Do you know of the Sousa guys at 8.2 as I played? They would not repeat that on the internet, right? Yeah, Joseph pointed out that Sousa, so in open Sousa, it's just the ButterFS driver just like it is in Fedora. And I think they have defaulted to it now or it was just with their latest release. Well, in SLEZ, what they did is they turned off all the fancy stuff. So there's no like transparent compression, you can't do RAID, it's basically snapshots and that's it. They only ship it on the root partition basically, right? So user data is still on like EAC4 or Exifast or something, I don't know. But we can't do that in Fedora because we've already shipped the driver for however long it's been out there since Fedora 15 or 16. And if we turn off those features then people who are using them will be broke. So we can't kind of cheat and get there before it's ready. And the open Sousa guys have worked on it for a while. They're definitely fixing things. They've, I mean, I see posts from, you know, at Sousa.com quite often for fixing bugs. So I'm sure they're running into issues in finding them, which is excellent because usually it's the way around, right? Fedora tends to be the distro that finds all the bugs and gets them fixed upstream just because they're so bleeding itch. But yeah, he just said it's not ready yet. Whether it will be ready around Fedora 24 is a wait and see. But, you know, unless you're really kind of hankering for snapshots or some of the other high level features that are awesome about it, it's really kind of a, it's like, why do you need it? Right. Yeah, I mean, if you want snapshots and rollbacks, you can do it with device snapper. You can do it with thin provisioning and things like that. It's not as integrated into the OS as we would like or simple from a command line standpoint to do. But I've been told by Steven Gallagher that they are working on it with a project called StorageD or something like that. I mean, the problem with Fedora is not really, might be the integrated by one of the features. Right. It's not really integrated from the desktop either. That's partly because it's not bad. Yes. We are told not to use it, so it would be integrated. Right, yep, absolutely. And I totally see the value of integrating it so that you can do transparent rollbacks and make it really simple.  It's just, every time I ask the person that works on it for his job, he tells me it's not ready. So I'm not gonna be like, yes, let's ship it. It's just, it's not there. But there are some other shiny things that have landed in the kernel. OverlayFS. So for the longest time, what's it called? AUFS was the kind of like the out-of-tree union filing system that everybody carried, Fedora even carried it at one point a long time ago for their, it was either their live image, I think their live images used it, long, long time ago. And then, sorry, what? Docker. Yes, Docker has a back-end layer that uses OverlayFS now. Didn't Alex do that? Alex Larson? Yeah, Alex Larson kind of wrote that. And it works. It was merged in 4.0. There's bugs here and there, but it's kind of the only, you know, it's the only upstream union filing system that we have. So. And I thought I would switch from ButterFS to... Yep, yep, butter switched what it does on the back end from ButterFS snapshots and things like that to OverlayFS to use that instead. It does now. Didn't they fix it in, didn't they fix OverlayFS and XFS? We did. Yeah, I believe so. There were other things. So XFS and Docker itself didn't get along because of namespaces. Username spaces did not work with XFS, but that's also been fixed. As of like 3.19? Yeah. So XFS should work. If it doesn't work, I would humbly be surprised and I'll go look at that because it should. KdBus. Leonard likes to talk about KdBus a lot. So if you've seen any presentation he's given in the past year or so, it's been about KdBus for the most part. Essentially all it is, is an internal IPC mechanism that kind of replaces user space debus. And it's kind of contentious. They posted it for inclusion in 4.1 and it was immediately shot down. Lots of upstream kernel developers were not very happy with how it was posted and how it was kind of positioned for inclusion. So they took 4.2 off. They said, okay, we're gonna go work on some of the feedback we got. We're not gonna post it for inclusion in 4.2. Maybe they'll try again in 4.3. Harold again, asked me, what can we do to get KdBus into Fedora in a way that it's actually usable? We used to have, we still do, have a repository called kernel playground where we kind of pulled in things that weren't quite ready upstream or things that were maybe in staging that we weren't willing to enable Fedora properly. And we pulled KdBus in there. And it worked. You can install it, but it wasn't getting the test coverage that they kind of needed. So what we did is we made this deal where we pulling KdBus into the raw hide kernel and raw hide kernel only. So if you go and you boot Fedora 23 alpha, KdBus is not in there because we said we're not gonna ship something in a release branch before it's merged upstream. So it's only raw hide. Fortunately, you can use raw hide kernels on Fedora 23 just fine. So if you wanna play with it, just install the raw hide. And then boot with KdBus equals one on the command line, kernel command line. Probably wanna set SeLinux to permissive if you're gonna do anything other than just boot. If you actually wanna log in, probably need to do that. SeLinux is being worked on. It needs kernel level support for how KdBus works. And it isn't there yet, but they are working on it. Is it something that absolutely has to happen? No. But when KdBus is merged, it will enable a lot of different things that you can't do today just because of the way if you have your init process start and it's not also your main system bus and you start your system bus later, lots of weird stuff happens when you restart the bus. So by shoving it into the kernel, you can just enable a lot more things. And then live patching. So major and I were just talking about this before the talk started. Live patching is commonly known as case splice, which is an Oracle product at the moment. It's basically exactly what it sounds like.  Fedora does not enable it. And it's simply because the amount of time and energy to produce those tiny little patches when we rebase the kernel as frequently as we do is not worth it. It is enabled in that kernel playground repository. So if you wanna play with it, you can enable it or you can go use it and then build your own K-patch patches. But we don't, I mean we just don't ship it. So for things like RHEL, which has a really long lifecycle, short, long time between updates, it makes a lot more sense. Whether or not RHEL is gonna actually use K-patch and live patching, I don't know. I don't get paid on RHEL stuff. So they might, it makes sense there, but it does it in Fedora. So that's pretty much the highlights list. Trying to think if there was one more that I was just thinking about or not. I can't think of it at the moment. So questions? Anybody? This talk goes much better when people ask me hard questions or any questions. Yeah. So like I said, it was posted for 401 that was shot down immediately. There have been patches that have come in since then to address a lot of the feedback on. There's a guy by the name of Andy Ludomirski, Ludomirski. That does a lot of like low level x86 stuff, but he also was really interested in like security aspects of things. Oh, that's what it was. Okay, I'll answer your question and then go back. So he's been reviewing the code pretty heavily and he keeps coming up with ways to boot a machine with KDBus, run a simple client and bring lots of destruction to the machine. When you take your message buffers and you move them from user space memory to kernel space memory, it's a lot easier to starve things, apparently. So it turns out that there's some bugs that he's still finding. Whether or not it gets merged for 4.3, we'll see. The way that it was kind of positioned was this is kind of like binder. You know what binder is? It's like an Android IPC system. The way Greg Crow Hartman pulled binder in is he basically said, look, there is a massive number of users for this piece of code. It doesn't impact anything outside of itself, so we're gonna pull it in. And it had some objections to it. There was some high profile kernel developers that did not wanna see binder get merged, but Linus can actually be pretty pragmatic at times. So he said, fine, if Greg's gonna maintain it and he's willing to spend the effort to do that, fine. That's kind of how KDBus was positioned. It just, it didn't work when they tried that 4.1. So the 4.3 development window will open probably in about a month and then we'll see what happens then. I'm kind of optimistic that something similar will happen because they are kind of taking the feedback that Andy and others have provided, but I don't know. I mean, if anybody knew, they would be Greg and Linus and I don't, I mean, they're pretty transparent. I don't think they're, you know, privately emailing to get it in. So we'll see what happens. Any other questions before I go back to what I was thinking of? That's an excellent question. Will real-time kernel patches ever be available? In Fedora? No, probably not. There was a discussion a couple months ago on one of the development IRC channels about somebody who wanted to provide a real-time kernel and they asked me what they could do to get it in and I just said, no, sorry. Again, it's a cycles thing, right? We're having enough trouble making sure everybody's machines work without real-time aspects to them. I don't want to provide a real-time kernel and have people expect that this robot arm that's moving is not gonna crush somebody's skull, right? So that being said, after a lengthy discussion, I said, if you want to have a real-time kernel in Fedora and you want to provide it in a Copert, which is, so if you know what, yeah. So I said, you're more than welcome to do that, it's fine. I don't care, just as long as it doesn't conflict with the actual kernel proper. It's not that we're not interested in a real-time necessarily, it's that it's a very hard problem to solve. Red Hat has paid one of our engineers, we have a whole real-time team, we have a real-time product. We've paid the kernel engineers for quite a while now to get those patches worked on and upstream and they still aren't there. So it's not only a hard problem just in terms of cycles of support, but it's a hard problem just getting the patches upstream because some of them, you know, they've made very, very good progress and now that the patches they have left are the really, really kind of needy-gritty hard ones, and so we're just not willing to carry that in Fedora. It's just the number of users we would get is very small versus the amount of effort we would have to spend rebasing patches, right? Yeah, yeah, I mean it's not like you can add the patches and have one kernel provide real-time and not real-time, it doesn't work that way. So it's a good question. I hope somebody does eventually build that copper. I don't think it's been done yet. Yeah, they've always had a real-time kernel that's kind of been based on Fedora's, I think. And I don't, maybe it's successful, maybe it's not, I don't know, but I know it's big in the audio projects and stuff like that. Any other questions? Anybody want to rant at me for not fixing a bug that they buy out? That's it. All right, well, thank you for coming.


Hey folks, my name is Pat Schloss and I'm the host of Code Club over the past couple dozen episodes I don't know how many at this point. We've been working through a project looking at exact and Amplicon sequence variants within the field of microbial ecology and how they're used I'm using this as a platform to talk about different practices in reproducible research We've talked about a lot of things like using the command line using make using version control using our using literate documents like our markdown and throwing it all together to look at this question of what threshold Should we be using to bin sequences together as a unit of inference for microbial ecology? Now again, this might sound like it's way far too into the weeds for microbiologists That you know if you're outside of microbiology, you shouldn't be paying attention But trust me you'll learn a lot as you watch about how we can use Reproducible practices to go through a data analysis plan and of course if you're a microbiologist who's interested in microbial ecology You might learn something about Whether or not we should be using exact sequence variants or Amplicon sequence variants So the goal of today's episode is to determine the threshold that we might want to use to get one ASV per genome and we could use exact sequence variants So where an ASV where the radius is zero Or diameter is zero or an ASV where perhaps the diameter is five percent five percent variation between any of the sequences within the ASV So you'll recall that in the last couple of episodes. We've talked about how we could build out those ASVs And today we're going to take those on and again what i'm thinking about creating is a plot where the y-axis Will be the number of ASVs and the x-axis will be the threshold And then each line would represent the number of ASVs that we see per genome For those different thresholds and then we could do some faceting by the region So we don't get overwhelmed with lines on the plot And the line that I want to draw isn't because we've got you know thousands of genomes in here Is I don't want to show the average What I want to show is the 95 confidence interval or the 95th percentile And so, you know if I see a threshold of say 0.03 and the number of ASVs is Two then that means that 95 percent of the genomes had Two or fewer ASVs per genome. Okay That's another way to summarize the data what we're going to see in the next few episodes really Are different strategies for thinking about all these variables. We have so let's get into the R There's really nothing new here except Thinking through the problem and seeing a lot of the commands we've seen in previous episodes In different contexts. So I've got my issue here issue number 35 I'm going to come over to my terminal get check out Our get issue, you know, actually, let me show you this other way We can check out a branch get check out hyphen b issue 35 This then does both create the branch and then checks it out So we've been doing get branch issue 35 and then get check out issue 35 get check out hyphen b Makes the branch for you. So it's a little bit more convenient. Um, I think I do both Both approaches and I don't know why I pick one over the other. Uh, you'll see it's red And that's because I've already put in a template for today's episode making this as an exploratory file with an r markdown document Before I forget, let me go ahead and open this Or create the rule for this in my make file. So I'll do add them period So here I am in my make file and I will go ahead and add it here. Um, so I think I copied it, but forget Um, tab that in and I want md and I need to add the backslash. So now my make file is good to go Um, I'll go ahead and close out Um, adam and I'll fire up our studio And so if I open up my new, um document The title I've given it is determine the threshold needed to get one asv per genome should update my date here The date being the 24th it's Thanksgiving week and I want my chunk output to go to the console as I'm developing this because If you've watched the other episodes, you know, I really struggle with this And what I've roughed in here are The code from previous episodes to read in the metadata as well as the easv data So we'll come back to this And so this is the overview that I've copied from the github issue And I want to start thinking about how we're going to do this, right? So I'll put in my code chunk and Trying to think through the problem would be to get a one genome per Um species, so we'll have to figure out how to do that. We've done that in previous episodes We then will want to, um aggregate data by Species or by genome Because they'll be the same thing for our purposes And we will then want to count The number of Easvs per genome Four so we want to aggregate by by species genome By the region, right? So v4 v35 v19 whatever and by threshold I think that's it and then we want to we want to count the number of easvs per Per grouping, right? And then we'd like to plot the number of um or determine the 90th or 95th Uh percentile Uh for each grouping So each um for each region And threshold And then we want to plot The 95th percentile Um as a function of um the region And threshold Okay, uh, so let me make sure it covers my issues. So create a line plot of 95th percentile for each threshold Um, um at a different number of operands per genome Okay, so that's the other thing that we have to remember Is that these genomes have anywhere between like one and 25 operands So I want to add to this also number of operands The rn operand per genome So we want to count that and then Determine the 95th percentile for each region threshold And a number of rn's Right um and here again Number of rn's And then another plot that we'd like to make um, so plot the Um, and maybe I'll indent this to give my Comments some organization. The other thing I want to plot is the 95th percentile um For the number of um Let's say e asv's For um those genomes with let's say seven copies um of the Of the rn operand All right And again one of the struggles that we have is that we have too many variables, right? Um, we have the region we have We have the we have the species right and so we're kind of aggregating across all species We have number of copies per genome or per species We have the threshold that we're interested in and then um Yeah, and so we've got these three or four variables And one strategy that we're going to look at in this case is this 95th percentile And so this will allow us to aggregate across all of the genomes And because some of our genomes have um Some of our species have far more genomes than others like the coli had like 900 or something And others might only have one we want a balance for that So that's that's one thing that we're going to do is this percentile And i'm not actually sure that 95th is what we want. That might be too stringent. It might not be stringent enough We don't know. Um In some ways, it's also good to know what it what it is, right? What percentile should we be using if we want to get um a You know one a e asv e asv or whatever you want to call it um per region per threshold um And and so that's also useful information to know All right, so we'll start with these two uh plots and of course in future episodes We'll we'll kick this issue around a little bit more to get better insight. Okay, so this is the game plan i'm going to come back up to My code here i'm going to go ahead and start by loading my libraries And as that's loading i'm going to think about What do I want to modify from our previous code to get it to work? Um, and you know what if I run these two lines, which is reading in my metadata I see that I get the genome ID and all this taxonomic information What i'm really interested in however um, I don't want to change any of that stuff is that i'm going to do a select um and Let's see i'm going to select to get um the genome ID And the species okay, and i'm going to So this then will get us our two columns i'm going to then Group by uh the species And as we saw a few episodes ago We can then do slice sample n equals one that will then give us One genome per species And we can then ungroup That and this should all work and because i'm using that random um slice slice sample. I need to set my seed This is fairly self. I don't know Differential or self-serving. I don't know what you call it, but I use my birth date 1976 06 20 june 20th 1976 um Maybe use your birthday. Maybe use one whatever so if I run that and then generate my metadata That then gives me uh 4774 rows, which is the number of species in the data set and I've got my genome ID and my species What I can use then is I can then as we see down here We can do our inner join to join those together. So we now turn to easv And let me make sure that this is Everything I want So we'll read in The easv column the genome the count the region the threshold. That's all good I'm going to remove these filters and selects because I want all that data. I'm not only looking at ESVs in this episode. So I'll go ahead and save this to easv We can then do our metadata. Ah You update the variable name run that and now we have metadata That should be es easv, but Make sure it looks good. And again, we've got our genome ID species The easv for that species the number of time that easv shows up in that species for that region and for that threshold All right. Um, and so this again needs to be metadata easv So one thing that you'll notice is that my threshold column is a character And that's because I've got esv's in here as well as numerical values Um, perhaps to clean that up. What I can do is I can do a mutate um on threshold And I can then say um What I'm going to do a recode So recode and we'll say, um 0.001 Sorry zero 0.000 Um Equals esv I think that works. Actually, you know what I think recode is the opposite of what You expect it to be based on the rename function. So let's do this. Um, and Run all that and see if it works Um problem with mutate input threshold argument dot x is missing. Ah, yeah, so I want to add threshold That and that worked and we now see that our threshold is 0.000 And again the syntax for recode is the opposite of rename. Um, That's kind of annoying, but whatever. Um, we still have this as a character type And what I'll do is Then add to this mutate where I'll then say threshold equals as dot numeric and then inside that threshold We could have done as dot numeric parentheses recode and all that kind of nested but It gets kind of hard to read and so now if we look at metadata esv We should see that this threshold column is a double. So we're good to go. That'll be very helpful for plotting Our threshold across the x axis as a continuous variable. Okay, so we've got metadata esv and We've already done the get one genome per species part. Maybe I'll pull this back up here to where I did that So right here And aggregate the data by species genome region threshold all this stuff So metadata easv and we're going to pipe that into this And then we'll do group by and we're going to group by Our region our threshold And our what? genome ID We could also leave in the species, but That's the basically the same thing as genome ID. I don't know that my analysis really cares what the species name is So, um, you know, maybe we'll Yeah, I don't care. I'm going to use genome ID And I think that's all I want. I don't actually want I need to figure out the number of rns per genome because I don't have a column for that So I still need to figure that out. So count the number of asv's per grouping and the number Of rns Per genome. Okay, so that's actually a secondary thing So we'll group that right And now we're going to count and so we'll do summarize And we will summarize the number of rns Per genome and that's going to be some of count right? um And our number of easv is going to be the end function. Okay, and so we run all this We will see that takes a moment or two And we see the region the threshold the genome ID The number of rns and the number of easv's. Okay. I want to go ahead and undo my grouping Um, so I'll do the dot groups drop again, we run that and the output here takes a moment And we see the region threshold genome ID And then the number of rns In that genome as well as the number of easv's so this this um genome has five copies of the six ns gene and it has four Different versions of that gene in that genome, right? So the question is That is at a threshold of zero But if we increase to a broader threshold does that number drop, right? And so we could look at that for this particular genome by doing something like filter genome ID equals equals that And this then would show us The number of times this shows up In the v19 it's got all of them here, but you can see In my output here that as we expand the definition expand the diameter of how we define an ample consequence, right? The number of unique easv's drops. Okay, and so that's what we want to look at but again across all of the genomes all the species Okay So Now we want to know the 95 5th percentile for each region threshold and number of rrns So what we'll do here is a group by And we will group by again region threshold And n rns Right And we will then Summarize And we will then summarize The upper bound as What are we kind of do so we'll do um quantile? And we will use the n easv column in our probe I will say 0.95 and that's going to get us the 95th percentile So again, this takes a moment or two to run Uh, it's complaining Um, which is kind of surprising. So let's look at what we're outputting here And so we've got n easv's Um, ah s i'm missing the s at the end silly things Um, so again, we run all that we'll get the correct output All this is running be sure that you've liked and subscribed to the channel Um, that way you'll know when the next episode is released. Uh, this is Thanksgiving week or may or may not release one on friday But hey, if you're a subscriber You'll find out you'll be the first to know and so what we see now Is that we've got our threshold and our number of rnn copies as well as the 95th percentile And so we see for this case um, that's basically linear uh, the number of rnn copies increases With uh, the number of copies in the genome But it's not easy to look at that table. I'm now going to go ahead and call this easv's by threshold and region And run that uh, so that we now have that um variable uh stored Um, something I forgot to do was to drop my groups So rerun that and we will now move on to plotting the 95th percentile As a function of region threshold and the number of copies And so we'll pipe this in a ggplot And our aes our x is going to be The nrns And our y is going to be what? the the um Yeah, not the threshold or yeah, we want to plot the 95th percentile. So it's going to be the upper bound Right and then our Color is going to be a threshold All right, and then we're going to make this a geom line And I'm going to facet it by region. So I'll then do facet a wrap And that is going to be a tilde region and let's do Yeah, that's cool. So we'll go ahead and run this And um, we get kind of crazy results and the problem is that I put threshold in quotes And now the problem is that it's treating my threshold as a continuous variable rather than as a discrete Or categorical variable. So for this case, what I'll do is I'm going to do as dot character Threshold and you'll recall that up ahead up above. We had done it by We we'd recast it from a character to numeric that will actually help us with the next plot in this case Um, uh, we actually wanted it as a character, but that's that's fine What I'm going to do actually because now that I see I've plotted this I've got all of The thresholds and it's really overwhelming to see in the output. So what we need to do is a filter On our threshold. So filter threshold And we could do a bunch of things like threshold equals equals zero Threshold and threshold or or threshold equals equals 0.01 or threshold equals equals 0.02 But that gets really long and tedious and painful what we can do instead Is percent in percent and you'll see from my typo that I just had that it's really hard for me to remember Not to use the pipe operator when I start typing with percents So what we're going to do is define a vector of thresholds that we want to look at so I'm going to look at zero Uh 0.01 0.02 0.03 0.04 and 0.05. Okay, so we'll look at Six different regions. It's probably too many. Um We might also instead want to look between zero and 0.01 So we'll look at this plot look at what it looks like and then we'll come back and reassess What threshold levels we're looking at but this should clean up the output a little bit And um, it does look better and we see What so this red line is exact sequence variance, right? So that's an esv 95 percent of the genomes have Many many copies, right? So if you have seven operons, then 95 percent of the genomes are going to have Seven different copies of the esv in the genome. So that's a lot, right? And if we look at the sub regions, we see that that goes down And again, if you look at higher thresholds or the higher diameter for how we're defining an amplicon sequence variant We see that that gets muted as well Maybe what I'll do to make this output a little bit easier to look at That will be to say n row equals four And I will make that taller. So I'll maximize that window Make it look a little bit more attractive and maybe what I'll do is I will go ahead and get rid of These higher to the 0.04 and 0.05 It looks like we do want to look between at these kind of hundreds place levels because It seems like there's still stuff going on at these levels and Again, you could kind of contextualize this however you'd like. The other thing I'm going to do to make Give more room on the right here is this as character threshold. I'm going to pull that out and make a mutate So threshold Is as dot numeric or as character, right? And we'll pipe that in and I can then remove this as character bit And that will then instead of saying as character threshold as the variable name in the legend It's going to give us threshold. And so we can see that like an exact sequence variant for v4 Still is 95 of the genomes at a decent level of Number of copies in the genome is still giving us three or four different ESVs, right? So it's splitting one genome into multiple taxonomic bins. And again, that's at a 90% confidence 95 percentile And so again, this is good because it It shows us the number of copies within a genome and that 95th percentile By our different regions and our different thresholds We can flip this in the second question by looking at the 95th percentile for the number of ESVs for those genomes with a defined number of copies per genome. And so we could pick seven And so what we'll do is very similar to what we did above So ESVs by threshold region And we will then say I'm going to leave out the filter for now and the mutate. Let's go to the ggplot AES and our x is in this case going to be threshold And our y is going to be the upper bound And we are going to color by region And We want to because I want to look at say seven copies. I'm going to go ahead and filter n our ends Equals equals seven and then pipe that into my plot. So let's go ahead and run that and see what this looks like Ah, I forgot a geome geome line So look at that And again for some reason. I don't know why I keep putting the color in quotes We now see our thresholds on the x-axis and our region Is a different each region is a different color. And so we see in this case That like v19 Never gets below two even if we go out to five percent and so even like a five percent otu 95 of the genomes are Getting split into Two two different bins and so that's not so desirable We see like v4 For the most part it does come down to One genome for like a three and five percent. I'm not sure what's going on up here, but you kind of see that as you Come down and you expand your threshold You're collapsing these esvs or asvs Into a single one per genome per species in in our analysis And so all these calls to use really fine thresholds um, it's really Really fracturing genomes into multiple bins and that's not biological, right? There's no reason why one genome would one part of the genome would respond differently than another part of the genome In your analysis that doesn't really make ecological sense, right? You know, my hand doesn't have a different ecology than my right hand, right? It's kind of a same in type of analogy One thing we can think about Is again, if I drop this probe to 90 and then rerun my analyses And looking at my plots that you'll see at a 90 threshold This does kind of dampen a bit As well as this one, right and so at a 90 confidence our v19 Does drop to one or two copies at that 3 threshold. Whereas all the other regions really Crush crash out at about one and a half percent to one copy per genome. So again I'm not sure that 95 percent tile is where we want to be at 90 percent Seems a little bit more generous ultimately it's really going to determine or depend on What you're trying to do and what gives you the the most taxonomic resolution Or the for your particular data set and so you'll recall that You know a lot of the things in our database are genomes that have been sequenced And so we're skewed by things that have genomes that have been sequenced As well as You know, those tend to also be more clinically relevant than perhaps environmentally relevant. So It all depends and again, this is a threshold of seven If I go back to a 95th percent tile run that and Let's look at four say like a genome that had four copies We see that even with a high threshold if I look at genomes that only have four copies Per genome that we do get down to one EASV for the entire genome if use a threshold of two or three percent It's certainly a much higher threshold and broader threshold than the proponents of asv asv's or esv's push Which is much lower down here, right? My conclusions Would include saying so i'm gonna Maybe I'll leave this here and I'll say Analysis depends on Our threshold or our Our comfort with uncertainty So prob equals 90 percentile or 95 percentile And the number of rn copies per genome, right and Kind of Regardless of the number of operons Or um region We need a significantly higher threshold Than es esv's or even um traditional i'll put that in quotes because Traditionals like a couple years Traditional asv's are defined at Frankly Zero point zero three percent or zero point i'm sorry three percent Doesn't look so bad for this type of analysis. I'll correct all my typos Someday I'll learn to type slower And more precise so there's a lot more we could do with this type of analysis, right? I could have bundled this in a package and I could have made a bunch of plots Um, perhaps I could have looked at different probabilities. So I could have picked um a threshold um Or I yeah, I could have picked Um, I could put put my probabilities across the x-axis my And my number and then perhaps my threshold on the y-axis and plotted the threshold that gives me one asv for each level of Uncertainty right So there's lots we could do but I think this gives us a pretty good sense of What thresholds we need to collapse the diversity of esv's down to a single one For genomes and genomes of different numbers of copies per genome So I'll go ahead and save this. I'm gonna Close that and I'll go ahead and make it so I'll do make exploratory 2020 11 24 And this we want to do md not rmd We'll generate that I'll go ahead and commit the uh change to the issue Um merge it to the main branch and we'll be good to go So over the next several episodes, we're probably not going to learn a whole lot of new r concepts But we're going to look at this problem that I talked about today Um, perhaps in different contexts the flip side of using a broader threshold to get fewer esvs per genome Then is that we increase the probability that the asv that we see in one genome Is also seen in another genome or another species And so something that we'll want to think about in the subsequent episodes is how do we balance that? And and perhaps what is more important to us? All right, so I'll go ahead and commit this And we'll see you next time for another episode of code club Please be sure that you like and subscribe to the channel Tell your friends about the videos and be sure if you're in the us or if you're anywhere Have a great week rest of your week. This is Thanksgiving here in the us and we'll talk to you next time