Test one, two, everybody hear me? Everybody not hear me? Yeah, right. I've been a speaker at every LCA since first Brisbane in 2002. And so in a lot of ways, this has kind of become sort of a regular and fairly important part of my calendar every year. And in fact, it's far from the first time that my birthday has overlapped. In fact, a year that I will probably never forget, in 2003, for lots of reasons in Perth, we actually had the banquet. And the keynote that I gave was on my birthday. So LCA and my birthday all get tangled up. And thank you very much for recognizing that this morning. Before we jump into the Q&A, though, this week, as I've talked to people, I've realized that there are a couple of loops that I should try to close. First of all, for those of you who were at LCA last year, how many of you sat in on the talk that I gave or have watched the video since? I'm really pleased to report that a year makes a heck of a difference. And we're able to move into the new house built on our property at the end of September. And the dust is literally starting to settle as is snow. It's that time of year in Colorado. Also earlier this week, my good friend Evan Moglen was asked about the status of Freedom Box and had this to say. I'm pleased to report that trading a little sleep for other things last night. I can tell you that. And to just be abundantly clear about this, the only contribution I've made to the Freedom Box O.3 release other than talking about it this morning is one package uploaded to Debian a couple of days ago since I arrived at LCA. There are folks like Sunil Mohan and Nick Daly and Peta Reinholson who've been carrying the torch over the last year or so. And I'm really appreciative of that. Go take a look at this. There's still lots of things to do. If this is a project that you're interested in and would like to help with, I'm sure they'd still love to have your help. So enough about those sorts of things. And me and all of that, well, maybe not quite. This is not the first time that I have shared the stage at LCA with these folks. Given the little ticket selling thing this morning, if my beard is worth on the order of $40,000 of Australian, a 3D printer's got to be worth something more than that. But in fact, if we go all the way back to 2003, part of the reason that year is so significant to me is that at that LCA, Linus and Tridge and I were asked to do something that hadn't been done until then at an LCA, which was to just take questions from the audience for a while. And I have said to the organizer several times since that that's one of my personal favorite sessions at an LCA ever. And part of the reason for that is that I actually like taking questions from people. And I like having some sense of what other people want to hear about as opposed to just standing and talking for a really long time. The other part was I think that was actually the first time that Linus and I or Tridge and I or indeed the three of us all together had actually shared a stage. And that has sort of set the stage, if you will, for lots of things that have happened since. Of course, there's some people who just couldn't leave well enough alone. And I think within two questions? Yeah, so that's my cue there. So I was reviewing the audio from 2003, which how many of you here were at the 2003 Q&A? Fantastic. Great. So can anyone remember who interjected in the first two questions? Rusty. Rusty, right. So the first two he interjected and then we realized that it was inevitable. And we called him down to the front to join the panel. And we thought we might as well just short circuit that now because thank you, Rusty. You notice we already set up the fourth chair ready for you. So the other thing we had at the 2003 event was we had chup-a-chups. And so there will be, again, chup-a-chup rewards for questions. So if you want a chup-a-chup, you have to ask a question. And that also happened in 2003. Although it was Linus that tried to grab one at the beginning. And we told him that he had to earn it first. So you'll get one at the end, Linus. Right, so shall we start off then? Any more photos, Speedo? No, that was all for the photos. But we did also have an audio recording of the session from 2003. Tridge, I think you thought there were a couple of questions from that that might be worth revisiting. Yes, indeed. So we had a few questions. So how many of you actually listened to the link that I sent out to the chat list, the audio? Right. So there was a few pertinent questions, I think, that deserve an update and answer from 12 years later to see how it's going. So the first question that was asked was, when is 2.6 going to be released? Oh. So do you remember when 2.6 was released? I have no idea. This is why I have source control these days. Right. So and there was a follow-up question, which, you know, is 2.5 ready for users to try yet? And I think we need an update on that. Do you think 2.5 is ready for users? That was such a huge disaster that we changed the whole development model. So I think the answer to that is no. That's a no. Right. OK. So there was a couple of other questions that I thought were worth revisiting. One was about trusted computing and signed code. And so a member of the audience asked about what Lena's thought about trusted computing and signed code. And so just to give you sort of what you answered at the time, you said that you would ask a vendor if there is an actual use case for a signed kernel. And the answer was no. And you thought it was just marketing. So would you like to update that answer? So these days we obviously do it even without vendors asking for it. We use it internally. And I use it internally, for example, for signing all my modules. And I think it's a great use, especially the way I do it, which is to generate random keys that you throw away after you've generated the modules, which means that nobody can ever try to insert anything in your kernel. So it's a great security measure. It can obviously, and I think that's something I've said for a long time, although I have no idea what I said 12 years ago, is that a lot of these security measures that are vilified in this community can be very useful. And sometimes they have really good reasons for being used, even if some people tend to hate them intensely. But things have clearly changed. We do use it ourselves. And I use it personally all the time. So the final question I think is worth reprising from the 2003 event was about IPv6. And at the time, if you recall, your answer was that IPv4 is good enough for most people because you don't want to give your shoe an IP address. And so you also said that IPv6 is a decade away. Now, 12 years later, it's probably less than a decade away. And most people here probably use it on your cell phones. I don't actually know how the New Zealand telcos work. It turns out IPv4 clearly is still majorly useful. And while IPv6 now works, I'd give it at least a decade before IPv4 is actually gone, if ever. We'll see. I always thought that IPv6 was way over-designed. And it made it more complicated and more painful that it needed to be. And private networks and things like that means that not everybody needs to use a public IP address anyway. And a lot of people do for no good reason. So your shoes, if they do want an IP address, maybe they could use one of your private network IP addresses. And you could still use your shoes with IPv4. So I think they're the key relevant questions from 2003. So you may have noticed I sent out an email to the chat list yesterday asking for people to think up good questions for today's session. And so how many people have got questions for today's session? A few people?  And if there's any applause at all at the end of answering your question, you get a chopper chop. So first person, please come down to the front and ask a question. Yep?  So Linus, nice, easy question to start with. Over the years, various people include myself and either reduce their involvement in the current community or steps away from this entirely due to the tone on LKML and especially your contribution to that. Why do you continue to argue that being really unpleasant to people is a good leadership strategy? Part of it, a fair question. That's not actually my real argument. My real argument is that people are different, and I'm a really unpleasant person. I mean, that's what it really boils down to is that, and some people think I'm nice. And some people are then shocked when they learn differently.  Really, seriously. I care about the technology and I care about the kernel. And I really think that a lot of projects in the open source community sometimes care about non-technical things too much. And the reason I say that is that the only thing we can actually agree on ever tend to be technical issues. And if we start making a big deal about non-technical issues that are important, don't get me wrong. I'm not saying they're not important. I'm saying that whenever we start making non-technical issues the primary issues, that just guarantees will never agree, right? And I'm not trying to really make excuses. I'm more trying to explain that this is where I come from. And I appreciate the diversity in open source. But to me, that diversity is not about gender. It's not about skin color. It's about people are different. And people are different in what they are interested in. People are different in what they're good at. Skin coloring, gender, and all these issues that get brought up as really important things, those are details. What is great about open source is that some people are unpleasant but they're technically really good, right? Some people are pleasant and like bringing other people in. And I think that's one of maybe the most important part about open source is that you can do what you're good at. So when you look at bringing in minorities, bringing in females, bringing in people who don't speak English, my argument has been and is that we should look for people who are good at being the people who can be between other people, right? There are lots of good kernel developers who are great at working with people. And they may not even be great technically. And that doesn't matter because we all have strengths, right? And this is my argument. And I refuse to change. Well, that sounds bad. That's not quite true. It's not that I refuse to change. It's that I don't think I want to try to bow down to what other people think. My personality is pretty abrasive. I love arguing. But bearing at the Museum of Technology, I spent the whole time just arguing with people, right?  And I know there are other kernel developers that are like me. And I know there are other kernel developers who are not like me. And that's all wonderful. And that was a really long answer to a really simple question. But the fact is that's how I feel. So please introduce yourself at the start of the question. Thank you. Good morning. My name is Tommy Richards. And Matthew, you kind of stole my question. Well, maybe I can. I don't work in kernel development. I'm not subscribed to the Linux kernel mailing list, but I am in the community. And I've noticed a worrying trend, which is that over the last year, there's been more hate. There's been more abuse, more vitriol in IRC, and mailing lists on Google Plus. And speaking as a professional, this really concerns me because I love my job. I want to love it in 20 years' time. But if this trend continues, I will not be in the industry in 20 years' time. So I'd like to know if Venice has already answered, but maybe the other three of you could maybe tell us, you all have a lot of influence in this room and in the wider community. Do you have any plans to try and improve the community for the rest of us? Yeah. So I think we have other people who want to answer that. I'll keep this quick. I think basically what you're saying is not actually true. I think the tone on LKML, I don't know about IRC. Maybe people in IRC are masters and really, it's nasty. LKML, I think, has been getting better. What has happened is that there's a lot of system data. There's a lot more talk, and it's become a thing that is a new thing. So I actually think LKML has, in many ways, come down, and then people try to make a big deal out of it and make news out of behavior that is not different. But does it mean it? So Rusty mentioned system data. Those of you who have been paying attention will realize that, I don't know, my life and that of Keith and various other folks who are here got really tangled up in this whole analysis and debate, and it really came to a head in the Debian community in the last year, in case you hadn't noticed. The interesting thing about that community is that, yes, we ended up in a situation where we had a question that was partially technical and partially not so technical, and emotions ended up running immensely hot, particularly for so many reasons. Emotions got hot to the borderline nuclear level for a little while, and it was immensely uncomfortable. It was one of the folks on the wrong end of some of the vitriol around that, uncomfortable. I'm being polite, because you are sort of all my extended family members. So I think it's a great opportunity and I'm being polite, because you are sort of all my extended family here, and I won't say things to you that I might say to other folks. But the really intriguing thing is all of that vitriol represents what I have described before as the vocal minority, and we do have a problem in the entire free software world where sometimes we are too willing to let the vocal minority represent us and to sort of set the tone or the agenda for some of the discussions. I'm here to tell you that in the Debian community in particular, the community is strong, the community is healthy, the community is recovering brilliantly from that whole ruckus around the internet system debate. Work is progressing. The next stable release is coming along pretty much on schedule. If you've got a little extra time, go help us chase down the remaining set of release critical bugs. And at the end of the day, part of what I think Linus might be trying to sort of get at around the edge a little bit is it's okay for us to have times where we all let our passions overflow and get in trouble with each other on sort of personal interaction and communication spaces. As long as at the end of the day, we somehow figure out how to make things right with all the folks that matter in the communities that we're working in. And I really hope that if nothing else, all of us here today can go forward into 2015 conscious of the fact that a lot of what we read and a lot of what gets amplified in the press and on blog posts and all of that sort of stuff is the opinion of the vocal minority. And many of us who would like to be in the silent majority are still just fine. Everything's good. We're going to do what we can to support the friends and diverse friends around us. And let's not get so distracted by all of this if we can help it. That's the thing that would help me to feel better about some of the stuff we've had to deal with over the last while. And I have to agree with Lainis. I think we have to be very, very careful not to assume that the amount of noise we see in the press and elsewhere is truly indicative of what it's like to be part of most of the open source development communities that we know of. Yeah, so as someone completely outside the system to debate, I haven't seen any escalation in verbal, you know, the trend has been pretty much downhill, I think, for the last at least five years, right? And by downhill, he means better. Better, yeah. So, yes. I do think we ought to give a shout out. There have been important folks in our community who have been doing really great things because it's what they're good at to try and help improve our openness to diversity counters. I'm really pleased by the diversity sponsorship at this and other recent LCA's and the fact that that's brought some interesting people into the LCA community. A shout out to Karen Sandler and the other folks involved in the outreach program for women. That has generated some really interesting new folks in the community who've done really useful technical work and are getting jobs on the basis of that and all those sorts of things. We've seen the same thing, you know, we've seen a number of projects making concrete efforts to try and increase their openness to contributions from different and diverse folks. It's not that there isn't a problem out there somewhere. I think we should be careful about focusing on what the problem actually is and recognize and reward when things move in the right direction instead of spending so much time pointing fingers when somebody gets emotional. Chop-a-chop. There you go. Next question. Hi, I'm Yaku. So is it the year of the Linux desktop yet? So I should interject there. We did have that question in 2003 and Rusty, you actually answered that. And your answer was that by the end of 2004 it would be 50% Linux desktop in the world. So how's that worked out for you, Rusty? You know, it's still so difficult to measure. It's so hard. 50% of my desktops are running on it. Extrapolating that, it's all good. And... In Smartfront... Who is running a Linux desktop? On your desktop or laptop? Excellent. Both of them. And on Smartfronts we are 80%. Right. Answer. There you go. Okay, something slightly different from the previous ones. Those of us at a certain age, gone to computing mostly, I think, from the old 8B computers. And I remember there were picture books for kids that taught advanced basic and even machine language. Now, the world's a different place now and there doesn't seem to be anything like that. Some people are trying, like, you know, we're asked to reply anything. But I'm just wondering what your thoughts are on getting the younger generations into our community because it's all about learning software literacy and understanding how our computers work. And it doesn't seem to be there's much encouragement for the younger ones to join our community and basically fill the ranks as we sort of age. So by younger ones you mean under 30? Somewhat, yes. I wish I had the answer to that because I've got three kids and none of them seem to be interested in technology except as users. Well, that's not true, but at least in computers, in the sense I'm interested in computers. And I think part of the problem is there's... The technology of today is so complicated that for somebody like me who is interested in kernels, it used to be easy to get in at the hardware level. And that's simply not true in computers anymore. And the world has moved on and we're all farts for a reason. That said, it's much easier to get involved in many other ways. Maybe you're not into computers and you're into the whole 3D printing thing and maybe you're into doing these user application things and they're way easier to do than they used to be. So I don't have an answer. At the same time, I don't really worry. It's not the same thing anymore. There's no point in teaching everybody machine language or basic or something like that. I'm looking at... Most young people tend to be pretty tech savvy. So I wouldn't really worry. They may not be kernel tech savvy, but we don't need everybody to do that. Yeah, I think there's also this immense trend that we've been aware of, and in fact there have been some talks around edges of this here at this conference, and that's towards accessibility of microcontroller platforms and other places where kids are really enthusiastic because they see a direct connection between little bits of programming and action that affects the real world. I discovered a really long time ago, well before I had kids of my own and it's something that I've enjoyed playing with them with, that on some level, computers are kind of boring. There's all sorts of things you can do with them and my kids are certainly far more literate and quick to go grab a computer to do something than even I am sometimes, but there's this immense sort of feedback loop thing that works well if you even get to the point where they're turning an LED on or off or getting a servo to move. So my strong suggestion if you've got kids or you're trying to figure out how to get younger people enthusiastic is look for places where there's an intersection between simple computing devices and real world effects and feedback loops and you know, if they figure out how to write a little bit of code and an Arduino thing or something then they might end up thinking trying to do kernel development is interesting someday. I'm not sure why they would, but... My question is where do you see system D being in the next five years? Is it just a new... Is it just a new Udev or maybe DevFS? Yes, I don't know why the system D question comes up but I don't even care. I'm a system D user and I have not been involved in any of those arguments at all that I know of apart from not liking one particular person. And I'm pretty sure Bdale doesn't want to take that question. So I use system D, most of you probably do use system D. It's all good. I didn't care either until yesterday apparently they've got masquerading support and I looked at the code, they put that in. They don't fork out to IP tables because that would be slower so they use it directly. And actually the code looked fairly sane. So, you know, as long as the code's pretty good, it's... I don't really understand what all of us is about. What do you try to... So just to be clear, the real reason I don't want to take that question is I don't try to predict five years in the future about much of anything. Events in my life in the last couple of years have made it clear that trying to predict the future is, you know, that way lies madness. Part of the reason that I ended up taking the decision that I made in the whole Debian system D thing is that it seemed to be the place where there was the most enthusiasm, activity, and good stuff being worked on for the future. And one of the reasons that we worked hard to constrain the actual decision that we made in the Debian project to the next stable release cycle is we have no idea what will be the hot, shiny, right thing to be using another release cycle later. My anticipation is that people continue to poke at it and do good things and hopefully it ends up being, you know, code that we're all okay with. I'll raise it. Okay, my question is for Linus. Linus, when you do like the current version control, you always wrote good and you didn't like, you don't find a good dive lock so you wrote your own dive lock. What are you working on now that is not on the kernel? Are you hacking on something nice? I'm not. Honestly, I'm a lazy person which is why I like open source. I get all these people to do all my job work for me. I don't really want to have other projects. What I want to do is just sit on a beach and sip a foofy drink and just let you guys do the work. And the fact that then occasionally I've had to say, okay, in order to be lazy for the next five years I have to do something, that's fine. Right now, the thing I've always cared about has been the kernel. And right now, knock wood or something, the kernel development process seems to be working really well and we don't have any huge looming concerns and I hope I'm not wrong about that. So we seem to have found a model that is working, is getting us where we want to go and we're uncoasting. Well, right now I'm coasting then the merge window comes along and it gets crazy for a few weeks again. So I don't have any projects I'm working on. I hope to next week work on my dive log thing that I started, but I gave away. But that's, even that hope is more like I hope things work so well that I don't need to. Thank you. We have a heritacle question here. I was just reading it the day about this new machine, that will have maybe petabytes of memristor so you don't have files because everything permanent and memory is the same thing. Therefore, Linux can't handle it. You need a much better offering system that doesn't deal with files but deals with memristructures. I wonder what you have to say about that. Call me when it's your turn. So a well-known member of our community has just rejoined HP in order to work on the Linux port to the machine. The one thing that might be worth, I mean, if Keith wants to come up and speak, but I don't think he does, one thing that might be worth mentioning is even when you have persistent memory and you can imagine a world where everything is in RAM and it never goes away, that won't necessarily really change things as much as some people say because you will still want to have a file system just to organize this thing. Persistence also means that... I mean, one of the advantages of RAM and it goes away is that you do something and then you throw it away and that means you don't have to manage it as much because the fact that it's ephemeral means that it's also something you don't need to care about long-term. If you have persistent RAM memorysters or any other kind of model, you will still need to have a file system just to have a way to organize these things that stay around for years and years. So it won't change that. It will change the details in the file system. When you have byte addressability, you can do things that you can't do with disks. But we already see that with SSDs that just the performance differences mean that you do different things in file systems than you used to do 20 years ago or 50 years ago. So on some level, one of the things I would suggest is that what's really interesting about the machine is that the intersection of a couple of parallel, long-term technology development trends that are going to hit the market at about the same time are going to give us the opportunity all at once to throw more existing assumptions up in the air, possibly bat them completely out the window than normally happens when we're watching the steady evolution of technologies. My interest is absolutely right. The introduction of SSDs change the way we think about performance in a storage subsystem. Gone are the days when elevator algorithms for managing block transfer sequencing were fundamental to being able to get any kind of useful performance out of a storage subsystem at all. The needs, the concerns, the pinch points just to move, it'll be different. But it won't be, it's not like the whole world magically gets simpler. Suspended resume is going to be really fast. There's been so many technical questions I want to ask a meta question instead so you can be purely opinionistic. Given that Australia and New Zealand technically share the same airspace, do you think that they should come together and form a combined space agency and that we're the only G20 nation without one? You should take that one. We technically all share the same airspace. It's expensive and I think we all need to get together to really go into space. I'm really the wrong person to ask of all the people in this room. I'm probably pretty low down on the list to really answer that question. Okay. So we already had a question about the machine and Linux++ and things like that. Do you think there'll be any innovation in hardware in the next few years where the Linux kernel possibly couldn't make the transition very quickly? And what do you think the implications would be for all of us and all of our workers as a whole if Linux loses the dominance it's achieved over the past decades? I wouldn't lose sleep over it. Yeah, and I really don't lose sleep over it. In fact, I think in the computer industry in general the big question is not what happens to software in the next few years but what happens to hardware when scaling stops. And it's not happening in a few years but it's happening in ten and it's getting much more gradual and maybe somebody will come up with something really radically new and there are people who argue that hey, we've been saying that scaling will stop for the last 50 years and it hasn't stopped for the last 50 years. It will stop. People who believe in quantum computers are. Before or after the year... Is it going to stop before or after the year of the Linux desktop? I really think what happened is the Linux desktop scaled down and now we have it in our pockets. But we're really at the point where in ten years, five nanometers that's not a lot of atoms and the whole computer industry... I mean software too, don't get me wrong but the whole computer industry has for a long time lived on the notion that the computer five years from now will be fundamentally different and faster. What happens when that's not true on a chip level anymore? You can still improve, right? You can do clever things and you can improve but it's not going to be the same kind of improvement but you've seen some of it already but it's going to get much worse and that will impact software too but I think it will impact the hardware people much, much more and that's going to be interesting in 10 to 15 years, what happens? I might mention there was a similar question actually in 2003 to that which I think is worth mentioning at this point. There was a question about somebody in the audience said that 20 years before, so in 1983 they had a computer on their desk that could write letters to their mum and print and now, 20 years later in 2003 they had a computer on their desk that could write letters to their mum and print. What's changed in that 20 years? Now the answer at the time from Betel actually was that it's got slower and that's what's changed. But I asked a couple of people about that before this and they said, well what's changed in the last 12 years and the answer from several people was it's got faster again and you can't print. It was two or three weeks ago there was this big surprise that it turned out that many current developers don't know how memory allocation works. I think LW.net wrote Too Small to Fail, that problem so I thought to myself well 10, 15 years ago the answer to that problem would probably be to have RDFM. The question today I would ask which fine manual we should have read. Do you think that documentation of the kernel is appropriate? Do we have any idea how to improve that? Well I clearly have no idea how to improve documentation of the kernel. This is where others sometimes step up and do a great job of explaining small details. At the same time there was a talk yesterday about how programming should be provably correct and it would be a great thing if we could prove correctness and that we all suck at it and I don't take that approach. I'm more of a touchy-feely guy and I believe in biological processes and I believe in evolution and I believe in all these making mistakes and trying and we don't have documentation for how humans work but humans work really, really well except when they break down and I actually think we're at the stage and we have been at the stage in kernel development a lot of technology where nobody really understands everything that's going on. The VM is complicated. Deal with it and just realize that the answer to that is not to make the VM simpler because the VM is complicated for a reason. It's doing all these crazy things and a lot of them are heuristics. They don't have hard reasons for it. There often is no this is what you need to do. We want to go in roughly that direction and we don't even know how to get there but if we do this most of the time it works. That's how a lot of real problems in real life are and that is hard to document when there is no real answer and we should continue to document the big rules we should continue to document the details like how memory ordering works which was another subject that came up yesterday but that still will never be in the situation where we can give a speck for the kernel and I don't think we should even try we should just realize that okay life is complex and we do the best we can and we need to have smart people but we also need to have a lot of people who test a lot of people who are willing to accept that I don't know what the solution is but I know what is better than what we do today in this area and if you continue to do that this is how biological systems work and it's a provably working process it doesn't get you to the perfect end result but it gets you to something that works better than anything else we've ever seen. Hang on, go to the chopper chop. So just to follow up on that Did anyone else want to chime in on that? Yeah, so when I wrote Netfilter the first time I actually wrote a hacking how-to that had always details on how it all worked and how you could extend it and everything else and the first time I received a major patch I was really excited the patch was great and I said what did you think of the hacking how-to? And of course he said what how-to? And so the lack of documentation does provide a filter for contributions It's terrible but it's true because they have to read the code anyway because the documentation will lie so why not skip the first point and make them read the code immediately because they'll have to deal with it so in practice I found that effort put into documentation in the kernel was not paid back So I actually disagree I mean we have areas where we document small details like locking rules and these kinds of very specific and very targeted areas we do need to have documentation because they are things that people can't keep in mind and they're a kind of big enough picture that you can't look at one piece of code and see what you should do So we do need documentation but I think the argument is documentation is not a panacea it doesn't solve everything it's not always the answer I noticed Tridge has not been answering a lot of questions you've been doing a lot of moderating and throwing out chop-a-chop so I'll crack out an old chestnut that everyone can answer current desktops, 2, 3, KDE, Emacs What do we run? Yeah Alright, okay It's checked from time to time and changes from time to time Okay, do you want me to start? Yeah, start off So you know, this is my badge Emacs, Git, I run Ubuntu trustee and I've switched between a number of different you know, window managers but actually a lot of my work is on embedded systems these days so most of the time my desktop is actually GNU screen and running GNU screen on a system that often the systems are on another continent and so really the closest thing to a desktop I have at the moment is GNU screen WN stable, XFCE4 and a lot of terminal windows I'm on Fedora 21 right now I'm actually known for being a GNOME hater I don't like some of the development models they have but I'm a GNOME 3 user with a couple of extensions to fix what I thought were big failures and I don't believe in Emacs or BI so I use this one editor that nobody sane would ever use but it's what I grew up with I just fixed that a bug where it would crash if you had a big screen and more than 128 lines on the screen that's the kind of crap I run I run stock Ubuntu on my desktop but my build machine is arch and the only window manager that has is when I SSH in SSH-X otherwise it's all text I'm Dave Lane just have a question this probably applies to all of you but I guess primarily for Linus in this case do you all see yourselves being at some point in your leadership roles usurped by a computer and if so, what kernel will it be running? your leadership role going to have that role usurped by a computer and if so, which kernel will it be running? I don't know will it be heard? yeah, I think my mind will be going before that happens but the argument there is when that happens and I don't know I think there is actually a partly serious answer to that question it harks back a little bit to some of the social interactions something I've noticed in recent times is processes for doing software development are becoming more mechanized and so the so often these days you're on GitHub or Getorius or whatever and it's like a pull request and it's just a patch and you respond there there's less social interaction you might as well be talking to a computer because it is getting more and more mechanized that process and I think that feeds into I've never seen any really heated discussions in a pull request chat so it is sort of moving in an organized process and being more computerized so if I end up being replaced by a computer I don't actually sort of care on some level as long as it's running free software my name is Laura Bell it's actually my first time at LCA cool I'm a security specialist while security bugs have always historically been found in Linux and open source products in the last 12 months through mainstream media has changed dramatically and our bash bugs in the last 12 months have kind of changed the way most people don't think about security interact what are your predictions for security in the open source Linux space for the next 12 months and which areas of code if you were going to go and look at one right now for vulnerability research which would you start with if you knew which piece of code to look at for security issues we wouldn't have security issues it's going to be something very obscure and stupid and it's going to be obvious in hindsight and people will say how did we ever miss that and that's but it's always going to be the security is just a hard problem well I think you're absolutely right and this falls into another one of those things that I try and be very predictive categories I'm really pleased that at least part of the response to this sort of escalation or change in the way things are being reported and talked about over the last year so is that there are some activities attempting to sort of shine more light on put more resources on some of the core elements of technology that we all depend on I'm referring of course to things like the Linux foundation core infrastructure initiative and so on but we actually immediately know sort of how that's going but the idea that we are at least trying to figure out how to get more eyeballs on some of the things that we all agree are some of the most important elements of code in our infrastructure and that we found the corporate members of our extended community willing to anti up to help make that happen is pretty cool and I hope that's a good step there's another thing that's been going on which I find personally very annoying which is that people are less willing sometimes to kind of brush the problem under the mat and leave it up to vendors and have disclosure like infinitely long disclosure times I'm a huge believer in just disclosing still somewhat responsibly but security problems need to be made public and there are people who argue and have argued for decades that you never want to talk about security problems because that only helps the black hats and the fact is that I think you absolutely need to report them and you need to report them in a reasonable time frame or reasonable for the kernel security list is admittedly five working days which some people think is a bit extreme and in other projects it might be a month or a couple of months but that's still much better than the years and years of silence which we used to have we're going to need to wrap it up there so I'm going to get the last question which is a real privilege as one of the organizing team as of now, as of today and perhaps looking forward a little bit what do you feel is your biggest legacy to the open source movement that posting about the first release of Linux or the momentum that caused you to release Git or something else Me personally I actually think one of the things that Linux has been really good at and this is going to raise a few hackles I like open source and I like this whole working together with commercial companies and this whole notion that you don't need to vilify people who also do closed source so for me personally one of the big things I'm happy about is that I was part of the group and Linux was a big example for the group who try to take and now this is when Tridge will stand up who try to take this very you us against the world approach of free software and made it more open not just in name but also in being acceptable to people who don't necessarily believe in our values but believe that our model is better and that's to me something that Linux was really instrumental in at the same time I'm really happy about Git too because I think Git has spread more than the kernel in some respects and maybe I'll be remembered more for Git than Linux we'll see if I would answer that on his behalf I think the long lens of history might find the most important is how both of those projects have positively impacted this notion that all of us as together collaboratively leads to better results than when we're all sort of hiding in separate corners and fighting and competing so we need to wrap up there but it's been a fantastic session and do you think that our panel has earned a chop a chop thank you also to the organisers for organising this session and for an absolutely brilliant conference so nearly as much more okay everybody thank you


All right, so we have the the pleasure nay the the honor of hearing David Hulton talk about the end of Kerberos and net TLM NTLM 1 and man, you really like giving me a hard one to go Cracking deaths, so please welcome and give give it up for Hakari David Hulton All right. Thanks a lot for showing up here. So yeah, I'm gonna be talking about cracking DES today Does everybody here know what DES stands for? Okay. Well, we're gonna be talking about a slight variant of DES. It's called the date encryption shark I Don't know how this made this into my slides here, but there may be some sharks in this presentation So did anybody here catch my talk I gave with moxie back in 2012 on cracking MS chaffee 2 Okay, a couple people so MS chaffee 2 stands for the micro shark challenge handshake authentication protocol and provides mutual authentication with a password and specifically focused on Usage with PBTV VPNs also used for WP enterprise and and a number of other protocols And this the research that we presented was nothing new It was actually the sim very similar attack was outlined by a bunch of cryptographers back in 1999 And we just kind of revisited that and basically the paper showed that state actors You know it could easily crack this and it was widely used in in Windows Do you guys see something that's weird, huh, okay So so basically how it works is that you connect to a server it sends a client challenge and And and then the then the the client generates a hash and then it basically uses their NT their their user password that's converted into an NT hash and and then encrypts the challenge to create the response and sends it back to to authenticate with the server and up until this point people had basically just been attacking the user password and And this was really good for people had weak passwords But if you're really trying to crack every pass go through every possible password this sends up being a very large key space they have to go through and so the attack that we presented was Basically just trying to crack des to recover the NT hash Which in a lot of cases can be used as the password equivalent to authenticate to to these networks and so Does you know people think that it's insecure? you know because to the 56 and Initially people thought oh, this is like similar to triple des but really this is three separate des operations So it's additive and not you know not doesn't explain exponentially increase So it's really you know they online that's closer to two to the 57. They have to you have to go through in order to actually brute force this and The the key thing here is that this last key is actually only 16 bits long And so that's pretty easy to crack just because your your NT hash is only 16 bytes long and so you're really just trying to crack two des keys and Another interesting thing here is that your challenge hash is the plain text that's used in both these operations and So normally how you'd implement this is you would go through the full key space twice To crack both of these keys, but because your plain text is the same You can go through it just once and then do two compares afterwards to see if you found the right key And so that's kind of the the attack that we presented and oh, whoa, what? weird oh Okay, okay get out of here sharks. Oh, thank you. Thank you so So yeah, we demonstrated that it could be do it done in two of the 56 des computations And the key thing is that we made it so anybody could do this instead of just you know governments and stuff like that so how this worked is a Moxie released this program called chap crack that you could point at a p-cap file and it would extract out the information to Basically create a token that you could then submit to cloud cracker and then that would go on to an FPGA cluster that we put together and Within 24 hours, it would send you the NT hash for that MS chap V2 Communication in this case we we supported a PPTB VPNs and WP enterprise and so Yeah, in this day and age des should be super easy to crack right back in 1999 the EFF does cracker was you know invented and It took around 9.2 days and so with more as law you think it'd be really fast now So looking at doing this in around 24 hours using CPU instances would take around 80,000 CPU cores to crack a key Which is pretty expensive Even in the cloud Then there's also GPU instances that are also fairly expensive and so what we did was we happened to have some FPGA hardware laying around and We were able to provide this at around $20 a key just using kind of spare FPGA hardware I mean one of these systems about five years ago would cost around 150 grand for the system but we we put this on as an online service just for to create awareness and It ended up getting relocated into my basement. So that's a That's my basement. Oh, whoa And So then of course everybody rushed to fix everything right that's that's what happens whatever you show that there's something's vulnerable Hi And so one one of the VPN providers that we called out in our talk was Specifically was I predator and all that they did was just added a little thing to their website discouraging people from using PPTP, but of course everybody still supports this And then people kind of dismissed WP enterprise Taxes being oh well, everybody does strong certificate checking. So this isn't really an issue And of course everybody here does strong certificate checking with every WP enterprise network they connect to I'm sure so Since then we got some interesting jobs Like for example, you know this first one up here looks like What you would kind of expect from? You know a challenge response that we're trying to crack But then we we saw these ones was like one one two two three three four four And we also saw these ones where the two ciphertexts are the same And so people were obviously using the service to crack other things besides just Emma's chat V2 and One day the traffic to our website dropped off and I noticed the cloudcracker.com was down I emailed Moxie and he never replied to me. So I don't know what happened to him But we ended up reinventing this as a service that Turkan provides and it basically is just a web interface to the FPGA cluster in my basement so So we decided to kind of reinvent the service a little bit figure out what people were actually using this for and try to Just add some additional support For all these different features and The real point of this whole service is to you know Like it's kind of the grand experiment of how do we finally kill a legacy crypto algorithm? that's like so pervasive and and And so hopefully you know after all this awareness eventually it'll finally get phased out of all these major products So one thing that we found out was that what people were using this for was With have it has anybody here use s&p capture or like responder or anything like that? And kind of windows pen testing engagements So landman and Intel MV one which are kind of legacy authentication Protocols now for just authentic age authenticating to a windows share They there's all these tools out there to kind of set up a fake samba server And then do downgrade attacks and capture this this challenge response and so the default The default Challenge the s&p capture sends out is this one one two two three three four four and people were using that to crack windows authentication and And so it turns out that you can basically take the values that S&P capture or a responder spit out and and It's essentially MS chap v1, which is pretty similar to MS chap v2 But you can provide the whole the whole challenge and so you can plug that right into our website and essentially crack someone's NT hash just with this authentication and And then this you know obviously works no matter how complex the person's password is because you're actually cracking the NT hash and not the not the password and So so yeah, we kind of made a update of the site to to take these hashes directly from from s&p capture and And then same with if you're using responder you can essentially just copy and paste it in the website and And then get the NT hash for for any sort of net capture No What was that So then also with WPA to enterprise things have kind of evolved a little bit since then now You know, I've talked to some people and it turns out there's still lots of environments out there that don't do authentic don't do proper certificate checking a big surprise and There's there's some better tools out now and so now if you ever come across a WPA enterprise network you can Run host APD and do kind of evil twin attacks and captured Essentially, they're they're net LM and copy and paste it into the website and for $20 within 24 hours You'll get basically access to the network because the NT hash that you get from us You can plug right into your WPA supplicant config file and then authenticate to the network or go through and decrypt The traffic or whatever you've captured for that network. Ah No, man, there's there's more and more of these sharks in my presentation And then also There's there's some research done some recently by Karsten Null about how doing over there updates to sim cards All of the authentication with that is basically done with a single des on a lot of carriers And so this is actually featured in mr. Robot a little while ago And so we we also added this just general-purpose interface for to allow anybody to crack Des if you find it in any any sort of applications out there and And so it's just based on simple rules with like masks and you provide essentially the information you know about About what's happening? It'll go through all the possible des keys and kind of send you a list of all the ones that match that criteria and so as a kind of a to demonstrate this general-purpose interface We thought we'd try just using Kerberos is kind of an example for this and so Does anybody here use Kerberos in their environments or looked at Kerberos a little bit? Has anybody out of those people have you seen des used in the wild at all does CBC or RC for? Okay, well Some people are still using des. I know that for my day job. We still use des on our network But it turns out that with Kerberos. It's really trivial to downgrade and so A lot of these legacy networks that still support it You can easily downgrade it by just using a simple header cap filter and so this is on our get hub where If you look over here in the encryption types We just substitute any encryption type with des CBC CRC and and that causes everything to just downgrade to that and then You can use that to You can also use wire shark to capture this I mean here's some wire shark captures here and I What's that I Guess I guess sharks are hackers too according to Google so and then once you once you capture this These Kerberos packets. It's all based on ASN one And so there's lots of known plain text just in the ASN one Encoding and so we we can automatically extract out basically the known plain text and create a token to submit it to the website and And because of CBC, it's relatively easy to to generate all of that So this is kind of an example of running it on a Kerberos capture that has some des traffic And then these are the submission tokens You can just submit directly to the website to crack the deskey and actually I I'm really horrible at Kerberos and so I kind of presented this at a couple conferences recently and somebody emailed me and Showed me how to use these keys to actually authenticate to the network. So now You can basically take the key that we crack for you within 24 hours And now you can authenticate to the Kerberos network and through Linux. And so anyway, there's some oh, whoa So I also started receiving emails on people asking if I could crack des crypt files Or a des crypt hashes from like old shadow files and password files And has anybody come across any des like crypt hashes and shadow files or a couple people? Okay? What sort of operating system did you? Okay, okay, buddy machine. Nice. Yeah So Yeah, yeah So so initially this was designed to run on a PDP 11 So it would take more than one second to compute and computers have surprisingly become a little bit faster since then But of course nobody uses this anymore, right? And so obviously there's some people that do another one that we found is that a lot of these people are requesting this We're doing car hacking and a lot of the a lot of the infotainment systems and cars use cunex Which I guess still supports a dozen. It's actually the more secure version. They also have another proprietary version of Their own their own password hashing algorithm. That's fully reversible. So so it's actually much more secure And so so we just went through the internet and tried to find as many of these hashes as we could Like this one appears from like the Charlie Miller Charlie Miller and Chris Vellisack g-pack There's another presentation that I just found found one of these in on car hacking There's tons on the internet and so I implemented this so it just goes through the full key space and So no matter what how difficult the password is it'll crack it and it takes around three days on the on the system that we have and so You can just plug it into the into the website and within you know a few days you'll get the password no matter what and so we started looking around for secure passwords and I Asked my buddy Carl if he had any really secure ones that you could send me and he sent me this one That's used in all of the on-star systems and so he ran it through here and it turns out the password is root And so And then and then like yeah the Jeep one is like do you know all these are like lowercase, you know Relatively easy to crack with just John the Ripper. So since then I've received some that are actually pretty complex They're using like boats and other things like that But if any of you are interested in trying to crack any of these. Yeah, just let me know and we can run on the system Elevators everywhere So when will this ever get fixed Does anybody here remember like session hijacking and you know stealing cookies and stuff like that And then also remember fire sheep where they basically made it so anybody and their grandma could You know log in to arbitrary Facebook accounts if you're just you know the Starbucks and so So kind of our our whole philosophy here is that you know if we make this easy for everybody to do then Eventually it'll get fixed and so that's kind of the current strategy and so The we're really asking ourselves like how do we motivate change and make this even more easier for people to do and So the biggest problem is that we charge for the service and the reason why we do that is mostly because you know I only have one of these FPGA clusters and it takes a decent amount of time to crack and And you know we do have to pay for air conditioning and power and stuff like that But it's mostly a form of just load balancing, you know and rate limiting with the service and So what if we can make the service free So so we looked into making a rainbow table for this and it's a relatively large key space like the largest off-crack table or you know standard Rainbow table I could find was somewhere around 2 to the 52 and It was around 2 terabytes and so our goal was basically to make You know one for the whole deskey space, which is about 10 times bigger than the largest table out there And and then also, you know making a reasonable size So so we could and a reasonable crack speed like close to real-time so we could just offer this as a service and And and not have to worry too much about rate limiting so the hardware that I picked up was six terabytes of NVMe storage and I kind of borrowed some FPGAs from from the office and also borrowed a server and It came up with some some rough parameters So I managed to borrow a decent amount of hardware to generate the tables And you know got everything all laid out and You know is ready to get going and then of course there's always hardware issues when you're dealing with hardware so like FPGAs started overheating we started having like we weren't Expecting FPGAs to go from like, you know drying say two amps up to drying 40 amps like and you know in a split second And so there they're all sorts of things we had to overcome But eventually all these things got fixed and we spent weeks generating basically kind of the first couple tables and ran into some issues and that we had collision rate problems and effectively made the tables unusable so we had to kind of go back to the drying board and Came up with some new parameters that made it more like around 12 seconds to crack a key instead of three seconds, but seemed totally fine and then of course we went back to To generate stuff and then some customer ordered hardware and so we had to like ship it out or something So anyway took took a while to generate everything, but eventually we had tables generated We got a this cracking system up right now We have you know the six FPGAs and these NVMe drives are pretty tiny so they're kind of off to the side and all of the air is just kind of directed straight onto the FPGAs and Turned out that our coverage was actually better than expected right now We're at about ninety nine point six five percent accuracy coverage, so you know one in two hundred times you'll you'll crack your key and and then Yeah, when you're cracking the two keys and your coverage drops slightly because you know It's relying on both of them getting cracked But then if we don't find it immediately then we just offload it to the cracking rig and then it takes you know a day to find it so Yeah, we currently have this up and running and on average it takes you know less around 20 seconds or something to find your keys No Shark So anyway, it's free and you can start using this right now We also have an API that we release so you can tie this directly into your tools And one of the key things to note is that this is specifically for cracking windows authentication So using SMB capture or or a responder and if we have We have some time as a Riverside in here. I think we have okay We have a couple minutes, so I can I can just show you this demo real quick here So using responder So Yeah, so normally, you know you fire a responder and then if there's any windows machines on your network They're vulnerable to this downgrade attack then Any sort of file share that they go to or you'll just start getting floods of servers connecting to you because it's basically Responding for any sort of net bios lookup request and so now now we have our NTLM view one authentication here and so we can copy this and Paste it into You basically just say NT hash and paste into here and you can give it your email address and I didn't want to do this connected to the internet And so you'll you'll basically receive an email kind of like this within about 20 seconds and then you can take your NTLM hash, which is right here and then go back over to here and Well, I'll just have this canned here So then you can do us and be client and now you're connected as administrator or whatever And then you can also, you know do any sort of like PS execs stuff to basically, you know get it pop a shell on this on the system as as that user and And then yeah, you're not relying on any of the other sort of you know injection or relay attacks and stuff like that It's so now you have basically the password equivalent for that user no matter how complex their password is So Yeah, anyway the kind of the big thing here is that you know, we've know that this is an issue since the late 90s We knew that you could build custom asics for about 200 grand that would do it in nine days And if you were to build the same thing nowadays in theory it could crack any deskey in around a minute and a half and And so it's you know, no surprise that FPGAs can do it in you know, a matter of hours or seconds doing different techniques so So yeah, I think this is kind of a bigger issue with with this algorithm Can we really get this phased out and get people to stop using it because this will probably become an even bigger issue You know in the future with with the other algorithms that we have out now, so Right now you can check out Crack.sh we have an API set up if you want to tie this directly into responder just to automatically crack these NT hashes We have some plugins like there's an Airbot plugin if you want to just like submit stuff directly through Slack or IRC or something like that and And we also have all the code for the rainbow table stuff up on GitHub So if you want to run one of these systems for yourself All the code is there We haven't figured out the best way to host or mirror this six terabyte rainbow table but if you have any ideas, let me know and There's yeah tons of people that contributed to all the all the research in this so lots of things to them and So help me kill legacy crypto if any of you want to try to crack some stuff Just shoot me an email and I'll let you run free jobs in the system tons of info online and And then of course come out to tour camp. I'm sure everybody's been barred been barred in you with that the whole weekend but We might have time for one or two questions if there are any anybody All right, thanks a lot for coming