So, my name is Phil Dibowitz. I hold many hats depending on where you've met me. I am a co-founder of Scale. I am a production engineer at Facebook. I do a bunch of different things. Today on this stage I am not wearing the Scale hat. I am wearing the Facebook hat. So if you don't like the things I say, you can go yell at Facebook. Don't yell at Scale. So today I'm going to be talking about building open source relationships from within Facebook. And Facebook has been building relationships for a long time in the open source community, but this is specific about my team and how we did it, and how we changed how we were doing it specifically, because we had already been doing it. And I want to take a quick moment to note that while I am the only person on the stage today, a colleague of mine, Davide Kalvaka, helped me build this talk and gave it with me in Brno, Czech Republic and the freezing cold in January. So thank you to Davide for his help there. And there's one disclaimer I need to make, which is I'm talking about my experience and I'm talking about my team's experience. This is not to say that every other team at Facebook is perfect at this or terrible at this or that nobody else does this. Open source is part of the ethos of Facebook, and lots of different teams work with the community in a variety of different ways. And this is our story, and we will go through a little bit of the history outside of my team, but fundamentally, let's talk about what my team did.  So I'm the tech lead of the operating system team at Facebook, and what that means is we work with the bare metal experience. So we are responsible for CentOS, we are responsible for configuration management, which in our world is Chef. We are responsible for packaging infrastructure, so the YUM servers and tooling around building RPMs and deploying them and the rate of change there. We are responsible for, you know, the system D and how we interact with all of the community that we serve below the container layer. And what that means is that our customers are any service that runs on bare metal, which in our world is databases and large storage infrastructure and those sorts of things, as well as the container management system we have internally. They are our direct customer, and then every other thing at Facebook is sort of their customer, right? So this is our agenda. We're going to talk a little bit about open source at Facebook first, give you a little history, a little overview, some of the good, some of the bad, I'll be super honest with you. And then we're going to jump from there into why would you want to do this, and given this is an open source conference, I'm pretty sure most of you know why you would use open source or why you would release open source, but at the same time you may not know why it's important to invest a lot of your personal time and effort into building relationships with these communities. And even if you do, you may not know how to communicate that or articulate that to your management and to the rest of your company. So we'll talk about that. And once we've talked about why you want to do this, we'll talk about how to do this. And then finally, the meat of this talk is the examples. It's not actually the meat in terms of minutes on the clock, but it's the important part. So we'll talk about very specifically the last year and some change of my team, the relationships we built, the good, the bad, and then we'll end with some lessons that we learned. So no discussion of open source at Facebook would be complete without mentioning the open source team. So I'll start there. We have an open source team at Facebook. We've had one, I've been at Facebook for six and a half years, and we've had one since way before I got there. Open source has always been part of the ethos of engineering at our company. So the open source team has grown and changed over the years, but primarily their job is to help shepherd projects into open source. And that means stuff like, hey, I wrote this piece of software internally, and it's awesome. And I think that every other company out there would love it. I don't know how to do that, because this chunk here talks to foobar internal service, and I'm like, what do I do with that? And they'll sit there and help you go, oh, we can factor this out into a plug-in or config file or whatever, and here's how you'd make this something you can release. Or you may sit down and say, hey, I'm about to be tasked with writing a service. I really like to make sure that when I'm done writing the service, it's something that I can go to legal and get open sourced. How do I do that? And they'll talk through you through doing that. They'll talk to you about when you go and say, hey, I really like to open source this. They can look at it and say, you know, I'm not sure there's a community for that. Are we just open sourcing it to say we did? Or is there going to be actual value in the community because there's a cost, there's a cost to us, there's a cost to the community when you open source stuff, and we don't want to do that. And the converse of that, hey, man, this would be really good for the community. But from what I understand, there's one person on your team who's wherever subscribed, are you going to have time to accept pull requests to look at issues, to build a community, to be a good community member? Because if not, maybe we shouldn't open source it. They can help you understand and navigate the legal waters. They'll work with legal with you. They are very technical and also very well educated on the law. So they will sit there and be that kind of middle man for you and help you understand what the lawyers are saying and help the lawyers to understand what you are saying. At the same time, they build a lot of tools. There's an open source tool called ShipIt that they have open sourced. That was redundant. And what it does is it can take internal repositories, get, material, whatever you've got, and sync it to GitHub commit by commit. And this is really different from the way a lot of big companies sync out their internal repos. Instead of taking the last day or week or month or quarter or year of shit and dumping it into GitHub where you as an external contributor don't know how you got there. Today you're in Los Angeles and then tomorrow you're in New York and you don't really know how you got there. This actually goes commit by commit. And so you can see as these commits are happening what's going on. You can see the commit message and the code that's changing and who's doing it. And generally speaking it's like between 15 minutes and a day out of date. So we're talking essentially life changes depending on the team and how sunset it is. And even better, your internal engineers actually get credit for what they're doing in the open source community which is nice. So they build a bunch of other tools to keep in sync before it's in blah blah blah blah. But fundamentally they provide guidance. That is their primary motivating thing they do is they provide guidance for internal engineers who would like to open source something. When Facebook started, as I'm sure most of you know, sorry I'm losing my voice a week of parting, it started as a LAMP stack in Mark's dorm. So he wrote a PHP app and that PHP app ran on Apache and it talked to my SQL on a Linux box. Many of us know the standard LAMP stack. So open source has been part of our history literally since day one. And how much we used it and how much we contributed to it as grown and shrink over the years. But it's always been a bigger part of who we are. And so as the company grew, PHP turned out to be great in terms of onboarding new engineers. But it turned out to be not so great in terms of scaling the site to what would eventually be more than a billion users. So people started working on PHP, they started contributing patches, they started trying to improve performance and they succeeded. They didn't succeed like orders of magnitude, which is what we were looking for. So another team sat down and they wrote a thing called HP HP, which eventually got open sourced as hip hop. And hip hop was a thing that took your PHP code and compiled it down C++ and stuck in a little embedded web server in there. And then you could take that C++ and compile that down to a binary.  It was super fast because it was compiled and that was our front end for many years. And we open sourced it and other companies started to use it. And it was pretty successful. But there was a huge drawback to this. If you're a developer running your code, your massive code base through two compile passes does not make for quick coding. Like I made a one line change. Let me hit compile and go get some coffee. And let me come back and hit compile again and go get lunch. So when the happening was all these developers were continuing to use NPHP in their development model and then using this entirely different runtime in prod. And for those of you who have worked in the industry, you know that like having dev and prod being different, not really what you want. And they were very close and it mostly worked except when it didn't. So eventually a team sat down and said, well, I think we can do one better. And so they wrote a just-in-time compiled VM, a JIT VM for PHP, heavily, heavily optimized. And it took a while to get the performance on par. But once they did, the open source that is HHVM and that's what we run today. And many other companies actually run this today. And it means that you can develop quickly and it's really, really, really fast. The interesting part of the story is not the technology, it's not the speed, it's not how they did it, it's not the order in which they did it. What's really impressive about the story is that team today continues to work with Zend, continues to work with the PHP community. They're part of the working groups, they contribute to the standards, they still send patches to Zend when they want to have a new feature added. They make sure that the thing they're writing is a compatible runtime for PHP. They didn't have to, right? They could have just started with PHP, forked off, and been like, look at our cool language, aren't we awesome? But they didn't. They continued to be a part of the community that they came from. And I think that's awesome. My SQL is another really good example. Sorry, I don't have a note. It's up here, so that's why I glance at my slides right now and then figure out where I am. So my SQL is another great example. We've contributed to my SQL for a long time and we're one of, if not the largest, installation of my SQL in the world. Which means that we have a MySQL team and they develop on MySQL and they add features and they do performance fixes and bug fixes and all of that sort of stuff. And there's a really good MySQL at Facebook engineering blog, if you guys are interested in that sort of stuff, you should follow. In the early days, as they were developing on this, they would send patches up to MySQL and then MySQL got bought by Sun and then Sun got bought by Oracle and this became kind of fuzzy. And those of you who follow this know that it was a tumultuous time in the MySQL universe. And eventually we ended up putting patches on GitHub because they took a while to get through stuff. I'm not going to say too many words about all of the internals of the MySQL community, but suffice to say we ended up putting patches on GitHub just to make sure people could see them. And eventually Facebook, Google, and a bunch of other big dot-com tech Silicon Valley companies decided to get together and work together on all of these things they were doing to help improve MySQL and they made a thing called Web Scale SQL. And you can go find it today, they have a web page in the GitHub repo and that worked for a while for a couple years. All of these companies were working together to improve the performance of MySQL. There was a single GitHub repo and everyone was one big happy family. Eventually these different companies decided they wanted different things to be performed within the code base and they were somewhat at odds with each other and so they all went their own way. And worked on different forks of MySQL and what have you. And today what we basically do is we have a GitHub repo under our organization that's just MySQL and it includes like MyRoxDB and a bunch of other stuff. The thing again that's impressive about the story is not MyRox or the way we contributed patches or anything else. It's the fact that through this long history, ups and downs, that team continued to work very hard to find the right way to interact with the community, to provide what they were doing, to be good community members. The kernel is another really good one.  I just couldn't care less. But I do care about scaling technology. I really, really love it. So I was at scale actually and I was being recruited by my Facebook. I was at Google at the time and I was like social media. And I started looking into Facebook because people kept telling me, oh, it's a cool open source company. I was like really? Big blue box. And one of the things I found was that they had open sourced, or well, sorry, they contributed a massive patch that to greatly improve UDP performance in Linux kernel. And no one had ever looked at UDP performance before this as far as I could tell because it was basically just for DNS and no one really cared if you could squeeze a couple extra whatever milliseconds out of that packet. But it turns out that at the time Facebook was using UDP for all of their cache traffic so they cared a lot. And so they basically rewrote UDP in the TCPIP stack and they contributed it. And I thought, well, that's cool. Maybe I should go look at this company. So that's kind of one of the things that made me start looking at the company. I get the job. I don't work on the kernel team, but I work with the kernel team being in the space that I'm in. And the team grows, company grows. We go from thousands to tens of thousands to hundreds of thousands of servers, from millions to tens of millions to hundreds of millions to billion users. And the number of machines grows. And the things we need from those machines grow. And the kernel team grows. And the way the kernel team worked for a long time was, here's a kernel. We have nine billion patches on top of it. Now we'd like to move to the next kernel. Great. Get next version of kernel, fork with billion patches on top of new kernel. And for those of you who are software developers, you know that this is a terrible way to do anything. It's very painful. It requires a lot of work for each one of those patches to try and re-merge it and all of that. And so eventually as the team grew and started to get more prevalent kernel developers on it, they decided that this was not going to work. And they moved to an upstream-first model. And so they said, okay, we're going to drop all those patches. We're going to take the linux tree. We're going to one by one by one by one through these patches. And we're either going to submit them upstream and get them upstream or we're going to drop them. It's like one or two corner case exceptions. There's always like the one thing you need or whatever, right? And they did. And it took a couple, I don't know how long it took. But eventually they got to an upstream-first model. And today we never developed a feature on an internal kernel ever. Every feature we write in kernel is written, submitted, and accepted in linux tree before we deploy it. And it doesn't mean we don't deploy it in a machine to test it or whatever, obviously. But we don't backport to our live production kernels until after that patch has been accepted. And the kernel team wrote a really good blog post that I've linked to in the other slides about this transformation, about how they made this. And the thing that came out of this is, well, the things that came out of this are many. So first when we want to move to a new kernel, instead of having to move a bunch of patches, we just drop the patches. Because when we moved to a new kernel, all of the patches we're carrying are already in the next kernel. So we just drop them all or at least most of them. So moving kernels is much, much, much faster. And when moving kernels is much faster you move kernels much faster. And when you move kernels much faster you're doing less backporting. And when you're doing less backporting you'll have more time to write more features. When you have more time to read more features, you're doing better. And then you move faster and it's a self-feeding cycle, and that's great, and that team is awesome. And that change helped to recruit more kernel engineers because they're not working on 2.6.24 or whatever, they get to come and work on Linux's tree. It's exciting for them. So again, the moral of the story here is not the work we do, it's the way the team worked hard to do the right thing to engage with the community. Of course, we haven't always been so perfect. HBase is a different kind of example. So we forked HBase really early on. Some guys sat down, some developers decided that they could make this better, and they started writing patches and they started improving performance. And part of the way they did that was by leveraging a lot of internal Facebook infrastructure. And they kept doing this until such time as it became very difficult to move patches back and forth between our internal fork and the upstream fork. And Hubers is the thing that exists. And they figured, ah, we know better. We got it. We got this. And so they did. And years went by. And then eventually somebody joined the team who said, you know, we should just try the open source version just for fun. And they did. And what they found was it was way faster in tons of different metrics.  There was like a couple ones that I picked up. But there was more of them. And so immediately they were like, we're doing it wrong. And so they started a project that took some time to move to the upstream version. And they took any patches that they actually needed and they submitted them upstream and today we're on fully open source HBase. And that team learned their lesson and went, we should do the same thing with Zookeeper. And so Zookeeper is one of those things that we had forked internally. And they looked at it and the upstream version had become much better and faster. And so now we're in the same process as Zookeeper. Chef is a very different example than both these two not necessarily initially positive examples as well as different from the other examples I gave. There's a whole slide on this later and we're going to go more into our relationship with Chef. But fundamentally the reason that I pointed out here is for four years we've engaged with both the company chef as well as the community chef. We've worked very hard in our relationship with both of them. And this has been great because for four years we've had a terrific relationship with them where we've been able to add features, we've been able to request features, and it's generally been a very, very positive relationship all the way through. We developed a ton of other stuff, right? I picked a handful of examples that I could use to show morals, but the reality of it is Presto, OSquare, React, RocksDB, McRider, Box3, Surround360 processor is a small sampling of things that we developed in the open with the community. And if you go look at the GitHub repos for these, what you'll see is that it's not just Facebook employees contributing in the occasional pull request. There are people from the community who are actively engaged, who are actively developing on this, and the commit logs show it. Most of these things are actively developed with the community and several of them have external contributors who are maintainers. And again, this is just another small list. We have tons of them. So that's a little history. Now, let's actually talk about why you might want to actually build these relationships more than just, say, release your own software. So a lot of these reasons that I'm going to go through sound a lot like the reasons that you might want to use open source software in the first place or release some internal pieces of software, and that's because they're related, right? So the first reason is that it's efficient.  And the same thing holds true when you're engaging with a community.   It's more efficient when you can actually engage with the community and say, hey, look, I really kind of wish this worked this way, or I really wish this had this feature, and either contribute it or because you have a good relationship, they couldn't do it. So that then always means less man hours because you're not maintaining these patches, these features, whatever. And as you build these relationships, now all of a sudden you're able to contribute code more easily. I don't know how many of you spend a lot of time on GitHub, but drive by pull requests tend to get merged a lot less often than the people who are engaged in the mailing list or in the IRC channel or in Slack. And as you can contribute this code easier with less stress with a conversation, it means that you generally tend to be happier about your life, your engineers are happier, and that's always good. And they feel that sense of community, that sense of shared ownership, and it's more people who are maintaining these features you've written and these bugs you've written are fixed. And then finally, the obvious more eyeballs equals better code, which I'm aware is not always true. But it's more likely to be true. The more people who are using your code, and the really big benefit of this is unlike, hey, I dumped some code into GitHub that no one's ever heard of, as opposed to, hey, I put a patch into bind, for example, there's a bunch of people using that software already. So they're going to get those bug fixes and they're going to see those features and they're going to poke at them. And so actually more likely to get tested. There's also less waste. So imagine instead of being super engaged with community X, you use version seven years ago that happens to be in rel six. And you're like happily chugging along on version X of whatever this package is. And it's really old, like really, really, really old. And then you have some bug. And so you spend some time debugging the bug. And we're doing S traces and taking debug logs and doing all the stuff and you're being the awesome engineer. And then finally you find this bug and you write the patch and you're like, great, I fixed the thing. Look at me, open source hero. And you go to submit the pull request and you're like, ah, that doesn't look like this. And so then you spend like three more hours forward porting it, only to find out that that bug was fixed two years ago and you just didn't know. Man, you just waste a lot of your own time. I've seen this happen time and time again. Or let's say that you don't have the time to fix that.  Man, you've got the S traces and you put them in markdown. You highlighted the right things. You've got the, here's my tarball of logs. Here's all the stuff. Here's my environment variables. I like, I got you. And now this maintainer sits down and he sorts. There were terabyte of data you uploaded, only to find out he fixed that bug two years ago. It's not going to go well for you. And these things cause animosity. They waste your time. They waste other people's time. This is not good. As opposed to being on something very modern where you probably didn't run into that bug, or when you did run into that bug, by the time you fixed it, you were able to contribute this and everyone was happy. Another good reason is direction. So knowing what's coming down the pipeline will make your life easier, whether you're on the more operation side or the more developer side. Say knowing that the next version of your OS is going to have system D and network D and whatever else. That's much better than being surprised by it. Like trying to roll it out and being like, oh, look, my in the scripts don't work.  And it can be small. It doesn't have to be these huge massive things. Let's say you wrote something that uses Netcat. Netcat's options change drastically between REL6 and REL7. CentOS6 and CentOS7. Like just knowing that's coming, knowing that they change where they're getting Netcat from. It can make a big difference if Netcat is your thing. If you're like, all of my stuff sits on top of Netcat, knowing what's happening in the Netcat community will help. So there's a bunch of different examples like this. But here's the thing. If you're involved in a community, when you get to have that conversation is not after it's been committed and then packaged and then send down the pipeline and then you downloaded it because you're not going to change their mind at that point. But if you're on the mailing list and they're like, oh man, I think we should change from library food to library bar and you're like, dude, that's going to drop these six features that I'm really, really, really need. Now you can actually be a part of that discussion. You can be a part of the direction of the community without even being a developer. This means that you're engaged with community and community is engaged with you so that you can actually start to be prepared for these things and be involved with these conversations when they're meaningful. That's not meaningful to have that conversation four years later. And then as you do this and as people start to respect you and as people start to think, man, that John guy, he's got some good ideas. He's always given us good feedback.  You're more likely to be heard. It's a community. There's clout. Doing engaging with a community more often is going to mean it's easier the next time you do it. We all talk about how code wins arguments and it does. But every line of code was written by a person or code that was written by a person or code that was written by code that was written by a person. It all comes back to people.  And you have to think about that because it turns out that when you're engaging with a community you're not engaging with that code. And I love daily engaging with code because code is not weird and mushy like people are. And people are like unpredictable and do weird stuff. But at the end of the day you have to realize that you're dealing with people. So when you start to build these relationships with the community and your engineers get to start contributing their code upstream. Start being a part of all of these communities that they know of, that they've heard of, that they've consumed for years. But now they get to be a part of that community. They're happier. And how much research has been done that shows that happier employees are more dedicated, work harder, do better work, right? So it's better for your company. And the thing that I've talked about a lot is being engaged with the community but it goes the other way. So as an example there are a handful of pieces of Chef that Facebook are super sensitive to changes on. And when somebody submits a pull request to one of those spaces in Chef it is very common that somebody in Chef will tag me or one of my co-workers and say hey what do you think of this pull request? Because we're not just engaged with the Chef community but they're engaged with us. They know what we care about. They know what we've contributed to. They know what bugs we've filed. They know how many times they've broken us or nearly broken us, right? So it goes both ways. And then there's this weird thing where like we did not at all plan this where this was not a goal but what we got out of this sort of superfluously was better recruiting. When we were out there talking about what we're doing contributing code, having discussions people like man Facebook uses that thing that I know that I use. They're not just this weird box somewhere that does their own thing. And so people who started coming up to us at conferences being like hey do you have any open positions I really like working on blah blah blah too. So that was really nice. And then good relationships are contagious. There's an example later in the talk about a sort of fortuitous happenstance. But that wasn't me, was it? No? I'm good. So but the reality is that if you're engaged in some community, you're a contributor, you're a great bug reporter, you're the person answering questions on IRC whatever it is, and then all of a sudden you want to engage with this other community. There's a pretty decent chance that like one of the maintainers or contributors to project A happens to be involved in project B. And so now you've already got a foot in the door. Now you've already done the first step of building a relationship. Good relationships are contagious. So with the why out of the way let's talk about how. And the how really boils down to one thing. And it's building social capital. Fundamentally building those relationships all comes down to having social capital. So how do you get social capital? What does that mean? Well there's a bunch of ways to do this, but the first thing, the primary thing, is wherever possible do this in person. Want to get involved in system D? Go to system D course. Most major projects have a conference around them and if they don't they probably have a meet-up around them. Go to meet-up. If they don't have a meet-up or a conference go to a conference that's related where those developers may be. Go look and see if somebody's giving a talk on that at some nearby conference. Go to the conference and meet these people.  If you can't do that, if this piece of software is owned by a company, send them an email. Ask to go to their office. Get on the plane. Go find them. Have a meeting. Sit in their office. Tell them why you like their software. Tell them how you would like to engage with their software to make it even better. And if they're not, maybe you can invite them to your office. Hey, I know you guys are only two people working in a garage. You want to come and see the Facebook campus? Want to come say maybe you don't work for a company with a really cool office, but you work in, say, Manhattan, and you're like, hey, want to go to a cool steakhouse in Manhattan, right? Invite them to you. Pretty much anywhere. We all know that it's so much easier to have discussions with people you've gone and had a drink with. Participation is important. Go find whatever mechanism they use for communication, whether that be Slack or IRC or mailing lists or whatever, and join and listen. But then participate. You actually have to be part of the conversation. It's not good enough to be a worker. Engage. When they start talking about things that you care about, talk, like, respond. As you respond, as you be engaged with these people, they'll know who you are, and you start to build a relationship. Again, go to relevant conferences and meetups. Two other things, and understand the goals of the project. Talk to them. Figure out where it is they see this project going. Because if they see this project going in some direction, and it's the opposite, the antithesis of what you'd like to do, it's going to be a harder conversation. And you should probably know that going in. Or maybe you will really want this thing to come out of this project. And what you find out from talking to them is that they also want that thing, and they're already working on it. Or they were excited to work on it, and they want to help you. Understand where they're going. But also understand their workflow. It turns out you're not going to get every open source project out there to use your workflow. Every time I went to an open source project, I was like, yeah, I mean GitHub is cool and all, but I think you should use Fabricator, which is our code review system. I don't think I'd get very far, right?  You need to meet them and their tools, and their best practices, and their approaches. So read the docs, figure out how they do testing, figure out how they do code reviews. Maybe they're an international team. And so after someone in the U.S. approves your request, they're going to wait 48 hours so that the offices around the world have two days to look and complain, and say, no, I don't agree. Hey, you didn't know about this thing. And maybe if you don't know that, you're sitting there going, come on, where's my, where's my pull request? Maybe they have protected branches. Maybe they have specific internal rules. Big companies often have a lot of rules about backporting changes to release branches. Maybe, maybe they have a bunch of internal, like, takes them a month to do it, and so they're just not going to do it. It's important to know these things. Context is everything. So understand their context. Kind of taking on from the previous slide, when we worked with a bunch of different teams that read out, what we found out was any developer could merge a pull request on to master, but if you wanted something on the raw five or the raw six branch or the raw seven branch, that was like, stuff had to happen. Like a lot of stuff. And we're like, oh, that's good to know. So any place where we can be on a release that's not directly from the distro, it's much easier for us to contribute. Easy peasy. Done. So understanding their context, we worked with a team that had a project that had a team in Brno, Czech Republic, and so they would add a tag that says one side of the team has approved this, and you had to wait until the other tag was added. That's the other team side of the team had approved it. This was their way to make sure that there was balance in everyone that you didn't miss context while the other team was sleeping. But then provide your context. If no one likes a 20,000 line drive by pull request, like, they just don't. Like, no matter how awesome your pull request is, no one likes a 20,000 line drive by pull request. Provide context. If you're going to rewrite a subsystem, jump on the mailing list, say, hey, I have this really cool idea to rewrite the subsystem. Like, how do you guys feel about that? Here's why I think we should do that. Here's what we're going to get out of it. Here's why I don't like the existing one. And what you may find out is someone's like, oh, yeah, I'm already halfway through that. Give me a week. Or, oh, yeah, that'd be great. Let me help you with that.  We tried that already. Right?  But if you haven't written the code yet, now you can respond better. Remember to find the appropriate medium for the conversation. If you want to make a three-line typo fix in their comments, maybe don't fly across the country and have a conversation with them. Like, maybe not necessary.  So one of the biggest, most difficult things about actually being engaged with the community is figuring out the right speed for you and for this project. This turns out to actually be quite challenging. What you want is to be following the project close enough to be able to engage meaningfully in both code contributions and discussions. And this is the key. In fact, I should have put that quote on the slide. I don't think I did. So the idea is you want to be following something equivalent to latest. And latest is going to change a little bit based on context. And you have to make a lot of considerations as you decide what this means. So you have to think about the cadence of this project. If this project is getting 600 commits a day and they're releasing twice a week, probably upgrading yearly is not going to be enough for you to have meaningful context on what's going on in this project. At the same time, if they're getting 100 commits a year and releasing once every six months, maybe trying to download, master, compile, and push it out every day is a waste of your time. So you need to understand they're the velocity of their development and the velocity of the release cycle and these are not the same. Some places release very regularly and get very little in terms of contribution. Some places are getting tons and tons of commits but never releasing. You actually have to understand both. You have to understand your own resources. If you have one person on a team who is swamped, maybe upgrading and pushing out this random piece of software you like to contribute every day is just not feasible. And so you have to understand that. You have to understand your own. Maybe this is a piece of code that you're putting into another project where you only release it once every month. And if so, does it really make sense for you to be trying to pull in the latest every day? So it's important to understand most of those things and pick a velocity that's going to allow you to the version you're actually running and familiar with to be close enough to what the developers are developing on that you can meaningfully get, contribute and get things out of discussions. And finally, be honest. That sounds silly but it's really, really true. If you come into a community with false pretenses and you're like, I'll do not. I'm just here to be a better community member. And that's it. I have no other ulterior motives. It's not going to go very well for you. Be upfront. Hey, I need this. These are the things that the problems I'm running into. Here's what I think I can contribute. Here's where I'd like to see the project go. And there's ways to work this into a conversation, but fundamentally you have to be honest. And then in turn you can expect them to be honest with you. So our journey by example. So as I said over the last year or year and a half, my team, the operating systems team, has worked really hard to expand the relationships we had to have more of them, to have deeper, better relationships with the community. And we sat down kind of late 2015 or so and started thinking about how we could do this because we really liked the way we worked with the chef community. But we had all these other projects that we were like then. We work around these bugs or really on really old versions or we don't know how to contribute or whatever. Like we should really try and grow this thing that we do pretty well over here. Can we do it better in more places? And how do we do that in a sustainable way?  And we'll highlight these as we go through the examples. But you need to make sure you're clear on what your goal is. You need to figure out what kind of speed for the previous slide is reasonable for you. You need to figure out the size of the change you want to make because that's going to change the engagement you're going to have. And then not so much applicable to these examples, but something you should keep in mind is where their previous engagements with this community. So let's say you want to engage with the community and you've never engaged with this community. But the team that sits 15 rows down from you in the building tried to engage with that community and were complete assholes. Like this is going to affect your ability to build that relationship and you probably want to know that because they're not going to know you're a different team at Company Fu.  Now we're trying to contribute to my project. So being aware of these sorts of things can make a big difference. So when we sat down to figure out where we were going to go next, the first thing we had to do was take stock of what went well, what we had done already. And as I said we have a great relationship with Chef. So we had been building this relationship for a long time and we had intentionally decided that our relationship had to be both with the community and with the company. So we buy support from Chef and so we can make support tickets. But if we ever have a problem with anything that's open source which is the client and now the server but for a while there was a closed server, we would always file that bug on GitHub in the open. And if we felt the need we could go and file support requests saying hey, we big customer care about that GitHub pull request but we never file those just internally because we wanted to engage with the community.  When there was two Chef servers, even though we paid to get access to the proprietary Chef server which had features and bug fixes ahead, we always ran open source and at least 10% of our clusters. And the reason we did this is because we made a promise to the community that any work we do, any tooling we write you guys should be able to use. And the only way we can guarantee that is if that tooling worked with open source Chef. And the only way I could guarantee that without having to set up way too much tooling was to just use it in prod. So I did. So when we sat down we had a goal. And our initial goal, our goal changed over time but our initial goal was scaling. So when we looked at a bunch of config management systems Chef scaled the best but it didn't scale enough.  And they were giving us builds like a couple of times a day, never sorry a couple of times a week. So we were running things like Chef server dash food dash Facebook dash get hash. Because some developer would be like I think I fixed the thing compile make RPM email to us. And we were upgrading constantly. And that was the velocity we needed to give them the feedback they needed in order to actually get where we wanted to go.   And so we stopped doing this. And we got on a major release and it was good. And we had a different goal. And that goal was to add features to the Chef client. So we started adding features to the Chef client but what we found was the rate of development on the Chef client was different from the rate of development on the Chef server. And so we were actually able to keep up without having to upgrade daily or weekly. We could actually be on kind of accordingly release. And it was more than sufficient to know what was going on that plus following the mailing list and for requests and stuff and be meaningful like meaningful contributors and have meaningful discussions. So we got to a point where we upgrade both about quarterly. And so the point here is that we picked a goal and picked a velocity based on that goal and when our goal changed we changed the velocity. The velocity is not just about the project it's also about the timing. We're not involved in community events. I've given talks at ChefConf. I've been on panels. I've gone to the Chef developer summit as is most of the people on my team. They've given talks. We've tried to stay involved with the community as much as possible. And the benefits of this relationship are many. They developed IPv6 support for us in the Chef server. They massively scaled the Chef server. We contributed multi-package support. We contributed much improved system de-support. When the client side of Facebook started using it for desktops and laptops and stuff they contributed significantly improved OS 10 support. All of these things the community benefited from, the company benefited from because they could sell it and then we obviously benefited from it. So, all around. And as we talked about these things people were like, dude, you wrote what for interacting with Chef? We would like that. And because we had had this relationship because we had worked to make sure we were using everything that was open source it was very easy for us to release the tooling and a lot of our cookbooks. So there will be a link towards the end where you can see the tools in the cookbooks we've released. So, with that said we had to pick a thing. And how do you pick a thing? We work with hundreds of pieces of software at least. And so we kind of looked at like, okay, well what are we working around? What do we have internal patches for? What meaningfully could we actually contribute to like today? Like if someone said, great, we'll interact with you. If you can deliver some useful pull request within the next six hours like what would that be? And there was a handful of them but this narrowed it down pretty significantly. And one of them was Anaconda. And Anaconda, for those of you don't know is the installer for Fedora and Red Hat OSes. So, Fedora, RHEL, CentOS. It is both the installer you get when you plug in the CD and get the pretty thing as well as the network-based installer. And so while we do not use the pretty thing with CD, we do use the network-based installer. And the thing about the way they build and develop Anaconda is when they fork the code for a given Fedora release, say Fedora 26 for example, in Master they drop every feature that would be required for 26 but not 27. And they continue to develop those features on the 26 branch in Master they do only what's required for 27. And one of our goals was we wanted one installer for all versions of RLS. As you can see, this isn't really a pull request.  So that was going to be a problem. We had features we wanted to add that was great. The other thing we wanted to do was we wanted to be able to use the latest bleeding edge version of Anaconda because it had hardware fixes and IPv6 support and worse huge IPv6 shop. So with our goal in mind it became very, very obvious that step one was send them an email and say hey, can we come to your office and talk? And I did. And so my colleague Brian who's sitting up here in front row and I flew out to Westford, Massachusetts which is in the middle of nowhere. And that's where the Anaconda team was at the time. And we met with them and we started that conversation very simply with what is being a good community member look like to you? And they were like what? And we're like community members like people who use your software like what what would be a good way of doing that. How can we be helpful? And they had to think about it and we had a conversation about it and then we asked them where they were going and why they did the things they did and why they developed the ways they did and what their branches meant. And they were really excited to tell us about all this stuff and they were like okay cool and they did all the things. And then we said cool within that context we were now able to talk about the things we wanted within the context of what they wanted and how they developed. And that conversation went really, really well because one of the things that they really wanted was people to be able to actually be using the code they were writing not to the code they wrote six years ago. It makes it much easier to actually go and track down bug when you don't have to be like wait what was I doing six years ago? So they were super excited about our idea and they said great let's just spend the next we were out there for two and two days and then halfway through the first day they said no let's just hack like next two days we're just going to sit down and we're going to hack we're only going to write code. Okay and that's what we did we just sat there with them and we hacked on code. And we went back to this to California kept in touch with IRC and over a couple of weeks all of those pull requests got merged and today instead of being on the whatever thing happened to have been released four billion years ago in CentOS 6 we cut a master build every month give or take and roll it out. We have one installer for CentOS 6 CentOS 7 Fedora Rahide it'll do them all. This code is not has no internal patches it's all in GitHub master. We have the latest hardware support we have the latest IPv6 support we've dropped all of our internal weird workarounds. So yeah pretty awesome. So we thought about the speed and we thought about all these things but primarily what we got out of what's interesting about the story is figuring out the right speed and their engagement model. Right. And so today we still continue to to communicate with them well contribute stuff we saw them my other colleague Davida and I went to Brno DevConf big Red Hat conference sat down with the Anaconda team hung out with them and we've continued to make a great relationship. So the other one that we decided was potentially useful was SystemD. Now we had started moving from CentOS 6 to CentOS 7 in late 2015 sorry I had to do math math is hard and with that most of you probably know comes SystemD. So it was very obvious SystemD was going to become this like very core underpinning of our infrastructure and we couldn't just ignore it and be like yeah yeah yeah it's fine right we had to actually engage we had to actually think about this. And so as we looked into it what we realized was there's a lot of really awesome features we wanted out of SystemD and none of them were in CentOS 7. They were all like way later and so we're like well that's cool like maybe we'll look at back boarding those or something and the other thing we really wanted to do is we wanted that C Group to support so Tejan who's actually giving a talk on C Group in the exact same slot time slot in some other room which is interesting he's the primary maintainer of C Group and me in the kernel and he wrote C Group 2 and we really wanted to use this but all of the support for C Group 2 and SystemD was like like weeks before this moment in time not years before this moment in time so we're like okay can we back for all this crap to CentOS 7 and it turns out that that's a nightmare they move way too fast so like okay well maybe we'll build the latest and then try and get that to jam that into CentOS and it turns out that didn't work very well either then finally what we realized is the version in FedoraRahide is either latest or latest minus one and somebody somewhere in the magical ethos of ethos in the magical world of Red Hat has done all the work to take this package it up and make it look Red Hat E and like that saves us a lot of time so we grabbed the FedoraRahide version and it took like a seven line patch to make that build and CentOS 7 and it was a spec file patch we didn't even have to write code and so we took that and we back ported that along with the nine or ten or dependencies and all of a sudden we were running latest and greatest systemd on CentOS 7 while that was cool and then we started looking at adding features fixing bugs blah blah blah and so we flew out to systemdconf in Berlin in October I think Davida and another coworker of mine and Zeal and another coworker of mine and I and Davida gave a talk on how we were intergaging with systemd and like what we were doing in this back port and blah blah blah and everyone's like that's awesome we want that can you give us that sure so we made a github repo and we stuck these patches and these spec files and on github and one of the packages of systemd in rel actually built a copper repo and you can now not just look at the patches you can actually go install these rpms that somebody at red hat has actually built and installed on CentOS 7 so like there's a huge benefit for the community that we didn't even expect to have we just thought we were in a weird corner and a room full of systemd people were like yes that is awesome please give it to us so that went really well and our speed is the latest release right finding the right speed there was critical but now we have cgroup mkos i is a fun one that's this thing they built to build a system image with systemd in it and fire it off in a container using systemd n spawn and which is how they do all of their testing and because we were engaged with the community as opposed to being off in our own world and we were discussing with them what we were going to do on the on the main list and join conferences and stuff we were able to leverage this not just for our systemd development but for a bunch of other internal testing that we do so that was really nice there's other examples I mentioned earlier that there was a fortuitous happenstance so we were at systemd cons in Berlin and we were discussing with the systemd packages for rel some of the stuff and somewhere the conversation moves and I said something about wanting to add to fix some things in network scripts and the guy looks at me and he goes I'm the maintainer for network scripts and I was like okay neat and so we started talking about all the things we want to do and all these weird workarounds that we have internally and within like an hour he had moved the entire code based from some internal fedora projecty thing to github and made davida and iMaintainers fortuitous happenstance it's awesome so I sat down and I was like sweet I'm going to take all these weird workarounds and make them all some features and bug fixes and contribute them all up this is going to be amazing and I vied the file and it was a mix of tabs and spaces and I wanted to cry so I immediately sat down in a bar in berlin to start rewiting spacing the entire code base and I did and I was about to send that pull request and I went that's not going to go well I'm going to be that guy I'm totally going to look like that guy who's just trying to get a github credit by rewiting spacing your project guys just met me terrible so I sent them an email and I was like hey I'd really like contribute all this stuff but like no like I can't like can I please send you a pull request to rewite space your entire project and there was like a brief discussion and they were like yeah sure go for it and so I did I've sent them a pull request to rewite space the whole thing they were cool about it and then I was able to contribute a bunch of v6 features some v6 big bug fixes and per device routes which is a thing that we had needed for a long time unblocked several internal projects and is now available in fedora and I think will be in rel74 so that was like one of those really cool engagements where we just happened to solve this other problem completely unrelated to the thing that we were solving and I think it's a really good example of what I was talking about earlier providing context if I just sent a pull request that touched every single line of their code but made no functional changes I imagine they would have been a little unhappy rpm yam and dnf is another one that we're just starting this engagement but we met with them in dev comp in burno we have a couple patches that we're trying to test and write internally we contributed a bit to our plugin for yam we want to make sure that we can contribute that to dnf we have some rpm db scaling issues we're working with them on so this has so far been a really good engagement as well so that was our journey those are the things the people we we engage with the things that they got out of it the things we got out of it we learned a couple lessons in some cases we just re-remembered lessons and the first one is to listen first like really and truly you got a bunch of people who are working really hard probably for free maybe not on something and walking in and declaring what it is you would like to fix or change doesn't tend to make them feel really good so listen first listen to what it is they're working on listen to where is they'd like to go listen to what it is they need in a pull request or a bug fix or a test plan or a unit test provide your context white space is the greatest example I can give of this like what should ostensibly in most cases be a complete waste of everyone's time was really really valuable but only because I was able to provide that context or going to Red Hat right if I'd gone if I'd submitted a pull request to like forward port grub one support into master of Anaconda I would have been laughed at but we could sit down and say hey look we really need to install CentOS 6 and that requires grub one and so can we forward port this thing that you deleted to your latest code base and it was that that conversation that enabled that to happen providing that context with made to the world of difference I want to point out somewhat tangentially that non-code contributions are also contributions figure out before you engage with the community what it is you can provide it may not always be code maybe you're a Python developer and this thing is in Scala and you don't have the time to learn Scala or maybe your team is just oversubscribed and you just don't have the time to develop maybe bunch of things but that doesn't mean you can't contribute and having some way to contribute to a project before you do a major engagement is going to be helpful maybe that's money maybe you can just donate money or hardware or time on the mailing list or time in IRC to answer questions or documentation fixes nobody likes writing documentation but we all need it there's a million ways that you may be able to contribute to this project even if you're a developer it may not be code maybe you have a place where they can test something that they don't get to test very often and you say okay I'm just going to take on testing this for you every two weeks I'm just going to run this on my foobar thing that you guys don't have access to one of the things that we tend to provide to open source communities we engage with is hey we have we can test things at scale so like I don't care if I lose a thousand machines if you have some really crazy thing you want to test like give it to me I'll throw it on a thousand machines and I'll let you know what happens right that's a thing we can provide that doesn't take me much time or effort that not everyone else can provide let's figure out what it is that you can provide you don't have to take a photo of this my slides will be available but this is the URL to basically everything I mentioned the blog post by the by the kernel team all the chef tooling the system detox that my co-worker gave links to a bunch of the projects that I mentioned our system the backboard all that sort of stuff and I know that we're right at the end of time but I'm happy to stay and answer questions and that's all are there questions I'm just that good going once going twice I'll repeat the question for everyone but before I do I want to clarification maturity curve of their project or maturity curve of Facebook so the question was is there a difference or do I think there's a difference in terms of the project the maturity curve of various projects as to whether we engaged early like we do with chef versus not engaging and forking as we did with age base that's an interesting question and I wasn't involved in the data infrastructure side so I don't know for sure however my suspicion is it was more to do with the team at hand and the maturity curve of that team internal to Facebook than it was anything else because frankly I think that they could have engaged with that team but it takes a certain level of maturity and understanding of the open source community and one of my favorite things about Facebook as with all features of anything there's good and bad but one of my favorite features of Facebook as a company is that each team is very empowered to make their own decisions and so while we have an open source ethos and while we have an open source team at the end of the day for example I've worked with companies that say okay every project must have 90% unit test coverage right at Facebook we don't have these rules it is expected for you to go oh I'm writing this really core critical piece of infrastructure I better have a lot of unit tests or I'm writing the weird widget on the front end that like is going to be replaced three times a week so writing a unit test is literally a waste of my time right and so you're empowered as a team to make those decisions and if you make poor decisions you're going to end up in in our no-blind course more than what we call sub review and eventually you're going to change your mind or someone's going to encourage you to change your mind um while I think that is absolutely the best thing about one of the best things about the company in terms of culture and in terms of really empowering engineers there is the downside that like sometimes you have a team that just doesn't doesn't know that they don't know the thing right like there's people talking about no knowns and no knowns and unknown unknowns right and so I think this is one of those cases where they didn't know they didn't know the thing they were they just didn't have the maturity within the community and within Facebook and no one caught it until it was too late and then once you're far enough like it becomes much harder to justify the cost of reemerging until there's something else there and finally somebody was able to show the value of investing deeply and in moving back to the upstream version so again take that with a grain of salt I wasn't there but that would be my suspicion any other questions going once going twice thank you guys so much for coming check check okay good all right so through control up here to switch this over to oh good looks like it's being done okay well while that pops up question mark I will walk you through who I am my name is Ryan Sypes I'm the community manager at system 76 we make computers that are born to run Linux and that means that we have a whole team of engineers who spend a lot of time making sure that the hardware is well supported in Linux we ship specifically Ubuntu and so we typically work really close with the Ubuntu guys in the canonical team in order to enable and patch what we need to do in order for you to just take that machine out of the box and have a pretty damn flawless experience the idea is when you get a machine and it comes to you in that nice package and you open it up and you start it up that you're not worried about whether the wireless works or whether the trackpad works correctly or whether airplane mode works it all just works and we support it throughout the lifetime of the product and so that means that as long as people are using it we're doing what we can to make sure that everything continues to work and that updates you can upgrade your system without worrying about breaking something or losing support for something and that's really really important for the adoption of Linux and we're very happy to have been doing this for 11 years and I'm very happy to share with you the hard lessons that we've learned so that you can avoid those lessons yourselves if you want to produce products built around Linux perfectly that's perfectly fine I can change over to VGA if I need to I have a VGA port on the other side it wouldn't be a Linux talk or it wouldn't be a computer talk period if every if everything just instantly instantly worked helps if I try to put the port in the right way and what's your name so thank you for helping me so so I'm really really good at just talking out of my ass when there's nothing happening the how many of you have heard about system 76 everybody except for this guy oh no you have heard about system 76 how many of you own a system 76 machine or have owned a system 76 machine one of you shame on all of you yeah so hardware is hard that I was before I joined system 76 I worked on a project called my craft we were working on essentially an open source Alexa or open source Amazon Echo and we we went into it a little blind in theory we were like hardware yeah that's a computer you know we can get the parts we can put them in this enclosure and it's just going to be easy and we're just going to then take those things put them in a box and mailed into a customer turns out that that is not at all how making hardware works it's actually a really intensive process and you get a you get sometimes you get hardware and the best priced hardware doesn't necessarily have the software support that's needed and so that's when you get to delve into Linux land and work with people making kernel modules or trying to get your stuff into the mainland kernel or sometimes you do you just have to write other you know pieces of software to act as kind of middleware in order to do interesting things and that was a fantastic learning experience and while I was doing that I was talking to Carl who's the CEO of System76 and he was walking me he was my sounding board for this is really hard how do we do this and fortunately he's been working with you know parts manufacturers and and ODMs and the whole gamut for years and years and years so he always had fantastic feedback so when I left Minecraft I was like well I still am really interested in making hardware so I decided that I would try to see if I could find a home over at System76 and I I was able to so now I live in wonderful Denver which is like 38 degrees right now or very cold we got off the plane and it was like 70 here and I was like yes so it's the warmest 38 can be and and it's also very pretty so that it makes up for it and just natural you know beauty yeah actually we were our if you're ever in Denver and you want to see where we work our a our we have an office that is right in the middle of downtown it's on 16th street mall and it's like you go down 16th street mall very good you go down 16th street mall or you go up 16th street mall and I think that we're pretty much right there in the middle we're on 16th and Champa and it is awesome like it there's everything there but talking about mass transit we all take mass transit to get down downtown take the train or the bus I I was really excited there and we had a we had something up on screen resolution I'm not too worried about there we go that's good enough that's the that's the presentation so thanks guys thanks a bunch for getting that working maybe can I dim the front lights guys okay so when I submitted the talk we were 10 years in I'm giving the talk and we're now 11 years in so I decided to add a little subtitle there so this is lessons from 10 years making machines born to run Linux but actually it's 11 years and so this is Carl he's our CEO that's him in his basement this is the only picture that exists from the first six years of system 76 he said he was too busy to take pictures and so his wife took this picture this is that is system 76 headquarters back in oh boy back in back at our inception which was Carl's basement and I've got a uh there's like an adapter for an adapter okay I'm just going to try really hard not to touch it so that is actually was taken during one of the binges he was working on the audio DAC drivers for this was called the darter back in the day and that's that laptop right there it kind of dates itself based on the monitor next to the laptop that is a nice CRT monitor he's got a phone right here a little handheld receiver for his landline phone that cat's dead now it's uh but yeah and uh he doesn't have any gray hairs in this picture so this is where we started he told a story about how he the website would sell not just linux laptops and uh desktops but actually things that worked with linux so printers that were known to work with linux other components that were known to work with linux and so the first machine that they sold that we sold was a printer so he sends it off he's very proud he puts it in the box he sends it with DHL and uh and the customer calls two weeks later and says I never got the printer and so he goes through a process printer gets returned to him and it's completely destroyed and it looks like the box was just tossed around for days and days and days and the printer is completely shattered the second machine he sold was to somebody who is in denver so he's like I'm going to do something really I'm going to go and personally drop off this computer so he gets there he walks up and he's like knocks on the door a guy opens the door and he says hi I'm Carl from system76 here's your computer and it wasn't the guy who ordered the computer the guy who ordered the computer was like a 13 year old kid it was his dad and he's like I don't know anything about this thanks and so that's how it was for a time um outside of that the community was generally interested in what he was doing back then the mootoo community was still fairly small this was in 2006 and uh the response was generally good but we almost didn't sure uh we almost didn't ship go boondoo when my display comes back it will show branding for another distribution called Yopur has anybody ever heard of Yopur before before I talked to him I hadn't either but it was a distribution that ran KDE and was really you know user friendly at the time and uh that's all I really want to say about it I'm glad we don't didn't ship it because that obviously was not the successful distro um it's been dead for quite some time just like that cat and uh if I get this back I'll show you guys a screenshot I'm not sure why it looks like it's fairly well seated in here not going to sleep the last two times it just took a little lot to come back I want you to know that this is not a limitation of the hardware okay I just won't move period and maybe that will help so this is Yopur there's the the only logo I could find and I was going to clean it up so that there wasn't the white stuff around it but then I decided it wasn't worth it for one slide for like two minutes on the other side you see KDE I don't know what version of KDE that is I remember using that version but I have no idea what version that is so this is our website back in 2006 the idea became an idea in 2004 but it wasn't put into effect until 2006 the Carl started the business with a buddy of his a co-founder and the co-founder designed the website not terrible by the standards of the day but now it kind of looks atrocious the at the top there's a little asterisk that says Linux rules that's because down there in the corner of the website there's an asterisk further up on the page that says something about Linux it says it says computers to change the world for business education and life with a little asterisk and then the corner of the page it says asterisk Linux rules on the right hand side of the page there's a few little key points that it calls out it says what is open source how can Linux benefit you what software is included and view screenshots at the time he said the advocacy was really important he said that a lot of people who are coming over were generally interested in technology in technology in general and were curious about Linux and but you know pretty high percentage of them was still in the process of learning about Linux and so it was helpful to try to give a high level view on what separated Linux why it was different than Windows and so he had to do a lot of advocacy the whole team did back then to be able to talk people through considering Linux as a platform and actually buying a machine with Linux preloaded on it up at the top there's notebooks desktops media centers servers monitors and TVs printers and software available for purchase on the website we narrowed the scope a lot the like I said the idea at one point was Linux compatible machines and now that has come in a lot which has also allowed us to grow fairly quickly the community back then was much smaller we had a direct line to everybody working on Ubuntu and we used to do stuff like this this was a flyer for this is not doing it for me guys I know I should have just grabbed another you get to see live troubleshooting let's see if that comes up here a second going to go into my system settings here going to go into displays I'll just walk through the slides and then if we get them back then we'll talk about them when we started the team was really small but that was okay it was really easy interface with people working on the projects you know when when Ubuntu is 18 people total or however many they started with it's really easy to just tap them in our IRC and say hey we're looking at a driver issue or we're looking at you know an issue where wireless is funky and get a pretty quick response they grew and we grew now it's a much more planned process going from a from here's an idea to it's in Ubuntu which means that we've had to build in a lot of processes around things that we want to implement the thing that is interesting about this is that back then though everyone had to wear a ton of hats being in this room with people who work with technology you'll probably understand that quite a lot the sysadmin always wears ends up wearing lots and lots and lots of hats and knowing a little bit about everything and so we have a culture that has been built on guys who are Swiss Army Knights you know a little bit of everything and so the the thing that is great about that is that we have folks who are who are an engineer a marketing person and a community manager that's me and so that's really awesome when we come to events like this because most of the team has gone through this thing of being on support call at one point and then working on driver issues the next minute and then finally coming up with a marketing plan and implementing that so we've grown quite a bit but we have a lot of experience across the board which is really just a fantastic asset to have just walking around the office the other thing is that we're all the nerdiest people that you'll ever meet we had a thing called superfan not too long ago where we flew people in to the office and and these were people who were doing interesting stuff with with our machines and what we what we did was we brought them in we showed them products that are not yet released and got their feedback on it but the most interesting thing was we went out to lunch and the entire lunch turned into a conversation about which Star Trek movie is the best Star Trek so we we like to think that we are our customers and that puts us in touch with the with them on a level that a lot of companies just aren't everyone who works there uses Linux full-time and has for years and years and that makes us uniquely qualified to advocate for Linux and for our customers when you know interacting with the projects that we do like Ubuntu and Linux proper the I'm going to go back to HMI because I had a much better experience with that even though it was going in and out the you can't see it but we have a key on our laptops that no one else has and that key is in place of the Windows key it's an Ubuntu key if you want lessons about how to run a hardware company it's to be true to your product and be true to what that product is we do not like the XPS 13 Ubuntu edition an edition means that it's not the original intent of the project so there are a lot of Windows first laptops out there they're held up as standard bearers for running Linux that is not the way that we wanted to do it we only ship Ubuntu on our machines and that is all that we will ever ship unless Ubuntu as a project disappears entirely and then we'll ship a different Linux distribution but everything is built around that process and so the the idea is that we will that the button on the laptop does what's on the button so when you press the Ubuntu key it opens up the launcher which if you use Ubuntu the that corresponds with the icon for the launcher this was breakthrough for us this was much bigger than we would have ever expected customers were genuinely impressed that we were shipping machines with a different key that reflected what they were running on the machine that is really interesting because it it says to them we care about this and we're there with you the that's a big deal it reflects what our values are it reflects that we care about open source and we're not just going to rebadge a laptop we're going to actually change it we're going to create something that has your values marketing open source is a totally different story entirely we we've recently open source solutions are permeating the mainstream in a way that I could have never imagined back in 2004 when I started using Linux it's it's crazy when I first got my first diss admin job I worked at a data center and we were a nearly complete Microsoft shop the only there were only two machines that were running Linux they were running Red Hat and they they were in the back and there was a it was a mail server and a web server and that same data center if you go back now it's totally flipped there are like two windows machines that are there for legacy software and eventually they'll probably be moved to virtual machines the rest of the data center is CentOS for the most part and some of them too at the beginning there was so many questions to answer so so many questions now even people on the street if I they if I explain to them what I do now granted I'm at Denver which there are quite a few engineers in there but the when we start talking I say hey they say what do you do I say I make computers uh well system 76 what what computers do you make other computers that run Linux they run Ubuntu and the recognition has changed dramatically but a lot of folks have these terms somewhere in their head they've run across them somewhere in their daily life and that makes our job a lot easier because we don't see ourselves as apologists anymore we say hey we make computers that are better and and that's born out to be true if you were here during the ubukan Carl talked about how he got two calls on the same day one guy who said hey he's a young guy 24 I think he was this admin he says hey I love what you guys do it's too bad this will never catch on mainstream the next call was a 73 year old man he said this is the first computer I've ever owned it's excellent I hope it's also you know the last computer I ever own which is a little macabre but the but it's a we have so many users across the entire gamut who came to the platform simply because they are actually think it's a better product and it is it's um often times I'll hear people apologize for things on Linux as if Windows and Mac don't have any problems and so I when I was setting up my grandfather what finally put the nail in the coffin on this to where I stopped apologizing for Linux was I tried to add a printer in Windows and it was this and it I used to be you know this admin I used to support a lot of Windows workstations but it was like so frustrating for this to be the modern state of Windows Windows 10 it's like I it's like oh we'll find the driver for you well okay that sounds great oh we didn't find the driver select from this long list of printers first start by selecting the type of printer that you have and then select the model so I was like okay well that's fine brother scroll down scroll down scroll down why is there not a search feature scroll down scroll down scroll down it's like 1760 dn 1780 dn I have a 1770 like what happened and then so I go to the website and first I just type in the driver or the name of the printer in driver and I get 18 websites that are not brother but they they're like download the driver here I'm like oh I'm not going to do that and so I finally find that and it's in some way off page and then it's like Windows 10 is not supported yet so so I just set him up with a an Ubuntu machine a while back about three months ago I clicked on the little Ubuntu key I typed in printers it brought up a nice little add button I hit add I hit I hit click the printer in the network like printers list and it found the driver printers done and so that's a sign of a superior product it truly is there's there's some other things is there's no there's barely any bit rot on Linux and it's it's just a fantastic experience especially for those who are doing your average every day computing go ahead I I'll take a question a new printer and new driver well it's reliant on it may have changed but main line of Ubuntu is reliant on cups and so is that right still the case does anybody know it's yeah so it actually does a search for those drivers from a list and I can't you can actually go and look I can't remember the website it might just be the cups project website and they keep a list and they just add the drivers to that list and so it pulls like it would any other software from a repo you know and pulls down the appropriate driver it's likely that you'll find it obviously I can't speak to every printer yeah yeah so I can't speak to the cycle for every printer because it relies on the manufacturer to make a Linux driver and so the generic Linux driver for printers is usually works with 90% of printers I've used it on printers that don't yet have a Linux driver and it works okay it depends your multifunction printers have a lot of extra features that are built on top of that so sometimes the the generic driver will work but it won't recognize some features of the printer so it's a so just like in Windows or just like a Mac Mac used to have this problem you're reliant on the manufacturer to contribute to a platform that you're using I would say that I personally have never run into a printer that didn't work with the generic driver but it is not exhaustive and I can't account for every single use case and every single printer so I hope that answers your question yeah they do they it's their project or was I think may have a different foundation yeah so the so marketing open source has become much much easier and the the difference between platforms has closed a lot of that is web web is a lot more dominant of a platform than it was 10 years ago for instance you have tons of electron apps now like Slack and like Telegram these are leading applications that are packaging essentially versions of the website that can in multiple cases be used offline but you still have access to all those features like the new Skype alpha is an electron app and so that has really made it easier to convert users and for users to stay on the platform application availability used to be an enormous problem one thing that we're really prideful about is our culture and if my slides are working we have a picture of a one of our our operations manager is this older Swedish man and he always has a very stern look on his face and so when he wasn't paying attention we snapped a picture of his stern look and printed out giant heads then we put them on pop sickle sickle sticks and we hold them up over our own faces and one time we invited them to a meeting and we were all standing there with the faces held up over our own we try to be an accessible transparent company we try we try to emulate the open source community in this respect and try to make sure that it doesn't seem like there's very many barriers between us and our users and the other communities who we work with and so in order to achieve that we try to make sure that we're available where people expect us to be available it's kind of interesting because we have an IRC channel and a telegram channel and it's confusing but I guess I understand it if you're thinking about Dell or HP people hop in there and they ask a question about system 76 and then a system 76 employee answers them and then they're like well what's your source and it's like well I'm the guy who's working on that and and I found that customers are really really surprised about that but that's because we don't put up barriers between who can talk to customers and who cannot all of our employees are I can say whatever they want to our customers there's no there's no real point in putting up these artificial barriers and as a result to your question earlier many of our customers feel for will ask us directly on social media or you know questions like yours and they'll answer it to the best of their ability or refer them to somebody who you know has a more exhaustive knowledge of the topic I've seen more and more companies going this way Red Hat has tried to move towards this what they call the open organization model I think when you're shipping open-source software it's very important to reflect the the values of the community because ultimately those are the people you have to work with every day to ship your product out the door and that community does not reward a closed culture it's just a matter of fact in fact any time that we have a closed process are I call them frenemies people will follow us but are very very quickly quick to call out when we're doing something that they don't like they tell us and it's very important to not let your ego get in the way and just listen to what they're saying and see if it's actually a worthwhile concept and so point is is when you're building an open-source company culture is key the this slide is a screenshot from our Slack channel but I'll just read it and I think it'll work just as good someone says in the Slack channel hey hey hey guys someone says hey what's up she April one of our support technicians who started the conversation thread says did you hear about the space bar and cast says okay and she says the drinks are great and he says oh no no no no terrible joke incoming but the atmosphere is terrible and then someone says I heard they're out of this world and then it shows like three people quitting that Slack channel don't really have anything to add to that I just think it's important that you guys knew that so engaging community I touch on this briefly how many of you actually work on an open-source software project for your day job and that's kind of sad too we have we try to make sure that when it comes to the community that that like I said we're as transparent as possible we try to go to where people are getting their information so if we want to tell you about a new product or do initiative that we're taking on we might post about it on reddit we might go on a podcast one of the Linux podcasts and talk about what we're doing they in the picture on the sides there's a picture of the Linux action show guys reviewing our laptops that has been fantastic for us it's funny because a lot of folks talk about how small the Linux community is if we now it looks like have maybe three percent of desktop computing market share based on net market share and I can't remember the other one off top of my head but so they went ahead and they put those two together and they're like we're approaching three percent and then and then windows users laugh about it three percent is a whole hell of a lot actually the Linux action show has monthly 200,000 listeners unique listeners unique downloads take what that with what that what you will but that results in a lot of customers who buy our machines and say I heard about you on this podcast and and I think that that is crazy I think that that is that the reach of some of our own community members is actually just incredible and it continues to grow we see with with the result of like Edward Snowden and privacy becoming more of a concept that we talked about in the mainstream media we have more and more people asking questions about Linux and asking us about Linux and asking okay well I heard about this tail sing you know the the distribution that does not save anything beyond your session and we get calls from journalists who say I found your website you know I've started using tails will this work well with the hardware and and so as we engage with these different communities with privacy communities with you know podcasters with Reddit we're finding that that results in a lot of great traction it also just helps raise awareness about who we are and what we do and the importance of what we're doing and then I have a few Reddit posts in here I really wish this worked and they just show us answering questions on Reddit and and doing it very detailed we have Cassidy who works on the elementary project anybody know what elementary OS is he is their designer and he is to be envied because he has the vigilance to stay up on top of Reddit posts about system 76 which is something that will crush your soul if you do it long enough so I respect him for his ability to engage in answering the same questions on Reddit over and over and over the next slide is us wearing medieval garb we we try to engage our community and person as well that's why I'm here we think it's important to get down and talk to people one on one and be an active contributor to the community and so I'm hoping even though I'm also kind of giving you guys a sales pitch I'm hoping that that by being here by taking questions by being available that you'll be able to learn for me and I'll be able to ultimately learn from you and that's really really important and that's why over the last two years we've worked really really hard to be at conferences be available be it's costly but the feedback we get and the information we're able to share we think is invaluable and we're wearing medieval garb because the event was our super fan event and the theme was Dungeons and Dragons and it's working that outreach is working who has heard this week in tech to it Leo owns an orcs pro and it was funny because he was in the middle of the show and they have a little chat you know that they they watch and engage in during the show and and Linux users salt of the world we really are but we also sometimes are idiots he's a new Linux user I mean he's tried it before but like he gets an orcs pro and he's using it and he's like oh I want to try some other distributions what should I try and I'm like arch gen two and so so so halfway through doing his distro hop to arch we get a message from someone who's like hey I'm watching this week in tech and he's moving over to arch on his orcs pro system and we're like oh no like if he can move to arch we would prefer that he not encounter all the problems learning arch while on a stream with a orcs pro and so we hop over there and we didn't try to change his mind we just were trying really really hard to engage him and say like no no no no this is how you do that in arch like and ultimately he's now on arch but he has a working stable system our drivers happen to be in the a u r which when I say drivers I mean all of our patches that we include in our in our repo that we ship on our machines before they get pulled into upstream I call it staging because we we can't obviously so we do things to fix the machines like for instance like boon to add a bug where where airplane mode didn't work on some models the airplane mode hotkey didn't work on some models laptop we caught it in staging we were in in no boon to staging we were able to include a patch so that our users never saw that but the but we we do submit those upstream but we hold them in our own repo until the maintainer accepts that change and so when I say the system 76 driver that's what I'm talking about this is not some crazy you know driver for some obscure piece of hardware that we write so point is that it pays off this constant engaging the community we get people like Leo to be a great advocate for our brand and for open source in general and I think that's so powerful if in 2004 or 2005 you had told me that a mainstream press tech guy would be using a machine that was designed to run Linux I would have laughed at you so I'm I can't even believe some days when I wake up that we are where we are the the other thing is we've started bringing folks into our organization and letting them help us stuff one thing is our knowledge base we've gotten lots and lots and lots of folks who ask you know the same questions they're you know they're it's every time you get a new machine you're asking the same questions or maybe you're trying out a boon two for the first time there's a lot of the same questions so what we did was we were writing all these knowledge articles ourselves and then we were like well what if what if we just dropped this on github and our entire doc site is on github and our users can write docs that they find useful or nuggets that they came up with and then that's our support I mean not all of our support but our I'm have a problem I'm not ready to talk to support technician I wonder if I can figure this out on my own support so some of those just include things like I want to install a different distribution on my system 76 laptop will everything work well we can't test every distribution on our own partially because we don't even know about some sometimes even though we are constantly in that world we'll get something like well how does Hannah Montana Linux work on this well I yeah it is it's actually a Wayland test environment I just learned that but it's a well we don't really know are you using it yeah it's like well how's it working for you oh you know and this doesn't work well let's let's document that and then let's work together to see if we can figure that out we have only benefited from bringing our customers in and letting them participate with us on helping to create an ecosystem around our products that is not just a company knowledge base it's a general knowledge base and it just happens to be in regards to our hardware although there are more and more people who are putting general Linux answers on there and so we are actually having to overhaul that website so that it's more searchable more usable for just your general inquiries the other thing is as we have more people who are buying our products we have a lot more opportunities to work with people who we never thought we'd get to work with in video we have direct contacts now in video we're considered an NVIDIA partner we hit for the last three releases of the driver we have engaged very closely on the Linux driver part of that's because we have a lot of users who are doing machine learning on our on our products and NVIDIA is very interested in that segment and so they ask us questions and then we get to be a sounding board for that you know here's what we've heard from our customers here's where the driver could be improved and that is really really powerful and it's good for the entire ecosystem that also I see a canonicaler in the crowd Sergio hi that also is allowed for us to collaborate more with canonical and and have back and forth and they've actually allowed us to post to their developer blog and talk about the type of things that we're focused on and kind of transition some of the Ubuntu community to focus on these things with us we didn't ship a high dpi display for a long time because support was not at a place where we felt comfortable doing that finally we kind of just got tired of not being able to ship a high dpi display and so we asked canonical hey we would like to improve this before we ship something would you like to work with us on this and and a few engineers got peeled off for a couple of weeks to help us engage with us and patch things with us so that we felt comfortable let at it and like we were at a point that we could ship that hardware and so as our community grows we're able to interface with more and more organizations more and more projects and really make a difference on a grander scale than we did whenever Carl was in his basement working on that one audio fix for that one machine and we played the long game if you were at the talk on Thursday Carl was talking about how we've begun building a manufacturing plant in Denver and all of our manufacturing is going to go from now we have a Taiwanese partner and a partner in San Francisco and it's this big long circle where we where we iterate and we iterate and we iterate and there's lots of differences between us and our Taiwanese partners and sometimes it's chimney sweeping week and everybody's out of the office over there and so it takes some something that we may have designed already and turns that process from perhaps the six month process to an 18 month process that doesn't really allow us to incorporate our customer feedback as quickly as we'd like to and so as we started looking at things we could automate in the manufacturing process and started to learn and hire people who were you know coming from a hardware background we started to think oh well actually this isn't as hard as we thought granted we'll see but the so we've designed a couple of desktops and we're starting to buy the equipment in order to build those chassis and build this thing we call snap fit which is our open source method of putting together a desktop so that it can easily be taken apart quickly and so we call it snap fit because everything snaps into place and so we're we have always been focused on creating an open ecosystem and that doesn't just mean software it means hardware and so we've been patient and we've been working towards the goal and that patience is really starting to pay off we're very very patient Sergio very patient someday someday we'll use Unity 8 I still believe I shouldn't do that to him he doesn't as he said he doesn't even work on that part of the of the project that I guess this slide is kind of pointless because it's a picture of the snap fit but if you would like to see it after this I will have my laptop down there somewhere and we can go through the pictures together as a family all right follow your passion when Carl started the company nobody was making hardware born to run Linux we're the largest Linux only manufacturer on the planet as far as we know and and he used to he told me a story that's actually super sad in the early days he said I thought he was joking at first he said I used to go down to the river and fish for salmon for food and and I was like and so I laughed really hard and he's like well I'm not really kidding you know the salmon the last if a salmon would last about two or three days and and why did why did he do that is because he was doing something he believed in and he was willing to sacrifice in order to achieve that and fortunately for him it paid off because the world changed and open source and Linux started getting deployed in data centers people wanted to work on the same platform on their work station that they deployed to and so ultimately we went from doing you know maybe 10,000 20,000 30,000 50,000 dollars you know a month to doing you know now tens of million dollars a year and in revenue so that has changed in a big way and that's because he followed his passion and that's something that we continue to do like I said there's a lot of there's a few elementary guys who work for us and and even though we ship a boot into we're constantly talking to them about what they're working on and how they incorporated certain things and taking that passion and letting that kind of make its way back into a boot too and also just letting them explore that you know we're hosting the elementary developer summit that they're doing that they raised money for here recently on Kickstarter it's happening in our office we're going to be loading up the imaging system with the different versions of elementary so that they can do their testing really quickly on our machines we have in our lab and it's important it's important to water and take care of that passion and so we make sure that as a company that we do that the final slide is have fun just generally if you're working on open source have fun doing it because you're one of the luckiest people on the planet and if you're part of a community you know there's I'm convinced there's nothing better to spend your time on because open source software is such a great way to move humanity as a whole forward it's this way of sharing your knowledge with the world and then collaborating with people from all over on a shared vision and I I heard that Linux as a project and I don't know if this is true but I like to think it's true before I go to bed turn out the lights Linux as a project has the most man hours in it of any project that's ever taken place on the planet that's pretty significant that's there's a ton of companies there's a ton of individuals contributing to this shared vision of a computing platform that anyone anyone can pull down and use for free modify and change that's pretty damn awesome and so if you're not having fun doing that then I don't know there's other things you could do I suppose I'm going to go ahead and put my cards out at the end when you guys are on your way out I'll probably put them on that thing back there I'm the community manager like I said at the beginning I exist to engage with the community that includes you you're in this room you're now part of the community so if you have any questions that you want to talk about if you have anything you want to talk about I mean I'm pretty open I won't probably talk to you about like you know really personal issues that you're having in your life but if you want to talk about technology I'm all for it so now I am more than happy to answer any questions we've got about 10 minutes I don't know if we're going to get kicked out after that so I'm willing to hang till somebody yells at me I was really impressed to add on to that I had a kid stop by the booth earlier today and I you know sometimes you you hear people say something like ah kids nowadays just sitting there playing xbox or whatever this kid stopped by and he looked like he was like seven or eight and and so I I walk over and I guess you know I didn't know what to expect so I was being patronizing because you know 78 year old I I wasn't going to give him the whole spiel from beginning to end but he forced me to because he started asking questions like what's in VME support like it's like okay yes there's great in VME support on this thing and I started you know walking him through everything he's like he tons of people came by the booth I told him about Thunderbolt 3 support and what that means I said this port has Thunderbolt 3 support and the kids like cool and he's and he's like I've got any GPU and I want to plug it into there and use the and make use of my NVIDIA GTX 1070 and he's like that sounds great I'm like I'm like you mean like nine out of 10 people had no idea what I was talking about and this kid's just like going through it and by the end I was like man I wish I could hire that guy so yeah I can attest to I really like the idea one thing that we try we've tried to do is we've been giving to kids on computers for a long a long time and we like the idea of giving them a machine that they can actually change and modify not a lockdown environment where they don't get to learn about how a computer works I think that's I think as a company we realize that it's vitally vitally important for you know not only use the computer as a tool but understand how a computer works I call it computing plasticity it's important like brain plasticity it's important for someone to understand the concepts of how to save a file not just that the floppy disk button is save like as a plot you need to understand that there are multiple ways to do this and there are multiple ways to accomplish this thing now on each computing platform but the concepts are the same and so that's why I'm super super hopeful that Luis worked at a school district I'm super super hopeful that even if we can't have like a whole bunch of systems 76 labs that they're able to get you know a ton of Raspberry Pi's in or some some computing platform where they can learn about how computers work any other questions or comments or okay yeah yeah all of our patches are in a are in a Debian based repo and so you can add the repo after the fact and have the patches that we carry before they're accepted upstream do it yeah no problem help yeah I can tell you that our office does not just comprise Ubuntu users sorry sir do you know we have a wide variety of what's that that's true yeah we have a wide variety of user users using different platforms even within our office and by platforms I mean distributions we have a long time Debian user and she says don't touch my Debian and we have a arch user and pointing because I'm thinking of the layout of the room we have an elementary user and we have an open sucy user so we have a lot of different distributions represented in the office and and that's why they engage with part the communities that they're part of in order to make sure that these distributions work well also we do things like the backlit keys and changing the color on the orcs pro are done in the firmware level and so it regardless of what platform you're on you'll still be able to change the color of the keys and we do little tweaks like that in order to make sure that you are able to jump to the platform of your choice in order to do what you need to do we officially support Ubuntu but that's because we're able to have those upstream relationships and we don't spread ourselves too thin because where do you stop with distributions there's hundreds and hundreds of distributions out there and eventually you have to say hey we can't possibly test all of these so no we we actually we actually order so our partner who makes our keyboards includes those when the keyboards are made we they are not a sticker they are actually a distinct key hint hint nudge nudge we have another hardware partner now and we're going to be releasing some interesting keyboards for our desktops that have the Ubuntu key so they if click click click click is the sound you like when you type on your keyboard you'll be very happy any other questions no not at all if the by the time we get out of here in what two minutes the expo hall will be closed so you won't be able to unfortunately you won't be able to is it two fifty eight for one fifty eight yeah okay with a more screwed uh you won't be able to go and probably check out the machines although we are here through the rest of the evening so if you see one of us and we might have them in our backpacks and we can show you the devices but also we you can contact us our contact information is on our website you can call and get a human being pretty much right away and you can ask them to your heart's content about the machines like I said we're pretty transparent so if you want to know like the tech specs of the new machine that we just announced Thursday you know we can send you the tech specs all of our machines have a complete product specification list that includes all of the stuff that's in it which is you know super important when you're taught when you're thinking about putting maybe God forbid but free bsd on it or something you need to know that all that stuff will work just fine to uh to the question over here about printers I'm not a support tech but our support texts use they use linux every day and they help our users use linux and install things like printers and things like that we do full hardware and software support which means that we actually support boon to up and down so if you have a question and you call and it's about a boon to then we will try our dam just to help you come to a solution same goes with other distributions but we'd say like hey we don't necessarily use this every day so you know we'll try our best the as such if you do have questions like that I know that the those guys are super super smart and they will and somebody was saying the other day about how he he just had this ongoing support ticket and it was not a specific thing it was just his general inquiry like support ticket that we had open for the for the past two years and it's just every time it comes up with a question you just go to the next line and ask it and we answer and and that's fine like we we we're your personal consider us your personal you know computer butlers you know and you you have a question or you want to know something about it doesn't matter the hardware the software we'll try our dam just to stick with you and and help you you know understand what it is that you want to work with and beyond that and we always appreciate contributors so like I said we have a knowledge base we have a few projects that we've open sourced ourselves that we use to run our business so if you're curious you should check us out at github.com so I system 76 we have lots and lots of resources and fun projects you can do on the website we have how to make a completely controlled Christmas light display we have how to make an augmented reality sandbox using our stuff and we also have some use cases for just we call them weekend projects just things you can do with friends family children in order to help them learn about Linux and learn about computing in general and so I encourage you to check those out and share them with friends and family so that's it guys feel free to come up afterwards and ask me any questions that you want to and great seeing you here at skill well I went to watch you know I flew there just for that and it was the first time in my life I said I don't want any more beer I don't I never thought in my life I'd be saying that but yeah but that that is not your product just a quick question too you guys like sell any like beer phone systems you know like in other words like any you know laptops you know I built my own custom desktop so but you know your laptops do you like you know full configuration and SSD drives and all that kind of stuff or so so what we do is uh on the configuration are you open next year no okay on the configuration portion of the website now do we get pretty granular is what you're able to put into it right bare bones all of we have an open hardware warranty that allows you to open up and replace any of the parts of the machine without avoiding the warranty obviously right and so um the so the what that allows you to do is you know obviously if you have a three-year warranty you can modify it as much as you want and not and not and that's a good question too so in other words your memory is a bit you know some some places now they're sold around nothing is sold around nothing is sold around so you get access to hard drives and memory no problem on the laptop correct that's great and that is not ever going to change that is that's been wonderful yeah uh that that is a core part of who we are and uh you should be able to change your wireless it's amazing you don't want to hear a lot of your users I can't believe it you know you know yeah no that's that's well we get rewarded from that too it's not entirely it's there's also a monetary you know yeah part to that because people buy our machines specifically because of stuff like that that's that's we had a what is this chief of staff for a United States has your running windows on our machine which is right but uh he bought it specifically because of that you know he said that's great I can't find any place where I can where I can change out my components and not get you know a tiny spot on the risk for it except for you guys your bios are you using UEFI or what are you guys doing um what is it um I keep hearing there's going to be an open source of bios I heard that from that five years and stuff like that here's the deal you know what's that we work with with on our machines that have an Intel bios we work really closely with Intel okay um we have not used to like move to like core boot or Libre boot yet partially because it's just there are some things that aren't implemented well there yet right and uh we walked this line of being open and being having machines that work and uh see and so we we I can't remember if our machines use UEFI because I know that this is like an ongoing conversation and I and I know that it's very easy to toggle it on off I don't remember if it comes installed with UEFI enabled right usually they're capable of UEFI but I know for a while we ship without it enabled right I don't know for still okay I know those two they have a finding key and so they're the big distros so they're you know we can still get around yeah yeah yeah it's just easier easy to just shut it off excellent and I'm sure you don't need it because you're asking the questions but our cortex have walked plenty of people like I said to Dell people don't know the hell I went through with Dell I mean you know you know it was a fried special cheap you know it's a cheap notebook that if I lose that I lose that I'll have my life on there and that Windows 10 is like get me out of here I you know like it was just a disaster yeah for everything I'm going to definitely look into your uh you know you know no problem man thanks for asking a question got interested yeah um so I'm sure it was going to be a problem why can't talk please yeah I have I have a call yeah it was a lot of oh yeah test okay oh it worked all right yeah I think I think so I'm trying not to trip on this yeah cool all right everybody uh so my name's Ian Iberg as he mentioned um this talk is about using union kernels in production today it's not necessarily like a high-level talk it's not a what is a unicolonial talk there's a thousand videos on youtube if that's if you're interested in that half of them are probably by somebody like me so if that's what you're learning you can go to the other unicolonial talk or you can chill here if you want to kind of follow along you can go to github.com slash defer panic slash vergo so a little tool that we'll dive into later on in the talk but real quick I just wanted to kind of go over some things if you watched any of the unicolonial talks online on youtube in the past six months or so you might hear phrases like this union kernels are not here now union kernels are not ready for production you know this is a reoccurring theme and uh my questions to these people that are giving these talks are have you actually booted a union kernel the answer is no and then I would ask some questions of have you actually compile the union kernel the answer would be no and so so then the question is kind of like do you even engineer you know no um and of course we're out of Linux conference so so why am I talking about union kernels well you know there's a there's there's a definitely reasons um but in terms of who's using union kernels this is another kind of question that kind of gets brought up and a lot of people point their fingers and hand waving so forth so just to set the record we at our company have been uh using them in production for over a year and a half but who cares about us let's talk about Ericsson I personally know some of the people over at Ericsson working with union kernels I know a couple of their lead engineers I know some of their business line guys that are actually using union kernels in customer facing products today not like yesterday not in the future like they're actually using it today meet down at their office from time to time for our union kernels meetups GE is using union kernels Dell EMC once again there's another union kernel talk it's on unique that came out of Dell EMC you know that class full of the company now and then NEC of course now all these companies I don't really need to tell you guys are mega mega large companies so if if you're talking about like you know is this kind of a enterprise interesting thing I would say the answer is yes IBM is using union kernels in fact we'll get into more people over at IBM that are using them what they're using them for Cisco is using them not just for the NFV stuff but also web facing stuff and web facing stuff is kind of like what we're primarily interested in and then there's like a whole bunch of other companies that some of them might come last and some of them I can't list for various reasons but there's a there's a ton of people actually using them right now so it's not like are they ready for me in the future it's it's a kind of a bullshit question and then just to kind of go through another thing real quick we're going to take a whirlwind tour of debugging union kernels because this is something that gets brought up every now and then you know defer panic is the company I work at and if you're a go programmer you might notice the two words defer and panic are two keywords from the go programming language and so so anyways we had built out a whole go APM tool two years ago to do kind of application performance monitoring on the go language and when we built the first go union kernel the very first one of the first things that we tried was like you know does our APM agent actually work inside the go union kernel and you know the answer was yes so like not only are we capturing errors and so forth with full back traces and also you know the memory usage which you can see is really low in here but you know everything else too and despite that you know gdb maybe the most famous debugger out there maybe outside of Microsoft Visual Studio works perfectly fine to debug union kernels in fact these are instructions taken directly from the Rump Run GitHub wiki on debugging a go union kernel using gdb so that argument is a little bit bullshit this is include os is kind of debugging interface I wouldn't call this debugging it's more of like a stack monitoring interface and as you can see they export all sorts of stuff everything from memory maps you can go pointer chasing to the stack sampling to the login like the whole shebang so this comes out of the box with include os and you know just looking at you can tell already that there's more than enough to go on here and then osb one of the original union kernel implementations this is an interface that they support now a lot of their stuff runs through the jbm right and so a lot of this stuff that you're looking at is obviously jbm specific but once again this is a lot of information if you're going to debug or you know figure out what's going on with that isolated app now let's talk security this one's this one's kind of a pet peeve of mine because everybody likes to throw in their two cents about it and the thing is is like some of the arguments are just not well founded right so so this is a headline taken from two weeks ago university suffers DDoS attacks from iot vending machines they're basically saying they got owned by a bunch of twinkies so like like this is a real headline this isn't like onion or anything I mean this is a real honest to truth headline from not only two weeks ago and so security obviously is it's been a problem and it's just it's getting worse by the day and so when people talk about unicornals and iot and then they start talking about like me robot net and all this sort of stuff you know it it all starts conflating and people start throwing around opinions and so forth but some of the people that throw around opinions about security or not giving well-sounded arguments so if you want to talk about security credentials they should probably like figure out who they're talking to first I am myself has spoken at numerous security conventions for like the past 15 years I've worked for multiple security companies and I've gotten in trouble when I was a teenager for facing websites and probably making people a little bit older than me lives miserable so in terms of security credentials I kind of know what I'm talking about here and you know one of the big things about unicornals in my opinion is not necessarily the attack surface that's small and this is something that gets banded around a lot yes that's true it is a lot smaller but that's like in my opinion not a not the greatest argument out there to me it's the fact that you can't just fork a shell every 10 seconds right so if I'm an attacker the very first thing I want to do is pop a shell on your system as fast as I can whether it's automated or manual attack it does really matter this is like the number one goal that I have the fact that I don't even have a shell to pop let alone a fork system call to use is a pretty big net negative for me trying to attack your system yes there's shell code that does not spawn shells but it does other things like add a account to Etsy password well that's kind of useless if you can't get into the system right so so this is this is huge in my in my eyes because if you can't pop a shell on any random system it's really hard to mass hack and why is mass hacking such a big problem well if you can't do that how do you exactly build a botnet this is something that doesn't get talked about enough out there this is this is such a huge huge thing a lot of the automated attacks are what preface these botnets and so the ability of not being able to pop shells is kind of at the root of this and unicolonels kind of deliver that you know from the get go so enough about that some stuff you might have missed about unicolonels in the past year if you're not kind of like in the ecosystem because if you do see some stuff it might just be the big headlines like mirageOS released in version 3 a couple weeks ago or something like that but let's let's run through these real quick so so last year and the year before we heard a lot of talk about serverless and function as a service and so forth and a lot of people were talking about like you know well how would you encourage us to play into this picture and you know there's a question that we kind of like looked into as well because as you can kind of see I don't know if you actually can see that we spun up like 1500 of these you know in a matter of minutes spun them up spun them down spun them up spun them down and yeah it's probably hard to see that column there but these are all about one megabyte big they're taking 32 megs of RAM each and so we were able to just spin and cycle these up and down a lot really fast and so it's kind of something that you wouldn't be able to necessarily do on the existing public cloud solutions and even like if you had that open stack solution this is kind of something that we would look at so so this is all this particular example is right in on KVM but you could theoretically do this with like any hypervisor then particularly has lots of good guarantees on performance in that regard some other interesting things that kind of came out last year guys their friends has on he's you know I mentioned Erickson was using Unicolonel so this is actually over at Erickson's office at a Unicolonel's meet-up we do in San Francisco and so he actually works for Siri or not Siri of SRI SRI spun out of Stanford Research like back in the 60s or 70s there was responsible for producing Siri which Apple later bought anyway so really smart dude right and his kind of research thing was we're going to take this arbitrary legacy code we're going to compile it to LLVM that code and then we're going to do optimizations and analyze that that code to produce a really optimized Unicolonel and so that's kind of one of his things that he's been working on once again it's not something that gets like advertised about or anything but it's some really incredibly interesting stuff that's been going on in Unicolonel land another thing somebody was joking on Twitter I don't know when this was I think it was October looking at the dates and they're like like you know Elk you know the Elk stack was such a pain in the ass to set up and this is like some dev ops dude and and then somebody responded to him he's like yeah well you know like try running that in a Unicolonel you know and it's like well that's that's interesting you know I I haven't looked at it I don't see why it would be a problem right but so we went out and spawned this up and so this is Elasticsearch running as a Unicolonel and there's no code modifications at all the stuff at the top the command line parameters and environment parameters in its system out there right so stuff like the class path and Java of course abuses the Hula command line arguments but so this is this is something that's like pretty straightforward you know for anybody who's in that domain of the ops world another thing that happened so so every now and then I reach out to Derek over at Sarah and one of his pet projects is NET so it's a it's a high-performance message messaging system written and go like super high performance and you know I reached out to him I was like hey you know have you tried any of these Unicolonels yet have you have you played around with it yet and he's like no no but you know so so he signed up and he he immediately tried out NET and of course it doesn't work for him right off the bat right but what was really interesting if you look at this code was it's all of 13 lines of code is what we had to stub out to make NET work as a Unicolonel and the only reason we even had to do this is if you look we had to specify a build tag for the Unicolonel implementation where we were using the Rump Run and so so it was very it was very minor edits to NETs and so this is still in the NETs source tree and you can run NETs as a Unicolonel this is the type of real world stuff that people can actually do today it's not it's not rocket science every you know you see all the PhDs and stuff talking about Unicolonels but this is something that like anybody can spin up and do themselves now okay speaking of PhDs these guys are PhDs but so this is also at another Unicolonels meetup the two that are talking here we're talking about doing get ready for the the buzzword heavy stuff but they were doing blue green deployments which were responsive in with the Diffy project with Unicolonels and basically what they're looking to do was do a deployment model similar to Docker and so forth but without the registry since they were so tiny the artifacts that they were deploying they're like you know what we don't need this middle man of a registry we can take that out and we can just push our updates that way and then do these blue green deploys and so that's a project that they've been working on for the past like month and a half two months maybe more that's that's that's what I know of that anyways another more recent thing that has come out recently Dan Williams you know I mentioned IBM people over at IBM have been doing some work with Unicolonels Dan Williams is one of those guys he's responsible enlarge for the solo five project which is a new virtual machine monitor for specifically for Unicolonels because Dan's like this one guy he's like you know what if we're throwing out the operating system let's just throw out this old hypervisor as well so anyways basically really smart dude and the gist of this announcement is that he's starting to integrate his solo five project with the native OSX hypervisor API which is really kick-ass news because the hypervisor situation on OSX sucks and it's honestly like one of the biggest things for the container already people out there too so this is this is really great news coming down the pipeline now let's talk about what sucks let's let's I'll be honest there's there's things that do suck you know and I'm not going to just spin bullshit I'll give you real life examples so sometime last year I was like let's take Postgres and let's turn this into a Unicolonel I just want to see like what is the work involved in this now I don't know how how much you all know about Postgres or how it works but I kind of knew going into this project the challenges that lie with this and how they compare and contrast in Unicolonel land so I went ahead and downloaded Postgres and the very first thing I did was look at the number for it calls right so on 953 which like August or July of last year there's about 46 calls there so all those are going to have to be turned to threads like right off the bat you have to transform all that so you know you put on your headphones and step one you just start mass commenting out everything right you know because I'm just trying to boot it you know what's the bare minimum to boot this project step two just go to it it's walking print out the bugging everywhere and then you know step three you know all these works we can just change the threads the vast majority of Unicolonel implementations out there allow many threads but not necessarily multiple processes and