ent at the same time. I'm actually a bit proud about this feature, but this is something that doesn't exist anywhere else. If you look at what Office 365 for Microsoft does, Skype for Business is also there, but it happens to a completely different window. And the same for Google Docs and Hangouts. So this is a bit, for me, also proved that crazy open source community can also do innovation, which is quite nice. Okay, so the summary is we want to build something that has the same use experience and features than Office 365 in Google Suite, but you host wherever you want, on your Raspberry Pi, on your Linux server, on your data center. You can deploy this one click on AWS if you want on a big cluster, whatever you want. It is compliant. I can say this because it's basically always compliant because you can always host it the way it is required. You can basically decide about availability, about how long files are backup or archive. If you have an auditing log, if you're what kind of authentication you use, it's super flexible because it's open source and on your infrastructure. Again, 100% open source, no vendor log in. You just can take the software, you use it. If you want to have an enterprise subscription, support from us, come to us, we help you. If you think that we're doing a bad job, the cancer subscription, you can keep on using the software. All right, there's no vendor log in. I think this is very useful and a lot of customers from us like this freedom. Yeah, if you want to use it, you want to try it, next-low.com slash install, and if you want to get involved, everything is open. You have a community of 1,800 contributors, only for code at the moment, so everybody's welcome to get involved. So, this was my presentation, thank you. It's a good question. I mean, you can imagine that there might be conflicts, but there are not a lot, I have to say. I mean, the decision happens basically in GitHub, in the tickets or in the pre-request, and I can rarely remember that there were some heated discussions. I mean, there were some exchanges about arguments. Some of the things here, you should do it like that, you should do it like that. Often, the people who have the highest reputation in the community basically are, I don't know, have high credibility, and if they say something, they're not to believe it, or there's also the rule that who codes decides, right? I mean, if someone does a feature and someone comes along and says, this is totally wrong, you should do it differently, then the answer is usually, okay, show me. And then they're free to build an alternative implementation, and that's fine, and then there's more discussion, and at the end of the day, then some solution is picked. I don't actually, don't recall a lot of conflicts there. Yeah, yeah, yeah, yeah. I'm not a big believer in voting of random people. I mean, you really have to do something to have a voice, I think. I mean, we have a feature that in feature requests, you can actually vote. I mean, everybody can go into GitHub and say, I like this feature plus one, and it's a little bit higher on the list, sure. But this doesn't mean that someone really develops it, right? I mean, it's developed if someone develops it, and not because it has a highest score. So I think, yeah, it's the code decides not to vote, yeah. How's the company organized? Like, up to 12, four, 12-cloud, yeah, yeah. Like, where are they now, and how's your organization organized? The company is boring organized. I think it has no magic source there. I mean, we have a, we have a sales department, we have a marketing department, we have an engineering department, man, this is maybe a little bit different. So for example, we don't have a separate support organization. Usually companies have a separate support organization, and engineering is different. I do this together. So the person who develops a feature is also the person who then has to help the customer to fix it. I think this is healthy, I think. And also the customer appreciates to directly talk to a competent person, not to a random call center guy who has escalated five times before, I don't know. So what we are selling is really premium support, right? This is also not cheap. But customers that really talk to the people who wrote the code and who can help with everything. And this is why we have support, engineering is the same thing. And then we have a small professional services department that doing projects. I am, project business is something you have to be a bit careful because it doesn't really scale. I mean, a lot of open source companies try to do projects to finance their product development, but there is a bit of a conflict between consulting work and product work. So we do projects or consulting work, but only for big customers. That's basically it. There is no real magic source. I mean, we are really distributed. So we have employees in nine countries. So we have offices in Stuttgart and in Berlin in Germany, two offices, but most of the people are just work from home. The reason is that hiring good people for me is really easy because I can just look in the community. If I need to develop in one area, I can look who is doing work there in a contain. What you do as a hobby, do you want to be paid to do the same? So this really, they know the code. They are motivated. They are behind the product. They already know the people. They are productive from day one. So this is hiring for me is very easy. The job I guess that they work from somewhere, right? They are not in the same city as the offices. But that's okay. I mean, the whole community works over the internet anyways. So employees can do the same. Speaking of money, how do you make provisions regarding money? When you are paying for offices, do you have contracts with your customers? Is there a lot of positions that you can make? I guess that's the same as with every company. I mean, me and the leadership team, we have to do the hard decisions at the end of the day. Sure, it's our job to make sure that everybody gets paid next month. So do you have like the leadership team? Yeah. They make budgeting decisions? Yeah. I mean, I don't know, 98% of the money is salaries of course. So budgeting is, yeah, but yeah. Yeah. No, we are, it's, I mean, I believe in a nice community. Everybody is welcome and everybody can be part of it. But as a company, I of course also am responsible that everything works, right? I also hire people if they're not the right people. I don't hire community people. I can do what I want. But for employees, I have to make sure that everything is working. Of course, yeah. Companies owned by me and some other early team members. They're no external investment. Basically everybody who owns shares of the company, it's just a few people have the interest of doing this the next 20, 30 years. So there's no real pressure there. At the moment, I think it's a good situation. I don't want to change it. I would never say never. But the problem is if you accept external investment, I mean, I get calls from VCs all the time, right? I say, hey, I want to have some money here.  I mean, it's relatively easy nowadays to get money. But the deal is that you basically pay like 10 times this money back in three years. Why 10 times? Because like only one from seven startups survive. So I have to compensate for the ones that fail. And the ones that survive really have to pay like 10 times this money back. And then they can make a profit. And the profit should be a little higher than on the normal stock market. So they really expect to get like 10 times back. So it would be super easy for me to get like, I don't know, 10 million of investment. If I somehow explain to them that I can get 100 million back in three years. Easy. The question is where does the 100 million come from, right? So it usually, I mean, you can get a lot of customers. That's tough. A lot of money. You just raise the next round. This is how it usually works. You just do the next round and pay the early investors back from the next round. But for that, you have to grow, grow, grow, grow, grow. And you lose like control of your company more and more and more. Or you do the big IPO, which, yeah, sure. I mean, that is like, I don't know, CEO.1% of all companies reach that state. So I mean, accepting investment is just like the huge risk. And if I can avoid it, I don't plan to do this at the moment. And then about VC funding, by the way, don't get me wrong. I'm not saying that this is always bad, right? It has a place and for some industries and some markets, maybe even open source, it can work out. The problem is I really think that everybody should understand what it means. Everybody, including the community, should be on the same page to understand what it means. Because if you decide this, then on the road, there are certain decisions already pre-made for you for the future. And if it's okay with you, it's all fine. I'm not saying it's bad. But often open source communities don't really understand what it means. That's all I wanted to say. But you re-question other mistakes. I mean, obviously license decisions are very important. You can't change them easily. You really have to think about it. And I'm not saying, I mean, we are happy with HEPL. But I'm not saying this is the perfect license. It really depends on what you're doing. You see these discussions with Redis and MongoDB and others are really struggling with their license. Pick the license and then they said, oh, it's no longer the right license and change it. And license is really an important thing. It decides about the playing field between all parties. That's something you really should think about it. And at the end of the day, community building. I think community is the most important value of open source. Open source has lots of values. You can audit the code. You have no Vandalogue and blah, blah, blah. But at the end of the day, the biggest differentiator of open source to other, the proper terriers that you have to community. I mean, we are still like small, right? 45 people is nothing compared to Microsoft and Google, right? Why can we do a product which is half very comparable? And only because we have 1,800 volunteers. Otherwise, there would be no chance. So I think thinking about how to build up a really good, working, happy community, it's also super important, I think. So you're referring to what Redis and MongoDB, the challenges? Yeah, yeah. That's a challenge, yes. That's a challenge. I don't have a good answer. I mean, just summarize it for everybody. It's basically, I mean, we are selling and end up by subscription, which give you like support and help or scalability and patches and like all kinds of things to make it run smoothly. But if you take the software from Amazon, take care of that. You basically don't really need our subscription anymore if you would get LexDot as a service from Amazon. And this is a challenge from MongoDB and from Redis. They also have the same end up by subscription as we have. But if you don't really need a subscription, if you just get a service from Amazon and then economy of scale, right? It's like, you don't need to care about patches. I mean, Amazon patches like all the instances in the world and that's done easy and no one needs to care. That's a challenge, yes. In fact, there are already, next lot is already uploaded by some other, I don't know, by some people to the AWS marketplace. And you could go there and say click, deploy this on this huge cluster. I don't know if this is properly done, but if it's properly done, that's all you need. You don't need our subscription anymore. Basically, you don't need our business model anymore. You just take the resources from Amazon, deploy with one click and then you have your cluster for 10,000 users. This is a challenge, yes, for our business model. I don't have a good answer for that. All I know is I'm not a fan of the comments clause licenses. I think this is not a good idea and I especially don't like that that call is open source because it's not, but I don't have a better answer either. I see a lot of product perspective. I see a lot of companies and enterprise organizations switching to Office 365 for Microsoft, a lot all over the world. And I think this would be just huge for them. Maybe this would be even more profitable than the old licensing business windows on Office for them. I think, I don't know, maybe a good idea to buy Microsoft stock. I don't know. But I think there will be like 10% or 15% or maybe 20% of the world who really want to be in control of their data and their privacy and after data and their jurisdiction and integrated with other software. Maybe want to audit the code, want to have their own encryption, blah, blah, blah, blah, blah. And this is something that we want to have as our customers. So because we are basically done, we want to be, we want to do the same than Office 365 from user perspective, but on-premise or at least where you want, not in the Microsoft cloud. And I think there will be a small part of the pie, but there will be a part of the pie who like that. That's our, that's our direction. Is it that you most enjoy doing? You look more into it. Me personally? Yes. I like innovation. I like crazy new ideas. I like, I don't know. We have like next week, we have another, another meeting in Germany where we plan the next release next month, 17, 16 will be out in a few weeks with nice new stuff. And then we are sitting together and think, okay, what is, what can we do for the next one? Just we do this group exercise with beer and pizza and takes like long time. And this is what I like the most. I mean, I sometimes look at what the competition is doing, but actually not that much. And just think, okay, what would be cool? What we can, what can we do? And then just, that's what I always liked about IT and software. Because I don't know, if I would be interested in cars, I couldn't build a car. No one can build a car at home. It's not possible or plane or whatever, but you can build software at home. I just find it's quite powerful. I always enjoyed it. We can all do something, some innovation on our own with open source, standing on the shoulders of giants. I find this amazing. I can do, I can be an inventor here. I can't invent a car, but I can invent software. And that's what I like. And let's see. Thanks. Okay. More questions? Thank you. I did not, I did not mean to get rid of the, can we adapt it? This is mostly just me checking mic level, so I understand how bad it is or how good it is. No. Is there a switch on the very, very bottom? We can just present for each other into our massive audience. Oh, no. And it's, it's sort of a nature of, of less technical talks, even though this is somewhere in between at scale. The technical talks definitely draw better. I shouldn't have said that too loudly. We could have lost 50% of our audience. I think it is. If it's too high level, there should be ample time for Q&A at the end for you to say, now wait a minute, that's too high level. I sure would like to see something a little more practical. Say that again. That's your feature request. Your feature request is that we are concrete and not conceptual. We get pretty specific about what we measure and where and how. All I have to do is keep my hands from freezing off. There we go. Yep. Let me turn this off. That makes a kind of sense. I'm going to start muttering quietly to myself as I put everything on mute. You should mute your phone, especially if it goes crazy on notifications. Let's go ahead and get started. I don't know if I'm supposed to remind everyone that it's being recorded. I'm just going to remind myself that I'm being recorded and try to stay in this approximate area to make that easy. Thank you for coming to this talk. We are in the open source and enterprise track. This talk is titled, your company cares about false sustainability, but are you measuring your contributions? My name is Dwayne O'Brien. I'll be speaking today with Danny Gellis. In my past life, I was sort of a Dev Ops-ish, Perl-ish guy before it was really referred to as Dev Ops. I was often referred to myself as the Perl guy on the Java team, but mostly responsible for making sure that things stayed up. After that, I was a JavaScript ruiner. That's not self-effacing. Legitimately, I was not a great JavaScript engineer, but I was hacking on JavaScript code for a while. After that, I stepped into sort of an agile facilitator role, helping a company through their agile transition. Then I was an open source enabler. I was running day-to-day operations of an open source program office over at PayPal. My current life, I am the head of open source at indeed the world's number one job site, which means I'm responsible for open source for the company, both in program and in sponsorship and in many other ways. My next life, I see myself as some intersection of a rock star vampire wizard robot. I haven't quite figured out what that is, but they always say that you should create the role for yourself that you want. That's where I'm trying to get. I'll be a full-time engineer, so I decided, well, I'll keep taking classes that I started teaching adults to code, which was great too. Now, my current life is I am a full-time engineer for an individual program, which makes me very happy. This was the first time I'd seen that, so now I know in what direction to help her develop. Apparently, both of us are trending toward automation and robots. Let's go through an overview of what we're going to cover in the talk. I'm going to give a really high-level overview of some of the goals and the thinking behind the program that we're building, mostly as context for the things that we decide to measure and how we measure them. We're going to look at what we decided to measure and why. We're going to look at the tools that we looked at, what we implemented, and the things that we built. Then we're going to take a look at what we've been able to learn from using those tools so far, and then where we're headed next with measurement and what we hope to use and build. Putting Indeed's open-source program together, we really spent a lot of time trying to align the team and a lot of thinking around a few core values. I don't want to share the whole doc today, but there's a couple of things I wanted to copy out. One of them is obviously that open-source is more than software, and this informs how we measure things because it means we want to look at more than just code contributions when we're looking at healthy open-source activity. We will meet you where you are.  Then that predictable long-term support is the ideal. Drive-by contributions on projects are great. Drive-by contributions of sponsorships and to foundations are also great, but the ideal state for both conferences, for foundations, for projects, for anyone is predictable long-term support, not just blips of contributions. With these core values in mind, we set about with this one goal, to build a culture of active and recurring open-source participation within Indeed. We want Indeed engineers and even people outside of engineering to be actively participating on a recurring basis. I called out in the abstract for the talk that you encourage the behavior that you measure, which calls back to Peter Drucker's management consultant, legendary person. If you can't measure it, you can't improve it. This here is Peter Drucker. Actually, that's Lord Kelvin. Lord Kelvin, you can trace Peter Drucker's comments back to a famous quote from Lord Kelvin. I often say when you can measure what you are speaking about, it expresses it in numbers. You know something about it, but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind. It may be the beginning of knowledge, but you have scarcely in your thoughts advanced to the stage of science, whatever the matter may be. Peter Drucker's comments, not particularly new, they go all the way back to Lord Kelvin. Let's take a look at what we decided to look at as a contribution. What are we going to count? Before we get into what we decided to count, let's talk about things that we decided not to count and things that we think make bad metrics in and of themselves. Lines of code is one. It's an easy thing to count, but easy things to count tend to be low-quality metrics. If you count lines of code, you're going to encourage the behavior of creating lots and lots of code, not necessarily any good code. You might unintentionally encourage the behavior that people are not writing very dense code, and so maybe the code will be a little easier to understand, but if the thing that you're measuring is lines of code, people are going to make lots and lots of code regardless of the quality. We also didn't want to count just individual contributions for the purposes of determining program success. If the thing that you're highlighting for the company is individual contributions, then you unintentionally incentivize a couple of bad behaviors. The first one is people who are prolific contributors will find other people who are prolific contributors, and they can turn it into a kind of competition, and this can be tempting in the beginning, especially a few years ago as gamification was a somewhat more common concept. Let's put up a leaderboard and see who's making the most contributions. Well, if you have really prolific contributors, people who are making 20 or 30 contributions of some kind today, if I show up to the company for the first time and the leaderboard has someone who's contributing at that velocity, I'm unlikely to want to jump in and play the game. There's just no way for me to catch up. Counted contributions, if it encourages competitive behavior among engineers, it can actually discourage new contributors, and it also puts the onus on projects because projects can then sometimes receive low-quality contributions as the people are trying to push their numbers up, and that isn't a burden that the company bears or that the individual bears, it's a burden that the receiving project ends up bearing as they have to push back contributions that are lower quality. There's some other considerations that you should keep in mind when you're looking at what you're deciding to measure, and one of them is whether or not you're going to share raw numbers inside the company, outside the company. And one of the values that I didn't list there is that we really try to default to open out everything. Our conversations are open. We try to be public about our metrics, at least in the confines that we can. But if you come back to that idea that competition will discourage new contributors, if you're sharing raw numbers widely, it is a thing that can affect your culture in ways that you might not intend. We also have to decide how you're going to account for personal projects and what that actually means. I, taking off any kind of Indeed hat and putting on my own hat, I think the division between company time and personal time in many tech companies is very artificial at best. And trying to figure out which projects happen on company time, which projects happen on personal time, and which ones actually are related to company work, or which ones might be less directly obviously related to company work, right? It's a thing that you have to work through when you're considering what to measure. So what we landed on as far as personal projects and what company-related contribution means is in the beginning we're really going to try to solve this problem one way or the other. We consider if you're making contributions to an open source project on your own time that are going to improve your collaboration skills that are going to make you a better collaborator internally and we're not going to try to filter those out, at least in the very beginning. Let's start very broad in what we're measuring and then as we gather more data, let's start to refine our metrics. Take a drink of water. So what we landed on for the main thing that we were going to measure is this idea of active recurring participants. And I'm going to unpack this. Let's go back to that goal that we have within the program. Active means that they have triggered some kind of an event that we care about. And when Danny is digging into the tooling, she's going to talk about what those events were and what those events weren't. So active, they've triggered some kind of event that we care about. Recurring, that means two or more days of activity within a given timeframe. Right now we're looking at this over a quarterly basis. It's not a terribly high bar, but that is the thing that we want to grow. Recurring participation. And participation means a broad category of events. So more than just code contribution, participating in code reviews and comments and so on. Does anybody, this looks familiar to this graphic here? Some seeing yes, some not seeing, okay. So this was rolled out, I believe in the last six months or so, on GitHub, it's the new activity indicator. And it gives you an idea for what kind of activity that you're involved in. Obviously we care about pull requests and commits, but we also care about the people who are participating in code reviews and issues. And this is I think indicative of a broader trend in the open source metrics community to recognize that just looking at code contributions is not the path to success for open source projects. Now before anybody starts getting ready to go collect stones to throw at me as I go through the project, I understand that this has been very GitHub focused so far and that open source happens in many places that is not GitHub. In fact, I have a talk about why you should not evaluate candidate during the hiring process based only on the GitHub profiles that centers on this very thing. In order to understand why it's very GitHub focused for now, I want to talk briefly about the difference between precision and accuracy. It is not important to us that very precise numbers that reflect all the contributions that are being made. What we want is a general sense for which direction the trend is going and is that trend, do we feel like we're more or less accurate? So we started measuring with GitHub, we know we're not collecting everything and in point of fact, there's a friend of mine, Cedric Williams who works a bit in the intersource community, sometimes talks about the concept if you went across the street to Cali Burger and said, how many French fries did you sell yesterday, individual French fries? They would say, I have no idea. That's kind of a ridiculous question to ask. But you might be able to say, how many pounds of potatoes did you go through yesterday? I think it's like, well, you know, 2,500 pounds and you can say, okay, well, 2,500 pounds of potatoes, maybe 12 potatoes to a pound, maybe a dozen and a half French fries for potato. Okay, you can get approximately an accurate guess, right?  So when it comes to non-GitHub contributions, we wanted to provide some mechanism for people to report that they were doing GitHub activity, open source activity outside of GitHub. And so we did what I think any reasonable multi-million dollar to put a Google form up and asked us to tell us more about their open source contributions if they weren't happening on GitHub. And this allowed us to do a couple of different things. It allowed us to determine if people were eligible to participate in the FOSS Contributor Fund, which I'll talk about in a little while. But it also gave us the ability to say, okay, now as we see, if we see a bunch of things come in that are happening on BitBucket, we know this is the place that we need to prioritize our next piece of work.  We know where to start, rather than trying to pick the thing that we think is going to be most useful to us next. So I'm going to hand it over to Danny for a little while. She's going to talk about the tools that we looked at, the things that we implemented, and then ultimately what we decided to build. An interesting challenge in that we're measuring something very different from what our open source programs are trying to measure. This is also an opportunity because that means it's an opportunity to contribute back to the open source community and it makes people think about this other way of thinking about contributing. Which is that existing tools measure the open source project health. What are our projects as a company and are people interacting with them, are people using them, things like that? It's Oregon Repo Focus. We however, at least for now, are more concerned with indeed employee contributions. Are we giving back to the projects that we use? Are we giving back to the community? How can we help the open source community essentially? And that is a more user focused thing to be looking at. So there are a number of different tools. This one is a list that came from the to-do group of various dashboards and crawlers and ways to get get contributions. And unfortunately they're all basically what we were talking about before. They're org and they're repo focus, not user focus. Getting the problem with trying to use those projects as they are for us if we're worried about user contributions is that if you're getting them, they will get all the users for an org and for a various repo, all the users that have interacted with it by doing events. But that will, for example, miss some of our indeed employees that are contributing to open source but not to indeed repos. On the other hand, it will also get us a bunch of users who aren't indeed employees that is not what we're trying to track right now. So we really needed to have a tool that can look at all of our employees and whether or not they're contributing. So the question is there were all those ones before can we figure out a way to adapt those tools to our needs that we're not creating the 15th competing standard. So we looked at some things. Let's see. So we decided on using the GitHub crawler for Microsoft to get our contribution data. As Dwayne mentioned, GitHub is not the be all end all but it is a place where there are a lot of tools that are made for it because it has a very robust API and it's where a lot of the contributions are happening. But Tergia also has a really great solution that looks at more than just GitHub but for us right now it's the easiest to do the GitHub crawler. A large part of that is just because up until very recently I was the only engineer on the open source team because we're a very new team and as you know, hiring engineers is hard and I know full stack JavaScript and I'd love to learn Python but we wanted to get some information so it made more sense to start with the crawler. What it does is it gets information about events on GitHub. Like all the others it is org and repo focus but we were able to adapt it to actually crawl through a user and that user's event. However, there are some pros and cons to using this crawler. Some of the positives are that it is a really expendable project for us to be using because right now we care about only our user events. Very soon we will also start measuring the things that everyone else cares about the org and repo health. It's one of the things that we care about and we'll be able to use this tool to do that very easily since it already does most of that. Another thing that's really nice about the crawler is that queuing and rate limiting are all handled in terms of using the GitHub API you have to get a token of course and it can do so many requests per minute which right now we're not hitting but when we're doing more we will. And the other cool thing about using the crawler is that I'm able to contribute back my contributions to the open source project so that other people will be able to also crawl user events. Now the negatives are that it being a large robust code base means it's also a large robust code base that's difficult for a newbie to figure out where things are how they're working and I've gotten to the point where it's working, it's doing the thing and I still know that it could be doing it a little better because sometimes I still want to tweak and make better before contributing back. The other negative to it is that one of the really cool things that the crawler does is it uses the webhooks from the GitHub API to see when a new event has been made and then and only then does it crawl it except that the GitHub API does not have webhooks for user events. It has webhooks for repos for orgs. I believe this is because of privacy concerns. I'm hoping that maybe we can convince them to create some webhooks for users. If not I'm starting to look into either using eTags or they do have an if modified header that you can do a little check and see like. If I am going to look at githubapi.com slash user slash danicella slash event, has anything been modified since the last time I looked at it.  There was definitely a moment when I was talking with the crawler people and the issues about what I wanted to do and they were like why are you using the crawler? You can't use our webhooks. We were like that's true but you do other cool things so that's why we are. Before we figured out how the crawler works as I said it's a large code base. We did just want to get up some numbers. I started at the very end of August and we had Hacktoberfest for October so it was pretty important to be able to get something there also when you think about quarterly things. You want to have some baseline metrics. So I basically just wrote some code that pokes to get hubapi for user events and then we filter it by the events that we care about. And then as I said before we just put that info into a Google spreadsheet. So we were able to look at it, get some basic metrics. What's kind of cool is that if you are just wanting to know some basic information that's really all you need to do. And I'm going to clean up my code for that and open source it so that's an option. For example, one of our initiatives means that we just need to know who has contributed over the course of the given month. For that you don't need the whole big solution that we created. You can just poke the GitHub API with the basic code. I wrote the interim dashboard code and it will give you that info. This was super useful just for counting Hacktoberfest. So I mentioned that we are filtering through the events once we get them. So these are the ones that we're measuring. Any sort of comments on various things, on issues, on pull requests, different things. We're measuring the pull requests, the issues in the code reviews. We're not measuring push events for two reasons. One is that often, probably even most of the time, push events are two personal projects. Like if I've got something up on GitHub I'm just going to push to it that it's my own thing. I don't want to be counting a few of those personal practice projects as possible. The other reason is that it's really not good open source hygiene to be using push events. So we're not counting them. So once we had that contribution information we needed to show it somehow. So we tried a project called Measure Off that's on the to-do list and it was also organ repo focused. How to make it work. It was going to be two completely separate paths that the code would take. Whether you were looking at an organ repo or looking at a user. The dashboard similarly would have two different parts. It would be like a flash repo and a flash user. It would have been really confusing. I was okay with it but when I talked to the Measure Off team this is how it seems like we're going to be doing it. They were like, we love what you're trying to do but that's not what our tool does. They were perfectly happy to let me fork in and do whatever I wanted. But specifically if we're doing all that work we wanted to be contributing it back to open source. It's an important thing in general if you're contributing to open source to ask the people whose project it is do you want this? They asked them and they said I'm not really sorry and I was like that's cool, we'll do something else. So we did. What we decided to use in our case was something that is specifically an Indeed project. It's called Imhotep and what it is is it allows you to build and query large data sets and then you can build tools for analysis and dashboards and things like that. Indeed it even came with a dashboard maker. This is a really big robust solution that made sense for us because we already had it set up at Indeed. Once I stopped trying to throw things into measure and just was like okay let's just do what Indeed is doing and now if I have any questions I can ask Indeed engineers it was up in no time. So that was pretty cool. It's something that you could use also it is open source but it would take a lot of work to get it set up and we're happy to help with that. So we made two different dashboards with it. The contributor dashboard for individuals and the aggregate dashboard for the program. And so the first one the individual one it provides contribution insights to a person which is nice for them, people love the little you know everyone always loves gold stars so you can see that you've made the contribution, you can show it to people it's also really useful for quarterly reviews and a number of our engineers have been have let me know that they're very glad we had that. Indeed is very data driven when we do our reviews and so being able to have a little like see this is what I did this is something you should care about is really nice. And then the second, oh and then this is what it looks like some of the things that it shows are weekly contributions total contributions you can see kind of the bottom of it the blue line is actually everyone at Indeed and then there's another line at the bottom that's an individual you can kind of see where you're at and as Dwayne said we don't want to be promoting people competing but people also do like to have a sense of where they fit in so we found that useful and the other thing that it shows which you'll be able to see a little bit on the next slide is there's a breakdown of every repo that you've contributed to and what sort of what sort of contribution a pull request or an issue comment things like that and at the end if you want to come up I can show you the dashboard in full there's loads of room on the slide for the whole thing and make it so legible the aggregate dashboard is just used by the open source program office and it provides us with program insights and it's useful for improving the program it tells us are our initiatives working essentially as Dwayne said we default to open on our team so there have been people who have asked me various questions that are answered by this dashboard and I've been happy to share the link with them and because we default to open which is not something that we're going around being like hey everyone look at all of these numbers and start comparing yourself so this one tells us all contributions contributions by month which has been super useful for figuring out how things are going with our initiative which projects you can kind of see at the bottom that little bit and then active recurring participants which Dwayne mentioned which is not currently on there but will be very very soon just because of the way the tooling worked out and that's one of the most important ones ok so what's next we are going to be adding other online code repositories like Bitbucket and Apache and stuff as soon as we can and also organization tools like JIRA and Slack anywhere that people are doing things that are helping open source projects and as Dwayne said it's not just the code design is important program management is important legal is important as much as we can figure out how to track any sort of contribution we want to put it in there as possible and of course the other reason to track everything there are we even already have people being like my stuff isn't on github and we're like we have the form and they're like not enough and we're like we get it soon soon soon people are also very nice they get that this is the one but we want to get it in as soon as we can so that's the first thing we're looking at adding soon another one is we mentioned we're not counting push events however the same way that if I have a personal project I'm likely to just push to it a lot of indeed engineers push to indeed open source projects it's not ideal and we kind of give them a little depth about it but it is what people do a lot so we're thinking about adding that in because it is still a contribution to an open source project we'll see as I mentioned this is the metric that everyone else is currently measuring and we want to pay attention to that as well we are looking at having some projects that open source projects the community will be excited about and doing more filtering of events so the question there is do some repos have more weight than others do we care more about contributions that were made to a project that indeed is using versus one of our most prolific contributors does a lot of work with a home automation repo not really helping indeed but it is still open source and that's something that we care about so figuring out if we're going to weight those differently or not and we want to as I mentioned stop measuring a user's personal project so it's not though as simple as am I contributing to a repo that's under my name because I could have made a tool that other people are using so we're going to look at things like are other people forking it are other people contributing it to it things like that so we can tell whether it's just a personal project that you happen to have up on github versus an actual open source project and then the last one is we might put in a toggle so that you can on the dashboard actually look and see what does this look like with only indeed projects without indeed projects with everything just be able to look at different factors now I'm going to turn it back over to Dwayne to talk about what we've learned so far there's a thing I really want to emphasize about Imhotep as the solution to the problem here it is a large tool to solve large data problems and if this was the only problem that we were going to solve with it it would not have been the right solution to use what made it one of the things that made it such a desirable solution was not just that it was well understood by people at Indeed who could help us implement it but it spoke a language that was well understood kind of across the organization it is the same tool that we use, for example to get insights about A.B.Test that we run on the site it is the same tool that is used to figure out if we make a change to something what was the actual end impact on revenue and it contains a vast amount of data that is used to answer other questions so by using the tool that already existed and was well understood within the company to answer the same questions but about open source we didn't have to do a whole lot of education around the tool itself people could just provide a link to a query and there it made sense and everybody understood exactly what that looked like and what that meant let's say I have Gilbert says we send things out to the people who contributed and so of course our community manager was like I really don't want to really find everyone's address can you do that programmatically I totally can because our information is in Inhotep which is the same place that our employee information is so I can do essentially a join it was super useful it was the right solution for us because of the constraints that we were working in but I would probably not encourage anyone in the room or anyone outside the room to build a solution on top of Inhotep just to answer this constrained set of questions there are much easier ways to get there and much more robust tools for that and another thing that I realized we were going to add to the slide deck and then forgot to was the problem of this list of users like what is the list of users that we care about right ultimately that means what is the list of GitHub IDs that we care about in this case now the Batergia lads in the room here already understand that managing identity across different places where open source is happening is a complex problem even if you're only looking at one solution GitHub and trying to figure out the correlation between an individual's corporate identity and their public GitHub identity GitHub doesn't make that easy for you unless you give them lots of money and there's no commonly used way even across all of the open source program offices to solve this problem and to get at this was as simple as taking the list of users in the indeed end org on GitHub that would be fine but there's not an easy correlation there either we don't force people into that process of joining the organization on GitHub so they've self registered they self identified their GitHub IDs the ones who want to participate in the program and now we have a list of end users NIDs that we care about that don't correspond to an org or a repo or anywhere that we want to look at and then we look at the publicly available data and using that let's talk about what we've learned so far so the first thing that we learned is that we had a couple of outliers they were very very active and when I say they're very active I'm not talking about 10 acts I'm talking about 100 acts like these are people who based on the group of events that we were paying attention to were 100 times more active than most of the other people that we were looking at these are heavily involved maintainers of very popular open source projects and it ended up being problematic in a couple of ways let's come back to that whole idea of competition discouraging new contributors again if we highlight numbers of the users across indeed these highly active outliers are going to be discouraging to new people or it could unintentionally set the expectation that this is what it looks like to participate in open source hundreds of events a day maybe for some people but definitely not for others we kind of had to reset ourselves because we were looking at this and trying to figure out how to account for everything we had to come back to the idea that you don't build for edge cases at least in terms of this particular solution if we wanted to be very precise we would have to include all the edge cases we want to be more or less accurate and so the outliers they don't particularly need our help they are already regular active recurring participants and making them more effective we want to help them but they don't seem to need our help and so we didn't need to build something that would take into account those edge cases the second thing is that we expect that the company who have never made an open source contribution and helping them get their first contributions made and we are also going to integrate into the onboarding training to catch people as they are coming in the door and similar learning mindset to see if we are helping people get acclimated and into the idea of contributing open source if we are creating new contributors for the first time I think that is a really great outcome for our program and then we want to pay particular attention to are we actually sustaining the projects that we use and this is going to mean and I am looking over here at Manrique this is going to mean marrying our open source compliance tool chain to our metrics and contribution measurement tool chain because there is not a solution that does this on the market right now there is no one thing that can look at a report from Black Duck or Sonotype or Palomita or SNCC or White Source and say ok here is all the things that you are using that we know about and also has visibility into the projects into which you are making contributions because if we can put those two things together then I can look at the report and I can say hey we are using this project in 2000 different products and indeed we have no contributors that is a place that we want to focus and encourage contribution and this I think is really the long term vision and the long term key to driving programs like this to help drive sustaining contribution into the projects that we use so quick review and then we should have 10 minutes for questions I think you need to start with values first and your goals second and then you get to the tooling when you need it right if you need to be very mindful about what it is that you measure and how it is going to be used that aggregate dashboard only used by the open source program the minute somebody comes to the program and says I want to see contribution numbers of everyone on my team so I can figure out who is doing better than the other like we have to have a long conversation about how these metrics are being used know why before you are building the how we have very specific things that we want to go after again growing that active recurring participation it is tempting to just look at the existing tools that are out there pick one that you think looks good start measuring things but if you don't know why you are measuring it or what you are going to do with it then you are still sort of flailing around to start with what exists again there is a lot of stuff out there it might not solve the solution for you but at least if you start with it you can look for opportunities to give back to the community and you should remember that the measurement is not the goal let's come back to Uncle Calvin over here Peter Drucker says you can't measure what you can't improve measure is the easy part and it is the part that everybody gets stuck on improvement is the goal if you are measuring it and not using the things from your measuring process to learn and improve on the work that you are doing then it is probably not worth it to measure in the first place and with that thank you for coming and listening we will happily take questions and we have got 10 minutes or so to stick around yes here I am curious how we can measure the effectiveness of COVID-19 that we are going to deal with though right now the events that are out there are full requests approved full requests denied and especially as we happily measuring the amount of time spent on COVID-19 and whether or not we are growing or encouraging COVID-19 that provides substance resistance versus rubber stance alright I am going to try to roll that up into a repeat for both the video and for everything else but as the maintainer of an open source project is a member of a community of an open source project one of the problems is that project maintainers have to spend a great deal of time reviewing full requests and it would be great to have some ability to determine when those are effective and when they are not and also helpful to grow people into the community who can take that load off of the maintainers so that they can maintain the project rather than just do code reviews is that fair enough a thing that I think would be useful to go after and I am going to look back at Ranrique here because I had a conversation with I am going to say your boss but I don't think he was your boss at the time about five years maybe four years ago on this the thing that I would really want to dig into there is for a given list of users over time are there pull requests taking more or less time to get landed so is it not about the time that it is open but how many iterations are involved in the pull requests now for a given user between two given users that number is going to be practically useless because no two pull requests are going to be the same you can't look at lines of code to determine if it is more or less complex but for a pool of users over a pool of pull requests I would expect to see that people who are getting better at reviewing pull requests the number of iterations go down over time in aggregate maybe that's right maybe that's wrong but this is the thing that we keyed into on the idea of how do you measure mentorship how do you measure someone's ability to be an effective measure if one project has ten people that are reviewing code reviews and you assume that they are just getting randomly assigned code reviews if one person is able to consistently get them in with three fewer iterations that sounds like that might be a place to really dig in and see if that is really a core competency if there is something else in play and that I think calls to the final thing I would say there that none of the metrics here in any of this will answer a question they will just tell you where to dig I think that's what I would have to say other questions yes we talked about GH archive as a data source and I really have gone back and forth on this to an individual showing up at GH archive there is no easy way to know it's going to be here tomorrow at least when we looked at it it left the impression that it was some person's project and if that person died or ran out of money or changed their mind that it could vaporize the next day now I have since learned that Google is involved in this GitHub is involved in this there are people behind it that give it more credibility than it seemed on the surface but then I actually had a conversation about whether or not to use this as a data source and ultimately decided no and I don't know that I would make that same decision today but I don't know I know that filmage out of the Adobe office the metrics work that he does all of it goes into GH archive a number of other people also look into the archive as well so it's hard to have people use GH archive it's hard to use we had already made the decision not to have all the tools based around it you had a question as well so the question was did we look at the tide lift because they are working on something similar so I know about tide lift I know about their model who don't their approximate model and if Luis hears this he might jump on me and correct me but the approximate model is we are going to take a group of open source projects we are going to offer support contracts on that group of open source projects and we are going to make sure that the money from those contracts goes to the people who are maintaining them and we are going to incentivize them to perform good maintenance work on the project it's much more complicated than that but that's essentially the model interesting model I think there are a lot of different things that are being tried in the world of open source sustainability to try to address the perceived problem that's there I'm interested to see how it works out it's but it's not something we would implement, it's something we would purchase from them if we wanted to get support from them my personal goal and my personal drive in the way I approach open source programs is that I want companies to be involved in the projects they used not just because altruism is great and I'm a very altruistic person I want that to happen just because it's good for the project but it's also just smart for the companies if you are involved in the projects that you use you can see problems that are coming down the pipe before they land and impact you you can help make sure the only way to make sure that project is going to be sustainable long term is if you're part of a sustainability solution for it tidelips model might help solve this open collectors model might help solve this and models like the fosk contributor fund that encourage both participation and donation into those projects may also help to solve this I'm excited that there's a whole bunch of different solutions out there and I think it means that the industry is taking it seriously and then you had a question in the back what is the business value for our company to measure the open source contribution of employees in the company the only so this is actually an interesting question and I think you may have nodded as well because you might have intuited on something we had one incident where we sent someone their individual contributor dashboard and they said please forget everything you know about me because they found it creepy right and that was you know we kind of acknowledged that that might be a possibility but it caused a little flat flooded so we forgot everything we knew about them and his issue was I only contributed on personal projects and having my company have visibility into that felt weird right it was publicly available data we didn't see a problem but we honored his request the business case for us measuring this is really around driving the wider program the company itself does not really care about the number of contributions the individual makes or outside the legal implications what projects they're contributing to as long as they follow good hygiene and good policy they're staying away from sort of the core IP areas identified for the company and that they're contributing consciously but the goal is to again drive wider participation to grow employee collaboration skills exposure to technologies and to help make sure that we're ahead of these kinds of problems that I was talking about earlier so it is really thought of more as an engineering culture ad as opposed to anything else we are literally the only ones who use the numbers outside of individual to manager conversations do I think it's possible to automate software management I think anything is possible dot dot dot that's probably a conversation we should have after the talk were you just giving a thumbs up or are you doing that because I think we're at time thanks for the great questions everyone I've got a little bit of time Danny have some time that we can take around as well thank you it's important to measure important to measure contribution like the health of your projects sometimes stars just don't cut it giving back to the project that's something no one else is measuring it's something that's actually kind of difficult to measure with the way they get to have ideas yeah even if you had a number of users a number of commits a number of merges one merge isn't really equitable to another merge or comparable rather we have some baseline numbers then we do things like we promote October Fed be like oh we really promoted this the numbers went way up November we didn't do anything and the numbers went not as far down but somewhat down and then we just went for bullet points and they went back up and even more so we just keep trying different initiatives and we're going to be giving classes and about contributing and things to try and get the numbers up and up and up that's fantastic there's actually I think a lot of a lot of organizations that can really benefit from a packaged open source program dashboard sort of the hell you know where we're using it, where we're contributing to it what's the activity, what's the velocity how long we have an outstanding issue almost the same way in which you'd run an internally developed proprietary piece of software but then even a little harder maybe to measure some of those intangibles about influence and health of the community and velocity and chaos and get up crawler that you were talking about and even for security the truffle hog or a number of things that might just crawl your repos for vulnerabilities you might have is that mostly for attribution for that recognition and credit for that developer or for different purposes of just measuring the company's overall contribution it's a lab to nominate a project and anyone who has contributed in a given month can vote on which open source project gets to be about 10,000 per month and so it of course has to be there are a few caveats that have an LTI license that have to be a way for us to give them money it can't be a project owned by an Indian I think those are the main three probably up do they have to be projects that are maintained by does it primarily maintained by your organization? no, no look at our react, it's still amazing we're all worried about that we're worried about are we giving back the project to the developer and this is is it Uber? oh indeed very good you just said lead by example is that what you just said? yeah now I totally interrupted your conversation I did it well entirely to kick off the next talk I guess so I'm not here today representing them a company that I work at full time is SolarWinds a monitoring software company but I'll disclaim it again not speaking on their behalf today but share experiences over the last couple years of trying to have established an open source program office and this is your opportunity now to not say if you don't want to say you may or may not find this insightful but to that point it took me about two years to form an OSPO an open source program office at my current company I'll repeat it of which I'm not here representing today but do consider that open source program offices are growing in their importance and that we'll see many many more of them popping up some of the experiences I've had the environment in which I've had these experiences have been in a company that has about 50 products products that are sold as a service, products that are sold as shrink wrap software delivered on premise products that are taken to market directly to end user and products that are taken to market through channel so pure software company hundreds of millions of lines of code of software so and have products that are using you know comprised of like 90% open source and other products that are comprised of 90% proprietary code and so and then everything in between open source happening in this environment and I don't know if this has been the experience of some of you but certainly for me when these two terms come together open and source that occasionally has the effect of producing an almost physical allergic reaction to those within our legal department being just very uncomfortable with what that means to the company and how they perceive the company's value how they perceive how much intellectual property is within the proprietary code that we have also how much just the heavy consumption of an ingestion of open source software and whether or not we are complying with those licenses gets people pretty animated or allergic hence the creation, the formation of open source office open source program offices are all the more important so I'm here today to share those experiences with you and not necessarily tell you that this is the one way and the cookie cutter way that this should be done but rather this is how you might do it. I'm going to reference the to-do group I'm going to reference the to-do group is an open source group a collection of companies that engage in open source communities and have put together a lot of their experiences and shared them I don't know if it's a sub-project to the Linux foundation but it's certainly associated with the Linux foundation I'm also going to reference a survey that had been done between the new stack, the Linux foundation and the to-do group and so my name is Lee Calcote this and other talks like you can find at CalcoteStudios.com so if you don't suffer enough today you can go and suffer some more there for some other recordings get a particular focus right now technologically toward service meshes and there's a recently published e-book available here in case that's your thing and for some of you I know it is so reasons why an open source office is needed to be created well there's a number of reasons within there some being for speed for pace of innovation to have influence in those communities in open source communities for compliance I think I think one of the ways to explain why you'd want to form an open source program office if you take this tweet here from my friend Nithya who recently established a small open source centric conference out in Philadelphia works within the open source I think stewards of the open source program office at Comcast it's great that she formed this put on this event but why did she do it and interestingly for this event this is the short description of her event and it says she's putting it on to help connect innovation and drive technology forward speaks in large part as to why you'd want to create an open source program office why it is that open source is so centric to really any digitally transformed company any company that has that software is used at the core of their business now most of you or all of you have probably heard of software eating the world certainly not something that I had said but since you're at this conference you're probably also aware that open source is eating software most major areas of innovation in software are happening in open source and in case you're living under a rock cloud is eating open source lots and lots and lots of the cloud offerings are based on open source so open source super important to many organizations be to more and more certainly for technology organizations and like I said any that have been digitally transformed if software is important to your business open source is important to your business of the benefits for creating an open source program office again this is from a survey that I said that I would reference a couple hundred between two and three hundred people went through and took this survey from various organizations and some organizations that have already established open source program offices some that desire to and some that have said that they never will or they don't have plans to irrespective amongst all of them the top benefits that they noted were awareness of how it is open source is being used within their by and large the focus of that being how open source is being used in their companies or in their products in their offerings or in delivery of their services they think that if they have got an open source program office that they benefit from development velocity the developers can work more quickly they've got an understanding and structure around when they can contribute externally when they can receive contribution how they can engage in community how they can represent their company or not and then to the extent and then the program office often times on compliance and ensuring that the legal rules of consuming or contributing to open source is complied with as well as them to be influential in the open source communities that are important to them what kind of an interesting take away here is that that open source is important to developers developers don't necessarily like to do things more than once don't necessarily like to repeat themselves if they're going to do it once many not all clearly would like to do things in the open and be able to leverage that wherever they go be able to assign attribution to the works that they've done be able to show off their works be able to have people reflect on them and see the things that they've done not necessarily have their works held behind the firewall so to speak that's certainly attractive to having an open source program office can be very beneficial to attracting talent to attracting developers and engineers so interestingly the respondents to the survey that already have an open source office are not really don't really consider that a top benefit is talent attraction which is kind of funny because they already have much of the talent that they're looking for because they've got the office in there they've brought in folks organizations that haven't established a program office consider that doing so will help them attract talent so I get kind of a chuckle out of that as you go to form an open source program office you need to consider part of your strategy in doing so and from my perspective there's kind of five C's to formulating an open source strategy and that is about consuming, complying having community receiving and giving contributions and considering how open source is potentially competitive to your offerings and so addressing each of these five C's is something that a well-rounded open source strategy needs to do it's not all about compliance that also if you consider these five C's I consider that there's a particular path to mastering open source and that an open source program office can help with that that most organization's journeys start with consumption that they start for needs of either speed or not reinventing the wheel they go and consume some open source they use those as either tools or they ship them in their products or they use them in the delivery of their services and as they begin to consume open source they probably very readily need to begin to comply with the licenses that are associated and so the journeys for most sort of start from bottom up they consume, they comply maybe they do maybe they begin to perform a little bit of inner source which means that they might have code bases internally that they govern and steward much like many open source communities govern and steward their projects companies might do inner source both to benefit from the flexibility that that provides the flexibility of how developers are assigned to projects they might do it to get ready to engage more earnestly externally and kind of understand the mechanism, the mechanics of how to how to engage in communities like that and then some companies ever graduate from these first two rungs and that in part, that in large part depends upon what's core of their business and whether or not contributing to external projects is important or hosting their own and receiving contribution is important that they do, I would consider that their next step they need to consider whether or not open sources if open source projects are competitive to their offerings how are they going to either complement open source projects play within those ecosystems or maybe just outright compete with them all throughout here there's a community whether that community is internal or external community is core to the open source way recognize that companies that software companies, those that have software offerings source code from a variety of places it's really kind of three places but the combinations of those are kind of interesting so it may be that the products or the commercial offerings that they have are just pure open source that's all that it's comprised of or maybe it's a bit of proprietary software and some open source software maybe it's third party commercial software so not necessarily open source software but software that they've licensed and OEM'd or licensed and incorporated in and so it's a combination of these three types that they might meld together and create a product out of this is I had mentioned that for the company that I work for of the 50 or so products that we have the gamut that we span it goes from very little open source in our commercial products to almost nearly entirely open source in the products and for on average most software products today have a healthy bit of open source in there and this is really measured in terms of like lines of code so as you go to address each of these five strategies for consumption there's a bunch of compelling reasons we highlighted a couple of these but it's to move more quickly to speed up the delivery of your solutions it's to not reinvent the wheel to share the cost of development of that software there are some projects that really only make sense done in the open maybe that's for compatibility reasons there's so many open source projects maybe it's to innovate I mentioned before that most of the innovation within software is happening in open source ecosystems in open source environments it's to maybe keep pace to attract talent to be influential as I said before all of these are they'll pile up to reasons why it is that companies, if they want to be competitive if they want to keep pace they're likely at least minimally going to need to consume open source if they do they're going to need to comply things that you need to address as you go to comply with open source is that not understanding which licenses are being incorporated in your projects may mean that you get delayed in shipping your product if you've got a license that's incompatible or that you consider to maybe copy left you need to understand whether or not that means that you'll lose some of what you consider your intellectual property if that license forces you to publish the work if you you might find that you're doing a bit of rework if you use a package that's incorporated into your product and turns out that license isn't compatible and you're not going to use it then you've got to go do some rework or if you ignore it and you ship it and later on get sued for not complying with the terms of the license it might be embarrassing to you you might have punitive damages so some of the goals that you need to that we had set forth as we went to go establish what it meant for us to comply it meant that whether we're shipping the product as a piece of software you download or whether we're offering the product as a service that not only were the packages that were incorporated into our software that those were using licenses that we approve of but also that the packages that we're incorporating in are secure at least with the world's known published vulnerabilities so it's both license compliance and it's also making sure you've got secure code so you need to make sure our goals here made sure that we were upholding those licenses that we're tracking vulnerabilities that we're also giving the appropriate attribution to those that are contributing to the projects that's both engineers internally and engineers externally so that means we need to wrap a process around it and part of that process was to ensure that we were educating internally on open source that we acknowledge that there's in fact a difference for many of the open source licenses depending upon whether or not you were shipping software and distributing it if you're redistributing it or if you're hosting as a service so there's different licenses that apply to those irrespective we wanted to try to hold all software to the same standards we also wanted to empower the engineering teams themselves to be as self-serviced as possible to make even, for the most part I don't think I've ever heard an engineer say that hey this is the fun part of writing software is focusing on what license there is and going through and trying to identify all that and fill in the forms and track all this that's generally not what they're excited about they want to write some more code so to the extent that your process can facilitate them self-servicing and make it as friendly and easy as possible those are part of our goals we wanted to streamline that execution we also wanted to make sure that we were counting for the fact that there's third-party license code that gets incorporated proprietary code and open source code that gets incorporated again the inner sourcing that companies may or may not choose to do so maybe they can establish an open source like culture inside their organization many organizations have yet to cut their teeth on participating in public communities so doing it a bit internally first maybe lets them work out some of those kinks or kind of come to understand it it's a little bit of a chicken before the egg maybe they want to go out external and contribute a little bit kind of play within the communities understand how they work, see a few of it bring it back in either way they can benefit from some of those efficiencies they might be able to standardize on tooling I don't know that I mentioned this before but the environment that I'm within hundreds of millions of lines of code that makes different types of source code management systems so Github and GitLab, SVN CVS, Perforce VSTS and then lots and lots of code in and amongst those so not even the tooling is standardized so I think there's a potential benefit of practicing some inner sourcing that in order to do that effectively if one engineer wants to go help advance a given component of code and they're using TeamCity for continuous integration in this organization, in this business unit in this team for this product they're using Travis or CircleCI or this language or that language or this type of repository that of the source code management systems that I just mentioned they're not even all Git based so even just standardizing on something like that can really help and it needs to be done to facilitate inner source hopefully part of inner sourcing then outside of establishing that open source like culture that also you begin to benefit from reuse and how many times certainly in larger organizations you find different teams creating the same capabilities and really it turns out half the time some of the biggest competitions that the teams have are each other like the internally who they've got to watch out for it's unfortunate sometimes companies actually do that maybe pit business units against one another and see who kind of let them race it off and see who comes out first but anyway that doesn't need to doesn't always need to happen they can certainly share and I think if they were adopting some of those open source ways maybe they would be sharing more so again I'd said that depending upon the type of organization you're within they never aspire to this third rung of contribution I think it confuses a lot of folks to understand that contribution means both maybe upstreaming some changes you've made in open source that you're consuming so you don't have to maintain a separately forked copy and float all of your changes that you can do the right thing both for yourself and being efficient by not having to maintain that and also in just being a good steward of the community or not steward of the community but rather being a good citizen in the community and maybe there's some good will that happens out of upstreaming your changes so contribution needs your strategy here needs to consider outbound contributions as well as inbound the outbound that you're giving is a lot less complicated than receiving inbound contributions and actually hosting a project and all of the things that come with that probably an easier place to start with with outbound is probably more natural place anyway if the reasons that people contribute externally most often is because they're consuming some open source project they either find a defect or they need to augment it a little bit to be compatible with add a feature or what have you and so they might start by contributing outbound which is kind of a nice organic way of starting it's the easier way of starting whether interestingly whether you're receiving contribution for a project that you're hosting or you're giving contribution externally you're probably going to run into or need to consider whether or not there needs to be a license agreement signed by those that are contributing code again whether they're contributing to your project or you're contributing to theirs you'll commonly see a contribution license agreement a CLA or a DCO a developer's certificate of origin a bit of different philosophies and opinions about the impediments that these present to people as they go to contribute and how what the reward is how much risk they reduce between them and so if you're not familiar with these maybe very quickly the DCO generally facilitates the process of ensuring that the person contributing code identifies themselves and kind of authenticates if you will identify who they are and as part of that also agrees to the fact that they've got the right to contribute this code so it's just code that they've written themselves and they're donating it to the project or they source it from a different project that has a compatible license and so they develop the DCO sort of addresses those two items by and large the CLA that type of license agreement takes a little bit further and also tip, you know, there's different agreements but we'll typically address things like intellectual property and copyrights who continues to own the intellectual property of the code that's being written things also like who has right to file a patent for that type of code lots of things to consider like things get much more hairy as you go to contribute again, either inbound or outbound whether you are contributing externally or internally or you're receiving contribution community then becomes important and how earnest you're engaging in open source means is an indicator of how important community is going to be to you, to your organization easy to get wrong, lots of lessons to learn from other open source projects lots of lessons that they've learned, also lots of things that open source projects are doing well if you are on the show floor here there's the cloud native computing foundation the CNCS, they have some very well to do emergent open source projects and their governance model and stewardship and support of those projects is probably a healthy example of how to get community right I've mentioned before competition whether or not you address, you have a strategy that addresses competition is entirely dependent upon your business but in the environment that I'm in we've kind of got two mostly competing schools of thought and my philosophy is and I consider that there are a number of open source projects that people will choose to use instead of our products and as we go to help people understand how these products are maybe better or maybe why they would want to buy these products is that the strategy there is to help complement the open source projects that are necessary to come in and try to displace them outright I think that's one approach but if that's not necessarily what's good for the customers and certainly why would you want to do that so go in and complement maybe fill in some areas within that open source project that are a bit weaker or don't integrate well into your product set so complementing is a great way of assuaging the competitive nature of open source projects just a couple weeks ago I'd seen a mail thread internal to my organization that said hey customers are asking whether or not our products are integrated with this open source project with this competing one in this case it was Prometheus Prometheus is an open source monitoring tool monitoring is centric to the company that I'm at and the product manager's perspective was well I scoured the mailing list I've looked for Prometheus as the keyword and every time it comes up really what customers are asking what customers are doing is like weighing whether or not they should go use this Prometheus open source project or buy our stuff from his perspective this customer that's asking for us to integrate we don't need to we just need to outpace this open source project and beat them by going deeper and better and that's certainly one approach to that I say we sort of with my corporate hat on and trying to raise money for the company but that doesn't mean that you wouldn't also want to have your software compliment that open source project and allow the customer to continue to run the open source projects but just augment them so anyway different strategies to consider companies so the role we talked about why you want to create an open source office why what I consider five C's that you're going to want to cover and address in your strategy we talk about the role of an open source program office it's tend to be kind of the center of the universe within your organization for all things open to provide a structure and a framework for addressing how and when you engage in either communities or use software of the survey that I referred to in the past that question was asked to a couple hundred organizations and interestingly there's a lot of aspects to the role of an open source program office some of the top ones here I'm going to talk about more in a moment but is number two is to ensure that compliance happens you know there's a long list but there's a lot of things to be covered it's actually ends up being the full time role for a number of people so an open source program office needs to be aligned with your business needs to certainly engage with your legal counsel if not be led by your legal counsel needs to be aligned with your product management you need to make sure that the way in which you are using or contributing to open source aligns with your product strategy if not then your open source program office isn't going to really reach its full potential you do a word of caution as you go to engage with legal counsel is that for the most part it's been my experience that those folks the role that they play is to minimize risk lawyers are risk assessors and what they want to do for their clients is minimize the risk that their clients have in this case being the open source program office or those that are engaging in open source it's easy to maybe overrevolve there so my word of caution is to make sure that you're striking the right balance the legal group may stifle innovation a bit that would be just the question of well if this is a cross functional mix of different teams that form your open source program office where should you land it what business unit or what function rather again the answer is it depends maybe if you're an engineering driven organization and you're looking to optimize developer productivity maybe you'd land that in engineering and you're just trying to facilitate for engineers consuming open source and contributing to it and the process by which they do that how you track it, how you dig through your code and identify packages and things register them, keep a list get approval or maybe you land it legal if you're more conservative really what you're concerned with is reducing risk and so on and so forth I think that if you could if you have your choice maybe that's within an operations team kind of a program management team maybe that's within an organization like that sometimes those organizations are sub organizations to engineering organizations like a build and release team an engineering ops team irrespective all of these functions probably need to be involved to some degree in your open source program office if you need infrastructure maybe your IT team needs to be included to the extent that you're publishing notifications and you're redistributing software that you're using notifications of what open source packages you're using maybe procurement to the extent that your licensing or OEM OEMing third party software your talent acquisition team, your HR team hopefully is there raw-rawing about what you're doing openly and can attract talent your corp dev team if you go to make an acquisition of software that needs to be scanned through and understood the disposition of all the packages that they're using and so on and so hopefully your marketing team can go talk about all the great things that you do so it's a really cross functional office here's an I-Chart that talks about a number of the specific responsibilities that those different organizations might have maybe you suffice to say that one of the organizations that I didn't list on the last slide is something of a committee maybe an open source executive committee so if you consider that the open source program office really probably handles much of the day-to-day stuff around someone saying hey can I use this license I want to use this software is this license okay hey I've got a project I'd like to open source is that okay I want to contribute to this project externally is that alright in this sort of process of approval that OSPO that office is dealing with day-to-day things there's going to be the time where use of an open source package that uses a license that you've identified in a gray area or maybe you've identified as hey we don't use that license don't do that there may be extenuating circumstances that need to be escalated up to a higher council if you will and so for our organizations that really falls between our executive engineering leadership and legal counsel and so we try to make the call between how risky is the use of those things and we wanted it to open up a certain code do we have intellectual property within there and so it's kind of that that's who forms our open source executive committee that compliance process that I talked about for the open source program office kind of the day-to-day for us it looks like this and it's really the notion that code comes in from two ways down below it's either that we've got ongoing scans of all of our code bases leveraging a variety of either proprietary tools or open source tools to scan either literally the lines and lines of code itself or talk to package management systems to identify the bill of materials of all of our software what are all the packages that we're using what are the associated licenses for them if we run we're scanning our code constantly for us too we also acquire things occasionally so our corp dev is sort of an ingest point the other ingest point is either just engineers themselves or product teams saying hey we've identified that we could really use this capability we'd like to incorporate this piece of open source into our product and they go over and submit a form for us so either way whether they're requesting to use it up front or maybe they just skip that they went and used it put it into the code base checked it in merged it that we're going to want to have a process that scans after the fact to catch that just in case and so then there's a process by which kind of different phases that are pretty straightforward about identifying what license it's using whether or not there's approval to use it we need to then publish the notice of it we need to redistribute the code you know itself so this compliance process of the survey that I've been speaking about this is like this is what like those organizations consider like 74% of the responsibility of an OSPO is that is that compliance process so another important thing for well whether you're whether you're creating a product or whether you've got an open source project and you're open you want to usually measure what you're doing measure the success of what you're doing and so if you an open source program office isn't any different if you've established one at some point someone's going to say you've got 30 people participating there's an org of like 30 people participating in that that's a lot of headcount that's a lot of salaries what value are they providing how is this why should we continue with this and so there's things like reducing the number of license violations you have maybe it's important to you to increase the number of contributors to open source projects that you're hosting there's a lot of different ways that you can measure success it just kind depends on what's important to your company maybe it's just measuring efficiency productivity how fast you deliver to market so to that end of the various things that you're measuring certainly for us and for many others that I've seen this is a snapshot of Microsoft's open source program office dashboard that you're going to want to measure kind of the health of your either the health of the projects that you're stewarding or maybe the health of the projects that you've become dependent on maybe you're dependent upon you know open SSL and all the maintainers have left and so you're you've got to figure something else out you should be paying attention to that so a lot of metrics to track both around compliance and contribution that we talked about and I also previously mentioned security there's all kinds of security vulnerabilities that are discovered on a daily basis understanding where you're vulnerable is important to do so having a dashboard that is important so as you go to establish an open source program office I told you guys that it's taken me about two years to get one established and that was that means that you've got to believe you got to keep the faith so sometimes I've seen more and more an increase of job postings that are hiring open source leaders someone to come in and establish this type of an office because open sources becomes so centric and important to businesses they're gonna have to have a bit of fire to them probably be passionate be a bit forward leaning and probably have some thick skin be ready for a lot of ignorance in a negative way the innuendo nuances of how open source ecosystems work and how their licenses work not necessarily isn't the day job of anyone outside of the program office so usually much education needs to happen so much patience needs to that you need to have when you go to establish one a lot of times these program offices start informally there's a working group of folks that get together that say hey we're using this they say any number of things maybe we're using this open source package we're having to maintain a private fork we need to get together and figure out how to contribute back or upstream that and so a lot of times there's sort of an evolution that happens over time of just people internally that are kind of passionate about either they're experiencing enough pain or they're passionate about trying to get this figured out within their company so it's about half the time little less than half the time that that starts within the engineering groups now I'd said it's going to require probably one or more people to champion the creation of an OSPO you gotta these are some of the top challenges you're going to face though I went through what I consider my five C's of strategy you gotta do your strategic planning there's probably a long form document in there that may need to start with an opinion piece about helping educate others as to what the heck open source is and why it's important and hopefully you can bring some statistics about how prevalently it's used within your company already or within the ecosystems that you're participating in then you go about the much more laborious and probably boring effort really defining company policy maybe based off that opinion paper and turning that probably into a couple of things maybe part of your employee handbook so as you go to hire a software engineer into your company they're signing the employee agreements that part of that is you generally hand them an employee handbook as well and that within that handbook you should probably have policy that says you should be proactive on my open source here's what you should do or shouldn't do another challenge here is executive support and that actually goes back to something I just said and that's education sort of ignorance if you consider today's this is a big generality of those that are in executive positions certainly in large organizations usually that's someone with a rare number of years of experience maybe that's 20 or 30 or 40 years of experience open source and its prevalence and its popularity I guess it's probably a safe guess to make that those executives probably didn't grow up in open source ecosystems that a little bit of this is unfamiliar turf to them that they're probably not coming to scale and sitting through talks like this they're not getting exposed to how open source works and so challenge number three is this executive support is just is helping them understand keep the faith the report that I was talking about says of the respondents there 70% of the respondents believe that are the ones that didn't have a program they believe that creating one would have a very positive impact on their company so I guess what I'm trying to say here is if you're often trying to create a program office keep the faith many people I think inherently see the value within there but the journey towards establishing one could be pretty hard fortunately there's some resources I mentioned this group earlier the to-do group and the Linux foundation and really the companies that comprise the to-do group and select individuals within these organizations that have contributed thought to much of the same things that I've discussed here today so there's certainly a great set of resources there with that feel free to reach out ask me questions off camera willing to be much more candid and share with you some of the stumbling blocks that I come across and what it how great it feels to have had established an office and how it is that at least for us we value that or which functions value that so don't be shy questions that I can take now let me ask this well I guess this is how many of you have are in an environment where there is an open source program office I know at least one of you that's true and then for the others of you I guess is software would you consider pretty centric to the value that your business delivers like either it's okay okay good and then of the software that you're using is there a fair bit of that okay very good yeah but sometimes it can be important well yeah I talked to a lot of consultants in that regard that's sort of the same position sometimes they end up engaging with customers and the customers say what we really need is this Microsoft product or this proprietary thing or an open source thing implemented but we need it changed we need you to develop a little bit around that for us to do that and as you go to see have that same ask from multiple customers you start to question whether or not you should be making each customer pay for you to repeat that implementation each time or maybe it's the first one that pays for it and then what to do with that I think for that first customer they say well wait a second I need to retain the rights to that the intellectual property and the copyrights and you can't go publish that if you're going to make me pay for it then as you go to you've got to make a choice I guess part of what I'm trying to say is relative to your environment you may end up facing a situation where you're asking yourself if open sourcing this type of integration so that you can use it a few times over make sense and one way to maybe assuage their concerns that company's concerns saying why am I paying for you to do this implementation where I know you're just going to go make money off of it a bunch of different times over they may feel better if it's an open source thing yeah and what people understand and know as well if they've been using a particular piece of software for a long time they understand how DNS runs within Microsoft in Windows Server versus Bind or whatever else so you could get big big challenge I've got a couple of employees in Germany actually in Berlin and yes Microsoft is prevalent there fair enough, very good any other questions before we close down yeah so the question was hey do I know of any other ways outside of the ways that we talked about to maybe measure the success of an open source program office are there others or maybe speak more to that so my mind immediately goes to a dashboard and the metrics of the dashboard some of these are intangible but others not I've certainly seen some of our engineers be choke on their own choke on the frustration of not being able to contribute something externally and for our company not having policy to do so and not being well understood or something that they can go self-service on so a metric being I probably don't want to have a dashboard on this but a metric being the count of how many engineers you've lost because you just didn't kind of get your act together around it's not that you need to that every company needs to say well yes any engineer can contribute whatever code they want to externally and that's okay rather to have at least a structure around their language around that so that they can engage in there's a process by which they can get those answers come to mind at least the end of our the the notice files that are published for the environment that I'm coming from some of those are updated like back in 2014 meaning we were selling, actively selling products and we'd be lots of open source but we haven't really told people what's in there for three years so anyway to not have published those things so the last updated distribution files the number of security vulnerabilities in the open source code that you're reliant on like hey that's an important thing to maybe track I think inherent to that is like if you're saying hey here's the number of here's the number of vulnerabilities that were within our code I think my assumption is that people are going to inherently understand oh there's a piece of value in there, there's risk in there that's being minimized and identified that oh maybe one of the things the metrics that you track is of all of the public facing code repositories that we have as a company how many of those have a contribution file that says here we either want to encourage contributions or we don't if maybe you're hosting on github okay how many of those for the environment that I'm in we've got like 16 github organizations and they have hundreds of repositories themselves there's lots of settings that you can have within github or any of your source code management systems some of those settings empower the users to open source things I was going to say willy-nilly just empower them to open source things maybe you wouldn't want to do that so some of those metrics can be things like hey what's the general health of our public facing in the environment that I'm in we've got hundreds of public facing repositories that have no license on them at all so inherently they're probably public domain because it's not defined so that's like an important healthy metric so I think some of these some of the metrics that I'm calling out are is like how exposed are you as a company in a variety of ways you I gotta imagine that people I've got to inherently understand that there's value in an open source program office tracking those things because you're just flying blind otherwise you have no idea how exposed you are although other things I think are if you're measuring like the even being able to understand of all of the this varies from environment to environment if you've got hundreds of thousands of repositories which ones are which ones have been updated which ones are moving the fastest are reviews being done across like this is I think some of these concerns are both centric to open source or public facing code and also just like interesting to understand hey how much how many lines of code do we even have and of that that's public facing what languages are those within or do we have how many how long the issue has been open on those projects do we even have a healthy community or so we said in the past part of our justification for opening up this piece of this project was because the engineers that were working on it said we strongly believe that there isn't intellectual property within this and that others would benefit and would bring goodwill to our company and it would make us happy and then we would also maybe hire some developers because they would get interested and etc maybe we'd be leaders in our space that things that you could use to measure the value of an open source project I'm sorry of an open source program office could be or is the health of those is how many contributors you have where they coming from but it both in good both in good ways and then also in bad ways like if you've got a bunch of unaddressed issues or if you've got or those are some of the metrics that come that literally you could have in an open in a dashboard some of them become a little less tangible I would think it was the first example that I'd given around engineers quitting or a little harder to attribute between them quitting and also some being attracted but our as an example and I try to use this occasionally internally I was giving a talk on Prometheus at KubeCon EU in Berlin 18 months ago a year right after I got on talking a gentleman came up by the name of Alex and said I've been thinking about I know a couple of people at the environment that I'm within and and and I'd really question whether or not it was right for me if the culture was right if I wanted to join now I see some of those individuals at the company here investing in the community sharing knowledge et cetera wow I'm a truck so anyway this individual is our director of platform for one of our more significant SaaS platforms and so my point is it's not always easy to say yeah they came in because they were but this is a good example of right after an open source talk where we're engaged in the community I didn't necessarily even open source any code we're just participating he came over we had a great talk he joined the company and has had a dramatic impact on the company how you can measure or like you know harder to explicitly quantify but very meaningful yeah the excellent question so I want to say that this screenshot of Microsoft open source program office dashboard is done in Node.js and that I want to say they had opened it up this was in advance of Microsoft having acquired github the the if you hit this slide deck which is public right now I think it's on the resources slide you will oh actually you can it's here it's where I was giving the disclaimer that hey this is how I did it maybe this isn't how you want to do it this quote comes from Jeff McAffer he's the director of open source program office at Microsoft these links right here this link right here there it is yeah very good I appreciate you guys coming today alright