And welcome, my name is Shannon Kemp and I'm the Executive Editor of Data Diversity. We'd like to thank you for joining the current installment of the Monthly Data Diversity Webinar Series Real World Data Governance with Bob Siner. Today, Bob will be discussing Big Data and BI Analytics Required Data Governance. Just a couple of points to get us started. Due to the large number of people that attend these sessions, he will be muted during the webinar. For questions, we'll be clicking them by the Q&A in the bottom right-hand corner of your screen, or if you like to tweet, we encourage you to share highlights of questions by Twitter using hashtag RWDG, Real World Data Governance. As always, we will send a follow-up email within two business days containing links to the slides, the recording of this session, and any additional information requested throughout the webinar. Now let me introduce to you our speaker for today, Bob Siner. Bob is the President and Principal of KIK Consulting and Educational Services and the Publisher of the Data Administration Newsletter, TDAN.com. Bob has been a recipient of the Dama Professional Award for a significant and demonstrable contribution to the data management industry. Bob specializes in non-invasive data governance, data stewardship, and metadata management solutions. And with that, I will give the floor to Bob to get today's webinar started. Hello and welcome. Hi, Shannon, I hope everything's going well with you. Thank you everybody for attending the webinar today. If you're new to the webinar series, welcome aboard if you've been a past attendee of the webinars, it's great to have you back. As Shannon mentioned, today's subject is Big Data and BI Analytics, Required Data Governance. And what I've found is that a lot of organizations, almost every organization, is talking about big data. They're also talking about business intelligence, data warehousing, and with an eye on being able to improve the way that they're able to analyze their data. So they're definitely focusing on BI analytics. And one of the things that I've found and I'm expecting that most of you have found this as well, that is that data governance is required in order for us to get the most out of our big data and to get the most out of our BI data as well. And so that's what we're gonna talk about today. We're not only gonna talk about the relationship between big data and data governance and the relationship between BI analytics and data governance, but we're also gonna talk about the relationship between the two of those big data and BI analytics. And hopefully give you some ideas as to where you can use the fact that you have these types of initiatives underway in order to jumpstart your data governance initiative. And the same thing in reverse. If you've got data governance, how you can then translate that into success in our big data initiatives and our BI analytics initiatives. So before I get started, I've got a couple of real quick slides that I wanna run through real quickly. The real world data governance series will continue through the end of the year at least. And we've got some really fascinating subjects coming up the next couple of months. Next month on August 20th, we'll be talking about data governance and data stewardship certification. That's always a hot topic. The month after that, we'll be talking about data modeling and data modeling is data governance.  We've got a very interesting special guest attending during that webinar. Dave Hay, data modeler extraordinaire, will be joining us. The month after that, we'll be talking about governing metadata. We'll be talking about agile, we'll be talking about data governance and the internet of things. Hopefully these subjects are of interest to you and we will have you back in another month or so for the upcoming webinars.  If you're not familiar with tdan.com, please go out, take a look at it, register so you can get the emails about the updated content. The publication has been around since 1997. However, this month, I just announced a partnership with Dataversity to not only redesign the site, but to relaunch the site, embracing things like social media, embracing things like RSS data feeds. I'm looking for authors, I'm looking for practitioners, people who are interested in sharing their success stories about their initiatives that focus on data management. It doesn't have to be a data governance initiative. Anything that you have that would be interesting, please pass it on and please be a regular visitor to the data administration newsletter. I think you'll find the redesign of the site was well worth it. There's all the content from the 17, 18 years that the publication has been around. So please take a look at that. And then real quickly, just to kind of follow up with the current events, I wanted to let you know about the book that I had published back in September called Non-Invasive Data Governance, also the relaunch of the kikconsulting.com website, and also put a real quick plug-in for the Dataversity events that I will be speaking at in the near future. One is coming up in September and that's the Data Governance Financial Services event. Data Governance Financial Services 2015 conference and I'll be speaking on considerations for starting or enhancing a financial data governance program. And then in November, I'll be speaking at the Enterprise Dataversity event where we will be talking about a strategic data framework based on data governance best practices. So please, I hope to see you there. Just some information about some upcoming events that you might be interested in. So this webinar is going to address the things that are listed on your screen here. We're gonna talk about existing governance applications or the ways that we can apply governance towards our BI efforts. We're also gonna talk about how we can apply data governance to our big data initiatives. We'll talk about what does the future of big data and BI data hold for us, the relationship between these three subjects, articulating the value of not only data governance to our big data and our BI analytics events, but also the reverse of that. How can we take advantage of the fact that our senior management is talking about big data. They're talking about leveraging as much information from the data warehouse and from our business intelligence initiatives as possible. We're gonna talk about how we can use to our advantage the fact that our executives are talking about those types of things to help to push our data governance program further into the industry, further into our organization. And then we'll talk about true intelligence that is derived from governed data. So we'll wrap up on that note and then take some questions at the end of the session. So as I do usually when I'm starting these webinars is I wanna share with you my definitions of data governance and data stewardship just to kind of get us all on the same page. There's lots of industry definitions out there, the definition that I've grabbed onto is that data governance is the execution and enforcement of authority over the management of data and data related assets. A lot of people look at that definition and say that it's worded really strongly. Well, the fact is your definition of data governance needs to be worded strongly. We need to be able to execute and enforce authority over the management of the data, whether that's from a data quality perspective or a data security or a compliance or a classification perspective and really at the end of the day what we're trying to do by putting governance in place is to execute and enforce authority over the data. Data stewardship, if you're familiar with the non-invasive approach to data governance, the one that I mentioned the book is about, the approach that I prefer to talk about focuses on formalizing accountability for the management of data. Where in a lot of organizations they think that data governance has to be about command and control, the non-invasive approach tells us that we're already governing data to a certain extent, we need to take advantage of that level of governance, that level of accountability that's already in place and formalize that so that we can be effective in governing data and stewarding data in our organization. Again, real quickly just to say with a definition of non-invasive data governance, it's the practice of applying that formal accountability through a non-invasive framework and I'm not here to talk to you about non-invasive data governance today, but I want you to know that that's the perspective that I take when I'm putting together these webinars is that we can do governance in a non-invasive way, we can apply it to things that we're already doing in our organization. Some of those things that we might be doing are big data related or our BI data related. So if we can take advantage of the existing levels of accountability in our organization and we can formalize those things, that is that kind of lays at the core of what non-invasive data governance is. So we're here to talk about big data and BI analytics data. So one of the things that I wanna do is I wanna just define some of those terms for us real quickly as well, where big data is really a broad term that's used for data that is so large or so complex that our typical way, our traditional way of processing the data in these applications are inadequate. And there's a lot of challenges that we address by that we come across as we're putting our big data initiatives in place. Some of those are listed on the screen for you here, analysis, capture, duration, search, sharing, all of those things that we need to do with this high volume, high velocity, high variety of data that we are all needing to address these days. And the fact is that in order to analyze the data, in order to capture and curate and search for the data, we must apply governance to all of these challenges that we have in our big data environment. And if we don't do that, then what we're gonna do is we're gonna have a lot of data coming from us at us from a lot of angles that is not being governed and therefore people don't thoroughly understand that data, they don't know where it came from, they don't trust the data. And so if we want to gain from the investment that we're making in big data, we wanna make sure that we're governing that big data. It does not mean that we need to put a separate program in place called big data governance, it means that we need to include the big data in the data that is being governed through our initiatives. So data governance must be applied to all of these challenges effectively in order for big data initiatives to demonstrate the return on investment, the proof of value that we are expecting from our big data initiatives. The same thing holds true for business intelligence. Business intelligence and BI and data warehousing, they're sets of techniques and tools for transforming the raw data that exists in our application into more meaningful and useful information for not only business analytical purposes, but for any purpose, any decision-making purposes that we have in the organization. So some of the common functions around BI include the reporting, that the process mining, the business performance management, the benchmarking, but also if you notice in some of those common functions that I've listed there, there's three of them that are very analytically based. Online analytical processing, OAP, there's predictive analytics, there's prescriptive analytics, and in order to get the most value out of our data in our business intelligence environment, in our data warehousing environment, we must apply governance to all of these functions as well. So it seems pretty sensible for us to understand that if we are focusing on big data and we're focusing on the investment of the data that's in our data warehouse, we wanna make certain that that data is of the highest possible value and the highest level of understanding that it can be for our organization. So that's really all I have as far as definitions is concerned, but we wanna make certain that if we're focusing on things like big data and business intelligence, that we wanna make sure that the data is of high value and that it's accurate and that it's put into the hands of the people that can use it. So I borrowed a couple paragraphs here from a recent article that talks about content management and the content really requires data governance. And so I'm just gonna go through these kind of quickly with you, but as companies generate increasingly more content and if we're talking about increasingly more content, we're really talking about big data. It's structured data, it's unstructured data, it's data that's coming at us from all angles. They're coming to recognize that the ability to be able to analyze that data is non-negotiable. Otherwise, why are we putting together big data environments or BI analytical environments if we're not going to use that data for positive purposes for our organization? So cleaning business insight from that data is really the end target for most organizations that are applying dollars towards these types of initiatives. So cleaning that business insight from the big data is still in the early adopters in the well-hyped stage. There's a lot of people talking about it, but there's less organizations that are getting the true value out of that data, perhaps because the level of governance associated with that data is not where it needs to be. Companies are beginning to see these results emerge in terms of smoothed workflows, improved search, better compliance, which really means better governance associated with the data that is being fed into our big data and to our BI analytics initiatives. So big data analytics is a business practice that analyzes and derives insight from data. That's what we're trying to do with the resources that we're applying to building these types of data resources for our organization. Big data analytics helps company manage their data lifecycle. With business metrics in place, companies can identify which of that data in our BI environments, in our big data environments is most value to our organization. And if we can identify which of that data is most valuable to the organization, then accordingly we can adjust our investments in the storage and the future analytical needs for our organization. So there's a very strong relationship between big data and BI analytics. There's certainly a big relationship between big data and governance and BI analytics and data governance. Let's talk a little bit more about that here. So when we talk about big data, and I love this slide just because the idea of it, first of all, it's the middle of the summertime, it's hot as heck out there and we've got a skier on the slide. But how much data is big data? So even dating back to when we were talking about data in terms of bits and bytes, up to kilobytes and megabytes and gigabytes, if you look at that progression of how the amount of data has grown in organizations, you'll see that where that red line sticks out, this is the most recent data that I could find, but as of 2012, there was 2.7 zettabytes of data that existed in the digital universe. 2.7 zettabytes, it's not even a term that's used by a lot of organizations. Well, the truth is that by the year 2020, they're predicting that we're gonna have 35 zettabytes of data being created every year. So we've got lots of data. We're putting a lot of money into managing that data and to taking advantage of that data for the best business purposes that we can. We need to make certain that that data is governed, that that data is well-defined and well-understood. So we're gonna focus on whatever you mean by big data in your organization, this is some type of scale to see where we're going. We're going to yoda bytes and xenobytes and selenobytes and so on and so forth. Some of these terms, I don't know if we'll be discussing them anytime soon, but the fact is that big data is just getting bigger and bigger daily, certainly annually, and we wanna make certain that we're putting governance around that data. If we are going to capture and collect the most important of the big data that's available to us, we wanna make sure that that data is of high quality, that it's accessible to people, that it's understood that we have formal accountability applied to make certain that that data is the best that it can be. So you've heard the expression from the US Army, be all you can be. Well, the fact is that we need our big data and we need our BI data to be all that it can be, and that starts with formalizing accountability around the management of that data. Couple of things that we imply by the name of this webinar, you know, the big data and BI analytics require data governance is, well, why does big data require data governance? Well, a lot of us in the data management industry, we use the term that data is an asset all the time, and some people are clear as to what we mean by that and some people are not as clear as to what we mean by that. But when we talk about big data and this volume of data that is now part of what we see daily in our organization, we need to recognize that not only is the data in our applications in our data warehouse, is that an asset, but any data that we can use as an organization is an asset to the organization. So that big data that is most important to us and most important for usage in the organization is certainly an asset. And we've got to look at big data as an asset in our organization, and if it's an asset of the organization, just like any other asset in the organization, we need to govern that data. We need to execute and enforce authority over the management data. We need to formalize accountability.  We need to make certain that this big data follows the rules that we've defined for us. We need to be consistent in how we apply big data to applications. And I'm gonna talk a little bit today in the webinar about how important metadata is to the big data and to the BI analytics environment and how important metadata is to our data governance environment as well. So another one of the reasons why big data requires governance is because in order to understand that data, in order to improve the value and the understanding of that data, we need metadata to support us, to improve both business and technical understanding of the data and the data-related assets. So we need to manage the definition of the big data. We need to manage the production of that big data, where it came from. We've got to manage how that big data is being used across the organization. So there's a lot of reasons, and I'm gonna address some additional reasons for why big data requires data governance and some of the additional slides that I'll be covering, but just real quickly, these are some of the primary reasons why we need to, why big data requires governance within our organization. So I'll talk about it from the BI analytics environment. Same thing holds true. If we're gonna put time and effort and resources into building out our business intelligence environment and that the data is going to drive the analytics, that data needs to be well-defined. That data needs to be understood. That data needs to be made available to people. And one of the ways to be able to do that is to, again, formalize accountability over the definition, production and usage of data in our BI environment. Again, it needs to follow the rules. We need to be consistent in how we apply analytics to the data in our organization. And again, the metadata has always been talked about as being a backbone of our business intelligence initiatives. We need metadata to improve the value and the understanding of our BI analytical data. We need to manage the BI data definition, production and usage of that data as well. In a recent webinar and also in a near future webinar, we're gonna be talking about metadata and its relationship to data governance in the more detail. But for most of us that have been active in this industry for a period of time, we understand the importance of having metadata to support our data warehouse. So we understand that. We can certainly translate that into why that metadata is necessary for our BI and our analytical environments and why it's important to our big data environments as well. So there's a couple of quick slides here just to share with you why data governance is important to these types of initiatives. So let's talk a little bit about the metadata angle. There's just for a couple of minutes here. And let's talk about where we've come from and where we're going with metadata. Well, in the past, metadata was always limited to things like the data dictionary, the business glossary, the names, the definitions, the labels for the data on the screens that people are seeing, on the reports that people are getting. But metadata is evolving just like the data, the big data and the analytical environments are evolving. And now the metadata encompasses things like not only the data, but the people that have accountability and responsibility for that data. The processes that are using that data and the processes that we follow in order to even get that data into a condition that's gonna be most usable by our organization. And now metadata is also focusing on things like big data as well. Where is the data coming from? What data do we have available to us? What is the definition of that data? Where do we get it from? How can it be used? How is it being applied in our organization? Metadata really lies at the core of pretty much everything that we're doing in the data management industry. So it really shouldn't be a surprise to us that metadata lies at the core of big data and BI analytical data as well. And the truth is, and I've done webinars on this in the past, I look at the relationship between data governance and metadata as being a two-way street. So when you're doing data governance, some of the things that are just gonna naturally fall out of and result from your data governance initiatives is metadata, metadata about people and the processes associated to the data. So that's the one direction of the two-way street is that we are going to have metadata that flushes out of any of the BI initiatives, the big data initiatives, and certainly out of the data governance initiatives. The other side of the street is that this metadata that we're now counting on to improve the value that we get from our big data and to improve the value that we get from our BI analytical environment, we need to govern that metadata as well. And again, just like I mentioned earlier where we don't need to have big data governance initiatives or metadata governance initiatives or data warehousing governance initiatives, really what we need is we need to define and roll out programs around data governance that are going to enable us to get better use out of our big data, out of our small data, out of our data warehouse data, out of our content, out of our records, out of everything. So metadata really lies at the core of what we're doing around the governance of data, whether it's big data or BI analytical data. So metadata continues to evolve. Initially, it was looked at as being primarily a data management asset, and then it became a business intelligence and a data warehousing asset, and then it became a data governance asset and a privacy and a confidentiality asset. If you look up in the news and you see the word metadata, most often it's associated with privacy and confidentiality and those types of things. So metadata has evolved not only from being a data management asset to a security and a privacy and a confidentiality asset, but now it has become an asset to all of our big data and analytical platforms that we're putting in place in our organization. So one takeaway I'd like to see you take from this webinar is that we understand how metadata really lies at the core of what we're doing around governing the big data and the analytical data that we have in our organization. So the other thing that lies at the core of the big data in the BI analytics is the data governance itself. We need to make certain that we have governance programs in place that will encompass not only the structured data that we have, the bits and bytes and the data and the tables and the databases that we have, but we also need to be able to apply governance to unstructured data. Years ago I was told not to even use the term unstructured data because somebody had really expressed to me that all data is structured to some degree. Okay, well what I mean by unstructured data is data that exists in documents, in audio, in video, in those things that are not your traditional databases. And a lot of the big data that is coming at us these days is unstructured data. So not only do we need to have metadata to address our structured data, but we need to have metadata and we need to have governance in place for unstructured data. Now other terms for unstructured data may include things like content management or records management or logs management. There is certainly a lot of social media data that is coming at our organizations as we start to embrace or we've already embraced social media for our organizations. There is a lot of data coming at us. And if we expect to be able to analyze that data and make use of that data, and we certainly need to govern how that data looks in our organization, govern how the quality of the definition, production and usage of that data. And that data includes not only big data, but if there's a such a thing as big data then I would be thinking that at some point in time we're also gonna have something that we call small data. And small data may be finely tuned to data sets that sit on the desks of managers. If we're gonna govern the data around big data, we're gonna govern data around small data, around data warehouse data, master data, metadata, all of this data needs to be governed. So if we understand that data governance and metadata kind of lie at the core of what we're doing around data management, it's for all data in the organization. Big data, small data, analytical data are certainly included. And so what are we talking about when we're talking about governing perspectives or governance perspectives for big data and analytical data? And so typically when we talk about governance for big data and analytical data, there's five perspectives that I wanna share with you around what it really means to govern the data in our big data and our analytical data environment. And the first one is the accountability perspective. From an accountability perspective, who has responsibility for making certain that the data is defined the way that it needs to be defined so that we can get the most value out of that data? Who has the responsibility for making certain that that data is being produced appropriately or that it's being used appropriately? So we're gonna wanna look at governing data from an accountability perspective. We're gonna wanna look at it from a process perspective. How are we gonna use data in the processes? What are the processes themselves for defining, producing, and using the data? We wanna talk about accountability when it comes to the rules perspective. From the inventory of data, what data do we have in our organization from that perspective? And then lastly from the decision-making perspective. So what I wanna do is I wanna walk through each of these relatively quickly here and talk about what is the governance perspective at the accountability perspective for data in our big data environment and then follow and do the same for the process perspective, rules perspective, and so on and so forth. So for the accountability perspective, and I've talked a little bit about this already in the webinar, is hope that if we're gonna govern data, then there's really three pieces or three processes that are associated with the data that need to be governed. And that is the governance around the definition of the data. So making certain that data that is defined to be used in the organization is not given what I've called in the past cheeseburger definitions. So what's a cheeseburger definition? A cheeseburger definition or the definition of a cheeseburger is that it's a burger with cheese or a client account number is the account number for a client where the definitions of the data don't really tell us anything more than the name of the data field itself. So if we wanna make certain that we're getting the most value out of our big data and our BI data, then we wanna make sure that we have valid definitions, definitions that have been validated, definitions that have been certified in the organization, so that anybody that is going to gain access to that data, whether it's in our BI environment, our big data environment has a good understanding of what that data really means to the organization. We also wanna have accountability for the production of that data. Where did the data come from? What did we do to that data in order to get it into the format that you see in your BI environment or your big data environment? We also then, it makes sense to govern the usage of that data. So if there are rules that are associated with how that data can and can't be used, then we wanna make certain that those rules are well-defined and that they're shared with people. And in order to get those rules well-defined for how you can use and how you can't use data, we need to have somebody in the organization who's accountable for doing that. And the fact is that if you're building a data warehouse, if you're building a BI or a big data environment, somebody typically has the responsibility for making sure that we're putting definition to the data that is included in that environment. And if we make certain that we have accountability for the definition of production and usage of the data in these environments, we're taking great steps towards having a governed data environment around these different types of initiatives. Certainly we need to have accountability for protecting the data. Now who can see the data? Who can't see the data? How can data be used? And there's something called data classification that a lot of organizations are talking about where data can be highly confidential or confidential or sensitive or public data. And all of those different classifications of data require governance. Not only do they require governance, but they require that we have classification handling rules. So if the data is identified as being highly confidential, what does that mean to the organization? How do we handle that data in relationship to data that's not classified? And if certain people can see it, certain people can have that data shared with them, we need to make certain that we have people in our organization that have accountability for protecting the data. And that doesn't just mean in our applications, in our data warehouse, it means in our BI analytical environment, it means in our big data environment as well. We need accountability for improving the understanding. And I talked about that a little bit with the definition is that we need to make certain that we get the appropriate people in our organization involved in putting definition to the data so that people that are using the data not only are they using it appropriately, but they're using it in such a way that they're really leveraging what that data means to the organization. And also from a retention and from elimination perspective, we need to make certain that we have accountability for making certain that we're retaining the data. And if it's big data that it's just more of it, we need to make sure that we're following the retention rules and that we're eliminating data in the appropriate manner as well. So there's accountability. The accountability perspective is very important when it comes to why these different types of applications require governance in our organization. And in fact, most governance organizations, most governance initiatives within organizations address these things. The definition, the production and the usage of data, the protection of the data, the understanding of the data. So the accountability perspective is key when it comes to being successful with our governance initiatives. Let's also talk about it from the process perspective where, and I've done webinars on things that I call bill of rights, getting the right people involved at the right time, using the right data to make the right decision. From a process perspective, that's what data governance is all about. We need to be able to identify who the owners of the data are. And I know that I typically shy away from using the term owner, but it seems to be used a lot in organizations where they've identified people in the organization that are owners of the data. So if there's owners of the data, then there's also gonna be owners of the process associated with the data. And we need to know who those people are as part of our governing perspectives for the big data and the analytical data as well. We wanna make sure, as I said before with the bill of rights, we get the right people involved at the right time. We wanna make certain that the appropriate people are participating in the efforts to improve the definition, the production and the usage of data in our organization. And from a process perspective, we're really looking at three things. We're looking at input and output and throughput. And what is it that is input and output and throughput to these processes? It's data. And that includes the big data and BI analytics data. They're all being used in different processes. If we understand what the data looks like coming into the process, what the data looks like as it leaves the process and what's done to that data during the process, we will have a better chance of successfully governing the data associated with these initiatives. From a rules perspective, well, there's a lot of rules, as we know, are associated with the data. There's business rules, which are rules that we define internally for how the fake data is defined, produced and used. There's compliance rules that are being handed to us from the outside organizations or from the federal government or from our local government or in our industry. We need to make certain that these compliance rules are well documented. The protection rules that I talked about, they need to be well documented. And any standards that we have for data need to also be followed when it comes to data in our data warehouse and data in our big data environment. So there's a rules perspective associated with governing the data in the BI and the big data environments.  I'm not sharing it in this presentation, but I'm sure that it will come up in a future webinar. But we want to know what data we have and where that data came from. We want to know the quantity of data. We want to know the quality of the data so that we can direct people to the most appropriate data in our environment. We want to know who owns the data and we want to know how the data is used. So from an inventory perspective, we need to know all these things about the data. And when we're talking about big data, data that comes in various formats from a variety of places, we need to make certain that we're looking at it from a quantitative perspective. Can we handle that volume of data from a qualitative perspective? Is that data suited for purpose within our organization? We want to make certain that from an inventory perspective, we're also looking at how are we going to govern data in our big data and our analytical data environments. And then lastly, from a decision-making perspective, we want to know that when we're talking about data in these environments, who has the accountability for being able to make the decisions? How are they going to make decisions? What are they going to make the decisions? And who's going to be impacted by those decisions? And then lastly, how are we tracking those decisions that we're making? Are they good decisions? Are they bad decisions? Requires governance. It requires that somebody has the formal accountability for executing and enforcing authority over that data, but also has the formal accountability for making sure that we track not only the data itself, but the decisions that we're making on the data and how they're adding value to our organization. So we want to look at the future. One of the items on the agenda was to talk about the future of big data and BI data, but in order to look at the future of these things, let's take a real quick look at the past, where our business intelligence strategies have evolved really from hierarchical databases to relational databases to dimensional to enterprise data warehousing, data marks, cues, distributed databases, data lakes, it may be an interesting topic for a future webinar, but data lakes, for those of you that may not be familiar with that term, they're an object-based storage repository that holds data in its native format. So if we've got big data, we're gonna throw all of our big data into this data link, and then as we pick and choose the data that's gonna add most value to our organization, we pull it out of our data lakes and we move it into our BI environments and into our big data environments. So we're not only concerned with the bits and bytes of data in tables and columns, we're talking about unstructured data, we're talking about big data as well, and we need to make certain that as the whole idea of business intelligence evolves, that we're evolving with it from a big data and from an analytical perspective. All right, so now let's look to the future. It's kind of funny, I read this somewhere some time ago that in business intelligence strategies, we're kind of moving from the data mining to the big data analytics perspectives around our big data sources. So we're in data mining, we used to be looking for that needle in the haystack. We'd be looking for the relationships between the data so that we can get that golden nugget of information out of our databases. Well now in the big data environments, we're talking about not only looking at the haystack and trying to find that needle, but we're looking at the entire haystack. And in order to make sure that all of that data in that haystack is of high value or high quality to people, then we wanna make sure that we're governing the definition production and usage of that data to add value to the individuals and the groups in the organization that are going to use that data. Can't really do a webinar on big data without talking about the three Ds of big data and I've seen them in different forms, but I usually refer to them as volume, velocity, and variety. We've already talked about the importance of the volume of the data. We know that we've got more data available to us coming from more sources than we've ever had before. It's coming at us faster than it's ever come at us before. And it's certainly in more varieties of formats than it's ever come before. One of the things that I suggest for organizations is you take a look at how are you handling the volume, the velocity, the variety of data now. Even before you enter into the big data environment, and you saw the chart on an earlier slide in the slide deck, how the volume of data is growing and growing in the organization. Well, if we're not able to govern the volume, the velocity, and the variety of the regular data that we have in our organization, we need to certainly put something in place to govern that and also be able to handle the data that's coming at us at the higher volume, at the higher velocity, and of different shapes and forms of data coming at us. So we wanna make certain that we understand what we're doing well with the data that we already have, but then as we embrace big data and BI analytics, we wanna make certain that we're governing that data consistently across the organization. So it's not just any data in the organization, it's the data that's going to be most useful to us, the data that we're investing most of our resources in. We wanna make certain that we have governance in place around the volume, around the velocity, around the variety of sources of data that we have coming at us. Really the question becomes, well, what are we gonna do with all this data? What are we gonna do with all this big data and this BI analytical data that we now know is well-defined and well-produced and we've documented the rules about how it needs to be used or how it can be used? Well, the first thing we need to do is we need to make sense of that data. We need to understand and trust the data. We need to protect the data. We need to know where the data comes from. We need to be able to govern the access to that data and also be able to govern how analysis is taking place on the data. So we need to ask all of these questions when we start trying to determine what we're going to do with all this data that now exists in our big data environments and our BI analytical environments. And the truth is, I can just get to the next slide here, that in order to achieve these goals of getting the return on investment from these investments that we're making in big data and BI data, is that we need to be able to do all these things, but we really need to have governed data. We need to have governed process and we really need to govern the use of tools in our environment. All these things need to be governed. Not only the data itself, which we've talked about a little bit here, but the process associated with defining, producing and using that data. And then the technologies that we're using to leverage and get the most out of our big data in our analytical environments. And you know, it's funny. So some of the organizations have something that they call information governance as compared to data governance. And oftentimes the information governance really encompasses all three of these areas. They encompass data governance, process governance and technology governance. And I can tell you that there's at least a handful of organizations that I've worked with that have called their initiatives information governance. And why information governance rather than data governance? Well, because we're not only really looking at the data, but we're looking at the process and we're looking at the associated technologies as well. So the next item on the agenda, things to talk about really was the value of governance to these initiatives. But what I really want to do here is I want to kind of turn that around a little bit. So we've already really talked about the value that data governance will bring to our big data and the value that the data governance will bring to our BI analytical data. Let's reverse that here for a moment. Let's talk about the value that big data and BI analytics are gonna bring to our data governance environments. So let's talk about that here for a minute. So again, now what we're gonna look at is what's the value of big data to our data governance initiative? What's the value of the BI analytics to our data governance initiatives? Well, the fact is that we can use the fact that our management is talking about big data and BI analytics to push our need for data governance in the organization. If they've already justified the investment in big data and justify the investment in analytics, then we need to make certain that we are gonna get the most return on investment out of these resources that we're building in our organization. We wanna make sure that we're maximizing the use of the resources across these different initiatives. We wanna make certain that we're providing trustable data in our analytical environment and in our big data environment. We wanna make certain that we're making better decisions from that data as well. So if we can look at the fact that our management teams are talking big data and talking analytics, then in order to truly get the most value out of their investment in these things, we need to have high quality data. We need to have understood data. We need to know all those things that we mentioned about, we talked about in the first 45 minutes of this webinar. So instead of again selling the value of data governance to these things, let's look at it the other way around. Let's look at how do we sell the value of these things to data governance? And it's a very different approach for selling why we need governance in our organization. But if they're making the investment in these things because these are the buzzwords of the industry and these are the things that everybody needs to address from the industry analyst perspective, from the vendors perspective, then we wanna make certain that the data that makes it into these environments is well-governed and most useful to the organization. So I'm gonna make two bold statements here. And if you don't agree with me through the chat or separate from the webinar, please reach out to me and tell me that you disagree. But these two bold statements, I'm hoping that most of you will agree with me, that organizations that govern their data effectively get more value from their big data and their analytics and their BI analytics efforts. So if you believe that to be true, I'd love to hear from you things that stand behind that bold statement. But if we can say that organizations that govern their data get more value out of their big data and their analytical environment, that's a pretty bold statement. And then we've gotta back it up with the actions that we put into place around data and around data governance. So my second really bold statement that I'm gonna share with you is that organizations that don't govern their data effectively do one of two things typically. Typically they don't even embark on these types of data activities or they get less value from their data-driven activities. So the first one is organizations that do govern their data effectively are getting the most value out of these initiatives than the organizations that aren't governing their data. Well, they may not even address these initiatives or they're certainly getting less value out of their big data resources and their BI analytical resources. So the last thing that I wanna talk to you about before we take a few questions is driving true intelligence from governed data. And so typically to derive intelligence from the data we need to have quality data. We need to have a high level of understanding around the data. We need to protect the data. We need to make the data available and we need to have formalized accountability. Again, that kind of lies at the core of what I talk about with non-invasive data governance is that if we look at the fact that there's already people in the organization that have accountability for the data and really what we need to start with is formalizing that accountability then we can make certain that we have the appropriate people involved in the appropriate times with the data to make certain that the data is the way that it needs to to derive that value from our big data resources from our BI resources. So let's walk through each of these real quickly from a quality perspective. We're talking about data standards and making certain that we have standards. It's very difficult to improve the quality of the data in the organization if we don't have a standard as to what this data needs to look like, how this data is going to be used. So by starting with data, if we're going to focus on data quality, we want to make certain that we have standards defined for the data in the organization and that includes data in our BI environment, data in our big data environment as well. We want to make certain that any processes that we put in place in our organization that they're being followed. You know, a lot of organizations do business process re-engineering and defined processes but then there's no level of accountability for making certain that we're engaging the appropriate people with the appropriate steps in the process. We want to make certain that we have escalation paths to resolve issues around data that where we're just coming to disagreement. So rather than bang heads and go our merry way, we need to have a formal way of being able to escalate decisions to the appropriate level of the organization so that again, we're looking at governing data as an asset and governing specifically data in our BI environment and our big data environments as an asset as well. We want to make sure that we have decision rights, we know who's responsible for making decisions, we want to make certain that we're certifying the data into these resources and making sure that it follows the standards that we've defined for ourselves. And then the other term that I would add to that is validation, we want to make sure that we're validating the data that makes it into these data resources. So we can derive intelligence from governed data, we certainly need to improve the quality of the data in order to do that. We need to improve the understanding whether or not that understanding is through a business glossary and a vocabulary or through a dictionary. That's making sure that we've captured the information about the mapping of the data that we've managed and that we make available information about the validation of the data, what steps have we gone through with this data to assure the people that are using the data in these environments that the data is what it needs to be in order for them to make decisions. So I call that the data validation piece. And then there's the understanding, there's the education piece of it as well. We need to make certain that people that have access to the BI analytical environments and people that have access to our big data environment that we give them some education around what that data is, how it's defined, where it came from, all the things that we do with traditional BI resources, we need to do the same thing with our big data in our BI analytical data environments. From a protection perspective, and we talked about this a little bit earlier, that we have classification rules, security rules, auditability rules. We need to make sure that all of these things are well-defined and that we have a way to be able to keep and hold people formally accountable for following the classification, the security rules, for making certain that we can work with our auditors to make certain that the data is what it needs to be and that it's trustable, and that whether it's internal or external auditors, that they have the metadata that is required to make that data truly as valuable as it can be for the organization. And that involves process and education, it involves enforcement and compliance as well, but protection is one of the things that we need to put in place to derive intelligence from our governed data. Availability, we need to make certain that the appropriate data is available to the appropriate people, that we've taken a look at what the best way is to store the big data, especially when it comes to big data, again, coming at us from many different directions and many different formats, we wanna make certain that we govern the storage of that data. We govern the timeliness of the data to make certain that the most appropriate data is available when it needs to be for the end users of these environments. Data is complete. Kinda goes back to the quality perspective as well, is we wanna make certain that our big data is complete, that our BI data is complete, and that it can be used to answer the questions that people have of the data. From an accountability perspective, we talked about that a little bit. We need to formalize accountability rather than hand it to people with new responsibilities. We need to have stewards of the data, whether it's big data or small data, or BI data or application data. We need to make certain that we have stewardship of the data. We need to improve our communications around the data, including the metadata that we've collected about the data. We need to improve how we enforce, execute and enforce authority over the management of that data. So the main points that we've covered in this webinar is that we talked about governance applications and how we can apply governance to our BI environments and our big data environments. We talked a little bit about the future of big data and the future of BI data. And I think we all know that it's growing and growing in importance in our organization. We talked about the relationship between the big data, BI, and governance. We talked about articulating value, not only in the value that governance will bring to our big data and our BI analytics efforts, but also that the reverse is true, that we use the fact that we're doing these things to really push the need for governance into our organization. And we talked a little bit about true intelligence is derived from the governed data in our environment. And with that, I'm getting ready to turn this back over to Shannon here for any questions that we have. But just a real quick reminder, in August, the third Thursday of the month falls on the 20th. We're gonna talk about data governance and data stewardship certification. That's a hot topic for a lot of organizations, so please welcome you back for that in future webinars as well. And the last thing that I'd like to share with you really is please, if you get a moment, go out and visit tdan.com, register for the emails that go out, announcing the new content. We'd love to have your participation in the tdan.com environment as well as the real world data governance environment. All of those, I'm so proud to say that I am glad to work with Dataversity on both of those things. It's really important that we recognize that Dataversity provides all of these resources, and if it wasn't for them, the series wouldn't be here and the publication may not be where it is today. So with that, I'd like to turn it back to Shannon. Do we have any questions, Shannon? We certainly do, yes. And of course, the most popular question is people inquiring about the slides. Just a reminder, I'll be sending out follow-up email by end of day Monday for this particular webinar with links to the slides, links to the recording, and anything else requested throughout the webinar. And if we don't have enough time for all the questions, keep the questions coming. One of the great things about this webinar series, Bob will answer the questions in written format that we don't have time to get to, and we'll make sure and get those out in the follow-up email as well. So Bob, the first question coming at you is, how realistic is it that rules are deeply flushed out? In my experience, I only know of two companies that do a good job of defining data. Most of the others at some point tried, but things are out of date or stale. Well, that's a great question. I appreciate that question. The fact is that in order to be able to get the value out of the data, we need to have people in the organization that have formal accountability for making certain that we capture that information. So it's really important. I mean, there are organizations that govern their data better than other organizations, but what it truly comes down to is can we define people as being formally accountable for not only the definition, but the protection for putting good definition? I would venture to guess there's a lot more than two organizations that do this well. What it really requires is, and to be successful with this, is that we need to have people in the organization that are formally accountable for making certain that they happen. Love it. Okay, so we've got another question coming in. Do you attempt governance of the data lakes or only data that was pulled out for a particular business reason? Well, by definition, really, a data lake is data in its native format. So if it's data that's coming from streams that are ungoverned, then the data in the data lake is not gonna be governed. So I would say, and again, I don't have a lot of experience with data lakes, but data in the data lakes is not governed. Data that is pulled out of the data lakes for a useful purpose, typically that's when governance is applied. If I'm off base around that, I'd love to hear about it, but that's the way that I view the data lakes versus the use of BI and big data environments. And that's the way I've heard it and understand it as well. Actually, all the questions that we have coming in, there may be a couple more if you guys wanna type in a couple additional questions, but Bob, thank you so much for another fantastic presentation. This is great as always, and I look forward to next month's presentation. As you mentioned, a very hot topic for a lot of people, the certification of data stewards. Anything else you wanna add before we end the session? No, I appreciate it. I think the next month's topic is gonna be great as well. Big data is a keyword that is used by a lot of organizations, and some organizations use it without having a firm definition of what that means. We're looking at how the volume of data is growing and the usage of that data for analytics, it certainly requires governance, which is really what this webinar was all about. Perfect, thanks so much, Bob, and thank you everyone for attending today and taking the time. Again, I will get that follow-up email out within two business days. Again, with links to the recording, the slides and additional information requested throughout the webinar. And I hope everyone has a great day. Thanks, Bob. Thanks, Shannon. Take care, everybody.


You guys, do you have a lab up here or do you need to give them? Okay. Okay. Okay, you got one of those beer? Yeah. Okay, I'll flash. All right. So you good to go? Do you have any audio or video with your presentation? Nope. Okay. All right. Lab is off for now. I'll turn it back on. You want to start talking? Level seemed okay and I'm loud anyway. With you ready, we're supposed to start at 10.40, but I'm going to probably give people another five minutes, just let everyone filter and think, you know, you can still hear someone talking. I don't know if they're still going or not. We will talk about this one super long anyway. Do you have any questions here in the course of the talk or later? Please signal your desire to have the question and I will take a minute or two so you can be recorded. So people might still filter in because it sounds like there's still chatter out there, but let's get started. So that's me in all of the places. I'm an infrastructure engineer at a company called Ride Cell and I'm here today to talk about open source sustainability. Before we talk about sustainability, let's lay out a little bit of scope. A lot of people have very specific opinions about what constitutes open source. I'm going to take a much more general stance away from the specifics of what precisely makes something open. I'm also not going to wade into open source versus free software at all. So that other way, let's look at a few types of projects of what I consider open source. The majority of projects buy the numbers. One person writes some code to do a thing and they put it up on GitHub because why not? This is how I got my start in open source writing plugins for a bug tracker called track. But the more visible projects are the big ones. Is a project so vast and omnipresent? It's easy to just think of them as part of the fabric of the universe. Some of those very big projects eventually actually have to build legal entities around them, usually projects that are partnerships between multiple large corporations. But a newer venue for open source projects has been VC backed startups. So the company acts as a steward for some form of open source project and they sell stuff on the side. We'll get to the selling later. What I call mirage projects feel like an offshoot of that so big it's part of the fabric of the universe idea, except these often have frighteningly few core maintainers. We all saw this during Heartbleed where the entire world sort of simultaneously realized that there were a single digit number of volunteers maintaining this enormously important core infrastructure. Some older projects, especially big Linux packages, tend to feel a little bit stuffy compared to the newer wave of GitHub generation projects. But these projects have a very important history. They've been designed to stick around for the long term as individual contributors come and go. Some open source projects can only be described as crazy ridiculous moonshots, usually led by a single person and sort of organized around, we just have to get this done. And in my very wide view of open source, there's a lot of things that aren't software too. Some of the legal specifics differ on how you license and maintain these, but the overall sort of contributor dynamics are basically the same. It's clearly not a complete list of every type of open source project out there, but it gives you a taste of the kind of breadth that I'm going to be talking about. So that's two words from the title, what about the third? Literally speaking, sustainability means taking a project that exists today and ensuring it will exist tomorrow, and the tomorrow after that, and so on. Money and time are important, and we'll get to talking about those in a few minutes. The fundamental limiting factor on most open source projects is the rate of burnout of contributors and maintainers. Most projects fail not for technical reasons, but because people don't want to work on them anymore. The most common symptoms of burnout in open source are these three things, a cycle of feeling guilty about letting down your users, leading to dread about working on the project, and finally bitterness towards the project itself for sometimes the users of the project. Open source projects have truly taken over the world. In almost every facet of technology, communal resources have pushed us further and faster, and if we want this pace of progress to continue, we need these communal and shared resources. But with this cross-pollination comes interdependence. While open licensing helps to ensure the possibility of someone picking up a fallow project, it doesn't help us if there's no maintainers there to pick up the fallen banner. And if I can digress for a few moments about why I think open source itself is important, I think we can build things together much better than any one team or organization. A lot of problems on software, there's so little value to the market that it's hard to define how the market would even come up with a solution for this. But collectively, all of us together can and we're all better off for it. So open source rules the world and I think that's great, but clearly I also think there's problems with the system or I wouldn't be giving this talk. How do we end up with a worldwide ecosystem of software projects built on the backs of a relatively tiny number of volunteers? If we want a globally simplistic answer, infrastructure investment has been lagging across the board. Putting time and money in things like bridges and dams doesn't show returns immediately. If a return can even be quantified, if your focus is only on the earnings report for the next quarter, things are gonna slip through the cracks. But a deeper structural answer, at least in my view, is in how we make things. I can only put my brain towards building one thing at a time, but while I'm doing that, I'm using probably hundreds of other projects and tools. Even in the best scenario, there's always going to be more users than contributors and that imbalance has spiraled far out of control for most open source projects. Even a simple project like a Ruby on Rails web app. I just sort of made this up off the top of my head. The Rails app at the top of the list sits on a mountain of projects and libraries. As we build better and better abstractions, which is great and we should keep doing that, it becomes easier to forget the shoulders of the giants on which we stand. But why all this interest and sustainability lately? Way back when, open source was just some crazy college students, several of them actually from right around here. Basically everyone that consumed open source also worked on it. So maybe I worked on GCC and you worked on Bash, but we all used the same sort of suite of tools and we all benefited from each other's work. Over time, this balance eroded.  As the third ish wave of open source developers are reaching middle age and deciding that maybe other pursuits are more important, often there's no longer fresh faces willing to take up the banner. And so here we are. It's hard to pick a patient zero, but it's probably either the lamp stack or Ruby on Rails that got the ball rolling on corporate use of open source. VC funded startups, usually in the early 90s, realized they could switch to friars and beer tools and save money. And the VCs didn't know about this, so the checks stayed the same size. And when being cynical, these startups used that money to just add their own profits. If I'm being nice, they used that additional money to double down and figure out how this whole web thing was gonna work, because that was still pretty new at the time. But all good bubbles must come to an end. And by now startups and big companies alike know that zero costs, software stacks exist, and many are of extremely high quality. Budgets have all adapted to this. So companies often have no choice, especially small new startups. They can't afford anything else. At the end of the day, open source and capitalism are at loggerheads. Capitalism requires, or at least strongly encourages, privatization, and open source is the exact opposite of that. As long as the bulk of companies continue to draw more value from the commas they put back in, we're going to have a sustainability problem. Much of the interaction between capitalism and open source is related to intellectual property, but I did a whole talk on that last year. So that URL will be up again later if you wanna check that talk out. So that's the general answer to why things went wrong. What about some specific projects that were ultimately unsustainable in one way or another? Two that are very close to my heart, React and RethinkDB. These were both originally VC-backed startups, but they were open source projects from the beginning. The founder of RethinkDB put up a fantastic postmortem on what he thinks went wrong with the company side of the equation, and I think it should be required reading for basically everyone at open source. But roughly, it was an extremely saturated market, both in finance and in mind share, and they just couldn't get a foothold. Whatever you think about MySQL and Postgres and whichever of those you prefer, they've set an extremely high bar for what is available for free and people are just not really willing to pay for databases anymore. Big player in the Android open source world, Sanagen, and their former OS distribution Sanagen mod. In 2016, there was a big management shakeup of the company and they looked at their books and they decided the benefits of being open source in terms of contributions they were bringing in were vastly outweighed by how much they were spending on user support and the amount of money that they could pull in from people that would pay for it if it weren't free. So they decided it just wasn't worth it anymore. Two notable individual cases, why the lucky stiff from the Ruby community and Mark Kilgore from the Python community. I say this not as a call out or critique of either of them, but it's cautionary tale if you put too much of a burden of a single community on one person, it rarely works out well for either side. And a more general example, drive by contributors. People that just sort of show up, they do one commit and they leave. Drive by contributors bring really important fixes. They tell you when your user experience is bad. They fix weird rare bugs and sometimes you can convert them into regular contributors or even maintainers eventually. On the other hand, it takes a lot of time and energy to shepherd that first contribution and doing that over and over and over again can be extremely draining. There's not a great answer here, but just be aware of the problem and step back from mentoring first time contributors if you just don't have the energy to deal with it anymore. That's okay. So that's things to track from sustainability. How about things that improve it? A big part of any project is how it's run and that can have a similarly large effect on sustainability. A benevolent dictator for life is where most projects start from, projects created by one person and more contributors slowly join, but that one person remains in charge of figuring out the overall direction where the project is going to go. This can work very well for a young project, but eventually having to run all those decisions through one person can slowly create more and more friction and frustration over time. So some projects that started out with one BDFL eventually fragment into a number of these smaller BDFL like maintainers that all just sort of exchange patches. Linux is the canonical example of this. There isn't actually one Linux kernel. There's like 40 of them and they just all work with each other strange ways. This can reduce friction for the maintainers that are exchanging patches, but it can be intimidating for a new contributor trying to figure out how to get a patch in and where it even goes. Some projects have a more rigid hierarchy for decisions. This makes it really easy for engineers to work out disputes. There's very clear dispute resolution processes, but it brings a lot of social overhead with it. Others use a federated council approach. Usually these are called SIGs, but that's sort of the newer term. Each group has final say over their area of expertise. And sometimes when the contributor base is stable enough over time, you can forego basically all the rules and just say, work on rough consensus. When enough people agree, just do the thing. And if we need to change it, we'll change it. This can feel great for long time contributors, but it can also really easily result in hidden power structures like the one person that is in charge, but is not marked as in charge. And that can be utterly incomprehensible for new people. And there's a lot of other models. As with the project types, I can't possibly hope to exhaustively list these. Every project isn't gonna be only one of these. They're gonna pull together a set of governance rules that matches, hopefully, their specific circumstances in people. Conway's law states that the output of any team is going to eventually look a lot like the org chart of that team. It's intended to be a bit of a tongue-in-cheek joke, but it's also remarkably accurate. Choice of leadership and decision-making structure doesn't just affect the humans. It affects the software that those humans produce. Unfortunately, it can be a bit hard sometimes to tell which direction to move in to improve sustainability for your project. For some projects, more structure can help to reduce the burden on overloaded maintainers. On other projects, giving them more paperwork to fill out will push them even further into burnout. So when you're playing with governance structures, always make sure to listen to the people that you are governing and make sure to change things if it's not working out for them. As I mentioned before, open licensing is an important component of open source. It's not directly related to governance model, but I needed to put this somewhere and one often influences the other. A good license ensures that even if all of the contributors from a project move on, at least in theory, the source code will remain available. It'll still be up somewhere and people can use it, they can fork it, they can continue the project. What about cases where the license can be more than just a backstop? Full intricacies of GPL or copy left general licensing are better left to that intellectual property talk I mentioned before, but the idea of copy left is in its own way sustainability. GPL doesn't actually care about the money side of sustainability, but we'll talk a little about dual licensing later, which sort of gets into that. Exactly how it works depends on the GPL flavor that you use, but the overall idea is that if someone is going to modify your code, it has to be available to the users and thus probably to you as the maintainer. One failure mode of those really big projects I mentioned before is that if they use a permissive license, someone can fork it, change it to keep the patches themselves, and in theory that can siphon off resources from your project onto their fork. Again, in theory, if you put your project under copy left license, it heads off this whole cycle because you can always get their patches and hypothetically if you want them, you can introduce them into your own project. But really how often has this happened? Definitely not zero big projects, notably the Linux kernel itself definitely have issues with proprietary extensions causing friction for both users and maintainers, but my personal view is that this is probably not going to be a thing for your project. I think that copy left probably for most projects causes more problems than it solves, but maybe your project is different. But okay, in the end, maintaining open source is about time and time is money. The simplest funding model is to just not have one, be a volunteer project, and those are great. Some people just like coding for its own sake, some people want to improve their professional portfolio, but remember, volunteering your time is an act of altruism. I speak from personal experience, it's real easy to end up in that cycle of guilt and bitterness that I mentioned before until you just never want to see a computer again. The next step up is working on open source on your normal work hours. This might be a project that your company maintains, that you're improving, or contributing a bug fix back to a project that your company uses, and it's a great option when you can do it, but a lot of companies don't really understand why they should pay someone to give something away for free, so not always an option. The next step up from one person using company time is everyone using company time. It's not a long-term solution, but it can be a way to get a new or nascent project off the ground. You'll need to look into some of these other options to eventually repay your investors. That's the idea at least. So now we have to do the round-up projects that are trying to be self-sustaining, to bring in enough income to support at least some of the time that you are spending on the project. By far, the most popular method is to sell support contracts, but this can be really hit or miss. Small companies often can't afford anything that takes away from their launch window, excuse me, and big projects just think we'll support ourselves internally. Examples of support contracts, there's a lot of them, but MySQL and Java are two particularly popular ones. Sometimes the project itself, or the community that stewards the project, isn't really set up to be a commercial entity, so the maintainers will sort of split into a bunch of smaller companies, and they'll sell support contracts and other services through those, but again, with the same idea of funding their work on the original project. This is the dominant model in the Postgres community, for example. There's no Postgres corporation, but there are a sort of fleet of smaller consulting groups around Postgres, and a lot of the contributors work for those. After selling support, the most common models are based around having an open core and then selling related close source components or differently licensed components that add additional functionality to cater to specific use case, usually big enterprise software. Sometimes there's clear places to add these paid add-ons, like big enterprises love Sammel login, they love cross-team reporting, et cetera, but other times it can be difficult to find a place to sell these features without harming your open source users in one way or another. It's been very popular in the DevOps space, so Chef, Pop and Salt and Ansible all sell commercial add-ons. Dual licensing is a special case where you sell the same software they could get for free, but under different license terms. So the common case is you have a copy-left-based project, somebody wants to fork and modify it, but they don't want to make the patches available, so they purchase a license from you that doesn't have the copy-left restrictions. It's not very common these days. QT is probably the best example of somebody still using this. Similar selling support contracts is professional services. So this is usually some mix of consulting to help them best use or deploy your product, customizations making custom plugins, custom modifications to the project for that customer, or hosting if it contains a server component. Again, examples too numerous to list, but Pivotal Software is a big one, they sell services around Cloud Foundry. Grants have been a mainstay of the academic world since time immemorial, and we're slowly seeing more of them available for software projects. Like with other kinds of grants, you write a list of things you want to do, why you think they're important, how long you think that'll take, and use that to come with a dollar amount, and you go and ask someone for that money. I'm sure the PSF wasn't the first offer open-source grants, but the first that I ran into. A lot of other groups have copied them in the Python ecosystem and offer project-specific or ecosystem-specific grants. Going a bit out from that, the Moss Grants Program, or Mozilla Grants more in general, are another example. Rather than being ecosystem-specific, they're looking for anything that helps with Mozilla's foundational mission of improving the open web. And then one step yet broader, places like Google's open-source programs office, they have a mandate to support basically every project that Google uses. That's basically all of them at the scale of Google. But a big problem with grants, like with VC funding, or any other non-renewable income stream, it's highly dependent on how good your time estimates were. If it turns out that you miss some complex requirements, and it takes you an extra six weeks to complete the project, that might end up coming out of your own project, or the project might fail. We all know that engineers are super great at estimation, right? So we reached the option most people go with. If you like this, how about sending me a few bucks on PayPal? This is so common, I wanna dive into some specifics. So the symbol of solution, straight up tip jar on PayPal, or Venmo, or Bitcoin, if you like that. Really easy to set up, but also very rarely useful. It can be heartening to get a few dollars every now and then, but if you're not okay with working for free anyway, it's probably not going to help on the actually funding your time approach as a maintainer, I'd much rather, and I see a worded thank you email from one of my users than like $5 from one person once. A more stable solution is some kind of recurring donations. This lets you plan out how much income you'll have for the month, you can figure out how much time you're gonna allocate. Patreon basically owns this market right now, but they haven't really reached out to the software world, so it can be a little bit difficult to get people aware that your Patreon exists, and they don't really have support for corporate giving very well, so it can be difficult to get donations from companies. Which leads into a problem that I think both recurring and one time donations share. Basically, it almost always ends up being friends paying each other rather than the companies that are driving economic benefit from my software donating to me. In general, I don't want individual users to ever feel like they owe me for my software. What I want is the companies that are making money off it to get some of those resources back to me so that we can both benefit. But okay, some other donation options. I did actually run a Kickstarter to fund some open source work. It was successful, technically, but it was extremely stressful, and I really wouldn't recommend it. Like I mentioned with grants, I misestimated my timeline by about three months, and Kickstarter's like a grant. The amount of money you have is the amount of money you have. You get it once, and you better hope that that was correct. Some mid-sized projects that are looking to tempt donations from companies rather than people will try to get nonprofit status, so their donations can be tax exempt, at least in the US. But creating a nonprofit from scratch is really hard, and I don't really recommend it, so it's usually a better bet to join an umbrella group, someone that already has nonprofit status and can accept donations on your behalf. Especially when highly the great work of Software Freedom Conservancy, they've helped dozens of projects set up this umbrella donations system. If you're interested in doing that for any of your projects, I highly recommend talking to them. You've got a contact form on their website for it. Outside of the simple logistical issues, a common problem with donations is working out how to divide them if there's more than one maintainer. Sometimes you can side-set this by spending the donations on project-level resources like server hosting or buying t-shirts or whatever. But it can very easily lead to hurt feelings or perverse incentives in the project if you end up paying by the line or by the commit. While money makes the world go round, many donations take the form of discounted or free credits to services. So for example, Rackspace has very graciously hosted part of the CI infrastructure for my chef cookbooks for many years. It's often easier to ask a company to donate whatever their service or their product is than to go and ask them for money. But okay, enough about donations. On to some less common but more interesting funding strategies. Various projects have tried to fund themselves by selling books. Most notably the Lua language did this, but it can be a really difficult line to walk because you still need some free documentation if anyone's gonna use your project. So you have to figure out how to balance those. Oftentimes these days it ends up being if the book is available online for free, but hard copies are available for purchase, but that'll severely impact your revenue. Selling ad space can theoretically be a very lucrative option. Ad sales basically run the entire world these days, right? Setting up Google ads on a website is super easy to get started with but doesn't actually bring in very much revenue. Negotiating your own direct ad deals can bring in a lot more money and give you a lot more control over what ads you show, but it's very time consuming and difficult to get those kinds of things set up and it all might be for naught if your highly technical audience are all using ad blockers anyway. Bug bounties are most common in the infosec world as a tool to try to incentivize people away from selling bug reports and attacks to the bad guys, but they're starting to appear for normal software, other non-security related things like normal features or bug reports, but they're probably still too rare to really qualify as a sustainable funding source. Maybe we'll see this get better in the future. Worker co-ops are a really deep rabbit hole and they all run differently, but we're starting to see some based on open source, Snowdrift and Tiedlift are both getting more interest, but we'll see over time if they are successful, I think right now it's still too early to say. And so we look further out into the future. With science medicine and the arts is a lot of national and even international organizations that provide funding for stuff like basic research. Basic research in science is things that do not really have direct rewards immediately, but they form the basis for future work. That sounds a lot like a lot of infrastructure level open source projects. So maybe one day we can see a national institute of software. There's some government grants available here and there, but it's not really a coordinated system. And if I can dream really, really big, universal basic income would allow people to spend time building things they think are important, even if the market doesn't directly value them. So hopefully by now I've convinced you that sustainable software is important and you've got some ideas brewing for how we can all move in that direction, but what can we do now? As I stressed before, the enemy of open source is most often burnout, not technical failings. Practice empathy both while using and maintaining open source software and it can help a lot. If you make money using open source, which is almost every company these days, be on the lookout for ways you can help or donate to projects that you depend heavily on. And most of all, always be ready for change. Maybe the only maintainer of a critical library for you will have a kid and decide that they don't really care about writing open source anymore. That's okay. Maybe you maintain a library or a tool or something and you get a new job and you decide, man, that thing, it's just not fun anymore. That's okay. And maybe the entire ecosystem will change. Maybe the DAOs on the blockchain will actually take off and that'll be how we write all software forever. That's okay too. I don't believe that last one's going to happen, but that's a different talk. We've all only been writing software like this for a couple of decades. We have a lot to learn about how to do it properly. Thank you very much. There's too many people that I can thank if I had time for their advice and wisdom over the years for how to run sustainable projects. But I'd like to extend an extra special thanks to Nadia Russell and Eric as true luminaries of the space. I highly recommend all of their writings if you're interested in diving deeper. And we have a few minutes for questions depending on when we're supposed to end. When are we supposed to be done? Just so I can keep mine on the clock. By the number 15. Okay. As someone contributing to like a 10 plus year old project that with multiple contributors, I'm kind of feeling mild burnout about like, oh, I don't want to change this. I might break, I could probably break users. I mean, the only thing I can think of is just try to have both, you know, a long-term support version and a branch and a more actively developed branch. Do you have any other advice? Unless someone is paying you, you do not owe them long-term support. I mean, if your project is robust enough and has the maintainer backing to support an LTS release, that's great. But otherwise, just don't. I mean, yes, users will be annoyed, but like, you have to factor in what is going to be best for you as a maintainer, as a group of maintainers. And like, you don't owe your users that much. It is important, but it's not that important. Thank you. Just a footnote on that. It isn't necessarily a binary thing. You know, I mean, Rails, for instance, is pretty, you know, is more aggressive than a lot of people would be about, you know, ditching features that no longer seem like a good idea. But what they do is, you know, what they do is to deprecate and delete on a schedule. Thank you. But, you know, that is a good approach, but users will still be mad. One of the projects that I maintain, we went two years, two and a half years doing no breaking releases because we were very worried that our users were gonna get, and it was terrible. We're never doing it again, ever. The users loved it, but basically all the maintainers said, if anyone ever suggests this again, we're all quitting because there was so much technical debt to clean up at the end of those two and a half years that we were all basically drowning in it. And in the end, I think it wasn't actually better for users. Like, burning out and making all of your maintainers hate their job is not actually in the best interest of users, even if they think in the short term that they would rather not have breaking changes. So it's, I think, a lot about communication and communicating expectations. Any other questions? Awesome, well, thank you very much and enjoy the rest of DevCon.