You're now live, John. Okay, perfect. Thanks, David. Everyone, welcome to the first Hyperledger virtual meetup of 2022. We're excited you could join us for this great session today. And, you know, 2021 was a great year for Hyperledger and the community. But we're looking forward to another great year in 2022. And we're really excited that everyone can join us today. So one of the things I really like about Hyperledger is there's always great projects coming out of the Hyperledger labs. And today we're going to really highlight one of those great projects coming out of labs. So I have with us today, Matt J. Pavologic. And he's going to talk to us a little bit about mirror BFT and ISS and ISS and give you a little background on Matt J. here. He's a distributed systems researcher, specializing in business team fault tolerant consensus protocols. He obtained a master's degree from Vienna University. And then he's worked on the permission lock chain solution. Now he's joining protocol labs to work on. We're going to have a great presentation here about scalable consent to mute, monitor the chat. Anyone who has any questions for Matt J. as we go along here, feel free to post them in the chat and we can answer them as we move through the presentation. Otherwise, when we wrap up the full presentation, then we're going to go ahead and just do a Q&A session. So Matt J, I'm going to turn it over to you and we're looking forward to the presentation. Thank you very much. Thank you very much. Can everybody hear me? Sound great. Perfect. So I hope everybody sees the slides and everybody hears me. So thanks a lot for this opportunity to show what we are currently trying to get going. Namely, MIR-BFT, which is both a protocol for highly scalable consensus and state mission replication and a library that is implementing this protocol. So I've been working on this together with Theresa, Marco and Jason, my former colleagues from IBM who are everybody, who are partly still there, partly only more. And so let's jump right in. What maybe one more remark. If anybody has any question, I like to be actually interrupted. And if this can be interactive, please make it interactive. If you have something that you want to ask immediately, just go ahead and shout because I think in the context where something is explained, it's also better to have the questions, not like at the end and then we need to reload all the context. All right, so first a little bit of background and putting things into place in terms of consensus, TOB, meaning total order broadcasts and blockchain and state mission replication and how these terms actually fit together. So let's start with state machine replication. State machine replication is a concept. It's a method for replicating a service that is usually online for making it highly available and resilient to folks. So state machine replication is based on having several replicas that each has a copy of the whole service state and they all start with some initial state. And we just, let's go to the next slide again. And then they apply operations to this state in some specific order. And each operation that is submitted by a client would update the replica state potentially and create some response that is sent back to the client. And each replica is appending these operations to like a chain one after the other. And each replica is executing these operations in the same order. Now the clients, they have to broadcast their operations to these replicas. And since the replicas need to execute the operations in the same order, they need to be considering those operations in the very same order. So this broadcast by the client of the operations to the replicas needs to happen in some total order. That needs to be consistent across all the replicas. That's why it's total order broadcast. So the clients use total order broadcast to distribute the operations to the replicas. Now these operations, we can see them as forming a chain. Sometimes they call it a log of operations. And at each replica locally, they're executed one after the other. And since we require each operation's execution to be deterministic, each operation will produce exactly the same state at each replica. So- Do I have a question? Yes. Sorry to interrupt you. Just curious about the latency. Do these requests have to come back at the, have to be fulfilled at the same time?  Or do they have to all have to act at the same time? Well, when you're talking about real time, this is actually even impossible to achieve to be for each operation to be applied at exactly the same time. So in general, in the model we're considering, this is the so-called partially synchronous model where there's not really a strong notion of real time. And the only thing we care about here is that each replica executes the operations in the same order, but in real time, this can very well be shifted compared to each other. Does that answer the question? Thank you. Yes, thank you. Yeah, perfect. Okay, so we have such a chain of operations. And these operations can also be blocks of operations in practice. This is really just a mechanism to make it perform better. Instead of broadcasting a single operation at a time, we can say each operation can be replaced by a block of operation and append it to this chain, which makes this chain a blockchain, right? So this is where the term blockchain comes in. If we have blocks of operations, append it to each other in such a chain, that's a blockchain. Now, another way to look at it is that each position in this log, the first, the second, the third and so on, is, it needs to be agreed upon by the replicas. They need to achieve consensus on what the first operation is, what the second operation is, what the third operation is, and so on. And achieving consensus on each of these positions yields total order broadcast that can then be used to implement state machine replication. And the underlying data structure can sometimes be called blockchain. So this is just to give the, like, level out the terminology here. Now, most of these things are used interchangeably very often because they are either, one is the means to achieve the other and so on. So we talk very often about consensus, but why do we need to have a scalable consensus? I guess most of you, since you are here, you already know the soil, skip fast through this. In private blockchain systems, a classic consensus algorithm is necessary to implement such a thing for, for example, banking purposes or for supply chain management or for consortia applications and so on. In public blockchain systems, some committees sometimes need to execute such a protocol to agree on the order of operations or on transactions and so on. Very often these are tens or hundreds of mutually untrusted participants and this is where our protocol comes in and this is where it can be used. So how do we do this? How do we approach scalable consent? How do we approach the problem of a consensus protocol that scales? Well, through modularity, log segmentation, duplicate prevention and censoring prevention. So one after the other, I will actually talk in detail about those. So, but very quickly, we actually take any leader-based protocol of which there has been so many already throughout the years. Some of them are new, some of them are old, but they share many commonalities. There's many leader-based protocols known today and what we do is we look at them as just modules that order some requests or some operations and we don't care how they do it, we just take them as black boxes and we plug them in as modules. Then what we do, we partition the log, like this is the log of operations and we just partition the log into what we call segments and each segment is handled by a different instance of this leader-based protocol in parallel. This creates some problems that need to be resolved. Specifically, it's what happens if the parallel instances are appending the same operations to the log like two different segments of the log is pretty useless if because it's just wasting resources and if each instance appends the same, just wasted their time and so we solve this by partitioning also the set of operations that each instance can propose and append to the log which in turn creates another problem, namely that if only one instance and one leader is responsible for some operation then and that leader decides not to propose certain operations because that leader is malicious for example, then this operation would never be attended to the log. In which case, so how we deal with this is by rotating the assignment of these operations to the different leaders and achieving that every operation will eventually be attended. So this is just an overview and I'll talk about exactly how we do this. I will do, so the rest of this presentation divided into two parts. One is the theory, basically what I was just talking about that's a lot more in depth and then in practice how we actually implemented and where we always look for hands-on approach of some people who might want to contribute and we're always open to enthusiasts who would like to put their hands on it and help implement this. All right, so replicating a log. So this state, this log is what we basically want to have in the end. Well, each replic comes to have a copy of such a log with certain operations inserted in it. Each replica initially starts with an empty log of operations and one of the classic approaches to implement filling this log is using a leader-based protocol that I've been talking about. And most of the leader-based protocols in some way work in three conceptual, in three phases. First, the leader proposes some operation for some position in the log.  There are variations to this, but this is like the basic idea of many of those protocols. So what happens is that the leader proposes some operation one and eventually everybody confirms and commits it and then some other operation and so on. And this log is filled, not necessarily in the order of the free spaces in the log, but operations are being proposed by the leader and committed and eventually all those operations are filled in and that's when they can be also executed by the application. So whenever there's an operation such that all of the previous operations have been already committed and executed, the application can execute the next one. Now the issue is that if we have a single leader in the protocol and the leader proposes the upper, it's the only one to propose the operations. The issue is that usually this proposed phase is by far the most expensive in terms of resources spent. And then the leader becomes the bottom. And so basically everybody, all the other nodes that are just mostly sitting there waiting for the leader to propose something then they quickly confirm and commit but they don't have much to do and the leader is doing most of the work. So and the leader is just one and it doesn't, so this is why many of the protocols don't scale. So what we do is we take a leader-based protocol and we say, okay, we call this a sequence broadcast protocol it's just some abstraction saying that it has some leader and it's just filling in some operations in the log. We don't care how this is done. We say, hey, Mr. Protocol just make sure that some operations are eventually committed and we don't care. So then what we do is we segment the log. So we take this log and we actually take part of the log which we call a segment and we spin up one instance of sequence broadcast of that leader-based protocol. Then we take another segment of the log and we spin up a separate instance of the protocol. We do this for each part of the log like this. I think there's a couple of questions that maybe we wanna take a look at just real quick right at this section. So we have from DE is the question log segmentation is then something similar to sharding? Well, in a sense it's similar to sharding as a concept. However, when I hear sharding what I first think of is having really separate logs and the operations in those logs not necessarily being ordered with respect to each other or not all of them being ordered. Having basically certain operations going to some log and certain operations going to another log. Here, the difference is that to the classic notion of sharding although this is some way of sharding but the difference here is that it's a single log it's just that there are different parts of it and each segment is basically a subset of the slots and each instance of the protocol is only responsible for filling in those corresponding slots in the log. Okay, then we had another question here from Alan and the question is is the commit phase an eventual or strict consistency model? That's the first part of the question. And then do we assume that each node as part of the network have the same compute capabilities as the others and no latency consideration for compute time mismatch? Yeah, so this is a very interesting question and it's a good point. So first the first part of the question. Yes, this is strong consistency. I mean, state machine replication is a method of achieving strong consistency among the copies of the service because each two operations are ordered with respect to each other. So if I'm a node and I have executed the first two operations and I execute the third one I know that this is exactly the same sequence of operations as everybody else executes and my state is at any time completely consistent with the state of any other replica after having executed as many operations. It can happen that I'm slightly behind and I'm still executing operation three and somebody already is executing operation eight but I will catch up and the sequence of states I'm going through is exactly the same as every other replica. Now for the execution speed this is a good point because in the first approach we consider everybody to have roughly the same computing power and thus everybody progressing roughly at the same pace. If this is not the case it doesn't break the protocol in any way. It's just that maybe a leader that is let's say slow would need to have a smaller segment than some other leader that is faster that has a better network connection that can handle more requests per second. So this is definitely possible to tune but for simplicity let's stick to roughly to the assumption of a roughly the same computing and communication resources per node. But this is a valid point and it has to be addressed but it can be addressed. Any more questions? Looks like we had one more question from DE and it's how do you avoid that this protocol now are real time constraints? Example, the leader processing segment six requires that the leader of set five broadcasts its results as it requires this as input and has new current state to complete its task. Yeah, exactly. So that's a good point. So the leader of segment, well, the leader of the first segment let's say hasn't managed yet to broadcast what goes in slot five. However, the leader actually let me go a little bit forward. I'll show what exactly is happening. I have it on the slide. So basically all these segments and all the instances of the leader-based protocols are almost independent. So the one only cares about filling in these. The second one only cares about filling these and so on. So and this can go in, everybody. Yeah, this can go in parallel and we spin up these instances such that each instance has a different leader. So the load is spread among several leaders. And this leader doesn't really care what's happening in this segment and this one doesn't care what's happening here and then so on. So that's why each is proposing operations in parallel and this can happen and they independently of each other and this situation, I guess this is what you were referring to in the question. Correct me if I'm not, correct me if I'm wrong. So. And feel free to come off mute if that would be helpful to describe it. Yes, because actually, I don't see the chat and I don't know where to, where to. Yeah, just at the bottom of your screen if you kind of dismounts over it, it'll pop up a little bar that will say chat, click on that. Ah, chat, here we go, yes. At DE, if you wanna come off mute and describe your question in greater detail, feel free to do so as well. Yeah. So for now, I assume that this situation is what you were referring to, right? That we have some gaps basically in the log. There's nothing at six, but we already have operation seven. So this just means that we just need to wait. Nobody can, no replica can execute operation seven. Before operation six has been committed. As whenever there's a gap-free prefix of the log, then that's when we can execute. But what we guarantee that eventually, and this is the property of each single leader-based protocol, that eventually each of these will be filled by something. And then, and eventually it's guaranteed that all of those positions will be filled. And all those will be able to execute. All right, now, what happens when leader two, for example, crashes? And so this is what I just said. The other nodes that are not leaders in the segment, they somehow need to agree that, okay, this leader has crashed, and these two positions will never be filled and we just need to skip them. So an important point to note is that every node is participating in every instance of this leader-based protocol. It's just that it's only leader in one of them. And in all the other instances, it's still participating just as a follower and confirming and committing, but it's not proposing. And that's when leader two crashes, then the other participants, as is the case for any leader-based protocol that is live, the other participants agree that, okay, leader two crashed and we're just not committing anything here and we're skipping those. So this can happen. And in the end, everything will be filled except for these two. And the, but since everybody knows that these are, these will never be filled because the leader crashed, we can just skip those positions and execute the rest of the operations. Okay, now we have a problem with this. The problem with operation duplication. So what is operation duplication? We said that these segments are independent of each other. So what if leader one and they propose operations for their own segments, but they don't really know what's going on in the other segments. And if a client comes and wants to broadcast an operation, the client needs to send it to multiple leaders. Otherwise he risks that the leader that the client is sending the operation to is malicious and would never propose the operation. So the client needs to send the operation to many leaders. And now, but since they don't know about each other, they might happen, it might happen that everybody proposes the same operation. And like this, and if this happens consistently, which in practice it actually does, if we don't do anything about it, then we lose the benefit of having parallel leaders. Because here, after the whole thing, we only committed four different operations. So we're basically wasting resources. So how do we deal with this? We partition the operation space as well. So what does that mean? That means that for each leader, we only allow a certain set of operations to be proposed. We assign some set of operations to the first segment and instantly we assign a different set of operations to the second one and so on. Like this, the first one can only propose and fill its own operations and the second one as well and so on. We call this set a bucket. So every leader gets a bucket of operations and it only is allowed to propose these operations. If it happens to propose some operations that is not in the bucket, well, then all the other nodes that are supposed to confirm and commit these operations, they would say, no, no, no, I'm not committing this because you're not allowed to propose this. Okay, we have a question. We have a question. Your operations in the chain don't have state dependencies on prior operations. I'm confused. Parallel leaders appear to run consecutive operations in the chain that depend on the computation of the prior operation. Well, if they do depend, then the dependency would have to be executed first. So it's the client's responsibility to not submit two operations that depend on each other because before an operation is committed, it is not ordered. So if I'm a client and I submit two operations that depend on each other, I still have to count on the fact that they might be reordered in the final log. But if some operation is committed and executed and it has its place in the log, since it's executed, I see the result of it. Otherwise, if it wasn't executed, I wouldn't see the result of it. And then based on that result, I can freely submit another operation that depends on that result because I know that it will never be ordered before the operation that has been already executed. So please, let me know if there's more explanation needed. Okay, so we have these buckets and each leader only is allowed to propose operations from one bucket. Now, this in turn creates another problem, namely operation censoring. So what does censoring mean? So remember this situation where we had one leader that crashed and that couldn't propose its operations. Well, what if that leader wasn't just crashed but actually malicious? If leader two says, you know what? I'm never proposing operation six. Well, then that's kind of a problem because if operation six is only in the second leader's bucket, then the client who wants to propose operation six will be sad all the time because nobody else can propose its operation only leader two. And the operation would actually never be appended to the law. So this is a problem. And since we do not know which leader can be malicious, this is not an acceptable situation. So the solution that we propose is bucket rotation. What is bucket rotation? So we have our segments and our log. And let's say in this particular example, there's 16 positions of the log divided into four segments. So what we do is that we call this epoch one. But the log it continues, right? The log continues indefinitely when new operations are added. So we say that another part, another section of the log, again divided into four segments would be the epoch two. And the log would be divided to a succession of epochs like this. And each epoch is subdivided into segments. Now here comes the trick. In the first epoch, we had different buckets assigned to different instances of the leader-based protocol. And in the next epoch, we actually reassign these buckets differently. So here we see that bucket one has been assigned to the first segment in epoch one and the same bucket is assigned to the fourth segment in epoch two. I see a question. If bucket rotation needs to happen, then it should be dynamically updated to exclude execute operations. Yes, that is definitely true. And in fact, a bucket is just a, it's basically an infinite subset of an operation. Imagine here in this case, it's like module of four. And each operation has a class, has a bucket which is a kind of a class of operation where it belongs. And it's not like a list of operations that can be proposed. It's really a class. Imagine that you take the operation, you hash it and you take the last two bits of that hash and that assigns a bucket. Okay, I see more questions. Similar before this, introduce another real-time constraint. You can only switch epochs after all operations in the previous exactly, that is true. So we switch, we go from one epoch to the other, once the first one is filled with operations. And thus clock synchronization need to be dealt with for epoch strategy, not necessarily. Since we know that each segment separately is guaranteed to eventually fill all the slots, then we know that this eventually will happen. And we don't really, in our model, we don't really care when this will happen. In practice, of course we do. And we can tune the implementation such that it's very unlikely to happen that we would need to wait for a long time. But in theory, we don't really care. In theory, we just are guaranteed that all of these operations will be eventually, will eventually appear in the log and this will be filled. And then we switch to the next epoch. And here, if we look at operation six, for example, imagine that the second leader would not submit operation six here. Then in the next epoch, the operation six is in the first leader's bucket. So if the first leader is correct, it would propose operation six. So in EVA, at least after a few rotations of the buckets, each operation would be in a bucket assigned to some leader that is correct. Okay, I see another question. Does clock synchronization need to be dealt with for if a malicious leader does not fill the bucket? Yeah, well, if a malicious leader does not, you mean does not fill the slot in the log. So if a malicious leader doesn't fill a slot in the log, it's basically this situation. What player are we here? Is this situation where all the other nodes that are participating in that protocol would declare the leader faulty and agree that these slots would be skipped or maybe filled with other operations. Okay, so we have seen how the buckets are rotating, how we can avoid the censoring of requests. So to summarize a theory. Sorry, maybe one question to the previous slide. Yeah, so to the previous slide. So in the case when the leader was crashed, the others were able to propose something like to agree on an empty or nil operation of the slot. And for the case where the leader is simply malicious and keeps some operations, although the clients already announced that they wish to execute certain operation. In protocols like PBFT, there is a mechanism of to trigger, to kick out the malicious leader and replace it with another replica that could propose the requests that are waiting for too long. And so do you just remove this functionality when you use, let's say PBFT as a slot as a segment protocol or is it completely replaced with epoch mechanism or is there any way that the replicas that know within the epoch one that certain leader is already misbehaving doesn't propose some operation for too long, although it's known that some client already long ago requested this operation somehow to kick it out or do some replacement early on. Yes, yes. So in fact, in our implementation, we do use exactly PBFT as the single leader protocol for a segment and PBFT has the view change sub protocol when they find out that some operation has not been submitted or nothing has been committed to some slot of the law. So in this case, only base, well, for two reasons. One reason is simplicity that it is simpler to, when the view change protocol in PBFT is initiated to just declare all the uncommitted slots as empty forever and call it a day and say, okay, this leader is faulty, we're done and this is the result. Of course, we could switch the leader in PBFT and have some other know become leader of an instance of that protocol and propose some other operations. The slight issue with that is it's only a performance related thing that usually when you switch the leader in one instance that the new leader in that instance probably already is leader in some other instance and it would have to have like double work to do to propose operation for its own instance and propose operations for the other instance. That's why we just take a shortcut there and say, okay, the new leader is actually elected, but the only task of the new leader is to propose empty batches for whatever hasn't been proposed yet or for whatever has not been committed yet. And like this, the epoch finishes and the uncommitted operations that the clients already announced they want to commit, they would be dealt with by somebody else in the next epoch. And what's the condition of epoch change then? The epoch change happens periodically. Like after just fixed number of operations. Yes, after fixed number of operations. This can, of course, we can go and try to tweak this and dynamically change it and it's totally possible and the implementation would actually support it that based on whatever criteria the system can decide how long the epoch should be, but for simplicity currently, it's just after fixed number of operations. And what actually is already happening when a leader crashes, for example here, if leader two misbehaves or crashes and there are some nil values that appear in the log in the next epoch, that node will actually not become leader anymore or it will be kicked out of the leader set and then in epoch two, for example, it would only be partitioned into three segments, noting four. But this is my first simplicity, I didn't mention this, but yes, this is all possible and it actually even happens. And you can specify whatever policy you want for kicking out suspected leaders and including them back in the leader set. And then was this your question? Operations are not redistributed if a node fails to perform those operations. Yeah, the client sends the operations to multiple nodes. So if some leader fails to commit that operation, then some other leader will succeed later. So that's basically the gist of it. Okay, I'm glad that there was actually interesting questions, we were running slightly out of time so I'll need to skip over some other things. So just to summarize, we have a scalable consensus protocol that is based on modularity, namely by using any leader-based protocol as a module and we use multiple instances of it, one for each segment of the log and this happens in parallel and that's why the resources of each leader can be used optimally. This creates one problem, which is how to deal with duplicates, how to make the leaders not propose the same operations at the same time. We solve this by partitioning the operations in buckets by hashing them. Basically just look at the hash of the operation and you know in which bucket it goes and assign one bucket to one leader which in turn creates another problem with censoring where if the leader that has a certain bucket happens to be malicious, operations from that bucket couldn't be proposed by anybody else anymore. And that is resolved by periodically, namely after a certain number of operations committed in the log by periodically rotating the assignment of these buckets to leaders. So eventually each operation will be assigned to a bucket with some leader that is correct that will be proposed. All right, so that was the theory and basically out of time. So I'll just show quickly how we implement this in practice. And this is a new protocol and we just, we're just starting the effort of having a really nice production ready implementation of this and when, if you're interested in contributing to this or in looking how it works or using it, just drop me a message on my email. I think it must be somewhere, if not, I will supply that. And let me know if you are interested in actually looking in the code and contributing or maybe even using that protocol. It's in an early stage of development. It's a rather fresh hyper lecture lab. And I'm just going to show you in a few slides the architecture of the implementation. I just, so you know what I'm talking about and see how it works. So the architect, so this maybe of this also library. It's a library that provides you with one main object, which is the node which represents the replica of the system we were talking about. A node is basically a data structure and it has a few methods. You can run the node, well, you can create the node, you can run it and you can submit requests to it and you can ask it what the status is for mostly debugging purposes. So a node consists of many modules. Some of them are shown here. The most important ones probably. So one module is the application. The application is basically the, well, the service that is supposed to be replicated is the implementation of the service. Even the protocol that I've been describing now is actually a module and it can be switched for some first for another one. Then there's a module for sending and receiving messages of the network and there's a right ahead log module which is crucial for situations where for example, the node wants to restart. Now, if you restart a node, then the node needs to keep persistent, persistently track of what it has done, what messages it sent and so on, such that when it restart, it can continue where it left off or at least not do anything contradictory. And now all these modules, they produce events. For example, when a message is received from some other node of the network, it's received by the net module. As an event, it is put in some buffer and then there's a processing function that would take these events from the event buffer and distribute them to whatever modules need to process them. For example, there's a message incoming that is part of the consensus protocol. It goes in the buffer, then the processing function says, oh, we have a message, we relate to the protocol module. The protocol module processes it and generates, let's say, some new messages that need to be sent. They go in the buffer, the processing function reads them and gives them to the net module to send them to whatever other nodes they need to go. Now, all these modules, they're working in parallel. So we think that this will be very efficient, although we haven't benchmarked it yet because not all of it is implemented. So some of the modules, the most important module from the user's perspective is obviously the application module, which only has a simple interface. It has an apply function that receives a request badge and it basically tells the application, this is the request you need to process. And then the system can ask it to create a snapshot of the state of the application. For example, when the thing needs to restart and so on and then you can restore the state of the application by giving it a certified version of the state. Then the protocol module actually has a very simple interface, but it's actually probably the most complicated one because all the theory that I was talking about is actually implemented in this protocol module. It has some protocol logic, it has some state and all these events that can be submitted requests or receive messages or some ticks of some timer and so on. They're processed by the module as events and then new events are produced for other modules. Oops, this is a typo in the copy paste error in the slide here is that the writer head log module I'll skip over this, it's not that crucial and I'm running out of time. And what is also very nice about this kind of approach is that we can have a separate module that is called interceptor and all these events that are happening here can be actually intercepted and recorded and fed into some debugger program and the debugger program can then analyze exactly what happened inside the node. It can even feed those events in the node and this is very, very nice for debugging and for showing what's happening inside. For example, you'll want to run it. You run it, you record the modules and something goes wrong.  So this is all in development though but it sounds like it is a very, very nice debugging tool. So this is how we use it. When you want to actually deploy some application on each node, you basically take the MirriFT library, you create a new node, you give it some configuration parameters like its own numeric ID and you tell it where to write the log messages and then you tell it which modules it should use. Like most of these modules are planned to be or are already implemented by the library itself but the user can replace them by more fancy versions if they want to. The most crucial module for the user is the application. For example, here we have a chat application. So now I only have maybe, what's the time? Yeah, maybe I have a few minutes left, I show quickly how it works. 10 minutes left for questions or any further description or demo that you wanna do. Perfect, perfect. So I show you a little quick pass over how that actually works and then we can have some more questions if you have any. So first, what do I need to do if I want to use this library to replicate my application? And here, let's say we have a small chat demo application. Do you see the code on my screen? Yep, we can see it. Perfect. So I've created a little simple chat application where it is just a data structure that stores the state of the application and the only state of the application is the history of the messages that have been sent over the chat. And one more thing that it requires is just a reference to a key value store that stores the payloads of the requests which are disseminated separately. I didn't talk about that too much for in the interest of time but the application basically receives batches of references to requests and then it goes to a storage module where it retrieves the data from the request. The only relevant thing that I need to do is implement the application logic. So here, what the application has to do when new requests arrive. And here we model the chat as each chat message is basically a request to the application to append the new message to the chat history. So here we just basically go through all the requests in the batch. We just retrieve the data from the key value store. So request data is the actual message. We construct a nicely formatted message that we can print. We add the message to the history and we print the message, that's it. So this is the application logic, it's simple. Now, we also implement the function in restore state. So I'll skip very fast over this. This is basically, we just serialize everything in a protobuf message and we return the bytes and to restore the state. If we lie behind and you really need to catch up with the state, the system might ask us to just restore the state from scratch and this is basically just restoring the chat history and we print everything. Okay, so here's the main function of the application and I only show you what you saw on the slide. So basically, we have a new node. We create some new modules like a request store, a networking module, a write-to-head log and a protocol instance. We pass it to the new node function at creation of the node. For example, the ISIS protocol is part of the library. ISIS is the name of the protocol there was describing. I should have said that actually. And so we just create an instance of the protocol. It has a well-defined interface that is in the documentation and that you saw on the slide. And then we just pass it to the node. And then we basically, we just run the node here and we submit requests. So how does it work? I already created a VM and I basically run the demo application and this can be running on a different machine, on the network. I just run each of the nodes. I give them their own ID. They basically initialize and whenever, when I type something in, say hello, I press enter, then everybody receives the request. Hey, please add client zero, say hello to the chat history. And here I can say hi and it is appended to the other ones. So this is running on my local machine but it's communicating over the network and it can be actually distributed. So this is just a hello world application, what can be done and you can of course imagine the application being much more involved. So I took a long time but if you have any more questions, please go ahead. I'm gonna go ahead and post a link to the labs in the chat as well. So if anyone wants to get engaged with that, I put in the link just now and then if you wanna go ahead and post your email. So if anyone has any follow up questions, that'd be great. So this is my email. And let me also go ahead and post the GitHub repository too. Ah, yes. Actually the labs website is maybe not as up to date. Everything that's happening is happening on the GitHub repository. Okay, perfect. Yeah, that's what I kind of figured but I wanted to give them kind of the overview as well. Yes, yes, absolutely. So please go check out the GitHub repository. And if you have any questions, just drop my email. Okay, let's see if there's any other final questions here. I think the thing is, you know, with the labs, we always wanna get in as much engagement in the community as possible to build out a labs project. So I welcome anyone to jump in and work with the team to put new code out there. And let's see if we have any... Yeah, it looks like everybody's very excited about the presentation. Thank you very much. Thank you very much. I see one question. That we can provide to the team as well. Yes, you have a copy. I sent it to you, right? Yeah, I'll go ahead and use that presentation that you sent to me earlier. Yes, perfect. So there's one question. Is this compatible with EG Hyperledger fabric deployments? Well, it is not, but it is designed to be. So currently Hyperledger fabric has an ordering service. And I think it is a matter of time when this library will be just pluggable in Hyperledger fabric to have a proper scalable BFT support. Now it is not possible yet. And also because the library itself is in an early stage of development. So many things that I've talked about are partially stubby or not quite yet implemented. That's why we need your help. And this is supposed to be integrable integrable with Hyperledger fabric. Okay, perfect. I don't know if there's any more questions. Yeah, go ahead. I'll share my email again. I already wrote it there. Oh yeah, please share your email, yeah. I'm pretty new to the store technology itself. So I would like to participate in that. Yeah. So do you see my email address or contact me on GitHub? I think there must be some email on GitHub as well. But here's my email again. Okay, well, thank you very much for the one presentation. Everyone should have the email there and I'll share the presentation with anyone who signed up for the meetup as well. And we look forward to seeing you for our next session that should be coming up over the next month. So have a great day everyone and we'll talk to you soon. Thank you. Thank you very much. Thank you very much. Have a nice day or evening or morning or wherever you are. Okay.


Notes from the edge of the narrative matrix. Pouring more weapons into Ukraine is not how you save lives. You save lives by negotiating a ceasefire. Pouring more weapons into Ukraine is how you create a long and expensive military quagmire for Russia at the cost of many thousands of lives to advance U.S. strategic interests. While making a vast fortune for the arms industry. Issuing a guarantee that you would never add a nation to NATO who you don't plan on adding anyway is a no-brainer when the alternative is mass military butchery. I mean, unless your goal was to provoke mass military butchery. If the Kremlin wanted to kill large numbers of people it should have done so with starvation sanctions and proxy militias like a civilized government. Fashion has moved on since the early 2000s, you savages. You're not allowed to talk about the known U.S. NATO and Ukraine actions which experts have been warning for years would lead us to where we're at. You're only allowed to say Putin attacked Ukraine completely unprovoked in a vacuum solely because he is evil and hates freedom. Only talking about the guilt of the invader and not the things that were done to provoke that invasion is exactly what I'd want people to be doing if I had just provoked an invasion. Believe it or not, it's actually always completely legitimate to criticize the actions of the most powerful empire that has ever existed especially when those actions clearly paved the way to a war. The fact that Western media cover Ukraine in a wildly different way from U.S.-led wars is actually immensely important and points to a problem that urgently needs attention.  Literally every single time I say NATO powers paved the way to the Ukraine invasion I get some liberal claiming that it's like asking what a rape victim was wearing. No, actually, victim blaming a rape survivor is not at all like criticizing the most powerful and deadly power structure in the world. Liberals have been bleeding this line ad nauseam for days and it's about the most obnoxious and most shit-lib thing you could possibly come up with. Empire apologists always try to distort power dynamics to make it seem like they're the brave up-punchers sticking up for the little guy. It's tiny Ukraine against the big bad Russia. Not Russia against a globe-dominating empire of which Ukraine is just one member state. It's the brave freedom fighters of Syria versus Assad. Not Assad against a planetary unipolar hegemon using proxy forces to effect regime change. It's Israel against the big strong Muslim nations which surround it. Not an entire empire of which Israel is just one member state picking on far weaker powers, etc. For years anti-imperialists have been calling for detente and warning that all this Cold War brinkmanship with Russia could lead to hot war. Now hot war is here as a direct result of refusing to pursue detente and they're trying to act like we're the assholes. Gonna be real fun when the consensus that it's fine for government-tied Silicon Valley corporations to censor online speech in the fight against misinformation moves from targeting COVID skeptics to targeting people who disagree with mainstream Cold War narratives about Russia. Nazis in Ukraine, Russian propaganda. US-backed coup in 2014, Russian propaganda. Donbass provocations, Russian propaganda. US and allies armed terrorists in Syria, Russian propaganda. Mass media is propaganda, Russian propaganda. The government's not your friend, Russian propaganda. There's a tweet by Senator Mark Warner. I'm concerned about Russian disinformation spreading online so today I wrote to the CEOs of major tech companies to ask them to restrict the spread of Russian propaganda. There you go. The solution to a crisis that was created by brinkmanship is not more brinkmanship. The solution to a crisis that was created by brinkmanship is détente. Saying the US government is still a far worse offender in the mass military slaughter department than Russia will be met with hysterical shrieking and the rending of garments now. But it's still indisputably true and if you disagree with it, it's because you're propagandized. That the US is the most murderous government in today's world is just an easily quantifiable fact. Putin will have to work very, very hard to catch up to those numbers. This is important to note not because of some genocidal dick measuring contest but because it points to what a healthy attitude toward US military butchery would look like through eyes untinted by propaganda manipulation. Theoretically the actual US and NATO military decision makers, no imposing a no fly zone over Kiev would be insane since it's a one way ticket to another world war. Probably a very fast and radioactive one. But the fact that so much of the official US political media class has been calling for one discredits it forever. They sincerely don't seem to understand what it is that would be stopping the Russian planes from flying under such a scenario. They think it's like a rule you make and then the Russians go, oh shucks I wanted to fly there but it's against the rules now.   It happened 19 years ago. The Simpsons stopped making good episodes longer ago than that. It just happened. The consequences are still unfolding. The occupation is still ongoing and they're still using the same old tricks. Hollywood teaches us that heroism looks like an individual stopping a bank robbery or leaving a criminal tied up outside the police station so that we don't realize that real heroism looks like a collective rising up against our plutocratic rulers and creating a healthy world.


Well, back to the ice in class two way. Brainard Little Falls is getting ready for their first ever trip to the state tournament. Clayton Castle has more on the Warriors as they prep for the X tomorrow evening. History is in the making for the Brainard Little Falls Warriors as the team heads down to the XL Energy Center as the first hockey team from Brainard to make a state tournament. It's a blessing actually because it's kind of like my last chance ever to take this opportunity and really run with it. Friday night's thrilling win over Rozo in the section finals was exhilarating for the team and the fans, but the girls know it's just another step towards the ultimate goal. I knew I had to come out confident. We all knew we had to come out strong right away from the start and keep playing through and keep playing our game. We kind of tired them out this year. I feel like last year we kind of ran out of gas a little bit, but it was like the roles were reversed this year. Waiting the Warriors at the X is top seeded Adina, which boasts multiple division one recruits, but that doesn't fret the Warriors as they will just stick to the game plan and give it all they've got. We need to remember to keep true to our preparation before the games and that's really what our coach stressed before the Rozo game. And I think if we do that, we can do well. No matter what happens at the X, the team has been gratified by the community support it has felt over the past week. It's just neat for the girls to be able to be part of that, you know? And it's one of those opportunities that you're never guaranteed anything, so you want to make the most of it. The X will play host to the Brayner Little Falls Adina match up on Thursday night at 6 p.m. Clayton Castle, Lakeland News. And of course, those two teams met earlier this season at the Adina tournament and they went four to one. So a little payback, a little rematch here for Brayner Little Falls and quarterfinals. If you enjoyed this segment of Lakeland News, please consider making a tax-deductible contribution to Lakeland PBS.


Today we celebrate Julian for all that he has contributed to our right to know. We condemn his arbitrary detention for 11 years and redouble our efforts to save his life and secure his release. Julian Assange has been punished for the extraordinary impact he's made in revealing the crimes of powerful states and for ensuring their widest publication by orchestrating a collaboration amongst the world's media the likes of which has never been seen before. What was less obvious though was the global network of eminent journalists enlisted before that to verify documents, provide analysis and advice regarding the significance and to make redactions. The redactions have never been properly recognised because of the determination by those he exposed to attest contrary to all evidence and their own inquiries that he caused harm. No one has been harmed but the victims of the crimes he exposed and for doing so Julian himself. His health has deteriorated further following a stroke and he is ever closer to extradition. The UK High Court's decision to accept untested US assurances that they will take into account his medical condition was based on superseded information as they failed to take into account that he had a stroke. Though his sought leave to appeal the UK's decision to extradite him, it is possible that leave will not be granted. Though he might appeal to the European Court of Human Rights, the UK is likely to ignore their ruling and is set to reform laws that will guarantee they can. We call on our governments to demand the UK receive the decision to extradite him and that the US drop the prosecution because of the impact it will have on press freedom worldwide. It is imperative our governments act to save the life of their citizen whose health has all been destroyed by this long term persecution which UN rapporteur Nils Meltzer has unequivocally acknowledged his torture. Julian's achievement stands as an act of enormous courage, vision, principle and resourcefulness and is arguably the greatest contribution to the pursuit of truth and justice made by an individual. Julian, you do not belong in jail. You should be free to enjoy life like those of us who have not risked what you risked for democracy. Your children should have their dad and know he is a hero. Thank you Mary.


Okay. Let's go ahead and get started. My name is Joe Brockmeyer and I am the speaker for this talk. I'm going to be talking about how you can build successful open source communities. I enjoy when people follow along on Twitter. So I'm JZB on Twitter. If you have questions, comments, concerns, whatever, feel free to post them on Twitter. Shout at me there. I work for Red Hat. I've been at Red Hat now for about three years on the open source and standards team right now. I've also been involved with the Apache Software Foundation, which knows a little bit about communities, open source communities. I've been on the cloud stack PMC. I'm now emeritus on that. Also work with the incubator PMC. How many people are familiar with the Apache incubator? I'm not seeing too many hands. Okay. Well, I'll talk a little bit about that during this presentation. I'm also a member of the Apache Software Foundation, formerly the open SUSE community manager from 2008-2010, and I am a recovering technology journalist. So not doing that anymore. And also I like books, beer, cats, and polar bears, not necessarily that order. And I'm a bit of a music fiend. If I may do one personal plug, if you check out Zonker.net, I've been working on a project. I decided I wasn't writing enough. So I've been working on a project writing about my favorite 100 albums, which has been a lot of fun, but not getting as much traffic as I want. So feel free to look at that if you want something that has almost nothing to do with technology. So let's talk about what we're going to cover today. I'm going to discuss why community over code, why community is important. Many of us are technologists, and it's not immediately evident why community is more important than code sometimes. I want to talk about defining success. A lot of people, you know, have an idea that they want community. And that's about as far as it goes. They're like, well, we need community. What does that mean to you? Because that means different things to different people. I want to talk a little bit about governance in community, why governance is important. I want to talk about infrastructure, which is important. I'll talk a little bit about marketing, and I'm not using the strict textbook definition of marketing here. I'm talking more just about promotion in general. This is an old version of these slides. So ignore the how can we help part LibreOffice helpfully crashed and restored a slightly older version of these slides. So this is actually, it looks like it retains some stuff from when I gave this talk earlier this year at Dev Comp CZ, which how many people have heard of that? It is a conference Red Hat does in the Czech Republic in Brno every year. Mostly about 60% Red Hat or 40% other. But a really good conference if you care about things like JBoss and Fedora and that kind of thing. So ignore that slide. Let's talk about what community over code means. In an open source community, sometimes how you do things is important is what you do. When you have a bunch of people who work for one company, for example, participating in a project, it's very easy to just have conversations and make decisions in the local office over lunch, which tend to exclude all the people who participate in your project that don't happen to work in that office. That's actually even true when you have a preponderance of developers in one area of the same company. You can tend to exclude people by making decisions locally. So it's important to have procedures in place sometimes that ensure that those things happen in public, even if it's slower, even if it's a little bit harder sometimes. Another thing that's really important is having a healthy community so it can roll with changes. How many folks are familiar with open source communities that have basically died when the main company involved has withdrawn its support? There are many examples of what looked like a healthy project until a company decided this isn't important to us, we're not paying people to work on it anymore, and suddenly the project basically is a ghost town because they never grew the community appropriately. Another thing I want to talk about is nobody is irreplaceable. People have heard of the bus factor, right? How many people can get hit by a bus and your project is still a viable concern? People used to really worry about the Linux kernel, for example. What happens if Linus is hit by a bus or decides that he just wakes up one day and this isn't what he wants to do anymore? You need to have a number of people who can step up, and it's just like being a manager at a company. If you don't know who is going to replace you, there's a problem because you're not going to be in that position forever. If you're a contributor in a community and you're not looking around and trying to grow a contributor to take your place or two or three, then there's a problem because eventually either life will get in the way or you'll find a new job or something, which means that you have to go do other things and there's no one to step up and replace you. Another problem that a lot of communities have is the person who is abrasive or hard to work with to put it in nice terms. A lot of communities suffer from people. Often they are great contributors technically, okay? There are a lot of people who are really, really abrasive and they're good contributors from a technical standpoint, so people are hesitant to say, you know, show them the door or correct their behavior. But what happens with those folks is I don't care how great a contributor one person is, they're not a replacement for five other people who will never get involved in your project because they don't want to deal with the abrasive person, which I have seen happen over and over and over again. Like I said, I'm a member of the Apache Software Foundation. I'm on the member and the board lists and I watch people come in and complain about different people in different communities and say, this person has driven this person and this person and this person away, you know, and sometimes it's hard because sometimes that behavior is borderline. It's not always as simple as one person just shouting things on a mailing list and calling people names. Sometimes it's somebody who always drags a conversation out longer than it needs to go or always has a technical question even about things that they're not involved in or somebody who's a little bit abrasive, but not, you know, not enough that you would call them out on it at any one instance, but it happens over and over and over again and people need to be willing to pull the plug on those folks, okay? So whenever you say, like, we have a problem with Felix or we have a problem with Alice, don't let people say, well, not but they're a great contributor. You need to set that aside. We'll do it right later. How many people have heard this in, yeah, so many scenarios? We'll do it right later. Later never comes, okay? Both technical and social decisions, when you make a technical decision, assume that it's going to be in place pretty much forever, okay? Some projects are good at revisiting decisions, but generally, especially when you're talking about technical debt, try to make the right decision the first time, not, you know, throw something in and do it right later, okay? And that also goes with governance, which I will talk about in a little bit. So there are some core principles that, you know, if you're trying to grow a healthy community, you should think about. My favorite with the Apache Software Foundation is if it didn't happen on the mailing list, it didn't happen, okay? You don't get to make decisions that affect the entire project in private and then, you know, just implement them, okay? Now, what does that mean? It doesn't mean that if you're developing something in Apache that you have to run every single patch by other people, okay? It doesn't mean that you can never commit something without having discussed it with other people. But when you're talking about architectural changes or major changes, changing the API or whatever's important to that project, you need to put it out on the mailing list for discussion, okay? You need to have an open community where you can do those things. You certainly can't have six people who work for one company having a voice conference and then coming back to the list with decisions. That, you know, really disenfranchises everybody else. You shouldn't be making decisions in IRC in real time when, you know, six people are on the East Coast in the United States and two contributors are in Australia because that's hostile to the people who can never join those conversations in real time. Mentorship. People often need guidance in a community. They need a little bit of help to learn how to do things. So it's important for people in the community. I was about to say senior people, but it doesn't necessarily have to be. One of the other communities I'm involved in is Fedora. And we have a contributor right now who's only been involved for, as far as I can tell, about 16 months. I think he's a sophomore in college and the guy is awesome. He is all over the place. He's doing things right and left. And whenever somebody new shows up to the community, he's right there like, what do you need help with? Yes, it's really tough to get this done. I know how to do it. Here, I'll show you. Like, he is the guy standing by the door showing everybody the punch bowl and is awesome. He's not one of the senior members of the community, but he's certainly a leader in just a very short time. But it's very important to provide mentorship to people. There's a, there's, I am not the most extroverted person in the world. And when I go to a party or something like that, I'm usually the person that, you know, if it's like a corporate party, I go find where the snacks are and I stand by that table. If it's a, if it's a party where people have pets, I'll be the person over in the corner talking to the cat. Okay.  That's what you really need in your community. You need somebody that when somebody shows up, they will take them by the hand and say, hey, everybody, everybody, this is Paul. They're awesome. So Paul wants to contribute. How can we help them? Okay. Earned rights are really, really important. Has anybody ever participated in a project where you've contributed for a long time and never seemed to get recognized? You know, where you've basically, you've done stuff for a while and other people have commit bits because they all work for the same company, you know, new hire comes in, hey, let's set you up with get commit. Yeah. Okay. You're good. And here's this person working for another company, maybe even a competitor who's been contributing for six months or a year, they still don't have the same rights that the person who was just hired in over here has. That's a problem. That is poisonous to communities. And it's something that people don't generally think about. A lot of times now in projects that I work with at Red Hat, we will set up a thing where even people who it's your job to commit to something, you don't necessarily get commit privileges right away. You don't necessarily walk in the door and get the keys to the project just because you work for Red Hat. There is a concept of earned rights. Okay. Another thing that's really important is quality is greater than speed. I don't think I have to do a lot of selling on that particular point. I think a lot of us know that it's better to ship things when they're ready as opposed to just as soon as possible. One thing that's really important to me as a non-coder is that all contributions are really important. And this is something that has taken open source communities well over a decade to really start coming around on, which is funny because really if you look at things like, let's just take an example, Debian. The preponderance of people involved in Debian are packages or technical folks, which to me means that they're doing a fairly decent job of attracting those people already. What they are missing is people to write documentation or do marketing or translation or some of the other things that aren't as immediately recognized as the technical quote unquote contributions. If you're trying to stand up a community or you're trying to write a community and get it working properly, you need to go out of your way to recognize contributions from people. Doesn't matter whether it's marketing, whether it's somebody who's on IRC every day answering user questions, which is really important and usually not something that developers want to do. You need to make sure that you are recognizing all those people. I've had a number of conversations in Apache communities where we've had to basically say, why isn't this person over here a committer? Why haven't they earned a voice in the project? Well, they're not a coder. They haven't contributed anything. It's like, really? Because every time somebody stands up over here and they say, I don't know how to do this, they're in IRC handholding. That takes time and it takes skill and it's not terribly rewarding, especially if the project you're trying to help doesn't recognize that as a contribution. And the last thing I want to talk about is that users matter too. Why are we doing this if the users don't matter? But there are a lot of projects when a user rolls up to the door, hey, I have this problem. Patches welcome is a hostile response. It's a friendly response when you have a developer that rolls up and says, hey, I found a bug and I think I know how to fix it. Hey, your patch is totally welcome. That's a friendly response. When you're talking to somebody who probably doesn't know the sharp end of an IDE, then you probably are going to, when you say, hey, I have this problem and you say patches are welcome, you're basically telling them pound sand. That's the nicest way I can put that, by the way. You're not giving them a friendly response. So we've talked about some of the things that are important in general, kind of more vague and soft things. But let's talk a little bit about what success looks like. Because, again, I've worked with a number of different companies and projects with inside companies and you go to them and they say, well, we want community. Great. What kind of community do you want? Who is important to you? What users matter to you? What developers matter to you? And they get this deer in the headlights look, which is basically like we want community. Isn't that clear enough? No, it's really not. You need to know what success looks like. You need to know what your project goals are. These look different for different projects. A successful and healthy Linux kernel community is drastically different than a successful and healthy LibreOffice community. The number of people who use the Linux kernel is larger, probably, than the number of people who are using LibreOffice. But they use it differently. You interact with that differently. And the number of people who can contribute to these communities differs as well. And a healthy community is very different. So what are you building and why? Some companies, for example, really don't care that much about having a core contributor community outside their company. But they care deeply about having a lot of users. And so it's, if you're building a community that you're trying to attract users, that's not the same thing as building one to attract core contributors. Another thing that's really important is making sure that your goals are actually measurable. More is generally not something that you can deliver convincingly. Okay? How about twice as many downloads in 12 months? That's something that you can deliver. Or twice as many patches in 12 months. Or three other companies contributing to your project. Those are metrics that you can actually track. Okay? If you're doing this in a position like mine, you also need to make sure that you communicate the goals often and to everybody. The reason I stress everybody is because it's really easy in a company, especially larger ones like Red Hat or Citrix or other ones, to tell the salespeople, here are your goals. And tell these people, here are your goals. And tell the PM group, here are your goals. And then tell the marketing group, here are your goals. And they don't look alike. And if you don't tell PM that part of the goal is a healthy community, then they're going to ride the developers to deliver features without any regard whatsoever for the work that it takes to work with external contributors. When I was with Citrix working on Apache Cloud Stack, for example, we had folks who, you know, would go out to customers and they would promise features in a certain timeframe. Regardless of the fact that those features needed to go through an open source project with very strict governance and with contributors from other companies that had a vested interest in the development of that project. So they would target some features that conflicted with folks from storage companies, for example. The folks from the storage companies who also have a voice in the project are trying to do thing A, and Citrix is trying to do thing B, and they don't quite mesh up. And the people in PM at Citrix wanted to just push things through regardless of what people from company B thought. That's a problem. It was a problem for the people working on Cloud Stack on the Apache side. It was a problem for the developers, and it was certainly a problem for the project. So you have to be really careful about those kind of things. There are projects that are successful accidentally, and it is really important to make sure that they are not held up as poster children. I don't want to name any projects specifically for names changed to protect the guilty and all that, or innocent, no, guilty in this case. But there are some projects that just happen to be in the right place at the right time and take off up and to the right very violently, and people say, oh, well, they must be doing everything right because look at all of the success. Not necessarily. They may just have gotten very lucky. So you need to know the difference. I want to talk a little bit about governance and community, but I'm going to pause to take a drink of water because this is my second talk of the day, and I'm starting to lose it a little bit. Talk amongst yourselves for a second. Okay, y'all are quiet out there. So governance and community. One size does not fit all. So what works for the Apache Software Foundation projects, for example, may not work for Gluster or for Fedora, for example. You need to have very clear guidelines on how things are done. This is both so that people are consistent and also because a lot of new contributors, even, I mean, I've been involved in open source now for a long time. I've been using Linux for 20 years, and for those of us who have been immersed in it for a while, the idea that you are empowered to do things in open source communities seems like it should have made its way, but it has not. People still show up to projects and they feel intimidated or they feel like they don't have permission to do things. And so it takes a while to make sure that people understand that yes, you do in fact have permission to do things. And having public guidelines on how those things are done is very important. You also need, and I alluded to this previously, you need to have clear guidelines on how someone becomes a contributor officially, so that they are applied equally across your project. How do they get core commit rights? How does somebody earn the right to just commit to master or whatever without any supervision or hand-holding? It's important to make sure that people know that. How do they do that? Also important is that your governance thinks about inclusion and diversity within your community, and make sure that you have something in your governance. I'm not going to say that every project absolutely has to have a code of conduct, but it is not a bad idea. And that code of conduct should specify, at a minimum, a level of behavior that everybody can live with. Any questions? Y'all are so quiet. I'm not sure if I'm making all of my points so well that there are no questions or people are just bored. So I'm going to assume it's the first one. A couple of very low-level governance suggestions. So everything happens on the mailing list. We've kind of already covered that. Give 72 hours for feedback. Give or take. Your community may differ, but I generally think 72 hours is long enough that if somebody sends out something to the mailing list and asks for feedback, 72 hours is long enough to be reasonable for people to have checked their mail. And I would put maybe a little asterisk next to this that says 72 business hours. Obviously, it's not cool to send something out on December 23rd and then make your change on December 27th or whatever. You need to make sure that it's a reasonable timeframe for people. Don't do it on Good Friday or something like that. I'm a strong advocate for lazy consensus. Is everybody familiar with lazy consensus? No. All right. So lazy consensus is basically the concept that send it to the mailing list. Send all the information that's needed to make a decision. If nobody squeaks, then you have consensus. So in other words, you're not trapped waiting for five people to act it and say, okay, you're setting forth the expectation that everyone is going to read the mailing list that cares. And if they have an objection, they're going to speak up. You don't get to revisit decisions three weeks later when something is already halfway implemented because you've decided now I have a question or now I have an objection that I didn't think of before. There are exceptions. Red Hat Legal, I'm sure, would like me to especially stress the legal angle. Lazy consensus does not work necessarily for things like trademark or license changes or things like that that require explicit permission from someone. So if you're in a patchy project and you want permission to use a trademark for a conference, if your company wants to run a patchy foo conference and you need to get explicit okay for that, lazy consensus doesn't work. Security things. You may need to push something out faster than 72 hours to fix a horrible security bug. So where common sense applies, the 72-hour rule can be set aside. Really important and puns sort of intended, you need to leave your hat at the door. No one likes participating in a community where one company gets special privileges. Nobody really wants to work with somebody who's clearly only vested interest in a project is whatever their employer wants. You need to be invested in the good health of the community, not just the health of your employer. And I can't stress that enough. You need to work with people you can trust that you know are going to do the best thing for the project. So any project you're participating in, nobody should come in and say, well, company foo says this or company blah says that we must do it this way. I already alluded to this a little bit, but do make decisions stick. It's both people will, some people have a habit of liking to revisit decisions given a couple of weeks. They'll bring a topic up again and hope they can get the group to change its mind. And also people, you know, people rotate and so people forget or weren't aware of decisions that were made. So one, find a way when decisions are made to record them somewhere so people know what those decisions were. Again, this is where everything happens on the mailing list, assuming you're using some sort of archiving software is very useful. But you need to have a culture where once a decision is made, you don't revisit it every two weeks. So document everything. Infrastructure. I won't spend a heck of a lot of time on this, but this is pretty obvious stuff. Every project should have mailing lists. It should have forums, maybe. I put a question mark there because a forum where no one is watching for questions is worse than one that doesn't have a forum at all. You probably need a bug tracker of some sort. You need documentation or a wiki. I like to say wikis are where information goes to die, but they can be useful for certain things. Obviously you're going to need a code repository if you're shipping code. Yes, sir. Hey. Okay. Sure. I mean, there are a number of different checklists on the Internet from different sources. Okay. Okay. Cool. Yeah. Also, I think I have this in the resources at the end, but you also want to look at Carl Fogel's work as well. Yeah. Okay. Cool. Thank you. You may want to also consider something like Trello or Kanban boards. I don't know how many folks are using those. I like them a lot because I like the visual nature as opposed to bug trackers and whatnot. A lot of projects. Now, this wasn't the case when I started doing open source stuff a long, long time ago, but a lot more projects are finding a deep need to have continuous integration and testing and infrastructure. If you are considering starting up an open source project these days, you may want to start from the idea that you're going to use CI when you're doing check-ins as opposed to, well, we'll just throw stuff together and assume that people are testing it. Then once every three months, we'll try to compile everything and then start debugging from there. I've been part of that culture too. It's not fun. Yeah. Translation tools. Depending on your project, you may want to consider whether translation tooling is necessary. Fedora and other projects, for example, make heavy use of translation. Quick shout out, since we are in fact at a conference, quick shout out to the usefulness of doing things face-to-face. They have yet to invent a technology to ship beer over TCPIP. It's a tragedy, but you're working on it. Good. Good. But there are very few things that beat buying somebody a beer in person and thanking them for their work. Meetups, lugs, daycares, maybe not daycares, but definitely getting to talk to people face-to-face. You find out, as a contributor, that people have questions you never dreamed of and you will learn a lot about the way that people use your software, a lot about the nature of your audience that you may not expect. One example, so I worked on Project Atomic with Red Hat, which is a container operating system, basically, and found out that people were using some of the technology, not because they really cared so much about running containers, but because they found the RPMOS tree stuff really good for Internet of Things devices. They wanted basically a way to push out the operating system and reboot without having to fuss with RPM updates. They just wanted to push out an update as one entire unit. We didn't see that happening, so we learned that by going to meetups and people coming up to us going, you know, we're doing this thing, and that's great. You also actually get to talk with other people who care about your technology, which, you know, depending on whether, if you're a remote worker and most of your friends, I used to live in St. Louis and most of my friends were involved in art groups and so forth, and so having a conversation about technology really didn't go very far, so I found that when I would go to a lug or some sort of meetup, it was like, holy cats, people actually, right, this is why I do this stuff. And then you can make sure you bring it back to the community. Don't hoard your information. Get into the habit of doing trip reports or whatnot. If you go to this, if you work for a company and you're at LinuxCon and it's not already a requirement, I strongly recommend that you make it a requirement when you come to one of these things to write up a trip report afterwards about what you learned, what you did while you were here, and share it with all of the people who didn't get to go to this event. Because there are a lot of people at your company, assuming you're with a company, that will not have gotten to learn anything here, and this will make the most of your company's investment and show why it's useful to send people. Yes, sir? Question on that. Yeah. You can comment on those. Okay. So at Red Hat, people do read them. I mean, when we send them out now, it's not 100%, right? I mean, it's usually, if I know there are 300 people on a mailing list, it's usually the same 20 people that will actually give me feedback on a trip report. But I guess the way to make people read it is have a culture where people are curious and want to know. So that probably doesn't really answer your question. It does, actually. Well, sure. And sending it to the right place, hey, start using the upworthy style headlines. Like, I went to LinuxCon. You won't believe what happened next. Do whatever you got to. I'll read it if people put cat pictures in there. All right. Marketing promotion. So this is a good topic, right? Again, audience and goals. This is deeply important.  Okay. If you don't know what peeps want, you can't give it to them. If you don't know what they care about, you can't craft messaging that works for them. So I recommend that people come up with personas of the people that they're trying to reach and keep them updated. Okay. Come up with personas. One thing that I have seen that is also very important when you're writing those personas, pro tip, they should not all be white males. Okay. So if you give names to your personas, if you, which I recommend, make sure that they are a diverse group of people so that when your developers are looking at these personas or when your teams are looking at these personas, they are thinking about a diverse group of users. Okay.  You do this by talking to users and not just assuming. Okay. Any questions on that? Yes. So the question is, are there less anecdotal ways to collect information than personas? Would that be a fair summary? Okay. Yeah. I mean, you can do user surveys. For example, the OpenStack Foundation does a survey every six months, which is both a blessing and a curse. It's a blessing because it's great information. It is a curse because they change the questions every six months, which means you cannot possibly use it for tracking things between OpenStack releases, which is infuriating. Not that I've run into this and it's made me annoyed or anything. But basically, yeah, you can do user surveys and you can gather statistics and so forth. And that's a good thing to do as well. Definitely. A couple other things, too. You obviously want to tailor your website to users, which means that maybe your lead developer is not the person who should write copy, or at least not the last person who should touch the copy before it goes to the website live. Have someone sit down and develop talking points and messaging for the product, for the project. Also, this is a great exercise if you are trying to get a larger audience. This is a good exercise when you can sit down with people and say, okay, what's the story for this release and worry if they do not have one? If you have a release and there is no overall theme of some sort, if you have a release and you can't produce a coherent tale with it, there's probably a problem there. Whenever you're writing about your project, the worst press release I think ever received when I was a reporter was a press release that said something to the effect of this release of software foo implements JSR, blah, blah, blah, and JSR, blah, blah, blah. I don't follow JSRs. That's basically like sending me the IETF codes or whatever for something. It's like I don't walk around knowing what the TCP IP over avian carrier number is. At least tell me what the name of the standard is, not just the number. Yeah. So show me the features and the benefits of the features. Don't tell me, for example, with Fedora we want to talk about Wayland. Users don't care about Wayland. Users care about X isn't going to crash as often. They care about fonts look prettier. They care about faster. They don't care about an obscure technology, generally speaking. Some users may care about Wayland for its own sake, but they probably already know that it's going to be in that release. So we're actually looking at making it the default in Fedora 25. So there are still some things that need to be implemented in Wayland and basically we are at that point now. I was just reading about this this morning on the plane, actually. We're at the point now where we have reached the most things are implemented, but the only way we're going to find out what's really broken is put this in front of enough users. So this is going to be the good news is we are going to make it easy to fall back to X for Fedora 25. So yeah. Well, we know how responsive Oracle is to the community, so I'm sure that will happen. Excuse me. I didn't say that. All right. And the last most effective thing is the best marketing is your tool lets me be successful at something and quickly. The tool that you install and can do something useful with in five to 10 minutes is the thing that you want. Okay. There are certain things that some users will stick with something because the end goal is so great they're willing to put up with the pain. But generally speaking, if I install something and I can't be successful with it quickly, I'm moving on to the next thing. The slide that I actually added in the plane that got dropped was and I wish I would have remembered this at the beginning but was basically all of this is academic if your project sucks. So no amount of makeup on the porcine creature is going to actually make it good, you know, if the project isn't useful. Okay. So that's step one. So if you can't be successful quickly, part of the process of marketing is going back to the developers and the rest of project and saying, folks, we got a problem that we need to fix before we can go out and tell people this is wonderful. Blogging and social media. This is always like pulling teeth but it's really important. Take the time to write about what you're doing. If you are a developer, spend just a few minutes occasionally writing about what you're doing. It doesn't have to be, you know, deathless prose or anything like that. Although I do appreciate it when developers actually find the shift key. Yeah, I'm thinking of a certain KDE developer who did blog prolifically but his blog looked like it was written by Hello Kitty. So take the time to write about what you're doing. It doesn't have to be John Irving but it does need to be out there somewhere where I can find it and follow what you're doing. If you are younger and like the videos more than the text, just sit down in front of your webcam and talk about what you're doing or create a screen cast that people can watch. For extra bonus points, really successful projects will do both. They will create screen casts that show off the technology and for old fogies like me, they will write text that goes along with it. Okay? They will reach out to both audiences. Again, this comes under the heading of people who can contribute to the project who are not technical. Find somebody to run the social media because odds are the developers on the project don't want to spend their time feeding Twitter. But it actually does help to have a active social media presence. Okay? Provide social media guidelines so people know what the voice of your project is and what to and more importantly what not to say. Okay? For example, your open source project probably should not be bashing other open source projects. That doesn't look good on anybody. Okay? For really advanced stuff, find someone to run an editorial calendar. Find somebody to do SEO on your website. Okay? Target people who are influencers in your industry. Releases. This is something that should be obvious but isn't. Start planning your release announcement early, ideally before the alpha is ready. Identify any publications that might be interested. Identify partner projects that might promote your project. So for example, Fedora and Python go together like peanut butter and chocolate. And so whenever a new version of Fedora comes out, the Python folks should be talking about all the stuff that we're doing in Python in Fedora and vice versa. We should be talking, and we do in Fedora, talk about all the stuff that we're doing with Python and the benefit to Python developers. Go big but do not just focus on releases. If your project only releases twice a year, that doesn't mean you should only be heard from twice a year. Okay? So summary and I think I'm getting close to out of time. You're never going to be done with tending to your community until they, you know, put your project in the attic. So a little bit about the Apache Software Foundation incubator since I skipped over that earlier. The incubator in Apache is a project that helps new projects before they become a top-level project and can govern themselves. The idea is that when a company or a group of people come to us and say, we would like this to be an Apache project, we say, great. Do you have this many people from different companies who can be involved? Is it going to, you know, can you put out releases? Is there enough steam behind it to actually work? And we work with companies or we work with groups until we see that yes, they can govern themselves and then we promote them to top-level project. And sometimes projects basically live out their lifespan and are no longer interesting or don't have enough people contributing. And at that point, we have a conversation about, is it time to put this in the attic? This is really important, okay? Because this gives people a chance to say, is this project done? Sometimes the project is, you know, there are open source projects that live out their life cycle and they're not that useful anymore to people. And it's time to signal to the remaining users like, hey, we're going to stop maintaining this. If you were expecting security updates, that's going to stop in September or whatever. Start making plans. Because it's just as important to let people know that something is coming to an end as that something is beginning. Successful communities change over time. Thank you. Thank you. Things have a beginning, a middle and an end. And in the middle, things may blow up. You know, look at Docker as a good example, okay? When Docker started, it was this tiny little thing that was part of a platform as a service company. And then all of a sudden they realized, hey, this is more interesting to everybody than the platform as a service that we're trying to work with, okay? And so that changed radically and continues to change. Single company projects are okay. I'm not going, there are a number of single company projects that are all right. But they're never as good as multi-company projects. Sorry about that. If you build it, that does not necessarily mean that they will come. You do actually have to go out and promote it. If you're not growing, you are dying. And the golden rule, treat people the way that you would want to be treated when you're working in a community. Any other questions? I have like four minutes for questions. Yes, sir? Yeah. Well, then you can revisit things. I mean, it's reasonable to revisit things when the world changes, right? It's just not reasonable to revisit things because either people have forgotten the original decision or because somebody didn't like the original decision. So knowing the difference between those things is really important. So does that answer your question? Yeah, yeah. So for example, you know, we should base this on this version of Java, okay? People may say, no, we don't want to base it on the newer version of Java because that's going to be painful for users. And then Oracle comes along and says, we're discontinuing the old version of Java. Time to revisit the decision, right? So yes, sir, back there. Actually, well, I think it depends partially on your audience, right? So the Linux kernel community is an example. I don't know that anybody that many people in the Linux kernel community would be terribly motivated by a leaderboard, right? However, in Fedora, we do see people who, for example, do package tagging and other things because there are badges attached. I have to admit this is really kind of terrible for me to admit, but I'm actually sort of motivated by those things. So how many people use an app called Untapped? Untapped? Yes, for beer, yep. So I'm actually a little motivated to get new badges on Untapped, and I will try beers that I wouldn't ordinarily try. Like my favorite, I know what my favorite beer is. My favorite beer is Paul and you're hip-wisin. And so if that's on tap, I'm like, that's what I want. But then I'm like, oh, wait, I'm in a new city and they have different things, and I can probably get two or three badges. I'm going to try this. So I'm actually motivated by that a little bit more than I should be. So it really, really depends on your community. And if you want something to implement that, I would recommend looking at the Fedora badge project and our unfortunately named Fed message bus, which is not actually Fedora specific. It can actually be used by other projects. But Fed message actually will watch things happen and let you collate data for that kind of thing. So I think at this point, I am actually out of time. So thanks everybody for showing up. I really appreciate it.