I just wanted to say that we have a tentative date set for our next event. We are looking to do that event in March, probably towards the end, so just have a March break. And once that's all complete, we will send all the information out to the other registry. We have details surrounding that event. We are looking at the topic of the discussion for that event. We'll be talking about hyperledger and blockchain, how it's used in today's supply chain industry. We will have some guests who are from the industry, and they will be speaking to us and giving us their insights and perspective about that topic. And just another second part of this is I just wanted to say is that there are some members, some people here tonight, guests that are not members. If you are interested in becoming a member, please see me at the end of the night, and I will get you the information that you need so we can get you signed up and registered and signed up to give you information about the next event and any other information that's for hyperledger foundation and the registry that you guys may want to be interested in other events. So for tonight, what we'll do before we get started, I will go through some slides just real quick for anyone who is new here. Just a bit of an overview, a good overview of hyperledger. And we'll move together. Sorry, you want to take the screen up there. Yeah. Okay, so here is hyperledger foundation hyperledger foundation is an open source community focused on developing the suite of stable frameworks tools and libraries for advice green blockchain deployments. It's, it is a global collaboration hosted by the next foundation, finance, banking, Internet of Things supply chains and manufacturing and technology. Sorry about that. Here are some of the foundations goals, you can have a look at the here and just click the plan section, and you can see. Next slide. So here are some of the hyperledger blockchain technologists with business here the graduate projects. I think most of you should be familiar or have seen that your fabric. That is the one that we discussed in tonight. In our talk, and then, we're going to be creating hyperledger projects. And again, probably noticed, we've seen hyperledger firefly and hyperledger caliper so here are some of the ones that are indicating, and we'll move forward. And here is all the information if you need there's a website.  And also at the end of the night, if you're interested in the PDF, I can send one out for you that you can have that for the reference and also have all the information that's on. So, for this, so pretty great. So I introduced it's a pretty same is the CEO again, so he will be discussing tonight. The topic that we mentioned. So great. First week. You have this, you have to share this one here. Take a look at it. Which light is it. Take yours. Take your, take your screen. This is your screen. I'll put it up there, but you need to put it up there just go ahead and share it. Let me just go ahead and start playing it. My husband. So I'm with the hiccups. Virtual and the first one. Again, so information for the company. So pretty much with it and it consulting. I'm going to talk a little bit about myself as an entrepreneur and how to experience that I am the IT industry comes specialized and just a transformation and twice black team. My passion is just technology. Okay, so we're going to do this. What we're going to do is we're going to talk about what we're going to be talking about without blockchain and technology which is more of a, I guess, not technical perspective or I guess, but in order to explain it fully, I thought that, you know, we're going to go through the basics of the architectural design and then we're going to do more of a comprehensive discussion such as why should we watch and show these blockchains and failing problems. So hopefully by the end of this visitation, you guys have a better understanding of the topics that we're talking about. We're going to go through was watching. I know most of you guys know it's going to go through the blockchains as well. Our blockchain is pretty much right. It's a transaction, a lot together and pretty much pretty much referring to the previous bar. So, all the transactions I have so as quick to go through this was a house is just a data set that's transferred into a single stream of things are no matter where our characters. So for example, you take that and we hash it you're going to get this hash. If you take the same wording, just kidding, but uppercase, maybe not changes the house so they understand that any type of data change, any type of data change and characters change the hash. Now, we're going to go to this. So, we're going to go to the blockchain. So first you have the Genesis block, and that's the first one. And with the first block was going to happen is that it's going to have a hash. The next box is going to have a bunch of transactions that are going to be put together and hash and that hash is going to be evident for that. So that has to be that block. And then we're going to refer to the previous block. The next is like an exact game that have another bunch of transactions that are going to be locked in. And that's going to be hashed again. If, for example, one of the transactions in the block, sorry, one transaction was about to change the compromise, you'll notice that the block hash had changed. And so if you look at the, if you look at the, if you look at the previous block, you'll notice that it doesn't match. So you obviously know that it's going to compromise.  It's pretty much is this a contract that's decentralized. Identifies a agreement, sets of conditions, code, business logic, and pretty much is going into the blockchain was used to. So, now, for the second part of what is the distributed ledger so pretty much what happens is that every no participant not watching it's going to have a copy of that. Because that is about that. So it's going to happen that anytime anytime the consensus is going to change the ledger and also the state, what's going to happen is that's going to be synchronized all the ledgers in the system. Well, watching to be able to watch it's a book about this. First, permission blockchain, which you can progress to the cryptocurrency, anyone has access to it. And all the permissions public second one is by the mission. This is customized blockchain here it's all the access control you have to have permission in order to join the blockchain. Then we have the hybrid, which is obviously the combination of the two. So now we're going to do is going to go into a brief about the hyper ledger. Currently, we're talking about hyper ledger properties because right now it's kind of known as the most mature. It's watching that there's also this popular term that's used that device. So we're going to talk about how to see every every organization in the blockchain is issued in the digital certificate. When that is just to figure out what happens is like that's pretty much your own. That's public key and your private keys. With that permission, an organization is able to create the network. The second part is the MSP. The MSP is pretty much defines the access to the different parts of the network. So if you're looking to read the public documents, you can pretty much compare the two as the CA being credit cards, American Express, Mastercard, and Visa. And you can think of the MSP as the merchant, not on merchant state American Express, although that although American Express is a valid part, but yet it's not accepted at that merchant. The next part is channel configuration. So configuration of the network is usually done through this and also you can do for others the history of tasks. Another thing that's very important is policies. The policies are pretty much used. These are conditions that are pre-defined that ultimately decide whether the channels actually be downloaded or not. Channels is a pretty much a mode of communications and also they can isolate it so different organizations could join different channels in order to communicate. The already know all to that consensus. So in order to have a valid transaction, the consensus will actually validate that transaction therefore send it into the legend. So here those are actually very important. Every organization has their peers. The peers are pretty much used for evaluating transactions, maintaining the state vector communicating with the rest of the network also holds the smart contracts. And the last one we have is smart contracts. I think you know what that is. There's a big role in here. So in fabrics that we call smart not just chain code. And so with this what happens at this chain code allows to validate other transactions update the legend and create it. And the last part is the lecture. So we have to understand in fabric, the ledger is actually composed of two things. It's actually composed of the state and the blockchain. But if you look at the state over there, you start to mistake. The state is a current state value of the watching that can always change. And the blockchain is actually all the transactions that have been performed whether they've been valid or not. And that's a miracle that is a change. Okay, so the next part we're going to go into before I go into explaining all the information we're going to do is pretty much explain which is why I use the hyper-legit topic images is that we saw go and try to read the documentation information kind of you have a better feeling better understanding if the diagnosis match. Okay, so what we're going to do first is that you notice the C1 C1 pretty much no C1 all of them are purple. So pretty much representing that all the organization one is going to be represented. So all of these all the purpose represent one organization. So you notice that you have a senior there in the digital search fields. The P1 actually represents peers and A1 actually represents the application. We'll get to what the application is a little bit. Obviously you notice all the organization too is represented by the other topic. O is the consensus which is the order and we have it which is one of the portfolio which is the channel configuration at the bottom over here and we have C for the channel. And obviously everyone that's on the channel is also going to have the ledger, sorry the down ledger press. Also, I want to explain one thing about the A1. A1 is actually representative of the application from the client side. So that's actually, for example, the ERP system of the existing companies. For example, Walmart, they actually use their own ERP system and they're going to use that right here. You'll see here there's going to be an STK fabric, sorry fabric STK which allows them to communicate with the network, right. So we're going to go a little more. So how do you create the network? So the first thing to do to create network is that what you do is that you create a configuration channel. With this configuration channel what it's called is called crypto, crypto, crypto file. Within that file what you're going to do is you're going to list all the organization, you're going to list the MSP directory. Also what you're going to do is you're going to put in the policies and you're going to see the direction of where the order is. All the files that have been used for the configuration is going to be the order YAML file which gives the configuration for the order and the peer configuration which is the core.yaml file. So I want to say another thing. And here in essence what happens before a configuration is actually launched, what's going to happen is that all the organizations are going to get together and they're going to decide pretty much the policies. How are the policies of endorsing policies and including policies and also any other configuration policies that are needed to run this channel. After that what's going to happen is that you're going to, what happens is that the admin is going to initiate or is going to create a channel. Once the channel is created it's going to add the order and the two peers. After that you'll notice that all the peers and the order have a copy of the venture. Well one thing I forgot to tell you, prior to this, what we say in here, what I've noticed is a lot of the currently there's examples of creating the CAs. A lot of people notice that they use crypto jet in order to create the CAs. However YAML said that that is actually a test environment. What the best thing to do is to actually use the fabric CAs or any other CAs that you can use. That is the most best and secure way to do it. The next part is we're going to launch, we're going to launch the smart contracts to different peers. This could be done, the CLI, but that's not necessarily recommended. It's recommended to actually have the applications integrated and through the SDK to actually have it to post the contract on the peers. If you want to add a channel, if you want to add a channel, it's going to happen that you're going to have to create another channel configuration. And it's pretty much the same process that we talked before. What's going to happen is in that fall you're going to have to pretty much list all the organizations that are going to be in there, all the policies and also the list of the directory of the SDKs. If you notice after that we're going to create a channel again and we're going to add the peers. If you notice in this channel currently, what has happened is that we have P1 and P2 organizations to be in two. They're connected to the channel. If you look at this, they are the only ones that haven't been able to communicate within that channel. Notice that P1 is not within the channel. So what happens in lessons, P1 or organization one has no access to that nor can they read the lecture. And you notice at the top P1 and P2, you'll have L1 and L1, which is representative of channel one. What is the order? So what is the order? It's the consensus. If you notice that the order is actually connected to both channel one and channel two. So when you notice that P2 has L1 and L2, what that means is that it's able to, it has the ledger of channel one and channel two. Also, you notice, keep this in mind at the end of the presentation because I'm going to bring it up again. Notice the order has both ledgers, right? So it's able to have both ledgers. But also take another thing is that the only way to, there is a way, another way, there is a way to query the ledger is if you have a smart contract. So now, say for example, you want to add, you want to add organizations three to channel one. So what's going to happen is you have to change the configuration for channel configuration one. So you have to go into the config TX file and you're going to have to add that, that organization to that channel, right? Once it's, once that's added to the channel, there's a, what organization is going to do is going to, through the SDK is going to launch smart contract, which is on the next one. Now, officially, P3, which is organization three, has access to both C1 channel one and channel two. However, notice that organization one still only has access to C1. So they don't have any communication here, nor do they have access, any access rights to here or see any of the information. So the transfer, how does this transfer actually occur?  Yeah, so what's happening is that who need, for example, a one organization one, and they want to perform transaction, they're going to do the SDK, they're going to invoke transaction limit smart contract. And what's going to happen is that that transaction is going to be sent to all the peers in the network. So what's going to happen is that that transaction is going to be sent to peer one, sorry, peer two and peer three. What peer one and peer two, three is going to do is they're going to endorse it. What they're going to do is they're going to check the chain code and also the endorsement policies of that chain code. They're going to also then check the judges to see if all the information is valid. If the information is valid, they're going to sign it with a digital signature and send it back to the application. The application is going to receive all the all of my files from P2 and P3. Then it's going to take that information and it's going to give the order of service. Once the order of services receives that information, again, it's going to check the endorsement policies and it's going to check whether these transactions are valid or not. But remember, at this stage, they're only checking the information from just the chain code side. Once that they validate that they're going to block information and then they're going to submit it to the committing peers. One thing is, don't think of the committing peers and endorsement peers of two to practically. Peers have multi, I guess, responsibility. So in essence, it could be the same thing. So now what's happening is that now those transactions are sent to the committed peers. Once it gets sent to the committed peers, they're going to be checking the endorsement policies, the ledger. And what they're going to be doing is they're going to see, for example, endorsement policies. For example, if that transaction needed the signature from peer 1, 2, and 3, what's going to happen is that since it has all the signatures, within that block, it's going to check, okay, does it have the three signatures, right? And also check against the general policies. So then what's going to happen is going to tell them whether it's valid or not, and if they're sent to the ordering service. The ordering service is again going to check whether that even meets the policies, whether they have significant signatures. If so, what's going to happen is it's going to send it to blockchain. Once it's sent to the blockchain, it's going to be distributed to all the nodes and the blockchain is going to be updated. So now we're going to go, we have any questions about that or not? All right. So now what's going to happen is that we're going to talk a little bit about the benefits of hypernetics. So obviously there's a lot of benefits. Some of them are that it's transparent. It's mutable. It has lost security at various costs. Okay, so we're going to quickly go through this. We enforce it. Obviously it's mutable because of the hash, right? Any, any, any information that has been changed is going to change the hashing. And then obviously you know it's going to compromise. And then obviously adding onto the distributed letter is very difficult to have much compromise information because all the nodes and all the participators are always going to have the latest and updated legend. So transparency. Uh, how to transfer that endorsement. Those are policies that remember before creating the channels all the organizations have to be upon the policies that they're going to have. So these policies are actually universal regardless of what organization it is or not. So this amount of organization, one, two or three there's no special treatment for either world. The pre-determined conditions have to be met. Second part is security. Why is it secure? Because everything's access control. Anything, any information you want to get, whether if you want to get into a channel, whether if you want to, sorry, you want to get a channel if you want to, if you want to create the ledger, everything's access control. Without access control, you're not going to be able to get the information and encryption. This is a very important part because all the transactions are encrypted, right? So, for example, a hacker will, it's very difficult for a hacker to get in because all the transactions are encrypted. So, all the organization, any transactions that they do, they have to sign off with their CAs, the private key and public key. And so, we're going to actually forget about hackers, even if we have, I guess you can't show it, but if you're not a part of that channel, you're not even allowed to see the information because of the encryption, right? So, it's very secure because of the encryption. Apart from transparency, security, the utility. So, this is all due to the consensus we talked about before. Since the consensus, as I said, it follows the protocols, it's the rules and policies. Again, it's all universal. So, no organization's going to have a special treatment for the organization. Security and utility, reason why is that? Because anytime when they validate the content, validate the transaction, it's going to be sent to the ledger and all the ledger, all the peers are going to have the same ledger. So, it's a very secure problem. Another part is the private data security, sometimes within a channel. Sometimes organizations want to communicate with each other. So, and they want to pretty much not withhold, but they want to keep up or withhold confidential information. In that situation, you can use private data, which is the private data collection. So, pretty much the information is going to be encrypted, right? And if one of the people that can receive it, for example, it will be pure one, appear to, on the ledger, you'll be able to see the information, but the information that pure one or pure two or organization one or two, wanted to encrypt or to hide would be encrypted in the private data. So, although you're in the same organization, for example, organization one, two, and three. Although organization three is in the same channel as the other two, yet it won't have access to that data because it's encrypted in the private data. Then also we have security. We won't go too much into this, but everything wants to smart contracts or smart code or chain code. And so whether you want to query the data, whether you want to post a transaction has to go in chain code. And all the chain code is also related to the digital circumstance. So, what benefits do companies have? What it reduces costs? Why is it automation?  There's a question. That's a question. Okay. Hold on. There's some back here. Okay. Yes. Yes. So if you want, I just, let's just take a look here. What is the goal? And then after that is there an integration platform similar to BlueSoft WebSphere that can be used to develop and run managed hyper-ledged components within the enterprise? And can I see this getting, and I can see this getting messy as a group of enterprises adapted? To tell you the truth, I'm not so familiar with those points right now. There, I guess, in and of works in is different partners that we have, which is how you go about it too, to where it was. Are there any tools to help and deploy and manage that work? Yeah, that's the thing, just give it an update. Okay. Okay. Can chain code be updated? Is it different from smart contract in the public one that can be, that can't be updated? Well, yeah, chain code can be updated. So the process of that what's going to happen is that when the chain code is updated, it has to, it has to ultimately have to meet the end of time and then ultimately meet the endorsements of policies. So it really depends. Okay. So now from a developer perspective, it really depends upon your configurations, right? So in the configurations, you might outline well, it can and cannot be changed or it can be changed or it can be changed by organization one or two or three or whatever it is, but it can be changed. But however, any change to the smart contract has to be validated by all of the peers. Did I ask the question or? That's the reply, so hopefully. That's your question. Yes, sir. So if my side and your side and then I'll add that up, communications contract, why didn't you tell me about that before? That's really exciting. What would the problem is that we don't know and trust. I don't know who you are, but you have a seed here. No, that's a question again. So I'm from the organization department in your district. So we have a substitute in the president. So every single time there's a way to see that we get it like an answer like a substitute action problem to the drop in power. So you're doing what's done to do. I'll think of you here. Yeah. The question is really simple. So an untrusted environment, simply change itself, where it's a strategic property. So I haven't, right, I think no one knows it's quite kind of complicated problem. However, here I have a seed, lots and you can't even the roots, the other things that's here. And the question like, I haven't tried things like anything. And you've got a substance. Get them and then you're in trouble. Why didn't they have a consensus for it? The CAAs are just means of communication, right? So say I have a time, you have to remember. I do. The CAA means next five to nine. Yes, yes, but remember, you have a CAA, all that means is you have a public key and private key. Pretty much as you have a public key and private key, you're just able to prove who you are, who you are, right? However, in channel one, if you're not included in that MSP, you're not able to get into that channel, right? So that has to be, so for any time when you try to enter that channel, you can get rejected by because the endorsement policy will first check or you're part of that channel, right? Because endorsement policy will specify the MSP. So for example, if you try to agree the leisure of whatever it is, you're going to get rejected because you're not in the endorsement policies nor are you in the MSP, it's access control. So you get what I'm saying or not? Yes, but actually my portion is not in the pending program. So I have a trust that's in the pending, I've got a trust that's in the pending program. I have a trust that's in the pending program. So would it be sufficient to not be just signing the document but the message and transmitting it? Would it be using blockchain or just physically signing it, doesn't that it? Yes, no, it does. Because we can get to that point, right? Oh, sorry, sorry, sorry, yeah. So why are you, is your question why are we using CAs? Yeah, like if you use a certificate authority, it means we're going to be attracted to each other. That's right. I'm attracted to your certificate. You're attracted to each other.  It's trustless. So what's happening is that we don't need to trust each other. So what's going to happen is that we're going to, for example, we're going to help line. For example, there's a transaction, right? For example, what are the transactions? We're going to, in our policies, we're going to say in order for that transaction to be validated, we need organization one, two, and three to sign off, right? Now, how do I know what you're going to be honest with me? It doesn't matter. Because what's going to happen is that, what's going to happen is that when you propose, I'll go back to you. If you see this, when you propose the transaction, this transaction is going to be sent to all peers, right? There's no need to trust. So what's going to happen is that, remember, we also agreed upon the smart contract. So for example, we have a smart contract. I'm going to sell you a product, 50 cents, right? Now that smart contract has been agreed upon. That can't be changed, right? How does that be agreed upon? Because that was proposed, that was accepted, and now that's on the legend, right? So if you were to try to change that information, for example, which we're going to change, a car for $5 or a product for $5, if you're trying to put it, okay, now it's $2, it's going to get rejected. Why is it going to get rejected? Because if you look on the ledger, and we look at that chain code, in that chain code, that number is actually $5, okay? So there's no need for us to trust it. But from the business perspective, we agreed on $5, and we cosigned it. We said, overpriced it. But we said, we said cosigned it, but like that's what I'm saying, we're saying cosigned it. So every time you try to change it in time, we should trust it, private key, so I can obviously trust it, you can trust it, right? Yeah, but what I'm saying, but we don't need, and this that's what I'm saying, that's what I'm saying, but what you're saying, we trust, how do we trust it? No, that's what I'm saying. There are certain things that it's doing, but what if the policy says something else? So what if the policy says, well, that's transactions, what if the policy says that we need three signatures? So then, but you understand, if in that policy, remember this, if in the policy to say it's only one signature, then in theory, what you can do is you can post anything that's gonna get to the edge, right, there's no consensus. You see, my point is like, and I'll be dealing on this in a minute, but that's what I'm trying to work in. I'll just point it to you in interpreting. Well, the reason why you're so specific is is that your policy, I was gonna get to the later slides, but what happens with the CA is that everything's transparent. Anything that you do will be seen.  So if you, if you do the transaction, you remember exactly who did that. How are you gonna know who did that? It's through your CA. So if they can't be a transaction, say, well, I didn't do it, or even if it's just two didn't, no, because you're the only one that has those digital certificates. It's impossible for anyone else to sign for that digital, anyone else to sign on your behalf, get it? Great, but brings if they can give you an example. I go to the band, I just start here, sign in the print. And thanks to sign this agreement, you just like another private team that belongs to the careful department. But when they go to a CA and I can see it, I don't get a pass, you can get all the certificates in the bank itself. Right, you may pass it to them. They'll be certificate. So, no, but that certificate is just authenticating that you are who you are. That's all it's going to do. So for example, we have, for example, we're gonna have me for the industry organizations, right? The street people, right? And me, I mean, you and John, right? We all said- I don't know how it works in an organization. In an organization, you don't work for yourself, you work for your CEO. So meaning like an office staff certificate could be like, you know, the best you can do on a person is actually stop the load for everything you say in your CEO, right? In your CEO. No, see, I think we're saying, so you're saying that if you're talking about information that's sent from the organization, well, that has nothing to do with, let's say, where the hypervisor, that's on the governance of that SDK. So for example, that ERP. So for example, from the ERP system, right? You're gonna submit a transaction. Well, hypervisor has nothing to do with your ERP system. That assets control is on your side of who can sign up and who can sign, whether they're a general labor signing up or a CEO, right? That has to be done pretty much on your side because remember, if a general labor or someone, if anyone gets access to your system and pretty much get access to your ERP and submitting the information to the SDK, that's pretty much on your side, get it? Because that assets control is on your side. You can't, fabric cannot be responsible for your security on your side to get that or not. For example, if you were to be able to set SAP, right? If someone, well, SAP secures your information, right? But if someone gets your password and they go in and they change it, well, that's the organization's fault. That's not SAP's fault. You can't say, well, why did you do that transition? You have to take that back, no. Well, your password got compromised because of your organization and your government and your security structure. You got that or not? There's no password and arbitration. Or standardization, right? Okay, I understand that. Well, you can make the password, but by general enterprise, you've always got to make sure that. You always got a security control. How is it more complicated? I think we're talking, we'll talk about it, but I think we're talking about two toys and things there because all the CAs does, it just certifies that if all the CAs does, it's just certifies that you are who you are, right? That's pretty much all it does, right? And pretty much, but with that, it isn't that you can join any channel, right? You get a sense.  But it doesn't mean that you have access to everything in the lecture. It's just saying that when the policies of the same organization has this access and that access, well, then you're able to pretty much transact on those rules because you are who you are. Do you understand that or no? No, we'll get to the next one. Okay, so yeah, so what we're gonna do is now we're gonna get into, we'll speed it up a little bit. So now what we're gonna do is we're gonna get into for a company. For example, why don't we be blockchain? A lot of companies think that, okay, this is great, blockchain is great, it can do everything for us and so on and so on. But the first question you have to ask yourself is do I even need a blockchain? It's not always the situation. So for example, if you don't want to try to database, it's actually doing what's supposed to be. It's hand it's security, the security efficiently, then would you need a blockchain at this point in time? Another thing is that if you as, I think what you're talking about, if you have trust, for example, in your centralized system, you have trust with your accountant or whoever it is. If you trust that person, then there would be no need for a watch because you have that trust with that person. Also another thing is that if you're, if you don't have any regulatory compliance with status courage, then you wouldn't necessarily need it. What I mean by this is that for example, take the pharmacy industry, all the data, everything that they have, they have to back that up. They have to prove that it is what it is and they record it at that time and so on and so forth. So there has to be some sort of system in place that shows that no, the data is accurate. If that is a situation that blockchain will probably be the best situation because since everyone should have seen the lecture, you can actually, in third parties, you can actually prove that yes, the information that you did submit is correct. All right, while you may need a blockchain, obviously if you want to remove the materials or brokers, another thing is that if you want permanent data set, I'm sorry, nothing that you want to permit records. For example, if you take example for a medical, right? Medical, what happens that there might have been mistreatment or misdiagnose or something that has to do with it. And so what a doctor or a hospital looking might do in order to, I guess, avoid any legal consequences, what they might do is they might compromise the data and change it so that there's no legal repercussions. But if you have a blockchain, it's very difficult for them to do that, right? Because of the distributed ledger, they don't have that central access that they're not able to do any, they're not able to do either rights, they're not able to change the data to be. Pretty much we're going to the other things that you need shared access. So with the shared access, a lot of clients, two companies need to share information. So in that situation, you're either going to use the third party, which is centralized, or one of the two organizations is going to host the data. In that situation, again, what's happening is that is centralization of data, right? You have to trust the other person, but in blockchain, since it's a shared, since it's from just shared ledger, you don't necessarily need to trust that person because it's blockchain that's going to pretty much use the consensus to validate those transactions. Does your business require performance transfer? Sorry, prevents high-performance transactions. So that's pretty much saying that it did a complexity level. If there's a lot of investment and there's a lot of components that you need, then blockchain's probably something that you want to work to. The last one, if you want to have trust between the contractors. Okay, so now we've decided that, okay, you know what? Now we've decided that, hey, blockchain is great, we want to do this. Now, what are the steps to move forward? The first thing that we have to understand is that there's no such thing as a blockchain developer that will build your blockchain. For some strange reason, we think that there's one person that could do everything for us. If we look at traditional, anytime we're about to launch a product or build anything, you'll notice by this jargon, there's a lot of pieces that are in there. There's a lot of positions, a lot of expertise, and so on and so forth, in order to get a product launched properly and successful. But with watching for some strange reason, we think that all of these positions can be covered with a single blockchain developer. No, that's not the situation, right? So building a blockchain is just like any other program. You're building a blockchain, you've got to gather information. And if you notice here, before building a blockchain, you have, before there's a lot of non-technical things, such as gathering information, identifying the scope of the project. These are more or less non-technical aspects that within the company and CEO has to make decisions of how this government's going to happen. Also identify any constraints that you require. Sometimes the transfer of the data or how you transfer the data or how you track the companies and what information you need. Ultimately, those are our legal devices. So that has to be developed by pretty much experts. Another part is designing the architectural design. That is, I guess, somewhat semi-technical, but at the end of the day, we happen to always start developing anything. We often understand what we're developing, what is the architectural design? And again, how this usually happens is that there's either a consultant hired to answer these questions or what's happened is that you hire people in your company to get these tasks done and completed. But again, what's happening in a blockchain is that we're asking that developer to tell us what to do and what to shouldn't do. For example, for the governments, and we're talking about the policies, do you ever rely on a blockchain developer to tell you, okay, well, what are the government services, what can happen, and what not, and so on and so forth? For example, is there any DevOps here or server admins? But for example, if you were asked to server admin, if a full-stack developer or a software engineer would you trust them with their tasks of launching the servers or configuring the servers? Not necessarily. It's not that that key, it's not competitive, but it just doesn't have the expertise. So it's the same thing for a blockchain. There has to be every position that has to have the expertise and knowledge. We can't depend upon just one person and everything in the blockchain. All right, so another thing, if we don't want to build our own blockchain, what we can do is we can join an existing blockchain. So for example, this blockchain here, there's an admin of this blockchain. You can be an admin of the blockchain that does not have an organization or you can be in our organization. However, in order to maintain this network, when we talked about previously, there's a lot of moving parts for a lot of different positions and a lot of things that have to be considered in order to launch, configure, and govern that blockchain. However, if you just want to join a blockchain, you might not need as many positions. So for the first thing is obviously, you're going to need to host the peer on your own, on your own node. Obviously, you're going to need a server admin for that and then also probably going to need a blockchain or a software engineer to actually use the SDKs in order to integrate with the network. But again, this is an exhaustive list. What might happen is that, for example, there might be a smart contract or a government's policy. And so what can happen is the front-end developer or the blockchain developer can take the machine language and turn it into a readable language, which then can be given to another expert, for example, a CEO or a lawyer or architect, and they can further inspect it and say, well, yes, this governance is within our business logic or so on and so forth. So when joining a network, it's not as labor-intensive, sorry, as intense as hosting the old network. Now, if you've not noticed late news happening, there's a lot of people saying, well, blockchain has failed, right? And there's a lot of things that are going on with the blockchain issue. And even the critics of blockchain saying, well, hey, I told you, this technology is not there. It doesn't do anything. We don't need this technology. But you have to understand, most of these, so you have to understand that, that most of the problems have nothing to do with the technology. It's actually the implementation of the technology. Now, let me get this further, right? For example, lack of clearing news. For example, I lost one situation when an entity, what they did was, there was a similar entity company, but they didn't want to watch it. They thought it'd be a great idea because all of the automation and so on and so forth, immutability and so on and so forth, what they did was, was that, for the CAs and organizations, for every department what they did was, they, the department what they did was, they issued the CAs for every department. That's great. But the mistake that they made was, is that if you've not noticed, everything was, everything was obviously on one machine. So what they did was, they actually used Docker. Obviously Docker, what Docker does is that, you're able to isolate different environments onto one machine, right? So yes, you have your peers, right? You have your work services, so on and so forth and everything, right?  But the one problem is, is that there's only one IT team managers. Even though we had different nodes, the access to this, server admin is one access and the developers are one team. So at any point in time, it can be compromised. If this, if we were to even separate this on a different server, or we would have these nodes on a different server, you will still have the same problem. Why? Because we still have a single IT team that has access to all the data. So although it's a watching, this situation is more or less a centralized watching because everyone that's controlling the servers that access to the information is just one entity. Another problem is that for industries where players lack technical infrastructure, right? So a lot of times what happens that there will be apps made or platforms made for industry that seem that it seems great. It seems like a way you can see a situation. For example, construction. In construction, there's many moving parts. There's a plumber, there's a house builder, there's certifications and so on and so forth. So in theory, that's a great, it's a great issue to use watching because you're going to be able to automate the systems. You don't need to have trust with each other and so on and so forth. So if you're ready to look at it, well, hey, you know what? You should do a blockchain. What I have a question, if you ever see in the construction industry, they're still using pen and paper, right? So if they're still using pen and paper, how do you expect them to create an infrastructure or create their own IT team two years of platform? They're just not going to do it. So although the ideas are great and it could be used, but the question is, is that is the infrastructure there for them? May I have your attention please? It is now 8 p.m. and the library will close in 30 minutes. If you need to register for a library card or up to a current library for the board of materials, please do so now at the customer service desk on the first floor. Please return all reference materials to the floors in which they were borrowed. Reference materials are for use in the library only. If you're using the wireless service, please end your session in 25 minutes time at 8.25. Once again, the library is closing in 30 minutes at 8.30 p.m. Thank you. Okay, so we're going to talk about the next part is that what people say, well, there's no speed, there's a speed problem with this, but we have to send this close back to the numbers. So for example, if you have a blockchain, you, for example, if you have brothers and sisters in your peers and what you want to do is you want to scan package. Well, when you scan the package, you can use a third party, right? When you're using a third party, that's kind of pretty much, it's kind of an unbiased information. So when you scan the package, do you really need all the entities to sign up? When you have all the entities signing up, peer one, two, and three, you have to understand that's going to take time, right? The peers might not be available, it might take time. So people complain about the speed, well, we have to understand the world, or in our configuration, how can configuration, where do we need these signatures and how can we speed up a process, right? Also, the problem is the architectural design. We have to understand, again, it's how we develop the blockchain, right? It doesn't make any sense. So yeah, it doesn't make any sense to make it more complicated or add policies that we don't necessarily need. And it comes back to what we talked about before, is that you cannot develop, depend upon a blockchain developer to tell you all this information. All of this has to be pretty much scoped out prior to building the blockchain. Most people think they complain about scalability. The biggest misconception of blockchain, in my opinion, is that blockchain is not a database. It is not a database. If you need to use blockchain as a database, you really have to reconsider why you're using the blockchain. First of all, the blockchain, remember, you have different nodes and different peers. Anytime you submit information to the ledger, it's gonna be obviously data intense, right? So what's happening is that people are coming, well, it's not scalable, it's not in a too slow world. If you're using data intense information, what the simple solution is, it's supposed to back to the architecture design and using a distributed model system. Does anyone know what that is or? So the distributed model system is pretty much a decentralized database, right? So what's gonna happen is, for example, you have a large image or a large data or even a very complex smart contract. Just like that, it's kind of the same concept of NFTs. Instead of sending it to the blockchain, what you can do is, you can send it to the distributed model system, have that information hashed, and then save that hash on the ledger. So what happens is now you're decreasing the amount of data that's on there. So anytime you need to invoke that contract, what's gonna happen is, it's gonna refer back to that database and how can you make sure that's the same? Well, because that database is distributed well. So whatever that contract is here, whatever hash that it has here, it's also gonna have the same hash. So it's kind of like the same thing as the distributed measure, but it's a distributed database. You could, other problems is also costs, right? Right now it's in this stages, so obviously there's not many developers out there and also that's gonna cost you more. And also lack of standards, talking about before, we don't know what the standards are, right? We don't know what the best practices are right now. And so because of that, a lot of practices are using the wrong standards, which may actually be too failure. Another one I'm gonna talk about is, if you guys not noticed lately, there's been a few projects that actually had to fail. They've been in production for a few years and they've recently failed. And again, critics were like, well, I told you blockchain doesn't work. But we have to understand, what is a blockchain? Blockchain is decentralized, equal rights and decisions and transparency, right? If we go into those situations, those blockchain that have failed, it really dig deeper. What was the complaint? The complaints of the organization was, was that they didn't feel it was decentralized. They didn't feel that they have all the rights. They didn't feel that it was transparent. And then you might ask yourself, well, how is this possible? You're using blockchain, right? If you use a blockchain, how can it be possible that there can't be transparency, can't be equal rights and so on and so forth? Well, that's very easy. Because remember, you admit to this, right? Can also be an organization, right? So particularly with this example, the event for this blockchain is also organization one, right? So what's happening is, is that, that what's happening is organization doesn't have access to channel two, right? It cannot see the information, right? Actually it can. Because since organization is the event, it also has control of the order. And since it has control of the order, it has a legend, you get it? So in all actuality, even though it's not a part of this, it can still get the information. And depending how they create the governances that have a chain code, they might have installed a smart contract, right? It's difficult to create the data because you need a smart contract. But to the CLI, you can still get information, you can still get data information from that. But then again, you don't know exactly what the governance was, right? In the smart contract, it could have said that, well, the order also is gonna have a smart contract. As soon as the order has a smart contract, you can create the data. So for example, P1 can have access to this. So example, if we have another peer we added, for example, organized zero, zero, they also become the admin, right? What in all actuality can happen is, is that now peer one and peer zero will create their own channel, right? The disadvantage in that channel will not be seen by anybody, right? But they're gonna be able to see everything within the system. So that kind of defeats the purpose of decentralization, right? Another problem that could have happened is the configuration, right? How the configuration, what the policies and so on and so forth. And it comes back to what I've talked about before, is that these applications should have read all this information. They should have access to this, understand what the policies is. But because of lack of information what the policies are, how the smart contracts work, they're just taken for face value, right? So now, as I said, this is not the technology's fault. It was the governance. So how can this be solved? Isn't private data, right? We could install private data, right? So although the order, right? Has access to the ledger, if it's private data they're not gonna be able to be able to be encrypted. And obviously if they don't have the authority encrypted, you won't have to see the information. The second thing is zero proof knowledge techniques, sorry, zero knowledge proof techniques, right? So pretty much you can allow, you can prove the authenticity of the transaction or the information, but you cannot see the underlying information, right? Also you can do as a decentralized architecture which actually can have different nodes or different channels. And then it goes back to where I originally talked about is necessary for architecture design. So if you notice that the creator of those projects are not necessarily the technology but how it was governed and how it was created. Pretty much we're coming to the end of the presentation. So pretty much what I see in the adoption of blockchain just like any technology, it takes time. As for example, oh, and that's order as you're going to be distributed. Well, no, the order is not distributed. So what happens is that the order is based on Kafka. To tell you the truth, I don't know what's in 2.1.1 because there's a lot of changes. In the prior in Kafka, what's happening is that in 2.5, that's not the situation. I think what we're talking about, we don't use Kafka in 2.5 or 2.0. So there's kind of a difference in that. Also in 1.0, if you notice that let's go back to here. The CAs are actually on the order, right? Now that they've separated in 2.0, that they actually separate the CAs from that. So there are a lot of differences in there. And also you notice that the consensus they didn't have, well, I think we could have graph, but they didn't have this in time talks. So that's the consensus difference. And so I think it's better. Yeah. I know that everything in the end or the end of the day. Yeah, so back to the question that was on there. That's all I was saying was that it's just central architecture. So what you guys are talking about is correct. And it's also, this can be stored on different independent, but it can be distributed. But that's originally what I was saying was that the failure of blockchain is not the technology, it's with the governments, right? So what's happening is that there are a lot of entities and companies are saying, well, how can we capitalize on this? And they're not only taking the essence of blockchain and saying, well, yeah, well, it has to be decentralized, has to be transparent and so on and so forth. So your architecture could be just for good, but that has to be included into the architectural design of it, right? So yeah, as I said, as the technology becomes more mature, people are gonna, as you gave me, but if you look at the 2000s, a person that knew how to do HTML or even though basic HTML was early about $100,000, but now I guess a five year old kid could be the same thing, right? Because it's becoming general knowledge and eventually what's gonna happen is is that it's gonna happen with also blockchain. People are gonna start understanding the basic technologies so on and so forth. It's gonna be general knowledge. Also, there should be more standards that will be created. So we have more standards of good practices. People are gonna start with what to do and not to do what to expect. Also, Moab is right now, it's kind of difficult to work with the CLI to pretty much your trade and configuration deploying. In the future, I think we had Dr. Koppos before, but the project's gone. And in the future, I'm predicting that there's gonna be probably a interface that's gonna be enabled to you to make it much simpler that you're gonna be able to deploy organizations and in the future of integration or interface-friendly, yeah, interface-friendly integration. And as I said, it's gonna be changes. So I apologize if I made any mistakes, but that's the end of today's presentation. Before we end this up, are there some questions or are there any questions from online? So, Kander, from every node that you create, you also will require invest on hardware as well. Which hardware comes in terms of the server, you see? Yeah, okay. So if you look at this, right, if you're talking about this network here, that network is gonna be obtained by the admin, right? So that's gonna be obtained by the admin. So all the processes are also obtained by the admin. But if you just join the blockchain and you're not creating this, it's gonna have a copy of the veteran, right? And through the SDK where you're gonna do this, you're gonna make transactions, right? But, okay, so you're gonna make the transactions through the SDK. Do you get that? So really, even though this is the admin, it is difficult because you have policies, right? The endorsement and the CA are connected to that server, right? So when you configure this, it's gonna be contributing to this, to the IP address of this. So when you get the CA, it's not just spot all the information, it's gonna be with it, right? So the configuration is gonna, obviously it's gonna define the IP address, the configuration is there. So for example, you have, I don't know if there's any IP address project running. Currently, I believe that there is. There's test environments of those. But in terms of production or something like that, I'm not sure. Right now, I've noticed that a lot of the projects are just in test mode. For example, I'm still seeing some production level hyperledger use in Docker or they're using CryptoGen to train CAs, to verify the hyperlinked project. What do you mean, verify the hyperlinked project? Verifying when you push yourself and then you go, the security check, right? That's the hyperlinked project. No, no, no, I'm not saying that there's, what I'm saying is, no, no, obviously if you have, yet we trade trace lands, there's a lot of projects out there, but I'm not saying there's no projects out there. Just saying that, that the IP's probably a project out there, right? About what I'm saying is I'm not aware of an actual project that's in, that's actually, the RRM, that's currently, that's the IP IP access. I do know that there's an, actually we're working on one right now, that's a, you're getting the IP assist. But right now, I'm not currently, I'm sorry, did I answer your question? Yeah, yeah. Any other questions? So last time, I guess any other questions from the chat? Okay, so yeah, so I guess I did answer that question. Can it be just, can we just say in a sense that it could be disputed through decentralization of having the nose on different, on different, what do you call it, on the nose? But again, then again, the two nose can conspire together. So, but then again, what shouldn't be constituted is the governance of it and how the network is governed and how the policies are made in order to secure the uses. All right. May I have your attention please? It is now 8.15 p.m. and the library will close in 15 minutes. If you're borrowing library materials, please use the nearest expression up to you. The library's computer equipment will shut down at 8.25 p.m. If you're using a wireless service, please enter the session in 10 minutes time at 8.25. Once again, the library is closing in 15 minutes at 8.30 p.m. Thank you. Great. I guess we have some interruption here. Thank you so much for your talk. I think I will speak to everybody here tonight and say that it is very informative and insightful. So thank you again. To everybody here tonight, thank you so much for your time tonight. We appreciate everybody being here. We appreciate your participation and all the questions that you have asked. Again, I apologize. Checking the difficulty into the people who are online. Again, that's an issue, but I hope you will get it right next time. So again, thank you so much. Thank you for coming and just look forward to our next event. I guess it's in March and we'll get that information out through our channels to you. And then we'll see you at the next event. So goodnight. Thank you and we'll see you again. Take care. Thank you so much. You guys work on the view of the library, right? You know me.


Hey everyone, I'm Victoria here with Columbia Animal Services to tell you about our February adoption special which is Meet Your Perfect Match Valentine $14 adoption special across the board all month long. Any animal available for adoption is already spayed or neutered and microchipped, dogs are heartworm tested as well. You get all of this vetting including that $14 adoption price. Adoptions are done same day at the shelter at 127 Humane Lane and our adoption hours are Monday through Friday 12 p.m. to 5 p.m. and Saturday 9 a.m. to 4 p.m. If you're looking to adopt, come on down with your photo ID. If you meet your perfect match, our team will get you squared away and you can take your pet home that same day.


Okay, so, hi everyone, my name is Jeremy Guilarno. I'll be presenting about continuous application monitoring using LTTNG. I work at EPCS. If you've never heard the name, we're a small consulting firm based in Montreal. We develop tracers, LTTNG being the most known of them. I personally maintain LTTNG tools and battle trace. I'll start the presentation with a few words about tracing in general and more specifically system tracing and LTTNG. How we struggled to meet the needs of people who wanted to do continuous monitoring with LTTNG and our solution, tracing session rotation, what it is, and the shortcomings and how it makes, well, the challenges that are involved in making continuous trace analysis at scale. So, LTTNG started in 2005. At this point it is more of an umbrella project. We have a user space tracer, kernel space tracer. The system demons that I maintain to control both tracers and control the data extraction paths. The LTTNG analysis project, which I'll touch a bit later in the talk, it's a set of scripts to work with traces and LTTNG scope, a graphical viewer. So I got a super great question in the interview in the week before the talk that is now on the FOSDEM website. What's different between logging and tracing? And really it's a hard question to answer because there is no clear definition for both. They both have the same goal. So the general goal is I want to extract information from my application or from the kernel and be able to understand the state of the application at that moment. That's the gist of it. And different logging frameworks and tracers will target either developers, sysadmins, or in the worst case end users. And it's always a matter of trade-offs between usefulness, ease of use, the performance impact on the system, and verbosity, which ties into ease of use. And then there are some concepts that will come back in every tracer and every logging framework, enabling only certain subset of events and logging levels. The big difference I find is when people think of system tracers like LTTNG, we're more in the low-level events that are very, very frequent. If we think of the kernel tracer, our goal will be to capture system calls and their payload. So we can have thousands of them every second. But we also trace scheduling information. So we track every task running on every CPU at any given time. And it will track file system activity, networking activity, power management events, and so on and so forth. In the case of LTTNG specifically, the user space tracer, it tries to be the equivalent for user space. So since we can capture a lot of events with a very low overhead, people will tend to funnel very, very verbose logging through LTTNG USD, and our job is to cope with that. When I say a lot of events, I want to give you just a rough figure of what that represents on this laptop. I did a quick measurement while it was completely idle. If you enable all kernel events, that represents around 54,000 events per second. And you're not dropping events at that point. On a busy eight-core server, you're more in the waters of 3 million events per second, so that starts to be a lot to cope with. And so this is where tracers and loggers are going to diverge in the trade-offs that they have to make. First trade-off, I'd say the central part of LTTNG is ring buffer. That's at the center of the whole thing. We have a lockless ring buffer implementation in user space and in kernel space that are very, very similar. There are only a few adaptations to make it work in both scenarios. And it is consumed asynchronously, so there's no locking in terms of waiting or blocking the application or the kernel, for that matter, when you're producing events. So worse comes to worse, you're dropping events, so you're not slowing down the application. And then there are a number of memory footprint and the security parameters that you can set on that ring buffer, which I'm not going to go into for this talk. And so that's one big part. It's the quality of the implementation of the ring buffer. The other thing is the best way to not influence the system is to not trace. So trace as little as possible. And this is why in LTTNG we have a very complete set of a rule-based system where you can enable only the events that you want and you can perform filtering at runtime on the events using our interpreter that's running both in the kernel and user space. So I gave two examples here. The first one, you can see that you're enabling the read syscall. You could enable it for a given process and then you can filter on the size of the read itself and not capture any read of four bytes or things like that. And to give you an idea of the performance in the kernel space, it's kind of a complicated question I won't go into, but it's pretty much the same performance as F trace. In user space, we're in the waters of 130 nanosecond per event. So that's very fast. As you understand, we can't go through string formatting to achieve that performance, so everything's going to be serialized to a binary file. If you follow the development of the Linux kernel, we have pushed the... Well, we've been working on that for a while, but now a restartable sequence. The new system call is in the mainline kernel from 4.18 and up. And the goal for us putting that much energy behind that was to improve the speed and the performance of the user space tracer. So right now, the codepads that use RSEC are not in the LTDNG or USD, but our prototype branch lowers that number to 90 nanosecond per event. So it's quite a significant improvement for us. The users of LTDNG are going to be kernel and application developers and embedded system developers and for a very long time in the project history, that was really the core of our users, and I'd say all of our users. But now there's a lot of interest from people running infrastructure, system administrators, DevOps, and those users come to us and say, hey, I've used LTDNG to debug that tiny race in a driver. They found firmware bugs in their mix. They found understanding why there was memory pressure at a given time. So things that are hard to reproduce that can be reproduced. So basically what they would do was they would set up tracing, start collecting everything, reproduce the problem, and then investigate manually through the trace. And that works. I mean, it's the way that two other classes of users tend to work. But it's not really continuous monitoring per se. So it's fine if you're hunting for a bug, but if you want to know if everything's running smooth, it's not really the solution. Something clever that I've seen reproduced at a number of our customers' site is that people who run a large infrastructure are going to use a sampling approach. So basically they set up tracing at random times on random machines. They will collect the trace for a given period of time, run a number of scripts on that, and they'll look for deviations to their known good response time resource usage. They'll look at error rates, error types, those kind of things. But it's not continuous again. So we add the snapshot mode, which is mostly used by embedded developers. And in this mode, it's flight recorder mode, basically. So you're tracing in memory. You're never touching the disk. And if you figure out something wrongs going on, you can capture your buffers and work with them. And for all intents and purposes, this is going to give you a regular CTF trace that you can use every tool out there that reads that format to work with. But there are some drawbacks, especially for monitoring.  If you're only keeping 8 megabytes of trace data around, by the time you realize that there's a problem, there's a good chance that you'll have overwritten everything. So no luck. And there's also the problem that if you manage to react fast enough, you only have a few millisecond of tracings. So you better hope that you're going to be able to understand what was going on with that information. I'm making that sound very negative, but I mean, it's useful in some cases, but it's a major drawback. And then we have the live mode. And live mode was something that we thought was good enough for monitoring for a long time. But really, it was not made for that. It was like a way to build, I don't want to say better S-trace, but a S-trace with kernel events that are not syscalls. And had user space tracing to that. So you could see what your application was doing and what the kernel is doing interleaved together and try to understand what is going on. But the goal was to have enough filtering going on that you could actually follow it with your eyes and not work with automated tools. And also we didn't want to slow the applications. Or at least not as much as P-trace does it. But when we discussed what users wanted to use that in production, well, first of all, our protocol, and we knew that is not designed to endow a high throughput, so it falls down in those cases. And it's not trivial to consume. There are a couple of people who have implemented that, but they're not making it open source. So it doesn't really exist for all intents and purposes. The only client that we have is the text viewer, which was for the original use case. And also maintaining connections to every target is not really realistic, typically in production, at least not for people that we talk to. And there's also the problem that you only get the most recent information.  So it became evident that there were missing pieces for continuous monitoring. Not that we had our head in the sand, but we thought, hey, if it makes the job for all of those users, why are you special, right? So, and I want to make a point, because I think there's an EBPF talk after this. Aggregation of system traces to monitor trends is different from what we're trying to do. What we're trying to do is have all the data available if there's a problem that goes on, and not establish trends or make an histogram of latencies and those kind of things, which are very, very useful. I'm not knocking that, but the big difference in the approach is that. So our modes are good for debugging, but for monitoring, they don't do the job. So we interacted, and we kind of were on the back foot to implement all of that because everyone had a very strong idea of what we should support, and they were all different. And we're not a project that has a ton of resources to tour around. For instance, everybody wanted us to support natively their own data store, their own ingest pipeline for logs, or their own message queues, and those are all legitimate ideas, but if we try to support everything, that's the only thing that we're going to be doing forever, right?  There's a problem that traces are huge. There are people who run traces in production and they'll capture hours and hours or days of traces, and they don't mind if the traces take hundreds of gigabytes. Those people are few and far between, so this is not a good solution. So we need a way to make it possible to continue tracing but discard the old traces and make it possible to compress traces as they age and archive them or throw them away. And the third point is that trace processing can be very slow and very resource-heavy. When you have hundreds of gigabytes of traces, there's not one tool that's going to be snappy on that without a bit of preparation. So I think it's a given that the analysis must not happen on a target. Typically, your target is busy doing something. You don't want it to crunch to 100 gigabyte trace just out of nowhere. So the solution in concept is very simple. We thought, OK, so why don't we have a log rotation but for tracing, right? It's not that we didn't think about it. It's just there are a lot of challenges in implementing that with the constraints that we have, and I'll go through those constraints and show you why it was complicated. But basically, the idea is we will trace and then we switch over to new files and then you can do whatever you want with those files. So then if you want to ingest them in an existing logging pipeline, you can do that. If you want to shuffle them around, you can do that. If you want to compress them, go ahead. And that trace should work with any viewer. That's also a big point since our tracing format is standard. It's open. It's under the Linux Foundation Diamond Work Group. We don't want to break the format just because we're implementing a new feature. So there's one problem. Our format doesn't make this easy. And the reason why it doesn't make easy is two-fold. First of all, an LTKNG trace is a set of files. It's not one file. So what I said before, we have a lockless ring buffer that we keep per CPU. We also have per CPU stream files. And those per CPU stream files exist for user space and they also exist for the kernel space. And they could exist, depending on your configuration, they could exist for each and every user or each and every application, depending on what you want to do. So it's not like a classical log rotation where you write, you open another file, start writing there, close this one, and we're done. And we also have a metadata file. That's one of the big challenges. Because LTKNG is fully dynamic, you can say you have an application that opens a shared library and this shared library has new events. They will have a layout that is completely different from what is known up to that point. So you need to add that layout. And we have just a simple event here that has a UID and a cookie file, a payload of the field. And if you don't know that layout, you can decode the trace. That's basically the problem. So rotating is a... There's also a step where we have to know, okay, up to that point, what were all the event layouts that were possibly created? We have to serialize that to the metadata file and that over to the user so that they don't end up with a trace that is not decodable. So the steps become... Basically, as we are tracing, we need to sample the production position for every CPU across domains and then establish a switchover point, determine what all the event layouts that we need to decode the trace are up to that point, flush that, and then basically switch over to a new set of files and mark this one as being ready for consumption. So you can either monitor the file system to see new files being created or you can use our Unix socket-based notification API to know that. So I'm very happy to say that this works now. It is not in a stable release, although we will be releasing this month. But now it is in release candidate stage, so if you want to try it out, if you find bugs, hit me up on GitHub or on the mailing list. I'd be very happy to hear about that before we release, although if it happens after, we'll fix it. So for a while, we were super happy. We thought, all right, this is done because we look at the world from a tracer's perspective. So we thought that all the analysis tools would be a good job of other people. It doesn't really work like that. People started playing with it, and there were two groups of users. And I'd say there are more people in the first group that were very happy with the feature. Basically, people who do stateless analysis. And stateless analysis is what you do when you just want to count events. So you want to figure out, you have a simple rule, maybe with grep, maybe with something else, where you just want to know how many of those events were there in my trace, how many errors, how many, you know, whatever, and you just need that, and it's fine. And then you can throw that in Grafana or whatever dashboard you use. The other class of users that were less happy were, okay, this is great from a theoretical point of view, but I want to perform stateful analysis. So what is stateful analysis? Basically, anytime you read a trace and you construct a model, so anything that's going to be interesting, which traces is going to use that, graphical viewers do that all the time, they basically have a simplified model of the kernel, for instance, and they'll follow every event that comes in. And for instance, if they see a syscall open and the open succeeds, they'll see, okay, now that fd number three is associated with that path name, fair enough. So this is going to make it easy for you to read a trace and understand what's going on without seeing a read on fd number three and having to go back perhaps hours before to see what that file was, if you even have it, and that's the thing with trace rotation, maybe that information is another chunk. So you'll never have that information. So basically the simplest use case is presenting data in a more familiar way. Even a command line viewer can do that, so instead of having the raw event, we replace the PID with the process name and it will replace the fd with the file's name. And this is what in LTT in general analysis we do. I'll present the project a little bit later, but basically if you want to see the longest read for a process in a given time range, it's going to give you that. It's not going to tell you there was a long read on fd 13 at that moment, it's going to give you the file name and the fd. So the way those tools work is that they track state changes of resources. So in concept it's very, very simple. They have, so a bit of background LTT in G is going to, when you start tracing the kernel for instance, it will dump the state of the kernel. So that means all of the open file descriptors, all the active processes, well I should say all the tasks and their PID, group ID, the user namespaces and all that. So everything's going to be dumped. And the tools can use that as an original model that they can then modify by reading the traces. Now what most graphical tools are going to do is track the changes and populate a state history database. And they make the reasonable assumption up to this point that all the information is going to be available all the time. The problem with independent chunks is that you don't have a complete model. So, well you have all the information, it's just split into a lot of tiny, tiny traces. So if you look at a classical LTT in G trace you're going to be able to follow the whole lifetime of a file. And if you look at chunks, well maybe you'll see the open and one are reading the other and the close and the last one. So if you point existing tools to any of those independent traces well they're not going to tell you much. So the challenge became, okay, people are going to have to adapt their tools, maybe I should try to do the exercise myself and try to help them. And I thought it would be very easy, turns out it's not. But there's a demo at the end and you'll see that it works, spoiler alert. So LTT in G analysis, we have a number of scripts. The most popular ones are there, system called latencies. There's one to see the interrupt and alert durations, IO usage, the highest latencies when accessing a file, scheduling latency so you can see, for instance, if you took a long time to answer a request, were you scheduled at all on the CPU during that time? Things like that. So I wanted to reimplement that analysis but with chunks and basically provide the same information as LTT in G analysis. So again, that analysis, very simple. You just have to track which is called active in any given thread at any given moment. When it completes, you can compute the duration and you can print that and use the kernel model to pretty print the information. So replacing file names, PIDs and all that. If we don't have all the information, we need what we call partial modeling. So basically the central idea of partial modeling is that any kernel object or operation for that matter can be represented as a span. So the lifetime of a Cisco is going to be bounded by the lifetime of the thread in which it happens. The thread is bounded by the process. File descriptors, let's assume that it's bounded by the lifetime of the process and not get into weird clone flags and share the file descriptor tables. CPUs, the same way, most tools don't... Well, most tools assume that CPUs are there all the time, but with CPU outplug, they can come in and come out. So the lifetime is also a span and the next kernel, that instance, is also a span in time. And the same applies to application. When you're serving a request, that's a span and that's not a new concept. I mean, Zipkin represents that information in that way. It's just that for system traces, it's not as usual. So this is where most analysis needs the most adjustments, and I'd say mostly a rewrite, typically, or at least of the back end. So intuitively, I thought, well, the analysis can happen in two phase. First one, I'm just going to go over the chunk, find every complete span and keep that aside. And if there are spans that span over the frontier between two chunks, I'll just keep that aside and when we do the merge phase, we'll be able to stitch that together. And this sounds a lot like a classical map-produced pipeline. OK, it's a bit more complicated than that and I realized that very quickly and it makes a lot of sense if you think about it. Syscalls are not atomic and system calls can fail. System calls can be interrupted and restarted. So you have to be careful. If you see it open, it doesn't mean that that file descriptor is going to exist at any point in the future. Maybe a system calls are going to be interrupted. Maybe it's going to hand it in the next chunk so you don't know a lot about it and your model has to account for that. So you have the example here. The other point is that can happen on different CPUs so you don't have to perform your analysis not on a CPU stream basis but on a chunk basis. Fair enough. The Babeltrace library makes that easy to collate all the streams together and do the merging. So during the mapping, we find every span. I gave an example here. Maybe we'll see in a chunk that there's a read on FD22. Keep in mind that this could be a socket. This could be a number of things. The only thing that we know is that at this point, if the read succeeds, there was a file descriptor 22. And we succeeded in reading 15 bytes. And we saw the end of the lifetime of that file descriptor. Fair enough. So now we have the end of that lifetime of a file descriptor. Reducing, well, to talk about my implementation, basically at the end of every mapping phase, I'm going to produce a JSON file that has all of the kernel model with the partial spans. And during the reduce, we're going to tie this back together. So at some point, there is a chunk that will tell us, I don't know the FD of that file, but I know that there's a file descriptor that points to this file. It's a block device normal file. It's not a socket. It's a nipple file descriptor. And we'll have all the information, and then any request that wanted to use that information is going to complete. I should say the report for a request, and it's going to give you the breakdown. OK, so in fact, it's a bit more complicated than that still. The rotation is not atomic. So if you look at the model that I had in the PDF traces before, well, it's easy to think, OK, I have a switchover point. Now from that point on, I have a new set of traces. It works until you try to do analysis on that. And really, it's simple. If you try to perform the switchover without blocking everything, you're going to end up with this staircase pattern where in time, the rotations happen and you switch files. I thought that this would be a lot quicker than it actually is. And for a time, I thought, ah, it's not the end of the world if we drop events for that little interval. But actually, rotating on a 12 logical core machine is going to take 28 milliseconds. So there's a lot that can happen. There can be complete lifetimes of processes during that time. So, well, processes, maybe not, but it could happen. But at least files, that can happen. And complete requests can happen during that time. So you can see that what ended up happening is that in some cases, you could see an open on the first CPU and then a dump on CPU number three. And then you think you know everything in that time span, but actually you're blind to what's happening during transition. So then you assume that, OK, I know what that FD is, but actually it was something else all that time. So you don't know what that FD refers to. So there are a number of solutions to this problem. The one that I used is a bit lazy, but if you use two chunks, you can see to a point where you'll have a complete chunk that fits your assumption that the switchover is atomic. I think it's a good moment to mention that we use the same clock source, clock monotonic, between kernel space and user space. This is what makes it possible to make the analysis reliable and really order events in the way that you would expect. So I'm happy to say that the prototype works. The goal was to provide the same data as LTTG analysis. And what I wanted to do is have a model of an application that performs a non-trivial query and then show the time that was spent in each phase of the query and then drill down what was the time spent in every syscall that happened while serving that phase of the query. And I tried to address the shortcomings of LTTG analysis. First, it is a lot faster. LTTG analysis is written in Python. So working with millions of events is not what Python is made for. So it's normal that it was slow, but it made it pretty hard to use in a continuous way. So that part is taken care of, but really the challenging part was the partial modeling. So the example I'm going to show you, it's a server that receives a request to generate a thumbnail. So basically it gets given an asset ID. It's going to seek to the requested frame, decode the video enough to decode that frame, scale to a thumbnail size and send the response back. And then we can generate a breakdown of time spent in every syscall. So I have this running in the background. It's feeding a Grafana instance. And basically what you can see here is the time for each and every request. Now I don't recommend that you do that in production because you don't want to be processing each and every trace chunk and individually graph each and every request, but it gives you an idea of what can be done. The thing that you can do is that if you have outliers like you have here, those points you could generate a report and drill down into. And this is what we're going to do. If you look at the report generated for one of the fast instances, we can see that most of the time is spent opening the video with FFMPEG and decoding a full frame. That's fine. That's what most of the work is about. Scaling, as you would expect, is mostly CPU bound. So opening the file, there's a lot of stuff that goes on. And you can see here that the combined duration of all GET TIDs is actually what is dominating that operation. Then we have a number of calls. I'm not going to go through all of them, but you can see the interest in augmenting the model here. As you can see, normally you would have a read on an FD that you don't know about. You would have M remap on FDs that you don't know about. Well, not M remap, but M map and so on and so forth. Seek, same thing. So seeking can see that not a lot happens, as you would expect. This is an indexed video file, so it's fine. Then decoding, nothing unusual. Scaling, nothing happens. It's a purely CPU. It's purely compute. And then writing the result to the socket. Now if we take one example that took a long time, we can see that the outlier is in the right. Actually, this one is not sending over in the network because I had some problems here, but it's writing to a plain old file. Basically you can see that writing doesn't take a long time, but during that time there was a sync call that took around 600 milliseconds. So that's the kind of insight that you can have very rapidly. Hopefully your applications don't perform sync during request handling, but it has been seen. So in conclusion, I think from the tracers perspective we can see that we have the infrastructure to distribute trace processing. Our users are very happy that now they have files and they can do whatever they want with them. But the open questions that I have for you guys is what form can this type of processing take that would be integrated with existing viewers? Or do you think that we need new viewers, new tools for that? Are there tools out there that you guys use to monitor very low level metrics continuously? I'd be very happy to have your feedback. Otherwise, do you have any questions? All right. Well, thank you.


A Float with Henry Morgan A float with Henry Morgan written for radio by Warren Barry and a George Edwards production When Kitty tells Deats that she is in love with Jeffrey Hunter and wishes to have nothing more to do with him Deats threatens to buy her from Bo-legged Jobson and thus have her as his possession His treatment of Kitty causes a riot in the Dolphin Tavern and the men's through the place with wreckage When Hunter hears of the incident he knocks Deats into the harbor warning him not to come aboard again The next day the governor Sir Thomas Modford tells Morgan that should another Dolphin Tavern incident occur The authorities will close the place He leaves the flying girl followed by Dolores on the way She is stopped by Deats who has been lurking in the shadow of a building. He tells her that he has recognized her How dare you stop and speak to me? Not so fast. Just don't want to know what you do here in Jamaica You are making a mistake May I remind you man that I am Antoinette DeLacy King's woman of the governor of Jamaica, Sir Thomas Modford. And not a make a mistake Your name is not Antoinette DeLacy. It is Dolores Bizarro When you came aboard the ship the flying girl I looked on your face I knew I had seen you someplace before and just a little while ago you came down off the ship and I remembered You see I was in Cuba when you arrived with your father Don Pietro Bizarro Governor of Cuba And I saw you leave the ship just like it did a moment ago, and I remembered Yeah, up to some misdeeds Yes, Spanish spy. I was screaming for help. You're screaming. I would expose you Go on take the risk Dolores what is keeping you? We have an appointment You will meet me tonight just outside the gates of government house and talk to me Else I tell the governor who you are. I certainly will not miss you. All right, then tomorrow. I go and see Sir Thomas Tonight it is to be a senior in earth. It's sorrow at nine o'clock or else tomorrow.  What's wrong? You've caught me at my secret cupboard See a sliding panel in the wall Until this moment. I'm the only person on the ship who knows of its existence. I'm sorry captain It doesn't matter so much. You know I got it You see I trust you lad. Sometimes your trust in me worries me. It's a great responsibility And I know an honest man and I see one Look here hunter I'll show you what I keep in the cupboard My face is the loot of all my bucketing days Take a look at that ruby See the fire of it How it catches the light and glows like haters and yet is so cool Like the heart of some woman Look you with these sapphires Look at their blue like the pacific ocean on the summer's day Took that little lot off the mayor of a coastal town. I once sacked Now I made him squeal before he'd tell me where he'd hidden them And look you hunter My biggest prize of all He has ticked necklace You've never held it in your hands long enough to take a good look at it Hold it now Beautifully fashioned by hand long since dead and crumbled to death. Tell me captain Morgan How much is the ransom of a bonded servant? What's that? I want to talk to you captain You better put those jewels away in your cupboard and close the secret door Well now What's troubling you lad? What is the price of a bonded servant? That depends Some are cheap if they're old and bent and ugly Kitty's price is far above what you've got kitty I'll know you I was speaking of kitty. There's not much that goes on in port royal that doesn't come to the ears of captain Henry Morgan Look you lad I took before to you about that when she isn't for the likes of you Well, I does not a man aboard the ship would not like the price to buy her from Boley good jobson and she knows it too I only want to know what her price would be Oh, and that is hard to estimate You'd best ask Boley good jobson If he kitties rather a special case His kitty that makes the dolphin tavern far and above the other key site devins As far as customers concerned When do we say on the next expedition? Oh, so you think you'll make enough out of buck and earing do you? We all get his share don't we? Sure But many take part in my expeditions and there are many to share the proceeds amongst You'd best forget kitty. Oh, I know that you've taken her eye and I know that although you had no hand in the actual ride at the tavern. It was started through you Forget kitty hunter.  Here, help me down off the flattery Here you are jump And right into your arms I come And what place could be nicer a kitty?   No, but I do all my life's been changed Whereas before life was just the sun getting up from the sun setting and sermon loud mass cursing men with their drinks Now it's all changed There's a song in the heart and it sings all day. I loved you kitty Please I want to talk to you where he said that after the life I've led You should come to suddenly into my life and and I'd fall in love with you You're not in love with me kitty It it's only because I'm a little different from the other men who have come to the dolphin town No jeff yet, isn't that I know in my own heart the way I feel If it's possible for a woman to fall in love with a man so that nothing else in the world matters Then that's the way I've fallen in love with you You have tried to help me kitty and I want so much to help you It isn't your pity. I'm looking for jeffrey. I want to get you away out of this place I don't want you to be free forever from the humiliation of being worked as a slave I would like you to be free forever from the demands of every cost of brutal sailor who shouts at you for his drinks And what you ask is impossible That could only come about by someone buying my freedom. There must be some other way How long have you been in cordial kitty? Seems like a lifetime, but I guess it's now on two years For how long is your sentence if you think I can work my way to my freedom you're mistaken I was sent out here for life. What did you do? I got two convictions both for the same thing must have been serious crimes. I Serious to the man who owned the bread Have you ever been hungry in a strange city being without friends? Yes, I I know you've been without friends. I was forgetting for the moment But have you ever been hungry really hungry when your whole inside craves out for food? No I was alone in England. I'd not come from Ireland long and I was hungry And the baker's basket was standing on the pavement. I couldn't help myself My hand went out and I took a loaf, but I was not quick enough. The baker came back and he caught me He sent me to nougat for two years The stealing a loaf of bread and when I came out of jail I was still as hungry as I was before I went in And I thought this time I'd become When I saw another basket of freshly baked bread I waited until the baker disappeared out of sight and then I helped miss her But someone else saw me and gave me in charge Already I had one conviction had not been long out of jail. So So they sent me to Jamaica for life Such barbaric injustice should be allowed. What are these times? We're living in and when I came out here I didn't care what happened four-legged jobs and took me as his possession. Well, I didn't care I knew what I was Just like cattle in the field So I just let me fancy play where it would How was I to know that that I'd meet you Jeffrey and that I'd regret what I'd done in the past two years Jeffrey You've not kissed me yet.  It's wrong. I should kiss you like this. But how can it be wrong? We're really alike your pasts are behind you and your future doesn't promise much Just let's be happy the past hanging on to you and and that's why you hesitated to kiss me Let me make you realize that you're gonna never again what you've lost And the future's only what you're going to make of it And here waiting Give me a start. I didn't see you standing there in the shadows So you thought it wise to give our appointment senorina? You do not understand you are making a mistake a mistake isn't Yet you're come secretly to meet me because you have frightened it. I'll go to the governor and tell him the truth about you How is it if you're masquerading as unto net the lazy? What has happened to the real woman? I might ask you the same question. What are you a spaniard doing here in port oil? I owe allegiance to no country. I just work for myself. How much money do you want to go away and never worry me again?  I have admitted nothing But I do not want you pestering me I will tell you something, huh? It is quite true that the english should be elizabeth hann was captured by the spanish It is quite true mamma moussel delice. It was a border and she was taking the cuba here And there you learned that the governor had never seen the mamma moussel So you came to port riley nested So what purpose, huh? You tell me We come from the same country Maybe we could help each other, huh?   Now you pay me well Maybe I help you in somewhere I can be loyal to you Because at the moment I hate the english and morgan included morgan So here's your enemy a Maybe you can help me a lot An unholy alliance is about to be made which could mean disaster to jeffrey kitty and captain morgan Listen to the next episode of a float with henry morgan