Hey folks, hello, sorry my mic was on the wrong one You have so many microphones. Yeah, no, it's a problem my my my list of one, two, three, four, five, six Inputs and another six outputs I should go disable some of these in the top Yeah, I hope it's not how not not happening to you when you are sitting in the headphones, but your output goes to like your External sound system and everyone enjoying your meeting So we'll need one or two scrubs today, so if You don't mind helping scrub that'll be and hey look Hey there, how you doing great? All right. We'll just give a couple moments usually wait five minutes for people to join in I'm gonna paste the The meeting minutes in the chat if you can please go to attendance and Put your name in Luke are we expecting a child's as well today? No, it's just myself. Okay. Yeah Okay, great. I think we can get started. So we're gonna start with check-ins and then today we have your care That we're presenting to us about key lime, which is some a new CNCF samples project So let me quickly go through this Emily do you want to give an update for CNCF security today? Posted CNCF tweeted about it. Are you treated about it? I'm going to drop it in the meeting notes So if you're curious as to what is on schedule, I'll make sure that you have that available to you But don't forget to sign up for Cognitive Security Day. It's not too late. Awesome. Thanks Emily Cool and Eli, thank you for helping to squirt Um and also Mark, could you have something on open telemetry? Yeah, one of my colleagues in the I should believe DevOps Communities working on open telemetry. I just wondered if anybody else here in this group is working on it or where of it or Trying to develop products that are compatible with that developing spec Okay, take that as a no. I'll try to track it and put some notes in here later. Yeah I Explored it a little bit for a project that I was working on but I'm also interested but I can't say that I'm working on something related. I'm just interested Yeah, yeah, same here really so they have a semantics for it, which One would think you could adapt the semantics for security audit and logging streams But it's not really designed for that so I'm curious, you know what where that ends up now my colleague who's working on this Is part of the DevOps security work in IEEE so that's kind of his bent if y'all ask him to come here and give a talk at some point That would be awesome If you can open a presentation Issue and then tag him and then we can pull off. Okay. Thanks. Let's see. I think that is all for updates today So we can jump right into the presentation. So Lucas show if I'll see someone be talking about Key lime, which is a new CNCS sandbox project congratulations silk and It's gonna be about attestation and DPMs things like that, but I'm gonna Give it good to plot you know it sure. How long do I have just to place it? So we have until the the end of the session. So I'm usually Up to 30 minutes and then yeah, yeah, sure sounds good Okay, let me just Not use too much before so hopefully I don't blow up my desktop manager. Hold on. Is that visible? Oh We can see yeah, okay. I'll go and try let's try presenter mode Yeah, that's that's good. Yeah Great. Okay. So so funny enough. I have these slides because originally We anticipated First of all showing the project to seek security, but then they changed the process for sandbox inclusion So I already had a branded deck available. So I'm getting the chance to use it at last, which is good So, yes a key lime So I'm Luke Kynes. I'm from Red Hat and work in the office of technology and worked around security for a good number of years now and Child it says Charles Charles isn't actually here Charles is Another guy that I work on key line with from MIT And he was going to be part of the original pitch as well, which is while he's which is why Excuse me. His name is on there So a little bit of history about the project Originally, it was a research paper That came out of MIT cells Charles and a new bill Who are both a couple of our researchers at MIT in their Lincoln labs? security Research department and they came up with this paper bootstrapping and maintaining trust in the cloud and they also wrote together a basic prototype and I've been Around this time about a year after they wrote the prototype. I've been hunting for Some sort of open source Remote attestation solution. I was either going to code my own or see if there was something around that I could contribute to and I found the Bill and Charlie's prototype and I really liked the Core design that they had how it was it was simple, but it was a very scalable architecture And I could see they'd solved a lot of the problems that are in an inherent part of TPMs around Performance and so forth. I just like the sort of the the use cases that they've built on top of this So rather than it just being remote attestation They had these other nice components which which made it a really good all-round solution So that's where I got involved and I I got in contact with those guys and I said, look are you Interested in we build a community around this and start to to get other people working on the project And we did so we started to sort of take it from a prototype Into being a good healthy open source project, which is what it is now And as you know recent bit of news we got accepted as a CNCF sandbox project So key lime itself the the main scope is that We have remote attestation Okay, now Imagine a lot of you are familiar with a TPM. If not a very very quick overview is a TPM is a A chip typically that resides on Another board of a server or device, okay And it's inaccessible to software and it has a very simple Crypto engine so consider it almost like a stripped-down version of OpenSSR So it's not a crypto accelerator, but it can perform very simple crypto functions such as Signing artifacts and generating keys. Okay, and there's this principle of measurement with a TPM, which is where An artifact an object whether that be a script or a binary or or or any sort of system element is measured So a hash a digest is generated And that hash is then signed by the TPM with a private key pair that's Locked away within the TPM. It's called an entity key And this then allows you to Using the public counterpart you can then Verify that it was the actual TPM That signs some measurements so you can remotely Retrieve these measurements from the system that you're monitoring and then you can attest them effectively using this What's called a hardware router trust and TPMs have been around for quite a while Um, it's it's it's not too far out there to say they're almost ubiquitous. They're on a lot of servers come with them Already there. They're not they're very Simple to retrofit usually quite cheap You can put them on a Raspberry Pi. I have a Raspberry Pi for about 20 euros or something You can get a TPM and put it on the GPIO So they're they're they're quite an accessible piece of inexpensive hardware. Okay And and there's also a virtual TPM which could be used in containers and virtual machines So with remote attestation what we do is we first of all, we've got something called measured boot Where we measure boot components. So this could be the the firmware the bootloader The kernel command line options There is a project called UEFI shim and the shim sits Above UEFI and below the bootloader and that can measure these components So some of the stuff that's measured is secure boots as well So secure boot they have like a mock list and certificates and so forth So all of these things are measured and then they're what's called extended into the TPM So it's to an extend is a like a one-way hash function. Okay, so you can kind of um, you can then Have a sort of a record a cryptographic replay of the boot occurring Okay, because it sort of concatenates and extends hashes to build like a one-way hash function And that is then signed And we remotely retrieve that Measurement set from the uh from the node. Okay, and then we attest it Is we attest the the hardware router truss using these cryptographic keys that I that I described earlier Now that sort of measures the boot of a system Once the system is running There's a linux security subsystem called IMA which is integrity measurement architecture And it's been in the kernel for quite some time. I think it's since Version 2.7 and it sits alongside se linux. Okay, and what IMA does is When a system call comes in so let's say for example, somebody runs a script as root IMA will measure that script And then it will extend it into the TPM. So I'll record it into the TPM and then that again that list can be Remotely retrieved and attested so you can measure the script that was run on the machine that you're monitoring Has the expected digest Obviously the digest is changed It suggests there's been some sort of compromise and so IMA sort of is like a A kind of a runtime measurement System okay, and then boot is obviously one time for the boot. Okay, so that's remote attestation We then have two things called encrypted payloads and the revocation framework. So encrypted payloads what these are is once a machine Passes its measurements. So it has the the expected state We that we Have set for that machine Then we will allow that machine to unload an Encrypted store or it could be a payload that's sent over the wire And we do this with something called a three-part key deviation protocol Where we have the machine attest itself to the verifier, which is a A node that's part of key line And the actual user themselves they can then there's there's like a two-part attestation that happens And then once that passes Two keys are released the keys are two. So a key is cryptographically split It's it's released by these different counterparts the agent which is the the end resulting machine the target machine can then Reconstruct the key which then allows them to get the the the encrypted payload And the other thing that we do is when when a machine passes its trust I don't really like the word trust when it's when it's cryptographically Of the state that we expect Okay, with its measurements Then we have something called an application bootstrap. So this is a script that runs it automatically runs Once all of the the measurements are verified And people use this app to do all sorts of things that Some folks have been using it to pull up a kubernetes cluster Or it could for example It could wrap around some sort of deployment framework such as ansible Or a puppet or chef. So what you would then have is you would have your secrets your deployment Your the secrets that you require for deploying an application would there be t less certificates or for example some ssh keys or An iini file that contains some passwords that that would be your encrypted payload And the application bootstrap would occur which would start the installation and then the secrets would be there Okay, but if a machine failed to pass its Its attestation And then it won't get its hands on that payload effectively And then last of all we've got the revocation framework And this occurs when a node fails attestation. Okay, and what we do is we send out Assigned revocation event to all of the nodes That share a trust boundary with that machine effectively telling them to Ring fence that machine. So this is like a It's a frame. It's got a framework because you can write your own scripts Which the key line agent will then execute for you. So people write scripts around shutting down network interfaces removing a machine from What's what is it in ssh your known host your trusted your authorized host you could for example remove The machine from there and i've seen other folks that have cordon and drained The kubernetes worker so effectively a po a pod will migrate to a a machine that still has a A trusted state so to say and all of these are built around this this router trust which is in the tpm and the vtpm So those are kind of the the central Trust framework to these to these three pieces So a little bit about the architecture So to the left consider that The the wild west. Okay, that's a machine that's in a It's sort of outside of our trust boundary So this could be in a another cloud provider's network or it could be an edge device or an iot device in the roof of an office building somewhere That's susceptible to physical tampering This is effectively the machine that you want to measure and on that machine we run something called a key lime agent Okay, and it's pretty simple. It just talks to the tpm And it has a rest interface.  Okay So the agent communicates with the tpm as this says and then over to the far right This is more sort of your on-premise. Okay, so you have the key lime verifier and this does the Verification of the measurements that are retrieved and then we have the register which is kind of like a database There's no cryptographic properties for the store in there That's where we have the agent uu id and we have a public key store For the intermediate certificates that are provided by the tpm manufacturers that help us Assess that it's a real tpm that we're talking to and you'll notice in the middle It says untrusted network So this connection we don't have to worry about a man in the middle here Because all of the communication coming back from the agent Is signed with a key pair that's locked away in that chip And the digests use this one-way hash function So for for somebody to be able to for example, if somebody changed Uh user bin Uh, I don't I don't know IP tables For example, they chose and iced it so they changed one hash They would have to cryptographically pick back through the the hash extend to then change a single hash and then reconstruct That chain of hashes again, which is impossible to do. It's it's computationally impossible to do So we don't really worry about this particular connection And it being there being a man in the middle attack So it is acceptable to run this over HTTP because like I say the we have this hardware router trust So, yeah, that's the architecture the The blue lines that you see here that's used in the trust the tpm tss stack And then the connection between the agent and the verifier and the register that's all on a rest API So we protect that with mtls mutual tls and we are also just working on getting json web tokens They're involved as well So you'll be able to use one or the other or both if you want to go belt embraces so to say So the remote test station use case. This is kind of kind of a simple overview but A user or a tenant would say could you attest a machine for my workload? The verifier will then measure the agent and there'll be two outcomes. So the first would be Passes which is we run the auto run and we release the encrypted payload or two is revocation Okay, so we look at those two particular outcomes from a remote attestation so the encrypted payload is The machine passes its attestation The payload is sent, okay And this is as I said, it's based on this three-part key deviation protocol And the execution of the auto run Happens once the system attests without failure. So you can see an example payload there So for example, you could have some passwords You could have a script With those particular Secret artifacts that you have and then you'll also see there's a couple of scripts in here called local actions Cube CTL and local actions IP tables So those are what that machine should run when another machine fails in its trust boundary So the idea is that um, you would then sort of ring fence a machine that's being compromised So that's part of the revocation The revocation framework which I'll go into next So um, we have IMA which I described earlier. So this is the integrity measurement architecture. It's a Linux subsystem So every syscall is measured and extended into the tpm The remote state is compared with the expected state. So on the verifier you would have A list of uh, like a posix path to a file for example And then a cryptographic digest so two five six A char two five six checks on okay And you would then query the remote system to find out what its file state is every time A system call is made into the into the kernel. IMA will intercept that And it will record it into the tpm and we continuously poll over the REST API. It's a very lightweight poll. We can do that you know to Point one two of a second if if we had to and so we would know we're in a very small window We would know somebody compromised a machine Because we continuously poll the the state of that machine and once that happens. So for example, if somebody Swapped out a binary trojanize the binary. Also with IMA you can let you can measure things such as su linux labels and kernel modules and various things so If a machine failed, which is what we're going to look at now This is revocation. So we've got the the key line verifier We have a simple ca that we work with But this can be integrated with CF SSL as well And we're looking to have like a plug plug in framework to to integrate with different ca's And then you can see that we've got Four nodes at the bottom. So key lime agents. Okay, these are all being monitored continuously by the verify And so one machine is compromised What happens then the key lime verifier Can make a call to revoke that node certificate So for example, if this node Had some TLS certificates that were part of its Connectivity it's trust it's trust connectivity. We could revoke that certificate. So a crl could be populated And then the other thing that we do which is uh Which is a lot more interesting is we send a revocation event So a revocation event is just some json that contains some metadata. Okay, and that's signed by the verifier So that that can then be you can be make sure that that's not sort of a A spoofed actor pretending to be a verifier Okay, so this signed metadata goes out to all of the machines That share some sort of trust relation with the machine that's been compromised. Okay and This revocation event will tell them to run local actions. So they will perform actions which typically somehow Ring fenced this compromise machine. So it's a pretty similar principle to some of you that work with load balancing with no stoniff Do you know shoot the other node in the head effectively? So that machine's compromised We just want to knock it out. We want to get it off the network Terminate any association with it and then it can be taken off of forensics and so forth. Okay So this is effectively this machine is bad We need to protect what is good still. Okay. There's no point trying to rescue this Let's let's just completely knock it out. So to say so an example here would be So the uh revocation is sent out to execute local actions And so some examples could be like I said earlier is to remove An entry from more fries keys or Cordon and join a node shut down a VPN tunnel amend an IP tables rule Anything that you can programmatically Do on a machine around some sort of sysadmin task Then the the agent will run that so for example, you could make an api call to a system which Sends out an sms message to a sysadmin to get them out of bed to look at something So it's it's anything you dream up really you can you can delegate to key line to run that once the machine fails It's uh, it's trust state. So to say so that's the revocation framework And um, so one thing at the moment the uh, the project is Uh developed in python. So we use python because we allowed us to quickly Develop and prototype and try different things out. But what we're doing for the rust agent So this is the one that runs on the target machine. Uh, we're porting this to rust at present Okay, that works taking place right now and we went for rust because um, there's a few things. It's um statically linked So we don't have to have a big less listed dependencies that need to be pulled in so that's a machine that's offline or it's uh, perhaps like a an immutable Sort of os tree type operating system where it's a read only operating system. We don't require This need to sort of pull in a load of dependencies And then of course it's memory safe and it's and it's very performant. It's a systems level Language so to say so that works underway at the moment and when that's complete That will become the default recommended agent Okay, and then in time we're open to considering Porting from the the other nodes from python that that's been discussed as well. So we thought about You know, if the demand is there then we could even go to golang or rust or something like that Because python's just been very very good for an hour and us to quickly iterate and develop features and and get everything in place So some of the future work that we're doing is um We're working with the uh, kernel folks to to make an iMA namespace. So for example this uh Integrity measurement architect that I described earlier that works perfectly well on a Sort of a single bare metal machine. Okay, you can get this to run in a virtual machine But it's not containerized. We require a namespace. So we have as you can see a prototype working and we're just trying to uh to get that Set so that we can then do Uh measurements of iMA within a container as well I mean, there are other things that you can measure with a container, but this is quite a key one So that's a piece of work that's underway Um upstream so to say We're also looking at um There's a vtpm which will run in the container. Okay, and uh, we're doing some work to include this As part of a container runtime and open to working with other container runtimes as well. This is the uh, the software tpm the vtpm code and then um We're looking to do some stuff to extend vtpm support as well. So One of the so one of the things with a vtpm is the When you have a hardware tpm Your secrets your the cryptographic keys are obviously they're on a hardware chip Okay, so like I say there It's only a specific bus that this cpu can use I think it's the lpc bus to communicate with the tpm. Okay now when you have a software tpm It looks and smells exactly the same as a tpm functionally what Around its functionality that it provides But you don't have that hardware router trust the keys are effectively in memory or on a disk so the underlying host can obviously See any sort of cryptographic Secrets that are stored in the vtpm. Okay, and so one of the things that we're looking at doing is Marrying up the the quotes so tpm quotes are Where we request The measurement list we're looking at how we can marry that up to a single hardware tpm Where we would aggregate the quotes into a merkle tree And then there'll be a single one-time poll to the hardware tpm to vouch for the software tpm And then that that extends the hardware router trust up to the software tpm So we've got some very basic code prototype that we've worked on here It's it's an idea that the folks at mit had and so we're looking to uh To to uh bring that into the main Keyline project The other thing as well is we're monitoring what's happening with Encrypted containers because if a container is encrypted And it's protected from the host then you can run the vtpm in there and not be concerned that There's an attack vector where the underlying host Can can can access the container and the vtpm so that will solve that problem as well So we've got a few avenues to to to solve this Principle of a vtpm being susceptible to attacks from the host So community-wise we're very Diverse there's obviously red hat mit ibm a contributing person from ibm recently became a a maintainer We also have Contributions from Intel in the city in the past. We've got another maintainer from canonical Netfix and quite a few int independent contributors and smaller companies as well that are interested in the technology We do all the things that we think a good open source project should do so All poor requests undergo unit tests and integration tests and we assess the code quality you know we we are Every poor request that's made is reviewed by a member of the core team and so forth and we're also very We try to be very welcome welcome into new contributors So we've got a lot of stuff the first time of support. So typically when somebody will make a first poor request somebody will go forward and You know with a helpful spirit around helping them Sort out any CI failures that there are and and just understanding things around How to format a good commit message and and just be a welcoming a friendly friendly, uh, I was going to say a friendly face a friendly bit of text for them to to uh Start to get involved in the community so um Yeah, that's it for now, which is quite good because my voice is just starting to give up But I guess we could go to questions now or I can roll back if any of that doesn't make any sense as well All right. Thank you. Look. I think we have some questions in the and the chat um Let me just um Maybe I can ask the question if it's not yeah, that'd be great. Thank you Sure So one of the questions I had at that point is the the syscall validation I was just curious as to what what exactly are you trying to guarantee validate? And if that is the case, how how do you achieve that? What is the use case around the syscall? aspect that you mentioned Sure. Yeah, so this would be in a okay and um this one so What happens is Typically You would build a list a kind of a golden list of hashes or digests for your The node that you're monitoring. Okay, so the list the format is very simple. It's two columns It's a a posix path to the file and then the hash of the file. Okay And that's stored in the verifier, which is on premise. Okay and On the node that you're monitoring It has ima which is part of the kernel. Okay, and then what happens is whenever somebody Executes so they run for example IP config or netstat or they run a script. Okay, what will happen is ima will capture the digest of that object that's Been executed. Okay, so let's say it's a script. It's a script that's been run as um a bash Okay, so that will be that digest will be captured now that digest ima will Record that into the tpm. Okay, so the digest will go into the tpm. Okay, and then the tpm Will Build that into what's called a one-way hash. Okay, so as entries come in It's kind of like a kind of like a burnt binary tree a merkle tree that concatenated Concatenated together and then a hash is made of those two and then you effectively have a root hash So you can sort of um by looking at the root hash. You can make sure it's not being tampered with Anyhow, I'm sorry. I'm going off a little bit of a tangent there So it measures the script it grabs the digest records it into the tpm. Okay And then the verifier the on premise node Will query the agent for hey, can I have a fresh? Quote and a quote is a measurement list. Okay And then it will receive that list It will perform a hardware root of trust validation So it will make sure that it's an actual real tpm that signed that list Okay, based on a hardware root of trust that we set up at the beginning when we we first Set the system up. Okay, and it will then compare that digest that signed digest With the golden list that it has locally Now if somebody's tampered with that shell script Then you would immediately know about it So that way you can remotely monitor the machine. Okay, and then you could then perform some sort of If you wanted to some sort of revocation action So that machine has been compromised you believe because somebody's tampered with something And You then perform an action to to ring fence it now there are stuff There's lots of noisy stuff on a system like for example var log Okay, but you would probably be quite difficult to track So we do support regular expressions where you can have like an ignore list So you can say there's part of the systems that you want that you wish to ignore Because it's not really of interest and it's very busy and it's very Mutable it's constantly changing and so forth. Does that make sense now a quick follow-up on that? Look, I think I think what growing me off is the granularity Because here you mentioned syscall But what you mentioned was actually a binary a posic posix path to some IP tables or if config so And when you say you're measuring every syscall, so these binaries could execute multiple syscalls So that's where I was getting thrown up. I think I see. Yeah. Okay. So yeah, we don't measure the syscall Yeah, the syscall the syscall is the event The the Inaccess to take the measurement. I guess. Yeah, I see what you mean. Yeah. I was a bit I put that in a bit of a confusing way. Yeah, so yeah, that now you're right the syscall is the Is the event that that kicks off the process of measuring whatever that's requested. This is got it. Thank you so much You know, no worries good question So, um, there's some more in the chat. How do I see the chat? I'm completely Use. Yeah, look, I had a question. Thanks for the presentation. That's I think very interesting stuff. I can see how can How can you use and integrate some of these? Ideas I had a question related to vtpm And he mentioned that it's like a software like do Underneath it's still use a hardware tpm. Does it use a key in a hardware tpm to wrap all the software keys In vtpm or or it's like pure Software information pure pure software. Yes. Yes. So with the vtpm How we're addressing that is is one There's a possibility of an encrypted container. Okay, so the the real person that you're concerned about here is a netfarious host Okay And so so an encrypted container would I use quotes protect That attack. Okay, but we're also looking to marry up the cryptographic Relation of the vtpm to the hardware tpm. Okay, and that's this That's his prototype code that we have at the moment and where we Where we can what we can do is we can take thousands of vtpm's We will aggregate their quotes, which is a request for a measurement Okay Into a merkle tree which is then sent back to the verifier Which will sort of percolate those quotes back out and perform the verification So that way we're not hammering a hardware tpm with a one-to-one relationship between a software tpm and a and a hardware tpm and there is also some other stuff happening around um People are looking at using utilizing fpga type tpm's which could be plugged into a system And could then I don't know too much about this, but they can Then sort of attest thousands of virtual tpm so to say so there's quite a bit going on in this space to To uh to address this problem. It's just which horse will win the win the races yet. I'm not sure yet Very Thank you Yeah, and look just to clarify on the terminology when you say encrypted container you mean something like like tees like enclaves and stuff like that Possibility yes. Yes. Yeah. Yeah, that's that's one model. Yeah So is there will say some other questions in the chat. I don't know how to access the chat. Maybe I think I mean maybe need to stop sharing um No, I can see it now chat. Sorry. I've got it. I've got it. Okay. I've got a vinnaze question Kapil Is I'm intended to be stacked on top of other lsm's But you know, I don't actually know the answer to that it's um, I don't believe so. I think it sits in parallel if that makes sense it's um Yeah, I must it's in parallel with the other lsm's Yeah, so it's it's a sort of a neighbor to I see linux and other lsm's aparma and so forth And it's been in there for go. I think about 10 years now. So it's it's quite well established in the kernel Cool. Thanks. Yeah, so yeah, I would say pretty much all distributions carry IMA gen 2 debbie and fedora Arch linux Is is very very well established. Yeah, so um Brandon is that a question or you will clarify in something? I think I was just clarifying things This is a question not in the chat, but sure this is new to me. So this Is probably unrelated. So feel free to tell me that but Is there any application of this technology beyond root of trust to assurance One of the challenges with security tools is the machine to machine communications Is difficult to do reliability engineering with So sort of machine identity Yeah, so if you lose connectivity, you know in a container to container scenario where It in a cloud scenario it may be different Could be different hosts could be the same Could be the same instance I suppose but Maybe launched under different circumstances or policies You know speaking in a very Generalized use case. It's just that one tool stops talking to another and the tooling for Handling assurance for that is is primitive and so is the troubleshooting So it occurs to me that it's similar to the problem of establishing trust. It's just that Uh assurance for connectivity is a more regularized event, right? Yes, yes so I know that Brandon George And folks they're using the entity key. Aren't they for machine identification? Yep. We use the Yeah, they use they use the ek key the entity key, which is the key that's locked away in the tpm as a means to to to uh Verify the identity of a machine with a hardware router trust. So that's possible and this is outside a key line, but From what I recall spiffy and spire. We're doing some sort of tpm Integration that might have been vault something to do with well I'm a bit. I'm a bit sketchy on the details there though But yeah, sorry to to answer your question is that is possible with the entity key Okay, yeah While we're on that topic also kind of I think one of the the things that I was exploring as you know, put this be also know the tester for for spiffy and for spire for example Where you would use some key line to bootstrap the workload identity Yeah, that's what it is. It's great for doing that really any sort of um Bootstrap in trust on a new machine That's another interesting topic right look and everybody. I mean, so we've had some very very interesting You know, you know project like a key cloak key line Spiffy spire. They have a lot of a number of use cases potentially. There could be some overlapping I think it would be interesting to explore or even explicitly articulate how these tools can potentially work together identify these kinds of use cases and really For the broader community help them understand how a lot of the This ecosystem of these kinds of specific tools. I think there are three or four I think we should go back and look into it and to see how it can The kinds of use cases that they address at least at most level Yeah, I would love to do that. I really would like to take part in that definitely I have a presentation drop on that Okay, Brandon. So I'm gonna bring you real less That as well because I think that's the same question that comes up right like even in key cloak and then the You know, how it works with spiffy inspire So so my team has been working with luke as well as Key cloak with both us team as well as working with the spiffy inspire community as well I think Really, we are trying to tie everything into the identity using key lime as the basis for trust with identity And then to take this identity and be able to make you know policy and authorization decisions Um in an open source. I am type solution. So we are looking at key cloak in terms of that Nice But that's like 10,000 feet level. Yeah I'll Clean up that and then get get feedback from community as well You can you can think about key lime as basically certified station infrastructure that you can use To make sure that's your Software and hardware. It's not being compromised And build something on top of it to issue like let's say short leave TTL T send certificate that you can use for like let's say spire uh type of hide station for this note you can Uh Use the satis station infrastructure. So if you're not as compromised, you're just not renewing uh The Certificate so you're not cannot be participate as a part of infrastructure because it has no proof That it can preserve show or to aspire server And and be part of infrastructure Anymore so that that's basically how we're looking to this Yeah, it's um I'm fairly familiar with spiffy inspire, but Yeah, I'll be really interested in exploring that definitely Um, we have one question to chat Um, do you have any thoughts on detection of key compromises? Okay, so um key compromises In general sort of people's Personal keys or within the tpm or uh right now. Do you do want to clarify on that question? In the tpm Somebody does get access to the keys then how do we detect? That's an entity is already compromised. So if somebody if somebody compromises the entity key I just fall into a heap on the floor and start crying Because my whole project's ruined. Um, so how would you detect it? But well, so this is the ek and this is the ek right? I think the ek is so No, it's a very good question. I don't mean to make sort of light of it with with humor um, so At present that There hasn't been any successful even theoretical attacks around getting the entity key from the tpm And people are doing trying side channel stuff putting microphones and oscilloscopes and trying to sort of measure keys that way So it's not being proved that there's any sort of attack. I guess If an if if a key was compromised Then they would also need to compromise the manufacturer because they have an intermediate signing certificate as well of the entity key so the An attack that happened it would It I believe it would be detected because suddenly there would be real tpm's Requesting attestation that don't have an intermediate certificate That's that's that's tied to them. So what we do in key lime is the the entity key We actually have a local storage of some certificates from the vendors themselves that allow us to To to attest that that's an actual entity key. That's part of a real tpm. So to say so that that would yeah, that would be the detection Sure, thank you. Yeah, no worries a good question other other option could be used a certificate transparency, but in this case They're basically tpm vendors should support this Uh with their infrastructure. I I'm not aware of anything like this But that could be another measure and if you use and if you use a hardware key to rob anything Inside you you can also use a certificate transparency for that So so interestingly on that note So this this is a project that i'm working on but it's obviously not a cncf project, but we are Working on a transparency log to record these sorts of things to pick up a key compromise specifically so It's it's not a cncf project. It's it's a prototype that i've been working on With somebody from google, but we use the same back end as the certificate transparency system Yeah, I think they use a trillian for that yeah, so we built a trillian personality as a sort of secure supply chain ledger manifest ledger And um, so but it's it's very early and it's it's quite more we've got You know and and one of the things you have to say one of the ways that that maps to key lime is We're going to look at Where you have these golden lists Okay, we're going to look at recording those into a transparency log to prevent some sort of freeze attack and to provide a source that people can Make sure that the list that they have is what everybody else is seeing there's some sort of consensus there And allow auditors to query the log as well and to check the log Who knows maybe a couple of months i'll be pitching that project I'll be great. Thank you. Yeah Okay, I think we're almost out of time so um Let's wrap up here. Thank you so much. I think this was a great session. Um, there's a lot of discussion around this Um, and it looks like there's some follow-up items that we can talk about as well, which is great as business suggested um, so just kind of along the lines of The process and now that key lime is in sandbox If there are any plans into going to incubation Um, we have a security assessment process um, so whenever You and other members of the team are thinking about incubation Um, you know, please reach out to us. We can figure out what what we have to do that Um to give recommendation for due diligence Um All right, so and I think next week. Um, we are going to have a conversation with Um, Santiago for in total again kind of get some updates. What's new with in total? Uh, in total is applying for incubation So we're talking about um, what are here the things that have changed since we've done the security assessment As well as identify points of recommendation for into the full incubation cool if not, um Any other topics that we want to bring out anybody that has a topic for the future Awesome. If not, thank you everyone Thanks. You're welcome Sorry, I was muted there. Yeah. Thank you everybody. I'm going to try and come along as well be a regular if I can Awesome. That would be great. Uh, look, do you mind to provide a link to presentation? Is it like public level? Yeah, yeah, yeah, of course. Yeah, it should be. Yes. Um, Yeah, you can you can you post it in the um the issue on Issue 411 for security. I'm gonna place it in the yeah, if you pop it into slack for me, um, I'll I'll I'll put it in now. I'll make sure it's shareable and Everybody can access Awesome. Great. Thanks everybody.


Hello and warm welcome everyone joining this session. We have Shakti Kandan today with us to share his experience that I love the doc in Fast and Curious benchmarking multi core OCaml. So without any further delay over to you Shakti. Welcome everyone. So this session is on benchmarking multi core OCaml and their experiences with the same. My name is Shakti Kandan. So when I actually submitted this presentation at FunctionCon we got part of Psychpol Systems based out of IIT Madras. And this is a continuation of the work done by my colleague Tom Kelly from the University of Cambridge in the UK. And now all of our OCaml Labs systems have been merged with the company Caridus based in Paris in France. This is my email address and my social media handle. So the outline of this talk I'm going to talk about what is the OCaml benchmarking suite that we have called Sandmat. And then I will talk about the documentation and the reason that we've been working on this for the last two years. Some results to share with you and also some solutions in the interest of time that we have found useful and also some key takeaways from the session. So what is Sandmat? Sandmat is the OCaml benchmarking suite which is available as free labor and open source software. It's available on GitHub. It has both sequential as well as parallel benchmarks. And if you just look at the top level directory of Sandmat we have a number of folders there from the benchmarks folder which actually contains the OCaml benchmark core. Then we have the dependencies that are required to build these benchmarks. They are again OCaml packages. And then we have the notebooks folder which contains some of the new notebooks which are used to analyze the results from the benchmark runs. And the OCaml versions folder has some of the OCaml variants, the branches that we want to run and test with. And there are some configuration JSON files and some dash scripts. So the source URL for the Sandmat repository is at github.com slash OCaml-bench slash Sandmat. So what we basically do here is we take the OCaml compiler. So the stock OCaml compiler is at github.com slash OCaml slash OCaml. And we basically build that compiler and then we use the build compiler to compile these benchmarks and we run them and we measure some metrics. That's pretty straightforward. So what could possibly go wrong with this? I don't know. What are the challenges that we currently face? So until January 2022, OCaml, the main stock OCaml or trunk OCaml had mostly sequential execution. And the parallel multi-core version was being implemented. And in January this year, we have merged the parallel and concurrency implementation with the stock OCaml. Actually, that was a big milestone of a lot of hard work over the years. It was a very big PR and we expect to have a fired-out release of OCaml in sometime June this year. But from running sequential benchmarks to parallel benchmarks, there has been a lot of changes that have happened. The language has evolved and there have been different variants of OCaml compiler that have been developed. So for example, here, if you see this 412.0 stock, which represents the core OCaml development, then you have 412.0 domains, which basically represents the implementation of parallelism when using OCaml. And there's an effect syntax with some of the academicians actually like to use. So the language is evolving. So it becomes a moving target for us to actually benchmark this. That's a challenge in itself. The Dune is the tool that we use to build the OCaml packages and the projects similar to how you have Cabal in Haskell. And so when this language changes, sometimes even the Dune build tool will fail to compile. So there are ways to mitigate that. We use a previous version of Dune to run and build the benchmarks. This is one hack that we have currently. But those are some things that we have to keep in motors. And for each of these benchmarks, there are a large number of dependency packages themselves. We used to have about 50 to 60 packages and every time we want to update to a new version of OCaml variant, we have to also make sure that these dependency packages work to build along with them. So that is a challenge in itself. Of course, the classification of benchmark runtimes, some run in less than one second, varies from one second to 100 seconds. Some benchmarks you want to run on the CI continuous integration pipeline. Some benchmarks like the Armin data store. Armin is a git based store in OCaml. So those are like database tests we want to run for like two, three days continuously. Those are longer v tests. So how do you classify the benchmark runs based on runtime? So those are things that are very important when we actually do benchmarking for OCaml. So this paper retrofitting palace into OCaml is what I talked about initially, where we have merged the OCaml team has merged the ballism and concurrency implementation with stock OCaml. And this paper is available, you can download it online. And this is the speed up graphs for some of the benchmarks. It's very similar to what we saw in the universal scalability law that Brooklyn mentioned the data for this day in our talk. So right now if you take OCaml, stock OCaml, you have palace and concurrency implemented, you can actually write programs for those. So we use these landmark benchmarks to review some of the OCaml PRs and changes that happen in the compiler itself. So this is an example of a bytecode regression that we caught recently. So bytecode is basically the OCaml interpreter, basically the repel that you use, where you can key in OCaml code, and then you can see the output immediately. So there was this camel ensures stack capacity function that is called very frequently and we're able to detect with this landmark benchmarks with this slowdown and the performance and this got fixed as well. So it's quite useful when you want to track down compiler PR changes for performance repressions. Another example is the runtime tracing. So this is an ongoing PR where we want to have instrumentation and tracing from the runs and say there is not much regression. So the main idea is if you take a sequential program and you run it on the parallel runtime as well, of course with multiple CPU cores you should have scalability, but there should not be much difference between the sequential runs in the parallel runtime as well. So it won't reduce the number of deviations between the two runs. So here we didn't see much deviation here and the PR is actually being renewed currently. So how do we do this? We have a lot of benchmarks and we have different runtimes that we want to select from and we have different configuration files on different hardware. So we use JQ and we use a tag system where you can pick the benchmarks that you want to run based on the tags for a specific configuration. So that's the three different dimensions that we have by which we can actually select the benchmarks that you want to execute on a specific machine. I'm not going to go into the details of these in the interest of time, but I'm happy to follow up with the hangout session if you want to learn more about these. So from the configuration perspective, we work closely with the compiler developers and what is it that they primarily need? So we need a way to specify the developer branch that we want to track. There might be some configuration options that they want to use when they actually build their compiler variant. When you're executing the compiler, you might want to specify specific options. Maybe there's some environment settings that you want to use. So we want to provide these options to the compiler developers when they want to actually run some of these benchmark runs and we have support for all of these. So here's one example where one of the developers wanted to see the impact of changing the minor heat size and be able to do that by changing the environment parameters. Of course, you can measure things using Linux performance tools like Perf and so on. But I will come to the metrics section shortly. So we're not targeting all the metrics. The main idea is to see what the compiler developers actually need. So the GC statistics is something that they are very interested in. This is one example where we implemented a code size feature for the F lambda variant. So F lambda variant has a lot of optimization passes that it does. And they wanted to see the number of camel symbols in the family code. And for some of the benchmarks, we have some counts here. This is a feature that we actually added as part of the metrics, which is basically the execution of the benchmark run comes out as a JSON dump, which we can use for analysis. So the key point here is you can have like 1000 metrics, but then what we really care is what are the metrics that are very relevant to the compiler developers. And that's what we need to focus on. So we have quite a few machines that we use, some important configuration settings that you should be aware of. So of course, this work builds on the initial work like Tom Kelly who presented in ICFP 2019, the International Conference on Functional Programming, where he does some initial benchmark experiments. There's a link to the detailed notes in the reference section. But I'll briefly mention here that something that we did was we disabled the hyper threading. We didn't want to have any crossover or resource sharing the CPU. Turbo Boost was also disabled. We didn't want to have any throttling from external factors. The next CPU isolation is done using Isold CPUs. We passed that in the camel boot configuration. So here is an example of the Navajo server. So this is about the configure course. And this is a benchmark run that basically runs for 64 cores. And so here we basically isolate the CPUs one to four right for the OS itself. So all the interrupts handling for the OS should really go into that particular CPU should not affect the CPUs that are actually running the benchmark. ASLR is also disabled because we want to have repeatable experiments. And of course, power state is also disabled. We are more able to power save more than actually running the benchmarks. So these are two configurations of systems we have that are currently running these benchmarks on a nightly basis. So one is the during machine, just 28 core machine, and the Navajo, which is an AMD one with 128 cores. NUMA is non-uniform memory access. A lot of the hardware these days come with this NUMA configuration, which we used to let it to experiment with, with something to consider. So we do have benchmarking as a service for the compiler developers. So where they can actually select the number of variants that they want to compare with across these two machines. And then we have those time collections count and so on that are shown in the web interface. The URL is sanmark.ocameralabs.io. So you can actually open that website and actually see the nightly run results for these branches. So the sanmark nightly config basically provides three types of entries for the compiler developers. One is they can specify a branch. The other option is to specify a branch commit. So you want to track all the changes on a specific branch from that commit. And of course, we have an expiry field. So let's say you want to work on this particular feature for like for two weeks. And you only want to see the results for those two weeks, then you can have an expiry field entry. And the nightly runs will stop running the benchmarks after that date. And you can also specify a specific pull request. So if you have something that you want to analyze and see if there are any regressions happening for your pull request. Then you can explicitly specify that in the URL. So the current sanmark nightly config is there in GitHub. So that's where developers can create a PR and then the machines will pick it up in the night and it will build the benchmarks for them. We also experimented with Docker. Current bench is a OCaml pipeline, which allows us to create custom builds. And we found between Docker versus native builds to be very close. We didn't find much difference. Of course, with all the tuning settings that I mentioned earlier, you're able to actually run sanmark inside Docker with the current bench pipeline. It's something we plan to expand in the future as well. Yeah, so the key takeaways from this talk, I would say when you're doing benchmarking, try to keep minimal package dependencies as much as possible. If you have some patches that you are building with specific OCaml or compiler variants, it's good to push those changes upstream so that the respective package maintainers can actually manage that for you. The nightly runs are very, very useful as we have seen. We were able to detect a few regressions in most of the PRs that we had. And yeah, it's important to work closely with the compiler developers to see what metrics they really need and only focus on that rather than trying to create a large Pythorov metrics and share more relevance. So it has to be very focused on what they're trying to measure and what they want to analyze. Build failures will happen a lot. The language is evolving and that's something that you need to embrace. The UI that we currently have for the Sanmark nightly config is very minimal. So we only have a few entries, mandatory entries that you can use. But the tooling as such has a lot of options available. So when somebody is really stopping to use the UI, it's good to just have a very minimal configuration to start with and always tune the hardware to measure, especially for the parallel benchmark runs. For the report analysis, we initially started with Jupyter notebooks, which are still there today in the repository. We find that very useful if you want to create new graphs, you want to test some things out. And then once you're satisfied with the analysis, you can do that. A lot of references here on Sanmark, the nightly repo and the nightly config. We do publish monthly reports on the multi-procami work, which are available. The paper references and of course the current bench that uses the docker for its pipeline. We are hiring as well. So if you're interested to work with OCaml, we have quite a number of positions. Feel free to reach out to us. And that ends my talk. Thank you. Thanks a lot, Shakti for the wonderful talk and sharing your experiences in various metrics involved in building the benchmarking framework in OCaml.


Cool morning everybody or afternoon evening wherever you happen to be We have the usual hackfest planning readout from Todd and and then we've got a couple of project reports and We can Pick up the discussion on hyperledger labs if there's time I Hope there's time and maybe we can Get that to closure Are there any other agenda items people need to bring up? Okay, if not Todd you want to take off with the hackfest stuff Sure thing before we go into the upcoming dates Just circling back. We did send out a survey after the Lisbon hackfest Thank you for everyone that submitted your feedback into that We do factor that into upcoming hackfest. So two things I do want to call out One is for future hackfest. We are extending it to three days So essentially what we're going to do is add a day zero Before the two-day hackfest and that's really going to be focused on Developers that are new to hyperledger Helping folks come up the learning curve To be able to better participate in the hackfest bring new developers into our ecosystem And in doing this it'll also allow the two-day hackfest to really be focused on more More hacking more advanced topics As opposed to kind of splitting time partly helping new people come up the learning curve and then trying to simultaneously hack So we think that format is going to work well And so the next hackfest we will be trying that for the first time And then the second piece of feedback that we're factoring in is it sounds like people do enjoy the flexibility of the Unconference format kind of figuring out sessions on the fly But there is a preference to have a little bit more structure in place for this so that people understand where to go when topics are happening And so we're gonna put a little bit more structure in place for Los Angeles So please if there's topics you want to cover Things you want to hack on projects you want to work on Cross-project collaboration get that dropped into the Google doc as early as possible And we'll start to map this out a bit in advance As always we'll leave some flexibility in the agenda for topics on the fly as they come up But we think this will be an improvement for the next hackfest as well So moving on from there We do have the next three hackfest pretty well mapped out So the next one will be February 20th to 22nd in Los Angeles That first date the 20th will be the the day for new developers 21st 22nd will really be the core hackfest This will be at UCLA We have a fantastic venue there. We were just waiting for the ink to dry on that But registration should go live today So I'll be sending that out later this afternoon Onward from there. It is looking like Dubai April 29th to May 1st There is an event there future blockchain summit. I believe is what it's called So we're just working out the details on this will be in touch with more soon And then the next one is for our European hackfest We'll be bringing that back to Amsterdam June 27th to 29th But that is firmed up so we will have more details on that soon So hopefully having these mapped out many months in advance will be better for everyone's travel schedule planning And for us to promote and get new new faces there Any questions on any of that I See one one question from heart in the chat window in terms of day one out of the veterans We're figuring that out right now You know, I think Tracy Dave and some other folks from our immediate staff will be there for sure But that's that's up for discussion right now. So I think let's come back in the TSC next week and figure out who from The more the veterans should be there Sounds great. Thanks. I think that oh, sorry So I think for you know, yes veterans. I think I think it would be ideal if If as many of us can show up so that we you know can handle the Questions and so forth. I definitely think though Todd that you know for planning these days eros that certainly there needs to be somebody who can Effectively represent each of the projects Yeah, good good point Chris So I guess we'll have to figure out how to How to handle that but okay any other questions This is Tracy. I just want to say maybe I can send some time and put together like a tentative Day one day, you know, whatever we're calling that proposal for what that day might look like and then That might help us determine who should be there from each of the different projects Yep, that sounds good All right, we'll take that for next week. Okay, where did my agenda go? Okay, next up is hyperledger. Exploder. No, I'm sorry Explorer Who is giving that readout? Hi, this is Satish from DTC C Hi, Satish Hey, good evening guys So How does it work? Do you open it on the screen chair or do I? I've dropped the link for it into rocket chat. So folks can pull it up on there Yeah, I think it's best if everybody has their own copy Okay, I'll drop the link I'm going to copy the link here I've dropped it into the the TSC channel for those in there. Oh, yeah, perfect. Thank you so So the project helps and we made the blockchain explorer compatible with the version one of fabric And then announcement will be sent out soon. So Part of those the sponsor for this project is working with Tracy Kurt with respect to publishing a blog and In this release we closed about 19 tickets slash user stories and We simplified the configuration to be able to connect to a network easily and there are no issues at this time and The release will be announced soon. So we made an informal announcement in the in the chat channels to sort of get an Early response from the people who are using it sort of release candidate and once we fix those issues We'll be able to send the announcement soon And the overall activity in the past quarter So we made some Significant contributions to be to make it compatible with the fabric one dot zero. The earlier version was working with zero point six and the activity on the Channel the blockchain explorer channel is The yeah, the channel is quite active. So we see people asking questions and we're trying to respond as soon as we can and The current plans in terms of current plan, you know, we are Reacting this to react is to be able to develop easily. So The current version of the code is not so convenient to roll out new changes. So That's what we are working on and a few things lined up on the road map and we are having the road map discussion with With the team of the explorer project. This includes people from DTCC, AmEx and Wan Chen and In terms of maintenance diversity, we added two members from Wan Chen in the past quarter And from contribution wise, so we have contributions from DTCC Wan Chen and other individual contributors That's what the status so far Any questions? Any question for Satish? Hi Satish. This is both I have two questions What is about the React.js? I saw the project is using React.js. So I'm not sure whether it's Compatible with the architecture projects now. We can use that When you say compatible with hyper ledger project meaning compatible with other hyper ledger project. I mean There should be some license issue before and not sure whether it's resolved now Okay, I Am not aware. Maybe we can check with the Linux Foundation legal team Okay, it's still the status. Yeah Okay, thanks, and I saw there are two new maintainers from Wan Chen so a We're doing the where if the companies is Wan Chen or An Chen. I just want to make sure there's no typo This is a company from China The website says one ten. Okay. Okay. That's to make sure Because there are two companies one is Wan Chen and the other is An Chen. The An Chen one should be in China too Okay, okay Thanks Thank you. Any other questions? Okay All right. Thanks Satish Next up is me. Here's the link TSC and Put it in the chat, too. Okay So this is the second installment we've been completely around the horn as they say in baseball and And so now we're in the second rev for all the different projects Obviously we we didn't get a readout on quilt, but they'll they'll join us in the in the next in the next round So in terms of the project health, I think things are still Going pretty well from my perspective. You know, we're continuing to Add new contributors and we're seeing Many of the contributors sticking around which is also good. And so we're starting to actually get to the point where we've got quite a number of Of the contributors Posting ten or more commits it's almost half which is really really a good sign from my perspective So we actually Reduced the the total mix of IBMers we dropped 2% from From the the previous report in October We did, you know, get a little bit of an uptick in terms of the code Commits that were landed since 1.0 from 49 to 50% From IBM, but that's still I think, you know, from my perspective, it's it's still a good sign I think that the bump is probably likely because we're coming up on a release and a lot of fixes and so forth and a lot of those are coming from IBM so We've added a number of commits, I think the number is something along the lines of 300 plus And we've been putting out releases on a monthly basis Consistently, I think the only month that we didn't put one out was the end of December Because of the holidays, but there really also weren't any any any serious defects to warrant it and we're in the process of Working towards a one-to-one alpha code freezes actually or not code free feature freezes tomorrow, and then we'll be working on the the death march to to knock down the bug count Although the good news is that the while the bug count is Something like 130 something a lot of those are actually questions and not really bugs So we just need to go through and triage some of those So I'm I'm pretty confident that we'll have one-to-one alpha out by the end of the month and and then probably beta in February and Final one that one release and in March I Think you know I skipped over the issues. So the issues that we have are still relating to Jira. I'd really like to sort of tee off on the Obtaining a Jira expert to help us configure Jira With github and Garrett so that we can be using it a little bit more effectively have it automatically changing status and stuff like that as well as You know just getting the the workflows and so forth Much more effectively configured Than they are so I don't know Todd or I don't know if Dave is on and what the status of Getting a Jira contractor into you know an expert to come in and help us with that but it's still a pressing need Yeah, hey, this is Dave We have been Making some improvements to Jira lately We've been able to get the security permissions and stuff all lined up We do not have anybody lined up necessarily to come in and fix it, but that's on my short to-do list actually so Expect an email from me out to the TSC to start gathering some of the requirements Yeah, it's actually not to say it's a bit difficult right because it's not just a Jira guy Right, and it's somebody to help us with the entire integration So when somebody has an issue that he works on and then he submits a change request We would like to see something like in review. It's the whole we need everything like the three mechanisms to be in sync So yeah Yep, that's challenge that would be great. Yeah, so I'm maybe Yeah, so in the coming weeks will will definitely I'll be doing a survey or something will be gathering requirements So we can figure out what we need to change Okay Continuing on so I covered the releases In terms of activity, you know, the mailing list continues to be fairly active. We had a Actually a nice spike in in December and November rather Traffic dipped in December, but it actually it dipped from the spike back down to just shy of normal levels of approaching 200 or so a month And and rocket chat again continues to to churn away And we we actually as I was putting this together I went to stack overflow And we had just ticked the 1000th question on hyper ledger fabric tag, which is Nice little milestone. We're about hundred one thousand and three actually this morning Where might current plan current plans in terms of Where are we going with fabric? Obviously we we we plan a one-to-one release in the first quarter That's tracking currently 21 major features and epics And and again as with as always is the case. We're tracking everything in Jira We have a dashboard that sort of highlights where we are in terms of getting there and You know, basically the the most important one obviously is going to be getting True live upgrade compatibility from 1.0 to 1.1 So that you don't have to bring a network down to do the upgrade Maintainer diversity actually grew by virtue of the fact that we actually lost a couple of IBMers to attrition To State Street and also to consensus and so now the the total mix is there 15 maintainers nine from IBM Three from State Street one from a Sarah one from Huawei and one from consensus And just a note. We did actually go through a bit of a retirement process for the maintainers from digital assets Sad sad note I although I I did hear that actually I think he's doing well, but Tamash actually had to retire from digital assets and he had to move to Germany for some to get some treatment for some health issues and And the other maintainer From from digital assets was reassigned and so he hasn't been working on the project for a better part of a year Go ahead with somebody chiming in no, no, I was just naming him. That's right. Yeah In terms of contributor diversity, we continue to add new contributors as I mentioned and we actually have about half of them that are You know starting to really engage where the number of commits is an excessive 10 So that's actually quite quite good news Basically 18 new contributors from four or more companies again We don't Always have true visibility into who people work for because they're using Gmail or other You know personal emails for their commits, but It's it's good to see the the increasing Contributions and I just sort of highlighted here the notable increased contributions from Secure key states 3 to Tachi SAP Oracle and IT people And that's pretty much it unless people have questions No questions. All right I guess we should move on to the labs proposal then Tracy or no one of you guys want to run this one. Yes. Hi, this is Sorry But I should be able to speak so just if he goes off don't worry, I'll be back shortly So Yes, the labs so I made a fairly significant revision to the document as I said in my email For those who haven't seen it. I did listen to the recording from last week's meeting and I Try to address You know the points that were made during the meeting. I especially like the change From maintainers to stewards for the labs I was actually struggling with that part to clarify They were like kind of two sets of maintainers the in the lab themselves and at the higher level of the org So the Stuart stuff makes that completely clear, which is great As I said in my email the main change for me the significant change I would say is The clarification in the terminology so that we avoid any possible confusion as much as possible between the official projects that are being you know under the governance directly of the TSE that goes through the approval process with the heap, you know and Starting an incubation and moving from there and the labs So we started with the terminology of hyperledger lab projects and somewhere actually while I was editing the document earlier I had started removing this because I thought it was a bit clunky But then you know, I had a somewhat of an epiphany when I was working the document yesterday that you know, I realized that in fact just avoiding any kind of you know Combination of lab and project in the same sentence could could be somewhat of a lifesaver in this instance because if we merely refer to labs as such as labs We can keep the term projects for the official projects and never mix the two and I mean, maybe you guys won't be so impressed for me it was kind of a breakthrough and I revised the whole document and In fact, I made slight modification again this morning when I got up. It was on my my pretty much all night and So that you know, even when we talk about moving from a lab Possibly to a project. I had sentences that were still talking about project in incubation Which seems to imply there's a some other kind of project And so I actually removed that to say things like in the first sentence or in the in the introduction the second sentence, you know Can be started with that the creation of a project That's it. There's no like, you know a project incubation. So it's very clear. It's like This really means a lab is not a project. So It I think clarifies a lot of things and I've you know I've revised the whole document I was searching for the use of the term project to make sure that every instance was okay with that kind of definition and I think the document is much clearer in that respect now There were several issues. They were raised that I kind of addressed based on discussion I hope I captured the intent and you know, although there was no formal agreement made during the call There was clearly a few things where they seem to be, you know, emerging consensus and I tried to capture that in the text and and You know, it's it's if people have a problem with what I put they should just raise issues but so as it stands I think There are a couple of issues or three left that we should discuss the first one has to do with what it takes what are the criteria to create a lab and So there was these discussions Yes, last week about a sponsor or mentor I figured it was meant to be the same kind of concept that there is some form of there is a person that will endorse the Lab and that's a condition for the project or the lab to be created rather and There's a question as to okay who constitute who qualifies as a sponsor They were discussions in the on the call as to whether it should be a commuter or whether You know or something else. So I put in the text today that it is a commuter of one of the hyperledger projects Dan already expressed himself in email that he would rather leave me that to actually maintain her I'm really open to that. The thing I actually you can see there There was a point three and point five that I scratched out. I voluntarily obviously, you know I left them there crossed out so that it wasn't just something that disappeared people didn't realize I had initially suggested we use five different people That was based on what the dirt we see does for committee groups There was this notion that you know, maybe we should have at least one commitment When commit that's from the proposer, right the requester of the lab I feel like we probably can get away with only have a sponsor We just need to define what that sponsor is. So I'd like to open the discussion on that if we could start So I put a comment in on this one and again, my recollection from the discussion was that the proposer had to be a committer that was I think we pretty much Agreed that you know to to help sort of Ensure that there's sort of continuity from a committee a Community perspective that they had contributed at least one commit So that and then the other part of it was the sponsor or mentor Hard and I thought that we had agreed that it would be a either a TSC member or a maintainer from another project Others may may disagree, but that's my recollection from last week now. I go back and listen to the tape, but Okay, so the first point you're making means that we would Scratch point five basically right Remove yeah, put it back. Yeah, put it back in does that have to be I One of the proposers because they may be more than one are we satisfied with one of them? Being a commuter or do they all have to be I guess the origin of that came from the patchy labs And yeah, what they they're using that that commit as I think they were they phrase it as a sort of social filter See that that somebody's at least participated once in a healthy way Since we also have this sponsor requirement, that's something that they don't have Feel like so long is is one of the is one of the proposers has a commit. That's fine I don't know if it's complicated to make it More or less specific than that but that this thing I didn't feel like we needed to have both the sponsor and Have the proposal be a commuter Chris you seem to be saying yeah, we want both. I I thought that's where we landed I mean others may disagree, but that's what I thought I mean We're speaking about both of them sort of in parallel and I don't know if we really had a good dialogue about whether we needed both I'm I'm kind of indifferent. I I Prefer the the sponsor be there for sure because that that seems like a higher threshold Yeah, the commit is nice to have but I could go either way on it. I Was my thought that I thought okay if we add the sponsor and now make it like it has to be at least a maintainer I feel like You know Forcing that the the proposal is also a commuter is somewhat you really even because the bar is higher anyway, I Guess it could be I guess it could be I Guess it could be said that you know the social filter is well, you've had to sort of solicit One of the TSC or a maintainer from another project to be your sponsor So that could be the social filters. There's there's a bit of a hurdle there. I Agree, I Do I do Good It seems like the filter around the idea of voting actually might make sense here as well because it might make sense to have Working group chairs also serve as sponsors and then also we might see that's a project come out of participation in one of these working groups That's a good That's a good add actually I think in addition to maintainers or TSC So let's capture that already the the sponsor where you seem to agree that it's either a maintainer of one of the project TSC member I'm writing it now Or a working group chair that sounds good to me if everybody's happy with that that solves that the issue Anybody disagrees That's good So now let's go back to that point five then given that do we want to also require that the proposer? Is a committer? I mean, you know One possibility is that it's up to sponsors to require it a sponsor might say who are you? You know, I don't know you have you contributed to anything and they might say I would like to see you contribute to the project before I sponsor your lab but Yeah, if we want that to be, you know, broadly Apply then it's better to put in the rules So Dan you brought that one up and you seem to be a little bit ambivalent now Is the sponsor? You know sufficient If we trust this one I mean If the sponsor would need some proof to kind of, you know, build up But I think you know the the point that Dan made was that the Apache labs uses it as a social filter in other words it's sort of Again with and and so forth I think that's fair Yeah, and part of what we were trying to also get here is that they're They're part of the community. They've they've been they've been interacting close enough with the community that What they're what their lab is about is is relevant And it's in concert in some way with with what hyper ledger is doing So I think a commit requiring a commit makes that a stronger requirement but Again, I don't feel so strongly about that that We could make it should We could also open it up They've done a Significant contribution to the working group. I know we do that sort of measurements when we do the voting list At least we did this last year But I because the lab is Code project in one way or another. I think it commit might make sense actually so Kelly I think that may be your point and maybe what we could just say is a proposer needs to be a member of the hyper ledger technical community and Leave it at that and so they could be a an active and You know active participant in a working group or committing, you know code Or serving on the TSC that any of those they're they're already part of the community, right? Yeah Doing some change request like I think the more we can encourage people kind of to to participate The better it is right as the TSC so anything that kind of you know encourages interaction. I think it's good So maybe it's another good opportunity to Recreate I know I think that The reference here should just be to you know, e.g. Has voting privileges in the annual Yeah, yeah Wait, wait, so I was trying to capture what you guys were talking about that the really what we're interested in making sure that the proposers I Think what we want to say is the proposers need to be a member of the hyper ledger technical community and by definition That means they have a vote in the annual election and Teria are established by whatever that criteria is Very good. I get it. I don't do we know the exact term? We'll need to put a link there otherwise I can search for I think a link would be to be good Yeah, we find it with VPN back in the day right before the election Yeah, well and where there's ambiguity. I think we err on the side of inclusion there, right? Sure. Yeah. Yeah Yes, of course, of course. We're not gonna be too Yeah, it should be a reasonable TSC And I think similar to the TSC or the votes in the annual election I think we deferred to the workgroup chairs to Say yes or no if someone was active participant so that could be easy way to look at it Yeah It just a quick question. Are there any sort of legal Issues that we have to go through for membership or otherwise or is that taken care of by the other clause? I Don't believe so, Nick. I think I mean from a Participation in the technical community. It's open to anybody. You don't have to be a member you're Making a contribution under the DCO and it's Apache license Yeah, as long as the DCO is I can care of that should at least cover most of the big concerns Yeah, I think the process we're in place where people have to do basically a pull request to to start the process of You know proposing the lab Forces them to have registered with the Linux Foundation and therefore committed So I think we're covered there Okay, it sounds like we're making progress. Can I scratch number three there? I scratch it off I can remove it that in based on what we did. I think it's a Okay, so I'll add a link to the definition we use for the for the vote for the technical community I think that settles this issue. That's great So next we have Anybody wants to add anything to this? No, so let's move on Then Tracy you added this thing. I heard you on the call talking about diversity and while I understand what you're looking for I have to admit. I had no idea how to capture that in the document Yeah, I I Think that's a hard thing to capture, right? So I would be perfectly fine if we didn't capture it, but yet we Recognize it, right? Especially the people who are going to be the sponsors Okay So that's more of a recommendation and it's in line with our code of conduct and all that stuff that you know, you shouldn't You know reject Lab proposal just because you don't like the guy or the person personally for whatever reason, right? So and again, there is you know in case you haven't seen it there is a clause somewhere Forget where now, but that says basically if there are any issues you can always escalate to the TSE Whether it's you know prior to the creation or a spot of the creation process or after for that matter, right? Okay, so Next there was an issue as to the name of the attic space in Github that we would use to move The dormants like deprecated projects or labs. I should say it's hard to change your language But that's what we need to do. So I Given all the concerns we have with mixing projects and labs, I felt that okay, it doesn't cost that much to create a new Github org and we'd be better off using hyper ledger labs attic as a new org for that then reason using the existing one, but I Know Chris you seem to you know, lean the other way around so I Just think it makes the landscape a little bit less confusing, but I I'm I don't really carry the way to be honest one particular concern I had was that we have these wording about You know the fact that things can be essentially Resurrected so a lab that was moved to the attic could then later on be brought back to life if commuters starts being more you know either responsive or active and then you know just looking at the Github Organization you wouldn't necessarily see. Oh, this is a lab We would have a clear situation at the higher level between Projects on one end and labs on the other and then when you look into the attic you just have everything in the same bucket I think it's fine to call it Anybody else as an opinion Dan was you was it you trying to speak up? Yeah, I was just trying to understand clarification what you were What you're trying to ferret out but Yeah, I don't I don't have any issue I just quite understand if it was it if it was an issue of naming the Github org or the process of pulling something out of the attic. Yeah, no, it's not the process We are really talking with the implementation of the Github level here You know you actually want to move away the repo for the lab that's been dormant or deprecated and where do you put it? So two options are either we move it to the existing Is that actually do we have that already the hyperledger attic org I Believe so We have an agreement to have one. I don't know if it exists But okay, so we could use that SM attic for everything or we separate the attic for the labs To keep a secretion between the two. Oh, we don't have an attic. I thought we did. Okay, so it's theoretical Yeah, because I would imagine we only need to Want to use it so that's why I wasn't sure it was even there yet. Yeah, I don't think we need to create it until we need it Yeah, I Thought we did have one dress I I could have sworn we did too Greg. I I Don't know. I just looked and it's not there. So isn't it called hyperledger archive right now. Oh Yeah, hyperledger gosh archive Okay, so we need to change the name there then but so Any other opinion on whether we should reuse the same archive space or or get a borg or if we should create a new one I Prefer keeping it separate for the reasons that you described earlier because as As we've put some things into the archive We've noticed that there are usually are lingering links and lingering activity on those projects And having a clear distinction between what was a lab and what was a project? I think will be useful for the six to twelve months as those links Disappear off the web Okay Any other opinions? Okay, so if there is no other opinion, I think we'll we'll use a different Or at least there are two of us that feel strongly about it Is that okay? Nobody wants to scream at me for saying that No All right, then let's call it done. We'll call it a different Name and I mean use a different space so that resolved that issue Yeah Put an S there. Oh Really? Yes, okay I'll fix and I'll go through the document because he talks about attic right now. So I'll change the name That's not a big deal It's really a tutorial in nature. I'll fix that and then the last one then is We got to the representation that's being made by you know that we anticipate people will do and so There is wording already and I updated that wording based on you know That's clear separation between what we call a lab and what we call a project the question was or is you know should we Make people sign up to you know commit to this to this rule as part of the You know making the proposal to create a lab and I don't know technically how we would do that if it's part of you know, what is it would be part of but It's almost like an eagle thingy, right? Yeah, so I think you know so there is a certain set of branding guidelines that Greg and team put out Not you Greg, but Greg Wallace from the marketing The director of marketing put out And there's a certain policing that goes on I think on a regular basis and so You know, somebody's looking at reports that say, you know, we found on the web Hyperledger something and I think they validate that I Think if it comes up then, you know, it's handled in a similar way, which is You know, I guess a couple of strikes and you're warned and then, you know, potentially For trademark violation basically, but I think that should I think you know rather than forcing somebody to sign You know in blood or something like that I think that part of the responsibility for the sponsor should be to make this known to the proposer and you know You know, I think that's part of the obligation of the sponsor is to sort of let them know what the rules of road are Did the marketing committee review if they were good with it being referred as lab name comma hyperledger lab? I Know that Dan and Greg were both in here. Yeah, cool. I mean I Think given it was, you know sort of gone through we should just make sure that Dan and Greg are given one last Yep So to review and I don't know if I mean I suspect that they've reviewed it with the the marketing team You know the marketing working group, but maybe no, I don't know but I Think, you know, I think we're so I think we're at end of The set of issues, right? Yeah, so just to finish on that one though I you know Practically speaking even though I think it might be nice to have a way to get them to click something saying yeah Yeah, I'll respect that I don't know how to implement it So I'm biased against it just because for a practical reason. I don't know how to implement it So I think you know if everybody is okay with what you said Obviously pending that, you know the marketing communication people Think that it's appropriate to have that But as I said, I mean this text initially comes from I forget if it was Dan or Greg But one of them Ray also was involved and Several people looked you know chime did not that that I updated it based on the language to clarify But that's their text. So I pretty confident they'll be fine with it. Yeah, okay, great Okay, so then I think what we ought to do then is give everybody another week to to review it and And then we'll come back next week and we'll do the hominah. I'm gonna or we could do email Actually, if people are okay with that as well Um, and I would invite Dan and Greg to weigh in as well Todd Cool, and I just tagged both of them into the google doc. Yeah Okay All right, let's do let's do email then, you know, maybe before next week's call Okay Thanks, Arno for Really yeoman's work there and tracy Good job Sure Okay, then, uh, I think we're at end of job and Give everybody eight minutes back and we'll talk at y'all next week Thanks all Cheers Our goatee


Good morning guys, welcome to another Vlogmas video. I was gonna do a get ready with me type video, but it was like so quiet and cycle bar that like I just wanted to like do my makeup and change and then just be done. But I got my egg bites here. I love these things so much. They are so good and it's so funny that like they didn't give me a fork this time, but luckily since they gave me like two forks last time it was almost like they knew that they weren't gonna give me another fork for these babies. And then I got my sugar cookie drink with no like sprinkles because like I don't really like the sprinkles. Not gonna lie, not a fan of those. They actually have like the good straws like I think this is the plastic one. Let me like check. So they finally have like the good straws. They're not paper or like super long and there's no sprinkles so A plus really good. The hair is looking all right, still a little sweaty from class, but it'll dry eventually. So today our big boss like the guy that's like in charge of like a lot of the TV stations like in the nation is gonna be at our station today.  So I was like okay so dress nicely for the occasion. I'll show you guys my outfit later today like I don't know when or maybe after I'm done eating. I'm gonna eat and then head to work or officially back in the building. So we just need to get like everything set up and that's what we're gonna do. Just gonna shell down here, pull out the Gucci belt because why not? Okay so this is the outfit. I'm just wearing this blazer.  I don't know if I'm doing like a good job showing you and sorry there's like someone mowing the grass. And then I'm wearing these shoes. I don't know if you could really see it but that's the outfit and I'm gonna get on the road. Luckily I'm just like 10 minutes away like 15 minutes. So not too far and yeah just feeling rejuvenated with this Starbucks and gotta brush you the hair a little bit more here. It's always so hard when you're like after a workout like trying to make your hair look good. Always so difficult man. So I got my monitor set up as you guys can see. I'm not gonna show like with all my screen just cuz anyway but I moved my stuff over to here. I sit in this little like corner area and I got this phone in my room my boss gave it to me and we went to lunch. I'll put the little boomerang on the screen so you guys can see but I'm gonna try this see how it is because it looks good. The monitors give good lighting yes this is amazing so okay I'm like talking to myself no one's in the room but um there's my project um anyway so Mickey Mouse came to welcome everyone back in the building. I wasn't here sadly but my boss was and he picked this up and was like oh I'll just get it for Alex to try hopefully it's good. It's like oh it's my high school colors teal and purple. Pretty good I hope my app doesn't quit this time. It has a hit of mint a little minty and sweet. Okay so I cleared off all my like desktop stuff pretty much so you can just see the apps I use but I still have Jordan up there he used to work here he was funny and everything but it's my little desk area not that extravagant I got my little plant here I know I already showed this but now I can finally show you more because there's like nothing open um I'd probably typically leave everything on the screen just because there's a lot of stuff like I have open but this is what it looks like and my bluetooth mouse I'm so happy about it I ordered myself a keyboard cover because I like the silicone covers and I'll just like keep it nicer I had to like clean it and everything um and then I like to have a mouse pad just because it's like it's softer this is like my little area I spent today dealing with like wires and like organizing it even though it doesn't look that organized it's a lot better than like what it was so I have like this here just like to keep everything in line this is the hard drive that's like where all my work is and I used to be at my house but it's not now um and just like my little thing here with like sticky notes my phone everything so this is what it looks like pretty much just here working and now I'm gonna go home what do we got cooking tonight well we got chicken from fresh market rotisserie chicken and asparagus yeah fresh asparagus in the oven yep so back up just a little bit and show the chicken there they are she is busy at work busy at work so that is gonna be it for this video I hope you guys enjoyed watching and I know it's probably on the shorter side I'm sorry about that but this is kind of like a day in my life because that's what vlogmas is during this time so yeah if you guys enjoyed this video make sure to give it a thumbs up don't forget to subscribe and I'll see you guys in my next video bye say bye sherry