Good day folks. I get somebody volunteer to make some noise. Make some noise. Thank you. Okay. My speaker is working. We have two way communication. All right. Let's get everybody to sign in. Today is the 15th. And Brandon, I just added some. Thoughts to the. Revoking revoking trust to an artifact. I think it's the problem with doing Monday mornings is we post stuff on Friday. We try to take the weekend off and then we're scrambling to get something in when we in between our. Eight a.m. Monday morning meeting and. Okay. Let's give the couple of minutes. Marina is here. We get some water and we'll, I'll be back in a minute. We have the stability of incoming people. I think we do. All right, Marina. So we have four things. I think yours is the most. Mike likely. Well, I don't know. Probably the first two. Well, maybe all three. My last one, which should go relatively quick. Let's try to do about 15 minutes each. And I'll just, that'll give us a little bit at the end for whatever conversation. Yeah, I think that the actual tech section I added is pretty short. So it should be, should be doable. Let me just get this ready for screen sharing really fast. Sorry. Everyone seeing the top notary design. Overview thing. Okay, cool. So the main thing I added was just this section here. Based on the discussion last week to kind of talk more about how this effort relates to the notary to goals stated in the. In the requirements repo. There's a link there. Yeah, so this goes over kind of which pieces of it. Are the reason we're doing this type of presentation and then we can go more into what the implementation is. So the first thing is the signatures attesting to the authenticity and or certification. I think this is similar to the existing design as well. With the difference that the tough metadata. Kind of provides information about who signed the image as well as who's authorized to sign an image. That's kind of a minor difference there. The other ones I think are a little bit bigger, like the key hierarchies and delegations was one of the stated requirements and tough very clearly through the tough, through the, sorry, the targets metadata provides that kind of hierarchy delegations that you really know who's trusted to sign each thing. And also that prevents some key sharing and some of those other things that we don't really want. Key revocation, including private and air graft, air graft registries. So this is just talking about the, you know, we've been talking about q application a lot. So this goes over that again. Briefly how tough uses the implicit explicit key revocation, which we, we have other conversations about, but yeah, that's kind of summarized there. And then the key acquisition must support users from hobbyist open source projects, large sector vendors. And that's kind of where our tough design has evolved to these different sort of different routes, whether it's the registry for larger registries, public registries with you, that makes more sense. Or whether it's for these organizations where you need some more separation of control, you can have the different routes for the different organizations when that separation is needed. I think kind of addresses that's that step. And as well as kind of addressing these, I kind of just add a note that is also, we also adapted the design to work with the other goals of the project. Tough doesn't specifically address those goals, but I think it can work well with other things that address those goals, like the moving between registries and other pieces like that. Like that's not why we're doing tough, but that's I think something that will work as well. Basically, do you have questions about this or kind of the, any other questions about like why we're using tough, piece of the design is important. I think the piece of it you're talking about the hierarchy of keys that can be used to sign makes a lot of sense, whether it's individuals or different nodes of a multi-node build server, right? Like, you know, you've got four different build servers and you want to know that things that came out of build server a or different from build server B and C. So if build server a winds up being compromised in some way, you can differentiate the services, the subservices versus people because we hope in most cases, these things aren't signed by people. So that hierarchy makes a lot of sense. You can see, hey, this all goes back. Yeah. Wabbit network signed it, but if you care about a level of detail, you know, because something happened, you can see that not only the Wabbit network signed it, but it was build server a to somebody that cares. That part makes a lot of sense. So the place that I was struggling a little bit was two parts. There's a larger constant updating thing, which we can talk about, but the multi-signature piece. I was trying to understand how you're proposing that, maybe that maybe I'm jumping down below.  That's fine. If you have questions there as well. If anyone has questions here, maybe we should do that first, but I'm happy to go over that as well. I'll give it a second and then I'll answer your question. Totally fine. If anyone else has questions here, and then I have to go into that as well. I had a comment here. I think, you know, it's great you're calling out the goals of the overall loader V2 project, but we'd also kind of come up with some requirements we'd agreed upon in the key management space. I think calling some of this out specifically would be helpful. Yeah, that makes sense. I think that the main reason I haven't yet is I think that document is still a little bit up in the air and I might be wrong, but I'm happy to add that here as well. Yeah, there's, there's I think the pull requests that we have out right now contains all the other requirements we have agreed upon. The three that are still up in the air are ones that I pulled out from that drop requirements. So I think that's a good place to start. The other three docs as we come to agreement there, so we'll need to come back and revise this stock to include those areas as well. Yeah, totally. That makes, yeah, I can, I'll go through the agreed upon ones this week and, you know, iterate as needed. I think that makes a lot of sense. And we can get them agreed. We should get a merge. I mean, that, that even this document, I realized a lot of the comments I made were stuff that was after it's merged and that's fine. But let's, it's really hard to track all the loose things that are PRs versus things are merged. So let's, let's say there's a group. Let's try to focus on things we could agree, get those committed. And it's not that we have to, they're etched in stone per se, we can go back and change it, but let's, it's really hard to reference something in a PR versus something that's been merged. So like Niaz, let's, let's try to get, and I, I don't know, I've been back, I've been trying to free up more time. But if you feel like the content is ready for review and ready to be merged and let's mark that and let's see if we can get that done. So let's see, back to your other question about the multi-signatures. So just to make sure I understand the question is, are you asking about like how tough, organizes signatures in the, in the targets metadata or you're asking about like how multiple different signatures, how you can require multiple signatures for a single artifact. It was a little bit of both. And I, I had a really good write up where I was quoting and commenting, quoting and commenting and the lovely browser internet failed me and lost it. So I was trying to reconstruct as much, but it was too frustrating to try to go back and get everything from detail. The, there was a reference, the way you were talking about, there's two parts, the way multiple signatures would get added. I was making, I wanted to make sure that we were maintaining this immutability aspect of a registry that if I add another signature, I'm not changing anything to do with the other content that's already there. Are they truly additive was the first part. And I'll come back to the, how many signatures should be considered trusted to, to move forward, but because I do that more as a policy. Yeah. And I guess I'll answer probably slightly different, but related questions. So like, so the way the targets metadata works is like each target's metadata is independent of the others. Well, mostly independent of the others. Like they do delegate to each other, but as far as uploading and being valid, they're independent of each other. Independent. I'll call them files for lack of a better word there, you know, independent things. And you could update one of those, we upload it and everything else would remain valid. As far as individual signatures on those target metadata files, I think that's a little bit trickier to answer because I think that the, you know, the targets metadata isn't valid really until it has a certain number of signatures attached to it. And I think this is one of the benefits of using the targets metadata instead of directly signing. The artifact is that you could, to update the signature, you just update the target metadata, which then points to the artifact. Not like, not like points in a registry sense, but it includes data about the artifact. You seem to be implying, like I view the number of signatures that must be associated with something to be considered valid, a policy decision.  It's like, they signed it and that's fine. In the acne rockets environment in dev, they might only need one signature to consider it valid to promote. And then in the next environment, they might need to, they might need a signature from the scanning piece of code and the functional test that says, okay, those two things passed, there's two signatures. So those get added to the image. And then when it goes to production, it says, what did it pass security and did it pass functional testing? Great. Okay. I think I mentioned the concern now. So basically the way this would, that's a particular situation would work is that you'd have, so basically you would encode that policy into a piece of targets metadata. And I think that in this design, we allow that to be either in the targets, metadata defined by the registry and mapped there or in some user to find one that just encodes this policy basically. And then the policy that's in that target metadata says, okay, I want signatures from rabbit networks and from acne rockets. I want, you know, three signatures from acne rockets, two signatures from our networks because that's, you know, that's our policy in that I want to make sure that those are the correct number of signatures. And then once like the client knows that policy, it goes and it tries to download the targets, metadata from web networks for this, which is basically the signature from web networks for this. And it looks at the number of signatures on that and make sure it matches their policy. Does the same thing for acne rockets. So those, those are two independent things and they don't, they don't have to be signing the same targets, metadata. They can sign two different pieces that point to the same artifact. Does that make sense? And I can make that a little loss. Yeah. I mean, it might, it might help to show like I've included in some of these where people have asked questions, I've included a document that shows like, here's what the manifest would look like for this particular case. Yeah. So I think a whiteboard helps a lot with this. So I can kind of draw it out too. Cause here's the thing. I really struggle that policy is embedded into an artifact. Like I think policy is at the same reason why I don't think vulnerabilities should be assumed to be in the artifact either, because things have, things have vulnerabilities that we discover after in most cases. So you want to have an added. It's an external entity that stores that information. Yeah. That wasn't clear. The policy isn't in the artifact. It's in a piece of targets, metadata, which can be defined by the client. You know, like any other configuration would be. Okay. So I don't have to push. I can push the individual signatures, the scanning software that scans it and found that it's whatever compliant, that pushes a signature to the registry. And then the functional test that says, yep, this thing passed. And it's been stamped a certified, stamped a certified. I can push that as a separate signature as well. And they're not, they're not even ordered. Those could happen in different orders. Yeah, totally. And then I could have a special policy at my deployment node, like through OPA. If you look at our scenarios diagram. Yeah. Somehow. So this targets metadata is not something that's pushed to the registry. So it's a little bit of both. And part of this is, is because part of the reason this is confusing is because of the adjustments we made. For the new review to version of tough. And I think maybe there's a better name for the. Client specified configuration thing, but that's.  Set of, um, configurations. And then the client can define their own. Um, you know, and I think that in some cases, like for prior interest use and stuff, the client will always define their own. They'll always have a configuration that, that does that. But if you imagine for something like Docker, it makes also makes sense to have a default.  This is the signature you should use for this, um, artifact kind of thing. So Marina, I want to hit on a quick point. You said there when you were saying that you could pull the targets from say the acne rockets repo and then pull it from your local web networks, um, registry server repo, wherever you're thinking of, are you also envisioning as a potential that we could have all of these targets metadata in the same repository, but separate. And the reason I'm thinking of that is you might want to clone the upstream repository to something local behind the firewall in your own environment and have that constantly synced. And you want to know that the upstream acne rockets, uh, signing data is identical to what's upstream. And so you're copying that verbatim. And you don't want to touch that. Is that something that makes sense in this model? I think so. Um, so the, um, like the piece of tech that would probably be most useful there is this like idea of mapping metadata, which is really just a client configuration saying from which repository to pull things you can do for multiple repositories. Um, I don't know if I have anything in this document mapping that out at the moment, but I think this would be a server side thing. And so from the client side, it's going to see multiple targets in the same repository. Yeah. So then, um, so if the, if the configuration is set up like that, the client could then be configured just to pull it all locally, which I think in a lot of cases would make more sense because then, you know, you can control your own uptime and all that. So yeah. Okay. Yeah. And I think these are definitely really good examples that, um, I think some of them are covered, but I think calling that specifically would make a lot of sense. And so I will, um, either to hear or to the other document. Um, yeah, with like exact scenarios and walking through this. Yeah. And not a big deal, but something that's bit me on some of the other pictures we've got is in dark mode. These don't look quite as pretty. Oh yeah. The black, the transparency doesn't work so well. Yeah. Okay. I just been putting a white background behind them. Keep it simple. I'm the rebel. It was, it was a good, a good example. Thank you for being the rebel that was being vocal as opposed to the others that were just struggling. So it's an easy fix when you think about it. Yeah. So the other piece that I had some questions on is the, um, if you go down to your diagram, it's the Coke and Pepsi problem that we keep discussing that happened. That's what he actually, sorry, you're right. Is the, uh, This one. I think it's that one. Yeah. Where we have contractual obligations that we can't even acknowledge that content exists in a registry. So we can't make any assumptions around. Yeah. Did you have any questions with regards to. To. To companies or even to orgs within the same company. Like we've got to have complete isolation. Was there something about this that suggests that otherwise? Um, no. So I think that this example image is using a registry. Um, cause I think it makes a simpler example here, but I think that we have loosened. I think up the idea that the route should be only tied to a registry. multiple routes on the same registry. For example, a Coke and Pepsi could each have their own independent one. If there's like, if you really truly can't combine them, I think that there are, you know, some security gains you get for some security and simplicity gains really you get from combining them. But when it, you know, if and when that's not possible, I don't see anything particularly wrong with just hosting doing the exact same system, but splitting it. Coke and Pepsi get their own independent metadata files. Yeah. Yeah. In this case, Coke and Pepsi would definitely get their own. The question is within Coke and Pepsi that they want to have subgroups that don't want their information shared or acknowledged across them as well. Is the other problem and the general thing that I saw, because I saw this as a reference to air gapped environments. There's this assumption that when things are all together and you have full visibility of everything, it's the most secure. But if you start breaking it down to smaller units, it's considered not secure. And yet those customers are asking for more isolation because they want more security. So the idea that somehow is less secure in that is going in the opposite direction. Yeah, I think there's two things here. I think, first of all, we created the system with the snapshot Merkel trees where the only information would be hashes, which I. So secure hashes, which are not backwards traceable, as far as we know. Using like, you know, shot shot by 12 or your hashtag with them of choice. So that's part one. And then part two is that like the reason that the the snapshot and other aggregate pieces give you a larger measure of security is that. You can no more as you get better rollback protection, basically, you get a better view of what's currently valid on the registry. And that gives you a better view of. Yeah, when something goes wrong in some other package, you then get a better view of that. I think these snapshot Merkel trees also in some ways address that concern in that if you have auditors auditing the the Merkel tree and are in the contents of that and they detect something that goes wrong, you can then get a similar protection using that. I think it's a little bit more work to make sure that that auditability goes through. But I think maybe in some cases that makes sense, even for smaller projects that don't need the scalability of the Merkel tree, they can use it for you know, the auditability kind of using these. So basically using it kind of as a transparency log to get a similar snapshot protection. Yeah, I think but you keep on coming back to the the same piece of that there's this concept of a public registry where I'm trying to make sure I have the most updated version of a specific labeled thing. And somehow I would do that across multiple companies. This is the part where it's just we can't even begin with the concept. There's any kind of data sharing whatsoever of any type, even acknowledgement that other data exists across two companies or two entities within even the same company. I the piece that I and it comes back to the other piece of where they having to do a constant update. I think I'm just getting more and more and I wanted to make this more of a discussion, but I wanted to get to the other and I was going to write it up more and talking about the the expansion of registries when you think about it from a couple of public to all the private to all the words within that private on how many things are getting constantly updated. It's not just that computers couldn't technically scale. I just don't think I'm struggling with why it's the priority when there's only a few things, few important things that are you do want to know they have the latest tag. And it kind of goes to Brandon's tag sign in conversation. But I just want to I want to make sure we're keep on coming back to whether it be content moving into an air gap environment. It can't be less secure because it's going in air gap environment. It has to be more secure. And there can't be any assumptions that that any kind of data is shared between multiple entities. Yeah, I think that the air gap one makes sense. And I think that with basically a there's a couple of options for basically just extending the timestamp allowances, including, I think, one where you make sure that it's more recent than the previous one and within some window or whatever makes sense for that. I think that the shared data one, I feel like it. I think that I've addressed it by allowing to split the routes into different things or allowing the snapshot, Merkel trees, which only have against secure hashes of information on the registry. And this is it's secure hashes, by the way, of the metadata, but not of the actual artifacts themselves. And so you couldn't even use it to. Yeah, that's what Obama said about metadata, phone records. So I'm just somewhat playing at the game. But there's secure hashes of it. It's not even the metadata itself, right? We can not acknowledge the content even exists. But that's where I think the splitting of the routes allows for that as well. So really to condense it down to to that. And really at that point, if you split, if for a specific use case, you have to split the route down that much. You could almost even combine. I don't want to say this too quickly without thinking about it, but the snapshot and timestamp would both be very small. And you would basically just be an auditable thing, saying, this is what it is now. We can check this later and make sure that, you know, if something went when something goes wrong, we can see when, you know, exactly the point in time when it went wrong because of these snapshots. Yeah. You made a comment earlier that combining routes would make it more secure. I'm not quite sure I followed the reasoning behind that. Yeah. So when you have a route for just, for example, sake, a whole registry that contains lots of different targets, metadata, lots of different images, that basically that the tag starting piece becomes more secure because the you get that the snapshot metadata you'd when you wanted client downloads, the snapshot metadata, they check that, you know, that nothing in that snapshot file has been decreased. So in the in the simple case without the Merkel trees and it's similar with them just for simplicity. So if a client on day one downloads the snapshot metadata file, they download, you know, image A from company X and then the next day they download that same image from company X, but then cover up some image like D from company Y, any better company names. They that that image has been rolled back. So the version number has decreased from that client's view. They can actually see that rollback on everything that's within that snapshot file and not just the particular images that they've downloaded because otherwise all they could see is that the the image that they downloaded is newer. They can't see that the next image they're going to download has been rolled back as well. That's kind of the protection you get. And that also gives you a little bit more of that again, that audit ability that if everybody's looking at this one document, they can then see when things are wrong anywhere on the registry. I think this is still on the assumption that there's a few versions of a few products that you're trying to make sure it's the most up to date. In when we scale to customers trying to deploy software, they're building their own software. Every build is a unique ID. It's not considered the latest because the newest thing might not actually need the thing that works. So they they have to make it. They make explicit deployments around what version they deploy. Yeah, I think that this is basically this is important as long as the we are supporting the tag signing scenario. And maybe we should talk there about that scenario. But as long as you're supporting tag signing, you need to make sure that you have the most recent tag. And so you want to make sure you have the most recent metadata about those talks. And yeah, as you was trying to say something. Yeah, I'm not quite sure I followed the interdependency there. Right. Like having, I think, like a diagram or like a step by step, like that comparison would make sense. Because if you had individual routes, you would have individual snapshots, Merkel trees, right? Like I am I'm still failing to follow how the individual structure would be any different than sort of like having like a single single key. So I can maybe draw it out better again. I think diagrams help here. So I can work on that. I think it'll make this a little more clear. Yeah, if I can try to rephrase it, maybe you can say if I'm on topic or on topic with this one. Are you trying to say when you have a combined route with a lot of different things under it, with a bigger scope to your route that any compromise so that snapshot and timestamp stuff just has a bigger blast radius. And so therefore it has more visibility and more opportunities to get caught. So yeah, I think that's that's one of the big pieces, more opportunities to get caught. And also if you as a user are downloading from multiple from two, for example, different places on that registry, you get that kind of rollback protection for both combined, which is just a little bit stronger protection. I can try and figure out a more like approachable way to present this visit. I can share a paper as well, but I don't know if that's the best way to to share that. You know, and this probably a good kick off to Brandon's next one for tech signing. I think this kind of goes back to this again, the same thing that is, do people reference a fewer set of things that I just want to know is the latest the latest version of that label, that tag, I'll use tag label. And I don't, and I'm not, the majority of quantity of content is not dependent on that. The majority of content would actually be a unique label a unique tag, sorry I flipped using label. It is a unique tag. No, it just so happens that a lot of the things that people build from in the container space do come from these, what I call stable tags, right? It is node nine. And I do want to know what's the latest version of node nine. At the same token, the latest version of node nine is not necessarily the one that I want because the latest version of node nine just did a change that broke me. So I think we just have to be, I get the, I mean, you think about tough, the update framework wants to make sure that the latest version, you're definitely getting the latest version of something. But that is how SolarWinds got exploited. They said, hey, here's an update. And it was the latest version and it did come from SolarWinds. So it would have certified that the latest update to SolarWinds should be installed because it's the latest version. It came out of their build environment. Yeah. And I think we could get into the whole supply chain aspect of this as well. But I think that the bigger thing is that if a particular user, a particular system doesn't want the actual most recent, I think the registry ecosystem allows them to not do that, to pin it to a particular digest that they're downloading instead. I think that the problem is that not everybody does that. And so that everybody who doesn't do that needs a way to make sure that the thing that they're downloading is the correct thing. Yeah. I think it's definitely a user choice.  Right. In fact, your copy of Node 9, so what you could do is as you, because hopefully you're not dependent on Node 9 from some public registry, you're taking that public image, bringing it into your environment. So let's you take Node 9 for the first time and a test and it's fine. And now you put it in your registry and it's signed by the Node community, whatever that means, and Acme Rockets. So you're good. But now there's a new version of Node 9. And as you test it, it turns out it doesn't work. And you don't know why. It could be security fix.  So it's not that it's insecure, it's just incompatible. The older version of Node 9 still stays tagged as Node 9 in your environment. That signature should still be valid. The search should still be valid. And then when maybe either the community figures out themselves, they put a third version of Node 9 out that either resolves the problem or Acme Rockets fixes their problem because it could be on either side. Then they have a choice when to float forward to the newest version of Node 9. And all of those cases, all of those signatures and search are still valid. Yeah, I totally agree. I think that there is a difference between just signing the digest and when there is tag signing needed. I think that this system would work totally for signing the digest as well. But I think that the problem is that registries today support polling by tag. And as long as registries today support it, I feel like there needs to be a way to make that secure. No, we do need to sign signatures, which is again, back to Tia for Brandon. But you, Brandon, you were saying something. I was gonna say to your comment there of saying that you're gonna clone this into your internal repo. And so now you've got Node 9 signed by Company Co and not Node Co, or it's gonna be signed by both. When you're using this internally, I don't think a lot of companies are gonna be looking at this and saying, okay, all of our builds are now looking that it's signed by Node. I think they're gonna say, we internally attested this, we need to verify our internal signatures on this thing. And that's all we hear about. And so they've transitioned, so they might only import if it's signed by Node or Docker Hub. But once it's inside, they're only deploying because of Acme Rockets. Yes. And that's fair. That's fair. I think that could be done with the configuration again. I think it's definitely a real world thing where the new version doesn't work, but I think that that's down to company policy more so than whatever we do. All right, tag signing. Tag signing. Yeah, I think this is a nice transition. Okay. So yeah, I've got two of these. It was originally just one and Justin had a great comment. So I've broken into two, but I'll start with 48. And that's adding this scenario 7.1. And effectively, what I'm looking at is just to say, let's make sure that we are signing tags and that we can verify if a tag was reverted to an older version. And so the risk I'm thinking of is an attacker gets into the registry similar to 7.0. That's why I made this 7.1. And the attackers, they will modify some stuff. They don't necessarily have to modify the actual signed content. They can just revert a tag to a previous state of the tag. And if we aren't signing the tags and verifying if those are current signatures, we could say, let's grab a version of Alpine version three. And that could give us an Alpine version three that has a known CVE. And we would say that's perfectly fine because someone at some point signed it. And so that's where this scenario is saying, let's make that not possible by verifying that tag signature and verifying that's current. And I'm looking at doing this as something that the signers deciding when they want to not sign the old ones. Maybe that's something that goes into tough makes that decision for us. But at least for right now, I'm saying that this is up to the signers to say we no longer want this digest to be valid for tag version three. So I can let people read that if they need a few minutes to read this. But this was kind of the overall point here was just to make sure that we couldn't have some kind of reverting of that tag to go to a previous state. Has anybody read it? Cause maybe that is the 940 till 940 or 40 minutes after we read for people in different time zones. And what I can do while people are debating that one, I could also bring the other one that kind of came out of this if we want to. Do we want to spend more time on this one or do we want to look at the other one and look on both together? I can't read and listen at the same time. Yeah. And I've, you guys have, you had this out for a couple of days so we could have read it before. Yeah. That's right. If it's just me, then we should go forward. A lot of the points here are almost verbatim from above from the 7.0. I just changed a few of them made sense to change. So this is the one. So what we were talking about here is how do you differentiate an evil person versus a good person? So use our previous example of node nine has an update and it turned out it didn't just break acting rockets broke everybody. It's the left pad scenario for instance. So they go, oops. And they do want to know whether they roll back or roll forward is a different thing because you can argue both cases. But let's say that they did intentionally roll back. They want to go back to digest A. How do we differentiate the valid scenario where digest A gets rolled back versus the evil person that wanted to put between digest, I could digest. To me, this is always a roll forward scenario. And so you're going to roll forward to a node nine. And so in that, in that signing of the node, I got to pull the comments up now, don't I? In that signing of this, you're going to need to make sure that you're probably moving forward to a newer version of this tag signature. And so maybe you've got some revisions on those tag signatures saying, here's the newest version of that one. And it may point back to the old Shaw, but it's just going to be a newer revision of this tag reference. So this is, if I'm good, and I'm sorry, I didn't read the whole distinguished me, good and evil, I was just laughing at that. I'll read that. But the evil person can pull the digest A and do a push, sorry, let's rephrase. Obviously there's no anonymous push registries or there shouldn't be. So they have to at least get push access to the registry. So if they get push access to the registry, can they push digest A back to node nine? So in that case, like, okay, so node digest A was signed, but it was not, and it was signed as tag nine, but it's not currently. So then the question, what we haven't solved today was tag signing. We said, we're doing digest signing. So the evil, was it Alice's evil? Do I remember this right? There's some names of people. Anyway, the evil person got push rights, but they didn't get the private key. So now as long as we can say that that tag has not signed, then we know that it wasn't valid. If the valid person did want to roll it back, then they have the access to the private key as well, and they can sign that digest. So that's part of it without fully reading the RFC here. Jay, does that kinda cover that or is there a completely different thing? I don't wanna skip past your pointer here. Oh, it's a joke. Okay, sorry. Well, it was fun. So this is extending scenario seven where we were saying scenario seven is that the attacker pushed some non-signed content. And this is saying, well, they've got some signed content from the previous version that this vulnerable release of this code was out there. And so they just changed all the pointers to point back to that. And if we're not signed tags, they can do that. And then everybody starts trying that vulnerable code. Yeah, we've had a lot of discussions about this issue in tough and basically rolling forward is the thing that makes the most sense in the most number of cases. Because once you've recovered from the attack, you can then just sign a valid thing. Just call version three, version five now. And it'll be all right. Although there is, I think a very subtle distinctions we make about a, what we call a fast forward attack where an attacker pushes like a version number two in max. And so that no, no feature version could be considered valid. And I think this is another thing that tough gets you is protection against that attack as well of the availability to recover from that by rotating keys. But yeah, this is a really good scenario to think about. Yeah, to me, it's what is a tag signature. And so if it's a pointer, there's probably gonna be some kind of signed metadata there that says this tag points to this digest. And then something in there should just have a revision number and some kind of expiration time on it, other things like that. So there just needs to be some reference like that. And then you have the ability to increment that. Well, I think the thing we're really saying is we should sign a tag. We haven't addressed that yet. And in fact, I was gonna part of, I don't want to tease too much into it, but that's part of what I was gonna get to prototype two and prototype three. Like I think prototype, like prototype two where we're making progress on, we'll talk about this, but prototype three, I definitely want to bring in like how should we sign tags? Because not just how, but I mean, I guess it is how, but the point is while I do believe the roll forward concept in many ways, the question is, do I have to rebuild old content? Which I may not actually be able to rebuild the same way. So I think it's important to be able to say that in a registry that has this concept of tags which are movable and digest which aren't, that we need to be able to say that a tag can point to an older thing and it be considered valid and attestifiable. Attestable, is that a word? Verifiable. So I think that's the key part. I think I absolutely should be able to. That's to me, that's something that a tag is just a reference, it's just a pointer. And so you should absolutely be able to make a new pointer to an old digest. Or update an existing pointer to be fair. Like a tag is a pointer. You could argue a tag is a pointer. How do I make sure that that tag can be moved to things and it's verifiable? Yeah. Makes sense. All right, so being conscious of the time here and I know that we've got a few things going on. I'm gonna jump forward in case unless somebody wants to stop me real quick. To the point that- Jan, was yours for this topic or just in general today? No, it's more of a general question. I can ask those after the agenda is done. Okay, so I'm gonna move forward faster so that Jan gets his question in. All right, so adding a scenario for revoking signatures. This kind of has snuck in. There's one of my little denim items there and Justin properly pointed out that, hey, I should probably break this out. And so what it's saying is it will be nice. There's a way to revoke a signature. And right now we've got the concept of revoking a key out there so that a signer can be said that everything he signs, we don't wanna trust anymore. But we don't really have the concept of, hey, I pushed this old digest out there of node version nine, CV was out there, left pad, who knows what went on, something massively broke. And so I as a signer wanna revoke my trust or my signature on just that one digest without revoking everything else. And so that's what this is kind of getting into. Originally I was thinking about just in the concept of a tag, but if you kind of expand it out to overall scope, I just wanna break into a different scenario because we're all society digest. So that's- If it's that bad, why not just delete it? Say it again. If it's that bad, why not just delete it? So can you delete it when we have the ability to copy things? You know, you have the ability to mirror these things out there. And so is it ever really deleted at that point? So this is the feedback that I was putting there is the, just because you don't like it doesn't mean somebody else doesn't depend on it. It's somewhat kind of like the example we just talked about before. And I think we might answer that earlier when we were saying that this goes back to the ACME company when they bring this internal they're probably gonna resign as their own signature. And so as long as they're trusting their own signature, I think we're okay on this one. So I guess that part of the question I was, and I was rushing because I realized I was getting time to the minute.  See if I became as crisp as what was in my head. But I think what I'm sort of wondering is this really a forwarding concept. Kind of like when you look up a part number that a manufacturer put out and sometime later they said, yeah, we don't not gonna make that part but this other part is a replaceable. I forget what that term is called. And I'm wondering is that something we should be adding as opposed to you're actually revoking a signature like revoking a key makes sense. Roking a signature would also, sorry, go ahead. Now I was gonna say, it really depends on what you mean by signing something because signing something to a lot of people means that you're just detesting this came from me at some point but then there gets into this other concept that you sign something you're saying this is secure. That's what a lot of people are trying to get out of the notary projects. They wanna know the stuff they're running is secure. And if you as a vendor that produce an artifact know that this isn't secure you should be able to tell the users that don't trust that anymore. I'm trying not to jump in from Niaz. Go ahead Niaz. Was that Niaz or is that Hank? Sorry, that was me. Yeah, as running like a container scanner obviously the things it's looking for over time changes. So if you wanna have it certified that something is free of every CVE it knows about that's only really good for one point in time. Like sort of as a use case here. And what Hank considers insecure I've just productized his product as Hank. Company A versus company A what the company B considers insecure and two companies that use companies B's security product are gonna be configured differently. So I struggle a little bit with this one as does a company wanna revoke as opposed to maybe be able to say, hey by the way you might wanna use this other thing instead but if you wanna use that it's still signed by us we still signed it. But I guess my point is more so like even within the company if you run it against your scanner and it gives you the thumbs up and signs your container image and you do that next week and it gives you the thumbs down do you just end up tacking on signatures or anti signatures at like regular intervals? How does that ever or does that ever interact? Do you just never worry about it? I'm provoking signatures. I was gonna say revoking signatures makes sense in the sense that it's a finer scalpel than revoking an entire key because the key could have been used to sign multiple artifacts and at the end of the day you're trying to say something I signed before is no longer valid whatever the reason for it may be. So the trade-off here comes in in terms of one information you are looking for from a revocation perspective. And this also kind of goes into the allow list denialist model from a tough perspective as well. Like what are you removing from an allow list versus are you publishing a signature to a denialist? The questions I had here are slightly different. So I'll just kind of wait until the individual signature one is addressed but I think that's a good requirement to have because the blast radius shrinks down significantly. I do want to touch on one thing though because what people want to get from Notary I think to someone implied by Hank's comment I think is there is a separation between here is the content that's in our registry and we can with all validity say it came from entity A, B or C. We can't say whether that content is secure or not. It's not really up. Like I'm not suggesting that Notary is trying to make any of those sayings. It's just saying that it did come from sorry integrity that was the word I was looking for. It's really an integrity stamp that says this content did come from this entity. Now if that entity got hacked and they're no longer valid or somebody the evil person stole entity A's identity then that should be that's the revocation kind of scenario. The idea that when and how and who determines a particular entity's security status I'm just suggesting is an external thing that can be compared to it. But I don't know if it ever gets attached to the thing. Maybe in a company you might want to put a flag that says hey by the way this thing is we don't consider this particular content usable in this company anymore. So that could be to your point of a blacklist. But I think we want to be careful that I think Notary is really around the integrity of something as opposed to its state of security at any one point. I think I generally agree but I think that even ignoring the security implications people make mistakes and they might sign something accidentally. And that doesn't mean that they're keen to be evoked in every single location but maybe they want to take back their mistake. They want to say oh yeah that was actually the dev version not the prod version of this thing that I pushed or whatever. Can you delete a signature or is that okay? Like if you can just delete the signature. Well, but if you delete the signature it's not caught.  Now I was going to say Steve I think what your question here is who is the owner of the revocation action, right? And I think the owner is always the owner that has the route or like owns the route itself. So if for example, if a CVE is discovered it's still not the, I would assume it's not the registry that's revoking the signature. It's still, I think, or at least it was implied to me that the person that originally signed the artifact or owns the route that signed the artifact is responsible for revoking that signature unless I miss something here. Yeah, it's definitely the person that originally signed it has the choice to do that revocation. And one of the implications I threw here kind of answers Steve's question was I really want to make sure an attacker can't go back and take something you revoked and replay it and say that something that was untrusted by the original signers now trusted to a client that's receiving this. Is that the tag signing? It mirrors with a lot of the tag signing. But it's, you know, same idea. They get into the registry they're able to push something out there. They don't have access to your key but they have access to something you previously revoked. They shouldn't be able to pass that off as something you trust. Okay. So with a couple of minutes left let's do one thing and I'm saying this for myself and if others can do as well let's try to take the stuff that we feel is stable and get it merged this week. So let's make a conscious effort to go through the various PRs and see what subset we can agree and what we don't agree. Let's, for the purpose of getting something merged let's see if we could cull it down. And if we culled it down to one line out of 500 then obviously we didn't make progress but let's see if we can try to do that. And he has, I'll get to your questions here in the PR year a bit. Go for it. Oh, I see you want to do that offline you're saying? Yeah, yeah. Okay. So Steve. Was that your other one? Did you cover both of yours? I made it through both of them. Okay, great. I'll let me just do a quick prototype two three so we can get to Jen's pieces. People have been asking like how we're making progress on notary and we obviously got the status report out but people want to know where they can engage and Brandon was asking about some places on this last week too in addition to the OPA work that people have been asking about. So, and there's some new stuff coming up and we've always said we wanted this incremental model. So prototype two, what I was going to propose and I've got it partially written up but I'll get it posted. I'd like to finish up the discovery and acquisition of the associated content the linked artifact stuff that we've been doing because I feel like there's good progress being made there. It's generally useful. It doesn't solve tag signing yet but it solves the other elements everything from how do you associate an artifact with it? Sorry, a signature artifact with it to an S-bomb because I'm seeing a bunch of people ask about S-bomb progress. So I'd like to make sure we get that stabilized in a place that people can actually use because prototype one was a hack to prove the concept. Nobody should run prototype one in any kind of scalable form. Prototype two, I think we have the design that we can make that in a scalable form. So that's part of what I focus on two and I think it's a defined range of stuff that I think we're solid to build upon. What I'd like to do with prototype three is pick up on the conversation that Brandon's kind of made a good point to run the signing of a tag to make sure that it's, we know that basically you need two pieces of information. Not only do you need push to the registry but you need to lease some access to the private key and that's signed. And I'm trying to be really careful to avoid any assumption around implementation detail but I totally convinced that we need to solve that. The other one that came up is this concept of the originating signature is an interesting one also. We spent a lot of time talking about multiple signatures which I think is extremely important but there is a differentiation between the original signature of content. How do you know that Wabbit Networks was the original signature? Now you may not care about Wabbit, you may only care about Docker Hub certified something and eventually intact new rockets. But there's something to be said of who was the originating author of the content. And something we have to kind of think about similar to the way tag signing is not something to registry natively supports today. So we're gonna have to figure out how to make some more changes to it. I think I wanna go back and look at the OCI artifact manifest to see whether it be that or something else. I don't know yet but I wanna see how can we break this loop that the thing we're signing is a digest. So if you take the digest of the thing you're signing and then sign it, you can't update the digest because then the digest is different and you're constantly signing something that you have to change because you've signed it. So I wanna figure out how we can do that with some level of confidence that we can to some extent stick the originating signature as a tightly coupled to the artifact. It's not a link, it's always a link of some sort. Link is a loose word there. But the point is that they're not, you know that that was the originating signature as opposed to an associated signature after the fact. I think that the one thing I would add here as well is that I think that working on the tough prototype aspect as well. And I think maybe this is more like a prototype for but figuring out what that integration could look like and whether that's something that makes sense moving forward. I really think that getting it working will I think help answer a lot of questions and get this whole thing moving. So I'll just throw out there that that's another thing to be working on for anyone who's interested is getting tough integrated with some of the OCI artifact work that's been happening. I wanna actually try to get a conversation with the tough folks together of this concept of something has to be constantly updated is the part that really gives me the most angst. So I wanted to, and we got two minutes left and wanted to give Jan the moment. But yes, so I wanna figure out how to make progress on the expertise you guys bring. I'll put it that way. Jan was yours quick. Yeah, okay, thanks Steve. This is kind of a related to what you just told about the prototyping, but I would definitely like to know the end goal. What is the end goal for this V2 work? Is it gonna be part of that CNCF project and is that gonna be a new version coming out from there or are we end up into the totally new kind of a product there in here? No, absolutely. Great point, cause I know new people are joining and wondering like what the hell's going on with this thing? What is this? So if you look at the Notary Project Notary Project I purposely split that out from the other experiments and prototypes we're doing to be kind of like the website of what is the content that we're working on. And the latest conversations have helped me kind of form the thought process to integrity of content. So you Brandon, Sharon, I can't tell who it's. Oh yeah, thank you, perfect. I wanted to thank somebody, but I see the green bar at the top. So it's really around the integrity of content. Once if you ship a package and you put that safety label on top of it when it gets to the other end you wanna know if it was cracked and modified. That's really kind of the main thing that I think we're trying to shoot for here and maybe just making that statement and getting agreement on that I think will help part of our goals that we've talked about. And the idea is that if you look at the goals of the Notary Project it is around signing content that can move between registries to support the consuming public content mantra. You should not be, the world cannot be dependent on a single point of failure. Everybody has to have their own isolation. But the question is if you pull content into your environment how do you know it's the same content that you wanted? And when you generate content in your own environment whether you publish it or keep it internal how do you know it's still your content? So that's the goal of what we're trying to deliver and then the timeline is I am hopeful that we have a runnable solution by the summer that customers can use in their production environments. It's not to say it won't change but I think we'll get out of a prototyping phase and the output of prototype two should give us the beginnings of pieces that we could run in a production-ish workload with a set of stability. It won't be complete. It won't have prototype two won't have tag signing for instance. And well, basically we're gonna keep on making iterations to the point where we feel like we've gotten to the point where this is good enough for now and because no security is every finite we got to get to a point where we feel comfortable and then we'll build from there. Okay, that sounds good. And I also saw that non-goals is not the compatibility with notary V1. So let's say I would like to build something on top of notary V1 and when V2 is coming out it's not the compatibility. Correct. Okay, so it's kind of a anyway kind of a on its own. This V2 will be kind of its own type of a thing. Well, we're learning what we did with notary V1 and the fact that content can't move within a registry much less across registries was a fundamental stopper. And I think I put it in some documents somewhere that if we, oh, it was a PowerPoint presentation I did this last week. If notary two isn't easy enough to sign the content you have, then we failed. Like the result of notary two should be like this is no big deal. It doesn't matter there was a V1 there's because in practicality there's almost no substantive usage of notary V1. Okay. Like we use it, we implemented it in ACR and it sounds great and customers start to use it like, oh, this doesn't do anything I want. And it's not just we didn't do it as a marketing thing we wanted to figure out what we want to learn we put it out there. We realized like, wow, this really doesn't work. So we haven't heard anybody say that other than it sounds bad is it really a problem? Yeah, I don't see it as a problem. But yeah, I was more interested about the actual timing and if you say that this next prototyping is we are more close to be able to test and run it kind of a not the full production environment but we get more ideas there. And you were saying like in maybe you get something or we get something more by the summer or did I understood that right? Yeah, so I wanna be really careful to not make this an Azure thing by any fashion. That's why I keep on leading to other people where we are gonna get a very like overall has already gotten some versions out. It's just, it's not stable enough to use anything at scale. We will make some changes to notary V, sorry to Docker distribution, CNCF distribution which we have a fork of under the notary projects. So if you got a notary project slash distribution there is some changes that he's done there. And yeah, if you wanna pull up the PRs, yeah, I don't know if he merged it there go to the pull request. We probably should just merge it because I realized this is a little bit prototype refer metadata store. The second one, the bottom one. So that was prototype one. And that is the demo. If you go to the NV2 slash docs slash NV2 demo that explains how to instance that registry and do the end to end demo. Like you can actually do this end to end demo today.  This one. Yeah, and then go to docs and then the NV2 again. And then there's a demo script. There's the read me but then click on the demo script. That demo script will take you from end to end and get the experience working. Okay, yeah. And I pushed an image to make it easier so you don't have to build the image. So we'll get a prototype two version going and you'll be able to instance distribution locally. And then assuming it works the way we want it to work we'll, and I'm hoping others will do this also we'll start running a stamp of it in ACR in Azure. So the customers can start using in Azure. And I hope AWS and Google and Red Hat and Harbor and whoever else will do so as well so that we can continue to get the diversity of feedback to know that we actually got this where we think it's good enough. And prototype two won't be finally good enough but I think if we can get digest signing we've started to make some progress and Brandon's completely convinced me of the tag signing. So we'll get tax on it figure out as well. Okay, thank you. These help a lot. We are over. I have to run about others. But I want to make a conscious effort. Let's try to get these things instead of referring to things that are in PRs let's try to get some stable basis committed. Thanks folks.


So, it's time to start, right? Okay, thanks for coming today. I'm Tom Ennebo. I'm Charles Nutter. And we're the JRuby guys. We've been working on JRuby combined for like 12 years. Over 30 years, right? Yeah. Well, combined, yeah, working together for 30 years. Yeah. And Red Hat is gracious enough to employ us to work on JRuby. Which leads to a question we've been getting a lot this week. Yes, Red Hat has been acquired by IBM. But we don't really know anything. We have no idea what this means. And if we did know something, we wouldn't be telling you right now either. But it sounds like the deal is going to take a long time to happen. So it'll be fun to see what happens. Or I hope it's fun. All right, so we're going to talk a little bit about JRuby. How many folks know JRuby? Have you used it a little bit? So most folks are fairly familiar with it. We always emphasize that JRuby is really just a Ruby implementation. That's our goal, is to make it as similar, as easy, and friendly as it is to use regular C Ruby. We want it to work exactly the same. Pretty much all pure Ruby gems should just work. Modulo little implementation bugs that we continue to fix over the years. But generally, anything that's pure Ruby, if it doesn't work right, just assume it's our problem. Let us know, and we'll fix any issues. But they should be good. It should be solid. There are a lot of C extensions out there. We have support for most of the big ones, like NokaGiri, JSON, stuff like that. We'll talk a little bit more about compatibility and C extensions later. But even though we are a Ruby implementation first and we want to be the best Ruby we possibly can be, we also are a JVM language, which means we have full access to the whole power of the JVM platform and all of the capabilities there. For example, JVM tooling. This is a view of a tool part of OpenJDK called VisualVM. I think it's been replaced by Mission Control now, a similar app. But this is a live view of the JVM's garbage collector, showing the different generations in the garbage collector filling up, getting collected, how much CPU time is being spent collecting and dealing with each generation. Over on the left, you can see a live view of how much CPU overall, how big the heap is, and the nice sawtooth there showing that it's being filled and cleaned up, just like you'd expect with a GC. Of course, JRuby runs on the JVM, which has real native parallel threads. So a single JRuby instance can saturate all cores in your system. You can take JRuby on an 8, 16, 32, whatever way system, run one process, and that's the entire site. You can run the entire application rather than having multiple processes as you need in CRuby. One JRuby could do the whole thing. We also have access to a lot of the fun stuff that's on the Java platform. This is a video of a Minecraft plugin using the plug-in Ruby API based on JRuby. Here, Tom's little script is modifying the number of chickens that hatch out of an egg from what is it normally, one maybe tune? Yeah, sometimes it's more than one, which is weird. To 120, and Tom always tells the story that he was playing around with this plug-in and basically destroyed one of his Minecraft worlds by creating way too many chickens and just couldn't get rid of them. I tried to solve it by actually changing the chicken to wolves, because wolves eat chickens, and that didn't work. Okay, so we have two versions of JRuby that we support right now. We have 9.1, which is the 2.3 support, and we have 9.2, which is 2.5. We, in fact, skipped 2.4 support. Whoa. Another slide that we didn't update. Okay, well, anyways. So, we're planning on retiring 2.3 support. We'll do at least one more point release if anyone's concerned with this. Come and talk to us afterwards, because we want to find out how stuck you are on 2.3, or if there's something we're not doing on 9.2 that we should be doing. Are there a lot of folks still running Ruby 2.4 or lower? Show of hands, yeah, a few folks. This is so great that it's 2.4. Okay. Okay. There's not a whole lot in 2.4 or 2.5 that really break 2.3 apps.  Sure. Sure? Okay. All right, so it's not too bad, but yeah, if anyone has concerns about this, let us know. We're probably not going to spend a lot of time on our 2.3 branch in the future. Yeah, and so the motivation for retiring it is that this Christmas 2.6 will come out, and we don't want to support three different versions of JRuby. So between 9.2.0 and 9.2.1, that was a five-month period. We're very sorry. And then in the last seven days, we put out four point releases. We made up for lost time. We actually, it would be like 30 minutes after each release, we'd get this report, and they're like, oh crap, we have to put out another release. But we're not going to keep putting out four releases a week. We should slow down to about once a month, or a month and a half, and try to stay true to that. Okay, so how do we continue going forward? Well, there's always lots of new features in every Ruby version. This is a great opportunity for anyone here to contribute. We do have a portion of JRuby that's implemented in Ruby. So if you see a new Ruby feature in addition to string or array or something, and you see Ruby code that can implement it, or you know how to implement it in Ruby, we certainly accept the patch. Feel free to submit it as Ruby code. Generally, that will be fine. Sometimes there's performance critical code that we might port into native, into Java or something. But generally, Ruby implementation of a new feature is fine. And this is really how we keep up with Ruby development. We've had tons of folks that have come in and helped us with individual one-off features. We work on some of the big features ourselves too, but we couldn't do it without community. We have an IRC channel on FreeNode, but we're also hosting a Gitter JRuby JRuby room, and of course on Twitter, email, whatever, just contact us, however you feel, and let us know what we can do to help you. So library compatibility. I mentioned that pure Ruby libraries run well. So Rails, Rake, all that stuff, those work great. They work just like they do on regular CRuby. And like I mentioned, if you have a pure Ruby library that doesn't work like you expect, don't assume it's your fault. Let us know, and we'll see if there's something we need to fix in JRuby. So I mentioned that we have support for lots of native extensions. Tom's gonna talk a little bit about some work that we're doing to support a new one, and that's kind of how we keep up with the C extension side. We don't support regular Ruby C extensions, but we're continuing to work on porting more of those over and working with the community. Yeah, so let's talk about OJ. It's optimized JSON. We've gotten a number of requests over the years to support this. In particular, it's a pain point for developers because if there's a direct dependency to OJ, they can always switch to JSON, but if they're using OJ's APIs, then they have to change a little bit of code. But even more problematic is people are increasingly putting OJ as a transitive dependency, and that makes it much more difficult to go and try JRuby out because you don't support it. And it's a native C extension, as Charles said. And the reason why we haven't done it is it's a pretty big one. Almost 20,000 lines of C. You wouldn't expect that in a JSON library, but part of the reason is that there's seven different modes where there's a string and a stream-based parser for each one and the ability to dump, and then there's a compatibility layer called Mimic that allows you to emulate the JSON gem. So our current, ooh, whoa, don't give it away too soon. So the Java side now is nearly complete and we're about half the lines of code. And this is mostly just due to the fact that Java has a better ability to share code. So Mimic hasn't been implemented. The WAB is the last mode that we're working on. And you can see that we're down to 50-ish bugs. So it's not gonna be too much longer. So let's look at some numbers. I went and got, I read this URL at the bottom and they have three different size JSON payloads and they kind of mix up the data that would be representative of what you would normally work with with JSON. I took their word for it. But the three numbers in each section represent JRuby running with the regular JSON gem and then the OJ port. And you can see there, it's about three times faster. So if you're already a JRuby user, this should help a lot. And you can even see that we're maybe two-ish to three-ish in cases against OJ on the CRuby side. So this is pleasantly surprising. When we go to the dump side of things, again, you can see that the JSON gem on JRuby just doesn't perform as well as the OJ version. And the MRI side of things, we're doing a little bit better, but not as much. This is one of the reasons that we're not really concerned about not having C extension support. We've tried to support it in the past, but compatibility-wise, it's just a really massive, difficult API to support. But here, whenever we take a C extension and port it into maybe Java, sometimes even Ruby, there's a red-black library that mimics a native extension, we often can end up being faster than the C version and less code. So usually there's an initial effort to get extensions ported over to JRuby, but then it's pretty solid from then point on. And most of the common ones have been done already. So the only thing we have to do is fix those final bugs and then submit a pretty epic PR. But we're committed to this. And the other thing to point out is there's been pretty much zero performance tuning. So there's a few things that we try to do a very literal port from C and now we can start leveraging some of the things that we know Java can do better. So a new section, we've been running Rails since 2006 and there's been plenty of people that have deployed JRuby on Rails applications. But about the time that we were working on Ruby 2.5 support, which was originally gonna be 2.4 support, we kind of fell behind a little bit. In particular, like most of the Rails stuff just worked as long as you didn't use a database, which is not very useful. So ActiveRecord JDBC fell a bit behind. But that's been solved now for probably a year or close to a year. And here's the results of us just running against Rails 5. Unfortunately, you see a few things that aren't working. This again, it was us falling a little bit behind and not engaging Rails core as much as we should. In fact, in Rails 5.1, we didn't have any errors in action cable or Rails ties. So this was just changes to the testing environment. But when we go and see fork present in tests, we get sad. We don't support fork and we'll never support fork. And those numbers are like 99.9% of Rails tests are passing and most of the other ones end up looking like this. Yeah, so this also makes us unhappy, but this is our problem and we'll do our best to fix that. If anyone is concerned about time precision down to like the 10 millionths place, let us know. So the tests are running well. In general, the takeaway here should be that you can just run Rails with JRuby right now and you shouldn't experience any problems. Obviously, if you do report them. One of the big outcomes of our Rails 5 work though is that we're only gonna support the big three database adapters. However, we've gotten enough community support that the MS SQL stuff is coming along. So we will probably be supporting that pretty soon. For Oracle and DB2, we request that you just go and use a third party gem like Oracle Enhanced. All right, so let's talk a little bit about performance. That's the main subject here today. So JRuby has what's called a mixed mode architecture. We have our own interpreter, we have our own compiler, the code gets parsed in, compiled into our internal representation, our Ruby instructions essentially. We interpret that for a little while. When it looks like certain methods are hot, are being used a lot, then we toss it into JVM bytecode and we toss it over to the JVM. Let the JVM chew on it for a while and the JVM is also a mixed mode runtime. So it runs the bytecode through its interpreter for a while and then eventually it uses one of its various JIT compilers to turn it into native code. And so in this way, JRuby was actually the first native JIT for Ruby just by leaning on the JVM and letting it JVM bytecode down to native code. So I'm gonna show a couple micro benchmarks here and here's the disclaimers. These are very fun to show off a lot of times and good for us as implementers to see them improve. A lot of these benchmarks aren't particularly useful unless you're generating a lot of fractals for your business. We have won these micro benchmarks for years and so we don't put a lot of stock in them necessarily comparing with other implementations. It's more for us to see improvement over time. But we will also have some Rails benchmarks a little bit later that are a bit more practical. So I wanted to mention one of the big features on the JVM that's been helping us since Java 7 timeframe or so about six, seven years ago is a feature called invokeDynamic. This is a bytecode at the JVM level that basically that's the JVM see our dynamic Ruby calls and constants and instance variables and everything else that's dynamic in Ruby as though it were static, as though it were written as Java code and it just inlines, optimizes and JITs like we'd expect it to. This has been improving over time. There's a little bit of a startup hit for the extra dynamicity of it. But performances have been improving over time. Startups have been improving over time. It is still only enabled in JRuby via a flag partially because of the startup time hit that you get from it. But if you flip this flag on and you don't see things get much faster, let me know because it should help quite a bit. So the first useless micro benchmark here is a Mandelbrow generator. It actually generates, I'm not sure if this one generates texture and actual graphic, but it generates a Mandelbrow fractal and then we just see how fast we can do that. So it's mostly a test of numeric performance. It's heavy floating point operations, some manager math as well. And this is another place where we lean on the JVM and newer JIT compilers like Grawl to help us optimize stuff. Here is the code. You've probably seen a variation of this, but I mean it's just a lot of math. A lot of loops and math and we, the big problem for us is that every numeric object we create, a float or an integer, is actually an object on the JVM. And so, another Mac List is gonna create millions upon millions of floating point objects in JRuby, whereas in CRuby they have various native tricks that they can do to get rid of those. So we're leaning on the JVM to get rid of that extra overhead, try and sweep those objects out of the way. So let's see what this looks like. So here is a comparison of JRuby with and without invoke dynamic compared to CRuby with and without the new JIT. And CRuby, oddly enough, the JIT does not help this particular case very much. It may be due to, if you saw Coco Boone's talk yesterday, not being able to escape away some of the extra math logic, might be the floating point math. I'm not exactly sure why. I'm pretty sure it's the floating point. It's the floating point stuff, yeah. So for this particular case, the CRuby JIT does not appear to help right now. But again, this is early days for the JIT. I wouldn't take this as how it's going to be in the future. JRuby comes in comfortably faster than CRuby just with the standard mode of execution. But then of course invoke dynamic really helps us. This gives us a big boost, gives us a two or three times improvement over CRuby. But we can go further than this. I mentioned that we are very dependent on the JVM and especially the JITs. And there's a new JVM JIT, a just-in-time compiler called Gral. It's written entirely in Java. It's been moving forward very quickly. But more important to us is that it has some advanced level of optimizations that the standard JVM JIT, the hotspot JIT doesn't normally have. This is available in standard JVMs. You can flip the flags on. If you're interested in trying it out, you can let us know. We have a wiki page we can point you to. But let's see how this changes things. So here we're just looking at JRuby. Here's our original JRuby result, about three seconds for this benchmark. Here is JRuby running invoke dynamic. And then because Gral has certain optimizations that can get rid, wipe away all of those extra objects that are unnecessary, we get numbers like this. So now we're getting into like 20 times CRuby range for performance. Your mileage may vary. Your mileage may vary. And again, this is a small benchmark. It's self-contained in one method, very math heavy. But on that sort of application or that sort of Ruby code, we are doing very well with the newer JVM JIT compilers. So another area that we've been working on improving JRuby is reducing memory, reducing memory size, reducing the indirection of objects. And that means trying to find better ways to dynamically shape the objects that we create. So Ruby instance variables are dynamic. They come in as they're assigned and we need to have space allocated for them to put them in the object. It really looks like a glorified hash, right? It's just a name, you put some value in it, you can put another name and another value in it and it always just takes on those values no problem. But if we were to actually implement this as a hash, of course it would be tremendously inefficient to have a full hash table for every object floating around the system. Plus the accessing of a hash is considerably slower than just going after like a native field. So we actually are doing optimizations that turn Ruby instance variables into JVM object fields. And then whenever you're accessing one of those instance variables, rather than going through the lookup process and accessing like a hash, you can go straight into a memory location, get that value out. This also reduces the memory overhead I'll show in a little bit. We're doing something similar for arrays. Objects now will all specialize however many instance variables are seen at first allocation time. Arrays, we're doing this by hand just for one and two element arrays. We just have a custom version of array that will have one field or two fields reduces that overhead of having a separate array object. And this is actually working out pretty well. We want to hook these two things together though. Obviously they're basically doing the same thing. They're creating a specific size object with specific fields for all of these values. We should be able to use a single code generator to do all of this optimization and make sure arrays and hashes and objects and structs all are nicely packed into memory. So how well does this help? So this is a single variable object. So you've assigned one instance variable. This is the memory reduction. On the left we have pre JRuby 9.2 or so about JRuby 9.1.17 before we did this work. And you can see that the object itself is about 400 bytes of memory. And then there's this object array, which is 320 bytes. Not only 64 of those are actually reference, bits, sorry, 400 bits of size. So there's 320 bits, 64 of those bits are the reference to the one object that it's holding. The rest is header information, array size, JVM, control flags, and all that kind of stuff. So that's what we're able to get rid of. We move that object into the, we move that reference into the object, get rid of this intermediate array, we have a significant reduction in memory. On a Rails benchmark that we're gonna show a little bit later, a select benchmark, pretty much all the objects in the system are actually getting specialized to these packed versions. So you can see each of these is generated. It just says, like Ruby object 25, somewhere in the system there are a whole bunch of objects with 25 instance variables. But it is right sizing them, putting the fields in there directly and saving that extra access. Like with the objects, here's the array version. So here's a one element array. We get rid of that extra boxing of the array stuff. We move that into the object itself, reduce the memory for every array. And you can imagine in Ruby there are tons and tons of single and double element arrays. To the point that just this simple by hand optimization, we're basically catching almost half of the arrays that are created during that same Rails benchmark. We can probably bump this up a little bit, but beyond like one, two, three elements, usually you're gonna be changing the size of it, you're gonna be adding things to it, you're gonna be a lot more mutable. If we keep these small ones hand optimized, it saves us for about half the arrays out there. We've reduced that memory load. All right, something a little bit more practical. JRuby on Rails performance. So we're gonna look at two aspects of JRuby in Rails performance. Active record, which obviously is the key thing for performance of most Rails apps.  If you're using Active Record, it's probably the main performance problem in your application, the main consumer of CPU. So it's very heavy on the CPU, it has a lot of objects that it's creating, it's a lot of GC overhead. We're gonna show just a few benchmarks, create, read, and update. If delete is your bottleneck, you can let us know, but these benchmarks I think were pretty representative of our Active Record performance. All right, so here's create. This is the one area where JRuby seems to be a little faster. It's probably due to fewer objects being created during the construction process of creating those initial records. But after this it all starts to look quite a bit better. So this is higher is better, this is fine operations per second. So here JRuby with invoke dynamic, comfortably faster than CRuby. I mentioned Grawl and we said your results might vary. This is one of those examples. It's very cool, very new technology, very interesting performance, but on a lot of things we run, it's still not quite there with the standard JVM Jits. So we're still playing with that. But in both cases, comfortably faster running Active Record operations compared to CRuby. Same thing with selects, again, comfortably faster invoke dynamic, really giving us a big boost here. Fines, also faster, looking pretty good. Grawl helps here for whatever reason, it was a little bit faster. That drops back down for update. Update is interesting because so few objects are being created, our performance really starts to shine. It doesn't become a memory bottleneck as much as how fast we can push stuff into the database. And here's where JRuby really starts to look good against CRuby. So if you can get Active Record to run well, then the next problem is just scaling out the entire application. This is a really a classic, difficult problem on CRuby. In fact, whole companies have been founded just to handle this scaling problem. It may help you scale out Rails applications. So on MRI we don't have real parallel threads. The only way you can really get two concurrent requests to be running on the processor is to use two separate processes. We have lots of nice tricks. There's some cool features like the copy on write stuff that CRuby does, so all these forked processes don't use as, they can share more. But inevitably, having multiple processes, you're duplicating some amount of runtime state somewhere. Application loads data in late. Application makes changes to what's in memory and you've now got two copies of it that you have to deal with. So this is a lot of effort to maintain all these instances. This is a lot of resource wasting, too. Each one of these VMs has to have its own garbage collector. It has to have its own optimizing. It does it all on its own and you don't get the benefits of that one shared heap, that one shared memory space. This is where JRuby really starts to shine. We can have a single process, multi-threads, run the entire site. And you can get rid of all of those processes and probably run faster on an individual request basis too. So how are we measuring this? This is not a terribly scientific one. It was just done on an EC2 instance. I ran the benchmark and the application and the database all on the same system. But it was a fairly large instance. It was a C4 extra large, so that's equivalent of four CPUs or so. I didn't use anywhere near that amount of memory for this benchmark, but it was there. So we warmed up for a little while. I will show what the warmup curve looks like so you can kind of get an idea. JRuby takes a little while to get going and then measuring every 10,000 requests. So after we let this warmup for a little while, here's where we come out for a full stack scaffolded application on JRuby. Again, about 30% faster than CRuby here and we'd expect this depending on what your load is. If you've got a lot more Ruby code or you've got some really CPU calculation intensive parts of your application, this will increase. Some of the overhead here is obviously just going back and forth to the database. But we're pretty happy with this result. Full disclosure, this is what the warmup curve looks like these days for JRuby application. So you can see CRuby and the JIT they don't really have a lot of warmup at this point. The JIT, when it does kick in, it kicks in really early so you don't see much of a change over time like this. But it takes us 20 to 30K requests on this particular app to really get up to full speed. So folks will sometimes have warmup scripts if they're deploying some of their production they'll hit it a few times, get everything booted and cached and that's not too unusual for large applications. And I did not get to the graph for this so I will do that later. You can imagine a graph here. So again in the effort to fully disclose things here JRuby does use more memory. JVM because it has a lot more internals, it has a generational garbage collector, it has a lot of its own memory space. It's gonna be 400 to 500 megabytes for a single Rails instance depending on how big it is. You can choke that down to like 200, 250 if you want but the GC will run better with the extra space. And so obviously if your site is a single CRuby instance you're probably not gonna see a whole lot of gain for moving to JRuby, you probably don't need JRuby. But when you get to the point of having 10 CRuby processes or 20 CRuby processes now it starts to make a whole lot of sense. Memory on any cloud environment, any shared environment is extremely expensive these days. That's what you're renting essentially. And 10 JRuby threads can do everything that 10 CRuby processes can probably more with about the same amount of memory use. And then if it goes up from there, site continues to scale, you can do the math yourself and see that JRuby really will end up using far less memory than a large CRuby Rails application. And we're kind of excited about this. This is probably the first year we've really been able to say this. In every way that we've been able to measure, JRuby is the fastest way to run Rails applications. And if the numbers that we have here hold out 20 to 30% faster in a lot of cases, database access and CPU intensive code can be maybe five to 10 times faster. So really excited that JRuby is the fastest way to run Rails applications these days. All right, so we're gonna shift gears. We're gonna talk about something that we're working on right now, which is method inlining. But before I talk about that, I thought I would describe what method inlining is for people. So in this little code snippet on the lower left, calculate cost, we're calling this method add a couple places. And we realize that the add method that it's calling is always exactly the same. So it becomes a candidate for inlining. When you inline a method, you just take the body of that method and replace the call with it and substitute the parameters. So it just becomes some simple math here. The body's there. So one of the obvious things about method inlining as a benefit is it gets, it eliminates the overhead of making the call in the first place. You're not deepening a stack, pushing values, and so forth. But the less obvious and probably more important benefit is once you bring that body back to the method that inlined it, you get additional information that you can do continual optimizations with because you've learned more. So here is the post inlined version from the example. At this point, we might look at it and go, why do we have any variables?  Then you have a single value and then this can just keep going on. That could get inlined into something and you can dramatically eliminate how much work you're doing. So method inlining is very important for performance. And in fact, Java's fantastic at inlining methods. This is probably the biggest reason why the OJ extension is so much faster because it all inlines based on how it's actually used and optimized. But there's always an asterisk. The one case that it doesn't help us is when we make a method call and pass a block, that case does not inline for us. So this is a tough slide. On the left side, there's a couple of places where we're calling times. This ends up being hot code. It ends up being two candidates where we wanna inline. So if you imagine the substitution example here, you might take something and replace the yield there and then you would inline that back to that first call. But that doesn't actually happen. What happens is because there's two blocks being passed into the time method, it sees that that yield is interacting with two types or two different versions. And it just says, nope, not gonna do it. So we've had this problem for the entirety of JRuby. It's interesting to note the JVM itself has this problem. JVM lambdas, which are their blocks, actually will not inline through these cases either. So we're fixing stuff the JVM doesn't do for us. If only they would have supported blocks to begin with. So JRuby has its own method inliner. We just treat the method call and the block as one unit and at the call site or the place where the method is invoked, a call site knows how to invoke the method for you. It also keeps track of some information. Is it always the same type? Is it always the same method? Has it been called enough? And if it has, then we inline it. And so as I said, we treat both the method and the block as a single unit. First, the process is very simple on a slide and very difficult in practice.  There's only one limitation.  So here's a contrived example, but as weird as it is, it represents a fairly common case. So Foo is calling inline me a lot and it's passing a block and in that method, it's yielding a lot. The wiles are just to get rid of having lots of extra blocks because we need to simplify it when we implement this stuff. But if we actually look at the difference with the inliner, we're almost five times faster in this case, but again, it's contrived, which is unfortunate because almost all of our core methods are implemented in Java, so we can't inline those. And of course, we're happy having Java implementations of both most of our methods because they start up very fast and they usually have pretty good performance, but they block us in this particular case if only there was a way around it. Well, in fact, there is, let's say for example, that integer.times method, we wanted to inline that. We go and look up internally to see whether we have a Ruby implementation. If we do, then we use that version of times to go and perform the inline. Excellent. I might not explain this slide due to time, but we decide to inline times. There's our special Ruby times implementation. We perform the inline and that version's on the right. And when we run this version, it's not as good, but it's still excellent. And it's not contrived in the sense that it's a real core method that you might commonly use. It even gets better. If we take the specialized knowledge we know at the call site, like if we were calling five dot times, we could maybe use a very specialized Ruby implementation that doesn't have any looping in it at all. It's an unrolled loop and we can get rid of bounds checking. So there's a lot of really, really fun potential here. And you can try it today. You can just do dash X, I or inliner and see if it works. Things have been, I've been running into less and less bugs with the larger and larger apps, but there's tons of constraints. So give it a try. And if you do get a failure and it's not a big program, I love to see those cases. The logging output's just massive. All right, so we'll wrap up here. Of course, if you're using a Ruby installer like RBN or RVM or whatever, you can just install JRuby, it'll pull the latest version and you'll have access to it. Otherwise, you can go to JRuby site, we've got tar balls, we've got installers for Windows. We try and always keep our Windows users happy. JRuby runs Rails and everything else very well on Windows. So if you have that as a deployment environment, definitely take a look at JRuby. If you are to pull down a current JVM or current JDK, it's probably gonna be a Java 9 or higher JVM. There's some stricter encapsulation here. So some of our tricks to convince the JVM that we need access to real native file descriptors and real native process IDs and whatnot, those things frustrate the JVM a bit. So you will see these warnings about us digging around inside JVM internals. We're slowly getting rid of these and dealing with them in the right way, but if you see them, don't worry about it. It doesn't affect anything in the application. We wanna thank all of our users around the world. This is a small sampling. A couple years ago, we just tweeted out a request for anybody using JRuby to send us a company logo. And then within a couple days, we got this much. Very exciting, a lot of these are really fun ones too. Like up there, NASA. I'm not sure if they're actually, they're associated with the Allen Telescope Array out here in California, but that's one of my favorite use cases of JRuby. The Allen Telescope Array used for the Search for Extraterrestrial Life Radio Telescope Array in Northern California. Of course, all the little bits and pieces are run by C++ or whatever, but the whole thing is orchestrated as a JRuby application. So JRuby is actually helping to search for extraterrestrial life. That's pretty cool. I'm not sure if we have them up here, but one of the scariest use cases of JRuby, the Oslo Airport, a friend of ours in Norway, wrote a terminal application, a GUI application, that is used for refueling all of the planes that come and go from Oslo International Airport. That is a JRuby application. So every time I fly into Oslo, I'm like, oh God, please, hope this thing works. But that's the kind of terrifying thing that's out there. So that's about all we have. Thanks very much. Here's contact information for us. And I think we are ready for questions. We've got about five, six minutes or so. Thank you.