Good morning. Good afternoon. Good evening wherever you're handling from welcome to another edition of cloud tech Thursdays here on OpenShift TV I am Chris short executive producer of OpenShift TV, and I'm joined by my fellow of Red Hatter's Josh Mike and Amy Mike. I'll hand it over to you for the starting of introductions Yep. Hi everyone. I'm Mike. I'm the stuff community architect for Red Hat. I'm Josh. I work on cloud-native projects here at Red Hat, mainly community support and facilitation Amy I'm the open-stack community person for Red Hat and you are an open-stack person. You know me as spots So I'm joined here with or rather we're all joined here with Jim Blair Who is the project lead for the Zule project gating system? And it's a founding member of the Open Dev Collaboratory team as a sysadmin and hacker He gets out of it gets that gets to write elegant code and then try to make that work in the real world You start ACME gating to provide Zule support and services. Zule is a project that is a gating system That helps you stop Stop merging broken code make sure that every change is fully tested and for merging and test those changes amongst Multiple services or microservices rather. So I'm gonna let the expert here Jim. Go ahead and take it away Hi it's good to be here with y'all and Here if why don't I share my screen real quick if that's okay So yeah, as as Mike said I'm one of the original authors of Zule and the current project lead and and and Zule is a project gating system and I Guess the first and obvious question is what is a project gating system? Is it a CI system? Is it a CD system? And to that I would answer. Yes, it is But I see sort of the the the the world of CI and CD as a continuum and The first CI systems if you look at them If you remember what CI stands for continuous integration, right? the first CI systems were you would merge a change to your Java project and you would run the build after doing that and see if the build broke and Basically, this was this was post merge testing. This is testing that was focused on the past Things have moved a bit since then and we have pre-merge testing now So you you write a change and SCI system tests that the change that you wrote works Before you merge it, which is great because you can find out that you that You can find out that it's broken before you've broken it for everybody else This kind of testing is focused on the present So I think gating is the continuation of this and it's a future-looking system. It's about saying not only You know does the change that you just wrote work? It's does the change that you just wrote work when it's When it's going to be merged with the tree which might have moved on since you wrote it other people might have merged changes since then Does it work with all of the other changes that are in flight? you know your colleagues are writing changes at the same time you are and You may or may not be talking to each other when those all get sequenced up to merge is is the thing still going to work and Does your project have dependencies that you know, is it going to work when this dependency changes? so That's that's how I see The purpose of a project gating system is its CI and CD that is focused on the future so a Quick little bit about Zoolz development history. How did we get here? How did we end up thinking that we needed to? make Yeah, but you go back on the the video is actually shown in the in the stream. Oh Yeah, let's see. How do I Can I share? I'm not actually sure if I can share the video and the screen at the same time. It should be able to If I share maybe do two connections Shouldn't have to I have to Yeah, it's normal for zoom if you don't specifically turn the video off. So you turn the video off So don't don't just don't turn the video. Oh, okay. So all right.  It looks like cool. Okay. Okay. So we've got we've got a slide and we've got me now Yep Okay, and the rest of us actually great Cool So yeah, any questions so far while we're There is a question, but I want you to get past the Past the intro first Okay, people have sort of context Okay, so how did we end up thinking that we needed to a system like this, right? Why why wasn't Why couldn't we just Check to see if our build was broken Every morning and so Zool came out of the open stack project it We started working on Zool fairly early on in open stack development and the reason is is You know, if you look at this slide here, I've got a lot of logos on here None of those are actually the open stack logo and that's because when you know, we present open stack Sort of to the world we talk about it as as a single product, but really underneath it's a bunch of Independent projects semi independent projects that are trying to work collaboratively to build a cloud system And so each of these little logos here is representing one of those projects and When one logo in particular is is I think near and dear to one of our panelists today So these I Guess the problem that we were presented with is how do we How do we develop all of these projects together and make sure that? That we can you know every six months release something called open stack where they all work together and And so that's kind of why we ended up with developing this system that was focused on Testing changes before they merge and testing them with other projects dependencies and other projects that sort of thing so that's that's sort of where Where Zool came from and why we we started to go down this road in the first place a Key aspect of how Zool does its work is something called speculative execution We we didn't invent that speculative execution is a pretty important concept in computer science and computer engineering and the idea is that instead of just You know It's basically about Saying what if we What's going to happen in the future if these things happen right so in in and processors You've got branch prediction where processors try to anticipate what's going what are the next few instructions going to be? and And if they're right then they save time if they're wrong then they discard their results And they start again and so that's actually something that that Zool does is it it anticipates that changes are going to merge to other projects and Incorporates them into its testing So we've got two little graphics here on the on the left It's kind of illustrating how you can use this with dependencies So you could have a front-end UI change that depends on a back-end server change that depends on Changes underlying libraries And and you can build up the sort of speculative chain of what is going to happen when all those changes merge and then on the right is is a illustration of How Zool sequences those changes in its testing right? It's it takes the latest merge change and then it it builds on top of that it says I need to test this change that somebody wants to merge and I need to test these three other changes that depend on it Right and if one of those fails it pulls it out of the sequence and keeps going So that's that's a pretty Pretty central concept to how Zool does its work And this my how do you track? How do you track which changes depend on which other changes? Yeah, so that's a that's a good question There's a couple of ways that that that we do that one is There's the there's of course get dependencies, right? So you actually If you're in GitHub, you know, you'll you'll see this as a series of commits in a in a pull request, right? If you're in Garrett Then each commit gets its own change, but it still recognizes the the git commit hierarchy and The git dependencies are Immutable right like there you can't you can't pull one of those out you you you have to merge all of those changes But that is part of the dependency sequence another is the implicit Sequence of changes that happen when when people approve them for merging, right? So I Guess an important part of what I think belongs in a gating system that I may have glossed over is that the gating system Is doing the merging, right? So there's no there's no person pressing the the the you know merge this PR button on github the gating system is responsible for doing that because it's also responsible for for Determining, you know at the moment of merge if the if the change passes tests or not so when when people Yeah, if they're in Github or Garrett or whatever and they're they're reviewing proposed changes and they say yes I want this thing to merge they do something to indicate to to Zool that they want it to merge So in in Garrett you leave actually in both you just you leave some kind of approval on the on the change or the PR and And Zool kind of takes it from there And so you can you can imagine that as you're going through your day and you're you're saying oh, I want this to merge This one needs more work go and make a change to it and then Maybe you find a couple more changes that are ready to merge and you approve them Zool is is is going to start start sequencing those as you approve them So there's this implicit ordering based on on that and That's actually kind of an illustration of what might be going on here on this slide, right? So perhaps somebody approved a nova change to Nova changes in a row that Nova is an open-stack component, right? There's another open-stack component called sender Somebody might somebody else might have approved that But these projects are both open-stack projects they need to be tested together We need to make sure that that Nova change isn't going to break that sender change and and that sort of thing So so this implied Dependency sequence is is kind of the next thing and that's that's the that's how you end up with these sequences of changes Looking at Zool here the third thing is I think the most powerful way of sequencing dependencies and And it's interesting because it isn't it's actually not something we started with in Zool It was it was really an emergent behavior but but now we see it as is absolutely critical to it and that is explicit dependencies between changes and So that's the idea where I guess going back to this thing where you you have a front-end change and you actually you say As part of the commit message This change depends on the back-end change and so not only do your reviewers see that when they're looking at the commit message but Zool sees that and it incorporates it into its Sequencing system and if If you want to indulge me for a minute or so I actually have a sort of Walkthrough for what that looks like if if folks would like a little visual Illustration of that Yeah, I think so. I don't know if it's would it be time for the first question. I think it might might be yeah So somebody actually asked from YouTube stream They're curious If you'd be willing to compare Zool and GitLab CI Depending on how much you know about GitLab CI Yeah, so there it's GitLab CI is is pretty powerful You know if you look at the merge trains Feature in GitLab CI that that certainly has some You know it kind of echoes somewhat of what I'm talking about here about their relationship between changes and particularly the sequencing aspect I think one of the There there are two things about Zool that I think make it really kind of a a standout System and and one is it works with a lot of different code review systems And it works in a lot of different cloud environments. So It's It's actually great for bridging Across projects that might be in Garrett or GitHub Or pack your or Or GitLab all of these we've got support for all of these in in Zool, we're still working on the GitLab driver but that's you know work in progress and and as well It supports a lot of different cloud environments. So Zools execution is it can be you know, not only in You can run it on a container system So you can you can run Zool jobs in Kubernetes or in OpenShift You can also run them on VMs in you know any number of public clouds Or private OpenStack clouds and that sort of thing. So there's really a breadth of what you can use Zool for as compared to some of these other systems and And I think this sort of explicit dependency thing that we're sort of dancing around right now Is is another really key unique feature? awesome Does that answer your question about a seven second delay? Yeah, well, we'll pick him up later on or we'll pick them up later on and If they have follow-up questions Yeah So, yeah, why don't I talk about that a little bit more? so I've I've contrived this example and I'm I've intentionally made it not OpenStack specific. Hopefully this is something that that everybody here can can relate to So I've got these three logos here because I apparently like slides with logos But you know with these logos, how can you how can you not love them? so over on the left we have The the logo for sender the OpenStack sender project. That's OpenStack's block storage system and in the middle we've got go for cloud and the right of course Kubernetes and and the example that I'd like to to walk you through here is this idea that that we want to add a new feature and Expose it through this entire stack. So we're Let's let's add Support for secure delete in in sender and you know Don't focus too much on the details here because I I This is this is a synthetic example and and I'm just trying to create something that sounds plausible Whether or not we could actually do this or need to do this, I don't know but to me this seems plausible, right? So we want to expose the idea of of you know Doing secure delete when block devices are removed From from sender and we want to make that happen in Kubernetes when say you've got a persistent volume That gets deleted you want you want that PVC to To cause the the underlying sender API to do this secure delete, right? So this is a new feature that we have to move all the way through that that entire stack so we we'd start by uploading a change or writing a change to sender and Uploading it to open devs Garrett Open stack uses a Garrett code review system that's hosted by open dev for its changes. So this is what that looks like This is basically a change that's in review We've got the commit message here on the screen where we we were adding the API option to to perform secure deletes, right? So the next thing what we need to do is add support for using that API to go for cloud, right? So we open a PR against go for cloud and And that lets the the the world of go access this sender API and then finally in Kubernetes we update to the latest go for cloud and and have it Use that secure delete option when sender when volumes are deleted So at this point, we've got three changes out there. None of them have merged yet And you know, we can test all of them independently But there's there's there's no relationship to each other. We don't The the people reviewing the Kubernetes change. They're kind of taking it on faith at this point that everything else works You know, they see this PR show up and say hey We totally added this API to go for cloud We're pretty sure it's gonna work You know, please merge our change. So Maybe maybe we want to use Zool to to show that this entire sequence can Can actually be tested together. So so we go back to our Kubernetes change and we add this depends on footer to to the change and and you can see that It says depends on and then it's got the URL for the go for cloud change. And so this is how you do that explicit dependency Specification in Zool. We just made it URLs so that So that you can actually support cross We call them cross source dependencies where where you have Changes in GitHub depending on changes in Garrett, right? So so anyway, this is the Kubernetes change saying I depend on the go for cloud change the go for cloud change depends on this change to sender and if we go back to So so at that point, oh, sorry So I just want to highlight again the depends on here That's it's it depends on to a different system. So we're not even saying we depend on another GitHub change We're actually depending on changing Garrett. That's okay Zool knows about GitHub It knows about Garrett it can stack those three changes up and this starts to look like that that That diagram earlier. We've got these three changes and the little lines Show the connection in Zool This is by the way, this is what the Zool status interface looks like it actually shows you these this relationship So so that's how you stack those up. We can extend this one step further though We This is a brand new feature and perhaps our Zool doesn't actually have a job that exercises it yet and so we can make we can extend this idea of speculative execution where we're saying I Zool I want you to run my tests with these three other changes. We can also say to Zool. Hey, I want you to run a job that you've never seen before So Zool's configuration is in YAML files, of course And so this is what a Zool job definition looks like So if we wanted to add a job that says, you know, exercise the sender secure delete option with Kubernetes We can call that job sender Kubernetes secure delete we can parent it on a job called say Dev Stack Kubernetes Dev Stack is open stacks environment for Performing development, right? So it's it's kind of like mini cube for open stack So you can imagine that there's a job somewhere in the system that sets up a dev stack for for testing open stack and sets up a mini cube for testing Kubernetes and maybe does both of those things for testing the two systems together Zool has a really powerful Inheritance system for its jobs. So we can actually take you know You can spend a lot of effort setting up a job that sets up your infrastructure and then make a bunch of little jobs that make small tweaks to that and so that's sort of what we're doing here we're saying let's take that Dev Stack Kubernetes job and Just Do just add on to that a little bit to test our secure delete system So underneath his job definition We've got the the project definition where we say I want you to run all the unit tests I want you to run the functional test and I also want you to run this sender Kubernetes secure delete So if we wrap all of that up into a change and upload it to To Garrett then in Zool Now we've got a sequence of four changes and the last one runs that extra job that we just defined and When that's done it'll report back to Garrett and you can actually see on the bottom of the screen here the little Sender Kubernetes secure delete job succeeded so so that's You know it's kind of a Whirlwind walkthrough and it's it's high-level, but that's how you can set up these complicated series of of changes across different projects and and See results see functionality changing in your dependencies before any of these changes merge So, you know in this example nobody had to actually merge a change to see this job run to completion and exercise our new feature and that's that's a really powerful tool and we use it all the time to test changes between open stack projects and between dependencies for You know external dependencies as well Now Jim at this point if we work flowed this it would then test this new Merge and patches against all of open stack if it was an internal open stack type deal Yeah, and and you know it's one of those things of Like what what's the test area that you want to cover right? We can They're sort of in in open stack. There's this core set of projects where we we kind of think that Keystone and Nova and sender and some other things that kind of they they're all really tightly Interrelated and we need to make sure that they never break each other and so we're always testing those projects against each other As open stack has grown over the years and as its interactions with other communities have have increased The surface area has has gotten huge right and so You know early on we We we were testing as much as we possibly could all the time that kind of grows exponentially And it's it's ultimately not sustainable to test everything and so we've gotten better about saying You know these things we always need to test This thing we only need to test some of the time You know when when this part of the system changes that sort of thing so So yeah, if we if we you know added the workflow vote on this test to approve it We would certainly be Merging This change only if it pasted past the new test that we added We might not want to always run that change though Sorry, we might not always want to run that test and so we then might make that contingent on well, you know Let's only run this job if we're touching the part of sender That's relevant to secure deletes and that sort of thing so and the other thing to know about The workflow here is that because of this dependency Chain that we've made This change isn't going to merge until the ones ahead of it have merged So since we depend on that Kubernetes pull request We're not going to merge this until that one passes and that one's not going to merge until the gopher cloud passes And the gopher cloud one isn't going to merge until the sender one passes. So Sorry not passes but merges right so you we can maintain that strict sequencing to make sure that not only can we prove out that this Whole new feature works in advance But we can make sure that we don't merge any changes that depend on it until You know the the dependency changes have merged as well So can you accidentally get a circular dependency? Has that happened in the past? Circular dependencies are a really interesting Thing we actually so we with Zool we started off with a pretty strong stance against circular dependencies because we wanted to make sure that the the Continuous deployment of an open stack cluster always worked right and so We wanted you to be able to upgrade your your nova nodes and then Upgrade or you know, maybe upgrade Keystone first and then upgrade Nova to use a new Keystone feature and so forth, right? So we kind of We said that we couldn't require cloud operators to Shut down their entire cloud and then bring everything up again. We had that we had to make Stepwise upgrades and so we mirrored that in Zool We made Zool enforce that by saying you can never create a circular dependency it always has to be a What we call a DAG, right? It's it's just like get it's a directed acyclic graph There it always has to be a strict strict tree structure And and you can't have the you can't have a cycle where one thing depends on another That worked out really great for open stack, but this is also kind of one of those it's an illustration of how we as a community have evolved over the years as More people who aren't open stack started using Zool. They said we don't have that restriction in fact Our software works a lot better If we don't do that, you know, we're doing things like So some of the other users now are are in like the In the manufacturing space. We have automobile manufacturers Embedded systems manufacturers stuff like that and they're like we don't have that upgrade cycle You know, if we're upgrading the firmware on a car, we kind of do the whole thing all at once We don't we don't we don't just upgrade the the entertainment system and then do a stepwise upgrade of the You know of the transmission control system, right? Like that doesn't make any sense. So so They said circular debensis are really important for us because that's that is how we do our development And so we've actually added the the ability to do circular debensis and they work kind of like how you'd expect or how you would As you suggested accidentally do right where you you know You stick this depends on header and change a that points to be and then do the same thing and be that points to a If you do that Zool will recognize that that is a circular dependency And then it actually changes its algorithm a little bit. It says I'm I'm not going to do a strict ordering between these I'm going to recognize that all of the Participants in the cycle need to be tested together And then merged or not merged as a unit So it's still kind of that the graph that will make still kind of looks like it normally does except that there's You know potentially a Grouping of changes that behave That move together and and all of the changes in that group have all of the Git changes checked out for them at the same time as well. So I think that for me to kind of sum up this line of Thought Zool Zool is a tool that lets you ask what if right? It's about saying What if I merge this change what if what if These two changes that depend on each other merge What if this other dependency merges like what what is what's going to happen in that case? And and I think that's where the the power comes from Just You know, I mentioned some of these other we've been talking about open stack and Kubernetes a lot, right? So these are these are some of the other Zool users That are out there you can actually on our website a lot of these users have Stories Posted there about how they're using Zool. So if you want to Read about them open lab is is an interesting one here because they they perform Sort of cross-community testing they're really at the heart of what I've been talking about They do cross-community testing between open stack and and Kubernetes so they're you know If you were to to do this whole sender secure delete thing that I was doing that's probably the most likely place for that job to run Because they're doing that kind of testing today between open stack and and Kubernetes But yeah, we've got a lot of different folks here using it for a lot of different kinds of development which I think is really interesting and exciting and These are some places you can find out some more information about Zool We've got a website at ZoolCI.org Where you know, we've got docs Link to our mailing list our IRC channel on on free node where you can chat with us and that sort of thing the docs have Instructions for getting started so you can get up get a Zool up and running pretty quickly You can also look at open dev.org which is a a very large public Zool system So that's where open stack. That's what opensack is using for its development So you can you can kind of see it in action there You can see the example jobs that we've got the standard library of jobs So that you know, if you if you do use Zool, there's a whole community of Folks working to make reusable jobs so that you don't have to start from scratch and then we've got And you know what I This is one of those things where it for me it almost goes without saying but I should say it And I can't believe we've gotten this far without me actually saying the words open source Of course Zool is a community developed open source Piece of software We're you know, we're not driven by One company this is actually You know a bunch of a bunch of people Working for a bunch of different companies as well as some people working on their own time Contributing to make this thing better so So anyway, yeah, it's open source. You can download and run it and there was also commercial support and hosting offerings that you can find out on the on the website and Awesome, you want to if that's it for the slides you can stop sure. Oh, actually you've got yeah So that's that yeah, that's it for the slides. I'm happy to This is probably a great time to stop for questions if there are any but if not I can Just sort of show you some of like I said, just show you what's going on on open dev.org live right now Here's it. What's the what's the technology stack with in Zool? Yeah, so it is written in in Python and It's actually kind of a it's a sort of microservice architecture and And so it's a it's a bunch of different Python services that that work cooperatively, so there's there's a web service. There's a scheduler that does all of the deciding, you know what to do with What jobs to run that sort of thing and then the executors are sort of there are most important scale out Feature so one of the things one of the Pain points early on actually was as as open stack was was growing exponentially We had to scale our CI system to to match it. So So we actually the the execution component of Zool is Is the scale all component as well And why oh slow computer there we go So if we if we look at This is the the Grafana instance for open dev and If we look at it we can See down here Yeah, okay, so these are the Looks like we've got 12 Executors online right now for open dev. So that's that's 12 servers That are just responsible for running the jobs That's not 12 test nodes. That's 12 Machines that are running the test jobs on the test nodes Right now. We're using Let's see almost Was it round Maybe eight nine hundred Test nodes right now. So these are about eight or nine hundred virtual machines that are that are in use right now Running running tests. So so yeah the the the stack internally is Python and Like I said microservices for scaling out We At this point We so we're kind of agnostic about whether you run Zool itself in containers or not you can you can you know Install it on a bear server or a VM or whatever in in the way that you would normally install install a Python program But I think at this point most of the larger Zool installations are running in containers Some of them are running Zool inside of Kubernetes. Some of them are just running it inside of Docker You know using Docker Docker compose that sort of thing but we that that seems to be the way What people are moving to And and it's really nice because you know we we build we build container images not only on every release but every commit So it's pretty easy to just run the exact container images that we're running in open dev for Zool's own Zool Let's see another Important thing to know about the tech stack is how the jobs are written. They are written in ansible and that's because Zool is a Multinode aware system. So, you know with some CI systems they they've They've got the idea of one test node down for running a job, right? But if you wanted to say, you know, you're doing work with Kubernetes and you wanted to actually stand up a full three node Kubernetes cluster With with Zool you can do that pretty easily. You can you can ask Zool for one node You can ask Zool for three nodes for your job and And Zool will happily provide you, you know, three virtual machines or three containers or whatever it is that you ask for and but if you if you think about writing a Job that uses multiple nodes, you kind of need an orchestration system You need a way of saying I want you to run this on node number one And then I need you to run this on node number two and this other thing on node number three and so we needed a system for saying I want to run scripts or programs or something on on arbitrary remote nodes and There was this great system called Ansible That was developed just for doing that and so we said well rather than you know reinvent the reel and and create our own Orchestration system for doing CI on on nodes. We said let's just use Ansible so what you get as a Zool job author is You write a playbook and Zool runs Zool sets up the inventory for you with the the nodes that it gets And then it runs your playbook against that inventory So if you if you know Ansible and you like Ansible then that'll be really familiar to you if you If you're like, oh, well, actually, you know, I don't I don't want my CI system to be opinionated about what You know how I do my deployment. That's fine, too Ansible as it turns out is really good at running other stuff So if you just have a shell script to do whatever it is that you need to do that's fine You can write two lines of Ansible that say run the shell script on the on the test node and you're done You've escaped Ansible and you know, you can you can do whatever you need to so Yeah, I think that's the the The highlights of the the tech stack there. There's you know an SQL database can be MySQL MariaDB Postgres Postgres girl, they're all supported and then Sort of behind the scenes we have a zookeeper system for helping those components Communicate with each other, but those are all Kind of behind the scenes details You know as a zoo Administrator you might need to know that they're there as a zoo user. You don't need to know that they exist at all so So yeah, this is a this is the current utilization of zoo on open dev and This is the the actual web interface for zoo So if we go look at zoo is a multi-tenant system. I should say so Starting here, we're actually looking at all the different tenants that we have in zoo We've got open stack, which is our largest tenant. We have zoo itself we have some Python projects the the cryptography and the packaging folks have some Have some projects in the system because we're actually All of the communities involved here are heavily dependent on these Python packages and And we want to make sure that this is going to that that cross-community testing that I was getting at earlier We want to help the Python community make sure that That they're able to test on all the platforms that they that are important to them And are important to us too. So we actually have some arm testing going on for these projects down here, but I'm going to Hop over the open stack tenant real quick and just show you what a fairly busy zoo system looks like so no Again, this is live so I don't really know what to expect here. This is This is what happens the the check pipeline is what happens when people upload changes So this is kind of before they've When you write a patch and you upload it for the first time this is when we run the the initial test I've even not gone through code review yet or or or that sort of thing. So If we actually go ahead and expand by default So if we look at this there's a bunch of in progress some have succeeded these jobs are paused All of these jobs are running for this change, right? So so each of these each of these little little boxes here is a change to a different project in the system and then if you expand that you can see jobs that have All of the different jobs that we run for that change.  So here's a here's a change that depends on several others so if This should start looking exactly like that slide that I showed you earlier And and this is you know exactly how we represent it in in Zool's status interface So these lines between changes are dependencies and then the gate pipeline is where we Changes that are about to merge, right? So Here's a queue of two related changes to Airship projects, right? So this one's going to merge first. This one is being tested with it In sequence and and I actually don't know if this is a get commit dependency or or just a dependency that's formed by the, you know, the approval sequence Or it depends on or what we'd actually have to go take a look to figure that out But regardless right now right now Zool considers that they are dependent on each other And so this one's being tested with the one ahead of it and And that's happening in parallel so that they merge more quickly This one these two are sharing a queue, but you can see that this one The one at the top here has failed which means that Zool has decided that This one no longer depends on it So we're still running the jobs on this because we're gonna we want to you know We've sunk a lot of time into running these tests already and we want to report back to the user as much information as possible So we're gonna keep running The jobs until Everything's done, but in the meantime As far as Zool is concerned this change is now The next one that's going to merge assuming that everything passes so no line connecting these two Is how we determine that Zool is has pulled that out Yeah You want to actually click on one of those I'm curious what this looks like in terms of like say if I'm a developer Like actually particularly into a failed test big deal developer trying to debug the failure what this looks like Yeah, so let me let me click on two things one Just while we're here. This is If you click on that open stack ansible one, I'll look at it from a core perspective Oh, okay Yeah, I just wanted to show you the streaming console output that we have so I mean kind of a standard feature bit You know if you if you like to watch console logs We've got that here What so what's handy or is that regex search? Yes, yeah, like, you know fail Right fail. Yeah, so Yeah, that's it's that's pretty cool So that's one tool that you you know if you see that something is is failing you can And and things are still in progress, you know, you can you can check on it while it's running. Okay, so where was the ansible? This one farther down, yeah the one thing so Do you see one? Yeah, and we just added some support for the latest CentOS and Ubuntu so else I think Debbie and Should I should I try this one then the ansible deploy AIO metals into okay? All right, so we'll and to be honest sometimes we just recheck and we go through So Yeah So we're we're actually kind of at a at a awkward place here in that Because Zool is still running this job we can access Yeah, we can we can access the logs, right? So we've we've got the logs here in In our log storage system, so you know, you can see the job output there's a Whole host like we collect a whole bunch of logs When when we're doing this I kind of want to show you the What it looks like when it's finished reporting Because Frankly because it's Easier to use and More There's some some neat Feature there, so I'm just gonna look for that same job and see if it failed recently And we'll look at that instead, okay so So so basically As long as the change is still in progress You know, which is what we've got here, right? We're we're still running which one we're still running this job Which means Zool hasn't like it's not final yet Zool It's clearly not going to merge but but it hasn't Zool hasn't made a definitive determination because all the the jobs aren't finished yet So it it hasn't yet reported back to the code review system the the final results things could still change So so that's why in progress we have access to the the raw dogs But but we don't actually have a final result yet to look at in Zool When it reports back to the code review system This is the the view that you get right and so there's actually a nice a nice Screen that shows you the summary of Over the build, you know when it started didn't finished and the branch the project the pipeline some other stuff We do this thing where we because the jobs are written in Ansible and You know Ansible is a series of tasks. We can actually look at the first task that failed And show it to you With the assumption that maybe there's something useful there. So So this is This is the this is the Ansible tasks that failed And this is the last few lines of it So often as a developer you can hop onto this You know this build summary page and and actually just see the error that tanked your job Right there This doesn't mean anything to me. I don't know if it means anything to you Amy Kind of looks like the container didn't build properly Which is why sometimes recheck works for us because we're a deployment Project so if the containers or the VMs didn't deploy properly will sometimes fail just from that Yeah, so this is this is the I'm feeling lucky Thing in Insul right you can look at this and and and and maybe this will just point you right at the error And you can go and fix it We've got all of the logs That So this is this is the slightly more Attractive version of that raw log storage that I showed you before right so, you know We've got all of these ETH tool config files things about things on the host and and And and this is you know something that we've we've learned after Years in the trenches in open stack development right this this job took an hour to run and My goodness after the end of that if it's failed you sure want to figure out why it failed, right? So that's why we collect so much log information so that when when something goes wrong Hopefully we've got everything that a developer needs to figure out what that was And they can go back and see everything that happened If the zoo has this artifact system, so you might We don't this particular job doesn't have a lot We've just got this script to download all of the logs and then the zoo manifest itself but examples of other artifacts that we have our documentation builds preview sites for documentation so if your job builds documentation you can just add a link here and then Developers and reviewers can click on it and and see the See the see a preview of the doc build Container images tar balls things like that. You can you can stick here as well. And then finally we have a View of The That's a big one A view of the the tasks in the Ansible playbook that make up the job so This is the really big thing that I'm going to hide But if you if you look at this The the job is made up a bunch of playbooks and each of those have tasks and you can you can kind of drill down into each of These tasks and and see what's going on So this particular job has one really long task in the middle that produces a lot of output, which is Yes, you know you can scroll through it all if you want but So anyway, that's those are some of the the tools available to Developers when they they look at failures like this So the folks have different kinds of learning methods. I'm curious Are there any ebooks or anything available for folks out there to learn Zool or what's the like? canonical source of info for folks to go grab So I would Go to Zool See I org and I don't think we have an ebook at this point But we Let's see we've got If you've watched this you you you've probably already seen everything that's in this video, so that's Kind of a short version of what I was doing earlier if you go to the get started page, we've Got a tutorial so This will get you up and so if you're if you're a hands-on sort of person This will get you up and running Just on your local laptop or workstation or whatever Using Docker compose so this will bootstrap an entire Zool system Using Docker compose and you can you can play with it there. We're actually in the process of expanding this tutorial system now so that You'd start with this and then build on that for for other tasks There are there's some other Parts of the Zool documentation that kind of Tackle this from different perspectives. So, you know, if you just want to read more about More about Zool concepts and things like that we've got discussing session sections there and I think there, you know, some videos on YouTube of Conference presentations and things like that And and finally there is a Zool from scratch section in the documentation as well so It's kind of the opposite of the quick start whereas the quick start is like just get me a Zool up and running So I can play with it if instead you are more of the No, you know, actually I want to I want to start with git clone And and go from there. You can do this all from scratch things. So Yeah, I'd love to have you know more of the ebook style stuff, but we do have You know a couple of different styles of Approaching this Web page and doing a ebook if you really want to So that's awesome. So thank you very much for this Overview and dive into the Zool. Oh, you're quite welcome. It's been Great to talk with you about it. Awesome Mike Josh Amy anything else? No word. I mean, I have other questions basically having worked on the Kubernetes project in prow We're once again, I'm looking at this and saying we didn't learn nearly enough from Zool. Yeah Although our biggest challenge is we have looked at the sort of multi project dependencies And our biggest blocker there is running out of cloud resources to actually run the tests Yeah the Because you know given microservice architecture there there are more projects in Kubernetes than there than there are an open stack The projects are smaller and they're more of them So but that could go on for quite a while and we're at times Yeah, no, so thank you very much James for coming on really appreciate it. Yeah, thank you for having very informative Thank you Thank you and without further ado We'll see everybody soon Coming up tomorrow on the channel. We will have an open shift commons briefing about Cloud native network functions. So if you're curious about telco space, which I know some of these people on this call are Tune in and we will discuss. Yeah, and then in two weeks for cloud tech Thursday We will have the folks from CERN Actually, Amy, you should plug this. Yeah, we'll have Delmero who is coming to talk to us about scaling at CERN on open stack So that'll be a really good talk because they're the largest user of open stack in the world So yeah, we're kind of to open stack you related things in a row I Personally takes all for granted because I use it all the time So it was really great to get some more of the history and some of the more in depth usage Which I'll go and use daily now. So I really appreciated this gym So yeah, we're looking forward to having everyone in two weeks And spread the love about Zool Yes, please share you out with folks who you see might need it. So thank you and stop merging broken code Best tag line ever All right, y'all take it easy be safe out there. Take care. I'll


The reason I'm doing this is because I put together this new one which motivates why we have extensions. So let's get to it very quickly. So I call this letter thousand flower blooms. It's actually going to be part of the keynote. The problem is the keynote, I have five minutes so you get the full set. I think it's interesting because it brings, it raises an important problem that we have. Why do we have extensions? This is the reason I think. So first thing is, what is this? We're in Switzerland. This is a cow, right? Most people know companies like IBM have products that are essentially cash cow. And I can talk about IBM but I'm sure it replicates everywhere. And what happens when you have a cash cow? You protect it. It's sacred. But it also is a problem. And the reason it's a problem, if my time would work, is innovators dilemma. I don't know if any of you are in business or if you've read this book. The issue is that every other company will try to kill that cow. And the problem is going to be that you are going to protect it as much as you can. And the reason you protect it is because it's your cash cow. And the problem with protecting it is that the people that are killing it, they're going to invent something new and you're going to miss out on it. This is a problem that happens everywhere. So for instance, Kodak. You've seen me taking pictures so I'm sort of a fan of cameras. But Kodak, as you know, is 100 plus year company. And they dominated film. And then eventually, if you guys remember, I can see all of you are old enough. They've also invented those little pocket cameras. They also invented digital camera. So Kodak could have been Apple because your iPhone, my iPhone is literally a camera. That's what people use the most. But Kodak is almost nonexistent. Why? Because they were protecting their cash cow. Another example of this is storage. IBM invented storage. This was like a gigabyte or a megabyte, sorry, of storage when it was invented. Less than about 50 years ago. And obviously what happened with storage is that it evolved into things like, for instance, this is a one gigabyte Seagate, which literally doesn't exist anymore. You can't even buy this. You could buy it on eBay, obviously, but you can't buy it new. Any of you remember these things? Zip drive, how many of you have still some of those, right? Exactly. And then now, that's the only thing you can really get, SSD. I mean, they are still computer shipping with hard drive, but pretty much SSD is what you get. The reason I mention this is because the guy that came up with the innovative dilemma from Harvard is Clayton Christensen. And if you read his book, the first few chapters talk about how storage, the storage business, literally is littered with these people, these companies, protecting their technology. Because as people went from DASD to things like SSD, it's a brand new storage technology. So if you had, for instance, the technology of the spinning disk, you protected that one, and then you lost on the next one, and then you kept doing that. So all of those companies died or they got merged and so on. So why am I talking about this in an extensions and also an open source? It's because I feel that there's also the open source dilemma. I think seriously that we need to look at ourselves and say, how do you keep your project relevant and people still interested in your project without necessarily protecting what you currently have? And if you look at Cloud Foundry, for instance, we have this problem. We literally was the king. But now we have to, I'm happy that we have Kubo and we have all these other things with Kubernetes, but we could have had what Kubernetes, the excitement around Kubernetes. We could have had the excitement of our Docker. You talk to other people, this might be a controversial statement, right? Because we have a way forward and I'm happy with it, so I'm not criticizing it. I'm saying that the next set of technology two years from now, we need to make sure as a community, we keep the excitement. So how do we make that happen? So how do we protect our, well, not protected, but how do we make sure we embrace the future? How do we make sure that we don't have a cash cow that we're protecting and then we get passed by the next thing? So I think they're example of these and open source. I didn't spend a lot of time trying to beat up on any particular, but I think a lot of people, I saw a couple of people smile when I put Apache struts here. They've been suffering recently from a huge problem in the security, but the point is that if you've ever used Apache struts, you know that Spring Boot is essentially a way better version of that and they protected their Apache struts and now it's sort of causing all kinds of problems. And I use struts. I was part of that community, right? So I'm not criticizing it per se, but I'm saying that how could it have maybe evolved into a project that still survived very well today? That's the question that I'm asking, right? And I have one part of the solution, I think, at the end. The other part, the other one is Tomcat. IBM, I can talk since I'm part of IBM. We were part of this and we still probably ship it, but we've evolved it, right? We've evolved it into WebSphere and various versions of that over time, right? Liberty and so on, and other companies have done the same. So there's a way forward to open source staying relevant while you can still protect a little bit of your cash cow. And I think the best way is this, is to let a thousand flower bloom, right? The idea is that you allow innovation for a platform like Car Foundry, which is massive, each one of those small pieces could be an Apache struts. You need this. You need to have a thousand flowers, right? So how do you make this happen? So embrace, invite people, right? And extend, and that gets us into extensions, right? So what is CF extensions? CF extensions is a process for any one of you, any one of your companies to contribute to Cloud Foundry. And it doesn't matter what you're trying to do. If you're trying to replace Diego or trying to replace a component of it, I want to talk to you. I want to hear. I want to have a discussion with you and we'll see how we can make it happen. Now, obviously there's a process. So it means that, you know, you come with your idea. Hopefully you have some code behind it and we review it. We spend time, I certainly spend time with people. And it's not just me. There's a whole community of people that care. You know, you heard from Dr. Nick, there's tons of people that will eventually get to reviewing your project. So you propose it. It gets reviewed. It gets voted on. And when it gets voted on, you add it to the Cloud Foundry incubator. And then the best part is once that goes through, and of course your project now is part of the incubation process, if it becomes something that any member of the community is releasing in production and making money and part of a business, you can actually go through the same process again, but this time to add it as a project. So whether it be runtime or Bosch or somewhere else. Okay? So there's a way. I don't care if you're pivotal, IBM, you know, whatever company you are, or a small company just new with Cloud Foundry, I want to know that you're doing something interesting and if you want to contribute it, you have a place and you have somebody in a process to make it happen. Okay? So that's the key. How do you participate? It's actually super easy. You just tag your GitHub project with a topic of CF extensions and it's going to start a whole process of asking you to create a proposal. Obviously, there's a link on how you get the proposal started. You have to, we should have a chat. I mean, obviously, I don't want myself to be slowing down the process. So if you think you want to have a chat with somebody else in the community, we can make that happen as well. The point here is so that we don't have duplicated projects, right? There is a place in the community that Dr. Nick created called the Cloud Foundry community where people can just come and just put as much as you want. So there could be duplication and so on. And it's great because it's a great fertile ground. What I'm trying to do is to take those flowers there and remove the weeds and make sure that we have maybe the groups of flowers that are red together and the groups that are yellow together to kind of make a nice garden. So that's kind of the idea. We have an extensions channel and stock where you can start some discussions. The cab now is officially the place where you can bring your projects forward to the community. We typically do this after you've gotten a chance to chat with me or somebody else in the community to make sure that this project is not a duplication or you should talk to other people and so on. So we do the first kind of step. In the Slack channel and the cab, you can actually get to present it and have some feedback. And then we have monthly calls that we're trying to get going as well. We've been doing this for about six months and Kubo came out of this. Now, obviously, Kubo officially started with Pivotal and Google Talking. Then they approached, you know, after they got some version of it, they started talking to me because the extensions was kind of getting started. Then I gave them some feedback. Typically, what I tend to do also is I talk to the relevant people, so Kubo being Kubernetes and Bosch. I spent time talking to the Bosch groups to give them feedback and so on. And it went through that process. And now, since it's becoming a product, I'm going to graduate to become its own thing. But there are other products also as well, right? So IBM contributed a backers. That's also part of it. Now SAP is deeply involved and pretty much, you know, leading the charge there. There's a new project called BBR, which is Bosch Backup Restore. If you didn't know about this, this is a great place to see it. SAP has just started the process for service fabric. I put it there because I wanted to highlight not just projects that have gone through the process and already are ready, but projects that are going through the process right now so that you can take a look at it. Obviously, there are different opinions, right? So there are going to be projects that die or that don't belong to Cloud Foundry. So there's a project called Unik, which was a Unik kernel project that hasn't gotten traction, although I was very excited when it was contributed. But this one probably will get out. So these are some of the projects. One thing I want to show you, and then I'll take some questions, is this thing here. So once you have your projects tagged or you added a topic of CF extension, it gets added to this list where we essentially are keeping track of who's the lead, a little bit of description, the links, the releases, and so on. This is an automatically generated page. The idea is that as we start having thousands of projects, I'm hoping maybe not a thousand, let's say a hundred, then we'll make this a little bit more browsable, searchable, you know, with different classification and so on. So as a member of the community, if you're new and you're like, well, I want to know what's going on in Cloud Foundry besides the core stuff. I want to see, like, say, new service brokers. I want to see new releases, anything like a new CPI, for instance, that's not part of Bosch yet. You come here and they'll be away for you to search. This is a project. It's called CF Extensions. I elated, I coded, but I'm taking contributions, so if you want to participate, feel free. So with that, let me stop. This is the second talk that just goes into more details, but I want to see if you have any questions, especially if you have a project that you have. Yes. I have a microphone. Which isn't why I get to talk first. I'll put my hand up first, but you have a microphone for your questions. Aside from, you know, your dismissal of the Cloud Foundry community organization, but it, you know, that was created three, four years ago, and then the Incubator idea came along and I've never really understood it. Like, the certain members of the foundation would be dismissive of or treat those projects as second class. We don't get to, I don't even know what we use. It's just projects. So why don't we get to, in the Cloud Foundry community, what does it take for those things to be able to have the signing certificates so that plugins and whatever they're distributed, or I don't get it. I think, at the end of the day, you know, it's, as you know, it's more than could, right? It's organization, it's matching regulations that make things successful. Meaning like, if you're going to try to sell to, say for instance, a company in Japan, they have regulations and it's very strong regulations on how, you know, things get put for your things, you can sell them. For instance, you can't have a product sold in Japan that is version zero point something. It has to be one point. Oh, that seems odd, right? Why does it have to be that way? No, do I go for an end then? Well, the point, the reason I'm mentioning this is that when you have a free for GitHub organization or GitHub itself, it makes things difficult to find the gems, like the diamonds. But I argue that the Cloud Foundry organization is full of, I mean, the people who have access to create new projects, create new projects. I don't have access, so I don't create projects there. No, actually, so...  You could argue that, I can see. I think part of what we're doing in Cloud Foundry Incubator and Cloud Foundry is to put some organization to what was lacking in the past, right? So obviously, because we didn't have a two-phase organization, things kind of were everywhere. Now it's starting to become more regulated, regimented. And the reason for this is to put structure so that when people come that are new, they have a place where they know where they can go and they can trust it. There's some level of initial trust. It doesn't mean... Let's explain trust then. Well, I mean by trust is that it's... Trust is reputation, one part of it. And the next part is to help you make decisions in a way that is easier than having to go and look at the source code and go to all the lawyering things and make sure the license is not going to make you ship beer to this one guy somewhere in China that wants to drink. So what is it else that goes in projects or the extension projects that adds trust that every other project... So for one, my employer IBM pays me to spend time and review each one of those projects. So you get my time. Now, maybe that's not valuable to you. That's fair. I'm just saying that I think it is valuable but I'm just saying that you may not think it's valuable. But the point I'm saying is that you get a process that makes sure that the project that gets added is not a duplication. It goes through all the checks. It has the right license. It's adding to the community. So in other words, you're not reinventing something that the community already has. You're not fragmenting it. So this fabric is they just took the dockerbush release and the CF container broker and forked it. So that's part of the reason why I should have put a store next to it. Because it's submitted but it has issues. And I'm working with them to solve those issues. And I'd love that you feedback on those issues as well because I don't want to be the only person pointing out problems. I've been making open source projects for 10 years and it's weird for this other thing to exist that says ours are better in some way, maybe more visible. Maybe there's an extra licensing thing. Maybe the CLA process. There's also more than that, right? So there's also the fact that do you want all the projects to go in? There's also the fact that right now, if you're new to this community you don't really get a sense for where can I go and add things? Where can I extend? There's this vast Nebulas thing and you know how to install it and get it ready and run your apps and you wanted to add your own build back. How do you get to do that? Now obviously you could work hard and see a copy and copy it and change it and so on but this is trying to put some structure and saying here's a way that you can actually go through a process and you can talk to people and you can get the support needed to have your projects added and get some visibility and so on. So it is a management solution, right? It's a very simple solution. Yeah, I just want to say that I really appreciate that there is a process and that people can know how to get into the project. I think that's healthy, that it's not this everybody just adds something on GitHub and it's the end. I think it's also good to kind of vet these expectations and the obligations which come from maintaining a project as part of a community. I think making this clear is a great thing. I have one question about projects like a new CPI, for example, or projects which would go into Bosch or into the runtime later. I'm a little bit confused what is the difference between extensions and incubation. I mean, for CPI I would expect that this is incubated in the Bosch PMC and not in the extensions PMC. How do you see that? Sure, sure. Let's talk about that. But thank you for highlighting the point that it's important to have these processes. So Dimitri and I Dimitri is the Bosch PMC lead. I spent a lot of time chatting about this question that you just asked, which is if I have a CPI I wanted to add it, does it belong in extension or does it belong in Bosch? I think I think it belongs in extension. Dimitri also does, but he thinks some belongs in Bosch and I think that's the confusion. And we're still arguing this, and so I'd love to hear your feedback about this the whole community. But let me give you my point. I don't want to speak for Dimitri. You can chat with him on Slack, I'm sure. I think it belongs in extension for the following reason. I see extensions as every part of the platform that needs to incubate goes to extensions. And obviously it can go to extensions very fast. So if it becomes a core CPI meaning that other your company for instance is putting it in production and it's following the rules so it's not breaking the process, not forking Bosch for instance then we can work to make it part of Bosch. Just like we'll do the same for runtime. So that's a simplest way to look at it. So everything always goes to extension so it gets a level of scrutiny. If you don't want that level of scrutiny then go to community. It's perfectly fine. It's brilliant that it exists. It's free form also so it's even better. But if you want the support, if you want the structure then you come to incubator and then eventually it goes to those different parts. So that's my view of it. But Dimitri has a different view. He has my view but he has an asterisk next to it. One more thing I want to mention is that everything I've discussed this is important for you too Dr. Nick the process, the CF extensions, everything is also incubation. What I mean by this is the process itself. I've had feedback. Konemi has given me feedback already. So if you have feedback on how this is being done you can fix it with me. So... Yeah, there's another question. Sendhi, you had a question? Yeah, so I think you actually partially answered my question but I think one of the things that may be helpful for people is, as you say the process is an incubation but I think one of the things that a lot of new potential contributors face is an obstacle is this sort of, as you described, the how do I contribute question but there's an implied sort of workflow here and it would be helpful I think for more contributors if that were published publicly, I mean there's good docs on as you say, how do I deploy Cloud Foundry? How do I use Cloud Foundry? And for individual projects like how do I run fork of this and test it and whatnot what isn't really well documented is what is the workflow to get new content into Cloud Foundry? And so I hope that as part of this incubation process of the Extensions Project one of the deliverables I would want to see would be documentation of the things that you mentioned that there is a slack channel that maybe there is a flow from you start off as an extension then you go into Incubator and then you eventually land up in the appropriate PMC maybe that's where it ends up, maybe it's not but that's what I would like to see and maybe even now just put a stake in the ground here's how you, your personal vision of that because I think I remember when I first came to the Cloud Foundry community and it was like, okay where do I start and obviously not where do I start as user where do I start as developer looking to enhance the platform I think that continues to be a point of frustration and question to this day You raise exactly why extension exists is to try to solve this problem definitely that's part of our goal I mean it's a journey right it's not going to happen immediately and I started by quoting Clayton Christensen and you should read one of his books I don't work for him I don't know the guy but what they are doing there is at Harvard under Clayton is to try to figure out solutions for this so there's even a new book called innovators solution and let me finish by saying that there's no one solution right so like this thousand flowers blooming that may not work you know Apple has a different way of doing of solving this issue that they have an open source should have different ways because it's not a business per se right so this is one way that I think it works by letting all in and then the success will flow but there may be different ways Linus Torval has a different way of doing this he's the benevolent dictator and and he you know for all practical purposes has been a resounding success with Linus so I don't know if this is the only solution but what I do know is that you have a way to make your voice heard and you have me and few others that I can introduce and I can kind of like you know get Cindy get you to talk to Cindy to make your voice heard and let's keep Cloud Foundry going okay so with that thank you


morning everybody. How are you guys awake? Or sober? Who slept in this room last night? That's the only reason you're here. One guy. Okay, so this is Git Digger creating useful wordlets from GitHub. I'm Wick. I'm Mubix. Go ahead, explain it. So last night at random, well not random for Mubix, but we ran into a taxi line, decided to go with them over to Pawn Stars. Everybody know Pawn Stars? So inside we're walking around, we're looking at the souvenirs and all of a sudden we notice this kiosk. Everybody's using it. What's that? Well we walk up to it and it has a camera. You can take a picture of yourself and they allow you to log in with your username and password to Facebook, Twitter to send the image to yourself or to tweet it out to the public. So I email to myself, I'm not giving them anything. And this is the result on the screen. Legit, right? So I did most of the research. I did all the research. That's me. Yeah, that's him. Here. So we're not the first ones to make word lists. In 2009 and 2012, Sebastian Riv, French something, made word lists from Wikipedia. He's an awesome guy. I'm not trying to make fun of him. But also all of Matt Weir's stuff, if you haven't used Matt Weir's keyboard dictionary, it's one of the best ones to find people who just used along the way. And the other people who make awesome word lists are Rock U, going on. Go ahead. So we weren't the first to go digging through source code repositories. Malvituna security, they released SVN Digger where they went through a ton of SVN repositories, looped through, and then published the frequency count of all the files and all the directories that they found and pulled down from, I forget exactly where they pulled them down from. Google code. So just to point out really quick in the slide, if you take a picture of that QR code, we're not trying to hack you. It's actually linked to the information. So it's easier. I made them not him, so they're good to go. The only problem with using Google code and stuff like that is they like to put these captures in, which make it really hard to automate stuff. So this is how everything got started. Two o'clock in the morning, somebody posed a link to SVN Digger. Everybody thinks it's cool. I haven't seen anything like it before then. Rob's like, that's awesome. And that one line, that's the only reason he's standing up here right now. Because of that one line of code. So I'm like, this is awesome.  I'll go to bed, wake up in the morning, code will be done, and I'll have an awesome word list. So my first problem was that I couldn't find at two in the morning, mind you, I couldn't find a good way to get all the repositories. So I started to go to their GitHub list as the top repositories, the most forked. So I used some Python and started web scraping all of that. So use some basic Python, I'm web scraping that, I'm saving it into SQLite, the user names and the project names, and then just set my computer loose cloning, all the repositories. So now what do I do with it? I have these repositories. I'm using OSWalk to go through each repository and keep account of the user, the file name and the directory. I'm doing a whole lot of said, grab, awk, just trying to clean everything up, make it nice and easy. There was a ton of manual review because I thought it would be easy to go through and pull out all the user names and passwords and email addresses I found in this code. So I spent about 17 hours total on my 30-minute project and all kinds of hours trying to pull out user names and passwords and I got a mile line of said that I just copy and paste and come back later. So OSWalk was taking forever to go through and find everything and I'm like, there's got to be a better way to do this. So after some Google fool I found BetterWalk which claims that OSWalk makes unnecessary API calls to go, is this a folder? Is this a file? We don't know API please tell me. And they cut that out of their loop with speeds things up to two and a half times. So the good news is I've got some awesome word lists. And I posted them out on IRC, everybody loved them, I was like great. But the bad news is I've only got some repositories. I've got maybe the most popular repositories and that's it. SQL transactions were extremely slow, took maybe about 30 seconds to go, is this already in my table? Yes? Okay, let's add one to the count.  If I could have got my goon to carry me in here, I would have. And my hard drive was full. I've had terabytes of this data.  How can I make this better? How can I stream on it? How can I not do 17 hours of manual labor? First problem, storage. How am I going to store all the data? So my first thought did some Googling and bit cost awesome. $99 a year unlimited space built in indexing so I could give other people access to all the code and they could search for whatever in the world they want and get it. At that time, six months ago, at that time there was only a Windows client. It crashed every time I tried to launch a robo copy or just simple copy and paste and it was extremely slow because they encrypted all the data on the upswing.  The next option which I thought was the option was to have a NAS. Everything was stored in one place. It was protected. I could download directly to it. But it's hard to get free money for these things. So I had three terabytes already. So my solution right there is the first ten terabytes of all the data. That's awesome. So the next problem is how can I make downloading these repositories better, easier? How can I get all of the repositories?  And it's nice because the API gives you all kinds of nice, useful information. The only thing that I haven't found is they'll tell you that it's a fork of a project, but they don't tell you who was the main project, who was forked from. So I can keep track of how popular a project is, but I have no idea which guy was the original. So database, SQLite sucks really bad when you're trying to store a lot of data. So I switched to MySQL. I've had questions in the past. Why didn't I use Postgres? Well, I know MySQL. And again, I'm lazy. Didn't want to learn something new. So let's put this all together now. So now I've got two main scripts. I've got the first Python script that is threaded, goes through, downloads all the data. It's got another mode that will go through and process all that data. And then I have another script which I'll talk a little bit more about that just takes a long list of usernames, passwords, email addresses, and I pass it a table name and it just goes and dumps all the data into that table. The MySQL database, that is what I upgraded the most. I've actually created a table to keep track of more product information, more project information, and the usernames and passwords and everything now has its own table. And I'm keeping track of the last scene ID so that I don't have to start over or repeat myself. So here's how the downloading works. Downloader goes out to the API and says, give me 100 repositories. I saw, I've already seen 5,000. So GitHub comes back at you and say, okay, here's the next 100. So it downloads it, dumps it into the database that I've got it and then automatically clones the repository to my hard drive. Unfortunately, the processing got a little better, but there's still a lot of manual work. So now the processor mode is checking my database going, okay, I don't have this repository, but I know it exists. It downloads it. Great. Or it goes through and auto loops it. Does the better walk on it. And now if you notice the red line, that's all my manual work. So I have to grep all this data, pull out usernames, passwords, emails, RSA keys, all kinds of fun stuff, and then clean it up, which can take for a one grep session for one day, can take four days for me to go through and clean it all up and dump it into the database. And then I have a bash script that will just go connect to the database, dump everything, and create the word list, and automatically send it back up to GitHub, which is the real irony. I'm downloading all their data and yet storing it on GitHub. So the updated news, I now have all the repositories. I can now get every single public one. Generating the word list with the bash script takes minutes, once all everything is in the database. Because of the updates I did in the database, I can now store the repositories on any hard drive I want, search the database, it will tell me which one to go to get. The sucky part about that is if I want to go back and grep for more stuff, I have to get this giant hub and plug all these hard drives in at the same time. It's awesome, you should see it. I'm estimating that it's going to take about 30 terabytes to download all the public repositories. However, I am pulling that number out of my butt based off of the amounts of repositories I got from the first 10 terabytes. Because everybody is uploading new stuff every single day. I could probably continue with this project forever and never see the end of GitHub. So this is the big data drinking game if you just heard me say big data, drink, but you guys are all hung over so I'm not going to ask you to do it. So obviously this is a build up to the actual word list. What did we get out of it? So anyone with kids knows exactly how this goes. So altogether. If you don't see kids, if you don't have kids, you can just get the movie and fast forward to that part because it is like the best part of the whole move. All right, so all directories, all files, these are pretty straightforward lists. But the cool things about them is what you see inside of them. And we're not just talking about password list. Password list is, you know, that's the obvious use, right? I'm going to have a set of passwords that I'm going to use against it. The all directories list and all files list is awesome when you're talking about web application attacks and the usernames. I didn't know that so many people loved Bob, but they do more than admin. So stats. Yeah. I promise this is the only stats. I just wanted to give an overview of how many passwords are in the database versus how many of them are actually unique to each section. So this is where it gets relevant to what I do. I'm a senior red teamer. And one of the things that I just break stuff. So I already talked about force browsing. The SVN digger kind of started that whole thing. The great thing about force browsing is when you get a set of directories or word lists or stuff like that, you can just exactly like a derbuster. You just go through it and find it. You can actually use these list with derbuster. The small default password list, which is not exactly the same thing that I would have expected as the default passwords. And you start with root, tour, blah. This actually got a lot of different ones. Static salts. It's hilarious when you find a repository that has a salt for passwords and then that repository is used as an application out there in the real world. I actually stopped pulling out static salts because there is so many. And I went, I'm never going to get this done in time to do a CFP on the project if all I did was pull out the static salts. So, five minutes. So number 22 on the list of files is exception.php. I have never, ever looked for that when I was looking at a web application. Even a PHP one. But after a week had done his research and shared the list, I used it, got code execution because the exception.php was actually loading the exception information from a file and you could just specify any one file you want. So it's on my list now. File.php I'm going to keep going through because this is five minutes left. That's pretty awesome. And this is one of my favorites, NTLM SSO magic. Anyone know what an NTLM SSO magic.php does? It has your username and password statically assigned in it so it does NTLM. All right. So, real world stuff. Anyone see this release? The secret tokens for Rails. If you have a secret token stored in your repository and it's also used in your production without you changing it, it's direct remote code execution. So, this is the gentleman and I'm going to butcher his name but I won't butcher his name. But he sent out an email. He was nice about it and sent out an email to all 1,000 users who had this in their repositories. I am much too lazy to do any of that. So not so obvious stuff. You start parsing every file from the revision history. Right now WIC isn't but if you store your password then the gentleman who just said it removes it but you can go back in the history if you don't nuke it. Mass static code analysis so you can find vulnerabilities in a ton of things really quickly. Dot SVN and dot settings are amazing things for some reason when you convert a SVN repository into a Git repository, sometimes people forget to delete those things and they can have configs including database configs and all kinds of things. Git ignores an amazing little file that tells your Git repository what files to never look to commit. Those are exactly the files that I want to look for because those are things that are important. So I usually look for that. 403s, empty directories, this is on GitHub or on Git. It doesn't, as well as SVN. It doesn't let you create a directory and commit it unless there is something in it. So empty directory and DS stores are usually how some people do it. Another thing is running OCR on all the images. We actually found a gentleman that or a girl that had their password stored in an image for the repository. It was awesome. Using a list of text files, grabbing all the emails which he already does and I'm stopping because it gives all ideas and we're done. Thank you. I actually want to give a quick thank you to Nova Hackers. Are there any Nova Hackers in the room?  Without their help and support and encouragement, I would have never kept going with this project because they helped me out with resources. I now have a file server which can store up to 34 terabytes of data. So once I get the original 10 bytes switched over, I'm going to start downloading and pulling out some more stuff. Cool stuff. No, everyone's waiting for the next talk. Questions? All right. Cool. Thanks. Thanks for coming. Did you do indexing on the SQLite database? No. No, I did not. And I probably should have. I'm not a programmer. I'm a problem solver. Yeah. Well, I've had one. I gave this talk at B-Sides and I had one guy come up to me and mentioned, well, why didn't you just do it in memory? I'm like, oh, I have three terabytes worth of data to go through. I don't think my computer would live to do that and store the database in memory. I'm afraid. Maybe. I don't know. Thanks a lot, guys. I appreciate it.


Hi, everyone. Joe for Jaspy's casebreaks.com coming at you with 2022. Tops Chrome Black Baseball 12 box picker team number 13. Lucky 13. All cards chip, full case break. Big thanks to this group for making it happen. I appreciate it. Tuesday the 17th, picker team 13. If you have a little rooftop next to your name, that means you won that spot in the filler we did. Ed, you ended up with Last Bot Mojo with the Mets. No, not Corey. Sorry, Corey. Pretty sure it's Ed. Picker team 13. No, it is Corey. Sorry, Ed. Well, Ed, you have second to Last Bot Mojo. How about that? Corey, you have official Last Bot Mojo. All right. Pop this case open and let's see what we got here. We have another case of Tops Chrome Black loaded up. Tops Chrome Black actually had a pretty solid price point. I feel like we can run it back or if we get it pretty close to single digits or into single digits, I'm sure building a filler, building a filler for this wouldn't be difficult at all and would probably sell out fairly quickly. We're doing the whole case. We got some base cards in there plus one encased card per box. Old school guys, new school guys, a lot of parallels, a lot of numbers, a lot of big hit potential in here. Good luck. I feel like of all the cases we've been doing this week, have we seen a one of one? I don't think we've seen a one of one. Got a Jaren Duran Duran to 150. So maybe we can cross our fingers for something like that. Also, I think the cards with the Big Rookie logo might be a little bit shorter printed. I'll be for the Cubs. All right, the encased card. Duran Duran will go to Howard and the Red Sox. Dustin has the Cubs. And our first encased auto is Trey Mancini. That will be for the O's, Orioles Edition going to Ed P. Oh, that is a case hit. All right, so maybe those fall one per case. There you go, nice. Yeah, I don't know how many cards are in this base set. I think probably a decent number, but they don't give you, but not a lot of cards per pack though. Not nice, Julio Rodriguez, his teammate Kyle Lewis and Brandon Marsh. It's a nice pack, 62 out of 150. Wait, is Kyle Lewis still his teammate? Did they trade Kyle Lewis? Why am I thinking there's... No, he's a diamond back now. That's right. All right, but this is still Seattle Edition in this set. Michael Loci with that 131 out of 99. And here's Julio Rodriguez, rookie card. Michael with the Mariners, former teammate of Kyle Lewis. All right, the encased is Aqua Parallel. Do I see a rookie logo here? I do. It's a good sign. It's Hoy Park, 85 out of 99 for the Pirates. Dustin with the buckos. They did, says Joe. Kyle Lewis, I think, supposed to be really good, but I think the injuries kind of have derailed him, I feel like. And we got a George Springer, one out of five. There's a train whistle, but looking for a one of one. We've done enough of these cases where I feel like I can start thinking about, hey, is a one of one gonna happen? Here's an out of five though, I'd love to see this. Anthony with the blue jays. Anthony, all aboard the big hit express. Woo-hoo. Daniel saying that that base Julio is 70 to 80 bucks just on its own. Nice. Yeah, I do love the tops chrome black. Really sharp looking set. All right, next player. Oh, this guy knows. Bo Jackson, Bo knows. Bo knows autographs. Michael Locia with the Royals. That's right, Justin. All right, if you think about it, one of one's just one card, not too much to ask for. Let's just see it. Let's see it. Got Nolan Aronato, 60 out of 99 Cardinals. It's gonna be for Matthew Wood. And we got some gold, 28 out of 50. Gold Rhino. And that'll be for Dustin and the Cubbies. There's Otani, the Starling Marte, and a gold Nick Madrigal, 25 out of 50. All right, next up is 16 out of 150, Ha Sung Kim. That's for the Padres, Dustin. Dustin at the Pirates and the Padres. Couple Korean wall players there. Rex has been noticing in the last three or four years, Rhino's been signing a lot. Or maybe Rex, maybe he's signing just exactly the same, but we're just pulling him more often. Maybe that's the case. It might be, huh? I think Korean. There's Manny Machado, Ha Sung Kim's teammate, Juan Yepes, and Vlad Guerrero Jr., 87 out of 99. What's Ryan Sandberg doing these days? Wasn't he aspiring manager at some point, but I think he had a not so good stint with the Phillies maybe. No job offers after that. All right, next, not numbered, it's Edgar Martinez. Nice. Mariners, that'll be for Michael. Michael with the Royals too. A couple old school guys for you, Edgar Martinez and Bo Jackson. One out of every 4,528 packs. How many cases of that? For, someone did that math, four, four, five, two, eight divided by 12 boxes per case. Spencer Torgelson, Brandon Marsh, and Jose Ramirez, Purple Chrome, 101 to 150. Oh, that's right. He got a statue in front of Wrigley. Torgelson going to Anthony and the Tigers. What other statues are out there, Rex? Do you know? I mean, yeah, I guess he's won a couple of MVP's with the Cubs. Is that statue worthy though? All right, next up is Raleigh Fingers, 58 out of 150. For the Oakland A's, John with the A's. Next box. Mike Tower, gonna try that joke again. Statue of Sammy Sosa, but it's corked. Ernie Banks, Harry Carey, Furry Gangs, Ron Sandov, Billy Wins.  Every single member of that World Series roster should have a statue. There's Dylan Carlson, 89 out of 199. All right, that's for Matthew and the Cardinals. And the next encased auto is, can you tell about that first letter right there? It's Reed Dettmer, I would not have guessed Reed Dettmer's. Rookie Autograph for Ryan and the Angels won that team in the filler. Oh, I see, they're going to the decades slowly, I guess. Both, Mike Tower. Both, they should get a statue because they won a World Series and it was so long since they won the last World Series. Kebron Hayes, 70 out of 199. Dilo remembering that Phillies traded Larry Boa and Sandberg four. Ivan, Ivan de Jesus, one of the worst trades in baseball history. I can't trade those guys. Ooh, we got a little red, red rookie. Three out of five, red rookie, red rookie. Joe Ryan. Little Joe Mojo going to Kenny and the twins. Kenny won that team in the filler. All aboard the Big Head Express. Woo-hoo. Still hunting for that one of one. I don't know, I think you're right. I don't know which teams actually have team statues. Statue might be more of an individual accomplishment thing. Well, Cubs could have been the first. Cubs could have been the first to have a big team statue. It'd be smaller though. It'd be the size of a regular statue, but with all the guys. There's Max Scherzer taking it to the max. 157 out of 199. We've got Gold Speckle, 35 out of 50. It's Jose Canseco. That's gonna go to John and the A's. You know what, I haven't. I don't think my facial hair grows thick enough to get a mustache, a proper one going. I haven't, I have not shaved in a bit, but this is kind of what it looks like. I think it probably looks better on camera than it does in person. I don't know, I wonder though, Justin, if I ended up on a deserted island for a year, what this facial hair would look like. It's Cabrion Hayes, 36 out of 50. But yeah, I wish I could grow a full motorcycle cop kind of mustache, you know what I mean? And then it's getting itchy and it bothers me. There's Ted Simmons for the Cardinals. That'll be for Matthew. All right, last box coming up. We got another case of this in the store, ladies and gentlemen. That's a good stat. A little Doug Pearson, Nick Foll statue. Is Jason rocking some sort of mustache?  I thought chicks did a facial hair. I don't think he's mad at like a thing. A scruffy look, maybe that maybe mutton chops a little too scruffy. You're on Alvarez, Ryan Velod, and Giancarlo Stanton, 15 out of 75. When I think mutton chops, I definitely think of like, you know, like a old, like an old politician from like the 1700s or something, like 1800s or something. Yeah, a bear chin and neck, but just the, I don't know, tell her, tell her. Wolverine has mutton chops, right? Wolverine's pretty cool. All right, last autograph is gonna be Jorge Posada, hip, hip, Jorge for Dustin and the Yankees. There you go, gang. Super Fractor, the one of one search continues in 2022, tops chrome black, but we've got another case loaded up in the store. Maybe we'll find one there. I'm Joe for jaspyscasebreaks.com and I'll see you next time for the next break. Bye-bye.