OK. Morning for everyone who just woke up, like probably me. Good evening. Noon to the rest of us. My name is Miguel Cinega. I'm going to be talking about continuous integration and employment using OpenStack. On December, eBay recruited me from EA and to join the cloud team. So I've been working for the past four and a half months into this. Right now, we're going to go through how we're planning on doing it. The presentation is pretty much about an application that I'm developing to go over and be able to deploy pretty much applications. The thing is that at the beginning, everything started to be used. We needed something that was able to go ahead and deploy OpenStack, like do stuff with continuous integration and everything. We'll look at the OpenStack CI. But after that, there were some other teams that they were a little bit interest. So we tried to actually make it a little bit more generic. So let's get started. As you know, the same marketing stuff as always. So I'm pretty new about it. So what you see there is what I know about these guys. Let's just skip to it. I have some slides, so I don't want to run out of time. The agenda is going to be pretty much what you see in there. It's going to be enter flow. That's the name of the application. The flow flow, flow code replication, package and artifacts, distribution packages, infrastructures go, deployment to production, screenshots, and a little bit of the roadmap. So first of all, enter flow from beginning to end. This is for what I've seen in my experience what most people have. And that wasn't until Google released Garret. Before that, it was just Jenkins. And you pretty much deploy stuff into a web server or an exos repository, something like that. As you know, Jenkins is really tied up with Java projects. So most of them are, it has a lot of options for Java only. Although there's a lot of plugins out there. So what we needed is pretty much some requirements that was a single interface. So developers are not jumping from one system to another trying to figure out where to use it or how to use it. And there was going to be scalable, simple to use, developer friendly, and generic. So most of the other developers could actually jump in there without any problems. So we decided to go with Fluo, which is pretty much a web UI, an API, a single pane of glass, hold on, oh, there I am. Cloud instance provision and decommissioner allow users to configure the system on the fly and has a role-based access configuration. How it works, Fluo is pretty much, you can see it as the management piece between Garrett, Jenkins, and a bunch of things that you'll see in the middle into it. It uses CRMQ to talk to other components, which doesn't have API stuff like HTTP API. Basically, it's just loading. Well, it's not actually loading. It has a supervisor model in the back end that actually uses to coordinate all these services. Every time the developer can go ahead and say, you know what? I want to create a new job in Jenkins, something like that. Or you know what? I want to create another step into the flow that I'm using. So if there's something that is not API compatible, we'll basically just tell ZeroMQ, go ahead, put in place. The workers go ahead and do whatever they need on the server that has the actual system in there and or the actual component and just reconfigure everything on the fly. The other thing is the recycles or provisions, the servers. Why we're putting this here.  That's a lot of money for some companies to have three environments. So a lot of them, they don't even use it. They're just like, OK, code it, production, there you go. Or as soon as it becomes something in development, out of the nothing, that server became production. So why are we using OpenStack? Because we can just pretty much provision on the VM, do everything in there, run the continuous integration, run the unit testing, run everything, and pull everything out of it. And as soon as it's done, pretty much destroy the VM. So that's what this stuff is pretty much doing, is just telling Jenkins, you know what, I got a new thing here that I needed to plug it in there. And after you're done, pretty much go ahead and destroy it. So now going back to the configuration file that I'm putting out right there. Well, if nobody else knows out there, GitHub provides a service that is called Travis CI, which I love, basically allows you to just create a Jamel file, put it into your code repository, and they'll basically do everything for you. They'll run your unit testing and everything like that. But what happens when you need to go over and do it like not on the internet, or you need to have it somewhere where it's pretty much private, or you don't have access to the internet? Well, that's another feature that we're putting in that place. So what is the configuration file? Basically, specification of requirements. It's just a specification of requirements file. And tells the developer, actually, you instruct through the configuration file what you need to do on the system itself. We'll see an example right now. So this is an example of the configuration file. As you can see, you can specify which language are you going to be using. So if you go and say, OK, I'm just going to run scripts. And the scripts are actually wrappers to whatever I'm going to be running. You can just tell it bash. If now you can just go over and tell, OK, I'm using Java 1.7, or I'm using Python, or I'm using something else. Any language, as long as you have the interpreter in the VMs, they're going to be pretty much set up. So now the package section. Basically, that's everything. You can go over and tell it APT, Jam, Python, I mean, or Python PIP, or Jam. It basically supports RPMs, Damian packages, Jams, and PIP at the moment. So all the things that you specify in the package installs pretty much gets deployed before executing your job. Now, for each of those steps that you actually have on your flow, you have to review name of the step and the name of the script. What this stuff does is basically just telling the system before you execute anything at all, go ahead and, I don't know, start my SQL, or pretty much load something into the database that I'm going to be using to run the unit testing after it. After that, it goes the system through the review script, which is basically the what are you going to be running. So you can specify it there. OK, you know why I'm developing OpenStack. Tox dash b e pi 27, something like that. If you're developing something in Rails or something in Ruby, run rake. If you're developing something in any other language, just specify it in there. The system will go over and execute one by one in that order. But at the same time, it's not restricted to be just something serial. If it goes over and says, OK, you know what, I have this guy that is running unit testing and this other guy just committed something into it, into the actual repository in the system, the system will just spin up an RBM and start executing the other stuff in parallel. And at the end, we have the notification sections, which basically just send out some email after it. Or right now, I have send out emails, HTTP hooks. And probably we're going to set up something to be able to some kind of notification system either. But that's pretty much the flow configuration file. So let's keep moving into the run of time. So this is the basic architecture of the application. I thought it was going to be a little bit bigger, but it's not. As you can see, we have the components right there. It's a fluid application and pretty much cloud provisioning instances. We have Gary. We have Jenkins. We're right now relying on Sule from the OpenStack CI to be able to use or scale a little bit in the way that we're executing all the jobs.  Something dies. I mean, you still have two of them. It has integration with Savic, so you can go over and monitor anything that you want in there. And we're basically using Puppet at the moment. The reason why we went with Puppet is because that was there when I came in. It wasn't like, OK, you can go over and use Chef or something like that. I mean, the company was using Puppet. And for repositories like Debian Package and RPMs, we're using M-Repo and pretty much our sync to just sync some stuff out there. So let's go through the flow. What are these things? So by default, you can go over and, well, actually, you have generic flows that a system provides in there. But you can go ahead and create your own flow. Where is basically a flow? The basic flow has five, kind of like six steps, which is review, which gets activated as soon as somebody commits a new change into Git or into the repository. The approval, which gets activated when somebody goes ahead and clicks, OK, I approve this change in Garrett. The integration you can call it any time just by putting a comment into your own history of Git. Or if you go into, like, Garrett and set up a comment, you can just go ahead and trigger it that way. We have the release and the periodic. Pretty much what it's doing at each of those steps are those five steps that are put in there. It goes builds a cloud instance, reads the configuration file, executes the scripts defining their report back, and destroy the instance. So how we're doing this, we're pretty much talking to Nova. That's pretty, that's it. It's not something fancier down there. It's not something magic trick. It's just basically telling Nova through the API, go ahead, build this stuff, have it in there running for some time, and there you go. So on the right side, you can see pretty much what is, which actions are going to execute every, or which actions trigger it, and pretty much each of those steps right there. So one of the things that I've been noticing a lot is that a lot of people doesn't actually do reviews. And I've got to be honest, I mean, before jumping into eBay, I was with electronic cards. I was given not support. I was helping the application teams, which were around 150 different application teams working all together to come up with games, to be able to have standardized. When you have a multiple environment where you go and say, you know what, I mean, I have this team that is doing its own stuff, this team that is doing its own stuff, and this team that is doing its own stuff, the best and easiest way to go over and have control of everything, not actually control, to come out with quality code is have a review system. Whether, I mean, I know that some people will go ahead and say, you know what, I mean, that's going to take me a lot of time to go over and have somebody just specifically saying, oh yeah, I approve the thing. I approve the thing, I approve the thing. But trust me, I mean, it's going to help you out a lot, specifically if you have a huge environment. So in order to have quality code with this basic workflow, like I mentioned that you can go over into the system and create your own. If you don't want to have review or approval or whatever, just destroy it or just create a new one. You define the name of the step or stage, which basically is the name inside of the application. And the system will go ahead and say, okay, you know what, this user is using a three-step review process. That's fine. I don't care. The configuration file, you just specify, like I said before, like you put it here on the example. If you have three steps in there, put it there. If you just want to skip one of the steps, don't put anything. I mean, it's simple enough to just go over and skip the steps or not. One of the things that, since Garrett is behind everything, as soon as you get the report back from Jenkins, we'll basically go ahead and say, okay, I approve this change or approve not. So in order to have quality code at least following the whole basic flow, this is what you need to have. You need to have a review that basically gets triggered as soon as the developer or somebody else commits a change or reviews a change into Garrett. The approval would require somebody to actually go over and approve it. Probably it's going to be the project lead or somebody else and the build. Why the build? I'm putting it there because the build gets activated or at least that step gets activated as soon as the Garrett system says, you know what, we're cool with this. Let's merge it. So most of the time builds, it's just you can use it for, I don't know, create packages or you can pretty much tell, send out notifications or something else. The rest of the flow, like I said, integration release and periodic, you can go over and trigger them any time you want. The period basically is just like a cron job where it's executing every single time at 2 a.m. in the morning or something like that. The way that we're using the periodic stuff is to, at the moment, to pull down changes from the OpenStack projects every single day and merge them either. What is happening is that as soon as the periodic gets activated, we pull down source on like the source code from OpenStack. We try to merge it with our repositories and if something goes wrong, we'll basically, that step will come out with a notification saying, okay, you know what, developer, X1 or developer, John, go ahead and fix this stuff. I couldn't merge last night. So it's really, it's like I said, it's a bunch of tools, but in order to have quality codes, at least the three of them up there are needed. One of the things that I've been crossing a lot of the before is that when you put out a Garrett review system and if there's somebody like in the rush, they'll probably just go ahead and say, okay, I approve, approve, approve, approve, approve, approve. That's not kind of like the way of going, you know?  They will basically come back and say, I give a plus one here, or that's okay. But however, to go into the approval, the way that we configure the system is that you need to pass over Jenkins also and Jenkins have to give the approval. And you need to go over and pass somebody, like the project leader, like a human, and some human will go ahead and say, okay, I approve this thing. Because that way you're not allowing to go and somebody just click, I approve, and pass it over. It actually needs to go through Jenkins first and execute all the unit testing integration or whatever you have specified on the configuration file. So how we're replicating things, if it is one more or multiple projects. One of the things is that we set up the system to be able to replicate anywhere where you are, anywhere where you want to put your code. Inside of Fluo, when you create a new project, we're telling the developer, okay, this is your repository. You can activate replicas. If you wanna go activate replicas, we're just letting the developer know, okay, give me where are you storing. Whether if it is github.com, if it is your private github, or if it is a local github, just give me the path and which user I'm gonna be working with. One of the things that we, it's kind of like a best practice is you need to categorize your code. First of all, will it be open source, or is it private for a team, is it private for a company? Do we need mirrors? Because a lot of times you just end up deploying everything into one thing and out of the nothing you got legal the next day saying why did you put out this out there? It's not even possible. So in companies that you have a lot of restrictions, or pretty much you're actually working with a really difficult, not kind of difficult, a legal team that doesn't allow you to go over and just open source everything that you want. The best way is to just go over and categorize and say, okay, for this project on this branch, let's go over and put it here. For this our project for this branch, let's put it here and this branch put it here. It will help you a lot. It will keep your code organized. So one of the things that, that's the whole reason why I put it here in Fluo is that any project can have as many replicates as you want. You can go ahead and just tell it, I'm gonna replicate it on GitHub, on these two, three, four different Git repositories. I'm gonna replicate it internally and I'm gonna replicate it into my laptop. That's fine. The other thing that you need to keep in account is that this thing will actually work better if you just have one user. Let's say that you have a user that it's on GitHub and that user just make it a contributor to everyone, to any of the projects that you're applying to replicate. That way you can just go ahead and have one single user. If you need to audit something, everything's gonna be there. It's not like something, oh, you know what, where this change come from or who is actually pushing into that. The system will not go ahead and put the, like the commit history as that user.   No, that's not the way it actually works. It basically just graphs every single art or from the Git history and put it in there just like any other, just like if you were replicating or cloning your Git repository. So let's move on into packages. I'm already half into it. Okay, so packages, artifacts, whatever you wanna call it there. One of the things that please, it will make your operations seems really happy is don't go over and deploy code from source. Don't ever do it. That's just something that you should not do. It's just wrong, you know? I mean, the package managers exist on the systems for a reason. It's not something that you are gonna go, oh yeah, Git pull, I have the new code in there, there you go. No, trust me, it will make your life easier if you just go over and say, okay, I created my package. So where and when do I, I'm gonna create it? Things to consider, if you're using this application is that Jenkins will be building the packages. So if you need something in there to build your package, you'll have to put it in your configuration file to be able to go ahead and say, okay, I build the package. If I'm building with RPM, let's go over and put all the dependencies that I needed, all the build require, sold all everything that requires by the RPM, same deal with devian packages.  We can basically use the package install or the before step, the before dash name of the step or stage. The other thing is once you have it in that specific section, you can go over and build the actual package in the script section. Why we're doing this is because sometimes where you go over and say, okay, I'm gonna build the package and I have everything into the same section. Okay, something failed. How do you know which thing failed? I mean, it's easier if you just separate the stuff into okay, now I'm preparing the system and the actual step for building the package, the actual script is actually building the package itself. So one of the other things that is really helpful, especially if you have a huge environment, is define a version in standard. Don't go over and say, okay, I wanna go over and put something like beta, one, two, three, four, five, six, seven and there you go, commit that. No, no, no, no, no, no, no, no, no. Define something that is understandable for everyone. Most of the common things that I've seen out there in some other companies and that's something that we use in my previous job was putting all the test packages in that format, zero, zero, zero, X and that's pretty much it. That's all the test packages. Nothing will actually, in testing, will have something like a naming convention or something like that or a name like beta or dev or something in there. No, that's the only thing. Most of the times you actually need to put something into like, this is for testing or this is beta, you use the iteration for it, not the actual versioning. The two main stages that we're using with Fluor right now to build is the build stage, which basically is using the commit number plus the timing seconds for the package version. So in this case, you will be like, zero, zero dot on a bunch of seconds and then the actual package version. Why we're using time here? Because it will make it easier when you're deploying. When you're deploying through a package manager, the package manager will basically just go ahead and say, okay, I need to install now this new version of the test package, which is telling me that it's all than the previous one that I have here. Okay, so if you go and put some crazy name in there and you try to just update it, you won't be able to see it. So that's the reason we're putting the actual or you can put a timestamp or whatever you want but you need to keep all the names, the versioning of the packages incremental. That way it's gonna pretty much, you can be able to achieve like continuous deployment right away. If you have something that is not really critical and you're just trying to deploy a library or something like that, just leave it there and tell your puppet or chef or whatever, push this package to latest always. So as soon as that new package with the new versioning in there that has the time that is an increment from the previous one, it will go ahead and deploy it without any problems. You won't have to go over and figure out what the hell's going on in there. So for the release stage or before releasing things into production, use the tags of Git. The tags of Git are there for a reason. It's actually, I mean, it's way easier if you just go over and use the tag. Why we're doing this is because tags are pretty much to set up something that is really important. There's a lot of people out there that will go ahead and say, okay, well, I'm just gonna go over and create a new branch and from this branch and now on, it's something like, oh yeah, by the way, this is the production branch. No, no, no, no, no, no, no, no, no, no, no, no, no, no. It's easier if you just go over and have everything. It's a little bit more organized and you can actually figure out where are you at a certain point. Even if you go over and see the Git history, you'll see like, okay, now we're on this patch and on this one and this other one. Also, don't commit tags with names in there. Use the same version in standard that I mentioned. And in this case, it would be like 1.1.0 or something like that. So the next step here into achieving everything is distribution of packages. Shipping the code into different locations. So now that you have everything set up in your really pretty package, how do we get it into servers themselves? First of all, use simple things. Use secure object storage if you're actually using something in there. If you're using Amazon, you can go over and put in S4E. If you're using OpenStack, put it on Swift. If you don't have anything, just do a simple rsync. I mean, the system right now where how we have it set up is that all the packages get uploaded to a specific repository within the system itself. After that, if something needs to get replicated, we're basically using mReput to go ahead and say, there you go, all the data centers, we need to have a mirror on each of those data centers. Grab the latest packages from it. And we're basically just telling, syncing between the data centers or we'll be syncing between the data centers. That's pretty much up to you. But one good thing is that if you have a really fast development cycle, you can just put it like sync it every hour or something like that so you can go ahead and pull pretty much the packages anywhere. Now, for that part that says replication is done every five minutes, it's not really recommended when you have or when you're creating repository of packages. The reason is because if you have five packages, that's not a problem. It will do it on the fly. But in the case that you have 100, 200 packages, every single time you commit something in there and creates a new package and gets deployed, it needs to rebuild the actual index of the package, of the package manager on the repository itself. So you can go over and start, I mean, like I mentioned that if you have small, like a few packages, you can put it every five minutes. But you will soon see that, okay, by the way, I have this system getting replicated and it's creating the index for the package manager and then you have the other index in there and the other index in there and the other index in there. Because probably creating the actual index or inside of the repository takes, I don't know, 20 minutes because of the amount of packages that you have. But still, one of the things that you should go over and do is just replicate everything any single place you have it. An example here is that you have data centers with cloud, you have multiple clouds. Again, this is always depending whether if it is secure enough for you or I mean, if it is pretty much possible to go over and do it because sometimes you will go and say, okay, I want to replicate into S4E out there in the cloud, oh, by the way, I have property, intellectual property in there, why are you replicating everything? So that goes back to the previous thing that I mentioned, like categorize your code to see where you're actually putting in everything. That pretty much applies the same principle here, just categorize your packages, you're gonna tell it, okay, this is gonna be for PCI, don't go and throw it out there. It's just like putting in PCI and just tell it to replicate to this specific repository in there. So let's move on into, now we have the packages set up in the repository, cool part, infrastructure and scope. Okay, so this part is a little bit more about for the operations people, but it's still, there are some companies or I mean, on most of the companies, controlling puppet and controlling chef, controlling even salt, requires a little bit of knowledge on the specific language that is running behind. So one of the things is, at the moment we're using puppet, and it doesn't matter if it is puppet or whatever it is, your configuration management should go through the same system itself. Or if you have another review system, put it in there.  No, that's really bad. It will give you bad, no, no, no, not at all. So one of recommendation is put it through the system itself, allow the operations teams to be able to say, okay, we're gonna push this thing into it. Put it also in git or subversion or whatever you wanna have in there. Put it in a source control management tool. The reason why is because sometimes where you have a huge environment, you're trying to figure out what happened. So the most common thing that would actually, somebody will go over and say, okay, you know what, we had a downtown last night and it was caused by a puppet. Puppet pushed something. How are you gonna debug that? I mean, if you wanna try to figure out what exactly happened, one of the things that is gonna help you a lot is just go over and say, okay, I have the git history here that basically is telling me, go ahead and put it in somebody at 12 or before midnight, commit a change that deploy a, I don't know, a package on a hundred servers. So put it inside of the git management or source control management or whatever. Some guidelines that we're giving our puppet coders is create your configuration management code in a service-oriented way. What does this mean? It means that as you know, puppet is kind of like deploying stuff randomly. I think that's gonna change in the latest versions, but if you're still stuck in like 2.6, 2.7 or something like that, we'll go randomly putting it in there. Putting it as a service-oriented means that you can either use stages in puppet or you can use just recursive class and just go over and have three or four main classes. The one that is gonna install your packages, the one that is gonna config your application, the one that is gonna do some post-config section and the one that starts the actual service. So as you can see right there, that's the kind of like the structure of the module where we have the virtual classes, then we have another directory that has some classes that is common to everyone and then you have the model scene that are basically same deal and we're splitting it up by OS. That way we are actually building both of them and we're actually, well, I mean the whole idea is to be able to code it for both OSes that you have there. If you're using any other type of OS, just put it in there, create another directory and put the same code in there.  You just only need one thing to go over and deploy it all across any OS and yeah, I mean most of the resources are available to work that way, but in case like Ubuntu and Red Hat, kind of like a real failure, the package names change. So that way you can just keep going and have everything really structured. The other thing is create your common modules and use parameterized classes. That will give you at least a little bit of a structure when you're deploying and just saying or actually making it a little bit easier to understand and okay, I have this node. I'm gonna go over and deploy this class which is gonna install JBoss and these are all the parameters that I'm passing into it. In the backend, I mean, Puppet is doing everything in there and the other thing, use also, always test your code. On Chef, there's, if you're using Chef or for all the Chef guys, go with MiniTest, I think it was the name of MiniTest, I don't remember, well, create tests for everything. On Puppet, validate the syntax. Try to keep the syntax a little bit clean. There's a lot of links out there that you can go around and use and also run it as no operations mode. What is doing the last part is basically just going over and telling okay, execute everything, like if you would actually go over and do it, but instead of actually doing it, just simulate it. And that way you can actually see not only syntax errors, you're gonna be able to see dependency errors where you can figure it out if there's a package that is requiring something else that is not there at the moment that Puppet is trying to put it in place. I'm almost ran out of time. So let's go to deployment now. So it's go time. First of all, centralize your configuration management. Put it in a central place, replicate it all across the different data centers, clouds, or wherever you have it. Have a mirror of the configuration management, but don't modify that mirror, it's just a mirror. Put it everywhere, anywhere you want it, but don't modify everything. That's the reason why you're putting everything into GitHub. You're committing to GitHub or you're committing to GitHub or whatever and just put it, let Puppet go over and get pulled the stuff in there. But don't go over and say, okay, I have this environment and I have these specific options for this environment, put it in there.  Keep it in sync. Usually, keep it in sync from like a get pulled doesn't take any more time. So one of the things that you need to keep it in sync as fast as you can, otherwise you're gonna run out of sync and you'll have these data center with specific things in your configuration management and these are data center with all our specific things that probably you needed both of them. You have an application here and here that they require the same package or the same changes at the same time. So keep them in sync and pull every two to five minutes. The tactical releases in a chronological meaning and keep it to the standard. Like I mentioned before, use things like 1.1, 1.2 that can stuff not be 0.1 or beta 1.1, 2, 3, something like that. That will actually make it easier for your ops team to go over and say, oh, I got the latest tag now. I'm not actually looking for, I have here 1.13 and up there it says 1. beta production. This is the final. Okay, so keep it simple, pretty much. And just keep in mind that the tag goes into the name of the package. This is pretty much when you build the package itself and you're using tags to create the package. Just remember it's going to be in there and you're going to help a lot the configuration, well, yeah, the configuration management tool and the package manager also to be able to figure out what do I need to go over and put in there. And lastly, not for all, the automatic deployment and scheduling. So it depends up to your business needs. If you can go over and say, okay, I have a really tight change management window, you can create a pre-approved ticket or something like that and let Puppet run at that specific night or that specific time of the day and go ahead and tell it, Puppet, on this day just pull everything from Git and deploy it. Or if you want to go over and you just need somebody to go and click the bottom, you can always go ahead and say, okay, wake up in the middle of the night and just Git pull from Puppet. Or I mean make a Git pull inside of Puppet so you can go ahead and deploy it right away. So it's up to the business needs whether you like it or not. Some of the things that is really helpful is if you have multiple environments like a testing environment or a QA, just set up Puppet to be able to pull the latest. And why you are doing this and not inside of a production because on the latest using the same configuration, actually the same version in standard that I mentioned that before is just keep, doesn't matter if you push a new package right now and another one in five minutes. Puppet will always go over and pull it down because it's looking for the latest one. That's one of the reasons why it's important to just keep everything chronologically. And like I said, some of the application screens, as you can see, that's what we kind of have right now. On the right side, you see all the projects that we have pretty much set up.  If something failed, it will basically just go and say, okay, it failed at this test. It will allow you to see the servers that you have and the latest jobs that are actually running in there. To be honest, that screen over there is huge. That's just like a half of it, but you actually see a lot of things. In fact, the screen, this one is pretty much the same deal. The only thing is that at the beginning, you start seeing a lot of graphics, like how much time it took to run a specific job, how much time it took to run X, Y, or C a specific job for this project, for this sort of project. Each developer will go ahead and has its own screen so he can figure out what's going on when they're running the unit testing. Some of the other screens is, this is the code review. Like I mentioned, we created the application to make it easier for the developer, single pane of glass, so everything you get in Garrett, you get it here. So in that case, you don't need to go into Garrett, figure out what's going on in there. No, nothing like that, just go over here into the application and you can review, see the changes, actually see the files. And if you click inside of one of those files, I wanted to put it in there, but I couldn't because some code that was sitting there. You'll see the difference between, okay, this is what I had before the commit, this is what I had after it. And as you mentioned, it comments all the way down there. So this is the review part, so you can see right down there, you can just go ahead and click minus two, I approve, minus one, zero, I don't care, whatever, and just submit it. So you don't have to be chasing everyone else, like okay, I need to go over into Garrett, no, just get into the application and do it. The listing of templates, what we're using in there is basically we're providing the developers something like basic flows. What I mean of flow is, remember all those stages that I mentioned it, that you have approval, review, build and that kind of stuff. We have multiple, well we have like five now, basic different flows that we're actually putting in there. Besides of giving the developers the capability of creating their own, we provide just like a basic stuff that they can go over and start using. And also, we're giving pretty much the output every single Jenkins job that we're passing into it. And also, pretty much a jump into the server itself. So in that case that you're actually seeing, okay, I need to debug something. I need to be able to go ahead and say, jump into the server. No, we're basically just telling the, we're gonna tell the developers, go here, there's the output of everything that's running there and you'll see the errors at that specific part. And if you need to jump into the server itself, just click on the link. That will basically spin up web UI kind of thing. It's not no BNC for the people out there. It's another JS application now which I don't even remember it. But basically allows the developer to jump into the system itself and create more and more, pretty much kind of like a web terminals. And they can go over and do whatever they want. We don't care because at the end, if you're putting in a static server, we'll basically get recycled in the middle of the night. That's pretty much helpful only if you're not using the automatically provisioning the commission that we're putting out there because otherwise it won't give you access to a server because it's already gone by that point. The roadmap that we have right now, like I mentioned, it's me and it's me. Let's see if we can put out a CI CD as a service so you can go over and put it in place and let everyone go ahead and just run. Don't worry about the stuff like, oh, I need to put out something specifically like this application in there.  This stuff is not gonna try to blueprint this thing. You just basically can go ahead and tell it if you have a server already done, specify the IP and which credentials are you gonna be jumping into it and the system will start using it. Integrate with other tools like Chef and Salt. The reason for this is because, I mean, not everyone likes Poppet, not everyone likes Chef, not everyone likes Salt. So we're gonna pretty much try to give support for all of them. What I'm saying give support to all of them is not like, oh, I'm gonna create the file for you so you can go over and deploy. It's not only that, right now we're taking information out of the Poppet servers through the API itself, so we can go over and query like, where this node is right now or give me the latest catalog of this node or let me know where, I mean, when was the last time or give me X, Y, or Z factor out of the Poppet. So that's pretty much what we're gonna be looking into. The other thing is container support. I know that cloud is fast and it's really fast, but they're not as fast as containers. Sometimes there were some problems where you can go ahead and say, okay, the cloud is taking a little bit more of time to go ahead and spin up the VM. The problem here is that that will actually increase the build time that you're working with or at least that Jenkins working with because it's actually waiting until the VM is ready to be able to register as a slave and then start actually putting up things in there. So the other thing that we're gonna be using is OpenBC and Docker.  And inside of the container that is sitting on the VM, deploy the actual unit testing or whatever you're actually doing in your script. Ask block storage as an option so you can actually use our things and have integration with ironic cobbler, foreman, and racer. The reason with this is because sometimes you can go ahead and say, okay, I have a huge instance out there, but I'm doing my SQL databases test in there. So that's not gonna work at all. Whether you like it or not, how fast it is, sometimes you need physical hardware. So we're gonna pretty much just allow the user to be able to provision systems by themselves like physical hardware. The reason why I'm putting all of them in there is because a lot of you or a lot of the people here probably don't have OpenStack set up anymore. I mean, not yet. But the thing is that the application we wanted to go over and put it out there to be able to use it anywhere where you have. Whether you're using AWS OpenStack, CloudStack, or you just have physical servers, you can go ahead and use it. Now the other thing, Jenkins and Sol, not sure if we're gonna keep using it. The reason is that, I gotta be honest, Sol is a really cool project. It's really awesome. Allows you to pretty much scale Jenkins. But the thing is that there are things that are already built in, like Erlang. Erlang and the OTP framework provides pretty much the same deal that or at least something similar to what Sol is trying to use. So we're looking into, instead of having and deploying Sol and deploying Jenkins, have just one pretty much huge, kind of like a Erlang application that is gonna be going, that is gonna be actually creating VMs, destroying them and doing everything in backend. Jenkins, it's also a really difficult to scale at least horizontally. Even though you have masters, okay, I'm actually running out of time here. If you have masters, you can just go ahead and add more states, but even with that, it's kind of like difficult. And as you see, improve the UI. I'm not really a good designer, so that's something that I need still to work. So with that in there, I'm running out of time here. If you have any questions, go ahead and thank you everyone.


so to get us started on this episode of questions from y'all we got to give a huge shout out to the two newest team keep it clean patrons uh one is milton w and the next one is izana appreciate y'all much love to team keep it clean thank you to all of y'all and and to show appreciation let's start off with the very first question from the newest team keep it clean patron uh izan he said good morning engraving hey good morning and when i'm recording this it actually is the morning time um but he said good morning i hope you're having a good day been watching your content ever since the titans lost in the playoffs okay thank you so appreciate it um i also wanted to say thanks for all the work you put in your content has become a part of my daily routine and i appreciate the community uh you and team keep it clean have created hey and that's that's important as far as um the the the community the family really not even community the family um and it's important to just for everybody to know that they're welcome for everybody to know that they can come through uh and for everybody just had fun just have a good time because that's all it's about again with this channel one of my uh first and biggest goals um that i wanted for the channel was for it to be a safe space and a place where everybody because we know that the world it comes with enough struggles and stresses and all that but i wanted this to be an opportunity to people for people to sort of get away from all that for a little bit talk about some football has some fun in the process so i appreciate it uh he said none of my question having looked at the roster as of august 30th um and i'm recording this on august 31st uh so a couple of things could change um but he said i believe the raven should incorporate a lot more for three fronts rather than three four and i also believe that they will uh now with um with mike mcdonnell um i believe that he ran uh multiple types of defense uh over at uh michigan so that could end up happening so you could be spot on he said um i think this is because in a base 34 front you need two outside linebackers and two middle linebackers we currently only have two guys that can play outside linebackers while five that can play middle linebacker and that that is definitely subject to change by the time you see this video it probably will have already changed because there is absolutely no way that they are going into the season with just a doffy away just in houston that's it i mean uh like my god cam he had reminded me of in a previous video they could also put malik here's an outside linebacker but there's no way that they're just going with the guys that they got on the team right now uh they're gonna make a move there for who we'll see um he said but if we switch to a base four three mike mcdonnell ran that at michigan see i should have kept reading um we can better the rotation at middle linebacker and let our pass rushers run forward to the quarterback instead of backwards like there's some db's uh this will also allow calleus gamble to kick back out to the edge where he was very productive back with the jacksonville jaguars lastly uh i have barely heard any news from the ravens first team d line since camp started and this has me thinking they could possibly switch it up and pull a surprise but i don't know really i don't really know what uh is going on right now and those are my thoughts on what front would best fit the roster have a good day and thanks for taking my question now you ain't got to thank me i thank you for supporting now um with uh what you mentioned as far as them doing a lot more four three and three four um for me and it's something that you brought out to you see you talked about how calleus gamble had a lot of success uh with that with the jacksonville jaguars and i think that's what it's about to me that's what it's about with offense with defense with everything uh just putting guys in the best position to where they can have success now you can't always cater to every single player's individual need at the same time um but collectively just coming to an agreement on all right this is where all these guys they can be the most useful that's where they can have the most production that's where they can do what they specialize in the most so hopefully ravens do that hopefully uh in your case which you're speaking specifically about the defense hopefully they do do that i think we all need to stop next question came from my guy outlaw and appreciate you for being a patron he said uh we can't really say what lamar contract will look like it starts with him we don't know if he started at a short contract to match up with the new tv deal or if he started with the ten year deal to the ravens if it comes to the franchise tag so be it with the tag he will be paid like his peers and not underpaid if he wins the superbowl on the tag he will get paid like joe did i hope when he signed his rookie deal he basically signed up for seven years including the franchise tag it's been five years no it's only been four years um and we need him to finish seasons or win the bowl uh who cares how much he makes if he wins it as long as the ravens don't screw up the cap behind his deal love lamar but ah there goes hey i told y'all whenever you hear that you know what's coming next he said uh love lamar but i have loved the ravens longer if he don't want to be here peace i'm pretty sure the ravens are not offering him something disrespectful knowing everyone is waiting to say lamar made a bad deal without an agent hashtag keep it clean uh your thoughts and thanks love the content appreciate it outlaw um when when you first signed your rookie contract especially your first round pick it's a four-year contract um and then it has the fifth year option so you write about that and then it has depending on how things go it has the possibility of reaching the franchise tag but again with the the fifth year option that is it's like with the fifth year option and with the franchise tag and even if it gets to the second franchise tag what are those fifth year options well and what is the franchise tag all of them are essentially proven deals and i'm glad you brought this to my attention because i didn't even think of the fifth year option as one but the the the fifth year option sort of serves as a all right we like you um and we want to keep you around another year uh so we can hopefully get the most out of you okay cool no problem um and then maybe we will even come to a contract extension with you so okay okay i get that and it allows teams to to just have more flexibility allows them to have buy some more time uh until they sign whatever that player is whoever that player is so what's the franchise take it's the same thing it's the same thing just a higher amount because it gives you that player for another year and you obviously haven't signed that player to long-term deal yet uh and it allows the team to buy some time until they can come up with a contract agreement or they'd be like you know what now uh we're straight calamari fifth year option no didn't happen josh allen fifth year option no didn't happen now let me know and there's plenty of the quarterbacks there we can go through all the quarterbacks we don't have to but the fifth year option with lemar jackson i know it takes two so you spot on about that ravens they got to come up with a deal lemar jackson he got to come up with a deal they both got agreed to it so write about that but from your wording it makes it sound like lemar jackson hasn't earned his contract and of course you could let me know if i'm wrong but from from your wording it makes it sound like lemar jackson still needs to earn this contract like he's not deserving of a contract and with that i would disagree next question and i'm gonna let y'all guess what day this question came on uh but it came from hotty and appreciate you being a patron instead my man engraving what's going on lemar has been going off on his twitter this past hour first responding to a fan's comment saying that the ravens did not offer him a 250 million dollar guaranteed contract then he likes a photo of himself in a dolphin's jersey i have a feeling his patience is running thin in a deal is not likely to have him before his deadline what do you think the organization is playing at here much love to you big bro lemar jackson patience running thin huh i mean at this point it could be i mean that they ain't come up with a deal yet and now as of today as of this morning of this recording just about a couple hours ago they restructured ronnie stanley's deal so with them restructuring ronnie stanley's deal um that could make it look like they already went to their last resort as far as getting some money for this season uh and one of the first resource would obviously be for uh lemar jackson to open up a lot of cap space because if he signed an extension that could open up a lot of cap space um but he ain't signed no extension so they went to ronnie stanley um and and brian mcfarland raven salary cap he reported that um that he he would think that that would be an indication that the ravens have given up on striking a deal with lemar this year i mean we'll see so we just we just got to hold our hope um but i'm sure for lemar um it would be frustrating uh because again i try to put myself in issues like man you've been working at a job uh you see a lot of peers that you know uh in the same position that either started around the same time as you or even something after you uh and they got paid you've accomplished a lot uh in a lot of cases you've accomplished some more than some other your peers who got paid um but you still haven't gotten paid yet next question came from one of the newer patrons uh m r.a.p. appreciate you today and graven just wanted to stop in and say thanks for being consistent uh that is so important in entrepreneurship hey i appreciate that thank you all he said now excuse me for josh ross i liked him so much that i had to become a patreon finally i've been watching you for at least a year now uh the last time i spoke up like this lemar jackson was entering the draft and i tweeted the jets to drafting the draft is when i became a ravens fan oh okay so you came in with the monster since 2018 cool cool cool now nonetheless this ross kid looks so instinctive and calculated uh it reminds me of luke keekley a tackle machine maybe even a ray someday oh that's now that's some high praise right there luke keekley and ray louis uh he said this is what we wanted patrick queen to be but he seems more like the second man in charge uh now if we could take a machine and combine josh ross his tackling and patrick queen to speed his athleticism uh just his burst put him together he had a perfect linebacker but anyway um he said ross is sort of ross is soda queen is mentos i think they will compliment each other very well now it all depends on um um how much uh if any playing time that he gets because he made the roster all right that's the first step cool that's great uh will he be active i know special teams is a big thing so it's likely that he'll be active because welch was active for a lot of games um welch also made the roster so we'll see what happens there there because i don't think they'll both be active on game there i think it's gonna either be one or the other but we'll see we won't know till we know um but as far as him complimenting queen that could end up happening slowly but surely because josh binds is not the ravens long-term answer i mean he's been there the answer for the past couple years but he's not the long-term answer um and the ravens they they they clearly need to to find out what's going to happen at inside linebacker over these next over this next season um this next season is huge for inside linebacker since patrick queen uh he is in his third year um so it's time to decide whether they're going to pick up the fifth year option or not so we'll see he said he looked like the best linebacker we have had in years oh wow oh that oh that's how i pray you really really like josh ross he said um his consistency and instincts are what sold me yeah the the instincts official and and yeah the consistency too because he kept doing it like every game uh he said along with his short tackling ability uh what do you think about the potential of him starting mid-season on defense yeah it could happen it could happen it obviously starts with special teams uh but if you get opportunities on defense he would just have to make the most of it next question came from another patron shout out to my guy raven he said ain't graven i got a question for you uh do you think we are going to go far in the playoffs if we make it there i do i think we will make it there but how far can we go hey as as far as the ravens take them in this it's on everybody um it takes a lot uh it takes a lot of stuff just going right for you health uh play calling uh execution just everybody doing their job and then some so if they can get that done they can go as far as they want to next question came from the andre uh and appreciate you being a patron you said ain't graven how are you in the family doing hope all is well yeah everything is really good all you said should we go after there you're slating he will be a huge addition for us and gives us another big body possession wide receiver see what there is slating i was watching some stuff on him a few days ago and i see that he he ain't afraid to go up and get it uh he got some he got some good speed um but he uh he's a smooth route runner not a crisp route runner not like a cutoff cut on a dime type of route runner but he's smooth with it um so i mean i wouldn't be mad at that and then somebody else to for the yak the yak the yak has been something ravens have been missing so if they were to add him to i ain't gonna complain i ain't gonna complain it off they would they want to add another quality wide receiver okay cool but then i think hmm i wonder who the odd man out would be next questions came from my god feel um and he said as and again this is a recording this on august 31st at 10 13 a.m so practice hasn't happened yet but for the past two days anyway he said as we already know ronnie stanley passed his physical and got taken off the puff list but on august 29th they reported on ravens.com that stanley still has not stepped on the field yet and hasn't practiced once yet um hardball had no answer for this and only wanted to talk about the rest of the team so if you ask me i don't see stanley start on week one probably comes off the sideline as a substitute to work his way back if he doesn't start uh one game this season i think edc should try to trade him or do a buyout for the remainder of his contract he couldn't trade he couldn't trade i mean he could trade him but the dead money especially with the restructure that that happened early this morning it would just be a crazy amount of dead money like i think before the restructure his dead money in 2023 would be like 29 mil and that was before the restructure so i think with the restructure would be like 35 mil because restructure was for a little over 6.3 mil so they they wouldn't they just wouldn't be able to do it he also said with the ravens interest uh the ravens uh interest in signing kenyon drake uh does this mean they will release mike davis or trade justice here i think it would mean that jk dobbins will end up going on ire um because i think it would mean that he might just not be all the way ready yet uh but that by the time you see this video we will have the answer to that uh he said this past tuesday on a pat macrophy show they showed the new NFL footballs amazon prime created the ball is thinner and longer than the duke football and travels through the air like rockets saying there are rumors in a video from amazon prime on youtube saying these balls will be used for thursday night football comes september 16th which amazon prime network took over they showed youtube videos russia wilson and herbert throwing them like darts and the video mat staff are throwing them from one end zone to the other end zone you need to go take a look at these things i'm telling you there'll be a lot more throwing on thursday night football this season can they do that can they like use different footballs for thursday night football like that that would be so weird man and i have seen the ball it's like yeah it's like longest it's it's yeah it's weird that would that would be crazy because i mean uh that would be that'd be crazy if that happened um but i guess we we just got to wait and see and the last patron that sent in the question for this episode came from my guy mar and a shout out to mar he said i don't understand why this front office hasn't already paid lemar the moment he walked through those doors you should have given him whatever he was asking for so what if he wants a fully guaranteed contract my thing is you're going to pay lemar a 100 on that contract either way because if it's a five to six year contract you already know you're going to keep them all six of those years anyway why not just go ahead and give him the fully guaranteed deal also i don't think this team is very good without lemar jackson there is in my opinion no other quarterback in this league that can carry this roster week in and week out like lemar ravens better pay lemar soon because as we saw with the browns desperate teams will give lemar whatever he wants and what a way to go out because they sure will you you already know that they will um so yeah ravens just hopefully they get on it sooner rather than later right now it's looking like i don't know what it's looking like right now they got some time that clock is ticking though uh september 11th is right around the corner and lemar said once week one rolls around then all them talks talks gonna get cut off so ravens just hey it i mean obviously the whatever deals they offered him wasn't good enough because if it was good enough then we would hear a lemar jackson a sign of contract dissension with the baltimore ravens but since we haven't heard that yet well yeah this feels like a dream that's what i mean youtube team keep it clean you see my boy he like automated automated well that's my homie ain't that right in graven right in graven