Before I start talking about configuration management, I'm going to give you some disclosures and caveats. I'm going to be talking in some detail this evening about four different products. I think most of you have heard of most of them, if not all of them. There's salt, of course. There's puppet, there's chef, and there's Ansible. Anyone not heard of those? We've all heard of most of them. Now, first off, I am going to be biased. I'm going to try and be objective, but you should expect me to have biases, because I work for Solstack. I'm going to try and approach this from my point of view, working in the field. I did web programming for years and years and years. Then I did ops for a few years. I did some training, I did some QA, and I even went to puppet training about three or four years ago. And I'm going to tell you about how I approach this and how I see these things, and that doesn't necessarily mean that my views reflect the marketing department of salt. So I want to get that out of the way. I have only used puppet and salt professionally. I've used them both extensively, though salt much more recently than puppet. I have a buddy at Rackspace that helped me out with the examples for Chef and Ansible. If I screwed something up, then you guys, I'm sure, will let me know. We have a lot of DevOps professionals here tonight, right? And I do want to point out that I have met people from every single one of these companies. In fact, I met Michael Dahon at puppet training for the, it was his, I think, his second week working for Puppet Labs and his second of three weeks. He wasn't there for very long, and then he took off and went and worked at a couple other places and then went off and found at Ansible. Michael is a great guy, he's very talented. I've, of course, met some of the puppet guys. My trainer was Jeff McEwn, who still works for Puppet. He's one of the core devs now, and a great guy, absolutely brilliant. And I met, I can't even think of the guy's name that works at Chef. He's actually a Python guy working at this Ruby shop, and absolutely brilliant guy. If you guys think I'm going to go and talk smack about these companies, no, we have some really great products out there, some really brilliant people behind these products. And if you happen to be using one of these products that isn't salt, then I'm not going to diss you on it, okay? There's some really great stuff out there. Let's go through a quick timeline here. And I'm going to talk about these projects in a small amount of detail based on when they first started up.  And I think that you know which of the one of those five it is. CF Engine has been around for a long, long time. Now it wasn't the first configuration management tool of this type, but it was the first really big one. It was the first one to kind of start revolutionizing the world of configuration management and start getting people to stop making snowflake servers and start defining a configuration that not only applied to all the servers, but was automatically applied to all the servers. CF Engine is this tool that would sit on the machine and watch for files to be changed. And if they weren't in line with what they were supposed to be, then it would change them back. Now first off I want to point out when you have files changing on servers, why is that happening? People, people are changing them. Are these people that should be changing them? Now the first company I worked at that had CF Engine, we had a three or four dozen production servers and I was in the QA department of that company and the guy that was senior to me in that department, what he would do if he wanted to test something and he wanted to make sure that he was in environment that was exactly like production, he would go grab a production server, he would take it out of the load balancer, he would apply code to it, he'd turn off CF Engine, he would apply code to it and he'd test it and if he was happy with it, then he would turn CF Engine back on, set it back to what it was and put it back in the load balancer and about 95% of the time that didn't cause any problems. But you guys can kind of see this problem here. We have a lot of people at a lot of companies that are getting out of production servers that should not be on production servers. If you're gonna be testing stuff in a production-like environment, you should be creating a production-like environment and not just using production, right? So if you're managing configuration drift, which is what the public guys call it, that's because somebody has done something wrong, okay? It could not just be the person that actually did it but also the person that allowed them to do it or all the people that allowed them to do it. CF Engine was not necessarily created for that, it was created because the university that Mark was going to at the time, they gave him a whole bunch of machines and he had to keep configuration the same on all these machines and he wrote a tool to do it. It's been through three different iterations now, it's on CF Engine 3, three different really refactorings. It is still a big tool in the marketing place. It is not nearly as big as the other tools we're gonna discuss at this point, but it is still big, a lot of people really like it. What a lot of people really don't like is the usage. This was designed back in 93 originally and this was not a time when we had a whole lot of Python being used. Python was around but it was really young. Heck, Perl was really young at that time. People, if they're doing system administration, they knew C and they're happy to deal with configuration that made sense to people that use C and so CF Engine worked really well for them. Nowadays, a lot of people see it to be a little bit arcane, okay? So one of these guys was Luke Kines. He was a core dev for CF Engine for a while. He was really unhappy with it. He was working on CF Engine 2. There were a lot of things he didn't like about it. In fact, he did a talk at least 10 years ago called What's Wrong with CF Engine? And shortly after he did this talk, he went off and started working on Puppet. And at some point he went and got some funding and built a company around it. But by that point, people were really liking Puppet it's much easier to use. It's built around the concept of having easy to put together configuration. When he was working on this, he actually went through a few different languages. He looked at Perl first and Perl wasn't doing what he wanted. He looked at Python. He didn't like what Python was doing. He looked at Ruby and he settled on Ruby and he really liked that. So he started working on Puppet as this configuration really for CF Engine. He considered it to be kind of the next generation of configuration management. And who here has used Puppet? Who used it when I was really, really young? A couple of you guys have. And it changed your world, didn't it? Things were great. Puppet's a really great tool and I thoroughly enjoyed using it for the first couple of years that I used it. One of the things that he uses, and again, I haven't used CF Engine myself. I just know that we had it in use of that company but Puppet has what's called a DSL, a Domain Specific Language. It's not Ruby, but it's kind of Ruby-like. If you're somebody that's been using Ruby, then it's something that will look kind of familiar to you. And if you've been using Perl or Python, then Ruby looks pretty familiar to you anyway. And in fact, if any of you guys have been using Perl, then the weird operators will look really familiar to you, right? So another guy came along, Adam Jacobs, and he liked kind of the direction it was going in but he didn't like that Puppet was non-deterministic. It's declarative. The idea is that by default, when you define something in Puppet, you say, okay, I need this to happen. I need this to happen. I need this to happen. Go for it, Puppet. And Puppet will just go out and do what you tell it to do and it's gonna do it in whatever order makes sense to Puppet, right? Chef is more imperative. You say, do this, then do this, then do this, then do this. And this makes sense to people that have been working in the Dev World for a while because when we write code, it's generally do this, then do this, do this. That's how scripts work, right? That's where we got the name script from anyway. It's like in Shakespeare. The actor does this and then the actor does this and then the actor does this. He doesn't just do things in whatever order or else Macbeth would be a whole lot weirder than it already is, right? So, and Chef I think was one of the first companies to really start pushing this infrastructure as code thing. They really started to push the idea of DevOps. Now, DevOps is a fine thing. And I've always thought that, in fact, I've been saying for years that the best sysadmans have some programming background with them. And the best programmers have some ops background, right? I've been saying this for a long time and I think that DevOps is a great move in that direction. Unfortunately, I have also worked for companies that have said, okay, we're gonna have you be one of the programmers and we're also gonna have you manage the servers because we're too cheap to hire ops. Has anyone worked with a company like that? And you may have discovered that when you start working for a company that starts doing that, they start cutting a lot of corners. Not just, let's have the programmers also do ops or one particular company, I was the tech guy. I did all the programming, I did all the ops and if something went wrong, then I called the rack space because they were hosting us. Okay, so that's kind of where Chef is coming from. Now, along comes Salt. Now, Tom and I were working for a music provider at the time and we were using a number of different tools. We had Puppet going. Tom has also been to Puppet training. Tom has submitted some code upstream to Puppet unlike me. He did a lot of work there and it was something that we really, really liked when we first got into it. He spent a few months trying to convince me to hire onto this company and a few months later, I finally did and the day that I started, he said, okay, we've been using these tools. In particular, one was called Funk and we're really unhappy with it. It keeps falling down. We can't get it to do what we needed to do. It's too difficult to use. So I'm gonna write a new tool and I'm gonna call it Salt. I said, well, that sounds great. And two or three weeks later, he came in. He'd been working out at home and he said, hey, it's working, more or less. So would you mind writing some modules for me? And so I started contributing code and that's how I got to be the second person writing code for Salt. And he started, you know, he put it up there. It was just a personal side project. He put it up. People started contributing to it. In fact, if any of you guys hang around in the Salt community, the fourth person to contribute was Seth House who is now an employee and the fifth person was Pedro Algarvio who is a community member right now but he's about to be an employee as well. These people have been around for a long, long time. And we discovered that in, I believe it was August of 2011, LinkedIn started using Salt. Now, Salt was really young back then. I believe they started with version 0.8.3 which was barely stable. And LinkedIn looked at it and said, hey, you know what? This is an open source project. We like the direction they're going in. We know that there's gonna be bugs because they're so early on. So we'll just, we'll roll with it and we'll see how things go. And we started using it in production at our company in October. So even LinkedIn has been using it longer than in production than we have. And we didn't even find out about LinkedIn until the next April. And then December of that year, that company gave us the opportunity to look at other opportunities. And I went with one for a few months until Tom finally hired me on. But Tom took a different approach to this. In fact, originally Salt was only designed for remote execution. We wanted to, our biggest need was to kick off monitoring jobs. We had a guy in charge of ops that was, he wanted to collect vitals about the systems every five seconds because he was insane. No, a great guy actually. But five seconds still seems like a lot to me. And he wanted to be able to analyze that more or less real time and be able to handle server load based on those stats.  It didn't even do remote execution for a long time. In fact, remote execution, LinkedIn, it wasn't even using it for their first year so that they used it. It was still really young at the time. But we realized that when you look at it, a lot of the things that we do in ops or even in development is, it's remote execution anyway. Configuration management is a type of remote execution. You are going out to a, well, with Puppet and Chef and CF Engine, you had the remote system keeping track of these things, but you still have the remote system executing something remotely based on something stored on a master, right?  Again, this was a music platform, so we would kick off MP3 encodings remotely. We did a lot of remote execution, and we realized that configuration management was just another type of remote execution. And at that point, it was trivial to just add it in. We could already do a package.install, so why not make it stateful? And rather than using a DSL, like Puppet and Chef were doing, we realized that when we define a system, we're just talking about data. We're saying, okay, we need these packages installed, we need these files to be managed by whatever service, we need these services to be running, and it's just a bullet point list, right? And so we started defining these things as data. And in fact, we went with YAML because it's text-based, it's easy to modify, it's really easy to learn, and people don't have to learn a new programming language just to use their configuration manager. In fact, one of Todd's big beliefs is your configuration manager should have as little configuration of its own as possible, at least required configuration, right? Why should you be putting so much effort into configuring your config manager, right? So even now, when you install Salt, you can install the master and have it running with zero configuration. When you install the Minion, you can have it running with, technically with zero configuration, but there are a couple of things that I think you should do. You should tell it where the master is instead of just telling it to look for a machine called Salt, and I think you should explicitly say the name of the Minion, but if you don't, it'll figure those things out, okay? We also had a few things that we came across when we were designing Salt. We were worried about scalability. Again, we were running Puppet, and has anyone run Puppet with Web Brick? How many clients can we have at a time? One, maybe two if you're lucky, right? And then we move up to Apache, how many can we have? Under 40 max? 140 max, okay? We actually had, when we hit about 40 or 50 servers, we had to put up a second Puppet Master. And again, these, we were doing everything out of VMs. Our boot server was a VM, it's very scary. We didn't do that. But yeah, we had scaling issues and we assumed that we would also have scaling issues with Salt, and so a lot of things were put into Salt in order to compensate for these scaling issues that we expected to have. Things like the Cintiq system. We were also running the Master behind HA Proxy. And we just assumed we'd have a lot of these problems. And once we got into it, we realized that Salt, because of how it runs the remote execution bus, it can actually handle hundreds if not thousands of servers at the same time.  Okay, next up, Ansible, Mr. DeHaan. He created this the next year. And now people that are really comfortable with SSH, which is most people that work in Linux, right? That was what he's targeting, was people that really just, he understood that SSH was another type of remote execution and he decided to go down that path. Now, the nice thing about SSH is it requires no agents on the remote machine except for SSH. And in most cases, SSH is gonna be there, right?  So you're still gonna have to install that SSH agent. But if you're installing a sent machine, a rail or something, or Ubuntu server, it's gonna be there, right? Designed for, again, he realized that this was remote execution. It was ad hoc. He also did a lot of config management because at that point, you've pretty much gotta be doing that and it was designed with simplicity in mind. It's a very small program, it doesn't have a whole lot of docs. And it's also very easy to get into, just like Salt.  First all, imperative versus declarative. We have the concept of imperative as the script-based approach. We do this, we do this, we do this, we do this. In that order. Declareative is, I declare that these things need to be done and leave it up to the compiler to do it in whatever order I tell it to do it in. Now, we can add requisites like this requires this, this notifies this and so on, but by default, it just does it in whatever order. Now, on the imperative side, Chef is imperative. We talked about that. Salt is imperative and Ansible is imperative. However, on the declarative side, Puppet is by default, it's declarative. Salt is also declarative. You can just define stuff and by default, it'll run it in the order that you tell it to do it, but you can also put in those requirements statements and it doesn't require that we do everything in order. I can say this thing up here requires this thing down here, so I go ahead and do it in those orders and then go back up and continue through your Salt run. DSLs. The problem with DSLs, domain-specific language, is it is specific to a domain, right? Once you've learned the Puppet manifest, can you use it anywhere else? Are you gonna be writing any actual code with it? Is it even turned complete? No. And again, with Chef, it's basically Ruby with a lot of extensions, right? But it's specific to Chef. Now, again, so DSLs apply to Chef and Puppet. Salt does have a number of DSLs available. In fact, they are all Python. We have straight Python, we have PyDSL, and we have PyObjects that are all available. My favorite one of those is the PyDSL, but I'm not gonna use it personally unless I absolutely need to. I'll probably fall back to YAML-based configuration, which goes on the no DSL side. And Ansible also uses Ansible. It's not a DSL, it's just straight YAML. Push versus pull. Push meaning that on the server we push a command down to the client and then maybe receive some results back, and pull means the client requests information from the server. That means the client is either going to be, we have to SSH in and kick it off manually, or we put it in a cron, or the server manages the time it connects back to the server. Ansible is push-based, and Salt is push-based. It makes sense for an SSH one to be push-based, right? It's gonna be hard to get the client to set up all these reverse tunnels or something like that. And Puppet and Chef are all pull-based, so they all have the client request from the server. And Salt, if you use the Salt call command, it can also request things from the server.  It's up to you. I've found that the vast majority of our users, though, prefer the push-based version. Kick things off from the master and have it do it immediately rather than waiting for cron to start something. In fact, we have a number of companies that are using, they've got years and years of formulas in Chef or of Puppet manifests, and they don't wanna throw that away, and who can blame them, right? I'm not gonna ask them to throw all that away. That's just, that's not even right. But they'll use Salt to kick off their Puppet jobs or their Chef jobs. And so they'll wrap it all together. Config management. Again, Puppet and Chef are designed explicitly for config management. Salt and Ansible do configuration management as some of the things that they do. Immediate remote execution. Again, config management is a type of remote execution. And Salt and Ansible do it now instead of waiting for the client to check in and do things. Going a little bit more into remote execution. This is something that I personally like about Salt since it is based on a foundation of remote execution. We have this whole foundation that other things can tie into. We could build monitoring on top of this. We could build our applications to just use those communications buses. Configure management. I already talked about how it's a type of execution, remote execution. SSH is still a type of remote execution. And on its own, it's not automated, right? Queuing mechanisms. Anyone here use RabbitMQ? And you guys use it to kick off basically remote execution, right? We just deal with a long term queue. We do this and then we do this and we do this as things come along in the queue. Whereas Salt isn't really designed for long term queues like that. One could be built in easily enough. It just doesn't have that design part in there right now. Let me go through some quick examples. I've got about a minute left here. As far as I'm concerned, the basis of configuration management are three big things. Packages, services and files. You lay down the package. You lay down the files that overwrite the ones that came with the package or augmented and then you start up a service, right? Now you can go into far, far more detail than this but this is the baseline I chose for my example. So with Puppet, we're installing Apache. So this is probably a red hat type system we're talking about here. Package, insure, installed, that makes sense, right? Service, again, we're talking about the same service name H2PD. Ensure that it's running. Make sure it's enabled. That means not only is it running now but when we restart the server, it will still be running, right? It requires the package to be installed. So we have now made this not quite imperative but we've put in some requisites there. And it's also gonna watch this file down here so that when this file gets changed, it automatically just restarts the service, right? And that's kind of the same thing we're gonna see in all these examples, chef. I'm not good at Ruby guys. But apparently this is correct. We give it a package. We set up a template here, ERB. This is a Ruby-based templating engine, right? And so this is assuming that we've got some templating going on. Puppet was as well, it was also using the ERBs. So we create a file and in the service we have a start and enable, okay? Assault. We set up a state that starts with HDDPD. First we do a package install and we do a service running. We have a file we've laid down and again you see the requires that I've set up here. Sorry, I'm just about done. Okay and it's pretty close to the same idea but you see that it's not in a DSL. It's just a list of the animal directives, right? And the shortest example I have here is Ansible. There we go, we're back up over here. So, host-sol, this is just gonna run on everything and again, very imperative. First we install Apache, then we copy down the Apache config, then we start the service and really all the work is being done down here. So name is just giving it a name and then we have, this is the module it's gonna use and any arguments it is gonna pass through, okay? Now, the question is which one do I use? Well, it depends. And I'm not just gonna tell you, start using SOLD now, everything else you've done is just throw it away, okay? I'm not gonna tell you that. There's a lot of things to consider. Do you have any legacy CM code, config management code? I don't, when I say legacy, I don't mean all things puppet, all things chef, all things Ansible. What I mean is code that we've been using for a couple years. We don't wanna just throw it away, right? And if you have a lot of code that you're not willing to throw away, I don't think you should throw it away. I think that SOLD can help you and can wrap around it and help you do things a little bit better. And when it's time to refactor that code, then it might be time to refactor it in SOLD or it might be time to just move into a different one and then still use SOLD to control it or throw a SOLD away immediately. Do you need immediate remote execution or are you willing to wait around for a job to kick in whenever it kicks in? That's really important, right? If you're okay with servers just watching themselves and getting into check, then by all means, chef and puppet will be just fine. If you wanna do it things immediately, then you might wanna at least think about controlling it with SOLD. Are you planning on using this layer programmatically and are you a Ruby shop or a Python shop or some other kind of shop? That's a really important thing to consider. And if you're a Ruby shop, then it could be entirely that chef is a better tool for you or that puppet is a better tool for you. Is it appropriate to use two of these things together or three of these things together? If you have legacy CM code, there's a good chance that that is appropriate, right? And last but not least, which one feels right? Which one is going to allow me to go home and be nice to people, be nice to my family and which one is gonna allow me to go home and yell about how rough my day was at work, right? If SOLD drives you crazy, don't use it. Use something else. If something else drives you crazy, go back to SOLD, okay? But use whichever one feels right, okay? And guys, spend a lot more time researching these things. This actually came from somebody at Rackspace that asked me, do you have a really quick way to tell me or for me to tell people what the difference is between the tools? No.  Hopefully I've gotten you guys started in the direction you need to be in. I'll go ahead and send these slides over to Matt and Matt and they can post them somewhere, I'm sure. And I guess there'll be videos online. If you guys have any more questions, we're gonna have to take off during that first break there but we'll be around for the first couple of talks and you're welcome to email me, josephasallstack.com and I'll be happy to respond and see what we can do. Okay?