naged service like a telco would for remote site connectivity. I work in both network and security stuff. We do large managed service environments and presently we're managing about 2,600 of remote CPEs in a variety of different environments. The tooling that I'm presenting today is fairly Cisco focused at least in the code that I've written. However, the principles behind it are things that would apply to really any platform and if you're fortunate enough to be using another platform that gives you XML interfaces more advanced methods of retrieving configuration and operational state the same stuff can be carried into those environments as well. On the state of our industry, it sucks. The networking industry is very much, if you think about where system administration was 10 or 15 years ago where people were very happy to manually craft each server and hone each individual element of its configuration. That's really changed a lot and even in more recent times DevOps tight integration between different groups. All this stuff has really changed a lot and what I've found is that in the network industry those changes haven't really happened so much and haven't happened in a structured way. There's a lot of manual processes involved. I know that there's a very major telco up in Canada where I work where they actually build the configuration for devices with their engineering teams using notepad because the engineering teams don't have access to the network devices and then field technicians paste those configurations into the devices in the field but they don't actually know how to work on network devices and hopefully they report back if errors occur during the device configuration process. Now hopefully no one here's situation is that bad but there's certainly a lot of room for improvement and when I look at the things that people are doing to improve the situation you have a lot of vendor proprietary tooling each of the different vendors has their kind of suite for managing or attempting to manage their devices and you also have commercial tools that are also quite expensive and often I suffice to say I've seen some very expensive implementations of SSH in a for loop. So I feel that there's a lot of room for evolution in this space and I think that the open switch stuff that we're starting to see from the open compute project, cumulus networks, things like that are I'd like to think the beginning of a sea of change like the transition from traditional Unix environments to Linux was and hopefully we'll be able to really improve things in the way that other parts of the IT industry have changed. In the configuration management space and I imagine this probably applies to most any part of configuration management not just for network devices. I first am looking to identify specific issues so we might see VPN flapping where specific devices are having a problem and seem to be disconnecting or reconnecting to the network. Doesn't really seem to have an operational impact but at some point it will and you wanna understand how big of an issue is this, how common of an issue is this. So I'm identifying either issues in the environment or new projects that need to be delivered for a customer and then we're generating reports that allow us to sort of examine the information about the systems and see how serious a given problem is to allow us to kind of prioritize specific changes that need to be made, what's more and less important or pressing. Finally, the tooling that I've developed allows us to then make those changes to the devices and in my environment with large numbers of remote devices in different field sites, I don't have all of my devices operational at any given time. Something's being serviced, there's an outage in a given area so it's important that the process that we use to make these changes is able to handle a device not being at the current reference, examining the state of the device and determining what changes need to be made it to bring it to the desired configuration. Some changes as well are gonna be dependent on operational state. For instance, if you wanna make a change to the cellular internet connection on a device but that's the active internet connection and the primary internet connection isn't present, that's not a change you wanna make at that time, especially if the consequence of it is losing the device. To achieve these goals, I've used a number of different approaches. Today I'm speaking about the trigger-based approach that I'm using now, but I've used a bunch of other tools and probably maybe some of you have as well. So Rancid, which I suspect is familiar to a lot of people in terms of a tool to collect network device configuration and report on changes in it, also has the ability to do custom scripts using C-Login, its tool for connecting to devices. So I started on that first migration project I spoke about before, writing scripts in TCL and pushing them out using C-Login. TCL is not the greatest language ever. Two little pet peeves I discovered during that project. If you put a comment that includes a parentheses, that will still be interpreted by the parser, but it does at least give you a warning. If you do like an if-else-if-else construct, but you typo else-if and don't spell it in the specific way TCL likes, which is different from Perl or Bash's, it'll tell you that else is an invalid command because it's, I think it might have interpreted the typo of else-if as an anonymous function or something. Anyway, it's really bad. I transitioned from that to using Perl and I'd move all the logic out of the TCL script. So I didn't need to do logic in TCL. I'd get state from a device, I'd parse that state, I'd decide on a method of changing its configuration. I'd generate a TCL script that I'd then run with C-Login. So it got rid of a lot of the pain of TCL, but it wasn't that much. It was a big improvement, but there was still much more ground to be gained. I've cut back on abusing Perl and switched over to Python, which has a much more structured approach. I know it's possible to write clean Perl, but I can't write clean Python, so it's not gonna happen with Perl. And then most recently I discovered Trigger, which is a framework for interacting with network devices. It provides a fairly elegant interface to connect to devices, but it is very much a framework. It's something that I've built tooling on and that other tooling can be built on either directly on Trigger or on the stuff I'll show you here today. And that has really improved the methods that I'm using and able to, yeah. So in terms of the steps that we have in the process of making change to a network configuration, first, I'm gonna talk a bit about the approach that I use within the code that I've got, which is generalized, whether you're doing reporting or making changes to the configuration. Then I'll talk a bit about the specific implementation of normalization. It goes a bit into detail on the Python code, but I think it's important to have an understanding of how that framework has been built, reporting which leans heavily on the normalization approach and then we'll talk about customization. And I've built out in the presentation a concrete example of a new type of reporting that I did just for this talk to show how easy it is to take the work I've done in understanding how these things work and carry it into customization for your own environment, which is a little more straightforward than the groundwork that I've done. And finally, we'll talk a bit about some improvements that I'd like to make to the tool. I left out something important about Trigger before. One of the big advantages compared to C-Login is that it is able to connect to multiple devices simultaneously and because of that, if you wanna process stuff for a thousand devices, it's a lot faster when you're doing them 10 at a time or 20 at a time as opposed to connecting to them in serial, especially when some of them are over cellular connections or are not online and you're waiting for them to time out because they're not responding. Specifically, the approach that I've taken is to first collect the configuration and device state. So we're looking at what the running config is on the device, what it's operational parameters on, then to parse and structure that information so that we have a clean representation using objects, dictionaries of the device state, which we can then analyze and determine if there's changes that need to be made to the config. If changes are required for a device, then you make those changes and you, I also store all of the information I collect from the device into a repository of device information. It's useful when you're doing reporting because you can do reporting without connecting to the devices so you can reference that stored state information if you wanna see how many devices have this specific config in place. You have the information local and you don't need to go out to the devices again. Inside the example code, we have four critical files. We have router.py, which has the kind of core logic to this system. There's normalize.py, which is a script which actually will perform a normalization on a device. Report, which is a reporting script. And I have a device list with three routers with highly traditional names. So if we look at how that, is that, okay good. A previous presentation I saw in this room a few hours ago, the text was illegible. I think this should be pretty okay, but if there is a problem, please let me know. So I ran normalize. It found that there were, it prompted to say, do you wanna run all sites because you can select a specific subset of sites? It chose to, I went forward with it, so I'm processing routers one, two, and three. One of them is down, couldn't ping it, wasn't processed. And one of them didn't have the trigger ACL present on the device. And that ACL that we're validating is example code, permit <IP_ADDRESS>. A lot of the stuff that I've developed is very specific to the device configurations in my environment. So what I have here is a very generic example. I'll come back to that a bit when we talk about the improvements though, because I think there are more common things that different people in this room might share and benefit from that could be done in a community way, even if a lot of things are gonna be specific to your own environment. So at the heart of the system with Trigger, which is based on the twisted framework for device interaction, I have a callback processing. And the callbacks, the way that they work is we take the list of devices and we get the details about them. The returned information from that is then processed by a script that validates the device's state. And then finally we initiate the normalization of the devices. So a device that needs a change is going to then build out the change to the configuration. If we, oops, just a second. Okay, so called out the critical elements of the function. So in normalize.py we have the get router details, which is an instance of a Trigger commando object. It's checking to, it's got a list of commands that'll be executed. And each device is checked to see if the device is available, tries to ping it, if it's available, then it's returned for processing. And within the common router.py code we have the actual show commands that will be run on the device. So we're doing a show run include IP access list and we're getting a show ver showing us the version of the device. When we then take that information back to parse it, the information, so each device that was processed and the result set that was returned is run through the validate function. And the validate function is first updating information on when the device was last contacted, which is used in the reporting. And we call the validate ACL function, which goes through the results of the show run IP access list, splits that out and uses a simple regex to determine if it's a standard or extended access list and to get the name of the access list, which is added into a list of ACL objects known on the device. We then have the functionality that normalizes the configuration. So for each device we call this normalize function, which is down at the bottom in the router.py. We look to see if trigger test one is present in the list of ACLs that were defined on the device and if it isn't present, we set a variable that tells that the device does need to have its configuration change and append to the list of commands the changes that need to be made to the configuration. So if you have, you have a bunch of different checks and those different checks, you'll see if it changes required to the device or not, and all those things are built together into a list of commands to be executed on the device. When we actually go to make the changes to the device, we just go through the list of devices returned from that function and if they do need to have their configuration changed, then the commands that were built for that device are pulled out and used as the commands to be executed on the device. And then finally, we validate the return from that to make sure that the right mem that's executed at the end succeeded. On that thousand device migration project, we ran into a few machines that when you saved their config reported that the flash was bad, which is always interesting and I wonder how long it had been the case. Most of them were fixed with receipts, but not all of them. Finally, we store the state of the device into, in my case, I'm just doing it with a JSON object. I take the list of objects and I represent it as a JSON object. If you had a larger number of devices or you're doing something more advanced, a better storage mechanism might be required, but it works quite well for the scale of devices I work on and would extend up to tens of thousands of devices fairly easily, admittedly perhaps not with the smallest file. For reporting, the output is fairly simple. You run report, it automatically will use all devices and this report just shows the device when the device was contacted and the firmware version running on the device. And then I've run it a second time where I've selected a specific device to be accessed and you see in the report, the access time for that device has changed. So when you've got a large field of devices, you're not going to reach all of them every time and knowing when information is from is extremely important in understanding its value or deciding how you want to act on it. So this slide should have come before the previous one and it talks about that reporting process. Okay, so this is actually the, this is the overview of how that actual code works. So we load the state information that's present right now, we identify the devices that require updating. We connect to the devices, we get updated information about them and we generate the report for them. And you'll see that there's a lot of commonality between the normalization process that I reviewed before and the reporting process now. So we're using most of the same functions that were present in the core router.py code. We just don't do the normalization. So we're doing the same validations, it's using the same validation code and the same commands to collect information but we don't need to make any changes to the device. So we have code to output a CSV which is pretty straightforward, goes through the different routers that are present and writes out when they were accessed the device name and the version. We speak about adapting this to your own environment. There's a lot that you can do with this to take the framework that I've made and to implement specific things that you're interested in, things that need to be validated in your own environment. I suggest that you work first on creating small tests that are exploring individual and smaller elements of what you're interested in doing and then build from that over time. You can also bring in information from other data sources. So I do a lot of stuff where I'm pulling information from monitoring systems about historical device availability. You, in larger environments, probably already have decent repositories of information about your network environment and that all can be pulled together and in. So what I built as an example of how it's relatively easy to extend is something to parse which ACLs are applied to which interfaces in a Cisco or Cisco-like device and this single slide actually shows the entire change that I made to do that. So it is a lot easier to add things on top of what I built than it was for me at least to build out that initial example. So I've added in a new show command that looks at the running config and gets the interface sections and then inside the validate function that we're calling to validate device state, I've added in validation of the interface, a new function called validate interfaces and inside there we create a dictionary of interfaces. I clear it at the start because I had some bad experiences with interfaces being removed that because I didn't rewrite the entire object each time, it would cheerfully report really old information about interfaces that were no longer on the device. So that is important. And then for in the config I go through the show run interface sections, I pull out the interface names. When you find an interface name it gets added to the interfaces. And then we build for each interface if it's got an input, an inbound and an outbound ACL those are sort of recorded into a dictionary underneath that for the specific interface. I've only done ACLs here but obviously this could be extended to all sorts of different stuff. IP address configurations, QOS configurations. And this is kind of something where there's a lot of, this part of the validation is something that a lot of different people have the same needs for as opposed to how you take this information and validate its relevancy to your own environment. So we have the output from it and I ran it against one device and we can see that the gigabit ethernet zero interface has an inbound ACL of test applied to it. There is a lot of room for improvement in what I've done. One thing is I've done a fairly, I've done a manual job of building this interface parsing code but first a lot of it is stuff that could be applied to any device in any environment and it's something where there's more common code that could be created. And there's also a Python library called Cisco ConfPars that will build a representation of a Cisco or Cisco-like device configuration kind of automatically it doesn't have as much understanding of how it works as something else might but it understands hierarchical command configurations and nested devices under each other and it's supposed to do a pretty decent job. I haven't really played with it that much but it is something that would be pretty cool. A much larger thing would be the development of a domain specific language. So right now, all of the stuff that I've done, you need to actually understand Python and unfortunately there's lots of people that know Python, there's lots of people that know networking but the overlap between them isn't necessarily as strong as it could be unfortunately and there's not as many people that seem to be you know, fluent in both we could say and bringing it to a domain specific language could make it easier for networking people with less programming ability to work on this type of system and to benefit from it. You also need to avoid creating a domain specific language more complex than Python which well, I think we've all seen examples where that could have gone better. Finally, another big area for improvement would be more community involvement. So I did this, I was fortunate enough to be able to kind of contribute back open source portions of the work that I've done and I'd like to think that there's other people that are interested in this subject that might also be able to work on this both for their own internal use and outside. There really doesn't seem to be much public work done in the network automation space and especially with open source tooling. I know that there's a lot of people doing stuff behind closed doors and that there's things that can't be shared with the outside but I think there's also a lot of room for more improvement and more change and I hope that this talk will maybe get a few people interested in it and that we can start to build more of a community around this and other related projects. I have many people to thank for being here today. First and foremost, my wife, my kids, my family who support me and put up with me and other stuff. Jathan, who created the trigger language framework and was able to also make that an open source project and available to other people without people doing open source work, there's nothing for other people to build upon. Stack 8, who's given me a great place to work and to develop my own skills and ability. My good friend, Henrik, who helped me with Python and actually didn't want credit here because of the quality of my Python but I try, I try. Charlene, who really turned this from a much more bare bones presentation into what it is now, the trigger community on FreeNode and my friends in the Linux channel, went through that a little and if you'll wait for the microphone. Why use trigger and not use something like Ansible? And not wish, I'm sorry? Ansible? Ansible. At the time that I did it, I don't think that Ansible had the support for the Cisco stuff. Also, I found it very, very hard to find anything on the subject. So when I was doing the C-Login stuff and hating it, I was looking for people doing this kind of work and I eventually found Jathan's presentation from three scales ago that spoke about trigger. I recently did learn about the Ansible, Cisco integration stuff, like the bare SSH support stuff. I'm interested in checking it out a bit more and really, while I'm happy with the work that I've done with this, I'm also open to other approaches. I'm looking for the best possible ways to do this. I'm looking for the best ways to scale it. I'm looking for devices, things that are more cross-platform and the more stuff that I can do with it, the better it is. I heard about some interesting stuff involving salt and network device automation as well, and I'm interested in checking out all that stuff. So while I'm presenting on Trigger today and I'm very happy with what I've done with it, doesn't mean that we couldn't be talking about something else in a few years. Hi, this is not a question, but I did some stuff with Trigger where it would log into devices and parse gigantic config files. And a problem in general with some of these devices is that there's no grammar for the, and you don't necessarily want to write a grammar, there's no codified grammar for things. So a lot of these things will actually dump XML and if that's available, I would definitely recommend using that versus trying to parse. So I know that developer devices do that. They have fairly strong XML support, as I understand it. Unfortunately, I haven't had much opportunity to work with them. I'm just saying if it's available. So obviously something that gives you a structured form of the configuration of the device would be preferred. I know that Cisco worked on some stuff with 1PK, which was supposed to be this kind of platform for automating their devices. And I went to a presentation in their offices in Montreal and many people were there for the lunch. I was there because I was interested in what they actually had to say, somewhat to the astonishment of the presenters. What I found, however, was device support was very limited and on a lot of non-Nexus devices, it was also necessary for you to run T-Train releases. Cisco subsequently- Oh yeah, no, I'm not saying it, you know, like everybody does it. I'm just saying if it is available, it can save you a lot. And sometimes when you're parsing, you can go line by line, but in some cases the configs are involved enough that you actually have to parse it, you know, into a parse tree and try to suss out like this ACL is attached to this device sort of stuff that is difficult to do without doing a full parse. And Jay, I think, can speak to this too, but I mean, you know, like they've done parsers that ended up being sort of frank in parsers for multiple devices, but it's difficult to say, you know, is it correct for any particular device, right? In Trigger's method, you do have a method of when you define the device, and I didn't really show the details of the CSV file. You also state what platform it is or what family of device it is, and that can be used to drive both how you connect to the device for devices that have like non-SSH support. And I'd imagine you could kind of feed that information back in in other ways as well. Yeah, yeah, no, I mean, there is parser support. It's just getting to the point where it was like, oh, this is a little different, oh, that's a little different. We'd actually talked about splitting into, you know, very manufacturer specific parsers. Anyway. You mentioned that you, and I apologize, I missed part of the first part of the talk, so maybe you covered this, but you mentioned that you were looking at other open source options or you didn't find any, but I was wondering if you looked at any of the prior works such as NetConf and Yang, which are standards-based protocols for manipulating figs on devices or even some of my colleagues are working on OpenConfig, which is an evolving standard right now, which is actually documented and available for driving device configuration in a vendor agnostic manner. So, unfortunately, I did not hear that question very well at all, and I think that the speaker facing is much better for the audience than the presenter, but I did hear you mention NetConf. Yeah, so I mentioned NetConf and Yang, which are sort of prior art from like 10 years ago on sort of programmatically driving device configuration, but it's also mentioned OpenConfig, which is a new standard protocol, that standard-based protocol that's being developed for device configuration and vendor agnostic way. I haven't heard about those before, but I'd be very interested to check them out. And maybe if you want to approach after, you could tell me a bit more about it. I'd welcome it. Any other questions? I have a bunch of references, starting with this presentation, which should be online on the website shortly, and I've got a relatively short URL if you want to grab it here. I've got a direct link that leads to the example code, which is in the GitHub Trigger repo, the documentation for Trigger, as well as the Trigger GitHub project page. For people that are doing, for people that don't know Python, but are interested in this subject, I pulled out one well-known Python guide, the Learn Python Hardway, which apparently is actually the easy way to learn Python. And it's supposed to be a very good approach to Python, especially for someone who doesn't necessarily come from a programming background. I think that if you do have programming ability, Python should be fairly easy to pick up if you don't already know it. I've got my website and my email address and my employer, Stack 8. If this stuff sounds interesting to you, but implementing it in your own environment sounds hard, we could also talk about that. And finally, I got all the graphics for this presentation from a website called flaticon.com that has a lot of really nice, clean little glyphs, great for presentations, other stuff. Here's the licensing on the different files. Thank you very much for your time. And if you're interested in talking more now or at any other time, I really am interested in learning more about this and hopefully other people have things to share too. Testing, testing, testing, testing. Hello, hello, testing, testing. How's the volume? Can you hear me okay at this level? How about that, is that? Okay. All right. Cool. Again, my name is Dave Boucher or for Salt Stack and I'm really excited about this talk. It's a fun one for me. It's kind of a crazy stuff. You never do it in production or in real life but kind of give you an idea of a good or vision of the possibilities of things you can do with Salt. Who here has used Salt either in a testing or production environment? Okay, so about 80% of us probably grabbed a guy's shake in his head. Yeah, he does. Okay, so I'm going to just talk a little about some of the very basics. Some of the general use cases of Salt that people use Salt for. And then I'm gonna talk about some of the other pieces of Salt that people often kind of miss when they think of config management. They don't realize there's so much more to Salt than just that. And I have a couple of demos that will be interesting. And so get to use your cell phone. If you have a cell phone sitting down here, mine doesn't, unfortunately, but we'll be able to, everybody will get a chance to participate. So beyond config management. All right, so Salt Basics. There's, generally you would use Salt with a running a Salt Master that controls other servers that are running an agent called the Salt Minion. If you have a situation where you don't need a constant connection to your servers, you can use Salt SSH to have Salt work across SSH in a, like, one time or, you know, not have that constant connection all the time, but just go over to SSH when needed. So this is kind of what they typically would look like. So you have your Salt Master, you have a couple Minions here connecting through Zerum Q, and then also Salt Master reaching out to one server through SSH. If you'll notice, the arrows are pointing from the Minions up to the Master because when you're using Zerum Q, Salt uses a PubSub interface and the Minions actually connect to the Master. So the Master needs to have two ports open for the Minions to connect to and the Minions listen on that published port. Whereas when you're using SSH, Salt Master does reach out to your Minions and does things. When you're using Zerum Q in this PubSub interface, that allows Salt to scale into the tens of thousands of servers because the Master doesn't have to, you know, specifically manage every server it's connecting to, so that's why it's done like that. You can also use the Salt Minion by itself, so with no Master at all, so everything's just done locally. Sometimes I'll do that on my laptop, I don't want, you know, sometimes I'll have a Master and a Minion running on my laptop but other times I'll just use the Minion by itself to configure my laptop. People do that in production as well. If for some reason they can't or don't need a Salt Master, they'll just use the Salt Minion by itself and do all the configurations locally. They'll pull them down from like S3 or other locations. A Cyndic allows you to have hierarchies of Masters. So you can have a top level Master that can see everything in your entire infrastructure but then maybe you have a lower level Master in different data centers or maybe different business groups have their own Master down below and that allows you to have kind of a high level view from this top Master but lower level control down below. It also helps with scaling as well. Minions can also talk to multiple Masters. So you can have a multi-Master situation where they'll accept commands from Masters. Some people use that for a kind of poor man's failover type thing for your Master or also for different people needing control or visibility into your servers. So advanced Salt features. One of the things that really differentiates Salt from others in this space is that fast encrypted dynamic connection that your servers have between the Master and the Minion. So we have an event bus that runs locally on the Minions and also runs from Minions up to your Master. So events, things that happen on your Salt infrastructure get passed up to the Master for various purposes. Salt itself sends lots of events. So when a Minion starts up, when a new Minion tries to connect to the Master, there'll be an event. There are a whole variety of things that would cause an event to be sent up to the Master. You can also send your own events. So you can have your own application send an event to the Salt Master. There's a whole variety of things you can do with that. One of those things is a beacon.  So we have beacons that'll monitor your file system, that will monitor your SSH logins, a whole variety of things. And we'll be talking about a Twilio beacon that monitors a Twilio text message queue and then send an event on the event bus. Reactors and engines. So the reactor sits on your Salt Master and it sits there and listens to that event bus and allows you to do things, react to those events.  Maybe you can check for that.  And then the reactor sees that event, restart your application and moves on, logs it somewhere and you know, barely a hiccup. Just a simple example. And the engines are similar to that. They're a Python interface. So you can create your own little programs that will listen to that event bus and do things in Python. So there's a little more flexibility there in Python. Tonight we'll also use a Salt API. You can turn on a REST interface. It allows you to reach out to Salt from any other application and do things with that. So if you have an existing CMDB and this is an application that manages things on your network, now Salt's entire, all the possibilities of Salt are now available to your application. So we have a Python interface as well as this REST API and really makes for some really interesting use cases there. So again, we'll demo that a little bit today as well. So again, it's just like an overview. You have a Minion here. Beacons and engines run on the Minions, reactors, engines as well, and the API run on your Salt Master. Okay, now we're gonna do live demos. So, yes. All right. I had to rebuild the server last night to make sure everything works smoothly, so. So as you see here, what's that? Yeah, exactly. All right, so you probably all, I think after four years I wouldn't remember that. Okay, so I have a Master, I have one Minion that I'm controlling here and you must be probably familiar with it. We have this whole slew of modules that you can run. When I want Minions, I'm going to run disk.usage. We're gonna get that data back almost instantly. Yeah, do we have, actually I could probably increase the, better? All right. This is how we work at Salt anyways. Over in the engineer section, all the lights are off like this, so. So you can see, you can also get that data back in different formats. So say I want that in JSON, so I can ingest that with some other tool. There's the xm command output in JSON. You can get it in raw text, in the raw Python data structures that it comes back in. Everything installed is a data structure, so that allows you to do a whole variety of things, including managed output like this. You can also do command.run, will allow you to run any arbitrary command on your systems here. And that was one of the things that got me really excited about Salt originally when I first found out about it was just, I felt like I had this power. I mean, finally I could reach out and know what's going on on the servers on my infrastructure. What's this space looking like? There's this one server that we're constantly checking. We have monitoring, but what's going on right now? You can run these commands, get that data back, and do something with that really, really fast. In fact, the remote execution was built first. When Salt was first written, it was just all it was remote execution, and that kind of informs the rest of Salt. All the Salt states, all the event bus. Everything is based off that initial remote execution that Salt provides. This is a brand new server. So if we want to run a Salt state, just for those who haven't seen one before, let's do an Apache server here. Okay, so here we, this is a simple Salt state. Apache 2 is the ID declaration, which also will become the the name of the package we want to use. We have a package state and installed is the function that we're going to call. And you notice it's installed, so if the package is already there, it won't do anything, but if it's not, then it'll do the right thing. So let's install this thing. So we're on the state.sls, Apache server, and the amount of time it's going to take here is mostly just apt checking to see if the package is installed and then installing it here. So I know for a lot of you, this is like the very basics, but I just wanted to kind of give a quick overview of how this works. Yeah, it probably does. He mentioned it was probably doing an apt update as well. So that's a simple Salt state. Okay, there we go. It took 32 seconds, installed all the dependencies we can see here. Again, this is all a data structure, so I could have gotten this back in JSON, but you can see here we got the new Apache 2, installed all the dependencies and gave us information about that. Now at any time, I can run that again and it will do the same thing. It'll check to see if it's installed and it already has installed, so it's not going to do anything. I can also, here on the command line, just come here and do, and we'll do the package.perge. And when you're on the command line, I'm using the actual execution module and this is actually going to do this. It's going to remove that package. So there we go. We're done. A friend of mine that runs a bunch of the websites for the major newspapers in Salt Lake was on a train heading home when Hartbley was released a while back and you hopped onto the crappy Wi-Fi on the train, ran the little check using this to see which servers were vulnerable to it, got that list, got that list, ran the command to fix all of them and it was done in the whole process. It took six seconds to actually do. And seriously, that's one of the things, literally, that got my attention about Salt was the fast control that you had and the simplicity, yeah. Okay, so now I'm going to talk about Beacon. So a Beacon, like I mentioned before, is a little service you can start on the Minion that allows you to watch something. So I'm going to set up a Beacon that will watch a specific file for any changes and when it sees a change in that file, it'll send that event up to the master saying, hey, this file changed. And I'm going to set up another one that will look in a directory and notify any changes there. So if I come in here, so here we have our Beacon's definition. Let me use the I Notify Beacon. This does require the, I think it's a Python notify Python module to be installed. So any event, anything that happens to this serve slash serve slash blah dot txt file, Salt will send a message up to the master. For this slash serve slash testing directory, I'm just going to look for the close write event and send an event up to the master and we're going to recurse through any directories that are created. We're going to auto add any new files to it and then the interval says we're going to check every 10 seconds. Okay, so down here I'm going to start a little listener so you can see what's happening on the event bus. This is a great way to see what's going on when you're debugging. So let me go into slash serve directory. Here's that blah dot txt. So let's open this thing up and another line, close that. And then you can see right there, we've got a notification that, let me increase that size a little bit, that the file was modified, the change that happened and the path to the file that happened. Going to that testing directory. So let's create a new file. You can see here in that slash serve testing directory, we had our close right, another right, close. You can also see there the swap file that Vim created was also modified there. So what could you do with this? Let's say you had some little sys admin that just won't, or some developer will not stop logging into production server, start editing files and I was always getting in there and modifying things without going through your change management. You could watch your Etsy directory and anytime there was a change an event was to be sent to the master and you could clobber it and say, nope, re-do it. We had one more clients do the same thing with, yeah, you can send an alert through PagerDuty, announce it to the world. You can also check for like SSH logins and things like that. So someone logs into a production, like, you know, web fraud end that you should never do.  Do you have a question? Does spam have the company when somebody modified a file and they shouldn't be? Automated pink slip, I guess? I don't know. Krister, yeah, so all this does is it sends an event on the vent bus. So then you can use an engine or a reactor to do whatever you wanna do. So it could be anything. It could be add something to your CMDB. Yeah, exactly. So Krister was talking about his security module for file integrity. This could be part of that. You know that file shouldn't be changing at four in the morning and you can get a notification. Then you could log that however, whatever monitoring system you have, depending on the file, maybe do an alert if necessary. Just because you have visibility into that. There are a variety of modules. So if you come here to the beacons section. So we have a disk usage beacon. This one allows you to set thresholds for notifications to be sent. So you can say, hey, if you ever get about 85% on these specific file systems, send us a notification, that type of thing. By notify we saw here, journal D versus system D. Same thing with load, you can do your criteria when you wanna send a message. People sometimes will use that for load balancing, that type of thing. Certain server gets maxed out. You can get above so many, your load gets so high it leaves me to speed up another VM to help manage the load. Services or PS for services that are running, a whole variety of things. So WTMP is for your logins, that type of thing. So and we're gonna use a Twilio. Take my message today. And let's look at that code for these. You'll find, you'll see that these are extremely simple. So if I come down to this Twilio thing here, you'll see that we have some documentation, we have some imports, we have a validate function. And then the only requirement you have is that you have a beacon function. You can see here we have our documentation. But this is the entirety of, so that's 30 lines of code including logging and nice clean formatting to check for these text messages as well as a check to see if they have images and things connected to it. So if you had a beacon, you had something else, say you had an application that you wanted to monitor locally on your server, you very easily could create your own beacon to run your checks and send events as needed. Any questions on these beacons or anything? Sure, yeah. So the question was could you have a beacon that would watch a log file and then if there's a specific error, notifies, dislog or something like that, you definitely could do that. So here we have our Twilio account here. I think I exposed my account credentials in the previous one up on the file thing, so I'll change those before this gets on the internet, so. Yeah, I do. This is live streamed? Okay, well, we'll be changing that quickly. Again, send the interval. This WTMP again is for if I wanna check logins, so actually the first time I gave this talk I got a guy from the Apple in the audience, I didn't know he's from Apple and I forgot to take it off my server. It's like the next day and a half he kept getting text messages that, I had to think so, if whenever somebody logged in it would send a text message to a person and he texted me back, like hey, could you take me off that server? So those are beacons. And again, you have reactors and engines, that can do things with those events. So in the demo that I'm doing today, I'm not actually going to use a reactor or beacon, I'm going to have a little webpage that the Salt API is going to expose and the JavaScript is actually going to bind to the event stream and listen for events from my Twilio account and update the webpage when it happens. So just a quick view of this, friend and co-worker at Seth House actually wrote this in like four minutes for me. He's more of a JavaScript guru than I am. So here we're going to listen to the events on this event source and then when we're going to update and create a new module or new elements on the webpage. So let's see what that looks like. Sorry, I'm losing, should I be here? Okay, so here's our simple webpage. Also promoting SaltConf in April. You got to come out, it's going to be awesome. A few of you have been to SaltConf and it's a good time, a lot of learning and it's really a fun conference. And is that you? No, it wasn't you, it would be giggling. Oh, that was Charles in the back. So if you text that number up there, you'll see the text will show up here. Somebody wants to send a, you can send a picture text as well. Please keep it g-rated. This will be on the internet, but somebody wants to send a picture not on my zipper down please. So again, what's happening here is my server is running a little beacon and it's checking every five seconds that Twilio will count and when it sees a text, it just sends an event on the event bus and there's a multitude of things we could do there. You know, we could have reactors happening and you can do many things with each event that comes through. In this example here, we're just listening to that event stream and on the Twilio ones, oh that's a big one. Yeah. And we're just dumping these to the web page here. So obviously you wouldn't want to do this in production, but maybe you have a monitoring system. You have pager duty, you have a whole variety of things you might do to monitor things and you can listen to these events and do things with that. Any questions about, let's go so far, yeah. Good question. So all it did was added that config file for it and then we started the Minio service and it'll process all the beacon definitions and do that. Yeah, oh yeah. No, you easily could have that run. You can see that events, take that text of that event and use command.run from the event loop and do that. Yeah, so obviously all of the security precautions you would take accepting input from external sources you would want to do here as well. Yeah. But this kind of thing you could do, if you have like a chat ops type scenario, people want to deploy from Slack or from IRC or something like that, you can do that with Salt. You could also set it up so they could text something and deploy from Salt. We had a customer that had this window application that would frequently lock up and we worked with them with their first line tech support and it was a pretty simple thing to do but their first line tech support people weren't technical enough to actually log in and restart this thing without screwing things up. They kept escalating it to someone who could do it correctly but it was fairly frequent like it had a lot of customers that would call in and do it so we worked with them to add just this red button in their thing and so the first level tech support could log in at the customer's account and say, hey, my thing, whatever's not working and hit that button. And that bottom one there scared me when I was a kid, Dave. Yeah, so they just hit this button and the higher level tech support people never had to bother with that again. They would just hit that and they gave them the ability to take care of the customers without having to manually go in and fix their problems there. So I was gonna text from here as well. I usually send it from my phone but I get no bars in here at all. Screen line, yeah, there we go. Any other questions about this here? Is there anything anyone, yeah, go ahead. Sure, should we get to the microphone? Hey, okay. First of all, the message queuing model that Salt uses seems to be quite different from any of the other configuration management systems that I've seen and as I understand it scales extremely well. What kind of volume can you actually handle in terms of agents that are connecting into that message queuing system for simultaneous processing on large fields of devices? Really good question. So it's one of those things that kind of depends but we have people that are using up to, 18,000, close to 20,000 minions connected to one master. You can, there are some difficulties when you get to that scale. Sometimes just the flood of data that's flying back.  We have lower level masters that handle all the cryptography and communication with the minions themselves that then wrap up all the results and send them back up to the high level master that type of thing but places like LinkedIn, I mean they have, you know, every time I talk to them it's like they're bigger but like one point they had like 30,000 servers and I think like 100,000 servers now. Not necessarily all of them running off one salt master but a couple of different masters there. So I mean it's really common for people to run salt on like one or two, three servers and then we have a lot that are in the, you know, five to 8,000 range very comfortably. Cool. I'm also very interested in network device configuration management. I don't know if there's anything you can speak to about salt being used in that kind of a context. Sure. So we have some modules that will reach out to network devices and things. There's a somewhat new feature of salt called a proxy minion that runs on a regular minion and allows you to take like a say a network switch that you can't install the salt minion on or something like that. The Adobe we're working with them on a, I hope I can mention this, but on a module we're talking to some server hardware so they can configure the hardware and then bring up the servers and that type of thing. So this proxy minion allows you to, the proxy minion handles the communication with a device and then allows you to use all of salt's modules and things like that as if it were a server and that type of thing. It's like a small subset of things that make sense on that type of hardware. Yeah, yeah you're welcome. I can actually get that to work. Emoji we're actually breaking it because I had, I was trying to cast everything to a string and so it was actually breaking the, so I had to go in and hot patch that code. You're getting close to the line people. Go ahead. When I run a command again, it's all my minions, which I don't have that many, like maybe 60 or something. The first time, if it hasn't been run in like a while, the first time a couple of the minions fail to respond, but then like if I hit it again, everything will be fine and happy for the next while as I'm using it. Is that normal? Are they like sleeping minions? You have to kind of warm them up. Exactly. So what happens is, by default, every 24 hours, the salt master will rotate its encryption key that uses for the communication on the publish port. And if the minions haven't communicated with the master, anyway, so the next time when that happens, the minions have to, when they connect to the publish port, they can't read, they can't decrypt the pack, the load, and so they'll request a key refresh.  So there is an option, that's what happens, so you have to run like two commands to get them all working. There is an option to have the master do a test dot ping after the key refresh so that it automatically updates all the keys like that. But that's what's happening there. Also, is there any kind of testing framework for salt? So salt tests itself internally, because states inherently test themselves, to an extent, but we don't have an official framework yet. We're kind of like, working on ideas on how to do that well. There is a plugin for Test Kitchen that's, employee of Field Packard wrote, so you can do, use Test Kitchen, which is a Chef tool to use salt to test your states and build up a VM with all your states and then make sure that they're behaving appropriately. Some of the ideas we have with that is to be able to specify success, parameters right next to the actual states themselves. So as you build up your states, you can specify, I spent my web server, when I access port 80, I should receive this type of text, that type of thing. But it's still kind of in the design stage at this point. And I'm not sure, yeah, I'm not sure if that'll even be gotten to this year, so for the moment, that's all we have. Any other questions? Would you like to do? Okay, so you get immediate audience feedback. It's awesome. Yes, and Utah Dave is my IRC handle. I'm Utah underscore Dave on Twitter because there's a real estate agent that has Utah Dave. He actually yelled at me on Twitter once. Who's using my copyrighted nickname or whatever. I've been Utah Dave, yeah, thanks for that. I actually don't have any on this server. So a reactor is a simple YAML file that says on your master, and you specify a tag that you want to listen to. And you can use Jinja to look at pieces of the data structure of the events that are coming through and do things with it. Now, when you're creating these, this reactor loop needs to be really, really fast. So you do have access to Jinja, which means you have access to the salt modules and states and all that kind of thing. So technically, you could run salt commands and various things from there, but you do not want to do that because you want the reactor matching on these tags really, really fast. Because then when it matches a tag, it will spin off another process, another worker process, to actually run the commands you want. Okay. All right, so here is a reactor that was listening for the Twilio thing. Or actually, no, no, no. Actually, this is actually to send an SMS. So in the other demo that I did, in the other demo that I was doing where every time somebody would log into the server, it would send a message saying, it's not so logged in. This is what would happen. So this first thing here is the ID declaration. So this is just a name I made up for this. It could be anything you want. Local is, means I'm going to run an assault execution module, which is Twilio, send SMS. Target is going to be Boucher, which was the name of my server. So that's the name of the ID of the server I had. I had to pass in some arguments. My Twilio count was just the arbitrary name I'd given, the credentials. That's from a Twilio count. And then this thing here, the information from the event that was sent is inside this data structure that's inside of Jinja. So this double brace indicates we're inside of the Jinja environment. So I'm going to get the host name. So the host name logged into and then the ID of the server and then the timestamp. I'm wondering if I had that, that's wrong. Seems like it. Yeah, so, yeah. I haven't used this in like a year, so I'm not sure if that should work. But essentially, here I'm creating a text that we're going to send in that. Yeah, so I think this could also just be name. It depends on the data structure. So I would have to look at the data structure that gets sent when someone logs in. So, you know, who's just named, you know, for example. That's essentially the text that we're sending in the data. I'm sorry, in the text we're sending. And then this is the two. So that's my cell phone number. And then this is the from, this is my Twilio number. So what that means is this gets called whenever a specific tag is met. So if we come in here. So in your master config, I don't have one right now, so actually maybe I do. So in your master config, you would have something like this. So your reactor is the initial config section. And then you have a tag you're going to match on. So in this example, the tag is salt minion. And then we have star, which is an actual wild card. So in reality, when a minion must start up, you would see like minion 01 there, where that asterisk is, and then start. So every time the master sees that, it will call first this start.sls, and then this monitor sls. So in our example here, we would have a reactor and then we would look for the tag that came up for that login beacon. So you can also send your own events as well. So here's another example. When you get an event with a specific key, you would take that key, delete it, you know, with this execution module. So really, you have the full power style to do whatever you want. So you can run stage, you can do, you know, a whole variety of things. So people that use reactors to fix problems that have known solutions that happen regularly, you know, the application developer just won't fix that thing. You can have salt fix that for you. You can send events, so if something happens, you can have it fix that. They fire in order. So they start at the top and then move its way down. You can also do things with engines. I haven't actually had a chance to use engines yet, but they allow you to create a Python interface.  See, there was, oh, it's just something else. I forgot what it was. Okay, anyway, that's pretty much it. Yeah. Well, they're more complementary than either or. So a beacon is the thing that's actually checking your file system. And it generates an event. So it's dumb in that it doesn't know what's going to happen.  It just check this thing, check it, check it, check it, and then send an event when the conditions are correct. And then you would use a reactor or an engine to listen for those events and then do something. It's not what you kind of get a separation of concerns, because you may not want the beacon running on that server, you know, it could be in a semi-hospital environment or something, or, you know, you may not want it to even know what's going on. So it would send an event, and then the master would take care of doing something. Yeah, a master always runs a reactor. Actually, you know what? I think you can run reactors on minions now in standalone mode as well. But in the general case, the master runs the reactors, yeah. Great question. Well, that's pretty much the end of my talk. I'm going to stay and answer questions. Yes. So the question was, when would you use a reactor versus using an engine? I think you would do that when you need more programmatic control of everything that's going on. So when you're using a reactor, it's simple in that you just have, you know, the reactor listens for a tag and then does execute this list of SLS files, and then those SLS files are still essentially, you know, YAML with some GINJA. And, you know, for more straightforward things, it's pretty easy to go. Sometimes it can be a little obtuse trying to figure out the data structure of the event that came up, you know, and that can be a little tough.  So you kind of just, you get to decide all the flow of that, what's going on. And you get more programmatic control. Whereas, you know, if you, my experience has been, if you're doing a lot of GINJA templating and things like that, I like to keep it at a minimum. If you're doing a whole bunch of it, you're going to get into a big mess and give you a big headache. And so if you were doing complex things, I would look at using an engine. You know, and you'd probably get to define what complex means to you, but that's kind of where I would do that. Yeah. Sure. Great question. This question was, he mentioned that there's a lot of configuration management tools out there and what led to the creation of SALT and what are the main differences with SALT. So Tom Hatch was the founder of SALT. He worked for, first for a lot of large federal government agencies managing large, large infrastructures and also worked for a music startup that also had some large infrastructures. And there were a couple of things he just craved as the systems architect was, one, to know what's going on with the servers, you know. They would keep track of servers like in a CMDB or just in a spreadsheet and it was always out of date, right? You know, everything should have this version of the software but you go check in reality and things that happen. People have updated things, some things auto-updated. You never knew what the current status was. And so he really wanted to be able to have that remote execution. He wanted fast control over his servers. That was kind of some of his main desires there. And he had used basically all the tools that were out there at the time too. And so that's where SALT started was the really fast remote execution. And that's one of the things that SALT stands out with is it's really fast remote execution. It's not just bolted on after the fact. It's the core part of what SALT is. SALT is also very pragmatic. So it's very rare. In the past, the first few years of SALT stack as a company, I did a lot of the professional services, working with big infrastructures, you know, JPMorgan Chase, LinkedIn, various places. Well, I didn't work on LinkedIn's infrastructure but worked with training on them and things.   We should have to configure our servers and we're not changing anything. We're not touching it. You know, it's not going to happen. And then there's other teams using another tool. And so the team has their own homegrown, bash, Python, Perl behemoth that they've got going. And, you know, it's rare that you can just swap it all out. SALT makes it very easy to use it where it really makes sense for your infrastructure. It's very pluggable. So both from what SALT can do, you know, if SALT doesn't do something you want it to do, you can create a new module and state for it very, very easily. If you have, you know, especially for an execution module, it's very, very basic Python. People that never programmed before, people that are experiencing other languages, you know, the Ruby developers, you know, it's very straightforward to get involved with Python. But also, connecting to SALT. So we have a whole bunch of external authentication modules. Some people in this room here have written their own external authentication modules, pulling data, configuration data from their existing CMDBs. You know, you know, they have their own CMDB that they had written, and it's not going anywhere. That's the source of truth for a lot of what they do. And just wrote a module, and now SALT pulls data out of their CMDB, and it just fits right in. It's very easy to tie into everything that you already have existing on your infrastructure. We have puppet modules and chef modules that allow you to do chef solo runs and puppet runs manage it. Wikipedia does that. They use SALT to orchestrate all their code releases and all their puppet runs, but they have something like, and the last time I talked to them, they had like 30,000 lines of puppet manifests, and they just didn't have the time or the desire to swap all that out. So they use SALT to manage that, to scale their puppet stuff out. We have Ansible, what do they call it, a list of servers? Playlists? No. Pardon? Not the playbooks, but they're a list of servers they have that they run commands against. I'm trying to blank, but you can use SALT. SALT can use those to run commands on servers.  So there's something like 15 or 16 different pluggable interfaces that SALT lets you touch and add onto if you need. One of our big customers in San Diego, they're spending like $5 million a year on their previous configuration management software that they were using, and they swapped that out for SALT. And the fact that they could get to the source code and they've basically entirely revamped one of the modules. So everything else pretty much worked great for them. There's one module that just wasn't quite doing what they wanted. The fact that they could get on, do a pull request, and make it work right, and fix that bug, was just fantastic for them. They just, you know, they loved it. So they have the support of SALT Stack as a company, but their expertise that they have on staff, Sysadmins, and developers can go back into SALT itself. In fact, a lot of times, when you're running code for SALT, you're running code that, you know, experts that LinkedIn that are managing 100,000 servers, you know, a lot of the scalability that SALT has was based off of some of the larger users running into things of scale, because we don't have 100,000 bare-metal servers sitting around a closet there at SALT Stack or headquarters to test against, right? So a lot of the best practices and lessons learned from some of the top people in the world get all the experience, flows back into SALT. Like the reactors, so there's people here in the room that were working with them, and we were working on re-imaging, so they had a tool that could re-image a server and just completely, you know, put a whole new OS and change it from like an application server backwards or vice versa. So we're working on automating that. Then it occurred to us that like, well maybe, what happens if something goes crazy and suddenly we want, it's thinking it wants to re-image, you know, 400 database servers, you know, that could cause a major problem. So we added an, we added an acu-ing system. I'm not sure if we actually got that fully implemented there, but added an acu-ing system that would allow you to add things to a queue, so if there was like five that need to be re-imaged, okay that's normal, but if suddenly there was like 400, you could say, stop, and you know, do that in an orderly way. And so, I don't know, I could go on and on, but there's really, I've done professional services with some incredibly talented people, and I've never used a tool that was so flexible. There's like, I've hardly ever run into a situation where Salt just could not do something. That's correct, so even in our enterprise software, Salt itself is Apache 2.0, completely open source, and for a couple years we actually provided packages specifically for our enterprise customers that we supported directly, and we're moving to a framework instead that they'll basically use the same packages, but we're building a tool called RAS, and a web GUI associated with that that sits on top of your masters that will allow you to, there's really large scale to keep all that data and scale out to the tens and tens and tens of thousands of servers and view that at a, at a pretty massive scale. So that allows us to provide value for enterprise customers at these difficult, large scale situations, but also have, you know, everything else open source. So if you contribute code back to Salt, it is all Apache 2.0, so you're not going to, like, contribute code back to Salt, and suddenly it's only in the enterprise version. I mean, it's all open source, and it's available for use. That being said, we do now have support offerings for just open source, so, you know, if you have, you know, 500 servers, and you would like to have support, but you're just running straight open source, and we have a lot of tracks for that as well. So, yeah, for a long time we didn't require you to do enterprise to get support, but we don't at this point. Yeah, so the question was the difference between writing a formula and having everything be, like, in your top file or, I guess, in your file roots. So, Salt Formula, if you go, we have a GitHub organization that we keep all of our Salt formulas in, and there's, I don't know, I think we're into the hundreds, I think, now at this point. So, Salt Formula is kind of a generalized, kind of a best practice as a way of doing something. So, at this point, they're pretty much generally, they're all community supported. We do provide some support to that, but our community really is rallied around and worked on those.  And so, like, most of them straight out of the box will give you an installation of, let's say, MySQL or something, with some sane defaults. And then they're set up to allow you to pass in variables to configure that simply through, like, the pillar system, that type of thing. So anytime you have, like, a generalized thing, Salt formulas are a good way to look at that. So you have a, you know, MySQL expert on staff, and she has everything just tuned just right. Well, other people in your organization can use her Salt formula to do the right thing without having to be an expert with MySQL, for example. And so that would be kind of where, whereas maybe somebody else has, you know, a light H2PD state that they just use and so they would just stick that on their file roots on their master and, you know, no one else really cares about it. So they wouldn't necessarily make that like a public, or at least public within your organization. The good thing about using the ones on online as well that we have is that, again, you're using something that people have already gone through the pain of experience to get going. So, you know, obviously you need to test them to make sure they work according to your needs, but sometimes it's a good way to get started on something complicated without having to do it all yourself. That's essentially what a Salt formula is. Yeah. There are three out right now. Actually, no, there's there's four. They're all pretty good. So there's one, there's two by Joseph Hall who is one of the original four co-founders, or original engineers of Salt. One's like kind of getting started with Salt Basics, and then he just released another one that's more advanced use cases. I haven't read the second one yet, but that just releases Joseph Hall. I'm not sure what the name of the book is. Yeah, there's another great one that's written by Colton Myers. And then a former engineer from LinkedIn also wrote one as well. Drawing a blank on the name of that, but there's like three or four books out. Yeah, we have some Salt there. We haven't done a lot of video tutorials yet. We do provide Salt training both in our office. We can do it in your office. And we also do it remotely. So we have a remote video conferencing system that works really well for training as well. So those are really, it's actually a really great training. Our founder was actually a four red hat for a long time. And so is the training class. How do I find that? If you go to www.saltstat.com, we'll have some things on training. We have a guy on the back here that would love to talk to you about that as well. He's got that green jacket. And I know I'm very biased, but it's actually one of the better trainings I've ever been a part of. We do have everything Salt can do.  So you really come out with a great boost. In fact, when I was doing professional services, I would basically force them to do a training the first week we did stuff because it just helped for them to like know what we were doing because they had the training already. All right, I think we're out of time. I can stand up for a few minutes afterwards,