that's delivered. We compare hashes to make sure that it makes the journey intact and we only use the finest of all the hashes. So it's efficient. If it doesn't change and it's trying to be reinserted, it stops, it doesn't reinsert it. If console thinks that the watch needs to re-deliver it, it checks to see if it's the same file and it doesn't re-deliver it. It's optimized for safety because we don't wanna write a blank file and that, cause that's generally pretty bad to whatever service depends on that config file. And you can also run commands after the file is delivered. It's also super fast. Once it triggers from a console watch, it's under 500 milliseconds to deliver a 40 kilobyte file to 1,000 nodes. The stats are weighted most heavily under about 300 milliseconds but there's always the stragglers that make my histograms and heat maps look terrible. I originally tried to measure the entire process from start to finish based on syslog timing but everything was happening within a single second so that wasn't helpful. So I added some higher precision logging and inserting to the key value store takes usually about 100 milliseconds in under and that's on the left and delivery from the key value store usually takes about under 300 milliseconds in down very often much closer to a hundred. When it insert records into the key value store, it can throw out an event to Datadog so that we can audit what's happening. You can see on the bottom we added some Postgres node somewhere and on the top there's a Kafka node that's getting removed as well. These events can be shown on a timeline, they can be graphed, they can be measured, you name it. Again, because we're a metrics company, we emit a ton of metrics, the size of the file, the lines in the file. We don't want, again, zero line files are bad. When it's firing, how often it's firing, how long it takes, we also emit like other things like panic metrics, not long enough metrics, checks on mismatch, things like that. And I have a very, very quick demo. It should take just a couple of minutes. I originally tried to make a video of the demo from start to finish and unfortunately even for me it was super boring because it's just, it's kind of one of those things that happens in the background. So what I'm gonna do today is I'm just gonna show the, I'm gonna clear that and I'm gonna show an ad hoc usage of KV Express on our prod cluster, so don't tell anyone. So in addition to be able to use in watches to deliver things, you can also, sometimes you might need to deliver a config file to a bunch of nodes quickly, right? Because you're having an outage, you need to change something and Chef has busted or something. So we also have something called, we call it ad hoc usage where I set aside the temp key space in that program called console sort of the temp key space and we can use that to write. So I'm just gonna cat my demo script. And so here, I'm just gonna sudo. So this command is the in command. It says KV Express in. I'm feeding it a URL and it's just a 600 line config file I grabbed from a cell stack website this morning. And I'm gonna insert that key into the scale 14x key and in the temp prefix. And so what, when you do it with a URL, it automatically shows the output just to make sure that the thing that you put in is the right thing. And if we take a look here, KV LS Crap Timp, we can see it's there. So console KV. So there's the SHA-256 checksum for that, that text file that we grabbed. So what I'm gonna do here, I'm gonna take this file. And so this is a console exec. Again, it's one of those scary things that sometimes you get to use. And what we're doing is saying for the service data dog, run KV Express out, minus D is send out the metrics for the, again, looking at the right key in the right prefix. And we wanna write the temp slash scale 14x.conf. And then the dash E at the end is the running the command at the end of the process. So you might, if it's a config file for Apache, you might wanna do sudo service Apache restart or something like that. In this case, I'm gonna, I'm just gonna, well, correct the file name first so that it's not remaining there. And I'm gonna change the name because I think I did this a few times testing already today. So like I said before, if it's the same file and the exact same text, it won't do it. So that's gonna go across 1,100 nodes, almost 1,200 nodes, and it's done, right? So it's kind of fun. Before we used to do that with Capistrano, again, it would take, sometimes we would just end up stopping, like stop everything and do it slowly because it would take so long to do everything in parallel. So, here, wow. So that's KV Express. It's available there as soon as I can click open. So it'll be available quite shortly. And I think we have time, a little bit for some questions, so. Sir. Of the leaders, okay. So, encryption for secrets. We don't have any secrets in there at the moment because there's no good way to do it at the moment other than maybe using something like Vault. So, hopefully that now that this has calmed down and the presentation is done, now I can start working on Vault to do that. Now it's another hash score product. It ties in with console. It's probably a better way to do it. We looked at a few other ways. It just, there's nothing really good. Far as the horizontal scalability of the servers, like I said earlier, you make them a little bit bigger until the leadership transitions go away. Now, I don't see a problem with console 06. You could probably get away with three nodes in production and it depends on your use case. There are some people who have very little traffic, very little read traffic. We had a lot. And sometimes we were hitting the key value store 1,000 nodes with some of them with 10 proceeds at once. We're reading 100 keys. We're still doing that and it's still not breaking any more under console 06. So it's great. It's ready for production unless you have 10 or 20,000 nodes. I think at that point it starts to break down. That's your question? Okay, anyone else? Oh, in the back, sir. Well, we're not saving any money, that's for sure. Now with those five C, the C32XLs, we're not saving anything. The biggest problem was around that service discovery and it really was whenever we were trying to update any of the things in that environment file, it was taking two hours to get out. And that's too long. If we had a problem, first thing people would do was, okay, stop chef everybody, right? And we would stop chef and then we would roll things out manually just because we didn't have enough time and no one trusted chef. It chefs great. And we use a lot of it, but just in that 500 milliseconds versus 30 minutes or two hours, you had a lot more capabilities that you could do. Like I mentioned earlier, when we replaced our entire Kafka cluster, we wouldn't have been able to do that with Chef without a lot of pain. And it would have taken us a lot longer than I think it was the two or three days that it took us. So it just gives us the agility that we need to do things, especially when things are hitting the fan. That's right. Sorry, sorry. You there. I know, that's crazy. We wrote at the same time. Now, console template is great. We still actually use it to template stuff from the service discovery. The dedupe feature and now Armand had a talk on Tuesday and Armand's one of the original authors of console. It essentially does a similar thing. The only thing that is different now, we've integrated into our Chef workflow so that we use an LWRP that when it goes through Chef, it looks to see if there's a file there. If there's no file, it grabs it through KV Express and then it writes the watch automatically. So we don't have to do any of the setup on every node for all of these watches because it's handled through the LWRP. Essentially, they're both doing very, very similar things. And if that had been available in November when I was writing this, then we might not have done this, but it wasn't so. Sorry, sir. Oh yeah, we do all that. We have stale, we have all that stuff. We actually never found some of the consistency modes to make it a very big difference in our environment. I might have been doing it wrong. I know I do a lot of things wrong, but we never found that to be a very big difference for us. We still have it on, but it really doesn't make a difference. Yes, we have some of the KV requests that we make are in that mode, yes. You know what? I'd never got around to timing it at that time. It was more just get it to work because everything's burning, so yeah. Any, sir, over here? Sure, so your first question. So Consul comes with all the stuff with ZooKeeper. Consul comes with all the service discovery built in. So that parallel service discovery app that we wrote was based on ZooKeeper because we have a bunch of clusters because of Kafka. But it doesn't have the DNS interface. It doesn't have the services. Like it's not built. You have to kind of handcraft it all yourself. And we like ZooKeeper, it's pretty good. But for this, we also want it to be able to use HTTP to query it and not have to have a really thick client and that just isn't available for ZooKeeper as far as I'm aware. A second question was around. Well, part of that is we don't get to depend on any stuff that only AWS offers. And we're doing about five million metrics per second. We've increased five times over the last year. A lot of the things that Amazon comes out with right away, they can't handle the volume that we're at at the moment. So at this point, it would be nice, I think for some low volume stuff, I'm looking at using some Lambda stuff to throw it into Kinesis for things like logging and some metrics that aren't kind of our critical path. But Kafka doesn't work, we're screwed. So that's just kind of how it is. Sure, and it totally depends on your volume too. Like if you have a similar volume, then it might work. But again, for us, we couldn't at the time. Anyone else? Sorry, I saw you first, sorry. Yeah, you. No, sorry, that's 300 milliseconds from the time that the watch fires in the console on the local mode to the time that it's done writing the file and finished. So it's all in one data center, but that's, so I actually wanted to measure for this talk. I wanted to measure, so how long does it take to write and then what's the gap in between the two before things start, before they start reading? Only problem was with all the, with NTP, I couldn't get the sub microsecond precision. So it was, before it was even done writing sometimes, nodes were already starting. I wanna figure that out exactly how long it takes, but it's so fast as to be, I couldn't measure it yet. No, no, we're still in, we're just in, we're in three AZs in US East at the moment. We have hundreds and hundreds of terabytes of data in S3 and in all sorts of other things. So it's moving to multi data centers is pretty interesting for us. So, sorry, you. Yes. Yeah, that'd be a problem for this. For sure. We don't have that, we don't have a situation that we're still all on VMs and we have consistent ports. Containers aren't, haven't been an option because of the amount of data that we have. We have nothing, almost nothing that's stateless. So we can't use containers at the moment. And when we go down that route, we might have to go and overlay network or something. Maybe you would do like live network or something else that has like IP addresses per container. But otherwise you'd have to use SRV records, which console supports as well. I just, we just haven't got, we just haven't needed to yet. Yeah, I think so. We haven't, we haven't tried that yet. So, Brent, if there's another one, sir, over there. Yeah, our applications are not currently aware of SRV. They're not querying for that. We are registering with a small JSON file in each node based on the role from Chef. So when we have a Chef role, it writes a specific JSON file for that role only and the JSON file registers the service. Again, once the node gets killed, the service goes away. Is that an answer? Okay. Who else? Yes? Oh, one back there. And then, sorry, you, after that. Not yet. Just, like I said, I just finished upgrading two weeks ago. And we haven't, but we're looking to that because of the AZs and some of the weird stuff we've seen between them. So we're gonna try it, but we haven't yet. Sorry. No worries. No, I've, so on my Octahos project, which is basically a very tiny, terrible, not terrible, but like a very lightweight Heroku that does, none of the things that Heroku does around state. I register with an API call and point to those containers. So you can do it that way as well. We just chose to do it with these JSON files because of the stability of our nodes and Chef. But you can do it with API calls and it's no problem to, you know, say, here's the port, here's the IP and you're good. Did that answer your question? Yeah, you could. I know I do that, you know, I host probably 60 containers on one box and all the config is built out of console, service discovery catalog that's registered via an API call. So, oh, sorry. Sorry, I didn't totally understand that. We haven't integrated with it, but I know there is some movement around that. I know HashiCorp is building their own Docker image. And I know there is some integration. I just haven't used much of it yet. Sorry. Thanks for having me, everybody. And if you have any questions, let me know. Test, test. All right, it's all working. Well, thank you all very much for coming. I'm Parker Abercrombie and I'm a software engineer at NASA's Jet Propulsion Lab. Today I'm going to be talking about how NASA uses the cloud to enable virtual reality Mars. So the project that I'm going to be talking about is called OnSight. And OnSight was developed as a collaboration between JPL and Microsoft. And it allows scientists and engineers to work on Mars through the power of virtual reality. So the way this works is the user puts on a virtual reality headset, and we use the HoloLens device from Microsoft, and then they see Mars around them in their office and they can walk around and explore the scene as if they were actually there. So the reason that we do this beyond just as fun is that you can get a different sense of the nature and the scale of the Martian terrain by looking at it in an immersive 3D experience that you can get by looking at a 2D image on your computer screen, and then trying to reconstruct the 3D scene in your head. So we work, our users are scientists who work with Curiosity Mars rover, and they're mostly geologists. And when geologists study a place on Earth, they actually go out in the field and they walk around and look at the rocks. So we're trying to give them as close to that experience as possible for Mars using the technology that we have available today. So to make this experience possible, we needed to create this virtual reality scene of Mars, but the rover moves every day. And we wanted this tool to be useful operationally. So we actually needed to not just do this once, but we needed a way to create these scenes easily and automatically as the rover moves and new imagery is downlinked. So the onsite team built a custom image processing pipeline that takes stereo images from the Curiosity rover and builds a 3D reconstruction of the train around the rover. And then we build a cloud architecture to let us run this automatically in the cloud as soon as new data comes down. So this is a cloud talk today. I'm not gonna say much about virtual reality and I'm not gonna get into much detail about image processing, but I'm gonna talk about how we've used cloud computing and open source technology to enable us to run this process automatically when new image comes down and push that out to our users. So by the end of my talk, I hope you've learned a little bit about how we've used the cloud to solve our problems. Hopefully you'll get some ideas that might apply to your own problem domains. And I hope you'll be excited about space exploration. So the data that we work with comes from this instrument, Curiosity. Curiosity is a rover on Mars. She landed in August of 2012, so she's been on Mars for about three years now. And to give you a sense of scale, this thing is about the size of a small Jeep. So it's pretty big. Curiosity had a lot of different instruments on board. The ones that we work with primarily are the stereo cameras on the rover mast here. You can actually kind of see the two eyes of the stereo mast cam there. So as the rover drives, it takes these pictures and it sends back the pictures to Earth, and then we take them and we process them into this. So what we're looking at here is a 3D reconstruction of a scene of Mars along Curiosity's drive. And I want to point out that everything you're seeing here is real. This is all real imagery sent back by the rover. There's no artistic retouching, and this is produced completely automatically with no human in the loop. In the front here, you can actually see a little bit of the rover's shadow that was captured in the imagery because the sun was behind the rover when that picture was taken. If anyone follows the Mars Science Laboratory mission, this mountain coming up in the background is Mount Sharp. This is Curiosity's final destination. And right now, the rover is exploring dark dunes that will be coming into scene just in a second here. And there's this band of dark dunes at the base of Mount Sharp, and that's where the rover is today. So this is where we want to get. So we're going to take the stereo images and make a scene that looks like this. But before we do that, let's back up a step. So this is Mars as seen from orbit. The Curiosity rover is exploring Gale Crater, which is the yellow dot here. We'll zoom in a little bit. So the blue path here is Curiosity's traverse. So everywhere that you see the blue path is somewhere that the rover has driven or stopped and taken images. So what we're looking at here is Mars as seen from above from an orbiter. And that instrument is the Mars or Constance orbiter. It has a number of cameras and sensors on board that produce these orbital maps of Mars. And this imagery is about a quarter meter per pixel, which is fairly good as orbital imagery goes, but if you're going to put it in virtual reality and stand on it, a quarter meter pixel is pretty big. So as Curiosity drives, you take these images and then we take those and we build these three scenes. And we need to do this for basically everywhere along the path. So everywhere the rover stops and takes pictures we consider as seen and we'll build a reconstruction of that part of Mars. At some point we would like to link these all together into one kind of super scene of all of Mars, but we're a little bit of ways from that today. So I'm not going to get too deep into image processing in this talk, but I'll give you at least the high level version of what we do. So we take the stereo images and using a stereo correlation algorithm, we derive a range from the camera position to each point on the terrain. And from those ranges, we can create a point cloud. And from that point cloud, we can do a surface reconstruction that gives us a mesh geometry for the scene. And then we can take the images from the rover and paint those back on top of the mesh to get the fully textured mesh. And you can see a little bit of that process here where the background is kind of hard to see on the projector, but you see only the wire frame mesh geometry of the scene. And then as you get closer in, you can see how the mesh and the texture start to interact until you have the final product appear next to the rover's wheels. So when we're doing this, there's a couple of different types of imagery that we need to combine because we don't have high-res curvilary imagery of all of Mars, unfortunately, especially places where the rover is just arriving. So the main types of images that we work with are black and white, oops, black and white images from the rover's navigation cameras, which give fairly low-resolution grayscale imagery in color mass-cam images. So this is the high-resolution science camera. And where we don't have either of these types of imageries, then we'll fall back to orbital imagery from the Mars or Constance orbiter. So for every part of the mesh, we'll have some type of imagery and some is better than others. So there's an element of sensor fusion and how to stitch these things together in a way that you're using the best possible data for each part of the mesh. Sir, it's an in-house pipeline. So there's an element of sensor fusion in combining these things into a good-looking final product. And for the purposes of my talk today, I'm gonna treat that as kind of a magic black box where images come in, magic happens, and then this textured mesh comes out the other end. And then once we have that textured mesh, then we can load it onto the onsite software running on the HoloLens and look at it in virtual reality. So we've been running this pipeline for about a year now. We've processed several hundred scenes all along the rover's drive. The size of the scenes varies quite a bit depending on how much input imagery is available in a place that the rover has been exploring for a while where we have a lot of images. We might have several thousand. In a place that we just arrived, we may only have a handful of images. Typical scene is about a thousand images or about five gigabytes of data. And over the course of our processing, we'll crunch that into a couple hundred megabytes of mesh and that process takes a couple hours running on a single note. So when we started this project, we developed the software just on our development workstations. And we would do kind of the simple thing that you would expect. You'd expect your input files to be in one directory and you'd put your output files in a separate directory. And when we went to run it, we'd copy the appropriate input files to directory A, hit go, wait a couple hours and then grab the results from directory B and then put them where they actually needed to be. And that worked pretty well for development but it obviously doesn't scale very well when you move into operations. So that's what led us to port this into the cloud. So at the high level, we have a batch processing problem where data comes in, we need to detect when that data is available. We need to pull it down, run this long resource intensive task that produces some output. We need to put that output somewhere where our users can get to it. We want to be able to see what's happening as the system is doing all of this and start new builds and stop builds that are running, to see the status of what's currently running. And we have a bursty workload where we get downlink from Mars on a roughly daily basis. So downlink happens, we need to spring in action, grab the new imagers and build a scene and then we kind of go back to idle tell the next downlinks. We don't want a lot of expensive competing resources running while not doing anything and costing money. So we built a system that we're running in Amazon's web service cloud to solve this problem. And here are some of the Amazon services and open source technologies that we're using in the system. We use the Jenkins Continuous Integration System both to compile and run our image processing code and tests but also to actually run the image processing jobs. So we can treat them as if they were compilation jobs. We use Ansible to deploy and configure our Linux servers. We use the loopback Node.js framework to expose a data API on top of our database. And we use loopback AngularJS and Bootstrap to create a dashboard view of the system. And I'll go into a little more detail on how we're using all of these. So here's the high level schematic view of the system. Things really start on Mars where the curiosity sends data down to Earth that goes into the mission data system and it's cataloged in that system and that's where we pick it up. Now our system falls into kind of three main pieces. We have a build manager whose job is to find out when new day is available, index that data into our database and request that builds start. We have a build cluster that has a master node and a fleet of worker nodes that actually do the terrain builds. And we have a distribution system which stores the results and pushes them out to our anti-users. So first thing happens, data comes down from Mars to Earth. The build manager periodically pulls for new data and as soon as it finds some, it indexes that data into our database. Then it requests that the build cluster start a new build. The build master Jenkins will find a worker to run the build, creating a new one if necessary. That worker pulls the data that it needs to do its job from the mission data system, runs the job and then pushes the results out to RS3 bucket. Then it notifies the build manager that new build is completed. And then our users the next time they launch onsite will pull down the new data and see the latest location on Mars. So here's how Amazon's web services fall out in our infrastructure. Imagine most people in this audience are probably familiar with these but I'll very briefly introduce them in case anyone's not. In the middle column here, these are all EC2 instances. So these are all virtual computers running in the cloud. The build manager and Jenkins are kind of medium sized instances and these are long running. They're pretty much always available. The workers are the beefy machines that actually do most of the image processing. And these we treat as disposable resources.  We use the SQS, the simple queue service to communicate between the build cluster and the build manager and within the build cluster itself. So we have a couple of queues set up for different types of communication. We use the relational database service. This is Amazon's relational database as a service. It's a kind of a veneer over a relational database to store information about the available source data in completed terrain builds. We use S3 to store the results of our builds. And CloudFront is Amazon's content distribution network. So CloudFront takes the files that we put in S3 and then pushes them out to data centers that are geographically closer to our end users so that they can experience faster download times. So this being a Linux conference, I'll also talk about operating systems. Our build manager in Jenkins, we run in Ubuntu Linux. And our worker knows we actually run on Windows. So it's a heterogeneous system where we run both Linux and Windows machines. And to communicate between those machines, Jenkins helps us. Jenkins is pretty good at sending jobs out to the Windows machines. And for other data, our usual solution is to use a simple queue if it's a small amount of data or to push results up to S3 and then pull them down on the other end. As long as it's not a terrible huge amount of data and it's all within Amazon's data center, it's pretty easy and it works. All right, so I'm gonna go through each part of the system in a little more detail. And I'll start with the build manager. So the build manager's job is to discover when new data is available, orchestrate the terrain reconstruction jobs and present a dashboard view of the system. It also is kind of the interface between the rest of the system and the database. So it exposes a REST API. We used the loopback as a Node.js framework to build a REST API on top of our database and the build manager exposes that to the rest of the system. So nothing else in the system needs to know anything about the database. We also have a dashboard view that we built using AngularJS and Bootstrap. And I'll demo that now. So this is our build manager dashboard. So this is created with AngularJS and Bootstrap. At the top we have some summary statistics about the number of scenes we've built and the number of builds of those scenes. I can see the status of the cluster. The time I recorded this, there were two nodes running, three allocated and one build in the queue that hadn't been allocated to a node yet. Moving down, this chart here shows me the trend in time over the past month of different builds. As I mouse over, I can see a little bit of information about each of those builds. The table on the side shows me the status of the build queue. So when I recorded this, there were two builds running. You probably can't read that, but it gives me the time that they've been running. Moving down, I have a table that shows me recent build failures. I'm happy to say that there aren't any in the last two months. So this table tells me a little bit about each of these builds. I can click through to view the log and see why it failed. And we build a feature to acknowledge failures so that we can track if they were something that we had actually looked at and determined it was not a problem or fixed or something that still needed to be reviewed. So this next view shows me all of the builds that have completed in the system. So each row in the table represents a reconstruction of one place on Mars and gives me some information about that build, where it was centered, if it succeeded. I can enable or disable the build, which means if it looks like something broke, then I'll disable that so our users don't see it. Fortunately, that doesn't happen very often, but it's handy when it does happen. Builds can be deleted from the system. The table is searchable if I'm looking for a particular build. And we generate these preview products as part of the reconstruction process. So I can click into any of these and preview what actually was built and see how it looks without loading into a 3D viewer. In addition to the PNRM previews, we also generate these fly-through movies that show a little bit of the 3D reconstruction. In these video preview products, it's just a camera that pans around in the actual on-site experience you can walk around. We have imagery, so the question for the recording is what happens if you walk toward an area where you have no imagery? We have imagery everywhere, but it may be orbital imagery, so it'll just get low resolution. Okay, so to store the data about source images and completed builds, we use Amazon's relational database service. So the reason we use RDS is that we get automatic snapshots. We're already running in Amazon's cloud, so automatic snapshots that can be very easily restored is a nice thing to have. It doesn't really take any maintenance. As the back end of this, we use a MySQL database. We did consider using a NoSQL solution when we were setting up the system and decided not to because databases aren't really our problem in this application. The amount of data that we're tracking is relatively small and is pretty easily handled with a single traditional relational database. On top of our database layer, we built a REST API, as I mentioned earlier, using Loopback. So Loopback makes it easy to add the standard REST API and features on top of a data source and it plugs into different database back ends. So this gives us a little bit of database independence. If we do decide to switch out MySQL for different technology later, then the changes are isolated to just this one layer. So other parts of our system don't have to query tables in MySQL. They can just use these REST endpoints to get them post-train builds or search with filters. And Loopback lets you add some logic in JavaScript as hooks that run where your tables are updated so you can add business logic at that level. Another Amazon service that we use in the build manager is CloudWatch log management. So this is a way of pushing logs from your servers up into Amazon's cloud so you can see them and process them from the AWS console. So this is, I think this is nice because if something's going on with my server, I can log into the AWS console and look at logs instead of having SSH in that box and then trying to remember where the log file is. And once they're in AWS, AWS, they can be filtered, they can push to Elasticsearch services if you wanna do more in-depth analysis. You can set up alarms based on keywords that appear in your logs such as errors. So this has worked out pretty well for us. It's pretty easy to set up on Linux. It's just a service that you install on your machine, some scripts from Amazon. I know that it can be done on Windows. I've not done it myself on Windows, so can't speak to that. So the next piece of the system is the build cluster. So the build cluster has a masternode which is running Jenkins. Jenkins is a continuous integration system. It's similar to tools such as bamboo and cruise control. And the typical use case for Jenkins is that you would have it monitor your source control repo and when code is committed, Jenkins would pull that code, run a build, and then make the results of that build available. So it turns out compiler code is actually not that much different than reconstructing my mastern. You have some input images that you run some executable on and then it produces some output that you need to put somewhere. So we use Jenkins to both compile our code for continuous integration and also to actually run the image processing jobs. So what Jenkins gives us is the ability to manage a fleet of worker nodes, configure scripts that will, or jobs or scripts that will run on those nodes and then Jenkins handles keeping track of the build queue and parceling that work out to the nodes in the cluster. So this is the Jenkins interface. If you've used Jenkins before, I think this will look very familiar to you. Over on the side, we have the view of the build queue. So at the time I took this, there was one staging build in the queue. Down below we have the status of the build cluster. So the master node was running a job called manage nodes, which I'll talk about in a second. We have a couple nodes offline and then that third one is running, I think a production build. And there's a couple more down below. This table on the right shows the job descriptions in Jenkins. So each of these is basically kind of a script that does a certain thing. So the top one is build scene staging. So that does a staging build for a certain scene. The next one is build production scene. There's one that builds the preview products and a couple other miscellaneous ones. We also have Jenkins jobs configured that can create and destroy instances in EC2 and manage the fleet of worker nodes. So if I wanna see information about a particular build in Jenkins, I can click through to see details on that build. I can click to see the console output to either after the build is completed or while it's running, if I wanna see what stage in the process it's at. Jenkins tells me that this build was started 59 minutes ago and it's been executing on this host and it gives me an estimate of when the build will finish. And it gives me some information about what version of the code this was running. So this was running the production code at that certain get hash. So Jenkins out of the box gives you the ability to keep track of worker nodes and parcel work out to those nodes. What it doesn't do is it dynamically create and destroy the cloud instances based on workload. So we extended Jenkins with some custom scripts to make that happen. So the way that we do this is we have a periodic task called manage nodes that runs every couple of minutes on the Jenkins master and it looks at the size of the work queue and it looks at the size of the build cluster. And if those two are too far out of whack, it will try to equalize them. So if it sees that there's more work in the cluster than there are nodes available to do it, then it will spin up a new node and then add that node to the cluster.  So the system scales within some parameters that we control to adapt the amount of work that is actually needed at the time. So there's a little kind of Jenkins specific trick that we use in the system. So our main job is this kind of large monolithic image processing pipeline that takes a lot of resources and runs on beefy computers. So we only wanna run one instance of this task on a node at a time. So we don't want the Jenkins master to ever try to run two of these on the same node at the same time. But we do find it convenient in our Jenkins scripts to be able to call out to sub jobs that don't take a lot of resources. So the way that we set this up so the Jenkins master will do what we want is Jenkins has this concept of execution slots on each of your nodes. So you can allocate more slots to nodes that are more powerful. So we set all of our nodes to have seven execution slots and then we set our main heavy weight jobs to take up four of them, which leads three slots for lighter weight jobs that the main one will call out to. So that will make the Jenkins master never try to allocate two big jobs to the same node. Okay, to summarize our use of Jenkins, we have periodic tasks that run to manage the scale of the worker cluster and bring nodes online and offline, depending on how much work is actually happening at the time. We use groovy scripts to automate Jenkins and we also call out to some command line scripts that use the Jenkins Resta APIs to view the size of the build cluster and status of the system and submit jobs. And we use tags to mark different types of nodes. So we maintain separate development of production environments and we tag some of our workers as reserve for production and others as reserve for developments. That way we know that there'll always be production nodes available when we need them. All right, so the next part of the system are the worker nodes themselves. These are the nodes that actually run the image processing code. And these are a little bit of a different beast because these are GPU enabled EC2 instances that run Windows 2012. So our image processing pipeline was implemented in .NET and it runs on Windows. So when we made the move to the cloud, we just chose to keep it in that environment. So the way that we manage these is we take a stock Amazon machine image for Windows 2012 and then we install all of the dependent software that we need onto that image. So this is not actually our image processing code, but it's all the dependencies of that code. So it's the right version of .NET and the right version of the Jenkins jars and the right versions of the different image processing tools that we call out to and a handful of other things. And then we bake that into our own AMI. And as our needs change, we revision that AMI and update it. We expect that to happen infrequently because it's kind of a pain to re-bake it. So we try to only put things that we expect to rev fairly infrequently onto this machine image. So these are GPU enabled instances. And in Amazon's cloud, there's two offerings in the GPU line. So they're both in the G2 family and there's the XXL and the XXL images. So the XXL has one GPU and eight virtual CPUs, 15 gigs of RAM, and the XXL is basically four of those put together. So it has four GPUs and 32 CPUs and 30 gigs of RAM. And it's really kind of a beast of a machine. We use both of these for different things. So the life cycle of a worker is it comes online, it registers itself with the Jenkins master and then it waits around until new work comes in. When work is available, that node will first pull our code from get and build the version of the pipeline that it's going to run. Then it'll pull the source data that it needs from the mission data system. Then it will run the image processing pipeline and then assuming everything goes successfully, it will push the results to the onsite S3 bucket. And then it will notify the build manager that a new build is available. And at that point, it can go back to waiting for more work, at which point the Jenkins master will either give it another job to do or it will shut it down if there's no work. So in setting up the system, there were a couple of things that we learned. One thing that we like to do is to keep the workers basically as dumb as possible. So we created and destroyed these things all the time. And we do have to install a lot of software on them and it's a manual process to do that. So we bake as much as we can that we expect to not change frequently into the Amazon machine image. And then we try to keep all of the rest of our code in get or other places that the worker can pull from. So we try to keep the workers in a state where they can be brought up very quickly and pull down the resources they need. Baking things in the AMI is necessary for us to get the workers to spin up quickly, but it's a little bit of a pain to manage. And the other thing that we found with these machines is using the GPU on the cloud. It can be a little bit troublesome. We had to fiddle around with this quite a bit to make it work. We would write code on our desktops that would work fine and then it would not work fine on the cloud. We did get it to work though, so it does work. So there's another thing that we use. This is another of Amazon's offerings. So EC2 has two kinds of types of instances. On-demand instances are the normal ones. They pay for by the hour and you pay a fixed rate. So there's also a spot instance market. So spot instances are a little different in that you bid what you want to pay and then if there's excess capacity in the system and no one outbids you, then you get your instance for that price, which is great because you can get instances at much less than market value. But there's a catch, of course, and the catch is that your instance can be terminated at any time. But actually for a lot of work, that's okay. So obviously we don't use this for anything time sensitive or production critical. But for a lot of work, it's actually all right. So when we get it down like we need to build that new scene for the place the rover just arrived as quickly as possible.  But for when we get new imagery for whether the rover was a couple days ago or for when we're running builds in our staging or development environments, those aren't really that time critical. And if one of the instances gets terminated, we'll just try again in a couple hours. So the question is if a spot instance is terminated, can you restart from the middle or from the beginning? And the answer is yes, you can restart from the middle if you program it that way. In our case, we actually choose to keep things simple and just restart from the beginning. In practice, our instances don't get shut down that often. And having said that, there were actually two that were killed in the last day. But other than that, they usually don't get shut down that often. But what we use this for is we bid on the 8x large instances for the price of the 2x large instances. So usually we're able to get four times the processing power for the same price as the small instances. So spot instances have worked out very well for us. So now I'll get into storage and distribution. This is by far the simplest part of our system. And it's really as simple as we put our builds into S3, then we use CloudFront to push those results out to data centers that are geographically closer to our end users. We have users at partner institutions all around the country. We are at the moment restricted to North America. And then, so when the on-site software on the whole runs needs to load to terrain, it's simply HTTP calls to CloudFront URLs. And those are just cured with signed cookies. So I'll talk a little now about how we deploy our system. We use the Ansible tool. So this is an IT automation tool that probably many people are familiar with. It's similar to tools such as Chef, Puppet, and SaltStack. And what Ansible gives us is the ability to capture the configuration of our Linux machines and code that we can check into our version control system. So the goal of this infrastructure is I would like things to be set up so that I never have to SSH into my servers. I'd like to be able to just create them and destroy them and deploy them automatically from scripts in source control and never have to SSH and manually configure things. And I would be exaggerating if I say that that's always true, sometimes we cheat and we SSH in and manually configure things, but that's the goal. And Ansible gets us a lot closer to that goal. So Ansible organizes things into playbooks. It's a very powerful tool, and actually the session in this room later today I think is about deploying web apps through Ansible. So if anyone's interested in Ansible, check that out. So I'll show a couple snippets of the playbook that we used to deploy our Jenkins instance. So these are written in YAML. So the first step is we need to create a new user named Jenkins, we'll run that command as sudo. And then we need to ensure that that user has the SSH directory with proper permissions. So we'll create a directory home slash SSH, again using sudo with proper mode. And then we need to copy some configuration files up. So we can point Ansible at the config file template and where we want to put it on the remote node and the mode we want. Ansible provides pretty good tools for templating these config files and then swapping out variables for different things. So we have a set of environment we can deploy for production and environment we can deploy for staging and development. And we've separated these two with these variable substitutions. So when it comes time to actually deploy a new one of these instances, it's as simple as creating a new EC2 instance, which we can do either through the AWS console or through an Ansible playbook. And then pointing Ansible at that instance with the dash i flag and then the script to run and the environment. And then this final argument is the vault password file. So the way that Ansible deals with secret keys and credentials that need to be deployed to the remote machine is that it stores them in a file called the vault, which is encrypted and stored with your project. And when you run Ansible, you need to provide the password to unlock the vault so that it can pull out the pieces that it needs. One time when I gave this talk at JPL, almost immediately after our talk or my talk, we accidentally terminated our build manager instance. It was complete user error on our part. So I can speak from experience that this actually really is as simple as these two commands to recreate that instance. Because if you check it into your source control system, you don't want unencrypted passwords and keys available. Also on that occasion, we've learned about EC2 termination protection instance, which is a good thing to turn on for your critical nodes. All right, so we use Ansible to configure all of our Linux machines. We're not yet using it for Windows. I would like to be in the near future. We haven't had time to get around to setting this up. What I would like to do is I described for our worker images, we manually configure a machine image that serves as the base of our worker node. I would like to be able to build those with Ansible instead of doing it manually because any manual process is error prone, especially if I'm doing it. So my goal for the near future is to use Ansible to automatically provision everything we need onto Amazon machine and then bake that into the base machine image. Okay, so to summarize a bit, things start in Mars. Curiosity rover captures new imagery. That imagery is downlinked to the mission data system on Earth. The build manager periodically pulls that system for new data. When new data is available, it's indexed into a database running MySQL on Amazon's relational database system. The build manager requests that the Jenkins master allocate a new node to do the work of running that terrain reconstruction. Jenkins will find a node to do the work, spinning up a new one if necessary. That node pulls the source files that it needs from the mission data system, runs the terrain reconstruction, and then pushes the results out to the onset S3 bucket and notifies the build manager that a new build is available. At that point, sir, that we use for a train bill? Oh, I'm afraid I don't know off the top of my head. That's a good question. The way that we run the pipeline right now is we actually run it on a single node at a time. So one job runs on one node. The reason that we've done that is we chose to optimize vertically before we scale that horizontally. So for this phase of the project, our time targets were to build terrain for scene in about the three hour regime, which we were able to hit by vertically optimizing and pushing some parts of the pipeline onto the GPU. So to keep things simple, we've kept things for the moment running on a single node. That said, I do not expect that to be the case going forward. That's correct. Yeah, yeah, welcome. Okay, so once the results are in the build manager and in the onsite S3 bucket, the next time our users launch the onsite software running on the HoloLens, they'll pull down the latest terrain and see the latest place on Mars next to where the rover is currently. Okay, so what I've described here is very much a snapshot of a system as we've built it now or as we've built it and as it's running now, I'm not gonna say that it's the best possible solution to this problem and we do expect to improve this going forward. One thing we'd like to work more on is improving our auto-scaling ability. So I described how we use Jenkins to manage the size of our worker cluster and scale it up and down based on the side of the queue. Now it may occur to you that most cloud providers offer kind of similar things that also can create mistransitions based on some kind of workload. We made the conscious decision not to use basic scaling features from a cloud provider because when we were designing the system we wanted to maintain some level of cloud independence. So we ended up rolling some of this stuff ourselves. And it's worked out pretty well for us but having gone through this, I think we would actually rather not be writing and maintaining that code ourselves. So we'll probably in the future be looking to see if we can swap out some of our auto-scaling logic with something that's just provided by a cloud service provider. As I mentioned before, I wanna start using Ansible to manage the Windows Worker AMIs more closely. That's one of the kind of the fiddley bits, manual bits of the deployment process right now that I don't like. And to your question, sir, right now the image processing pipeline is this monolithic thing that goes all the way from stereo images from the rover all the way to a 3D terrain reconstruction. Now obviously there's a lot that happens between these two end points and we'd like to split that up into a little bit more modular services that do different things and can be scaled out more easily because going forward, we will need to exceed the three hour threshold by a lot, especially looking toward 2020 when the Mars 2020 rover will land. So I do expect to do a lot more of smaller services that are scaled out horizontally, probably all still using the GPU and then we'll need to look into how to distribute the source images to the different workers that need particularly those images. Right now we're in a kind of easy parallelization world where each worker can just pull all of the stuff that it needs and it works out fine. Okay, I'd like to acknowledge the great work of all of my colleagues on the on-site team and our partners at Microsoft. I hope you've enjoyed this talk. I hope you've learned something. If you have any questions for me during the conference, feel free to grab me or to say hi in the hallway. If you have questions afterward, feel free to email me at parker.apicromby at jpl.nasa.gov. If you'd like more information about the on-site project, please see our website, opslab.jpl.nasa.gov. And if you have any feedback about this talk, I would love to hear it. I'd love to hear what you liked and what you didn't like at the Google forum at the bottom. With that, we have some time for questions. Sir. Mm-hmm. Okay, yeah. So the question is why are we using Windows for the worker instances? To be honest, that decision was made before I joined the project, so I don't have a completely historically valid answer for you. In general, though, at the time the decision was made, they felt that was what fit the problem best. Sir. So the question is why can't we just throw more workers at it to reduce the three-hour window? And yes, we can, and that's probably what we will do. We will need to rewrite parts of our pipeline to run in parallel on multiple workers. So up until now, we focused on, by design, we run on a single machine and we focused on making it run as fast as possible on that single machine. We do parallelize extensively on cores on one machine, but we made the intentional choice not to scale out to multiple machines yet. But going forward, I think we'll do exactly what you have in mind. There's one up here and then. Okay, so the first question was, how does CloudFront scale, I believe, even geographically? Yeah, so I'd recommend looking at the CloudFront documentation on Amazon for its worldwide reach within the continental US, which is our region of interest. It has very adequate coverage. And the second question was, what kind of GPU problems did we run into? The challenge we ran into is that on our windows machines, for some reason, the GPUs sometimes weren't recognized, possibly due to misconfiguration on our part, or I'm not 100% sure, I'm actually talking to some engineers at Amazon next week about exactly what was happening. There was a question back here. Yes, we are. To be honest, I haven't looked into a lot of detail on this. One open source tool that caught my attention recently that I have not evaluated, so I can't really speak to is Spinnaker, which is an open source project from Netflix. I saw an interesting presentation by some of their engineers last week, and we'll be looking more into that. Yeah, okay, the question is, is there anything in the open source community, open CV and other image processing libraries like that can do to improve our situation? And that's a deep question, and the short answer is yes. My hope would actually be that some of the innovations that we've made in this pipeline can be pushed out into the open source community. I don't have the details enough in my head right now to speak to that in much depth. I'd be happy to chat with you offline though, in the back for archiving. Our strategy at the moment is to keep things in Amazon's data system. Archival hasn't been a big focus of ours yet, but probably moving forward, we'll want to sync up with other parts of the Mars mission and leverage whatever technologies they're using. I'm afraid I don't know off the top of my head how they do it. In terms of run, I'm afraid I can't answer that. Oh, okay, the question is, are each of the scenes that we build complies entirely of, I think you mean, new imagery from the most recent downlink, as opposed to older imagery from earlier in the mission? And the answer is the latter. So what we will do is for a scene, we'll find all of the images that are taken in that region of Mars. So that includes what was most recently downlinked and anything that might have been downlinked in the past. So in theory, the rover could actually drive in a circle and you would pick up data from actually much later in the mission. This has actually happened a couple times in the mission. Yeah, so deciding which pieces to use, I showed a slide earlier that showed color imagery and black and white imagery next to each other coming from different places. So this is actually a very deep question. How do you decide if you're doing a mesh reconstruction and you have a set of images and for any given part of that mesh, multiple images saw that place. So which one are you gonna use? We have some heuristics that optimize this for our use case. There is a lot of different ways that you can do this. To give you a couple of examples, we kind of pretend that Mars is a static thing where nothing ever changes and this is not true. This is a lie. The most obvious thing that changes is this huge Jeep-sized rover runs over it.  But you could imagine using your heuristics of which imagery you're gonna choose to skew the reconstruction to capture the state before the rover drove or after the rover drove, depending on what you're interested in or any number of other things. In this application, no, we did not have such a requirement. In the other parts of the Mars Science Laboratory data system, there are images available to the public that can be searched by different things. I can't speak to exactly which technologies they're using. And this application was to finish on your point. We limited the amount of metadata that we tracked to really just the bare minimum of things that we need to know about. And thus far the searching has been within the scale that we can do within a relational database. I think a question over here. The question is, is the final data product available to the public? The answer right now is no, but there are some thoughts in that area that unfortunately I can't share right now. But stay tuned. Not that I know of when the decision was made, Amazon was what we chose. Yes, I think that is possible. I think the more interesting question is, can you send the data from a data system directly to the HoloLens? Getting the data from Mars to Earth is actually a whole other question. And maybe not surprisingly, the bandwidth between Mars and Earth is kind of limited. Yeah, sir. So the question is, how soon do our users want the data? I think to paraphrase. And the answer is now. It's actually a very subtle question that you're asking. So the way that Mars planning works is there are certain planning meetings that are time sensitive for planning rover operations. The way that those meetings line up with how data comes down from Mars is very complicated. Ideally, our terrain would always be ready at the beginning of the meetings. That's not always the case. So in some cases we may get it down like in the middle of the night and after the rest of the night to do the work, in which case no one is calling us. And in some cases we may get down like half an hour before it's supposed to be ready, in which case it won't be ready. So the answer is that the users would like the data as fast as possible, and we were trying to meet that need. Scouter. The question is, is there a way to view the data without using the HoloLens? In onsite today, no, it is a HoloLens only application. However, I don't expect that to stay that way forever. In fact, yeah, not for long. Sir. Well, that's what we aim to make it. Okay, that's a great question. And that really speaks to why did we even do this? Because we have these 2D images already, so why even bother with this 3D reconstruction? And yes, the reason that we do this is that I think there is something perceptually different in looking at images in 3D and versus 2D. Ideally, you would actually send a scientist to Mars and they would look at real looks for real. We can't do that. So the next best thing is we can put a scientist in virtual reality. And using the HoloLens, you're not tied to a different sort of place, you're tracked, so you can move around the scene and walk around it as if you were there. Clearly it's not quite as good as being there, but you can use some of the spatial cues that you use in your everyday life on Earth to understand the scene around you. And at the end of the day, geologists are looking at rocks. They're trying to understand how big are the rocks? How are the rocks laid out in relation to the scene around them? And it's very hard to do this in a 2D image. And there was some work before this project started that motivated the whole thing that tried to quantify how people understand a scene that they see in a panorama image versus a scene that they see in virtual reality.  So that's the whole motivation for building this application that lets you look at Mars immersively. Skylar. So the question is, I think you're asking if you walked from one scene to another along the rover's path, would you switch from orbital into the new scene? That's something that we would like to support at some point, and unfortunately we don't today. Well, conveniently we have a rover on Mars. So the question is, why did we choose Mars as opposed to other planets? And I'm sorry to be a little bit snarky there. We have much better data on the surface of Mars than we have for other planets. If you think back to the orbital maps that I showed at the beginning of my talk, there's a lot that you can see from orbit, and Mars is absolutely beautiful when seen from orbit, as are other planets and moons, but for a virtual reality experience where you're trying to put someone on the surface, orbital data just doesn't really get you there. It's just not high enough resolution. So where we have really good surface data is Mars, where we have robots that collect this for us. At the moment, we don't have such data for other places. Earth, yeah, okay, Earth is a counter example. I'm not sure who was first, man. Okay, the question is, when in the process of planning the Mars mission did this idea of creating a virtual reality tool come up? I'm not entirely sure, to be honest. I've been at JPL for a year and a half, so I was not at JPL when the curiosity landed. I know that the ideals of looking at Mars and virtual reality have been around for years and years. This particular project has been going on for about two years and this group has been doing work in this area for much longer than that, but when the germ of the idea that is now on-site was born, I'm afraid I don't know. Okay, the question is there one for the moon? We are not doing one for the moon today. I don't know if we will or not. I think you were first, sir. Sure, so the question is, is there a reason that we chose an augmented reality technology, such as the HoloLens versus a virtual reality technology such as Oculus? One thing that we get from augmented reality is that we wanted our users to be able to continue using the tools that they're used to while they're in their virtual world. So I didn't really go into how on-site works in user interface, but we actually detect where your computer screen is and then we cut that out from Mars.  So the reason we do that is you can continue using the tools that you usually use to work on data from Mars while you're looking at Mars in virtual reality. Sure. Yes, so the question is how accurately are the spatial details maintained when you're going from 2D to a 3D reconstruction? This is a deep question. So I'm not going to get too deep into it. We aim for the kind of centimeter-ish level accuracy in our mesh. There's a lot of ways that noise can come into the system and in fact the source images that we use, we get range products out of those and just by the nature of how stereo correlation works, the quality of that range falls off as you get further from the rover. Fortunately, usually the things we're interested in are near the rover, so that's not too much of a problem for us. One of our next steps actually is going to be doing more quantitative analysis of our mesh versus the input images. So the question is, how specific is this technology to curiosity and is it possible to process data from other missions? I assume you were talking about perhaps a spirit or opportunity or other Mars rovers. In general, this is not specific to curiosity. This is something that can be applied to any kind of stereo reconstruction process. Currently we are a little over-fit to curiosity. We have talked about supporting other Mars missions. Right now we don't have any plans to do so because it will take a bit of development effort to make it happen. Schaller. The question is, do we use location data about where the rover is to inform the image stitching process and the answer is yes, we do. I think that's a little more detailed than I want to get into today. I'd be happy to chat with you offline. Okay, maybe one more question. Well, we'll see. Okay, well thank you all very much. Is it working? About now? Is it muffled or anything? Okay, you'll get the high notes. Awesome. Okay, my name is Rami Alvani. I'm a, oops, that came before. Is it auto advancing? Sorry, that was a bad one. Okay, so let's start again. My name is Rami Alvani. I'm an engineer at Symmetric Corporation here at Culver City. And today I'll be talking about one-click deployment of cloud applications using Ansible. To begin with, of course you have my details there. If you want to reach me on social media or the best social media ever, IRC, you can find me there. But I want to start a small experiment. I always try to pitch to companies that are not usually, or at least parts of companies that are not usually involved in open source to try to support open source a little bit more. Since my manager is here, actually. Rami, can you say hi? Since my manager is here, I thought if you enjoy this talk or you tweet about this talk or you like something about it, when you post to social media, would you mind using the hashtag scale OCD as in one-click deployment? And if you don't like something about this talk, you can meet Rami after the meeting or call more in support. Both of them will help you out. Semantic owns more in just in case you didn't get that job. Okay, so currently I'm at a DevOps position. I'm my team at Semantic. And the reason I'm here, or one of the reasons I'm here is I believe in what the DevOps dude said. He said at DevOps, we are awesome. So we take the awesome that's in our head and give it to people who don't have that awesome in their head. Because one important part of being a DevOps person is sharing. It's one of the core values of DevOps. And this is one of the things I'm doing here, okay? And I actually found out who that person was. His name is Jason Freeman and he works for Stackstorm right now. So, and my slides are auto advancing and I don't know why. So this is gonna be a little bit difficult. So I work on a product that's called Semantic Unified In-Point Protection. And yes, it is soup. So, that's supposed to be a joke. So, thank you, I appreciate it, thank you. So, and it's a wonderful product. Actually, if you saw, come on, this is. Sorry, I'm gonna exit the normal slide thing and use this, okay. I hope you guys don't mind. So, the product that I work for and this is the last you're gonna hear about Semantic in this regard, basically manages cloud applications. It's a brand new product that Semantic grew from the ground up to manage end points that the company manages through different products, okay? And like the gentleman who was presenting him from JPL app before me, he actually has a Semantic endpoint installed on his box. That's why you saw the Semantic logo there. And this is the product that will end up managing all those. I also identify as a PhD student of computer science at USC. Can I get a fight on from the Trojans in the run? Awesome. So, I also identify there. I also attempted to reboot the USC Linux users group. So, no, you don't get to, you should have said fight on. And we have some good work there. So, one of the major things before we go to road click is one other thing I identify myself as is a husband and a father. Then I have my family at the end of the room there. So, thank you for letting me come to the scale. So, on the road to one click, you need to know and understand where you're coming from in order to know where you're going, okay? Because every organization has some sort of system in the way they deploy and orchestrate their systems. I don't believe anyone here is still doing it by hand if they have more than three or four boxes, okay? And even if you're doing that, you're not cool if you don't have the configuration management system.   So, your goal is to actually sit down, have a monitor in front of you and start looking at everything as it goes along. So, the actual idea is you have everything pegged out, all your code, all your infrastructure ready and all you need to do is just watch it go through and see the green okays and everything is nice go forward. That's, you can say the engineering fantasy or what you're looking forward to. So, who knows what this is? Yes? Excuse me? That's Morpheus, where is Morpheus? He's in the construct, right? Now, this is where one click starts, okay? One click starts in the equivalent, in the cloud equivalent of the construct. Basically, a big one with nothing. You have no cables, you have no networks, you have no routers, no switches, not even power. Well, cloud doesn't need power, so let's forget that on the side, okay? But you don't have anything and that's the true nature of one click because if you see that everything I have is gonna work like it is or it can stand up an environment which is a big deal in a lot of places, stand up an environment from scratch just using code that can be triggered only by one command. That's where you need to start from. You need to start from the construct. But in reality, even the construct is not completely empty, right? You still have a TV that you can use to see the matrix. Now, does the TV show the matrix in the green squiggly characters? Mindful color. Does anyone know why? Ah, the nerds in the room. Why? They can see it in full color and not as they see it in the screens in the nambucca nether. Okay, that's homework. So, regardless of what happens, you still need some components that you cannot always start from one click, okay? You can incorporate their own code that you can use to stand up at any point of time, okay? But still, they need to be independent from your infrastructure as you go along. So basically what you're asking for is to go from this to this to this. Well, this is a gif that should be working actually, okay? Where all the guns are coming in the background. To this, where, whoa, I'm up. And actually the first time or first two times you do it, people will be four a.m. in the office looking at each other, wow, it actually works. I can't believe it, okay? And it's a good feeling. And it's a feeling of engineering triumph. And we all at DevOps, we actually need to have that. Can you hear me really well? So, one of the things, I used to be a teacher at USC. I used to TA a little bit, so I'm used to talking to people. What I'm not used to is not people talking to me back. So I would love to have this more of a discussion kind of thing and we can speed up and slow down as we go along depending on how the time is. But more value will come out of this talk if we start, if you engage with me. So feel free to talk back to me and ask questions. So we're gonna go through some steps that will help take you through the one click process and ending in some things that you would like. So, first of all, you need to understand where you are deploying. One of the main things that you need to know is in terms of the sizing of your tenants. Now, some of the mistakes that I've been seeing talking to people in meetups and things like that, all your environments are bundled in one account or they're not segregated logically from each other. They're actually at some point under the same network that are not even subnetted away from each other. So that's something that you really need to keep in mind. Also, you need to maintain access control to your environment. So you may have some people who can see the VM list of VMs and routers and switches, but they can edit them. But you may have engineers that you don't want even to go close to that. There are engineers that you want them to be able to see logs for production, but not the productions VM themselves, but not, and you want a combination of all those. So this is another thing that you need to keep in mind when you do your own click, because as you configure your environment, you need to make sure that these access controls are in place and not because the code base or the current state of the system doesn't support it. You are forced to give SSH keys or passwords to people that technically you should not give to them, okay? Also, there are some resources that need to be taken care of. Like for example, package managers and code repos and code repositories. Are they shared in both environments? Is the same package repo that you use in prod is the same one in stage, or is it the same one in development? That's something you need to consider because mistakes can happen and will happen, and we need to take care of that. Also, you need to make sure that you segregate your private and public networks. Now, I'll be adding extra emphasis on some private networking and some of the aspects that may not be standard for the usual deployments that people have, just because those are some of the things that at Symantec we emphasize, making sure that everything is isolated two or three degrees of checks and balances for anything that we do. So it may be a little bit too tight for your taste, but at least you can go back and make it easier from there. Does that make any sense? Any questions so far? One question, please? Yes, no? I'm gonna start pointing out, yeah. I need to build servers. How you do what? Well, the build servers need to push to the package repositories, right? So if everybody shares the same, you need to make sure that you don't have development build servers having access to your production package repositories just because they happen to be on the same network. And when you whitelisted the firewall, you just gave the slash 24, for example, in order for all of them to access. So the main thing is you need to outline all those as you go along and document them. That way, the initial documentation is gonna be on paper. But as you get more and more into your DevOps work, into your configuration management, that will be actually documented in code. And what will happen is, especially if you're using Ansible, we're gonna come to that. It's YAML, so you can technically write code to generate the documentation from your configuration itself. And tools like Ansible Tower, et cetera, can help with that. Thank you for the question. Also, there are some intricacies for infrastructure that you need to keep an eye out for. And I'm gonna point out some of them. Hardware versus software accelerators. For example, you wanna ask, I have SSL termination, for example. Provider comes to you, say, I have a box with SSL termination. Well, is that box a software one or a hardware one? If it's a hardware one, where do you keep your keys when you reboot the box? Do you have an HSM? Where do you keep your keys just in case? What kind of ciphers does it support? Does it have all the PCI-approved packages, et cetera?  Is there an F5 sitting up front? Are there two F5s sitting up front? Now, not everyone in your DevOps team needs to know that information, but one or two key engineers, when they start debugging something catastrophic, you actually need an understanding at that layer to make sure that you got what you're, that to be able to understand. Also, some intricacies. Are my security groups or firewalls all the stateful or stateless? Do you all know what the difference between stateful and stateless firewalls? Okay, so there are some people who don't. The idea is, a firewall, you block traffic coming in, right? Okay, and you can do it going out, but let's take traffic coming in. Traffic coming in, you either block or you allow. If you're blocked, you're blocked, it's done. But let's say you allow the block from within your company, for some reason, a whole block and somebody keeps calling. If it's a stateless firewall, anyone who calls in will get in, okay? Even if someone is sending a sin flood your way, as long as they're whitelisted and they're open to come in, a stateless firewall will let the sin flood come in. Whereas a stateful firewall will understand that there was no negotiation, so I'm just gonna drop that packet because there is no state for this connection between these two hosts. Make sense? So, small things, especially if you're not going with your standard AWS or your standard workspace some of these things actually come into play. Also, the definition of availability zones, okay? If you go, I work with OpenStack primarily. If you go to multiple OpenStack providers, you see some of them define their availability zone as their whole data center, okay? So technically if your database and it's failover are in the same data center, they're considered one availability zone per the SLA. So if both of them goes down, you're expected to have one in another availability zone. Whereas other providers have each rock as its own availability zone. So per the SLA, if you have them in two different locks, they're good. Maybe not per your own confidence, but still. Others have it as a hypervisor, which is scary, but at least you have a clear definition of that and you know it when you're going in to design your system. I'm gonna try to speed up a little bit so we have time for the rest of the info. So I'm gonna pick TTLs for load balancers. Who uses RabbitMQ or message queuing service? For example, in order to maintain a state between RabbitMQ and some other devices, you have a device, RabbitMQ cluster. There is a connection between one node and RabbitMQ, okay? And that connection stays up until a new connection is made and goes to another node. Now this node may only know the address of that exact RabbitMQ. So if it boots up and that node is down, it doesn't know how to reach a cluster. So how do you load balancers? You can use load balancers to do that. However, RabbitMQ has strict policies on what's the TTL for that load balancer. So if your cloud provider does not support it, you'll be debugging RabbitMQ for like three days and you don't know why until you figure out that it was actually the TTL that's the load balancer not enforced. So some things like this happen quite a bit. So in this talk, we're gonna talk about how we deploy on top of OpenStack. So I think I'll give a five minute preview of OpenStack in general. So OpenStack basically is your open source infrastructure for cloud virtualization. You have your compute nodes, your networking nodes and your storage nodes. And then you have an interface that you use to interface with all of those. The way OpenStack works, they have different projects. Each project is in charge of a specific component of OpenStack. And as a person dealing with your infrastructure, you need to understand what each component does and how to deal with it. That said, not every OpenStack provider provides each of these components, okay? Now, do you guys mind if I walk down here? I really like to move when I speak. So Nova is the compute layer. So you do all the provisioning, et cetera, through Nova. Swift is for object store. It's one of those things that get dropped if you're in a private cloud and you don't request it. Glance is the imaging as a service. So it's your image management, uploading images, downloading images, et cetera. Keystone is for identity. And it's one of the cornerstones of OpenStack. And it's the bane of my existence after one other one. After Neutron, Neutron is software defined networks. And it is the... Anyone from OpenStack Foundation here or anything? It is the messiest part of OpenStack. It drives me nuts, okay? Because you have software components, hardware components, software components for anyone. Other software components, it's just crazy. And Horizon here, it's actually a dashboard you use to modify and do a portal basically for OpenStack. Any questions? Okay. So the next step after understanding infrastructure is making sure you understand your application. And there are certain things that you need to keep an eye out in your application. So the application that we're working on right now, there's a Spring has a known Hello World sort of application that's called PetClink, okay? And this is the one that I'm using for this exercise here. So you're here, you're gonna be talking to this example to an engine X node that's acting as a proxy. And it will do a proxy pass to all the traffic to Tomcat which has the application deployed on it that talks to MariaDB in the backend, okay? We have some supporting cast in terms of Git and the second most thing I hate, Jenkins after Perforce, of course. Jenkins and also the package manager, okay? So this is the example that we will be following going on through this exercise. So one of the, so as you start your application, your application is mostly still in development and your developers are developing on i7 machine, i7 hardware CPUs with at least 32 gigs of RAM. So if the application does anything wrong, they really don't see it. And it's really hard for them to profile it before they actually send it down to a profiler or even discover it until they deploy it on a VM that's yay big compared to their computer, right? So as you start deploying your application, you'll notice that you need to account for your CPU and RAM requirements and document them, okay?  How big do I really need to make it, okay? And this will come in handy. How much object store, how many block storage do you need? Also, and I emphasize this is an unusual scenario, but your VM egress and your application egress requirements. You need to make sure you account for anybody calling into any VM or container that you have, okay? You need to account for everyone calling in. You need to make sure you allow them explicitly to call in, okay? And your application in general calls out to different locations, okay? And you need to make sure you account for all of those. I went through the exercise where we had this application running completely in a lab environment, okay? And we took it to an open stack instance that was completely firewalled. And egress access is only allowed by permission. So if I wanna go to Google.com, okay, they're not a competitor. So if I wanna go to Google.com from a VM, I actually need to file a firewall change request in order to access Google.com, okay? So in order, of course, you can just do a VPN to another node that actually has access, but nobody knows about that one.  But better yet, environment is environment. You can open up tickets. You'll see your own CM code failing, okay? Why? Because this specific node package required to go to that weird repository instead of GitHub that you got white listed for two minutes. Oh, then I need to actually clone that locally so and change the configuration. That way, node will pull it from there as opposed to pulling it from GitHub, for example. Or there's a trend going nowadays for a little bit more complex open source software where you say, oh, just curl this URL, pipe, bash, set up, and hey, it works. That never works if you're sitting down the firewall. And it drives me nuts not knowing what's installed on my machine. I actually, what I ended up doing, I installed CoreOS and put in a huge Docker container and installed the application and did a diff and sat down to see everything in order to be able to deploy the application. Stackstone, if you're watching, that's you. Okay. Yes. So that's actually the first thing that I did when I was diagnosing that issue. What I did is I set up Squid Proxy on an environment that has complete egress access, full access to the outside. I set up Squid Proxy and I deployed my whole application and I sent all my logs to Elasticsearch and then I started combing through them, got a lot of grip and regex and I got the main set of URLs that I need access to. Okay. Then there's the, you want to hear about intricacies. There's one small intricacy, okay. There's a certain set of firewalls that does not allow you to whitelist URLs. They only look at up by IP. Where is that a problem? Any guesses? SSL, that's a problem. Yes, but mostly I'm lagging, I'm calming something outside. Anyone who works with mobile devices develop apps for any reason, okay. Your phone talks to Google and any service you interact with needs to talk to Google, okay. On the back end to make sure that you can push notifications, you can push profiles that goes down to the devices. Good luck tracking Google IPs. You can't do that.  Give me the whole thing. That will never happen. Okay. So you need to work out some tricks. So you want to know a trick? How I dealt with that one? Hands. So what I did actually, I have a job in the same environment polling the same URLs, okay. And whenever a new IP comes in, spits it out, sends an alert, then we follow a request for it. All IPv4 in our case. We're all private networking. So IPv6 doesn't really make sense. And if you have a partner engineer who became a Java engineer and you want to explain how to ping local host using IPv6, it's going to take some time. So we kind of switch to that. Okay. So going back here, also bootstrapping requirements. You learn a lot about the application if you go into a clean arm environment. How did I ever bootstrap Caltrace? Where did that user come from? Oh, only because that deep match the SSL certificate? That's how they got it and we couldn't deploy it anymore. So there are certain intricacies and lessons learned.com on. And I'm looking at it now because I'm bored. It's because I want to make sure we give enough time for other components of the talk. Okay. Also one important thing, you need to know where you stand. In terms of we're all mature dev ops engineers or aspiring dev ops engineers or hoping we can be dev ops engineers, you need to know the tool set that you have. Like a lot of people, we were invested in Puppet for quite a bit of time. Let's, I love the Puppet guys. Actually last scale, we sponsored Puppet labs at scale. And I'm so proud of that. But you need to know your tool and you need to know its limitations. Okay. For those who are using Puppet, one of the main issues that we had with Puppet is the construct. Okay. You want a tool that can bring up itself. A chicken and egg kind of problem. So guys, how can I bring up an environment where you need a Puppet master? How do I bring up a Puppet master? A Puppet master. Or you do it by hand. That's one thing. Also it actually Puppet doesn't, at least at the time and until recently does not have support to provision resources in OpenStack natively. The support that they have is repurposed EC2 costs. So if I actually wanna provision to an OpenStack using Puppet, that won't work for me. So you need to understand the limitations. And some of the things that you, that you need to look into also decisions that you need to make about environment. Are you gonna use this environment as pet or as cattle? If a node misbehaves, are you gonna just pet it? Please come back to health. I'll patch you. I'll do whatever you want. Just come back again. Are you just gonna kick it to the curb and, well, Peter's not here, but at least put it on the side and try to, and just bring up another one that's healthy. Okay. Also you need to decide what layers of tolerance do you have for that misbehavior. Do you say, okay, if it's just a misconfiguration, property files went away? Maybe the last Puppet one or Ansible one didn't catch something? Will I throw it away? That may be more costly than just deploying your property files. So you need to have some sort of measure there on what to do, as opposed to operational failure, that you just totally kick it out and bring something else. Also, you need to make a key decision, especially if you're going to Ansible about your machine state. One of the main powers of Puppet is that it ensures that a specific node is in the state you expect it to be. So even if a work developer went in and changed something, if your Puppet code is right, five minutes later, 30 minutes later, that node is gonna be back the way it was before. Now, if that has value to you, then yes, you may think you may Puppet, you may go Chef, you may go other resources, that will work for you. Ansible may not, it could be, but it may not be, it may miss some of the components that you really like. If that thing is important. For example, in our team, we adapted that if it's bad, kick it to the curb mentality. So machine state is not really that big of a problem. And if you have enough sensors in that let you know what the machine state is at a certain point of time, you can put in the code to actually auto heal as you go along. Also, you need to make sure that if you're using Puppet or any language that has a DSL in it, you need to make sure that the more qualified people are writing code for it. Otherwise, you're gonna end up with Puppet code, and almost everyone I talk to who uses Puppet are even one of camps. They either have a dedicated team doing their Puppet code and they're extremely happy and they even have Puppet Enterprise, and I am with it, okay? Or on the other hand, everyone in the company is writing Puppet code and the guy who actually needs to run it is going crazy. Okay, all kind of weirdness in the code and everything at the end is an exec. Okay, to a shell script. So if you're really into code or coders as shippers, then you may need to revaluate the tool that you're working in. Also, no, you don't necessarily need a Dumber Deployment Tool. You can have a combination of both. Like a lot of shops have a Puppet and Ansible Deployment where they manage their machine state using Puppet but do the orchestration using Ansible. Well, then you need to start putting the same qualities you have for your code, for your DevOps code. Testing, code reviews, making sure no bad code goes in and at the end you start phasing things out and that's what we're doing actually at this point because you can tolerate it up to a certain point but afterwards it's like, I have bigger fish to fry, I cannot deal with this anymore. But if you're in an organization that they can write that code, I think it's harder for you to do that unless you're in a place of authority which usually equal developers don't have that within a team. You should do it regardless but you'll find people who I'm reviewing the code that's next to me or Oscar comes to me and say, hey, I'm sending a code review, can you approve? It's not even in my inbox yet. Okay, so things like this happen and it's the way things work. Something is really broken, it needs to be fixed but at the end you're gonna incur so much engineering debt that at some point you're gonna just need to invest in cutting the code. Does that make sense? And then we can discuss it more offline so we can go along with the presentation. Okay, also this is, if you take anything, anything out of this presentation, I hope you take a lot, but if you take anything, you need to take inter potency. It is the golden rule of demos. And it is so important that any tool you pick, if you achieve a cut out of potency coverage of at least 80%, you're in really great shape. Okay, what does that potency mean? It means I have my code, I run it once, it'll deploy my whole system A to Z. Okay, I run it twice, if nothing is wrong, it'll come back, nothing is wrong. If I run it three times and a node was down, it's gonna run, fix that node and finish the run and report that one node back, okay? So much headache and grief can go that way. And that's the fastest way where you can achieve this new buzzword auto healing, okay?  The problem with that, okay, that shell script itself could be at your point. I could be making an API call that is mundane.  If you run an exit script on a curl call, every time that curl call runs, it is considered a change in the state of that node. So you will get our report back saying there is a change in state. That is bad. I should not, and I repeat, I should not get a notification of a change of state unless something actually changed. Does that make sense? So part of your effort, part of your work will be to make sure that even if there's a shell script, and by the way, this is something Ansible shines in. Because you can write a bash script that if the bash script understands that nothing happened, it can spit back nothing happens and Ansible will return nothing happened. So in that case, the actual call will be fine. But in that case, it's not a shell script, it's an Ansible module. Make sense? Okay. Any questions? Do I need to start pointing? For questions? You know I'm gonna call you out. Okay. So one important thing to know also is when to shut up. I know a lot of you here are actually here for Ansible. So now's the time we switch there. So before, so let's Ansible. So if you ask me what Ansible is doing and what it is, as far as I'm concerned, it comes, it ranks like in the top 10 of the best thing after sliced bread. At least in my case. Why in my case? Because my responsibility is to provision a complete environment from construct to guns, lots of guns. That's a quote from The Matrix, okay. Complete provisioning of everything, setting up everything, calling out to external services, reporting back that anything goes down, okay. I happen to use OpenStack. And there's amazing support for OpenStack and Ansible. It's not just because Monty Taylor, one of the Ansible, one of the OpenStack board members is in charge of the Ansible code for OpenStack. Has nothing to do with that. But actually there's a huge community behind it, okay. And if you go, puppet takes the idea of you get anything you need from the forge. Ansible is the complete opposite. Everything you need is baked in. So they have two repositories. They have a core modules and an extra modules repository. And people just contribute code there, like crazy. So anything, especially for a cloud site, anything you need is there. The orchestration component of Ansible is very mature. So what does Ansible do? People say a lot of things, but for me these are the three big things. The first one is cloud provisioning. If you're in my case, you're using OpenStack. You can have in one file, okay, the credentials and details for four OpenStack providers in one file, it's called cloud.m. You can have four OpenStack providers and you can run the same code base once for each provider and you'll get the same outcome in all four. Okay, I did it for two and it works great. I didn't test it with four, but with two and it works.  So it does that. Infrastructure has a service orchestration, so adding a load balancer when you need one, removing a load balancer when you need one. Like, I'll give you an example. We have this case that we call the magic packet case, okay, where we have this client that calls to a load balancer, okay. This load balancer is an HA proxy in a active passive mode, okay. So you have an active HA proxy that load balances the nodes and then you have a passive one that if this one goes down, this one will come up and a new one will be provisioned somewhere else and it will become the passive one. So far so good? Okay, so what happens is there's this, as far as we can tell, and as far as Juniper Network says, there are SDM provider, there's this magic packet that comes in and will go to this load balancer, it crashes the HA proxy. This one comes back to pick up the slot, packs our buffer, the same packet goes to this one, it crashes by the time the next one is up and it crashes by the time the next one is up and it crashes and we have this cascading failure of load balancers, okay. All we needed to do is to provision another load balancer and change the DNS entry from that load balancer to this load balancer, since we were using local DNS on the same hardware, we are golden, okay. Make sense? So things like this is really easy and fast to deal with Ansible for multiple reasons, one of them being how easy it is to pick up as opposed to other tools. So the way it works is, anyone has a pointer by any chance? I'm really used to pointing and stuff, okay. So what happens is you have an inventory. Everything in Ansible is based on an inventory, okay. I have these sets of nodes and each node has certain properties, so I say give me all web servers, it'll give me all, oh, thank you so much. Great, appreciate it. So you have the inventory file here and any VM you own is part of that inventory, okay. And you can tag your files in INI format, your VMs, any way you want. So if you say all my Phoenix two nodes, you can list 20 and you can have all my web server nodes and it can have five of those 20, okay. So when you query Ansible to do something, it'll just pick one of those, whatever group you give it to deal with. And also playbooks. And if I have one wish, we stop calling things recipes, playbooks, scripts, whatever hats and knives, et cetera, just pick a terminology. We should write out a name for everything in computer science, except configuration management. So I would love to change those, but I'm partial to playbooks so we'll go with them. So you give an inventory and a certain playbook, which is a set of tasks to Ansible to run. And Ansible will figure out where to run it on. It'll SSH to those boxes. Download its mini small agent, put it on that box. That box has, does not need anything, except Python 2.4, by the way, to be installed on it. And then that whole thing will run. Come back and return what happened. That's it. Now, if you happen to have an evil shell script, it'll take that shell script there, it'll run it, and it'll come back telling you that it changed regardless of what happened. Questions? Okay. It sounds rosy, trust me it is, until it's not. So, how do you install Ansible? Let's say you have an existing system right now, and you want to use Ansible to run it. You have VMs provision. If there are Linux boxes, and there are more than CentOS 6, which most likely is the case, or if it's the last LTS for Ubuntu, you have a Python that's big enough to handle Ansible, okay? All you need to do is clone the latest version of Ansible if you're so inclined. Or of course, you can use your package manager or pip to install it. Just make sure from now on, that you're using Ansible 2.0 or more. Because there are major differences between the previous Ansible and Ansible 2.0, okay?  Now, if you want to deal with OpenStack, you need to add these lines here. Sorry, Ansible needs these on your local box, and these are for OpenStack, okay? OpenStack, basically you need the Nova client, Neutron client, Keystone client, and OpenStack client. And Shade. Shade is actually the Python library that all the calls that Ansible makes to OpenStack are translated from Ansible-ish to OpenStack-ish using Shade. Yes. Yes, it is agentless in the sense that you don't need to install an agent on any box. However, the lead functions, when you tell it, go install nginx on those boxes. It'll SSH there, create a temporary directory under your user, install its agent, which is basically a Python shell, that will run everything that it needs to run, and then it will clean up after itself and come back. The agent, oh, you just got it from here. It's part of the Ansible code base, okay? And I looked at Ansible code base, it's actually beautiful. If you're not very good in Python, and you wanna look at good Python code, in my opinion, I'm not sure are other people, but this is actually some good code to look at. And you can make sense out of things. There are some intricacies here and there, and we're gonna see some of them in the code samples, but it's some good code. Now, small hint, if you ever have a problem when you SSH, especially if you're using a Bastion box, you need to install this small package that's called SSH Pass, okay? Just a small hint, if you want into an SSH problem, just install this and we'll fix it. I have no idea why, but it does. Though, don't get it from Source, it's hosted on Sourceforge, and they have so many ads and stuff on their website, so just any version from your package manager will suffice. And yes?  Because MIT is too liberal for you, right? No, no, it's not for security reasoning. It's for, okay, when you use a Bastion host, the Bastion host, all the SSH work is offloaded to the Bastion host, like a jump box. So, instead of you as a switching to the end server, you need to SSH this server, which can SSH over. And you need your SSH agent to follow through. There are components of SSH Pass that you need for that. Okay? If you need your password, not your key. I can reproduce the problem. Don't get me started. Can I trash Symantec? No? We'll talk offline. And by the way, ladies and gentlemen, Professor Ted Faber, he's my open source mentor, so that's why he's giving me a hard time. Yeah, I'm gonna blame him, and you can blame Scale on him. He's actually called the Godfather of Scale. He was there when it all started. So, thank you. Round of applause, everyone, thank you. That'll get you to stop talking. You're gonna ask questions. Okay, everybody else. Okay, so this is when we're gonna go to code. Yes. Okay, so think of it this way. And I'll give you another example afterwards. But this one is, so as part of securing your own network, okay, not all your boxes need to be accessible from the outside. So what you do is you create a private network. Think of it like your home network, okay? You have your own router that's set up. All your computers that are inside your home network are using a 192.168.something. That's something IP address, right? But you cannot SSH that box directly. But let's say you need to manage if you're a user inclined, and I'm sure a lot of people are. You want SSH to your home box from Office. So how do you do that? You either put your box in a DMZ zone, or you have a box inside your network that has a public IP address, okay? And you SSH to that box, and that box and only that box has, think of it as two network cards, one network card attached to the outside world, another one attached to your inside network, and you can talk to your inside network from it. So that's where you use a bastion box. So especially if you're using open stack or private network, you have the luxury of having none of your boxes accessible from the outside, which is almost a requirement, okay? And you have one box that you can use to jump to all those other boxes. Yes. You usually have, what happens is, like in our case for example, we have one box in each tenant. So if we have a specific project, we have one box in each. That's, and that's called the overlay network because that's the virtualized software defined network. Then you have actually physical boxes on the underlay network, okay? That are controlled, that are almost bare metal, basically. And you allow those to calling, okay? And you can also whitelist an external box as long as it has an IP address that's publicly accessible and known.  So apparently you only have 10 minutes to go through the meat of the talk and you can blame your non-professor favor, okay? So, okay, and I officially hate this. Okay, so I put up this QR code before the talk started in hope that people will actually, I can, you know what? I can use, just in, do you mind coming on my computer, please? Okay, so this is the code base where we have the examples for here. So the first thing that I want you to see is, can you please go to the Ansible configuration file? Okay, now let me bring up a case and tell me what problem you see coming there. I have, I'm provisioning brand new boxes, boxes I never talked to before and I'm gonna SSH to them, give me one problem I'm gonna face. The user's provision as part of the image. Host key, correct. So how do I deal with that? Yes, that's the easy way of doing it. However, in the production environment, you might wanna actually have the machines call back with their host key so you store it locally. That way you don't, you keep the keys as you expect. So, host key is checking, you just set it to false, okay? Also, you might wanna allow the forwarding agent. What is the forwarding agent? Let's say you're using a jump box like we talked to about before.  I don't own that jump box. Joe Schmoe owns it. I don't wanna put a public and private key that are attached to me on that box because Joe Schmoe has pseudo access and he can read my private key and he can practically impersonate me, right? So I shouldn't be doing that. So what I do, I actually load an agent and that agent forwards my credentials with me as I jump through boxes. Make sense? So these are things that make my life easy as I go along. Okay, let's go back. And if we scroll down, let's go to clouds.default.yaml. The one under, yeah, that's why you're up there and I'm down here. Yes. Okay, scroll down please. So this is the, if you're dealing with OpenStack or even you're dealing with AWS, you need to define your credentials and your details somewhere. So this is called your clouds.yaml file where you can define an n number of clouds in this file and remember it's yaml based. So indentation, like Python indentation does matter, okay? And you really said if it's a private cloud or a public cloud, if there's a region prefix, so if you're in Virginia one, Virginia two and Keystone is the authentication details and the username, agent Smith, the password, I hit oracles and all these details, you actually get them from OpenStack itself.  Make sense? And the code is on GitHub so we can follow along. So let's go one step back. So the first thing I wanna do is go to the construct. So the goal of the construct is to basically build my network infrastructure, okay? And pay attention with me here because we're actually going through some of the Ansible basics as we go along. So first things first, we said Ansible is based on inventory, right? But in order to create VMs, I'm calling an external server. I'm not calling a server that already exists. Should I put that server as part of my inventory? No, I just say use local host. I'm just making API calls. I'm not SSHing anywhere, right? So you can actually have modules that do not need to run on boxes. They just need to send alerts. They just need to make an API call. They just need to make something. Even email someone, okay? And you just set the host that it runs on a local host, okay? And you set the connection to local because the default it was SSH to your local host if you don't tell it. So you set it to local and gather facts. Everybody familiar with Factor in Puppet? That's basically a script that you call and gives you everything that could be known about your box, okay? And that's how Puppet uses it to keep track of everything. So you're telling it, don't waste the two seconds you need to collect information. I don't care about the box that I'm, I don't care about the box information right now. Now, Ansible has something called pre-tasks and tasks. It's simple. You have a set of tasks that you wanna run, okay? It goes one by one, predictable. Anyone tried to do one by one in Puppet? Have you ever succeeded? I don't think so, okay? You can, okay, I know you can but you need to be an engine, okay? So sometimes you wanna run tasks but in order to get tasks you need to have some information collected beforehand. And that's what we do in pre-tasks. So in this case, I said I have a file that's called project.yaml, okay? That I need you to import all the information from. So would you mind duplicating the tab? Go one step back and go to group vars directory. Here, yes. And project.yaml, okay, scroll down. So here I'm putting some information that are variables. This is where I keep my variables basically. The whole idea is I wanna keep my playbook as generic as possible. So if I run it on different environments, all I need to do is change the variables, okay? So I define, I'm calling my project the matrix. I give it the user credentials. I also provided the SSH key. Why do I need to give it my SSH key? I'm SSH'ing into boxes. So I need to give it my bubbly key, otherwise I need to put my password everywhere, okay? By the way, hint, hint, if you're using Ansible, all your boxes should have SSH. You should have your keys installed on all the boxes. And you should have password list pseudo, okay? And the most common playbook that people write is the playbook that SSH to all the boxes and installs your key. There's actually a module for it. There's a question? Yes, very, very good question. So Ansible Tower, which is the product that the Ansible company says has an SSH key management system in it. In our case, we have a hybrid SSH for full production. We actually have a routine that goes in and deletes all the keys and even deletes all the users because we use LDAP to authenticate. So if you're gonna use something in production, provision a key on the fly and plug it in for you. And it's taken out after a while, so nobody stays in production for longer than they need. Okay, so there are different ways to do it, but if it's not production, you can be a little bit lenient about it. There are projects that you can use to do it, but I really didn't find one that I liked other than vault.io, but I didn't have a chance to vet it properly. Okay, so also I wanna define some networks. Application needs some networks, right? So I'm gonna define a network that I'm gonna call the jump network and in OpenStack, each network needs to have a subnet because the network can have multiple subnets. So I'm saying, great, this network that has this subnet and this is the IP address range that I want that network to have. And I go down for five networks. Would you mind going down there? And also I wanna define a router that I can use to connect all these networks together. So far, so good. And basically these variables define everything. So let's go to the previous tab. And so the first thing you do in here, I'm assuming the most vanilla application you could have. You don't even have your key uploaded. So what you do here, the way Ansible works is you call a task, you start with a tick here and you give it a name, this is a comment basically. So when you do an Ansible run, this is the comment that's gonna come out. This is a module. Notice that the name and the module are in the same indentation level.  So keep that in mind. Then we indent a little and these are the variables that this module takes. It takes a cloud because remember the clouds.yaml file that we had? It needs to know which cloud to pull out of that file because you can have 10. So you need to tell it which one. And I, because I'm smart, I kept it in a variable file. And I wanna say present because you can say absent and you can go in and delete a key from being there. Okay? And then I give it a name because I need to use it later and I gave it the public key file. Makes sense? By the way guys, I kept my key there so if you wanna put it on any machine you want, feel free to do so. Okay, then I wanna start creating the networks. Now we're doing a little bit more in the advanced Ansible-ishness, okay? And two minutes I promise I'll stop. So this is basically what I'm doing is I'm doing the loop. Okay? So same thing, I have a module, I have the properties of each module but I'm looping over project networks because remember I have project networks and I didn't have a set of networks under it and then each network had its own properties. Okay? So I said loop over all the networks and from each item in that you're looping over give me the net name. Network name. Fair enough? Let's scroll down. This is creating the subnets, okay? And it's the same thing, I'm looping over it and but I'm using more variables. So forgive me the net subnet, the CIDR and actually the network name also. So far so good. Now the last one, it's called OS subnet task. Sometimes when you're running something, the inventory is not kept alive. The inventory is right at the beginning and it's used forward. Sometimes you need to load stuff from memory as you go along. So you have module like this that tells me just give me the facts please. Okay? And you give it the facts and it'll store in a variable for you that you can use and it's basically just a dictionary and you can parse it any way you want. Okay? So unfortunately, thank you, Justine. Unfortunately this is as much we can go to at the moment but I'm gonna sprint through a couple of things quickly and if you don't mind sticking around for five or 10 minutes. So the second step that we'll actually go through is the provision document and this one is where I actually provision the VMs and in this case I'm using an Ansible role that I created. The role is called provision VM and the role takes input that are the cloud name and the specifications of the VM itself. Now, since I'm a good boy, what I do is I store all those into variable files. You can see I stored the VM specifications in a variable file. So the name of the VM, the key name that I attached to it, the size of the VM and basically how many I want of it. So it'll create as much as I need. So going back to the presentation. So you start basically from an empty set.  And when that happens, actually the agent technically lost because you actually provisioned everything that you wanted to provision and I guess I will say now you know kung fu as Neo famously said. So we can discuss this more offline. I appreciate your time and your patience. And I hope you learned something new today. Thank you very much. They're gonna be posted on the talk itself, on Skype.