Welcome to the home lab show episode 73. How you doing Jay? I'm doing well. How are you? I'm doing great The thing we're gonna be talking about today is automation and this is gonna be led a lot more by Jay I'm a observer here. We're gonna be learning a lot more. I familiar with the words I've not done as much deploying with any of these but the automation strategies are really important because when you Build something out, especially if it's complicated now It's fun to tinker with things in the lab but for a business you want to have like a wind up and wind down process and replacement process to Build in an automated way or build a dev environment in a scalable way and a very Repeatable way. That's where the automation comes in because you know If you're used to working in a Windows world people like oh You just keep cloning windows and do some of that but in the Linux world There's orchestration tools we're gonna be talking about to make your life easier when rebuilding all of your VMs Or your applications and everything so that's what our topic is for today. The first place we want to Talk about though is Linode and they have all kinds of automation tools We're actually gonna be talking about a few of them that you can absolutely use in Linode I think Jay's gonna have some expanded topics later on that but of course This is all standards open things we're talking about an open source and Linode is a great place a great sponsor The show who run all of your open source ideas and project when you want them public-facing And I said first sponsor because I'll think Linode for this But we want to give Jay a shout out because there's something Jay wants to talk about Yeah, I want to basically plug my new book Mastering Ubuntu Server fourth edition It's been out for just under a week now and it's doing really well So I just wanted to thank everybody that has checked it out It's currently the number two book when it comes to Linux servers It was and it also held the number one and number three spots both when it comes to New releases and servers and like several other categories. So I'm like really blown away right now at the response It's just been so great. So if you haven't checked it out yet, you could go to a boot to server book calm That's a special little micro site I've created that just links to the places where you can get the book from so definitely check it out If you haven't already I am like super excited for this one because I don't have any numbers But it really does seem like it's doing even better than the last one, which is definitely a great feeling So yeah, really super happy about that I'll say make sure you leave a review and let people know what you thought of the book Not just on like here, but like the places where you get the book. It's where you can do that Is it available on Amazon as well Jay? It is Kendall Amazon the publisher has Their website for it and then I also did a Google search and I it looks like other other Publishers websites are all are also stocking it. So I'm like, that's pretty cool more exposure. So Yeah, very cool. So there's our announcements and now we can dive into automation But yeah, it's cool that Jay's got the book. I know I know Jay won't have I got an autograph copy of the first one. So I don't even remember autographing that one. That's cool. I think you did I had your name in it. I think last time I got that one And then I do have I already got a copy is why I've not read it yet So it is on my to-do list to sit down and it's perfect because it's winter so I can sit down and read some books That's I'm not a big summer book reader, but I'll I'll read a few books in the once it gets colder up because I don't want to be outside anymore Yeah, yeah, same here actually and get my Linux knowledge on I want to be as hard as she on Linux. He's All right. Yeah, now let's just dive into the topic here. What's the first tool we're gonna be talking about? Yeah, so I have a little list that I've put together that I'm going to be following along with and some of these things I want to be clear we have talked about some of these things but Some of the things that I'm going to going to be talking about are brand new I've never mentioned before some of them I did and then it's more along the lines of how you orchestrate everything from beginning to end and how these tools fit together because we've gone over like you know singular tools, but this is more like how they fit together and In the case of fabric, I don't think I've ever mentioned that at least I don't remember mentioning that so we're going to talk about that as well so the ultimate goal is to automate every step of the process of building servers and Tearing down servers if you can't achieve that then that's great because then if you know something happens You need to rebuild it's I mean you've been training for this right? You have all the tools and the scripts and everything in line and you could just go through and do what you have to do So that's the ultimate goal But it takes a long time to get there if now a bonus goal is if you get you could get to a point where you can like Let's just say delete a virtual machine. Just delete it Just just randomly delete it and then you know it just comes back a new one just automatically Spins up your data is untouched. Everything is there. That's a bonus goal If you can get to that then you're pretty much at the level that enterprises should be You know, you might think they're all you know at that level but most If you could get to that point then it kind of helps you you know with your job if you're tasked with Implementing some of these things so the first thing I'm going to talk about is a concept more than it's a tool and There's probably different words you could use for this I'll just call them stateless servers the idea being that you could have a virtual machine or a container It really doesn't matter. It could be a container or virtual machine so you could just insert container anytime I say virtual machine and It has no data inside and that's pretty much the case when it comes to Docker anyway But that's no data no database nothing. It's just the VM or the container Just attaches to maybe an NFS share a remote database server so that you can delete or you know Delete the VM or if it goes down you can recreate it and that's how you get to a point where there's No data loss because there's no data to lose on that instance because you don't put anything important on that instance So one one working example of this for me is Plex by and I've talked about this before But if you haven't heard me talk about it the basic gist is that Plex if you didn't already know is a Server for multi-media especially movies, but they also support music to allow you an easy way to consume your movie collection Now obviously movies are a big pain point when it comes to storage Now my VM is 16 gigabytes Gigabytes not terabytes gigabytes and it's completely stateless What happens is that everything is mounted via NFS? So and this has happened if something happens to that VM I just you know restore a backup or whatever and it comes back and whatever I don't miss any movies It I still have my progress and everything is great So if you can get to that point that's awesome But the very least we want to be to a point where you can just automate the building and Destruction of your servers you should be able to spin up and tear down things whenever you feel that's necessary So that's going to be the overall You know storyline I guess we're following here in the saga of Automation and home lab. So that's where we start to get into some of the tools Now again, some of these I've mentioned before But I'm going to throw a bit of a curveball and I'm going to mention one right now that I've never mentioned before and that's fabric Fabric is a Python library that you can download and you will need to know some Python You don't have to know a lot. In fact, if you want to learn Python As long as you are willing to you know struggle a little bit at first by just learning the basics of it It really is a great way to learn Python because with fabric what it does is it essentially is Basically like Ansible but not as powerful. So it's really a good way for bootstrap scripts. For example, if you want to Do an initial settings or config run on a VM for the very first time You could just use fabric for that and you can point it to the server over SSH just like Ansible It doesn't have as many features as Ansible So if you were to use this as your only automation solution, you can do that But at some point you will outgrow it because it's not it's just going to hit a ceiling and then at that point It's Ansible. So you may as well just use this for bootstrap scripts So it lets you do common things like copy a file to a server run a command on the server and things like that and admittedly it's been Probably five years since I've used it. I had to actually just Google it before the show and make sure it still existed It didn't get deprecated It does I was using it back with Python 2.0 something. I don't remember which version So obviously the syntax with Python 3 is going to be a little bit different But if you're looking for a project to learn Python, that's going to give you like an applicable Systems-related thing because let's face it you buy a Python book. What's going to happen? It could be a good one or you could read this book and it's like here's how you create a mortgage calculator with Python And then you're thinking gee how exciting is that? And that's why yeah Projects that get you to a goal that work, especially in your home lab to help solve a problem help you dive into them more I that's one of those things when people ask for to get started find a product Trying a project like that that's going to like hey I want to get my Plex server automated cool You have a goal because not everyone why maybe you're working to find the instantion you want to build a mortgage calculator But I'm gonna tell you right now. I don't want to build one So and the interesting thing is that's that Fabric is actually how I learned Python because with my ADD I'm trying to read these Python books and they're putting me to sleep no offense to the author I'm sure the problem's mine, but at the end of the day This is why when I write I try to make it more like You know related to the actual field, so it's more exciting rather than You know you read a programming book. It's like objects are like dog or are like dogs You have type dog, but then you have subtypes like poodle. It's like that's great But how do I use that in the real world and fabric is absolutely that because What it does is it it shows you how you could use Python to actually interact with you know with systems and Connective systems and you know deploy something to them, which is really great. So One use case for fabric That would be a good one is installing ansible on a target server, right? Because we're gonna talk about ansible a bit So you want to use ansible, but it's a chicken and egg problem You could build ansible into the image that you deploy That's fine or you could just have a bootstrap script that does the initial connection Installs ansible, you know things like that. That's fair and then while you do this You'll learn Python if you didn't already know Python So it's kind of like a two-for-one, which is why I wanted to mention that first You know, I have a question though and I've wondered about this Are there any of these that? Do it in reverse so to speak like hey, I have this existing server Let's say my gray log server and it can it reverse out how I built it at all I grabbed that configuration to bring it back to that state Where it pulls like the change configuration files. I guess I'd have to point it at at that point, but Um Yeah, there is it kind of depends on the tool though now with terraform. We'll talk about that Okay, doesn't allow you to do that. However with terraform. It's actually going to tear everything down. So if you wanted to Do like a one-off You know remove a part of it or go, you know move one config file back But not the whole system I would say ansible is a great way to do that if you have it under git for example version control Then you could just revert the get commit back to a previous one and then you know, it'll run So that's probably the best way to do it. But if you wanted to tear it down Terraform can absolutely do that. Yeah, because my thought is like there's a lot of packages and extras like you install the base So as then I had to throw some packages I mean easy way to do it I've done is just you know look all the as I'm usually using a debium based or a bunch you base system I could just dump a list of the packages and make sure I reinstall those I know how to do that. That's been my kind of easy go-to way to get it done Why I need to say I just want to build server just like this one. Here's everything that was installed on it I'll just run this again to install But I don't know if there was an ansible builder that did that for you or some other related No, we'll talk about some tools and then I think that'll probably Better frame your question because if your question isn't answered by the end And I think it'd be great to talk about that perfect and we can and anything else related to what about this What about that? I love those questions because that's how we learn, right? So we've learned those edge cases and that's just audience. This is how Tom learns to so Yeah, and sometimes I'm learning from you like like there's another topic. We're thinking about Covering we mentioned yesterday. I'm not gonna mention it. I don't know if the episode is gonna happen But I don't know anything about it really so I would probably be looking at you for that so The next tool that I'm going to talk about is vagrant, but I also want to mention packer as well I'll talk about both in more detail But there might be some confusion about vagrant versus packer when you would use one versus the other so But I mentioned what they do you might think that they do the same thing until I get into more detail so vagrant allows you to spin up reference virtual machines and Then packer allows you to also spin up virtual machines, but it focuses more on images now Let's talk about the life cycle. So vagrant is something that you would use before packer What vagrant allows you to do is it can hook into a virtualization solution which could even be just virtual box on your local computer It doesn't have to be a server. So for example, let's just say You heard about a new feature in Ubuntu or Debian or whatever or a new package or something And you really want really want to try this out now What this can allow you to do is you can use vagrant to spin up a Debian VM You could point it to virtual box. You could point it to other sources as well In my notes, I have virtual box VMware and AWS There's also a third-party plugins to add support for other things But vagrant is more for test environments It's not for like and this is the main difference between it and packer So packer is something you would use to create something that you plan on keeping around that's going to be possibly production vagrant is more like I want a reference VM to test something so I want to see what this app behaves like on Debian versus Ubuntu or I want to run my Ansible scripts and I want to change my ansible scripts, but I don't want to run them against a production server I'd rather test them out on the test server before I point them to production There's even as I was googling around a way to use vagrant to spin up a proxmox host So if you wanted a reference proxmox host, you could absolutely do that So I think the most common use case for vagrant is to point it to virtual box That's what I've seen the most so you have a you know developer or devops person They might have virtual box in their system. They use vagrant to spin up VMs as they're testing maybe their company's app They want to test on different VMs. They want a pristine Virtual machine they can create it with vagrant run their tests and then use vagrant to bring it right back down So vagrant can delete the VM when you're done with it So that could be a good way to automate something Now Packer and a lot of these tools are made by hashy corp actually I just kind of now realize that because vagrant Packer and Terraform You know our hashy corp hashy corp hashy corp makes a lot of devops tool So I don't think it's possible to even study devops for longer than a week and not at least hear about hashy corp At least one time. Yeah, there's a market leader in that I would say it really are and it's open source By the way, so even though that is a business behind there These are open source tools. You have access to the route talking about yeah, so Let's look at the next stage. So You want to build a virtual machine? Maybe you want or even a physical server doesn't really matter, but You just want to build a VM. You're ready to build it so you can get an ISO image You can install a Linux distro or you could even create a template That's fine and a lot of people do that near virtualization solution Just create a template use that template to you know create other virtual machines You could build in things to the template if you'd like But what I like about packer is that it allows you to automate the building of the image or the template layer so you think about Your virtual machine solution and the work you go through when it comes to building in Ubuntu template, for example You could obviously just download Ubuntu server install it on a VM and then make it a template Or you could create the config within packer point it towards, you know proxmox for example I want to build a proxmox image or an AWS image or whatever it is and it'll absolutely build that image for you So you could say I want to boot to 2204 and then it just does the thing and you have the thing so packer automates the image layer of this So that's why you know, I'm going kind of in order here So vagrant to test things then packer to actually deploy an image although vagrant is something you'd probably use at every step because you're probably always going to want a test subject every now and then to test something but The actual first stage of a production deploy is probably going to be packer because that creates the image But it doesn't actually deploy the image You know for you your virtualization solution you could deploy the image or template or you can move on to the next step Which is terraform and terraform can take that image that you created with packer or even a template if you didn't use packer But terraform can create the environment. It can spin up the virtual machines using whatever image you want it to use You know when it comes to cloud providers that can set up your networking subnets security groups, I mean whatever features your Provider has and hashi corp uses the word provider for what you're Building something inside of so if you are building it in AWS or proxmox and proxmox is the provider in that case AWS you know AWS is the provider and there's provider plugins for these tools that you download same with packer and Obviously terraform so a terraform that creates the things that you would would then be maintaining or keeping going And then terraform allows you to also destroy if you want so if you have 10 virtual machines in your config You know a terraform and you spin up those 10 virtual machines It'll let you do that But if you do terraform destroy it'll destroy all 10 of those VMs so you better be careful with terraform destroy Another thing terraform allows you to do is update existing things that you've rolled out I don't use it for that As I talked about in another episode. It's like ansible is probably a better fit So what I like to do is packer creates the image Terraform takes the image and creates a VM with it Ansible takes the VM that you've created with terraform and then makes it to the spec that you want it to be so If it's going to be a web server then ansible will you know, you could have a role web server It'll make the server into a web server or whatever you have to find And then if you make changes to your ansible config then those changes will be rolled out to all the machines if you want to do a I don't know a security update. There's a vulnerability. That's out right now So you could just put it in ansible once and ansible goes out and just Hits all the machines and make sure that they have that security update rather than signing into each one one by one And doing it manually, which you know ain't nobody got time for that Which is probably it should be the slogan for configuration management in general I think So So those are some of the tools and there's other ones that I want to mention as well, but Now when it comes to ansible, I have a short URL for that for my ansible series if you haven't learned it yet It's a Linux video Ansible one, I'll take it to the first episode in that series if you want to go right to that We'll have all the links in the show notes But so so far my question to you Tom is if anything I've gone over at so far answers the question that you had Yes, a lot better and one of the things I'll mention too if you look up some of the terraform providers as they call them You're gonna find everything from AWS Azure Google Cloud Oracle Cloud there's all your major big cloud companies in there, but there's also proxmox Then there's also attachment for Zen Orchestra So all the different platforms that we've talked about here on the channel, especially Zen and proxmox are Supported as well and they even have some windows integrations, which I didn't realize I've never used them for that But yeah, pretty cool that you can do terraform with some windows automation as well So that's that part center not just Azure, but like it has some active directory options I sent it outside of the scope of this particular video, but yeah, there's terraforms pretty cool to to get things built up Yep so other concepts that I want to cover because those are the some of the tools there I mentioned fabric for creating the bootstrap script so Fabric you could then have it do the first ansible provision for example it can install ansible and then you know run the first provision That's a valid use case for it. And then I mentioned the purposes for the tools from hashy corp Those are all great things to learn highly recommended now another concept that I think is important here is Continuous integration continuous development or is it continuous deployment? I can never remember that CICD and Immediately a lot of people think about things like Jenkins, which is like a really really popular CICD solution So the idea is if you use get which you should use get for everything like you're you know all the configs and everything you do But even better is if you have a system where when you make a change to those configs and you push those changes Then you have something like a CICD solution that Notices that you made a commit to the repository and then runs it on an agent or whatever it is They have it due to prove that it actually works and if you have an error it'll let you know But I don't want to make this episode overly complicated Jenkins is a whole another thing But what I've done is I've created what I call a poor man CICD solution, which I feel like it fits the absolute definition of CICD But it doesn't require Jenkins or anything special so for me when I use ansible I have two repositories. I have the main repository, which is what everything gets that's production And I also have staging as well staging is where I put all changes basically And anything I put into staging gets run on reference VMs And then I'll get a message back that you know the deploy finished or it didn't and I'll fix whatever I have to fix So I commit everything to staging and then if I commit something to staging and then it works I don't get any errors and I just merge it into main and then all of my machines Get that change, but how did I set that up? So what I've done is I've had I have a reference virtual machine for each in my case My use case is gonna be a lot different because I'm a you know, I do a YouTube thing So I need like reference Dabian reference of boon to and all these different distributions So for each of the distributions that I support via ansible I have a reference VM for it that is looking in the VM itself via cron is looking for a commit to Staging so I don't have Jenkins or anything like that I just have actual Linux servers that are looking for changes And then if they see that there's been a commit made to the staging repository They'll pull them down run them send me an alert Let me know if it worked or if it didn't and if it passed on each of those VMs Meaning it works on all the distros and I merge it into main and it goes to the rest So that's one way it's kind of manual But it is a way of having a CID CICD solution without also having to deploy Jenkins But I'm not telling you guys to avoid Jenkins If you are really serious about CICD and this interests you and that's absolutely the direction I would recommend going but if you just want something in place quickly just to test your code or whatnot It's probably easier just to do it the way I've done it But also keep in mind that doesn't scale well So if you then want to test this and then you want to test that you have things outside of Ansible at that point It probably is better to use Jenkins because that's what it's built for and if you manually build everything That it gets to a point where that's just too much technical debt But CICD is great because again You could just push your change to a special repository that production servers don't have access to It'll run against reference agents or VMs let you know the status if you like what happened out of your config Then you could push it to the others and I think that's a really important thing to do If not for homelab, it's especially important But it comes to enterprise and if you work in IT then it's probably just a good idea to get some practice with this If you don't already have practice with this And one of the things you touched on that I want to repeat for the crowd here Technical debt it the technical get starts the moment you start the project you have to from the time It's on the drawing board look at the scalability of it And you'll save yourself so much time a year or two years from now So if you're just someone who's building a homelab awesome the technical debt is part of the learning process and things like that but if you're an IT person in charge and responsible for your Internal IT or external because your next general administrator really think about the scalability where that project's going to go Because man, there's so many times people contact us and they're full of technical gr at a technical regret It's just that's probably where we put it. No, I kind of like that. Can we just use that? We just use that it's not technical debt It's technical regret if I only would have known not to try to do it They built something so complicated that it's become unmanageable because the company had 10 employees when they did it now The company has a thousand employees. So you always have to be thinking about that It's just a consideration take away we're going with it So as we said if you were looking at or go with something like Jenkins There's a reason that's used and almost every new young technician thinks they're super innovative in some are But they just try to go against the grain and write it all by hand and build their firewalls from scratch with Rules and then they realize wow, I'm the only one I can manage any of this So always kind of be thinking about that. It's a very beginning Absolutely, and I want to underscore using get for everything. I think it's a very important thing to get into the habit of this So what I'll often do is create a get folder somewhere on the file system and put the servers configs there So for example, I might have a private repository and all these are going to be private unless you're sure you have no Personally identifiable information in the repository, you know passwords or anything like that Well, you should never do that actually regardless of whether it's private or public but What you could do is have private Repositories for everything and then in the case of a web server, obviously you're going to have Apache or nginx config files, for example And if something happens if you have another person that you work with I don't know if anyone in our audience has a home lab club or something like that It'd be really cool if you did to where you know, you might share some of that It's probably a good idea to have this because if something stops working You might be like, okay, what's changed if it's under version control, you just run get status You can find out right then and there what's changed because you'll know you can run get diff and and I have a whole video on on how to use get if you guys are interested but That allows you to see Configuration changes historically over time and roll them back and a lot of people will think of get and we have again We have a whole episode about this actually a lot of people think of get is something that you know software developers use It's not for me. I'm a Linux admin. That's for the developers actually it's for everybody I don't even care as I said in a previous episode if you are writing notes for yourself. It's put them in get Why not? We actually comment on our get episode because someone puts their recipes in get Right, right. Yeah, so that's another example of something you may not think of at the you know off the top of your head that it could be used for But I've had servers for example, and I think this is how you separate the you know expert threat actors From the beginner or new threat actors because I remember one time I had a server and I'll remember how somebody got in You know they did something and then I just sense everything's under get and this is like a WordPress site actually I just get revert and then everything the whole state of WordPress just Rolled back to before that person even broken into the server I mean obviously you can make a case for net never trusting the server again if someone got into it But I thought it was kind of funny because I'm oh look at that person they they put a B64 thing in there and just a bunch of gibberish everywhere get revert, you know back to a previous commit. Yeah, that's fine You know good luck with that. Obviously it was a threat actor that was You know probably intermediate or advanced then they probably probably would be looking for a dot get directory and getting rid of that first because obviously that's how you revert back, but That's just an example of a use case forget that some people might not think about That also might have a bit of technical debt with WordPress That could have been a bad example because WordPress the plug-ins updating and this updating and that updating It could be a full-time job watching version control for a WordPress installation of all things But yeah, um, you could absolutely use get where you wouldn't even think to if nothing else If you're Apache or nginx config files are in there That could be a good use case for it So you could keep history on that or whatever apps you might be running on your System, but another thing that works really well with get is Let's say you have a template or a script that builds something Your server goes down it breaks or whatever What you could do is you could just have the server come back and then just do a get pull of its You know all of its configs and it's right back to where it was Right before it died or had an issue, right? So at that point it's just so easy to get everything back because you have to configs and get you have Ansible managing it So even if you restore an image from like four months ago Ansible will catch that up and Get the packages up to date if you have that set up and then from there you just do a get pull Actually a get clone in this case And then you can have your configs back to and you can even automate the initial get clone If you wanted to do that and I've had servers that absolutely, you know, absolutely do that when they start up They just do a get clone of its config files and then Ansible runs and I'm in this later It's like nothing ever happened, which is just such a great feeling if you can get to this point because it's just so cool It is and I seen someone comment that they that's where their home lab automation is currently not an existing But that's where we all started and you slowly just stack and build on there all of it Mine is still mediocre Existing in my home lab my business stuff's way more tight than the home lab stuff I do and I can't really tell there's always such a split between them sometimes, but definitely It's not easy. It seems so hard until it isn't as a best way to describe It's a lot of things it's like even the Linux command line is daunting for people until Something clicks in your head or even networking and understanding subnetting is really difficult and then all of a sudden one day It's just like you you kind of wake up in the matrix. Wow. I know networking Yeah, yeah, and sometimes it's like, you know, it's not even you it's just The person teaching you or wherever you're wherever you're getting that the instruction from it might be a great teacher But maybe they're just not your kind of teacher because everybody has a different kind of Instructional process that really resonates with them as for me. I like watching videos It's funny because I can write books, but they're not my go-to when it comes to learning something It's watching a video is learning something but when I want to teach it's making videos But also writing books, but everybody's different right some people right I've actually met people that can read a book and do only that and they could know and they have good memory and retention They I'm like, wow, you actually remembered all that just by reading the book one time. That's like a superpower I wish I could do that but but that's just how it goes everybody learns differently and I Think the most important thing to think about when it comes to automation is Yes, I'm telling you to automate everything, but that's just a general term There's some things you probably don't want to automate So if you're doing something that you know for a fact, it's unquestionable You're only going to do it one time. There's no reason to automate that so some people will write because and it's good for practice But if you're automating things that have a very unlikely chance of ever needing to be done again You could probably make a good argument that it's a waste of time But if it's something that is going to make things easier for you, don't go too crazy like me I'm fascinated with it. You might not be as fascinated with it as I am maybe for you It's just a means to an end if that's the case automate what you need And if something is taking you hours to do and it's just drudgery then automate it because then it wouldn't take you So long That's a good reason to automate something and if nothing else even if you roll out ansible to all of your machines And you have all your all your machines checking in and whatnot Even if you have no config and no automation at all and it's just a skeleton that can still help you at one point because if there's a again if there's a security vulnerability and Something's being exploited. You need to get this patch out there and you have 20 servers When the day comes that you could just put in a play within your ansible config to push that Security update to everything in one shot. You're going to be very happy that you've implemented that at that time Even if that's all you do is you know, like a skeleton configure something like that. That's fine That everything is fair game here But for the person that mentioned they don't have automation at all Well, you always don't until you do right and if it's not important to you and it's not something that resonates with you maybe just Building servers manually is fun And for some of us that is fun and for some of us automation might take that fun away because maybe we like the process of Installing Linux and we don't mind doing that over and over again Okay, fine in that case, you know why automate if it's not going to help you out but at the end of the day if you at least automate the things that drive you crazy or just annoy you or Just otherwise tedious. I think that alone adds value to automation Yes, and by the way, this is easy to get wrong at scale ask Facebook about their outage because That was really nice automation for servers, but the automation itself Was what led to The cascade effect of hey build this rebuild this and these conditions have been you know other times in an example might be Oh if these servers are down for some reason you can rebuild them But what if they're not down for the right reason and then the automation tools are trying to build something on top of that? This is I've seen people mess up HA systems in a similar way by not having the proper heartbeat to do things and the HA Will try to start on more than one to try restart your BMs on more than one system So with any of these automation or redundancies You had to be very careful and think about any of the race conditions that may exist Where there's a collision within here because man when that goes awry Facebook managed to for those you don't recall they lock themselves out of their own building because they their Identification management system also relied on their physical building access relied on that so people in the building could stay in Or not get out and definitely the people on that side couldn't get in and couldn't get into Reset the systems that were down. I remember that you know What's funny and I'm probably not supposed to be saying this but I'm just going to Because I um well the first thing I don't I don't think anybody cares for me to say but it was actually quite a long time ago I don't remember exactly when but at least You know five years ago probably closer to seven or eight or something. I actually interviewed at facebook It's not something I've mentioned Many times it was it was an over-the-phone interview that I've done I figured I wasn't really interested in working there, but I just figured okay This might be interesting and even in my career videos. I tell people if someone wants to interview you There's no harm in letting them Because even if you're not looking for a job you could just be practicing your interviewing skills And who knows you might be super surprised and find out you actually love the company but for me I was just kind of curious and One of the things that the person said and disclaimer the person I talked to was you know The initial HR person it wasn't like someone from their technical team But she did say that the motto was at facebook internally and this is where I might get into trouble The slogan is break fast fix faster Yes, this time they didn't fix fast enough Yeah, that one it's a startup culture problem There's a lot of the companies as well. There's this thought to do it that way It sounds it sounds good for a slogan like you're going to work for real innovative place But then it it doesn't always lead to the best outcomes If you're a unicorn like facebook sure it's going to make a lot of money, but it's not for everyone And you know recent incidents with uber probably is because they have some of those same policies Which is why someone undoubtedly temporarily put the master Break glass password into a script that allowed for the elevation of privileges Now that is something that happens when someone's like, I don't know get this done We're like, well, it's not reading from the external script. I don't got time to solve why So the short term is stick the variable in there That's and if no one goes back to fix it. Well, there's nothing more permanent than a temporary solution to a problem so That's where you can be you got to be careful when you're thinking about that And how you place any of the data in there because by the way as you build your ansible scripts You have to think about that if you're setting passwords How are you embedding any of the very customized very sensitive data when you're doing that? That's uh, a lot of people get in trouble for that. So it's another consideration is the security implementations that can come from doing Exactly that problem. So by the way, my scripts don't have them in there because i'm accidentally sent I have more than one friend with a similar name and I sent my scripts to they're both technical people The other friend and when he replied back, he goes, I don't know why you sent me this But good job because your script just references where Where each one of the private credentials are but doesn't actually have them in your scripts. I'm like, oh, yeah, yeah, true This is my mount scripts do not have any They need creds, but the creds are located a very specific place Now, um, I want to touch on something you brought up though because you know, you mentioned facebook, you know got locked out Right. Um, and some people might be faking. Well, what does this have to do with homelab?  I don't have like my Door access actually with homelab you could have your door access in the same system nowadays with iot But it's probably unlikely But you'd be surprised how often a situation like that happens For example, and this is an example. I think that would be likely to happen to a homelab person more than any other Um, let's just say you do the right thing and you put two factor on everything Maybe it's a uber key. That's awesome because that's great. Um, that's something you should do But then you lose the uber key. So you lost access To the uber key which is your access to the rest of your system So that's why they're I don't care if you have to have like a backup code that you put in a safe somewhere Or maybe another uber key that you put in a safe somewhere just in case your primary one goes I mean with homelab you run into those things where you can legitimately lock yourself out of your own things It's happening to me. I think it happens to I'd be surprised if There's any homelab are out there where this didn't happen to them at least once so if you have been A homelab person and I don't mean just one server I mean you have more than one server And you've been doing this for at least four years and you have not locked yourself out of something Please email us. We'll probably call you a liar But um, you know, maybe we'll mention it because I really don't think that it's um a thing That doesn't happen to everybody at some point everyone gets locked out I mean I've locked myself out of the house and out of the car before so maybe I'm a bad example But um, you got to think about having another means of access Don't make it too easy to where it's just sms to your cell phone and that's so easy to um, you know For a threat actor to get into But another uber key is a great idea or some other means of getting in I mean that's that's something that's important to have Maybe maybe that's a whole show topic is talking or you know, it'll be part of our rata show We'll talk about some methods because I have some thoughts on that about password management And where you should hold on to certain pieces of it But that'll send us way off topic. We start down that road But I think we've covered all the automation or do we have anything else we get? I think that's the Overall gist of it because we can go into more detail and into any of these things and some of these we already have Like we like episode 10 was our ansible episode if you're interested anyone listening So if you haven't caught that one episode 10 is the one where I go into more detail about ansible than I also have The ansible series There's some videos about some of hashii corpse tools on my channel. I really do got to update those I i'm not going to recommend any of them because if I do have any they're probably you know way too old Maybe I should go around um and do more videos about that because I think that's something that people would enjoy seeing Like maybe I can have um No, a linoad instance spun off as well as a proxmox instance spun up via these tools Oh, yeah, that would be fun and our episode on git was episode 25. So we've got that so run through our back catalog of episodes There's plenty to listen to and of course jay's got lots of these as we mentioned in videos Check out either of our channels for all these different fun things we talk about and more in-depth discussions on them all right Well, thanks for joining us and uh, everybody take care looking forward to next time Um, we're still deciding because of some timing of whether or not we'll have a show next week or not So, uh, we'll leave that up in the air, but if you see if you see the live stream get posted Um, it's going on if not we may have to skip one because of some timing and uh collisions of things Well, we'll get it all sorted out At a minimum, so yeah, we we just don't worry it may have to skip one small but we're still moving forward we have more ideas and uh That one we're talking about I I think I have some thoughts on that. I'll share with jay. So all right. Thanks