Welcome to the homelab show, episode 81, Building Lab Server Templates. This is a topic me and Jay have tried to figure out how to word because it's gonna cover a few different things and well, we're just gonna throw it all in this episode because I think the question comes up a lot about, you know, how do you get your template set up? How do you get, you know, some of those, you know, simple to deploy templates because you wanna play with something and there's a couple of different methodologies we're gonna talk about and not one of them is right. It's about which one is right for you. That's what a lot of this homelab stuff, we wanna remind people of. You can all have your opinions of the way you wanna do it but the way you're most comfortable doing it because that's what I kept replying to someone in my own forums, do it the way you're most comfortable with because they didn't like my method. And I said, that's fine, you don't have to. You asked you what it was. Fair enough? Yeah, simple enough. Before we jump into this, let's talk about a company it has plenty of templates to play with. In fact, about how many different templates do you say are in Linode's store there, Jay? I have never counted. It's just this list I scroll through until I find what I want. They have a lot of stuff out there. A lot of great stuff. So sponsor of today's show is Linode and they have templating figured out. It's actually something pretty cool that, I mean, one of these times or have you done a video on that directly with some of the Linode templates, Jay? Yeah, I've done so many videos with them, like either on their channel or on my own. It's like I'm losing count at this point. So I think I covered it, but sometimes it gets a little blurry between my channel and theirs. Yeah. So there's a lot you can do with all these templates. We thank them for being a sponsor. It is a great place to run any of the different things we've talked about here. Well, most all the things we talked about here on Home Lab, there's a few exceptions. You don't run your hypervisor inside Linode. They have one they provide for you. But many of the projects you talk about, if you need to host it somewhere, Linode is a great place to host it. We have an offer code in the description down below. Thanks, Linode, for being a sponsor and let's talk about some templates now. Yeah, I wanted to start with, so basically I'm trying to think of the best place to start. And I'm thinking the best place to start is the whole idea of not having a template or image just so I could kind of address that because I know that mindset does exist. So what I mean in particular, I'm not talking about people preferring to set up everything over and over again manually, because let's be honest, that's just a waste of time. Right. Some people do and some cloud providers prefer is the concept of user data, which I don't really know where that term came from. I guess I could Google it, but either way I'm not going to agree with it because the idea is that user data is run when a instance is created in a cloud environment, whether it's OpenStack, you're running your own or it's AWS, a number of others. The idea is you just take a vanilla, Unchanged, Ubuntu, Debian, CentOS image, whatever it is and you apply user data to it, which would just have bash commands inside, maybe something like sudo apt install dash y apache2, if you want apache to be straight out of the gate, as soon as you create that particular instance. So I mean, there's still an image, right? Because the cloud provider has the image, you're just using theirs and then using a script to install packages, maybe even bootstrap your automation service. I just want to acknowledge that that exists. And one of the reasons why that some people might like this idea is they don't have to maintain an image themselves, they just cause the instance to get bootstrapped from the user data with whatever command you would normally do. If you have like 10 commands, you might just paste them in there one after another. Maybe you'll pull in an Ansible script or something. So I just want to acknowledge that. And the case can be made for no images at all, but I don't think there's any right way or wrong way. Just like you mentioned, it's whatever works best for you. Yeah, cause a lot of simplicity I try to do is I have a lot of just a bunch of different variations. I leave running that I all just designate the word template to them. There's not really anything special I've done. Other than I've done all the things. I got my SSH keys in them and things like that. And I have them running unattended upgrades.  Or sometimes I just take the template itself snapshot, it try something. And if it's not something I would want permanently added to that template, I just hit revert back in the snapshot to do it. It's all about speed at which I want to get something done or try a new thing. That's why I tell people, once you've come up with a secure methodology and a good workflow, it comes down to what you're comfortable doing. And there's some things I wanted to test. Like I wrote a MinIO installer. So I kept reverting the same template back over and over again until the installer worked because that's the fastest way to do it. Because I was like, oh, the installer failed on this part here. Right, right. You know, going forward from here, because I just wanted to get that mentioned out of the way because that's again, user data is more cloud specific. Again, though you can run OpenStack on your home lab and have user data and cloud in it can use user data. I'll talk about cloud in it later. But here on out, I'm going to use the term template interchangeably.  I don't care if it's VMware, Proxmox or physical disk images. Maybe you're going to use something like Clonzilla to take an image of the hard drive. No matter what kind of hardware it is, whether it even is hardware, the goal with templates, again interchangeable. I know it could be an image if it's physical. The goal is to have something to start from that lowers the amount of time that you're going to spend getting the instance set up for its actual purpose in life. Maybe you want some packages installed on everything. You just want those packages, you know, always there right out of the box. Maybe there could be some config files that you prefer. You want to make sure those are present. So there could be all kinds of different things that you might want to do. And in the enterprise, there's going to be the, I believe it's CISA, right? Because we had this conversation. I can sometimes remember CISA. They have a hardening image, hardening guidelines. If you type in CISA hardening guides, this is something nice. Then they've actually come a long way. We always think of the government. I think the first joke Jay made and I laughed because he's like, oh, do they work in Internet Explorer? Because the government's traditionally a little behind when it comes to technology, but CISA has actually improved a lot of the years. And, you know, they have good, really solid hardening guides and standards by which you can follow.  Exactly. So, and that kind of brings up another point too. Like how deep into this rabbit hole do you want to go? Now, you could literally have everything you could possibly ever want or need in one image, but that's going to take a very long time. And then if you start getting into security, security is a bottomless black hole. I mean, there's just no limit. I mean, you could go, like, there's always going to be someone else that can outsmart you because security is security. So you're never going to have 100% perfect security, but there might be some things that you might want to make sure are there from a security standpoint. Companies go a lot further. A lot of them, especially if they're, you know, have some kind of business certification for compliance or something like that, they might have to follow the CISA image guidelines. Now, HomeLab, we don't even have to follow that at all. We could just install our updates if, you know, something bad does happen.  Anyway, hopefully everything is just going to carry you through between your backups and your automation scripts to get everything back and running. But again, it's probably a good idea to have some security in there. I'm not recommending that, you know, you do everything in that list of guidelines because it's huge. You can explore it and look at that. I just wanted to acknowledge that it does exist if that's something that you're interested in, especially if your HomeLab was created for the purposes of getting a security certification. I know at least some people are, you know, trying to get a certification or something and that might be why they have a HomeLab because I want to work with real, you know, equipment. So you might actually follow that all the way if you want to follow the security track. But either way, it's probably outside the scope, I would say at least 75% of the audience if I had to guess. I want to make everyone aware of that. We'll have links in the show notes for that. And then from here, we can actually go even, you know, more into detail and more specific to our topic. Absolutely. Which requires some hydration. Should we just get the Windows one out of the way real quick? Yeah, because I do want to, you know, at least address that because, you know, we were kind of talking and I do agree there's going to be a smaller percentage that of people that want a Windows image for deployment because more often than not you have that Windows VM for a specific purpose. Maybe there's this app that you need to run and you might only turn on this VM because I don't know, maybe it's a Windows XP VM and you have this thing that configures another thing. It's very common legacy stuff. So you're probably not likely to want a Windows image. Maybe just taking a backup of your VM is good enough. But if you do, you know, spin up Windows images or Windows instances, maybe you're also following the Microsoft track and the Linux track. I mean, who knows what's running in anyone's home lab? As much as I know the industry has moved away from it, I still use sysprep anytime I create a Windows image which is very rare. But, you know, Microsoft has their own imaging system. I don't remember what it's called now. I know it's deployment services. What's that? Windows deployment services. Okay, so it did change its name. I'm trying to remember what I knew it as like way back in 2008, but they have their own thing and you could absolutely follow that and learn it, but for me, sysprep works. It generalizes the image enough and or the installation enough. And then you could just use that as your base. The problem with sysprep and I, it really annoys me. You have three times that you could do it. So on the third sysprep, you can't sysprep that installation again. You have to pave and reload it. So of course, as home lab people, we're going to run that reference Windows VM in a VM. We're gonna snapshot before we sysprep. So after we create our image, you know, just shut it down and take an image of it, then we're gonna revert the snapshot so we have infinite retries. It's very, very, very easy to work around Microsoft's requirements so much so that I think they should probably just patch that requirement out because it's pretty much useless. I think any first year person working with Windows deployment knows to just probably use a snapshot and circumvent that requirement or that limitation. But anyway, I wanted to address that. I think people that are deep into the Microsoft world will probably be like red faced right now. Like you're still using sysprep. You should be using their service. I know, I know, but sysprep does work well when you create images enough for home lab people and going any deeper is probably only a good idea for people that are serious about the Microsoft side. But again, just something to acknowledge that it does exist. That's the thing you can still use. Yep, and if you Google Windows setup automation, there's some different guides for it. We as a company, the business side of me, we don't use that as much and the reason why is a lot of times it's more part of their domain deployment that we're doing. So we'll have something more along those lines to maybe scripts that we deploy that push them to a certain setting on theirs. So there's definitely a lot you can do. What is that? It's called the Windows autopilot system as well for when you're joining it to a domains. There's like, they've built some automation but it's all licensed. It's not that you can't build a lot of this with some EVL licenses in your home lab, but I don't know. Let us know. And if someone gives them feedback on this, we can bring guest in that is an expert at this if there's enough interest. I don't know if there's enough interest in this and I also don't know how good it would be as a podcast other than talking about it. You know, yeah, it's a little off, but we're gonna most of this, I figured that's the Windows stuff we wanna get out of the way that we know there's some that exists. And obviously part of the problem is there's several different ways to do it in Windows because there's different options they have. So we're gonna focus a little more on some of the Linux stuff. Yeah, and one last thing about Windows, I think Sysprep is also useful, especially when I get the question, how do I move this Windows installation from A to B and A could be a physical machine, B could be a virtual machine solution or they could both be virtual machine solutions in either direction. And in that case, in my opinion, you would back up, take an image of the unchanged Windows installation, then Sysprep because Sysprep can fail and even if it fails, it still counts as an attempt. That's one of the biggest annoyances there. Yeah. So, yeah, even though it's not your fault, right? You're doing everything you're supposed to do and maybe there's something in the installation that doesn't generalize well. But anyway, you create a backup of that and then you can Sysprep it, take another image of it. I know this is the long way of doing it but it's more accurate than that Sysprep image, you could then restore that onto B, whatever the target is and have a greater chance, and I say chance because it's Windows, but you have a greater chance of having that restored properly. And then the only thing you have to figure out is the drivers because of course, the drivers will be different, but if you use Windows, you already know how to deal with that. So that could be another use case for Sysprep when you're just moving from one place to another, which that question does come up. Now, we're gonna get into the main course here now and part of this is going to include cloud init, which I used to say that Linux is a lot simpler when it comes to, when compared to Windows deployment and how you're supposed to do it on Windows with the preferred tools on Windows, I would always argue that Linux is easier. Nowadays it's been so much more over complicated that it's like, I can't make that argument anymore, especially with cloud init because now it's like, congratulations Linux community, we have officially matched Microsoft in complexity and difficulty, but I'll get to cloud init, I'm getting a little ahead of myself. So again, when it comes to templates interchangeable, the idea is to have certain things set up. You don't wanna go overboard, but just have some of the things that are more tedious, you may as well just put it in the image. Now, when it comes to Linux, the story is changing a bit with how you do this and it also depends on the distribution too because some are more easier than others. I do have a feeling that the added complexity is going to make its way to other distributions. I primarily use Ubuntu server even though I complain a little bit, I still like it a lot and that's still what I use, but even they have made some things a little bit more complicated than it needs to be. But anyway, the first goal when you create an image is to generalize it. Even on Linux, you have to do that. In the past, it was pretty much just the SSH host keys. Everything else was fine. That was the only thing you had to do. And Raspberry Pi OS, and I covered this in one of my videos, they have a system descript that is enabled in the image. And when you create a flash drive or SD card from the Raspberry Pi OS image, the first thing it's gonna do is generate new SSH host keys. If the SSH host keys are the same in your image or actually in your image, then every instance you create from your image, you're gonna have the same SSH host keys. If you go into the Etsy SSH directory and list the storage there, you'll find a bunch of files that have host in the name, those are your host keys. And those need to be different. If they're not different, then you're gonna get this annoying error message. There's something wrong here. I forgot what the verbage is when you SSH. It actually cancels your SSH attempt and tells you that there could be something going on because it's checking for a man in the middle attack. The SSH host keys are different. They're not the same keys that you've previously used. They're the same, right. They're the same on each instance, but the IP is what's different or the host name is what's different, which kind of confuses the whole system. So it's better not to have those in there, but they won't regenerate themselves. You have to have something there to do that for you, which that system descript and we'll link to it in the show notes is going to take care of that for you. That's the easy mode version.  And then you'll basically make sure that no instance created from that image will have the same host keys. And that's the first thing. And it used to be the only thing that you had to worry about, but now there's a little bit more. I threw the commands in the chat as well. It's just a matter of, like Jay said, it's delete everything under etsy ssh, ssh underscore host underscore asterisk. You'll get rid of all those. Then you just do a dpackage reconfigure open SSH server and that'll rebuild it. When you do the dpackage reconfigure, it goes, hey, look, you said reconfigure but I can't find any host keys. So it just generates them. And then you restart SSH and it implements more. You can rebuild the system. Both of those options work. Exactly, yeah, that's exactly right. Now I mentioned that that used to be the only thing you have to do and some distributions that's still gonna be the case. But one time famously, and I think this is probably a recurring thread for me, when I encounter a very frustrating problem that just annoys me to no end and it just takes me a while to figure out, you guys see the end result of that. I'll just make a video and I'll say, yep, here's how you do it. You do this, you do this and you do that. And I'm sure I sound very confident and of course I am because I've usually rehearsed like 10 different times before I even hit the record button. But what you're not seeing is a frustration that went into discovering what it is to tell you guys to do. And one example of this, and this is getting into the thing that's kind of different now is the Etsy machine ID, which is an Ubuntu and I think it's gonna be another distribution. So I think everyone that understands DHCP knows that the way that it works is a new instance comes online, it hits the network, if it's in within reach of a DHCP server, it's gonna ask for an IP address, if one's available, it's gonna get one and it's gonna present its MAC address as its identifier to that DHCP server. Now, for one reason or another, I'm not gonna get into the reasoning here. The Etsy machine ID is now a factor in the process of getting an IP address. So what happened to me is I created this image, I was pretty happy with it. I thought I had some pretty cool defaults in there, but I noticed that my SSH connection would keep dropping. I'm like, excuse me. I'm like, what's going on here? But then I noticed another instance has the same IP address. I started creating more VM instances from that template and they also got the same IP address. And it was then that I realized that the MAC address was not the factor, I made sure it's different on each of the instances than it was, especially in Proxmox. It knows to randomly generate a MAC address, so you don't have to worry about that. And then it was when I discovered the Etsy machine ID and now if that's in the image, that's gonna be a problem and you're gonna have to clear that out. Now you can't delete the file. And that's interesting because most things in Linux, if you delete a file, it'll probably just get recreated for you which is what you might assume would happen. If you delete this file, it just, it won't be there and strange things will happen. But if you empty out the file, literally truncate dash s zero Etsy machine ID, just empty out the contents, then when the machine boots the next time it's gonna say, oh, there's supposed to be a machine ID here. There isn't, so I'm gonna randomly generate one and put it there. That's basically what system D is doing nowadays. And you also have another file and I know people are probably driving and they probably won't remember any of this when it comes to paths. But var live debus machine ID is a duplicate of Etsy machine ID in most Ubuntu server installations. One has priority over the other. But what you basically do is delete the var live debus machine ID. And this is in a video of mine, if you're curious. You just sim link that to Etsy machine ID. So you only have that one file, empty that out. Next time it boots, you get a new machine ID. Except now you won't. It'll still be the same. How is it gonna be the same? Well, the SM bios serial is the machine ID. And that's actually kind of hard coded into the VM or the instance. So, but that's not gonna be a factor though. I just bring that up. If you're testing this and it keeps getting the same machine ID, that's why. As long as you create the image or template with Etsy machine ID empty and var live debus machine ID, you know, simling to that, then any new instance you create will have a different SM bios and get a different Etsy machine ID. But I mentioned this because it's a very common question. Why are all of my VMs getting the same IP address? And before we would immediately blame the MAC address. We would assume that that's probably the same in every instance that was something we used to run into. That's largely solved now, but the actual truth of the matter is now we have the Etsy machine ID to contend with. And my big issue with this is that it's not like Ubuntu made a blog post, at least none that I'm aware of. Telling everyone, by the way, this is happening. If you don't take this into account, you might have a situation where you have the same IP address. No, it's just a bunch of people that encountered this or scratching their heads, wondering why it's happening and figuring it out independently rather than the company coming out and telling everyone what they're doing. And then of course I created a blog post about this a long time ago. It got pretty popular because, you know, a lot of people are running into this, but it's yet another thing we have to take into consideration. So now we have two things that we wanna make sure are not in the image, the machine ID and the host keys. And let's validate something here real quick, Jake. Someone in the comments here had something I didn't know and maybe you didn't know either. In that plan, if you add DHCP dash identifier colon MAC under DHCP options, then we'll go buy your MAC address instead. Yeah, I did know about that. I'm a little cautious about circumventing the design of the distribution because I don't know how much or if anything they've built around the machine ID because there's always more, especially when it comes to system D. So although that's 100% valid and 100% true, I caution again, again, going against their design for the distribution because again, there could be more to the story than that. So, but then again, I mean, we're homelabbers. So let's just play around with it and see what happens, right? We could just discover first know what it does. Because obviously the default behavior is to create that machine ID, but without a better explainer as to why. And by the way, if someone knows the answer why or you happen to be the person that developed that and you would like to email the show, have at it. Feel out our contact form. Send us an email homelab2022 at the homelabshow.com. Right, and I feel like I knew the reason at one point, but maybe I was too irritated at that point to soak that in. I wouldn't be surprised if it's a security thing because I know a lot of hardware manufacturers that ship phones or something, we're catching onto the fact that people walk into a store and it's following their MAC address as their phone is reaching out, looking for a wifi access points to track them and their spending habits and where they're going and things like that. But then again, if the machine ID is the same on that device all the time, then you'd argue there's no security benefit there. So yeah, there's that. Yeah, also I just said the name wrong. Our email is the feedback2022 at the homelab.show. So if you wanna email us questions, comments, concerns, that's where that goes, feedback2022. That one's, we're gonna keep that one valid and of course we're gonna be creating one for 2023. We just wanna know if a spammer gets a hold of it so we're putting years on that. We should, we sent it before the end of the show so we're doing better. We didn't say it at the beginning. It's gotta, I gotta put that in our notes. Now, at this point, we're getting into cloud in it territory. And cloud in it is one of those things that I think is absolutely worthwhile to learn. I've even done a video about it, but that doesn't mean I'm extra generous. It doesn't mean that I'm an expert though. Anytime I create a video, it's I'm teaching everyone at the level I'm currently at. So I usually just, you know, as my knowledge grows, I'll create more videos around it. And I've had a lot of time to go deeper into cloud in it lately than I've ever done before. Now, cloud in it for those of you that aren't aware is primarily designed for cloud providers, but it's not just for them, we can use it too. And you could use it on, you know, metal. Like if you're installing a Linux distribution on a physical piece of hardware, cloud in it is basically a way of automating first run tasks or things that should happen when a machine comes online. And one of the things that cloud in it can do, and I believe it does by default, is it will generalize your SSH host keys. So if you know how to trigger cloud in it to run, basically you trigger it to run in the reference instance, shut it down, create an image. And then when you boot up or create a new instance from that image, the goal could then be getting cloud in it to run for that very first time. Now, the problem with cloud in it is useful as it is. And there's all kinds of automations that you could add in there from installing packages, creating users, whatever you want the case to be when you bring up a new instance. The documentation is good-ish at times and lacking other times. And the complexity is a little frustrating. And I think the issue here and why that is is because cloud in it seems like a secret sauce that cloud providers use. And it was developed for them primarily. So someone who has created their own company and runs servers that are creating hundreds of VMs a minute, I think that they know their way around cloud in it fairly well, you could probably argue that they don't really need the documentation. But that's not an excuse not to have documentation. I think other people need documentation, other people want to learn it. But sometimes cloud in it is just kind of off on an island and a lot of people don't understand how it works. And to make matters worse, sometimes distributions change it up a little bit and add some customizations to it, especially Ubuntu that make it work differently than the documentation. So even if the documentation was up to date, you might still run into things. But cloud in it is very useful, but I also understand if someone made the decision that it's not for me, I'm just going to add that system descript to generalize SSH host keys and just, there's nothing wrong with that. If that solution works for you, it's valid, just do that. If you have interest in cloud in it, absolutely learn it, especially if you plan on getting into OpenStack or maybe at work, you plan on getting into managing cloud stuff. It's a good idea to try to learn that. And as I learn it, I'm going to probably keep coming out with content around it. And who knows, maybe at some point, then people just send everyone to me to learn cloud in it and it won't be an issue, but there's going to be some challenges there. One of which is the metadata. Cloud in it needs metadata. It needs, for example, the default username, default password, and you have to have a server to serve that, that cloud that can then catch, call in and make the case. It'll also silently fail if it has no way of getting a hold of that, said metadata. Now, Proxmox works around this by creating what they call a cloud in it drive. And in the cloud in it drive, it, as far as I know, and I'm going to paraphrase this here, but it seems to me the way that it's working. And again, it's complex. Is, yeah, cloud in it, don't worry about looking for a server for this metadata you're looking for. It's right on this drive, just use that. And that way you're not required to spin up a web server just for that one purpose, although you can. I kind of feel like for a home labber spinning up a web service, just for that is kind of a waste of resources. It's probably just better to use a cloud in it drive. And with that box being checked, cloud in it will then run. Well, you have to run cloud in it clean to generalize it first, but that plus the existence of the cloud in it drive in the case of Proxmox or whatever, or however your individual platform makes that available, that'll give you the benefit of cloud in it. I see a few comments on this as well, that the documentation is, yeah, the machine ID and the cloud in it install and installing Ansible, which that's a very valid way to do this as well. So you can use your cloud in it to get your Ansible and then get your Ansible Pulse setup. So it pulls everything down there. Absolutely that will work. And it does work, because that's also how I do it as well, especially on Raspberry Pi. Raspberry Pi, cloud in it works very well on Raspberry Pi. It's actually kind of crazy to me how well it works. It works really, really well. And I just have this cloud in it config that I keep personal and I've only changed like two things, like the default username, and then also the command that pulls Ansible is the only other thing that I've added to it. And whenever I restore an image or Raspberry Pi, like if I download Raspberry Pi OS or Ubuntu for the Pi, I just drop that right into Etsy cloud, cloud.cfg, the one that I curate, and then it just does everything. And then I don't worry about it. I get a message in my phone, machine provisioned. I know it's been provisioned, I can log in. I do feel though, that it's complicated as cloud in it can be. It's still fun, well, maybe not to everyone, but you will get a feeling of satisfaction and also feeling like you have more power because you have more tools at your disposal for getting these images taken care of. The cloud in it also is what makes the complexity nowadays of creating deployment images for Linux machines about the same complexity as I remember Windows to be back when I used to manage Windows machines, but it's not insurmountable. It's actually probably not as hard as it seems when you learn it. And you'll probably have a mindset of, oh yeah, that doesn't seem so bad. But then again, considering some distributions add their own things to it, you might run into some things that are gonna take you a minute. Like for example, last weekend, I had an issue where every Ubuntu instance I created from a cloud in it provisioned image would have zero networking out of the box. No IP address, nothing, literally nothing. And that's when I found the hard way after looking through, again, you guys are getting the benefit not going through the pane I went through. And there was a file called, well, it's actually in a directory called clean.d inside Etsy cloud. And that directory contains scripts that'll run when you run the command cloud in it clean. Cloud in it clean again is how you generalize it an instance. If you want to utilize cloud in it, what that'll do is it'll remove the state of the instance that cloud in it has run. So at that point, the next time it boots, cloud in it is gonna say, hey, I've never run on this machine, even if it has. It's gonna think that it hasn't and it's gonna go ahead and do everything it needs to do. So you have to run that command. But when you do so, everything in that cloud, excuse me, the clean.d directory will be run as well. Now that's a pretty cool thing if you think about it because you could just put some scripts in there if you want some more things to be done to the image. But then what I found out is that Ubuntu had a command to delete your netplan.yaml file, just knock it out. And sure enough, every instance I created at CNET plan was empty. And it was hard-coded right in clean.d, right off of Ubuntu server. If this file exists and the name is, you know, zero one, no, zero zero installer.yaml, something like that, it's gonna delete it. Now, I think the reason why is because cloud providers generally have their own system for assigning network configuration and they probably want to avoid a situation where you have a default netplan file plus the cloud provider's, you know, network configuration on top of that, which could probably cause some interesting things to happen. But, you know, again, we run into an issue where cloud in it, as a project, they kind of seem to not understand that creating deployment images is an industry-wide thing, cloud or not. Sure, it has the biggest use case in the cloud, but people still provision VMs internally and they want to generalize things that needs to work for them. So I feel like this mistake that's clearing out network config is just not taking into consideration that, you know, it's used for other things. However, when Tom and I were chatting last night, they must have fixed it because I couldn't reproduce it again. So right now it's working, even if you run cloud in it clean, it works fine. I think how many times do we try that at least twice, if not three times last night to reproduce that? And it was 100% reproducible for me last weekend. Anytime you run cloud in it clean, your network config is gone every single time and now it works. So there's that. That's the thing we run into with HomeLab is that, you know, we have a process that we've gone through a lot and then one day it doesn't work, why doesn't it work? Well, there's a bug now and something that's causing it not to work, but that's one of the trade-offs that we go through. Right, and do take the time to fill out those bug reports because I thought I was doing a video on that because I have, well, I'm gonna go through my testing process probably in a video soon where I show how I find the bug. Well, the bug, you know, I kind of found me, but I'll show how I produce a lab environment where I can reproduce it. And my goal is to always give the developers the accurate answer. Like this is the system, like here's a clean system. These are the steps that reproduce that particular bug. That way, you know, they understand the scenario. We found it in a production scenario which is where we found the bug, but we're not 100% clear on it yet. So I'm actually gonna reproduce it in our lab. It's some rule creation in PF sense, but it's an edge case, but I know enough people probably have this edge case and I'll solve it for them. So always think about that when you find these bugs is making sure you file good detailed bug reports that's what gets the developers, you know, they really wanna solve the problems and not have them just lingering.  But there's some optional things you can consider. If you decide to do none of what I'm mentioning, that's fine, doesn't matter. It just matters how much do you care about some of the particulars. And specifically for me and what I'm talking about here is clearing out things like bash history.  And do you really want that, you know, every instance to have, you know, for that user have a bash history that's there from like before you took the image, it just seems kind of weird. So what I do is I just delete the dot bash history file for every user. And then what I'll also do is look for any file. So like find, you know, dash type F you wanna look for files in the var log directory and just pipe that into truncate that dash S zero, be careful. And that'll empty out all the logs because in my opinion, what you don't want is logs from the process of creating the image because that was from before. You want things to pretty much start out as a clean slate. So emptying out log files is not outside the norm.  If it's absent, which is kind of rare, most will, but then some won't, which is the reason why I recommend truncate with the size of zero because at least then you're not gonna run into a situation where you need logs from a service just to find that it hasn't ever been logging because the file isn't there and it doesn't wanna create it. So again, rare, but it happens. Another thing you might wanna do is go into your distributions package cache locally. So I think it's like varlib, EPKG or something like that off the top of my head in Debian and Ubuntu. So anytime you run like apt install, just upgrade, it's gonna download the Debian or dev packages into that directory. You could run sudo apt clean to make sure that those get wiped out. There's just, it's just a waste of space. The image is gonna be bigger and you probably don't want that to be the case. Just wipe that out. You can always download everything later. So that's probably another thing you might want to clear. And then finally another tool I want to make everyone aware of is PyShrink, which I just love. And this is specific to people that might want to create a deployment image for Raspberry Pi. And what this will do, let's just say for example, and this has happened to me, I had a 128 gig SD card for my Kubernetes controller. And when I ran the DD command to back it up, I also had a 120 gigabyte image file. Now, forget the fact that probably two or three gigs were being used most, but DD is DD. It's gonna grab the entire thing. What PyShrink will do is it'll actually truncate the unused space that's in the image, knocking it down to a ridiculously small size. But what it'll also do is automatically trigger the service that runs to check that the installed image is taking up the entirety of the hard drive. So that way you don't have a situation where you have 128 gigabyte SD card, but like three gigabytes maximum usable space, but you're missing out on the majority of your space there. So what PyShrink will do is set it up that when it first boots, however much free space you should have, it's gonna expand the boundaries of the restored image to that so you can take advantage of that. So it's been an indispensable tool for me, especially as I back up my SD cards. And sure, you could just G-zip your image and that'll knock down the size too, but it's still gonna take a long time to restore. May as well just use PyShrink. I think it's a good tool and it's specific for Raspberry Pi purposes. Yeah, someone mentioned and I pulled the website up because it actually looks cool. So dietpi.com and it's actually supports quite a few different single board computers. It's a very minimalized install of highly optimized minimal install of Debbie and OS. So that's actually rather clever. So if you're looking for something else, thank you for that suggestion there. That was definitely great. I'm curious that the one I used to use is even still around. I haven't heard of anyone say it in a long time. I doubt it. I used to use something called, if I'm even saying this right, Minibian, like M-I-N-I-B-I-A-N. I'm not even sure. This isn't a recommendation because it's been a long time but that used to be my go-to. Nowadays, I just do things the hard way. I just have Ansible remove everything that shouldn't be there. So regardless of what it starts out as, it's gonna become what I want. But there is definitely a use for it. Jay froze for a second there. But yeah, Jay froze for a second there. But I already can finish your sentence. Jay does use Ansible very well. He's got a whole video on that. I'm using Ansible Poll and then customizing it, including not just installing the things you need but removing the extra things you don't. So kind of minimizing the install a bit. Yep, that's exactly right. So I guess all I'm saying is I'm not the use case for it, but then again, I think there is a valid use case for something like that and absolutely something that you might want to look into as an option. And if you're lazy like me, and I'll reiterate how a lot of what I do, I have an Ubuntu server 20 and a server 2022 and whatever the subversion is. Those are both running with the name template sitting in my stack of all my other hypervisors that I have running. I have unattended upgrades running. So they're always pulling whenever there's a cycle that goes through of whatever updates are needed. And it just loads, this is a template. That way anytime this week, next week, three weeks from now when I go, hey, I have this idea. I want to load Portainer. I go grab that template and maybe I'll do one more update but it's usually already up today just in case then I'll stop that template, shut it down, fork it and let it live again and then take that fork and build my idea, play with whatever I want to play with. And it's always just running it up to date. For me, and this is simplicity for people running if you want to own the whole lab, all the different more complicated ways we talked about are great. Now, if you're running a large scale dev team and things like that, yeah, you're going to want to have automation, there's even Packard, there's all kinds of different methodologies for having ready images made. But you can also just leave one running with unattended upgrades. And because it's idling, it's not doing much. It doesn't take up much CPU power. It doesn't need to have even that many cores. You can assign it like one core and a gig of RAM is usually enough. It just can sit in the background doing its thing, pulling those updates. And then you just clone with your hypervisor whether that's Proxmox or Xtp and G doesn't matter, they both have the same option. You can then do a full clone or a linked clone depending on how you want to do that. And then you have a ready template all the time. Cause I like the things in my template. We, one, not just having the package up to date but two, we have a common user at my office that we have on there that the employees log in as, it's just a lab user. And it's just already set up on those templates. Now, if I'm doing something for production, I usually don't pull from those templates. Like if I decided I'm going to do a new install of something production. I generally like to start from like a new install scratch. I'll grab the latest ISO of the distribution. And then I have my customizations I'll put on from that. I kind of like going through that raw install process. It just kind of makes me feel more engaged and involved. We don't replace production service very often. That's something it's, it's kind of like a one-off rare thing. So I like to just go through the process again on there. So there's a couple of different methodologies whichever one really you're comfortable with on there. As long as you're doing some of the important things and one of the things like Jay said, and we, you know, I threw that little a couple of commands that are resetting SSH host keys, resetting the machine ID, or as someone noted, putting a net plan to you make sure you're not pulling it so you get a different MAC address presented for DHC client. Cause you don't want to have same MAC addresses as your template that would cause drama. Make sure you've got those bases covered cause those basics will cause you some headaches when you have your template still running and all of a sudden the other thing looks just like it. And then the other thing I'll mention is probably the last thing for today for me is what do you do after? I have a rule at my company, I know it's just me, but we'll just pretend I have a whole team of people. I will eventually, but I also had this rule when I managed people before I went on my own that nothing is allowed to be accessible from the public internet unless it absolutely has to be. Well, that's pretty obvious. We've said that a million times, but also it goes through a security check. So before I allow anything that is supposed to be accessible from the public internet to be accessible, then I'm going to make sure everything is updated. You know, I understand what ports are listening, should they be listening, and I just check everything and then I will allow it to be publicly accessible. But like it's even to the point where if I'm working with someone else, which is generally going to be probably remote, maybe we're tag-teaming an instance or something, I might say, well, what's your IP address? And I'll whitelist their ability to get in. I still won't make it accessible from any IP regardless until I know that it has security standards in place. Obviously, you don't want to go too far down that rabbit hole because there's no end to it, but you definitely want to check some boxes updating being the obvious one. Closing ports should also be obvious, but then also just kind of take a look at it, make sure that everything checks out and then if you need to make it publicly available, you can, but then again, if you could get away with nothing being publicly available, even better. Yeah, someone else drew in the comments here, cloud init, space clean, tac-tac machine-id-sl. Yeah, I tried that last weekend. I'm trying to remember what happened because it didn't work, but I don't know why. I don't remember why. And this is another issue about the documentation. My understanding is, maybe this is wrong is that cloud init clean will clean everything, but if you add an option, it's going to just go after that one thing because there's also dash-logs as well. So you could wipe out the logs by adding that as an option to cloud init clean, but I'm going to have to look into that to be sure that clean is inclusive, which I thought it was, but again, maybe I'm wrong here, but if there is something that you want to clean in particular, then that's one way to do it. But again, I just wish I remembered what went wrong when I tried that. I spent all day figuring this out last weekend, so I probably am missing some details. So apologies for that. More testing needed on that. So we look forward to hear some feedback from any of you that have that. And thank you, Justin, for that suggestion on there. I've seen Justin also mentioned he used Arch. Were you trying that cloud init clean on an Arch machine or an Ubuntu machine? Jason, Jay was doing them all on Ubuntu, so. Yep, yep. I haven't run Arch on servers in a while. So right now it was just Ubuntu. And I was also running into Ubuntu-isms that were, again, specific to that distribution, not in the cloud init documentation. So that was further complicating things, too. Justin says he was using Arch, so there goes some more testing for you, Jay. There's the next video Jay can work on, cloud init on each different distribution and why it's not the same. You know, now that Arch was mentioned, I feel like there's just this stigma around rolling distributions that I don't think is fair. I would absolutely run a server on Arch. No question about it. Sure, you're gonna have a little bit more work sometimes, but the whole idea of Arch is to install as few packages as possible to serve your goal. So as long as you follow that, you don't have like 10,000 things installed, especially a bunch of things from the AUR, and it's just the things you absolutely need, especially considering you could literally just install something on your Arch server just to run a container and everything runs in containers. They don't even have any extra packages beyond that. You can absolutely have a stable rolling distribution and then benefit from literally never having to reinstall everything the next time a new Ubuntu release comes out. But again, everyone makes the assumption that rolling is bad for stability. No, just means you are in charge of the stability. If you're up to that, then you should absolutely do it because you are benefiting from not having to reinstall. And that's why we also see CentOS, CentOS Stream, Amazon Linux is, you know, semi-rolling. So it's taken off and I think it's gonna be very common. But then again, if you want something simple, Debbie and Ubuntu, there's Fedora server, there's a number of others that might just be a little bit less chaotic. Yeah, all right. Well, this was definitely a fun talk and we sent some homework for a few people. So they got a few different things they can try. And so do we. So I think there's gonna be a lot of things in here. So I haven't done a video at all on cloud on it and XCPNG and it's been kind of my two lists because to kind of talk about that, I just don't use it, but I want to. And so I've got a kind of a goal set for that. And there's been some discussion in the XCPNG forums on cloud on it. So that'll be my source materials for it. But thank you everyone for joining us. And we look forward to hearing from you. We wanna do another Q&A episode. We think we're doing, me and Jay, are we gonna do an episode we didn't talk about this for on that between Christmas and New Year's? Is it, was there any plans or? I don't, yeah, I don't think I'm gonna be available. I mean, things are a little up in the air because there may or may not be a blizzard coming. Yeah. If I knew for a fact it was coming, I'd say, yes, I'm available because I'd have nothing else to do. But I think it's probably safe to call it like next week, probably we'll be a break. Yeah, okay. We might be a break next week. Just let everyone know. That means we won't be back till after the first of the year, but we look forward to hearing from all of you. And thank you for all the supporters of the channel. Leave your comments down below or hit us up on the socials. Thank you very much. Take care.


Today, we're going to focus on why the United States suddenly became interested in acquiring overseas territories at the end of the 19th century. We're going to begin by focusing on America's changing perspective from a continental nation to in a ways an intercontinental nation, a nation that wanted to acquire control over this vast territory stretching from the Pacific to the Atlantic Ocean. We're going to begin by focusing on changing policy concerns from continental concerns to intercontinental concerns. We're also going to look at growing American concerns about competition with European nations. And finally, we're going to focus on why and how the United States, it begins to expand from the continent of North America into the Caribbean and Central and South America, as well as into the Pacific in East Asia in the closing decades of the 19th century. American foreign policy prior to the 1890s had been focused primarily on dealing with affairs within boundaries of the United States itself. In many ways, this focus was on defending against foreign invasions, resolving conflicts with our immediate neighbors, British Canada, Spain, and then later independent Mexico, and ultimately sort of ensuring that the United States would be a strong and united nation as Americans gradually expanded westward across the Mississippi River and ultimately began to fill in the boundaries of the United States itself. And really sort of a final focus during this period was enforcing the Monroe Doctrine. And the Monroe Doctrine was a policy that influenced American foreign affairs during the 19th century that essentially argued that the United States would vigorously defend against any efforts by European nations to establish new colonies in North America or South America. And that we felt that the age of colonization had come to an end. And those were really the factors that influenced American policy during much of the 19th century. But beginning in the late 1880s and 1890s, the United States begins to grow more interested in issues growing on in the Atlantic, in the Pacific Ocean, and elsewhere in the world, especially within the Caribbean itself, the issues that were going on in the islands and countries of the Caribbean. And we begin to see a renewed focus in expansion from westward expansion, growing west across the continent, to international expansion, expansion into the Caribbean, to the Atlantic, to the Pacific Oceans, particularly for a number of reasons. One of these was simply economic, that American businesses and business leaders were very concerned that American businesses would be cut off from expanding into countries in the western hemisphere if we allowed European nations to gain a foothold in these nations. So as a result, American economic leaders pushed the political leaders in the United States very hard to increase American political interaction with nations of the Caribbean, Central and South America, East Asia, for instance, during this time period. The American Navy was also very concerned with expansion. American Navy was growing significantly during this period. And the American Navy wanted to gain control over a forward operating basis once again in the Caribbean, once again in the Pacific Ocean, as a way of using the Navy to support America's commerce with Central and South America, with East Asia, and so forth. And finally, there was a growing emphasis on a much more sort of militaristic enforcement of the Monroe Doctrine. Now, throughout the 19th century, the US government used politics and policies as a way of sort of enforcing the Monroe Doctrine, but we began to grow more interested in using military force as a way of preventing European nations from either trying to establish new colonies in the western hemisphere or interfering with newly independent nations of the western hemisphere as well. And so it's really within this broader foreign policy context that we begin to see growing conflicts between the United States and European nations at the end of the 19th century. And the most violent of these foreign conflicts is the Spanish-American War, just a brief context about the war itself. The war ultimately has to do with Spain's collapsing empires. Spain had lost control of Mexico, had lost control of Central and South America during most of the 19th century and only retained a handful of possessions, the island of Cuba, island of Puerto Rico, and the island of the Philippines and Guam in the western Pacific. And the United States was very interested in these territories, especially Cubans, being just 90 miles off the coast of Florida. Well, Spain had lost all these territories as a result of independence movements during this time period, and these countries had broken away and gained their independence. And only Cuba, Puerto Rico, and the Philippines and Guam had managed to remain part of the Spanish Empire. But over the course of the 19th century, increasingly Cuba wanted to break away, and there were a number of attempts at revolutions in Cuba. By the 1890s, another revolutionary attempt had begun, and Spain responded to this by sending over harsh Spanish officers who used all sorts of violent measures to try to crush this guerrilla war and this rebellion and restore civilian rule within the area. And for Americans, they looked at this efforts by the Spanish and they saw it as misrule. They saw it as horrible attempts to kill the population in order to reestablish control of the nation. And Americans were growing increasingly frustrated with Spanish efforts to regain control of the population in Cuba, including such things as concentration camps and killing people in some ways indiscriminately. And as a result, the United States was growing increasingly interested in doing something about this struggle. And Cuban revolutionaries were certainly depicting Spain as this evil nation, somewhat like Great Britain had been before the American Revolution, who didn't respect the United States. And we're trying to get Americans interested in supporting these brave freedom fighters in Cuba in their struggle for independence against this sort of evil Spanish government. At least this is how the Spanish and the Cubans depicted it. This was how many American journalists depicted this struggle. America's president at the time, William McKinley, attempted to mediate the conflict in order to satisfy both sides, but it proved to be fairly unsuccessful. Well, America had managed to stay out of this conflict until the year 1898. And it's in 1898 that we have a series of crises that ultimately pull the United States into the war. And they began in February of 1898 as a result of a letter written by Enrique de Puede de Lom, who was a Spanish minister in Cuba. And he wrote a letter to his government. You know, we have pictured on the right here in which he called President William McKinley, America's president at the time. He called him a weak president and basically said he was really sort of an unfit leader and not particularly interested in actually mediating this conflict. Here we have McKinley depicted. And as a result, this letter became public at least. Journalists got ahold of it and they were infuriated. Here we have the Spanish government insulting American American president and basically calling him weak and not somebody who was an effective leader for the United States. And this just infuriates Americans. And then just a few days later, another incident really seals the fate of the relationship between the United States and Spain. The battleship main, this beautiful battleship that the United States had just completed is destroyed in a violent explosion in Havana, Harvard on February 15th. The U.S. suspected Spanish sabotage or they suspected an attack of some sort. And they put a huge amount of pressure on McKinley and a huge amount of pressure on Congress to respond to this attack. And McKinley is sort of forced to go to Congress in April of 1898 and ask for a declaration of war between the United States and Spain. Spanish attacks and Spanish efforts to attack the United States in the form of the battleship main had convinced Americans that the time was right to deal with Spain. Well, America began to prepare for this attack. And of course, the United States had never fought an overseas war before. The last major war we had fought had been the Civil War. And as a result, American military was not particularly prepared for this conflict. And the early preparation, the first month or so of preparation, largely amounted to simply trying to get enough troops to Florida in order to prepare for the invasion. And we have some photographs taken from Tampa, Florida that show the camps that were set up and the efforts to load weapons and guns and supplies onto American ships and figure out a way to transport all these supplies to Cuba. Again, we'd never invaded an unoverseased territory before. And American military was really not prepared to take action against Spain. Well, eventually, military forces were organized. And in early June and during the month of June and July, American forces set sail around the island of Cuba and eventually landed forces near Guantanamo Bay, Cuba with the assistance of Cuban guerrillas. Probably wouldn't have been a successful landing without the Cuban guerrillas really paving the way for the invasion. And in a series of short battles that take place after the landing, including the Battle of San Juan Hill that we depicted here, the American forces achieve a pretty profound victory over the Spanish forces. It's a tight battle, but thanks to the efforts of a number of organizations, including Theodore Roosevelt's Rough Riders, which was a group that was organized by Roosevelt, who later goes on to be president. Here he is depicted in this black mic photo. And, of course, along with the Rough Riders, he's ably assisted by a number of regular army troops, including African American soldiers of the 9th and 10th U.S. cavalry and 24th and 25th U.S. infantry regiments who really did the bulk of the fighting, suffered the bulk of the casualties and really helped Theodore Roosevelt and his Rough Riders achieve kind of the ultimate victory. And of course, in many ways, Roosevelt and his Rough Riders become the famous fighters of the battle and we largely forget about the African American troops of the 9th and 10th U.S. Cavs and the 24th and 25th infantry regiments. Well, ultimately, despite the poor organization, despite poor leadership in many ways, despite poor food, poor equipment, in spite of all these obstacles, the U.S. manages to win the war in the Philippines. And we occupy the Philippines by July of 1898, excuse me, the war in Cuba. Now, the Philippines, the other side of the conflict, in the Philippines, the U.S. Navy had been dispatched to the very beginning of the war by Theodore Roosevelt. He had been the assistant secretary of the Navy before he resigned to become a Rough Rider and he had dispatched a Maudui in the American fleet to the Philippines to destroy the Spanish fleet and take over the Philippines. And Dewey does so in a very important battle in May of 1898. Utterly devastates the Spanish fleet. Americans barely suffer a scratch. And as a result with the battles in the Philippines, the battles that take place in Cuba, a landing in Puerto Rico, as well, by the summer of 1898, the United States has acquired control over Puerto Rico. We've liberated Cuba from Spanish rule. We'd acquired the Philippines in Guam, and we had begun to stretch America's overseas empire all the way from the Atlantic to the Pacific Oceans. And in a very short time period, it was this dramatic, dramatic outcome. The Spanish eventually, formerly, surrendered these territories in December 18 of 1898 in the Treaty of Paris. And the U.S. gains permanent possession of the Philippines in Guam. We gain permanent possession of Puerto Rico. And we gain an influence over Cuba as a result of an important amendment to the Cuban Constitution called the PLAT Amendment, which gives the United States the authority to intervene in Cuban affairs anytime we wish for any reason necessary. And the end result of this conflict is that the U.S. acquires economic, political, military influence in the Atlantic and the Pacific, especially in the Caribbean. We gain new base for naval operations in Cuba, at Guantanamo Bay. We gain bases for naval operations in the Philippines and Guam as well. And we use these bases to support American commerce in Central and South America, support American commerce in East Asia, Japan. And as a result, we gain the United States gains a tremendous advantage in this very short war in that we expand American influence nearly halfway across the globe. And so for the first time, the United States has to begin considering itself an imperial nation, a nation that controls overseas territories. But for the United States, they wanted to distinguish themselves from European imperial nations. Americans argued that American imperialism brought about freedom, brought about democracy and was ultimately a benevolent form of imperialism. The reality was American imperialism was very little different than European imperialism, especially in the Philippines, where Filipino independence fighters fight against Americans for independence of the Philippines for nearly four years. And as a result of this conflict, probably about 100,000 Filipinos are killed in this war and about 4,200 Americans are killed in this war. So the reality is, while Americans might have claimed that this was benevolent imperialism and it was imperialism that was for the benefit of the people who were started to put under American rule. The reality was it was a violent, violent struggle, ultimately for the benefit of American political and economic figures and really not for the benefit of those who were colonized by the United States as a result of the war between the United States and Spain in 1898.


Welcome back, in the previous segment we saw how an array could be passed to a function. Now we are going to use those ideas to build a somewhat more elaborate function. So this function is going to sort an array, so the input is an array containing numbers. The output, well the output is going to be present in the same array and you can think of the goal as being to rearrange the numbers so that they appear in non-decreasing order. So as an example, suppose the array initially is 35, 12, 29, 70, 18, 29, then at the end we want this desired order. So 12, 18, 29, 29, 35, 70, okay, so how do we do this? Well before we say how do we do this, I should point out that this is an important operation and chapter 16 will give us a clue as to why this is the case. So we are going to do that, so we will just wait until then. There are many algorithms for sorting and chapter 16 will discuss a clever and fast algorithm. Here we discuss a slow but an easy to understand algorithm called selection sort. The basic idea of selection sort is find the largest number in the array. Then we exchange it with the element in the last position. Now we have made progress, why? Because the last position now contains the largest number which is really what we wanted, the last element of the array to indeed contain the largest and we have placed it there. So what happens now? We just have to do the same thing for the first n minus 1 elements. So we apply the same idea to the first n minus 1 elements of the array where n is the length of the array, then to first n minus 2 and so on. So a primitive step in all of this is finding the index of the largest element. Well we said we want the largest element but actually it is more useful to find the index of the largest element. Where is that largest element present in that array? So here is a function which does that. So this function takes as argument an array, well an array name but as you know it is the starting pointer to the start of the array and the length of the array. But we can think of a as an array name as we saw that when the parameter passing values are copied it really behaves like the name of the array. This function is going to return the index of the largest element in a. So what is the invariant for this iteration? Well so for the i-th iteration the invariant is that max index which will be a variable that we will define soon will equal the index of the largest element in a0 through i minus 1. Or I should say a index if there are multiple elements which are maximum it will be the index of one of those elements. So we start by setting i equal to 1 and max index equal to 0. So this makes the invariant hold why because if i equals 1 a0 through i minus 1 is just a0 and max index is indeed the index of the largest element. Well there is only one element a0 and its index is a fine index of the largest element in a kind of trivial but we will do. Now we are going to increase i until l minus 1 and as we increase i we want the invariant to hold. So this is like our max so far but instead of keeping track of the max we are keeping track of the index. So a of max index is max so far and if this is smaller than the new element then we are going to set max index equal to i. So either way max index will point to the largest element or be the index of the largest element and so at the end we return max index. I should note that in doing this the number of comparisons needed is going to be l minus 1. So for each value of i going from 1 to 1 to l minus 1 so l minus 1 comparisons will be needed. We are later on going to estimate the time taken for this and this value will be needed over there. So what is the main function and the main program? So the main function is the function selection sort. It again receives the name of the array which is the address of the 0th element but which we can pretend is the name of the array because the name of the array means the same thing and the length of the array. Then we are going to start with n. Here we said that we are going to place the largest element in the nth position then the second largest element in the n minus 1 position and so on so that is what this loop is for. We first calculate max index which is cos of max of ai. If you remember cos of max was going to return an index of the maximum element in the array a of length i. Now our array a actually has length n. The first iteration this will be n but in the subsequent iterations this will be smaller. But if you remember according to our discussion in the first iteration we wanted to find the largest element in the entire array. In the next iteration the largest element was already in position n. We wanted to find the largest element in the first n minus 1 positions and that is exactly what this statement will do because in the next iteration i will have come down one step so this will be n minus 1. So this is exactly what we want. So cos of max this function thinks that the array only has i elements but that is okay. So it is going to only tell us the index of the largest element in the first i elements that it is going to consider from this array. Now we want to do the exchange we want to exchange the i minus 1th element with max index okay. Why i minus 1th element well in the i-th iteration we are going to consider i elements okay. So this is element 0, 1, 2 and this last element is i minus 1. So we found some max index over here and we are going to exchange the element over here and the element over here. So what this will do is we already before starting would have had the correct values over here okay the largest values over here. Now the largest value in this entire region is going to be pushed to this point and so the good region will extend a little bit more that is what this is doing okay. So we need to do this exchange. So how is this exchange done well the exchange is simple enough we copy the value at max index to max value. So this is max index so this value goes into this variable called max value okay. Then we copy the value in i minus 1 into this position and finally we copy the value over here into this position okay. So the exchange is done okay. So that is it that is our selection sort okay. So in our main program we will create this array and then we will call make us call to sort okay. So we will print it but I will show that printing step when we do a demonstration okay alright. So before we do the demonstration let us review what we have done a little bit okay. So this is our function and let us say let us try to count how expensive how much time this whole thing takes okay. So we said that cos of max our function will perform l minus 1 comparisons where l is the length of the array pass to cos of max okay. So this is what I am talking about so this red part this red part will require me to do l comparisons l minus 1 comparisons okay if the array has size l. So if the array has size i or if this argument is i it will do i minus 1 comparisons the first iteration it will do n minus 1 comparisons the next iteration it will do n minus 2 comparisons n minus 3 comparisons and so on okay. So Celsot calls pass of max for l equals n n minus 1 2 so the number of comparisons it does is n plus n minus 1 all the way down to 2. So it is n plus 2 times n minus 1 upon 2 okay so just about n squared by 2 comparisons. So even if I just count the number of comparisons Celsot will do about n squared give or take a factor of 2 okay and n squared turns out to be not such a fast algorithm. You can do far fewer comparisons and we will see that a little bit later okay but anyway this algorithm will sort correctly and now we are going to see a demo in which that sorting is going to happen. So let us take a look. So this is our function okay so this is our code for pass max I have added a print statement because I would like to print the array at different point so we will see where this is going to be used okay. So let us just take a look at the Celsot function okay. So I have modified it a little bit and the modifications are stored shown by these stars. So what I am doing is at the very beginning at the beginning of each iteration I am going to print out the entire array. So this way you can see how the array is going to change and to make sure that okay so then this pass of max is going to be called and I am also going to tell you where that index is what index was reported. So again we can check how pass of max is working how the entire thing is working and I do not want the iterations to go off very fast okay so here I am going to just have a dummy variable okay and I am going to read a number into it I am not going to use this but this will just force C++ to stop at this point and wait for me to type in something so that I can continue. So the value of the dummy variable is not important over here I will just put in so that I can force C++ to stop okay. So the rest of the code is exactly the same and at the end also I am going to print out a message and I am going to print out the value of the array okay. So we had the print function at the top and that simply prints out the value of the array value of all the elements in the array. So let us execute compile and execute this. So let us run it okay so this is the print out at the beginning of the first iteration of cell sort and then it so this is at the time when nothing has been done we are just beginning the very first iteration and we have issued max pass and it says that the maximum value is found at index 3 let us check if that is true. So 0, 1, 2, 3, 70 is at index 3 and it is indeed the maximum value. So max pass has behaved correctly. So let us just type in some nonsense value so that the program will continue okay. So now you see that the second iteration is being executed and at this point 70 has already gone to the end because of our exchange and we have also issued the call to one more max pass and that is limited to this region and in this max pass is found at index 0 which is indeed true. So again let us continue okay and now 35 has gone to the end and now we are finding that max pass is found at index 2 0, 1, 2 yeah there are 2, 29s but it has picked one of those and so and it is indeed at this position there is indeed the maximum a maximum value. So let us again make it go forward so this time 29 has gone to the end well we do not know whether it was exchange or not but presumably there was an exchange but now again a maximum is found at this position okay. So let us again type 0 so it has gone forward and this time it is really searching within this region max pass is being found within this region. So the largest value is over here and so index 0 is returned by max pass and so at this sorry let me type 0 so at this point the function is returning and after returning to main program 12, 18, 29, 29, 35, 70 is printed and it is indeed correctly sorted alright. So POS of max will perform L minus 1 comparisons where L is the length of the array passed to POS of max okay. So we saw all these things okay we saw the demo and we can do a variation we can express sell sort as recursive program okay. So I am going to leave this I am going to show you the code but I am going to leave this as an exercise for you okay in fact I am going to put out the code as well and you can run it yourself and check it okay. So that essentially concludes this lecture and so I would like to make some remarks and in this lecture we saw what sort of happens behind the scenes when we access arrays okay and we said what does an array name mean and what happens for the indexing operator and how do you pass an array to a function okay and we said that when you pass an array to a function it does not copy values but it merely copies the address of the zeroth element and if you want to access the entire if you want the function to access the entire array you should tell the function how many elements the array has. So that also must be passed okay writing functions on arrays is useful and this is definitely a skill that you must master the selection sort runs in time proportional to the square of the number of elements being sorted and in chapter 16 we will discuss faster algorithms. I will stop here but I will urge you to solve the problems at the end of chapter 14 and of course read chapter 14. Thank you.


What's going on everybody welcome back to another draw preview and we have the Barcelona open an ATP 500 event on the calendar It's a big tournament for the clay court season. We didn't get it last year It got cancelled of course, but we have it this year. It's a big names are playing but some big names have pulled out All right going to the withdrawal list. We have Grigory Dmitrov He's pulled out Harris Nick Kerios is also out Opelka and Struth has also pulled out of this one All right starting at the top of the draw We have 16 seeds and all 16 seeds get a buy in the first round Rafa Nadal He's the number one seed he gets a buy in the first round He'll take on the winner of Manorino or a qualifier in the second round Then we have Nishikori versus Palla in the first round. That's a good first round matchup winner That's gonna take on number 13 seed Garin in the second round Then we have the number 12 seed Hashinov into the second round He'll await the winner of Nori versus Caruso who are playing in the first round Then we have a qualifier versus Herbea winner of that's gonna take on number 8 seed David Goffin So one of those tournaments where everyone gets a buy in the first round So we're not gonna get too many good matchups to watch on day one, but some plays to watch out for obviously We've got to watch out for Rafa Nadal also David Goffin He had a good week in Monte Carlo last week and Nishikori He's done well at this tournament in the past so keep an eye on him as well second quarter of the draw now We've got the number four seed Schwartzman. He gets a buy in the first round He'll take on the winner of Wildcard Alcares or Tiafo in the first round circle That one is a popcorn match huge first-round match and another chance to see baby Nadal in action Then we have Coppa versus Mutay winner of that's gonna take on number 16 seed Dan Evans Who is in Korea best form beat Novak Djokovic last week in Monte Carlo eventually lost to City pass so keep an eye on Evans Of course, then we have the number nine seed Finini He'll play a qualifier in the second round no matter what because we've got two qualies going at it Then we've got Thompson versus Gascade the winner of that's gonna take on number six seed Carino Buster in the second round so some of the players we've got to look out for obviously We've got to look out for Dan Evans. He had a great week last week Can he back it up this week Diego Schwartzman made the semi finals of the French Open last year to see how he can recover some of his form After a pretty poor week last week and I'm very keen to see how baby Nadal aka Alcares does this week in his home country of Spain All right bottom half of the draw the top half of the bottom half We have the number five seed Batista Ragu he gets a buy in the first round He'll take on the winner of Simón and then Dooha who are playing in the first round Then we have Songa versus Jerissimov winner of that's gonna take on number 11 seed Yannick Sinner in the second round then we've got the number 15 seed Rude buying the first round He'll take on the winner of Ramos Vanoles or a qualifier in the second round Then we have Pear versus a qualifier winner That's gonna take on number three seed Rublev who made the final in Monte Carlo last week He is playing Stefano since he passed in that final later today So definitely want to watch out for but also watch out for Rude, Sinner and RBA because they've all proven that they can play On the clay courts and in terms of unseated players I'm very interested to see how Joe Woodford Songa does because we haven't seen him for a long long time All right final quarter of the draw we have number seven seed Dennis Shabavala first time seeing him on clay for the year He gets a buy in the first round He'll take on the winner of Shadi versus Basilash Vili Which is a very tough first-round match and potentially second round match for Shapo as well Then we have Lopez versus Moussedi winner That's gonna take on number 10 seed Ojea Eliassim in the second round Then we got the number 14 seed Diminor buying the first round He'll take on the winner of Fakina or Bublek in that second round Then we have Montiero versus Munea winner of that's gonna take on number two seed and Monte Carlo finalist Stefano Sinti pass So there are some big names here a lot of next-gen names here, but you got to keep an eye on Sinti pass obviously I want to see how Shapovalov does this week as well because we haven't seen him on the clay in 2021 And also keep an eye on Moussedi as well because he has proven that he can play on the clay If you can get through his first round or two could cause him damage But I feel this is the most stacked part of the draw full of dangerous players and a lot of good clay quarters as well So there it is the Barcelona open for 2021. It's got some big names playing. It's a stacked event It's an ATP 500 event. So it's worth a lot of points. So obviously all the big names are playing Let me know down in the comments below Who do you have winning this event because we saw Rafa lost last week didn't look great in his loss as well When he did lose to Rubelev and we obviously have Rubelev and Sinti pass who did well last week playing in this one So let me know down in the comments below. Who do you have winning this one? This is an interesting one for me to pick because there are a lot of players that are informed obviously from Monte Carlo last week I was like Dan Evans, Kasper Rudd, Rubelev, Sinti pass, and then you've got Rafa up the top But I'm gonna have to go with Rafa Nadal. I know it's the easy pick I said that last week when I picked him from Monte Carlo didn't play out that way But I feel like in front of his home crowd in Spain that Rafa's gonna pull through and he's got a really good draw The top half of the draw the only real worries for him are gonna be Schwarzman for Nini and Goffin So I think he has a pretty good road to the final Let me know down in the comments below. Who do you have winning this one? So the Barcelona open didn't have it last year, but it's back for 2021


A new understanding of our galaxy structure began in an unlikely way on Twitter. A research effort sparked by tweets led scientists to confirm that the Milky Way's central bulge forms an X shape. The newly published study used data from NASA's Wide Field Infrared Survey Explorer Y's mission. Infrared light allows astronomers to see the structures of galaxies in spite of dust, which blocks critical details in visible light. But it was the Milky Way's appearance in the tweets that got the attention of other astronomers. Some chimed in about the appearance of the bulge, a football shaped central structure that is three dimensional compared to the galaxy's flat disc. Within the bulge the wise data seemed to show a surprising X structure which had never been as clearly demonstrated before in the Milky Way. The bulge is a key signature of formation of the Milky Way. If we understand the bulge we will understand the key processes that have formed and shaped our galaxy. The Milky Way is an example of a disc galaxy, a collection of stars and gas in a rotating disc. In these kinds of galaxies when the thin disc of gas and stars is sufficiently massive a stellar bar may form consisting of stars moving in a boxed shape orbit around the center. Our own Milky Way has a bar as do nearly two thirds of all nearby disc galaxies. Over time the bar may become unstable and buckle in the center. The resulting bulge would contain stars that move around the galactic center perpendicular to the plane of the galaxy and in and out radially. When viewed from the side the stars would appear distributed in a box like or peanut like shape as they orbit. In the structure according to the new study there is a giant X shaped structure of stars crossing at the center of the galaxy. A bulge can also form when galaxies merge but the Milky Way has not merged with any large galaxy in the last nine billion years. We see the box shape and the X within it clearly in the wise image which demonstrates that internal formation processes have driven the bulge formation, Ness said. This also reinforces the idea that our galaxy has let a fairly quiet life without major merging events since the bulge was formed as this shape would have been disrupted if we had any major interactions with other galaxies. The Milky Way's X shaped bulge can be reported in previous studies. Images from the NASA Cosmic Background Explorer, COPE, Diffuse Infrared Background Experiment suggest a box structure for the bulge. Thanks for watching and remember the ways by which we arrive at knowledge are hardly less wonderful than the discovery of these things themselves.


Oh, this is a very interesting question actually. Can you elaborate on the association between fatty liver disease and cardiovascular disease? Fatty liver is on the rise. Fatty liver is related to the obesity and the diabetes epidemic and people that have excess fat put it in weird spots. A liver being one of them, the sac around the heart is another one.  It acts aggressively. It makes chemicals that are inflammatory. And it increases your risk of heart disease. It increases your risk of heart attacks, of strokes, and of cirrhosis of the liver, particularly if you're Hispanic and if you drink. So all the medicines that we use to treat lipids and that treat diabetes and diets that are all cardiovascular beneficial are good for liver disease, for fatty liver disease. And I think that, you know, the obesity epidemic like Dr. Coulter said is really something that us as a cardiology community we're combating now and in the future we'll continue to do so as diabetes is on the rise, obesity is on the rise.  But now we know. We're much more ambitious about it. Yeah. And we know that there are things, you know, in daily routines that we can change to improve it and medicines of course that we use to help too. In this article there's a big chart of what medicines, in fact it was shocking because when I was reading the, you know, evidence-based medicine for the treatment of fatty liver disease it was like reading the diagram of how I prevent heart disease. It was basically the exact same thing. So we're going to leave you with some tips for the future. Lead a heart healthy lifestyle, encourage your friends and family members to do the same, monitor your blood pressure and see your doctor for routine screening, you know, really certainly after the age of 50 or sooner if you have a strong family history.