The Mac Observers' Mac Geek Gap, Episode 796 for Monday, January 6th, 2020. Greetings, folks, and welcome to The Mac Observers' Mac Geek Gap, the show where we take your questions, your tips, your cool stuff found. We mash them all together. And the goal, by sharing your tips, sharing your questions, sharing your cool stuff found, answering some of those questions even, the goal is for each of us, all of us on this side of the mic and that side to learn at least five new things every single time we get together. Sponsors for this episode include ExpressVPN.com slash MGG and TextExpander.com slash podcast will tell you why you want to visit those URLs in a little bit. For now, here in Durham, New Hampshire, at least while we're recording this, I'm Dave Hamilton. And here in Fairfield, Connecticut, this is John F. Braun, but wait, there's more. That's true, folks. I would like to introduce you all to someone that if you've been listening for the last couple of weeks, you know, is coming on the show, the founder of Bombic Software, I believe he will correct me if I'm wrong. The chief programmer behind one of our favorite apps, Carbon Copy Cloner and an all around great guy, Mike Bombic. Mike, thanks so much for coming on the show. Hey, thanks for inviting me. Yeah, absolutely. Well, we talk about Carbon Copy Cloner a lot. We also talk a lot about APFS and encryption and Catalina. And sometimes we get the answers right and sometimes we don't. And we're hoping that you can help us maybe enhance our relative knowledge base here today with with some stuff. So we've pulled together some questions for you.  We've got some some tips to share too. So it's great to have you, my friend. It's I we were talking before the show. This is your first appearance on Matt Mac Geek. You've been on other podcasts here at Mac Observer before. But let's let's do a little bit of history. I don't want to I don't want to go too deep into it. But prior to turning Carbon Copy Cloner into a for for profit for business venture, you worked at at Apple. Is that right? Yeah, I worked at Apple for eight years and I was in the field engineering group. And I was basically putting together solutions for higher education and enterprise customers right in the middle of the US. So I participated in things like the seven hundred and sixty eight node supercomputer cluster at the University of Illinois Urbana Champagne. Oh, wow. I was never a rocket scientist, but I got to rub elbows with them. So that was pretty cool. That's cool. Wow. Yeah. And eventually it was it was weird. I felt like I could be more creative outside of Apple's end inside. So here I am. And here you are. Yeah. And that and it's been I mean, we I say we we at Mac Observer interviewed you at a WWDC right after you left Apple, I believe. But I mean, that's been what almost 10 years. Is that right? Yeah. Yeah. Ten years.  Well, that it seems like it was a good decision. Here you are. You've got you've got not only do you have carbon copy cloner as a business, but we all have it as a product that we can use and rely on. And so that like that's a win-win if you ask me. So yeah, good. Yeah. All right. Well, let's let's dive right in and let's get some of these questions and then I'm sure as always these things will lead to tangents. But one of the most popular questions that we get related to not just carbon copy cloner, but but Catalina and APFS in general is sort of epitomized by by Graham's question where he says just today I was doing some updates to my CCC tasks after having bitted in my Catalina upgrade and had caused to read through some of the very detailed user guide information that you've put together. I read various items, but the one area that is touched on and has been of concern of mine ever since APFS came out is the reporting of file sizes and how the cloning feature of APFS would seem to leave no utility able to accurately report the actual disk space usage anymore. Allied to that is how APFS snapshots are or are not reported or included by various measuring apps. So the question to you, Mike, is do you see any evidence from Apple that they may in the future expose a better way to see and report actual disk space usage? Or is there already a go to measurement that is accurately reporting this? There's actually a couple, a couple of different pieces to this. There's actually going to be two different answers for the snapshots part. It is pretty surprising that we are all the way into Catalina. So we've got three full production OS releases with APFS as being the default startup file system. And Apple still has no way of visualizing snapshots listed on a volume or snapshot disk usage. They added snapshot listing and removal to the disk utility command line utility. But right now there's really no way of figuring out what the snapshot disk usage is on any disk. We've had that within CCC as soon as they gave us access to the snapshot API. So for a couple years now, you can select a volume and CCC sidebar and we'll list the snapshots there. You can click on one and delete it. And then you can also see the disk usage. And the disk usage is actually kind of funny that the calculations are a lot more complicated than you would think they are. If you look at the disk usage listed in the snapshots table, and then you add all of them up, that doesn't actually equal the total snapshot disk usage. And we could go into it. We don't really have to I've got a video that kind of demonstrates why that is the case. But the end of it is just that if you want to see the snapshot disk usage, the amount of space that snapshots are consuming on a volume, just click on that volume and CCC sidebar and we show it to you. Okay, so is it because we do like to get a little bit geeky here. So I'll ask the question, is it because the same files might be included in multiple snapshots, akin to the way Time Machine does things with with hard links? Is that sort of what's going on there? Or is it am I completely off base? That is precisely it. So when you look at the snapshot disk usage that CCC lists for each individual snapshot, that's an indication of the amount of space that would be freed if you were to delete that one snapshot. So if you've got three snapshots, and they each have a reference to a five gigabyte file, each one of those snapshots the size is zero. If they only have if they all have a reference to the same file, then deleting that snapshot is not going to remove that five gig file. Right. When you delete two of the snapshots and there's one left suddenly that third snapshot is five gigabytes. Because when you delete that snapshot, the last reference to that large file is now removed.  That makes sense. Okay. Yeah. Okay. Yeah, which is which is what what time machine is doing essentially. So yep. So the other side of that question, though, is about the APFS cloning feature. And that's a little bit more complicated. So the way that the cloning feature of APF and I wish they picked a different word. I mean, come on, they're stepping on my toes here. My guess is that they didn't spend a whole lot of time worrying, what will my thing? Unfortunately, so the way that it works though is if you duplicate a file on the finder, let's suppose you've got a one gigabyte video file on your desktop in the fire in the finder, and you duplicate it. What APFS will do is instead of making a copy of all those bytes, it will make a reference to those blocks, it'll create a new file and the new file just has references to the same blocks. Sure. So it looks like you have two files. But in reality, they're still only using one gigabyte of disk space. What is complicated, though, and excruciatingly frustrating, is that even finder does not do the math correctly. So when you duplicate those files and then ask finder, you know, get info on your desktop, it does the math as if those are two separate copies of the file. No kidding. I had no idea. Oh, answer so much. Sorry. It does. And well, and that's where it starts to get frustrating for any users, because they're trying to figure out why things don't fit when they copy it to another volume. And it's just there are no tools to determine that one file is a clone of another, or even to what extent one file is a clone of another. And not only are there no tools for this, there's not even API's that developers could use to make those tools. There's no access to the internals of APFS that would allow us to make that assessment. And that's it's a frustrating situation. And I actually pressed Apple about this at the developer conference a few years ago. And the issue is that let's suppose you've got a 256 gigabyte SSD, and it's got 200 gigs of stuff on it. But you've got a clone of your photos library. And let's suppose your photos library is 100 gig. So if you ask the finder how much data do I have on in my home folder, it's going to give you a value that's like beyond the capacity of your startup disk. We've seen this we've had questions about this and couldn't put our finger on it at the time. So this is this is really enlightening. Yeah, Yes, and that's precisely why so it gets worse because if you then try to copy that content to another disk, because there's no API that allows us to determine that one file is a clone of another, we have to copy those to separate files. So if I were to copy that 256 gig SSD to another 256 gig SSD, the data won't fit. And I pressed Apple. Can you give us an API to, you know, to work with this? And they said no, they didn't have anything that would do that. They gave me the impression that Oh, maybe we could do this or this. But they literally said, take this to the bank, get a larger destination disk. Seriously? Yes. And the really frustrating thing about that is let's suppose I get a bigger destination disk, and I'm making regular backups. And now my original SSD fails, or, you know, it's corrupted or whatever, I need to restore back to it. You now need a larger boot disk or origin disk, whatever that would be. Yeah, right. And, and I've had a few cases where I have to give the person the bad news that, you know, their dog's dying of cancer, that they basically have to figure out, with no tools that can actually do this, which files are the clones? And honestly, the end users of it's the worst case scenario, but the end users, probably the best person that could figure out, Oh, yeah, you know, I remember duplicating that really large folder. Yeah, right, right, they might, they might have that, that, that situational memory. Yeah, right.  And it's really, really frustrating. And I'm hoping that Apple addresses this. But I'm actually not very optimistic that they have plans to address it in the near term. Well, so here's, here's a thought that came up, and maybe this will change your optimism, or, or maybe it will cement your lack of it. Does migration assistant deal with this properly? If, for example, I have a 256 gig, you know, drive in, in machine A, and now I get machine B and a migration assistant from one to the other, is it smart enough somehow to know that that's a duplicate, or is it going to say I don't have room on my destination, even though the two disks are the same size? That's a really good question. I don't have the answer to that. Okay. I kind of want to test that. I know, same. Now that, now that I know. Yeah. Yeah. I'm flabbergasted. I think that's the end of the show, folks. No, no, no, no. John, do you have any thoughts on this? Um, actually, can we go back to the snapshots? Yeah, yeah, we're done with this here. Yeah. Yeah, I mean, my observation here, so one, so, you know, I have one of my backups in front of me here right now. So if I just click on one of these snapshots, and it looks like there's not a lot of them, there's only, they only go back like a day or so. So that seems kind of weird. I mean, why would you, I don't know, but from what I see in a support article that they have, that's what's supposed to happen. I'm only supposed to have a couple of days worth. But so if I click on one of these, you know, there's a little restore arrow, that will put my drive in this state that it was when that snapshot was taken. Is that correct? Yeah. I mean, you'd have to boot from the backup if you wanted to restore your system from a snapshot. But yeah, that's the general idea. Okay. The other thing I'm observing is that looking at time machine, there's no correlation between these snapshots that I see in the CCC interface and what time machine is showing me. If I say run time machine. Yeah, I mean, time machine is going to create its own snapshots. And that's actually a point of contention for some people. They don't realize that just by having time machine enabled, it's going to create snapshots on an hourly basis on any APFS volume that's connected to your Mac that's not explicitly excluded. So if you look at the list of snapshots in CCC, when you click on that volume in the sidebar, you'll see time machine snapshots right there alongside CCC snapshots. And time machine will only retain snapshots for 24 hours, and then it deletes them. CCC will retain snapshots according to the settings that you have right there in CCC. And to address the first part of your question, if you're looking at your startup disk, we have some pretty conservative snapshot retention settings for the startup disk. We'll only store hourly snapshots for up to a day, daily snapshots for up to three days, and then nothing beyond that. And that's based upon some feedback that we got really early on when we first started providing support for snapshots. Even in cases where people have ample space on their startup disk to store snapshots, they were a little bit surprised that any space was used at all for snapshots. And initially, we thought, okay, we'll just scale back the snapshot retention policy and make it more conservative so we're not storing as many snapshots. But eventually in CCC 518, we decided to just stop enabling snapshots support on the startup disk altogether. You could certainly enable it manually. But in the default configuration, we're just not going to enable it on the startup disk because people just found it too surprising. And people would get really frustrated when they empty stuff from the trash and they're not seeing their free space come back. Right. Right. Yeah. In fact, we had a listener that sent a tip in that essentially he was running into this problem where his disk was showing more space used than was possible given the various numbers. And what he said was he realized, oh, carbon copy cloner, just by enabling it, like time machine enables a snapshot retention policy. And once he found that, then he tweaked it sort of to his liking. But his tip was FYI, if you're running carbon copy cloner or time machine, snapshots will be enabled and you might want to you might want to go in and tweak that. So. Right. And hopefully by now, most people are not seeing snapshots enabled on their startup disk. It's been a while. Okay. Hopefully getting in front of that. Got it. Oh, so that. Okay. So I missed that. My apologies. So that is no longer a default setting to do that. Is that right? That's correct. Certainly anybody who's upgraded from prior to 518, they may still have snapshot support enabled on the startup disk. Yeah. But I mean, even in those cases, it's it's not necessarily a bad thing. Right. You know, we have a pretty aggressive file watching or file space watching service. If free space on your startup disk starts to drop below the free space retention policy, we'll start removing CCC created snapshots. So we don't ever want people to get into a position where like the startup disk fills up completely. And that that's a very rare occurrence. But we also don't want people to be surprised at all. And concerning free space on their startup disk. So that's why we changed the default setting to not enable snapshot support. Yeah, that makes sense. That makes Yeah, yeah. So that users aren't surprised by this thing that they don't know is happening and can't explain. Yeah. Yeah. Makes makes good sense. Yeah. And I think in an earlier version of Catalina we would have and I think I actually ran into this as well is that there was this mysterious category in. So if you go to about this max storage and it would show this like system blob. And for a lot of people at one point, so I think it must have been a bug in something. But it would keep growing and growing and everybody was like, how do I get rid of this? And I think yeah, the solution was to manually whack the Wax some of those files free up some space. Yeah. Yeah. And one thing that's important to point out is that CCC is not the only thing that will create snapshots nor is time machine. If you apply an update to Mac OS, it's going to create a snapshot right before you create the update, which is pretty cool. But at the same time, you know, if you're poking around stuff right after you've done an update and you're deleting doing some housekeeping, that snapshot still there and it's going to be there for 24 hours. So even if you don't have time machine enabled or even if you don't have CCC enabled or installed, you could still see the same issue. Right. Right. Yeah. Right. Oh, that makes sense. That makes sense. Yeah. Fascinating. Fascinating. I love this stuff. All right. While we're here, we had a tip, a carbon copycloner tip from listener Keith that I wanted to share with everyone, but also with you, Mike. He said he's a recent customer and he says, I put my Mac to sleep at night and my backup is set to run at 2 AM every day when I'm normally fast asleep. He says, I broke my arm last week and my sleeping patterns are a bit disturbed. So I happened to be passing the shack last night and saw the screens were on. He says, of course, you know, the carbon copycloner wakes up the Mac at the scheduled time, does the backup and then puts it back to sleep. He says, because I've set it to do a find and replace corrupted files once a week, which takes hours compared to the usual few minutes, I decided to do something about this because he says, I don't want my screens on for hours at night, even if it's only one night per week. He says, I wrote a very short script which turns off the screens and then told carbon copy cloner via the advanced setting to run before copying the files. He says, I tested this during the day and it works perfectly. Carbon copycloner opens, goes through the preparing backup process and then turns the monitors off. At the end, it goes to sleep. He says, really, the script is one command. He says he named it blank screens.sh, gave it execute permissions and I will put the contents of his script. But really, what it does is it runs PM set display sleep now is what the what the command does. I love it when our listeners come up with these creative solutions to solve these problems and I thought you might appreciate that one too much. That is pretty neat, but I do have one comment. There's an even easier way to solve this. I wondered if there was. It's to not have the screens turn on at all. And what's happening is the default. It's funny, the default settings that we choose sometimes, you know, nobody would have a default setting that says, I want my backup task to go really slow. Right. Or I want my backup task to run, you know, give or take an hour. It's fine. You know, they set it for two o'clock and they expect it to run at two o'clock. And if it runs at, you know, two thirty seven, it's like, man, what's going on there? Sure. So the default setting when you schedule a task is to wake the system if it's asleep. And what that does, CCC schedules a wake event with the system management controller. Yep. And that is a full wake event. So that wakes the display, the CP, the video card, everything. And one of the other options, instead of telling it to wake the system is to run when the system next wakes. And by default, your Mac is going to go through dark wake sleep cycles every couple of hours. So if, you know, you can handle it, give or take an hour on the runtime and, you know, this one's running at two a.m. Then configure the task to just run when the system next wakes. And if the runtime has expired and the system makes it two thirty seven, it'll run then and the screen won't be on. Oh, really? Huh. And would laptops be the same or no, probably not. Or if they're powered, would they be the same? Yes. As long as they're plugged into AC power, they also go through a sleep weight cycle, even if it's in your backpack and plugged in. Right. Right. Yeah. Fold it up and plug in. Huh. Interesting. So. Oh, yeah. So I definitely need to go change like all of my backup tasks to be just do it the next time you wake up. I had no idea that those dark wakes, I didn't even know they were called dark wakes. I also didn't know that third party developers could sort of tap into the the sort of coalesced happenings at those times. I thought it was only Apple could do what do they call it? Power nap or something like that. Right. Where it is. It had three different names over the years. That's why I don't know. Power nap, dark wake. And there was another dark week. I like it. Dark weeks. Yeah. Dark wakes with Mike Bombick. It sounds like we might have an episode title. So I like it. All right. That's cool. That's cool. Wow. Man. All right, John. Any anything more before we before we move on here? No. OK. I am going to take a minute and talk about our two sponsors here. Our first sponsor for today is TextExpander at TextExpander.com slash podcast, as I mentioned before, and that is the correct URL. I know it's a generic URL. It's OK. It's the correct one. TextExpander is an app that I truly could not live without. And if it happens to not be running on my Mac, I notice it immediately as soon as I start doing anything. Because what I use TextExpander for is to ensure both accuracy and efficiency. These two things are usually at odds with one another, but not with TextExpander because what TextExpander lets me do is all those things that I type repeatedly, things like customer service responses, certainly for people at the business, but also just those short things like the command to join our auto recorder to our our Discord channel. Like I can't remember how to do that. I don't have to. I have a little TextExpander snippet that remembers that command for me and types it in. This way, A, I'm efficient because I don't have to go look it up. I don't have to think about it. And I also don't have to type the big long command out and B, I'm accurate. I never get it wrong because it's always right just inside TextExpander. So think about it for like your customer service responses, your little commands like that, your email address for form fields, your phone number, all of that stuff can be stored right there. And if you are using it at your business, well, then TextExpander for teams lets you share your snippets with everyone so you're not just stuck using them yourself. Check it out. Go to TextExpander.com slash podcast.  Our next sponsor is Express VPN. I love these folks. I started using them a little over a year ago when they first became a sponsor of the show and I very quickly came to love them because not of the fact that they were a sponsor, but because of the fact that it just works. Express VPN I can put on all of my devices iOS MacOS doesn't matter. They just works one click and I'm connected. And what's cool is how attentive they are. I mentioned I think in the last sponsor spot that I had watched Rick and Morty on Netflix by connecting to Netflix Paris or by connecting to Express VPN Paris. And then some of you wrote in and said we tried it and it couldn't work. Well, because Netflix was blocking one of Express VPNs endpoints, but their support people were very quick to tell me, oh, use this different France endpoint or something and then boom, I connected and I was able to stream it again. So they are on top of this stuff. They know what you want to do with a VPN. They know how to protect you. They hide your IP address and let you control where you want sites to think you're located. You can also ensure that your connections when you're at like a coffee shop and things like that are secure in the tunnel and that way no one knows where you're browsing or what you're doing. And that's the kind of privacy that is a good thing in today's world. You got to check it out. And here's the deal. If you go to express VPN dot com slash MGG, you get an extra three months of Express VPN for free so you can support the show, watch what you want and protect yourself at Express VPN dot com slash MGG. Our thanks to Express VPN for sponsoring this episode. All right, Mike, one thing we were talking about before the show began was the concept of Catalina volume groups. So I just kind of like to give you the reins and take us on a little tour if you wouldn't. And then John and I will probably have some questions along the way, if you don't mind. Yeah, yeah, sure. This is actually, I think Catalina is one of the biggest changes to Mac OS probably since the beginning. I mean, APFS alone is huge, but the change with volume groups, especially as far as backups are concerned, it's it's really huge.  The first time I saw the little notification that an update is available for your Mac, it just sent a wave of panic through me because there are like millions of people getting this little notification as if it's this tiny little update to your Mac and they click it. And, you know, eight gigs later they're applying this enormous change to their Mac. Yeah. And it totally happened, you know, that that first week after Catalina came out, we had lots of people who like just were not really computer savvy making this enormous change to their system. And we kind of had to teach everybody how it works. Yeah. So we have a great article that goes into depth on how this works. But basically what happens when you upgrade to Catalina, your current Macintosh HD volume is renamed to Macintosh HD dash data and its volume role is changed to data. And then the installer will create a new system volume and kind of attach it or associate it with that data volume and create an APS volume group. And a volume group, it's not a different kind of volume. It's just an association between two volumes that the operating system and applications will treat as if they're one volume. And there are special links between the two volumes that will link up folders on one volume to folders on another volume, such that when you navigate to the user's folder on Macintosh HD it instantly transports you to the user's folder on the data volume. And this is this is the concept of firm links that we talked about after I got back from Mac Tech and I'll I'll put a link to the episode in there too. But that's that Tim Standing did a great talk about firm links. And man, that guy dug way deep into this stuff. So, yeah, yeah, good stuff. Yeah. Yeah. Yeah. So that's that's the gist of it. And when we make a backup we have to basically recreate that volume group to destination. So we actually walk through that same set of changes for your backup disk. We convert it to a volume, a data volume and rename it and then create that system volume, attach the two together, create the firm links and then populate the two volumes separately. And hopefully we pull it off just as seamlessly as it is in the finder. You know, you still just pick that one Macintosh HD volume as the source and you pick your CCC volume as the destination. And that part still works the same. And we can tell if it's a volume group. If it's a volume group, then we deal with the two volumes separately as necessary and present just the the main system volume in the interface. The one exception to that, which people see this every day is in the sidebar, we list both the volumes, just like in disk utility. If you open this utility, you'll see Macintosh HD and Macintosh HD data. Right. In our sidebar, we reveal both of them. And that's because there are times when you have to get into the weeds and take a look at each volume individually. Yeah. Still, I mean, the thing is, though, there are millions of people out there using Catalina now. And it's a brand new concept to them. And for most people, it's hidden. So, I mean, you can just imagine I'm using my computer and upgrade and I got to replace all these applications because the 32 bit apps don't work anymore. And sure. And then some time later, you open disk utility and say, what on earth? What is this Macintosh HD volume? I've literally had people ask, should I delete that volume? Yeah, that's right. There was no like even the first time I launched disk utility, there was no, hey, this is going to look different. And here's why kind of thing. Like I get that you I mean, not you, but Apple wants to make this as seamless as possible for the user.  Because most people wouldn't care to to understand that. But if you are the type of person that launches disk utility, that's a very like I knew about the split in volume before I launched Catalina's disk utility, but only because I had gone to the the session at WWDC, the state of the developer state of the union session, where they sort of casually mentioned, oh, yeah, you know, we've got these two volumes. It's like, wait, what? Even there, they sort of downplayed it, at least at the state of the union.  But I wouldn't even say that they went that deep into it. I mean, they definitely describe firm firm links, but surely not as deep as I expected them to at the developer conference. Right. Yeah, right. Yeah. So I don't know. I'm a huge fan of contextually appropriate documentation. Yes. And I think you're right that when you open disk utility, that's totally the time to tell people, hey, so there's something new here. And if you want to learn more, click here, like even just a button in the toolbar about volume groups. Right. That would be that would be great. And it could take a link to my documentation. They don't even have to write their own. You're so helpful, Mike. This is great. Huh. Yeah. Yeah. I mean, the nice part is that, A, by and large, it that like this this concept seems to be working very well for most people. And it does provide this layer of security because I don't think you mentioned it in your description, but just so everybody is aware, the data volume is writable, but the non data volume, a.k.a. the system volume is essentially not writable. There is one little moment that Tim Standing found where it is writable, but that's only after all other processes have been shut down and it is writable for a split second right before the machine either shuts down or reboots so that any changes that need to be made to the system volume can be made. But but but that's it. So it means that that you can you can be relatively more secure than you than you were with a writable system volume. It also means that when there is an update to Mac OS, it can do it on a a clone of that not a clone, a snapshot of that volume. Right. And then just make sure the update worked and then wipe out the old one. So you've got sort of a safety net there, not to co-opt your terms in some carbon copy cloner, Mike, but it's a good term. Right. So there is that safety net with updates that now is just sort of fundamentally more possible because of these volume groups and the fact that system and data are split. So yes, the one thing, the one comment I would make to that, though, is I wish that they had made this split even even wider. I don't I don't feel like there has to be an association between the system and the data volume. You know, we've already got a recovery volume and a preview volume and these are dedicated APFS helper partitions. And when Apple first announced APFS and the whole concept of a container and space sharing within the container, I thought, aha, I know exactly where Apple's going with this. They're going to create a new system volume that's completely independent from the data volume. Right. And they kind of did that. But then they got this whole concept of we got to glue them together. And if we could just have a system volume that's completely independent of the data volume, it would be so much easier. You could hide that completely from the user. All the user has to see is their own data. And, you know, even as I say that, I know there are huge logistical constraints. There are parts of the operating system that have to remain writable. And right to be able to make that sort of change. I don't think you could do it in a one year release cycle. I kind of hope Apple is moving in that direction, though, if someday they found a way to have a completely separate system volume. I think that would make me happy. Right. Huh. Yeah. That's I hadn't thought about it that way, but they certainly that seems not only does that seem like a good idea. That seems like the kind of thing Apple would want to do anyway. It didn't hold. Thou shalt not touch. You know what? Don't even look at it. You know what? It's not even here. Like that that that seems like a natural progression for them. Yeah. Interesting. Huh. Any thoughts on this, Mr. Braun? Uh oh. Do we lose John? I think we might have lost John. He's on mute. He is on mute. OK. No, I'm on mute. Welcome back. That's all right, man. Yeah, I thought I thought I'd mute when you did the read. But anyway. And I forgot to unmute, but we're here now. So Mike, you mentioned Preboot and you know, that was actually created a little confusion when I first installed Catalina because I think at one point it actually showed up on one of my volumes. And I'm like, hmm, what should I do with that?  But what is Preboot doing? It looks pretty small. I see it along with recovery and VM and all of that. But yeah, it's like 26 megabytes. But what's going on in there? Yeah. So that's that's basically APFS's answer to starting the computer using firmware that that doesn't have to know a lot about how the file system works or how the file system is secured. So within the APFS container, any any volume that has an installation of Mac OS will have these associated helper partitions Preboot and recovery. Recovery is not really necessary for the operating system to work, but you know, you know, the recovery volume does. But with Preboot, there's just a small collection of resources there that allow Mac OS firmware to kickstart the startup process. So there's the bootstrap file boot.efi. And then there's just some configuration files. And then there's some user interface files, things that, for example, show you the unlock interface for a file wall encrypted vine. So the Preboot volume itself isn't encrypted.  What will happen is the Mac OS firmware will see the Preboot volume and it will exit when you choose to start from it. It will execute the bootloader. The bootloader will then execute a kernel, a kernel extension cache. And then finally, that will load some resources, enough resources to mount the file system, unlock file system if necessary, and then kickstart launch D and the rest of the startup process. Huh. OK. All right.  So maybe that was a bug in a prior version because at one point I saw it. You shouldn't ever see it, but occasionally some process will mount it. I will mount CCC will mount those volumes, but it mounts it with the no browse flag. So it specifically does not show up in the finder, but I've certainly seen cases and reports of it showing up in the sidebar. I've seen some reports in the Apple discussions. There's always somebody saying, should I delete this thing? Probs not. Yeah. Thankfully, system integrity protection prevents you from deleting it. So I was going to say it be very difficult to delete, right? OK, that's good. Yeah. Yeah. Yeah, with the with the data volume, if someone went into disc utility and I'm I'm just folks, I'm just posing this as a as a thought exercise. I'm not necessarily suggesting anyone attempt this, but it like if somebody went into disc utility and instead of asking, should I delete this data volume actually went and deleted it? How like, could you do that if it's the data volume that's attached via firm links to your current boot disk or would it also stop you? No, you can't do that. OK, that that is actually a really good question because it raises another topic that we've seen a lot since Catalina came out. Actually, in the very first release of Catalina, if you let's just suppose you boot from your backup and you want to erase the disk or let's say you boot in recovery mode and you want to reinstall Mac OS. You just want to erase everything and start over. If you so you open up disc utility and lacking the contextual information about volume groups, you see Mac and dash HD Mac and dash HD data. Click on either one of them and click erase. Now what happened initially in the early versions of Catalina is that it would only erase that one volume and that would break the volume group. So when you're done erasing that volume, you now have two separate volumes and neither of them belong to a volume group. If you then go and install Mac OS, it's going to go through that procedure again to create a data volume and a volume group. And at the end of it, you're going to have a second Mac and dash HD data volume. And we've seen this a lot. I've seen cases where people had five of these and it's people don't don't know what to do in 10 15.1. I think Apple enhanced disc utility such that when you select a volume and volume group and click the erase button, it'll prompt you to erase the volume group. But it's still not the default selection and people are still making the wrong choice. And we're still seeing these orphaned data volumes. But getting back to your original question, let's suppose you're in disc utility and you see Macintosh HD, Macintosh HD data, and then a second Macintosh HD data. The concern is, all right, I see that I've got a duplicate, obviously, but which one do I delete? And that's where, thankfully, system integrity protection prevents you from deleting one of them. So you know which one or which ones you can delete. It's the ones that you're allowed to delete. Oh, OK. So that's good. All right. OK. So it's actually really easy to clean up the the vestigial volumes there. Yeah. But people often don't know that they need to or, you know, sometimes you're asking, can I delete Macintosh HD data? And the answer is yes. And that's just adding to the confusion. Yeah, right. Yeah, exactly. Exactly. Huh. Interesting. All right. Cool. You know, I'm going to jump to a listener question here about formatting external SSDs and how that should be done. So listener Andrew wrote in and said, I would like you to ask, Mike, if there is any reason why I should stop partitioning and formatting my SSDs, including SDXC cards that I use for clones, he says, as Mac OS extended journal, AKA HFS plus, as a bit of background, he says, I'm using 1014 Mojave now, but will probably upgrade to Catalina in the next year. Should I expect to have any issues booting from Mac OS extended drives or SDXC cards if I move to Catalina? Says, I don't I still don't fully trust APFS, but maybe my fears are unfounded. Is there some advantage to using APFS for my clone volumes? So I mean, I think there's there's some obvious answers here, but but also just sort of a more general question. I'll answer his first question, which is that once you move to Catalina, your clone drive must be APFS because Catalina won't boot from HFS plus. Is that correct, Mike? Yes, and we also will convert that to APFS when possible. Got it. OK, so so Catalina in order for Catalina to boot to a volume and therefore in order to have an effective clone of a bootable Catalina volume both need to be APFS and and that will almost happen as a matter of course, without really being able to avoid it. So I guess the real question is for his other SSD, his external SSDs, thinking about APFS versus HFS plus, is there any reason he should be on one or the other for non bootable external drives? Yeah, and I'm kind of right there in the middle.  If you're going to take advantage of the features like snapshots or file cloning, then I would definitely encourage you to use APFS. If you want to have multiple volumes and you want them to share the storage in that volume without having to partition the disk and make your your storage allotment choices up front, then APFS is a really nice fit. But I I don't blame him for not fully trusting APFS. You know, one of the issues that we still see in the wild and I can't really comment on the the frequency of this because, you know, the doctor sees all the sick patients. Sure. But we we've seen a lot of cases of file system corruption that, well, I can't honestly, I don't even know if we can call it file system corruption. We've seen reports from disc utilities first aid that have just hundreds of errors. And at the end, disc utility says that it attempts to repair. Oh, wait, repairs are disabled. And then it concludes, hey, everything's OK. And there are no other utilities that have the level of access to APFS that's required to repair the volumes. So with disc utility being our only choice and disc utility behaving like that, I can certainly see why people would be reluctant to use APFS on a production volume, or it's not completely necessary. Certainly for, you know, a backup of your startup disk for your startup disk, APFS is here and it's just something that we have to, you know, live. But for other volumes, you know, if you really want to take advantage of snapshots and it's an SSD, absolutely, I would that would be my first choice, APFS. That's where I'm feeling with it for right now. But if I really didn't want to take advantage of APFS's features, you know, then the disk is an HDD, a rotational disk, then I might consider sticking it out with macOS Extended for a little bit longer, especially if you have significant performance demands from that disk and performance specifically in relation to file system enumeration. You do a lot of searching on that volume. You just have millions of files on the volume and, you know, you tend to browse through it a lot on the finder. APFS just does not perform nearly as well on rotational media as HFS does. And I wrote a blog post that kind of goes through, at a technical level, why that it's the case. But I think eventually, you know, it's like the floppy disk. Right. Oh, yeah. Eventually, we'll be there for sure. Yeah. Yeah. Any thoughts on this, Mr. Braun? Any questions, thoughts? No, I think, Mike, brought up a really good point, though, is that it's of concern that the only utility you can use to detect or repair damage is disutility. And actually, I did that a while ago because I started getting time machine was I had a machine that I had migrated. I had not freshly formatted it. And we've had more than one person report problems eventually creeping in. And at some point, I had that happened where this utility reported some sort of block allocation, some weird error. I had no idea what it meant, probably. It's not even documented. But yeah, I had to do a fresh reformat to get things working right again. And I should probably stay for the record that the issue that I described, I did actually report to Apple looking for some advice on how I can advise my own customers that are reporting that same kind of issue. And I have yet to receive a response. Yeah. Yeah. Well, I think they are probably scrambling on a lot of that, too. You know, it's one thing and an impressive thing to develop a brand new file system as quickly as they did and then roll it out to as many devices as they have, right? It's on both iOS and MacOS. But until you have it rolled out to mass quantity of devices, you don't know what problems you're going to run into that you then have to write tools to fix.  But until until people run it and, you know, turn off the machines at exactly the wrong time or the right time or whatever you want to call it, like all of those perfect storms, each one of them sort of requires probably an in-depth analysis. And then, OK, here's how we either could solve this or couldn't or maybe could. And here's how we would then write the tool. Now let's write the tool. And so, you know, as I've said many times, HFS plus is very mature and we've seen most, if not all of the problems that it will have. And that's why these tools can exist. So yeah, it's tough. It's tough. I do want to sort of shine a light on one thing that you said, just so folks understand, with APF, well, with HFS plus, if you wanted multiple volumes, and I'm really talking about this in the context of an external drive, but everything is true for whether it's internal or external. It doesn't matter. If you want multiple volumes, you had to partition them. And as Mike said, you had to decide how much space was going to be allocated to each of those volumes upfront. With APFS, you don't have to do that. You can approximate that with quotas, but you can change that basically at any time. APFS does not carve out separate partitions unless you really force it to. It just creates one big blob of data that your volumes use together. And that can be a really nice thing in many circumstances, not all, but many. And so that is a benefit to APFS, especially if you're doing it on an SSD externally. So just wanted to make sure folks understood what that was. So good. Any more thoughts on this before we move on to BitRot? OK. BitRot. Yeah. So I think it was the same listener Andrew wrote, and he says, I'm very concerned about BitRot or other file corruption, especially with my aging spinning drives sitting on shelves. Are there ways that a tool like carbon copy cloner can help avoid copying a corrupted file to a drive? He says, I know there's an option called find and replace corrupted files. And I see that you recommend choosing that on a weekly or monthly basis. Can you explain a little bit more about that works, about how that works? And what about files that are corrupt before they are ever copied to another drive? Is there are their best practices about that? And also with regards to storing drives that aren't in use. So any thoughts on that, Mike? Yeah, so there's there's often some controversy about how BitRot is actually defined precisely. Sure. The way that I define it is when magnetic media is no longer able to discern the difference between zero and one and the sector is basically unreadable. When CCC encounters an unreadable file, it gets a specific error from the file system and an IO error. And if we run into an IO error on a source file, we will throw up an error and say, you know, this file is corrupted. We'll actually make two attempts to read the file. Statistically speaking, a second attempt is sometimes successful at reading that failed sector. After that, though, there's really no point in making additional attempts. It's just not going to be recoverable. So in that case, we flag it and we report it as an error at the end of the task. But we do move on to the next file and then copy as many files as possible. That's in stark contrast to some other tools and also to the finder itself, where if you try to copy a folder and it hits any error at all, it just throws up its arms and I can't cope with this. So the way that dealing with it on the destination is a little different. We have the find and replace corrupted files option, which will basically cause CCC to reread every file on the source and destination. Typically, we'll just compare the file list and if a file is different based upon size or modification date, we'll recopy it. But if those values are the same, we'll just skip it and move on. With the find and replace corrupted file setting, we'll reread every file and if the file is unreadable on the destination, so that's for my definition of bit rot, we actually get an error reading a file or if the checksum just doesn't match, then we'll replace that file on the destination. That is based upon the conclusion that the source file, if it's entirely readable, it is authoritative. Got it. So other people have a different thought about corruption in general. If something maliciously corrupts a file on the source, but the file is still readable, I can't really tell you that that file is corrupt just because somebody flips some bits around. Yeah, there would be no way for you to, even if you compared it to a known good file on the destination, all you'd be able to say is these two are different. You don't know why. So that's certainly where snapshots come into play. For restoring older versions of files and you find out I've been attacked by malware, you can go to an older snapshot and recover files from there. But as far as protecting yourself from the gradual decay of magnetic charge, so to speak, on your destination volume, rereading those files and comparing the checksums, that's going to be effective at preventing at dealing with bit rot. It's also effective at preventing bit rot. There are discs where the firmware, if it detects that the polarity of the media is in decline, it will rewrite that media to kind of recharge it. So just a periodic rereading of a disc is kind of like giving it some exercise. Makes sense. Makes sense. Yeah. Yeah. Okay. Yeah, yeah, yeah. That's good. Thoughts, Mr. Braun? Okay, now you mentioned, I remember looking into this or you mentioned checksums. So every file, and I don't know if I think this was the case with prior file systems, but a checksum or CRC or whatever is created for each file? As I read it, yes. Okay. All right. So that's one way you could tell if there's bit rot as well, I guess, is compare it. Because I remember we looked into this, a time machine claims for 10, 11 and beyond that they actually do calculating checksum and you can do a compare operation and it'll point out the files that that differ. Oh, yeah. That's right. That's right. Yeah, yeah, yeah. Yeah. Because we have that question come up sometimes. It's like, well, how do I know my, how do I verify my backup? And the, you know, the word verify gets tossed around quite a bit. It's like, well, what exactly do you mean by that? And like I use the option that you mentioned or that was mentioned. And that, you know, I'll do fine and replace corrupted files. To me, that's the best assurance that I can get that my backup is correct. Yeah. I like the idea of giving the disk exercise. Yeah. That's amazing. Well, I didn't know that. That's good. Yeah. At some level, you will cause a disk to fail by running it too much. But, but it will fail anyway at some point. So, yeah. Right. And it's certainly not something I'd recommend that you do every day. Even every week, I think, is probably excessive. We have options to that setting to do it on a weekly or a monthly basis. And I have my go on a month. I think one of them goes on a weekly basis. Yeah. And the others on a monthly basis. Huh. Yeah. Yeah. Cool. All right. I want to take us on a little detour here. Listener Gary wrote in and says, is there some reasonable way to audit the behavior of my iMac? He says, I don't know how fast any given procedure should take. I know the more complex, the slower. But if it is, if it is reaching out to an external drive, especially say a spinning USB drive, maybe even slower. He says, but I'd like to know if my iMac is firing on all cylinders or not. But I don't know how to perform this kind of an audit. So when this question came in, I thought, wow. Yeah. Same. And then I thought a little bit more and it was like, OK, you know, what tells me something weird is going on with my Mac? So I'll start with a couple of ideas, but I'd like to kind of have the three of us pass it around a little bit. My general thought is if I'm waiting for my Mac to finish a task and I look at the CPU usage and see that it's being maxed out, then I know, OK, we are waiting for the CPU and then I could dig a little deeper and look an activity monitor and make sure that the majority of the CPU usage is being used by a process or processes that are related to the task that I think I'm doing as opposed to some background task that's, you know, bottlenecking the system for some reason. But if I'm waiting for this in the CPU isn't maxed out, that's when things get a little more interesting because it's not always easy to tell what I am actually waiting for or what an application is waiting for. What do you what do you guys do, Mike? I'll start with you and then John will will bounce to you and see if we can't figure if we can't give people some advice here. I'll hail the spin dump. The spin down. That is my mantra. And I mean, that's certainly by a developer folks. Typically the the issue that I'm tracking down when I'm looking at a performance issue is people say that the backup task is running slow. And the very first thing that we look at is hardware. Sometimes we'll find cases where people have their USB disk attached via USB one. And that immediately explains the issue. It could be that the we're waiting on hardware. And like you said, if CPU is is not really being maxed out, then it could be a hardware weight. And any time I'm trying to figure out what on earth is my process waiting for, go to activity monitor, click on the gear menu and choose run spin dump. And what the spin dump does is it basically creates a sample of every single process running on the system for 10 seconds just to see where in its code it's running. And it's not necessarily going to be easy for everybody to evaluate a spin dump. But if you know exactly what process is waiting, you can go into that process and then you can you can look at its stack trace and you can see what kind of functions it's waiting on. So in my case, if I'm waiting on a file system call, like get extended attribute, I'll see it get exeter call. And then you can see that it's just waiting. It's waiting for file system code. And if you go a little bit further down, we'll see all sorts of interesting things. Sometimes it's just a stall and an APFS IO read write lock. And that's basically just there's something wrong with the file system. Sometimes we will see something like a sofos or another antivirus software has a kernel extension and we've got a chaos request to that kernel extension. And basically what that is is the the AV software taps into file open and close requests read and write requests. And it gives authorization for those requests to go through. And if everything's functioning well, those are seamless and have no performance impact. But those requests rely on some process running in the background to go through its logic and make a decision on whether or not it's allowed. And if that process itself is stalled, then our file system request is stalled. But yeah, I mean, we cannot see all sorts of things from this bin dump. Just other processes that are running. You know, the CPU usage of other processes at a particular moment. So that's that's usually my go to picking a part of performance issue. I I I grok, but I don't grok. I mean, I've looked at spin dumps. I guess this is one of those things where, you know, experience and familiarity breeds the ability to to affect this. Yeah, to digest it. But what we say on the show a lot is learn what normal looks like so that you know if what you're seeing is not normal. And and perhaps there's some exercise with spin dump where running a spin dump when everything is running fine and seeing what a few of those look like might. And I this is a very emphasized might allow one to look and say, ah, OK, this is a little bit different. Maybe maybe that's the maybe that's the thing because that there's a lot of data, as you said, to digest in those. So yeah, the one time I think to that is even if you don't know how to digest the spin dump, having a spin dump captured at the moment that you're experiencing the performance issue, if you can at least narrow it down to one or two applications, you can reach out to the developers. If somebody reached out to me with a spin dump, I could look at it in about 30 seconds decide is it my issue or not. Because you know Europe. Right. Right. Exactly. Right. Oh, ah. OK, right. And in the spin dump, we will see which apps are appearing most frequently. Is that correct? To you'll see every app. Right. OK. All right. OK, got it. And that's actually an important point, you know, for privacy concerns, when you hand off a spin dump, it's got a list of every single process. It can't see files that those processes are accessing. OK. But you know, if you've got apps that you're embarrassed to you know, of using this. Yeah. If you if your Bitcoin miner is there and you don't necessarily want your favorite troubleshooter to know that you're, you know, mining Bitcoins, I don't know why you wouldn't want people to know that, but there you go. There's an example that keeps this all safe here. That would actually explain your performance problem. Yeah, perhaps. Yeah, that's true. That is a fair point. All right, John, what do you do when you're in these scenarios? I got a simple answer. I set menus. OK. For the most part. Yep. So activity monitor is nice and it's free, but I don't think it presents information in as useful a format as I set menus does and that I set menus for a lot of the stats. It'll show you the top candidates are the ones that are chewing the most whatever be it CPU or IO or whatever. So that's one of the first things that I install on any of my Macs.  I think the only time I see my Macs out is when I'm doing ripping media is when it fully realizes the CPU. Otherwise, so that'll be one, one suggestion because yeah, I think the spin-dup thing. That's cool, too. I should actually look at one of those, but that could be a bit overwhelming. But if you're dealing with a developer, I think it will be for all of us. Yeah. Yeah. Yeah. Yeah. I mean, the same thing is with, you know, kernel panics. You know, some of those reports, it's like, what's going on here? I don't know. Actually, I have a general kernel panics. There's extensions and backtrace. That's the key part of it. We'll get a lot of kernel panic reports sent to us. And people immediately look at the process that's identified and they're like, oh, that's the one that crashed the system. But if you look at extensions and backtrace in backtrace, that lists exactly the name of the kernel extension that caused the panic. And that just gets you to the answer a lot faster. Okay. And sometimes I'll see like last loaded or last unloaded. I think that that sometimes points in the right direction. I don't know. Yeah. I used to... Oh, go ahead. Sorry, John. No, go ahead. No, I was going to say, I used to... I'm nearly certain this has changed. Activity monitor has never shown what I'm about to talk about, but this command line utility that is very similar to Activity Monitor called TOP, used to, in macOS, show IO weight states as part of CPU usage. So you could see how much of the CPU's time was being spent waiting for input output, usually disk operations, though not always. And I always found that kind of handy when troubleshooting various things. And the various Linux machines that I still administered, the versions of TOP on those show IO weight states. But when I was looking the other day as this question came in, I realized Catalina does not. I'm not quite sure when it went away, but it sure was nice to see there, but I don't get to see it anymore. At least I haven't found a way. So if someone knows feedback at MackieCab.com. He said feedback at MackieCab.com. Feedback at MackieCab.com. That's correct. So so do you know of any way to to see those, Mike? IO weight states are probably a thing that enters your world quite a bit, I would think. Boy, only because I can see that the information that I need from a spindome. I don't remember the last time I looked at TOP. OK, got it. Yeah, makes sense. Makes sense. Yeah. Yeah. All right. Well, interesting, though. The other thing I could think of if you just want to, you know, once you get a machine would be to get a benchmarking utility to just make sure that it's working as it should. I think the last one that I used geekbench is one of them. For disks, we like a black magic speed test. That's a good one to make sure your disk is running as fast as it should. So those would be some other suggestions. Yeah. Yeah. Black magic speed test is super helpful. Any time, you know, I get a new disk, either one that I bought to use or one that I've gotten to review, the first thing I do is run black magic on it and just make sure that I can see the speeds that I'm expecting out of it. And I've seen a couple. There's a, in fact, one company I've been going back and forth with they've shipped me a few disks and it's like, it's super slow and they can't understand why, but every model they've sent me is super slow. So I'm curious about my meeting with them at CES next week, but we shall see. So cool. Do you want to talk about encryption a little bit, Mike? I know this episode is sort of winding towards what our natural completion would be, but encryption was a topic that came up in our pre-show discussions. Is there something that you can encapsulate sort of an ad here for us? Yeah. The one thing I wanted to point out is it's been a real top support driver. One of the frustrating things in Catalina is that we can't take an encrypted volume and create a volume group with it. So if you've been making a backup on Mojave to an encrypted volume of your startup disk, when you upgrade to Catalina the first thing that you see when you try to run your backup tasks is you're given one or two equally sour choices. You have to either erase your backup disk and start from scratch or decrypt your backup disk, wait for it to finish decrypting and then run the backup and then re-enable the encryption. And it's been really frustrating. You know, the only tools that we have to work with APFS are the command line utilities, primarily DiskUtil. And for this one particular task, creating a volume group, we just can't do it with an encrypted volume. So if you're, you know, still out there and running Mojave and thinking of upgrading to Catalina, that will be potentially one thing that you have to work with. And it's not so terrible. You know, even erasing the backup and starting from scratch, it's a one-time thing once the backup is complete, you boot from the backup and you enable encryption and then you're on your merry way. But yeah, that's certainly been a driver. And it's frustrating for us having to take the heat on it because the tools for working with it just aren't there. That's the handboard dealt. Could you turn off encryption on an external drive, migrate it, and then turn encryption back on? Yes. That's one of the options that we offer. Got it. Okay. Okay. That's just, it's a really time-consuming option. Very. Yeah, right. And I mean, you have to wait for the decryption to complete. And aside from the disk utility, command line utility, there's no interface for it. So. Oh, right. Because when you encrypt an external drive, it's not using the T2 chip even if you've got one in your Mac. Right. Right. Yeah. Because encrypting with the T2 chip is, it, but it's, it's always encrypting. Yeah. Right. You know, it's either, are you adding your own key to this process or just letting the T2 chip handle it is really what encryption means with internal drives on T2 equipment machines. But with external, it's got to be that old style. Let's go do the entire drive process. Read and write. Yeah. Yeah. Hadn't thought about that. That makes sense. Oh, yeah. Okay. Man, this has been so enlightening. Okay. Before we all go here, I do want to do a little bit of fun and that is cool stuff found. So we've got a couple, and I think Mike has one. I have two, but only because they're related. I've started playing around with smart locks, specifically fingerprint locks. And I think I'm going to see a whole lot more of this stuff. Or we are, John, next week at CES. But the Lockly Secure Pro is one that I just installed in my house. You can get it for either a deadbolt or a latch. It becomes your... If you get it for a latch, it becomes your door handle. It replaces either a deadbolt or a latch in their typical position. And you can unlock it with a key. You can also unlock it with a passcode. You can also unlock it with your phone. And you can unlock it with a fingerprint if you get the Secure Plus model, which is pretty cool. We've got one door in our house that we use a lot. I use a lot for coming back from the office, but oftentimes in the morning, it is not the door I go out. And it is frustrating, especially when it's cold outside, to get to the door, maybe have a couple of things in your hand and bam, nope. And of course, I don't have my key in my pocket. So I have to climb up the stairs and go around. And I mean, look, first world problems, but they can be solved with a first world solution with this Lockly Secure Pro. It's pretty cool. And it works. The I think the pro version is the part that includes a Wi-Fi connection. So I can have the lock linked to like the Amazon A-Lady or Google Assistant or just my phone, where I can A, know whether the door is locked and or open or closed. But also I can use the A-Lady to tell it to lock the doors or unlock the door. So now that door, I don't have to worry about locking it at night. I have my Amazon routine that just locks the door and now I'm good to go, which is pretty cool. And to unlock the door with the A-Lady, I have to manually say a pin code that I have taught it. So not anyone can unlock the door, which is good. Along the same lines, the tap lock is at tapplock.com is a fingerprint-capable padlock, which works really, really well. And they say it can store, depending on which one you get, the light can store up to 100 fingerprints per lock and the one-plus can store up to 500 fingerprints per lock. But super handy to be able to open a padlock with just your fingerprint and of course manage it from wherever you want, which is pretty cool. So both of those are interesting. I like what we can do with our fingerprints, so it's pretty good. Mike, what's your cool stuff found for today? Yeah, one of my Christmas presents was a smart oil gauge. And for people who don't live out in the Northeast, they may not use fuel oil. So it may be a completely foreign concept. But here in the Northeast, a lot of people use fuel oil. And one of your concentrators in the winter is checking on the level of your fuel and ordering fuel. And the neat thing about this smart oil gauge is it connects to your Wi-Fi network and it tells you the level of your fuel and it checks it every hour. So not only do you know what the fuel level is, you can also see how your usage changes over time. And we actually just recently got a heat pump. So I was really curious to see how much does our oil usage change, especially on a much colder day versus a day where the heat pump can be used primarily. Right. So I'm a home automation nerd. And this is just data that is fueling my hobby. You know, it's funny. I mean, we were all in the Northeast here. And just yesterday, my wife texted me. She said, you know, I noticed that our oil tank was low. We're on auto delivery, which is the typical thing in this area. It's not necessarily what everybody has, but we certainly do. And for whatever reason, our usage is a little higher than it previously has been. So they weren't planning to come for, I don't know, 10 days or something. And they're like, oh, yeah, if it's down below a quarter, they're like, we'll be there Monday. So yeah, this would have been, and she just happened to notice, you know, this kind of thing would have been a good addition. How difficult was it to install on your tank? I have a Ross tank, so it didn't have, I didn't need to use any tools at all.  Yeah. Actually, the hardest part of setting it up was getting it onto the right wireless network. Isn't that always the problem? I had that problem with the Lockly, too. I don't know why, but it, I think because we have our 2.4 and our five gigahertz networks all named the same within our mesh, you know, meshified world here, that sometimes confuses some of these smart home devices, but it took two tries, I guess, for me, and then I got it there. So yeah. Thoughts on any of this, Mr. Braun? Don't they have a little, yeah, that most oil tanks have like a little float that shows you the level, but I guess they're not too reliable, and you have to go outside and look at it. Yeah. Our tank is actually, our oil tank is inside. I think building code around here anyway says that propane has to be out above ground and outside, but oil we're allowed to have inside as long as it's above ground. Well, that's what we have. I don't know if that's current building code. Interesting. So you have gas. No, I have, well, I have oil in the house and I have propane in the office. Yeah, yeah. You don't, you get yours from the municipal utility, right? So you don't have a tank, correct? Right, yeah. Yeah, I'm just on gas and less supply or as much as I want. Christ. Yeah, yeah, I know. Yeah, yeah, yeah. Yeah, that's good. That's good.  They replaced the metal pipes with plastic and also moved my meter to more easily read it, but. Right, you would need a meter for that. That makes sense. Yep. Yeah, well, they've pretty much all my meters are F enabled right now. Yeah, my water. Yeah, they've, so that poor guy doesn't have to drive around and read it manually. Water, yeah, the only, the only, we have, I mean, we have a well. So the only thing we have pumped in from the street is electric, well, and cable because heaven forbid, we don't have wired internet to the house here. That would, that would be terrible. So, well, Mike, thank you so much for joining us here on the on the show. This has been, I've certainly learned way more than my quota of five things. And a lot of that is very much thanks to you. So, so thank you, Mike. You're welcome. Thanks for having me. Yeah. And I also want to point out that the timing of this is, is, you know, this show will come out on Monday the 6th, which is after John and I, well, while John and I are in the thick of CES, that is coincidental timing. We had Mike on as a, as, as the guy that knows the stuff that he just told you all about. This was not an episode that Mike paid to be on or anything like that. It was our pleasure to have you as a guest. However, I, while I have you here, I do want to thank you for yet again, but also many times in the past, being a coverage sponsor for our CES coverage, Mike. So thank you to you and the team there for that. Very, very much appreciated, my friend.  And it really makes a difference, you know, waiting through all the, the waiting through at CES begins, for me, about six weeks ago, as the requests start coming in and I start deciding what is worth covering. I think somebody did the math. If you didn't have any travel time between booths and certainly no travel time between expo halls, if you were to want to see every vendor, you have 23 seconds to spend with each vendor. So then that's assuming you don't take lunch or bio breaks and you are at the show floor, the entirety of the available time that it's open. So we have to wade through it all and then go and do the coverage. So thank you, Mike. And thank you to all of our CES sponsors. So yeah, good stuff. John, you have anything to add before we, before we wrap this up and move on? I'm with you and that you got to have a plan for CES. I didn't really have one the first time. I'm just like, ah, no, I can cover a trade show in a day, or maybe two. No, I laugh only because I had exactly the same thought my first time. And it doesn't matter. People told me, dude, you got to have a plan. I'm like, come on, I'm an old pro. I got this. And then I go and it was like, wow, that was a waste of a week. Cool. I made it home. I'll do it differently next year.  You're like, ah, I got this. And then you get home and you're like, okay, I got to have a plan. Yeah. Yeah, it makes the week more productive, but also more enjoyable, which is part of the key. There are a lot of people that dread CES. I really, since I kind of came up with my, my ability or my decision to plan for it, I suppose I always had the ability but makes it makes it way better. So we'll be reporting from the show there. We will likely take over the daily observations podcast here at Mac Observer once or twice throughout the week to to chime in on some stuff. And we might even throw this in the Makikeb feed too, because why not? You know, we'll make sure everybody can hear. And oh, a bit of Makikeb premium business. Last night, Adam Christensen and I stayed up and solved the Apple pay problem. So no matter what you want to do, a subscription, a one-time contribution, anything like that, from what we did last night and the tests that I did and the money that I spent myself on Makikeb to make sure the test actually worked, Apple pay actually works. So if that is your preferred way of doing it, by all means do it and let us know how it goes. So and thank you, of course. All right. Well, that does it. We told you how to email us. We told you about Makikeb premium at makikeb.com slash premium. And John, pick one favorite way for people to get in touch with us. Oh, my favorite way, at least for now, is Twitter. How do you reach us on Twitter? I am John Fron. He is Dave Hamilton. The podcast is Makikeb publication is Mac Observer. There's Pilot Pete, who is piloting right now, I assume. And Mike, where are you on the Twitter? If you want to share that. At Bombik Software. All right, very good. BOMBICH software on Twitter. As your very first business card I ever got from you, said on the back, like the pen, not the dog, which I really, really like. That's right. Where else can people find you, Mike, just in case they want to find you and maybe pepper you with even more questions, but hopefully we got everything answered here in the show. Certainly our website, bombik.com. But if you've got a question about CCC, you know, right there in the help menu, choose ask the question about CCC and that goes to our help desk staffed by six people and we're stoked and ready to help you out. You rock. Thank you so much for coming on the show. Thanks everybody for listening. Thanks, of course, to our sponsors for this episode, which were techsexpanner.com slash podcast and expressvbn.com slash MGG, all of our ongoing sponsors, including OWC, Barebones, Iro, Linode, thanks to all of them as well. As I said, thanks to our CES sponsors, which include Bombik Software because you guys rock. And we'll see you next week, but we'll also probably see you throughout the week. So thanks for listening. See you next time. John, before we leave for CES, any advice to send to all of us, including Mike, even though he's not going, do you have any universal advice that might apply everywhere? Yeah, probably since we have to get on airplanes and deal with the TSA. For anyone heading out to CES, my advice to you is don't get caught. Made up.