Welcome to the second day of GPN 2022. And yeah, hi. This is a talk about Ersing, which I personally use every day to transfer files around. And Michael Stabelberg prepared a talk on his re-implementation of this in Goal. Probably some of you know him already because he's the creator or behind the I3 time manager in Linux. And I personally am very curious to learn something about why you did this and what you discovered doing it. Thank you so much for being here and enjoy the talk. Thanks very much. So, you know, as you see, the talk is titled Why I Wrote My Own Ersing. In order to explain this to you, I'll need to first explain a little bit what Ersing is about. Then we're gonna talk about Fiber 7 and about my projects, Router 7 and Go Crazy. And hopefully, by the end, it will all make a little more sense to you. So, can I get a quick show of hands? Who here has used Ersing before? That is most hands. Who has never used Ersing in their life? One per two, oh, okay, cool. Okay, so let me show you a quick demonstration, also maybe as a refresher for all the rest of you. What I'm doing here in this video is I'm copying the slash ETC directory of a Debian server with SCP on the right-hand side and with Ersing on the left-hand side. And you can see that Ersing just completed almost instantaneously. It's much, much, much faster than using SCP, right? You've probably made this experience yourself, which might be one of the reasons why you're using Ersing. When you look at the documentation, the way that Ersing describes itself is as a fast, versatile, remote file copying tool. What does that mean in practice? It means that whenever you have two file sets that you wanna synchronize, you can use Ersing. There are many use cases for which this is a useful tool. One of the original usages actually was to synchronize mail directories. So if hosting your own email server is your kink, then you can use that to read emails on your laptop and not on your server. Then, obviously, many of you might be doing backups using Ersing. You can use Ersing to do partial backups, to do full backups, to do incremental backups using hard links. Really, when they say it's versatile, they mean it. There's an option for almost any behavior in Ersing. One of the things that I like to use it for is to mirror source trees. So typically, before I travel to a conference like this, I wanna just grab my work from my PC at home so that I can continue it here. One option to do this is to commit everything nicely, create branches, et cetera, but I'm way too lazy for that. So what I do is I just Ersing my dirty git directory over, and then I have all of the state, right? I don't need to clean it up beforehand. It's very handy. Then, if you've ever used a Linux distribution and installed a Linux package, that package will have touched at least one, most probably multiple Ersing copies in the process of getting to you because the whole Linux distribution network is all built using Ersing. And then lastly, if you just wanna upload your website to your server, to your web poster, or if you wanna go all the way and build your own CDN across multiple continents, Ersing is a great tool to just distribute all of your content like that. So a little bit of history. Some of you might be surprised to know for how long Ersing has been around. The guy on the right here is Andrew Tritchell, and he is famous for starting the Samba project in 1991, 1992. A couple years later in 1996, he first published a post about Ersing in the CompOS Linux Announce group. Then a couple of years later, he published a PhD thesis that outlines how the Ersing algorithm works. So if you really wanna know about it in detail, you can check out the PhD thesis. But there's also a presentation by Andrew at the Ottawa Linux Symposium just a year later where he develops Ersing from first principles, essentially, in the presentation. And I thought it's a very nice presentation. So if you wanna get to know a little bit of the reasoning behind it and how you would come up with a design like this, then I would encourage you to look at that presentation. All right, so how does Ersing work? I have a sketch here of protocol version 27. Why is it protocol version 27? Because in later protocol versions, the protocol got a lot more complex. In fact, most of the alternative Ersing implementations and there's multiple of them. For example, there's open Ersing from the BSD world. They implement protocol version 27 because it gets you most of the way and then afterwards you have sort of diminishing returns. So in protocol version 27, you can see that on the top side, we have the Ersing sender and on the bottom half of the diagram, we have the Ersing receiver. The protocol works in one direction, but the roles of a sender and a receiver are not tied to the client's server roles. Again, Ersing is very versatile and flexible, so both the server or the client can be the sender or the receiver, it really doesn't matter. These are distinct from each other. The Ersing sender starts the protocol. The sender walks the file system, so that is the file tree that you wanna synchronize, and it sends the file list over the network to the receiver. The three arrows that I have here on the diagram indicate that all of this is pipelined. So there's no synchronization here or anything. Like the sender just goes, it walks the file system, it sends as fast as it can. It doesn't need to wait for the receiver to acknowledge the transfer. Then afterwards, or actually let me dive into the file list. So I'm saying send file list, receive file list, this is what the file list looks like on the wire. You can see here the smallest file list entry is about 18 bytes. This is pretty compact for a file list, and Ersing uses a couple of tricks to make this file list very, very small. For example, device node data and simulink data is only sent when it's actually non-zero. Or maybe even more complex, the file lengths are sent as an int32 by default, and only if they need an int64 to be represented, then it switches to an int64. So this makes like the encoding and decoding of the file is more complicated, but also a lot more space efficient. All right, so back to how does it work. After sending and receiving the file list, there is one step where both of the sites need to synchronize with each other. And that's because they both need to sort the file list so that they have it in the same order. And then afterwards, the protocol only uses handles into the sorted file list. So it's kind of like an index into that array. In later protocol versions, this step has also been eliminated. And our sync became even more streaming and incremental. But in this older version, that's the only stage where you have like one synchronization. Then afterwards, it's no longer the sender who is active, but now it's the receiver who starts doing something. And the receiver goes through the file list that it has received. And for each file, it either creates the file locally and sends checksums if the file already existed, or if it didn't exist, then it just tells the remote site, hey, I just created this in you and I don't have any contents yet. Now, based on the checksums, the sender side does what's called the hash search. And I'm gonna cover that in more detail in a second. This is sort of the main thing that most people think of when they think about our sync and the algorithm, right? Because here is where a rolling checksum is used to then figure out which parts of the file are already on the other side. And the sender can then either send a token or actual partial contents if the contents are not on the other side yet. And the receiver will use either the tokens or the contents to reconstruct the file from scratch. And again, here I have these three pipeline arrows. So all of the directions are heavily pipelined and are only constrained by the bottlenecks that you have on your transfer. So that might be your CPU or your disk or your network, any of these, but our sync in general has no problem maxing out any of these dimensions. All right, so I mentioned the hash search is like the key of the synchronization algorithm, right? So how does it work? Well, our sync first determines a block length for the algorithm to work. The block length is chosen depending on the file size. It's actually the rounded square root of it. And then the receiver sends all of the checksums of the existing file. We talked about this just a slide ago. And the sender goes through each byte of the file and matches existing content. So this sounds like it could be super slow, right? It needs to go through each file. So how can it be efficient? Well, our sync uses a multi-tiered checksum architecture. It has two different algorithms. It has the so-called sum two that's the more expensive algorithm that's a 16 byte MD4 checksum. And why is it MD4? Because end-retritual happened to have an MD4 implementation for Samba. So you just reused it. And then you have the sum one and the sum one is much less expensive. It's essentially two U in 16s. And then I have here this sort of multi-tiered structure where you first have a tag table. The tag is one half of the sum one. You first calculate the sum one. You look into the tag table and if that gives you a target's index, you check if the target's tag two also matches. So now you've verified the sum one. This is very, very cheap. You don't need to do any MD4 calculation up until this point. And you can use this in a rolling checksum way. Then with that index, you again do a lookup into the sums. And in here, these are the transfer checksums. You have the full metadata about the block, right? You have the sum one. You have offset length and index, but you also have the sum two and that's the MD4 checksum. So R-Sync can be efficient because it tries to do as little work as possible, right? All right, and then as I mentioned, the sender now sends either a token, which is a reference to existing file content or the actual file data and then the receiver reconstructs the file. You can actually visualize this quite nicely to verify your understanding of the algorithm. You can just look at the R-Sync receiver and sender and look at the system stats while you start an R-Sync process, right? So you can see at roughly at this point here, I started the R-Sync process and I started it in order to transfer a full Debian server with like 21 gigabytes in 800,000 files. And what you can see here, the rectangle marks the file list sending phase, right? You can see, okay, the server now reads from disk with approximately 40 to 50 megabytes per second and it sends out to the network but with only one megabyte per second. So here you can see sort of the compression rate of the R-Sync file list encoding, right? It needs to read 50 but then only sends a megabyte to the other side. This takes approximately 17 seconds and then you can see that the transfer characteristics change, right? On the other side, previously you had like these reads from the network and if you watch very closely, you can see that it's reading from the network and using more RAM because it needs to buffer all of it and then sort it, right? But then afterwards, once you're at the end of the file list building, you can see that now it actually reads file data from the network and writes it out to disk and on the other side, you can see, okay, it reads the file data from disk and writes it out to the network and these go as fast as possible, right? So this slowly ramps up to a full gigabit which is the connection I have to this particular server. Okay, so now you know roughly how R-Sync works under the hood, right? The question remains, why did I write my own version of it? So previously I was running the server in my kitchen as one does, right? And it was hosting R-Sync D. The server was connected to the internet with a project of mine called Router 7. Router 7, you can see it at the bottom right of the slide. It was running on a PC Engine's APU.  It has a one gigahertz CPU for gigs of RAM but most importantly it had three different gigabit ethernet ports. So it's like an ideal platform for doing a firewall, a router or any other sort of network appliance. So then at some point I had to upgrade the hardware for this router because it was becoming a bottleneck in my network. And I decided, well, if I need to upgrade from like a small little embedded device to a fully blown PC or similar, I don't want to run like a server and a router and both of them consume power, I want to merge the server and the router machines. So why did I need to upgrade? So I'm with this ISP called Fiber 7 and they did a hardware upgrade recently. They upgraded their core network from 10 gigabit connections to 100 gigabit connections. But more importantly, they also upgrade their access network, which is the network for the customers. And that network used to be limited to one gigabits and now they added 10 gigabits and also 25 gigabits. So obviously it was clear what I had to do, I immediately ordered the 25 gigabits. I have a blog post about this whole upgrade if you want to learn more about that. It's linked here, the slides will be online afterwards. What you can see here on the right is a picture of the actual hardware that INIT7 runs in their access network. So these are like the new Cisco switches. Each of these lines here is a fiber to a residential customer and they can run with either a one gigabit SFP module, a 10 gig or 25 gig. And then the pink wires here, they go up from the access switches to the aggregation switches. And then from those, you have another fiber that would then be like the previously 10, now 100 gigabits between the pops. All right, so how do you do 25 gigabits at home? Well, you can no longer use an APU embedded device. They're just not fast enough. Instead, what I ended up doing is I evaluated commercial off the shelf hardware, but the problem was that these are typically made to like plug them into your rack. They're like loud, they have fan control that spins up every minute or so, it's very annoying. It's not living room compatible. So instead, I decided to do my own build. I did a custom PC build for 25 gigabits and the way you do that is you just build a regular PC and you plug in fast network cards. On the right hand side, you can see the left card of the two. That's an Intel network card that can do two times 25 gigabits. So that's where the uplink goes into. And then the right network card here is four times 10 gigabits. That's essentially distributing it across my home. So if I have multiple machines that need to plug into fast internet, they just go into the 10 gig ports here. So that way I don't need like an expensive switch that has a lot of switching capacity. I can just do all of it from this one custom PC build. And if you're curious about more details on that, again, there's a blog post I've linked it here. Feel free to browse that later. Okay, okay, so now this is why I had to upgrade. Now I was talking about router seven, right? What is router seven? Why did I build it? Well, previously I was using the tourist omnia. And I suspect many of you have heard of this router. It's a great project. It's open source software. It's open hardware. It was even sold on the official in its seven store and they subtitled it. You can see the ultimate nerd router, right? So I was super excited to get it. It worked well. The reason why I originally got it is that it does auto updates, right? So it automatically updates to newer versions of the software, meaning I no longer need to care and shepherd it across the updates. However, in May 2018, it did an automatic update and it broke my IPv6 connectivity. And that's just not great. And the reason why it broke is that the fiber seven DHCP servers, they always sent an option with it where essentially the option says, here's your DHCP lease. And for any further traffic, please contact the DHCP server directly at the following address. Now the open WRT DHCP client used to ignore this option. It was just not implemented. It started to honor this option at some point and it turns out that the address which fiber seven gave out was actually not reachable. So obviously it wouldn't work, right? And I contacted both of the parties in order to get a fix for this. I contacted fiber seven and they told me, well, it looks like a bug, but we don't have the manpower to investigate it right now because we're in the middle of a full DHCP platform revamp. Like any debugging work we spent now just delays the actual project. So it's not worth looking into it too much. Also it didn't actually affect many customers. And the author of the open WRT DHCP six client was actually against my suggestion of making the program more robust. I suggested to them that, if you can't reach the DHCP server at this address, instead of timing out and expiring the lease, why don't you start over? And they wouldn't have it, I don't know why. So unfortunately I was stuck between the rock and the hard place, right? Nowhere to go, like the provider can't fix it, the client can't, what do I do, right? So obviously I can just pin that release and stop the auto update, but that was like not the point of the device, right? I really wanted the auto updates. That's why I got the router in the first place. So at that point I decided, well, you know, it would be fun and more reliable to build my own stuff. And when I say build my own, I don't mean just installing standards like network software on Debian or open the BSD or something like that. I mean, I'm gonna rewrite the whole thing and go, right? So that's what I did. So router seven is a home router that is built on Go Crazy. Okay, now what is Go Crazy? It turns your Go programs into appliances. And what does it mean? It means that no longer do you need to like install Raspbian and update it and maintain it. No, instead you only have four moving parts. You have the Linux kernel, you kind of need a kernel. You have the Raspberry Pi firmware, you need that to boot the Raspberry Pi. You have the Go compiler and standard library in order to build your programs and then you have your actual programs, right?  And notably there's like no C environment, there's no dynamic linker even, no shared libraries, nothing like that. There's no memory unsaved programming languages, no security holes like that. There's only the security holes that I program myself, right? So that's comforting. Then Go Crazy, it originally was created for the Raspberry Pi three. It also runs on the Pi four. We recently added support for the Pi zero two W. And for router seven, I added support for running it on PCs. So with Go Crazy, the way you would use it in your day to day is you just have this one Go Crazy packer command that you use to override an SD card. And the packer command creates a reproducible read-only root file system, which afterwards you can then update over the network. And I have a demonstration here. It's embedded as a video, so that I don't have to have like the hardware here, which is inconvenient to show. So let's sit back and enjoy the demo. Quick demonstration of Go Crazy. In this demonstration, I assume that you have already done the preparation work. So I'll skip straight to step one. In step one, we're gonna run this watch command while I insert an SD card into my computer. You can see that the new device has appeared. And now we're ready to move to step two, where we create a new Go Crazy instance in a new directory. Now, we're gonna use the packer command and tell it which SD card to override. I'll remove the SD card from my computer and plug it into the Raspberry Pi. All right, let me switch over to the Raspberry Pi's monitor output. We can see that Go Crazy started up. The system is now obtaining its IP addresses, time, and showing the utilization on the connected monitor. Switching back to the computer, we can now use the URL that it printed to access to Go Crazy Web Interface. Here, we see all of the running programs and now we're gonna add one more program. To do that, we're gonna modify the packer command for I, specify the new program, and also instead of overriding an SD card, we're just gonna tell it to update the existing Go Crazy instance over the network. All right, now that the device is back and ready to use again, we can see that the new program we installed is listening on port 6618. And indeed, we see the system utilization in our web browser now and can disconnect the monitor. So this is like what the day-to-day feels like for Go Crazy, right? Now, you might be wondering, well, what did I write this platform for? And one of my favorite use cases that I have is what's called scan to drive. So whenever I get a new paper mail in my mailbox, right, I just unpack it and immediately throw it away if it's not important. But if it is important, I put it into the scanner, I push the scanner button, and then a couple of seconds later, the connected Raspberry Pi will have scanned the whole pages, arranged them in the proper order, converted them into a black and white PDF, which is then uploaded to Google Drive where I can use full text search on all of the documents that I have there automatically, right? So this is very convenient. There's also an option to not store it on the cloud if you prefer to do your stuff locally. Scan to Drive is compatible with any scanner that can use AirScan. So maybe it actually works already with the device that you have at your home. But going back to router seven, right?  Well, you don't have to have that many pieces in order to implement a router. On the left hand side, you can see the internet, right? And then in order to obtain access to the internet, what I need to do is I need to at least get a DHCP v4 lease, right? So I talked to my ISP with my DHCP version four client. I also do the same for DHCP version six. And then both of these clients create a lease file, which is then picked up by what's called the net config D, which is a demon that continuously runs and configures network interfaces. Okay, so at this point, the router itself actually has internet connectivity, right? It has obtained an address and it can reach targets on the internet. In order to pass along that internet connectivity to your local network, you need to have a few more components. You need to have a DHCP version four demon so that, you know, your computer on the LAN can also obtain an IP address. And for IPv6, you need to have a router advertisement demon so that your client can use Zerocon to obtain an IPv6 address. Strictly speaking, this would be enough, but it turns out there's one more component here which is useful to have and that's the DNS forwarder. It's not necessary to have it, right? You can just configure a public DNS server on all of your clients that works fine, but it's good to have it for this particular arrow here where the DHCP four demon feeds the host names into the DNS server so that you can just, you know, do a ping laptop and it will just automatically work, right? So this is like a simplified architecture diagram of how router seven works. There are a couple of other features that I decided to add into router seven because, you know, while you're at it, why not experiment with a couple of nice features and see where it goes. So one of the features that I specifically wanted is I wanted to keep DHCP addresses in order to maximize internet connectivity. And the reason behind this is that the DHCP specification actually says that when your DHCP lease expires, you have to un-configure the IP address. Now, that's typically good advice, but in the specific case of the connection between my router and my ISP, I know that the ISP has a feature enabled which is called port security, which means that if I have the wrong IP address, I will not be able to send out any packets into the network anyway. So the only couple of times where I've seen issues with a five or seven connection is when they had trouble with their DHCP servers. So sometimes what would happen is the DHCP server disappears, and then a regular router would go offline after a couple of minutes because the lease expired and it un-configures the address. But with my router, it will actually just leave the address there until it gets a new one. And chances are that this is good enough to like paper over any connectivity issues that you have because by the time the DHCP server comes back, like you have not even noticed that there was an issue. So this is like a very small tweak that has a big effect, right? There are a couple of other measures that I did to ensure that there would be no more breakages of my internet connectivity, such as the one that I had experienced with the Taurus Omnia before. For example, for all of the unit and integration tests that I wrote for router seven, the test data is actually packaged capture files from five or seven. So really like the specific bytes that I'm sending and that I'm receiving from my ISP, that's the test data, right? So it's very unlikely that I'll introduce a bug and not notice it at unit testing time. Another super cool feature is that router seven is maintaining a ring buffer for any sort of network related configuration packets. So any sort of DHCP traffic, neighbor discovery traffic, that sort of thing is automatically captured into a ring buffer and you can then attach wire shark via SSH to view the contents of the ring buffer and whatever else comes into the ring buffer anew. And essentially what that means is that whenever there is a bug, like one time I had a bug where DHCP addresses in my local network would be reused too quickly, like with the modern Macbooks, Apple does like super quick configuration stuff and apparently there was a bug in there. What I did was when I noticed that the Mac was complaining about a duplicate address in the network, I attached wire shark and got like the last couple of hours of traffic and just scrolled up a little bit and saw, oh yeah. This is like the host that previously had the lease, right? It's super convenient. It's like traveling back in time. So especially if you have the RAM, just capture this and then expose it for wire shark. It's very cool. And then lastly, I wanted to have safe updates and especially also automated rollback so that I can very quickly recover any issues that the router has. In fact, I've decided to set it up by using network boot. So the original installation is via network boot and then all subsequent updates, they either go over the network or they fall back to doing rollback with network boot. And in fact, the safe updater, it actually looks at whether the machine is currently connected to the internet. So I have another demon on router seven which is called the diagnostics demon and that demon essentially does all of the troubleshooting steps that you and I would do if we noticed that our internet connectivity might have some trouble. So for example, it looks at is there an IP address configured on my ethernet interface? Can I ping the default gateway? Can I resolve host names? Can I then establish a TCP connection to a host name, right? Like all of these sort of troubleshooting graph steps, it does them automatically and in parallel. So if you ever think, oh, like, I don't know, my Netflix is hanging, is that a problem with my connection? You just open up the diagnostics demon and it will immediately tell you, no, all check marks, everything's fine or it will tell you, look, your IPv6 connectivity has a problem. So we can not only use this in order to save ourselves some troubleshooting time, we can also use this programmatically. So the save updater will actually query the diagnostics demon and say, well, if you're not online, I'm gonna abort the update to not make things worse. But if you are online, I'm gonna do an update, check if you come back online and if so, everything's fine and if not, I'm gonna do an automated recovery. So this is sort of the gold standard for updates, I think. And it's very good that it can work like this. And I've been automatically updating my router every day for the last couple of years without any trouble. Okay, okay, so I mentioned router seven, it needed to be updated, right? It needed to have better hardware and it needed to be merged with my other kitchen server. So what I had to do with was I had to move all of the server software from the Debian installation on my kitchen server to go crazy. And that means because there's no C code on go crazy, no user land, I need to use Go programs, right? Luckily for many of the services, this is not a problem. For example, I see a couple of torrent images of my own Linux distribution, I'm gonna tell you that in a second. That program was already written in Go, so just install it on go crazy, configure it, it runs perfect. I was serving files with Nginx, but instead I could use Caddy, which is a good web server that's written in Go, just configure it on go crazy, it runs perfect. But here comes the problem. For R-Sync, there's an R-Sync server, the original one, but there was none in Go at the time, right? So obviously the right way to fix this is we're gonna first do some prototyping, run the original R-Sync and then re-implement our own, right? So for prototyping, this is a relatively laborious procedure, but just to illustrate how it could work, right? So on a Linux computer, there is a tool that's called the Freeze program. And what the Freeze program does is you can point it at R-Sync, for example, and it will copy the R-Sync binary and all of the libraries that it depends on. So this includes the linking loader, this includes, I don't know, lib SSL, lib Z, whatever else that R-Sync links into.  But then obviously you don't have the benefit of having a unified installation where everything is in Go. You don't have the insight into it and most importantly you need to manually maintain this, right? Like if you do a snapshot of your software, you are on the hook for updating it and the whole point of Go crazy is to have auto-updating devices. So this is not a great permanent solution.   And while we're at it in implementing a new R-Sync implementation, we can achieve a little bit stricter security by default for the particular use case of serving data to the world. So in the Go crazy R-Sync implementation, all of the configured modules are mounted read only into a Linux mount namespace and then obviously as most servers do, the server will drop its privileges and change route into that mount namespace. So even if there is like an issue, if there is a bug in my implementation, all an attacker can do is look at the files that I told them to be world readable anyway, right? So the attack surface is massively reduced by that. And then there are a couple of other nice properties that we have here. I mentioned the privilege dropping. It will actually verify that the privilege dropping is configured correctly by trying to write a new file into what you told it is a read only module. And if that succeeds at just the boards, right? There's no chance that you can accidentally misconfigure it. You need to be very explicit in what you configure.  So just by virtue of doing like a clean re-implementation with better starting properties, we end up better off. I've also decided to build in an SSH listener, which is very convenient in Go. There's great ecosystem of SSH related tool. The SSH listener can be anonymous for if you just want to have a secure transfer but allow anybody to do that transfer, or it can be authorized. And it's entirely separate from your system SSH setup. So you don't need to create a user account where you could have misconfiguration or data leaks or anything like that. It's entirely separate. In fact, Go Crazy by default doesn't even have an SSH demon running. You can start one, but it doesn't have one by default. And yet, our sync on Go Crazy will work because it brings its own SSH listener.  You can do a simple yet performant implementation and you can leverage the full Go tool chain and ecosystem. So you can use Pprof profiling to figure out where the bottlenecks are. You can add Prometheus metrics to monitor your stuff. The list goes on and on. So what did I actually implement in this project? When I say an our sync implementation, that sounds like a lot. So the first program that I implemented was called Go Crazy our sync D for demon. And this would be the role of the sender as we saw in the architecture before and the endpoint of a server. So colloquially, the use case would be to serve files to anyone who wants them. Then I later added the Go Crazy our sync tool which is a receiver because somebody was like, well, now you have the sender but the receiver would also be cool. I have a good use case for it. So I decided, okay, let's look into it. It wasn't too much work. So we have the receiver role as the endpoint of a client. So this would mean it downloads files for you. To be done are the other remaining role and endpoint combinations. So we don't yet have a sender as a client which would be the use case of uploading files. And we don't have a receiver as a server for receiving files. But given that the functionality of the others is already there, this should be, I'm gonna say an elaborate refactoring exercise but nothing groundbreaking. All right, I mentioned that Go allows for simple yet performant implementations and I was actually surprised when I benchmarked this myself because it turns out that at protocol version 27, the original Tridge our sync achieves about four to five gigabits and Go Crazy our sync without me even trying too hard was already faster at five to seven gigabits. So how can that be? Well, the bottleneck is the MD4 checks and calculation and the our sync protocol is sequential as you've seen on the slides before. But what you can do very easily in Go and what is much harder to do in C correctly is you can move the hashing, the MD4 hash calculation into a separate Go routine and that way use two cores instead of just one core and that makes the whole thing much faster. So you get two extra gigabits out of the same protocol. Now, Tridge our sync actually didn't stop at protocol 27 and one of the cool features of the newer protocol versions is that they can negotiate which hash algorithm they want to use. And in particular, they default to the XX hash algorithm which is much more modern and much faster. And then you can actually reach 10.4 gigabits with Tridge our sync. I actually tested it on a 25 gigabit line. So this is really, it's not constrained by running on a 10 gig line. It's really just achieving 10 gigs and no more. I haven't been able to test how good Go Crazy our sync would be in that situation because we don't implement the newer protocol versions yet. So that is a future optimization that we could do as well. Cool, okay. So I've talked about the kitchen server but what does it actually serve? So let me introduce you to Distree Linux. Distree Linux is a Linux distribution to research fast package management. What do I mean by that? If you measure how fast common Linux distributions are when you install your packages, like when you upgrade your system to many new packages or when you install new software on your existing system, it turns out that Fedora, Debian, and NixOS all achieve about six megabytes per second. Arch Linux comes in slightly faster at 12 megabytes per second. They have switched to using the Z-Standard decompression algorithm, for example, that gives them a good boost. And Alpine Linux is the fastest of the bunch that I've tested, which comes in at 18 megabytes per second. And Alpine is sort of like this very small stripped down thing that's primarily targeting containers. So Distree Linux tries to approach this issue of how can we install packages faster or how can we install them as fast as possible by using a lazily mounted read-only SquashFS image for all of the packages, including system packages. So you might be hearing, oh, SquashFS images, that's kind of like, I don't know, App Image or Flatpak or one of these other new application container formats or distribution formats. And those, yeah, they do use images, but what I'm showing here with this Linux distribution is you can use the same approach for any package, even system packages, not just your application. You can use it for everything on your machine. So with this architecture, package installation becomes just inserting a new file into an append-only package store. And this you can do very quickly because you don't need to synchronize your rights to disk. It's not a big deal if the file is like half added to the package store because it's not effective, right? Like any sort of temporary file, you can just clean up afterwards and retry. And with this architecture, you can saturate even 10 gigabit network links. So you can install packages at 920 megabytes per second. And this is not just like a synthetic HTTP download. This is really like, you type distree install, your packages, and at the end, it clocks in at 920 megabytes per second, including writing to disk, right? So this is like much, much, much faster than even the fastest of Linux distributions. And it just shows what huge potential we have in order to make our stuff faster, right? There's no good reason that it has to be this slow. Many more details about Distree Linux you can find in my talk here at the GPN conference from 2019. Recording and slides are linked at my website. All right, so achieving these 10 plus gigabit installation speeds requires that obviously people who wanna try it out have a fast internet connection and that they have servers that are closely located to where they are. So I need efficient distribution of all of the Distree files across continents. And as I mentioned before, Linux mirror distribution is one of the prime use cases of R-Sync. So with a good R-Sync demon, we can sort of build out our own mirror structure. I also wanna do more work on figuring out how to set up this infrastructure. So to that end, I've recently started implementing like a little 25 gig test lab setup at home to really like max out where we can go. Now, this was the original motivation, right? To have like a good auto updating, secure safe R-Sync server for Distree. But then I realized, wait, now that I have an R-Sync implementation, we might as well use it for other things. For example, circling back to the scan to drive appliance with which I scan my paper mail into the cloud, I can now actually use the R-Sync demon as a backup server for it, right? And the way it works is the packer command that we talked about before. I just add the R-Sync demon so that it becomes included in the image and installed on my Go Crazy instance. Then I configure the flags for it. This might look a little laborious, but it's not too much. The two flags we need are just the Go Crazy config file path and the fact that it should start as a demon. And then in the Go Crazy config file for R-Sync D, we tell it to not actually namespace. So the file permissions are actually honored. This is not well readable data. This is not Distree. This is personal private data that should not be well readable. So I wanna actually authorize everything, right? I have the authorized SSH listener configured and I configured just one module that is just the scans module. So all of the scans that I do, they go onto Google Drive, but also onto the SD card. And then I use R-Sync from a PC to do a backup of all of that. So on the PC side, it works as you would expect. You just configure the SSH ports and the hostname and a passwordless identity file so that you can then actually use a cron job to automate that without having to enter your password. And then afterwards, you can run R-Sync via SSH as usual, right? Just like any other R-Sync demon, you can just use this command and it syncs over your backups onto your computer. So at this point, you might be thinking, okay, that sounds cool, but how does one approach such open-ended projects, right? Like if I wanted to reimplement something, where do I even start? A couple of tips that I can give you is to start in the right mindset. You need to identify really the smallest possible milestones and then focus on achieving just that. And then afterwards, extended step by step, right? It really helps to clearly document the current status and any limitations that you have, not just for yourself, but so that you can also then publish it as a community project. And then you rely on community contributions. So when I started the project, as I mentioned, I just implemented the serving side. I published it, somebody asked me for the receiving side. Somebody else actually thought, hey, it would be cool to use this as a library and not just as a program. So they contributed a couple of pull requests to turn it into a library. And then there's a couple of other people who are interested in even like paying me to implement features into this, right? So it can go quite quickly, quite far. And you never know what people might be interested in before you actually publish it and see what the response is. Then in the process of doing the building, you need to build tools first, right? You need to focus on visualization and custom tooling in order to make your job as easy as possible. At least for me, I know that I need everything visualized. And a quick step for this is in R-Sync, there's already debug output. It's not graphical visualization, but it does visualize what R-Sync does under the hood. And maybe you can just add a couple of more print messages, recompile R-Sync, it's quick to recompile it, and then go from there, right? That could be like a first step. Maybe you also want to do like a diagram, right? Or some other sort of graphical visualization. Then in the test suite, compatibility tests really help. So in order to be sure that my R-Sync server actually serves stuff that other people's clients can use, I did extensive compatibility testing against the Tritch R-Sync implementation, that's like the original that you all know and use, but also against Open R-Sync from the OpenBSD people. And if there's other R-Sync implementations, it might make sense to add them to the test suite as well, just to make sure that they're all compatible. All right, so that's all the slides I have. I have put a couple of links up here to the Go Crazy application platform, sorry, to router seven, to the Go Crazy R-Sync implementation, if you want to try it out, just one go install command, you have it on your computer. I have actually another repository I haven't talked about yet here, which is R-Sync Prom. If you wanna have like Prometheus monitoring and alerting for your R-Sync transfers, like if you have automated jobs that you do for backups, for synchronization, then you can have an alert when it no longer works. And you can also track how big are my file sets, how long does it take to transfer them, et cetera, and build like a nice dashboard. And then lastly, a link to Distry Linux. So if you're curious about that, feel free to try it out. You can just download it on to like run it in a VM or download it onto a USB stick, put your computer from it and try a couple of cool things that it allows you to do. Lastly, if you have any feedback for me for the presentation, you can use the QR code here or later on just click on the slide to fill in the feedback form. So with that, we're gonna open it up for Q&A and I'm gonna say thanks for your attention. Thank you so much for this thorough explanation. If you have any questions from the audience, raise your hand, I will come to you with the microphone so everyone can hear you. First question coming up. Pass the mic, please. Yeah, thank you for rewriting this in Go. I'm asking about synergies because I'm also in a place where we just rewrote a ton of things in Go, which is the Eurot project. I guess you know it.  And my question in turn would be because you're using I think the C Busybox, right? Right, yeah. Have you considered using Eurot or some other Go clone of that? Yeah, we could totally do that. So the only case where we're using Busybox just to be clear is whenever you use the break glass program. And the reason why it's called break glass is because it departs from the Go crazy model where the primary idea is everything on your device should be in Go, right? But sometimes it's just so useful to use like a Busybox shell and interactively poke at I don't know the sys file system or proc or whatever else you wanna do, right? And for that we have the break glass program where essentially it breaks the class in that it opens up an SSH listener, you SSH in and you bring your own user land. So what I did is I just built a Busybox statically myself copy that over as a tar ball and execute that. But you're free to like build Eurot yourself copy that over and use that. And if you think that works well, if you try it out we could even like replace the standard Busybox or have like a prepackaged Eurot that people could just pull in very easily by just adding the package name to their Go crazy packer command line. I think that'd be cool. It's certainly in the spirit of the project to use Eurot, so I'd be happy to make that work. Yeah, awesome, thank you. And regarding break glass, it sounds a bit like the CPU command that we also brought. Yeah, it's a bit similar, but it doesn't go quite as far. Yeah, the CPU commands definitely very cool. So if people are unfamiliar, they should check it out. Yeah, okay, cool, thank you. Further questions from the audience? No more hands. Okay, well if any question comes to your mind please feel free to find me at the conference. I'm always happy to chat about this project or any of my other projects.  I should be easy to find. All right, thanks, and enjoy the conference.


We're now going to take a look at a concept that is quite useful to use when you're solving problems involving heat transfer. It relates to the flux of thermal energy going through surfaces and what happens when you go from one surface or one material into another. And it's referred to as being the surface energy balance. So let's take a look at that now. And so what we're going to do, we're going to consider a scenario where we have a solid, so we have conduction, and then we have a fluid outside of the solid, we have convection, and then we have some surroundings. So we have radiative heat transfer. Okay, so what we have here is we have a scenario where we have a solid, and so the solid is shown here, so this here is all solid, and then we get to an interface where we then transition into a fluid, so there's a fluid outside of the solid, and then far away we get to some surroundings that is at temperatures surrounding. And when we're dealing with these types of problems, first of all we know that within the solid we have conduction, and we've shown the temperature distribution here, but we know that we have conduction coming through in this direction because it's going from the higher temperature to the lower T2. Now when we get to the wall, what's going to happen is the conductive heat transfer, or the energy flowing in, is going to leave via two mechanisms. One of these is going to be radiative, let me use a different color there, I will use red. So we will have radiative heat transfer, and we are also going to have convective heat transfer, and given that I put the fluid in green, what we'll do is we will put convection in green. So we have convection radiation, and so in solving these types of problems, it's often useful to draw a control surface right on the interface between where the solid transitions into the fluid, and so what I'm going to do, I'm going to sketch a control surface, I'll do that red so that it's easier to see. So what we're going to do, we're going to assume that these control surfaces exist on either side of the solid fluid interface, and we're also going to assume that they are infinitesimally large, so that the delta between them is sorry infinitesimally small, that they're right next to each other, they include no mass or volume, and so the control surface is going to be kind of a theoretical construct, but it's a very useful one that we can use for solving problems whenever heat is transferring from one media to another, and one material to another. So let's take a look at conservation of energy across the control surfaces, and so we know conservation of energy says Q in minus Q out is equal to zero, so we're assuming that this is a steady state scenario, so there's no build-up of energy, but even if it was, the control surface is infinitesimally small, so there would be no place for it to build up. But let's look back at our diagram here, so what is coming in? We know we have conduction coming in, and then what is leaving? Well we know we have radiation and we have convection, so those are the three things that we will put into this equation, so in here we can then rewrite what is coming in is Q conduction, and what is leaving is going to be heat transfer due to convection and heat transfer due to radiation, and those all need to sum to zero. So this is a very useful equation that can help you solve a lot of different problems. Basically it's an energy balance that you do on this infinitesimally thick surface, we call it a control surface, and we will use this while solving problems. Okay, so that is the idea, the control surface and the surface energy balance. What we'll be doing in the next segment is we're going to be looking at an example problem involving all three, conduction, convection, and radiation. We'll be using the energy balance in order to solve it.


Hi, good morning, and congratulations to the Class of 2015 of Immaculata. I'd like to say first of all thank you to the administration of Immaculata for thinking of us here in Sudan and for giving this award. I'm rather humbled by it, but I would rather quite appreciate it also. So thank you for that. My name is Tom Katina, and I'm an American medical missionary and a medical doctor and surgeon. I've been here in the New Amounts of Sudan for the past seven years. People often ask me why I do this kind of work, why would I give up a practice in the U.S. and the life there and come out here to the middle of nowhere to work in this field. My answer is very simple. Christ gave us a directive to take care of the least of my brothers and sisters, and I've taken those words quite seriously and said, no, let me go ahead and do that. What He's asked me to do. So that's the simple reason why I've come out here. Christ also said, you've got to pick up your cross every day and follow me, and often that's what it feels like here, where there's a lot of challenges and a lot of heavy work. But these words are a great encouragement. We know that Christ suffered himself, and that's all we can do to try to follow him. We know it's also going to entail carrying a cross. So I know now everybody's trying to give you advice and telling you what you should be doing for your future and all that sort of thing. My only small piece of advice would be whatever job you do, whether it's in corporate America or as a teacher, as a nurse, as a doctor, whatever, take some part of your time to take care of the least of your brothers and sisters in some type, in some capacity, whether it's working in a soup kitchen or volunteering or doing as a full-time job or what have you. Take the words of Jesus very seriously, and I think the rewards you'll get out of that will outweigh any sacrifice you have to make. I think one of the problems we have now in the Western world is a lack of fulfillment. People are not fulfilled. They don't feel happy in their lives. There's a certain emptiness and materialism. I think this is all born out of this idea that material things matter. I would just encourage you to do away with that idea. Remember that people are what matters. Taking care of people is what matters, no matter what job you do. Forget about the material things and pursuing the good life. I promise you if you try to pursue what Christ is asking of us, you will find the good life. Maybe not with a yacht and a swimming pool and all that sort of thing, but you'll find something much more valuable. So I really would encourage you to follow this line of work. Again, I say congratulations and I wish you all the best of luck in the future and God bless you all. Thank you.


I think that we need today to follow faith and reason and not only philosophical reason but also scientific reason and put together in a real dialogue these things like do Pope Francis in Laudato Si that described the situation of the world especially the question of climate change and put this in relation of our task in the world in that come from the scriptures that we are the stewards of of the of there we don't say creation because too much but there and and for this we need faith reason, philosophical reason, values, moral reason and scientific reason. So in understanding how we change the world in order to achieve goals that we can defend and I agree there can be but there isn't necessarily overlap between the goals of say secular humanism and a religious worldview but to the extent that we agree if we agree that those goals are human flourishing namely life, health, education, happiness, richness of experience then how we maximize those is going to very much depend on our understanding of what causes what in the world and our understanding of what causes what in the world comes from science by definition because that's what science is in the business of doing namely understanding the world with the under with the method that none of us is omniscient that we cannot understand the world by revelation by faith by dogma by sacred texts but rather by proposing hypotheses and seeing letting the world tell us whether they are true or false. The fact that we can see ways in which we've improved the world by the application of knowledge emboldens us to think that we can improve it still further. This part for the visible war I completely agree but there are another part of the world that we can say invisible war that is not enough science. We need for this for the great meaning we need something that comes from revelation we say. Why should we believe revelation? Well, revelation to put in relation with faith revelation with that is to say some new intervention in the world that is not the only the creation but is a new intervention of God in the world. But if it's revelation it has not been revealed to me it's been revealed to you. What are you going to do about the fact that there are people like me people in this audience people in the world who have not seen the same revelation that you may have seen who have no plans to accept Jesus as our savior I'm a Jew I'm I'm going to tell you I'm not going to accept Jesus sorry I don't think I'm I don't think I'm that's a big problem in terms of making the world a better place I don't think that the billions of people who have not and will not accept Jesus I don't think that's the world's problem they have not experienced the same revelation perhaps that you have I don't think that's the problem either because we don't need revelation if we can appeal to dimensions of experience that are available to everyone such as evidence such as reason such as a concern with human flourishing.


Hello. In this video I will identify the eight bones of the cranium, 14 facial bones, 26 bones of the vertebral column, and the 25 bones of the thoracic cage. The axial skeleton are the bones found along the midline in contrast to the appendicular skeleton that are the bones found in the limbs and the girdles that connect the limb bones to the axial skeleton. There are eight cranial bones and six names of those eight cranial bones to learn because there are two a left and a right for the parietal bone as well as two a left and a right for the temporal bone. A pneumonic device that you could use to help you remember the names of the cranial bones is pest of six P E S T O F six. And so six because there's six different names to learn and then P stands for the parietal bone. So I'll highlight that here. You can see the parietal bone and we're looking at a right parietal bone in this view but there is also a left parietal bone. The parietal bones are the large flat bones that form the convex superior lateral walls of the cranium. Then E stands for ethmoid bone. So we can see here the ethmoid bone. So there's just one ethmoid bone and it's located in the superior roof of the nasal cavity. It does contain air filled sinuses that connect with the nasal cavity. So the ethmoid sinuses are one set of sinuses that are air filled spaces inside of our bones that connect to the nasal cavity. Then S stands for sphenoid bone. So we just have one sphenoid bone. You can see right here the greater wing of the sphenoid bone is visible on the right side with this lateral view of the skull. The sphenoid bone is a medium sized irregular shape bone that forms the anterior floor of the cranium and contacts all seven of the other cranial bones. Then T stands for temporal. So right here we can see the temporal bone is a purple color here in the illustration. The temporal bone, we have a right temporal bone as well as a left temporal bone. So there are two temporal bones just like there are two parietal bones in the cranium. Each temporal bone is a medium sized irregular shape bone that forms the inferior lateral wall of the cranium and the temporal bones contain the middle ear and inner ear and there are three small bones within the middle ear known as the auditory ossicles. So now O stands for occipital bone. Here we can see the occipital bone is a large flat convex posterior wall and base of the cranium. The occipital bone contains a very large opening for the brain to contact with the spinal cord. The last bone in the mnemonic here F stands for frontal bone. So we have one frontal bone that's the large flat bone forming the convex anterior roof of the cranium. So here we have a posterior view of the skull where we can see the occipital bone is forming that posterior wall and then the parietal bones are forming the superior and lateral walls of the cranium whereas the temporal bones are located just on the inferior lateral walls of the cranium. Another major feature that stands out here is this large opening right here that's the foramen magnum it's part of the occipital bone. So occipital bone has a large opening foramen magnum. Magnum meaning really large and foramen meaning an opening a hole through the bone. This is the large opening where the brain connects to the spinal cord. Now here we have an inferior view of the skull so again we have a nice perspective on this foramen magnum that allows the brain to come down out of the cranium and contact with the spinal cord. So that foramen magnum is a part of the occipital bone and that will help you remember which bone is found there on the posterior and inferior region of the cranium. Now we'll go through the 14 facial bones and I also have a mnemonic device a memory tool that will help you remember these bones the mnemonic device is Virgil can not make my pet zebra laugh and the first letter from each of those words is the first letter of the name of one of the facial bones and there are 14 facial bones we have some facial bones that are unpaired there's only one and some facial bones are paired meaning we have a left and a right. So let's start off with v v stands for vomer so here you can see the vomer bone it's located in the inferior of the nasal cavity right along the midline so there's only one vomer bone and it has a plow shape that's where the word vomer comes from it's forming the inferior portion of the bony nasal septum that separates the left and the right halves of the nasal cavity. So next we have c c stands for inferior nasal concha so the c stands for the last word in this name concha and inferior nasal concha is a facial bone there is also a middle nasal concha and a superior nasal concha but those are parts of the ethmoid bone so the inferior nasal concha is its own facial bone and there are two inferior nasal concha a left and a right for the inferior nasal concha the inferior nasal concha are small irregular shaped bones that form a scroll shaped extension that projects out into the inferior lateral wall of the nasal cavity and it functions to produce turbulence in the air as we're breathing through our nose this helps to warm and humidify the incoming air if you look into the nasal cavity of a dog or a seal these animals have really elaborate nasal concha in order to help them with breathing very cold air helping to warm that air up so that it's not too harsh for the respiratory system so now in the mnemonic device we've gone through v and c next is n so that n stands for nasal and right here we can see the nasal bone there is a right nasal bone as well as a left nasal bone so there's two nasal bones and they're forming the bridge the nasal bridge just superior to the nasal cavity okay so next on our list is an m and m can stand for maxilla we have two maxillary bones two maxilly so let's find where did we label the maxilla on this there we have it so the maxilla is the upper jaw bone and there are two maxillia left as well as a right you can see that the maxilla also forms the medial portion of the inferior margin of the orbit or eye socket and it's well as it's also forming this lateral border of the entrance to the nasal cavity so there's another m the next m stands for mandible and we just have one mandible it's the lower jaw bone so here we can see mandible is the lower jaw bone and a interesting thing about the mandible is the mandible contacts the temporal bone to form the only movable joint of the skull the the jaw joint the temporal mandibular joint is the joint that enables us to chew our food or to talk to move our jaw so next we have p p stands for palatine bone now I can highlight here palatine bone and there's a very small part of the palatine bone that's visible when looking into the orbit looking into the eye socket but the word palette means roof of the mouth and if we were looking at the roof of the mouth we would see the palatine bone is forming the posterior third of the bony hard palate the portion of the roof of the mouth that's formed from bone the anterior portion of the hard palate is formed from the maxilla and the palatine bone just forms the posterior third of the hard palate and there are two palatine bones both a left and a right okay so next in our mnemonic device we have z z stands for zygomatic bone we have two zygomatic bones let's highlight with yellow instead of red so we have two of these zygomatic bones and the zygomatic bones are your cheek bones located just lateral to the maxilla they're just anterior to the temporal bones and they do form the lateral margin the lateral wall of the orbits now last on our list here we see L and the L stands for lacrimal bone so lacrimal bone is found in the medial margin of the orbit and there's a nasal lacrimal duct that is an opening through the lacrimal bone into the nasal cavity which allows tears to drain into the nasal cavity so the lacrimal gland produces tears that wash across the eyes from the superior lateral region of the eye and drains down to the medial region into this lacrimal bone through the nasal lacrimal duct into your nasal cavity those tears are constantly draining through that pathway as a way of just lubricating nourishing and cleaning the anterior surface of your eye but the word lacrima means tear so lacrimal gets its word gets its name from the word for tear and it's found there in the medial part of the orbit of the eye socket and there's one in each orbit so we do have two of these two of these lacrimal bones as well so next we'll move on to study the bones of the vertebral column so there are 26 bones in the vertebral column seven of them found in the cervical region 12 in the thoracic region five in the lumbar region and then the sacrum is one bone formed from five sacral vertebrae that fuse together and then the the coccyx or tailbone is the most inferior of the vertebrae so here we can see the structure of a cervical vertebrae cervical vertebrae are relatively small compared to the other vertebrae they don't have to support a large amount of body weight essentially it's just the weight of the head that's supported by the cervical vertebrae and the vertebral foramen the large opening in the center of the cervical vertebrae is relatively large because the spinal cord is relatively large at this region close to the brain so here we see the structure of thoracic vertebrae the most unique feature of thoracic vertebrae that really helps them stand out from other vertebrae is they have costal facets that is they have these little indentations that are the locations that the ribs attach so the ribs attach to the thoracic vertebrae and those little costal facets are the locations where the ribs attach then the lumbar vertebrae are relatively large they're much larger than cervical vertebrae and even a little larger than the thoracic vertebrae because the lumbar vertebrae have to support a relatively large amount of our body weight here we can see the structure of the sacrum and coccyx so the sacrum is found in the posterior pelvic region whereas the lumbar vertebrae would be in the posterior of the abdominal region and then just inferior to the sacrum is the coccyx the tailbone and so the coccyx is the most inferior of the vertebrae so next we'll move on to the thoracic cage which contains 25 bones these are all flat bones and the large flat bone found on the anterior of the thoracic cage this is the sternum so then attached to the sternum and the thoracic vertebrae are the ribs although only the first 10 ribs the most superior 10 ribs will connect to the sternum the the ribs are all attached to the thoracic vertebrae and ribs one through seven are known as the the true ribs we call those the true ribs because they have a costal cartilage attaching directly to the sternum then ribs eight nine and ten all share the costal cartilage of rib number seven and last rib 11 and 12 are the most inferior of the ribs and these are floating ribs that do not attach to the sternum at all so the ribs provide a bony wall to the thoracic cage to the thorax which helps to protect the organs the lungs and the heart that are deep within the thoracic cage