So, hello and welcome to the third day of GPN. I guess I can still tell good morning to everyone because we are at a hacker conference, so I'm not up that early. We are here to listen to some, yeah, new things regarding GO that you can do on a Raspberry Pi for whatever you want to do on a Raspberry Pi. We had a quick, yeah, questionnaire here. A lot of people have a Raspberry Pi in the audience and a lot of people do home automation and a little backup on there. And I'm hoping that there is a new way of doing that. Maybe some of you will then go there. But without further ado, I'm here to introduce Michael. He will, yeah, tell you a little bit more about it. And a warm applause for him. Yeah, thank you very much for the introduction and thanks everyone for coming. So the title of this talk is Build Go Appliances for the Raspberry Pi using Go Crazy. So first I'm gonna give you an overview of what I'm gonna talk about. First of all, I'm gonna introduce what is Go Crazy and I'm gonna do that by showing you a couple of different aspects of the system, like how does it work, but also notably how does it look and feel like when you use it. I'm gonna show a quick demonstration of how you would just install it and get started. I'm gonna talk about which hardware you can use it with and how we implemented fully automated Linux kernel updates. Then we're gonna switch gears a little bit and talk about a couple of notable Go Crazy appliances. So these would be things that other people have built that you might find interesting and wanna run. Then we're gonna talk about notable Go software that you can run.  And then we're gonna talk about building your own stuff with Go Crazy because the conference here is called Gulasch Programmier Nacht, so we better do some programming. And then a very quick outlook to future developments and we're gonna have some time for questions and answers at the very end. All right, so let's jump right in. What is Go Crazy? Well, I told you already it's for appliances. You can use it to deploy your Go programs as appliances to a Raspberry Pi or a PC. What do I mean when I say an appliance? An appliance is sort of the opposite of a general purpose computer. So with the general purpose computer, you just get like, let's say maybe an Ubuntu or a Debian installation or what have you and you can add anything and everything you want, right? An appliance sort of looks at it from a different way. It's not like you want a broad base computing platform and then you can do anything and everything.  So specifically Go Crazy is 100% written in Go, right? And what do I mean by that? The only non-Go parts of the entire system are the Linux kernel itself, which you might know is written in C and it would be futile to try to reimplement it, right? We really wanna use what the Linux kernel already offers. And then there is the Raspberry Pi bootloader, which if you're into the Raspberry Pi, you might know, you need these binary files in order to start up the system. But aside from these two components, there is no C user land on the device. This means there's no G-Lip C, there's no open SSL with security vulnerabilities. There are no package managers. There's nothing like that, right? There's only the Go programs that you decide should be on the device. So the philosophy is sort of a build minimally from the ground up and add only what you need. And that way also keep the complexity and the attack surface very low and self constrained, right? Notably also, like why is it written in Go or what does that give you? Well, you can enjoy all of the strength of Go uniformly for the entire system. This means that all of the components are managed as Go modules.  So in order to do that, you would use the same replace directive that you might already know from working with Go. Also, you'll have all of the other strength and to pick just one that I personally like very much, that would be the very quick compilation times. So working in a system that is written in Go is great for interactive development. It's very quick to make progress and the same is true for Go crazy and I'm gonna show you in a little bit. But first, I wanna show you this demonstration video to give you an idea about the look and feel. Welcome to this quick demonstration of how to first install Go crazy on your Raspberry Pi.  I have prepared an entirely empty SD card and we're gonna jump straight to step two of creating a new Go crazy instance. With the Gock new command, we can see that we now have a new instance created and we can use the Gock override command to write it to the empty SD card. Now I'm going to insert the SD card into my Raspberry Pi. Now that the Raspberry Pi has started up, we can copy and paste the link of the Go crazy CLI gave us to open up the web interface of Go crazy. Let's add another program to this Go crazy instance. We're gonna use the web stat command and this time instead of using Gock override, we're just gonna say Gock update. Now that the device is back, we can reload the web interface and indeed we see the web stat command has been added and it should now be available on port 6618 and indeed we now see the same output in our web browser as on the connected monitor. So I think this should give you a good overview of what it feels like to use it. Now let's explain in a little bit of detail what happened. What did we just see? So the stuff that we're writing onto the SD card on disk, what's happening here?  The first partition is called the boot partition. On this one, you will have the bootloader files I already mentioned for the Raspberry Pi you need these binary blobs to start it up. If you're installing Go crazy on a PC, you don't need those and then the boot partition is your EFI ESP, right? Go crazy also supports the older MBR based boot protocol instead of only UFI. Then after the boot partition, you have not just one root partition but you have two root partitions. These are called root A and root B and they're using an AB update scheme so that typically the first time you write it, so in the demo that we just did, we have written boot and root A and then with the GOG update command that we used, we actually wrote the new system to root B so that if anything goes wrong in the update, we would still have a bootable system and then in the reboot it switches over to start from root B and then the next update goes to root A and vice versa, right? So it's a classical AB update scheme and the actual contents of the root file system are read-only compressed SquashFS images. So the idea here is that most of your appliance typically is stateless but obviously you can only get so far with a stateless system and then typically you will want to store some configs, some data, some what have you. For that, you will have the fourth partition for your own usage. This is called a permanent partition and you can have an X4 file system there or really any file system that the Linux kernel supports out of the box. So you could also use ButterFS if you wanted to but you cannot for example use CFS because it's not part of the Linux kernel. It's a separate module that we don't support yet. So the fourth partition you can do with it whatever you want, you can also say, oh, I don't want a file system on there at all. I just want to directly write to the block device if you have strange custom use cases. That's entirely up to you. Go crazy, only manages boot and root. All right, so what happens with that SD card once you plug it in? What happens at runtime? Well, first of all, obviously the Raspberry Pi bootloader starts up and then the Linux kernel starts up and then once control is handed over to us from the Linux kernel, the go crazy init system starts. So the init system is the core component of the system that brings up all of the other processes. So this means it supervises the installed programs. So for example, the WebStat program that we added in the demo, it will be supervised by the go crazy init system, which means that whenever it exits because it crashed or something, it will just be restarted, right? It's a very simple supervision mechanism. The init system is very integrated in go crazy, meaning it's also the component that displays the web interface that we just saw. So when you click on stop or restart in the web interface, you're directly telling the init system what to do. You can also configure it to send the logs of all of the programs centrally to a syslog server. Now, obviously this depends heavily on your setup, whether that makes sense. If you only have one Raspberry Pi, don't bother with remote syslogging. But as soon as you have like five or 10 or the numbers start to rise, it makes sense to like centrally aggregate and filter. The init system is also what provides the network update functionality. So the gock update command, it will send over the new stuff to that system, which will then write it and reboot into it. Aside from the init system, the only components that we really, really need are a DHCP server and an NTP client. So the DHCP server is for IP address configuration so that the device can actually participate in your home network or in the internet. And the NTP client is required because the Raspberry Pi does not actually have a real-time clock that would survive across the power loss. So whenever you start up a Raspberry Pi, you will need to set its clock from scratch, which is why those are like the three components that you really need in the user land on the Raspberry Pi. Then there's one other component which also gets added by default, which is gonna be very handy if you develop anything on Go Crazy and that's a component that we call break glass. It's used for interactive debugging and it is an SSH server. It's Go Crazy's SSH server. It's called break glass because you sort of break the glass of the model. The model is you have a stateless appliance, you have only Go code, there's no C code running on there. With break glass, you're starting up an SSH server that's still written in Go. But then it starts up a busy box, statically compiled environment, which is an embedded Linux tool set for those who haven't used it, which is written in C. And the whole idea of break glass is that you bring your own software. So this is like, I'm gonna say remote code execution as a service, but for a good purpose, right? So you could, for example, if you wanna debug, I don't know, what your CIS calls your program is doing, you could use S-Trace. If you wanna debug what's happening on the network, you would copy over TCP dump. And that way, even though the system is very small and self-contained by default, you can still add all of these useful tools that you might need during development. All right, so now that we established what happens on disk and when you started up, where can you actually plug in that disk, right? Which hardware is supported? The first device that was supported by Go Crazy is the Raspberry Pi 3. You might wonder why exactly version 3? Why didn't we start sooner or later? And the Pi 3 was the first ARM64 Pi that was supported by the upstream Linux kernel. So in case you're not familiar, upstream Linux is developed for all of the devices, whereas the Raspberry Pi Foundation maintains their own fork of the Linux kernel that is specifically targeting only the Raspberry Pi. For the longest time it was necessary that if you wanted to use the Pi, you would need to boot the kernel provided by the Raspberry Pi Foundation. You couldn't just use any arbitrary Linux kernel. The downside of that is that whenever there's security issues in the Linux kernel, you need to wait until the fixes have propagated to the Raspberry Pi kernel foundations, the Raspberry Pi Foundation's kernel fork. And that was a delay that we didn't want. So we thought it would be nicer if all of the components could be updated very quickly, including the Linux kernel. So we were so happy to see that with the Pi 3, we first had a CPU that was supported by upstream Linux. Then later on came the Pi 4 and also the Pi 0 2W, and all of these are supported and recommended for usage with Go Crazy. You can also install the system on standard PCs. And as long as you have an AMD64 CPU, this can be anything, right? It can be a very small embedded PC, like the PC Engines APU was sort of my go-to example that I would recommend. Unfortunately, PC Engines is no longer extending the series of APU processors, so they're like selling all the stock they have and then it's game over. So if you want the PC Engines APU, now's your chance to buy one. But it could also be something more abstract, like a virtual machine. So you can totally run it in like your Proxmox cluster that you have on your Home Lab or something like that. And as I mentioned already, Go Crazy supports both the more modern UEFI-based boot but also the older MBR boot. So whatever embedded PC that you have, as long as it's enough PC-like, you will be able to use Go Crazy. Now, this is the officially supported stuff. And what that means is that whenever there's a new Linux kernel version, it will be automatically built, imported, and then tested on a range of continuous integration devices that I have at my home. However, there are also other alternatives, community-supported alternatives, where the community has stepped up to provide the Raspberry Pi Foundation's Linux kernel fork without as much testing. But if you, for example, have an older Pi, like the original one or a Pi 2 or a Pi 0W, which are not supported by the official upstream Linux kernel, you could still use Go Crazy on these as long as you're okay with using the community-supported kernel. And then lastly, there's also alternatives to the Raspberry Pi, such as the Odroid family of devices, which are also supported, also community-supported. So I can only do so much, but other people are interested in running the system on the other boards, so they're providing the support for that. All right, so typically, you don't only wanna boot an operating system, you actually wanna use some hardware with it. The most common request for hardware that we got was certainly using Wi-Fi. And on the Pi, the situation is that Wi-Fi is the same chip as the Bluetooth chip. It's like an integrated thing. So as soon as we add a Wi-Fi support, we can also use Bluetooth. Now, let me qualify this a little bit. We've have had Wi-Fi support since the very beginning, just not encrypted Wi-Fi, and people really wanna use encrypted Wi-Fi typically. And that was pretty hard because we thought you would need to use a WPA supplicant or something to do the Wi-Fi cryptography, and turns out there's actually a cool feature, which is called Handshake Offloading, where the Linux kernel tells the firmware, in this case, like the Cypress or Broadcom firmware of the Raspberry Pi, hey, why don't you do the crypto? And then you have a very nice API with which you can just connect to a Wi-Fi network. So since March 2022, you can use encrypted Wi-Fi on all of the supported Go Crazy Raspberry Pi models. The configuration is very easy. You just give it like this JSON file for a configuration. And then I mentioned you can also use Bluetooth, but I do wanna qualify that because what you can use is the Bluetooth Low Energy part. So this is called BLE, and devices that will be popular with it are any sort of IoT sensors, right? Like a temperature sensor, humidity sensor, that sort of stuff. They use BLE, but what you cannot use is like a full Bluetooth stack. So don't expect to like connect your wireless keyboard or your headphones or stuff like that, because for those you need like a lot more software components which are just not available in Go because nobody has bothered to write a Bluetooth stack from scratch in Go. So those are sort of the limits, but encrypted Wi-Fi works, and that's very exciting. So then I mentioned we have automated Linux kernel updates, and I wanna talk a little about that. The goal here is to make new Linux releases available as soon as possible. So whenever there's a new Linux minor or major release, usually they will also address some things security critical, not always, but usually. So we wanna pull in these fixes ASAP, right? Because don't run unpatch stuff in your home network. And the idea here was to make it really as easy as possible. So the way we designed the automation is to actually mirror what a human would do if the human were to import the new version manually. So we have a daily cron job that is running on GitHub Actions. It looks for whether there's new releases on kernel.org. And if so, it submits a GitHub pull request which just changes the URL from which the kernel is supposed to be built. Then that pull request triggers a GitHub action, CI run. The automation builds the new Linux kernel, which by the way is built reproducibly, so it really doesn't matter if I'm building it on GitHub CI or I'm building it on my local machine or any of you is building it on their laptop, it will be bit for bit identical. Then the newly built kernel will be added to the existing PR, which will then be force pushed. That triggers another run of the whole GitHub Actions CI automation, which then goes on and deploys the newly built kernel on these sacrificial Raspberry Pis that are running in my living room. So for those, they're just there to support the CI infrastructure. It doesn't really matter if I break them. So it's better that I break them where it doesn't matter rather than you break them where you're gonna be sad afterwards, right? So if those devices boot and pass the test suite successfully and they're all hooked up to like a serial port on the coordinator, then the PR automatically gets merged. And at that point, you as a user and pull in the new kernel versions, right? And the beautiful thing about this is that if any of these steps fails, it's very obvious where it failed and how you continue, right? Because the automation really does exactly the same steps that I would be doing if I were to manually update the kernel. Now you might be wondering how well this works and the track record for this automation is that we have most Linux kernel versions available in less than 24 hours. And mostly this is just constrained on the GitHub Actions cron job running just once per day.  But at some point, you also need to stop. If there are issues in this automation, it typically means that there was a crazy specific kernel patch that fails to apply to a new version. And in that case, you would need to change the patch, rebase the patch or drop the patch or whatever is the appropriate action. Over the last couple of years, we've tried to minimize the patches that we need and it's only gonna say less than five or so at this point. So, you know, this typically works really well. All right, so now that you have a rough overview of, you know, how it works, how it feels where to run it and what hardware is supported, let's talk about a couple of appliances. Like what do people actually do with this system? One appliance I wanna highlight first is called Conserve. Many of you might have like a home lab at home or are responsible for a small rack of computers at work. So, you might know that the serial console is very handy as soon as you have an issue with your server where after you rebooted it, it doesn't come back online on the internet and you can no longer log in using SSH. So, that's why many people still have a serial console as sort of an out of band access mechanism. Now, either you can buy these very expensive serial console appliances or you just build something from scratch. If we're only talking about one, two or a small handful of servers, right? And this is what Conserve is. It's an appliance that uses a Raspberry Pi. You plug in via USB, a USB to serial adapter. You connect the serial end to your server and then you can actually SSH into Conserve and directly get onto the serial port, right? So, it's not like you would need to log in and then start mini-com, figure out which device, et cetera, right? No, it's much more convenient and integrated and configurable. What's notable about this particular appliance is that Matt Leher, who wrote it and runs it for his personal home lab, started the project on stream together with me. So, if you really wanna look at, you're starting from zero, how do you build such an appliance? You can just watch the recording of that stream and you'll be walked through it. All right, so not every one of you might have a serial port on a server or something, but I bet almost every one of you will get physical paper mail in your inbox, usually more than once a week. So, maybe you might wanna do something about it in terms of organizing it. Like, I bet not every one of you is diligently scanning and organizing every mail they get. Maybe you don't need all the mail, but typically you would want to subset. So, what I built at some point is an appliance called Scan to Drive, where the idea is that whenever I get a letter in the mail, I just put it into the scanner that you can see on the picture here and then I push the button and it automatically will be scanned and uploaded into Google Drive, where the cool thing about that is that for the PDFs that you upload into Google Drive, you have full text search automatically using OCR that you don't have to implement yourself. So, from my perspective, I'm just putting it into my scanner. The Scan to Drive appliance converts the full colored scan into a black and white PDF, compresses it so that it fits into the size limit for OCR on Google Drive, where you can't have more than a couple of megabytes on your PDF, I think, if I remember correctly. And then you can use full text search over whatever many years you have stored in there. So, if I needed to pull up an urgent, I don't know, insurance invoice from three years ago while I'm here for the conference, I could totally do it, right? So, this appliance supports the Fujitsu Scan Snap, which is the very model that you see on the picture. I actually implemented a driver for it in Go for this particular model, but it's a couple of years old at this point, so many of you will not have this exact model, I think you can't even buy it anymore. The good news is that any AirScan compatible device is also supported. And what does that mean? Well, AirScan is the standard that Apple introduced and uses for its scanner slash printer support on the iPhone. So, if you go into, I don't know, media market and you look at which boxes say compatible with the iPhone, you just buy any of those and it will work with Scan2Drive, which is very cool. If you don't want to use Scan2Drive, you can also use just the regular AirScan library or tool, both of which are on my GitHub. Under the covers is really just HTTP and JPEGs, very nice small little protocol. So, if you have ignored printing and scanning for many years because you just couldn't, then have a look at it again, it's much better now. You don't need all of the crazy software. All right, this is Scan2Drive. Let's ramp it up one level in complexity. This appliance here is called a router seven. You can see the original version on the top right. This is a PC Engines APU that I mentioned before already. Like many of you, I was using OpenWRT for my home router for a number of years. I was using it on a Turris Omnia, which is a nice open hardware router from the Czech internet registry. This was nice because it's also an auto updating device, but at some point, the auto update feature really annoyed me because in May of 2018, there was an update in the DHCP v6 client which broke my IPv6 connectivity. Now you might be wondering, well, okay, why didn't you just fix the bug? And I tried, I reported it to the author and the author said, well, this is sort of a weird edge case. The standard doesn't really cover it. No, I will not fix it. Talk to your ISP. So I talked to my ISP and the ISP was, well, yes. I mean, the DHCP platform is old, we're migrating to a newer platform. We cannot help you find weird bugs in the current platform. Just wait until the new platform is rolled out. So I can totally understand both the author and the ISP saying, no, we can't fix it. Talk to the other one, but it didn't leave me in like a good position, right? So I figured, well, I could either turn off the auto updates which defeats the purpose for which I bought the device, or I could pin the DHCP v6 client indefinitely which defeats the purpose of having auto updates and security updates and stuff. So I decided it would be fun at this point to just build my own router. So I did. So router seven is entirely written in go. It contains a DHCP client server, DNS forwarder, all of the different components that you need in order to run your small home network. And because I implemented the entire stack myself, that also means that I could focus on a couple of goals that I really wanted to try. So for example, router seven is built to maximize connectivity. What does that mean? Typically, if you have a DHCP client and it cannot reach its server anymore, the standard says the DHCP client should expire the lease, not use it anymore, give it back, right? But if you're talking about DHCP between your router and your ISPs access platform, these are like in VLANs anyway, there's no other clients to take care of. So it's actually better to ignore the standard and say, well, I can't really reach the server, but my lease is gonna be fine. It'll still keep working. And that actually helps me get over brief connectivity issues like that. So the DHCP servers of my ISP are much better now, luckily, but whenever they had small blips and issues, I would just stay online with this approach. The router seven is also very reliable as its goal and also very debuggable. Meaning that for example, if you notice that there's an issue in your local network, so for example, when my girlfriend got a new Mac, it couldn't really connect. And it turns out I had a little logic bug in the DHCP stuff. But in order to debug this, it was very nice because I could just say, okay, I noticed now that your computer can't come online. I can just connect using Wireshark to a ring buffer that router seven maintains, which will give me the last couple of hours of network configuration traffic retroactively, right? So it's kind of like time travel debugging a little bit where you don't need to know that you're gonna debug something in two hours. You can at the point when you encountered a problem, just jump back in time and get all of the stuff. And this is very simple to do whenever you're controlling the entire platform because on the PC Engines APU, I have a couple of gigabytes of memory. So I don't really care to set aside a 50 megabyte ring buffer to just proactively capture all the network traffic. However, for OpenWRT for example, they run on much, much smaller routers where you don't have that RAM so you can't easily do that, right? But I found it very instructive and enlightening to see that approach work as very nice for development. So you might be wondering why the two pictures? Well, the picture at the top is the PC Engines APU, that's sort of the small version with which I started out. As I mentioned, it's a very nice device. You can see it has like the serial console at the top. It has a couple of ethernet ports.  But at a later point, I needed to upgrade because my ISP started offering 25 gigabit connections so I built my own PC and that's also the reason why Go Crazy now supports both MBR for the PC Engines and UEFI for the regular PC. But both are well-tested and both work. All right, so you don't need to go as far. You don't need to like build your own router from scratch. Nobody's asking that. In fact, you can go the other way. You can say, oh, I don't really have a use case to build my own appliance but there are a couple of cool Go tools that I would like to run and maybe that's also something for which Go Crazy is interesting to you. So I'm gonna list a couple of examples and the one that I wanna start with is called Prometheus. Prometheus is a monitoring system and time series database. For those of you who haven't seen it, I would encourage you to check it out. It's pretty cool. It comes with a couple of different components. For example, the Prometheus node exporter is the software that you would run on every one of your nodes in your network. So this allows to export metrics which in turn allows to monitor the Raspberry Pi's CPU usage, RAM usage, whether your disk runs full in a couple of days, that sort of stuff, right? You can also go one step further and you can run what's called the Prometheus black box exporter which you can somewhat compare to like traditional monitoring like Nagios or other stuff where you probe targets to ensure they're still good. For example, you could probe your company's website to ensure the SSL certificate is still valid for a couple of days and you would get an alert if that's not the case anymore. You can run that on Go Crazy as well and you can even go one step further and you can run the entirety of Prometheus yourself on the Raspberry Pi. So including the time series database storage and all of that stuff, but I would encourage you to use good durable storage, not like a cheap SD card that you found in your drawer, right? Because if you write that many time series database updates, you will want to have something that survives for more than a couple of months.  All right, let's switch to the next example. This would be Tailscale, which markets itself as a secure network that just works. To put it into more technical terms, Tailscale is a zero-config VPN which uses WireGuard in a mesh network configuration. So that means if I'm here at the conference and I have my laptop and I connect it into Tailscale, I can reach the computers in my home network without having to deal with port-for-wordings, natural-versal and all that sort of stuff. And the traffic doesn't go through a central server, it goes directly between my machines. So that's very appealing and you can also just install Tailscale on Go Crazy. And then as soon as you have installed it, that means the services that you're running on your Raspberry Pi, be it your backup console, your media center, what have you, will be available over Tailscale. You can go one step further and you can run it in what's called a subnet router configuration where Tailscale will not only make all of the services on your local Pi available, but all of the services in your entire local area network. So this is very appealing because it means that, for example, for my parent's place, I could just prepare a Raspberry Pi, install Go Crazy, install Tailscale as a subnet router, place it at my parent's place when I'm next visiting and then just leave it there, update it remotely and have access to it and all of the other devices, PCs, laptops, printers, tablets, what have you via the Tailscale subnet router. So very appealing use case, I think. Okay, and then most notably, we can also run Docker containers on Go Crazy and this really is a great feature that we added in April, 2022. The reason behind it is that Potman is written in Go, well Docker mostly is as well, but Potman has the additional advantage that's available as a standalone static build which lends itself well to be packaged for Go Crazy where you don't have most of the traditional components of a Linux system. So we did that in April, 2022 and this means that we now have an escape hatch for running non-Go programs on Go Crazy. You might sometimes need or want to do that. One of the examples I've put here on the slide is the Ubiquiti Unify Controller which you need to configure the Ubiquiti Wi-Fi access points, that's like a pretty complicated thing that uses, I believe, Java, Redis and a couple of other components. So you can just run that as a Docker container and then you don't need to install it on any of your systems, you can just run it, right? Similarly, maybe it's not so much about third-party software that just isn't written in Go but maybe it's about older setups, right? And one example I have here is that for chatting on IRC I still use IRC which is very old software at this point but there are a couple of Perl scripts for it that are not in the mood to rewrite in anything else. So I've just put that into a Docker container and I'm running that myself, right? So this is very interesting because it means that you can really start using Go crazy with nothing else but Docker containers and then sort of move all your stuff in there and then gradually replacing Go what you want to replace in Go, if at all, right? All right, so now you have an impression of what appliances can you build? What software could you just install to build an appliance without really building much? But let's talk about what if you wanted to build something from scratch, right? How does that work? Well, the only thing that you need to supply is a runnable Go program, right? The command line flags with which it will be started you can customize entirely, you can customize the environment variables. These would be typically the stuff that you would expect to be customizable in traditionally a system disservice unit but in the case of Go crazy just in your Go crazy instance configuration. There's a couple of more details in terms of what environment variables does Go crazy additionally provide to tell the program where it's running, what's going on? Like is this the first attempt at starting it or are we talking about a subsequent attempt after the supervision figured out, hey, you crashed? Or are we talking about a program which shouldn't be supervised at all? All of these sort of nuances you can look at at our documentation where the process interface between Go crazy and you as the software author is clearly defined. Then aside from the runnable Go program that you provide, the stuff that works out of the box is essentially network because we ship the DHCP clients, the NTP client. So from your perspective, the program that you write you can totally use the network. You can assume the network is there and working. So one example here that I found very instructive is that at some point I needed an MQTT server for my home automation and I didn't have one. So I looked at which MQTT server is written in Go. I found this one here at github.com slash FHMQ and I just installed it on Go crazy and I didn't have to change anything. I didn't have to set a flag, I didn't have to set environment variables, nothing. I could just run it and would just work by default because the network is there, you connect your devices to the MQTT server and everything just works. So if you're mostly interested in home automation or in network services and that sort of stuff that's very convenient in Go crazy. Then in terms of hardware support, what hardware can you actually use? Well, whatever is included in the upstream Linux kernel. Sometimes there might be a little surprising differences between the Raspberry Pi foundations Linux kernel and what is upstream, right? But mostly most of the features that you would expect to work also do work. Just last week for example, I merged a pull request from someone who enabled USB webcam support for Go crazy. So that just works. And another thing that you might wanna do is connect GPIO pins on your Raspberry Pi to I don't know, LEDs, speakers, all that sort of stuff. You could use Perif IO for that, which is like a little Go library for GPIO access. And there's a documentation page about how you would do it, how you would verify that it works using a multimeter, that sort of stuff. All right, let's do a little live demo about how to add a new Go package from scratch, right? So let's see, wanna have a terminal here.  What I wanna do is I wanna create a new instance called gpn. So I use goc-igpn new. You've already seen this in the demo, right? So now we can go into the gpn directory. We see there is a config.json, which we have just now created. And let me switch back to the slides. So this is the first command, create the instance. The second command is, okay, now we need the actual program.  I'm gonna say, go modern it, hey, as you would do typically in Go development. And then we need sort of a minimal program that just does nothing. So this is what it would look like package main, func main, it's empty. Let's say, hey.go. And now I say, okay, use the goc CLI, goc-igpn and add the current directory to it, right? So now I can use goc override dash dash root temp root dot squash fs and just write it out, right? So this is what it would look like. And within a couple of seconds, you have the SD card image prepared. We can see in the resulting file system, there's a lot of system stuff, but then there's also the program that we just added, which it would be empty. So it doesn't really print anything, it doesn't really do anything, but this is what you need to start with an empty directory and you can just write your go code in there and then it runs on the device, right? All right, so once you have that, the interactive development, you could use goc update for it, but if you run it in real time, the demo was sort of a little bit accelerated, but if you run it in real time, it takes about 45 seconds on the pi four. This, as you just saw, like the go crazy part of it is very quick. It generates the entire root file system within a couple of seconds, or even less that if you're only changing a little bit of it, but then you need to also transfer that over the network, write it to an SD card, have the pi reboot, and that all takes a little bit of time. So what I prefer to do instead of using goc update, which updates the entire thing, is I just use goc run, which is a command which just cross-compiles what I'm working on, uploads the resulting program into the RAM of your running go crazy instance and then just restarts that service. So just like a very quick edit run loop where if I interactively develop for the system, that's how I do it. Okay, so now I've already told you that there's like no C on go crazy, right? But that's not actually strictly fully 100% true. There's a couple of escape hatches that you can do if you really want to. If you want to stay go only, that's fine. You can do it, that's the default, but sometimes you might need like the one C library or the one routine that you have. And one example is for the scan to drive appliance that I showed you, at some point I noticed that the image slash JPEG package that comes with go is a very nice and readable JPEG implementation. So I really recommend you read the source if you're curious about how JPEG works, but it's not as optimized as the other implementations that we have for JPEG. So the standard implementation is lip JPEG turbo at this point, and it contains an optimized version that uses the ARM neon instruction set that really accelerates stuff on the pie. And this really makes a difference. So this changes the scan time of a single page from, I don't know, 30 seconds to 10 seconds or something on that order of magnitude. So it's really worth it to use the optimized version of this library. But then I told you, okay, go crazy, it's 100%. Go, no C user, no G-Lip C, so how does it work? How can we still use that library? And the answer is that some of the C libraries can be used if they're statically linked, if they have no dependencies on G-Lip C or anything else, right? And lip JPEG is a perfect example here because really it's like you give it bytes, it does an encoding, it gives you back the bytes, right? It's what we call a compute kernel in the sense that it just does compute, it doesn't have any dependencies on the outside world. So these sorts of libraries you can totally use, you would just build them with like a typical cross compiler GCC in your environment and then you specify a link mode external and you give the external linker the static flag and you see it's a bit involved but you can do it if you want to, it's supposed to be the takeaway of this slide. There's one more escape hatch that I wanna show you which is that if you wanna run C software maybe temporarily on the go crazy device, you can just put it into a Docker container but sometimes you don't want that. Sometimes you don't want it containerized in the way from the host. Sometimes it really needs to run on your host. What can you do in those situations? The one example that I have here is TC, the traffic shaping and control program on Linux where if you just run TC, it doesn't support anything.  So when I used router seven to run an event network much like this conference network here, I wanted to cap the traffic limits to not overwhelm our uplink and I used TC for this, right? So how did I get TC to run when it obviously needs to see user land to the extent that you can dynamically load code at runtime? Well, there is a little helper tool which we call freeze, which looks at the binary that you wanna freeze and it looks at all of these shared library dependencies and it freezes those as well. And then it generates like a little tar ball that contains your program and all of its library dependencies and you can just copy that over to your go crazy device and run it there. And obviously that's not great because the software is frozen. So that means it doesn't get automated updates, right? It's like entirely outside of the system that go crazy manages. But this is very helpful for stuff that isn't security critical and doesn't really change. So one example is that we use this approach in order to ship the MKFS program so that we can create the permanent data partition using the X4 file system. We didn't implement like, creating an X4 file system from scratch in go. We just use MKFS and it's easy enough to just freeze it because it doesn't really change. All right, so with that all out of the way, let's talk a little bit about future developments and then I wanna open it up for questions and answers. So one project that one of our contributors and I am working on is called GUS, the go crazy update service. And the update flow that you have already seen is sort of the default update flow, the synchronous update flow where you run GOK update and it does the update right then and there. But sometimes that might not fit the model of devices that you have deployed. So if you have a number of devices that are only sometimes online, you don't necessarily wanna coordinate between where you create the update and run it and where you wanna deploy the update, right? So what GUS allows you to do is one to many updates. So you build the image once, you upload it to GUS and then all of the different Raspberry Pi's that are running the same image can come to GUS at their convenience and pull down the update, right? So it just uncouples these two steps. All right, so with that, I'm at the end of my slides. There's more information at gocrazy.org. I've also laid a couple of go crazy stickers on the table here up front. So if after the presentation you think that's cool, you wanna grab a sticker, please feel free. Also, if you wanna talk about any aspect of go crazy, feel free to talk to me while I'm still here at the conference. I love talking about the stuff and I will also give you a sticker as a gift if you want. Please also scan the QR code and give me feedback on this presentation either now or later. And with that, I'm gonna open it up for Q&A and say thank you for your attention. Thank you very much for the great presentation. Yeah, the question in the front row, please. Thank you. Right, I'm gonna repeat the question for the recording and stream. The question was, do I develop this on the side or is this like a commercially available thing? This is entirely a hobby of mine, which I'm using for all of my Raspberry Pi's and more and more computers at this point, but there's no business behind it. In fact, we've gotten the question at some point where somebody was like, well, I wanna build like my IoT system on this, on go crazy, like are you okay with it? And I'm like, look, it has crazy in the name. Do you really wanna base anything on this? But if it works for you, be my guest, right? Just no support other than what I can provide best effort. Thank you very much. Any other questions? Question in the front row. Right, yeah, I'm gonna repeat for the recording. When did I start with Go and what was my motivation for it? I actually looked at Go when it was first announced in 2009 and then I tried a couple of toy demos and I was like, oh yeah, this looks cool. And then I put it back until Go 1.0 was released in 2012 where they introduced what's called the stability guarantee. So all of the stuff that is in Go 1.0, it will not be broken until you know, go too hypothetically. But that was really like the status, like the signal for me to really use it, right? In earnest. So I started building services such as Debian Code Search, which I'm still running to this day. So if you're interested in searching through open source software using regular expression matching, you can go to codesearch.debian.net and grab over all of what's in Debian. That is a Go system that I built as my bachelor thesis back then. So I was like, oh yeah, this looks like a natural fit for network services and for that sort of stuff. And I'm enjoying it to this day. And I'm doing more and more in Go as you can see. Question in the second row. Oh, there's Mike, can you use the mic? Yeah. What is your experience regarding the performance of Go because you said you'll vote some drivers and so on, is it compatible with C or needs it some fixes or tweaks or something like that? Right, in general, my experience is that the performance of Go is always sufficient and in the very few rare cases where it isn't, such as in the hot loop that's encoding the JPEGs, right? That's the spots that you find using profiling. The profiling story is very good in Go, so it's very easy to figure out where are your bottlenecks in terms of CPU usage or memory allocation or that sort of stuff. And then you implement that more efficiently, right? Most often, I try to just optimize the Go code itself, but sometimes if that's really not sufficient, like for example, if you wanna use the ARM neon instruction set for which the Go compiler doesn't have any intrinsic support, you will need to use the escape hatch and say, okay, now I'm gonna use C or I don't know, maybe you could even use Rust or something else and just pull that into your Go program and keep the rest in Go. Yeah. There's a question at the very back, right? The question was, is there any separation between the processes and the rest of the system, like Seccom or Landlog or that sort of stuff? Yes, but you're responsible for it on your own. So for example, the NTP client that I mentioned and I believe also the DHCP client to a lesser extent, they use privilege separation, so they sandbox themselves essentially. So when you're running on Go crazy, you get full access like sort of as root, right? To like anything you wanna do with the Linux kernel. But if you think, oh, for this particular use case, Seccom would be great or Landlog would be great, then you can totally do that, but it's something that you need to do yourself. Yeah, there's currently no facilities to make that easier, but I'm also not opposed to adding something at some point if it turns out, oh, most people actually need to implement this and that, it would be nice to have a common helper for it. And another story where it's pretty similar is with the Docker containers that I mentioned, which can also sometimes function as a sort of boundary. For those, you can currently run Docker containers, but there's no sort of container manager or something, which you could configure to just tell it, look, just run this and that. So there's no Docker compose essentially, right? So for that, I'm also into space where it's like, for now, just do it yourself. So because you know what you need, you know it best, right? And if it turns out that there's a common need that we should address with like a central component or a repository or a library or something like that, I'm open to add that in the future or to merge that if someone wants to step up and contribute it. Another question in the very front, right? I was wondering since it was also attached to Google, did you ever meet up on the topic of how it involved this with Google nowadays? Because I saw that there's very little information in general from any of these points on the internet. Right, right. I'm going to try and summarize that for the recording.  So for other parts of the Go ecosystem, are there other replacements like that? For example, the Go module proxy, which is currently centrally run by Google. Fun fact, for this presentation, I actually did use a module proxy that was running on my laptop locally. So that is totally supported. I don't think for the context of Go Crazy that there's really any dependencies like that. You can sort of mix and match everything and anything. So if, for example, you say, well, you know, I don't trust the Google engineers. I want to use a different compiler. You can use GCC Go, right? Or you could use Go LLVM or whatever other implementation you have. The Go Crazy system itself is not really tied too much. It does use the Go tool though in order to build software, right? So, you know, just getting all of the modules, working with the modules, all of that, we don't implement that ourselves. We reuse what the Go tool already provides.  Right, that's written in Go and you could provide a replacement, you're right. Yes. Yeah, I think, you know, for, I think to summarize it and feel free to talk to me later outside, but to summarize it, I think for any level of paranoia and as you can get comfortable with the system as long as you're willing to exchange a couple of parts here and there. Yeah. Further questions? Still have a couple of minutes, so feel free. Okay, no one wants to ask, I have a question. Please, I'm interested in your build pipeline. You mentioned, yeah, Raspberry Pi's at home. Like, how do you ensure that if they break or like if a separate job comes along, how do you do the duplication of that? Right, so there's actually locking in the coordinator. So the CI software, like the CI pipeline that's running on GitHub Actions, it talks to my coordinator at home and it tells it, look, I want to deploy this new image. Can you do it? And then either it immediately gets the lock and it has access to all of the devices or it doesn't get it and then it just waits. And the reason why this needs the locking is that not only are these devices coordinated in terms of which software is running on them, but in fact, they're also turned off to save power. So the coordinator actually talks to a smart plug, which then turns on all of the devices, lets it boot, et cetera. And then the CI pipeline only uses the energy that it really has to use, right? But yeah, the jobs can't stomp over each other because there's locking involved here. Yep, another question in the front. Okay, the question was, which was the trickiest part I had to build and which one is the proudest part? I think for the proudest, I'm just gonna give you the origin story of how we built it, which was we were sitting around at a Chaos Communication Congress at some point and we were like, isn't it so super involved to have all of these distribution installations that you need to manage? So one example we had at the point was like, back in 2013 when I did my last big house move, I set up an installation on a Raspberry Pi and that just left it running for years and years without really updating it because it was so involved to update it and I didn't wanna risk anything. So that was sort of like, we were like, isn't there a better way? And then a friend of mine was like, well, what do you really need? And then I figured out, well, if we have the Linux kernel, could we just give it in it equals and then a go program that we statically build and cross compile? And to statically build it, you just need cgo enabled equals zero and to cross compile, you just need the go arch environment variable. So I just entered a simple command to my computer and out came a binary and we just booted that up on a Pi and it worked. And we were like, wow, that was way easier than we thought, right? So could we just build the rest of the system that we need? And then it sort of spiraled from there, right? And then the trickiest part that we had to build was probably when go did the transition from using go path to using go modules, because that really changed a couple of different things in particular, because we're sort of a downstream user that's built around the go tool, right? So you really need to be in tune with all of the nuances there. And that also changed a little bit how everything is structured on disk. So that was sort of a bit of a disruptive change from our perspective, but I feel like at this point we have mostly gotten everything, right? But we can talk about the details if anyone wants to. I think we might have time for one more question or so. So don't be shy. Yeah, yeah, about five minutes left. So there's definitely room for more questions. Okay. Okay, but if you can't think of anything, as I said, feel free to approach me outside any time, talk to me about stuff, or just send me an email, open a GitHub issue. Contributions are very welcome. So I'm gonna say thank you again and see you all soon. Thank you very much.


Got there, I have some trivia. You have some trivia in a box? Some Bollywood trivia. What? What? Ready? I'm ready. What actor plays the role of Dr. Cliff Patel in Silver Linings' playbook? I thought you said this was Bollywood. It is. Silver Linings' playbook is Hollywood. I'm guessing he was in it? You know who this is. I'm gonna guess it's Irfan. Anupamkar. Okay. Yeah. Actually. That's a poignant. Did you choose that specifically for this intro? I just saw his name.  I'm Anupamkar. It's actually Kair, I've been told. It's actually Kkar. But you know, that's Samantha. Yeah. Not to be confused with Samantha's. Samantha's? Yes, Samantha's and Samantha's. What happens is Samantha goes hiking and then she comes back covered in ticks and so it's Samantha's. It's me, Jessica.  You are my Sonia.  Classic. Today we're doing a movie review. And we're doing it over the 2009 film. What? Sid? Seriously? Do you know what time it is? It is. It's movie time. Movie time. Directed and I believe written by Ayan Mukherjee, right? Great last name. Who is also the director of Bra Mastra and Yaijuani, Yaijuani. I wonder if they were talking about Bra Mastra back then. 2009. Not yet, not yet. He says it was a 10 year process. I believe this was his director debut. So probably not yet. Maybe he had it in the back of his mind, but probably wasn't. I'd imagine planning the biggest, most expensive movie ever yet. No, but that's how far back they go. Yeah, yeah. He only likes to work with Ranbeer. He is exclusive Ranbeer. They have a contract together. They're the Scorsese and Leo of Bollywood. But it's obviously starring Ranbeer Kapoor, but Konkanasen's Sharma, and then Supriapathik and Anapom Kure. Or Kare, as my wife would say. Kare is that Bengali? Yeah. Kure. Anapom Kure. Because you're Bengali. No. You know she speaks Hindi. No, she doesn't. It's Ranbeer Kapoor. Yeah, Kapoor and Konkanasen Shomal. I don't know why I'm doing this. But she's Bengali. Yes, and that is the wonderful actress, also known as Shay Kapoor's stepmother. That's how she wants to be known. Maybe. She wants to be known that way. I don't know. Maybe she's very proud of being. Do you think? Konkanasen's Sharma? Yeah, I don't think so. No, she's not her stepmom. Yeah. Anyway, obviously this came out in 2009. Almost all of you have seen this movie. If you haven't seen it, wake up! Sid? If your name is Sid, let us know down in the comments below. We're off the rails already. I'm sorry, guys. But it'll be 100% for all of you. You've all seen this movie. So if you haven't watched it, Netflix? Netflix. It was on Netflix. At least here in the States. You can go check that bad boy out. Rick, your initial thoughts, please. This is my kind of movie. Hold on, wait.  No, I'm really not. I don't know why. Is this justice for a Ranbeer? You've hated every single Ranbeer movie. No, that is such a lie. All times! Lies. You've never lied. Did you think he's ugly? Lies. And you think he has a small wiener. I don't say it. I didn't make that up. That's all you. That is all you. It's so funny. Because I had the initial response that I did and then went back and revisited and realized I was wrong. Something you never do. I'm never wrong, Nick. Is to discover that, no. This kind of film is already gonna be, it already has a win for being like a rom-com because I love them so much. But it also could have gone wrong in a lot of ways. In fact, we were watching it and the primary thing for me is why it works is just because he and Kankana are just so good together and it's just a really feel-good, lots of fun. I enjoyed it. I believed them throughout and it's not one of those films that is intending at all to be like this award-winning cinema. It's not that kind of movie. Not trying to be in the slightest. It's the kind of movie that makes you feel good and I really liked it a lot. Right when I was done, I texted the team and I said just watch, wake up, sit and they said please tell me what you thought and I said, I think, I don't know because I have to, it's difficult. I said it could be my favorite rent beer movie. Oh! It could be but also, because obviously the other two in the running are obviously going to be Tamasha. It doesn't surpass Tamasha for me. No, Tamasha and Barfi. Both of whom, artistically, obviously better films. Yes. But though I- Agreed. I was like, I would never tell somebody 10 Things is one of Heath Ledger's best performances or best movies. But it's going to be one of my favorites regardless. I will watch 10 Things I Had About You which has the same kind of cliches, almost dated stuff that's going on but it's such a feel good movie. I will watch that over better performances and Broke Out Mountain or different performances like that of his that I really, really enjoy. It's different because those like Broke Out Mountain is a really dramatic film as opposed to Tamasha which is still a really fun film and a good film like that and Barfi as well. Great performances. But I agree with you. But this is one of the ways, it's an easy watch. Yes.  I would, I have to be in a particular mood to watch Tamasha and Barfi. This could come on at any time. Yeah, yeah. It almost gets that feeling of a Shahrukh Khan movie of I'm going to sit down and I'm going to enjoy myself not as almost whimsical I think as a Shahrukh Khan movie. Yeah. Those are their own beasts I feel like especially the 90s ones and all that kind of stuff. No, this is like Dildo Hock Nado. It's like that just everybody's really good and when it's over you feel good and you really like the message. You know what's going to happen in the entire time. Of course you do. Like immediately. When I said go run after the girl, he obliged and he ran after it. Right when he talked about the rain in the beginning of the monsoons. Me and my wife were like, that's how it's going to end. That's how it's going to happen. And it did. Nothing surprises you. You don't care. When I was talking to the group, they said yeah, it's not the best movie and it's a lot of cliches. I'm like, this film's supposed to have cliches. Exactly. Like you're not looking for like something to invent the wheel, the new wheel in these style films especially obviously it's 2009. So it's like what almost 15 years old even. And so you're just looking to like have a good time. You don't care that it's cliched. That's what these films are for. No, but the thing about it is that even though it is cliche and even though it's going to be predictable, the writing and the acting are done in such a way as is the directing and the score. I want to talk about the score that's done in such a way that everybody's committed to the believability. And so it works. They're not playing the cliches. They're not looking to do this in a way that's anything other than just grounded and as believable. This isn't Shakespeare. It's not supposed to be Shakespeare. This is supposed to be this kind of film. I believe these guys were these people as well as their friends. I believe the people in the workplace and you just wind up walking away thinking about this as a believable story. And we're Americans and we've related to this film on many levels but I'm sure as Indians, especially young Indians growing up. If you've obviously know somebody who has grown up very entitled or just had everything handed to them. Or also you're trying to find your own path in life that doesn't line up with everyone else's which is very relatable I think to anybody in the world. Which is also with Thiem and Tamasha. Yeah, it's yeah. And that's one of the things I really enjoyed because I was nervous because I was like, are they gonna make them try to like get like, this is not like I said, this is not like a new inventing of the wheel.  But sometimes I'm like, not everybody fits into that career mold. Don't put this guy who might not wanna be in a traditional office space. He wants to do something a little more creative. Let them do that. I'm glad that they went that avenue that allowed to show cause obviously in India, it's even more taboo than it is here. I mentioned that to Indrani which is something I knew going in. I just, she confirmed it was how many, how many, has she seen this? Oh yeah, that's what she wanted to watch it again. She loves it. So we watched it together and I said, that's really common, isn't it? And kind of rhetorical question where a son who would be born into a family and when he's through college, he still isn't really sure about the direction he wants to go. Then it's just a foregone. You are following in dad's footsteps. I mean, that is the expectation. And sometimes that's the expectation from much, much younger too. So one of my favorite things about the story that I thought was really great was adding the element that his, adding the element that his dad has had a love for photography too. I thought that added a really nice aspect in their relationship that would have made this film less lovely without it. That's one of my favorite things about the film. I really enjoyed Anupam Kerr. Oh yeah. Obviously, I think he's a fantastic actor. He's a very good in his comedic roles but then I also, I think he's an underrated dramatic actor because he brought it and normally we've seen him as the corky dad in a couple of things. He was a strict, straight forward dad but it also wasn't the stereotypical in my dad either.  No, he obviously had his principles and what he wanted and stuff like that but he also loved his son. Yep. The beginning, I was like, you're gonna give him a car for working 30 days?  Right, exactly. What is going on? There are rich people that do that. But then you saw his justification. He's like, I wanna work with my son. I also wanna help him get going in life and so this is how he thought he was helping and then you had that moment and you kind of understood the dad. Sure. When they had that big falling out, you're like, which is good writing. Yep. You wanna understand both sides even though oftentimes you're not supposed to understand Ranbir side because he's a spoiled brat. Yeah, he's very spoiled. Absolutely. And he doesn't really know what he's doing and it's kind of his whole story but I liked the arc and the way Anupamkara portrayed his, and obviously I don't wanna just call her shade stepmom. I wanna know about him. I'm sorry. Cause I love that actress. I've loved her ever since Ramleela. Yeah. Supriya Patha. As well, very different performance. A much more subtle. Oh yeah. Still, she brought that weight to it and she's one of the, I think, better underrated actresses in all of India. And again, great writing. The fact that when he goes to see her, he, all he does is, it was very obvious that the mangoes couldn't have been from anyone but his mom. And she thought she could maybe just, you know, don't tell them they're from me. And he's like, yeah, I know my mom. Yeah, and it was so interesting how they wrote these parents because obviously when she went, she wanted to see where he was living and she wanted to give her money to her cause she knew her son probably wasn't gonna have any. Wasn't gonna pay him. Of course. But then at the end when she was like, I don't want you to get the wrong idea. We're not, we're just friends. But she didn't actually say anything which I thought was way better than like, her actually like them coming to like this under, she just touched her face. She, you don't know if she believed her. You don't know if she didn't believe her. Exactly. She's like, I don't care you're taking care of my son and I wanna thank you. Yes. I like that representation of, even though they're parents and they want their son to grow up and obviously take responsibilities. Right. It's still their son. Exactly. And they care more about him than they do him in any certain avenue. And assuredly would not approve of him living with a girl. But that, I agree with you. Not scripting anything, just having her touch her face is way more powerful. It wouldn't surprise me if that wasn't written either. Yeah, absolutely. With could have just been in the morning. With an actress like her, it wouldn't surprise me at all. Rand Beer, I wanna talk about obviously, he's the lead and that's why we got a little off track here. His character was relatable in a lot of ways, but also obviously not relatable in certain ways. Obviously, because it's fun. Like when he was making the eggs, when I was dating my wife, I did not know how to make eggs. I'm not even kidding. I had to call, when I was, I lived in San Diego when I was probably 20, I wanna say, or maybe 19. But I had to, I wanted to make some scrambled eggs. Wow, forget, like over easy, you're talking, you didn't even know how to make a scrambled egg. I don't enjoy cooking. Like there's people that, like my mom apparently tried to teach us all, but apparently we showed no interest. And then all three of my brothers left. They all showed interest in learning how to cook and doing all three of my brothers.  It's maybe it's the ADD, ADHD, I just, I can't focus and even though there's recipes I can make, I still have to, I have to have the recipe in front of me and I have to know what's going on. I can understand that with you. And so it was very relatable. I had no idea how to make eggs. Were you as excited as he was when you finally figured it out? Oh, I was very proud. Oh yeah, but I've also burnt eggs. Apparently it's hard to mess up eggs.  Yeah, that's tough. It's real bad. It's real bad.  I'm much older than I was then. But it was very relatable at that point. Not the rich parents, never had. No, I can't relate to that at all. I was working for my age of 14, because my parents were like, if you want money, you're gonna have to work. Yeah. Can not relate to that. That was not relatable, but it was, I liked his trajectory arc. Ranbir's a fantastic actor. Of course. I mean, obviously this is not one, like the rock stars or the Tamashas that was really asking him to give a really Oscar-worldly performance, but he did a fantastic job in what he was supposed to do. No, he, you know what, he's very relatable. He doesn't remind me of the actor I'm gonna reference. But his capacity as an actor is comparable to this guy. Tom Hanks. That's an interesting comparison. Because Tom Hanks can do, Maybe young Tom Hanks. Captain Phillips and Castaway, but he can also do Sleepless in Seattle and You've Got Mail. And it's because he's himself in those moments and in those places. And he's a very lovable person, but he is an incredible actor, so he can do, or saving Private Ryan, he can do drama. And so it's the depth and the breadth of what he can do as an actor because I could tell he was taking this very seriously. He was putting in, I feel like he probably puts in no different kind of focus as an actor on a film like Wake Up Sid, as he would something where he had to be far more in character and like in Barfi or Rockstar. And it shows in his work and he has a quality about him that's perfect for this character because there's an innate likeability and a naivete he can convey that like, not as strong as like the one who can do it the best for me in all cinema. I can't imagine anybody playing naivete more aptly than Amir Khan. He plays characters who are naive and innocent because I think it's just part of who he naturally is. And that's one of my favorite attributes of his character in this is that I believed and I liked, like I felt bad for him when she said, and she was being honest when she said, I wanna tell you this, it's gonna hurt your feelings. No, no, no, tell me.  You're a boy and I need a man. And I was like, she's right, but all. Yeah, and once again, I wanna go back to the writing because obviously you knew what she was talking about in terms of he's not really responsible. He takes, and for a second, I was like, you're wanting him to like just like grow up really? You don't want any of that. And in the end, she was like, I'm a little childish too. I kind of like, I don't want somebody who likes, even though I love jazz, I don't know about that. Jazz is fantastic. I love jazz. But she realized in the end, she was like, I kind of like somebody who has a little child. And obviously he needed to grow up in certain areas. Yeah, so the difference between childish and childlike. And it was as he started to gain responsibility and respect her space more, he didn't lose his childlikeness. And that's why I think she was able to fall in love with him because she realized, you know what? How much of this is being just irresponsible childish versus a whimsical childlikeness that most men lose, that he has, which makes him so endearing. And I thought she did a, I 100% believed she was falling in love with him. Yeah. 100%. She's fantastic. Yes, she is. She also has such a beautiful, unique look to her. I like. Conkana. Conkana. Such a beautiful look to her. I agree. I'm also partial to five foot tall, dark skinned. Well, why don't you marry Conkana? God. Bengali girls. And obviously we've seen her before in a couple of things, but I've always thought she was extremely talented actress. She's very good. And it was cool seeing her in this kind of like. They're fun together. Cause I feel like she is, she likes more heavy duty roles oftentimes, based off some of, she was in Okara, she was in, she's the lipstick on my burqa, which you know, I have not seen that one yet. Let's talk about her deriding, directing Death and the Gunge. Death and the Gunge and the Tolvar. She's formidable. And then there's, there's a couple, oh, this one. Oh, she was in Seven Kun Moth? Yeah. I totally forgot that. And she was obviously in Ithidaian, which we saw last Halloween. I totally forgot she was in Seven Kun Moth. So I feel like she likes a little, even though she does commercial stuff, I feel like she likes a little more kind of intense. I do too. Roles, but you know, seeing her in this kind of will they, won't they kind of movie? Rom-Kami movie. But also I like the writing a lot because it set it up immediately.   I just want you to know that.  But then this movie kept going on. I'm like, there's not a lot of time left. No, and I love their friendship. But even though love their friendship. In the back of your mind, you always knew. It was like, you're gonna get together in the rain. I know you're gonna get together in the rain. I literally, what point was it? Oh, it was like very notebooky. Shh, he wasn't getting the fact that he was being a slob and disrespecting her new place that she designed. And I straight up, I wasn't being silly. I yelled at the TV in the moment and I said, come on, what are you doing, man? Wake up, Sid. We were dying. We were dying. Speaking of which, you don't want to talk about the soundtrack. I do. Wake up, Sid. Okay, so yeah, my wife was singing every song. She knew every song. Oh, she knew every song. I used to listen to it like while, you know, my friends drove and like long drives. It used to be like on the- A soundtrack? Yeah. Yeah. So here's what I want to know. If you can check, I didn't check. There was a point in the scoring that was so beautiful. It was toward the latter part when he's about to move away.  Let's see this, come on. And I thought, this, I bet money that the composer or one of the composers did Laos sing Chata. But I haven't checked yet. Shankar Mavanan was always a singer. Music. Kaho Nahao, my name is Khan. My name is Khan. Music. Amit Trivedi, my name is Khan. Man. Amit. Amit Trivedi was the composer of this movie. I don't know who wrote some of the, but some of the moments in it. Hard non-bar. Were as pretty as the score in Laos sing Chata. And I also loved moments where the score is playing and then somebody says something and it stops and there's silence for a minute and then it picks back up when the mood re-established. I thought that was a very nice use of scoring. Where was Kalki? Huh? She credited here. She's Aditi? Did you see Kalki in this? No. Sorry, I'm a little distracted here. I was just looking at the cast of the film and it says Kalki, was she like, was she cut? Is she in a party scene? Sid, maybe it's, no, that's a song. Yeah, I don't, that's weird. Yeah, it's, maybe it's like the Alia thing when in ugly Alia. She's in like a clip. That's strange, but if you know the answer to that. Yeah, why is Kalki? It's Kalki Keklin on there. Credited on INDV. Credited actress in the film, I don't know. But yeah, I thought the score was nice. It was fun to see all those lovely songs and sometimes it's in certain films and I don't know why this is a run time too. Sometimes I don't know why some of the English dialogue seems cringy sometimes. I know. Do you know why? I do, why? And it always does. And I think it's just because it's so, there's an affectation in the way most Indians when they speak English in a rom-com and when they're saying things in like a party way, it's just like, I think this is the best way to describe it. So Indians will regularly say isn't it at the end of a sentence but it grammatically doesn't work the way we use isn't it. Right? I think that's the same thing. Especially if they're like college-aged party-goers, there's just something about the way they say it and the way they phrase the words that is so different than the way we say them that you question the actor's authenticity in speaking the lines. Well, obviously it's not an issue because it's in every single thing. But it is a little cringy. Yeah, and I wanted to figure out why exactly it is.  Yep. But it just went like when they're speaking English essentially sometimes and they say certain things and I'm like, oh, that was, it does. It sounds like they're trying to give an affectation to the phrasing, but that's just the way they speak. It's just strange. And it's because it's so different than the way we say those things I think it hits English speakers who that's native to them and the same way that if we tried to say stuff in Hindi it would always sound weird.  I'm sure he did. Because this had a lot of heart of a Kieran Johar film. I mean, he's the producer. But I feel like that's something he's really, even though obviously he can write stuff that's cringy or he can do stuff that's way over the top but he also always brings heart in his films. Oh yeah. Every single one, which is something we said about Kaohonaho that it lacked, I felt, the warm heart feel which apparently people said that he actually regretted not directing that film. He apparently handed it off to somebody and he later said he regretted not doing the film. Yeah, I think you'd have to find out as well because just because it says somebody's producing or exec producing doesn't mean they're necessarily fulfilling those roles the same way in every film. And some films, if they're producing, it's a far more collaborative thing for them and in other times, all they did was write a check. Yeah, and it's, so this, Ayan, the director. So we've seen what, three films of his? Yajwani, Hajiwani, this and Brahmastra. Brahmastra. He can't have done it a lot because he's probably been working on Brahmastra for ever, right? Yeah, wake up, say director of debut, I'm pretty sure. Oh, oh no, it's just a special thanks. I saw luck by chance and I was like, wait a minute, that couldn't have been him. He was an associate director on Swadesh. Hey, wait, wait, wait, writer, Swadesh? Associate, writer screenplay. Wow. Which is funny because I thought one of the weak things in Brahmastra was the screenplay in terms of the dialogue writing. And that was a very common, it was, I didn't feel that, but there were, you weren't the only one. It was a common critique of the film that the dialogue felt very forced, which is something that I feel like is not in this film at all. Definitely is not in Swadesh. Yeah, and I agree with you, it's not in this. I felt like everything was justified and was really well done. But clearly he also likes love stories. That's like, that's one of his driving forces, I feel like. And I think that's the heart in all of Karan Johar's things eventually, because I think he's a big believer in love stories. Why do you think he's, I don't think it's just gossip. I think he genuinely loves and gets so happy to see people get together, and that's why that's one of the topics on the couch, is who are you with and what's going on? Because I think he genuinely just gets thrilled to hear about love. Yeah, yeah. I think he did a fantastic job. Some of the critiques, like some of the supporting cast, I didn't love too much, didn't care too much though. And nobody was awful, thankfully. Yeah, nobody was awful, just some of the supporting ones weren't as strong as others. I agree. So that would be a critique that I can say.  Me too. When I go into these films, I just want to feel good. Yep, and if you did that, you did your job correctly and what can I critique? If that's what I got out of the film, right? This is exactly what you were expecting and I would watch it again in a heartbeat. Absolutely. So let us know what you think about this film, which will be our next Ranbir and Konkana. Yes. Films that we should watch and clean up after yourselves.


It is the 9th of December 1946 and 23 men are led into courtroom 600 at the Palace of Justice, Nuremberg, Germany. This marks the beginning of a fascinating yet chilling Nuremberg doctors' trial. Part one of the indictments laid at the defendant's feet was war crimes, performing medical experiments without the subject's consent on prisoners of war and civilians of occupied countries, in the course of which experiments, the defendants committed murders, brutalities, cruelties, tortures, atrocities and other inhuman acts. The trials aimed to prove and hold responsible those who conducted truly horrific experiments on prisoners at a number of concentration The evidence was plentiful and damning, ranging over several gruesome experiments on both live and dead subjects, including the study of malaria. A difficult case to rebut, but the defense counsel thought they had an ace up their sleeve. The US had been studying malaria as well, in a widely publicized experiment, and on the face of it similarly on imprisoned individuals, albeit the stateside experiments involve criminals and not victims of genocide. But the defense's attempt at what aboutism has meant that the prosecution now needs to show the difference, and this will bring in a questionable expert witness. Although USA versus Karl Brandt et al is a fascinating court case that would lead to the vital Nuremberg code, today we are focusing on the study that offered a glimpse of hope for the evil men on the stand in 1946 to 1947. My name is John and we're delving into the 1944 Stateville Penitentiary Malaria study. Welcome to the dark side of science. Our story starts with the USA's entry into World War II, with Japan launching multiple attacks on US and British holdings in Asia and the West Pacific. Pretty soon, the full power of the American military was brought to bear, and this resulted in thousands of soldiers being deployed to the Pacific and Africa. A long known hazard to soldiers working in tropical and sub-tropical zones has been that of malaria. This mosquito-borne infectious disease has throughout history stopped armies in their tracks. The diseases symptoms include fever, tiredness, vomiting and headaches. This is caused by mosquito bites transmitting single-celled microorganisms of the plasmodium group, the most deadly of which, plasmodium falchiparum, has a history closely linked with our own. When around 10,000 years ago, its population exploded, around the same time humans became an agrarian species. As populations grew in towns and cities, so did plasmodium falchiparum. The bite passes the parasites into the victim's body and into the bloodstream, from there they pass the liver and reproduce, with symptoms beginning to be shown between 10 and 14 days. For reference, in the year 2020, there were 241 million worldwide known cases, of which 627,000 were fatal. But even if you don't die from malaria, you won't have a great time if untreated. Relapses can happen for years after exposure. The best way to defeat malaria is to prevent infection in the first place. As such, mosquito nets are an essential part of life in parts of the world where malaria is common. The evolution of treatment followed many wars and colonial disputes in the first half of the 20th century. Pre-1920s, quinine was the main form of treatment. It is derived from the tree Chincona calicea. It was discovered by the indigenous peoples of Peru, who used it as a muscle relaxant. And with most great things from South America, it was taken and exploited by Europeans from the 16th century. It was later discovered by Jesuit, Agostino, Sal Eubrino and Apothecary by training, who lived in Lima, observed the quechua using the bark of the tree to treat shivering, a common symptom of malaria. Anyway, to cut a long story short, it was found to be an effective treatment for malaria. And throughout the 1800s, became a crucial component in the colonization of Africa by Europeans, which could be a dark side of science video in its own right. During the First World War, blockades of Germany resulted in restricted access to Chincona calicea, which by now was grown in South America and in Dutch plantations in Java and Sumatra. This led to the first successful synthesis by German scientist Peter Mullens, resulting in the drug Parmaquine or Plasmochin. The drug, although very effective at killing malaria, was also effective at killing people if treatment wasn't closely supervised. This was discovered when the drug was tested on United Fruit's West Indian African Descended Workforce. Ah, lovely, the story just keeps on getting darker and darker. Of course, these impoverished laborers were not given informed consent of the treatment with a new drug. As Japan swept through Asia in the late 1930s and early parts of the 1940s, the Allies were cut off from the vital Dutch plantations, which by the 1930s produced over 90% of the world's supply of quinine. With dwindling stocks of the drug, the age old foe of an army came back with a vengeance. And this was the issue the US Armed Forces were facing in the Second World War. In 1942, the US Army Medical Corps were estimating 251 cases per 100,000 troops of malaria. Needless to say, this can be a massive drain on military resources, and not surprisingly searches for more effective treatments were a top priority for militaries around the world. This is where the University of Chicago comes into our story. Starting in July 1941, the newly created US Committee of Medical Research sought out to find the most effective treatment of malaria sufferers. Their plan was twofold. Firstly, to attempt to source quinine from the South American market, which by now was pretty derelict after Dutch dominance in their plantations. And secondly, to develop new malaria drugs. The committee found human testing desirable, as it would speed up drug development and the study of the disease. Malaria would, however, not be the first for human experimentation, as the US military had several experiments running looking at many illnesses linked to maintaining an army, such as STDs. Although eager to use human subjects, a number of rules were set, including, when any risks are involved, volunteers only should be utilised as subjects. And these, only after the risks, have been fully explained, and consent forms would be required to be signed for later access. But who to use? The military was not eager to use their own soldiers, as well they were needed for soldiering. But there were two sections of society that in the eyes of military officials had a debt to pay. The first was conscientious objectors, and the second, prisoners. The latter would be preferable, as they offered unique opportunity in that their environment was completely controlled, thus offering the perfect human guinea pigs. A contract was drafted with the University of Chicago to launch an investigation into the disease. Just 30 miles south of the University of Chicago would be the setting for the study. State Phil Penitentiary was, at the time of the experiment's conception, just under 20 years old. It featured a roundhouse based on Jeremy Bentham's Panopticon concept, a layout that would help the experimenters more than they would know. You see, the Panopticon architectural design is a stroke of genius. Prisons have always had one big issue, and that is how to constantly surveil their prisoners. The Panopticon concept fixes this by placing every cell entrance on a walkway around the circumference of a roundhouse. A tower is built in the centre. This allows guards in the tower to be able to observe any prisoner at any time. But the prisoners view the tower is obscured, which means they can't see if the guards are watching them. This in theory results in the concept of constant supervision, as you can't tell whether or not you're being watched. As such, the prisoners self-regulate their behaviour and act as if they are always being watched. And this would help the experimenters throughout the study. The team was headed up by Alf Sven Alving, a neurologist at the University of Chicago. Much of the team were doctors just past residency doing their military service. Interestingly, the prisoners would also be employed not just as subjects, but as technicians. They would study two types of malaria, P vivax and P fauciparum. The former is mild, but causes repeat relapses of sickness, and the latter is short-lived but can be deadly. However, most tests were performed with vivax, as it was more common in Asia. Before the study began, mosquitoes were bred at the University of Chicago and infected with the chest and strain of vivax. The source of malaria came from a US soldier who had been infected in the Pacific campaign. During the setup of the experiment, the prison warden allocated the top floor of the prison hospital for the study. This was away from the roundhouse and thus gave a kind of relief to the prisoners of constant supervision. Initially, the participants were immune to any punishment, but quickly this was revoked when the request was put out for participation beginning in March over the prison radio over two times the required 200 men applied for the first round of experiments. The opportunity for earlier parole and a payment of between $25 and $100 also didn't hurt. Each infected mosquito was kept inside a circular plastic cage with a gall's bottom. This allowed the cage to be placed on the subject's skin and the mosquito to bite without escaping. The subjects selected were sentenced for longer than 18 months. This allowed an almost 100% follow-up rate and were all white men of a similar age and health. A medical history was taken and physical examinations were made on the candidates.  Deliberately infecting a person with malaria wasn't a new thing, however. It was actually a form of treatment of syphilis in the early 1900s, pre the discovery of penicillin.  The patient would then be treated later on for malaria with quinine. But I digress. The inmates were usually infected in groups of three, two of whom would be used to test out potential anti-malarial drugs with the final person acting as a control by not receiving anything. However, sometimes the infection group would be done in just pairs. As part of the infection process, one mosquito would be used to infect all three men, and each man would be bitten by 10 mosquitoes in total. After infection, the mosquitoes would be dissected, usually by a technician who was actually a prisoner. But infection is just the start of the experiment. Testing of potential drugs came next. For the prophylactic tests, the drugs were administered to the inmates the day before infection, the day of infection, and for six days after. They were encapsulated in gelatin, and were strictly controlled to ensure the right doses were issued. This test was to see if any drugs could prevent the onset of malaria, or at least reduce its symptoms. Blood tests were taken after the eighth day post infection. At the same time, temperature and pulse measurements were taken. Subjects were admitted to hospital when fever and symptoms became too much to handle in their cell. During any hospitalization period, rectal temperatures, pulses and respirations were monitored every four hours. If the fever ran above 103 degrees Fahrenheit, then this would be increased to every 30 minutes. Every day, blood pressure, urine samples, fluid intake, and special tests to establish drug toxicity were taken. The next set of tests were aimed at curative treatments, that is to stop sickness after the onset of disease. The curative tests could make use of previous participants, as long as the prophylaxis test had failed in the individual or they had been used as a control. They would be administered drugs once their temperature had exceeded 103 degrees Fahrenheit. This is where various different versions of synthetic quinine would be tested at four hour intervals. In a typical scenario, this could be for around 14 days. Throughout the study, over 30 new compounds were tested on the volunteers at Stateville, one of which was Primer Queen and the study marked the first case of human testing of the compound. They also tested analogues of Pammer Queen, an existing alternative to quinine but had high toxicity levels. These analogues were tested in the human subjects, even though their toxicity was not fully known, thus risking the health of the participants. Some doses were increased past what was known to be safe in order to observe the side effects of the substances. The testing of SN8233 resulted in one prisoner's death after it caused a heart attack. Many subjects were complained of heart conditions after the studies. The experiment allowed the military to find out how much of each analogue could be administered before adverse side effects would be felt. There is no doubt, however, that the experiment proved useful, especially in saving the lives of Allied soldiers. But we need to talk about the ethical considerations. Although the study for its time was considered an ethical milestone by getting written consent from its participants, the administrators still left a lot of the risks, shall we say, unexplained. The form only stated that the prisoner was agreeing to take part in investigations of the life cycle of the malarial parasite, and to accept all risks connected with the experiment. Pretty vague by modern standards. But the idea of informed consent is hard to define, especially when the participants are in an inherently coercive environment. The motives of the subjects can be pretty much split into three groups. Mercenary, for the money on offer. Corporal, in the sense that the punishing life of prison may be lessened. Or philosophical, in that participation offered a feeling of working towards a greater good for the war effort. An notorious participant was Nathan Leopold, who would later insist it was pure altruism that led him to volunteer. Leopold, along with an accomplice, were serving a life sentence for the kidnap and murder of Bobby Franks. He would be paroled in 1958, partially influenced by his time during the malaria study. Even amongst large swathes of the prison population, the war at the time was seen as a good thing. The pitching of good versus evil and all that. So really the question is how unethical were the actions of the research team. They most certainly made most of the use of the prisoners, in that I can't think of any other population that could be tested on with potentially lethal drugs and society not be too concerned. Even with volunteers from the civilian or military world, the testing of unknown drugs couldn't be done. As in the eyes of the public, a death from Eva would be considered a tragedy, something that is unlikely for a prisoner. It is really open for debate, the ethics of the study. Something that is still questioned today. And this brings us back to 1946 and 1947, during the Doctors Nuremberg trial. The malaria study at the time was widely publicised, leading to it being the most well known top secret project. It even featured in Life magazine. As such, and in preparing an argument for their clients in the USA versus Karl Brandt trial, the defence council sought to compare the German malaria study undertaken in concentration camps to the Statefield study. Even though the US study by today's standards stands on ethically shaky ground, the crimes committed in the name of science at Dachau are on another level. Talking specifically about the malaria study, the victims in the concentration camps were not given any form of information about what they would have to endure, let alone being asked about participation. It is estimated that around 1200 people were used to test malaria at Dachau. Approximately half would die. Any who received any debilitating side effects but survived were murdered. Needless to say, in the Doctors trial, this poor defence of what aboutism didn't work. And out of the 23 defendants, 7 would be acquitted, 7 be given the death penalty and the remaining serving various terms in prison. In the wake of the trial, a new code for human experimentation was set out, and that would take the name of the city, Nuremberg. Ironically, the experiments used to defend the perpetrators, if compared to the new code, would actually have failed to meet its standards. The malaria study would continue past the Second World War, and some really bizarre experiments would happen with prisoner-to-prisoner drug transfusions. But that would be a story maybe for another time. Now where would you rate this subject on my ethical scale? 1 being all okay, and 10 being pure evil. I'm going to say around a 3. This is a plain difficult production. All videos on the channel are creative commons attribution share alike licensed. Plain difficult videos are produced by me, John, in the currently sunny corner of south London, UK. Help the channel grow by liking, commenting and subscribing. Check out my Twitter, all sorts of odds and sods, and hints on future videos. I've got YouTube membership and Patreon as well, so check them out if you fancy supporting the channel financially. If you like the outro music on this video, please make your way over to my second channel made by John, and all that's left to say is Mr Music, play us out please.