Got it. Makes sense. Sweet. This should be fun. So let's talk about Edge Computer a little bit more. Our definition of it, we'll get some common ground. And all the things we did wrong, that hopefully, you'll not do wrong, which is very important. And the first thing I like to do is a little bit of a level set. So Edge Compute, what is it to me? So you have an understanding. So maybe it will make you go like, hey, wait, that is entirely different than what we're talking about. Basically, we're looking at compute that takes place at or near the physical location. Apologies for the creaky floor here. Computing that takes place at or near the physical location of the producer or consumer of data. Beautiful wording, super exciting. Edge locations may be properly built up data centers. More often than not, they're not properly built up data centers. They're shoddy hardware held together by stuff that you could find last minute at Home Depot or whatever your local equivalent is. And, well, you sold it to the customer and they signed for it, and that's great. Way before we had Edge Compute, beginning of the 90s, any point of presence for CDN, content delivery network was pretty much Edge Compute. Wasn't as smart as it is today, not a lot of functionality running on the Edge, usually serving some HTML computationally, still very important, still very impressive, but certainly not as beautiful as mobile wireless deployments that we see these days where you can run very high-end computing, very far out, in the middle of nowhere. Good stuff, though. Good stuff, way more fun than some of the things that we're running on. Point is, Edge Compute isn't new. And yet, what we see with a lot of projects is that a lot of folks and engineering teams, good engineering teams, are making the same mistakes at the age that we made in our data center that we made in the cloud, irrespective of that actually being a mistake you need to be making. So for today, I want to look at three cases, three engineering projects, share some of the things that we learned from them, I want to talk about over-engineering and thermostat, I want to talk about making a really fancy counting script, and I want to dive deep into feeding fish without the frenzy. And my goal is really for you to leave this session with enough information that you can make new and better mistakes than the ones we did. So before I joined HashiCorp, I worked on industrial IoT at the Amsterdam Airport. Super exciting. Airports, fun places to work at. Quick shout out to my favorite airport, 52 million passengers a year. That is a lot. Puts us in third or fourth place in the world. That's post-pandemic, obviously. Amsterdam Airport. Script poll. Yes, sir. Thank you. I'll take all the credit for that, except when the security lines are long. That's definitely somebody else. If you've been at airports, most people come to conferences via airports. You know that the primary function of an airport is to get people from A to B. Load and unload humans meet where it works. The parting passengers take a route that starts at the curb or train station, obviously in the Netherlands you show up by bike. You head through baggage drop-off, you go through security, pass chops, hopefully enough that they can offset their very high retail fees and real estate fees. And to the gate, cross the jet bridge into the plane with the goal of having butts in seat before the plane is supposed to depart. Plane departs late. Somebody is paying for that. And next year your ticket prices are going to go up. Not very great. Arriving passengers, pretty much the reverse of this process, leave the plane, head on out to the curb. Usually don't go past the shops. Much is chagrin of all those shop owners. And of course, if you're one of the passengers that has a connecting flight, we do about 30 million of those in Amsterdam every year. Not 30 million flights, 30 million passengers. Should point that out. Not that bad for the environment. Your journey ends and starts at jet bridges or in planes. And computationally, every person is a couple of those in API requests. When you show up, you scan your ticket.  And the airline has to know where you are at. The airport has to know where you're at. We're at a venue that has a fire code. An airport has a fire code. Slightly stricter. Because panic there can be much more troublesome. If you've ever been at airports, where there's a fire alarm, you'll find that that is the smoothest experience that you'll ever experience for a fire alarm. Because people know their stuff. Because there's data that tells them these are the areas that we need to focus on first. And so to make those protocols work, there's a lot of data that goes around. Literally every step that you take in the airport and every status change in terms of drop off, checking in, going through security, there's an API request at least for us attached to it. There's an acronym attached to it because, well, aviation is very much like the military in that sense. Anything that can be a three-letter acronym will be a three-letter acronym. But as engineers, of course, we know all these API requests, they get stored somewhere. So imagine yourself at a busy airport. It seems like a good thing. I mean, if it's busy there, it's good for you, good for the airline. In reality, it's not. Other than fire and safety hazards, keeping those aside for a second, busy airport where you're spending too much time not deep-planing and actually planning and leaving the area means it's a heavier burden on the API, the Twitter API, because people end up complaining. So airport design is all about risk mitigation, which means PAX ops, passenger operations, is very much about understanding what the pulse of an area is. And one way to ensure that is to use some sensing equipment. And the mission statement is always fun. We would like to know how our airport is doing. Imagine stepping into a control room, beautiful, high-risk screens, and all you want to know is, how's my baby doing? Problem with that is how your baby is doing is not a binary question. It is a question of about 150 different data points that somehow need to end up in a database that you then can correlate from, correlate the right insights from, and basically deduce the answer to that question. So if we're talking about sensing equipment, well, it's easy. Airport, big area, so let's deploy a wireless IoT network. I'm some airport-use Laura, one of the coolest technologies, if you ever get to work with it. On paper, 16 kilometers or 10-mile range, six miles and 10 kilometers. Math is very hard right now, which is why we usually use spreadsheets for this. And to put that into perspective, if you have a home Wi-Fi, even fancy outdoor Wi-Fi, best you're going to get is like 500 feet. So we're talking range. Except if you've ever worked with radio and spectrums like that, it's not the range you're going to get. Luckily, our software can account for that. If you have beacons that don't transmit and get a check that, hey, your data has actually been transmitted, they'll happily retransmit. Of course, you're going to pay some battery for that. So we'll look into that a little bit. The cool thing about Laura is it uses a chirp protocol. Spectrum, there is too difficult to explain because I lack the way of simplifying it. But imagine a beacon that can't have a battery can last from five to 10 years. My phone lasts about 18 hours. So five to 10 years sounds pretty good to me. And of course, you're not going to get the same fidelity. Like, your phone does much more. It's high-powered. But a beacon at the edge that can calculate humidity, temperature, noise level, assign those values to an area and then help you figure out, hey, is this area safe or not? Very important. Because that means you can make operational choices. So if we simplify the statement a little bit, let's deploy 1,000 plus sensors that can track occupancy. Is the check-in desk open? Very important because it means that little check in the app that you have from the airline that says go to this counter actually sends you to a counter that is open at that point. Temperature and humidity. Do we need to kick in AC? Do we need to make sure we have a temperature that's useful for people that's healthy? Very important. Amsterdam does get three days of summer. And we like to be cold at that point. Not chill, just actually cold. Because the Netherlands, complaining about the weather, is very important. It's one of the first skills people learn. And of course, safety. We carry around phones. You have a wireless network. We have whatever the PC term is of sniffers that figure out how many people are actually in an area to make security assessments. If there's too many people, you need more security personnel because there's a ratio you need to hit. Lots and lots of different data points. And remember, again, those sensors can last five to 10 years from an operational perspective. Well, let me take a step back here. Who here stayed at a company for more than three years? That is surprisingly different than I thought this was going to go. How about five years? All right, there we go. See? Already getting a lower group. Most software engineers stay two years. You're never going to worry about the battery having to be replaced. So it's not a problem that you're going to face early on. It's very much said it in forget-it kind of territory. You deploy it, and by the time the battery needs replacement, you're probably going to get the third generation of that product. Operationally, though, that means you have a mass problem. And the PAXOps team would like to know how long this stuff is going to run before they have to write a new check. And luckily, simple formula for it. We'll start by defining the capacity of a single battery. This is going to be fun, I think. We'll define the current that's used for various active and passive operations. We'll, of course, define the amount of measurements we're taking every time you're measuring, you're using battery. Every time you switch on your phone, you're using battery. And then we define how long each operation takes. A measurement might take half a second. A transmission of the payload that you have might actually take five seconds. There's a cost attached to that. And finally, we have to account for self-discharging, because even when not in use, batteries bleed capacity. And so with all those defined, we've got a pretty simple formula. It is so simple that you could run this as a shell script, as a bash script. You probably shouldn't. We didn't. We found a vendor that commercialized a very nice open source, big data solution and allowed us to store this. Reason for that, really simple. If you deploy 10,000 of these devices and you need to do these calculations a lot, it is nice to have an integrated solution. Ultimately, if we go through this, take the amount of time that the device is in measurement mode, times the, looking at my preview here, the current, when the device is in measurement mode, divide it by the amount of milliseconds per hour, it's a lot. I don't like it. And then just some more math. And once we get through that, we're figuring out every part here. And we were really smart. 365.25. You know why the 25 is in there. Better is going to last 10 years. We're going to hit at least two leap days in those 10 years. I think that's great. We were so smart. We considered ourselves a powerful engineering culture. And we got our number. We're kind of disappointed. Package said eight to nine years. We'll take 7.6. I mean, it's still a beautiful number. None of your AAA batteries get there. But our plan was to exchange the batteries after six years for one simple reason. Operational overhead. And we wanted to make sure stuff doesn't break when we don't need it to break. And so, if we're going back to that mission control room, we graphed this for every sensor. And we got a nice little line. Line looked great. We started in 2018. By 2023, we were going to be at just above 50% of battery left. It's great. Like at that point, I was long going to be rotated out of that project. It's going to be somebody else's problem. But at least it was an immediate problem. And so the next thing we did was we started tracking it with the actual battery. Rural usage versus what our spec sheet said. And you might say we were confused. You might think the formula is wrong, but the formula is simple enough that if we spend 15 minutes together, you can completely understand that formula. It was not complex after all. So we kept measuring. We went from confused to, I think puzzled is a word that we would use. Because something was happening. What was happening? Nobody knew. Summer was coming, so most of us just decided to take a break. We came back. It was great. This was not the 7.6 years that we were promised. You're no longer puzzled at this point. You're just deeply frustrated. But it gets better. After we hit 4%, we hit 0% pretty quickly thereafter. Battery capacity that bleeds like this, when you now have to deploy an army of people to replace batteries in small hidden devices, is very costly. The cost of the batteries pales in comparison to the people and the hours they're going to spend fixing this problem, which was a software engineering problem from the get-go. What we didn't account for when you buy 10k batteries is that buying them in batches of 1k batteries is great, because you should never buy lots of hardware from the same vendor in the same batch. It also means that you can just test 5% of them and extrapolate from that. That will not tell you if the whole lot is good to go. The other thing that is very important, software engineers bring a certain confidence or hubris, depending on which side of this conversation you're on, to engineering projects. As a software engineer, I know how to make code go from left to right. I can also line it on PowerPoint slides. I'm less well-versed in thermal engineering and figuring out how the sensor that is attached to a ceiling might get affected by the ambient temperature. The surrounding humidity and the fact that an airport is built like a bunker in terms of radio transmissions, nothing goes out through the windows. That's great because we don't want to mess with the planes. It also means that all your devices actually transmit at about 50 times the power they were supposed to. Sorry, not 50 times the power, 50 times more often. For one simple reason, the device is told to send that payload, that measurement, to our central server. And it will do so until the battery is empty. Great, great experience. We were confused, puzzled, frustrated, all the above. It didn't look like we expected it to look like. And one reason for that is we had our very own NASA moment. NASA at one point kinetically landed a probe by misunderstanding metric and imperial and mixing both of those up in the same projects. And when I say kinetically landed, I mean that they smashed it right into the surface. I think the term for that is unscheduled disassembly. In our case, Laura has this fun way of calculating batteries. Most sensors give you battery range on a scale of 0 to 254. The engineer that worked on this project and the reviewer both did not read that part of the spec sheet. So the 83 in your calculation is not 83%. It's slightly less than that. It's about 2 thirds empty. Makes projecting things a lot harder. So first learning for today, when you do engineering projects, edge computing projects, understand the whole flow. Understand what you can be monitoring. I think that is literally the most critical thing. You are going to be deploying a network, usually, because most edge places are like dogs. You can't have just one. You have multiple deployments. Have monitoring in place that can account for the stuff that we don't traditionally account for. If you're running stuff in a cloud, things are easy. I don't have to worry about the temperature of my data center. I don't usually have to worry about it getting flooded unless you have something deployed in Paris. Different story, of course. And understand how math works, which is really the hardest part here. In our case, Laura does not give you enough compute on an edge node to run the monitoring you want to run. This is good, because in theory, you get many years of battery. In reality, it also means that you being able to monitor what's going on there is very far from the edge. In our case, you have to go to the packet forwarders. And then usually, in our case, we used our asset management system to figure out what was going to be needing replacement next. So first learning is really go beyond the software. I feel for this project, a software engineer would almost do well to have a brief course on electrical engineering and material sciences. When you walk around airports, you generally not see a lot of the IoT most airports deploy. That's because all that stuff is hidden. It's painted the same exact way as a ceiling. It's made to not stand out. That pain that looks like the ceiling of your airport might have some effect on the radio transmission. Not enough for it to be noticeable in a lab test, noticeable in a real world deployment, though. No contact survives, sorry, no plan survives contact with the enemy. This was pretty much that. For us being able to put these parts together, not just by understanding the spec sheets, but by having real world data, could have made a lot of difference. But Glasshuffful, we were also able to give job security to many maintenance engineers who were kind enough to replace batteries. Glasshuffful. So airports are cool, but let's talk about grocers. And in this case, building a fancy counting script and d-dossing ourselves from within. So imagine a company, an organization, that has 1,200 retail locations. 12% of these, 150, are in the hypermarket category. Basically, department store, grocer in one location. Same physical location. 80% on the go, kind of shops like the stuff you see at airports, the stuff you see at train stations. There's a lot of them. And then between the two, of course, the traditional supermarket. Just a retail location, nothing else attached to it. Altogether, sizable amount. This is the beautiful edge deployment. Each of those has between two and 75 checkout registers. All running outdated versions of Windows, not even Linux. Outdated version of Linux would be fine. Outdated version of Windows is just, well. Good times. We all know how retail stores work. You pay for a product and someone in the back of a storage area, somebody gets a notification that, hey, that was the last manga that we could sell for today. We need to restock and reorder. And usually, that notification is pretty seamless. You might also swipe your loyalty card and get points for the transaction that you carried out. Of course, that mapping is never one-to-one. It's usually not in your favor. But hey, we still chase those points because those points eventually mean we get a free drink after spending about $561. And we love it, right? Free stuff is good. The thing with that is really, there's a lot of engineering that happens for the store or anyone who uses a loyalty scheme to cheat you. Because all that screwing around with numbers has to happen somewhere. In our case, it happened on the point of sales. POS, not the other POS. And let me put it this way. The about this operating system screen had a number that started with a 1. That was not because the operating system was version 1. That was a copyright. So you can figure out how modern our tooling was and what kind of access we had to in terms of anything remotely modern. So one way to work around this, if the checkout register is your absolute edge, and it's moderately locked down. Sure, USB ports are open. And as an attacker, there's about 15 different talks you could have for any supply chain security conference, any security conference as well. But we couldn't install anything. The good team was not allowed to install software in there. We just invited the hackers to do it. Well, all of the stores have their own processing. Slightly more powerful devices, usually unattended. In our case, I wanted to put the picture in that we had. Unattended Dell desktops that collected all the data. And I see you shaking your head. Sorry, nodding your head. It is not as far-fetched as it sounds. When you see that in movies, they didn't pick those computers because that was the only thing that was available in the prop department.  And it's painful. Usually the Windows license key stickers also scratched off. So the data that gets sent around is not super special for each transaction in our case. We built loyalty card software, and we wanted to make sure people got their loyalty. You've got about four different things that we want to have from you. We've got a transaction ID, very important, because loyalty points are technically equivalent to money, PCI DSS, and everything else applies. We, of course, need to know when the transaction took place, which card was swiped, or scanned, or whatever, and the amount before taxes, because obviously you don't get the points with taxes included. That would be bad for everyone. These transactions are 120 to 200 bytes, nothing big. Even in this case, this was a store in Asia, so if you're using UTF-16, it's still not a lot. One megabyte is good for about 4.1K transactions. And of course, you're accounting for envelopes. So in our case, we had a factor of 25 on top of it, 25%, because we want to be generous, because we want to make sure we have enough storage. And per store, average 2.5K transactions a day. Of course, the hypermarts much more, but all the small ones much less. Averages are always a lie, but the 2.5 is actually what we worked with, and it worked for us. Apologies for that. Per store, transaction size minus Q. If I'm downloading an app update from the app store right now, it's going to be about 100 times that. Here's the problem. 723 megs is not a lot of data. What is a lot of data is 3 million API calls, especially because all of these stores are in a single time zone, and they all close at the same time or within 15 minutes of each other. You cannot build an API that withstands that many inputs without having any brokering in between. We didn't, so we deduced ourselves, which was great. All these devices came from a trusted network. So of course, the web application firewall appliance that was not cheap did not account for any of that, because if you come from within, then who's going to do that at scale? Who's going to attack us from the inside? It was great. Our network security team had the brilliant idea of, why don't we just have the web application firewall use as a throttling mechanism?  So now the stores are closed. The stores are going to open again in about seven hours, and we are blocking stores from actually sending the data. All those POS devices did not have a lot of storage, because they were not designed to have a lot of storage. The edge location was not meant to be the point where you store the data. It was purely a pass-through. So if we block devices and don't get our transactions actually transacted, there are problems. There are problems in terms of did you get your points assigned? Did you get enough points assigned for that special deal because this transaction happened at this point?  So in our case, the easiest solution was build a local cache. Lots of back and forth, lots of very silly ideas. I know we started every engineering meeting with, there's no silly idea.  There weren't so many.  We built a local cache in the coolest way possible, and the reason I think it's cool is because it worked. And code that works and solutions that work always beat solutions that don't work but have more features. We stored all store transactions per store in a single SQLite database. That was our local cache. They were eventually going to go to a Postgres database. But by having all of them per store on that one Dell machine in a single database, we had a solution. Inserting all those things in there, not a problem. Transmitting a single file, R-sync, in our case, SCP, super easy. We can account for that and we can then process all of that stuff, not at the age. And you might think, doesn't that give you a different problem? Isn't that like just shifting it? It's not. So I want to present you this from the SQLite documentation. The entire database ends up in a single cross-platform compatible file. I can give you one of those files now after not having transacted it for the past five years and you can run it. That is power. That is power because it's compatibility. And if there's one thing we figured out in IT, it's how to get files from A to B. Did we use the Dropbox client to do this? I cannot comment on that. But if one were to do that, it would certainly work. And I think that's what's important. Turns out that grabbing a thousand files, one each per location, processing them, super easy. And yes, we still run them through an API just because it's cleaner than actually inserting them. So the learning here, design simpler flows. That is the absolute key. Remove everything that you don't need at the age. Don't build it with your machine in mind. Build it with the edge in mind. Deploy early to the edge so you know the stuff you're gonna run into. If I were to find myself in that same situation today, we'd probably not use Dropbox, but we would definitely build the same thing and just encrypt it to be on the safe side. Simple reason there is these are basically financial transactions and what you want to be doing there is make sure you have an audit log. Constraints are your friend, not your enemy. Generally speaking and rebuilding from an API driven workflow that we all love that was very microservices and everything to basically let sync a file was not easy, but it allowed us to keep the security model intact. It allowed us to not open huge problems for everyone, including the ordering department and people got their loyalty rewards sooner. I feel like that's a win for everyone. And then while we're at it, build for offline first and this is the hardest part. Putting a fair day cage around your hardware and seeing how it behaves at that point is beautiful. Beautifully hard as well. Think about what you need in terms of storage, power and application architecture to support that. When you build your edge, basically build a mini data center. Think about job orchestration. If you're running these processes at the edge, how do you account for failures? These are unattended devices. People in the store are not generally the people that know how to fix all of that stuff. They can flip the power, that's about it. They can make sure the cable is working and plugged in. They can't even make sure the cable is working. They can make sure it's plugged in. These are very different things. Have a strategy that allows you to process enough at the edge and recover from failure. And so for the last five minutes, I wanna look at one of my favorite US cases, agricultural food production. I'm a huge fan of agriculture tech or agri tech. For one simple reason, we have more mouth to feed. If we don't use automation, shit's not gonna work, and people are gonna go hungry. We have lofty goals on every continent. We have lofty goals with every organization that wants to get into the news. The reality is it's a hard problem to solve. Lots of farms use IoT. John Deere, very famous for having a lot of stuff in their machinery, these are edge compute devices. Edge compute devices you don't own and you pay a licensing fee for, at least nowadays you get to change some of them. In Europe, I don't have the exact stats for all the other places, two thirds of farms use industrialized IoT. Robotics, basically farms managed by robots and industrial IoT deployments. Problem for that is the traditional farm is very good at caring for livestock, knows what to plant, which grains are ready for harvest, they're not traditionally software engineers. And that's okay. I'm not a farmer myself, I just enjoy the final product. In their case, if you think about it from our side, our industry, IT struggles with massive skill shortage in traditional IT, right? Even just getting our group back in the engineers to understand more front end, front end engineers to understand operations, security people to be well-liked by anyone. We struggle with that and we can figure out with our group, getting a farmer to become enough of a software engineer to be able to figure stuff out, impossible. When you're the kind of farmer that runs a fishery, so fish farm, your livestock gets even harder to maintain. You can't just walk up to the cow and look at it, you have a million different things, not a million, but a lot of different things that matter. You want to have lots of fish because lots of fish you can sell for a lot, it's a good thing, but to get to that stage, you need to figure out how to feed them, how to care for them. Not too much, not too little. If one of those kicks in, then you get that beautiful animation where your fish now no longer has any skin on their bones. And that creates a problem. You have to figure out when you're feeding, is the wind that's blowing over the pond shifting all the feed to the side of the water? Are the fish actually getting any of that? Again, just deploying a workload to the edge, doesn't mean it's actually being consumed. In this case, how do we measure that? Well, we can measure various parts of the water. The salinity is usually a good indication. The pH values, because they tell you how much of the food is actually hitting the water and changing the water makeup. Problem with changing the water makeup is also, too much of it lands and goes down, your algae grows. And most fish don't do well in that environment. If you were to account for all of these variables, you have a lot of stuff to keep in your head. So of course, in our case, we need aggregate tech to the rescue and we can figure this out. That animation was definitely slightly more colorful the last time I opened this file, which was yesterday morning. So close your eyes.  Imagine a box of hardware filled with a dispensing mechanism for food, various sensors attached to it, one cable for power, and that could be actual electrical power or power that comes from a solar panel or a battery pack. We all imagine beautiful device, beautiful edge deployment. It works, right? The first thing that we need to deal with here is we're dealing with a highly unqualified operator for that device. So we need to optimize generously. Every little tweak that you can make to your system that makes it stronger, more reliable and more trustworthy to somebody who doesn't want to expend time on figuring out how the tech works is worth its weight in gold. Not only will you have fast operations, you'll have lower CPU usage, all that great. You'll have a better working system. And for that, you need to be very, very strict with what you're building. We see a lot of architecture that talks about being compatible with the edge. Unless you're custom building stuff or working with projects that have stuff for the very edge, you're gonna have a system that uses more CPU than you want to be expanding. For a single device in your lab, nobody cares. For a deployment in the middle of nowhere, 150 miles from the closest repair shop, all of these things start to matter. So the real hardcore thing here, compile CPU-specific binaries, strip everything that you don't need. Second learning, use boring hardware. The fancy hardware your vendors will sell you is not the stuff you want. Figure out what you need, figure out what you can build. This project was built on Raspberry Pi 2s. We're talking 2022 using 2015, 2014 hardware. One gig of RAM, 900 megahertz of CPU. And at least in production until 2026. 11 years after it was, 11, 12 years after it was originally introduced. That is power. Boring hardware is good hardware. Readily having access to large amounts of that hardware is even better. And so when we last chatted about this with somebody, they were like, well, boring hardware, I don't want to use that. I want to have something fancy at the age. I want to have lots of LEDs that tell me everything is going well. And this just doesn't really matter. I present to you a setup that had 115 compute nodes, all Raspberry Pis. It was a single cluster running HashiCorp Nomad. All together, 115 gigabytes. My Mac at home has 48 gigs, so that amount is not a lot. But if that amount puts 115 fish farms in a position where they can monitor food production for a cost of less than 50 bucks in terms of hardware, those 115 gigs are the single coolest 115 gigs that you will ever come across. Small things add up, especially when used with the right orchestration. And so with that, I want to finish on a lighter node. I came across this one this morning on ours. The airport project struggled heavily with battery consumption. The farming project entirely focused on producing food better and more sustainably. And it looks like those two worlds are coming together. Technology on the edge and everywhere is always changing. Sometimes food are better. I don't know how I feel about edible batteries, but I think it's cool. So I want to leave you with one thought. All of these projects are hard, especially if you're one of a few engineers that works on it. Engineering, security, networking, everything in this area is a team sport. Don't do it all by yourself. Make friends along the way. Learn about the other areas of engineering and build a better final product. Thank you so much. Definitely taking questions if you have any. It is sadly not a very isolated setup. Windows at one point sold licenses for point of sale systems for very cheap. That was before everyone started switching to iPads and we will pay for that until 2045. Did I actually admit to using Dropbox? I just hinted at, yes, well, error correction is important and we solve the problem. And as engineers, that's key to what we do. Thank you.


Welcome to my talk mega code to facilitate gates and in this talk I will simply present what I did with this remote It will be About electronics about microcontrollers about software defined radio, but I won't go into details Also because they are not a lot of details. It's a simple level So everyone should be able to understand it and even if you didn't do any hardware hacking yet This talk should emphasize you to do it because it doesn't have to be hard Also, it's not a new attack. I will spoil it from beginning on the remote is sending a fixed code And I just replay this code so no new attack on there, but the way to achieve it. This was quite fun so whenever if you ever wanted to make some electronics and You find that LEDs are boring But you don't know what to do or if you wanted to play with software defined radio But you think that software defined radio or radio transmission are complicated, which they are then this is the perfect talk for you Yeah, there's one which is who is happy. That's good one happy people So I looked at this remote. I was in the US for a couple of months and they gave me this remote to access my building They gave it to everyone and I wanted to know how secure is this access gates or the access gate to the building So this remote is used for the garage in the US You're almost forced to have a car because public transport is not that good So they give it to access the garage, but also they give it to access the main entry nobody uses it But it's still there and what's important. They gave you to access to pool area. Don't ask me why it's fenced I also find it a bit stupid, but it's fenced You have the remote you can access it and we will see that the hot tub will play in very important parts in in the Reason why I looked at it simply because there is this gate you can jump over it, but it's a lot more fun to hack it, isn't it? So they provide you one remote which is only for one part of the building You shouldn't have access to the other buildings and as you can see this complex this falsity complex has a lot of buildings And this remote is not only used in this complex. It's used In a lot of complexes in in California or at least just by driving why I could recognize some of the things and if you Have seen this remote and it's exactly the same system But before starting to already disassemble the remote It's important to find information particularly if you're a beginner if you already Have done a lot of hardware hacking then you can already disassemble it look at it figure out What component is what what it does if you already software defined radio? You will probably find out which kind of modulation it uses in the beginning, but if you've never done it It's pretty hard. So finding some documents about the the remote will help you to indicate, okay? This is how it works. This is how it modulates and so on and I can only in facet. It will save a lot of time So we need to identify the remote we can see on the front There's not a lot of information on the back nothing written there is even a sticker missing and If you open it you see that it doesn't have the electronic is pretty simply it doesn't have a lot of components But it doesn't tell you who is the vendor or what's the product is on the back to it doesn't tell you They're even to stick it with I think it's the code which is transmitted But they've never really figured out how they encoded it how they transmitted it. It's not too important though But we didn't find any information whenever there is a Transmitter there is a receiver So you just run around the building and try to find the receivers to know if they have markings on the garage gate The receiver is on the top. It's just a black box. It doesn't tell you a lot on the pool It's also just aluminium box no markings on it. We still don't know at the main entry this is where you find a fancy dial pad to Contact the resident and we see on the top. It says it's from linear so at least we know the vendor probably and after such hard work and running around the building you enjoy the hot tub and This is where you use the second skill your social skills simply because every resident has a remote So you make the acquaintance you make your friends and you ask please can have a look at your remote and This one had the sticker and it's particularly useful from from the sticker We find the vendor is linear and the product is act 34 B on the website. We can find it So the product still exists We know it operates on 318 megahertz it can send 1 million codes but the manual doesn't tell a lot and We already know with frequency in transmission, but we want to continue and find more information Again, if we look at the remote on the top you will see there is an FCC ID and this is one thing I know of in the US is that Whenever a manufacturer wants to produce something which transmits radio They have to comply to some regulations and they send test reports to the regulation authority FCC to show Okay, I transmitted that power and it complies to FCC part 15 for radio transmission and the FCC Shows you these documents and compared to the manual is a technical documents So this is where we find the really neat information and we can see that they provide a lot of Data, although they don't have to provide so many so first of the test reports to tell you what kind of transmission It uses and here we see its Amplitude modulation something which is very simple. So the signal if it's strong or not gives you the level It's also pulse position a 1d and if you look in Wikipedia a 1d stands for amplitude One is just one channel 380 megahertz D for digital I think so there's only two levels either on or off very simple transmission and the Data is probably coded in pulse position I even provide you some information about how the mega code code is sent you have 25 bit frames 24 bit frames each bit frames it's six milliseconds long between each other and within each bit frame You have a one millisecond pulse So you have a the remote goes on and just transmit very loud at this at this frequency And they were so kind to provide with a timing diagram and everything just because of the you we found everything to the FCC ID and we almost already know how it's encoded. So we spared a lot of time And as you can see we have a sync bit so the remote receiver knows, okay, there's a signal I record the 23 20 bit system code bits the three data bits And then there's a blank cell before you next to send the next code And as I written is each bit frame is six milliseconds long each pulse is one millisecond long now we know how it transmits it start to To play with software defined radio and even for entry level you have this cheap RTL STR Which everyone speaks about only $20. It's nice software defined radio You look at the frequency here used as the range love you tune to the frequency and This is on the left of frequency on the top You can see a fast free transformation which will tell you at which frequency there is a strong signal and we see There's a peak at 318 megahertz on the bottom. You see a waterfall diagram It's almost the same that on the top, but you have the timing component So you can see over time how the signal is and you can clearly see the pulses every time on and off Whenever there's a yellow peak, it's a it's a pulse It seems to correspond to them to the Specification and that's that's good Software defined radio can be complicated and canoe radio is complicated and I don't I'm not a fan of it Oh, I don't know how to use it. It's probably very good But we want to keep things very easy and we know it's a m modulation. So I just use a tool called RTL FM which Cannot do a m modulation. It's thought for Listening to audio using the software defined radio. So you tuned your frequency you tell it a m frequency It's a m modulation You put it in a file and then you open the file using any audio editing tool and here we can see again Two times 24 pulses if we look at the details We find the pulses are one many seconds long and you have groups You have bursts and you have bit frames of six milliseconds and we know the information It's pulse position. It's quite useful because it's a bit frame So there's only one bit per frame and then there's a position and if you look at it If the burst isn't the first half, it's a zero if it's a second half It's a one. This is not written into documentation, but you figure it out pretty easily And we know it's encoded so we route a program which just takes this Demodulated data and finds out the code. It's pretty short hundred seven lines of code It detects the edges it groups the edges into pulses of one milliseconds Then it knows it groups the one millisecond pulses in group of 24 and then it decodes it and here we can see on the left the value which are decoded the 24 bits this three bytes and We immediately see it's exactly the same code all the time It's individual per remote, but it's actually the same So we have a replay attack if we can record it and if we can send it we have a clone of the remote Only using this documentation which we have So we have we want to send it Look again at the FCC ID. Can we reflash the remote using our own code in electronics? You have a Board layout which tells you which component is where this this was also provided You can see footprints for putting a switch as one and every component has a Reference as one is for switch you one is for the microcontroller and so on basics of So you will learn how to read printed secret boards There are also schematics schematics a bit more abstract way Before you do your secret board you want to Just to know which component is connected to which other where they are placed and how they're connected You don't really care. You just want to find out how they're connected. This is what schematics are for And if you get if you learn a bit how how they work You will find out there the switches on the left the microcontroller in the middle The clock just just behind it for 380 megahertz and then the antenna This is also what we see on the identify again on the board because we have the reference which tells us U1 is the microcontroller Look for you one on the board You will find the microcontroller and you will find the clock to switch the antenna and some passive components And even if you didn't do electronics, you already pretty much know how to read schematics How to read boards and how to identify components The problem with that one is that it uses microchip pick 12c microcontroller and This one is only one time programmable So you can look at the I know this because of the schematic the schematic told me it uses this chip And if you look for microchip, which is a big Microcontroller very known for hobby projects. You'll find is only one time programmable So I cannot reflash the code on it. It also has code protection. So I cannot read the firmware on it This way I cannot flash it. That's that's a bit of a shame But the simple is so easy that probably somebody already did a compatible device So you look you use you Amazon and eb skills you look for linear mega code compatible remote and you find one which is not by the same Vendor it's by transmitter solutions. It's the Monarch 31 30 18 lip w1k It tells you 318 megahertz perfect the same frequency. It's compatible with the linear act 31b That's what we have and it's programmable. That's interesting. The other one was not programmable The manual doesn't tell you how to program it. There's a small section tell Contact your manufacturer, but there's the fccid on the right and we've seen the fccid is pretty interesting So you Look at the fcid you don't find as much information as the previous one But at least you have a picture of the interior and you will almost immediately see that on the top There are some pin header which is on soldered and this indicates that this is the programming header Which you should connect to program it to? We buy it the programming it had is the same We can read from the top markings on the chip with kind of micro controller it is and this one is flashable It's again a micro chip pick micro controller from but it's based on flash and I can program it The next skill will be soldering If you're even if you never did any hardware you just have to solder the pins on the right side So you can connect your programmer and program the chip And for entry level is soldering. It's pretty good. It's pretty easy to make We also Want to know how the things are connected this time They didn't provide the schematic, but we learn a bit how to read boards We can find the micro controller the clock one LED one switch the programming header And if you check with the multimeter how they are connected you find the schematic again And you can write your own schematic this time Microcontroller clock switch pretty simple board actually even simpler than any Arduino Things like that. Oh, so even if you didn't use Arduino, you can use this as an entry programming We figured out how everything is connected. We know how the signal is modulated how you have to send it So it's time to write our own firmware and just enable the the clock and the transmission at one One millisecond long every six milliseconds, and we know the pattern And if you could switch to to the camera here, I'll show so yeah, you wrote the program It's pretty simple 125 lines of code and this way I can flash my code on it Oh Where is my terminal? Ah, here does some Can you show the camera? Okay, they will while they do it I will just start to software define radio so we can really see the transmission Which terminal is it software as they range love so here we have to software find you with the remote with antenna the remote is here now we enable It and if I press on it You see the the transmission which is made and you see it's on. Oh, you don't see it anymore This is the this is a receiver. I'll talk a bit later. And if you see on the side Whenever the code is transmitted the lad stays on so it means This transmitter will react to this code normally should blink blue. So I will do it by hand. It's somehow broken Like always demo so it will blink blue. It only Works with this remote. It does not work with this remote as you can see it doesn't do anything for now. Oh Yeah, that's So we've identified the frequency well just quit it and stand we just use the decoder which we had press on the remote a Couple of times You exit and here we see the value which we decoded on the left side We know which value it uses and if we look at the other remote which is here Triton I think I have to enable it Yeah make on Operation succeeded if I transmit this one So here you can see that it transmits using the LED I'll find that it uses another code Now I will simply edit the EEPROM With this code which is 218 a Flash it on it. So you use a standard. I use a picture to Program, I this is generally what you used to flash this micro This these microcontrollers Please flash please No picnic to found The USB is here Connected directly here. That's not USB. This is USB flashing flash EEPROM make on So now the thing is open and if we look Here again, and if we send the code Then we see that it showed open the gate And this is how we remote we clone codes Back to the presentation if I find it here, I didn't stop at I didn't stop at just cloning the remote I We have one code for one remote. We want to have even more How about getting codes from other remotes? You can do it with software defined radio again But it gets so the stimulator which you have is very central of frequency and the bandwidth is not It's very narrow. So with using software defined radio is quite complicated to record far away one signal You have to play with the and Gain and so on. I'm not good at software defined radio I'm a bit better electronics. So when there's a sender, there's a receiver just buy a receiver online for me They make a code look at the receiver open it you see again, not a lot of components and We already learned how to identify them. You have microcontroller the antenna Radio filter the voltage then some memory which memory where we you can store codes and generally This is the memory which is used to read out which code is allowed or not It's a single layer design so on the back you will see all the connections to all the other pins you can read them visually and Using you the components which you can identify in the path. You can already figure out who is connected to what he recreated the schematics again It uses a pick which is only one time programmable. We don't want that So we on solder the chip. That's your next skill which you will learn Not too hard vacuum just vacuum pop or solder wake and they're pretty resistant to this chip It's hard to break you put your own chip which is the same but just flashable The name is a theme you program everything you want You already know how the modulation works and you already wrote code for demodulating you already wrote code for the microcontroller So you do exactly the same, but just on a microcontroller on this microcontroller and then you put it just next to a garage door and You power it over USB and you wait until lots of people go home and go and go back and record lots of codes So you can impersonate anyone you want Just because you have the code and they are fixed but another problem is that you know when they leave home And you know where they come back its individual code So you probably could go home and then steal everything and have enough time to to leave back The pool again. This is the last piece of information Very important simply because the pool is fence at 10 o'clock. They kick you out So they have a security guard which comes in at 10 o'clock triggers his remote I wait with my decoder. I record the security code and I have the security code and and enjoy the hot water And with the security code, I can access the pool after 10 o'clock I can access every building and you could access the security room with all the all the TV and If you don't want to wait for security You take one working code you flip the bit you flash it you test if the gate still opens and this way you Find which bit is important which bit isn't which business are relevant and from the 24 bits you have only 15 bits Which are relevant so you could write exact you could write a firmware on this remote Which just starts brute-forcing the code and you don't have to find one code you have thousands of residents So with 15 bits of thousands of residents. It's pretty easy to find the right code to to enter the same building But because we're not evil hackers we tell the vendor and as always the vendor just doesn't care even if they provide security products and I also Don't I'm not I also show this talk because solutions to it are quite easy. They're called rolling codes where the code changes all the time But you have a fixed seed If you do such a system you should use rolling code. So it's therefore if they don't do it What we've learned is that It's not hard to do hardware hacking we reverse engineer a real device used if for security for gates We improved how we search for documents if you see we've made new friends at hot tub We know how to program easy microcontroller. There are lots of code examples, which will help you to program this one. It's really Very well documented. We use software define radio. It's at this level. It's not voodoo. It works We know how to solder and desolder. We had fun and if you want money info, there's Wiki there are two videos which go more in the details and the source code is also available and with that I hope much for questions Thank you very much for this great talk. So any questions, please line up behind the microphones. Oh, yeah, number four, please Hello, I work at a research facility. It's an accelerator facility and I've got access to my laboratory rooms with this stuff and There are really expensive Equipment behind the doors. So the first thing I'm gonna find out is if this works If it doesn't what what you mentioned this rolling codes So if it doesn't work like that, what is rolling cold and is it easy to break or is it's no Their solution for not cloning remotes and this is partly important. For example, if you have a car Because you don't want everyone to steal the car and you have cars on remote the car industry I'll use a lot of time rolling codes and how it works is that you have one seed in the In the remote and every time you press on the button It's generates and it calculates a hash some kind of a hash of the seed and every time it changes because the counter increments And so under in the car, it has exactly the same seed. It knows this remote has this seed So I know which code will be next and the code every time will be completely random So you don't know for which come from you don't know the seed and if you have a central building then you have a central remote system Which we which knows where? Which knows which code is transmitted So you should they should use rolling codes if they have central system If they don't have central system, it's a bit harder because you need to synchronize the two But yeah, try to find one with rolling codes and you can using software find radio You can see easily if it's uses rolling code or not Okay, thank you The internet has a question Okay, yeah, the internet wants to know Or somebody on the internet wants to know be why you didn't get Into or if you did to get into any kind of trouble because of you breaking into the hot tub or like Well act being able to open any of them doors I've I haven't been to the security room because I'm not interested in the security room I'm interested in relaxing and also in this building It's not really important to have a remote and to have all this hassle of reverse engineering because you see in the hot tub Barry is pretty low so you jump over it and around the building you will always find a door which is open So you can always come in the building and I had no problem even with the security I mean, they don't see if it's a clone remote or not one more question from the internet, please Yeah, somebody wants to know About if it's possible to open cars with this No, not not new cars new cars use rolling codes and still have strong stronger encryption. So If your car uses this you should sue them But generally cause have rolling cars have rolling codes and you can use not use this technique So it's a lot more advanced in cars try to have use something which is less attacked Okay, thank you So I guess we have time for one more question from the internet once more well Okay, I was going to say that there's none, but there's one now And that is what other stuff can I open with your method? Well, we've did it for one Garbage remoto and generally garage remoto's are not very secure try to find another one This is just one one product from one company which is in California in Europe They probably will use for something completely differently So look at together at your garage door or the garage door of your neighbor things like that Okay, so very very simple devices when they achieve generally they are simple so look at that Okay, thank you Thank you very much. Thank you


half the time, maybe a bit less, if you just ask for a cross compiler please, it will complain that it can't install because they get out of sync with, Debian changes theirs, it takes us a while to rebuild, and that breaks, which is very annoying. I get quite a lot of complaints and there's a lot on the mailing list saying, you told me to install this compiler but it says, well, apt can't because of long complicated list of complaints. There's this building them in Debian plan, I don't know if that will help, do you want to say something about the uninstallability problem? You just want to talk about how this works. Hello? Yeah, we have agreed a mechanism to have proper cross compilers under Debian instead of having them in Debian repositories. I guess nobody can read this, can you? No, sorry. I had to make it fit on the screen so I made it smaller. We could have it so that you could only see the top three lines, then you could read it. We'll try and explain. Yeah, okay. In fact, I'll keep that in case I need it later. The uninstallability problem is just some dependencies that the cross-touching has with the GCC-based package and yeah, I think that's it. Maybe some other libraries, there was a dependency with LibGump with this library for big machines with embedded, doesn't care, but this has been removed because it also caused problems with the native compiler and it has been removed. So we are planning to, and the problem is we have one source package and we upload this source package into the build-it-is and that builds the native compiler. There isn't a PDF. I did make. Well, you upload a source package to the auto-builder and it builds the native compiler and we don't want to depend on the cross-touching as well in the same source package. So we need to have other source packages to upload into the build-it-is and this, we have designed a way to do that by, maybe the other one, the, by. You don't need the diagram. Yeah, we have some source packages that depend on the binary packages containing the source, like the linker, the compiler, the kernel headers. You have the linker, you have the compiler, you have the C-library and the kernel headers. All this is involved to build a cross-touching. We have circular dependencies with C-library and the compiler library and we need to implement the first stage bootstrapping in the Debian compiler, because right now it only builds two states and in order to fulfill these dependencies correctly, from source, you need to do a three-stage bootstrapping of the tool chain. And the first stage needs to be implemented. And basically, this is like a dependency tree and that's how we are going to build this tool chain. I know you have some questions about it or you want me to explain a little bit more or this more technical. You can come up to me, I'm around and we can talk about it. So the point about this is it doesn't need any cross dependencies so we can do it with the existing builders, is that right? Yeah, and this will be done in Debian so when GCC uploads and then this will upload in the same day probably and it will have the same versions. It will be right in sync or in sync. The problem is that if you type de-package build package in the G-Lib C sources nothing happens. That's what I'm talking about. Oh. You mean our diagram didn't make sense to you. I can't see the diagram. I'm sick. Okay. Yes, right, good. Do you know when, how long will this take before we move to doing it this way? This can be done very well. We have to implement the first stage, bootstrapping and then when that's done then we can, yeah. Okay, yeah. So that should be good. That'll be cool, I hope. Maybe take half a year or... So, for those of you who've been following the multi-arch debate, this actually is quite fundamental to how people cross-build things at the moment. Like I say, to some degree this is a cross-compiling discussion and not an MDB in discussion but obviously quite a lot of us are cross-building things. I know how many people here cross-build things? Okay, a few. How many people have ever used MDB for anything? Slightly more. Wait for the mic. In my packages, Neil have sent me patches to support cross-compilation. Is there any way I could test that this is still working because during one of the changes we dropped one of the variables and it stopped cross-compiling. So is there anything I could do to test that cross-compilation is still working in my packages? Yeah, there's a couple of ways. So there's two parts to this. There's cross-building it at all. Are you exactly the same package or there's cross-building it in MDB and which implies applying our patches as well. Do you want to test the bare package just the cross-build part? No, what I mean is if there's any way like running few parts to cross-compile my packages so it's still working and I see if it's working or it's not working. You can get the cross-compiler from the MDB in the repository or zoom from MDB directly and try compiling your package with the package build package dash a some target architecture you have a cross-compiler for. So fundamentally it's that simple. Instead of typing dpackage build package dash rfake root you type dpackage build package dash a rml dash rfake root see if it works. But the problem with that is that you need all the cross-dependencies installed that allow it to build and that's the hard part. Now you know what your cross-dependencies are for so your own package is not too difficult to install all cross-dependencies which you do with dpackage cross or apt cross you should be able to say apt cross package your package and it goes and gets all the cross-dependencies and installs them. But in practice that doesn't always work it works if your package is very simple. Yes, but it would be very nice if this could be integrated in let's say pure parts because I always run pure parts with my packages. So you don't want to know anything you just want it to work. Okay, I mean yes you're right that will be really nice and no it's not in pu parts I don't know how easy that is. Yeah, that'll be a useful thing for somebody to do. I don't know how hard it is. Also I have set up an experimental repository of dpackage cross packages for army L. So you can just point apt edit and you get all the like lip something dash army L dash cross packages. Yeah, so the point is the cross packages you can either build them from the arm repository and actually generate a cross package now this minute or if somebody somebody could do it in all advance so that every package that needs crossing is already crossed and then apt can just get it. So you don't need to do the crossing you don't need the cross machinery to work but you still need to install the dependencies. Yeah, anyway I'm going to figure out how to do this and send the patch to the people. So yeah, I'd like to be able to use P builder effectively to do you know P builders good it would be nice if P builder cross worked but fundamentally it doesn't yet. So that's actually the bigger the in practice our cross building for the M. Debbie and the light writer do at work works except for the fact that the automatic dependent installation of cross dependencies isn't really ready yet. So you have to you have to tend your troops and make sure they've got the right things in advance and you depend on the cross compiler. So the first step is to get the cross compiler into proper deviant. So you can integrate in pew parts or whatever build the build systems and also this is very helpful for QMU people. We have a talk afterwards to build some bios a boss or this kind of things they they build for other architectures and and right now they they do it manually or do it or either have to hack it a little bit. And this is so the first step is getting the cross compilers into deviant and then everything will will come. Yeah, then there's something for all the charooting build tools to depend on and say well you need this part. So yeah, you're right that'd be really nice that when that's that's nearly where we're at where we have all these things starting to happen so that it's it's looking like this will be possible. And also there's this very few few people of us and not all not all of us are working a hundred percent on on the deviant. If this is like a few more people getting excited about going oh really I could do that that sounds useful and would have things go faster. Did you want to say something else on it? I am the light writer. There we are that light writer that's all cross built and in fact is running MW and Grip. So people really do use it. Where is it? It's sat at the back plugged into the sound system. So yes so I get to do this for work. She's good. Dependency stuff all changes. Yeah so this is about the transition to multi-arch really. I guess we've covered that. One thing that just popped on my mind. Okay for Debian not for variants would it make sense to... Yes sorry that's what I was gonna say. To have another triplet for variants and basically treat your variants as a separate architecture and still remain within the multi-arch system. No because so the problem here is multi-arch is great lets you install all the cross dependencies side by side and build things and it's all very simple. The whole process actually gets simpler because now you just want the package. You don't need a special package munged into the native architecture with a different name so that we can install them side by side. It all just works. However it only works because the host... Now for those of you not familiar the terminology here is thoroughly confusing. The build machine is called build. The machine you're building for is called host which most people kind of think of as the build machine and you don't call it target because that's something else only to do with cross-compilers. So when I say host I mean the thing we're building for. This stuff... a multi-arch only really works where the host versions of stuff that you're trying to build is exactly the same as the build versions of stuff that you're trying to build because as soon as you run binaries or check config files if they're different it'll all fall over because the only thing you've got the foreign version of is the libraries. You haven't got the foreign version of the config files or users share anything which defines what was supported by this build of the package. So it only works basically if you're trying to build something that's almost exactly the same as the system you're building it on which is a major limitation as soon as you're doing proper embedded you tend to want to build it without LDAP and without X. You actually want to make changes in the packages and now the config files have changed but you can't install the host version of the config file and the target version of the config file on top of each other so you have to do it in a charoute or at least a GCC sysroute maybe in order to have a whole file system containing all the stuff you need and the problem with that is that you also have to have all the tools in there you've got to install scrollkeeper and latex for building documents and sgml tools and all sorts of garbage you didn't want just so that you can build the package. So a lot of the stuff we've been talking about this week allows you to cross build Debian. Doesn't necessarily allow you to cross build something a bit like Debian which in practice is what most embedded people want to do. That's a more complicated problem and it's important to bear that in mind when thinking about different ways of doing things. Did that make sense? I lost everybody yet. Basically if you'll just link against the libraries you don't care about the config files. Yeah but you don't just link against libraries when you build something. It's a very simple package you just link against libraries but anything at all GUI runs all sorts of weird checks and things to determine. I mean even just the tools the autocomf can go and check stuff whether such and such is supported and if it finds a library for x it'll go and build it with x it's supported and that's not what you wanted so you have to do configure without blah. Yeah well but the autocomf checks usually know about cross compiling. The autocomf check should work and if it's not we can fix that but as soon as you run binaries like oh I don't know someone find me a good example. You need to look sorry. Okay font config I think is actually quite a good example. Font config knows where it should look for its config so it's very likely to get the build systems config not the target systems config and then generate stuff for the whole load of fonts you haven't got and didn't want on the machine you're trying to build but I think in practice there are lots of examples of this problem. We haven't come across it too much because we build relatively simple things that just needs libraries and tools but as soon as you get into GUI building have we got any of the mayor people here? You know about this don't you? You see you just avoided the whole thing by using scratchbox so it all goes away. But I mean the reason scratchbox was created well one of the reasons was that it is very hard to get all those tools and things to be right if you try building in the the whole system without a charude. So we can build all of GPE and that works but we mostly do it in a charude anyway just to be sure. So it's something to think about but yeah I quite like your idea of a separate architecture. Well no that solves it's interesting but I don't think it solves the version skew problem as perhaps I shall call it. So another big part of this is what happens if you want to build a different version of things. There's relatively simple stuff like taking the docks out which works very well in Grip. It will be nice if we could get no docks support in the build tools like debhelper and CDBS as a debbin build option like no strip no test which I think are the only two that are normally supported isn't that right? Is there a debug option? Debug is sort of implied by no strip. Yeah okay. Basically the policies as we always build with debug symbols and then strip them off.  That works fine but it does mean you don't end up with a copyright file either which you might argue is not to kosher with the GPL. Mike? You cannot trust debhelper and CDBS to do all the things that gets installed in dock. No true. There's nothing to stop people just copying them in as well. Yeah exactly. So we'd have to patch that too but most things these days just list the files and get debhelper to do them. Do they not? Or CDBS. Or CDBS exactly. I mean I see fewer and fewer exception lines doing things manually in a lot of packages. It just does the right thing but you're right you can't rely on that it's not a complete solution. Or they use them and then they rename things afterwards. Yes. So that renaming will break if you try to override or suppress these parts of the debhelper. It's true. Basically that's a major limitation in debhelper. You cannot rename files while installing them. DH install only supports a target directory not a target file name. So if you want to have a file with a different name in the resulting package than it has had in the source package you need to rename it either before DH install time or afterwards. Yes true. So the debhelper no docs patch has been installed since May 2007 or something. So you have to persuade Joey it's a good idea. It is a bit crafty but it's a very easy win. Effectively that's what grip does. Pretty much. And it is quite effective. He's not here is he? Lazy bugger. There is an alternative to this which is to use d-package filters so you make the package the same as before but then you filter at install time which is more reliable because you'll definitely remove everything in user share doc. So the package size doesn't decrease. You still want to download it. But in a lot of context that doesn't matter much. So I guess that's something we should look at some more. Does grip use d-package filters to actually do the filtering? Nobody knows we need Neil. I think so. What else? A much harder problem. There was an interesting idea which Jonas had yesterday. So one thing we've discovered from doing this. If you make an MW and grip route it's about 90 megabytes for a base system which is quite a lot better than 400 meg or whatever 200 and something out of it. That's our base system with a fair amount of libraries in it. However as soon as you do app get update the first time and point it at W and it gets 50 or 60 megabytes bigger. So it gets more than half as big again just because of the package files, the list files, the cache files, all that stuff for managing our marvelous package system. And that's a big overhead. If it was 10 meg that would be good. 20 meg that wouldn't be too bad. 60 meg is far too much. And a lot of that is just the size of the package list. 22 megabytes compressed. So one thing we do in grip is chop off all the long description apart from the first four lines which makes an enormous difference to the size of the package file. So that's a good idea. Jonas suggested we could generate better package files on the fly. You could have a little proxy thing. So this would allow you to use real Debian but still to have a sensible size packages list. And also filter it further using DevTags. So it only includes, in the final list, includes the packages that is tagged in DevTags as being relevant for this embedded device. So we can tag everything embedded which would get you part of the list. And I don't know. You'd have to work out what suitable tags. But we can have any tags we like. Is that right? Or we have to ask Enrico for new tags? I believe we can throw in tags. We can even have a repository of DevTags ourselves. But that's more cool to work together with the DevTags people of course. So that's actually quite interesting idea because you now make an amazing array of distributions just by making new lists and pointing at the normal thing and then using de-package filter to filter stuff out. Interesting idea. So de-package vendor. How are we going to explain this? I don't know how this works. I tried to find out last night. I don't know if anybody else does either apart from, Neil is about any personal world who's actually used it. Simon's thought about it quite a lot. So this came out of the Debian Work session a bit over a year ago here where we were trying to work out a mechanism for dealing with the fact that hundreds of people make variants of Debian. Ubuntu is the most famous. Are we second most famous yet? I guess not. Limpus or somebody. Nopix. Yeah, okay. Do they actually build from Debian sources? I'm not sure what they do.  I think they do now, yeah. Okay. So and they all want to build something with some changes in the rules files generally. We certainly found that we can do everything we want by only changing stuff in the Debian directory. We never have to change anything in the package itself or almost never. You can pretty much do what you need by changing the Debian directory stuff, which is nice. That limits the surface of the problem. But it would be nice to have a generic way. So the problem with all the patches is they're hard to maintain. We want to put it into the package, but we can't really have every package listing if Debian do what I normally do, if Mdebian do something they told me to do, if Ubuntu do something else they told me to do, and so on for a list of 100 distributions. That doesn't seem very maintainable. The deep package vendor stuff also has inheritance. So basically, we just tell it to if it's Debian or derived from Debian and not derived from Mdebian or Ubuntu then do this. Okay, so we can have hierarchical rules. Yes. Okay, which might work. Doesn't that mean you need to know what the other guys did in order to decide what to do? Basically, you base your distribution off an existing one and so you tell the package vendor that you are deriving there and all the packages will basically default to the distribution you're deriving from and then you just have to add the patches you need. Okay, so the attractive idea is that the 97% of packages which are the same as whatever you chose for your upstream, you don't do anything to and the few you want to change, you add some information. So in theory, this is quite a nice generic thing. But as you can see, this is the sort of thing people have to put in there. So the top half is the obvious way of doing this in a rules file. You know, you check the vendor thing. And if it's not Debian, then do something at a whole lot of different configure args, which isn't very beautiful. Actually, it's actually it's even worse than this because you have to check the inheritance tree. Like, you have to do yourself. It doesn't kind of magically happen. Yeah, basically, you have a list of things you check. For example, you start by checking, am I am I on Mdebian? And the answer is either yes or no. If the answer is no, then you can ask, am I on Ubuntu? If the answer is no, then am I on Debian? The answer is probably going to be yes, because every distribution is derived from Debian. Yes. Okay, did somebody else? This is not very like the nicer solution. This is the better solution we have come up with. And if you come up with a better solution, it'll be good. Because the package maintainers don't like to have like, let's say you have 10 or 15 this derived distributions, you don't want to pollute all your rules files with all these these an entrance for each variant. So the idea of the lower half of this screen is a recent suggestion for making this generic or more generic.  And basically we have sorry. Oh, there is. Yes. Hi. Please don't do the lower one because because if it's not equal to Debian, if it's a bun to say, then we get the embedded version of the package. And while we like it, we don't necessarily want to have a bun to be an embedded distribution. That's true. Yeah. But the point was, you'd set your vendor to whatever. So you would have set when you're doing a build. Overall, you say my distribution is a bun to. So now it would say Mvendor, vendor a bun to package of R he key whatever. Yeah, but that require that that would then require us to modify every single source package in Debian to say, to say a bun to is like Debian. Whereas the DPKG vendor stuff supports the inheritance which which we which we were involved in because because that's useful for us. So we just if we want to make so we can say we are derived from Debian and then we can have DPKG vendor derives from a bun to which we would use for our customization so that our derivatives then inherit.  It's very new. So we haven't we haven't deployed much yet, but we were involved in the creation of it. So yeah, so we like it because because it allows us to and I think that's more of the reason why it's happened does asking for stuff doesn't necessarily cause much action. I guess a bun to asking for stuff is a little bit more that I don't think we asked for it. But I think we thought of it and people went we should tell you bun to people because they might like this might be useful. Yeah, it's good. And the inheritance stuff is good for us because we we want to we want to say that we inherit from Debian. So we get 99% of the packages correct, like you said, did but also support our derivatives. Okay, so I guess we should sit down afterwards and look at what I agree. I agree. There could be a better way of doing this. So the tools support it more.  You want some magic so that you know, it's just set and stuff happens. But I think somewhere somewhere it has to say what happens for the different variants. And the question is, where should it say that really should go in the rules? The maintainer doesn't necessarily want to know about every single different variant of there being there is. But there's also an argument that when you open a Debian rules file, you should know what it's going to build. You shouldn't have to you shouldn't have to then go and check a separate repository, which says, here are the changes that are made for this distribution. And that's what the second flavor does that actually goes and looks off in slash ETC for, you know, ETC package name, key name, blah, you know, basically put the database of things. And the problem with that is, as you say, you've broken the fairly fundamental idea that the rules file does tell you what's going to happen. You could, you could, you could, we could actually use the second version, but just write it in a slightly different way so that you don't necessarily, so you don't get the embedded version if you're not Debian. So it should be, it should be a check for if we're derived from Mdebian, then we use the second version or something. But yeah, something in the tool so you can do package name dot install dot Ubuntu or package name dot install dot and Debian, which doesn't include the documentation or something like that. That might be, that might be an interesting thing to pursue later. Okay. Good. Thank you for the clarification. Who are you? Are you representing Ubuntu or are you talking about something else? Just curious. Ubuntu. Okay. Thanks. Okay. This is an enormous subject and we've only got a quarter of an hour left.  Or again, just for clarification. This example you're showing here, that is intended for being put into each package or that is intended to be put into an DPK vendor configuration file master on the system that you're building. That goes in the rules file. Oh, it's not pretty, is it? Yeah, the biggest problem basically is if we want to use the inheritance stuff, then basically since there is no ordering on the names of the of the different distributions, it's very hard to decide. For example, if we have a, if we have a, if that helped gain support for that deep package render, and there was some install.dbn, install.mdbn and install.ubuntu, then it's pretty, that's a simple case basically because it's, it's pretty obvious which one is the most specific because it's, because these form a tree, but at some point it becomes, but for example, if you just have install files for two base systems, like embedded Debian and embedded Ubuntu or something like that, or embedded Debian and Ubuntu and you end up with an embedded Ubuntu system which install file should take precedence. Yes, it is tricky. I mean, I think something to think about, bear in mind when thinking about this, is that there's a set of changes which are actually changes by type and not really anything to do with the distribution as such. You know, no docs is actually just that, take out the docs. Now, that's something that a set of distributions would like, but there will also be a set of changes which really is quite specific to the distro. And this is a mechanism to let you choose both of those things, but I think we should try to have generic mechanisms like no strip and things which are, you know, entirely agnostic and just have a purpose, some purpose we can actually define that is useful and is worth doing because one thing that is, given the binary distribution focus of Debian, there is a limit to how customized things can be, you know, without rebuilding. We can only build a fairly small set of things, you know, a headless Debian.  Like Hecton, we also discussed a little bit yesterday. I see the difference between some of the flags we are using today as the no strip and then some of the things that are needed here. The difference I see in this is that things like no strip, you don't want to ship such a package typically. You want to build it for testing purposes and then you want to throw it away again and have it go back to the proper package. Whereas things like no ex or no doc is typically things that you want to ship. So I'll throw in the idea, the way I see it is like what we're talking about here, yes, technically the mechanism should be like flagging similar to the ones we know today, but the difference would be like, I would call it flavoring. Like a very few packages, Debian packages today have multiple builds providing actually separate binaries, binary packages for variants of Francis libraries or variants of packages like no GNOME versions of the package or like I do with one package that people have shouted at me about was providing a no ex version of LibGD. That is really different flavors. And I think in the discussion of trying to find a common sense of what kind of flavors do we want separate from the discussion of what kind of flags do we want to signal for the compilation. You're right, there's a lot to be said for a package knowing how to build these several variants. Basically different configures, sets, and it spits all those out. And the problem at the moment is that you always have to build all of them, I guess, which is a bit slow, perhaps. One thing that's really important is if you change the dependencies by not having X, for example, it does need to be a different package name, at least if you want it to go anywhere near the rest of the Debian packages. If you want to mostly use Debian and change something like that, it has to be a different package name. You can't just go changing it. That is exactly why I'm talking about this not being a compile flag, but a handling of flavors. Because I'm working on, I have support for this, not officially, but I am working on pushing it into the official CDPS support for flavoring so that you simply say, oh, there is two different flavors. Then it actually, from the full build system, makes sure that it gets rerouted into multiple binary packages. There's different places in the chain that could use this flavoring mechanism. Not as a, we force all Debian developers, they should all support these and obey all these flavors. No, no, as optional. If you want to be nice to the Debian people, then have in mind if this package might want to provide a No-X version also. If you say, what the fuck? This is so tied to X that it only makes sense to build it together with X, then don't support this flavor. But just having a common consensus on what flavors are, what are we talking at, multiple optional variants. Yeah, I think that's very sensible. So we're a bit short of time, so I think we need to talk some more about that. I think that's a really good idea. Just to cover the other issues that we care about in case anyone gets excited. C-Library, it's not a big deal, but it does ship an enormous amount of time zone crap and, not time zone, all the locale information which we then, from which you generate the actual locale information. And it kind of annoys me that to get my two megabyte C-Library, which I've got to have pretty much, ignoring the whole UC-Libsy thing, I have to have four megabytes of files for locales I never wanted to use. So in familiar, they do this by pre-generating them all and you only install a tiny little bit. And that's much nicer. The Aurelian doesn't actually want to change it all for Debian, although it is gonna get compressed, which will make it a lot smaller than it is now. But if anyone wants to work on a better packaging of G-Libsy for our purposes, that would be nice. The biggest problem we've found in Crush or using Busybox instead of CoreUtils is the millions of maintainer scripts and init scripts that use command options that aren't in Busybox. grep.x-x is a good example. Quite a lot of things do grep-x in their scripts. If you replace the normal grep with Busybox grep, doesn't work. Script blows up, packages aren't installed. That's a big deal. I think this should be treated kind of the same way as shell versus bash. You need to declare whether you're using fancy GNU stuff or not. But we have to work out where. The problem is that CoreUtils is essential. So for Debian's purposes, you don't need to declare that you depended on it. It just kind of is. But somewhere it should say, and I'm not quite sure where somewhere is. But if anyone has a good idea, we could then start working through and filing bugs against things that use the extra options without saying so and we could fix them. Well, we could use an x-control field just to mark packages as requiring CoreUtils blah. Yeah, or as not requiring. And at the start, we could have like a field saying, okay, this package is clean. And at some point just decide that we are now going to start taking the other way around, okay? Or maybe use DevTax to have external to the package. The other thing we could do is basically have a wrapper package that just diverts all the CoreUtils and replaces them with just a small script that checks if any options that are not inside in the busy box variant of the tool. You mean an easy way for maintainers to actually check because often they don't know. Yeah, for maintainers and also users basically to have verifying that this system is still going to work when switching to busy box. For example, we could just lock, this script just called this tool with this flag which is not supported. And so we can find them easily. Just have a lock file of all. Yeah, sounds good. So I want to write that, that'd be really useful. The way I hear it sounds nice but it puts a burden on all of those doing the Mdebian to test it really. I'm thinking a way to do the opposite is to provide a package, like provide a build. I don't know how busy box is provided now if it's possible to install busy box without exploding your normal system. There's a version on Debian which doesn't explode your system, I believe. Okay, so install a busy box and then make a package that makes the semblance for grep and provides grep. Then in the normal Debian system, you have an alternative grep which then makes it easier for people to actually install the alternative similar to dash instead of bash. And then we can start the cultural thing of saying to people, hey, whoa, we really have to make sure that if you're using this funky grep, the GNU grep options, then it breaks for other variants of grep that is actually available for normal users. And then we can start pushing, please do this. And then so we can have it in the lintian and we can start finding bug reports with concrete examples of how they can verify this. Instead of it being a geeky thing to compile Mdebian so just to test this. Yep, that's actually kind of related to the other point which is making it possible to have a busy box say system but install the real versions. If you need real mount because you want to do something exotic or you need real find or something, making it so that, this certainly didn't work a while ago and I'm not sure if it does work now because busy box has moved on. We need to check, I know we run out of time. Install busy box and if you install find, find utils, you get the real version of find overwriting the busy box version without barfing that it's impossible and the files conflict and then ideally you take it away again, you get the busy box one back. That would be really neat. Somebody needs to test it. So, we've run out of time. There are lots of things to do. I hope some of that was interesting. We could do with some more people doing them because it's complicated and you get to kind of persuade the whole world. Very slow process. But it is nice to see, we started doing this seriously two or three years ago and quite a lot of stuff is now starting to change. So it does work and please come and talk to those of us with big mouths about any of this stuff if you have ideas or can help. Is there any way we could do something about this with the OpenMoco people? Yes, I'm sure they have exactly the same desires for many of these problems. I already have my free runner and I want to do something with that. Clearly, Mdebian on free runner is the thing I should be doing today. So let's have a go. I don't know if it works. Does it work? Right, okay. We have to stop. Thank you.