Welcome, everybody. Hopefully you're back from lunch, or enjoy the free IBM lunch, as far as I heard. So the next around 30 to 40 minutes, I'm going to talk a little bit on something that I've done about a year ago on trying to see how small we can actually Linux get and what you have to differently to make it a little bit smaller. So the talk is called Running Linux on Tiny Peripherals, and we're going to go through this one step by step and give you an idea on how small we can get and how we got there. My name is Marcel Holtmann. I work for the Intel Open Source Technology Center. And quick stuff over me, you will realize by some of the things in the talk I've picked this way. I've been working on Bluetooth since about 2000, maintained the Linux Bluetooth stack since 2004. I have done a couple of other projects, like a connection manager for back in the days for the Mego and Tizen operating systems, a telephony stack, a proxy manager, and something that will become interesting in a little bit of future on the Meta Linux library. And I will make have a quick detail on that one for a second later in the talk. I'm with Intel for over nine years now. I joined in 2007. I do a lot of standards work lately. So this was a little more of a fun project to get going and see if we can actually push the boundaries far enough to actually make this work. I'd like to start off with a quote that Jonathan Corbett did at the Kernel Summit in Korea last year, and fundamentally goes the way it's like, if we are not able to shrink Linux, then Linux will play absolutely no role in the IoT space when it comes to the devices that are really small and tiny. And someone else is going to eat our lunch. And I think he's spot on with this one. Mainly we keep moving Linux bigger and bigger. We put it onto enterprise systems, but we keep forgetting that it was actually really big in the early days and helped us to establish Linux dominance on the small devices. So backing on this comment, I said, look, let's see what we can actually do. And the question we're going to really, I'm going through in the next 20 to 30 minutes, is how small can we make Linux? What's our target? Where are we going? And where we are stuck? And where we should probably use something different. So the way I did this is I looked a little bit and fundamentally, and this is not to scale, if you can't read the bottom part of it. It's not a scale. I just picked some numbers in there to actually see where we are, where Linux is good, where Linux definitely has a use case in the embedded space and the IoT space, and where you really want to actually take an RTOS, like the recently released Zephyr from Intel, for example, and we're going to do this. So the generally scale goes, if you're more than 512 megabyte, there's no questions to ask. You want to run Linux. 32 to 512, sure you run Linux as well. And even heard that some companies have problems getting any RAM chips that are less than 32 megabyte these days, if you don't make custom designs. So 8 to 32 megabyte, I still think there's no questions to ask. If you have 8 megabyte RAM or Flash, you're going to run Linux. There's really no questions. You're not going to bother with bringing up an RTOS to that size and scaling it to the features you're going to need. Now we come into the space that's a little bit of the rambling zone, where you actually can consider running an RTOS or something else. So 4 to 8 megabyte, I don't think you should be running an RTOS, but I hear people want to actually blow up an RTOS on the level that they actually want to do this. I think Linux where we run perfectly fine on 4 to 8 megabyte, no problem. 2 to 4 megabyte now gets really tricky. Can you actually shrink your Linux call down to something where you can actually run this? And you will see in later talks that we actually can. And the really interesting part is when it's 1 to 2 megabyte, because that's the really tricky one where you actually have to think a little bit different, where you might not going to run your standard Linux distro anymore or do whatever you think you were doing in the past by doing a tiny build of Debian or something else. You just want to boot it up. If you go less than a megabyte, I think it's really on none of the Linux business anymore. That's where the RTOSes are going to shine, like Zephyr. But then you have to make the assumption, OK, you might not going to have Wi-Fi support. You might not going to have the dedicated IP stack support. You might not going to have XYZ and so on. So that's something you have to scale up your RTOS a little bit. But the rambling zone is somewhere in between a 1 megabyte and 8 megabyte, where we can have a discussion about this. Once you go over that one, it's Linux. If you don't use Linux, you're wasting your time. So there's another one between Jonathan's comment. It's what I call the Timberts Serial Scale. And he basically has given a couple of talks. And the latest one that I reference is the Stutters of Embedded Linux in October 2015, where he basically looked at, OK, what's the cheapest Android phone you can get? What does it cost you? And was it still running Linux? So the cheapest Android phone he phoned back in the day was $29 in Russia. That's a whole Linux phone that can run Android. So it runs Linux kind of plus a whole Android stack. And then there's also the chip with a $9 board computer. You can also run it on. So I think the cheapest estimated Linux capable of SOC is around about $3 right now. So that gets us pretty close. And we know that the price of silicon is on a steady decline. But fundamentally, his was like, look, we want to put a Linux system into every serial box. And the rule of thumb for doing this one is that it needs to cost less than a dollar. And that's SOC, display, battery, input, RAM, et cetera. And you have to run Linux on this. So that's where we have to go. We have to get a less than a dollar. And that's pretty much his scale I'm going after in this talk is like, look, what do we need to do to get us there? Now to actually getting us there. So when I did this, I was like, look, let's pick a really use case where we actually can do this. So with the IoT space, you have to imagine some sort of devices. And I pretty much picked three corner case that I want to do with the device. The first one, it has to measure and access sensor data. So the symbols would be like a thermometer, or kind of wind speed measurement system, or something really simple that measures something, or even just your weight, or something really simple that is tiny. But it has to measure something that you actually have as an input. The second one is, you want to get this data back out. And most of it is an IoT device. It has to be wireless. You're not going to attach an Ethernet cable to it, because that's not really an IoT device anymore. So it will be Bluetooth, 15.4 Wi-Fi, or even LTE in some cases. But we have to pick a wireless technology to get the data back out. And the third one is, you need to have a battery powered, so it actually has a long lifetime. Because if your IoT device that measures data and sends it out over any wireless technologies keeps running out of battery after a day, they become really useless. So that was my good thing. We're looking at these kind of device types, and we want to run Linux on them. So which device types we would actually going to pick? So I have a really simple categorization for any of these. So we have the really simple ones, which I call sensors. And they're basically using Bluetooth low energy, Bluetooth smart, and they will measure your temperature, measure your heart rate, or something else. That's really simple. And the only thing they're going to do is that they measure, and you can access it over Bluetooth. It's a simple peripheral, and you can get the data back out. They don't have any advanced security or anything else. There's a second category somewhere in between where you actually have what I call an IP node. And it's pretty much, OK, we're going to run IPv6 on this one, and we run it either over Bluetooth or we run it over Threat. And that's pretty much then, OK, we have real internet connectivity. So we'd be really internet of things kind of device. And the third one is you always need an IoT. Well, you need the gateway that will connect this to your larger scale, to your home entertainment system, or the back ends, or somewhere else. I don't know what. And actually, I should have probably put these days LTE and LTEU on these ones as well. But fundamentally, we have the really small ones. We have the IP ones, and we have the gateway ones. So I think for the gateway ones, there's really no questions. Most gateways in the IoT vault that currently run Linux. And they're doing well. And the DSL and cable modem manufacturers have been doing this for years. So putting a little bit of extra IoT juice they need into these boxes is pretty easy and straightforward. The IP nodes, Linux has a really great IPv6 stack. And I have a couple of comments on the Linux IP stack as well. But Linux has a generally good IPv6 stack. We need to probably look into actually just utilizing this instead of having a half-baked IP stack from some RTOS where we actually have to fix half of it. Linux is well-proven because they run this in the data center. Companies like Facebook and Google run fully IPv6 backends. And they know what they're doing with IPv6, so why not leverage that one? So the sensor world is a little bit tricky because historically, Linux hasn't really been used on any Bluetooth-enabled sensors. That's all custom design running on the same MCU. So that's something. Let's see if we can actually do this. What's the size of building, for example, a temperature sensor? And I'm going into this one in this talk and how we going to get there. So I picked a few ingredients to actually do this. I put the logos up there. The bottom ones are a little bit more. I had to pick a logo, it makes sense. Does any of you actually know the top three logos, what they are, which open source projects they are? None? Oh, wow, OK. OK, so the left one is gummy boot. It's a UFI bootloader. The middle one is the Linux kernel. You should at least guess that one. And the right one is the Muscle C library. And I picked this pretty much for a reason because I'm not going to starting out with a G-Lib C, but I wanted a C library that actually was pretty much compliant when it comes to we behave the same way and we proud ourselves on doing this one, while being small. And Muscles seem to be like a good fit. They're more than them, and you can probably use DietLib C or ULib C or UCLib and some others, but I just ended up with picking Muscle. The Linux kernel was kind of obvious, since I want to make Linux smaller. And gummy boot is I needed a UFI bootloader that I actually can work with. And you will get in the later slide to see what I'm doing with this one. The two bottom ones, I said the target was a Bluetooth peripheral, so I thought I'd have to take the Bluetooth stack for a Linux, Bluezy, and make that one work. And I needed a helper library for a little bit of doing the basic stuff like setup, main loops, and so on and so forth, which is L, which is the embedded Linux library which I worked at on Intel, and we open sourced it for about a year and a half ago. So now I'm putting this all together, and now it gets a little bit of the fun starts. So what I'm actually doing, I'm building a PE executable. Everybody knows what a PE executable is? It's a Windows binary. So I'm building a PE executable that I can actually directly put onto an UFI bootloader, and we'll just start it. So to do this is I'm borrowing the gummy boot EFI stub, which you see in the bottom of the diagram, which is either the IA32 or X64 version, depending on if you're on a 32-bit or 64-bit X86. We're stuffing the OS release information into this one. We're using the kernel command line put it into this one. We're building a static kernel image without modules, putting this in there. And we're going to put a built-in in drumFS, and put this in there as well, and we're linking this all together, and we're linking it into a PE executable that you can just literally put on your SD card and just boot it off your system. The in a drumFS consists of a single process. We're actually building the PID1. We're building the in-it process. And the in-it process is actually the Bluetooth stack itself that runs in user space, using Muzzle as a C library and LS an embedded Linux library for different functionality. If you put this all together, you get your boot IA32 on 32-bit platforms, boot X64 on 64-bit platforms as an EFI binary, which I said is a PE executable, and you can then just put this on your UFI system and just boot it. You don't need anything else. On the left side, you have still the case that you actually can sign these, since UFI supports a secure boot. So you can sign this whole image and have also secure boot, getting at this one for free. So to summarize again, the IEFI stub comes from gummyboot. Gummyboot is a little bit of a deprecated project. Most of the stuff with the EFI stubs has moved into SystemD boot, which is part of SystemD, but it's fundamentally the same thing. I just kept gummyboot because it had a nicer logo. The operating system identification is pretty much only that you can actually analyze this and see what's in there, so you can put version numbers in there, et cetera. The kernel command line is important to put in there, because you want to boot this kernel command line, and we actually want to, if you do secure boot, you also want to sign this, not that you actually can circumvent the boot command and disable certain things. The kernel image is obvious, and the user space is custom-built in a drama phase.  And really, any UFI system that you have out in the market can actually boot this. And we've tested this with Galileo and the MinoBoat Macs.  And secure boot is just a nice thing. A neat trick, besides secure boot, is that if you want to do an update, you just put a new EFI binary inside the boot partition, turn the boot loader entries for UFI to the new one with a fallback to the old one. It boots the new one. If something goes wrong, it falls back to the old one, and everything is taken care of, so you don't have to do a large-scale software update process. It's just pretty much, here's the new binary, put it on there, off you go. Looking into PID1, so as I said, we're not going to use any system D or system 5 in it or anything else. We actually have to build our own PID1. We have to build our own init system. And Bluezy, as a user space demon, doesn't have support with, so we actually created a project that is called Bluezy Purple, which is part of Bluezy, that does exactly that. So it becomes a Bluetooth stack and an init system in one binary. And what these things have to do is they have to fundamentally create basic device links, since you need to create a couple of device links in your dev route. You have to mount some file system like dev tempfs and some other ones to actually get the system up and running. You need to initialize and set up the Bluetooth hardware. So that still needs to be done. So either you need the firmware to load or something else. If you want to additionally have some watchdog support, you actually start the watchdog at that point, which is pretty much, OK, we actually open the watchdog support from the kernel and make sure that we're actually going to get restarted, which is pretty important if you build an IoT device that's supposed to be somewhere lying in the ocean for three years. You want to actually have that automatically restart itself, and you don't have to go diving back down there. And then, obviously, at some point you say, OK, I'm exposing the Bluetooth information. Oh, my sensors are up and running. So you can actually connect to me. Muzzle, the basic C library. Works really well for this one. Nice static linking, nice easy to build. And L, we have been using this for main loops, hashes, and queues, which is sometimes needed for some product operation that you need to do with Bluetooth. The interesting part is really on the left side. So that when the system boots up, we don't have a route file system. We basically keep dev route as the kernel gives us, which is the inner drum of S as you build it. So that stays there. We don't pivot route into a new file system. So this system has no idea about route file system. It's read only. But these systems sometimes need to store some information. Like for Bluetooth, for example, if you're going to pair the device, you get a security key that you want to save over reboots or something, or you might have to some configuration that you have to remember, or some states, and you want to store this. The interesting part is, if I actually provide you with persistent storage, so you can use the UVI-VARFS file system to actually access the storage data. And we have built this directly into this demon that you actually can store a couple of information in the UVI-VARFS, and then just works it. So pretty much we're running without a file system there, which is great. It saves us a lot of space. And the UVI persistent storage, we just make that's the problem of the UVI system to provide us. But it works really well, and it is nicely standardized. So we built a proof of concept. So this idea actually works. So the idea was simple Bluetooth sensor. It boots perfectly fine in QAMO. The Bluetooth controller in QAMO is just forwarded from the host. We have nice support in Linux that you can just move the Bluetooth controllers into the host and just appears a UART, and then you can just use it nice. If you're going to build a Bluetooth 64 EFI binary with that one that runs in QAMO, the size is less than a megabyte. And that's still the 64-bit version. I didn't even bother to try in the 32-bit version, mainly because I was lazy as luck. We're going to do this a 64-bit, and if I find time and find the need for it, I'm actually trying 32-bit version. But even the 64-bit version, I was all in the space that I previously reserved for the RTOSes. So we're already less than a megabyte. If you want to put this on a Minibot Max, you need a little bit more, since you actually need PCI and USB support to actually get a USB dongle that is a Bluetooth USB dongle that is on USB attached to the system, so you can actually operate as a full peripheral. Then you end up with 1.3 megabyte in size, and it's still in the range that we were actually targeting. So and this is just a simple Bluetooth peripheral where you can read your temperature sensor and so on and so forth. So as I said, we could do this for 32-bit. I just was too lazy, since I would have to refresh the firmware of the Minibot and make a build system that builds this for 32-bit. So that's something for someone who wants to actually play with this one. Try to make this to build a 32-bit and see what it actually saves compared to a 64-bit build. Now we're getting into the level that I'm going a little bit on details on what we have done and where still working areas are going. But so far, any questions? OK, then let's get a little bit into the nitty gritty details on what we have done and where things are actually funky and what still needs to be done to actually make this all a reality. So the first thing that we had to do, we have to build a user space that actually isn't bloated. And that's you have to really take it. If you're aiming for a size between a megabyte and two, you have to heavily look at this one that you actually build something where your user space doesn't blow your limitations out. You can easily blame the kernel of being too big, but in all of cases, it's a user space. And the first thing we had to do, we had to remove the IPC. Bluezy generally supports debuts and supports a couple of other IPCs to actually work on this one. But that's mainly because you have a multi-user system. In this case, we're running a single process, which is PID-1. So there's really no need of IPC. So we had to restructure pieces in Bluezy. It was luckily relatively easy to do so. But we had to restructure a lot of pieces to just run as a single binary that actually has no input for anybody else. So it runs for a given time. We needed L and Muzzle to actually make the small. Because once you use the G-Lib C or something like that, you end up still with a megabyte or two megabyte just in user space size. Because it's G-Lib C is just large. But Muzzle worked out really well. And we have written L specifically for the case to be used with Bluezy and with Muzzle. The kernel site. So you're going to start out with make tiny config. And there's the wiki link for this one. It has been through the press a couple of times. And it's improving. So you want to actually make a kernel that actually has almost everything disabled that you don't want. And then you manually enable things else. So you can start with an empty .config and then build your kernel. But you still want to find one option that's tinyfication where you actually have the expert menu that allows you to disable even more features. The one thing that will make this really interesting as a future is when you actually be able to use execute in place on x86, then you can share your RAM and flash. Because the kernel sadly still consumes a lot of RAM. Even while itself is small, the RAM that it keeps persistent for itself is still pretty high. So it would be great if we can actually reduce that further. After I made besides the execution place, we haven't looked at this one. The patches have been posted. So there's still some work to do. An interesting part is then if you want to do optimize your linker that links this all together. And there have been kernel support for this one as well. The CTO seems to be really interesting for this kind of support, especially in addition with cold code compression. And I really urge you to see or read Tim Byrd's Advanced Size Optimization for the Linux kernel. It's a really nice read. And he has done a great job on that presentation. So find this and look it up. So when we got this far, we ended up getting a little further. The user space, we built it dedicatedly for this one. So there's really not much more optimization. It's super tiny. It does only the job that it has to do. And everything else is now in the kernel. So the only size optimization we can go further is actually getting something removed from the kernel. So the first one is we don't need any inter-process communication. So any kind of IPC the kernel is providing, we don't need it whatsoever. There's running a single process. We don't need UID and GID support, mainly re-running PID1, which is rude all the time. So we don't actually have to worry about multi-user system setups. And there have been patches in the kernel that allow you to actually disable this now. So that's pretty great. We wouldn't need fork and exec. So these system calls we could disable. I haven't really succeeded in doing this nicely. They have a little bit of tangles, tentacles spread over everything. So we have to see how we're going to go there. And on that line, we really only need a small subset of system calls to actually operate our small PID1. So you can disable certain system calls, but you cannot enable all system calls. You're still carrying a lot of baggage around this. So you would have to actually put a lot of extra config options into the kernel to re-enable the rest of it. The command line parsing is rather complex in the kernel. We actually wouldn't need a lot, since we know exactly what we're doing. We don't really need the full blown VFS, since we don't have a root file system. We're running on the inner ROM-FS and the EFI-RFS. So that's some automation as well. We would not really need a heavy scheduler, since the only thing we're going to have is running PID1 and a couple of kernel threads, so that could be just really simple scheduler. So that could be made an argument to have an alternative scheduler or a breakout that actually makes the simpler and we roof a lot of complexity in that code. Especially when you build a Bluetooth peripheral, we have limited need for networking. That's obviously changes when you want to build an IP node. But even there, you have limited need for the whole networking stack that Linux provides. So you actually want to do a small separation, say I need x, y, z feature. For example, if you only need IPv6, what are you going to do with IPv4 support? What we especially did for this one, because we actually focused on Bluetooth low energy. So there was no Bluetooth classic or Bluetooth high speed support. And if you look at this one, start looking at this one, you will find a lot of other areas you'll say, look, this is not really needed for this one. It's always built in. It's like, why is this actually there in the first place? So let's figure out if you can get some kernel options in there. So how far have we gotten? As I mentioned earlier, single user support went into Linux 4.1. It's a big step. It saves a lot. We were pushing Bluetooth LE only options into the kernel. So previously, we had the tendency to have the Bluetooth stack always be supporting everything. So fundamentally, if you selected it, you get the full blown support. So there were a couple of options in there that we can say, we want to support only low energy. And they even patch this now for a decentralized security manager, you say, I'm a super dumb center, I don't need the security, which means you avoid having like elliptic curve, Diffie Hellman, and some of the AS operations built into the kernel, which saves you quite a bit as well, since the crypto subsystem in the kernel is not as slim as you think because it's built for enterprise. On the other hand, it's great because you would have crypto functionality in one place and don't have to duplicate it in user space. So there are pros and cons in that direction. I think the biggest challenge that we have gotten, and I will get a little bit on the details where we actually have been really challenged here, is that getting new config options in the kernel to actually disable things not as easy as you think. Even when hiding them behind something like config tiny or something else, there's still a lot of resistance to actually adding new kernel options that disable code. And the reason for that was always being the testing complexity that everybody's worried about, that you're going to break something and you put a new config option in there. So that's still work in progress, and some of these ones will make it eventually into the kernel, but I'm not sure if you get all of them there. But hey, if you have found an option that you think might be useful in helping this one, if you toy with this idea, please let me know and please try to get it upstream. So our biggest construction areas. The Bluetooth sensor node is kind of nice. That works pretty well. We have a little bit of cross-termination if you enable networking, since Bluetooth is still a networking stack. So it compiles in some of these net filter stuff while they're only empty stubs, but they're causing a little bit of size. So there's a little bit of cleanup to do with the cross-termination once you enable config net, which Bluetooth depends on. If you have to do an IP node, you want to enable this one anyway, that's all fine. But the biggest problem is, can we actually build an IPv6-only networking stack in Linux? The answer is right now, is no, you can't. And frankly, I don't know if we ever will be able to. So there's a lot of work needed to make Linux IPv6-only. And some enterprise customers are actually wanting this, because they're running IPv6-only backbones. IPv4 packs are completely useless to them. So for them, that would be not a size optimization that would just, OK, we don't have to deal with this one, because it wouldn't accept the packet in the first place and don't even push it through any queues. The reason for IPv6-only mode is so hard, because historically, IPv4 was the one and IPv6 was the one that was the last thing glued on, and a lot of pieces are still interlinked between IPv4 and IPv6, and they're sharing code. So that would be neat, a lot of separation and cleanup to actually make this an IPv6-only Linux. What we have succeeded, while not upstream yet, but we have succeeded a config TCP equals N mode. And the reason for that one is pretty simple, that most of these embedded IoT devices, they will actually not talk TCP, since they don't want to keep any states. They may only take UDP, and then DTLS over UDP actually do their work, and they don't have to keep that many TCP states and sliding windows and so on and so forth. So we managed an option where you have config TCP equals N and that disables TCP, but it will not actually allow for any user-space TCP. It will still handle the TCP packets, but it will just all reject them, which was one of the main requirements from the network guys that, look, we don't want to enable user-space TCP, but if you think you just want to take the code out, just behave it as a dummy that has no idea that there's any port open or anything else. That patch is not upstream yet, but I hope you understand this upstream at some point so we can actually get this included, because I think this is a big size saver. So the other interesting part is what we are looking into is actually kernel-side TLS. And this is an ongoing work item, mainly driven by enterprise folks. And you have seen probably patches for KCM, a kernel connection manager, where you actually can split out a TCP connection into multiple sockets and gets pre-passed in the kernel for you and pre-framed. Something similar will happen for TLS and that you can do TLS in the kernel so you can make use of crypto accelerators. While for the enterprise side, this is purely about speed. While for us, it's about size. We don't want to carry two TLS implementations around. And the kernel already has support for TLS and it needs support for TLS because of secure boot, or RSA in that particular TLS, but RSA. But if you're going to go to the IP notes that I showed earlier, then they would use DTLS, which is fundamentally TLS over UDP. And we need TLS support again. And we don't want to drag a TLS stack into multiple areas or have copies of AS encryption algorithms or any kind of RSA or Diffy Helm. So our goal was to actually put TLS into the kernel, put the certificate authority into the kernel. The kernel has a concept of keyrings, so you can actually have keyrings that you can seal. And hopefully we get this all working at some point where user space process can inherit kernel keyrings. So they would start out with a CA that is backed up by kernel. The interesting part is once you do this, you can actually access TPM or secure enclave type devices where your keys are actually not in the kernel memory, where your keys are actually in hardware. And you can ship devices with pre-built in keys, and they will often go and work perfectly. I think there will be updates on the TLS and DTLS support in the networking or NetDev 1.2 conference in Tokyo at some point in the next month or two. So if you're interested in this one, I will look out for that or find me and talk to me afterwards. An interesting error is when you actually want to go towards, we want to build a tiny peripheral, but we want to use Wi-Fi. And previously, I always thought this is really a weird case. If you want to run a Wi-Fi, then run a bigger system. But I've seen the Wi-Fi requirement for artists is coming up lately quite a bit. And I've seen the Wi-Fi requirement up even for a small Linux system. So we have the problem that the current way of handling Wi-Fi is only with WP-supplicant, because if you need the crypto, and it drags in either GNU-TLS, OpenSSL, it's on crypto libraries. And it has pretty much no management capabilities. So you need to put another process on top of it that actually manages the Wi-Fi network then. So you have something that only can speak the crypto protocols, a little bit of the scanning, but doesn't want to manage anything. Then you need inter-process communication again. You can't really easily link it. So this turned into something, OK, this doesn't look really feasible. So this is not an open source project yet, but we plan to open source it at the Open IoT Summit in Berlin in about a month. I think it's the second week of September. I forgot this right, or maybe October, second week of October, which is probably we have been working inside Intel phone for about a year and a half now. And it's a new attempt to actually do a wireless demon. So we're going to replace W-supplicant while using kernel keyrings and kernel TLS and RSA support for actually WP Enterprise. Since that's one of the funny things, if you actually build an IoT device and it's an Enterprise device, you're not going to talk WP personal for your home network. You're actually going to talk Enterprise network, and you have to deal with certificates to actually get this device securely authenticated to the network. Watch out for this one. The plan is to open source it at the Open IoT Summit in Berlin. So note to the elephant in the room, I'm basing this out of a U of I. And I'm talking about I want to go less than a megabyte, and probably everybody has been silently laughing at me. It's like, what the heck am I doing here? Because U of I is already humongous. And yes, there are huge U of I versions out there, but U of I is based on the U of I standard. And there's nothing stopping you from actually building a small U of I bootloader that just does the basics that you're going to need. So there's a huge amount of opportunities. And when I gave this talk the first time internally, this was still as it was. But as time moved on about a month later, there was an interesting email from Alexander Graf, who came up with like, look, I've actually built the U of I primitives into U boot.  Make this work on U boot. It's like, look, I work for Intel. We do U of I, and I think the problem that you want to solve is not make this work on U boot, you want to make U of I smaller. And Alexander Graf from SUSE actually did.  And which is kind of interesting, since we had the fundamental problem that the Edison devices from Intel were actually using U boot and was really hard for us to boot on this one. But out of the sun we were able to actually boot these devices as well using the PE executable that we were building on this one. The email is public. I think he has sent version three by now, something like February of this year. And as far as I know, this is actually merged upstream and he's continued working on getting support for this one. He had a little bit of caveats that he had no persistent, so no EFI variables so far, but he has plans on making this work within VRAM, so they will happen as well. So this becomes a real alternative. And in this part, he then does this for ARM and ARM64. And ARM also supports EFI as well, so you actually can build this tiny peripheral using a PE executable to actually also run on ARM. It's no problem, it's just I didn't bother testing this on ARM. I tested this on Intel platforms. So what we get with this one? Testing and verification of this one was really interesting since it was super easy. It was super easy to bring this up. It was super easy to test this. And I didn't have to deal with certain hardware problems. OK, does this hardware boot? Does it the hardware doesn't boot? At the end of the day, I was even building extra code pass into the PID one where it allowed me to test all what I was going to do either as a process running directly or either in a namespace or either in QMO. So before this even saw real hardware, I had all the nitty-gritty details figured out. And that's just awesome, so you don't have to deal with a lot of things. Emulating virtual hardware, emulating virtual and real hardware is especially easy when it comes to wireless devices because the Linux stack has so many features in there. So I've seen so many engineers struggle when, oh, I need to make this work on the real hardware. Well, I go, look, can you just run this on your desktop? And once you actually like 90% there, then you put this on real hardware to find the other problems. This made it really easy to get this all working without actually having touch on hardware. So you get 90% done, and then you fix the rest through 10% of bugs that you're going to have. Seriously, KVM and QEMO is the friend. If you're any doing any of this one, my only advice is if you want to play with UFI, build your own BIOS that you strip down from the EDK2 because that saves you so much time. If you know what you're doing with this one, you'll probably spend an hour or two trying to understand EDK2 and how to build it. But just do this once, and then you will have so many things easier because you just only do 10% of EDK2 ready to run any of this one. The interesting part gets really when it comes to the wireless space where you normally would hand off the device in a large-scale certification lab, and they take the device for three or four months. If you can emulate all of this one, you can cover a lot of the ground before this even has to hit a physical hardware test step. That's kind of interesting because, hey, what's the difference between spawning one key, in one instance, compared to spawning 300? Doesn't really make a difference. And the interesting part is really you will easier find a Linux developer than you find an ARTOS developer. I think that's something that everybody needs to consider when they make the arguments. Oh, we need to scale up the ARTOS because the Linux is too large. Or like figure in the case that you have to find your ARTOS developer while Linux developer, you find quite a lot of them. Do we have any questions? So there's another project. So we open-sourced this. We put some of this stuff into Bluezy Stack. You can find it. You can test with it. But there's also another internal project that actually took this idea and allows you to build it. The Soleta project has an API that you can use to build your applications. And it has options to actually build this for an ARTOS, build this for Riot, build this for Zephyr, build this for anything else. But it also has an option to build you an EFI binary similar to the process I just described. And then you can integrate this with any kind of IoT solution it's going to have. And then you can actually test this out by yourself. So I would urge you to actually start looking to this one and then want to toy with this one. If you want to look at any kind of the work that we have done manually to build this, it's all in the Bluezy source code under Bluezy peripheral, which is pretty much a dedicated mode to actually run this as an EFI binary. You still need a little bit of magic linking tools to actually build this all together. If you don't want to look into that great details, Soleta is probably the better approach, because you can just start designing your IoT device and then just build this for a PE binary. And then have fun putting this on a minute board on Edison or Galileo, or any favorite other IoT platform from a different manufacturer. That's all I have. Thank you very much. Any questions? Yes, please. So the question is, I didn't really strip down Linux. So what did I use? So we actually took, we selected the right configuration options for the kernel. We put a couple of patches in to actually put more options in there to scale down the Linux kernel. But we walked away from having the idea that we have the standard Linux way of booting a system. So there's no system 5 in it, and there's no system D. Because that's just killing your budget that you have right there. We're building a custom user space that is really only doing one job and one job only. Say again? Oh, so debugging, the thing is, as I said earlier, the user space that we build, you can run this as a standard process in your desktop Linux. And then you attach GDB to it, and you're good to go. So that's pretty easy. That makes it kind of nice and easy to try.  Any other questions? If not, then thanks, everybody, and have a good day.


Now that we've implemented the software and tested the counter in software, what we're going to do is implement it in hardware. And specifically, we are going to use this board. This is a Digilent Cmod A7 board. The link is down below. Here is the webpage. There are actually two versions. There is a 15T and a 35T. You will actually want to get the 35T simply because it has more gates in it. So there is more room for stuff to put in. It's got two buttons. It's got two single color LEDs and it's got a three color LED. You can program it through USB. So this is quite a nice board. It's also relatively cheap compared to all the other FPGA evaluation boards. So that's kind of nice. The Digilent page has links to the datasheet and to the Schematics, which is quite nice. There is also a page on how to install the IDE that we are going to use. This is actually Xilinx's IDE called Vivado. So there is getting started with Vivado, installing Vivado, and especially installing Vivado board files for Digilent boards because there is some configuration that you do for the board itself to let Vivado know what kind of FPGA is on it. There is also a programming guide which contains some other useful information, especially some configuration information. This is really important when you want to program the board and essentially get the program to stick so that you can remove USB from the board and then just pop it into your breadboard, power it on, and hey, your hardware configuration is downloaded to the FPGA automatically and it just runs. So this is a very important page to know about. I have also created a reference card to the board that we are going to use. You can see that we have the pins down here, along with which FPGA pin those correspond to, and then some extra information about the pins. There is this connector up here called PMOD or modules, or pluggable modules, or plug for modules, or whatever. Digilent sells a whole bunch of modules that you can stick in here. They are not cascadable like you would normally expect with say Arduino boards, but nevertheless, if you want some sort of hardware capability like GPS or Wi-Fi, this is kind of convenient. We are not actually going to do any of that. We are going to connect it to an Arduino to stimulate the hardware. There is also some information about the buttons and LEDs. The other thing is that the Digilent board has 512K by 8 SRAM on board. It is 10 nanosecond SRAM. Here is some information about the address lines, the data lines, and so on. There is a link to the datasheet. Also, the reference card contains all the documentation, which is important. There is a little legend down below and a little warning about how to hook up external power. The basic idea is that you either power the board from USB, or you power it from this external pin, not both at the same time. Because basically there is a diode connecting the two, and if you connect power on both ends, current will flow through the diode, unimpeded by any resistance, which is not so good. There is also a recommended voltage for actually powering the board through the external voltage input anywhere from 3.5 to 5.5 volts. The other thing that I should mention is that the board is strictly 3.3 volts. Now you can configure the pins for other voltage standards less than 3.3 volts, but we are just going to be using 3.3 volts. Do not connect any 5 volt equipment to this board because you will fry it. That is really bad. That is another important consideration since we are going to be stimulating this using an Arduino. You will need to get a 3V3 version of Arduino, such as the Arduino Due, which I highly recommend. Here is the Digi-Key link to it. You can see that it is $89. Again, not that expensive compared to a lot of other FPGA development boards. The version of Vivato that you will download is the Webpack Edition. It is free. The reason that it is free is that it supports only a limited number of chips. There is the RTX 7, the Kinetic 7, and a couple of other boards. Generally, the Kinetics are larger than the RTX, I think, and I hope that the RTX 7 that we are going to use, the 7A35T, is large enough to implement our CPU. That would be awesome. If not, well, we are going to have to go back to the drawing board. But in any case, the Webpack Edition is free. The installation instructions, like I said earlier, are in this page from Digilent, which is really nice. Again, all the links are down below. Let's get started with a project. I am going to start up Vivato. Here it comes. Great, we are going to create a new project. I am going to put it in my Plug1 directory. Now, you might notice that the directory is F colon. Yes, I am using Windows. You can run Vivato on Linux. I haven't tried it, but in any case, my main machine is Windows, so I am just going to run it under Windows. The project name, I am just going to call it Plug1A. This is basically the first revision. Plug1A. I am going to create a project sub-directory because Vivato creates a whole bunch of directories in there, and I want to keep things clean, so the sub-directory is going to be called Plug1A, and then all the other sub-directories that Vivato creates are going under there. Next. This is an RTL project. RTL stands for Register Transfer Language. This is basically the level that we are programming at when we use System Verilog, Register Transfer Language. We do not specify the sources at this time because we will put them into the project a little later. Now, here we go with selecting the parts. Remember earlier, I said that Vivato had some board files, which are important. If you go to boards and you go to vendors, we have digitalandinc.com. Normally, if you download Vivato and you did not download the board files, you would only get AfNet and Xilinx boards, so that is important to do. Digital and Inc. The display name is going to be the CMOD A735T. That is great. There is only one choice. We pick that, and we go to the next, and we go to finish, and it creates our project. Let us just open this up. I am a complete newbie at FPGA programming. There are probably many of you who know how to do this a lot better than I do. I may stumble through this a little bit, but we will see how far we can get. From what I have been able to tell, you go through a flow. The first thing is you add sources. Let us go to add sources, and we want to add design sources. We are going to add files. When we add files, we are going to ... Where are my files?  Plug1. There we go. I have copied this from the Linux machine. This is exactly the same as we had in the earlier video. We are going to copy both of these files. We are going to hit OK. We are not going to copy the sources into the project because why would we have an extra copy lying around? We might edit one copy and then wonder why our edits did not stick. That is it. Finish. Great. If you look under design sources, you will see syntax error files. These are files that contain syntax errors. Why do they contain syntax errors? Well, let us click on one, and you can see that the type says verilog. The type is not verilog. The type is system verilog. That may have something to do with the file extension. I do not know. Once we change all these files to system verilog, you will see that there are no more errors. Great. There are the design sources. The next thing that we do is ... There is an IP integrator. We do not need ... There is an IP catalog. We do not need any of that. IP stands for intellectual property. It is a term that rubs me the wrong way. It basically means a library of hardware. For example, an SD card interface or a memory interface or something like that. Those are all called IPs or IP blocks. I do not like it at all. You see my library over there? That is all my copyrights. Those are my copyright blocks. It is kind of weird. We are not going to do any simulation because we have already done so. We have already run our tests using our verilator test bench. The next thing that we have to do is ... I really do not know why this is not done automatically for you when you select a board. You need to go to Tools and ... It is not here. Great. We need to open the elaborated design. Elaboration is another word for compiling your system verilog files. We are going to open the elaborated design first. Great. This is a footprint of the device. It tells you where all the pins are. We are going to use that in a moment, but not quite yet.  This is the really important part. You go to Edit Device Properties. Under General, this is all mentioned in the Digilent Cmod Programming Guide. Let us see. We go to Enable Bitstream Compression. You set that to True because apparently bitstreams are big. The bitstream is the thing that actually programs the FPGA. In terms of configuration, change the configuration rate to 33 MHz. For configuration modes, this is basically how to ... It tells Vivato what hardware is being used to program the FPGA. The bitstream is typically stored in maybe a flash memory somewhere. Vivato needs to be able to configure the FPGA to tell it how to load itself, basically. We select Master SPI X4. I will just close. I will select that. Hit OK. That was one pretty important thing. Let's take a look at DRC, or actually schematic. This is what Vivato has determined our hardware description describes. You can see that it's fairly straightforward. We've got some inputs and we've got some outputs. We've got a bunch of flip-flops. This is our address register. You can see that we have an adder here. This is a thing that adds one. Then we've got two multiplexers, which are driven from our command. One multiplexer is for the clock enable on these flip-flops, and the other multiplexer is for the data. You can see that the data is multiplexed between adding one, the load address, or zero, or nothing. You can see that the clock enable is multiplexed between ones and zeros, depending on the command. This is basically a schematic representation of what we wrote in System Verilog. Great. If you look down below, and if you don't see this, up here there is a layout selection. You want to select IO planning. IO planning is what you do when you say which inputs and outputs go to which pins. The first thing that we are going to do is, let's see, I'm pretty sure that there was another step that I'm actually missing. Let me go to Tools, Edit Device Properties again. Let me just check something. Startup. No. Yes. Under Configuration Voltage, you also have to specify what is connected to the FPGAs configurations. Configuration bank voltage selection should be set to VCCO. Again, I really don't know what these mean, and I'm pretty sure that there is something in the thousands of pages of data sheets that Xilinx has that explains this. And Configuration Voltage is 3.3 volts. Now we've done it. If we go down here to the IO ports, we can see that there is an IO standard that you can select. The default is apparently low voltage CMOS 1.8 volts. We want to change that. We're going to change everything to low voltage TTL. What's the difference between low voltage CMOS 3.3 volts and low voltage TTL? They're both 3.3 volts, but CMOS is really for driving very low loads in the basically up to about 100 microamp load. Low voltage TTL is for when you want to drive loads that are maybe up to about 20 times higher than that, like maybe 2 milliamps or so. So we're going to select low voltage TTL for pretty much everything. Low voltage TTL, low voltage TTL. And then we've got two scalar ports. The scalar ports are 1-bit vectors essentially, and of course a 1-bit vector is like a scalar. So we're going to select also low voltage TTL on these things. Great. Now we have to decide what pins we want to use. So if we open up address, we're going from 10 down to 0, and we have to select what package pin. For whatever reason, board part pin isn't coming up. I have no idea why. So let's take a look at our handy reference card. Here we have a whole bunch of pins that we can use. It would be nice to put the address lines together. So let's just start from the top, and we'll declare this pin as a 10. The next pin down is a 9 and so on. So we have M3L3A16. Oops, I have just somehow changed the direction. There we go. So again, let me make this smaller, move this over, and now we just start typing it in. M3L3 and so on. And the last pin is J1. Okay, that's pin 11 right here. So we've defined the address lines. Great, I'll close that. We have the load address. It would be kind of nice to have the load address on the exact opposite side. So we'll start with pin 48 here as a 10 and go down to a 0. So we start with a V8 and so on. V5 and U4, great. We also have the command lines, command 1 and command 0. I'm going to, let's see, I'm going to put those on the next two lines. No, actually, I'm going to put them all the way down here, T3 and R3. And I'll explain why in a moment. So T3 and R3. Okay, now we have some scalar ports, we have reset and clock. Now, I'm going to define the clock first, and here's why. If you look over here in the clock column, you can see MR and SR. And down here in the legend, it says multi-region clock capable and single-region clock capable. We want, and we actually need, a multi-region clock capable pin for the clock. The reason is that FPGA is, at least for Xilinx, are divided into regions. And clocks are, or clock pins, are preferably bounded to one region, but there are some clock lines that can go to multiple regions. Now, because we really have no idea how the tool is going to place all of our modules, we're going to pick a multi-region clock. Now, again, this is probably not the right way to do it. From what I can read, clock specification is a whole science, which I don't have. So anyway, I'm going to specify this pin over here, W5, as the clock, W5. And that pretty much explains why I didn't want to put the two command lines on V4 and W5, because then my multi-region clock would not be accessible. Yeah, I could have used these ones down here, I suppose, but anyway. I just want the inputs all on one side. And the reset line we're going to put on V3. V3. Command 1 and Command 2 are already placed and will be unplaced first. No? Did I screw that up? Okay, so this thing I've seen several times already. And I think it must be a bug. There is absolutely no reason why it should say that some other line is already placed and will be unplaced. So first I'm going to unplace these. Okay. So if I say T3, it gives me this buggy error. If I go to name and I change the order of the names, then I select this and say T3. It works. Buggy software. And believe it or not, Xilinx charges thousands of dollars for the fully unlocked version of this. So I have no idea why they actually charge for the software at all, considering that they don't make their money in software. They're a hardware company. They should be charging for the chips, which they are. So anyway, the reset line. Okay, where does the reset line go? It's just below the clock. So at V3. V3. Excellent. Okay, all of our lines should be now allocated. Yes, they are all allocated. So we save the constraints file. If we go to sources, you can see that we now have a constraints file, which basically constrains the design to placing certain pins at certain ports. Great. So let's go to report DRC and see what happens. So DRC is the design rule check. So we want to make sure that we didn't do strange things like connect inputs to inputs or outputs to outputs. So we're just going to run this and no violations found. That's awesome. So once we've done that, we are going to run synthesis. And in the upper right here, we will see that we're running the synthesize design program. And it just keeps going. Great. So we can open the synthesize design. Now what synthesize means is it takes the register transfer design and converts it into gate level designs. So the things like the multiplexers and the clocks get expanded. So if we open this, we can see, maybe if we go to where is synthesis schematic, you can see that this is a lot more complicated. So basically what it's done is it has broken down every single multiplexer and it's even placed buffers in there. It's broken up every flip flop. So yeah, it's a lot more complicated. It's put in the pin buffers, the output buffers, and so on. So this is the gate level version of our circuit. There's another tab here called clock resources, and you can see that some of the pins are labeled SRCC and MRCC. Again, those are single region clock capable pins and multi region clock capable pins. So you can see that we were quite clever in assigning our clock to a multi region clock capable pin. If we didn't do that, then from my experience, a whole bunch of warnings are created. And in fact, what happens is to get from one end of the chip to the other, you will have lots and lots and lots of delays in your clock. Maybe as much as five or even 10 nanoseconds to get from one end of the chip to the other, as the signal has to pass through all these different multiplexers that go between the regions. So anyway, that's quite nice. Now again, we can report any design rule problems. So if we click on design rule and we click OK, no violations, that's awesome. So the next thing is to run the implementation. Now before we do so, there is one thing that we should do, and this is run the constraints wizard in synthesize. The reason that I say this is that when I try this before, if you don't do this, implementation just sort of hangs. The implementation is essentially placing all the gates and placing them on the chip, deciding where to put them on the chip, optimizing the routes and so on. And basically it hung during optimization. I suspect that it's because it knew absolutely nothing about timing constraints. Basically say I want, I think it means I want this signal from here to there to take no more than X nanoseconds. A lot of this, I have no idea what it does. I'm just going to go with what actually worked. So the first thing it says is that the clock has an undefined frequency. And I believe it really needs to know the frequency or the maximum frequency of the clock in order to be able to place the clock lines properly and ensure that to get from one end of the chip to the other, that can happen within the clock period. So we're just going to say 100 megahertz. Now we're not actually going to run our design at 100 megahertz. I think we're probably not going to run our design at any more than about 10 megahertz because any higher than that and you start running into high frequency design problems on boards and especially on breadboards. Now the reason for that is that as your clock frequency gets higher, your edge has to get sharper and sharper. And the sharper your edge, the more high frequency harmonics that edge requires in order to maintain the sharpness of that edge, sometimes up into the gigahertz range for a fairly sharp edge. And there's no way that you're going to run one gigahertz signals on a breadboard. So this is why on breadboards you typically want to limit your frequency to very low frequencies and as a rule of thumb that's maybe 10 megahertz, maybe even below that. But anyway, we're going to specify that we want ideally our circuit to be able to run at 100 megahertz. So if we go through the next things, there are no recommended constraints, there are input constraints here. I have not been able to figure this out. I don't really know why the data signal is referenced to the positive edge because these are inputs. And why would an input happen after the input clock? I did notice that if you change this synchronization to source, then it looks a little more reasonable to me in that if you have input data and an edge, there's usually a setup time and a hold time. And I guess that's what this means. But I have discovered that you can click that off and just continue. And these are output delays. And again, for the same reason I don't really understand this diagram. Click that off. Just go to next, go to next, next, next, next. And those last few pages had to do with multiple clocks, which we don't have. We have a single clock. So finish. And what that did is it added some lines to our constraints file about the clocks. So now I'm just going to save the design, control S. Now I'm going to run the implementation. Synthesis is out of date. Okay, that's good. Probably because we added a constraint, maybe. So okay to launch synthesis first. Yes. Implementation will automatically start when synthesis completes. That's great. So we're going to re-synthesize. So there it goes, synthesizing the design. So that's great. And that's just going to keep going. Okay, synthesis complete. And now it is going to start the implementation. Okay, this is optimizing the design. So it's basically placing things and routing them. And this is the place where it hung before when you didn't actually specify the clock speed. Okay, now it's placing and routing. Okay, I guess optimization wasn't place and route. But this is definitely place and route. And presumably what it does during place and route is it tries to maintain the clock constraints that you set up. Because obviously you can come up with a really inefficient design that takes signals forever to get from one end to the other. Okay, that's it. We could open the implemented design and take a look at what it did. So you can see that this is the FPGA device. There are six regions. One of the region, one of these regions I think is unbonded, which means that there aren't any actual pins that go to it. But there are these other five regions. And if we take a closer look, here are some pins. Take an even closer look, see what it did. Okay, what's that? Is that a pin? That is a pin. That's another pin. So these are the pins and these are the routes. We can even zoom in even further and look at all sorts of crazy stuff. So here you can see that there is a logic block here having to do with the pin and there is a wire and it goes through this logic block and then it goes through this sort of crossbar thing and then it goes off and so on. So, and you can see that we're using, we're going across this region and to this region down into this region. And I think some of these lines here might be the clock lines, typically the multi-region clock lines run down the middle of the chip. And there are some outputs over here. So that's pretty cool. It doesn't really tell me anything. So this is interesting down here, the design timing summary.  The worst pulse width slack is 4.5 nanoseconds. I'm not sure what slack is. I'm going to click on 7.0 nanoseconds and you can see that there is a path that goes from an address register C, that's clock, to an address register that's data. I don't really know what that means. Why would address register a zero clock? Oh yeah, of course, all the clocks are connected together. So this is from clock to data. I guess that would be the delay, total delay 2.9, logic delay 0.9 nanoseconds, net delay 2 nanoseconds. And the requirement is 10 nanoseconds. Ah, okay, so slack means how much of your clock period do you have left after all the path delays? So you can see here that the total delay is about 2.95 or about 50 picoseconds shy of 3 nanoseconds out of the 10 nanoseconds that we've told. Xilinx, how much we require the chip to go from end to end. So that's pretty cool. There's a warning here and basically it says, oh, we didn't specify input and output delays. Those are those clock constraints that I know nothing about. So presumably, I guess, if we put in some clock delays, we can get some better timing here. This says set up and hold. I don't know why there was a clock set up and hold. But yeah, this is a little bit of a mystery to me, but that's okay because we're just going to test it and continue. So now that we have run the implementation, the last thing that we need to do is generate the bitstream. So this is the, in the Arduino world, this would be the flash or the executable that you're going to download. So the first thing we need to do first is go to the bitstream settings. And under bitstream settings, there's minus bin file. You need to check that because what that does is it sets up, there are two formats of file. There's a bit file and a bin file. The bit file is used to just load the FPGA from USB directly. That means that you can load the FPGA and run it. But when you power it off and power it back on again, the FPGA is blank essentially. The bin file is what's used to go into the flash. So the bin file gets programmed over USB into the flash memory of the board and then the board resets and the flash memory will load the FPGA. So this way you can disconnect it from USB, power it off, power it back on again, and the FPGA will immediately load itself from the flash and your hardware will go. So we want to make sure that that is checked. So we hit OK. Then we generate bitstream. So you can see that we're going through a flow where we're adding sources, we're elaborating, we are synthesizing, we are implementing, and we are generating the bitstream. So here we're doing write bitstream. Now if you have any comments about what I might be doing wrong and I'm sure I'm doing plenty wrong, or if you have any comments about what these clock delays actually mean, please log on to the EEVblog.com forum and go to Projects and find my project, which is building a CPU on an FPGA that can play Zork. Join that thread and comment away. I'm not enabling YouTube comments because we all know about YouTube comments. So all right, we have generated our bitstream. So we can view reports or we can do whatever. What we're going to do is we're going to open the hardware manager. Okay, so it says no hardware target is open. That is correct. That's because I have not hooked up my board to USB and I'm going to do that now. So I'm plugging in one end of the USB cable and I'm plugging in the other end of the USB cable. And the first time you do this, there will be a driver that gets installed and it takes a little while, but it will get installed. I've already done that, so I can just hit open target and auto connect. So this will automatically detect the board and you can see that indeed we have detected our XC7A35T FPGA. Now to program it, I can go down here to program device and we select the FPGA and we need to select the correct bitstream file. This is the .bin file, so we find it here and hit OK and hit program. Okay, now that took, that was fast. Ah, okay, sorry, my bad. What that did was it programmed the board over USB. That's not what we wanted to do. We wanted a program flash. So what you have to do instead of program device, you need to add configuration memory device. And we're going to click on that. So again, this is in the programming guide. Yes, now this is in the programming guide for the Cmod chip. Programming the Cmod A7 using JTAG is what we did when we programmed over USB, not directly into flash. This is basically directly into the FPGA. What you want is programming through quad SPI. The quad SPI is the flash chip. So you can see that we're going to add configuration device and you can see that we're going to select micron. And you want to select the N25Q. So we're just going to search for N25Q.  We specifically want the Q32 3.3 volts. Hit OK. Do you want to program the device now? Yes. Now we can specify the configuration file. It's the .bin file. Where is this? It is in PLUG1A. PLUG1A runs .imple and then the .bin file. So we hit OK. And all of this we can leave alone. Basically this is going to erase the configuration first. It's going to program the configuration. Then it's going to verify it. That's pretty good. So we'll just hit OK. And this will take longer because we're writing a lot of data into flash, which is always kind of a slow thing. Great. Programming was successful. Now at this point you would want to reset your board. So all I'm going to do is unplug the USB. I don't think I can actually reset the board here. I can refresh the target. What does that do? Well, it certainly doesn't reset anything. Refresh? Yeah. Okay. I don't think there's any reset here. So what I'm going to do is I'm going to unplug the USB. I suggest that you don't unplug it from the micro USB port on the board because that just adds more fatigue to that connector. You want to disconnect it at the computer. So I did that.  We don't really care.  So when I power up my board, the red, green, and blue LEDs are lit because those LEDs are active low LEDs if you look at the reference guide. So I think by default they would just be on, which is good because it means that the FPGA is alive and doing something. Or is it? Well, in the next video what we're going to do is actually hook this up to an Arduino and write a little Arduino sketch that can send commands to the board and read back the address that it's in. So until then, take care.


Okay, get into the vault. No, get out of the vault. Yeah. You wanna show some more videos? Yeah, I have one little video and then we have some still images. Some of these, we just released the products. It's not exactly top secret, but kind of it was working for our guys at the time. Whatever. So here is- Behind the scenes. Here is us testing our Feather 2040s with the teensy. Okay, Lamar, what is this? This is my RP2040 tester for the RP2040 Feather. We finished this up and I wanna show how the testing procedure works. I use a teensy and the USB host and this is really cool. I actually load the UF2 over mass storage onto here. This is my test jig. So I test all the pin connections. We just milled this on our other mill. If I crank it down here and then press this button, it loads the UF2 file, does a serial test and it finishes testing two and a half seconds. Which is awesome. Over here we've got the Eagle CAD file for the testing jig. You can see I test alternate pins that are not next to each other. And then this is the Pico SDK code that I load onto the Feather to do the GPIO test. This is in raw C, so it's really, really fast. So two and a half seconds is awesome. Yes, go test. One weird thing, so I put this up on YouTube as well. And because I guess Gmail does this feature and then also YouTube does this now, when you're logged in and it's your account for YouTube, it looks at the comments and it has suggestions on what to automatically reply with. And some folks had said, hey, I wanna see the code. And all of the things already filled, if I click one button it says, look in the description. Look in the description, look in the description. Yeah, because I guess it analyzes the notes. That's such a common thing. And then so it had no, I don't know, look in the description. So anyways, if you want to cut for that, it's in the description of the YouTube video or you can look on the blog post for that. Okay, other things we have. Cutie Pie that we were. Oh, the Cutie Pie RP2040. So I actually finished these up and just doing some testing and making sure everything was working. These are ordered, very exciting. TCA4307, this is an older breakout but we're gonna finish that up real soon. And it's a cool I squared T bus buffer. It's a DVI breakout, it's not HDMI, it's DVI. I was gonna use this with the RP2040 stuff to get DVI video out to displays that have DVI in. And sadly the STMPE811 is no longer being manufactured but we're looking at other SPI or I squared C. We just have touch controllers, check out our great search for a couple of weeks ago. I think this is the TSC20407. I don't know, I don't know the number. And then more experiments in NeoPixel keys. So what if we took four of those like Neo keys, put them side by side and then we have like a seesaw to connect them to I squared C so we can make like I squared C plug and play stuff. I don't know, cool. And then Rotary Trinkie, next door neighbor to Neo Trinkie. They're best friends, they used to play together as kids and now Rotary Trinkie is coming to life. Get back to the ball. That's in the ball. Okay.


There, without any further ado, I want to bring up one of the newest Red Hat-Hers and one of our very favorite ones as well, Clayton Coleman, the lead architect for OpenShift, and Brandon Phillips who has joined us from CoreOS and is now a Red Hatter, so I'm going to let them give you the State of the Union. All right. Good morning, everyone.  I don't know if you guys have heard that in the last couple of months. So I'm Clayton Coleman. I'm Brandon Phillips. So when we talk about State of the Union, normally this talk is somewhat about what's happened in the last six months or the years since we talked about it, but I figured when we talk about Union, we're talking about CoreOS and Red Hat coming together, what are the things that matter to the people in the room, give a real high level overview. To me, the idea for the Red Hat CoreOS merger was really about bringing two people, two groups of people who cared very deeply about the technology in Kubernetes and the open source ecosystem, bringing them together and being able to build something greater out of it. I think when this went through, there was kind of this idea that we were going to bring the two smartest sets of people in Kubernetes together. I don't know if that's actually the case, but I'd like to think so. So before we get into the technical weeds, I wanted to just give a little context on where CoreOS came from and really how we thought about the space and our overall mission as a company and as a set of products. So there's 3.5 billion internet users today and there's about any point, this is a rough estimate, 29 million practitioners of IT and development in the world. And the simple fact is we're horribly outnumbered. The number of folks that are using the internet, actually the number of folks coming on to the internet versus the people who are experts in how the whole thing works. And if you think that we can just educate our way out of it or if we can expand the number of jobs and catch up, that's simply not the case. There's a quarter billion of people being out of the internet every year. These are just staggering numbers.  And so at CoreOS, with these kind of staggering numbers, we thought about, well, how do we make sure that the roughly 100 million servers that exist worldwide, how do we ensure that those things are well managed? How do we ensure that the people who are in charge of managing them have the tools to do the best possible job by security, by the customers, ensuring that there's uptime for those three and a half billion people. And what we came up with was we needed to change the way that enterprise software ends up being delivered into people's data centers and onto people's public clouds. Kind of the culmination of what we've been doing over the last few years is this idea of operators. We'll talk about it in the context of both the platform, the OpenShift and Kubernetes platform, but then also in the context of applications on top. But really what it comes down to is we want to enable this idea that no matter where you end up running your application across the hybrid cloud, we want to make sure that the experience is the same, the application infrastructure around your app is the same, and that you're able to manage it in a way that removes a lot of the complexity and enables you to simply use necessary infrastructure to make your application and your team successful. So that's a little bit about what the Coros mission was about. We'll get into some of the weeds now. And on the flip side on the OpenShift mission, for those of you who've been with OpenShift for a very long time, we started out as a platform as a service. Platform as a service is about making things easier, deploying software specifically. There have been lots of attempts at platform as a service and lots of different platforms as a service. Everyone, different requirements, different use cases come together to mean that people build things slightly differently. For us, the opportunity with Kubernetes and with OpenShift starting just three years ago was we want to find the easiest way to ship the most software. Iteration has to be easy. We want to reduce that difference between development and production, make deployment repeatable and automatable. And Kubernetes and containers and the ideas of building powerful tools that can be composed together to build applications just enough of the standard patterns that we all used, but still allow things on top to have their own complexity and flexibility. We've seen that in the Kubernetes ecosystem. We've seen even in the last, I think, three months, there have been something like 15 or 20 different projects for building images or for making it easier for people to iterate locally and then push to a Kubernetes server. This idea of empowering developers while still keeping that under operational control and having the tools that we as operations teams need to understand what our users are doing to keep the applications running, to standardize, and to provide a common security base. All of those pieces, I think, have been really borne out maybe three years ago when we stood up and announced OpenShift V3GA. There were some doubters, I would guess for most of the people in this room that there's not as many doubters as there were then. There's thousands of OpenShift clusters worldwide, we've done nine releases, ecosystem is exploding. And so that idea, making it easier to get ideas from, to go from ideas to production as quickly as possible, this is where that complement with what the CoreOS mission really came to was. If one side of the story is about keeping control of the world with automation because it's just going to grow out of our hands, and on the other side it's about empowering developers to be as efficient as possible, but still to do that in a form that's under control. What's the next step? Sure, so essentially our shared mission, and I think what we'll be talking about a lot over the next year or so as we bring products together is automated operations. And the way that we think about, the way that we thought about automated operations was essentially this. Cloud is essentially two parts. The traditional hosting business where you rent some servers from somebody. And then the other part, which I think is the interesting property that has caused kind of the explosion around cloud usage is, well, I make an API call and now the service is available to me. I make an API call and that service is now updated. And overall we call that automated operations. And that's kind of the shift in enterprise software that I think we share as an interesting change, particularly as you start to think about hybrid cloud over the next year or two. So there's a lot of things that could be automated and you were somewhat vague today because there'll be a bunch of big announcements coming up tomorrow and at the summit sessions that I will have a list at the end for everybody to go see. So we're going to stay really high level. Just tease you a little bit and want to get everybody to go into the sessions. We don't want to steal anybody's thunder. But the things we want to automate are applications in the platform. Whether that's the operating system in Kubernetes, the install and update mechanisms, the infrastructure. Again, all of these things exist to serve us, the developers, and us, the operations teams that keep all of this working. On the application side, development and test, being able to deploy anytime around the world. And not just one type of software, not just these really limited applications, but every kind of software. And keeping those up to date, because again, a ship software. And once it's out in the world, my job's not done. Because that problem that Brandon spoke about is, that's another point of vulnerability to my organization. If I don't know where it is, where it's at and can deliver fixes to it over time. All right, so what I wanted to show here is the automation that we ended up building into CoroS Tectonic. Which was the name of the product that we had built as part of CoroS Kubernetes product. Seems like, there we go. All right, so this is the CoroS Tectonic dashboard. When you first log into it, we have a few things that are useful and interesting. When you first get in, we have high level metrics. One of the things that we wanted to do right off the bat was to ensure that the system gave you monitoring information. So whether the system is up to date, whether the system is healthy, etc. So that's the first thing you land at the login screen with. But one of the unique properties in going to this whole concept of automated operations that we had was that CoroS Tectonic had kind of a one click update experience. So if you're familiar with the experience on your iPhone, your Android device, where you click a button and then your device updates over time. CoroS Tectonic had a very similar experience. Now, this cluster has automated updates already running. So the last upgrade that we released a few weeks ago has already been applied to this cluster. But it's really an interesting experience for somebody who's spent their career in enterprise software where this cluster has been running for nearly about nine months now. And I get to see as somebody who's been involved in the product, I get to see things go from mocks on our UX team and our design teams laptops all the way through to actually being delivered on my cluster every time I hit the button to update to the latest version. And this is kind of a large change for how software is delivered. But I think an important one if we're thinking about getting software that runs across hybrid cloud environments. And the way that we actually make this all happen is pretty interesting, a little recursive, and it's 9 AM in the morning, so it may hurt your heads a little bit. But just bear with me. What we actually do is we run the entire platform from monitoring to the Kubernetes components like the scheduler and API server on top of Kubernetes itself. All right. So what you see in here is this is the Kubernetes scheduler. So the thing in charge of scheduling the applications across the cluster. And they're all just running as pods on top of Kubernetes. Which gives us a bunch of really, really interesting automated properties. One of which is that the exact same software that we use to monitor your application and alert on your application health is the exact same software that we use to monitor the health of the cluster and alert on the cluster's health. And so we get all of these properties automatically for free. Like what is the CPU and RAM and memory usage of the scheduler, for example? Or if I need to write a piece of software to update the cluster, I can use the Kubernetes API to do that. So that's how we've built automated operations in Tectonic. It's all built against the Kubernetes API. And this is kind of a sneak preview of some of the ideas that will be flowing into OpenShift and complimenting some of the features of OpenShift over time. So, and I think that to add to that compliment while we're going back to the presentations, since the beginning, the focus of Red Hat has been making Kubernetes the best place to run applications bar none. And to build on top of that a layer to make it easy to deploy and run containerized applications. And as we worked with Brandon and the rest of the CoreOS team, it was really obvious that we had picked complimentary approaches, automating the platform by using those pieces. Every bit of an investment that Red Hat has made over the last, almost four years now, four and a half years now, has been about making Kubernetes the best place to run those applications. And it turns out that a platform that's really good at running applications is also really good at running that platform. And so each of these points has really made me really excited about what's gonna be coming this year and next. Because every one of these builds on the strengths that we already have to deliver for every end user and reinforces those. So that every time we make a user's life a little bit better, who's building applications on top, we make administrating the platform just that little bit easier. And then the last thing I wanna touch on was that this isn't just administration of the system, the Kubernetes system. We put automated operations and the namesake of the company is this operating system we built called CoreOS. We put these automated operations all the way through the platform down to the individual host in the cluster. So this is just a looping demo of where when a machine gets an update, we actually coordinate that update through the Kubernetes API as well. So you get this full system view all the way down to the hardware, the virtual machine of whether the system's up to date, what the state of the update is for each individual host in the system and actually control the rollout of that. The big difference between automated operations and these updates experiences between your phone and the server environment is obviously we have to keep the server environment running. And so it's slightly inconvenient when your phone is down for like three minutes, it's probably like the worst feeling in the world at this point in 2018, you're like, I don't know where I need to go. I don't know, I'm not in touch with anyone. But on the server side, we're able to control these things so that only one machine or a handful of machines in the cluster are down at a given time while the updates are happening. Really taking advantage of the distributed nature of Kubernetes. All right, this is your baby. So the other thing that we've been thinking about is over the last five years, we've been really focused on how to deliver a platform that has this concept of automation. But then the next bit is, well, how do we make it easier for open source projects, for ISPs, for really anyone to deliver software on top of Kubernetes? For the sole reason of supporting applications.  And those pieces of software today are at least on the top 95% side of the market. They're really easy to deploy from a cloud provider via an API call. But we wanted to think about what about the long tail, all the other systems that exist. And so last week at KubeCon, we introduced this concept called the operator framework. And operators of this term that we have for cloud native applications, where applications again like caches or databases or complex stateful applications or load balancers or machine learning systems can be wrapped up into a Kubernetes API and on demand deployed to support an application by anyone who has access to a Kubernetes cluster. And so over time at CoreOS, we really want to see more and more of these operators come into existence. And at Red Hat, we kind of have the horsepower to ensure that that happens. So this is an open source project to ensure delivery of not just a platform filled with automated operations, but supporting services and applications on top of that platform. And it's really about extensibility. So if you're going to build a platform, it's really critical that platform can actually support the use cases on top of it. So from the very beginning, Kubernetes has had this idea that it was a small core of a distributed operating system for the cloud. And each of the pieces that you build on top of it would complement other pieces. So if you bring in a new service, for instance, to let you, I don't know, get access to vault at scale. And to allow people to deploy their individual monitoring applications that not just the people who are consuming it benefit from the effort that gets put into it, but that everybody can benefit equally. And so the pieces that we've built into Kubernetes over the last year or two features that folks may be familiar with, like custom resources and API aggregation, and making it easier and easier to extend Kubernetes. This is the next step in that story because we're building not just the tools that you need to extend, but the concepts and putting them together in a framework that makes it easy and approachable. Yeah, so to give a practical example. So two years ago, we introduced a thing called the Prometheus operator, which makes it really easy to deploy instances of a monitoring service on top of Kubernetes. And what we'd hoped was that application developers, if getting monitoring for the application was just an API call away, that they would more properly monitor their application. As a proof point to this, Ticketmaster, which is a customer, ended up making this Prometheus operator available to all their users across all their clusters, all their Kubernetes clusters. Ended up with about 400 copies of Prometheus supporting their applications, making it possible for the application engineers to set up alerting, set up monitoring, and support themselves with the services they need to make sure that their application had good uptime. And this is just an evolution of the story that we've believed for a really long time. We're building a platform that makes it easy to take ideas, turn them into reality, and to keep them running. Automation is a key part of that, having capabilities available. So I drew this really quickly and just threw a bunch of buzzwords on there. Whatever the buzzword of the last six months is, all of these ideas, all the pieces that make up an application, those will change. And people will evolve and will build new tools and new ways of taking advantage. Serverless is up here because that's the hot topic today, is how can I make these really, really simple applications run easily and at scale. And I think for us, when we talk about things you'll hear this week and over the next year for us, is that we think of these as just points along a spectrum. The underlying idea, making it easy to build an application, to deliver it anywhere in the world rapidly, to keep it up to date, and to keep the platform that it runs on secure. At the heart of it is automated operations, and we will continue to build upon that for a long time to come. So we alluded to some of the sessions, Brandon and I are speaking in these a number of PM. These are the top four if you're interested in some of the deeper details on the ideas that we talked about here. The CoreOS and Red Hat session on Tuesday, the Kubernetes and the platform of the future talk that Brandon and I are giving with Steve Watt. 330 on Tuesday will kind of go into some of the deeper details of the trends here, Container Linux and Red Hat Enterprise Linux, the road ahead. We'll talk about what the story is for the operating system pieces that Brandon alluded to, and the OpenShift roadmap on Wednesday is always a perennial favorite. So we're kind of at the end of the prepared stuff, and Joe is gonna come and join us. All right, guys. So hi, everybody. I'm Joe Fernandes, I run the product management team. Myself and Mike Barrett will be running around with mics. So if we have time for questions after each session, just raise your hand. I do have a couple of questions to help clarify things for the audience. So, Brandon, you showed the tectonic console, and I know one of the things we've been doing is bringing that together with the existing OpenShift console. You want to talk about how those things come together and what users they're targeted for in terms of? Sure, so the tectonic console, really the focus of that was people who are administrating Kubernetes objects. And so that meant people who were caring about deployments, were caring about application monitoring, were caring about the cluster settings. We went in identity upgrades, role-based access control for a cluster level. And so that's been kind of the focus of that, which to go to this idea that a lot of these ideas are complementary. The OpenShift console for since the beginning has been focused very much on the application engineer and the application developer's concerns. And so these two consoles kind of serve two different purposes. And I think you'll see some, our story is always going to be, we're trying to make sure that if you fit a role on the platform, that there's an experience that works really well for you. And so I think, as we'll talk about in some of the sessions coming up, those two users where they blend, you'll actually be able to see those pieces coming together. And where there's value in focusing on one of those, we'll have a really strong experience around that particular user, like a cluster admin. Yeah, and at the OpenShift roadmap session, Mike Barrett's saying that you'll be able to actually see a demo of how these things are already coming together. So we have actually mocks where the consoles are now converged. It'll be tied to your role. So if you log into OpenShift as a user, as a developer, say you'll see the OpenShift console, the service catalog, and be able to act as a user. If you log in as an administrator, you'll get to see the user's view, but then you'll get to see all of the administration views that Brandon showed. So state of your cluster, state of the services and stuff, the things that a cluster admin should have access to. And so it should be seamless just based on how you log in. The other question I get a lot is about operator framework and service operators. Last year, we were spending a lot of time talking about the service catalog and service brokers. You want to talk about how those two concepts, the Kubernetes service catalog and then service operators, how those things come together.  And in a sense, the operator idea when we make these concepts Kubernetes native is they're no different than any other tool that a developer might have. And so I think you'll see that, again, it's more about use case and who you are and how you consume software than necessarily the deep details of where the technology is. Yeah, a way of thinking about an operator is imagine that you had a public cloud provider that allowed you to inject new services. So you were able to say, well, this cloud provider doesn't provide a Cassandra service. So I'm going to go find somebody who's an expert in Cassandra and plug in the Cassandra service onto this public cloud. This doesn't exist, except for in the case of Kubernetes and this concept of hybrid cloud. And so what we're trying to do with operators is make it so that you can find a component, an operator, injected into your OpenShift cluster, and then make available whatever new service your application developers need, whether it be through Service Broker or something else. Make that available to them, an API call away. And that's kind of the analogy that we're going towards with operators. Yeah, and to make one more plug, tomorrow afternoon, there'll be a Red Hat keynote that also features some of our partners like Microsoft and IBM. We'll actually be doing a demo of the operator framework with one of our ISV partners, which is CouchBase. And kind of to echo what these guys said, the service catalog is about end users consuming those services. And for those of folks who have used it, that there's provision and bind operations to make that seamless, it also deals with on-platform services as well as off-platform services. So last year, we did a demonstration with Amazon. And then at AWS Reinvent, we launched an Amazon service catalog where you could bind and provision and bind to Amazon services from OpenShift itself. But what you'll see tomorrow in the keynote is answering this question of when you're running a platform service on the platform, who operates it for you? The developer who wants to consume CouchBase is not the DBA who needs to run it. And at the end of the day, we need to be able to automate the operations of the services or of the apps, not just automate the operations of the platform. So please try to attend that keynote. And you'll see, I think, a pretty cool demo of how this all comes together. Sue, how are we doing on time? Well, you're done. All right. Well, thanks. And didn't you see the good job of getting you all to ask questions? Right. All right. Thank you all for coming. We'll see you. Thanks.