Welcome everybody Hi, so yeah, thanks for the nice introduction Yeah, so today we're gonna talk about PVFD, which is a project. I started to make it easier to run BCC which is a powerful tracing tools toolset on an embedded remote target I'm gonna start off with a demo and then I'm going to show I'm gonna build up towards BCC talking about Linux tracing in general and then how that builds up to BCC and what BPFD is and Hopefully the demos will make it clear how how I'm using this stuff So a little bit about me first I work in the Android kernel team at Google We ship all the kernels on all the Android devices out there and my two areas are Linux scheduler which is the piece of code in the column that placing tasks on different CPUs and Load balancing the task between CPUs and scaling the CPU frequency and stuff like that. The other thing I work on is tracing Which is frays and perf and all these powerful tracing Technologies available and in the Linux kernel Yeah, and BCC is the is the latest thing that I mean so Yeah, so I'll start off with a demo of a tool that lets you look at all the File reads and writes that are happening across the system. Just show you how powerful this thing is how powerful BCC is And because we're in the kernel we can't miss anything Anything that's you know involving the kernel itself across all Processes we will see that in the kernel I'll just show you an example tool file top So just showing my setup. I have Embedded target high-key 964 Which is which has the high silicon 960 SOC? and it's running Android and It's connected. It's connected over Wi-Fi to my laptop And so everything I'm going to be doing is over the Wi-Fi network I'm going to like be tracing the Linux kernel remotely over Wi-Fi So we had to before I run five top. I had to set up some environment variables Where my kernel sources are how I'm communicating with the target what the target architecture is And so I run five top now you can see there's some some warnings that happen when the tool runs I want to go into that now, but right off the top you can see the Wi-Fi service in Android is Continuously reading from I fight that format to get information about the Wi-Fi interface So you can see that you see this you have this global view of what's happening on on the Android system And so I'm going to like open an app called the contacts the contacts app You can see it did a lot of reads and you know, it's reading the contact database there And I'm going to create a new contact just like how you would create a contact in a regular Android device and I just created a contact and you can see that There is you know, this entry here that shows that the contacts database was written into 53 rights in total So it's kind of nice to be able to do this over Wi-Fi To get that kind of observability you can see that There's not a lot of data transfer happening because All the data collection is happening in kernel As these like Thousands of events are firing the data has been collected in kernel aggregated there and periodically sent to user space in this case over Wi-Fi, so So all this builds upon a lot of technology. That's not visible. I just ran files out But there's so much going on under the hood that you do not need to know that's the beauty of this So probably speaking there are these six different Signals that you have in the kernel that you can monitor This is not related to BCC or BPPPF or anything just tracing in general. These are the signals you have the static trace points first of all, which are Points in the kernel that are inserted Place points that are inserted at compile time So at compile time you you can like have a static trace point that emits some information if it's when it's enabled So they call static because done at compile time These are the S trace events that you have and then you also have dynamic trace points which are which are Trace point probes you can insert at runtime That you don't know in advance That's how file top works actually it Instruments VFS read and VFS write in the in the VFS layer Dynamically so these are not these trace points are not decided ahead of time. That's what I call dynamic That's the K probes infrastructure In the kernel.  Let's you do that for user space applications You probes is dynamic Again, you don't need to do it ahead of time. Then you have something called USDT which is the static Trace points for user space and then last to our first event So you have your performance counters. Those are also signals that you can attach hooks on to like cycles and cache misses and stuff Finally you have the sampling signal Which which lets you run some profiling code or some code Periodically a certain number of times a second. That's how perf profiling works You have an interrupt that fires 100 or 500 times a second and you can run some code that can observe the system So these so these are the various signals that we can build on top of to get to get the observability that we need So now let's come to BPS. So BPS lets you Write programs easily that can run when any of those signals are triggered Any of this any of the signals happen? the way it works is you have a BPS program that's compiled to BPS by code and That's loaded and verified by the kernel and so once it's loaded it's sitting in the kernel ready to run and Then you can attach that program to any of those signals. I just mentioned in this slide and program runs in kernel collect some data That the signal might have emitted or you might just want to read some kernel data structure or something like that to get that information the information that you're trying to observe and And Finally the BPS program in the kernel can output that in different ways using Maps there are masses a Data structure that lets you aggregate information as you get it like so for example histograms is a kind of a map And there are various other ways to to image information from the BPS program So BPS is a technology that lets you Let's you do that So now let's come to BCC. So what is BCC? So BCC actually puts all of all of that together. So it lets you set up all of these signals I was talking about the dynamic Signals especially activates them and all of that and it also does all of this stuff takes the program that That does the observability like files off this in the demo I just showed you and builds a BPS program out of that and Finally it attaches it does all that automatically for you attach it to the signals that are required by the tracing tool and It also has a set off Libraries written in Python and stuff that can read those read the maps and present the information to the user periodically BCC kind of it makes it easy the beauty of it is it does all this powerful stuff at the at the Behind the scenes that you don't even know like when I ran file top You didn't see any of that happening. Just had to say file top and it was showing you You know all this information there are like a hundred plus tools that you don't even know need to know how they work And you can just run them That is opposed to running tools like first or something like that where the running those commands might be more complicated and Interpreting the information so BCC does all the powerful stuff for you, but Really is very easy for easy to run So yeah, feel free to interrupt if you have any questions so far, please Okay so traditionally BCC has been Has has been typically typically run on the same device where you have your kernel sources and your clang stack and all of that And the kernel is running on the same machine where you're running the tracing tool and That would have been fine But unfortunately you also need kernel sources to be able to run the tools as I showed you you needed to set an environment variable To run file top that pointed to the kernel sources so all that is That doesn't work very well for cross development kind of Development model where your target is remote remote and your your development tools are on a different machine So That was so that so that makes it very difficult to run these powerful tools So what I did is I separated out the whole stack Into two into two parts one is the stuff that doesn't need to run on the on the Device you're tracing and then the stuff that needs to run So the stuff that needs to run on the target is just the stuff that that does a low-level Things that are not compiler related like Loading a BPS program once the bytecode is available or creating You know attaching the BPS program that was loaded to a trace point for example these things are low-level and are not like Not related to compiling the tool or Compiling the program that Produces the BPS bytecode and stuff like that. So, you know, that's and the way I got that to to work is I I started this project BP FD which Kind of encapsulates all the low-level things that are needed that BCC needs and have that have BP FD run on the target so in the in the demo, I just showed you BP FD is running on the arm target and BCC is running on my laptop Just to summarize why we want to do this You know, we don't want to sync kernel sources to our remote target We don't want to download them onto the target. We might not have space for that Or the sources might get out of sync with the target all these things. It's really unnecessary With this kind of a model you don't need to cook across compile Python Clang All of that stuff it can run on your host quite easy quite a very well run any host and This fits the cross-development flow where as I was saying you build on your host and run on your target You don't do it the other way and also your host machines are more powerful They have more processing power typically than battery power devices So it makes sense to do the compilation of the tool on your host And also many embedded targets don't have symbols in their executables things like that On the host you you have a bigger chance of having that information and the tools need that information to Need the symbol information in many cases to produce meaningful results So that's why these are all the reasons why we want to do it Any questions so far? Not I'll move on to the demos and show you some more demos Okay, so I Guess I'll just move on to the demos then So this is a tool, so I just showed you five top the other tools like hard irqs dot py Which summarizes the amount of time That I spent in interrupt handlers Over the period of time that he specified in this case ten seconds every ten seconds It's going to show me the total time interrupt interrupt handlers spend on CPU and That's useful because You might run into real-time issues or latencies and stuff like that if interrupts are spending too much time You need to as a designer you need to make sure that your interests are not taking up too much time These are all the interrupts that Spend time in the last ten seconds The second column is in microseconds So I'm just going to do some UI activity and show you that the GPU interrupts are now going to take a lot of time You can see that Mali here just like 141 microseconds milliseconds Ten seconds, so that's hard irq dot py just to show you the tool itself What it looks like? Tool itself has an embedded seep it has a C program embedded in it that is compiled into BPS Should yeah, no problem. So where this tool works is it uses K probes to instrument Handle IRQ events per CPU function in the kernel And it attaches the count on the Function and then when that runs it There are two modes actually you can either count or you can time I think the default is time so For the for the mode in which it times let me show you how that works because that's what I demo so for that trace start is is executed whenever handle IRQ per CPU runs and and it makes note of the starting timestamp and Trace completion Runs whenever handle IRQ per CPU finishes and It you it uses the the time it stored in the start takes a difference between that and Adds that to the to the histogram So some other tools that I wanted to show you This one I won't be demoing, but it's useful to see How much time was spent doing block IO operations that summarizes? Shows you a summary of every every so often of all the processes that we're doing block Ios how much time They took to run here by your top and biosnope there in the slide the Difference between biotop and biosnope is that biosnope shows you an entry per it shows you it doesn't refresh every every so often it It shows you like a tracing type of output It doesn't clear the screen but by top clears the screen and both these tools show you the the latencies that the bio operations took So this can give you an idea about whether the disk I was slow or not for example So another tool is cash that Which is useful to monitor what's happening in the page cache in the kernel whether you're having page cache hits or misses Stuff like that if you have too many misses in the page cache that can indicate that You know that is what is called causing a performance issue. I'll just show you a demo of cash that so I'm going to remote into my target board over adb And I'm going to run a while loop that reads a hundred MB file So there's a hundred MB file I'm gonna I'm gonna write a loop that that reads the file and then in another window I'm going to run cash tax every two seconds you can see in the third column here a cash status showing me that all the Cash accesses were hit Because the first time the hundred MB file was What's ready? It was cashed into the page cash Then I'm going to I'm just going to drop all pages in the page cash And so the next occurrence of cash that you can see that there were a ton of misses That's because everything the cat the page cast has had to be three prop populated with the file So the fourth and fifth column are not working because it's reading Files that are on my laptop instead of the remote target to populate them But the first four columns are valid again to run this tool all that you know, it was very trivial I just have to run the tool. I didn't have to specify any complex arguments or anything like that It's very easy to use if you want to do more complex stuff and and don't want to write your own program There's there's the set of tools called the multi-tools. It's one of my favorites the truth trace multi-tools and The multi-tool Automatically will generate a ebpf program for you and compile it with bcc based on the arguments you give it You can do a lot of powerful things with it related to k-probes and stuff like that in this This slide here. I want to see all the open The file opens that are happening on the system so this function in the kernel called do this open of which receives arguments from user space to open files and this you this uses k-probes to instrument do this open behind the scenes So I'm telling the trace multi-tool. I want to instrument do this open and print the Second argument that is faster do this open second argument is the file name I can just do that It's already showing me. I think that's the the while loop that I wrote in the other window that's continuously running So I'll stop that and so if I do some activity in Android It shows me all the opens that are happening surface finger and all that Yeah, some some files are being opened a lot of times, you know, it might be better to open them just once And that might give you an idea of maybe something is unnecessarily done And this tool will tell you that yeah, you're doing this and it's not necessary It's very useful for that kind of performance debugging And I wouldn't be running a demo of this so but still called run queue length Which uses the first sampling mode and every time per sampling Event occurs the BPS program runs and it looks at the run queues on all the CPUs and as information to histogram about what The run queue length like how many times a certain run queue length occur The histogram has that information in the end This will give you an idea about whether there are too many tasks running on the system causing performance issues Sometimes you might have so you can either have too many tasks running on system or you might have too few CPUs Like if you have too many tasks in the run queue, then it's you know The run queue is overloaded and you might have performance issues. So The show this will let you let you look at get that information So many things are not working right now, but they are very close to working I call these the boring issues because there's a clear path to getting them work But just need some work to be done so that involves user space dynamic and static tracing symbol lookups new tools tools that need to Read stuff on the remote target, but they read stuff on the on the On the local machine That is my laptop in this case There's a full set of issues. I've created in the issue list I would really welcome any contributions there Maybe have some more interesting issues That is not clear how to solve like One of the issues is that BPFT itself the reads and writes it does can cause or the other system cause it makes can cause a lot of Tracing activity itself. So sometimes that interferes with the tracing to a lot for something. Maybe we can blacklist the BPFT process from being created somehow The some tools can generate a lot of output. So We have to Reduce the amount of data that is being sent somehow maybe Batch the data together stuff like that Perf polling is another issue that we're looking at and also Logging is not working. So there are there are a couple of interesting issues that it's not clear How do how we solve but we have to solve them So that's all I had, you know This project is completely open source. So you can go to GitHub and download it and contribute to it I wrote an LW an article that goes in a much more detail about how it works and A Brandon Greg has set up nice resources on how to run these tools So I would encourage you to Go check it out. I guess we have a lot of time So we can just do a Q&A and any discussion So usually You usually as an embedded developer you have the kernel sources available to you because you build them on your host right So you just find BCC to that using the work that I did now it's possible for you to set an environment variable pointing your kernel sources in that and BCC will look at the kernel sources on your host not on the target on the host and use that to build your BCC program the box symbols are not needed in the general case To run a lot of these tools. They're needed to interpret the output sometimes like there are tools that Do profiling of stacks and stuff like that and so then the output will just have Have instruction pointers if you don't have symbols So for things like that you need Symbols on the host But otherwise you don't really need symbols Something Yes, so the first question About the other ways you can use this the way it's been written the architecture right now It's very plug-and-play kind of thing. So for Android. I have an adb module that does Android specific stuff But PPFT itself is written with a standard IO interface kind of thing So every command is like one line and you send that to be you can you can run it locally and just give it a command String and it'll respond to it. So to load a program. We give it the bytecode in basic base 64 encoding so I somebody else was telling they wanted to use it on servers and Over SSH so that's you know, that's definitely possible to add very easy to add. I would say The other question was failure mode. So right now it's very early stage. So we haven't really thought about that So any kind of cleanup? Will have to be done you know for Android what happens is the adb connection dies the Processes that was started remotely will will also be killed automatically So and then all the resources will be freed that way but yeah But clean up will have to be thought about this There's a lot of those kind of low hanging fruit issues that have to be worked on So yeah, feel free to pull the code on github and ping me if you want to work on this stuff and contribute Yeah, yeah, right now. It's just a single device but Potentially that could be done. Yeah, I don't see It's just a matter of somebody taking it and applying it to their use case. I don't see a reason why that cannot be done Testing Testing one two Mike check test test. Hello. Hello, everyone. Can you hear me? Can everyone in the back hear me? Okay Hello everyone, I'm Robert call. I'm the developer of the livery CMC project I was the co-founder of the livery wrt project. It's the project that essentially came before this livery CMC It was a fully free And very guinea-lack distribution had some different goals I'm also I've worked for a few nonprofits that worked for providing free livery education resources one of those nonprofits worked to make a Place for students to share their notes when they're done from class with classes and share them under a free Creative Commons license I'm in my free time. I do advocate for the usage of free software both in my personal and in my professional career I also break stuff for other people Sometimes for the better or worse So on before we get to talking about livery CMC we need to talk about livery wrt. This is pretty much where it all started Five of us at livery planet 2010 got together and wanted to create free embedded distribution a fully free embedded distribution for The focus is around about PDAs and music players. So long. There's this Fully free PDA device called the banana. Has anyone ever heard of this? Does it ring a bell? Awesome so It was essentially probably I would say it was one of the freest Computers that we've had to date at this point Considering that all the schematics are free the CPU was free It didn't need any microcode or initialization blobs and the like So we set out to with livery wrt to create a fully free distribution hope with the hopes of Getting the banana know our wife by the fsf the fsf's our wife program hadn't existed as of yet But there were internal talks about starting such a program and we wanted to banana note to Meet all those goals. So we need a fully free operating system that didn't have any non-free blobs or any Non-free packages if you will Sadly, I Ran that I was limited on time. So I kind of had to walk away from the project in 2011 I also had some design Issues that I wanted to fix but a lot of the people using livery wrt Really didn't want to Didn't feel that was really necessary. They wanted to keep it mostly a clean open to VRT With the Lenox livery kernel and we've all most of the packages in So what's livery CMCA it stands for the livery Concurrent machine cluster the name at this point. I'll get you in a minute. I doesn't really reflect what it does now, but It is a fork of open to VRT then we now open the VRT again And is anyone familiar with the open to VRT the for the forking situation? So I believe it was back in 2015 Seven of the open to VRT developers weren't really happy with the way that the open to VRT project was being managed So many of these main contributors went off and forked it into the lead project They forked it with the intent of making it a more community focused endeavor they wanted to Make anyone who's a contributor have voting rights and they wanted to focus on certain things one of the notable things that they did want to focus on was Focusing on the IPv6 stack in open to VRT and then about Was it three months ago four months ago? they finally Re-merged into open to VRT. It's essentially they took lead and Through the open to VRT branding on it and just now it's open to VRT LibreCMC uses the Linux Libre kernel. Has anyone heard of the Linux Libre kernel? So the idea behind the links Libre kernel and that effort is to remove all the non-free blobs and components from the Linux kernel So we use that instead because we don't want to advertise non-free software and we Don't we don't use want to use non-free drivers because part of the FSS Requirements for being a GNU free mean the GNU free software distribution guidelines We remove targets which have a hard dependency on blobs these targets can be Might require a non-free driver mean the source code for the driver is available We're dependent on the OEM to what the chips that manufacture to build the driver for it or it requires a loadable firmware blob which again the community doesn't have the sources for it or they're under a restrictive license So we remove those but we do leave in some targets which do might have a Soft dependency on these blobs, but still would function as intended or might function in a limited fashion So on some of our targets With the 802 of an AC stuff, we don't really have any drivers however the first generation of AC routers shipped with a removable mini PCIe or a Double mini PCIe type card, which could be replaced. So those are where we make the exception We don't officially support announce support for those devices, but we do leave it in So if people actually do want to use liver CMC with with those devices, they actually can We also don't include support for devices which require initialization blobs. So The rabbit's very pie is one example that people are familiar with however there has been Work progress being made in creating a fully free initialization blob for Free There isn't work Okay, so the Raspberry Pi requires a non free initialization blob There's been progress made in making a free software initialization blob as a drop in replacement So the liver liver CMC's original delivery CMC project as itself started in October of 2012 its original goal was not to target routers or some of these devices, but I wanted to create a essentially a dropping component that would allow for distributed compilation of Because at the time I really didn't have means to go by expensive hardware for building library WRT I Needed to use the hardware that I had just lying around and this was an endeavor to do that It failed because I just lacked the time not to mention dependency issues and other complications I'm there's a lot of other pack of stuff there's a lot of drop-in replacements that already do this, but I wanted something that was a little bit leaner and Then don't fell off the face of the earth at this point So the delivery CMC name doesn't really reflect what its initial goal was I was approached in 2013 by an OEM to They wanted a fully free embedded distribution because they wanted to Be part of the SSF RYF program. So We I sent you I couldn't think of a better name. So we're CMC stuck and we created a Maintained version maintain embedded distribution Liberty WRT had kind of Been erratic with Releases sometimes happening every six months sometimes a year. So we need something that was a little bit more up-to-date So the Liberty CMC project right now does target routers and NAS devices at this point in time We release every three months Six to three months and then usually will of course have some security updates depending on the severity of the security update So things that allow us to allow someone outside of the device at running stock Be able to get route or access to the router that would be really bad. So we always patch those Usually within 24 to 48 hours of being made aware of that Right now. We also we want to try to start targeting Divide having a 10-year lifespan for the distribution for long-term LPS releases and possibly longer the problem that we have with the embedded space right now is the fact that we While we have some we don't really have long-term support in terms of being able to build it 10 years from now Which it can be a huge problem So um Why would we so why do all of this work right now? We have open to VRT We have a whole bunch of other projects and we have a lot of stuff in this open space But some of it doesn't quite fit what we're trying to achieve And then I don't want So How many people in here? I'm sure a lot of people in here are working the embedded space right in some shape or form Right raise your hand if you do actually okay A lot of times many of these embedded devices they run non-free software or they're closed in some fashion They restrict users Freedoms in some ways or they don't they're not compliant with the sources so Many of the solutions that exist for providing a open source of free Distributions for these devices some of them require Provide blobs or non-free packages or packages which are encumbered by patent issues or The light that makes them non-free Many of these devices can tend to be black boxes, which is a huge problem Has anyone recently bought a car within the last 42 or three years Okay, how many of them included those infotainment systems Okay, how many of them have stopped working because of a forced update so as um So if it has anyone heard recently is anywhere familiar with fiat Chrysler and their you connect Consoles so recently they just force pushed a an update which causes it their Center console unit to reboot every 40 seconds So we get this with no mechanism to revert back Which of course if you're going to be have work with embedded devices and you're going to be doing updates You might want to have some sort of mechanism for reverting back So a lot of these people are going to have to hit their vehicles in and they're non-free So anyone in the community can't go and actually Fix these problems The other problem we have in the embedded world is forced obsolescence Who here has smart light bulbs? anyone How many have you have used that Philips hue light bulbs so With their first generation light bulbs they forced an update to brick these devices so they just stopped working This is a huge problem. It's not just this Just smart light bulbs a lot of these smart devices Where companies can see well, they're not going to be paying more. So they're not good They're just going to make it dead which I guess some that is like one way to go about your security model If you just don't want people to use it anymore once it goes in stops working You don't have to worry about updates anymore, but the problem is that The user in a lot of cases doesn't have control over this and a lot of these devices are running free software But you're kept from that. They don't distribute. They don't comply with the licenses And or they they do a lot of design They design it in such a way that it makes it really really difficult for you to go and run your own software on these devices Another mechanism is secure boot I spun arm for instance, so some of these devices have Signed firmware and they have signature checking which the keys are few is in the CPU die at the factory So you can't you yourself can't actually go and change these keys Without a lot of work and a lot of times. It's actually just impossible sometimes Manufacturers love to cut serial lines Or make them really really difficult to break out. So if you wanted to at all start playing around with it, you can't A lot of times if you change the software on the device The vendor might be say, okay, you can put whatever software you want on it. However You might get they might When you unlock it, they might make the bootloader show a scary screen saying this device may cause death if it's unlocked essentially usually says that they're not responsible for injuries as caused as a result of Changing the software on these devices that for a lot of people that probably would be scary They go they go pretty far with that one This isn't the case if does anyone have Motorola phones? Has anyone seen the scary message? Yeah, I've been wanting to change that I just haven't had the time to actually go forward with doing that and then some devices they'll go and They'll just brick those if it detects that soft unauthorized software is being used. They might actually brick it this was made famous with Release of the first Nintendo 3ds. I don't know. I don't get their naming scheme at this point But it was the first and 10 of the 3ds system that Nintendo released back in 2010 The SSF did a campaign about this and on the box of the intent of 3ds the first time it says that If you running off or I software on this device may result in it not being non-functional so there are many options in in this space in terms of being freer than What normally would ship on a lot of devices that these projects talk about many of these projects listed are great They they just are not great for some of the focus of some individuals Open WRT it does have some non-free packages Some non-free drivers the driver situation has gotten better. There are more free software drivers But some of these chipsets require loadable from our blobs and some of these can be really small like maybe a few dozen kilobytes or whatever Or it might be that they Sometimes it's a half a meg of stuff that has to be loaded to some of these chipsets They also include some patent encumbered packages which have a gray area, but And those can cause some issues and some Devices have blobs have restrictions where you can't reverse engineer these blobs. It violates the license that the firmware blob is under Also, they sometimes have restrictions on how they can be distributed. So does anyone here with anyone here running? any like Nvidia cards back in the Late the early the early the early to mid 2000s to late 2000s Does anyone remember the restriction on Nvidia's? Distribution of their firmware blob their portable blobs So so it's not just graphics cards, but there are a lot of devices which require these types of blobs Video Has anyone here heard of the DDWRT? I know it's very very popular And it does provide lots of functionality However, it is a little bit restrictive Some features on certain platforms do require activation. I told can fully sympathize with them It's their business model But some of the sources that they provide don't always build or Sometimes they build with some tinkering if you know which path to go down or Just not at all And I can understand sometimes it doesn't because it requires maybe a specific distribution which this is Really a huge problem, especially if you're wanting to build it to Five or ten years down the road. Sometimes it's not really practical to do that sometimes though You might need it because maybe it supported a particular platform But newer versions of what available is not there So you might want to look down and see what exactly they did to get support working Open them bedding in Yachto. I Thought they're really really great platforms. I mean they do distribute the vanilla likes kernel which FFF does have a problem with but The other thing is blow it in my view it's For what it provides. It's a great tool and it makes it really easy to build for embedded platforms It's just a little bit too blooded for our focus The bunch of cores available is anyone using it in anything right now It's essentially they're targeting for Embedded in internet of thing devices and sometimes some networking appliances This has changed a little bit. I probably shouldn't have that but a bunch of cores a little bit bigger than our target platform at this point We definitely our target devices are Half of our devices that we support we only have four megaflash and 16 to 32 max of RAM so That's a huge problem and a lot of Conautical and a bunch who have been working to reduce some of these footprint issues, but um This is the way they're going. I for what they for what a bunch of core provides it is definitely a great way to To ensure that you can easily roll back images. So there's like a There's like a last time I left there is a one gigabyte fallback for each system running on it So you essentially have two installs one to fall back if in case something breaks No, what no, you don't you don't want this Wait Wait, no one's speaking in here, right? Because I mean Okay, well Did I come in okay, I'm still speaking I just Yep, okay Okay testing, okay, I think I think we're good again. I mean um, so no so um again Within the last seven months. I have not followed it went to core. I will admit that they are getting better in terms of flow Okay, well Okay, I think I'll continue now So they are making they are making they are trying to work on Okay, I don't know Testing testing Every time Wouldn't it be easier to Okay, I don't know So I don't know if they want anyone changing that because I Think Really really high Okay Yeah, they're gonna change the frequency it's not gonna I Don't have the handheld one Testing am I good to go? Okay, awesome Okay, so So again, my information on a bunch of core might be out of date again, they they are doing things to rain in the bloat however We want to make we want to make our core a lot smaller than what their vote with their goals I mean we sort of I'll talk about that later There are those also Embedded Debian, but again, it's a little bit too large to be running on routers And if you're looking for package management, well the kind of all bets are off at that point considering again How most of our targets are only have four megs of flash and might only have 16 to 32 megs of RAM And we definitely don't want to make those targets obsolete. We should definitely be able to still Provide a router distribution that will fit in that space given and with the minimal amount of features that people are looking for So again, package management comes at a cost. It does add just a little bit of overhead in terms of resources and a lot of these embedded devices, it's Not really packed practical to have full package management support But we are trying to we this is one of the areas that we do want to work work on as the Liberty CMC project Many people Complain that manufacturers don't update their platforms often enough, but the real problem is a lot of these manufacturers Don't really focus much on really focused on their security model you should be able to have these devices working for long periods of time and Prepare to be able to mitigate for some security issues that might pop up Properly configure device definitely goes a really long way in terms of preventing a lot of Holes that might pop up Not everything needs to be connected should need to be connected to the internet I mean how many people in here can help to own a smart fridge or smart device or smart light bulbs. I Mean, it's a convenience thing, but I'm not I personally don't want all my light bulbs I mean, I can understand having a separate network Completely isolated, but a lot of these people are connecting them directly to the internet in some way of fashion and so it Pretty easy to you know, turn on someone's light or possibly do something even more dangerous. I Mean, there's no toaster ovens that now run Android. Has anyone seen one of these? I Mean there is definitely a market these things definitely go a long way And they do make our lives more convenient, but we should definitely think about what networks that they're being connected to So also a lot of these devices should all fail gracefully have Has anyone seen message board or signs that have a Windows blue screen of death or a kernel panic? Yep, and a lot of times That you really shouldn't be happening It's just showing Information sometimes it's because the device overheats or sometimes it's because some resources aren't available or there's some other weird bug A lot of these devices really need if they're being used in critical pieces sign. I would say as a critical device, but They should definitely fall back gracefully. They shouldn't have to They should be the manufacturer should prepare them device in the event that some component of it will fail and have a plan for dealing with that So these are all the things that I've talked about at this point that are wrong with the embedded space we would I like to address some of these issues with using Libre CMC as a platform for moving that work forward so I'm going to talk more about Libre CMC, but I get a lot of people saying well Libre CMC doesn't support X Y and Z so I'm not going to use it. Okay. Well, Libre CMC doesn't support Everything in the Sun given our constraints and the fact that we want to be a fully free software distribution We can't support everything under the Sun because of that. We can't support some non free project packages or device drivers and There are people out there that that really that do want care about that and want that focus So that's what the Libre CMC project is here to do We want We also want to focus more on security and bloat as well There are more platforms that we would love to target, but we need to get our base down to in the region ballpark of two megs of flash and eight megs of using only eight megs of RAM This may not be practical today given what people really want in their devices But we're certainly going to try and it might involve a lot of work We're not competing with projects. We are standing on our own right for these reasons that I've laid out So if it works great if it doesn't well Maybe it might be we might be able to make it work in the future and maybe a manufacturer might see oh there might be some benefit in releasing the firmware blobs for Particular chips that they do that. That's great. We can then support that device, but at this point in time we can't so right now the Libre CMC project is essentially two projects one is Focused at consumers and people who actually want to use this in on their routers and other devices and the other is research which of course is unstable and Some of it doesn't see the light some of it does but Our consumer-facing project helps fund the research side of the project so We work with a few different OEMs We encourage them to make a small donation if they're going to ship with Libre CMC. We encourage them to make a small donation for the hosting space possibly of custom images and source code distribution disks and the preparation of those so one of the What we push OEMs to do that are shipping our devices devices with Libre CMC is We want them to provide a source code disk with the device and it must build on a recent GNULEx distribution We work with them to prepare these disks to make sure that they build all the sources are available in the respective image and That's pretty much it and that's a huge thing because a lot of These devices running GNULEx Or Linux the Linux kernel They don't provide source code disks and half the time They you might be able to write them and they're you're lucky if they're going to respond and sometimes if they do respond They might provide you a disk, but the what you get isn't always the source code that they distributed and a lot of kit companies just feel oh they can just They don't have to worry about it because the litigation aspect of a lot of the licenses is pretty well the enforcement is Constantly and there's not no enough people doing enforcement in that area Also, it's a nice thing to give your customers the source code disk that actually builds an image that can be flashed on it Considering that they can then expand the capabilities of the device later whether it be Now when the device releases or maybe five years from now when the when the device is on a site is on the second-hand market Why these many people don't realize that not everyone has the resources to go buy a brand new router every two years or something ridiculous like that And in the case of Liberty MC sometimes some research Some stuff that we do doesn't always pan out. We just move on it's expensive work And it does take a lot of effort So that's a great question. Thanks for bringing that up. So a lot of work has gone into Open to VRT and lead and open to VRT again for the idea of reproducible builds. So Some aspects of Liberty MC are some aspects and packages are reproducible. So not so much some kernel stuff A lot of the kernel stuff is reproducible On some platforms, but not all platforms lead I Think it might have gone on the back burner, but I'm not sure what the progress. They've made lately But again, we've received see since we're basing it on Leeds stable 1701 We do benefit from some of those The stuff that we does Did I answer your question? Oh No So we're gonna get to that in a second it's um I didn't okay Hold the questions for a minute as we're still gonna move along here. I guess There's it. So Where's it? I think wait So a lot of our focus again in terms of research is into to blow Our what our build system right now. It's currently using we're pretty much just a fork of lead Lengthly recurrent or removed. So we're using essentially modified version of build route To put it simply We'd look to look into We want what we would like to focus on is to have a build system that can build on systems for the 10-year span and the reason that's difficult is because of Dependency issues so a newer version of certain components Might break compatibility down the road. So that's a huge problem Especially in terms of build systems for embedded devices Build route does a really great job at this Reducing the breakage, but it's it's still quite It still does it probably once every two to four years so if I take a version of OpenWRT or any arbitrary version of OpenWRT or LiveryCMC and I want to build it Four years from now. There's a large chance that some of the things will break. The other problem is Some packages might drop off the face of the earth So if you don't actually have a copy of that package then it might fail if the blink is dead So we want to reduce all those things anytime. We have a dependency. We always make sure all those All those dependencies are available for each and every version of LiveryCMC We started doing this early and we plan on keeping all those packages For long spans of time as long as the LiveryCMC project is around It's something that we need to address I guess in the free software world where Projects don't Projects some projects disappear. Sometimes they're hosting changes. Does anyone remember Getorius at all? Anyone anyone you actually use Getorius? Anyone remember the breakage some of the break. Oh wait, Getorius and Google code Both of those were depreciated and there's a lot of projects that were dependent on those platforms for hosting and when those went away a Lot of the links broke Google code they did keep some of links on for quite a while I think they're getting about to the point in time where they're going to start Making them completely dead But that's a huge change and a lot of time if you have this old version of the project you then have to go find Where they're hosting the code now and they might not even have that old version of the package And this is an issue that we need to address And that's what I would like to address with the LiveryCMC not just by us hosting some of those older packages but also Coming up with ideas to better manage these dependencies 15 years long extended long-term support is probably going to be sound ridiculous It probably is ridiculous in terms of effort needed to actually make that happen It requires probably giving up a lot of Shaving off a lot of different dependencies of a build route and a lot of the tools that meet packages depend upon for building but If it might be worth it it might not be that's probably going to be up to Me and the community to decide on that it may not be a great option So I mean right now we're targeting routers and controllers and PDAs are out of fashion at this point. Well, they're kind of out of fashion in 2010 But I still like the idea of having a small Organizer like device that I could control all the software on and communication applications We would like to Get LiveryCMC Some of the extended packages. We'd like to get support ready for some federated and self-hosted services like on cloud or and Like the new social or pump. I owe is anyone you are diaspora. Does anyone use any of those? Okay, so we would like to make it so that we have a nice wing core that you can install some of or we Actually package up some of these things and make them available to people to use because right now a lot of these services are really really difficult to install and deploy especially for people who have no background in GNUX or The like so we'd like to meet be able to make it so you can just download this image and Maybe answer a few questions and it would configure it for you But this is a long way around down the road I know there's other projects that do focus on this like anyone heard the freedom box project So that but they're targeting raspberry pies and they're using embedded Debian But we we kind of don't want to go down that road So our first Area, I think So our first area is research design We want to not just limit ourselves to the Linux the vanilla X kernel There are other Linux variations Linux kernel variations that we could possibly use You see Has anyone heard of UC Linux? So those do run on some even more limited devices Again, I'd in some Area divide areas. I don't think that this is still the right route to go I could be wrong on that but um any other paradigm is going to be really expensive in terms of research There's a lot of cool projects going on and we'd like to probably take some of those pieces and work run with them if resources allow for it We already talked about dependencies So the build system again, we do we use Essentially, it's build route. It's my modified by open of air T And that's pretty much what we're sticking with everything is based around the fact that we use build route So yeah in terms of bloat though one of the recommendations from One of the Routers is the However, it doesn't really fit well if you want Flash plus have a full Sadly right now we're going to removing the package management and While I Now a lot of people might not have an IPv4 connection provide to them right there ISP, so they have to use So again Liberty seems he's not for everyone, but if you care about software freedom and pushing that idea forward in the embedded space That's what it's that's what we're here for we want to make long-term support much easier and guaranteed You should be able to take the sources that we provide you now and build them Hopefully five and ten years from now if you follow the dependencies As long as we make sure that the dependencies are cut down in such a way that they'll be guaranteed to be supported Down that far down the road That also helps in terms of compliance. So a lot of times Has anyone actually got requested sources and then they give you build instructions that are very big And at the end it says that you need for Dora course three or something like that. Has anyone run into that? Yeah, so A lot of times you might not be able to find that version of Fedora course three. Luckily we have the internet archive But years ago there have been versions that I just you couldn't find anywhere. They weren't uploaded yet They weren't the right person. You didn't get a hold of the right person. So You're just out of luck you can't you couldn't build it and I've actually have run into that recently because Version of you, but I couldn't get to build on any modern System and I need to I couldn't I was hoping it would build on this particular version of the door if I couldn't find it so We are open to incorporating any new features as long as they are free software and they don't add too much Blow for additional dependencies that are required. I guess I'm going to leave it up to Q&A Anyone has any questions? I That's not we given the the routers that we build for it's not entirely That could be done in a lot in a better fashion On our AR 71 xx targets. I don't think that's the case. There are I mean everything else is a module But that adds more bloat to the to these images More static linking could possibly be done, but we can we could probably modify it slightly so that we could do that That would probably save a lot of space It's just a matter of time constraints. So just maintaining what we have now Does take some some effort in terms of the rebranding Making sure links we read kernel work sometimes some patches their conflicts. So we have to do other patch those out or rewrite certain patches to actually make the thing built so Hopefully down the road we can approach blow from that perspective, but I don't We could be doing that better. Yes, that's a great point. So any other questions? I have a comment and a question Okay So the thing is that no matter how small you make it somebody gonna come with a smaller device and Where that goes is that remember you mentioned Yuclunux or Yeah, I think that the point behind that is it doesn't require an MMU So my cork on something like an M-series Arm machine They're really really small things typically around M-series arms Do you have Do you require an MMU to run? Sadly at this point. Yes, we do require that. I don't think I don't know if there are there any I don't think there are any platforms that that don't require one that's open to beer cheese, of course I could be wrong on that one. We Could possibly do that it just be a prior matter of using the Yuclunux kernel, but I haven't built it in quite a while So I don't know what I like Is the real problem The problem with this is Is time because it's all this stuff does take time to like play with and actually see how well that would pan out Right. Um, I mean The thing is is we are getting more and more these resources in this space. The problem though is those are not guaranteed Has anyone looked at RAM prices lately? I don't know They've recently in the last year or so or two years. They've gone through the roof So those types of fluctuations can happen and what the manufacturer decides to use Might change and what the manufacturer has the option to use so Many projects don't really see this as We need to shave off every single light ten kilobits here ten kilobytes here. It's Yeah kilobytes, sorry But some people don't see the justification it but it can be a heck of real profound effects when someone's going to try to build some of these devices Is there any more questions? Yes, that so that does come wait so Can I address that or do you have more? So um, yes, I do realize that that is a concern in this space A lot of these blogs are getting pretty huge and if anyone's familiar with the it's in the last year There was a huge security vulnerability with some broad con chipset that Were Revolved around some of these blogs and we as a community couldn't fix them. We'd have to wait on Broadcom to actually go and push up a new push out a new update I understand that we're not going to be liver seems he's not going to be popular with a lot of people for that particular reason We I didn't start this project because to take easy way out That a lot some of us you care about software free. I want to be able to See what that hard what what the software on our device our embedded devices is doing and be able to fix issues that come up So Okay, so we do support some other targets and not officially though for some reasons we do have some We do support some of the all-winner boards that use the a20 So see The eagle bone The lack is not officially supported. I'm If anyone can answer this they I'm still having issues with the license around one of the the blogs that are initialization blocks Not really a blob in the first is available for it, but under some Texas Instruments license anyone to clarify that? Okay, cool Yes, but what about I thought there was a there's a component that you have to build drop in to Get it to run on the beagle bone black Mainline you would does have some initialization blocks sometimes around memory initialization and some other CPUs set up stuff. So we should Is also the other people are Right Right, so I'm not an expert in this we're not gonna initial We're gonna for some things if it looks good or clean like if the art we can build a male image with it using only free software Then we'll support it But we only want to announce support for things where we're guaranteed that all parts of the stack are free software Yeah, we're still in question I have the Okay, thank you. Thank you for supporting the project that that really does help Run run what exactly So our bottleneck right now we could and the problem is that again wife 802 open AC chipsets There's no one right now that we know of that is releasing Versus for the loadable firmware and there's a lot of things that we're actually restricted from doing because of that as well In terms of making mesh networking that's slightly more difficult And in terms of initial bugs Not at 10k had quite a few bugs initially so our bottleneck right now is mostly Blobs in that case, so that's the reason why we're not supporting any slightly newer targets We also have been looking at possibly Getting a more modular router out the door that we might be able to support faster Have faster CPUs and things so we can do Some cooler things and actually have some resources on some of these devices, but as of right now I Mean if it's free software will be more than glad to support it So yes, in some cases There isn't there's not really so it only applies to five gigahertz devices and the approach That TP-Link made was not really the right way to go But that's kind of what we feared initially that companies would do because they Wanted because that's if you required that you couldn't make the Wi-Fi chipset operate and a An approved way so TP-Link went throughout of just walking down all the firmware You pretty much had to TFTP flash it you couldn't flash it from the web UI for a lot, oh, and they went to further and cut the Serial line, so you have to solder to solder for solder pads on the board to get a serial interface Because of those so the serial is there they just cut the line and put to solder to pads there four pads there, so Yeah, oh Okay, so So there is someone else who was Did a lot of work around that whole thing What's his name Eric what yeah Eric Schultz? He's given a few different talks which are online. I believe he talked actually last year about this very same thing It hasn't affected yet But it could definitely affect us At some point actually I would like to add one thing some so some vendors have actually gone the route of having Doing signature checking on the particular loadable firmware to get around that but I mean The only real way to do it with the F9k stuff is to just lock down the whole firmware People from doing that It's not really it's not really. We're really You would just go that route and they they tried really really push that What we said wouldn't happen when TP like actually went and did the exact same thing that we said would happen But can you Speak up a little bit louder like, you know So the other thing is on our targets do we do Try to focus on targets that allow you to flash an image through the web UI And we made the exception for some TP link routers where you have to use TFTP But you did not actually have to open the router unless you need to debug something that had failed But they start locking it down So in TP link, it's not it's kind of probably wrong to actually say it's a full lock down it's more or less a Restriction on the web the web UI is that you would load the firmware to and so they made it so you could still TFTP flash it you just had to name it to a certain strength So what you do is run your TFTP server and see what file name it requested and then rename the firmware you're trying to flash it to to that name and then it would Magically flash it it didn't do any signature checking on it Yeah But it's still more difficult for regular users When you try to get regular people to use some of this stuff, it's it's challenge especially in terms of the steps The hopes that you have to go through Is there any more comments or questions? Okay, I believe we're We've got two minutes Well, thanks, I'm glad that I was able to make it and my flight was We luckily were able to bump our flight up otherwise would have missed this because of Thank you everyone Testing testing just call my pocket. Oh shut the screen off Okay, yeah, and if I want to toss the presentation over there All right Conference Being Thank you, so yeah, I had a slide about this actually So my talk is called Linux I scored C and me and I'll take care of it and me part first He did go through it I'm an electrical engineer by schooling But I have a lot of science experience as well I do pretty circuit board design and some of these boards up here Are boards that I helped on or I actually all three of the ones that are shown there are boards myself and He said that I worked on the pocket Beagle I actually worked on the pocket bone, which was the predecessor to the pocket Beagle But it was kind of the I do a lot of the Linux board support Colonel you boot I do a little bit of real-time stuff and Some user space but mostly low-level I've worked on several community initiatives including the big board org GSOC 96 boards the EALE training that's going on next door and key cat Training is coming up in Reno next month. So anybody's interested I'll have some information about that on social media Okay So we're first we're going to talk about What I squared C is Now we're going to run through some example devices Talk a little bit about the protocol and we're going to skip to the I squared C subsystem Give a little history there some links back to different resources Talk a little bit about each type of driver. Well, not all of them. I'll mention them But there's a there's a few other ones other pieces, but I figure there might not be enough time to go through it all Then I'm going to talk about instantiating devices and there's quite a few questions about by tree in the last talk over in the other side of the wall So I'll go over a little bit of how to instantiate devices using different interfaces Talk a little bit about the user space tools Character device a generic character device for I squared C and then I'll run little short demo and Pocket beagle with Baking bits cape so I scored C Stands for inter integrated circuit. So it's kind of a mouthful But it was built in 1982 by Phillips So it's been around for a long time Synchronous multi master multi slave interface Not typical, but it's it's possible because of the type of signaling it uses Half duplex meaning it can't communicate in both directions Simultaneously, that's where it differs from SPI He is open open drain signaling so If two devices are trying to be masters at the same time it won't cause electrical issues It'll just follow the transaction The arbitration to determine which masters which but for the majority of cases you're a single master multiple slaves You only have two wires for signaling Sda and Sdl they are Usually a hundred killers signaling speed not super fast There is a link down there for Wikipedia She goes on and on about the protocol. I really recommend taking a look at that. They did a really good job on that article But here's some of the highlights so it's a seven bit Bus addressing protocol At least originally and the original back also called out a hundred killers and then Progressively got faster and faster and added different features so version one added foreign fast mode with 10-bit addressing which is part of the protocol and Then they kept boosting it up The really fast ultra fast is actually unidirectional being that it is push-pull instead of open drain that means it'll drive both low and high states and and so it It's just one way and then Intel came up with this thing called SM bus which is a subset of I squared C which is used on motherboards particularly for monitoring devices temperature band control stuff like that they constrained the protocol electrically and they added some optional software adjusting address resolution so I squirt see you it doesn't really have a way of Broadcasting back What's on the bus so that that's kind of the limitation of the I squirt C in SM bus Optionally takes care of that So here's some example devices that would use it I squirt C We got the real-time clocks a problems analog converters, which I use quite a bit Sensors temperature pressure accelerometer, etc a microcontrollers can actually be Master or slave and we'll talk a little bit about that What the difference is There's touch screen controllers Usually four wire resistive if they're I squared C GPIO controllers and the monitors and TVs have a these things called DCD, which is kind of I squirt C for Detecting the the parameters of the monitor when you plug it in so that it automatically works so here's here's a few sample circuits that I actually I Granted from a couple of my designs This here up top is an accelerometer That's actually what was on This guy. There's an accelerometer on here, and I will show you guys Accessing that's accelerometer towards the end This guy down here is a PWM controller this I squirt C and It was on this board called Robo Mezzi and it's also on some data for boards different breakout boards So you can drive up to 16 servos from the two I squirt C lines The one of the important thing about I squirt C is you have you have to have at least you have to have the Pull-up resistors on there. You'll see that there's pull-up resistors there They're up to 3.3 and the signaling voltage is dependent on the controller you're interfacing with and If you don't put those resistors on there, it just won't work. It won't pull up so the open collector could pull it down and If you don't put the right values Then you also have problems because of the rise times There's a really good talk by Dave Anders The talks it goes into detail about how the mechanism works. I'm not going to get too far into the hardware stuff If you're interested I can point you to that. I was going to add a link, but I forgot Okay, so here's a little Picture of the protocol This is just the the addressing in seven bit mode It's a basic transaction so what happens is the lines will fall Low in a specific sequence that to signal the start of transmission That's called a start bit really just a transition sequence and then the address bid is shifted out Along with a read write signal Which will be one for read zero for write and then an act bid will be either zero one based on the acknowledgement and then The data comes out either if it's a if it's a read The slave interface will send the data back on the on the lines If it's right that the master will send the data out on the same line So it goes it's bite. It's not a bidirectional Communication but it goes either way so it goes from the from the host to the slave during a write and Then you'll have a little bit from the host and then the rest comes back from the slave with a read And then there's another transmission Signaling the end now typically a device will have another this data is actually per device. There's a definition of the internal chip addressing which will be you know what register internally and Then it'll have the values you want to send to that register or what is read back from it So there there is a bit more to the protocol, but it wouldn't fit on the slide with you know being able to see it all So here we'll get to the to the Linux portion of it the early implementations were pretty early on in the two the two point X era and But they were there's one out of tree and one was kind of tied I think the out of tree one was the LM sensors but either way eventually Greg Carhardtman came along and kind of pulled it towards the new Developing device model which came out in the two five and the two five early two six era and then David Brahmo and John Belver did the standard device model port and two six and a lot of things happen since then I can't really go through everything, but Then Wolfram saying is the is the current maintainer We'll talk about how you would get to a hold of the maintainer in a minute here so Here's an overview of what the subsystem looks like There's a couple of pieces that are required for the overall system You have the the the core peripheral inside of the SOC or Linux capable processor We'll have a peripheral controller internally and this thing called the adapter will allow the the the the core portion of the I squad C subsystem to talk to the host controllers peripheral adapter or whatever and then there's an algorithm that's usually built for each Host they'll have an adapter which will It's pretty much pass along a function that allows the core to talk to the hardware essentially And then on the on the top side, which is where most people are interested you have the drivers the client and The driver driver it's kind of a silly name, but It's we'll go into that a little bit so The the the slave device has it in a instantiation talks the ice core C core, which abstracts the the SOC's peripheral interface So that it's the same across different platforms. So your your ice core C driver shall work across the whole ice core C subsystem and that's part of the device model and Then yes, you can map it and go There is mention of ice core C dev, which is a user space interface. So we'll talk about it towards the end okay The Linux ice core C subsystem mailing list and if you're a kernel developer You want to get to know how to use the mailing list? So I get this is actually straight from the page I cut pasted it in You can subscribe to the mailing list for the link there and there's the the web interface for the mailing list and The mailing list is is your entry point into the subsystem For mainline. So if you wanted to do mainline development, you'll have to go through the subsystem which the subsystem maintainer will Monitor and then he has his own separate git repository Which we he'll he'll apply the the patches that you send to him And then he sends them up with a merge request to Greg Hartman, which eventually goes up the stream to the mainline kernel. So This is going to talk a bit more about the algorithm and adapter so The algorithm driver contains general code that can be used for a whole class of ice core adapters This is actually directly from the documentation, which I have a link down here. I Can actually show you that it's towards the end We have time the actual documentation and each Specific adapter driver depends on one over them or includes its own implementation Which I've stated previously okay, and then for device drivers, they call they have split to driver and client and It actually says that right in the documentation a driver driver It's kind of silly, but That's what they call it This is essentially the iSquad C device driver where You'll contribute if you wanted to add like a new touchscreen controller You'd create one of these type of device driver for iSquad C and iSquad C subsystems The protocol Drivers are spread throughout the drivers subsystems. They're not contained to any one specific subsystem So if you have an analog converter You'd be in the IIO subsystem or HW mon hardware monitor So the the the host controllers are actually in a specific driver directory drivers iSquad C in the kernel Source where these these are everywhere they're across the board is any kind of device you can imagine That has iSquad C will be spread across the kernel drivers I'm gonna talk a little bit about this, but I don't have a whole lot of time to go into it Recently, I think 2014 iSquads or yeah iSquad C maintainer added slave support what that means is if you're if your controller supports it Your Linux SLC can actually act as a slave instead of a host controller that means you could actually talk from one from a Linux SLC to another SLC and It'll emulate of a slave device and there's some bindings that are required This is a SysFS interface that you use to kind of poke in hey, I want this particular type of slave The slave Interface documentation is down here at the bottom and there's a nice presentation from the iSquad C maintainer down there at the bottom It he goes through in pretty much pretty pretty good detail okay, so I Haven't really covered like the anatomy of of the driver. I didn't think I'd have enough time, but maybe I Can show you at the end But here's how we how we instantiate devices so one method is the device tree and this is preferred method for you know arm and Other you know device tree compatible devices, so the device tree Binding will say hey. I have this particular device Which is specified by the the register and The compatible string so the register is iSquad C address and The compatible string is what binds the driver to the registration here I can show you inside of one of the drivers where that is and Then say you have multiple devices on the on the bus Then they'll use head head further Registrations this one is special because it's a iSquad C controller for GPIO So you have to specify that it's a GPI controller There's actually a GPIO controller on on the bacon bits, but it's SPI controlled Okay, okay now back in the day before the transition to Device tree and when I first started There's thing called platform devices own platform Devices that are a platform bus is still used But it's abstracted a little bit. This was actually a physical C file in arc arm Mach whatever architecture and then a board file and board file contained the registration for your different devices And it was a C code I don't really use this so much anymore There are maybe a few cases for this still around but they kind of transitioned away from it over time Okay, and there's a way to do Instance eating from user space, which is a neat feature as long as your driver supports it and You could you echo the driver the driver name address And they have a sys bus iSquad C devices and then you have for each bus that you have on your system You'll have a different iSquad C dash whatever Edit new device I didn't want to break it down too much if you're interested in how to do that. There's an actual there's a whole For all the instantiation methods. There's a documentation in the kernel source directly okay, so We have some user space tools they're used Kind of abused at times A character device which is kind of an abstracted device which can access pretty much anything on a iSquad C bus The device note will come up and dev iSquad C dash Whatever the bus number is and then you have an iO control that sets a slave address Which was the number that the address in the slave protocol and then you have you can use simple read and write Mechanisms so you can say hey write this register a read from this register There's in a capsule a version called iSquad C SM bus rewrite If you don't understand that notation I can break it down for you. Yeah. Yeah Well the protocol depends on the device you're accessing but typically You send the address of the of the device which is the iSquad C address right then you have an internal register address Which is set as a set of data, so you'll send the address and Sometimes there's other bits involved depends on the control or the controller on the slave So you'll just send a sequence of bytes to it and the sequence of bytes will determine what address Register internally is addressed so that's part of the protocol essentially What's that? No the recommand is kind of raw whereas this will actually break it down into separate like it I'll actually do a little bit more for you and the dev the dev interface is also linked down here at the bottom. I know I've kind of pushed everything off to the links, but this is more of a higher level kind of thing and You can you can go and kind of dig in if you're really interested These these instructions are pretty important for the iSquad C user space implementation there's a whole slew of defines and and I didn't want to have to put them on the slides. It'd be slide after slide And then there's some tools that use that interface one is called I2C detect which it'll just kind of They'll go out on each address and kind of see if anything acknowledges and it'll print it out on the screen a nice little tape tabular form It will ask you if you really want to do this when you use it. It's kind of real kind of tricky about it. This is This is used for mostly for early testing and I can I'll actually show a demo of that as well as a demo of a driver running Here at the end. Okay. So there here's the demo Of course, this is always where the problems start right? Okay, let me sure first. We got to get the count flow up there, huh? Get out of that probably not large enough is it I'm going to plug in the bacon bits keep We use the serial debug header because it's just what I'm used to I'm actually in an it currently in a mini-com instance I believe it's the wrong header Should know this There we go So for the first the demo amazing the standard Debbie an image Debbie an image takes a little while It's a full-blown OS like All the Gadgets and what have you you'll notice I have a build root File system that I'm going to boot up next for the next portion of the demo That's what's a lot faster. That's why we don't do live demos All right Debbie is okay There were prompt here or not. Can you see the prompt? There it is. Okay? so I'm going to show an example of Oh, I know it's disappeared Oh, I'm guessing it's down further. Hold on. I'm going to use I support see Utils to access the accelerometer on the board directly This is good for when you're first trying your hardware. You want to say hey, this was hard work working This will give you a real real good idea real real fast So first, let's do an I score C detect. See what comes up if I take that till the front there And then I think it's on bus to it tells you what protocols are supported by the bus Tells you the different adapters that you have Of course, I don't remember the command off the top of my head again Q either way That's our the last one you try of course I Can tell I don't use this often But it's very useful when you first start you see here. It says one Charlie or one C That's the address of the device The accelerometer that's on the board So now I'm going to try and I'm going to poke at the registers internally on that on that device to set it up for sampling on the on the Converter this is going to poke a register internal So this to a is the internal register on the I squared C device and this is the value that I want to write to that register That one turns the it's shutting off the conversion actually and this one here is configuring the conversion to 2g or 2g Okay, and then we got I'll do that right And the last piece is enabling the conversion Okay Yes All right, so I've written out some registers and it's doing the conversion internally and what I'm going to do is I'm going to use a the I squared C double command Which I It's really just a read across a bunch of registers a watch Or why I want kind of loop to so that it shows the values as it kind of moves around Okay, so you see how those you see all those registers are kind of wiggling around Those are like the raw hex counts of the different Excesses of the accelerometer. So if I move around the device You'll change the values that are coming up down there Not really readable in this form at least not easily readable It's a signed value. So you'll see that you'll have F E F Whatever and then the number it's a for this one. It's a 10-bit conversion so but if you're going to Really use an escort C device you should make a real driver So what I'm going to demonstrate next is The use of an I an industrial IO or IIO driver to access the same device So Well, I got to get the other I Was going to put it on there directly, but I kind of last minute did this one This is a Builder file system. Okay, you see that? Blinding fast. All right, so Let's log into this guy All right, so tried to access the same device on this with this build because it has a the registration It won't allow you to access it with the other tool. So If I were to do this I squirts you set on here, it'll probably walk at me That's advice or resource busy. I mean, that's the driver is bound to the I squirts he bus at that address and then Let's see. Let's go back into the word of our registration comes through I scored C and we have a device this directory So anything that's that's registered will show up here and you see there's the the second bus 001 Charlie, that's our device that means the platform Register there was a registration for this which is in the device tree And I can I can show you guys that if you'd like to see it So yeah tells you that a bit about the device so you see that it's an industrial IO device so that Think about the the device model is it's kind of recursive and so if you register something on I squirts you buzz and It comes up as a different type of bus. It can kind of go down from there. So I I Is in Now bus the whole there's a whole other talk about how I works But we're just going to show you this that the device isn't attached here Okay, so you see there up at the top you'll have The accelerometer scale and then accelerometer readings are all broken down into virtual files So if you can't those files, I'll give you the raw account. That's a signed account. So If we do a Y reading I guess why would you do this to? And that shows you that it's oriented either right side up And then you have one for each axis. I've created this little special script that accesses the Accelerometer So let's take a look at it first Okay, and what it's doing is it's getting those files into variables and then Rendering them out to the screen that I'm not really complicated So then you got Just and there you go. So you can watch it kind of live If they move it around you can see the different accesses change so The protocol driver in this case isn't is the industrial IO driver The adapter driver is an old map Controller driver that's based on the so see on the bottom here So I guess I could show you some of the various pieces if we have some time Do we have some time? Yeah, it looks like we have plenty of time So let's take a look at Some of the Different pieces here So a lot of those links I showed were hyperlinks But they're also all the documentation is actually in the in the source tree directly. This is the kernel tree here and You'll see I have them all full Different Sub-repose origin is the mainline kernel here and then Documentation And a lot of the text that I kind of just kind of scammed out some of these files If you want to get into more detail on anything Particular it's going here and say oh, I want to learn more about how the slave interface works. There's a separate file for that One thing that I didn't mention was multiplexers or muxes This allows multiple devices of the same type to be on the same bus, but multiplex This is useful because the address space of I squids is quite limited and there are going to be times when They overlap or you want to have more of the same device Sometimes we just use pins change the address slightly, but when you're on that option you use a multiplexer So anything that that you're interested in no should be here Make this a little bigger Okay, and then let's go to the Registration I guess that would be for the platform registration For this system You use the device tree This is the controller registration. I'll be in here I scored C2 and there's I scored C2 and What this does is it maps a compatible string which will be in the driver and It gives you the registers the platform registers our memory map registers inside of the so see and This is used to pass the driver These are kind of fixed Usually have to worry about this if you have a system that has Good Linux support. This will already be taken care of for you But then which one Hw mod Yeah, I'm not real sure on that one. It's a TI specific see how it has PI in the front So you'd have to look at the driver and see how it parses that what it does with it So the driver for the ice court C court C the drivers and then anything that's a Bus or a an adapter Would be in the bus's directory so every so see or implementation is going to be for the for the Adapter side We'll be in here and the TI one would be I think it's High-square C All map you'll have a compatible string here All right, I guess it would be Townthorne's in So it's a platform driver. So there's there's gonna be a lot of boiler flake code here And then you have the open from our match table Here and you see you'll have the compatible strings and And when it when it probes what what it sees a match a compatible match It'll probe the driver and this data will be passed. So this data will give you a little bit more information this isn't necessary for all different controllers, but they like to combine the Controllers for different like series of processors into the same driver to eliminate redone redundancy in the code and then you have That gives you that tells you what's going on with that particular device and then There's quite a bit involved in making a Pay adapter driver check really couldn't cover in one presentation. Maybe it could be a separate session but There's gonna be callback functions that are called from the iSquad C core driver to Accomplish what you need to There's a lot involved so let's Quit out of that one, and then let's look at the protocol or a driver Yeah, the name of the accelerometer a camera it's MMA something now I don't think about it aren't that 84 52 There's a there's a series of them that are going to be supported by it and 84 53 is the one that we're using on the red one of the of the Cape, this is the one that was we usually are used originally, but we leveraged stock from GHI So we we had to go down to this guy, but no big deal and so this this is the the driver that's Quite a lot of code as well Let's go to the end where the important stuff is I guess So here's your typical, of course the driver will have this driver struct and it'll have some callbacks here and Pointers to tables that are useful in matching the device so again, there's a device tree Pointer table and That'll match with the device reentry So we put this free scale underscore 84 53 into the device tree under the I squared C sub node and then off it goes And it'll probe that advice on boot if the driver is loaded and if the driver is loaded dynamically it'll detect and load And then what will happen is the probe? the I squared C probe will be called and it's The I squared C client Struct will pass then a relevant information down to this guy, and then you'll have first per controller Data, so so for each you have to allocate a little bit of memory DevM IO IO device allocates specific to the IO and anything that starts with DevM is nice because When you won't have to do a free call and the remove function It'll just do that magically through the through the course So and then you got you know your basic Some bus read byte data. Who am I and then it figures out who it is and Attaches yeah It's going to tell you a little bit about the device It's a register internal to the to the accelerometer the and then this these are just kind of Defines to say hey what which register reads out this value and it can be slightly different based on the different types of devices But you get the idea Back to the bottom and this table here is is you can use for kind of like dynamic probing and Platform registration outside of device tree Yeah, the platform files if you guys would like to see an example of the platform files This is kind of legacy But they used to end up in Linux Ark Ark arm a lot of them in factor out especially in arm because it was quite a mess and Linus 12 all got really mad and They fixed it. That's a Oma to there's not going to be a whole lot of examples here, but There is still platform file. There was one for each different board And here I got it. It just gets a little out of hand after a while But inside of these they'll have those that platform registration, which I showed in the slides earlier So I guess that's enough for the demo. We can move on to questions now that we have like 10 minutes device Yeah, yeah, it's already it's already instantiated. So I don't know if that doesn't work for all drivers but It well it depends on the probe function. So let's go back into the driver this particular driver I think Jason had problems getting it dynamically allocate or whatever So he had to use a device tree overlay. I think The problem was they didn't have any code for handling that so it was either you had a device tree Registration or just wasn't there and the driver I suppose I could show you that What's that Overlays are kind of new but oh is it I to CD twice. Yeah distracted typing again. There we go 84 And then we go down to the probe function and Then you see this device tree of this is the open firmware device match and you see how it says if not match Unknown device model So it doesn't it doesn't really expect anything but a device tree for this particular driver every more code platform code and I have a little Session about I scored see that we'll cover a bit more of this detail tomorrow in the embedded a If you're registered you can come and see that as well So I guess questions It's pretty different This is this is a busy box image, and it's very small. It could be smaller actually, but on that freedom H is free dot Okay, I think two case There we go. We don't want to megabytes kill bites Gigabytes, okay, so we're only using 13 mags Not bad It's it's can be up there depends on what you have running the Debbie in the stock image does run a lot of web services and stuff So really this this is about as pretty bare bones. It doesn't have a whole lot of features installed So builder it is good for kind of a small footprint type of builds Anybody else sure Yeah, I Would be pretty tricky because these hex addresses on Unique in that they can be designed to multiple devices So you could have two separate completely different types of i-square C device. They'll have the same address but you could perhaps kind of Maybe grep through the driver code. It's really not easy Yeah, it's kind of tricky I guess other way around would be trying to find the schematic for the Board that you're working on and if you're doing boards before you usually have that If it's still something you can't figure out then I guess you kind of go upstream to the the hardware manufacturer And see what's going on. I'll get one back here first It seems like that but what I'm doing is I'm I'm actually pulling their advice continually Yeah Yeah, though if there's multiple devices on the bus They won't they won't talk to the to the master unless their address So that's a way of arbitrating between so if and there's really no streaming involved You have to you have to from the from the The master you have to say hey Read from this specific device. It's not going to stream across the Because otherwise it would cause all sorts of collisions on the bus Yeah, that's something specific to that a kilometer Right Yeah, we'd have to dig into the protocol a little bit there are There probably is something like that the who am I register might be Standard, but I don't know I don't go that far into the spec usually I just try to get the device at hand working The the link for Wikipedia has some pretty Detailed explanation of the protocol and it also has a from their links back to the actual spec Where you might be able to find something like that For I squared C protocol I The protocol is defined and the address of the device has to be specified before you do a read The registers on its device are really dependent on the implementation of the device itself Yeah, there's no no direct standardization I don't think but there there is kind of a Kind of a guidelines so that you can use basic The same kind of algorithm or the same kind of register mapping from one device to the next so if you have like From the same vendor you have like say from TI you have Touchscreen controller and then you have an ABC Accessing the registers the address probably would be in the same place because they're probably going to reuse that internal hardware slave core or whatever, but it's really up to the To the data sheet to determine the protocol beyond the addressing of the device itself So there it's kind of algorithm Let me try to Get a feel for what you're asking No Yeah, I think there are just I don't know them off the top of my head Yeah, it wouldn't be quite like that It would be other types of parameters. I like the the protocol would maybe determine the sequence of bytes or whatever but I Usually do you you the protocol is really just saying hey jam these two lines the adapter is pretty much saying hey Call back to this function and then that function will do what you want it to do There's not a whole bunch to it there If I was an I squirts the expert or a Linux kernel I squirts the expert I'd probably give you a better answer on that but I Just kind of get things working essentially and how that Yeah, there's there's a you mean the name of the device Yeah, there's there's a mapping called at aliases and the in the device tree file, which will determine the order If you don't specify it, they'll kind of pick the order for you I could probably show you that I'm guessing probably pretty long time But let's check it out. Okay. Here's your aliases list and this will determine the order so It's pretty straightforward on this particular device There's one to one mapping but you can change you could say if you wanted the peripheral controller that's mapped it at I squirts you one on this Come up as I squirts see zero It really depends on this alias here and that it's useful for the ordering of the serial ports as well and The USB controllers, etc. Ethernet, so that that's used to kind of determine the order and the device Essentially, it's completely possible Yeah, you if you don't register in the device tree and you enable it it's not going to show up so and the top-level DTS You'll have one and this this is the DTS. I which is kind of like for the so see and You'll notice that I'll see I squirts see You have the status equals disabled right here So as long if the if you're not mapping a device to it then This will stay disabled and they'll it won't show up in the devices if you if you if you have The device tree file for your specific device then you can probably enable it It's just a matter of getting to it another thing to worry about is the pin boxing though, so there may be a five or six controllers But they're all coming out through a limited number of pins So they have to be multiplexed out to the SOC pins So if you want to use them as I would see you got to make sure they don't contend with anything And they're actually broken out to pad somewhere and then you have to set the multiplexer so that it sets that pain for that specific function And then you can enable it and go essentially so it's kind of like there's a few hurdles But you could do something like that. Yes Yeah, so you're kind of piggybacking on to another bus. Yeah, okay Yeah, they probably some kind of maybe for firmware or something that they don't want you messing with Yeah, that that's Either happens earlier on in the boot and then they don't expose the interfaces to the user space or to You know even to the kernel depends on how it's mapped in the in the device tree Yeah, if you if you have mainline support for your device I'm not sure about that particular one, but Yeah, it's one thing I didn't bring up is pin boxing can be kind of One of the first things like if I can't get my device working First thing you do is determine if your pin boxing is correct From there you gotta determine if you have the the pull-ups on the lines and not in multiple places Not too small of values and then from there. It's just kind of Software pretty much went a little over didn't we The The original boards I have a few of those I hadn't populated some of those Yeah, the guys over at Osh Park got you those early on I Brought him to the expose show off. I brought him back and then I actually assembled hand assembled A few units like five units. I have a few that I hand assembled then Before that Jason ordered some from Handful we were gonna try to go with it, you know at group kits we had like a campaign We just kind of It was an economical Because Okay, well, and they I think they were really You It's a different So you have to use a different build, but then anything is all right and you can try to make it as good as you can. Maybe from bike drivers, as long as it is affordable it should be good. Now, I only broke out a few years ago with IEL, so it was just FBI. It is as many as you could. Yeah, there was very much space on the board. Yeah, this one, you can see the ship was a lot smaller than this one, and it makes a lot more matters worse. Sorry, I couldn't answer your question earlier. For which one? For the algorithm. Yeah, the algorithm. I never heard. I had no idea that was something that... Yeah, the algorithm is just a pointer back to the function that makes it happen. They're all doing it kind of the same way. They have a function that's passed in this depth, or in the function, or in a steady structure. It just kind of pushes around this function pointer so that it's natural for this hardware. I would have showed you that on there, but I haven't been flustered. I had never seen that that was something that could be different. Yeah, you say the 10-bit protocol, say you want to do the 10-bit, whatever, you might have a different way of doing that. It's just a matter of working out what's what's what's what. I really haven't done a ton of development on that. It's really just kind of driver stuff. Yeah. A lot of the... I really like the unified thing. Yeah, it's interesting. The new device? Yeah. Is that what you call it when you kind of echo the address in the driver into that file and load it? Yeah. Yeah, that's from user space in the fancyation. Yeah. It's pretty cool. As long as you have the table set up and the probe function set up to handle it properly, it should work fine. A lot of the older drivers will have a platform code, and then they adapted it to the drive street. So those would probably work. Because I.O.9... Well, they just do a device with that more recently, and it's not a hard-to-padded requirement to have every different kind of stampede more than those. So that was an I.O. driver and the probe didn't have it. Are there other I.O. drivers that have that in their probe? There might be something I'm missing. There's a string that's a passion that might be a little different, but I don't think that's the truth. It's one of the best effects of the driver. Yeah. I might spot it off any falsities I'd like people to tell me so that I don't do that in the future.  You guys got all the terms mixed up. What happens when you push them to do something and then you don't work on until the last minute? Yeah. I think it's good on jager and it's going more in-depth in the workshop. Yeah. I see how my hours are going to go. It's less than 500,000,000. Because it's running on because of the limit. But... Do you change what's between the revisions, right? What's that? The part? The part got changed? The accelerometer? Yeah. Okay. Because of what they had to stop? Okay. Because jagen, he was confused. He didn't know what was going on. Why isn't it working? Why isn't it working? Right. Last night, he was going to get his dimple. Test, test, test, test. All right. I'm live. Jason, president of the authority. In your aesthetic, Texas Instruments was 90 years of experience with doing event processes. And it also is driving force behind the legal form of legal laws, which are oxy-legal legal systems, as one of the founding members of the legal board, that holds foundations. And they have a big influence, though, attracting bakers and learners of all sorts to the open-source world through the legal platform. This is going to be a good opportunity to have a lot of slides. I won't go into my personal life horror stories and stuff, but I spent all my time preparing the slides for the talk this morning on the Pocket Beagle Walkthrough. If anybody was in that, but it didn't look like that made familiar faces. But next door, they're giving the embedded, apprentice Linux engineer training, where they're using Pocket Beagles and these add-on boards to teach people about developing with Linux. And I am spending my time there. This presentation was one that I was originally... I'm going to give it this next week at Embedded Linux Conference. But unfortunately, I was explained already how the slides are done by now, but I was just going to reuse it. So forgive me. I don't actually have any slides. But if you want to walk out, I'm not going to be offended. But I will try to show you some of the things about Net Console. I don't have it working in this setup, but I can show you where some of the gaps are and some of the things that allow you to get there when I have all the rest of the... Anyway, I've got a couple of nights of work ahead of me still. But if you've never heard of Net Console, it's generally a way to use a network interface to do serial type of tasks. All it is doing is sending UDP packets, pretty much raw checks in UDP packets for the data. And that could be all sorts of different. But the Linux kernel, it's the console messages for the kernel log. So everything is printing out, gets sent out across UDP ports that's pre-configured for Uboot, which I think is, in a lot of ways, more interesting for the early boot phases. The somewhat more challenging part of Net Console is that interactive portion of the boot loader. So you can actually... You know, you can specify to do TFTP, you can specify to do all those things using UDP packets. And you would just use a tool like NetCats if you've heard of it, seen NetCats. I mean, people have heard NetCats somewhere. Okay. You showed up to a Net Console talk because you're actually ready to learn about, like, see Net Console and things working. So, yeah, you can use NetCats and you can just, you know, the port 6666 and you just... You can start interacting with the Uboot console directly from Net Console to see the commands coming from Uboot. You can type the commands. The extra little bit that I'm really trying to add here is doing that all over USB. Right? So if you... You know, we've got... So this is the Pocket Beagle. So it's a little Linux computer running off of the powered off the USB cable. And that USB cable also, you know... So it's connected to the Gadget controller on the ROG controller on the processor. So I can actually... And the ROM has a boot loader. So I can... Let me just take my... My disk out, right? So that's... Normally it boots off of this microSD card. But the interesting thing is when you start booting up off of the USB. So I've got a little Node.js app that essentially implements... It uses the Node USB or it essentially uses LibUSB if you're familiar with LibUSB that gives you an API to directly initiate packet transfers.  So you can do all the individual transfers from the API rather than writing a driver if you're doing everything in user space. So with Node USB, it's just doing that exact same stuff in JavaScript. It may not be your choice, but I'm trying to put this stuff together so we actually do it with the new Web USB stuff. So the overall objective is to be able to boot your computer off of your web browser and then even to go to the point of seeing the U-boot traffic in your web browser, right? And being able to interact with it that way directly from just the ROM on the processor. That's where we're headed. Check my password in, right? So this is that little daemon. You could kind of do the same thing all with services, right? You don't necessarily, like, because what I'm doing is instead I'm grabbing that USB device here, I could let it get enumerated by the computer and look at it as like a network device and do things that way, but instead I'm grabbing it here in that sort of web USB framework. But it's essentially a TFTP server, right? So the ROM code on this processor actually comes up and says it is a network device and then will allow you to TFTP code to it. So we TFTP SPL to it, then once SPL is loaded, SPL TFTPs, or it loads another USB driver and it TFTPs U-boots, and then from there you can interact with it via Net Console or you can boot to the kernel. SPL is the secondary program loader. So it is U-boot. It's a different compilation of U-boot. So it goes through a different set of if-dif and different linking, such that it trims it down to a much smaller version. So it's a U-boot meant to load U-boot. It doesn't have things like the command line that you could run different... It's more fixed purpose. The reason that it exists is because for some of these processors, including this one, you need to configure external memory with some initial code before you can load into that memory. So the bootloader itself doesn't... The ROM bootloader doesn't configure that external memory. So in our case, that's exactly what it is. So the SPL will fit entirely inside the internal RAM of the processor, configure the DDR that's sitting external to the processor, and then that SPL can then load U-boot that's bigger than what the internal RAM is on the processor. MLO and SPL are the same thing. MLO historically comes as a TI name. It came from a proprietary approach, which was to... So the TI developers started with U-boot and they stripped it down into an internal version and then loaded U-boot. But then it got pushed upstream and the MLO... I don't even know what MLO stands for, but the ROM code actually looks... On an SD card, if you're booting from FAT, it'll look for a file called MLO. So it looks for a file with that particular name, but the file is SPL. And did I say that SPL is a secondary program loader? I did, but I don't know what MLO stands for at all. Yeah. Right off the... Well, and it's just meant to load... It's just meant to load the... Well, the primary one here is U-boot. So the re-of-the-ROM is first. That's Treasury Area. Yeah, okay. Yeah, I couldn't... We need Tom Rini around to answer that one. Oh, is it? Oh, but that's the hands-on U-boot. The... So I'll just show you this. So that's doing the boot pee and, you know, it'll just... It'll be sitting there in U-boot right now. So let me... It's not very interesting. U-boot is trying to boot. And so, like some of the default configurations it has is to try to TFDP this PXE config file. That's just part of what Mainline U-boot does. I don't know why it's trying to do that. My little program doesn't have any idea how to handle that. So normally, I would stop it before handling it. I'm going to go ahead and bring up the serial port, which is the whole thing that I'm trying to get rid of. But the way you would normally interact with U-boot is you would have a serial connection to be able to work with.  Figure out how to configure things just to be able to read boot logs. They should almost never have to do. But in the case they do, I wanted to work all over the USB pseudo screen. You have to put the bot right after it. So it happens to be 115.2. I hope that's the right location where this guy enumerates. I should have looked at my message log first. So if I...  It's about to start rebooting again. My little daemon is going to sit there running in the background now. But... Oh, it kept the power up at the serial port. Hit the reset button. And again, so the ROM grabbed the SPL. Okay, and that's how I stopped it. So at this point we're inside of the U-boot command line. So what's run here is... So if we go back to the very beginning of the boot on the serial output, we see this very first line. That U-boot SPL line. So that has... So SPL will print something out that says I'm going to try to boot from USB Ethernet. It tried to look for environments on the FAT. Of course I have no SD card attached, so it's not going to find any environment variables saved off in the FAT. And then it ends up getting the U-boot load question. And the boot ROM itself on this particular device includes that it looks like an R-indist USB R-indist device. And it tries to do an ARP and a boot P.  And then it'll start doing a TFTP transfer. So that first part was satisfying those requests by the ROM code. And then the second part was satisfying those similar requests from SPL. You couldn't then go from here, if we wanted to TFTP the kernel, we could use the same daemon to TFTP the kernel or TFTP a root file system. So I mean that part is pretty well handled. What's not handled is the debug part, which is why the net console part is really important. So as long as all that works, you're great. But if you're trying to develop a new kernel, you're trying to develop a new, you know, making changes to U-boot without having a serial port, you're not getting any real feedback. You're just not getting the packet request and you have no idea why. Or you're not getting your kernel to boot up. And so when you want to do, when you want to do net console, essentially it's pretty straightforward. You have to compile a few variables in the configuration that you say enable net console. But from a practical standpoint in U-boot, you also want to probably have the thing that allows you to muck the output. So it's possible to have multiple consoles in U-boot. And maybe you'll eventually have to switch over to all net console, but typically it's serial. So if you print, like you'll see these, I don't know how many people, how many of you have played with U-boot's command line very much, right?  Well, I'm good. But it's actually a very, very flexible programming system that it's got in here. So you can print, you can help, get a list of all the different commands. There's a lot of different commands. You can find out a different version information. So you can query quite a bit. If I wanted to be able to turn this into a, like a USB dongle, I actually run UMS right from here, which means USB mass storage. So from this, I wouldn't even have to have an SD card writer that's separate. I could actually just, straight from U-boot, I could run UMS, have UMS compiled into this version. Yeah, so if you run the USB mass storage thing, you can actually just use it to serve up the card and write it directly from the card. That's another aspect of what we're trying to do here, is so we can use the web USB and it'll automatically download the images and program it. Is there a question or comment? So the device tree, this is super, super useful stuff if your system's not booting because you tried to load the wrong driver or other things. So U-boot has the ability to patch the device tree. So the device tree is the thing that describes all the hardware to the kernel. And so you, for each of the different target boards, you'll have a device tree that gets loaded at the same time as the kernel. The boot loader will load that. For Beagle, we also have an overlaid scheme where we'll look at a bunch of different variables that you set up in a text file and use that to load different overlays. And U-boot will actually apply those overlays for us. So in the kernel boot, it's all just set up and running. There's also ways to dynamically do it in the kernel, but for things like, oh, I'm adding on an LCD. I would like for the LCD to actually show the boot messages or something like that. You know, I want it to have it very early on. So you can do that in U-boot before the kernel ever loads. So you can modify the status structure that describes the hardware to the kernel. All things that would be really nice to do without having to go get a serial adapter and a number of things that I found that I can't do in the kernel that I can do in U-boot. One is, so even if you have like a fleshed out device tree, I don't think there's a way to easily, I mean, I see remove nodes, but honestly, I really, really struggle in any sort of order to really remove nodes practically, but it's pretty easy to go and add a status equal disabled to an individual node so that when the kernel loads it doesn't load that module. There are a lot of, there's a number of hack modules essentially in the world out there that you might want to load that don't unload themselves properly. So anytime if you try to do it dynamically in the kernel, it needs to have the remove module support done properly. Yeah, and that's, I mean, the config at FES Pro is how I'm doing it now on Pocket Beagle. So as far as like live patches, but we just generally discourage people from doing live patches instead of doing the patches as U-boot. But there's a few places where I'm showing people how to do the live patches with config FES. But the Cate Manager, the Cate Manager is not part of Pocket Beagle. It's pretty quiet, I know the rest of the room, I'm sure the rest of the room can't hear you. The advantages of config FES over what? Well, for the most part people using it really aren't doing editing, they're just wanting to use it, and so that way they can have it at boot up that it's all just running. You know, it's persistent, right, when you edit the config for U-boot. I'd say it is a challenge to try to do certain things like we now have in our default device tree for Pocket Beagle.  But just trying to disable those, like the helpers for that is all like one big thing, it doesn't unload cleanly from the config FES, but if we patch it in U-boot it never loads. So we just set status equal to disabled as part of our overlays, and then we export whatever GPIO pins we do want to export, so there's just a few cases where you really needed it to be in U-boot, and choosing one or the other, like the LCDs, like that was one of the biggest ones was the LCDs. Well, but we have config FES, it's already in the kernel, when you talk about the config FES. The CAPE manager, one of the things that we do in U-boot now is we actually scan the eProm, so the bone CAPE manager, so if you've heard of CAPE manager, it goes and looks at eProm values, looks for the device descriptor string in the eProms of add-on boards for Beagle board, for Beagle bones, and if it sees that it loads the device tree overlays that is associated with that board automatically, so it just dynamically configures and goes at boot. We were doing that in CAPE manager at the kernel level, but now we're doing it at U-boot because CAPE manager, they didn't want to accept upstream. They've got dynamic overlays, but because they think a lot of it has to do with the terminology of calling the CAPE the bus, and say, well, that's not a bus, right now it works. By default, it's going to run this, U-boot's going to run boot commands, which goes through this list of different boot sources and tries to look for some valid image. We want to spend a whole lot of time going through U-boots, but I did want it like so. The Enverma variables standard in, standard out, they all say serial. If I put serial comma in C in those variables, that starts in that console. When I'm doing that, I've always created a variable that sets all three of them at once. I don't know if it really matters or not, but if there's a right order, you can do it in, but I've always set all three of them at once. The set environment, I'm going to standard in to serial, comma in C, standard out, and C. Now it's going to complain because I haven't set the server IP or my IP address. This is where I need to kind of think about a few things, but if I do run, start, it brought the network up, but it says there's no IP address set, so I need to be able to set IP address before that. Does that quite make sense to me? So we need to try to do something to fetch an address before that, and I don't know. There's a couple of steps. I've had this running before like six months ago. I had this running, but I was actually using this to do board testing. I was actually booting the boards over USB and using the net console to issue commands. So I'm going to pull that code up and try to understand everything that I did to get that again. But I abandoned it at the time because it was too slow compared to booting off the microSD card. So it took a couple seconds to load up everything over USB, whereas I could be done well under a second with microSD cards, so I ended up doing all the board tests. For the pocket-beagle testing, I was all done booting over microSD and then using USB to send the serial number across. So we still make a USB connection and test the USB connection when we build the pocket-beagles, but we boot them off of the microSD, which tests the microSD card, and then we send the serial numbers to U-boot but over the USB cable. We don't boot it over USB. So the serial number is going... I have to try to look at the code. The tester has the serial number and it's sending it to the board. The test code is still on a gist. I don't think it ever made its own repo. So the test code... This is the DTS file that I was using. A lot of this stuff in this repo is also about bring up. So this is the tester code, so it says JavaScript code. So it's actually using NetConsul to send... Oh, yeah, for sure. So we just increase the font. So this JavaScript file is using NetConsul over U-boot, so U-boot boots up over the microSD, but then it exports the NetConsul. And so I actually... I just listened for the UDP ports here in... I saved off my magic foo. I probably should have just copy and pasted it, right? But this is the magic foo for starting a NetConsul where I've got the... I'm setting a server IP and a personal IP address. And I'm setting all the other items. But the commands that I'm issuing... This is what it actually issues during the test. So it turns on an LED. This is the write protect signal. The GPO is the write protect signal for the eProm. So it's tied to a line. Then I've got this script that sets up the eProm with a base header that's common across all the pocket vehicles. And then I've got this serial text. It's actually the serial number for the board. And then I'm doing... What's one of these memories? Oh, I'm putting it into memory chunks at a time. And then I'm going to do a write of the whole memory dump. Right? So here I'm doing the eProm write... I've got the reading the data that I've put into this location in memory on the... What is that? Is that the link? I have to look at the eProm write parameters exactly. That's the... That's the iSports theater for the eProm address that's reading out of memory. It started off at zero. And I think 1C is the link. And then it sets another GPIO if the write succeeded. I read it back. I set another GPIO to say that the write succeeded. I do a memory compare to actually see that the eProm value that I wrote is the eProm value that I read back. And then if that works, I set another GPIO. And we've got all four LEDs on board good shipping. So there's the testing here. You've got the USB for all the communications. You've got the micro SD card for doing that. Plus you've got the eProm writing. So it ends up doing a number of tests on the hardware in order to get here. But this is what's being used to test all the pocket vehicles before they ship out. It's really not very well documented at all. There's a reasonable amount of stuff for doing that console from the kernel. But when you start doing that console in UBoot, it's largely undocumented. And my hope is to kind of bootstrap some of that and to kind of give a really good example. Unfortunately, I wasn't prepared today. You can definitely miss some of the early stuff. You need to set a reasonable boot delay to allow time for the interface to come up before you start doing interactions. Yes. I will recommend that people doing this actually do either both if they still want to be able to access the serial or as a default. There's a way to test if there's a network connection or not essentially by doing the ping. And if there's no network, if the if-net console still pre-doing the boot, we do a ping of the server IP, and if we can't ping it, then we don't start net console. I think for your individual target, it may not. That's what we're trying to get to. But where the usage is right now is everybody depends on serial until there's a lot of good examples, a lot of people coming to talk here and really prepared showing people how they can do everything with net console. Until that happens, and lots of people are using it on a regular basis, I don't think the documentation and the examples are going to be clear enough for people to follow it enough that people migrate away from using serial as their primary debug mechanism. And that's where I want to try to get to. Let's not pay for the USB to serial adapter in order to have people be able to bring up Linux. And especially with something like what you've got with Pocket Beagle, you've got that SIP on there that's pretty easy to lay out and make a board for. I think you can actually probably do it with a two-layer board, which currently are a four-layer board. Michael is in the room, we'll have that debate. All you need is power and a crystal, and you still need to do some strapping of the boot config lines. You put resistors on there to kind of tell it what boot mode it should try, or what boot mode, because it tries four boot modes, depending on what the boot config is. So you need that much hardware, but really nothing else. You get power and you can get the USB connection just from the USB cable. And so if you're trying to build your own board, you don't have a lot of other system dependencies. I haven't gone out, it doesn't matter what's on the other pin. I can boot up and run Linux, and then from within Linux, I can go and try to turn on other things and turn on a subsystem at a time and test it, see how it works on the board, and go about that approach to bring up a new system, which is pretty, pretty different from the way people do it today. And although things have evolved a lot, and when I started doing these, and everybody would have this, you start with a JTAG connection. First you get JTAG, and then from JTAG you start loading, okay, I'm going to do some memory tests to make sure my memory is all right. Okay, memory is good. Now I'm going to go to the serial port. I'm going to get my serial port connection, and I'm going to get that data going. Individual test code items that are personally hard to maintain and move forward on the different platforms. That's the way you traditionally would bring up a board. It's evolved somewhat now, because there's a lot of open source support. There's still the initial processor bring up, but if somebody's already brought up the processor, you can probably get to that serial port and do some other stuff there. But I think this takes it to a whole other level of simplicity. You can essentially do your board bring up with just the USB connection, providing your power, providing your communications, and everything else. Then you can use the Linux to actually running within a system, booting into RAM, to start going out and doing anything else, sitting outside of that. But also you want, so I try to look at it both ways, right? What is the professional need and what does somebody need to do as a kid or a hobbyist or something, just trying to get into it to understand what's going on in bootloaders and get that visibility. If they can get it in their web browser, they didn't have to install any other tools and it was less dependent on what that system is. That's to me sort of the magic. There are some tools to give the USB, that means to get to your browser to serial, but they still depend on external hardware and connections that aren't just trivial. This is the one connection to the board. Anyway, that's what I'm trying to get to and we've got the Node.js project. I think the big thing we have to kind of make a change is get people using Net Console because it doesn't get used enough to get really debugged and fleshed out. Maybe I'm wrong, but I haven't heard anybody here actually using Net Console on a regular basis. Anybody use it on a regular basis? Have you used it at the kernel or at the bugoo? Both. Oh, wow. I need to... Why haven't you done this presentation? If you had some documentation, I've been like something right up on that. That would be extremely interesting. The gentleman with the necessary. He looks familiar, but I don't... So what do you use Net Console for?  But you've already run the Ethernet, so... Did you boot? Were you using you boot? But you didn't do Net Console. I mean, doing it in the kernel is not that uncommon. Doing it in you boot, it's the support there, but in terms of its function. But you can tell, I mean, I've done this a few times and I still struggle with getting it set up right. For the kernel, it's just because at the kernel command line, you provide all the parameters for... Where do you want the IP address of where you want it to be sent, and it goes. So it's just kernel command line. Chris, are you telling me you're asking me? You're probably...  Especially if you can get into EFI, where you can actually run, you know, you can run some nice scripts from the bootloader to get things. If you can get into EFI, then I think you can do the configuration of the kernel command line stuff from there. I think it makes it real easy. Well, this doesn't have either, right? So that's why we're doing it over USB, which makes it a little bit trickier because you've got to get the drivers to come up first, and then you've got to try to get the services. Like, if you're sitting on a static network, all you have to do is you have to have somebody running and listening for the R request, you know, and the boot P request for when the device shows up on the network to get it as IP address to get it as base code. It's a little bit different when you're doing it over USB. Yeah. I don't think it's that much more complicated. It's just that there's a couple twists to it. I mean, so in this case, we have a ROM bootloader that notes how to boot over USB. You're going to still need something that knows how to talk to the Bluetooth stack. If you really wanted to add something like that, there are some Bluetooth serial dongles that I think could be an interesting solution, but you're still depending on firmware within that dongle to support the serial connection and doing it that way. Well, at least it's not, you're not depending on software in your target. You're just depending on software in some external little module that you can control and replicate. But, yeah, I don't know anything that's kind of got a solved just boot interface on Bluetooth. Maybe some six low-pan experts will speak up. Thanks for coming by. I apologize for the lack of slides. If you have any other questions, feel free to come up. If you didn't want to shout them out, feel free to come up to me. If you have any pocket-beagle questions, I can certainly entertain those. Oh, one more. Yeah, free LC. I can absolutely upload it on the scale slides as well. So I'll make my slides over the next couple of days. And, yeah, so give it until after you'll see and then look and the link should be live. My apologies. Yes, again, feel free to come up to me afterwards and I appreciate your understanding. Thanks. Inside the pocket. It's not going inside the pocket. I'll try and help him back here. No, please don't. What? Put