Check check, okay, well, let me get my myself tucked in here Oh, okay, great Okay Okay, great. Thank you Okay, that's your I gotta know what time it is in Tokyo. Yeah, you need to know Okay Hello everyone, welcome to this session of embedded Linux conference And I'm happy to be here with my co-presenter Marta Ribczynska Ribczynska, yep, yep, I think it's better and better every time we practice I get a little bit better at it Anyway, so I want to talk about the nature of this talk is just We periodically take a look at what's going on in the kernel and in the embedded industry And this is not comprehensive. There's no way we could cover everything that's going on And so this is just a few things that Marta and I saw And There are a few extra slides at the end of this deck. I actually prepare a longer version of this Talk that I give at other events So there's an hour and a half version of this talk But we're not going to do that today We've distilled it down to the important points But the slide deck has extra slides if you go past the end and they're already uploaded So just a quick overview what we're going to be doing We'll talk about the Linux kernel technology and then some technology areas a little bit of industry news About the community and some conclusions. So let's start with the Linux kernel I think John Corbett famously said the looking at the Linux kernels schedule is a little bit like looking at a tide chart It's not very exciting because it's so consistent So you can see that it's either nine or ten weeks between kernel versions So I mean you can pretty much set your clock by it We've had about five in the last year. That's pretty normal very very normal We've been at this kind of cadence for many years and so that's good. It's Companies know what to expect So I'm just gonna go through a couple of things. Oh, no, I'm gonna let Marta go through a couple of things with these kernel versions So We have distilled the embedded information important for the embedded developers from different kind of versions starting from 6.4 So in 6.4 there was the slope allocated that got removed new documentation for building a kernel The model license declaration removed for anything that you cannot build as a module and user trace events API that Was merged in that version then six five Has seen Work on parallelization of the boot CPUs if you have a multi CPU architecture there is important to Actually boot all of them and their optimization to To reduce that boot time of everyone Every single one of them x86 only for now Quite likely others will follow a big organization of 42 bit device tree files to more to match other Like armist 64 MIPS and and so on And a trace and path updates Of course all links that we are seeing in the slides They are clickable in the PDF version of this light. So you can download and read everything up 6.6 is It's possible with 6.6 to build a kernel without buffer head support. It has been in the works to replace it with different Different mechanisms and yeah, there's this work. This was this work is happening We have new event FS subsystem in this version to for improved memory efficiency and tracing and we have shadow stacks on Intel platforms only for now and let's get a little bit into this You have probably all heard about attacks true Overwriting the stack pointer and then going somewhere nasty so they are hardware mechanics that allow to prevent that and They have they have been long discussions in the kernel about how to handle that has Actually landed in 6.6 There are patches for arm 64 and work on muscle Also has been started then six seven You can enable disable felt it to a bit emulation on x86 on the command line also to reduce attack surface one architecture remove ea64 More efficient kernel merging of same page and And Part of the real-time work Print case supporting per personal locking six eight deadline servers the first rust driver Refactoring of network code data structures in some use cases up to 40 percent improvement Yet another memory allocator removed the slab one so the slope is the only one that Is remaining Probably all reducing confusion also because all those abbreviations with one letter difference that was complicated for years and There's a new path that are a type profiling and Deadline servers Okay So one of the things that was added in 6.8 is known as deadline servers so this is actually a replacement for real-time throttling and So the what you want to do when you're running in real-time is you want to avoid the situation where The real-time tasks are starving the regular scheduled tasks Because when that happens you can't like break in with SSH or as an admin and fix what's going on So we've always had something called the real-time throttling That was in the kernel, but it required a fixed amount of overhead and We don't like to kind of waste CPU cycles and so they now have a new mechanism. So there's there's a new thread It turns out that there's another scheduling category the deadline scheduling category And it's actually got a higher priority than real-time. So they've got a way this works as a deadline server set up to be started And it's running all the time, but it only activates when it detects That Possible starvation so it's actually not consuming any any cycles And then even when it is active it only takes about 5% of the quanta the CPU quanta So it's actually more efficient. So that's just something shows you that Real-time stuff is continue to be work on The other thing that I thought was really cool in this release was the perf data type profiling So this is a mechanism where you can annotate the perf data On on a perf field basis. So you can actually see these charts saying how much time was spent accessing a particular field in a data structure and this is really really handy to go out and look at Whether or not your data structures are well aligned for cache boundaries, right? So you want the most use the most frequently accessed fields to be all within the same cache line and This is a tool that helps you do that and I don't think I've ever seen this level of granularity in the tracing system I think people have already said that they found some things Where they're able to optimize the data structures in the kernel So this is this is I think a really nice new thing In terms of Linux 6.9, it's not out yet, of course, but we've had the merge window so there's a couple of things the ext2 file system is now deprecated and So that doesn't mean that if you have any ext2 file system on disk you're out of luck Because the ext2 4 can handle the ext2 in backwards compatibility But it does mean that you may be out of think about moving forward I don't think many of us are still using ext2 file systems But you never know if you have like an old design just just be paying attention to this issue there's been some Timer subsystem rework and this kind of blows my mind because the timer subsystem has been around For forever basically and they still find ways to make it more efficient and You think we wouldn't find fundamental Areas of improvement like this, but but we do and this had to do with Matt how Timer expiration expired timers were handled And let's see there was work on reducing network stack contention the kernel energy model can be updated at runtime and then You had there was a vulnerability on Intel Register file sampling and there's a mitigation for that so we continue to see security work in the kernel I want to talk just a little bit about LTS, right? So this is a big issue for us in the embedded space So LTS kernels have been historically supported for six years or they were for a while supported for six years but the new plan is to Only pick one per year and maintain it for two years So the old the LTS support is going to kind of run out and So there are some places in the industry like CIP or Ubuntu LTS where you might be able to pick up some of the patches But upstream is not going to be doing this. There's also some commercial efforts actually around doing LTS support or You could consider switching kernels during your device's lifetime, but we all know we never do that so Let's see. This is just a quick chart showing contributions by embedded Linux companies and You shouldn't read too much into this because companies end up going off and doing different things Consulco has kind of a lot a low number But they're doing a whole bunch of stuff for automotive that's out of tree not I mean Like in user space and other places and the same with other companies But you could see kind of the top contributor for a lot of open source a lot of these Consulting companies and what they've been working on and it's kind of the same stuff you expect, you know, I to see Media tech processor support a lot of driver cleanups We client conic Pangu Tron is doing you it turns out that if you do a reaffactor you can get a pretty high commit count out of that So so that's just kind of interesting stuff. So now we'll talk about some technology areas We've noticed any an interesting talk from T.I. during LPC 20 23 so already a little bit Sometime ago about Looking into how we want to improve the boot time of Linux systems on embedded on on automotive Systems in an automatic in automotive you do not want to wait seconds long seconds until your system is ready so they were thinking about Using the bootloader to do device initialization Then it requires to pass the whole initialization to the Linux itself And Also working on Pre-initializing the market drivers. We know that synchronization of Ethernet that takes a little bit of time So if we can do it earlier, then we have it ready when a Linux needs that you can see the slides and the videos for this talk Then on the graphic sides, we can see on open source Vulkan driver fan video that Has been announced by Collabora and They have an article about this that you can have a look Then on the file system and IO This work on empty the SPI nor so Locking work to support reads While writing that could be interesting for embedded use cases area FS and and husband on the cache caching a speed and the block IO speed up work that Is the problem we haven't checked it had if it has been merged Then on real-time preemptive RT that is not the year when we announced that the whole preemptive T has been merged Not yet But what is left is? Is not that much and it's applying cleanly The big blocking point is the print K Apart from that there's a little bit of TTY serial DRM and Few things here and there are related to architecture But they're really the really the elephant in the room is is the print K on the security side So we have talked about the stacks It turns out that There was information About the kernel addresses from the elf nodes that was exposed in the system there was a work how to to remove that information because it could be used during attacks and BPF meets security in In many places there is a talk yet again about unprivileged Users being able to run BPF modules Well, we are going to see how how it goes Then also on the security side CV handling the big news at the beginning of this year The Linux kernel becoming a CNA. What is CNA? CNA stands for CV numbering authority If you are CNA you can create or assign the CV numbers So the identifiers for security issues The process itself Has seen a lot of discussions about the problems it has The big the big thing that the kernel community has seen that sometimes the project members disagree with the researcher or the project members disagree with them between them about if it's a really Security issue what is the impact and so on and so on and if you are a CNA you control what is assigned to your project so the kernel community has decided to make the pass and become a CNA to be able to design to assign vulnerabilities to the kernel before it was done by Well, basically everyone So they have decided to have a policy that is a little bit different that different that all the CNA policies So they are the assigning CVs only to fix the issues. They have a plan to invalidate Entries that were assigned by someone else. That is a pretty complicated thing to do talking from experience And the kernel community is not the only one that became a CNA in the recent months there's also gdpc there's curl python and The list is growing Kind of every week If your project is not the CNA the paperwork is not complicated. It's the cost is zero in money So you may consider doing that to to control what is assigned to your project Now we will pass to the System size Okay, so now I go on a little bit of a rant because The slob memory allocator was removed like okay fair enough. There's probably not that many people using it my main my main problem with that is that It was it was between the time it was deprecated in 6-2 and the time it was removed 6-4 It was only like two kernel releases and there's this like no way I'm sorry, but there's no way that the embedded people were going to notice that You know we should we're supposed to be working upstream But a lot of it, you know, it's not going to actually impact our our products for usually were a year behind or two years behind And so I just didn't think that was enough time. I would have liked to have seen a longer deprecation period And then the slab memory allocator also got removed in the same year So only slob remains. This is not the end of the world I think most people were using slob anyway and there is a config slob tiny option It's just not going to be as small though as slob was and so I I am not doing MMU list stuff But I worry that some of the people who are who are trying to get Linux to run on the deep deep embedded You know lost Facility here that was beneficial to them You know, I understand kernel maintainers don't want to maintain multiple memory allocators And so I understand the direction, but I thought it was unfortunate Really interesting in thing in terms of testing that showed up. This has not been mainline yet, but Kolobora Did this thing where they are submitting the CI pipelines upstream to put into the kernel source tree And Lena's initially kind of rejected this Didn't want it at top of tree But there was some discussion it looks like it could go in not at the at the top of tree in a in a directory Or he doesn't really have to pay attention to it. This is interesting because This is just one CI system that people use GitLab CI But the way they structured it was they actually had directory they put it in its own directory And you could envision other CI systems also putting materials parallel to this upstream So if you're using GitLab when this goes upstream or if it if it manages to get upstream You'll be able to just run CI right out of the box on the kernel source tree without having to add additional stuff And I think this is actually a really good thing I think other other CI systems may add their materials We may find ways to share CI materials And so I mean we have case self-test, but we don't have like kind of framework integration That's upstream and there's no reason not to be sharing that in terms of tool chains GCC was released 13.2, which is the current one was released in July. I didn't check in the last week 14.0 is supposed to come out This month sometime. I may have already come out, but if not, it should hasn't come out yet. Okay It's supposed to have more improvements for the static analyzer some actually really cool stuff Where it'll find problems before you even run the code so You can look at GCC actually as a test program It finds more bugs than probably any other test program out there, but luckily it finds them before they make it into your code and then the other tool chain that's used a lot is LLVM and there's kind of a They're improving their support for architectures risk 5 new ARM processors and Interestingly LLVM is not just used on the kernel. It's increasingly people are using it to Do whole distributions There are some resources I'll just I'm not going to stay on this one, but Bernard Did a great talk at ELC last year talking about a bunch of alternative tool Toolchain elements highly recommend that talk and then also You may not know this, but bootlin has this area where you can download Toolchains Pre-built tool chains that they've gotten from build route.  The reason I'm mentioning it because I had a board that I It's a long story, but I needed an old compiler and These guys had a pre-built one that was lying around so Anyway, so and they're they're available in multiple flavors, you know for different C libraries and stuff And then on the tracing front I talked about perf data type tracing We did and then eBPF Increasingly is used for tracing and other things Including on itself. So there's now a tool Called BPF top which does pretty much what you would think it can monitor BPF programs and tell you how much they're using the system And they're overhead and stuff like that. So So that's a nice bit of introspection there and then I'll turn the time over to Marta for more stuff On the other project the 50 is coming any day or any hour right now Officially expected in April that will be the new LTS and what it has That is the support of the generic arm 64 for system ready setups So the first first time you can build a generic arm image and boot it on different platforms The default corner is 6.6 and there are hundreds of Package updates as you would expect the previous latest for Yachter is the 4.3 non-built Rest in November this one we've kind of six point five on the industry side Oh, we have chosen three events The first even the the C array that's the first time I'm talking at this conference about C array I've has been I have been stopping myself from talking about C array What is the C array for the US audience you may be less aware? C array is a mandatory cyber security framework for all software and hardware in they put on the market in the European Union it has Many requirements some examples of them being that Software updates will become mandatory by default five years for everything Every window will be obliged to do due diligence of all components that they use including all open source components There are requirements on documentation. It will be mostly based on self-assessments, but it's a little bit more complicated in that It it's extending the C marking it is Nearly done The final publication is expected in October 24 Then it will be three years for Up to full application. So we are arriving at 27 But three years will not be that long in this case, especially that Standards need to be created And the C array is referencing a lot of standards and they are not written yet So during the first part of those three years, they will be writing and approving of all those standards and Only then we'll know What the exact requirements actually are? Then HDMI when we are back we understand that's in some places We are lacking open standards in the case of HDMI the lack of open standards Causes the lack of a possibility to write an open source driver for HDMI 2.1 It's a little bit of impact for embedded development Yep Some people still haven't understood the open standards and the nice most helicopter probably you have heard it had an Accident the blades have been damaged and it cannot cannot the nice helicopter cannot fly anymore NASA announced end of the mission Unfortunately, but probably we'll see Many other projects and we are releasing many other projects with Linux in space happening back on the community to theme Okay, so I'm just going to talk really quickly about conferences and about the elinux wiki So conferences embedded Linux conference, okay, so our grand plan to do it once a year it only lasted one year and There's a lot of reasons for that. I'm not going to get into all the details You can ask me in the hallway and I'll explain some things but One can view this as a positive thing So we're in the spring in North America in the fall in Europe. That's the way we had been for quite a while We have this one going on right now. The next one will have a mini ELC in Vienna So it's only going to be two days and two tracks a little bit smaller in scope But the CFP is open for that now and the deadline is the end of this month. So So please consider Submitting some entries for that Okay, so why the change away from EOS is a standalone event Again, I don't want to get into a lot of details, but it had to do with some some aspects of sponsorship and some aspects of attendees we did a survey a while a couple years ago and found that about 50% of our end users like to be Co-located and about 50% of our users prefer a standalone and so it's like It's hard to please both camps Hopefully This this year. I was actually very pleased with the technical content. Sometimes open-source summit I don't know if I'm even allowed to say this They don't have like the technical content, but they have had this time. There's been several really good talks and Linux con and so Sometimes it's good to be co-located. You get some extra benefits from that This is in red. I believe yeah So I'm I'm planning an embedded Linux micro-conference of plumbers, which is also in Vienna That's one of the reasons we were not going to do ELC in Vienna as we're going to kind of replace it but plumbers and ELC are not the exact same thing right plumbers is kind of more of a deep dive on a Individual topic. The reason this is in red is because it's been submitted as a proposal But has not been approved yet. So don't don't count on that happening But I'm hoping to organize that I My personal opinion is that the plumbers content and embedded has been a little on the weak side and I'd like to beef that up Get more involvement with the the kernel engineers on on deep technical topics The other thing is the elinx wiki, okay, so we were concerned about the elinx wiki because we lost our funding for Our administrator bill trainer turns out bill was hired as a full-time employee of Linux foundation so and This is not the only thing he does, but he is paid to continue working on the elinx wiki As part of his job. So this site it could use a facelift There are some areas where the content is kind of stale But it does have we continue to maintain all of the slides and all of the videos For every instance of ELC and so there's a lot of great material out there Also, a lot of board vendors use this site to publish, you know technical specs and stuff. So I Think this site is underutilized. I'd like to see more people using it I am looking to get some volunteers to maybe help walk through the site and try to clean up and Update portions of the site. We actually do have some money the last gasp of the of the project that was funding this Took their remaining funds and they put it into a crowdfunding site So there is some money that we can use to add content to the site or do some other things And then I think I'm gonna turn the conclusions over to Marta. Yeah The conclusions is overall If it's not the year of the Linux desktop, it's definitely yet another year of embedded Linux We have embedded Linux deployed in billions of devices We have the core kernel supporting Very well the embedded use cases But of course we are getting new devices. So new drivers new things to upstream to modify to optimize There are investment needed in the infrastructures infrastructure and community not only talking about XZ but All all the related subjects and we are Looking for your ideas what we are missing for the support of embedded Linux and what Yeah, and you can you can Talk to us During the hallways and I think that we managed for the first time to have first time in the history of this talk to have Time for questions. It is So if you have questions, we have a mic People are not used to having questions to this session Okay, if there are no questions, I'm gonna I'm gonna talk a little bit about the Mars helicopter Because I think it's really cool. I just found out this week so the Mars helicopter is is Grounded it cannot fly because of the damage to its blades But well, I got to be okay You guys are getting a freebie because this is one of my questions later this week in the closing game So But the base station there's a base station on the rover That is also running Linux that continues to run and the helicopter itself also continues to boot up every day and it and They're going to over the over as long as it continues to do so they're going to use it to get temperature data and They're going to take pictures Because the camera is still working so it continues to be an active science instrument on Mars running Linux and they say that it has enough capacity in the storage to Handle 20 years worth of saving data So it will soon the rover is soon going to leave that area and it will not be in radio communication It can't send that data back to earth, but hypothetically When some human lands on Mars or some or the rover comes back around for a pass and gets near enough for radio contact They can receive all that data. So it's still doing science on Mars. So we're excited that we still have Linux running on Mars That's that's all I got to say about that. I find that so exciting that I just can't stop talking about it Any other questions about any other topics? Yeah, that's dangerous. Oh, yeah. Oh, I think it's on now try it We get the mic on Should Okay, just repeat I'll repeat the question or Okay, so the announcement was that I think we have a new mic back there, too Yeah, that's weird So the announcement was that kernel recipes will still be in Europe every year and they're Embedded recipes. Yeah, embedded well, so kernel recipes to embedded recipes will be in Europe every year, but they're adjusting their dates to not conflict with Embedded Linux conference Any other any other questions? There's also you for those of you in Japan make sure you go to the fan technical jamborees We have those about once a quarter and there's Interesting content about embedded in on those as well Okay, if there are no more questions, then thank you for your time. Thank you everyone Hello, okay testing Okay, so Good morning. I think it's time. We should be starting the talk My name is Laura more I work for a guy and today we're talking about Automated testing of embedded web applications I met the guy there since 2020 working in the webkit team Mainly I was working on key rate tasks and now there's I'm working mostly on the web drivers support for that be okay, but you take a and in the past I have been always Since the beginning of my career I've been working on offices tools like the in the old Nokia times of my ammo in the Python binds and we had back in Brazil Port of webkit called nix it was spread cut similar to WP I streamlined port a keyboard and also worked on the Enlightenment Foundation libraries also in the binders for JavaScript and C-Shop Igale is a Open-source consultancy. It's over 20 years old. It's from Spain, but we are fully emotes nowadays We are working on that cooperative. So we had a flexi instructor We are mostly known for our work in the web browsers and giants It's the main contributors to webkit and chromium outside Apple and Google respectively we also have been working on gecko and servo recently and Not just browsers, but we are growing our contribution in other kind of projects like different JavaScript and Giants compilers and lot of projects in Jeff or Jay streamer and a growing number of contributions to Graphics drivers in Mesa and the kernel due to our open source background we are very proud of our working in the Different working groups to standardize the technologies to work together with others and move the technology forward For today, we will be talking a little bit about waters. What do I mean when I talk about embedded web applications some challenges that we face developing them and testing them and how browser Automation can help you dealing with those challenges So why web applications? web applications they are like my colleague Adrian Mentioned in yesterday talk. They are eating the world. They are growing everywhere because they enable first development cycles the developer can See what's happening quite fast Even with the added complexity of today's JavaScript tooling it's still very fast to develop new applications It's also enables fast deployment like if you are using a remote server to serve the web application You can just update the new version on the on the seven instead of having to To deploy directly to the physical device with either a new major or another kind and It's also a larger existing of both different tools different technologies Vibrant community of the developers and It's growing but in real life web applications you write the answer and you run mostly anywhere for example, we are back. It's looks like in some ways we are back to the works better with browser days and Why does this happen? because despite all the work in the standards, we have some enjoying differences Blink which enables Chrome based the browsers might support Difference Have a different level support for a given feature to advocate versus Gecko. This is one of the most common kind of differences especially when we have some new shiny new standard that one of the engines might be a little bit more conservative to to implement but even if we get the same engine we might have browser difference some of the Features might be Implemented at the browser level So this we might have some cases where the Chrome behaves different for rave that they behave different for edgy and Because of the web kit we might have so far different from the number of it and Also, we have the trouble with fashioning because the one system might be using a Different fashion with a different kind of level of implementation for a given feature There's a great talk linked from the last year's Selenium conference by David Burns That talks how those kind of differences appears as browser engines are different from desktop browsers in mobile browsers Because it's also about other Is the web application running on a desktop like machine on a tablet on On a phone The size of the screen it's 4k Full HD. Are you using a scale of native resolution and What kind of interaction the web app will have it will be using if we just a display will it be using mouse or gestures and Given that what do I mean about embedded web applications? just Question first who here already works with some kind of embedded web application, please raise your hand Okay for people So embedded application differs from generic web applications Because it's not just a generic website that I have That's running on a generic mobile browser. Let's say just my sister just lost from me with some kind of stuff I Mean that I Embedded web application it will load some specific resources. I Can control the user will not get the access to the URL bar and load some malware website In some cases, I might even be loading the resources from Locally in the machine to isolate from the external and network and We control what kind of browser is running the Web application so we don't have to deal with some esoteric lab different browsers and ages and I think the most importantly the web application can explain the browsers with new APIs to access some Sapses from the system But even with this kind of differences in real life We might have different devices with different hardware capabilities for example different CPU Different GPUs the even different CPUs different screen sizes. For example, they are dealing with Project that the same application is expected to run I max 6 and I max 8 with very different GPUs with different levels OpenGL support. So this also brings some complexity to test Also, you might have different browser versions because Due to hardware capabilities or different device generations, we might have let's say another Webkit version running on a other version of the project, but it's still consuming the the new version of the web application But thankfully this is still less variables than doing generic website development and as I mentioned generic websites are usually sandboxed the browser prevents them from Accessing the system only through some very specific APIs defined by storage groups and even so it's still very limited But in better devices, sometimes they need to interact with the system. For example We have a pro we had a project where they Through a web application you control the mixer a blender for to So you need to be able to make the web application interact with the system and As such testing the web application in a live system is a must There are cases where we need to be able to test the live The web application in a running system It's big this differs from also from generic website development because usually generic testing tools for Websites they kind of launch a sandboxed browser just to start the load the page run the test Okay, it's over and in some cases the browser starting the the process of starting the browser might connect some systems in Some services in the system and we need to be able to to see it working on real life especially on Entrant tests for for those applications But then How do you test Web application first of all there's a confusion between automation and test Usually when people talk about automation they already may always lead to lead to Talk about testing But automation is just a tool that we have to Control a web browser For example, we can use automation tools to crawl the web it's having a Use it extensively in the past to Crawl the web For let's say getting some informations. I remember some friend friend that they had a project that was crawling some government sites for information We can use automation to field access for example to do some boring stuff but also we can use automation to running test steps which is our focus here and originally Automation was Designed in a way to mimic user actions for example navigate by giving URL find the input for Given form with the valid name Enter the text That input Then find the boot with the label confirm and click this button Also click this button search for the message user created so the whole most tools of the the tooling of automation was created originally mimicking user Actions and these led to some design this decisions that Still reflect today So in the beginning It was about early 2000s that the first real efforts appeared Namely with the Selenium and the web driver projects that after some time they measure together and we can what we call today the Selenium web driver that driven the development of those tools and And it was widely adopted, but it's still kind of depending on some browser specific Communication eventually it became WTC standard in 2018 So the browser zone had only had to implement the standard and the tooling that sits on the client side could talk with to any browser and And these web drivers standards nowadays it's supported by all major browsers and giants Chrome WebKit Firefox and it's what we call today by web driver class and It's this protocol that Mediates that the communication between the client the out the automation script or test script and the browser it's based on HTTP and It allows remote connections which which is interesting for the better those kids for example the developer can be writing his automation scripts on the desktop normally launching the scripts and it will connect it to a browser running on a embedded board like a has very pie or AI max and as it's based on HTTP under the hood the user sends comments to the browser which Sends the response the HTTP response back to the user With the result of the action This worked well for some time, but Just mimicking the user nowadays. It's it's not enough why because For our applications logic has been steadily moving to the through PEM We have huge applications nowadays running on the browser instead of thin just thin layers and Given the nature of JavaScript lot of synchronous tasks and the synchronous tasks does doesn't mix very well with the Common in the responses scheme of the original web driver protocol and Pauling is tricky in those cases because for example you send a common to do something in the web app and doing this will trigger some Task on the browser that will take an ink no time to finish The automation script is waiting for some kind of signal, but the browser can push back so the The web driver classic script you will be Pauling checking is this happening is this happening So this introduces some kind of sampling error that for example, you might be waiting for a specific value That might be in the past you might kind of lose some exception that might have happened in the browser So The web driver class key has this kind of limitation in the meantime also, we have been developing new tools like the Chrome developer tools the remote Spectrum and WebKit and Those tools are very very powerful they enable true two-way communication with the browser because we will open the For example in WebKit with the remote inspector you can open the Spectrum in a in a different browser targeting a Given browser that you want to expect and it will react to things that happen in in the browser and So given those capabilities we end up developing automation tools based on those protocols and one of the most famous ones on top of those protocols, for example, we have the Since the Puppeteer that's based on Chrome Developer tools, which is the protocol that Chrome uses to do some debugging inside the Chrome based browsers and those kind of tools Enabled new capabilities that help it address those limitations of web driver class, but back again They were some kind of browser specific solutions Based on those experience the industry sit together and we are Developing constantly they what we call the web driver by die protocol by die for bi-directional and It's already press well supported in Chrome and in the chromium engine chromium engine and in for Firefox and WebKit we at Igalia have been working to add support for it and It's progressing Prior to also be implementing the coming months under the hood Web driver by die As I mentioned it enables true two-way communications between the communication between the user and the browser because the user now can still send comments to the To the browser, but the browser can push back information to the user script and Web driver by dies implement these by using web sockets instead of HTTP. It's just just to set up the Connection It's built on top of the experience with a driver class. So if I drive a by die We should be able to do whatever we could do with web driver class For example, you can still perform user actions Like I said clicking getting information from the page Find elements so on but also you can move toward the browser state without polling Which is the game changer in for testing for automation What can we do it is we can mount or log messages by the heads in specified how the Client can subscribe to events and whenever there's a log message from the browser for example console.log message or a javascript Exception you can assign some cowbacks to be called from the client Also, we can observe for don'ts changes in the web application Which is something that was also missing as I mentioned you have to be polling now You can just react to to those don't changes and for testing it's also Very interesting that with by die we can intercept and mock browser requests So when the browser Going to start to fetch a network resource by die allows you have to have a by die allows you to Intercept that request create a mock response and the browser will use that mock at response so this also helps and Testing and as it's based on web drive a class key by die also allows you to connect to him or Browsers running on other boards And what about some browse automation tools because by die and web driver adjust specification just a protocol between the those two The area they have an implementation a reference implementation inside the web platform sample story called It was formally called WD client, but it's just a t-layer on top And Given that we now have That in specification for web driver and by die it makes it see easier For two developers to create new tools on top of those the most famous one probably is Selenium because it was let's say the one that started it all in the in the past and It deeply influenced the web driver class key standard Do it to these it was the first one widely adopted and And it's has implementations for JavaScript, Python, Java, Ruby, C-Shot, many languages And Selenium is not just the Selenium web driver library It's a huge project with a lot of the different tools like the Selenium and Anger for to handle different browsers And now so the Selenium grid where you can assign different Different workers that can run the test is in parallel to reduce the size The the time that they spent running those tests Another tool that's getting traction is what we call WebDriver.io it's also based on the W3C WebDriver protocol and It's written since the beginning in JavaScript. So it is deeply it's very well integrated with the The let's say the workflow of front-end developers So it helps it makes it easier for people doing the front-end development also use it for automation and It also right it it has built-in support to run with the frameworks behavioral driven developer frameworks like Cucumber, Mocha and Jasmine so It's also a plus that it's tightly integrated with the front-end development environment Another major player that's also gaining traction is Playwright That it has some oranges in the Puppeteer So it's tied to the kind of tied to the Chromium DevTools protocol Nowadays it's being developed by Microsoft it's under active development. We at Igalia work with Playwright in the especially in the supporting WebKit for it it has a Very interesting feature called outweights that Before performing an action the outweights features it performs some checks In the in the given element for example, and you go to click a button It checks if the boot is visible if the boot is enabled If it's visible, it's not obscured by some kind of other element that could Receive the event so it greatly helps organizing the and Reducing the flakiness of the the tests But unfortunately Playwright it depends on patched Firefox or WebKit versions for the Generic web app the developer this might not be much an issue but for if you we are developing a An embedded version of WebKit basic browser for example We might have some downstream patches specific for the platform and the WebKit patch for playwright is quite huge so it might add some complexity to testing Now we Do a recap of a few Points many points of the the talk Embedded web applications they usually run on more tightly controlled Scenarios than generic applications. So these greatly reduces the complexity of testing but at the same time that we need to test how it integrates with the With the system so We might need to to run different kind of tests as we might connect to The system running on a board. It's not that we can just do some Hemo testing on the desktop of the front end developer and everything is alright But thankfully especially the new protocol web that everybody has some Features like the bit by the bidirectional Communication that enables deeply inspecting the the the status of the web application within the the target browser and We can better test it That's it so Questions anyone Yeah, okay, the question is about maintenance of the automation script is how Often they change how well can we maintain the intro the lifetime of the application? One of the I think one of the important things in these To maintain this is the coordination between the automation team and the people writing the web application one one fragment pain point for example is The selectors used in the web application to identify the the a button important a field that we want to test especially nowadays with the Those frameworks that move elements in the dome around all the time for example some People nowadays are a bit wary of using XPath to select so The both teams need to agree in How how are we going to label to identify the the main elements to be to be tested there? I think there are some There are already starting to appear some AI power tools. I think to Identify to that visually we identify the the elements What is I think mainly it's something that need to be agreed upon between The people writing the tests and people writing the the front end code. That's why probably using the JavaScript based automation tools will help when talking to people from writing the JavaScript based web applications to make easier to make the Web developers help you if testing did find the the points and such Okay, the question was About integrating those kind of automation based tests into the pipeline to to make sure that you're taking account to the the The duration of the test is to make sure it integrates well with the the pipeline For this kind of tests the automation tools, I think they are let's say Very agnostic in the sense of they can be integrated in various kinds of pipeline For example, we are working a project that will be using cucumber to to run the tests So to describe it the tests I mean so they'll be written in that Language and the steps the test steps will be used We are still defining the the kind of the library that we be using either Selenium or web drive IO or other and In turn those tests that those and cucumber tests that can be integrated in other kind of testing frameworks that the company use they I think the main point is that the The way that web driver is designed allowing to connect to a running browser To a remote browser, etc. It's also great flexibility to choosing the to to using it with the tool that's already being used being inside the Organization, I think the the next talk in this building will be exactly about some testing tools to to integrate the To be running on embedded system, so probably the automation tools could be also be running on over Okay, so this is it Thank you very much and See you around Hello everyone My name is Paweł Wierczorek. I work with Colabora and I've been involved in testing automation efforts since around 2016 Maintaining various parts of automated testing laboratories ranging from devices under test Maintenance as well as their supporting hardware through server side components profiling and development Up to co-architecting custom solutions in this domain Today, I would like to share with you some of the experiences I had with the various Various software solutions and some of the encounters or let's face it Straight-up mistakes made along the way I would also like to share with you how you could make use of these lessons learned In your laboratories and your workflows As well as maybe having a better start in some of the topics in the domain of Automated testing labs. You might have not given a try before I Will start with a short introduction to why this talk Was created next I will move to describing all the candidates that I Chose for comparison After that, I will also elaborate a bit on the use cases I tried to cover and then I will give you a short summary of My experience with all those various software solutions Finally, I would like to share with you what might come next What would be my recommendation what might come next and also some final thoughts before we move to Q&A session For now, let's start with motivation for this talk. Why? Why we're all here? And let me take you a few years back to the ELCE 2017 in Prague where the topic of comparing strengths of various solutions got a bit more attention during birds of the feather session called farming together Led by Andrew Murray It was for me one of the first times when I started looking outside my bubble of automated testing laboratories Or maybe before we go there if you haven't seen this session, I Encourage you to review the recording From the link down below However, it wasn't the first or the only effort aimed at working towards experience sharing in this community There were others like two automated testing summits That were held in 2018 and 2019 with that 2019 not only covering Status updates for various test lab stacks as well as survey conducted by Timberd the 2019 event also had multiple talks on various tracks ranging from board and lab management Through covering test definitions up to most common QA systems and their issues It also sparked many follow-up actions like for example open automated testing standards Call which is held every month and provides a forum for discussing various topics Or the workflows mailing list just to name a few of those follow-up actions What I believe is that these efforts have shaped testing automation community To what testing automation community is today and Actually still play an important role in developing new solutions improving existing ones and onboarding new engineers to the topic of automated tests during the previous Embedded open source summit in Prague. I had the chance to discuss the topics that were raised in 2017 with Andrew And we were wondering whether the questions he asked back in 2017 and the discussions that followed Needs revisiting and if so What would be the best way to do it and Let me be crystal clear with you I cannot express how much I appreciate continuous work and support from members of this community and All that All the help you show on the daily on the daily basis Sharing all those Insights of what you develop what you use what you continue on improving is invaluable and I realized that many of you have gone great lengths to help others to try different solutions out and That is actually exactly what I intend to encourage you to do today If you're at the start of your journey I would like to provide you with some hints on what the available solutions for you are And how well they can fit in your requirements If you're already maintain a test lab or some of its components I would like encourage you to take a look at your solution then check out how it compares to other solutions and maybe think of the ways how both of them could benefit from each other and These actions were actually what I've been up to for the last couple of weeks But I made a few assumptions. I wanted to just start small maybe with a virtual device only and To make things a bit more challenging. I also Decided that I would like to Work with either no or as little as possible of external support What I meant by that Was that I wanted to find out how easy it is to to access and use the documentation already available when starting from scratch or Without much experience in new testing laboratory. I hope that by the end of today's talk You will have some ideas how to approach such a challenge yourself and you will feel encouraged to do so But please don't make a mistake that I did of Trying to tackle this challenge alone, but we'll get to this in just a second First let me introduce you to the champions will be taking a closer look at today And I would like to make a quick note that I chose Those only as an examples of various paradigms You can go with when creating or maintaining your own setup If you see an important one missing, I will be more than happy to learn about it And we can discuss those as well either during Q&A or we can take it offline Also, I realize that some of them support multi-paradigm and Certain functionalities can be achieved in various ways. I'm not questioning this at all. I Will be just highlighting what I really enjoyed about all those solutions the most and first Candidate is lava which in just Maybe over simplification but It's mainly set of Django apps that connect to postgres database So obviously that's a Python based solution it Also provides an existing library of YAML test definitions and is well equipped with dashboard results querying helpers and various other tools its performance in is continuously profiled and pushed and By that I mean that the performance limits of these solutions are continuously pushed forward And it supports quite large workloads. You can even say that it's Cloud deployment ready as well as it's well integrated with various external tools like Prometheus for monitoring its ongoing Its ongoing work or GitLab for submitting Submitting tests to be executed in lava The next candidate is Labrith, which is also a Python based solution Which introduced a place term, which is not only device under test that you would use in your testing But also accompanying support equipment it already provides supports for large set of such hardware and Also provides an integration for pie test for writing test scenarios Interactive use is actually a first-class citizen for this solution And what's also important and interesting is that there is set of Tutorials on how to get started with lab grids available as a playlist on a YouTube channel of the creators of this tool and Next one I Had a chance to play with is Bordswarm, which is relatively new solution and in And other Then the previous ones this one is based on the Rust tooling It also provides an interactive access to your boards But it is focused on providing Hardware abstractions to unified how you can access your devices for example from storage point of view or serial Console that you would like to access So the test definitions might not necessarily be useful in this context instead It is meant to be a building block for more complex solutions Or maybe just a unified base layer replacement for existing ones It also follows the UNIX philosophy and UNIX principle of doing one thing, but doing it well. I Also wanted to take a look at More at a solution that is aimed at more Enterprisey Let's call it that way environments and for that I chose beaker Which again is a Python tool It's not necessarily meant for embedded environments, but a similar set of Functionality and requirements on its side are also present. So power state toggle Some additional supporting hardware And so on this is also a fully fledged Solution with various dashboards and you guys available, but there is no obligation whatsoever To use all of those components if that's not your thing You can still benefit from its subsystems like for example cobbler just for having a bare metal Provisioning layer if that is your requirement. I Also gave a try to custom hardware assisted Solutions and I realize there are various solutions available for that ranging from USB maxes Micro SD card maxes to more complete solutions and one of those Which is Testing in a box by coating was the one that I chose to give a try I could try to Also specify what language it actually Uses for its supporting software, but That's not the feature I would like to highlight here What what is actually a really interesting is that the designs for this custom hardware and it's a software integration layer Although tailored to specific system requirements were kept generic enough and modular enough that they should be that you should be able to Reuse those tools in your setups as well But before we even think of trying to do so, let's make an important note that it's much harder, obviously to do iterations Using someone else's solution unless it ticks all the boxes for you There will also be some invested Some investment needed in both costs labor know-how Before you could even try to incorporate those in your setup, but it probably can bring nice returns in long run Also, this is one exception to the rule. I made that I would like to try all those solutions in virtual environment Because it's simply not applicable to Do such an approach? Another remark I would like to make is that I did not manage to get my hands on Testing in a box hardware. So I made use of what I have what I had available Which is a simple micro SD Demultiplexer and custom PDU To sum it up We've got the five contestants that That started in at various points in time as you can see most of them are Python based but not all of them and the test definitions or task definitions are Described in various ways I Would also like to Make another reminder that I do realize that That for example lava provides hacking sessions or multi-node tests And also lab grid works with custom hardware for assisting its test runs But I chose those Candidates as the just representation of the use cases we will be covering and now let's move to those use cases the first one which I believe many of you might be interested and Is interactive access to your devices under tests most often it's just to get an access to device that is Not available in your physical location, but instead it's somewhere in a server room on the other side of the globe possibly or just Downstairs for cooling purposes If more than one user wants to access Such such a device It's also important to remember that there will be a need for some reservation mechanism For example to time out sessions or just Kick users that no longer make use of device they reserved Also just Access to serial console is a good start, but for embedded purposes some additional control would really be useful for example to GPIOs to Massive storage or to additional measurement equipment like ACME boards Next use case I covered was patch processing of submissions And I just tried to throw a bunch of test submissions and see how well it would be handled by scheduler that That was either built-in or external to a solutions to those solutions of course some of the candidates I showed you earlier are Do not have an built-in scheduler, but instead only provide a reservation mechanism and for those I Cannot Recommend you know with how well this scheduler works because it's just an external tool Other than that it would be very useful to have some notification mechanism or preferably even a callback one and this is probably mostly used as Acceptance tests on wide range of device types when a single result does not really matter before the whole test suite is completed Up to the last device you would like to check Next I will also wanted to cover version control systems because while many embedded projects and Developers work using complex dependency structures some of the projects use just a single repository and actually need per patch verification in Such cases it might be a deal breaker if the test lab system integrates with your VCS or not Otherwise, it will also only bring unnecessary overhead to the projects you develop. I Also realized that all those features won't you won't get you anywhere if a new user gets stuck at first try and I Was also curious whether some of these solutions already come with some evaluation Environments to get started quickly and try out the basic features some of them in fact Come with extensive documentation some like I mentioned with tutorial videos but These and I guess you'd agree with me work best if they do not overwhelm user at first glance It Also might be a bit hard to find the right pace for the user to learn a new tool And from the developer point of view and I mean developer of such a solution It might be hard to provide Such a documentation, but I believe it brings many benefits in a long run for example by creating Scenarios for basic use cases If you automate them you already have a verification environment for your test lab solution So just to recap quickly the aspects that I analyzed was interactive access batch processing version control systems and integration with them as well as documentation and First steps that you would take with such a solution Now, let me share some of the experiences I had with those test lab software for interactive access Lab grid and board swarm performed really well for me. I enjoyed using both of these solutions And although lab grid provided much more feature rich experience Board swarm allowed me to keep things simple. I Realize I still have to evaluate for example new approach to hacking session in lava which now is teammate based But that's still to be evaluated Also lab grid ticks all the boxes I mentioned earlier When I described what I would like a solution that provides interactive access to be equipped with As for the batch processing Lava was the solution that ticked all the boxes for me So it provides it provided me with a scheduler with the right priorities with call back mechanism and even Q clean up in case the test submissions that I made Were waiting in a queue so long that they became no longer relevant and I know that it might seem like recurring theme that those solutions tick all the boxes for me, but I Think you would agree with me that these are very mature test lab software and That is why They they work so well in those use cases as for the solutions that do not come with Internal scheduler I had to pick one from the version control system that I tried to use with a given example and That's probably the best way to work with such Solution for yourself just pick the one that you would use as your version control system and Speaking of version control system When when we discuss the interactive Mainly interactive use for the solutions the Pipeline provided by your version control system are Quite a good candidate for a scheduler together with a reservation system in lab grid or board swarm for Lava the integration with the VCS Could be made just by submitting jobs and either poll for the results or use the Previously mentioned notification mechanism or even use custom runner integrated with VCS You might have noticed that Some of the text on the slide is slightly different color. I linked all the relevant additional software and If you'd like to know more about these solutions go ahead to a slides on the schedule and Try those links out as for the Tools that were built around version control system like for example testing in a box the key benefit for me was having the whole config of Such a solution kept in a single place So that I did not have to actually care Where do I get test definitions from? How is my laboratory configured and so on? On the documentation front I have to admit that the Beaker documentation was the most extensive one I came across Ranging from setup documentation through administration one up to user guides Not far behind was Lava document Lava documentation with both classic one and the new one hosted on read the docs and I think I already mentioned it once or twice that I really enjoyed the tutorials provided by lab grid development team And going to Examples that you could reuse as a basis For your own setup and they're testing in a box Initiative provides examples in most of their sub repositories While lab grid just holds an example sub directory in the main repository, which is also fully equipped And Lava already provides a library of various test definitions that you could simply reuse Many of those solutions also came with ready-to-reuse evaluation environments beaker with a set of Ansible Playbooks to just provision some virtual machine or Possibly other system that you would like to try it on As for the lab grid My colleague Chris Obart created also a set of Ansible playbooks to try Lab grid in your own setup and you could check out his talk from Embedded Linux conference Two years back Linked on the slides As for the lava you've got multiple choices Ranging from just a simple Docker compose which sets up a server for you to try things out Up to more device centric approach with Lava Docker that Generates actually a configuration for Lava based on the description of your testing environment and I what stood out the most for me from all those solutions was that some of the Testing automation software Makes sure that all the design decisions are well documented like for example lab grid and board swarm I was also really impressed by the amount and Specificity of the documentation provided by projects like beaker and Lava and what I liked about the testing in a box is that The hardware designs that were created for the specific use case Could possibly be reused and recreated in any other testing laboratory And now let's move on to what I would like to Recommend to you and I realize that some of them might sound a bit tribal maybe even Trueism like like a trueism, but I guess you'd agree with me that there is never a silver bullet in any of these approaches and if you ever find a Solution that is ticking all of the boxes for your requirements, then you could be called an extremely lucky Also each of the testing champions. I showed you might require some adjustments for your specific needs And that comes with some integration effort As well So it's really important to decide on what your key requirements are so that It's not just a guess what would what could work best for you, but it's a choice It's important not to over invest in a solution that could become a dead end for your use cases Also If it is a choice there will probably be a fewer integration bumps for the adjustment of the whole setup to your needs And that would also help you to set your expectations, right? So that no one ends up disappointed with any solution What probably could work best is a hybrid approach So for example Getting to work a board swarm and using it as a base layer replacement for some existing Testing automation solution that could make use of hardware abstractions or for example using lab grid with its dedicated hardware solution as well as Providing some additional integration to to an existing test lab like the git lab runner I mentioned earlier or for example exporter of various metrics to your Prometheus database But the most important recommendation I would like to Give to you is to not make the same mistake I did on trying to work everything out on my own the best decision I made was by it was to approach a User that already had some experience With one of those solution is one of those solutions when I really got stuck and almost gave up on Trying to to make it work Because it reduced It could have reduced greatly the time that I spent on trying it out for myself but also from a Viewpoint of a developer of such a solution it could provide a really important feedback on What works well what could be further improved and what use cases are Actually covered by a given solution As for the next steps I still need to upstream a few changes in my Evaluation environments, but only if they are actually applicable according to Core developers, maybe I made some mistakes Probably that's The reason for that is that I tried to do it without communicating first But those virtual only quick starts Probably could be beneficial for reusing if you Consider trying out a new software solution Also, maybe it would be useful to create a set of requirements for such a virtual only quick start environment but That's actually for you to decide and that's what I would love to hear from you Do you think that it would be beneficial? Would you like to use it? And if there is actually a need for that Related to it. I also Wanted to Provide you with some setup that is not only bound to virtual hardware, but Actually can be reused with the physical one. There were some previous discussions on choosing the right candidate for that, but that's also a slightly more complex solution to provide and Again, what I would love to hear from you is whether this action whether there actually is a need for Such an environment or it would never be used by anyone The last thing I had in mind is To try to scale up all of those solutions I know there are certain efforts in some of them to Keep it testing and Check how it behaves with higher and higher workloads and if you'd like to Get More details on this topic. I linked three different talks covering it for you to review the recording to sum it up I Wanted to show you that there are Many various solutions to choose from if you'd like to check something out and Maybe even migrate to a new one. I Realize this might be a high investment But I hope that some of the features that I highlighted Got your interest and maybe even encourage you to take up the challenge and again, I would like to make another remark that The biggest lessons the biggest lesson I learned is to try to engage with core developers and The community around given solution right away so that you don't get burned out. You don't Get stuck on possibly simple issues and You also provide a feedback to those developers and That's all that I've got for you today If you've got any questions, I will be happy to answer them so you mentioned a lot of upfront work to to Get everything working and that it is difficult how much time Does it take to or how many people does it take to administer these types of? Projects like a lava instance or so I guess that's hard for me to estimate and that's because I Did this experiment that I didn't want to Engage with anyone and try it only on my own. I believe that if you get some maybe mentoring or just Some hints from core developers that time to get your new system ready would be greatly reduced But for me that was just Fun experiment it was about two hours per week for a few weeks and That's what I ended up with Thank you Thanks for your talk Do you have any experience with some kind of peer-to-peer testing? You know often you have some kind of a device on the other side You want to test some interface and you might need to configure sites on things on both sides any comments on this So let me rephrase so that I make sure that I got you right You would like to perform a test that involves multiple devices under test that communicate one with another right, so first solution that comes to my mind is a Multi-node test on in lava, but I don't have much experience with that I mostly used the lava for just a single device under test As as far as I'm concerned although we would have to try it out we can do that in just a minute a Place in lab grid terms could also Provide you with multiple devices under test that would be reserved for your test run and Possibly communicate with each other And that would be the two solutions that I would try out first Okay. Thank you very much There's another solution that Intel had when they were doing their Android testing and it's called TCF stands for test case framework That one also has a lot of similarity to a lab grid and it also had the ability to Set up like a virtual network as part of your test framework and everything and that allowed you to do multiple devices and talk to each other Through a while it was in Python 2 and then it finally got upgraded to Python 3 So it's on github Intel TCF so that one's quite similar to lab grid though Thanks for your comment. I will definitely have a closer look at it But if there are no more questions Thanks for your attention And have a good one Hey, everybody getting quiet. I guess that means it's time to start so welcome I'll be talking about some adventures I've been having in the industrial IO IO subsystem and Of course what you all really came for hopefully something about some Lego robots So I'm David Lekner. I'm an embedded software engineer at Bay Libre Bay Libre is a consulting Firm work based in Nice, France, but we've got remote people scattered all over the world as you can tell I don't have a French accent We love everything embedded and open source especially Linux kernel Android we're starting to get into Zephyr and we also recently hired the code sorcery team. So now we've got tool chain services We can offer I've only been working at Bay Libre for about six months now So my background in Linux kernel hacking actually comes from playing with Lego robots the Lego Mindstorms EV3 that came out in 2013 runs Linux and I spent quite a few years hacking on it reported Debbie and Linux To it and I started sending kernel patches upstream and that's how I Learned everything I know about being a kernel developer maintainer and then more recently Another gentleman and I Lawrence Volk Started that project called Pybrics, which the more recent Lego robots don't run Linux there are more constrained microcontrollers, but we're running micro Python on them and Before that I was I actually started my career in industrial automation automating water and was what wastewater treatment plants. So I had kind of a bit of a non traditional career path to get where I am today, but it's all come together In the industrial IO system now, so I've got five little adventures to tell you about today If you're following along at home watching the reruns on YouTube or the PDF you can choose your own adventure But I've got a captive audience today. So we'll go through them all the first couple ones are going to be talking about Hardware aspects of what working on and kind of day-to-day how we Managed to get remote workers working on industrial IO analog digital converter type hardware and then the last three ones are a little bit of a report on interesting things that are going on upstream in the IO subsystem So for those of you don't know industrial IO subsystem is the corner of the convert kernel or Anything that's an analog digital converter or digital to analog converter hangs out that it's all about Measuring and signals and things like that So our first adventure the adventure of the remote IO developers As we mentioned we've got Developers all over the world at Bay Libra so our team that's working on this particular project of Upstreaming some support for some analog digital converters and also developing other User space tools. We're also doing a little Zephyr bare metal type stuff for the same analog digital converters We've got about six fully remote people a couple Hybrid that are in the office some days Wrote some days and one person the office full-time so We kind of like to share what we're doing and hopefully in the Q&A compare notes with anybody else who's Doing any similar kind of work like this to see You know, what's working? What's not working? so To start with Early on then on this project. We decided that we're going to keep all of our hardware in our lab in France So as a developer we're connecting remotely to a hardware stack like this and In the next slide I'll zoom in on What we're actually Looking at but for now. We'll just appreciate the aesthetics of it shout out to Fabian on our team for Making this all nice and neat getting all the cable management done so here we've got a Development board set up that people can remotely connect to to develop their drivers test them out And it looks pretty cool at night, too. So Here's what we actually have on each shelf there We've got of course our remote developer wherever you are connecting in through a VPN and I forgot my pointer so we connect to the Development board which is running Linux and So much for that idea So that's our z-board it's a development board that runs Linux and we've got an analog digital converter evaluation board connected to it That that's what we're developing a driver for and To be able to test it. We've got a what we call the m2k. That's like a Swiss army knife of signal generator logic analyzer oscilloscope all-in-one To generate a signal and do any kind of testing I have a motto that if we can measure it we can understand it So we need our instruments to measure what we're doing to be able to test and verify that our driver is working We also have a special power switch. This is something we've developed Internally at bay labor we call our co-pilot that can allow us to remotely Cycle power on our development board. So if it ever hangs crashes, whatever we can reboot it remotely and All these things are connected with USB to a little mini PC and We also connect to that through our VPN and On our mini PC We're running graphical tools to be able to actually visualize what we're doing the scopey is the graphical interface for that Swiss army knife m2k gizmo that we can use to generate a signal or Use it as an oscilloscope if we need to and then we're also using the IiO oscilloscope program to actually capture data From our analog digital converter to be able to see are we actually Capturing the same signal that we're generating to know if our drivers working or not so as Some of you might guess if you've tried to do work like this before Trying to do this remotely without being able to actually get your hands on the hardware has some issues with it So these are kind of the biggest pain points over the last six months that we've identified The First couple of ones we've more or less figured out The second two are still kind of causing us problems. So Sharing resources and when we first started we had at one point four different people sharing one laptop That was connected to four development boards and had about 20 USB things plugged into it That didn't work so well. So We sprung the money got eight mini PCs now. Everybody's got their own That problem solved just took a little money a little time But easy problem to solve the next one the booting files. We tried a few different things I've got a couple more slides on that. So I won't talk about the details just yet Then the last two is on our setups We often have to swap out that analog digital converter that We might have five or ten similar ones that we're developing a single driver for X number of chips and so we have to swap out the board to make sure it's working on all the different chips And so since we're working remotely Some we have to ping somebody in the office and say hey, can you come switch out the board for us? Which is you know kind of disruptive to them if they're trying to do other work, but that's also at the same time That's their job. So they do it at the Maybe the biggest pain point has been the debugging actual physical electronics issues that The issues that are like noise Cross-talk that's not a problem with your software. It's actually a physical electrical problem So the remote developer is Can't do anything about it at all. We have to rely on whoever's in the office to have the time and knowledge and skills to be able to debug that and so that one's particularly a problem because the remote developers the one who's been reading all the data sheets becoming the expert on that specific chip and the Person the office doesn't know anything about it and you're saying hey Can you fix this this for me and they're like sure yeah? I have to learn the whole thing how it works, but so jumping back to our Boot files this one we turn there's you know a few different things people recommend We tried using a SD card mux But those actually turned out to be more of a problem than a help we had lots of electrical issues They just didn't work The we would switch it and then not be able to connect to it anymore so we gave up on using those and You can also do like a TFTP boot boot from network We tried that, but we have three boot files that we need to update and Only two of those work over TFTP so it didn't really completely solve the problem for us So we decided to just keep things simple as possible our development board Mounts the boot partition automatically at slash boot already So we can just use secure shell as long as we can boot into linux. We can use secure shell the Copy files into the boot partition reboot our board It comes up Worst can I scenario you have a bad kernel it crashes or something well We've got our mini PC connected with a serial console so we can get into the u-boot prompt and Recover if we've got a bad kernel or bad device tree So that only leaves the one bad case of the boot bin which has the u-boot in it If we botched that one well, we're out of luck somebody at the office is gonna have to fix it for us But luckily that one just doesn't happen very often So that hasn't been a big problem so Kind of wrapping it up, you know, is it possible to actually work like this? You know some of the reasons why we decided to work this way of having the hardware remotely is that We need multiple engineers all sharing the same hardware because we're developing a linux driver a bare metal driver User space stuff all for the same board. That's not always the same engineer so there would be a lot of logistics trying to ship around the same board to every engineer and We wanted to avoid that things get lost in the mail the shipping costs add up and Not all the remote engineers have all of the equipment you need to test those hard problems So that's our main motivation for having our setups the way they are but You know, there is a loss of efficiency of not being able to get your hands on the hardware yourself so we have been making some exceptions for that that when One engineer or there's only one software task for one engineer then we've shipped them the board Or if we had a duplicate board or some of the chips are just so incredibly complex that it makes more sense to Ship it off and ship it back So I don't know if everyone else on my team agrees with this conclusion, but I think overall it's been working Well enough for us We've been managing to get things upstream so that leads me to my second adventure Which is the first Linux driver. I got upstream Using this method So this is the adventure of the resolver the digital converter. So remember these tires. They come into the story later so I'm guessing most of you don't know what a resolver is I didn't either when I started this so here's a 30 second crash course on What's a resolver? So it's this? two-piece thing here This is called a frameless resolver and it's got a rotor a little smaller part rotates It's connected to electrical coil that rotates and there's a stator That's the bigger ring it stays still and it has connected to two more coils that are at 90 degrees to each other and the rotor coil we energize with an electric field and That induces a voltage in the stator coils and those signals go into a Resolver the digital converter chip and since they're 90 degrees apart We have a right triangle the converter chip does some trigonometry And you can figure out what angle the rotor is Compared to the stator. So this is measuring The rotational position and it also has a clock in it so we can measure rotational speed of how fast this rotor is spinning So when I started this is our first test setup that they had set up at our lab That I would connect to develop the deriver It's got the same board running Linux Connected to the evaluation board with the resolver to digital converter chip and then we have our hardware store Set up of We added a shaft and some brackets to mount our resolver so we could actually spin that rotor But there's a few problems with this setup number one. It's human powered that Whenever we wanted to test our driver, we had to ping somebody on slack to say, hey, can you go turn that shaft so we can make sure my driver is working and Also since this has to measure speed Humans don't turns out they don't have a constant speed setting, you know, can you turn this one revolution per minute? Or they don't have a high speed setting. Can you turn this 10,000 revolutions per minute? And also since that came in two pieces That have no connection at all. It's called the floating resolver for a reason You can't touch and there is a point five millimeter air gap in between so Our high precision hardware from the hardware store Wasn't really up to that task, but do you know what is precision engineered to a fraction of a millimeter? That's right Legos So I went to the office and got to visit the office in France and I brought back one of the Duplicate boards we had with us and since I've got that background in Lego robotics I built a our version to set up or we can actually Test this properly So remember those tires I told you about from the 1980s Space sets well it turns out they're a perfect fit for the bevel in the resolver because I needed something to grip it to spin it With the motor and then I built these little Gizmos with some worm gears to get precision alignment that you can see there's four of them So that I can there's two degrees of freedom on each side of the resolver So we can get it exactly lined up the two donut shapes, so they don't touch each other at all so Here's what it looks like in action this is just doing a rotation test by hand to it's measuring and radians and then to measure speed the Lego motors actually have a Encoder in them as well So we can know that Exact speed of the Lego motor and compare that to our resolver So that's actually how I validated that our driver is in fact working and converting the speed correctly To radians per second. That's what the industrial Ios subsystem uses so hardware problem solved onto The next part of this is it turns out to get this driver upstream there was actually a Driver that's been in staging which is the corner alert of the kernel where Things that aren't quite ready to upstream sometimes go But need more work So there'd been a driver in there for 13 years waiting for someone to love it. So I Also got to learn the lesson in this adventure of how do you get a driver out of staging? So number one fix the bugs first always good practice when you're up streaming anything number two fix all the user space interface in Industrial Ios subsystem we have tons and tons of sysfs attributes for All the different aspects of these kind of parts and we try to use the standard ones as much as possible So that we don't end up with every single chip having its own unique Attributes so the driver in staging was basically reading and writing registers through these attributes, which is something maybe you do in debug FS, but not in Industrial Ios so we had to fix all that then Stuff in staging usually doesn't have device tree bindings So you have to write that and one thing to watch out for is don't let using a device tree property in your Driver sneak through without documenting it in the bindings YAML you have grumpy DT maintainers and And Then finally once you fix all those things we can actually move it out of staging I did a goat get format patch with no renames so that we could see the full driver on The mailing list give it one final review as it was moving out and then Once you finally get it out of staging then you can start adding all those shiny new features you wanted So mission accomplished just took a little ingenuity and about 40 patches and finally got that upstreamed So speaking of upstream that brings us to our last quick little adventures of Some other things going on upstream in the industrial Ios subsystem So we have the invest adventure of the spy offload What's the spy offload? Well the problem we're trying to solve is some of these analog digital converter chips can do a million two million samples per second and we need to get that into the Linux kernel over a spy bus and Using what's out there in the kernel right now we can get 10 maybe a hundred thousand if you've got a really fast system but yeah nowhere near two million and There's also all sorts of sources of jitter because when you're dealing with interrupts They're not always happening any at a regular interval So you end up something like this? What's wrong with this picture? Yeah, we missed almost the entire sine wave at least in this capture because the kernel must be any busy Sending the previous capture over the network or something. I don't know So Our solution to this is spy offloading so that's a way to do all of the spy transfers Completely without any CPU intervention. So avoiding all of the interrupts all together So the way this works is we basically Record a series of spy transactions whatever it takes to read one sample from our analog digital converter chip And then we're triggering it with a hardware trigger. So for example a pulse width modulation signal that can Make a pulse one million times per second at regular intervals. So we get evenly spaced sample our received data Gets piped to a DMA buffer So we're getting all of the samples in one big buffer At one time then we just get one single interrupt to tell us the buffers full and we can Read that buffer. So that's eliminating all the gender problems and it's allowing us to get way more samples per second so Ta-da, it works This is actually getting two million samples per second from a chip and Linux So the good news is it works the bad news is it's not quite all upstreamed yet So I sent a series of patches for this upstream and It got Quite a few criticisms, but it's also nobody said it was a bad idea It was just the way I was going about it Wasn't great. And so it's been suggested to break this down into smaller parts So the first part that's already been upstreamed already is this idea of pre-cooking or pre-preparing spy messages So this is an optimization that Actually was talked about about 13 years ago on the spy mailing list We resurrected this idea that Whenever we're doing a spy Transfer in the kernel There's a lot of code running to verify all the parameters of the spy transaction and so in a Like in a analog digital converter when we're reading samples, we're doing the same Transfer over and over and over again. So there's a lot of CPU time wasted Reverifying that same transfer over and over again. So now since the six point nine kernel there's a new API to pre-optimize a message or spy message to avoid some of that overhead and So the next steps for Breaking this down because remember that the idea is to eventually have a way to record and playback spy transfers completely from a hardware trigger So The spy maintainer wants this to be kind of something even more general purpose that can Solve more problems than just ours because that's how we do things in the Linux kernel. So I'm kind of looking for other people with these cases feedback hopefully to Figure out how we can make this work the best for everybody the In the IISO system we can already start using that pre-optimization to make II drivers more efficient The one of the ideas also thrown around was a lot of Spy drivers can do DMA mapping that that part could be moved to the pre-optimization to Optimize that even more A lot of people interested in this we're using can controllers So if you're interested in that, this is probably something you want to look into the There's a couple spy controllers that are specialized for spy memory That have a lookup table where you can pre-program spy commands and play them back by a hardware trigger So that exactly fits that paradigm that I mentioned before So maybe there's an opportunity to also improve some performance with spy flash memory using this and The last ideas from the spy maintainer of a lot of the Embedded systems nowadays have extra Microcontroller cores like maybe the PRU on a Beagle bone that we could use to also be a spy offload to improve Port performance in any way you want. So that was spy offload Next quick adventure is the adventure of the IIO back-end framework. This is also something new in the industrial IO subsystem What was the problem we're solving? Well some folks from STM 32 had this ADC IP block and a digital filter IP block that are separate blocks So have separate drivers, but they needed to somehow connect them into a single Logical device in the IO subsystem Analog devices had something similar where they have an analog chip and then they have this axi ADC FPGA block that reads data at super high rates that they need to somehow connect into a single Logical device in the kernel so we came up the idea of a front end and a back end So the front end is going to be that analog digital converter and the back end is going to be the digital filter or whatever that other block is and Now you can write Device tree bindings with a provider consumer thing to connect them together very simply and then your drivers You can register a back end and a front-end gets that back end that so the IO back-end framework is doing all the heavy lifting of connecting those together for you and Now you can have the easy way to write a single IO driver that's using multiple pieces of hardware And last little quick one is the adventure of the IO DMA buff the problem we're solving for here this one is getting more data from point a to point B for example this Use initial use case was some software to find radio stuff where they wanted to get 64 megabytes of samples From the IO subsystem out over a USB connection as fast as possible So we have this new DMA buff framework the way it works DMA buff is a API in the kernel to Move DMA buffers between subsystems or user space and the kernel So we can create a buffer in user space we can attach it to an IO device now using some my octals and capture some samples in that buffer Detach that buffer from the IO subsystem and Then reattach it to the USB function and then tell USB to send that out over The USB so that really improves performance a lot by avoiding copying the data from The IO buffer to the USB buffer. It just we just move the handle instead. So and This one test case they were over doubled their throughput While at the same time Reducing CPU usage. So that's a really nice optimization there, too so that concludes my adventures in the IO subsystem and we'll open it up for questions With the spy offloading my assumption is that you were starting not with bit banging But you started with a spy driver that had hardware support. Is that correct? Yes, yeah, and then offloading was one level beyond hardware support your That makes sense now, right? Yeah, anything that's going to do offloading is going to require special hardware that knows about Being able to have a hardware trigger or Piping the Received data somewhere else. Thanks So the the peripheral in question is an axi FPGA peripheral that analog devices makes you can find documentation it's open source can find documentation on wiki.analog.com and The hgl is also available Hi, thank you for the presentation and I think it was really great because it has always been very challenging to test Sensor drivers remotely at least So I just have a basic question. Did you require any kind of human input at some point? Or any challenges in that case While you were testing the driver for it. Yeah, usually we need somebody to set up the hardware and attach a signal generator or Logic analyzer whatever we need to the hardware, but once that's done We can connect to that remote PC and control all that remotely. So it's we're trying to minimize the extra human need but So once the basic setup is done after that you still need some human support for that Not unless there's a really strange problem. Oh, yeah Thank you. So on your oscilloscope and stuff. Are you able to Is the remote user able to see the The screens from the oscilloscope app or they yes We're using a remote desktop to access that many PC. Okay, so we can see everything on it and use it Just like we're there. Okay, that's great. Really cool So the the m2k that was being used is called other on 2000 and in fact, it's actually a complete Linux system so it's based on a sync MP no on a sync 7000 runs Linux and Also runs IO for all the different blocks in inside that that unit for Signal generator ADC stacks and so on and so you can also remotely connect to that unit So you you don't really need needed Kind of the scope the evaluations of running directly connected to the device you can also use that across a VPN and run that Software on on a different PC maybe in a different location The latency is a little better though when you're local Thank you for the talk I was wondering about for what I understand you were mostly using it for development your setup With the computer remote computer and stuff, but I was also using it for testing and see eye purposes Yeah, most of this is going just manual Testing since we have to generate a physical signal but I mean my typical setup is I'm writing the kernel Stuff at my computer at home Compiling the kernel there and then using SCP to copy the kernel to the remote board and then reboot the remote board and And then the remote mini PC is just the user interface for the logic analyzer any tools like that That fully answered your question. Thank you, right If you were to take it the next step and have it be more of a CI type of setup does the does that m2k? Scope and stuff provide the ability to script it and not have it visual But to maybe capture some logs and do some comparing. Yeah automated tooling. Yeah, like Michael said we can connect her remotely with Anything that talks IIO User space stuff. It's all using the IO Stack so any compatible tool Works with that and we've made a Python script So far just to create a simple sign wave, but it could be automated much more real quick Reservations for the boards so you can share them between developers. Did you homebrew a script for that or did you use something that you found in the community? I'm not sure I fully understand the question So like to prevent say two developers from trying to use a board at the same time. Oh, yeah the We have a we use snipe it for Keeping track of assets. So we made a Setup asset that people check out One board or two board or two setups and then we'll we can use that to see who's using them Okay, thank you But we try not to have two people working on the same thing at the same time so that it's just not a problem of having to take turns With the continued development of cheaper SPC's and such what stopped you from building more custom test harnesses to test some of these devices instead of building off of what looks to be I think you said it's a xylene zinc Development kit why not build your own platform The a lot of the analog digital converter chips we have are using an evaluation board with the FMC connector And that's already built into the zinc and there's lots of history of Testing developing and testing these devices with that hardware. So we're just sticking with it. All right. Thanks all for your questions And thanks for listening Test Everybody hear me good Let's go ahead and get started Want to welcome you to this session. We're going to be talking about quality beyond kernel CI in upstream Linux for TI SOC's Want to take a moment and highlight You know something that we feel is very important, you know, TI has a very long history in the open-source community and Collaboration as you can see from you know all the the different organizations there that you know, we have relations with with Including kernel CI Which we are very happy to announce TI is now a platinum member of kernel CI as of this week We're very pleased about that And I know you guys have probably seen this some of the other talks, but I think it's important to highlight upstream first very important In our view to have a upstream first mentality And as it relates to this presentation, if you're going to have it upstream first mentality Better have a way to test the upstream. So we're going to talk a little bit about that here Quick intro of speakers. My name is Barry Sharop. I've been with TI Long time going on 18 years now Started my life at TI and analog been in a AP for about two and a half going on three years I am the manager of the system integration and test infrastructure team We're responsible for managing the physical infrastructure hardware as well as software for doing test at TI for Linux I Am also pleased and privileged to be up here with Nishant Menon Who also has almost two decades of experience at TI and he is everything Linux? So Linux system architect Kernel development kernel maintainer He has a very very very deep experience in Linux We have also worked with Minas humbard Zimian Couldn't be here today, but his contribution is No less important He is the lead developer for the system test team that does a lot of the automation tools as well as the Kernel CI implementation the test and kernel CI testing that we do in this presentation Okay Disclaimers this is technology presentation. This is not a product readiness or road map commitment in the opinions here are Our own not TI So get mad at us don't get mad at TI All right quick overview Give a little bit of background a little motivation of you know why we're doing this as it relates to kernel CI And then talk about how we're doing upstream testing and this breaks down into a few buckets here You know, how are we doing the kernel CI testing? How are we doing? CI CD testing as it relates to upstream and then a full system test of our upstream builds and then I want to transition a little bit into You know talking about how do we get to a? Common testing framework, you know towards common testing. All right, so motivation. Why do we use kernel CI? Kernel CI provides a common platform Obviously to help kernel maintainers gauge our Lennox kernel And they do that by publishing the test results You know on the the website on the the the you know customer-facing portal We need to use that to get notifications if there is a regression We want to know about it obviously And it helps us gain confidence about our builds since the images are built outside of TI You know it sort of helps us make sure there's been some secret thing hidden in there that makes it work inside TI Whereas it's gonna break outside TI So quick aside on how we do tests so as I said we maintain a test farm my team does We have two sites that have test EVMs one in Dallas, which you can see on the left Please excuse the cabling. It's utilitarian and then one in Bangalore on the right We have about 300 boards roughly split between the two sites Those are used for both Lennox testing as well as our toss You know the key point here is that it is Truly a global test farm so we can initiate tests from one site run them on boards at either site in parallel Bring all the test collateral back to the original site. So it really helps with our Parallelism and scalability across, you know all these different test platforms Okay, so let's talk about the upstream testing Start with kernel CI So the upstream testing we're doing in kernel CI we do automated build a test collateral So we build our boot images locally But everything else is built at kernel CI and we use the API to you know pull it down to our site So, you know the the build has the kernel files the DTB files the file system And then you know we pull that down from the kernel CI cloud to TI and run those tests on our boards in our test farm You know in Dallas in Bangalore The results are automatically, you know collected and uploaded to the storage site on kernel CI for web-based review So pictures worth a thousand words it kind of looks something like this you have the API server the storage server Web dashboard query the recent kernel build run the tests on the TI farm Publishes us up to the storage server and then make that available, you know through the kernel CI web dashboard So what is the status of this right now? So we are running nightly boot tests We would like to get the LTB and kernel self-test those will be coming soon I think Venus is back in the office next week. So hopefully very soon We are currently running 13 upstream SOC's and that covers both our satara platforms and our Jacinto platforms And you can see the the you know the web screenshots there probably looks very similar to what y'all are used to and you can see that You know green is good, but it's not always green Okay upstream CI CD testing Our upstream CI CD testing does end-to-end automation of the latest upstream build test and release so the builds are Automatically initiated test plans are automatically executed again on the boards the results are tabulated into a report and The build collateral and those test reports are then automatically pushed up to a customer-facing portal So the way this looks you got a next to pride branch Do the build assuming that passes you promote to actually run the tests push all that collateral out onto the public portal if For some reason the build fails then that branch essentially is dead doesn't go anywhere until it's fixed And then the next cycle has the next set of results So these execute the CSED tests execute nightly The tests that we run tend to be a more expanded set of tests beyond what we're doing with kernel CI They tend to include device driver tests that are a little bit more complicated, but you know still fairly simple You know localized to a board One of the challenges with this since we do want to run it nightly is we tend to have to make trade-offs You know between the the test coverage versus the amount of time it takes to execute You know so we want to run this thing in you know, you know three four hours tops Across you know all 13 so see so you know that time tends to add up So the test reports that we have are really useful for our developers you can see that You got you know green for passing red for not passing And it gives you the ability to drill down into an individual test and Drill all the way down to the log file to triage and figure out you know what went wrong So for the the upstream snapshot we take As I said all the build collateral and those test reports push them up to our software DL website For anyone who wants to look at them to go see Much of the content is similar in the sense that you have all the SOC's that are supported in upstream You can drill down to an individual snapshot down to the test results for that snapshot You know all the way down to the log file to you know dig down and figure out what weren't wrong if a test does not pass Okay full upstream test So We recently started running full system test on our upstream so we automatically Generate those test plans based on requirements. So this is a requirements based testing strategy As you can probably guess that can tend to generate a lot of test cases So hundreds per SOC across all the SOC's we can easily run thousands of tests and these tend to be even Longer running more complicated tests than the CICD Test plan so we can have you know device driver tests that actually require the device to be plugged into the SOC, which is a challenge you can have you know test cases that you know could run for 20 minutes 30 minutes an hour I think we have a test in there that does a thousand boots. We can run for several hours So these test cases take you know upwards of two days to run in some cases. So it's a You know a very very long cycle We've implemented Automation to flash the SD cards And obviously there's automation to collect all the results tabulate them together generate reports for all the SOCs So as I said on a mega generate test reports, you can see an example here These are not published to the community today That is one of the things we would like to do if it makes sense One of the you know key highlights here versus the CICD Which is sort of a functional testing is these test reports do link back to requirements So you can get an idea of you know, how much of our requirements are satisfied with actual passing test cases and again from a From a triage standpoint you can drill down to the individual Test case results all the way down to the log file figure out what went wrong if something isn't passing Okay So to recap where we are We are running the kernel CI tests. So those go out to the kernel CI upload the results to the web dashboard We also have these sort of private builds these private tests for CICD and system tests Some of those are supported uploading into the portal and I suspect at least in the conversations with folks over the last few days that y'all have something kind of similar a Lot of you kernel CI tests Maybe some tests running privately Maybe some of those results are uploaded somewhere where they can be viewed But it's it's it's a lot of similar things implemented in different ways One of the nice things would be Can we get to a place where all of that kind of testing is common? So we're all doing it sort of the same way and Can kernel CI sort of be the umbrella to sort of bring that together a sort of a foundation? And I will pass it off in the shunt Dig into those details a little deeper Thank You very Good can hear me why I mean why would we care about doing this? Collaboratively we all talk about doing upstream testing. Why can't people make products from upstream? Simple they don't trust upstream it all comes down to this little keyword over here trust. I Don't trust the quality of software. That's the I can't trust that you won't regret in the future I can't trust the features that you have in the the solution that we are proposing here Involves the standard stuff to manage trust you standardize stuff so that hey if I see it on one vendor Is the same on other vendors too? You decentralize if I am able to do it you can replicate my results too If it's like it right look at the shark on it. Nobody has injected a new problem into the system Reporting can be trust the reporting that you're doing. There's no bias in the reporting What if you are changing the performance number a little bit skewing the result towards your product, but that that's not really true transparency is the key Today we don't have that transparency currency I is moving to the transparency What can we do next stage? so I understand most of the folks who Work on Linux our focus not purely on Linux, but the guys who are actually creating products They look at the product solution and most of these systems are heterogeneous in nature. You have Linux You have our tosses like free our tosses sapphire thread eggs. There's so many different noises coming to the They all have the same problem. How do I test an interface? How do I test a peripheral and camera sensor? For example, it doesn't change whether it's with ti and xp Adi, what are we interfacing you're using? If you're a seed like if you're using a serializer DC lights of it like fpd link is the same thing Why are we testing and differently if you dig down, right? I like to use this little thingy on the side. We all use different languages. This is the standard communication Delta We all would have sung Kumbaya if you were all talking English together But it's a diverse world I get it but if the communication were to get standardized between this it would be much easier for folks to reduce the cost down and Move between different vendors easily There is limited value in looking at testing as something of a differentiator. It is already commodity Let's come out of place it There are a few topics that we kind of put on here But it's just examples of various problems that we have some folks like lava. I'm a big lab grid fan. Sorry Barry He likes lava There are different frameworks that people have created ti decided fork LTP Linux test project and added device driver test Framework on top of it which ti Managers year on year. We have to rebase. It's completely out of tree solution We also on the whole side created something called a VATF There's actually two different frameworks that we have on the PC side Because many testing involving peripherals have to send something from the board and has to be cross verified from the PC side as well When you look at industrial protocols, for example, yeah, maybe you have a big list of things to do They're all standard problems. We all repeat the same solution, but in different ways Decentralizing and this is key. You saw the farm that ti has right.  We have 300 maybe we can go to 800,000 boards But that limit is coming soon. If I look at these products or 10 years We say in Linux. Hey, we love Linux We support any product that people use for years on the go really do we Do you do testing for the same product 10 years from now the way that you're testing today? most of us don't It's a matter of economics The only way to spread this cost out is to let the guy who's interested whose stake is high to give the importance that he wants Instead of doing this centrally Currency I does that today. It's already decentralized. There's a framework. There's a standard for us to do it We can leverage it to the next level So many things if you guys are in corporate world, you know that our business priorities change Ti has a lot of history of doing that often Our omap processors are no longer Actively supported. There's a lot of folks who have complaints about it. There are other products that we have slowly scaled back But there are real products out there that actually use these processors. What do they do? They have no way to replicate what we have done internally This gives them an option of doing that Anyway, long story short. There is so much duplicate duplication that is going on here The other thing which is very scary to my marketing people at least and again, I'll claim the disclosure I'm speaking as an individual right now Nobody likes to publish performance results out there Without marketing team review. Why because a D MIPS number change is millions of dollars lost But guess what? There's nothing hiding it. I can run Tristron as you can you'll get different numbers But if you could reproduce the same test results, that's great. We are scared about saying hey, we failed at this case It's normal There are regression like for today for example You boot master branch is completely broken None of the K3 devices boot today morning Yesterday it was today's broken. We know internally because we are monitoring Yes, we are monitoring both you boot master and you boot next and we know masters records and we know why by now There's nothing wrong in reporting that the other thing is The quality of reporting as well if you are sitting so the way that we looked at this problem is if we think From external out in what would you like to see? How would you trust the data to be able to create a product based on mainline colonel mainline you boot mainline TFA? everything mainline Call it your master directly You need to be able to reproduce the data. You need to be able to trust the trust the test reports that are coming out But there's no standardization. You're entirely dependent on TI's translation of the test report if we say that I2C's function you can't replicate that you have to trust us long story short Currency I today is solving a humongous problem already The infrastructure for that a distributed reporting system the amount of platforms that they're trying to solve is already a complex problem But that's just the tip of the iceberg The real thing that we as embedded developers have is the platform sensor the products themselves Here's a scope for us to grow further down buddy so Call for action common testing framework beyond Magnus kernel as much as I had leveraged the shared solutions, you know kernel CI can be that umbrella You know, it's it's part way there You know, it would be a good place to to congregate to do that.  Thank you Obviously Texas Instruments Linux Foundation Colonel CI Lonaro Panganitronics Calabra None is what it would be possible without their help So big big. Thank you and Contact information Myself and the shunt mean us emails and IRCs, etc. And We will open it up for questions We really wanted this to be like a conversation where What stops us from doing this? Okay, so I can tell you what's stopping me from using kernel CI right now and that's that I'm waiting for v2 to drop It's in its next generation phase and I'm a little bit worried that they're in a Mode well, I talked to you about this in the hallway I'm a little bit worried that they're in a mode where the next Version of it is gonna be like a kitchen sink version. They kind of have second system syndrome Anyone want to comment on that am I wrong I could be wrong I Think we gave that impression and I think we were right in the direction we've been fixing this strategy for the past three months there and We have a system that's working. It's supporting build results departing boot results there is much better testing for device probe for example enabled and We are scaling up that like slowly like involving a few maintainers Hipporting a lot of this they stuff like manually at the moment because we want to we want like a data that we can trust And we want like to not make the mistakes of the past system that was just if you if you start like looking To how the test results is a big mess and in the in the legacy system because we can't really understand like the bookness of the data so we're spending a lot of time on on on that process and More communication is coming up. We had like some blog post last week There's a bunch of other things on the way There is a community engagement working group that has been set up and Shua Khan is going to join us to leave that group So that's kind of the new states of things. I think that that kitchen sink is no more You say something who's a request race ability right and There is a tool called ballot that has been starting to get built and this will probably play a big role in As putting that in a story some toda along with trace ability that That's not the point which I want to make I was more on the Sigh where you had an old one slide where we're showing all the different kind of like like bridge or if you go With whatever you like It's not a little bit like you would like to look for how can we get more standardization? But the question is would it make more sense to go on like this is the agreed Standard we are talking not about the tool which we're using I'm asking this because I come from an environment where I have LTP test cases. I have robot fresh cases. I have CTS BTS STS test case from Android LTP I have all these different kind of things I don't want to spend any effort in migrating one test framework to the other But the nice thing about the test framework if they come up with saying I report a test case It goes with a zero one minus one whatever they have as a result and we directly translate and create like a mapping on it And with this we can see independent which Testing environment we are using we get a comparable list of results for the reports because at the end you want to see a green red cross check whatever and maybe find something like configuration and Often to try to get a standard So I see the Jenny V for example Jenny V started with generic IVI system There was so much discussion going on they couldn't agree on which audio manager to use which framework here or there Other parts there was no strong alignment with a GL What turned out is that the Jenny V changed into a coviz are where they do a lot of vehicle signal specification and this vehicle signal Specification give the room to say we are discussing. We're talking a similar language. We have an interface open standard and I Would see this is much more a way to go and they get the traction and it helps that OEM suddenly work together and say well, I need these kind of signals You need that and we make this as the interface and this could be an interface also Torx Colonel Si or whatever? Three different problems that you introduced but let's talk through them In terms of information exchange that has to be a standard And that's the best way to do it have the abstraction at API or data exchange level and we need to do that step number one step number two is the Even if we standardize something there will always be someone who is not lining up the standard There'll be some BTI be with anybody in that case. So there's a cost to move to the standard Why should I do that? Be the same source code beat a different source code or beat the same source code the translation has a cost We'll have to justify that to our management. Everybody will have to that the standard actually has value by moving to this API standard the third vector standardizing the test code itself There are benefits in some areas where they are commodity but as we mentioned it can be a very hotly debated topic and We know the VI versus emacs discussion, but we won't bring that up, right? Again, we have not set up the working group yet And this is part of the discussion in terms of we would like to do that and we would like members to come and say Where do we want to draw the line? We don't want to come and say okay. We are willing to provide the code that we have right We'll dump it out there and say this is what we do you guys want to use it Do you want to write rewrite it or do we reduce the scope to just purely API? From where I stand I Feel there's a lot of value in sharing code Directly and contributing together because from TI's perspective There are vendors like fight echo here who work across multiple associate vendors, right? There's companies like Toradex who do the same They all invent the same thing And there's a lot of value at least at embedded level that we can share and start differentiating at all Now there's a cost associated which means I will have to rewrite a lot of stuff on my end From my perspective it pays off in 10 years. It's worth it Now, are you convinced that's a different question? Okay, as you know, I've been You're preaching to the choir or maybe singing to the preacher here So About six years ago. I had Fuego adopt the kernel CI Format as it's out as it's native output format. Well, he's translated into json And then Colonel CI changed so that was a bummer because it was a lot of work and so If we adopt something It would be good if we kept it The only way I think you can deal with this is you're not going to go back to things like LTP these massive suites that already exist and Have them change their output format and and so I think you have to have you have to have a parser per test or at least a parser for per Test output category. So that's one of the reasons we pushed k-tap Into the kernel k-tap is not the greatest output format but it's good enough and particularly for the types of tests that the colonel was running at the time we adopted it and It's uniform, right? So and I really appreciate the efforts by the way to go into the existing kernel tests and make them Compliant with that output format because now you can at least run k-tap on you can run a tap parser on the kernel and put it Into whatever you want the Linguafrank would it be for for test output? You can do the same thing with LTP without you know, you make a single parser for LTP You get a huge amount of win off of all that But I would love to see a single common format Maybe kernel CI as it is today is now that current format I don't I haven't looked recently because it scares me to have to go back and rewrite my My generator again their generator was one of the hardest things to write To try to do do that in Fuego was to write a generator that could handle all the different cases And I have all these parsing modules But I would love to have them land somewhere that other people could reuse them, right? So I ozone results I can put out into old kernel CI. I can take Bonnie results I can take stress test results and we just need to Find a place to put that stuff, but right all my parsers are Fuego parsers All right, they're in Python. They just take output and convert it into a Python module But we need to have the discussion and we just need to pick one and actually start doing it But the harder part is not coding. It's it's finding someone who's gonna host that and You know, you have to have a place where this code can live This has been a big talk on the current cycle meeting the past like a few weeks and yesterday We spent like a few hours on this topic like not only test specifications per se, but the work texture of what currency I will be for now on and We feel that like how this how the different CIs they have like their own test specifications They're on like they start in their data that and we want to move that out of the different size systems And maybe move a lot of that inside the current tree like that's part of the effort that they're pushing with CII improving the test quality test quality of things like and There is a test catalog a proposal from from Nikolai Kondershov as well But how those things I think are taking us towards like a more common place to start like a Not on the tests the the specification for the tests the kernel configs like which bars should be running which things So I think that's that's where Kernel CIs is building a lot of a lot of grounds for people to come and share that I guess With all the power thing and adjusting to something which is a project at the question if this is really The way where you want to put your bad on if I think for example on container framework There's a large set of container frameworks available different kind of containers But there's also the OCI compliant container the open container initiative and what I hear in the industries I'm talking to that they say okay We may have like there is the LXC which is not compatible where they have certain use cases where they accept it And because I say that's for a performance reason or so we need to go with it But I know a lot of other places where they say we will only accept new technology which is in accordance to an open standard interface and then it may be better to have like an open testing infrastructure testing interface whatever which could be hooked under Existing an eye environment like Kernel CIs about having more open and of course you will also find then even in OCI compliance decks extensions so Where you just say okay the OCI is not fulfilling all my needs I need extra features I need extra things, but they are not for all cases Which allows you also differentiating parts maybe for testing is less on the differentiating elements But you can have all like extra fields You see this also with SPX where you have like mandatory things extension things You're more compatible as compatible and I get this is for me the perspective where you leave a lot of openness And you bring it to the standard Our framework and then you have maybe parts. There's adapters and so here this is how I can have the if we go to the Intermediate parting competing compatible with kernels the iron so on so to just have this as an interface and not changing all the Things again and again because one project decide that we change it to a different format because we like it better And don't see the full ecosystem of all our users I mean that's that the thing which you have in a traditional way also when you do Open source or a fork open source projects utilize them, but you don't give back your requirements You don't give back your modifications There is a strong risk of breaking with the upstream and this would be something similar like we have all this different kind of test Types all these different kind of things they all need to be then aware to the kernels CI environment and kernels See I need to know how they are used in order not by mistake breaking something and getting good then all the feedback of the Community and maybe this works more if we say here We have a standard you comply to issue change it and then we have like a governing board on the standard Which is not like a collecting pool of that's where we have the result or Texas test execution It's going to be a journey the first thing is to say that you know we can If there's one message we were to take from this presentation, right? We can go and attack the easy problems right of saying that a currency I is let's solve this problem of reporting and Reporting of boot systems for the maintainers. Yes, we can maintain some pressure t