Hi and welcome to this session about software update solutions for the Yocto project and Open Embedded. It's a pleasure to be here with you. Thank you very much to the Linux Foundation team for organizing this event. My name is Leonom Aviv and I'm a senior software engineer and an open source enthusiast. I work for Kunsuko Group. This is a services company specializing in embedded Linux and open source software. My colleagues and I have a lot of upstream contributions to popular open source projects such as the Yocto project, Open Embedded, the Linux kernel, Ubooth, Automotive Grade Linux and many more. Kunsuko Group is specialized in hardware, software, build, design, development and training services. The company is with a headquarter in San Jose, California, but has some engineering presence worldwide. I'm personally working from Poldeburg area remotely and I have colleagues from around the world including the United States, Canada, Spain, Sweden and Germany. The agenda for the talk today is about software over-the-year updates from the perspective of the Yocto project and Open Embedded. This is not the first talk on this topic, neither the last one. However, the special thing about it that the focus is from the point of view of developers familiar with the Yocto project and Open Embedded. In the beginning we'll briefly mention what is the Yocto project and Open Embedded. After that we'll have a look at the challenges for software updates of embedded Linux devices. We'll continue with an overview of the various open source software update solutions. There are many of them, so we'll have a closer look at three of them, Mender, Raug and LiboS3, which was previously known as OS3. Finally, we'll wrap up with some conclusions and there will be time for questions and answers. The duration of the talk is approximately 40-45 minutes or we'll have 5-10 minutes for answering questions at the end of it. So let's talk about embedded Linux devices at the beginning. Embedded Linux devices dominate various different industries. Here at Consulgo Group we have customers from the automotive industry, from medical industry, from telecommunications. Various companies making very interesting intranet of things. Obviously those devices are completely different in terms of purposes. However, there are certain similarities between them. All of them need a build system to build a custom Linux distribution for their own need and a software update mechanism. Among the best practices of the industry is to use popular stable open source products for the build system and for the software update mechanisms. In order to save time and money and to focus on the core features of the embedded device that you are working on. The Yocto project is not the only build system available there on the market. However, it's my preferred. Alternatives are BuildTrue, PDX Linux or even using general purpose distributions such as Debian and customizing it. However, my preferred choice is the Yocto project because of its flexibility. A few words, what is the Yocto project? It's an open source collaborative project of Linux foundation for creating a custom Linux based system for embedded devices using the open embedded build system. The open embedded build system brings BitBake and open embedded core. Furthermore, the Yocto project has a reference distribution called Pocky. It is provided as metadata without any binary files. The idea of Pocky is to bootstrap your own distribution for embedded Linux devices as a quick start and after that to increment it by adding new features to it. The Yocto project has a b-annual release cycle twice a year. There is a release. The latest release is expected this month in October. Furthermore, something new that happened in the Yocto project and open embedded ecosystem recently. In 2020, the long term support release was introduced. The first LTS release in the Yocto project is Dunfel. The idea of the long term support releases in the Yocto project is to cover two-year period. As you know, developing an embedded device has its own pace and sometimes making the hardware and after that the custom software for the hardware takes a significant amount of efforts. Therefore, having a long term support release simplifies the development efforts. Now, let's have a look at the Yocto project current releases. As I mentioned, there is a new release on every six months. The long term support release is Dunfel. This is version 3.1. It was released in April this year. And the new release coming in October is version 3.2, Gathers Guard. A brief introduction to some of the important keywords in the Yocto and open embedded ecosystem. Although I'm pretty sure that the majority of the audience of this talk is already familiar with them. There are receipts and layers. Receipt is the most common form of metadata. It contains instructions as a list of settings and tasks for building packages that are then used to build a binary image which you flush on your device. Receives describes source code, additional patches, dependencies for libraries or for other recipes as well as configuration and compilation options. The layer is a collection of related recipes and configurations. Layers isolate information used when building for multiple architectures. Each layer is packaged in a separate directory. The common notion is that the prefix of this directory name starts with meta. Furthermore, the Yocto and Open Embedded has an excellent documentation known as the Yocto Project Mega Manual. Here is a link to it. With each release of the Yocto project, there is an update of the documentation. And it's a great source of information. I use it on a daily basis and I recommend to anyone to have a look at it while working with the Yocto project and Open Embedded. Now let's talk about software updates over the air or over an internet cable. There are so many things to consider. First of all, are there any limitations of disk space for the downloaded update? Can you afford to have two identical partitions or your constraint of disk space? Are there any limitations of the network bandwidth for the data transfer? For example, if you have a setup box that's connected with an Ethernet cable with very fast internet connectivity, that's something that you should not worry. If you are updating a wearable device or a vehicle, that's another story because you may need to use cellular data, which is not as fast and not as reliable. Do you need a container-based solution? Nowadays containers are very modern. We'll briefly talk about them. Do you need an AB? This means like two identical partitions on which you have different versions as you switch between the versions. Or you need a binary data update where you have a small footprint that you download only the differences and incrementally upgrade your device. How do you upgrade over the air, over USB stick manually or cable or something else? Is the device that you are updating mission critical? Furthermore, there are more questions that you should answer when you are choosing which open source software update mechanism to use. Is there a Yocto and Open Embedded USB layer for the hardware use you use as well as integration for the software update technology that you would like to use? Which Yocto project release do you need for your product? As I mentioned on the previous slides, the Yocto project has a new release every six months, but there is also long-term support. Maybe this is the one that you should use for your update. Finally, how to update a fleet of devices with Internet of Things? You might end up with a lot of embedded Linux devices that you need to update simultaneously or to manage the different version on different devices. Which is your preferred choice? So many questions and the answers are different depending on the specific project. So different projects, different people have different needs. That's why there is a big variety of open source solutions on the market how to achieve all these things. Let's have a look at them. This is a list of some of the popular open source solutions for software updates, actually not all of them. Mender, Raoq, SW UPD, Update Hub, Balena, Snap, which are container-based solution, Ostrich, which is coming from the desktop world. However, nowadays it's a core technology of various software update services and technologies for embedded Linux devices. Actualizer and actualizer line are applications that are compatible with Ostrich and are supplementary applications for doing updates. We'll discuss them in more details in the coming slides. Other solutions such as QT, OTA, QDIS, relying also on Ostrich, Verizon, again a solution, relying on Ostrich, which is coming from Toradex, or RPM, Ostrich, which is used in Project Atomic. So many choices. Which one is good for you? Well, first of all, we have to say that all these options have different strategies for performing updates. Some of them are similar to each other. Some of them are completely different. So in my opinion, there are four categories in which we can divide these open source solutions. The first one is AB updates. This is dual redundant scheme where you have two identical partitions. The idea is that you download the update to the B partition. After that, you reboot the board and the boot loader boots from this B partition which you have previously downloaded. The file and it becomes a partition. The delta updates are quite more challenging because here we're talking about binary diffs of the file system and downloading only this binary diff which represents the difference between the update. This is convenient for devices with constrained bandwidth of the network as well as disk space because binary deltas are smaller, which means that they are easier to download and easier to store. However, the advantage of the AB updates is that you always have one partition that is known to work. While with the delta updates, the standard approach is that you have just a single partition on which you incrementally download the binary deltas and apply them. Furthermore, there are container-based updates. Balena, previously known as Raisinio, is doing this. If you are looking for container-based updates as listed here on the previous slide, you have Balena and Snap as an option for this. Furthermore, there is something really interesting that is happening in the past few years and this is a combined strategy. An example for a combined strategy is to combine container technology with AB updates. The container technology has changed the way application developers, specifically working on web applications, interact with the cloud and some of the good practices are nowadays applied to development work for embedded devices and especially for Internet of Things. In general, containers make applications faster to deploy, easier to update and more secure through isolation within the container. Yocto and Open Embedded provides a layer called meta virtualization. It brings support for various virtualization technologies such as Xen, KVM, LiveVirge, Docker and associated packages necessary for constructing open embedded based virtualized solutions. In my experience, there are many, many cases nowadays of companies working on products where they are bringing containers to embedded Linux devices but still need AB updates for the base Linux distribution built with Yocto and Open Embedded. I quite often nowadays make images with the Yocto project that include meta virtualization with, for example, Mender or Raug for AB updates. This is something new with some kind of a trend which is really interesting. I think it's convenient and I expect that in future we'll see more of this approach. Now, let's have a look at some of the most popular solutions with which I'm familiar, starting with Mender. Mender is available as a free and open source plan or paid commercial subscription and enterprise plans. So, of course, the open source plan for Mender is completely free. However, it lacks some of the features that the commercial and enterprise plans have. For example, delta updates are available only for the paid plans. AB updates are part of the open source and Mender is known for its AB updates. The delta updates, which I repeat, are part of the commercial and enterprise plans of Mender were introduced a year ago just in time for embedded Linux conference 2019 in Lyon, France. Furthermore, Mender provides a back-end service. It's a hosted Mender which allows you to manage the devices. You can host it on your own computer. Mender is written in modern programming languages. It's a relatively new but very stable technology. So, Mender is written in Go, Python, and of course JavaScript. The Yocto and OpenM Edit integration of Mender is done through an extra BSP layer called Metamender. The whole source code is licensed under Apache 2.0 and it's hosted in GitHub. Furthermore, Mender has these Mender community layers in which you have various supported hardware and integration. Pretty much these are the bits that are specific for certain platforms and not part of Metamender. The source code is available in GitHub, as I mentioned. The supported devices out of the box for Mender through this Metamender community are Raspberry Pi, Beaglebone, various Intel X86-64 machines, Rockchip, O-Winner, NXP, and Furthermore. The idea is that you can easily get started with Mender. Build it, for example, for Raspberry Pi or Beaglebone, because this is a common target and pretty much any embedded Linux developer has one of these boards on his desk. Of course, if you want to work on a more professional project, and most probably you will not be using Raspberry Pi for it, then they have these simple integrations for the most popular systems on the chip. Furthermore, you can do the same integration if you have a specific hardware. This is a screenshot from GitHub of Metamender community. Here you can see how this layer has sub-layers also with Prefix Metamender, followed by the specific target that they are made for, such as Beaglebone, ClearFox, Coral, Atmell, and so on, including Raspberry Pi that we've briefed and managed. The most popular way to get started with Mender, and to give it a try in my opinion, is just to build an image for Raspberry Pi. Mender has an excellent documentation and a good community. As you can see here in this Metamender community layer, there are 10 contributors which are from various companies. So Mender is something that has been already adopted by the industry, and a lot of people are using it. Mender AB updates supports two client modes. The first one is the manage, and by default, the manage client mode is supported. The idea is that the client is running as a demon. It pulls from the server, and when it detects an update, it applies it. The standalone update is another client mode that's supported by Mender. It's pretty much a solution, which is convenient if you are doing an update with physical media, such as USB stick, or any network in pool mode. The updates are triggered locally, which is suitable in this very particular use cases. Here is an example how you can run a Mender client in standalone mode if you build an image with the Yocto project and open embedded. In your local .conf, you have to configure, or either to do a BBAPend, you have to configure this variable system out to enable for the Mender recipe and to set it to disable. As I mentioned by default, the manage mode is enabled, so if you set this variable in your local .conf to disable, you make sure that you are switching to a standalone mode. Once you've built the image with the update artifacts, what you can do is to run a simple .script as here, and after that, this is happening on the build machine, of course, and after that on the embedded device, for this example, I have done it with Raspberry Pi 4, you need to run this command, which is Mender install, the path to the HTTP server that you have started, and the update artifact, which is with extension Mender. The steps to install a Mender update are actually three. First, you need to apply the update in standalone approach. Let's have a look back in this slide. On the last step, when you run Mender install, this is when you apply the update. In order for the update to take effect, you need to reboot the system. On the first boot, after a successful update, the Mender client checks the status, if it's successful, it commits the update. So this is the standard approach how to apply an update, and this is valid for both the standalone and the managed mode when Mender is working. You still need to do a reboot after installing successfully an update. I forgot to mention that Mender works not only on ARM devices, but also on Intel X8664, and for ARM devices, the recommended boot loader is Uboot. Moving on to the next solution. For AB updates, again, it's called Raoq. It's a lightweight update client that runs on your embedded device and will allow you to control the process of updating it with new firmware revisions. Furthermore, Raoq provides the tools for the build system to create and inspect and modify update bundles. Furthermore, from security perspective, it uses cryptography to sign update bundles. Furthermore, it's compatible not only with the Yocto project, but furthermore with BuildRoot and PDXDist. However, we'll focus on the Yocto project for this presentation. Raoq has several components that are available under different licenses. The Raoq software itself is available under general public license version 2.1. It's hosted in GitHub. Actually, all components of Raoq are hosted in GitHub. Metro Raoq is available under mid-license. This is the Yocto and OpenEmbedded layer, which brings Raoq to your embedded device. Raoq Hubbit is a plugin in Raoq Hubbit data. These are the plugins needed for the Hubbit, Eclipse Hubbit web system with the user interface for managing fleet of devices and applying Raoq update bundles to these devices. Raoq is recommending to use Hubbit, but actually Hubbit is an open-source project developed by Eclipse. The last slides of the presentation will talk a little bit more about it because it's recommended not only by Raoq, but by other software update systems. The challenge about Raoq are the integration steps. The integration steps are actually not so different from Mender. However, unlike Mender, which has this community layer with a lot of boards that are already supported and most common boards such as Raspberry Pi or Beaglebone are already supported, this is not the case with Raoq because Raoq just gives you Mender Raoq and furthermore you have to go through and apply the specific steps for your device. You need to select an appropriate boot loader. For example, if you're building a Raoq image for Raspberry Pi this is going to be your boot in the next slides. I'm going to show you the exact steps for Raspberry Pi. You need to enable SquashFS in the Linux kernel configuration. Now this is important. Raoq supports only X4 root file system. Raoq does not have an X2 or X3 file types so make sure that you are using X4 or a file system that can be mounted as X4 and recognized by Raoq. You need to create specific partitions that match the Raoq slots. Raoq supports various slots which you can update and there is some flexibility how you can have several slots. For example, two redundant slots and one slot that is separate. Furthermore, you need to configure the boot load environment and create a script to switch between the Raoq slots. In the most common case you have A and B slots and you need a partition for both of these slots. Furthermore, you need to configure a specific boot loader script for example for your boot. You need to write the script that switches between the slots and the partitions after an update. Update is applied with a reboot with Raoq just like in Mandarin. Last but not least, you need to create a certificate and key ring to Raoq's file system conf. These six major steps are time consuming and if you are a newcomer to Raoq, it could be a little bit difficult to get started. However, recently, a few months ago, I created this layer called Meta Raoq Community. At the moment the only sub-layer available in it is for Meta Raoq Raspberry Pi and specifically it's for Raspberry Pi 4. At the website of consulka.com I have published a blog post that explains the exact steps how to build a Raoq image for Raspberry Pi 4. There are several steps that you should do. However, these six steps that I've explained are incorporated in Meta Raoq Raspberry Pi so you can just reuse this layer and change your certificates because that could be a security issue. You can get Raoq working with the Yocto project and open and bed it on a Raspberry Pi 4 with a simple minimal or base image. Let's have a look at the configurations that you need to apply in your local.com if you are using Meta Raoq plus this layer that I have created. First of all, you need to specify the machine. Of course Raspberry Pi 4 is provided by the popular BSP layer Meta Raoq Raspberry Pi. All those layers have to use the same Yocto release. The particular example that I have created is using release.dunfo. After that I'm switching to using SystemD. What's important is that I'm adding Raoq to the image with image install append. The image FS type has to be set to X4. That Raoq is compatible only with X4. Furthermore, we need to select the bootloader. The default Raspberry Pi bootloader is not compatible with Raspberry Pi. It's not a good choice. You need to use Uboot as a bootloader because this way I have already created Uboot environment variables and a script to manage switching partitions after Raoq update. This is something that is part of the layer available as Meta Raoq Raspberry Pi in the Meta Raoq community. Here is how to perform an update. After building an image and an update bundle, the update bundle is the phrase that Raoq uses for the artifact that you download and install on the device. After building an image, booting in on a Raspberry Pi, after that you need to make an update bundle to build it with BitBake. Then you can go to the directory where it has been built to start a simple Python or whatever is more convenient for you, a simple HTTP server. On the embedded device, you have to download the Raoq bundle. Here the example is very straightforward. I'm just downloading it with wget into the temporary directory. After that, I'm using the command line command for Raoq to install this update. I'm typing Raoq, install the temporary directory and the name of the update bundle. In this case, the recipe for creating this update bundle is update bundle followed by the machine name which is Raspberry Pi 4. After applying the update, I need to reboot. If everything is okay, the scripts for UBOOT provided through meta Raoq Raspberry Pi, we'll check the status and if everything is okay, we'll boot the new partition which we have booted. Here is a diagram which has been taken from the documentation of Raoq. You can see how the bootloader first boots from A partition, downloads the update bundle, flashes it on the B partition and after reboot, if everything is okay, the B partition is booted and it becomes an A partition so you can do another update if you want. Moving on to something completely different. So far, we've covered Mender and Raoq which both use AB updates and now we're going to talk about Libo history. This is a shared library and a suit of command line tools for committing and downloading bootable file system trees. The interesting part here which makes it very different from the other technologies that we've already reviewed is that Libo history supports Git like model for incremental automatic atomic updates of a file system using binary deltas. So pretty much you generate just the differences of binary delta, you download it to embedded device and you apply it. After an update, a reboot is required so that the changes will take effect. Persistent data is kept in var and atc directory. Previously, Libo history was known as oyster and it's still valid to call it oyster so on many occasions you might hear me and see it in various resources as oyster. Libo history has a very good documentation. I have to say that this is a quite old project. It started with GNOME continuous so it's a project coming from desktop to the embedded devices. Exact steps for adapting an existing mainstream new Linux distribution to Libo history are available as part of the documentation. The source code of Libo history is written in C but language bindings are available through G object introspection. So it's something that you can use with various programming languages. Libo history is compatible with multiple bootloader options including group Uboot and Netram FS. The source code is available at Github under GPL version 2 license. Nowadays there are more than 100 contributors. I have to say that in the past three to four years Libo history became very, very popular because nowadays it's widely used on embedded devices so it's not something that was only used for desktop. The documentation is available at oyster.github.io. Speaking about the integration of oyster in the help to an open and better I have to say that four years ago I was involved in writing the very, very first recipe for oyster. On the left side here you can see a screenshot from Automotive Grade Linux. Automotive Grade Linux is a project of the Linux foundation. It uses oyster for doing software over-year updates. The initial contribution of oyster was done in the summer of 2016 and it was part of the layers of AGL. After that they were moved as external layers known as meta-updater which will explain more details in the next slides. So nowadays oyster is not part directly in AGL but instead it's used through other layers that are incorporated in AGL. Furthermore, last year in 2019 Alex Kiernan included oyster as a recipe in Meta Open Embedded. Meta Open Embedded is probably the most popular extra often open embedded layer providing various recipes. I'm a regular contributor to Meta Open Embedded primarily to Meta Open Embedded. Meta OE and Meta Python. So continuing further, speaking about oyster we need to mention Actualizer and Actualizer Lite because both of these are important for embedded devices. Actualizer is an open source client that runs on the embedded device. It relies on oyster to download and install the update and Actualize communicates with the server in order to understand that there is a date and to apply it. Initially Actualizer was developed by a company called ADS Advanced Elimatic Systems in 2015, 2016. Initially they had an application called RVI Soda Client written in the RIS program in languages, however they decided to completely rewrite the program in C++ and to call it Actualizer. The company with an office in Berlin that's why the name Actualizer sounds so German. I was involved in this process. Actualizer became part of Automotive Great Linux. The company that I worked for consumed the group was under a contract with ADS for this development. After that here acquired ADS and nowadays Actualizer is part of the here portfolio. Actualizer is compatible with Geneva Soda and obtain requirements. As I mentioned it's written in C++ the source code is available in GitHub under a Museo public license version 2. Now there is another company, Founders IEO which decided to create a simple version of Actualizer. They called it Actualizer Lite. As the name suggests it's a lightweight source version of Actualizer which allows two things. These are the two major differences between Actualizer and Actualizer Lite. Actualizer Lite allows anonymous access and requires devices to be always up to date. This is something that's different from the obtained requirements. So why is this different difference? Well obviously things become easier with Actualizer Lite. However Actualizer itself is made by the requirements of the automotive industries for vehicles and Actualizer Lite is more suitable for Internet of Things. There are many Austria based solutions for embedded Linux software updates nowadays on the market. I'm starting with here OTA Connect which is with Actualizer, the layer meta-updater as well as a bunch of additional meta-updater BSP layers, for example meta-updater Raspberry Pi. So the supported devices are Raspberry Pi, Cuemo, Intel X8664 with Miniboard being the reference board RISC 5, Texas Instruments as well as Renesas Ports. Renesas Ports are used in automotive grade Linux. Unfortunately on 31st of August this year here announced that they are removing OTA Connect from their portfolio. So any users subscribed for the free evaluation version will have access until the end of November. Of course Actualizer and meta-updater are open source projects that are available in Github so the source code will remain. However they will be not, in long term they will be not actively maintained by here. But there are many other companies contributing to these projects because they have been already integrated in different products and projects. One of these projects is the automotive grade Linux. The AGO sort of feature in AGL is based on meta-updater, Actualizer and oyster. I've also mentioned Foundry's IEO. They are developing Actualizer Lite as well as they are contributing to meta-updater, Yocto Layer, and they have their own layer called Meta LMP. Furthermore, Toradex is developing the Verizon OTA solution for Toradex Apalis, Colibri, Vergin IMX devices with IMX 6, 7 and 8 systems on the chip. Also these devices must have EMMC for Verizon OTA to use. A Torizon OTA is using Actualizer and Layer Meta Toradex Torizon. You can follow the link here. By the way I'll upload the slides at the end of the presentation. So you can follow the link and learn more details. More oyster-based solutions. Qt OTA also uses oyster and its for embedded devices. It's a home-continuous project to make a platform. A lot of solutions are using oyster. Finally, we're coming to the end of this presentation. I would like to mention Eclipse Hockbit. This is not a solution for performing an update, but rather this is a web-based interface with a server backend for performing an update. It's a domain-independent backend framework for rolling out software updates to constrained edge devices as well as more powerful controllers and gateways connected to IP-based network infrastructure. Eclipse Hockbit is written in Java. It's available on GitHub under the EPL version 1 license. Furthermore, both RAUK and SW Update are using Hockbit for perform updates. In the previous slides I've explained about the RAUK bundles, how you can create a RAUK bundle. Well, with Eclipse Hockbit you can apply this bundle on your device. A couple of screenshots here. This is taken from Eclipse Hockbit documentation. You can see on these two screenshots that from your web browser you can manage the deployments to the devices as well as to upload update artifacts which should be deployed to the devices. At the end of this presentation, let's make some conclusions. There are many open-source software solutions for software updates which have already support for the Yocto Project and Open Embedded. Pretty much all of the popular have support for the Yocto Project and Open Embedded. If you have already selected the Yocto Project and Open Embedded for a build system, there are plenty of options that you have. Obviously, you need to select a solution that fits your need best and there are a lot of solutions on the market, as I have explained. So things have changed since the past five years. So it is very important and recommended to use and actively maintain the Yocto released. For example, the LTS release, furthermore to choose an open-source software update mechanism that is also actively maintained. Software updates depend on the boot loader, especially for ARM devices, Uboot is preferred choice. I've explained this in more details about RAV, but it's pretty much similar for other solutions as well. For AB updates, probably Mender is the best choice because it provides you excellent documentation, easy getting started, and furthermore, the paid plans for Mender have some very important extras such as the Robust Delta update. An alternative for AB update is RALF and SW update. Speaking about Delta updates, Libs Oystery is commonly used as core technology in various open-source solutions, no matter if we're speaking about a product or a service, because nowadays there are so many companies providing as a service software updates. Furthermore, combining AB updates on the host OS with containers coming from meta virtualization, for example Dock Air, is nowadays also often used for embedded devices. This is an interesting opportunity, especially if you have in your organization developers already familiar with the workflow for submitting and using containers. There are many talks about software over-the-air updates. Here is a list of some of them, but before I read all of them, I would like to highlight a talk from last year by Chris Simmons, which was comparing using Debian or the Yoctur project for embedded device. Chris made a great presentation with the advantages and disadvantages. Have a look at it, and if you're still wondering whether to use Yoctur project, this is something that might help you make the choice. Furthermore, the CTO of the company that I work for, Consuelco Group, Matt Porter, did a very interesting comparison of Linux software update technologies back in 2016. Four years ago, things were quite different, and Matt Porter made a great presentation, a very technical presentation, so you can have a look at it and you can compare how things have evolved since then. A few of you once did another presentation about AGL. I've mentioned AGL in my talk, automotive grade Linux. In this presentation from 2017, you can learn internals and details about the software over-the-air updates in AGL. Furthermore, there are interesting talks from Fosland by Enrico Jones and from Bartos at EOC last year, as well as Stefano Babic. There are all maintainers of open-source software solutions for over-the-air updates, and the talks are very interesting and contain a lot of details. Thank you very much for joining this session. Here are some useful links in the description below if you would like to read and learn more about the things that we've discussed. Now we have time for some questions and answers. I'll be happy to hear your questions. Thank you very much for your attention.


What kind of release is that for you? That was awesome. That was awesome. I didn't I didn't start a soft like judge you did but it was still great You know I said we did great. That's all we could ask for I know there's one of the reasons you came here because you thought this team was good enough to do something like this How excited are you for Boston? This is awesome. It's gonna be great Go there tomorrow put it to him this weekend and you know see what happens excited to be here. Yes, sir. Thank you, John. Yep, no problem