Hello everybody, and welcome to this presentation about Leap Camera. For those of you who are lucky enough to join us live from Seattle today, thank you for waking up early. This is the first lot of the day, so I know how difficult it can be. My name is Laurent Panchard. I'm the Chief Architect and Project Manager of Leap Camera, and today I'm going to take you on Leap Camera's fabulous journey. So let's dive into the subject. Before we actually start talking about Leap Camera itself, I think it's important to understand the context and where cameras support Linux came from. What you're seeing on the screen is my very first webcam. It's made of a VHS camcorder connected through a composite video cable to a PCI capture card. It was very bulky. It required an actual tape inside a camcorder to operate, but it did the job. After a month of reverse engineering, it even worked on Linux. That was my idea of a well-spent free time in those days. And after a brief period of crazy ideas, such as connecting cameras to the printer parallel pod, which Linux actually supported, at the end of the 20th century, the world moved to USB connected cameras. For all purpose but nostalgia, it was a very good idea. What did all those devices have in common? A large portion of them had Linux drivers. They were mostly developed through reverse engineering. And all those drivers implemented a video for Linux API. Now, in the rest of the presentation, I will use the words video for Linux, V4L and V4L2 completely interchangeably. But the old V4L1 API is only of interest to historians. Video for Linux is a broad API with lots of features to accommodate different kinds of video capture devices, from TV grabbers to webcams. It's fairly monolithic in the sense that it tries to apply the same model to all those devices. As a result, an application that supports V4L2 will likely work with TV grabbers and webcams alike. At least for the basic features. V4L is also fairly high level because it maps directly to the features that those devices implement and expose. My camcorder, for instance, handled all the gory details of auto exposure and auto white balance internally. And the USB webcams follow the same model. So V4L2 was designed to map to those features and didn't have to care about what happened internally under the hood inside the cameras. So what happened next? Fast forward about five years, the industry standardized on a common USB protocol for webcams that was in 2003. It took about two years to develop a Linux driver for that. It was my humble first real steps in the Linux kernel world. And it got merged in the kernel in 2008. As virtually all new webcams were expected to implement the USB video class, I thought that the camera support in Linux was solved once and for good. The future proved me completely wrong. What did I miss? Well, not all cameras were connected over USB. Linux was gaining fast adoption in the embedded world and there the situation was very different. In embedded devices, cameras are made of an image sensor that outputs data on a dedicated hardware interface, such as MPC-SI2, for instance, in today's devices. DSOC, the processor that is running Linux, integrates a receiver compatible with that interface, which then transfers images to memory using DMA. Unlike in webcams, both the camera sensor and the receiver are directly controlled by Linux. What implications does this have? Well, to answer that question, we need to understand what a camera actually is. The core of a camera, as you likely all expect, is an imaging sensor. This is a device that, broadly speaking, converts light into digital values. It is made of tiny light-sensitive photodiodes with the electronics required to convert the charge into a voltage. Each of them has a microlens together as much slice as possible into the diode, and those are what we call pixels. The pixels are assembled into an array with rows and columns, and additional electronics to rod the voltage of each pixel, one after the other, to an analog amplifier, and then an analog to digital converter. The sensor outputs a digital value for each pixel that's proportional to the amount of light that has reached the photodiode. And this is our first trouble. The diodes are sensitive to an amount of light, so they produce a grayscale image. How do we even get colors? Well, colors are related to physical properties of the light and its spectral contents. It's very important here to realize that the concept of color is deeply tied to the human eye, and the perception that the human eye has of light. Without going into too much details, let's just remember that color processing in a camera is mostly about outputting images that appear as realistic as possible to the human eye. What most cameras sensor do is simply put a tiny color filter in front of each pixel, with the colors corresponding to the sensitivity of the human eye to red, green, and blue. The most common arrangement of such filter groups are in cells of 2x2 pixels, with one red, one blue, and two green filters, because the eye is more sensitive to green than red or blue. This color filter arrangement is called a buyer pattern. Let's look at the impact this has on the image. Image 1 shows a sample scene that we can then capture with 120x80 pixels buyer sensor. Image 2 shows the values of all captured pixels. As you can see from the red flower, for instance, we have a checker pattern of light and dark pixels. The light pixels correspond to the ones with the red filter, which lets the red light through, while the dark pixels correspond to the green and blue filters, which block most of the red light. Image 3 shows the exact same image, but with each pixel colorized with the color of the corresponding filter. You can see a bit better what's happening. We can see that color information is present, but each pixel misses two out of the three color channels. This leads to the typical checker pattern of colorized buyer images. Image 4 then is obtained by interpolating the missing color components using the values of neighboring pixels. For instance, a red pixel in the image is surrounded by four green pixels above, below, on the left, and on the right. So its green value can be estimated by taking the average of those four green neighbors. In practice, to obtain a good image quality, we need more complex interpolation, and we need to take into account more neighbors with more complex mathematical operations, but the base principles still apply. It would be very easy if it stopped there. You see, there's more processing that needs to be applied to the image to achieve an acceptable quality. This is called, in part, operations in the optics and in the camera sensor. For instance, as shown here, the lens will let less light through on the periphery than in the center, which causes an undesired vignetting effect. You can see that the corners of the image are darker than the center. There are plenty of other issues. The sensor may have defective pixels, for instance, that will need to be hidden. The total absence of light when the scene is completely dark isn't rendered black, because there are leakage currents in the photodiode. There's also noise that will affect the image at all stages from the photodiode to the analog to digital conversion, and the list goes on and on. So images need to go through a complex camera pipeline, which is way too expensive to implement in software in real time. Cameras that thus need hardware assistance, and this is provided by specialized devices called image signal processors, or ISP, in short. And if you thought this was complex enough, it's not at all. The luminance of the scene in front of the camera typically varies constantly. You can have a cloud passing in front of the sun, for instance. This requires adjusting the integration time and the gain of the sensor accordingly to produce an image that is neither underexposed or overexposed. The same is true for white balance, for instance, which requires adjusting the color gains based on the light of the scene, or for the focus, as people will have a tendency to move in front of the camera. The parameters that control the lens, the sensor, and the ISP, they all need to be computed in real time. And that needs to be based on the analysis of the capture images. This is, again, very computationally intensive, but fortunately for us, the ISP comes to the rescue by computing statistics on the images, such as histograms, for instance. Based on those statistics, the algorithms will then compute the processing parameters for the next frame and apply them to the device. The same process is repeated for every frame, forever. Those algorithms are often referred to as 3A for auto-exposure, auto-white balance, and auto-focus, but they can include more processing than that. All of this is really key to getting the final quality of the image. Last but not least, the algorithms need to be calibrated and tuned for every combination of a camera sensor and the optics. This adds even more complexity to the development process. Now, do we really have to implement all these formatted cameras? Fortunately for us, no. There are sensors that embed the ISP and a microcontroller to run the algorithms inside a sensor. They are called smart sensors, or YUV sensors, because they output processed images in YUV format, as opposed to the raw sensors, or non-smart sensors output unprocessed by images. On a side note, USB webcams most of the time use smart sensors and when they don't, they integrate a separate ISP internally inside a camera. And integrated webcams in laptops are also USB devices, so that's why Linux didn't have to care so far. Everything was handled on the hardware side, on the device side, under the hood. So that they saved. We can use a smart sensor, we can connect it to the SoC. There's a bit more to deal with than with a USB webcam, because the camera receiver inside the SoC may have additional features, such as the ability to scale, for instance. Still, the complexity of the camera pipeline is very limited. And it can all be exported through video for Linux, because it supports scaling already and all the features that are required for these embedded pipelines. Existing video for Linux application can keep using the exact same API and can be completely unaware that the SoC has a scaler inside the SoC, not inside the camera sensor. Everything works like if the embedded camera was a webcam. Then trouble happened. This is the Nokia N900. It's one of the first Linux phones from a large manufacturer. It's very special because it marks the beginning of the whole camera journey. You will ask me how can a phone release in 2009 mark the beginning of the camera, which was announced nearly 10 years later in 2018. That's a very good question. You see the N900 is based on an OMAP3 SoC from Texas Instruments. Its camera architecture uses raw biosensors connected to the USB inside the OMAP3. This marks the turning point of Linux having to care about all the complexity of image processing. All that I just explained all the required processing is something that no Linux had to be aware about and to implement. Back then I was working with the Nokia Kernel team on drivers for the OMAP3 USB. Video for Linux wasn't ready for this use case. It was missing features. So we designed and developed a new API called the Media Controller API along with many VFOR to extensions to expose the full set of features of the ISP to user space. This is only for the kernel but not for applications. Applications had to program the sensors program the ISP parameters. They had to capture the raw images they had to pass those raw images to the ISP they had to capture the process frames and the output of the ISP, capture the statistics as well and implement all the image processing algorithms that I mentioned before. All this has to be diverse dependent as well because different ISPs behave quite differently and algorithms cannot be generalized. They cannot be portable between different devices. The idea of portable video for Linux applications that would work with different cameras got completely shattered at that point. Nokia had the resources to develop a custom camera stack in user space. It was partly open source, partly proprietary it was based on G-streamer actually at the time. Also had the resources to develop custom applications. But this was beyond the reach of most developers. Whether hobbies, working for small or medium sized companies the complexity was just not manageable if you wanted to develop a camera application running on that. We envisioned solutions to this problem. We didn't develop kernel drivers without caring about user space. So we had designs that were based on platform specific plugins for LibV for IL for instance and were able to wrap a library in user space. Oops, sorry. But unfortunately on February 11th in 2011 Nokia decided to cancel its line of Linux based mobile phones and switch to Windows phone. Development of user space solutions completely stopped at that point. From 2011, Linux was without an embedded camera stack. Developers continued on the kernel side but nobody could or would commit enough resources to fix the situation in user space. Meanwhile, new devices got developed with RAW camera sensors and an ISP in the SoC. This architecture was spreading from phones to ARM based tablets for instance and from tablets to laptops even on Intel based devices. This particular laptop for instance uses a RAW sensor with an Intel Cabil Lake SoC that integrates on ISP. Why did vendors decide to use RAW sensors and not smart sensors or USB webcams integrated inside the device case? Or masochism? Well, it turns out that there are multiple reasons. They are related to cost for instance to size because a USB webcam module can be quite thick so it requires size to integrate it inside the laptop but most importantly to image quality and to features. You see with a RAW sensor no silicon space is used on the sensor itself to implement an ISP so we can achieve larger pixel sizes and larger resolutions and possibly because the pixels are larger you can also have better light sensitivity for instance. And with a separate controllable ISP the vendors can implement more advanced image algorithms. They have full control on those algorithms. A couple of very simple examples are for instance focus assistance based on face detection. Why you would run algorithms that detect faces in the image and automatically focus on them or advanced HDR processing? There are many more complex use cases that exist as well. As those features can be fairly advanced they are often considered by vendors as a key differentiating factor that needs to be covered by the uttermost secrecy. This makes embedded cameras and free software very unlikely friends. At the end of 2018 after contacts with the industry over the summer we announced the Leap Camera Project. The goals were ambitious. Leap Camera was to provide a complete user space camera stack with a new native API and a feature sender that would at least match the capabilities of the Android Camera API. This was way beyond by the way what Vidovo Linux supported natively. And of course it had to be free software. While our initial team was actually coming from current development we wanted to focus as much as possible in this space. Leap Camera thus uses the existing V4L and media controller kernel drivers. It does not come with a new kernel API. Leap Camera had to handle camera enumeration and support multiple cameras concurrently. And it also had to support multiple concurrent streams per camera to be able to capture the same frame in different resolutions and formats. This way an application can for instance use the native screen resolution to display it locally. And a second stream with a different resolution for the purpose of video recording or streaming for instance. Leap Camera would support per frame controls. This means it would guarantee to applications that control parameters get applied precisely to the requested frame. And of course Leap Camera had to control ISPs. So it would need to implement image processing algorithms. We have seen how those algorithms are some of the most protected applications for video recording. So we decided to isolate them in plugins that we named Image Processing Algorithm Modules or IPA modules. This architecture allowed vendors to provide close source implementation of algorithms that can coexist with the open source implementations even for the same platform. We will see later in this presentation how this was done without compromising on the ability to use cameras with three software only. At the beginning we considered the need to be backward compatible with existing APIs to facilitate Leap Camera's adoption. In particular, we wanted Leap Camera to be usable with most Vfile 2 applications without having to even recompile them. We will also see later how that was achieved. We also considered as a main goal support for Android and Chrome OS which are both based on the same Android Camera API. For those who are not familiar with the camera implementation on Android it defines an API for camera provider modules that is named the Camera Hardware Abstraction Layer or HAL. They require the device vendors to provide the Camera HAL implementation and then Android builds the camera service on top of that HAL implementation. With a single implementation of the Android Camera HAL based on the camera we can support both operating systems and Android and Chrome OS. So at the end of 2018 we had a goal we had an architecture and plenty of motivation. The adventure at that point could begin. We started development by targeting two very different devices initially. Our main goal was the ISP found in the Intel Kaby Lake SOCs which is named IPU3. We picked a Chromebook device as a development platform as it had an open source for implementation it has also kernel drivers and a supportive team at Google. We will see later why the open firmware is actually important. The second targeted device was well any plain old UVC compatible webcam. While the webcams don't benefit so much from the camera as the ISPs for the reason we've seen before, we wanted to show that the camera stack could also perfectly support what most Linux users are using today. From the very beginning we decided that tests in a particle unit test were crucial to the success of the project. I cannot emphasize more strongly how that decision turned out to be right. The project would have completely collapsed from regressions all the time if we didn't have that. To support unit testing of the lib camera core without the need for a particle hardware platform, the first device that we actually supported was the virtual media controller device which is called VimC and that was before we even implemented support for the IPU3 or for UVC webcams. VimC is a kernel module that emulates a camera sensor and a camera receiver without having a real hardware device. It has been extremely useful for testing unit camera. Shortly afterwards, we added one final device to the set with the ESP found in the ROGCHIP RK3399. Its architecture is quite different compared to the IPU3 and we wanted to test the lib camera designed with different device architectures to make sure that it could support more. Our initial development device was also Chromebook for that SOC but the same code works exactly the same on other RK3399 machines such as the ROGPY 4 development board for instance. At that point something unexpected happened. Among the platforms that we hadn't considered for lib camera was Raspberry Pi. Raspberry Pi released their first camera module in 2013 with a camera stack that was implemented by a closed source firmware that was out of reach of Linux and as a consequence out of reach of lib camera as well. The Raspberry Pi user base however wanted better access to the internals of the camera system and Raspberry Pi listened to them. They had the code to control the camera pipeline they had a complete implementation of image processing algorithms for the platform and they had a will to open source it but what they were missing was a standard camera stack for Linux that could host those components and that's where lib camera comes in the picture. So Raspberry Pi got in touch with us towards the end of 2019 we worked together for about 6 months and which to be fair was really mostly them doing the work with our guidance and in May 2020 they announced a new camera stack based on lib camera. The code they released was as far as I know the world's first open source production quality implementation of image processing algorithms for an ISP along with that came a tuning tool and very detailed documentation as well. This made the Raspberry Pi a great platform and playground for image processing algorithm development. It was the world's first and even if nobody noticed at the time we knew it was a major milestone. lib camera had its first experience of collaboration and its first real users. The sudden and expecting encounter was indirectly due to Microsoft. We knew about Windows based laptops with an IPv3 and we knew they would be challenging to support. You see the Linux kernel needs to know what camera sensors are present in the system and how they are connected to the system and it needs to control the power to turn them on and off on the x86 machines this is normally handled by the ACPI firmware which describes the hardware and controls the power automatically. However, on those machines unlike on the Chromebooks the ACPI description is very badly designed. It's missing critical information and it requires drivers to implement power management manually. Without support from the device manufacturer which we didn't have as to the schematics which we didn't have either without firmware on Windows driver source code there was really little that could be done to support those machines. That's where the Linux Surface community comes into play. There are a set of users turned into developers who had different Microsoft Surface machines and they teamed together to try and get Linux up and running on them. Needless to say they were at least with the lack of camera support. They studied the camera ACPI description, they went to reverse and they needed a firmware to obtain the missing information. We held them with that and it ended up requiring a significant effort to understand what was going on behind the hood in the firmware side. After great work on the kernel side from some of the community members and nearly being driven crazy by ACPI atrocities they managed to get the sensors detected. The camera was at that point ready to capture the first images from those devices and that's how they looked like. You see, at the time, Linux camera had support for the IPv3 but no corresponding image processing algorithms. And without the IPAs, this is how bad it gets. Still, it was a breakthrough achievement and I really wouldn't have bet on that six months earlier. Far from being discouraged by the appalling quality, I tried to hack around to improve it. Some of them even got a bit poetic actually. A user noticed that pictures had a tendency to be very purple in low light conditions and very green in bright light conditions. So that user took a picture through the window on a bright day and this part of the picture was named the sky after rain with leaf. The story doesn't tell if it was later sold as a non-fungible token. Art is great and it doesn't make for a great webcam. So we decided to fix this and one of our developers spent a few months implementing initial algorithms for the IPv3. This is what he achieved. As you can see, this is lots of room for improvement. For instance, to address the lens shading, the corners are definitely darker than the image center. But the reason starts looking like a camera. So at that point, more than two years of work is being done in to the camera. This underlies my earlier point. Without a user space camera framework, the development effort to write a camera application two years is simply not realistic. Still, this reality isn't widely known among Linux users. This screenshot, for instance, shows a bounty for the camera support in Linux on the Microsoft Surface machines. $810 to cover all the counter in user space development multi-year work that's a bit of an effort under estimation. Of course, lots of free software gets developed by hobbyists in a free time, if a free. And the community has great talents. These can be seen, for instance, by the amazing GP reverse engineering projects. Nonetheless, with the number of different ISPs on the market today and the fact that new ones are being developed all the time, solving the issue of camera support on Linux will definitely require involving camera vendors. The journey leads us to today. What have we achieved? What have we done so far with the camera and what are we busy working on? I think it's fair to say, to start with, that we have managed to create a user space camera stack for Linux. At the central piece we have a camera manager. It can enumerate all media devices in the system. It supports hot plug to notify applications of camera audition and removal. And the camera manager pairs the media devices that are detecting in the system with the device specific back ends that we call the pipeline handlers. The pipeline handlers, in turn, create one or more cameras and register them with the camera manager. From that point the cameras are visible to applications and they're ready to be used. What does it work behind the scenes? As we've seen before a camera needs a complex pipeline of operations. So to support this the device specific back end is split in two parts. First we have the pipeline handler. This is the code that is responsible for all the interactions with kernel devices. It configures data routing inside the devices. It manages internal memory buffers. It handles image captures. Overall it strikes the details of device operation and exposes the video streams to applications. The pipeline handler doesn't implement the image processing algorithms that produce the ISP parameters. This task is performed by the separate IPA module. The module passes statistics produced by the hardware and it computes parameters for the sensor and the ISP. IPA modules do not have direct access to kernel devices. That's a very important architecture limitation that we have put in place in the camera. The only way to communicate with the rest of the system is through the pipeline handler. They will receive the statistics and other metadata from the pipeline handler and they will send the computed parameters back to the pipeline handler. The parameters will then be applied to the device which means that IPA modules cannot cheat. They cannot go and access undocumented device interfaces behind the scenes. They are bound by what the pipeline handler implements. The IPA modules are implemented as shared objects that are loaded dynamically which, as I mentioned before, allows the coexistence of IPA modules developed by the live camera project and third party modules supplied by camera vendors whether they would be closed source or open source. We allow vendors to supply closed source IPA modules. This sandboxes them in a separate process to make sure that they will not affect system stability. We enforce the rule that it must not communicate with devices directly. This does not mean that live camera requires closed source component to function. The pipeline handler is part of the live camera core and its open source. Its communication protocol with the IPA module is documented and we also require vendors to provide an open IPA module implementation that enables at least image capture with basic image quality. The format of the ISP parameters of the statistics as well must be documented and this enables the community to then start with the basic IPA module and write free software implementations of algorithms and improve them. As the impact of sandboxing to keep development simple the IPC mechanism to communicate with the IPA module process is handled completely transparently for both the pipeline handler and the IPA module. To further its development live camera provides a large set of helper classes. They cover various areas such as wrappers for the VFOL2 and media control API such as pixel format abstractions threading and message passing helpers logging infrastructure and the list goes on. These are not areas where anyone innovates but today everybody, every vendor ends up reinventing the same wheel in the camera stack. With live camera the vendors can rely on well-tested helpers and they can focus solely in the pipeline handler and the IPA modules where the real added value is. All of this exposes cameras to applications through the live camera native API but live camera offers more than that. The adaptation layer that sits on top of the native API interfaces live camera with other APIs and frameworks because all the device specific code is located in the pipeline handler and in the IPA module the adaptation layer is implemented once and works on all devices. In no particular order the first component in the adaptation layer is an Android camera model module which implements a camera provider for the Android camera service. It gives free Android support for any platform supported by live camera. So once again we have one implementation of an Android camera module inside live camera and it supports any device that has a pipeline handler and an IPA module. As mentioned earlier because ChromeWise uses the same camera module API as Android, live camera also supports ChromeWise out of the box. The second adaptation component is a Gstreamer source element called live camera source. It provides multi-stream capture in Gstreamer pipelines for any camera supported by live camera. It has been successfully tested for instance with cheese the GNOME camera application. The last adaptation component that we have in live camera implements vFile2 emulation. It allows existing vFile2 applications to use live camera without any modification being recompiled. So this can be used with closed source applications as well. We achieved this with a shared object that is pre-loaded inside the application and that intercepts all the calls to the C-limber. When it detects that those calls attempt to access a vFile2 device it will then redirect them to the vFile2 emulation code that translates the calls to the live camera API. This is a best effort approach because it's really impossible to fully emulate a 100% of vFile2 so we can't have complete API coverage. But we have implemented the most used features and we have for instance successfully made video calls with Firefox using this technique. Let's summarize the currently supported features in one slide. We have a camera stack it supports multiple cameras with multiple streams per camera and perferring controls and hot plug. The list of supported devices is limited with a flagship implementation being Raspberry Pi today. Raspberry Pi and IP3 both have an IP module and the last two devices on the list NXP and Allwinner don't have any ESP so only smart sensors can be used with them. We've just seen the three components of the adaptation layer with GStreamer, Android and vFile2 being supported. Libcrum also includes tools such as a camera tuning tool for Raspberry Pi for instance and a tracing infrastructure for debugging. We have three simple applications one command line application that implements all supported features, one graphical user interface application and one simple application that we use as a tutorial to show how to implement Libcrum native application. We have extensive documentation on the API with 100% coverage as well as high level tutorials and tutorials. The API documentation is published on the Libcrum website with Nightly Builds so you can get it from there. The guides are currently available from the sans code repository only. They can be compiled to HTML and we will publish them on the website too. Many developers are better at coding than writing documentation. I'm sure many of you are aware of that fact. When developers do not viscerally hate it completely. It's an investment. But one of the lessons that Libcrum wrote to me is that it pays off to enforce a good documentation policy from day one. We are of course on sitting idle. We are working on support for additional platforms. This includes an SOC from MediaTek using many IOT applications. The ISP is left out for the time being so only smart sensors can be used with a platform right now. We have also started implementing support for the NXP IMX8M+. This is the first SOC from NXP that includes an ISP. And here we are targeting the ISP support array. Another platform that we have started looking at is the Libcrum 5 phone. It's based on raw sensors but doesn't have a hardware ISP. So we are exploring the implementation of a software ISP that will be running on the GPU because the software implementation running on the CPU would be way too slow. There is more work in progress. We are improving IPM modules for the Intel IP3 and the ROG chip ISP with the goal to bring the quality on par with the Raspberry Pi algorithms. We are also extending the Libcrum API with new features and new use cases and at the same time we are cleaning the API to move towards the first 1.0 release. That's a big scary milestone. There is lots of work in progress on the Android Camera High Implementation as well to pass the Android Confirmance Test Suite and we are also implementing Python bindings for Libcrum to support new communities of users. We expect that bindings for other languages will also be developed in the future. I'm sure that's some of you in the audience are thinking about Rust already for instance. On the integration side we have a prototype of natively camera support for the Chromium API. This is a screenshot of a video call in Chromium with Jean-Michel, one of our developers on the top right, who is using a surface go-to running Libcrum. This is running an old version of Libcrum from before we had IP3 algorithms which explain why the images are green. Today the quality is much better and you would actually not notice that Libcrum is involved if I took a new screenshot of what we can do today. This I think is how we will judge if Libcrum is successful in the end. We'll have done a good job if users don't even notice it exists. Another part of the work that is not very visible is our participation in the industry initiatives. We are an active member of the embedded camera exploratory group that is hosted by Cronus and the European machine vision association. The goal of the group is to explore opportunities to standardize a camera API. This is something that was tried before by Cronus with the OpenKCam API proposal years ago but the working group was shut down before any proposal was published. Quite obviously we think that Libcrum is the right solution to this problem. In parallel to that we also have multiple bilateral contacts with SoC vendors to try and work on Libcrum adoption in the market. Speaking of vendors, we've already seen some advantages that Libcrum brings. Let's now have a look at what implementing the Libcrum stack entails for that. At the bottom of the stack we have the kernel drivers. As long as they implement the media control API, no change is required in the kernel site to work with Libcrum. This being said if you want to upstream your kernel drivers this may of course require changes as part of the review process with the kernel community. Libcrum has driven the development of extensions in video for Linux and the media controller to fulfill the needs of the platforms that we work with. We have also encountered ambiguities and design deficiencies in video for Linux and we work on fixing them. The Libcrum team has extensive experience with kernel development in the media subsystem so we can also help vendors in this area if their platforms have needs that are not covered by video for Linux yet. On a side note, it was quite interesting experience of humility to realize that some of the problems are mentioned in video for Linux were actually in APIs that I had designed myself. And that's another lesson learned from Libcrum, a kernel API that only gets validated with tiny testing tools for instance, without a real user space stack will most likely have defects. Libcrum is however user framework. It's not a hostile takeover of kernel development so we cannot help vendors to bypass the requirements of the kernel community. On the kernel side, you have to do homework and get the drivers upstream. We have already seen that Libcrum provides an extensive set of helpers that help producing the development complexity and development time by avoiding the need to reinvent the wheel. The adaptation layer is also shared by all cameras as we've seen through the vendors from having to write the gstreamer support manually. The early components really that need to be developed specifically for a platform apart from the kernel drivers are the pipeline handler and the IPM manual. That's all a vendor needs to do. This is the responsibility of the vendor, of course we can help if needed. There's extensive documentation. There are example pipeline handlers that can be used to provide guidance and we also here to provide support. Aware on licensing because it's important. The Libcrum recall and the adaptation layer are licensed under the LGPL. This includes the pipeline handlers which need to be published according to the license. The IPM modules however are not covered by the LGPL. Only publishing the code is required to comply with the LGPL. Upstreaming is not a requirement. We however strongly recommend upstreaming because the best results are achieved by working together. Forks are very costly to maintain. The kernel code of course is covered by its own license which is out of scope for the camera. Let's also note that both the pipeline handlers and the IPM modules can link to third-party libraries if desired as long obviously as the licenses are compatible. Close source IPM modules are fully supported. I mentioned that before. Even if you would like to encourage vendors to follow the lead of Raspberry Pi and the algorithms as well. This presentation would of course not be complete without talking about the future of the lead camera. I present here the features that we envision but haven't started developing it. In the lead camera core itself we have already thought about quite a few interesting features. We have per-frame controls today but they are currently global to the camera and we want the ability to set per-stream controls too. This could allow setting for instance digital zoom factors for different streams. We are also considering a higher level use cases on top of lead camera to offer features such as zero shutter light capture or exposure bracketing HDR. Both of these would be implemented by capturing raw frames pre-processing them in software and sending them back to the ISP for further processing. Two other important features that will eventually make their way in lead camera are still image triggering with focus support as well as logical cameras. Logical cameras is something that's available today in many phones in the market. It's the ability to combine multiple physical camera sensors in order to create one logical camera. There are a variety of use cases for that such as for instance seamlessly switching between a wide angle lens and a tele lens when zooming in the image or using multiple sensors to infer depth information in the scene for instance. We really want to expand the number of supported devices. I've already mentioned ongoing work with the LibreM 5 and a GPU based ISP implementation and we have ongoing discussions with more vendors. Support for older devices would also be great. I would personally be very, very happy to see the OMAP3 ISP the one that started all this supporting lead camera and finally bringing an open source camera stack to the Nokia N900 and the N9 phones. That would be a great community project. I would love to mentor that. So please volunteer and don't be shy. Paul Koszalkowski from Butlin will talk tomorrow about his work on a color driver for an ISP fund in all winner SOCs. He has posted a code a few weeks ago for review and it would be an interesting platform to support the Libre camera as well with a community based effort. If you would like to attend his presentation, which I recommend doing, don't hesitate to tell him you want to volunteer. We will of course continue working on the Android camera howl and support the newer versions of the howl API as they are getting developed as part of Android. Support for additional features such as zero shutter lag in Android for instance is also in scope. Lib camera wouldn't be very useful of course if it wasn't integrated in frameworks in applications and in distributions. In the frameworks category there is no scope. But so are OpenCV for instance or Qt Multimedia, Electron or just your favorite framework. On the application side while Firefox already works with Lib camera using V-Fold to emulation native support would be much better. I was also very tempted to add native Lib camera support to OBS when recording this presentation but unfortunately didn't have time to do so and I hope that many other applications will also follow. On the operating system side now we package by Chrome OS by Build Roots by Debian and Stable too and work isn't going for Fedora as well. This one will take more time but being included in Android is something that I'm really looking forward to. This concludes the presentation. The slides should now be available from the Linux Foundation website. I hope you found this interesting and regardless of whether you're a user, an application developer or a device vendor, please don't hesitate to come and talk to us. The Lib Camera team can be contacted through our public mailing list and the NSC channel and you can also contact me directly by email if you prefer. I am now available for questions in the conference chat channel. I would like to thank you all for attending the talk and I hope that you will enjoy the rest of the conference. Thank you.


Smart homes are becoming increasingly popular as they offer convenience and efficiency through automated appliances and communication with users. Technologies such as smart grids, which connect homes to the power grid, provide energy savings and improve reliability. Smart homes and smart grids are advancing rapidly, offering many opportunities for innovative solutions. This article was authored by Gabrielle Lobacaro, Salvatore Carlucci, and Erika Lawström.