your software efficiently and safely. And we'll get into a little bit more about why I chose those specific words. And it's also a way to protect your users from intrusive third-party applications. And it will provide more security for what are typically shipped as native applications today. It's decentralized to avoid App Store lock-in and it has a rich suite of developer tooling, some of which I've built, some of which some other people have built and we'll demo some of that today. And what we think this will allow you to do is to ship more frequently and with less latency from your release times than any of the distributions or both distribution channels and Linux distributions have allowed for us to do. So just to give you a background of who we are and like our understanding and knowledge and why we think and we have the hubris to believe that we can solve this problem. A bunch of us have worked on distributions but Alex Larson is the lead project and two very important projects which he worked on the past were called GLIC and GLIC2. And these were single file bundle applications for Linux much like people were really enamored with the .apps on Mac years ago. So it was an attempt to kind of try to figure out how we could do that on Linux. He also happens to be the person that fixed Docker so it could run on a lot more operating systems with different file systems than the AUFS that was shipped years ago. So incredibly, he has an incredible strong understanding of how the mountain name spaces work and the details and the difficulties involved in these very cutting edge file systems like the problems with overlay FS for example which we run into a lot of them or whether or not we should rely on butter FS and whatnot. And another very key person in the design of this tooling is Colin Walters. Colin Walters invented and designed a product called OS Tree. And, excuse me, additionally Linux user to root. And then we have a large group of collaborators on this. Red Hat obviously where I work. Endless computers are shipping an entire OS where all their applications are flat packed. Collaborate, Code Think, Intel, Convoke, Solis, KDE is working on some stuff as well and a bunch of individual contributors. So I'm gonna split this talk kind of up into three sections and I'm gonna try to get through the details of it fairly quickly so we have a little time for demos and questions. But I'm gonna split it up into what flat pack brings to users, which I assume will be everyone here. What flat pack brings to developers which will be some subset of us here and then what flat pack brings to distributions which will also be some subset. So for users, Linux has been fragmented for those of you that haven't been around for the 30 year history of it. Linux has been fragmented from the very beginning and it's been very difficult to make and ship software that could run on all distributions. About 10 years ago some technology started to come around that allowed us to think about this differently but it's taken a long time to stabilize and flat pack was built from the ground up, day zero designed, implemented and designed before implemented to be cross distribution. And it actually follows through on this. We have some really good examples of how this works today and we'll talk about the technology that allows you to do that. And we have really easy installation. Currently that's mostly through GNOME software and that's getting used by a few different distributions. But it's also, GNOME software is not gonna be the only software installation tool that can do this. There's software patches for different projects out there that are starting to land. And we have buy-in from a couple different desktops. Obviously GNOME cares about it. KDE, Solas, Fedora. And for those that aren't into the GUI stuff we have command line tools and we think that we're the first system here that can support proprietary graphic drivers top to bottom. And the reason we think that is because we did a lot of work with those graphics driver companies to be able to do that. So like NVIDIA, we have NVIDIA working inside a container and we'll discuss a little bit why that's so difficult later. In particularly with static linking and such. We don't rely at all on the host system. The only thing that relies on the host system is the setup process. So you don't have to deal with some of the libraries coming from the host, some of the libraries come from this like bundling or whatever. It's very clear where your application will use stuff. And really what that means to users before I get in too much of it. What that means to users is you don't have to worry about it. You can have applications that will work on some of your older machines. You can have applications that are gonna work on your machines when you've dist-upgraded like to releases later. They're still going to work. That's something that we've traditionally had a problem with. Those of you that maybe like bought a game 10 years ago or something. Like I bought the Quake 3 and like Unreal Tournament for Linux. And there's no way those run today, right? Like they were using libc 2.96 and like different ABI's and Linux threads for people that knew that before POSIX threads. Here's something that's kind of important to me. With efficient installs and updates. I used to live in the forest like way out in the forest. I had a T1 line. And this is up till about two months ago. I had like 1.53 megabit symmetric internet. Having an efficient download system is absolutely critical to a majority of the people on the internet. Very few people have fast internet. And everybody has only partially connected internet. So if you have a system that has large downloads. Whether that be an upfront cost to get all of the package header metadata. Fedora was terrible with this for years. Or your downloads are the entire bundling over and over again.  This is terrible with phones. I'm sure you have those of you that use like an Android or iPhone. Maybe have updated Twitter or Facebook. And burnt through 200 megabytes of updates. And they update their apps five times a month. And you just burn through all of your data. So this, and that's here in an area where we have fast internet, we have access to this. So this is a big problem. And it is exclusionary if we don't get it right. So something that's built into the layer underneath Flatpak, which is OS tree is this concept of static deltas. What we can do is take two different versions of your application. We can compare them. And we can compress it into the minimal diff between the two. We use a tool that was invented on BSD years ago called BSDF to do efficient differences between binary applications. So those are into the geeky details. It looks at the elf headers and it tries to figure out if like instructions were slightly off and compresses this down. So between that and the static deltas, we get very small updates. Also going from zero to an installation is just an empty tree compared to an installation tree. So we can do a static delta for that. So it too is very fast. It's like just doing a single download. As I mentioned there, they are trees. Those that are familiar with Git, it's basically Git for binaries. OS tree can be simplified into that. Now that said, we don't have that problem last week. I'm sure some of you saw that there was the big SHA-1 attack on Git. This does not affect OS tree due to its difference in design. The problem with Git was that SHA-1 verification was only happening at your commit level. It did not go and validate each level all the way to the root of the tree. The design of OS tree from the beginning and if you're using Git, it's called GitEV tag which was written by the OS tree author. It does a defense in depth of checksum. So it includes the entire height of the tree. So even if there was this attack on it because you get that check all the way to the root, it still would not have been vulnerable to these attacks. One of the ways we also get smaller download sizes with Flatpak is this concept of an application versus runtime split. And what that means is you don't have to bundle everything. Your application is only gonna bundle the parts that you need that are not in your runtime. And we do that for a couple different reasons which we'll talk more about in the distribution section which allows us to share some of the burden of us kind of rethinking how we build and distribute applications. So by splitting these things out, we can make the applications very small, that adds some more work to the runtime problem and we think that by us coordinating together we can make that less of a problem. It's a little nebulous at the moment but it'll be more clear later. Another cool thing we have to provide for applications or for users of applications, as I mentioned the Git one issue. Applications will update automatically or sorry not automatically but atomically. We do not automatically update your applications today. We may do that in the future, I'm not sure. But the interesting thing and what I mean by atomically is the update to the application either succeeds or fails. And this is important because universally all of our distributions are actually broken with this today in very subtle ways. For example, if you're running Firefox and you update Firefox while you're running Firefox there's a small percentage chance that it could have not accessed a file yet that gets accessed while the update's in place and you end up getting like a new version of the file versus the old version. And so now you're running this inconsistent state of the application. This is not a problem with OS tree due to how we lay out the, how we explode out the application at runtime. So you are guaranteed to have either the entire old version and the entire new version and atomically switch between the two. And I guess with trustworthiness comes sandboxing now. This is kind of a state of the world moving forward. And what that means is we do not give applications access to system resources. You do not have access to the sound card directly. There are some caveats to this, like yes for graphics drivers we'll be doing it but you do not get access to the camera, right? Because if you have access to the camera you could potentially start taking pictures of the user or things like that, right? We don't even give you access by default to the home directory. We have a really neat trick for this. This concept is called portals. We learned from various previous security analysis that just asking people if they want to do something turns out to be a really bad security UI. This is the case with Firefox, right? When Firefox would ask you, hey do you want to accept this SSL certificate? I don't know. I don't know if I want to accept it. So how we provide this UI to people is an important part to keeping the integrity of the security. What we call this as portals and for example to give you access to files in your home directory we'll show you a file browser just like normal. Except for that file browser is not run in your application space. It's run on behalf of the operating system for lack of a better term. When you select a file that's implicit security grants to the application that yes you deserve to have access to this file. And then the way it works is we do some combination of fuse and file descriptor passing to get that file into your container. Give it access but never access to the raw file. You get access to a intermediary who can then write the file back. And the reason we do this is because there's actually a lot of programs that do not do POSIX renames correctly. What I mean by that is if you open a file for writing and you start writing and your program crashes in the middle of it, you might be left in an inconsistent state. So the way to do this correctly on Unix is you write a new file, a temporary file and then you rename it into the place where you want it to be on a destination. That way it's either in the old state or it's in the new state. And that's because the guarantee on POSIX that rename is atomic. So we piggyback on that and use that to make all the stuff work. So you can do like a lot of the other systems to be practical to, I couldn't repeat the question actually, what happens when you have applications that need access to .files? So there's a couple of different things. If it needs access to .files that already existed on the system, then it would need to have some override set that like, hey, I need to use these files. And that's because the sandboxing right now is opt into as much as you want. And I think what's gonna end up happening with the UI is based on how much of the sandbox you wanna roll back or like prevent from running will be how we show the trustworthiness of the application to the user. So there'll be some sort of like implicit security flow. I guess on Android it asks you what permissions. I don't know if that's the design we're gonna use because we're definitely gonna put it through a design phase to do that. But if you only need .files in your home directory and they only matter for that application, you probably shouldn't be putting them in home, you should be putting them in what's called like XDG dataters or user dataters or one of the .config or .local somewhere in there. And those are mapped for every application that starts. So they're persistent between runs, none of the other applications can see them, only that application can see them. So you can persist all the state you want but never have your state exposed to other applications. So that helps prevent like coordinating application attacks. Okay, yeah. So Wayland is an important part for us here and that's because fixing X, it's difficult to say. So fixing X is possible, but it's really, really, really, really not possible. If you ever wanna like argue about whether or not that's true I'd be happy to because I've worked on a lot of X extensions over the years and tried to use them and make them work and they don't. Plus it really all comes down to like what people have with X and like what you connect to and nobody has these extensions installed. It's like necessary to try to do some partial bits of the security. So and then those are almost in direct conflicts with how applications like to draw. So you have to re-architect all the toolkits to try to work in this different way as well. So anyway, once you have Wayland and you have the rest of the security stuff set up, the goal is that applications can't snoop on each other. We don't have that attack where like you could just get an update every time this other application repainted and then like screen scrape their data from the X server and figure out what passwords they're doing between that and like snooping on their key presses. Like that type of stuff will no longer be a problem. All right, I think that's enough for users. We can go on to what we have for developers. Oh wait, did I go backwards? Where am I? That was interesting. All right, like I talked earlier, cross distribution. That helps developers as much as it helps users. For users it's nice because you don't have to choose a distribution based on what application you use. You can choose a distribution on what you care about, like what community you're part of or what not and not restrict your application use. But for the developers it's really huge because you don't have to go test something five, 10, 15 different times and then for each of their last five LTS releases versus rel and all that type of stuff. So some of the other systems out there do rely on the host ABI and as someone who works and maintains some of the upstream host ABI, that's a really bad idea. Don't rely on us to be absolutely perfect with ABI because it's really difficult and we think we do it and we still don't do it. And we can get into a little bit of that later. But for example, the one here that I find very problematic to building a robust system long time is it's something the app image does and they whitelist certain libraries from the system and relies on them. That's something we tried to do at VMware since about like many years before I worked there, like 1999 and it was an absolute disaster. Trying to get this set up in such a way that you could have your applications run and rely on any mismatch of host system libraries, I mean just everything broke. And then the common complaint would be like, oh, upstream should fix their stuff. Well, it's like, well, that stuff's out in the wild, so how are you gonna work around it? We don't rely on any out-of-tree LSMs. Most of the LSMs are upstream these days. The one that we do support is SE Linux. It's not required, it's there. And you can use, you can do this all just like mainline Linux kernel. There's no like exotic kernel features we need to be able to make this stuff work. And if you are on a really old kernel that doesn't support some of the newer features, the one in particular I'm thinking of is called username spaces. If you do not have username spaces, we can make all this stuff work with a suid helper. And that suid helper is 2,000 lines of C, very easy to omit and is in the process of having that done so. We only require a POSIX compliant file system. So pretty much everyone has those unless you're using overlay.fs and I will tell you that you should stop using overlay.fs. But people won't. We do hardlink farms, content addressing, those that don't get familiar with it. There's this really cool feature in butter.fs and it's landing in XFS right now, it's called reflink. And what reflink is, it lets you do hardlinks between files except for if you change the file that is hardlinked it will become copy on write. So you get very, very cheap file system copies and only when you make a change should they become copy on write. So whether or not you take any of this in, something that you should totally keep an eye on is XFS reflink. And I say XFS reflink because who knows when we're gonna use butter.fs. Atomic upgrades, you don't have to worry about your applications breaking. And then the final one here, runtimes provide predictable and reliable user spaces. We think this is super, super awesome. Every time someone runs your application, maybe you have a problem where something was in user lib64 versus user lib, where do you find plug-ins, all sorts of stuff. And it's different on every single person's computer that is running your application. And you always get those like, it's broken for me. And I don't know, it works for me, you know this seems to work, right? How many times will we close bugs like works for me, sorry. This single piece allows us to simplify testing drastically. Every time your application runs, the user space is going to look almost identical. I say almost because there's a couple things that can change. The GL library for the system hardware enablement and extensions, which I don't really get into too much here because it's not super important, but extensions let you do things like gstreamer plugins so they like get overlaid and like no matter what applications you have, you can have additional plug-in enablement. So for the general case, what your application sees as their system is always going to be the same no matter where you run it. So here's one of the things that I work on more and it's the build tooling. Flatpak Builder is a simple command line program and you give it like a JSON file that describes your project, like your name, a list of dependencies, maybe those dependencies have like a patch that you need to apply because upstream doesn't maintain things well or doesn't do something in quite the way you want or maybe you want some performance improvement. There can be tons of things that you need to patch and people and distributions often do. And it can take all of that, wrangle it together and generate your final application. One of the really cool things about that is it does it in a build environment that almost identically matches your runtime environment. The only difference is it still has the compiler tooling. So you have a compiler inside that runtime, you have the headers, the debug symbols, stuff like that. But it's the same runtime that your application is running with the additional bits for building stuff. We have this thing called OS tree cache points. And what that does is at each stage of the pipeline building your application like each of your dependency libraries maybe, it takes a cache of the entire state of that build system and it stores it away for later. So if you have a continuous build system based on this and one of your upstream libraries updates get, it will only rebuild from that point forward. So we do nightly builds of all the good home projects like this and it could take like 20 seconds to rebuild the project. It's really, really fast. It all depends how deep in the pipeline it was changed. And it is linear or serial in this fashion in that we will build from the change point all the way forward and we do that to try to be somewhat close to reproducible builds. And we're not there yet. Reproducible builds are really, really, really hard. And that's because maybe your compiler takes the date and shoves it into your build or something, right? Like you can't make that right every time if you're dynamically generating the date from running the command on the shell. So you can't do things like that to get reproducible builds and you also can't do things like you can't use compilers that will use uninitialized memory in the output because that could be different each time. But one of the best things we've done to get close to reproducible builds in this process is sharing compilers between different people building the system and having a predictable user space. Those two things get us a lot further with a lot less work than all of the other reproducible build stuff, which don't get me wrong, is hugely important. But it's that 80% easy versus that 20% really, really hard. So if we can get everyone forward to a certain amount, we all end up getting more secure systems in the process. QA testing is never done enough, which means it's even worse if everybody has to do it 10 different times.  Like there's a big push for automated QA, that's great. It never fully replaces manual testing. You have to still do manual testing. Now, if you have a project that you use and one distribution is shipping one version of one dependency on theirs and another distribution ships a different dependency of theirs, you have to test both.  Not to mention that maybe the library broke in between the two versions and now you can't ship your application on both distributions because what you need has been ABI broken on one and doesn't exist on another. Ask me how I know. As I mentioned, build and test in the same environment as your users, big use there. And then the last few things are things that I'm working on now. I've built this IDE called Builder and I demoed it a couple of years ago here and the primary build engine behind Builder is Flatback Builder. So it's in the process of making us get contributors to GNOME into our life cycle of creating and distributing software, very easy. We're basically down to cut and paste a URL and click build and I'll demo that in a bit. The profiler here that I wrote and maintained called SysProf and it integrates, I think it's the first profiler to work with containers and that we do all the work that's necessary to, if doing profiling in a container, be able to extract symbols from the L files, everything natively and it just kind of works so. And then the part I'm working on now, hope to land here before too long, is a new debugger for containers and this is traditionally very difficult because GDB has required a lot of patches to do container stuff. Anyway, I'll have a big blog post about that in the future. It's kind of nebulous at the moment. Okay, so runtimes versus SDKs. Runtimes are this shared base system for applications to target and different platforms might have different ones. So GNOME has a platform, Free Desktop has a platform, KDE has a platform. I assume that Solis and some of the other projects that are working on things will have their own platform. You probably don't want to create your own and it's because there's a lot of burden, right? If you have this runtime and it's doing libraries like GTK, GLIB, Geo Objects, stuff like that, you probably don't want to deal with all the CVE potentials, right? For like SSL, OpenSSL and stuff like that. So we want to keep the number of runtimes small so that we actually get the value of like a shared group of people working on runtimes together because that CVE tracking is a lot of burden. But the feature that we get out of it is this small per app downloads because most of the things that your application's going to use is going to be in the runtime that you and all the other applications that depend on that same runtime don't ever have to download the runtime bits again. You only have to download those once. So what we find is like even Builder, which is an entire IDE with like 50 some plugins and 200,000 lines of C, like the application is less than five megabytes. So having these like runtimes really, really does bring the size of the applications down. So as I mentioned, you probably don't want to make your own. I personally suggest the GNOME one, but obviously I'm a GNOME hacker, so I take that for what you will. I mentioned this just briefly earlier that an SDK is just a runtime without the developer bits removed. So that could be debug symbols, headers, package config files with valid, it's Vapi files, any of those little accessory stuff. And you can rely on an SDK instead of a runtime because it is a runtime with all of the stuff in there and that's what we do for Builder. Builder depends on the GNOME SDK, not the GNOME runtime. And that means that I have access to like libclang to do auto completion for C and C++ and all of those little bits. So a little bit more on portals. They run out of process as I mentioned and they don't get raw access to things. And these portals are meant to make things seem as if everything's on your local computer, like you shouldn't know that you're using a portal, but it should be very clear that it's authorizing elevated privileges to the application, but only through safe APIs. You can open documents as well, URLs, you can share things, that's one of the newest portals is the ability to share and that might like mean you wanna copy and paste something and share it to an email or share it to a website, Facebook, all that type of stuff. A capture portal to use to take a photo without giving your application access to the raw camera device. And we're actively going and working upstream. So Wim has been working on both Pulse Audio and Pinos and this is allowing us to do sandboxing of these frameworks. The difficulty here with audio, very low latency problem, getting Pulse Audio to be safe is a decent amount of work. The problem here is you may need to add echo cancellation to your recording stream so that you can do video conferencing. The way that works is you load these modules into the Pulse Audio Damon so they can do it as efficient as possible. In some cases it's even pushed off into hardware. Now if you have a client that's talking to Pulse Audio Damon to stream audio and you end up loading one of these modules because you know there's an exploit and then you can use that to get elevated privileges, not really gonna help us from a sandboxing standpoint. Pinos is the same thing but it does it for video. So video capture devices. It's basically like a Pulse Audio for video. So and I've heard some rumors that maybe it's going to subsume Pulse Audio. I don't know if that's the case but we'll find out. Anyway, we have plenty more portals to write. They're not difficult to write because of how we design this. And so if there's something that your application needs you can either talk to us to write the portal for you or come help write portals. They're not difficult. They don't take too long. For us by using Flatpak we're getting a ton more testing before release. This means we're shipping more stable builds every release of GNOME. I have a nightly channel for Builder which means I have people testing my application within like six hours of me shipping code to get. This has been huge. I get bug reports the same day that I'm pushing code so I'm, you know, the code is still fresh in my memory and it's very easy for you to set up if you set up yourself a Flatpak repo. Some people are doing betas and having like the normal stable beta nightly so you can do that. The other stuff, the last one here is actually pretty cool. Multiple architecture support. Thanks to QEMU getting some features fixed in the Linux kernel this last cycle. You can take, well you've always been able to do this for a while. You can take an ARM binary and you can run it through QEMU on an x8664 system. Not even a whole virtual machine. You can take like a single binary and it translates the syscalls to the host syscall interface from the ARM syscall interface. But now you can do it inside a container seamlessly as of like 4.9 Linux kernel I think. So this means that we can actually run ARM Flatpaks on x8664 seamlessly without you doing anything. It does all the hooks correctly to set up QEMU. So this might be useful as we get to the point of doing simulators where you actually wanna do a production build and test it and have all that type of stuff. So I expect to see more on that in the next year but just know that it's there, it works. Distributions. Distributions are very, very unhappy people for very, very good reasons. They are probably the most overworked people in free software. And I work a lot and they are more overworked than me. We think that Flatpak can help reduce the load of the distributions and allow them to compete with each other on the parts that they're actually good at rather than having to have the only way to be a successful distro, to have so many people working on it that you win based on the number of applications that you can package good or bad. And for those that don't know what Sisyphean is, he's like the guy that was climbing up the rock up the mountain and then rock would roll down and you'd have to keep doing it over and over again. So I like to think of that when I think about problems that just can't be fully solved. So one of the big complaints that I've heard so far about application developers taking apps back from the distros, right? That's kind of like the scary statement from the distros, is that distros provide these really good things like CVE tracking. And it's true, today they do provide CVE tracking and it's nice. But it's actually not terribly difficult. You get on the right security lists for the things you're bundling in your runtime.  Yes, Joe Ranham developers should not be making runtimes. Right, but that's why the sandboxing doesn't give you access to security related stuff. Like I get it, like sandboxing is not 100%, right? Like it'll never be perfect and there's exploits in the Linux kernel, there's lots of different layers. But SSL, for example, should be bundled by the runtimes. And the runtimes hopefully share a common core. So runtimes can, they don't inherit but they can build off of each other. We just don't track the dependency chain of them. So for us, we all, at least all the runtimes that I know of today, all depend on Yachto stable. So they're all part of the same day CVE releases, part of Yachto and can get updates, automated build, signed, pushed. Yeah, I mean, so I guess the question, the areas where you can have problems are like, okay, you're doing network communication in a library and it crashes, right? This is why the important thing for us to not give the application access to your home directory where the only things that it has gotten access to are things that we've implicitly granted it access to. And that doesn't mean there's not surface area but it does drastically reduce the surface area, right? Like if I use my web browser and then it steals my GNU cache file, like that's a pretty serious problem. But if I've never opened up, if I, let's say I'm using a flat packed version of Epiphany, like if I've never opened up the GNU cache file, it's not even available in the container space for Epiphany to read and steal through like, you know, remote code execution. So in that sense, you know, like all security, it's not absolute, but we do try to put in many, many layers of mitigation factors. And it's better than it is today because today that web browser like Epiphany or Firefox isn't even protected with any sandbox. And if we can get the distributions to be doing that for our core applications, we increase the status quo rather than continuing for it to be, you know, where it is. So yeah, the sandboxing we think is good. It can always be better and we'll continue to make it better. And I do think that the automated CVE tracking for developers could be useful. The difficulty is getting on the embargo list. So it might be the type of thing that we have to do through like an intermediary where, you know, we're watching JSON files and get repos or something and we're on the embargo list. I mean, certainly I would be at Red Hat, I'd be on the embargo list, right? And then we notify developers the same day and we could even provide them updates. But I mean, it is a difficult problem, but it's not one that's intractable and it's not even one that's takes an advanced developer to do. It just has to be done. Okay, so a little bit more on the security. Like I said, the sandboxing should help us protect, you know, more from third party applications, right? Like if I get Spotify, it's only binary, right? It's proprietary software. And, you know, most of us have come to the terms of accept that some people will want to run some level of proprietary software, whether that's Chrome, whether that's Spotify, you know, who knows. But we can also improve the security story of the standard distribution. You know, photos, calendar, you know, web browser, any of these apps can have their security story improved over what it is today, which has access to everything, ran in your same user session, can snoop on other applications, right? Like each one of these, we can improve the state without making things worse. Again, we support LSMs, SC Linux in particular, and D-Bus filtering so that your applications can't attack your system through service access. This is important because for all the years we've been using D-Bus, it's just been kind of implied that if you could communicate with the D-Bus service, that implied that you could do things with it. And that meant that pretty much anybody in your session could do things with D-Bus APIs. Now that we're gonna be running potentially untrusted applications, we need to prevent who you can talk to, who you get signals from, and all sorts of stuff. So the D-Bus filter runs inside the flat pack, but inside the process namespace, but after the point where P-Trace can be attached and with no privileges. So it's both inside this handbox, but with absolutely no privileges. So you can't say, like your application, despite that it's in the same Pid namespace, can't P-Trace the D-Bus filter, hijack the process and get it to do whatever you want, so you can talk to the host system. So we get both the sandboxing and the inability to attach and attack. We mentioned earlier, Xorg, not really suitable for security. You can kind of do the security, you can kind of do efficient graphics, but you can't really do both at the same time. Really not much more about it, it's just, that's the end of it. Okay, so ABI's inversion skew, bane of my existence as an upstream developer. So libraries break their ABI in very subtle ways. How many people here know what an ABI is? You're wrong. So I guess, I mean, most people say like ABI, that's like the functions that are available right, and perhaps the calling convention for it, and also the structures, and the layout of the fields and the structures. Well, how about if you embed a file in your library, and it's not really public API, but people know how to access it and have been, and then you decide to move that resource. Is that breaking ABI? So what if you support CSS to style your widgets, and it's never been documented as public API, but instead of supporting just CSS2, you decide to support CSS3, and do all these really fancy features, or 3D transforms, all these other cool things, but it slightly breaks some old CSS element names. Is that breaking ABI? Yeah, it always, it always, it depends. Okay, I have another fun one. So let's say you do something that performs an asynchronous operation, and then you do something else that performs an asynchronous operation, and which callbacks get called first? They're both asynchronous operations. They have nothing to do with each other, but you might be doing something in a certain order that expects them, when that changes in the future. Yeah, yeah, so these are all things that aren't necessarily ABI, but people get really ticked off in their program breaks, and refuse to accept it like, okay, I made a mistake. That's just step one. Like, you still have to fix the problem, but it doesn't help. So now that we know that ABI is very nebulous, and we've been trying for 20 years, we have SimVir, we have ABI versioning, like we understand what we thought, how to solve these problems, but it's just, it's not enough, because it doesn't accept the reality of how every single piece of thing we do in a library is, what's the term, compounding, right? And so getting everything right and adding features is just like, at odds. So it's really, really, really hard, and I don't think that we're ever gonna get it fully correctly. So in my opinion, I think we should probably just accept defeat on that solution, and try to come up with something that actually works. And yeah, it's even worse in these other languages. Like, I hear it's absolutely terrible in Node. I haven't done Node, but I hear it's just like a disaster. The one that bit me last week was Python, Debian, and some other distributions have updated their auto-completion Python library to a version that totally changed their API. And so I have all this auto-completion code for Builder that uses the previous version from six months ago. And if I ship it on Flatpak, everything works. Any sort of container-based system like this isn't a problem, because I wrote this to a particular version. There's no new features to the users by doing the updated one. It's just bug fixes, refactoring, stuff like that. But at the end of the day, like the user just gets auto-completion for Python. It's not like they changed the language in this, so. And you know, we might consider and say that, oh, they don't have the strict rigid adherence to soname and ABI versioning that we do and see, and rarely, but sometimes in C++. And sometimes you just can't even deal with it, right? Like a Python application might have one, and a different Python application might require a different version of that dependency. And Python, the language itself, doesn't provide a mechanism to have different versions of the same package unless you go through virtual M and have like entire different Python installs. And the distributions have never really bought into that. Like do you have a different virtual M for every Python application you install? It's just kind of outrageous. People complain that, oh, bundling is gonna add all this file system overhead, which I demonstrated earlier, does not. But that really would. So this one's kind of a, just a little fluff piece. Reducing host costs. You know, if we get the applications off the backs of the distributions and then having to host all of the updates and continue updates and the mirroring of the updates, right? The core OS is only a couple hundred megabytes usually, maybe a gigabyte with the core applications installed.  Anyway, we get smaller incremental updates. That's kind of nice. Some of the distributions do incremental updates, but maybe we can get updated mirrors faster and whatnot. So, okay, fun part. That's a really big terminal. Do something you shouldn't do, which is run out of tree against git. Where is my mouse? So running an IDE at 1024 by 768 is a pretty terrible idea.  But I'll just kind of, let's hear a demo quick to what we would have done. Where is my mouse? Well, since I can't get my neck around this thing to see this well, I'll just take the easy route and we'll just open GNOME to do. All we have to do to clone this is you just put git at, or git hub.com slash whatever. And then we'll get brought into the project loading and we're kind of displayed here. And wow, that is, I'm just gonna mirror my screen so I can actually see stuff. All right, so I opened up the project. This is GNOME to do. For those who don't know, this is a simple to do app or whatever. And all we have to do is click on build here. And what happened in the background is, it went through and it downloaded and built all of the dependencies. In this case, I built them right before the talk. So it kind of like skipped past it. But they're just, they're automatically built. They're right in a JSON file, which I can show you quick. The JSON file here is, well it's JSON plus we allow comments. It just describes some of the deboss things we wanna talk to. Build options, if you want optimizations, some cleanup phases, and the different dependencies we depend on and how to configure them. Not super important to the talk, so I'm not gonna go into depth on how to go about doing that. But when we click build, it goes through, it gets all the dependencies, make sure that works. We click run. It's actually deploying it inside the container namespace, setting everything up, and hooking it up into Wayland, all those different types of things we need so that we can run it. And it's running out of a completely different mountain namespace right now. So this is like as the user would run it, it's in kind of like that same sort of runtime. Come in here, play around with it. You know, nothing entirely surprising. But one of the more fun things that we have is that you can also run with the profiler. And I landed profiler support with SysProf to support container namespaces. So now we actually can come in here. We'll try to like exercise it so the kernel gets some samples in our program. And we have profiler, right? So this goes through all the different hooks of like looking into your programs and what libraries were there, trying to extract symbols if the debug symbols were available. You can even go seamlessly between the kernel and the user space into your runtime and all that type of stuff still works, right? So it's very easy to like go figure out where you had a performance problem, even if it's in the shell or whatever. Select in here, some range you care about. Anyway, it's a profiler, nothing too exciting. But now that we've simplified this whole process of getting applications, building it, I mean really, it's about as easy as it gets now for us to put newcomers through the joining the developer process. Like you really just paste to get URL, click build and you're off, you can run, you can test, you can profile, you can debug. Well, you'll be able to debug soon. Right now it's still printfs. And then I have one more demo for you. Now, Endless Computers is a software company that's building computers to try to make them very easy for the next billion users on the internet. And they do build on top of GNOME and they've done this really cool thing with Builder and Flatpak because all of their applications are Flatpak on the OS. And they care about helping people learn how to code, right, this idea of turning users into developers. Is that aspect skewerly? Okay, cool. So this is the weather application.  And in the meantime, it's cloning all the repositories from Git based on that Flatpak manifest which are included in the Flatpak files. Going through it, we can build it and run it. So it's like the same thing we saw except for you can seamlessly go through the application you're looking on and seeing like the code that's behind it and why that matters, right? So this idea of explorability, we're going to be able to take people going from a user into becoming a contributor to help ease our load. That one, so. Can you repeat the question? Yeah, so the question was, that one was the same being the SDK in the runtime. That one does not use the SDK. So the way that works is it's not actually the same application on both sides. That is a patch to GNOME shell that Endless has done that hopefully will have an upcoming release of GNOME. The intention I believe is for an upcoming release of GNOME. And it's just a GPU texture. So on the backside of it is the GNOME builder application that's been sent like a D-Bus signal or command line option to told to load this project. And then it's GPU textures rendered on the backside of the other application. And when you hit run, it'll be able to flip back and run the modified version of the program rather than the program before. So we kind of have like a really cool way of bringing people like an explorability to the system that is both users and developers. Yeah. Okay, so the question is, how much does this share with say LXC or Docker because they use the term container? And I apologize for using the word container. I generally try to avoid saying that word. But the reality is the infrastructure for which containers are built on does share a lot of the same Linux APIs. We use mount namespaces, network namespaces, PID namespaces, IPC namespaces. Yeah, yeah, it's all there. So C groups are more used from the session management. So we're not really doing a lot of C groups ourselves. We would rely on say like system D, login D and like the system D layers to do the session management for C groups, like how much memory or CPU we get, right? But we do rely on the namespaces. So if your application doesn't need network, we're not gonna give you the network namespace. And there are some problematic things with the plumbing today, like IPC namespaces are global in some sense. So you can't or abstract UNIX sockets are global. So like giving someone access to IP namespace might actually give them access to the XORG server, even though you didn't want to give them access to XORG. There's all sorts of like weird, funny edge cases with things, but you know, we're aware of most of them and they're documented in what needs to be done. And it's a matter of like getting things pushed upstream. So, yes. Yeah, so the question is what happens if you have an API or ABI break in a runtime? I would assume that this would be the same thing as it is today, right? Like your application's gonna break. You know, it's like your applications break if you break the runtime. The runtimes shouldn't be doing ABI breaks and generally only patch releases, right? So for the GNOME runtime, the GNOME, you wouldn't say like you have a GNOME runtime and it's like forevermore just a GNOME runtime. You have a version, just like a Git branch, right? So you have the GNOME 320 runtime comes out with 320, 322 came out with 322. Exactly, right? You know, for some period of time, yeah. But it's like, you know, the question is like, how far back do you go, right? Because it is an ever larger burden. So let's say I bought uplink in 2003, but like we had Flatpak and that type of stuff. Do we really want to be maintaining that for retro gaming? You know, like five years maybe. And I also would assume maybe like distros might choose a longer term, you know, like the equivalent of an LTS, right? The question is do applications specify a version or a minimum or max? And you specify a branch. So like Builder today targets the three, or if you're running nightly, it targets the master branch. Otherwise it's at release as we pin them. So there's a 320, 322, and it'll be 324 in a week or two. And then those only get dot updates, right? Like the patch level updates. Yes. Yeah, it's kind of like on a person by person basis, like the reactions. Not only do we have various distributions that are partaking in so many patches upstream for like especially things like bubble wrap, which is our sandboxing. And we extracted it out of Flatpak for people that like maybe didn't want to use Flatpak, but still want to go down the sandboxing route that we can get the maximum buy-in on that. So yes, we are working upstream. The other area which has been working out very well is the various free software consultancies have been doing a lot of work around Flatpak. So there's distributions, but then also people that aren't distributions, but like solution providers. So you're gonna see a proliferate from there. Yes. Sure. Oh yeah, I guess I didn't really dive into that too much. Graphic drivers are really a pain because one they're often broken in subtle ways. And so the developers are often releasing new versions of the graphics drivers and they'll release them for different distributions that they've done different builds. And until recently they were dynamically linking. And C++, which a lot of the graphics drivers use at some layer have happened to subtly break ABI, especially over some time period. Like we had the big C++ 11 move which did ABI breaking and whatnot. So it was very important that we get the driver vendors to start at minimum statically linking their graphics drivers. The important part of that is so that we can bring in a libgl into the application namespace, so that the application can use GL. And this is important not only for games but GTK4 is a fully Vulcan and GL based renderer. We have like Colleen and render trees and all sorts of really fancy stuff in GTK and we need access to like real graphics pipelines to do it. The other situation with those, having a brain moment. Anyway, we're gonna be doing some other things, neat things I guess with virtualization in there too. Like maybe we want to not give full access to a GL renderer and you wanna do a virtual renderer, it'd be possible to do that. Yeah, I don't know if there's anything else in particular. Yeah, so the question is, are the different driver vendors up and ready to use? NVIDIA, yeah, is there AMD ones? There was some stuff going on this week with it. It's their discussion on the mailing list. If it's not there, it's like imminent. We've been working upstream with Intel and the Mesa team. So that stuff's all there and remember just now, the other important component of this that makes it work is a thing called GL-VND. It's the vendor neutral dispatch for GL and this is problematic when you start having multi-head or just switching dynamically between drivers. So Red Hat did a lot of the work to get that in Mesa and I don't know if it finally landed but the idea was we were gonna ship it as part of Fedora 25. But that is another important piece into getting into the mountain namespace so that we can also support different GL drivers in an application, maybe when you switch monitors or something, right? Yeah, for me, I think it would be really cool if we had all the tooling in place to not need to rely on Travis. That's a missing piece, right? We need people to be very easy. You can do it, you can plumb it together yourself but you can't just go in to GitHub and say, oh, I wanna point this at a flat pack builder. It's not difficult to do, like I said, it's a command line, you pass the JSON file and it builds and you're done. It's not hard, it's just taking that and having someone work on that. That problem would be super key. Yeah. Sure, yeah, if anybody wants to, like hacking on like say GitHub hooks or whatever, I don't think it's a terribly lot amount of work, it's just, I'm already busy, I can't do it, I don't have time. Yeah. So the question is, is it using package kit, policy kit? What I'm not sure on is whether or not the code went in as a GNOME shell, or not GNOME shell, but as a GNOME software plugin or whether or not it went under package kit. I would be surprised if it didn't go under package kit but I don't know the implementation of it to be able to say. But the big thing is that this all uses app data and so it's like this very simple XML descriptions of applications and everything. So it's a fairly easy format for us to get access to the metadata without having to care about like was it RPM or DEB or like what the package type was underneath it. Oh, cool. And I guess you're using policy kit with like the JavaScript, like SpiderMonkey interpreted policy controls. Yeah, yeah. Cool. Yes. We do not do server stuff. Yeah. We focused very heavily, this is solving desktop. That's what we care about solving as desktop. That said, there are a couple things that I would like to see the server side things consider. And the design of portals for safe flow of elevation of privileges is hugely important and I think applicable to server type container technologies, right? Like I don't want to permanently have to give this process access to the kernel key ring to be able to do encrypted type stuff, right? There might be some workflow where you want to use a safe UI progression and curses, terminal, whatever to be able to put in your decryption key for your PEM files, right? And then it gets loaded in the kernel key ring and all that type of stuff, right? You have this like progression of flow. That is agnostic. The design of that is agnostic to flat pack. That is just good security policy. So we're not solving server stuff. I have no intention to, but we do hope that some of the design ethics cross pollinate. One more? Yeah.  Yeah, we want you forking software and playing with it and sharing it. I actually hope to land this before 324 is out. Translators don't kill me. I'm gonna add a new string, but being able to actually do the export and everything right from there. So you'll be able to like share it. That feature's also mainline. Like it's really, I just have to do a sub-process launcher and I just have to insert it in the build pipeline. It's like, I don't know, 20 lines of code. So when you build a flat pack, the manifest that you use to build it, there's like a truncated version or you can have the whole version that gets, it's like a file inside the namespace. So we're gonna just like look inside the mountain namespace and get access to the flat pack manifest and then use that to prime all the projects to download. So it's very simple. If you're doing a commercial app, you wouldn't put the sources of your repos in there, but for the free software apps, it's a great way to take people from users to becoming partial or learners to developers. So KDE is probably the next biggest. And then there's the one that, you know, Alex also maintains the free desktop.org one. And free desktop.org is more of like your core lower level libraries plus like G-streamer and like the plugin infrastructure and SSL and stuff. It's kind of like our Yachto plus free desktop components. So because we're not focused on servers, that's kind of like our top level, right? Free desktop down. I mean, theoretically, but like I said, like we're not focused on servers. Now, I hope that they start using bubble wrap. And I think that may happen with Project Atomic or something like bubble wrap is fairly agnostic. It's a way to like set up your mountain namespaces and all that type of stuff. But like don't expect us with flat pack to be going to solve server technologies. Any more questions? Cool. Okay, one more. Yeah, so the question is like, is Yachto like a base layer image? We're all, those of us are, many of us are paid to work on GNOME. Like I'm paid by Red Hat to work on GNOME. And, but I really do just work on GNOME. Like I'm a Red Hat employee, but I just work on GNOME. But because of that, we're sensitive to this idea of being neutral in the community and we don't want to choose distributions as like a favorite. So Yachto is a great core because it's fairly agnostic. It's, Intel does a lot with it, Linux Foundation does a lot with it, but it's not choosing favorites. And it still gets CVE updates and like compiler tool chain management. It's like it has all the important things of a distribution, but because it's part of this like trade organization where like the automotive Linux uses it and various container systems use it. It's like, it's the best neutral hub that we had to choose from at the time. And I still think it is. Cool, any more? All right, I just want to say thank you to Endless for their awesome demo here because I did not do that flipping, right? Like I just wrote the program, like they did all the really cool 3D textures and stuff. So thank you. With that, I think we're pretty much done. Do we have another talk after this? Yeah, okay, cool.  The build is triggered. You want CI. Yeah, builder does all of the difficulties of the building and the rendering. So it's really just listen to some web hooks. Yeah, I see. Exactly. Got it. Thank you. Yeah.