Hello everyone. My name is Michael of the NACA from Bootlin and today I'm going to talk about using Visual Studio code for embedded Linux development. So to introduce myself, I'm the founder and an embedded Linux engineer at Bootlin, which is an embedded Linux engineering company. We are specialized in low-level developments, kernel bootloaders, working on embedded Linux build systems. We do boot time reduction projects, implementing secure booting, graphics layers and many other aspects that are low-level enough, right? Contributing to the community is what we try to do as much as possible through code, of course, with experience sharing like in such conferences, but also by sharing all our training materials freely online through a free documentation license. I am also the current maintainer of the Elixir cross-referencer, which is a tool to index the source code of Linux, Uboot and Busybox through an online interface. So check it out if you're interested in such projects and in browsing source code. I'm always interested in discovering new tools and sharing my experience and the learning experience with the community. And what's special about this talk, I have to admit it's the first time I'm speaking about the Microsoft tool and actually so far I've only used Microsoft tools with the purpose of replacing them. So I initially proposed this talk after stumbling upon a survey from Stack Overflow from last year stating that visual studio code was ranked as the most popular development environment in the survey with more than 50% of the respondents claiming to use it, so that's an impressive number. And I was actually looking for other tools, other editors, but they eventually was amazed by this number. So I wanted to know more about that. So let's start with a disclaimer and the goals of this talk. First, at not a visual studio code guru at all, I'm just learning. But after hearing about VS Code also from many bootlin customers who started showing the tool to me, I wanted to do my own research on it and share that with you. So the main focus of this research was to find out to what extent VS Code can help with embedded Linux development tasks, and also how it compares to the Elixir cross the refer-referencer that I've been contributing to in terms of code browsing and indexing. So please share your experience while this pre-recorded talk is playing by posting questions and comments. And I'll be available during the talk on the chat and also after the talk, of course. I'm most interested in your feedback and your experience too, as this is relatively new for me. Thanks. Let's start by an introduction to Visual Studio Code. It's hard to beat these summaries from Wikipedia, so here it is. So Wikipedia says that Visual Studio Code is a free source code editor made by Microsoft for Windows, Linux, and macOS. Features include support for debugging, syntax highlighting, intelligent code completion, snippets, code refactoring, and embedded git. Users can change the theme, the keyboard shortcuts, preferences, and install extensions that add additional functionality. The first release was made in 2015 and over the years, over 16,000 extensions were developed by the community showing how extensible, easy to extend the solution is. It's important to keep in mind that this is not a free software or open source project. You can check on the interface itself. The license is stated as a Microsoft proprietary license. However, most of the project is still based on the VS Code open source project from Microsoft, which is under the MIT license. So check out this GitHub repository where most of the source code is. So what are the issues with Visual Studio Code? The first one is telemetry. Visual Studio Code collects usage data and sends it to Microsoft. There's a link to the Microsoft privacy statement in the tool, but it's all about Microsoft products and doesn't reveal anything specific about the data collection in VS Code. However, this can be disabled by tweaking the user settings. Just look for telemetry that's very easy to disable. And anyway, the source code for this data collection is available. Another issue is with the extensions marketplace. So the issue is that Microsoft prohibits non-visual studio applications from downloading binaries from their extensions marketplace, the Visual Studio Code marketplace. So you can't implement a third-party alternative that fetches from the Microsoft marketplace. You can see that in section 1.b of their marketplace terms. It's very explicit. I find this disappointing as most extensions are open source anyway and not developed by Microsoft. So as a consequence, the extension owners have to push their extensions to other extension registries too. And that's manual, that's not automatic. They have to do that explicitly. So as I'm always looking for 100% open source solutions, I looked for one. And I eventually found VS Code, which is 100% open source build of Visual Studio Code with telemetry disabled, essentially, and proprietary extensions disabled too. The strengths of this project is that it's making very frequent updates, staying in sync with the latest VS Code base very quickly. So as soon as there's a VS Code release and you release there, you get VS Chromium update very quickly too through the repositories. So that's really impressive. It has limitations though. Like it doesn't have all the VS Code extensions, as indeed it cannot use the same marketplace, as I will explain in the next slides. It also missed some extensions I found under VS Code that I found useful for embedded Linux development such as Devastory, Kconfig, embedded Linux, Linux kernel dev. And another issue was that in particular, C and C++ support is apparently built in according to the documentation, but doesn't work out of the box as it does with VS Code. Like when I imported a new project, a C project, normally in VS Code, it just detects that, proposes the C++ plugin, and there it goes. Everything is ready to be used, but it didn't work out of the box with VS Code. So I could do a few things to improve this, but that's probably due to my limited knowledge of VS Code. So that's a project definitely worth supporting, and I will continue my investigations with it. But since I had limited time, I decided to stick to VS Code for the moment and really understand what features it can bring, and then we'll see about the alternatives to it. Before we dive into the capabilities of VS Code itself, I have to mention other related solutions that I found during my research. One of them is the Thea IDE from the Eclipse Foundation, which is a completely different project, still supporting all VS Code extensions, but having its own architecture being more modular and allowing for more customizations, according to their developers, of course. It's also designed from the ground up to run on both cloud and desktop, and I'm not sure that's the case with VS Code. So it's developed and hosted by a vendor-neutral open-source foundation and not by a single company, as is the case with VS Code. It's now adopted in many IDEs across the industry, such as the new Arduino Pro IDE, ARM's embedded IDE, Eclipse J, which is a flavor of Eclipse, and the proprietary WebID from SAP. Another related project is Eclipse OpenVSX, which is a vendor-neutral open-source alternative to the Visual Studio marketplace that solutions such as VS Code and Thea can use. There is a very nice article actually, right here, introducing you to Thea OpenVSX and the limitations that are brought by the Visual Studio Code marketplace. Last but not least, here are some terms that you will find when you read documentation about VS Code or when you go through its menus to explore them. So the first one is IntelliSense, it's a generic code-completion tool that's built into Microsoft Visual Studio and it appears in support for all types of languages. There's nice documentation about it in here. Another one is IMET, but it's shown in the interface, but apparently it's for web developers, so that's not for us here. And the last one I wanted to mention is the Language Server Protocol, which is a new generic open protocol for communication between code editors and IDEs, so that when you use a new language to support, you don't have to create a new communication scheme between the IDE and the editor. Wikipedia has a worth-checking page about that. The first thing I want to show you is how to install VS Code on Ubuntu. Then I'll quickly guide you through the interface and its various components. I also show you how to disable telemetry and I will just show you the most important shortcuts in my opinion. So here's how you can install VS Code on Ubuntu 20.04. It could be different with other distributions. In the case of Ubuntu, it's available as a snap package. And then the package name is just Code. Yes, here it says that you don't have a classic confinement and not a security sandbox that would be better in terms of security, but let's go ahead and do it. So I just add minus, minus classic code and there you are. So here's what the interface looks like when you start it for the first time. Well, I already started it as you can see because there are files somewhere that are open. So well, you have a welcome page with summaries, information, documentation, shortcuts, et cetera. That's quite nice. So you have like the code explorer here for product code. Then you have a search and replace functionality, which is quite, quite good actually. So like replacing hello by hello in a set of projects. Just works fine. So it finds the projects that are applicable and does that. Then you have the source control window that for a repository can work with Git, typically. Then the debugger window, at least the debugger button and then the extensions that are available. So here you see all the most popular extensions apparently. Then I wanted to show, yes, here you have like the settings part. That's where you can set your settings. Here it's time to disable telemetry. So I'm going to show you how to do this. It's very easy. Just input it and disable it here. So there should be no more reporting to Microsoft in case you don't want that. And last but not least, I wanted to show the shortcuts. The most important one in my opinion is control shift P, like command palette. It shows you various tasks that are available, like to build a project. And that's the second shortcut I would recommend, which is control shift B, like build. And then of course there are other ways, other ones you can customize. You have plenty of choice here. This is very flexible. Now I will demonstrate the CPP tools extension for the C and C++ languages. Once again, if you look at the interface and the description of the extension, you'll see that the license is proprietary, but once again based on MIT license code. So I will open the Linux kernel source directory and then VS code will propose to install the plugin, which I will do. And then I'll show you the features of this plugin. First looking up identifiers, as I would do in Elixir, testing the expansion of the finds and also starting to type code and try auto-completion as proposed by the IntelliSense mechanism. So here I opened VS code again on the Linux kernel source directory. I started browsing a little bit. I got a few warnings I can dismiss. Normally when you open a C source file for the first time, it pops up and automatically suggests to install the C++ extension. But here I've done that before and it doesn't want to bug me with the same suggestions. So I would have to install it manually, but it comes next very close to the top. So here it is. I can install support for that. That was quick. I don't want that. And then you can go back to browsing the code again. There we are. So now it takes a bit of time for VS code to index the source code. You can see this. It's scanning. So it's ongoing. Like if you look at the definitions here, they're not available yet. So let's wait a little time before this is over. But it's making progress. Usually it happens even before you realize it. You start browsing and while you're doing that, the indexing is over. Okay, I'm back now. I wanted to show now that information about symbols is available like this. Just pass the mouse over the symbol and you get the information. What's nice too is that you have also defined expansion. It's really nice. Very, very convenient. Actually, to make this happen, I also had to accept the update or maybe the installation of extra settings. So I just accepted the suggestion that I was given. And now we have the code information right here. Of course, then you have lots of more detailed things like go to declaration, definition, references. All this go to switch header source like that. Right, okay.  And that's about it. Yes, now I can show you how we can edit the source code, of course, using auto-completion. So here, if I want to add a printk, it will show me the possibilities and the prototype. But I can also use the same command device create and see all the commands that match like this one. So you see the prototype here. I don't know exactly how to expand it, but there must be a way, right? That's about it for the moment. You see that indexing is still going on. But I did that again, actually, on the project. It's apparently, I guess it gives priority to the file that the files that are being explored. So this way, you always get similar information once you're browsing a given source file. So that's very convenient too, so that you have a fast response for all your requests. Next, I will show you the VIM extension. It's available under an MIT license, as most extensions do. And you will see that it's easy to enable or commands seem to work as usual. You feel like at home within. Note that other extensions also exist for other editors, such as Emacs, Atom and many others. Well, now I want to show you the VIM extension. There we are. VIM Emulation for Visual Studio Code. Let's install it. Hey, there we are. It's enabled globally, so everywhere it should be available. So let's go and try to edit our code now using features of edit of VIs. So I'm moving the code with the J, K, L, H keys. That's great, they work. Let's try to delete this word. CWU, yeah, it does work with BI commands. The last one thing I like is also rectangular selection. So let's see if this works. So it's Ctrl Shift to Ctrl V. Now, I probably made a mistake. Ctrl V. There you are. Yes, it does work. Almost, at least. Yeah, kind of it. It works. Currently, so now I can... It's not as rectangular as I used to have, but almost. Then I can delete that and undelete that, of course, with undo. So it seems to work mostly. Well, we don't want to explore the whole of Vim, but at least I'm back to my editor for editing the code. I'm feeling at home now. The next extension I want to show you is called CheckPatchLint. It's available again under the MIT license. What you need to do is first install the checkpatch.pl script on your system coming from the Linux kernel source directory. And it's very useful to create code that's compliant with the Linux coding style and rules right from the start. So directly in the editor, you get feedback from CheckPatch if something is not right in the code you are creating. Of course, you can also post-process your code with this and see the CheckPatch report. So let's install the CheckPatch extension. There you are. So to actually use it, you have to install a CheckPatch on your workstation or maybe provide a link to copy it from the kernel sources to somewhere on your workstation. I already did that. So now it's just a matter of exploring the source code with CheckPatch. So I'll do that. Actually, if I go back to the properties of the extension right here, you see properties. I'm supposed to enable it. It's not only installed, but I need to enable it explicitly. And now I can go back and explore the code. So now I open another file, a Ctrl and Shift P to run a command. CheckPatch selected file. There we are. So I'm going to checkPatch on this file. I don't see lots of warnings up here. Right here. Which one is that? Oh, okay. Two ones. We prefer PR Info to PrintK, Kern Info. That's to recommend it now as PrintK is re-deprecated. And there's also spacing. There's a space prohibited between function name and open parenthesis. So we should take that away, right? I could run CheckPatch again from the commands. Ctrl Shift P again. CheckPatch. And at least this should be gone. Apparently I need to save, I guess. Save. Ctrl Shift P. Check. Oops, sorry. One command. And now this should be the only one problem that's reported. So you see the CheckPatch errors that actually reported by being underlined in blue. A bit like syntax checking. No, spelling checking. But with a blue color. So here you can see where all the issues here, they're reported in the problems as well. And by the way, I failed to say that there are a few issues with the kernel sources. It cannot find some generative.h files. And that's expected, of course, because they are generated. So there are probably things to do to properly support the Linux kernel sources, like maybe either generate the headers ahead of time or we'll do something else. This is something I haven't figured out yet. Another resource I found is called Kconfig syntax highlighting with MIT code again. But it's just for syntax highlighting in the Linux kernel and bit root configuration parameter definitions. There's no support for symbol lookup. I'll compare that to the more advanced capabilities of Elixir in that field. So now I'm going to show you syntax highlighting for Kconfig files. So you see you have syntax highlighting by default with MIT files, at least for CNC++.  So let's go ahead and look for the extension that's missing. Kconfig, there it is. So let's install this one, enable it. Go back to the code.  However, it's just syntax highlighting. It doesn't help, for example, to do something about the settings and see where they're used, for example. And this is in contrast to what you have with Elixir. So I'm going to fire Elixir and show you what kind of information you get with Elixir compared to this. So there I am. I opened Elixir on driver's chart Kconfig. And this time you can see that for each, you have syntax highlighting as well that we contributed, if I remember correctly, to the printing print project. I forgot about the name exactly. So now here for each symbol here, you have a link to each parameter. And you see where, in which Kconfig is defined. So that's exactly where we were. But also where it's referenced, especially in the makefile. So here the reference, thanks to dependencies, I guess, what is the definition itself. But there's probably a dependency somewhere else. Depends on printers. You can see the dependencies on that parameter. And also where it's used in the makefiles, typically here. So you see where it's used. To what file it's being used for compiling. And the same applies to other settings in makefiles. So you can click on those and see how this one is defined. So that also helps to explore the makefiles themselves. So that's something we don't have in VS Code yet. Hopefully one day. Another extension that I found is called embedded Elix kernel dev extension. It's also licensed with the MIT license. It's trying to support symbol.completion, function and symbol navigation for C, Kconfig, devconfig.config and device tree files. So all types of kernel files. And it's trying to add automation to match device tree compatible and open their respective driver or documentation files. Like Elix here, it's based on universal syntax, but it's less advanced than Elix here, as we will see. So I'm going to install this new extension. So it's called embedded Elix kernel dev. There we are. It's installed. We enable it. And then I'm going to show you how it tries to match, to find the device tree compatible definitions. So let's look for a file. So the easiest way to look for a file in a project, you go to the command palette, control shift P. You actually do a backspace here. You want to look through commands, but rather through files. So backspace here. And then you get a chance to look for files by name. So here I'm going to look for SAME5D3 DTS sign. That's I want to go to. Now I'm going to look at, oops, control F. Sorry. So looking for SPI. I'm looking for, of course. Let's this one. So that's the compatible string. I'm going to try to use this to go to the definition of it. Oh, no. Device tree doc from compatible. And there we are. Let's wait. There we are. So what it found was atmelflexcom.h. And I believe it's actually wrong, because it just probably mentioned of this string inside this file SPI. Yes. That's an example here. But it's just a child node to the main child, to the main node that's been demonstrated. So here the binding is for flexcom, not for atmel SPI. So here we have a false positive. It does find the compatible string in the binding, but not at the right place. So that's the one, this one that should match. So actually it's somewhere else. And let's use Alex here to see that. Otherwise, let me just close this, get back to the original file, and then look at device driver match from compatible, and see if this works. At least you see the syntax highlighting for device tree, which is good. And here there's another one, XFDcore. So if I look for SPI, I'm not sure I got the right one. I should look for compatible. Nope, doesn't seem to match. I don't know why this one matched in particular. Okay, so that's not completely perfect. It's not really mature yet. So it needs perfecting. So now let's have a look at how this is handled by Alex here instead. So I've opened the DTSI for SAME5D3, right? I want to look for the SPI, SPI node. That's right. That's the one I was looking for. So here in Alex here as well, you have a happily corresponding to this compatible string. And let's see if it works. So the definition is actually the implementation like the driver implementing that compatible string. So there you are. That's the SPI admin driver as expected. So there's no, you don't have the false positive that you had before. Effectively, we still find the one that was found by VS, virtual studio code, but visual studio code and virtual studio. But that's also false positive here, I believe. So here it's actually should be, that's the right answer here. So that's the binding for the SPI admin driver, right? And then what you also have is all the device resources that our device resource includes that make a reference to this compatible string. So all the boards are essentially all the SOC or board devices that use this particular compatible string. So that's useful too. So Alex here has the false positive as well, and I'll report that, but at least it also has the correct information. Next, I'm going to quickly show you GitLens, which is a quite exhaustive extension for Git access and management. It's a monthly license as well. I'm just going to show you the Git blame feature, which I find useful when working with kernel code, but otherwise they are countless possibilities related to Git. So now it's time to show a few details about Git integration. The first thing I wanted to show you is here. It shows actually the changes that have been made.  What happened is that we fixed the check back issue if you remember. So now it could actually create a commit out of that change. It's also, I can actually discard a change if I want. Let's do it. You see, it's nice. So this is out of the box. You get support from VS Code, right? You could also update, get the latest changes from Mainline, from upstream. Let's do it. Yes, please. So it takes a bit of time. Shouldn't be so slow because there are probably not many changes to check. And that was done. All right. So I guess it pulled some other files. I could see. But I don't want to spend too much time on that. Git is a very huge topic and you can assume that it's pretty well covered. What I want to show is an extension that's well known. It's Git Lens. There you are. So it's super charged. Git capability is built into Visual Studio Code. So there we are. Let's install it. It's immensely popular, as you can see. In particular, it's going to show Git blame information for the code. So there we are. It should be ready. Get back to the code. I'll select a file and we'll be back with you. So I'm back. I open the blog, ioctl.c. I'm trying to use Git blame, for example. And just after enabling Git Lens. So here, when I click on the line, you can see here in gray, or you can see a description of the commit that corresponds to it, to this change. So you know who last changed that line. So you even recognize the photo of Arne Bergman, if you know him. That may be a gravitar or something like that. Then you can click on, you also have that information right here. So on every line that you click on, you see who last modified the file. So then you can go to here and see the changes associated to that commit. Oh no, sorry. My mistake. You have to go. You can go here and do show commits in search commits view. And that's the commit we want to check. And you can see the changes in the files, right? Corresponding the difference between the original file and the modified file. Nice, isn't it? Yeah, it's really nice to have such integration directly in VS Code. So Git blame, very easy to run this time. Next, I'm going to show you how to cross-compile the Linux kernel inside VS Code. So I'll show you how to set the environment for cross-compiling in Terminal. And using the Terminal to configure and build the kernel. Here, VS Code doesn't add much value. Here, CPP tools could help finding tool chains, setting up the environment maybe for the Linux kernel. But I guess a more dedicated extension would be more suitable than a generic one. But it doesn't seem to exist yet. So now I'm back in the Linux kernel source directory. And what I want to do now is just compile the kernel. So I haven't found anything special yet to compile the kernel. So you can do it though in VS Code just by opening a terminal. So there we are. And it's business as usual. So I need to check the target architecture, the target compiler. Which was already set. There you are. I can make the terminal a little bigger. Notice that the terminal is automatically opened in the directory corresponding to the project. So that's the right directory. I can run make menu config. Hopefully the terminal is wide enough. Yes, it is. So I can check various things. Actually, I don't have the right setting. So I'm going to run make some A5 DevConfig. Because I want to compile the kernel for the some A5D3 explain board. There we are. So now I can check the system type. And it's the right one. So I could make further changes of course. And then just make running multiple jobs in parallel. And then here it goes. Obviously it would take a bit of time. So here nothing really special about kernel combining. You just had the convenience of directly having a terminal in the right project directory. And there you are. You don't have any special highlighting for warnings, I guess, in terminal and output. That's something that you might get. But then of course you can't expect that from a generic C++ extension. It should be specific to the Linux kernel, right? So that's it. We don't wait for the end of the compile work. It's going to take maybe five minutes. So no need to wait. Now I'm going to show you how to cross compile a simple C program. Hello world, of course. So I'll show you how to specify the use of a cross compiler in the interface. Actually it's just replacing the name of the compiler executable. And then I will show you how to define the default build task. So that you can easily compile a program every time you change a line of code, right? Just by pressing a button. So now I'm going to show you how to cross compile a simple hello.world file with Visual Studio Code, right? So just what you have to do is just define the build task as done here. So I'm going to take the GCC build active file for the current file, right? That's very easy. And the only thing you have to do is just customize the compiler. And that should just be enough. I'm going to find the compiler file in the terminal. There it is. There you are. You can actually, this creates actually a tasks.json file. That's what defines the task, the build tasks, right? So tasks.json, you can save it. Then you move back to hello.c and just run build task, which is going to be run directly through control shift B. There you are. Success. So just click on that to close. And then I can see I've got an executable that's called hello. And that has the right type. It's an ARM executable. You see how easy it was. I'm sure you can do that again easily. Another tool I found is called CMakeTools from Microsoft. It was actually created by somebody else, but Microsoft took over. So this can generate a template project for you to get started. And a nice feature is the capability to detect cross-toolchains. Cross-toolchains, it calls such toolchains kits, but that's partially broken, as we will see. There's support for multiple compiling profiles, like building a production ready executable or an executable with debugging symbols, or to optimize the executable for size of a speed. Another limitation is that there's no example code or CMake files for languages other than C++. I didn't see any C support and URAS support. And that's missing. So I've just installed CMakeTools, the extension.  So it's a CMake project. So it's easy to do. You do a CMake quick start, which asks you for a project name. Hello, of course, it will be an executable. So it proposes to create a CMakeList file, right? And that should be all right. So it did create a main file with Hello World and a CMakeList file. All right. So the next thing I need to do is Ctrl Shift P, select scan for kits. I'm looking for cross compilers. So it did manage to find some Ctrl Shift P. Now select the kit. That's the one. So I'm going to use my standard C cross compiler. There we are. And now the next thing you need to do is CMake configure. So that's going to build the infrastructure to compile my executable on Linux. And now after doing that, I can do Ctrl Shift P build, CMake build. There we are. So let's look at the output. It did work. So if I go to the build directory, there's a file that's gone. Hello, as expected. Ouch. It's an x8664 executable. So it didn't really take the cross compiler into account. So there's something I need to fix. I know how to fix that I found out. So let me take the path to my cross compiler on Linux GCC. That's the one. I found out that the quick hack is to modify the CMake cache file and look for C++. That's the wrong path, you see. It didn't get it right. From this kit, the kit I selected. Unfortunately, so if I build again, Ctrl Shift P CMake build, it succeeded. Apparently it called my terminal, my compiler. And now it's an arm file. So I need to fix this to understand why it doesn't work as expected. But now I have a CMake file and it's ready to get started with this project. Then the last thing I want to show you is something closer to real life. That is remote debugging with GDB server. So here I'll show you how to implement a script to cross compile code, deploy the generated executable on the target through SSH, and still using SSH started remotely through GDB server. Then I'll show you how to customize VS code for the use of a remote debugger. And then we'll be able to use the remote debugger with all its features like inserting breakpoints through the interface, step-by-step execution, reading variables, analyzing a segmentation fault. And that's the first time I use a graphical interface for driving GDB and it was really a pleasant experience. A special thanks to the author of this article which really helped to get this to work. So now about remote debugging, let me show you what I've done. So first, I'm going to show you what I've defined some process, some script to actually build my executable. Cross compile my executable, so it's called Mr. C executable, it's called vstimulator.c. I compile it through prepdebug.sh. So this cross compiles the executable, kills GDB server on the target, copies the executable to the target through SSH. So here it assumes that you have to set up password less access to your board through SSH. This is relatively easy to do. Just look at the URL, the article I shared. And then still through SSH, you restart the newly compiled application through GDB server on the TCP IP port, right? And this means that the GDB server is waiting for a connection from the host.  So here I just had to create a new task. I mean, VS Code will help you to do that. You just essentially need to fill the path to your executable instead of just invoking the cross compiler just called your script instead. So that cross compiles and copies to the target. So you can look at this. It's json.tasks.json. So that's where I run, when I run build task, terminal, run build task. And then, so let's actually run it. Just run build task. And you see it, there's no GDB server for the moment. So it's not killed. It did do the job, right? In the terminal. So it should be okay now. So now I can go and actually debug my application remotely. And this is done by creating and customizing launch.json. Launch.json is always for debugging. There are a few things that I needed to specify the target architecture, the debugger, that's GDB. And especially I added two parameters that correspond to remote debugging. That's the debugger path. Here I'm invoking ARM Linux GDB on my host. So that's part of my tool chain. And that's for the GDB server address. So that's another parameter I had to add. That's the target address and the remote port. That's all you have to do if I recall correctly. You have to mention also the path to the local executable, which was built with debugging symbols, right? So now I can go ahead and actually launch the debugger by just pressing start debugging. All right. An exception has occurred, segmentation fault. So now I could debug that. So you see it worked. It showed me exactly where this happens. It actually happens because I'm allocating too many buffers. And after some point, of course, the allocation fails. And there's a segmentation fault because I didn't check the return value from KMLO. So here I could run that again. I can insert breakpoints, for example, by clicking on a line, abort the debugger and debug it again. Wait. OK. Now I hit the breakpoint. Good. And it's nice here because you can read the values from the variables in this context, like file is 0x0. And filename has this value. You can read the string value from here. So if I had other local variables, I could check their value. I could step in and run again. And then I hit the segmentation fault again. I would expect to have to step in multiple times and remove the breakpoints. So if I do that, it would stop and directly hit the segmentation fault again. Yeah. I apologize. It's a bit quick. But I'm running out of time now. But I could change some code and fix the issue, recompile. Like if I recompile the code here, I would build it again and be able to debug it again until I managed to get rid of the error, of course. So it just works out of the box if you know how to configure those things, how to configure remote debugging. And set it, I mean, of course, here in that case, I built a root file system with build route with GDB server on the file system and also with SSH, DropBear SSH, so that I could directly copy the executable over SSH from Visual Studio Code to the target, right, to the target by using SSH. And then restarting GDB server. Restarting the target executable through GDB server. Right. But this works. That's quite nice interactive. It's a nice experience for me to use a debugger in an interactive way, not only from the command line. It's now getting time to draw conclusions. So in terms of pure code browsing, I believe the Elixir cross-referencer wins. For C, C++ make files, device tree files, it provides links to included files, which the current extensions in VS Code can do. In the device tree, it supports finding driver bindings, which is just partially implemented in VS Code extensions. There's also full key config language support. It's also available as a way to identify users in this space. But however, VS Code would win for exposing git information, such as git blame information, and showing in-place information, like symbol prototypes, expansion of defines, which is much easier to get than through links that you have to click on in the browser. But of course, VS Code offers much more than code browsing. Now, I would like to share the limitations I found in Visual Studio Code during my research. The first one is that there's no way to filter extensions by license, whether they're open source or proprietary, and I have a clear preference for open source one, so I'd like to have that option. I have the exact same issue on Android, by the way. An issue with CPP tools, like that is C and C++ support, I observed clogging the CPU, or maybe the IO, looking for the declaration of some symbols. I had to close VS Code to end this. This is quite rare though, but I noticed that some other people report this kind of issue about when they rate the extension. Another issue, which is the outcome of this research, is that there is a current lack of extensions for embedded Linux, like applications and tools. So support for cross-compiling tool chains, support for Valgrind, except output syntax highlighting, that's just that. Support for S-Trace, L-Trace, Yocto project, build route, there's nothing for kernel modules. There's no serial port emulators that I could find. So currently, the VS Code and extensions seem better suited for microcontroller and RTOS work. There are plenty of plugins for that for embedded microcontrollers, for OpenOCD and things like that, but it's more focused on microcontrollers, apparently, than for embedded Linux itself, but of course that's a gap that can be filled by the community. And now come the final conclusions about Visual Studio Code and its suitability for embedded Linux work. All in all, I think Visual Studio still misses important features for embedded Linux development. However, it is very flexible. It's easy to extend and therefore has a great potential for further improvements. So it's already a great solution for debugging programs, even remotely. It just works out of the box and fine. Not out of the box, but just fine. And it makes me feel like contributing new extensions. For example, to support bootlin cross-compiling tool chains. This will also benefit to more open platforms, VS Code, Yocto, everything you contribute to VS Code can also support to the more open platforms. So think about it. Hey, that's already the end of this talk. It's time for questions, suggestions, any comments. I'll be most interested in your feedback about your usage of VS Code, what extensions you discovered, and I could talk about those ones in my next talk in the updates to this presentation. So I'll be available for further questions in a time that remains and also we'll be hanging around in the instant messaging channels that's provided by the conference. Thank you in advance and thanks again.


Welcome everyone. This is a Jenkins platform Sieg at gender. We are on the 31st of October 2022 and today we've got Damian DuPortal, Kevin Martens and myself. Welcome folks. We have quite a few subject in the agenda today. The first one being open actions items. What haven't we done yet? We used to have some PPC64 LE agents on CI Jenkins IO, but it looks like we lost them somehow. The low node program is finished now. So we have to drop all mention of PPC64 LE in the infraindex and if you make a quick search on Jenkins IO, I think we will still find some. So it's still open. We have to do that. It could take quite some time. It's okay for you. I'm opening an infrastructure issue on the desk about that task. We will take care of listing the elements so we can start working on that part. Yeah, that's a good thing. Thanks a lot. And let me know when it's done so that we can modify the agenda. Yeah, thanks a lot. Next subject. It's on you also Damian, so Git 238.1 update campaign. What's going on? So two CVs have been fixed by the version 2.38.1 of Git that covers two. So the version 2.38.0 was concerned by the security issue, but also the previous one that we're using today, 2.37. So we have to update to that latest version to avoid we don't risk any CV, even if the exploitation is not that easy. It's not high-criticity, but still we have. So that should be a campaign of updating the Git version on all our images. That's an action item. Yeah, so let me just change the indentation that it looks like it's part the action items. Cool. Done. So it should take quite some time because we have the Linux VM, the Linux container for CI Jenkins IO and Windows VMs too. Well, it has to be done, nonetheless. Any other comments on that subject? Nope. Okay, thank you. Anyone interesting on being, if you are faster than me, don't hesitate to send your pull request. We can spread the pull request. There are multiple repositories, so don't hesitate to get started. That will be a nice Actoberfest subject, of course. Yeah, that's what I wanted to add. It's still Actoberfest period, so you can get some swag just by changing to Git to 38.1. Thank you, Damian. Next subject is also you, Damian. We had some issue open on the infra saying that we maybe had a problem with the volumes in the Docker agent. So something is about to happen. Somebody should work in the following weeks about that. Because for the time being, the volume for home Jenkins has some trouble. We have some issues with that. Maybe we shouldn't detail them here or now. But yes, that's something we have to work on in the following weeks. So we'll have to change it to home Jenkins and to agent work there. I don't have the reference of the issue. Let me add it on the note. Documentation is important here. Just a note that's nice that we have Kevin there. Because I've seen that quite often the past weeks and months of user lost when configuring a Jenkins agent. They were lost on understanding what is the root directory when configuring an agent. The root directory is the directory in the remote system where the agents download the temporary files and then start a workspace and will create one workspace for each build that is going to handle. That one is not always by default, depending on the images or the plugins or the setups. That directory may change. But that directory usually is recommended to be under the home directory of the user executing the agent. Because you could have multiple agents with multiple working there. And you want it to be multi-toned. The error we did a few months or years ago was to define slash home Jenkins as a data volume, which creates a usability concern. Because if you want to extend our own image, you cannot copy custom tooling inside home Jenkins. That will be lost. Yeah, been there, done that. Absolutely. And I've been beaten and everyone has been beaten by that one. So we have a usability issue, but also documentation because we have numerous locations where we have numerous documentation for each agent. If I only focus on Docker in bound agent and Docker SSH agent, one say you have to configure an object on Jenkins, an agent, and then you have to connect it to the agent and you pass the working there as a flag. So it's one direction. With SSH, it's the opposite. You have to tell the controller that will decide where is the agent. So that's why having a documentation for each agent that mentioned, hey, the working directory by default is that path, a screenshot eventually. And if we have an agent configuration reference on the main doc, we can point to that reference each time. That will limit the mess of each plugin having a different default version between Docker cloud, EC2, Azure VMs, Azure containers, SSH, et cetera, et cetera. So that's why having a single change for at least the Docker images is a good start. Sorry. Sorry. It was just a good idea at the beginning regarding the performance. I think you explained me one day that it was because using it as a volume gives some better performance than just working inside the machine without any volume. Am I right? Okay. But still, it has to be a volume in the case of Docker images. We have users complaining that it creates empty volumes on Docker. That one is not a problem because that one is absolutely the normal behavior of all Docker images that you can discover on the Docker hub. It's a normal Docker process. So we can help them on learning and growing on Docker usage, even for development. But that's a normal Docker pattern. And the other way around, if we don't define the data volume for the agent work there, the performances will be so much worse, like then to 55 times slower. So we needed that volume. And that's all. Okay. And for the volume that gets forgotten, nothing that the Docker system prune or whatever could get rid of, we can remove them pretty easily. It's easy. Thank you, Damian. Oh, I still see some references to JDK8. I thought we were about to forget everything about JDK8. So what's going on? So, Team Yakom already and I think Alex and some contributors, usual maintainers, have removed JDK8 from our Docker agent images. The controller has been already done a month ago. However, with the recent issues with the Docker inbound agent and Docker agent tagging and release that were going back in the past. So we have issues and we are totally bad at that. So I'm working on fixing that. It takes time because it's not easy. But one of the last elements will be to have a JDK8 branch that will allow us to do one last release of these agents that should be frozen on time. And then we can close or forget about that branch. That branch will be used in case of security issue though and would help us to have most of the fixes that we already did on JDK11 that could be done on the JDK8. The volume agent is one of these fixes that have to be cherry picked. So once this element will be done, we can then continue forward with JDK11 and 17 on the main branches. So that's a proposal that comes from me. I haven't seen anyone since the past three weeks going against that on the repositories on the usual maintainer. So I'm going to work on that part. Cool. Thank you. Next is finish fixing main branch releases, avoid overriding existing versions. I think it's linked to what you just talked about. Exactly. Absolutely. That's fixing deployment. Yeah. And it's also linked to something that we'll talk about later or maybe we can just address it just now. The container repository management origin agent. Let's discuss that later. You already have the documentation. Thank you. And so the last subject of Docker agent is the proposal to merge the three repos into a single one, the atonal, you know, the mono repo versus multi repos. I can see some fights on the web just about every week on that subject. Yeah. Let's wait for the, you have more details on that topic. Next is container image deprecation for the Blue Ocean container. Kevin, you made an amazing job of putting warnings just about everywhere on the Blue Ocean documentation. Please stop using that. I know that it's deprecated or won't be, it's maintained, but you won't see any new features coming in the following months. It's not dead, but anyhow. And we still have to do, that's kind of the same thing for the containers of Blue Ocean. So maybe Mark will do that. I don't know. We'll see that later on. We have to update the page on Docker hub. We have to make a change log, a great guidance on, and it's not just because we should stop using Blue Ocean. The container images are pretty much updated. In fact, I even don't know somebody is taking care of them. So anyway, it's not a good idea to use them these days. Oh, let me know if I'm wrong, Damien. I think that's a good summary. Blue Ocean is still used, but not really well maintained, except for security. There are projects, but nothing to replace it as it right now. However, maintaining that image was created on a time where Jenkins configuration as code and the Jenkins plugins CLI weren't a thing. So it was easier for the community to push pre-built images with Blue Ocean already installed. As for today, if you start a brand new Jenkins, the setup, the default setup proposes you to install Blue Ocean by default. You can always install it manually on your existing Jenkins instance. Most of the time it's already there. Third, you can use Jenkins plugins CLI either on the default container installation or on your own system if you don't use container. It's already pre-default installed on Kubernetes installations. So yeah, we have a lot of tools and ways of easily going from zero to Blue Ocean in one command line. So that's why it doesn't make any more sense to maintain such image, because that image is always not up to date. Cool. Thanks for the details. So as it's written, not likely to make progress until November, I guess Mark wanted to address that when he comes back from PTO. We'll see that later on. Now, container repository management for Jenkins agents. We already addressed some parts of this subject, because yes, we may want to unify the existing repositories for the agents. And we also have to work on the release process, which is not that easy these days. Yep. So the main driver for these merging repositories is the following. We have inbound agent that depends on agents, which mean for each GDK version, then each operating system, which has multiple variations, we have to duplicate the same update of the operating system and GDK version. Inbound has a mapping 101, but already we have different naming and different operating system between both. Second, Docker agent alone is a nice route that we can extend for the inbound agents, but we never were able as a community to do the same with SSH agents that could iterate from this one without having too much duplication or unused item. So the goal will be to have that element. And now, since we have Docker multi-stage builds, we have Docker bake or even just a single Docker build for Windows machines, the goal will be to avoid the spread of this dependency, because the matrix with GDK's remote inversion, eventually SSH, all the installation and the tools that were already like Git and the rest, when we have to update one, it's a nightmare. So the goal will be to have only one list of dependencies centralized and then a system that build in parallel with the same layers. And so we can iterate what is inheritable or should be, and we can have specialization depending on the use case. That's a tool that has been paved since years by a lot of contributors. So thanks for this contributor. And now we have everything in place to merge everywhere. That will not change the name of the Docker images if you are using in bond agent today. That will keep the same. However, that will change how you see the release and release notes, because they will be all in the same repository. So we will change the tagging convention. For instance, one of the proposal, it's not reality, but one of the proposal will be to have, let's say a tag name inbound dash agent dash the tag that you have currently on the other repositories. So for the same version, we can have different tags that point to different artifacts. And multiple tags could point to the same commits. It's only for audits and contributors. But on Docker Hub, we will keep the same images. That's the whole idea. I see. Technically speaking, that sounds really interesting. And for the maintainers to come, that should be easier to maintain when the heavy lifting to convert everything to a single repo will be done. Nice. Thank you, Damian. What else? I think we're done with that subject. Require Java 11 on your for Jenkins core. I think so, the dropping Java 8 from the Docker agents has been merged. And you said me earlier, it had been released. So we're done with that. Pretty cool. And some plugins, I don't have the numbers yet, are beginning to require Java 11. So it's an ongoing process. I guess that's a good idea to move the plugins to the GDK 11. Do you know, Damian, why plugin maintainers shouldn't move to GDK 11 or 17? We don't force them to do so. Can we expect their plugins to work and be able to release new releases in the coming future, even if they're still using GDK 8? I don't want to force anyone, I just want to have the status. Should we or do we have to move to GDK 11 for the plugins? So today, if you are maintaining a plugin, you should test it with both Java 8 and Java 11. Java 8, because there are still some people using Java 8. So you need backward compatibilities and Java 11 because it's the standard default. If you don't use Java 11 today, that means your plugin cannot be installed on the new LTS. So that's quite problematic. Going to Java 17, it's not required today, but you should really start going on that direction. If you don't know how to build and test your plugin already today on CI Jenkins, I always the free GDK that we provide. Don't hesitate to look at the build plugin documentation or open the topic on IRC, Gitter or Community Jenkins. But today, you can absolutely set your plugin to build and test on the free GDK on Linux and Windows. So you can have six builds in parallel that try all the available. My bad. Sorry. Java 17 on Windows is not available yet, but it will soon. So my recommendation is if you are maintaining a plugin and when you are hearing this word, please enable for the free. If you already did the work for dropping Java 8, no problem. You can remove the hate from your CI process. However, communicate correctly in the readme. Don't hesitate to mention that Java 8 has been dropped for your plugin. Okay. And I've seen, I know if it's a urban legend or not, but I think that for Java 19, we could have some problems with Jenkins Core or even Jenkins plugins because some things, you know, we always have some warnings about illegals, status exceptions, something like that. And I heard that maybe with Java 19, we will have some problems. This will not be a warning anymore, but growing into an error. So we'll see. Definitely you should upgrade your plugins to Java. Sorry. I have no idea. Okay. I know. It's maybe a urban legend. I don't know. I should have checked. And now it's a good idea. What I remember from what you said that it's just a good idea to move to GDK 11 and CI. Jenkins IO, if you are a plugin maintainer, supplies you with builds for 8, 11 and 17, except on Windows where there is no 17 yet. Pretty cool. Thank you. And regarding GDK 17, by the way, you're using it, maybe not everywhere, but you're using it on CI Jenkins IO, am I right? Seven. Yes, but Linux only. Linux only. Yeah, of course. There is an open item and we are working on that one. So I don't have the numbers for October. Unfortunately, there were 11,000 GDK 17 install for Jenkins last month. I can only help this as grown. This month. I don't know where to find the numbers, by the way. I think it's Mark who can accept that. Anyhow, that's a good progression from August months and the forum. Anyway, next and last subject is a contributor summit remix we work. Kevin, you may be the one who knows everything about that. So I know what you have here on the page. That's pretty much it. Honestly, I think that that still has to be developed and figured out a little bit more. And at least obviously those scheduling still needs to happen for sure. So yeah, more to come. Thank you, Kevin. Folks, if you have any other questions, subject comments to do, it's the right time. If not, that's okay, too. We had a great meeting. Thanks a lot for coming. The recording should be available from 24 to 48 hours on YouTube. And yeah, see you in two weeks from now. Have a nice weekend. Bye bye.


From my experience over the last 20 years involved in organizing major sport events, it's really an important part to bring senior officials from across the world, from different events from different sports, from different policing units together in one room and to discuss the current issues and the challenges. And therefore, I thank Interpol and also the Supreme Committee for delivering a legacy to organizing a conference like this. This is very helpful. I just had also a meeting with the Secretary General of Interpol yesterday and we are trying to work a little bit closer together with FIFA and at the end of the day also with Qatar 2022. I know everybody here from the Supreme Committee are lived five years here and I'm very much looking forward to have a close cooperation to really make sure to deliver a successful World Cup 2022. The annual conference like this would be very helpful and I hope that there is also a kind of a legacy that when the contract ends after 2022 between projects like Interpol and Qatar 2022 that the cooperation will stop. I think there is a need also for the region for other countries. They are looking forward to host major sport events and I think there should be a legacy.


moving forward. He was also asked about Roldes Chapman said he could see the look in his eyes, the pain in his eyes about the way this ended. Let's hear from Chapman. It's frustrating. You know we have we have to hunger to keep going and to keep doing our thing and like you said you don't say have to season end like this you know it's of course it's frustrating. Obviously a battle would pour so there just take me through that at bat as you're trying to put him away. Yeah you know he put up a good at bat there. I thought I made some good pitches in that moment. I think he had bad activity to into a full count and I think it was a fast ball towards the middle middle end. He put good contact on it and there was the winning hit. Marley Rivera you have the next question. Roldes the fact that it happened what broke so after the scuffle that you guys had earlier this year was that tougher? Marley I'm so sorry about that. No not at all. I'm not thinking about that at all. You know like that happened about a month ago you know just this is the way it worked out. He put a good swing on that pitch and you know I gotta give him credit. Just a quick follow up Roldes so then the fact that it happened last year without Toubet and that was really tough how do you feel this year? I don't want to throw away any games. Unfortunately it's my turn now. These situations. Well I'm a striker and I'm practically the end of the game. And almost always when these things happen I think it's almost always going to happen to me. I'm the one who's coming to finish the game. The one who's coming to finish the game, to win or to lose. These things are practically going to happen to me in my career. But I'm a strong person and I'm in a bad moment but I gotta think ahead. Yeah I mean I feel terrible you know nobody in my team wants to lose the game. Nobody wants to lose. I particularly do not want to lose. You know as a closer of this team it's part of my responsibility to go into a game where it's either we're going to win or we're going to lose and it's going to happen you know it's going to happen both ways you know it's tough but I do have a tough mind and you got to put it aside and you got to keep moving moving forward.


Mae'r ddweud y gweld cael ei gallu gweld, sy'n ymddych chi'n ddweud, daeth eich gweld yn ymddych chi'n gweld o'r economiaeth, ac yn ymddych chi'n ddim yn ym hyn. Y ddechrau yn ymwneud, byddwn yn yn ymddych chi'n gweithredu os ymddych chi o'r methau o'i ysgol, mae'r ddechrau yn ymddych chi'n ymddych chi o'r gweithredu ..y'r amser o'r ddweud. Felly, rydyn ni'n ddweud ymddangos. Felly, y gallwch yn ddweud o litr. Ymddangos o'r amser, rydyn ni'n gweithio'n gwybod... ..y'r amser o'r amser o'r ddweud o'r amser o'r gweithio... ..y'r amser o'r gweithio'n gweithio. Yn gyfnodd, dyma'r amser o archiologe. Felly, tebyg i diddoriaethar honno'r amser o'r amser o'r amser o ddweud o amser o'r amser o'r amser ope carry brandon... ..eg yma, ym explodingig, y dyma llaw The UK is not smiling at the特cek. Why in the name of God didn't you predict the economic crash of 2008? What are they there for? I won't pretend to be able to take on the economists on anything like that kind of level. What I would like to do if something else is... and also to compare it with what I know, which is not much about... about the origins of the particular English approach to it, or British I suppose. Now, the mistakes, Measai's, Hayek, Schumpeter, Mae'r rhaglen o'r cyfnodol yn ymddangos i'r llwyddoedd. Mae'n rhan o'r ddod o'r ddod o'r ddod o'r cyfnodol, ond yn ymddangos, Paul Krugman, yn y New York Times, oedd rhan o'r ddod o'r ddod o'r ddod o Joseph Stiglitz, oedd yn ymddangos i'r llwyddoedd o'r cyflwynt cyfnodol, oedd y Llywodraeth ymddangos i'r ddod o'r ddod o'r ddod o'r ddod, oedd yma yn y 1970 oedd yn ymddangos i'r ddod. felly mae'r iawn ond gofodd. The Austrians came into their own again in the 1980s when I think it's fair to say that you could sense, I won't call it civil war, but an atmosphere of crisis in this country where there was a military coup and where 20 people were being killed every day. There was a crisis in England. And it was a very bad time Ac mae'n meddwl i'w gweithio'n gwneud hynny yn y 1980. Prydddon ni'n feddwl bod ni'n meddwl i'r ysgrifennu. Yn y... Roeddb yn llwyddo i'r... Roeddwn yn ysgrifennu Llywodraeth Precaen, ond mae'n ffgwrdd yn ynnig gweithio, ysgrifennu. Ac rwy'n meddwl, mae'n meddwl i'r ddweud hynny, mae'n meddwl i'r ddweud hynny yn ysgrifennu i'r ddweud hynny, mae'n meddwl i ddweud hynny o'r Ysgol, Fylltwch Cwiym. Mae'n ysgol yma 1860. Mae'n ddigonio'r ysgol yma yma yma yma yma, mae'n ystod o'r cyfle. Mae'r fformillau efo'r cyflwyno oherwydd y cyflwyno blynydol o'r gweithio yma. Mae'n gweithio. Mae'n parlymu, ymgyrchau, mae'r banyddol, ac yn ymwneud. A dyna'r ffordd y Llyfrgell Llyfrgell yn ymdweud yma, yn y ffordd yma yn y 1870s, ac mae'r rhaglen yn ymdweud yn ymdweud yn ymdweud. Mae'n ddweud o'r ffordd yma, sy'n ddweud yma yn y ffordd yma, yn ymdweud yn ymdweud, yn ymdweud yn ymdweud, ac mae'n ddweud yn ymdweud. Ond, ddych chi'n mynd yn ymdweud, yn ym 1860s, ac yn ymdweud yn ym 1900s, mae'r rhaglen yn ymdweud, yn ymdweud, mae'n ddweud, yn ymdweud. Mae'n ddweud ymdweud yma, sy'n ddweud, mae'n ddweud, mae'n ddweud yma. Mae'r ddweud wedi'u ffrindwfyr. Mae'r ddweud yma, mae'r ddweud. Mae'r ddweud yn ymdweud, mae'n ddweud. Mae'r ddweud yn unig, iawn yn ymdweud, a dywedohol sydd yn 48. Mae'r ddweud yma, mae'r tufeic yn ymdweud yn ymdweud. Mae'r tarn yw'r teulu, yna o'r hir. Mae'r tarn dda'u tarn. The little girl is fed two very large bottles of whiskeys, and the door is slammed. Half the tooth came off, so she was given another couple of whiskeys. And they dug the tooth out with the sort of thing that you take stones out of horses hooves. Now, a dental operation in 1900 is another case altogether because they have ether. And by the time my son was in the audience was going to the dentist, he regarded it just the same as having a haircut. Now that kind of progress multiplied everywhere, everywhere. You start off with horse and carts in 1860. Somebody's coming up with the principles of the jet engine in 1900. A Russian is coming up with the principles which will launch the Sputnik into space in 1903. That kind of thing. I don't need to ram this home. And it's a terrifically interesting time with bursts of progress everywhere. And that is the background to it. No, I don't want to sound unpatriotic. Really I don't. But you don't regard modern England. Scotland, where I'm from, is a little better. You don't regard modern England as a particularly efficient state. It has a ministry of death. It has a ministry of immobility. And it's, as Tony Daniels has said, we have achieved the unique feat of combining prosperity with discomfort. However, in the 1860s, this miracle period, England is at the top of everybody's mind. How do they do it? London, for instance, abolished cholera about 30 years before Hamburg did. It built the first elaborate underground railway, which was a very considerable engineering feat. When the Crystal Palace went up for that exhibition, it was more or less three weeks from idea to draft and then another couple of weeks before it got built. There's a wonderful book on this by Andrew Wilson. And England's the country people look at, and they say, how is it possible, constitution, and the key seems to be the adaptation of old institutions. For instance, the local government in England was run through the parish vestre, as it was called. Justice is not particularly... They didn't make a fuss about it too much. A local government in England, right up to 1914, consisted of a man with a booming voice and the sort of face that you want to measure the ipot in use of the square that I put in use for, and one secretary, and he runs the place, and they create places like Leeds and Glasgow. It's all done with, we call it privatisation, and it seemed to work remarkably well, and I think the same is true for the states, although I'm on territory, I don't really know there. England did seem to be the country which you had to imitate. My favourite Italian prime minister is a man called Agostino de Pretis, who dyed his beard white in order to look like Gladstone. So it was until, well, things changed a bit, and on the whole Germany became the model, but England seemed to operate on a principle of privatisation. Now, and if you look at English economics, I'm sorry, I keep using English as a shorthand, it's British, of course, and if you look at English economics, it's got a moral component to it. The Cambridge economics faculty was actually part of what was called the Faculty of Moral Sciences, and that's where Marshall came from. Now, the English state ran into troubles. It was a product of a unique period when England produced one-half of the world's trade, and of course, with France, most of the investment. It was a world which in nature couldn't last because the Germans, the Americans picked up and worked things better. And England, the state had to take things on, and the making of the English state was really Ireland. You know, the unique trick in English history is the abolition of the peasant. In 1900, 8% of the British population lived on the land. Compare that with France, 50, Italy, 60 and so on. And if you're not dealing with peasants, life is a lot easier if you're one way or another, not least with cheap food. But they ran into the problem of Ireland where the peasant problem was still there and the state had to do things. It had to move into more or less to force the big landowners to give up some of their land, that kind of thing. And by the time you reach 1911, the state is taking over things like pensions, unemployment insurance, all of which had been done quite successfully, privately before then. And the English state got a good reputation in, if you look at the official histories of the First World War, for instance, the way in which the English ran their war effort. It had far less inflation than anywhere else for a start. Then comes the second war. Then comes the big disaster. And this is something where I'm a bit critical of the Austrian school. The big disaster, the slump of the 1930s, when Keynes came up and said, in effect, I'm simplifying absurdly, that the state must come in, wave its wand, don't bother about inflationary finance, just get people working again. And in the Second World War, I can remember this as quite a small boy. We all thought that the British state had performed wonderfully. We were all together, we had won the war. Now that picture has been chipped away at, not least by Corelli Barnett in a very good book, pointing out just what went wrong with the state. And the fact is that a lot of the British war effort really came from America, even before America came into the war. And it worked. It worked quite well. And there's an illusion that the state can do things. So British economics abandoned the pre-Keynesian lines, if you want, Marshall, and took up all the economists in 1960, the Keynesians. And there is one point of this that I just do not understand. Round about 1960, the general idea was that the state would step in and abolish unemployment. There wasn't any unemployment. Everybody suddenly starts taking on debt. The Americans with Kennedy, England with the Macmillan government. Even Germany took on a tiny debt in 1964, probably as a manoeuvre to get rid of Adenauer. And quite why Keynes triumphed in the 1960s, a decade of stunning prosperity, is just something I find inexplicable. Now that ended badly in the 70s, inflation, stagnation, all that. So the Austrians, there's a kind of counter-attack with Margaret Thatcher and monetaries, this kind of thing. But it's curious that the English tradition in economics moved off in a Keynesian direction, whereas with the Austrians it's another matter. And I want to talk a bit now about what the background to this is. Central Europe is, I don't need to tell you, a powerhouse intellectually. I'm not so sure that you can claim it now, but in fact you couldn't. But both Britain and America were hit by great force by this wave of immigrants. Some, but by no means all. Of Jewish origin. And whatever you look into in Britain or America, you will find the presence of these Central Europeans. I mean, I've been around for a long time, so I knew some of them. And I think my prize goes to a man called Ernst Gombrich. I don't know if he's at all remembered. He wrote a stunningly good history of Western art. And he was a great globe-shaped figure with globe eyes. And I met him at something rather. I said, are you the son of the Gombrich who developed the iconographic tradition in art history? And he said, no, it was me. You could find people of this kind in any position of great. We've all profited enormously from them, of course. And that tradition of the education system, I've got my doubts about it. There were 40 economists associated with the Austrian school. There are two-thirds of the students in the Habsburg monarchy studied law. And if you study law in German, it is, oh, that is something. And so they have to go through that. Their range of references is formidable. They can look at things from a philosophical perspective. And they can also handle mathematics. So it wasn't all important to them. And that educational tradition with things that are right and things that are wrong is responsible for an awful lot of what happens. Now, the next thing is in the Habsburg monarchy is that the state did not work even in 1914. There's a famous book about it, what's his name, Musil, who said, this is a state which imitates itself. I'd start, does he sell by mit macht. And you were aware all the time of being in a kind of surrealist condition. In 1914, at a time when public employment was not common, certainly not in England. In Austria, 25% of people were employed by the state, which is an enormous figure for that time. There were more judges in the Supreme Court of the Kingdom of Bohemia in Prague than there were in the Supreme Court of the whole of the British Empire. And with this multiplication of public jobs, the whole business, everything becomes waterlogged. Taxes were three huge volumes, very thinly printed on both sides of paper. And you couldn't really find your way around it. I think in the states in Britain now it's not unlike that, is it? So in a sense, Austria presents the problem of the modern state in a way that we're now conscious of. And the Austrian economists, as they come up, really are looking for some way out of this terrible tangle that they are having to live with. Now, why did it develop like that? And here I might allude to something that Tony Daniels was talking about yesterday, the idea of rights. That's where monarchy was a complicated place. When they mobilized in 1914, the mobilization posters went up in 15 languages. And of these nationality disputes, the worst was the one between Czechs and Germans in Bohemia. The Slovene one in the borders of Italy also became poisonous, but not to that extent, and there were others. Now, I could wish that the European Union would look at the history of the Habsburg monarchy, because it is a terrible mistake in these circumstances to try to run anything by a central parliament. You can't do it. Up to about 1860, if the town council of a place like Budweis Budirovice met, there were Germans and there were Czechs. They knew each other. Some old Czech might speak broken German. He was allowed to address the thing in Czech and they would somehow informally work it out. And in 1860, a decisive moment came when the Habsburg monarchy ran out of money and it approached the bankers in Vienna for the Rothschilds, and they said it must be a precondition of help that you have a proper constitution. So they produced, first of all, the October diploma, which in effect decentralized everything. And the bankers said we're keeping our hands in our pockets. So a couple of months later, they produced a programme for a central parliament which is dominated by the German liberal element who thought they had the answer to everything, and they did. They knew about insurance, they could do tax law, they knew how to run a parliament, they would make terrifically good speeches, littered with literary illusions in Latin, and they thought very highly of themselves, which is one of their vast weaknesses. So Austria in the 1860s gets its central parliament, the Reichsrat. Now what happens? In the next 20 years there's a great boom of progress and the Czechs develop a middle class and they say we want education in Czech. We want a Czech university in Prague, they got one. There's a German university as well. And this is where the collective rights nonsense gets revealed. One third of the students in that German university were Czech because they wanted to get on in a language in which there are plenty of books to read and so on and so forth. And the result of all this centralisation on Vienna is that the nationality problems get transferred from places like Budweis to Vienna and the Czechs define themselves by nationality. Everybody else defines themselves by nationality. Everybody then starts hating everybody. And it's a test case of how you do not deal with problems of that sort. And I wish the European Union would learn it. I live in Hungary now and I just think what is the European Parliament doing voting sanctions against Viktor Orbán who's got the support of the vast majority of the Hungarian people. It's none of the European Parliament's business. And if at the end of it they create a great secessionist bloc in the European Parliament electing Viktor Orbán as president of Europe, they'll deserve it. Now if you're living in this kind of world, you're looking for some kind of formula which will get you away from the state which imitates itself with its absurd taxes and its unreformability and its terrible era of surrealism. And I think that's where the Austrian school comes in if I understand it. Mises biography is wonderful when you look at it. It's heroic. Now I will make one criticism of them that they're still right as if the slump hadn't happened. That is really high-ex weakness when you look into it is that he couldn't find a proper explanation for that. And they were not at home at all in a world in which there were world wars. The depression was caused by the problems of the first world war. That's obvious. And I think the Austrian school were rather at sea when it came to that sort of world. And I'll end up with something which I happen to know about because I lived in Turkey to some extent still do for decades. One of the finest moments in this country's history was when Attaturk changed the alphabet which meant we have a literate country. And the old university which was religious objected so he got it closed down and he opened up for business as it happens the first of the 30th of January 1933 when Adolf Hitler was appointed Chancellor of Germany and immediately a thousand of the best German academics came to, well not to, actually yes to Istanbul and Ankara. There were some wonderful characters among them. One of them was Ludwig von Mises' brother. Another was Wilhelm Lubke who can be called one of the makers of the German economic miracle after the war. It was in those days, I'll stop in a minute, it was in those days a precondition for teaching in a Turkish university that you should learn Turkish in three years and teaching it in five. Now it is a very difficult language and poor Ulrubka had a tenir. He just couldn't do it. So rather disconsolately went off to Geneva in 1939 and he bumped into Ludwig von Mises on the station platform. Now this, I don't want to be too critical but this is what's wrong with the Austrian school. Mises said to Ulrubka, if Copten and Bright had been able to sign the free trade treaty with Louis Napoleon's France in 1868, this war would not have happened. I praise the Austrian economists but there are limits. Thank you.