Hi, my name is José and I came here today to talk to you about a program I have been working on by myself in the last couple of years. And finally I made it basically to do something useful and I was so happy about it that they published it like two or three weeks ago. And well, this is it, it's called POC and it is a sort of editor for binary data to which you can describe the structure and then edit it in terms of the abstractions that you are defining. I know this is not maybe not a DC2 grasp at first but that's why okay I am going to do a little demo and everything. So, first of all, POC is not finished. I mean you can already use it to do useful things. Actually it helps me a lot in my daily work but this is work in progress. Work that you are by the way welcome to join. I will give you no pointers about if you are interested in contributing at the end of the talk. So, why writing something like this? Well, this is an excerpt of a real of one of the many, many, many, many, many scripts that they have to do my work. I work on the GNU toolchain. I am a compiler hacker mainly. So, I work on GCC, on binutils, linker, assembler and what not. And then I find myself very often in the need to vandalize L files, an object files and libraries and executables so I can reproduce bugs in for example the linker. So, I find myself very often in the need to edit binary files that have some structure. For example, L files, for me it's very common. And I use things like this. I used in the past things like this. So, for example, using object dump to get the contents of some information about the offset of the text section of an L file with object dump, to parse the output, to somehow operate it with a shell script and then finally to use DD, the DD command to patch the object file or to get information from it. Ok, this works. Yeah, sure it does.  Why? I mean, look at it.  It works but it is fragile and it breaks so often. Also, it is very specific, obviously. If I wanted to do something slightly different, I would need to write another script. Not good. So, at some point I was like, ok, you know, this is it. I'm not going to continue like this because my amount of scripts, it's increasing all the time, they are breaking all the time and I am investing so many times, so much time of my work, you know, instead of doing real work to maintain my infrastructure scripts. So then I decided back in 2017, during the summer, I was like, ok, enough. I'm going to write myself a binary editor, you know, that should be generic.  So, you know, I mean, it took actually a while because initially it was like, ok, very easy. I want to be able to describe the structure of binary data, for example of L files, right? They have a header, they have the locations, they have this field, these other fields, things like that. So then it was ok. Most of the data I want to describe, usually it is described already by some C library, C header, you know. So it should be, you know, the way I want to describe the structure of the data, it should look like C structs, it should not be that much different. But of course I'm sure that you all know that C actually is not very good when it comes to describe physical layouts of data because everything is undefined, right? And then the C compiler can introduce padding, can introduce alignment, can reorder bit fields, you know, which are less than one octet, things like that. So it should be, ok, it should be C structs plus something, right? Something extra. Then after one month or a couple of months thinking about it, I found some existing stuff. One is called Datascript by Godma Bach, who is a professor from some American university, for what, which one, who wrote back in 2010, 2011 a paper about something he called Datascript, which is very similar to what I wanted, actually. But it was not that much satisfactory for some reasons I will talk about after. And then there is an aberration, which is called 010 editor, which is proprietary, which means that it is unusable and it's very bad for the freedom of everyone and we don't talk any more about it. So then I just spent a long time, you know, like saying, ok, how can I have a description language that at the same time it is flexible enough and at the same time allows me to edit data in a transparent way and so on. You will see it working now. And then, well, finally, I got something that makes sense and something that makes sense and something that me in my general stupidity I am able to implement. So this is the program, this is how it looks like. Now I just told you what poke does in a very abstract way. Probably you are still like, ok, what? So it's demo time, alright? I'm going to use poke very fast because we don't have time. I'm going to use poke very fast to poke at relocation in an L file, which basically corresponds to some real stuff that I have to do, you know, like often. Ok, so I can use not installed poke, you know? So this is poke, oh sorry, first we need an L file. We create an L file and then this L file it has a relocation. I'm using redelf, you know, which is part of binodils. This is not poke yet. Ok, so let's poke it. So I just open it with poke and what can I do? Ok, first I can take a look. This is the dump command, you know, that basically tells me the bit and bytes on whatever that is in a cell file. Ok, nothing very exciting yet. But I'm always talking about structure binary data. So what is the structure of this binary data? This is an L file. How can you define in poke the structure of the data you want to edit? Well, using poke with big p, which is a programming language, which happens to be a full-fledged programming language, where you can describe data and operate with it. So of course I have already written a file for elf, which is called elfpk. The files containing poke code I call them pickles. And basically you can see here that in a language which is poke, you know, you can define structs, right? You can define structs, you can define types, you can define things like that. I will explain this later, but very fast for the demo. So, here there is a struct which is elf64eHDR. This is the structure, you know, of an elf header, right? And you see here that you don't always, you don't only specify the different fields, but you can actually also specify, you know, like constraints. Like for example, this is a constraint which is an arbitrary poke expression that tells that the elf magic number should be like that. So let's poke it. First, I have to load the elf pickle.  Now poke knows about those types. So, again, dump. Well, there should be an elf file at the beginning of the file. How do I get it? Well, I map it. This weird thing with the hash b is an offset, which is zero bytes. I will explain more about it later. And it gives me, it gives me the value. Of course, I can put it in a variable. So it's a EHTR, it's a struct variable that basically contains the elf header that is at the beginning of the file. Of course, once I map a value and put it in a variable, I can access, you know, the different fields. And I can also, you know, update them. Okay. What happens if I try to map an elf file and start in the first byte of the file instead of the zero byte of the file? Oops. I get a constraint violated exception. Why? Because the constraints which define, you know which are defined on this specific struct, which in this case is the elf header, they are not satisfied with the data, this offset in the file. So then I get a constraint violation error. Right, exception. Okay, but I have a header. So what is our goal? To vandalize that relocation. How can I get to a relocation in an elf file? Okay, I have the elf header, I don't know how you familiar with this format, but you have the elf header, and then in the elf header you have a field, which is called ESHOF for sectioned header offset, which contains the offset in the file to the beginning of the sectioned header table. Okay. The sectioned header table, as its name implies, is basically a sequence of different things of header entries, right, of sectioned headers. How many of them? Well, it's also in the header. It's called SHNAM. So how can I get it from the file? I map. At what offset? Oh, sorry. What are the entities that I want to map here? Sectioned headers. I have this another extract definition here. So this is basically map this number of sectioned headers, you know, at this specific offset from the file. Okay, what happened? EHDR. There you go. So this is an array. I can put it also in a variable. So this is an array. The size of the array is that in bytes. It contains, okay, let's put it in decimal. It contains 11 sections, right? So which section are we interested in the one containing the locations, right? How can we identify that section well by the section, flux or by the name, for example? How can we get the name of an elf section? For example, let's pick the first section, right? Or the seventh section, for example. It's its name. What is its name? Its name in another file is not a real string. It's the offset of the string in the elf file, a string table, you know, that gives you the string. Okay, you see that this is not very nice, you know, like the format, but this is the kind of stuff we have to work, you know, like usually with.  Oh, it is in the string table. Okay, in what string table? Well, usually it is pointed by the header, blah, blah. So that's why in poke, you can also define functions like this one here. This is a poke function that given an elf header and an offset, it looks in the, it looks for the string table and gives you, you know, the proper string. So for example, you can call it like this. And pop, this is the section we were looking for. This is not by chance, you know, I mean, I have done this before, before the talk, I mean. So this is the section we are interested in. So we know that we are interested in SHTR7. Okay, fine. SHTR7, it has a name, a type, a flags, what do we want? The section headers in the elf files, they have a pointer, which is another offset, yes, it's always like this, to the contents of the section in the file. Where it starts? SHTR, where it starts in SH offset? So what do we want to map at SH offsets? We want to map in this case relocations, because we know that this section contains relocations. So there is a strike definition for relocations too.  How many of them? Well, here you see one of the peculiarities that usually you find in object files and in object formats. Elf is not telling you how many elements in a section you have. It's telling you how much space occupies, you know, the elements in the section. Fortunately, POKE allows you to map arrays not only by number of elements, but also by size. And it does the right thing. So here you can pass this. SH size. So here we have an array of one relocation, because we only have one relocation. So we can do the like of my relocation. It is this array. And this is the relocation I want to vandalize. Let's do it. Let's put an add end of 666. Done. We get out, we do redelf and mission accomplished. All right? So this was the demo. Now, this is what you can do with POKE.   But you can do something slightly different non-completely different with a completely different object format by writing a pickle of 50 lines. And this saves a lot of time, at least for people like me. So, you saw here that I was using a pickle and loading it and, you know, and using sort of a language. The language is called POKE, right? And now I'm going to tell you very fast, very quickly the different characteristics of it, but only the interesting ones. What makes it different to other programming languages, right? First, the language has support for values, like any other, you know, language. You can specify integers in different numeration bases. You have strings which are null terminated. You have arrays. You don't have multidimensional arrays, but you can have arrays of arrays. And you have structs, right? Nothing special here. But then, let's see the first characteristic that makes POKE, you know, special. When I designed this program, one of the first problems I found was, should I make it byte-oriented or bit-oriented? Right? I mean option A. Ok, I'm going to make it byte-oriented. Why? Because 99% of the object formats around are byte-oriented, right? So when it comes to specify objects and think like, or sizes or whatever, should be bytes. Ok, fine. Cons of this approach. Well, that if you are one of the one percent, you know, who has, is unfortunate enough to have to implement deflate, for example, or any other bit-oriented format, then this program is not for you, I'm sorry. And you did not want that. Option B. Ok, wanted to make it general. But if I make it bit-oriented, it's going to be a real, real pain for 99 of the users because you can imagine. You know?  So, I was like, ok, byte-speed, byte-speed, ok, bytes but not bits. I was getting crazy. But unfortunately, in Frankfurt we do, from time to time, I do with some friends, we are called, we call ourselves the rabbit herd, we do like hacking weekends. And then one of those hacking weekends, I told my friend say, look, ok, I have this problem. So then we brainstorm and then we come with an idea, which is united values.  And also, you have only for offsets, what they call offsets in POC because usually you edit files, but sizes, memory, you have united name units, united types which are called values, which are called offset values. So you can specify something like this, like 8 bits, 23 bytes to kilobytes. That was the initial idea. Now this has many advantages and actually it's a pretty concept. But then I was like, but you know, to have a list of predefined units, you know, it's something limited. So why not allowing you to specify any arbitrary unit? So for example, this is an offset of 8 units of 8 bits each, 8 bytes basically. And this is 2 units of 3 bits each. Ok? But then I thought, well, why are they stopping here? Why can't you be able to specify offsets and sizes also in terms of your own types? So for example, this is a value, which is 23 packets, a packet bin, you know, that is the structure you are defining just before that. Of course, this only works for that structure in Poc, whose size is known at compile time, right? Because that's not always the case. But it's useful enough. So you can operate in terms of packets. Of course, these are the operations that makes a little algebra of offsets. If you add an offset to another offset, what do you get? Another offset. If you multiply an offset by an integral, but a magnitude you get, another offset. If you divide you get a magnitude, right? If it's like if you divide meters by meters, what do you get? A pure magnitude. And you also have the rest, the modulus, which is another offset, obviously. The offsets are beautiful also because it allows you, you know, to think in terms of units, like when you are doing like physics for example. So for example, how many, how many if you have, if you define this type packet which is a structure of an integer and along, like in the example in the slides, right? How many bytes are in one packet? Well, you divide. In one packet, how many bytes? Oops. Yeah, okay. 12 bytes. In poke, when you write a unit and in 23 packets, 276 bytes. If you write a united value, you can omit the magnitude if it is one. So if I write it like this, how does this looks like?  I think it's quite cute and nice. And also allows you to operate and do conversions without having to have size of and things like that, right? It's very nice. Also it has a very nice side effect which is that often in object formats and in object files you have fields which give you an offset in the file or the size of something else in some specific unit. Some formats, for example, this is in elf. This is the size of the section pointed by the section header. It happens that the size is in bytes. This is how you specify in poke a type or an offset type, right? So if you were using C, Python or whatever to edit your object file, you need to remember what is the unit every time before writing into it, right? And you have to convert to bytes every time. So if you are working in relocations, for example, and you want to write 10 relocations in this section from your C program or your Python program or whatever, then you have to remember the unit and you have to do the conversion. In poke you can just assign because poke knows the unit of the destination of the assignment and also of the source. So it will do it for you. So this was the first thing of the language which is different to probably what you have seen until now. Of course, this was the values. Now, poke has types. It has integral types, which is for sign integers and unsigned integers. Another thing that is different in poke than in most other programming languages. In other programming languages, normally you have integers which by default are, I don't know, maybe 32 bits, right? Or maybe 64 bits or whatever. Or in poke at an integer can be of any number of bits from 1 to 64 and actually plan to expand it to infinite number of bits. And I'm not talking about bitmasks or anything like that. I'm talking about proper values of 7 bits or 3 bits, 1 bit, 5 bits, whatever. And you can operate with them accordingly. So you can define those types using this syntax which I think is pretty readable. Also in the offset types you have offset types which is a proper value in poke. Offsets are a first class citizen in poke and then one string type because there is only one day for strings. And of course, you also have component types which is what you use to define the structure of the data you want to use. Arrays are picky in poke because to be honest, when I started it, I always thought, ok, arrays will be easy and it is structs that are going to be painful to implement and design. Structs were easy. Arrays were the complicated ones, surprisingly. Basically, in poke you have three types of arrays, the array types in the language. You have unbounded arrays which is an array of non-defined number of integers, for example, like in this example. Then you have arrays bounded by number of elements which can be constant, like two integers or can be variable. You know poke is a lexical-escoped plug-oriented language, so it has closures and you can do all sort of unspeakable things with it. Or it can be variable. Or as we saw in the demo, editing the L file, you can also bound an array type by size. So for example, this array can contain the same number of integers than that array, array of this type of an array of that type, which is two, right? But this one is bounded by size and that one is bounded by number of elements and we will see in the next slide or in one of the next slides that has an impact when you map it. And also of course, you know the queen of poke, destruct, all right? Which is what you use to actually define your data structures. I will go very fast with this and I'm very sorry but I have no time. First, okay, a packet, you know? Okay, this packet consists in a file or in memory of a byte which is a magic number of an unsigned integer of 32 bits which is the length of what follows and then an array of bytes of data length. You see you can use, you know, fields that has been read just before or before to define the data after that which is the data. This will be the typical definition of a variable length packet, for example. Also, although this is not implemented yet, you can pass arguments to the structs because a struct is also a closure in poke, you know? You can actually also define variables inside of it and functions. You can pass arguments that sometimes it's useful. Also, it is very typical in object files, in object formats that the structs, they have holes in it. You know, the structure of something it has holes in it, for example, a typical example, there are files who has two headers. One at the beginning of the file, one at the end. Or if you have a header and then you have an offset to whatever other data, you know? Think about, for example, a poke struct which is an extended to file system which is a header or a super block that points to a super block and then the super block points to the different super blocks. It is a sparse, there are holes in it, right? So, in poke, you can specify... I'm sorry, I don't have a pointer. In poke, you can specify at the offset of a field using what they call a label. A note, here you can put any expression that evaluates to an offset, right? And this is how I fix this problem of byte bit of byte. And this offset, for example, in this case is part of the struct itself. You see? It is very flexible. I know this syntax is very bad. But this is one of the syntax words that they will fix as soon as they can get rid of the bison parser I'm using at the moment and use a recursive descent written by hand. Because I want to use the normal C syntax of a label, you know, like a prefix and a colon. But for syntax issues, you know? And LALR grammars, it's not a choice right now. Also, you can have pinterest structs, which is exactly what we understand by unions in C. I call them pinned because it's like if the fields inside they are pinned to the same...  Why? Because in a pinned struct, usually in poke, this field starts immediately after the first one, right? But if the struct you define it to be pinned, the different fields start at the same offset in iospace. I call iospace the file that you are editing. So it's like a C struct, right? So this is basically also from ELF and it's telling you that you have an ST info, which is you can interpret it either as an integer of 32 bits or as a 28 bits ST bind or 4 bits ST type. All right? It's like a C union. And then you may ask, why did you not call it union? Because poke has union types too. What is the poke union? This is a concept I got from that script and I really love it. Basically, you have seen that a poke struct definition is basically the specification of a decoding process because if you look at it from an astronaut point of view, poke is nothing else than from the normal process of decoding, computing with data and encoding back, basically subtracting you from the encoding and the decoding and you can focus on the computing. So when you write in poke struct, you are basically in a sort of a declarative way you are teaching poke, you know how to decode the data, right? The unions give you conditionals. So in the struct type, in the struct type, you can use constraints, which is to every field you can specify an arbitrary poke expression that contains calls to functions and whatnot, anything you can imagine, also mappings, although that is a very obscure, I am not sure I want to get into very much yet, and to specify constraints associated with the fields. So how do you do conditionals in poke in your data structures with unions? So in unions, you have different fields too, like in this one. So how does it work? Poke will try to decode every alternative in the union starting from the first one, and then the first alternative for which no constraint is violated is chosen. And this is recursive, you know? I mean, the constraint should not be immediately in the field of the union. The union can have a struct, we can have a struct and so on. Any constraint that fails that invalidates this union. So for example here you have an example which is from the tag format in MP3 files, artist's name of the song and things like that. It uses this format. So here for example you have an ID of this frame whose is four charts, the first one cannot be zero, then you have a size and then what comes next, it depends. If the first byte here equals t, then what comes next is depends of the value of size. If it is bigger than one, it is two fields, you know? This idea is string zero and then an array of size minus one. Otherwise there is an array of characters of size size which is called frame data. If the ID zero is not t, then it comes an array of size charts frame data. Now you may wonder how is this different to this? Well, it is different because you have this, this happens only if ID zero is not t. And this happens if ID zero is t and size is not bigger than one. I know it takes a little bit to get used to those unions but when you do, you know, it's quite nice. Ok, POC supports polymorphic types, so you can write generic code. It is lexical-escoped, you have variables and whatnot. And then mapping. And this is worth it to waste time on this. So I was, ok, map this, I map that. Ok, in POC you have variables, like this bar A, like for example, this is an array of three elements, you can access them, right? But if I open a file, I can have also, I map three integers at this offset. So B is also an array of three integers and A is an array of three integers. So what is the difference? Well, the difference is that A is not mapped and B is mapped. So if I do A equal ten, ok, change the value of the second element of A, but if I say B1 equals ten, I change it in the variable as well, but I change it, you know, in the IUSPACE. So it has a side effect. So B has an offset, A have not an offset. So for example, you know, ok, this is it, right? So the central idea of POC is that you should be able to work with normal not mapped values and mapped values transparently. So you can write, and actually you can do it, you can write, I don't know, a function that sorts relocations and you can sort an array of relocations in memory, like in normal variable, but if that variable is mapped in some file or some memory, it will also sort it, you know, in the back end. This is that sounds very simple.  What is mapped? The value of the type. No, it is the type. No, it is the variable. No, it is the value. No, it is the variable. No, no, no. And I think I got it right. It is values which are mapped or not and only complex values. So this is the mapping. What they have been telling, right? You use the map operator, which is like that. Now, in Pog, you have functions, right? It is, again, lexically a scope, it is nice. It supports optional arguments. It supports variable length argument list 2 as an array of any. Also, I am a huge fan of Algo 68 and one of the things I like more about Algo 68 is that if a function doesn't get arguments you can actually use it in the same way that if it was sort of a variable, which I like much. And this was about the language. So now, unfortunately, I wanted to tell you how it works internally, but it's going to be super fast. So this is the architecture of the thing, of the application. Do you have a command part, which is the read line, you know, and all the things boring. Then you have a compiler, right? Which actually compiles pickle Pog, sorry, into a virtual machine, which is the Pog virtual machine. And it is the virtual machine through the instructions that access the IUSPACE, in this case, the file. So here, this is the structure of the compiler. At the right you have the disassembly of the Pog virtual machine instructions, which is a stack machine, because I love stack machines, of the expression that you see there. Please ignore the prologue and the epilogue. Those are the different passes and facets of the compiler. It does constant folding, some optimizations. I mean, it's not a toy. It's actually a very cute compiler. I invite you to see it in the source code. I made myself a macro assembler to not go crazy while writing the runtime and the execut generator. Actually, I wrote an AWK assembler so I can write the runtime of Pog and also you know the code generator routines like in proper assembly like this. Then the IOSAP system basically abstracts what you are editing from space of bytes into an space of IO objects, which can be those integers, strings or whatever. The important detail of this is that it doesn't have to be a file. It could be a process that you access the memory using Ptrace or whatever. It could be a file system because you maybe want to edit your extended tool or whatever, right? It doesn't matter. Anything that can be addressed by bytes can be poked. Then poked is extensible. Why? Because you can extend the application in poked. So for example, you saw, this is a syntax we can feel very proud of but I don't have time to explain it to you. Anyway, the dump command they have been using is basically written in one Pog function, right? You will see that the argument you know they all have a default value. So for example, you can say dump from the offset size besides, right? So you can pass arguments like that. And those arguments to define the new command, basically, you do it by writing a Pog function. What happened here? Yeah, there you go. A Pog function here, alright? So that's all you have to do to extend it. Now, pickles are Pog source files containing a collection of related goodies. Type definitions, functions, whatnot. Like elf, I have a test suite, it needs more tests and it sort of works. This is reassuring, right? I hope it is. So what works? Most of what I have told you today is working in Pog. Only one kind of IO devices which is files and things like that. But we need more commands and also there is a lot before a first release can be done. Supporting unions, por exemplo, which is work in progress. Support for sets, for enumeración, bitmaps, bitmaps, and things like that. More control sequences in the language. I have a for loop. But it is so cheap to add control sentences that, well, I don't know. And then after the first release, this is a list of big projects I want to do in general. But there is a lot before a first release in general. But there is still a lot of work to do. So those are the links for the project if you want to contribute. I have a home page, a many list, and also we have an ASC challenge in free note. And if you think that Pog could be useful for you and you want to have fun, please see the hacking file in the source directory, in the source tree because it contains a lot of information that could be useful for you. All right? So, done. In time, yes? Yes. I have a question. We can have six minutes for questions. How do you handle padding and bit endianness? Endianness, at the moment, you basically can set the endian to little, big or host, basically. I want to add a function that you can put in a struct constraint that will change the endianness run time when it is decoding. But it's not implemented yet. And regarding padding, there is no padding. I mean, in Pog, if you have an integer which is 7 bits, it's 7 bits. And in that sense, it is bit oriented. It doesn't pad, it doesn't align. What you describe is what you poke. Ah, there you go. Yeah, yeah. Okay. Did you take a look at KAITI structs already? So, do you know this project? KAITI is a K-A-I-T-A-I. Okay, no. Because they, it's an open source program as well. But it's meant completely the other way around. You define some binary structures and it's meant to compile into several languages reader, so parser of this. You can take their definitions, convert it to yours as well. We have a big repository of binary file definitions. Okay, no, but I will take a look because I want to translate all that to Pog. Yeah, very nice, yes. But it's oriented to write encoders and decoders, right? Decoders only. Encoders, okay. Is it possible to invoke any of these things as a library for say, if you needed to parse files and use the pick all definitions that are available with Pog? It will, yes. Actually, Pog, it started as an editor, you know, that's the main idea. But right now, I really think that it could be a very nice foundation for writing prototypes or binary utilities. Like one of the things that I'm going to write soon is a proper diff and patch for structure binary data, right? Based on Pog descriptions, yes. More questions? Just to make sure that I got it right. Would it be a tool where you can just define a binary schema once and then use it for writing and reading binary files? So all that you do with your files, you could use this loop Pog for... From Pog language in this case, yes. I mean, could be also, many people is asking me, hey, can I have a Pog to see? Can I have Pog to generate an encoder like this other tool is doing or another encoder? Yeah, why not, we can have a pickle written in Pog itself, do you know, that writes you out. Yeah, sure, yeah. That would be welcome actually, would be useful. We can still have one question. Okay, well thank you. Xau.


So welcome everyone to the session closure through the lens of music by Dave Yardwood, so thank you Dave We are so glad that you were able to join us over to you Dave Hello, everyone Good evening. It's actually morning here. It's about 9 o'clock in the morning. I'm 30 in the morning here in Durham, North Carolina on the east coast of the US and I've just had my breakfast and I'm ready to go so So I'm going to talk about the closure standard library And how we can understand What the functions in the library are doing by visualizing them through music But before we get into that a little bit about me I'm a software engineer at Keville But before that my educational background is in music composition and in the mid-2000s I Picked up a strange hobby, which is text-based music composition And I did not invent this concept There are there are actually a lot of languages out there for audio programming or music composition programming But at the time I couldn't find anything that really got at my goals Which were to generate or to create classical music By writing code and To have the full power of Western standard musical notation at my disposal So I ended up creating my own language called Alda And that's been my passion project for the last ten years So a little bit more about Alda As I've said Alda is a text-based music composition programming language The idea is text in music out and I've designed the syntax to be as easy as possible for people with little to no programming experience to learn Partially because at the time I myself had very little programming experience So the the language is a markup. It's what you see is what you get So this is very easy for new people to pick up even if they don't know about Kind of higher-order concepts like functions or classes or anything like that So Especially if you have a little bit of a music background learning this is very similar to it should map very cleanly to your understanding of concepts of music like notes and chords But as programmers we have additional superpowers and Markup languages are very easy to generate From from programming languages like a more touring complete programming language So I've been having a lot of fun exploring that So in this talk, I'll be showing you examples of ways that I've created music by generating This markup language from a programming language in this case closure So the library that I've created for this is called Alda CLJ and Under the hood what this is doing is it's putting together a string of valid Alda markup code Like we saw in the previous slide But now instead of working at the markup level, we're working a level higher where we're generating the markup So now this is great because we can create our own abstractions on top of the base Ideas like music like like notes and chords So we're going to see some examples of that I also want to talk about a couple of big ideas while I'm given this demonstration One of them is an idea that as functional programmers. I think we all understand inherently But you may not have heard it expressed in these terms So I'm going to borrow some terms from Eric Normand who is a well-known figure in the closure community And he just recently wrote this excellent book called Grocking Simplicity and in this book he talks about Ways that functional programmers approach writing new code and maintaining existing code By applying what he calls functional thinking So this is a lot like functional programming, but it's kind of coming from more of an angle of Practical skills that you would use in the workforce And less of an academic angle So even I think even if you have a lot of background in functional programming This is a great book to read because it might put to words some of the ideas that you've had in the back of your head for a long time So in his book Eric talks about one of the first things he talks about is this trichotomy of data calculations and actions and He describes this as skill number one that functional programmers must have in order to do what they need to do So functional programmer is always good at distinguishing between data calculations and actions Every definition in your code will either be data or a calculation or an action So data is just static data. It has no behavior It must be interpreted by some calculation in order for it to have any kind of meaning But it's really just data for example in JavaScript the data Examples of data would be a JSON object or an array And then functions every function is either a calculation or an action And a calculation is what we think of as a pure function. So this is a function that has no side effects It's not printing something. It's not making an HTTP request. It's not launching any missiles It is really just taking data as input and producing data as output and that is all that it's doing an important property of calculations is that You can call them any number of times with the same input and receive the same output So a random number generating function would not be a calculation That would be an action because it matters when you call it. You'll get a different result each time Another important property is that the sequencing of the calculations doesn't matter It doesn't matter what order you call them in because every time you call them you always get the same output from the same input By contrast an action is typically very It matters a lot when you call it and what order you call different actions in because they all tend to rely on global state and You know, let's for a metaphorical example here. Let's let's say we're writing some functions that represent making a peanut butter and jelly sandwich so You might have some functions that represent actions like slicing the bread and spreading the peanut butter But if you're not careful to call the actions in the correct order, you'll end up with a mess so Kind of the lesson to be learned here is that as functional programmers It's really best if we try to stick with data and calculations as much as possible But of course actions are inevitable. We always need to do some kind of action We need to make a database query or put an item in a queue You know actions are important as well, but we can help keep our code manageable by Deferring taking the action until we absolutely need to So why am I mentioning this? Well, I've been finding myself even before I read this book I think I was following this pattern as I've been writing code in Alda or Alda CLJ To create programmatic music. It's always a similar pattern to this where we start with some kind of data An example of some data that I've actually used in a musical composition is Weather forecast data that I pulled from the nether national weather service API So it might be a list of high and low temperatures. It's just data and then The calculation is some kind of algorithm that I come up with that Produces some kind of data that represents music in what I think is an interesting way Now this is just a calculation. We're not we're not actually playing the music data. There's no side effects here We're just taking this input data and producing an output data, which is it represents music We'll see some examples of this in later in the talk And then the action is just taking that data and performing it The other big idea that I want to talk about here is the power of visualization This is a screenshot of this great tutorial for the racket programming language Which I encourage you all to check out This is great because they're using a graphical environment called dr. Rackett it's a sort of like an IDE for racket and The way that they're this is a tutorial to the racket language And so the way that they're presenting the concepts is By using this library of graphics so that you can actually see You know a representation of what the what each function is doing So for example here, we can see that that series is a function that takes You know some kind of shape producing function and it produces three shapes of three different sizes Now we could also understand that by reading through the code and and trying to understand what hcapend is and what all this is doing But at a glance our brains are very good at understanding What this is and we're very good at abstracting especially in a visual way like this so the experiment here in this in this demo is To visualize things not visually but with with audio and with music so I'm going to jump into my editor here and Hopefully the audio is is going to come through here if it isn't please interrupt me But so we're going to use the Alda CLJ library here and I have both an Alda repel and a closure repel running So I can connect the closure repel to the Alda repel and Start a new score and Here we have just the markup syntax that I talked about at the very beginning of the talk So we can play this we can just hand it a string of Alda code and it will just hand that over to Alda So this is you this is useful, but as programmers we really want to work with data and calculations so Alda CLJ gives you this library of Functions that represent musical events and when you evaluate them, this is just a calculation Like we take a string and we end up with a record that represents An event that says to play play the following notes on a piano So this is just a closure map with a indication that this is supposed to be a piano and then we can see that these note Calls evaluate to records as well So these are closure maps that described a note So it has a pitch how high or low the note is and a duration how long the note is to be played And if we play this it produces the same result And it even returns the string of Alda code that it generated But I'm aware that not all of you are musicians So this I'm aware that some of this might confuse you with we're talking about half notes and eighth notes and dotted quarter notes and sharps and flats so actually to facilitate the programmatic Music that we're about to create. We're not going to work with Western music notation instead. We're going to work with just numbers So here's another example of something you can do with Alda CLJ Instead of specifying that a note is a C sharp or an E flat or something like that You can give it a MIDI note number. So the general MIDI spec has a range a very wide range of notes from zero through 128 and we can just give it a number and so what we hear here is The each note will be one semitone higher than the previous And we can also express durations not in musical terms like half notes But in milliseconds So here we have 100 milliseconds 200 milliseconds and we'll hear the that the duration of each note is double of That that was played before it. So this is all set up to kind of show What we can do with programmatic composition And what we just played we heard the notes we saw the notes Explicitly written out here. We have a note. That's 40 the 40 the note 42 For 100 milliseconds and the note 43 at 200 milliseconds But we have a clear pattern here We have a range of notes from 42 through 47 and we have a pattern where we start with 100 and then we double it every time So in closure we can use the iterate function to to multiply a Number by 2 and and keep producing that You know keep multiplying by 2 to get the next item in the sequence So that's an infinite sequence A lazy sequence and we can make it finite by just using by using the take function to just to just take six of those those numbers So we can see this result here So I've annotated this as one source of data and another source another data Source is the range of numbers from 42 through 48 and then we have another calculation here Which is that we're mapping this this this function So this is this is the calculation. We're taking our numbers for example the number 42 and the number 100 and The calculation is that we return a note that has that MIDI note and that number of milliseconds So we can kind of see this if we want to without playing it We can just do this calculation and and see the result So we have a list of six notes six actually six records that represent notes And they have the the note numbers and the durations that we've just talked about Then if we play it we can hear that this is the same as the example above All right, so now we've kind of established the basics of what I'm showing here So we can start to explore some of the functions in the standard library. Here's an easy one you know, there's map so map takes a function and List of values and it produces another list of values That is the result of applying the function to each item in the original list So we can use this to define a helper function called chromatic scale So what this does is it returns a list of notes Where they're just incrementing by one every time So if you were to play this on a piano, you would play every single key on the piano one at a time going up So we can hear what this sounds like on a harp So now we can play around with this a little bit more and bring in some more functions This is just a helper function that I've defined to grab the note number given one of these note records so this is the note record and We can use this function to pull out just the note number part of it So this is useful because we can use this in a predicate with filter So as I'm sure you're all aware filter is a function that takes a function which is a predicate and a list of values and it will return a list of values that match the predicate So we're going to keep only the notes in this chromatic scale That are even numbered So instead of every note in the chromatic scale will have just every other note And we can see if we count these in our original chromatic scale We had 78 notes and now we have half of that 39 notes So here again is so that we can hear the difference. Here's what we just played the full chromatic scale And then here is only the even numbered notes Now if you've watched TV or movies, I'm sure you've heard this sound before This is what we call a whole tone scale where you're moving up by two semi tones every time And this has a very it has kind of a dreamlike quality to it Which is why they often use this in like dream sequences and film and TV So just to play around with this a little bit more This this expression is equivalent to the previous one We've only changed the predicate function and now instead of saying if the note is even we're saying if the note number Modulo two equals zero in other words does the note evenly divide by two? It sounds the same But now we can adjust the number and hear what other intervals sound like So we can hear what it sounds like if you space apart the notes by three instead of two This is what we musicians call a diminished Quality and if we change it to four, this is an augmented quality So it's interesting to hear how these all have different kind of emotional characteristics. Let's hear a few more. Here's five and seven And then here's 12 which is octaves Let's learn about some more functions These are a couple of my favorite ones. Here's Rand Enth. It takes a list of things And it returns randomly selects one of the things in the list So if we call this a few times we can see that we get four and five and one five So it's randomly choosing each time And repeatedly is a great function that takes a number of times and a function And it calls that function that number of times So we can and we collect the results into a sequence or a list So we can collect three random numbers from a list using repeatedly and Rand Enth So let's Let's grab three random numbers from this chromatic range. That's the chromatic range on the harp So we'll get numbers like 33 82 73 So this is a I think this is an interesting way to come up with musical ideas I've done this myself in the past by Generating random numbers and just seeing how it sounds and then finding things that sound good And then I'll keep those numbers and I'll write out a more deliberate composition Using those notes Here's another great function. It's cycle. So this takes A list of things And it produces an infinite lazy sequence Where you repeat those items over and over again So for example, if you cycle one two three you get the infinite list one two three one two three one two three, etc So then we can take that infinite lazy sequence And make it finite by taking just the first 48 items The first 48 note numbers y48 well, it's it's a multiple of four four works well in music Um, I think it's all might also be a multiple of three, which may have been why I chose that because there are three notes Um, so this will sound even rhythmically And then we just take those numbers and map this function that turns each number into a note That one sounds pretty good. So the cool thing is if I evaluate this again All the clj will just depend the new notes onto the end and we'll hear them as soon as it's done playing what it's playing I just really enjoy this. I feel like I could do this all day And just listen for interesting musical ideas So this is great, right? We're generating music by using functional programming techniques Here's another example another function from the closure standard library. This is interpose And this one you can see pretty easily what it does by just if I just evaluate this Um, so we you know if we use interpose x with this list of letters, we get a x b x c x So it's inserting the item in between every item in the list So we can also easily hear what this does by interposing a high note between a bunch of low notes All right, here's another one. Um, and don't look too closely at this code It might not be the most optimized code for generating prime factors But I just wanted to play around with a prime factor function So if we if we map this prime factor function Over the range of numbers from zero through 10 We we will get the prime factors of every number so Zero and one don't have factors prime factors not sure how mathematically correct that is but that's what I came up with Um, and then prime numbers the only factor is the number itself. So two three five seven Those are prime numbers And then the other ones like for example eight is two times two times two So this is just to show that map cat. There's a function called map cat in the closure standard library And it's just like map Except uh, you use it when you have a function like this that produces a list of lists Or it produces a list so that when you map it over a list you get a list of lists Uh, but you might not want a list of lists. You might want a single, uh list So you can concatenate them all together by using map cat instead of map So this is this is also useful for the kind of exploration that we're doing here because we can use map cat to For each number in the list. So here's our data part. It's just the range of numbers between 40 and 60 We can map a function over this list that returns a list of notes for each of those numbers And then we'll we'll concatenate them together because we use map cat So this algorithm is a little bit trickier to explain it's a little bit more complex Um, but what we're doing is for each number in this list We are generating, uh One or more notes And the more notes there are the the faster they'll be so that we can all fit them into the same 500 millisecond span of time So 500 milliseconds for each of those numbers Uh to show the prime factors of each number And we're also, um adding each prime factor to the note itself So that might be confusing to hear verbally But we can um for one thing we can look at this Just look at the data that that this generates And um just for example the first four notes will represent the prime factors for the number 40 And so then those are two and two and two and five So we're adding each of those to the number 40 Because that's the number that we're getting the prime factors of And the duration of the notes will be 500 divided by four the number of notes And then the next number is 41, which is a prime number So that will take up the entire 500 milliseconds because that's the only factor And the note number is 41 plus 41 So there's a pattern here that will allow us to easily pick out the prime numbers because they tend to have high note numbers So, um What I'd like to demonstrate here is that as you listen to this Try to listen for the prime numbers. There'll be they'll they will be the notes that are higher in pitch Did you hear the prime numbers? So I think this is great because even if we you know if you were to look at this list as data You could find all the prime numbers by looking through and scanning for numbers that look higher than the rest But your ear is very quick at picking up the high notes the high frequencies as compared to the low frequencies All right now. I want to give a shout out to aditia. He uh, he asked me to think about generating music from fizz buzz And I got really inspired by that. So I came up with this So, um, we're going to represent fizz buzz just as a quick introduction to fizz buzz this is a elementary programming problem often used to test your skill and programming In any language And the idea is if the number is divisible by three, it's fizz. So three is fizz If it's divisible by five, it's buzz If it's divisible by neither you just get the number itself So one and two are not divisible by three or five and of course when you get to 15 That's divisible by both three and five. So you get fizz buzz So we can actually hear the fizz buzz cycle Uh here. So what I've done is I've defined some functions here without looking too deeply at the code Um from a high level what this is doing is it's taking Uh the divisor so three or five for fizz or buzz And a note number And it's producing a sequence where every n notes. So maybe every three notes we play a note So we'll the oboe will represent the fizz And the buzz will be the clarinet. So every five notes will play this other note And I've also added some percussion to make this So we can hear more of the kind of the rhythm of fizz buzz I'm going to comment these out for now. So we can hear just the pattern So what this is this is periodic notes inverse Now this is like given these factors Play a note whenever it's not divisible by any of those numbers. So this represents like the The the numbers that are neither divisible by three or five So if we hear this we can hear A hi-hat playing the notes that are not divisible by the numbers that are not divisible by three or five And the oboe is the fizz and the clarinet is the buzz Now the interesting the interesting thing about this is that this is actually it kind of has a natural rhythm to it And that's because it's cyclical Right mathematically Once you get to 15 and you go to 16 you've you've actually started over at one essentially with as far as Whether the notes are divisible by three or five And so we actually get this repeating pattern and we can hear this better. We can hear how it repeats if I bring in These other voices in the in the drums. So we have a kick drum and a triangle So these will just kind of show you What the what the pulse is so that you can better hear the rhythms of What the other instruments are doing. So there you go fizz buzz as music Now the last thing I want to show you before I go is an experiment that I did recently Which is applying similar concepts Um I wanted to take words from the dictionary or just any word any arbitrary word that you provide and And interpret it as music And so I was grabbing random words from this dictionary file on my on my hard drive For some reason a lot of these words have apostrophe s on the end I don't quite understand why there's so many of those but I filtered those out just to keep things a little bit more Saying more interesting maybe Um, so, uh You know kind of glossing over what all these functions are doing exactly I came up with um a way of interpreting Oh a string as a list of note numbers So we can see this if I map this function over the word hello We get 76 79 72 72 69 So these are all numbers that represent each character in the string so We can take you know the string functional conf for example And hear what that sounds like if we interpret those letters as notes And the space by the way represents a pause So you'll hear that pause in between And we're going to play the functional conf on a random instrument. You know, that's not a very good one You can't really hear the pitch And I'm going to stop that Let's try that again. This is like live coding in action Oh boy Hold on Sorry I'm going to restart this Okay, uh, let's grab another random instrument that hopefully has pitched this time I think that's kind of a nice rhythm, don't you? So let's uh, let's try this again But this time the words are going to be randomly chosen from that dictionary file that I showed earlier So for example, we might have uh, uh, prin, principe Presumes plaque. Oh, that has a bunch of p words Or we could get rover exterior's biggles I guess we have some proper names in there as well Um, and so each each of these words Is going to be played by a different instrument all at the same time And why this is interesting is that you can hear Uh, kind of the the rhythmic way that these words play off of each other The words are left Transmissible and moisturized The longer words kind of linger at the end because they it takes longer for them to finish All right, and uh, let's actually let's look let's listen to another one. I think we have time This is kind of a crazier one This is we have despoiled daggers and axis And lastly, uh, this is a an absurd poem by e ecomings. Um called me up at does Here this is the poem me up at does out of the floor quietly stare a poisoned mouse Still who alive is asking what have I done that you wouldn't have And so let's hear, uh, how this poem can be interpreted by this algorithm. All right, that's all I've got Thank you very much for listening to my talk And enjoy the rest of the conference Thank you, Dave. Uh, such an amazing and interesting session today. Uh, we have some time so if anybody has any questions, uh, Dave is available for the next Um, seven to eight minutes to answer There are some amazing comments on the chat as well, uh, Dave for For the session so oh, yeah, I'm looking at those now People are actually saying like, uh, I think nadej is saying once more once more Isn't for an encore Yes, absolutely Am I taking dj requests? If yes aditya wants to hear appease demo gods. Yeah, okay. I could do that Let me share my screen again All right, so we'll change this to appease Oh, actually we could do three different words. We could do, um appease Demo gods Oh, I like that one Any other questions? The whole poem up thread. Oh, right, right. Okay Yeah, let's do it This is going to be very interesting because I don't know how well my algorithm handle its, uh, characters like the lambda character there But let's give it a try It'll either want it'll either work or it won't so let's see what happens Let's maybe define this as a var appease demo gods So I think this will be very long. So I might need to cut it short, but we'll just hear what it sounds like It's pretty mysterious In the interest of time, I think I'll cut that off, but yeah It's always interesting to hear what the different, um the like how uh What kind of musical ideas emerge from listening to just random, uh, you know programmatic Output of uh transforming a string into, uh, you know music All right, so rakesh asks are you using a lib to generate the midi instructions or have you written the midi bindings yourself? Ah, yes, so the uh, the midi is actually happening in a background process uh called alda player And it's um these are spawned by alda. It's kind of all under the hood Um, but the alda player process is a jvm process Uh, I actually wrote it in kotlin and Uh, it's using the the java the java actually has a very good midi library built into it There's the java x dot sound midi package And so i'm using a midi synthesizer and midi uh sequencer So, uh, thank you dav and dav do you also want to share uh where people can reach out to you? That would be very helpful for you audience I'm glad you mentioned that I actually had another slide that I totally forgot to show at the end Uh, so yeah, let's chat. Um, uh, I actually uh There's there's a slack group for alda Which is um, not very active at the moment. I'd love to see more activity there. So if anybody is interested in alda Um, please feel free to join for free and you can share your scores that you come up with Uh, share your music with with the community Um, you can you can at mention me there for if you have any questions about alda or anything Um, you can also feel free to reach out to me personally email and twitter are both good ways to do that Wonderful. So thank you so much. Dave. It was I think wonderful Start for the weekend and great, uh presentation. Thank you


Thank you, Ingo. Yeah, GammaRay. Who of you has already heard about this? Who has already used it? I hope that will be a few more after this presentation. So what is GammaRay? GammaRay is a runtime introspection tool for Qt. If you might have encountered the tooling you find in current web browsers like the Web Inspector or Firebug, imagine something like that just for Qt applications. So it's something that at runtime allows you to look at what's going on inside your application, but at a much higher level than a conventional instruction level debugger. So you don't want to debug layouting issues by stepping through the Q layout implementation. You want to debug that by seeing the actual layout and fine-tuning the various layouting parameters until it actually matches what you have in mind. And that's the kind of stuff that we can do with GammaRay. But instead of doing all theory here, what I actually want to do in this session is primarily show you how to use GammaRay and show you a few real-world problems that we investigated using GammaRay. So to get started, let's look at how to actually open your application with GammaRay and look at what's going inside here. Is this half-ray readable or should I increase the font? It's okay? Good. So, larger? Okay. Better? Okay. So there's two ways you can connect to an application similar to how you would use a normal debugger, right? So you can either launch your application just by prefixing it with GammaRay. And that launches the application. This is just some dummy widget layouting example here. And then you get the GammaRay main window started as a second window. Is that half-ray readable? I'm abusing the high-DPI support in Qt here to scale it up. So there will be a few rendering glitches. I have no idea if the high-DPI team is here. So in the end, what we get with GammaRay here is the... This is actually not visible at all. So we have the tool selector on the left starting with the object tree. That's the most basic tool. That's what you have in the middle then. That's basically the entire hierarchy of all Q objects in your application. And then you have the property editor where you see the specific properties of that object. And that's kind of the foundation for everything else inside GammaRay. And that's 20 or so specialized tools that focus on one specific aspect of Qt. For example, widgets. There you get some layouting overlays and it limits your view to that specific kind of object. But we'll see that in much more detail later. So that's basic UI and how you start your application at the beginning. Alternatively, you can also attach at runtime. So start your application. Either you pass the process ID on a command line like you would do it for a debugger. Or you just use the GammaRay launcher UI that you get by just launching GammaRay without any additional arguments. You select the process you want to attach to. It does some magic in the background. It takes a while because like attaching a debugger, you need to load all the debug information. And then you get the same main window. As you notice, there's fewer objects in there. If you attach at runtime, GammaRay is unfortunately not able to find all the objects that have been created since application startup. But only those that can be found by watching out for events or by walking down the object hierarchy. So usually prefer the launching with GammaRay attaching is just if you see your application misbehave, it's very hard to start. Then attaching at runtime comes in handy. Right. So back to this. Yeah, there's a few things to keep in mind. The way this works is there's a small part that's called probe. That's basically a DLL that's loaded into your application. And that needs to match the Qt version you are using in your target application exactly. That means the Qt version, that means the compiler, and that means the compiler settings on Windows. So debug release needs to match. If that doesn't match, GammaRay will notify you and will not even allow you to launch the application. That's probably one of the most common problems you might run into. And as a side effect of that, you most of the time will need to compile GammaRay yourself for your specific Qt configuration to make this work. That's a bit inconvenient, but the implementation of this heavily relies on all kinds of non-public internal Qt API. So, and that keeps changing. So there is unfortunately no way around this. Right. But then the interesting question is what can you actually do with this? So what would I use it for? And for that I have a few examples that are all based on real-world problems. So stuff we actually encountered in customer projects or we run into in our own code. And where we found GammaRay particularly useful for debugging this. All the examples are massively simplified, which might make some of the cases a bit too obvious, but if you imagine that in millions of lines of code, you might not even know having the tool that points you right to the spot where you have the problem becomes really valuable. So let's start with the first one. Slots. So this is a tiny widget-based application that has two buttons and an LCD widget. One of the few remaining uses of the LCD widget. It has two basic functions. I can click on the emit button and the LCD widget increments. And I have another button that reconnects the signal from the emit button to the widget for incrementing the number. So you see that still works. Reconnect again. And suddenly our increment gets bigger. So let's try to fix that with another reconnect. And our increments keep getting bigger. So something is wrong there. Probably somehow related to signals and slots. If you try to diagnose that with a conventional debugger, you put a break point at the code. I can actually show you the code. Not in this one. So maybe you can spot the problem even without the debugger. Readable. So it's layout, two buttons. We have the reconnect signal slot. So that's the code that's executed when you click the reconnect button. And we have the signal emitted slot that's connected to the clicked signal of the first button. So if your number keeps incrementing somehow wrongly, what you would usually do is use your instruction level debugger. You put a break point on the increment of the LCD widget and you see what triggered it, right? So what you will see is it's the click signal from the push button. That's not really helping, right? That's what we expected. But it doesn't really tell us why we get that multiple times. So let's... I already launched it just gamma ray. So let me make this a bit bigger again. Works. Reconnect. Still works. Reconnect again. Increments get bigger. Reconnect a few times. Gets even bigger. So how do we find anything remotely related to this in gamma ray? So the good starting point would be the button that actually emits the click signal. So let's find the corresponding object.  But most people don't give their objects sensible names. You have the address in the class names. It's not really helpful. What you can do with widgets and also with quick applications is to in-app picking of graphical elements. So if I control shift left click on the button, the right button is automatically selected. And it's even opening the widget inspector because it knows I clicked on the widget. And then it gives me the properties and the methods and so on for that specific object. So our problem is the signals. So we can check in the methods tab. Signals are technically methods in QMeta object speak. So that's where we're probably going to find the clicked signal. And first of all, let's check if how often it's actually emitted. So I can right click on it and connect to it. And if I now trigger it, you see in the output per click it's emitted exactly once without argument. So that's kind of what we expected. So at least at that side, it's all fine. I can also do the other side. I can manually emit it. It doesn't have any arguments. I can just confirm this. And we see it has the same effect. So I have the ability to monitor individual signals and I have the ability to manually trigger them, which in this case already helps me to bisect the problem. So the problem is not the emitting side, it's probably the receiver side. On a slightly bigger scale, we also have the signal tool that plots all the signal emissions in the entire application. So for the one click case, that is overkill. So this is the clicks coming in. But if I have problems with signal emissions on a larger scale, this comes in handy too. There's a useful feature to navigate between the various views you have in GammaRay. That's also in the context menu. So it offers you to basically select the same object in whatever other tool that might be applicable. So let's go back to widgets. So if the problem isn't on the sender side, we can look at, well, it's obviously not at the receiver side either because that is a one-line piece of code that just increments. So it might be worth looking at the connections in between. And we also have an option for that here. So for each object, you can look at both the inbound and the outbound connections. And on that button here, we of course expect an outbound connection to the code actually incrementing our value. And the fact that we see some warning icons down here already is kind of suspicious. And we see there's a clicked signal connected to the signal emitted slot. And once more, and once more, and once more. So apparently, we are accumulating connections. And that of course would explain our problem. And we can just reconnect that another time to verify it. And now we are already at six inbound connections on the slot. You can also by a context menu jump between sender and receiver to observe both sides. So this, I mean, now we have an explanation for our problem. It doesn't actually answer the question, why there is this problem. But if you look at the code now for the connection handling, you see one is done string-based, one is done pointer-based. They aren't interchangeable, so you can't disconnect a string-based connection with a pointer-based disconnect or the other way around. But at least GammaRay guided us to that specific connection handling code that we needed to look at. Yeah. So that's the first example on primarily on signal and slot handling. And if you try that with an instruction level debugger, it's very hard to find out what is actually connected to a specific object there. I mean, you need to look into the key object private internals or you see what is a print object info function that dumps out the connection. But then you need to find the right object first in your debugger and trigger that. So that comes in quite handy to have a nice visual view of this. Okay. So let's look at something else, something QML-based for a change. Also still a rather simple problem. Let me launch it with GammaRay right away. That's quick event handling. So even simpler UI. Let me make this slightly bigger. Just a single button is very hard to do anything wrong with just a single button. I can click it. I get some debug outfit from that. As long as I click like on the left side, if I click on the right side, nothing happens. So something is apparently eating our events. So the approach for this is similar again, right? I launched with GammaRay. The tool of choice for QML is the quick scene view. This looks slightly broken. There we go. Just needed a repaint. So we saw that clicking on the left side works. So we would expect that the element at that point in the screen is our actual button. So I can verify that with the picking tool in the preview here. So I just click somewhere on the screen and it selects the corresponding object. So if I now click on the right-hand side where clicking doesn't work, I see that there's something else selected. So there's something on top of the button eating my events. And we see it's apparently a second button that is not really visible. So let's check where this is in code. So this is another nice feature you can directly jump to code. It navigates your editor into the right place. That already works for QML. We are also working on making that work for C++ if you have debug information around. So indeed our application has a second button, but that's supposed to be hidden. But somebody tried to hide it by making it fully transparent rather than by actually not making it visible. At least that would be the theory, right? So another thing that GammaRay enables you to do is to actually live edit the properties, which is very useful on verifying this kind of series, right? So if we would actually hide the button by setting it to invisible, there we go. Then clicking on the right-hand side should work, right? So view this and indeed this now works. So this doesn't change the code. So I still need to fix the code, but with the code navigation to the right place, that's fairly easy. But I can test this theory without even restarting the application. Right. So we have the code navigation in QML, but as I mentioned earlier, we also produce debug output whenever I click on it. And for debug output Qt also provides source code information. So the same navigation works here. In the debug output view, I can jump directly to the line that produced the debug information. That's also useful if you get random warnings, for example, from within Qt. And you want to find out what's actually triggering this and how to get rid of it. Right. So that's the QML case. And I mean, this looks like a trivial example, but imagine having that in an application with many, many screens, and there's suddenly the not completely invisible on-screen keyboard hiding half of your screen. And you can click on the upper half, but the lower half doesn't work. Identifying that one element in a big code base that is messing up your event handling is just so much easier if you can do this kind of just click on this place and show me everything that's there. Actually, let me show you one more thing. That's a rather new feature in it. If I just use the picker, I get the top-level element at this specific location, which is most of the time the element I actually want. I can also do control shift click on that location, and I get the complete set of all elements that stack on top of each other there. For event handling, that's sometimes helpful, but that also comes in very useful for painting artifacts or just seeing what actually is visible there on top of each other, because all of this has rendering costs. So you get a fairly detailed view on what's actually stacked up there. And the same is also possible for widget layouts. Okay. Now we have my favorite example, something slightly more advanced than the event handling. Let me do this without gamma ray first. So this is two pretty QML sliders. It consists of a few rectangles in a list view and slightly changes colors. So pretty basic stuff one would think. If we look at the code, that's actually all right. So it's a fairly simple program, but once you run this through, say, API trace, which you would usually not do for something that simple, but which you would do once somebody complains that your application using your fancy sliders is slightly too slow. Oops. That's all we need. We see that the two simple sliders need 740 GL calls. We dive down there. We see there's a lot of them are draw calls. So that's the expensive part in OpenGL. Usually, one would assume that this is maybe four or five draw calls. So something is clearly wrong there. And if we look anywhere in this, in what's actually happening, we see it's actually drawing rect angle by rect angle. And this is like the slowest possible way you can do OpenGL. And then you lose all the advantages of Qt Quick. Now, how to debug this? I mean, there's a tiny bit of background information needed for this. Let me just jump to the magic code for this from the batch renderer. So how the way Qt Quick renders its content is it tries to batch together as many visual elements as possible and then merge them together in a single draw call. And then a single draw call it tries to do as much as possible in parallel. And that's where the GPU is really efficient. So this is like the sweet spot for rendering performance. That's where you want to get to. And this is the like the core part of the logic that decides if these batches can be merged together. But this isn't really, I mean, getting from this to why do my sliders not batch correctly in itself isn't really useful. So we need something to actually look into the running application and see, okay, this is the transformations we are doing. This doesn't match the logic and so on. So let me keep this here and search back to you. So let me run this in gamma ray. So for some of this Qt Quick has built in diagnostics because I guess certain other people also had similar questions when debugging the scene graph renderer. If they implemented this with debugging categories, gamma ray would give you the option to enable them at one time. That comes in handy in particularly for the rendering code because that produces huge amount of output every frame. So you want to see, usually you want to see diagnostics for like one moment somewhere at one time where you actually have the problem and that's not necessarily right from application startup. So you want to navigate to the screen that is broken, then enable the diagnostics and then look at what's actually going on. And thanks to categorized logging in Qt, you can actually do this with gamma ray and it also shows you the kind of different categories that Qt provides and that you could enable. Rendering is particularly interesting and input handling as well. Unfortunately the scene graph batching code uses some approach with environment variables. That still needs to be ported to this. So that's not actually helping us in this case. But it's the place where we started to see if we can actually get some information. So we need to look a lot deeper into what's actually going on here. And for that we have the scene graph view. So this actually shows us the internal structure that you hardly get access to from the outside of what the quick renderer uses to... Actually not here. This is the view. What's happening inside. And if you remember the complex condition for merging, one of this in there is the transformation matrix save for 2D only transformations. You would assume that our non-transformed rectangles or seemingly non-transformed rectangles would be fine for this. There's no rotation, no affine transformation or anything like this in there. So it looks from the result. But if you look at the actual transformation matrix that is applied, you'll notice in the combined matrix for one of the small rectangles in our slider that there's a negative number on the diagonal. And then anything that's not like same numbers on the diagonal is somewhat like a rotation. And that's actually triggering the condition in the merger. So the merger thinks this is a tricky rotation that it can't merge. And yeah, that's giving us the problem we have. So we have one step further already. We know that something is introducing a transformation that breaks the merger. But now we need to find out where this is actually coming from. And if you look, there's two properties here per node. There's matrix and combined matrix. Matrix is the transformation that happens in this specific step. And this is harmless. This is a simple translation. And then combined matrix is basically everything along the hierarchy up to this point. So we can go up the hierarchy and identify the node. This one, for example, there you see the matrix one actually is the one introducing the minus on the diagonal. So we can switch back to items and it identifies the specific element introducing the rotation. And, yeah, we see that there's some place in the code that tries to be clever regarding list view layouting and just flips it around by 180 degrees. Looks in the result as if there would be no transformation involved. But that's enough already to break the batch merger. So this is one and not that much time. So I'll just skip finding the other one and take this out as well. Same approach. So if you remove the first one, you find the second one. Just quickly showing you that this is actually the problem. So, of course, now the sliders are the wrong way around and the handles are all on the same side. So I actually broke the application. But if I run this with API trace, right now we are down to 76 GL calls. So this is, yeah, what was the other one? 700 something. So about the factor of 10 faster rendering or less GL calls, which is usually comparable to faster rendering. That, of course, still leaves me with an application where my sliders are the wrong way around. But that just means I need to find a solution that does this without rotating. Again, navigating through the scene graph tree with the conventional instruction level debugger, trying to find the matrix, print out the matrix in a way that you can actually read it, and then identify the node that broke it, map that back to code. Fairly complicated unless you have tools that actually support you in doing this. Okay, so I think that's the most complicated example I have for this. Let's look at something from a somewhat different field, classical old-school widget layouting problem. So we have this complicated dialogue. It's somewhat resizable, but yeah, this is as small as it gets. So apparently something is blocking it from making it smaller in the vertical direction. If you look at that, you would assume it's those address group boxes at the side. The street field sounds like something that can use several lines of text. So maybe this is actually all correct, but if I need the dialogue smaller, then something needs to be rearranged. So GammaRay allows me to verify that at runtime without actually changing anything. So I again select the widget I want with Ctrl Shift, click in the application, or I can also use the preview down here and the picking mode in there. So let's just hide this into a box to see if this actually fixes our problem, and apparently it doesn't. So something else is keeping up the space. So this approach is rather useful for kind of bisecting the layouting problem. Usually you would do exactly the same with recompiled cycles. So you take out something, you recompile, and you see if that fixed the problem, then the problem is in the part you took out and so on. With GammaRay you can do this in the live application. Saves you a lot of round trips in particular when you are working on embedded targets with slow cross compilation and expensive deploy cycles. So how do we find the thing that prevents this from getting smaller? You already see this blue area that's basically highlighting the spacing areas and the elements in the individual layout. So I can just click through those and see what's actually in here. And that very quickly shows me that there's apparently one widget down here, a label which currently doesn't have a text, which I don't see, but by just selecting it on screen, GammaRay gives me the ability to identify the, yeah, invisible things, but things that still need space. And for that, I mean, we can now investigate why it's not resizable. And again, for this the tool of choice is the property view. I mean, there's usual suspects like size policy. So as you can see here, GammaRay supports even the more complicated types in the property editor, at least the queue types or basically anything that's a gadget or a queue object or a type that we manually edit. So you can look at the individual details and modify them. Size policy looks sane in this thing, but the minimum size here looks like it could be the problem. So we can again edit that life and see if that fixes our problem. So again, this doesn't actually fix your problem in code yet, but you have verified that you have found the reason for your problem. And now you can actually fix this in code without even having a single recompiled cycle. Okay, so that's a midget layout thing. Right, something else I can show you quickly is state machines. Let me show the code for this as well. So this is a simple example that implements a traffic light with a declarative state machine, which I think is the standard example for anything state machine related. But you see, this is already, like, doesn't fit on a single screen anymore. Finding anything in there that, why is this actually not running? Let me start that. Finding any logic errors in a state machine just by looking at the code or stepping through the implementation is fairly tedious. And if you look at the flow of the traffic light, you notice that now it's going from green to red without going through the yellow state, but from looking at the code, finding the right place there for something that has four or five states you will manage, but in reality you have something with hundreds of states. Finding that in code is fairly messy. So what we offer for that is a graphical display of the state machine layout and a live view of the configuration. And there we immediately see that apparently we put one transition in the wrong place and have the green to red state dangling unreachable on the top. And you already see it's live updating and if I put the pause button on, so all the various state types in QDA supported by this. Right, and then one more. That's a brand new one that isn't even completely enabled yet in the released version. And I think then we are running out of time. That's the Q3D inspector. So here we have a rotating cylinder and apparently something is messed up there. We notice that the light reflections are somewhat often most importantly there's apparently some part missing. So if we look at that in the geometry view, and never mind the black thing that is a high DPI scaling issue, so in a normal version you wouldn't see that. We can look at the geometry of the elements we have in the Q3D scene. Middle mouse button doesn't work. If we look at the bottom of this, we see that apparently something is missing there. That's actually a fairly common problem in 3D geometry. If you render the triangles in the wrong order, you don't see them because you're looking at them back. Here we have a very useful diagnostic feature where we just disable that backface culling and render the stuff you're looking at from the wrong side in red. So we see that our geometry is actually correct. We are just looking at it from the wrong side. So it's just a matter of changing the index order. But again, the difference from you don't see anything to you can at least verify that your vertices are in the right place already. You just index order is wrong. It helps a lot to diagnose this problem. So that at least explains the missing bit in our cylinder. And the other feature we have in here is showing the normal vectors. And that explains the weird light issue we have. I'm not sure how well this is visible on the big screen. You see these tiny blue hairs? That's the normal vectors per vertex. So if we look at this somewhat from the side, there you see it, you see that they are somewhat bent towards the center rather than being perfectly straight off the surface. So that's again something that is that has some visual impact. But it's very hard to diagnose this without actually seeing what's happening. If you want to see more details, we can of course also look at like the raw buffer data that's sent there. But this is much harder to read. This is mostly if you're generating the geometry and want to compare if that's actually what you put in there. Or to find memory leaks. If you suddenly see bizarre numbers in there and far more than you expected and apparently something else is wrong. But the visualization is that is what really makes the thing useful, I think. Right. So I don't think we have time for the remaining demo. So let me switch back to the slide deck. All the examples I showed here and also those that I couldn't show here are all part of the gamma ray manual, including a description on basically what I showed you here and a few more examples that I didn't have time for. That leaves us with the question on where you can actually get gamma ray. The tool is available on GitHub. So it's available as GPL and also under commercial license. And it's also part of the Qt Automotive Suite where it also comes with a creator integration to make it even easier to use. If you want to see it in action or play with it at our table downstairs, there's a demo connected to an embedded device. So while everything I showed you here is desktop only, it also works remotely with an embedded target. Yeah. I think that's all I have. Thank you. Hello. How hard is a gamma ray to maintain? If it's all using all this private API, I can see just now what you showed. I probably broke something that was there in 5.8. So both how easy is it to maintain and how do you maintain it between different versions of Qt? Yeah. That's indeed a problem given that we support anything back to Qt 4.8. And we use a lot of private API that you guys keep breaking. Well, the best thing we can do is large CI coverage and adapt it for every Qt version. So I mean, if you look at the code, there's a lot of iftaves, there's a lot of special cases for different Qt versions. A lot of features are only available in certain Qt versions as there's new stuff being added. So we try to release with every Qt version and try to keep up. I don't think there's a way to make this easier. I mean, we already added a few not entirely public but somewhat stable APIs to Qt to make this easier, like the hooks for object creation and object destruction. Before that, it was a lot more messier. But yeah, it's work. I don't think there's much you can do to simplify this. Thank you. Can you use gamma ray on an embedded device like debug or edit properties, whatever? Do you have a server running on the embedded device? Right. The probe part, the part running in the application actually opens a socket and that's what the client connects to. So this works locally as well as remotely. We have that downstairs if you look at our table there, it's connected to an embedded device. Thank you. And it supports Linux, QNX, Android on the embedded target and Linux, Windows and OS X on the host. Any more questions? Thank you, Fokar. Thank you.


Right now at ReadLion.com it's an important new study and Dr. Susan Berry wrote all about it. You should read it. If you're a parent or thinking about becoming a parent, it's probably not a huge secret to most common sense folks. Parenting matters. Doing it well matters.  And it doesn't matter race, it doesn't matter socioeconomic background. This study bears it out from Gallup and the National Institutes of Family. If you're a parent who's attentive, loving, and also disciplines your child, turns out your child's likely to not have nearly the mental health problems that so many kids have today. Read all of the particulars in today's study at ReadLion.com.