ng the opposite way in this idea of starting with the dance and seeing what algorithms are already there and then applying them to the technology. And then sort of my last robot piece is my collaboration with Alex McLean, which hopefully you all come and see tonight. So we've been exploring movement, drawing and live coding using strudel as a way to choreograph robots. And again, there's this layering of human input, live coding, improvisation, and machine motion to create an overall performance composition. Recently, I've also been on a collaborative project funded by the Canadian New Frontiers in Research Fund. And this is a project with Dr. David Ogborn of McMaster in Hamilton, Canada. We've been creating something called Locomotion, which is a live coding language for the movement of 3D avatars on the web. This project has a huge team of research assistants, including Shaden Ahmed, Ashmeet Deval, Melissa Hennick, Vic Wiskowski, Misha Jiao, Esther Kim, Sneedy Goldstein, and Andrew Parris. What happens in this is there's sort of these layers of choreography as I see it. We are using motion capture. So there's a human being captured by a machine. This motion capture is then live codable with our language. And then it's reproduced by these digital figures on a screen. So the project brings together screen dance, live coding, and avataring. Here's a little clip. This particular clip was done by... Oops, it does this every time. Let's see if I can go back. It skips this one video. It's very annoying. Let's see if I can get it to go. This particular clip was done by Misha Jiao and Esther Kim. Now we'll go to the lab one. There we go. Locomotion can work as a standalone language or within estuary. So you can do networked performance. This approach to collaborative choreography with motion capture in 3D models is quite unique. In our last performance, I also had this sort of dual role of I was controlling a 3D... like a 3D camera in the space, which to me was this great... like I felt like I was making a screen dance piece because I always think about the camera as another dancer in screen dance. So I had that sort of effect going on. But I also had this role where I was instructing how to move their avatar in the virtual space. So I was like meta-live coding. And I was also doing this translocally as I was in Richmond, Virginia, and the other performers were in Hamilton, Canada. This video clip isn't from that performance, but it's just a really short clip to give you a sense of like multiple avatars moving in the space. This is actually a very old demo now. We really need to update. So to briefly summarize my time with you today, there are always new ways of placing ourselves in relation to our code. Live coding allows us to shift, change, repeat, all while emerging compositions are displacing those that have come before. It's important that we embrace the slippage between placement and displacement, allow for human agency, and see the possibilities of improvised performance with programming, whether it be algorithmic choreography, animated visuals, or techno music. Thank you for choosing algorithms, and I hope your ICLC is full of human live coding experiences. I think there's a little bit of time for questions. Anyone has any burning questions? I don't know where the other mic one is. Are there any questions? So thank you. That was a lot of fun, I think, and inspiring. I mean, have you found like that programming can also learn something from dancers in the way that... Indeed, this idea of agency, of course, is very interesting, but also on a core, kind of almost technological level, like how bodies move and interact. Do you maybe have... I mean, you've shown some things, but... Yeah, yeah, absolutely. I mean, that was kind of the starting point of the robot work, was that as dancers we have so much information about movement, that it doesn't need us. Yeah, so this idea that like... Yeah, just... Even simple things of like... Yeah, like different dynamics, like this idea of like, you know, you don't always want to move at one speed, like that gets really boring. That's boring repetition, right? Like what happens if you like, yeah, start to develop that as a motif, right? What if you do it in a different direction, right? Like there's all these ways we think as choreographers that lend themselves to that kind of work. I think also the direction we're going with the foresight work is really interesting, in that we've like deconstructed these sort of improvisational strategies, and then we're trying to teach the computer to do them. So we're directly taking the algorithms from dance and putting them into a coding system. Another question over here. Hi, Kate. I have a question, so... Yeah, clearly. I'm interested in like sonifying different genres of dance, like for example, like Chicago footwork, like dances that have classical like just like move sets, I guess, or like associated with them. I want to know like if there's been like... Like what are the challenges with being able to capture those move sets and then being able to sonify dance and like collaborate with different kinds of dancers and things like that? Yeah, I think, yeah, when dance forms have like, yeah, these set move sets, I think the hardest thing to then capture is like someone's style or essence, or like what makes it special when they do it? And I think like, yeah, when we start to capture things, like we can lose that, because that's like that's almost impossible to capture, right? That like, oh man, like look at that person doing that one step, it's magic. So that's like the hardest thing to capture in that kind of dance for sure. I mean, I think the tempo thing that Zia and Alex have been working on is quite interesting. This idea of like, oh, if something happens repeatedly, then maybe we can like pull off of that. And then of course the dancer I'm sure will change. Maybe that's a good thing because then we can measure that change and we can follow that. So that's like an interesting approach is to follow the dancer rather than being like when you move this arm, it's going to map to this sound, right? Because that gets really like, I really hate one-to-one mappings and anything. I'm really much more interested in like, yeah, the balance between, yeah, doing something and like different things happening, yeah. Okay, got one question from the online audience. Lee asks if if the team has published any of the robotic controls. Yeah, all of our robotics work is open source. You can find it at the HiveLab lab GitHub. And then we've also, we've published some papers on it too. If they look at the movement computing conference, that's where we tend to publish. Okay, thanks. I think we'll wrap it up here. And of course we're looking forward to the performance tonight. Yeah, thank you. Thank you. Okay, is everybody inside? Are we still waiting for people? It seems like most people are here. So welcome to the third and last paper session of the day. This one will be a bit longer. We'll have four presentations and they will pick up some threads from this morning and slowly fade over to things we'll have tomorrow. We'll start with a top-level view on life-coded music and then I forgot the order and I will do it on the fly as it's habit and the life-coding scene. So there's a slight change of format because the first presenters prefer to do a shorter session and answer questions right away, so we'll hear one short presentation and we can ask questions right away. We'll continue with algorithmic composition and numbered notation. Interfaces with machine learning. Interfaces with other technology and life-coding will be picked tomorrow as well. And we will finish the day with addressing accessibility for blind and visually impaired life coders. Don't run away right after the question and discussion session because we'll have an announcement from the nine conference, a brief announcement by Jan Verdonk. So with that, I'm giving the microphone to Georges Diapoulos and Martin Kalle. So hello. I am Georges Diapoulos and Martin Kalle. I am doctoral student at Salmers University of Technology and University of Gothenburg in Gothenburg, Sweden. And Martin is postdoctoral researcher in Ionian University in Corfu, Greece and other laboratories. We'll be presenting irreproducible musical analysis of life-coding performances using information retrieval a case study on the Algorithm 10th anniversary. So the Algorithm 10th anniversary was a big event celebrated with many live streams and physical performances. I wasn't sure if I have attended the physical Algorithm but due to collective memory I thank Patrick. Finally, I learned that I have attend one in Kalsure in 2013. Yeah, so the first study is by Collins and McLean in 2014. So at that point the notion was quite early. There were 10 algorithms up to then for this period 2012 to 2014. As I said, the 10th Algorithm had a 24-hour stream that was on in two days and it was just a live stream with 144 slots and 133 performances in total or this is what has been this is the repository in the Internet Archive. There were 13 physical and virtual events in total internationally. Today up to December 2022 there were 322 Algorithms worldwide. So in the 10th birthday party which was the live stream 24 hours there were 363 performances in Europe and in North America 20 in South America 20 in Asia, 2 in Oceania 1 in Africa. The good thing is that all 5 continents were represented somehow western sorry 6 continents yeah well yeah yeah now the programming language is used there was tidal cycles with 52 annotations so these were retrieved by the Internet Archive so below there was a description and then there was a manual work that's why they are annotations. 18 Super Collider 14 Sonic Pi 8 Fox.7 Orca and one C SoundTuck and the other maybe pure data or yeah I don't know you name it. Not an answer I couldn't figure out what is this about very individualistic setups now the study that we present is a reproducible musical analysis based on musical information retrieval so musical information retrieval is an interdisciplinary field coming from news college informatics to psychology and machine learning DSP digital signal processing the applications they are recommended systems like in Spotify you get the next recommendation source operation we use in VSTs to separate the drums from trumpets authorizing recognition to recognize if this is a forest or a city for example now we follow the cognitive MIR approach meaning that and the focus is on the musical form musical structure so from music recognition theoretically separate between primary and secondary parameters so the primary parameters they exhibit proportional relationships between them so the pitch for example we can hear if a note is higher than the other and we can make a mental sort of understanding of this rhythm and harmony so these are the three primary parameters the secondary may be seen as byproducts so for example brightness it's a byproduct of pitch I mean it's higher pitch is like 2 to 4 kilohertz and then pulse, bit, meter many other characteristics now in this study we analyzed 133 performance and we constructed a visual representation of musical form based on acoustical features so this informed by music recognition specifically we focus on the shorter memory which is a time window between 0.5 seconds to 8 seconds and then this based on this assumption that we took we kind of confirmed some gestalt principles like good continuation and proximity so the visual form looks like this we later do a demo and we will explain more but the shorter memory is shown here with each slice so each slice is up to 8 seconds and this is like a representation of a 10 minutes performance now we go to Martin and to the reducible part of this study to us was a very important feature of the study for two reasons to get the data from the internet archive there is some conversion necessary to kind of extract the audio from the videos that are there and then do processing on pre-processing on the audio do the analysis so this is all work that runs for hours for a very long time and there is software involved to do that so we want you to trust us that we are presenting should be reproducible by you to verify it so that's the classical use of or need for reproducible research but we also want it to have accessible this kind of pre-processing we have done the algorithms are used you should be able to use them right away without doing all this analysis that you would probably not do so these are the two exciting factors for having it reproducible then reproducibility is actually related to literate programming in the sense that to reproduce something you need to kind of understand what were the steps and it's nice to have instructions around it better the code being inside of the instructions rather than just comments to the code so next and org mode I was very pleased to see in the second presentation of today I think somebody was using org mode and especially orc babel that kind of was used to run the superglider code so this is useful for for doing live coding as well so I'm more into that but in this case we are running a jupiter set up environment where you can do coding even live coding if you like on the code that we bring along and we do this through a Docker container infrastructure you might be very familiar with it if you're not just a few words on that so it's a way to have a self-contained shipment of all the code necessary to run and in our case, as I mentioned before we already have made a Docker image that also contains the data so we can start off that thing and run it right away we will demonstrate that next so the build is done by a docker file then you have a container runtime and docker hub is where the image is so this all together makes this quite popular infrastructure of container handling and building shipment which we employed here you should need nothing else but go to docker hub and download the image and you can run it and all the code will be there and finally what was there? that has been we can step forward to the next slide this has been realized by multi-stage builds which is a bit more familiar with it the analysis is run while building the docker file so you can do that you can build the image yourself and that means that all the analytical steps will run during the build and it will take 16 hours or so the multi-stage build goes to the docker so this is the overall I mean, yeah that's the structure how it's done so we use docker and so we have a docker hub which looks like this essentially what you do is that you can go there take this line copy paste it and if you have docker installed you go here and you get the pre-built image so now I copy paste this to my the last line yeah the last line it's the very last line it's just a local host connection to a port and I open it in my browser and this is a full development environment here so here I can open a jupyter notebook for those who are familiar with jupyter notebooks, a console, a terminal I can open files, read files etc. now we have one working directory and here we have some prepared data sets so these are the acoustical features we extracted with Librosa and these are some configuration files and for the annotations and for the just configurations files very small this docker hub doesn't have the the data from the internet archive and we mirrored this repository to a github to be more reproducible because you have the commit there and from the git for large files we take the video data to the docker and then we extract, we transform the video to mp3 also and we do the analysis to the audio and then we run the runtime which is this so we have the data an empty image folder and the notebooks it's one notebook officially and which is this, this is runnable now I load some libraries I make some custom functions this is an encoding of the acoustical features some of the features they are it's not a vector, it's a multi-dimensional vector I mean I load the prepared data to descriptive variables here so for example these are the mfccs the sentroid etc and then I do the general sanity checks in the between unique identifiers of its performance time tags, namely and then this kind of a very bad kind of descriptor like one number for each performance for ten minutes performance just the mean value of one feature but nevertheless we make some plots okay and yeah and then we can go to the videos, we can play a video so for example just the first maybe this is how the original data look like you get the idea these are the annotations to the language so we made a plot based on the language which means make this plot which means that the language they do not define how it sounds so you can, regardless the language you use, you can make all sounds which is a good thing so there are more figures in the notebook then and the publication that's interesting to go there if you're really interested and see what analytical steps they have been done there's more than actually fitted in the paper and all these end up to this visualization which is a representation of musical form so we have here this is a 3D representation so the radius corresponds to an indirect measure of the event density and the heat map corresponds to brightness so the longer the more events the brighter the color the brighter the sound so here for example in 40% there are some simple sounds and here are some more like events and here we see this a blank slate live coding so we see nothing in the beginning no sound this is how here we stop us I think you should show the full comparison of all ah yeah and to see the purpose of it so yeah and yes we can also run this will take 45 seconds in between I can open this and I can open another so you go here you open another terminal and I can plot I can plot the the tempo distribution across the performances and it's here and now we have also the 121 performances which looks like this so you have a comparison it's a performance this normalized based on its own so we don't do any cross comparison between performances this is its own kind of evolution and the vision there is to use this also for interactive experimentation so you can kind of have the visualization of your progress how the the performance starts for example from 0% and then here you can see some brightness and then less bright etc so do we have time for questions there you go so my question is as I understand these are plotting the relative duration of each performance one piece of information that is not so easy to extract from videos but could be possible is how regularly new code is evaluated or how long between different evaluations of code so then it could be interesting to also plot the difference in sound between evaluations and to see how different performers treat evaluation as more or less drastic changes to the sound are there any other similar things that you thought about but didn't have the time or the resources to do so this corresponds to the video data if from the video you can really identify when because some systems they do have this code highlighting this analysis only based on the audio so we don't use any video so you can do that do some computer vision analysis and then some systems they don't have this it's quite hard to know when something is evaluated but for Tidal or Strudel or Supercolider you can definitely do that that would be very interesting I guess it would also be really interesting if you had access to the OSC data the patterns that the code generates I think that's impossible it's impossible for concerts that have happened without that being taken into account but maybe moving forward if life-coding instruments could have an option to actually keep a record of all that that could also be interesting exactly, forward your question there's a session on how to document other apps oh right that could be so yeah anyone here who makes life-coding instruments thank you any other questions? thank you of course here you can see the form of each individual performance where you're also able to maybe see if there's some some trends coming up like for example in the form of that because one might think or sometimes critique on life-coding might be that there's always a very slow build-up so is it true where you're able to show that with your data? yeah if I understand so for example here we see a climax in sound events so the good thing here is that you can easily spot some perceptual you can easily see something and kind of hear how it sounds so if we go actually on this video and we go on 40% we're gonna hear this so you have fast recognition of climaxes in the music etc was that your question? this image would confirm yeah right this would confirm the prejudice if you change to the image where you can see all of them it's very surprising that you see very different forms actually what this shows and this is only for one feature it's not a 3D but if you do it 3D etc what it shows is that everything is totally different first is enormous nobody plays as anybody else because it's an improvisational practice that's thank you yeah with these specific ones with much yellow and the big ones does that mean that there is less dynamic than in the other ones because there are less quiet parts or so these specific ah okay but the length is what's the length so the color here goes with the length I thought the length would also be this is a 3D representation so here what's your question exactly if yeah so here the length is try to analyze the dynamic of the loudness or the difference in dynamics in different performances that was my question yeah I mean for example if you map the length here to the this feature that you talk and the other to whatever some pitch so the length here is a rhythm based feature and the color map is a pitch based feature so you can use this too whatever you want like the dynamics you say and then the I don't know the spectral centroid or some other like how flat is the spectrum maybe spectral flatness and you can get that maybe the typical answer here would be change a little bit of code and you plot it differently and you answer your question that's actually a value of that so that a lot of questions rather easily by changing and following the code that is already there changing a few mapping parameters and you get a different plot that kind of gets you to that point and easily to an overview of all of them so that's an interesting way of integrating the data yeah thank you next we have Michalo, he'll present something that was already hinted at this morning ciphers or ciphers I'm not sure how sharp he wanted that in there algorithmic or number notation for algorithmic composition and let's give the stage to Michalo okay hi everyone my name is Michalo and I come from Finland Helsinki area I'm a part-time doctoral student at Aldo University in the Department of Computer Science my main research field is link data and information management so this is more of a distraction from my main research field but still I'm really really serious about live coding and learning a lot about music meanwhile my background is more of a software architect and I work as software architect consultant and doing deep part-time doctoral studies the other author is Raphael who has helped a lot and encouraged me to write about this subject and he also pointed out that there's some well-concepts and it's good thing to write about this okay what is GIFRS it's a number based notation for live coding and algorithmic composition so it's a iterative result of unifying different kind of notations used in traditional score historic numbered notations and live coding it started as a personal project that could run in sonic pie buffer but it has grown over the years it's now about five years old and basically I first I reinvented the numbered notation then I started to learn about the history of notations and uses in live coding and different mini languages other than sonic pie also it has evolved and grown in the different directions and it has also developed through this course within the sonic pie community in the forums and there's been a lot of helpful comments and posts about how to improve the notation okay let's try if this works so this is the basic notation numbered numbered notation here means that these are basically pitch classes in a scale and by default it's a major scale because I haven't defined anything else other than that I can also use characters to describe some samples and then I can do different cut-offs and things like that what sonic pie basically offers okay when I first started to live code with sonic pie I got frustrated with the kind of data structures that were used in sonic pie to create the pitches and the rhythm so there were separate structures for creating sequences of pitches and different structures for creating a rhythm I wanted something that describes the both in an easy way so I first reinvented the numbered notation but then started to learn about the history of numbered notations and then named a new notation after Sifras system that has been developed by Natorp in early 19th century and basically it looked like this and I wanted to create live programming and algebraic composition language that can be used as simple as this so you can define different lines of notations and create loops out of that then I went really deep into the history of numbered notations so let's do a quick overview it's really interesting and complex and contradiction history where there's always some reference an article that mentions numbered notations in a sideline but there's no comprehensive history about numbered notations specifically the first occurrence of numbered notations was in the 16th century at least it's the only one that has survived so Pierre Damantes published Deep Sounds of David using numbered notations and his idea was that it would help singing usually the notations is only for the first first verse and then he used numbers for the second verses you can see behind you have the melody in one line then you can have numbers in the following lines so it would help sight singing and the next one was Antoine Paran and he used this notation for more advanced numbered notations for creating these polyphonic presentations and he also devised first numeric stuff and then Kirsten Antanasius invented Arsamusa Ritmika in the 17th century he used combinatoric techniques probably these were unique inventions they might have known about each other but it's also possible that they didn't know that they were inventing the numbered notations separately and this was also the kind of first algorithmic composition in a physical form and then probably the most famous Rousseau developed his numbered notation in the 18th century and it was widely ignored you could say they praised that it was good but it didn't get widespread use then Ludwig Nador in the 18th century he reformed music pedagogy in schools and he used numbered notations to teach singing in elementary schools but after that, Gallim Pierre extended Rousseau's notation and even though these are in a timeline it's probably possible that they didn't know about each other's publications because books were still a bit scarce in these eras and then the Nordic instrument psalmodicon was very heavily influenced by these numbered notations and first psalmodicon was the developer invented by Lars Rohrund who got the idea from German monocord and he also created his own notation, numeric notations for teaching this one-stringed instrument and it was used widely in the 19th century in schools and churches similarly psalmodicon was invented in Sweden and it's also not known if they knew actually about each other but and I also managed to find this Finnish numeric notation book from local bookstore that defines this building instructions for psalmodicons and also way to perform a ceremony and in the mid 18th century the Gallin-Parishev method was built on top of roses and ideas to they were actually building this kind of a school they were teaching this notation and they gave certificates to people who learned this notation you can see the certification on the background then Heinrich Franz published a choral book that was used widely across Europe in Mennonite faith and also in Russia in the beginning of the 20th century and bit before the numbered notation also spread to Asia through Christian missionaries and in different countries for example in Indonesia they created their own numbered notation for example in Indonesia and also Yampu in China that was improved the local notation and of course this is basically an unrelated innovation by Schoenberg but in Zifers these ideas about using numbers and arithmetics are combined so many ideas come from the 12-tone technique and post-tonal bits class theory okay, so the objectives for the notation was always this minimalism what is the most minimal notation you can use to compose music and live code and what is the most minimal in a way that isn't mentally challenging if you compare esolongs like organ bite beat bite beat is much more mentally challenging how does the organ make the language simple but very effective so in a similar way Schifers aimed to create most minimal numbered notation for live coding and flexibility so it should be suitable for live coding generating sheet music do numerical analysis and be open for multiple interpretations for the numbers so bits, classes, decrees, interval sequences or MIDI or CC message or anything and also the learnability was a big objective so it should be really easy to learn and write any musical sequences in any key and possible to write down ideas quickly in anywhere for example notebook or do the live coding on live setting and also the interoperability so the idea was to create a notation that would work in different live coding environments not only one but also enable transformation from traditional score to live coding and from live coding to traditional score so the implementation as said the first implementation was in Sonic Pi Buffering 2018 then after many iterations I also did MuScore export plugin in 2021 and new generative parts are using 3Dop in 2021-22 and also just months ago this gala scale parcel that enables you to mix and also the first implementation for Python that enables it to be used in Sardin and also in Music21 Algorithmic composition framework so the architecture for for the Sonic Pi looks like this it's built on top of Sonic Pi as extension using monkey patches and all kind of hacky and ugly things but it works you can use any type of or most type of input as a parameter so you can do string patterns you can do integer patterns you can use arrays or lambdas or these enumerables which can generate any kinds of number sequences and you can create music from Pi sequence or any kind of mathematical sequences so there's few main methods chat play method is basically playing the melody once then the slope method and also parse method that both both of these methods uses and those are using those parsers there's three different parsers the basic parser for basic melody and generative for creating these more complex melodies and for for creating different kind of algorithmic patterns was that two minutes or okay so I already told about the chifers for music or plugin you can select part of the melody and it outputs chiffer notation so you can play it and you can add for example different kind of transformations to the notation but then when the melody goes on it creates for example retrograde or octaves and things like that and then you could start messing with the melody and start to create different kind of let's go really quickly the basic notation and stuff so as you saw it uses integers to define pitch classes and then you can use scales you can use integers arrays or enumerables pitches are defined you can use negative pitches or positive pitches the new version uses the pitch class notation and not the degree based notation that is used because in our experiments we notice that it is much better to use the pitch class notation because zero provides this kind of a natural inversion point and also it's not really useful if you create these arithmetics that create zeros that would be rests in the notation because in the traditional notation the zero meant rest but in jifers the zero is the root of the scale and in operations you can use these characters so for each typical note length there's different characters you can use all these I'll know this melody you can also use dotted notes and things like that it also includes ideas from tidal cycle so you can use the note lengths or you can use rhythms of division things like that so these are the same melodies with different kind of notations there's a lot of different characters for different durations and the chords uses the most minimal syntax I could think of so chords are grouped as numbers and it's a trade-off between scale decrease that has more digits than nine but it's the most simplistic way to define a chord and you can also use those Roman numerals and then the generative notation I mentioned had also these minimalistic syntax for creating random numbers and loops and different kind of things and some of the inspiration also comes from traditional notation so this is this is the notation for traditional loop so it repeated two times and then you could also do loops with random numbers and things like that and then there's much more complex list operations you can do create lists which are basically containers for numbers and you can sum them together to Cartesian sums and create this algorithmic sequences and it has a lot of other stuff that I don't have time to go through now but you can also use it within Sonic Pi so you can use it as a way to play a scale and let's jump to conclusions so numbered notation is really well suitable for algorithmic composition and provides easy introduction to this exploratory live coding and for especially those who are not familiar with the music theory you can just use numbers to explore scales and define your own scales and come see what comes up from the random numbers and loops and transformations and as mentioned there's also new version for Sardine so you can use Cyphers with Sardine and there's also some support for D1 so you can also use it for generating scores for example with the music 21 framework ok thank you cool thanks Mika let's keep questions for the end and well kind of let's do them now because the topics today are a little bit divergent so let's go ahead and say hi for this session and yeah does anybody have a question so yes go ahead so one of the things just because you're drawing a parallel between Western notation and other more contemporary forms the thing that I find really powerful about Western notation is that it uses the Y axis as well for really important information and usually when you're reading chords it's much easier to look at the overall shape and the first note and then you extrapolate what the other notes would be based on their distance so it's a relative specification for the chord so my question is have you played around with specifying chords by using the first number to specify the absolute note and then each other as a relative number for that chord so let's say if you wanted 5 to 5 it would be the fifth note and then two notes over that yeah something basically using intervals yeah but only for the rest of the chord not for the first note and then my second question is have you played around with using superscripts and subscripts as some kind of way of faking that Y axis of Western notation so having a number and then a superscript number over that or a subscript number underneath that I'm not sure what you mean so it's when you have 2 to the power of 2 for example that's different to having 2 next to 2 but would that be some, would that be useful or interesting do you think so the question is instead of just having numbers right next to each other if you could stack them vertically somehow but the only way you can do that today in let's say a browser environment is with a superscript yeah of course this check first check back in notation that if you have stacked multiple lines it means different things for example you can do polyphonic sequences so if you have multiple lines they sound the same time or you can do also the same but if there was a kind of tracker on notation okay any urgent questions we're kind of short bit behind so maybe we can still people are still around so there's still plenty of questions now we have from algorithmic composition to interaction with machine learning models Junichi Shimitsu presenting the Jenny system so let's give the stage to Junichi do you want to mirror your screen or I didn't do that some technical problems sorry for that the other one thank you thank you yes thanks so much hi there I'm my name is Junichi I'm visiting researcher of creative computing institute and today I'm going to introduce about Jenny designing and exploring live coding interface for generated models my co-author is Rebecca Fiebrink she's also a professor of creative computing institute so what is Jenny Jenny is a web based live coding interface we're built new environment from scratch and we're with some generated model API for interacting with our clients and also we provide right-weight code patterns to generate rhythm sequence and also we discuss transparency and authorship and creative expression and visualization technique and another aspect is also we're exploring design pattern with using this interface first one is we made some model API to generate the rhythm pattern to get from generated models and also not only the using generated models we prepare some pattern functions which mainly for making rhythm patterns with live coders and of course we can combine this technique with combination of machine driven generation and also human pattern making process and yeah we provide several avenues to make feedback group to machine and humans for the live coders so let me begin start off the experiment little bit about background of generated models for live coding first one is more generally more broadly generate machine learning algorithm for music creations so I think you already know about and they have a long history first one is using macros chain model like more to generate the musical sequence and more essentially our NSVA or even transfer model can doing a lot of capable things to do so they can also generate the entire novel musical sequence from scratch or even generate continuous or partial sequence from provided users data and if you look at tools for the interface and generated models machine learning in the music they have some several works in here most famous one is maybe here Google machine industry they are building music production too available in the VST world with applications but basically mainly for the music productions and some researchers even try to figure out how not only providing GUI too but also a combination of development the model itself and how can apply these controllable models with GUI but most of the work is the recent as a GUI and there's no because they don't want to focus on programming environment for the users but if you look at some more live coding context they have some similar works in trying to apply to generate models to the live coding or programming so there is a mimic there's a web-based programming environment they're providing really high-level API wrapping machine learning functionality and at this moment they didn't focus on the live coding stuff but they can possibly apply in this program environment to live coding interface and as an example is a SEMA this one might be most relevant work for our research they're also building some live coding environment but focusing on user's own live coding designing user's own live coding language and this programming environment require really substantial bio-project code and lacrosse efficiency so designing live coding language for generate models still has a huge challenge because if you look at generate models they have online offline inference we need to care and the model size is huge we have to use pre-trained model and what kind of data media world or audio generate from this model and the live coding environment of course we need to inference in real time and of course current the live coding environment mostly not designed for the machine learning or generate models so how can we design the whole workflow more friendly, accessible for everyone what is the gap so we're looking at making some API for more accessible the researcher and also live coding context so this one is our implementation of how generate models can apply in the live coding editor so here is a configuration file which mainly for use for model developer so here is a list of which model the model is using or any other properties in the configuration file and also not only the model parameters but also some described inference methods to get the result from generate models but of course the interfaces don't know what kind of models API are available from these interfaces so connecting this configuration file to the interfaces and then live code can easily access each property through this looking at this configuration file so how Jenny works I will try to explain a little bit about short expression using the video I will explain more detail later Jenny has three UI components first one is code editors as you can see and also they have console view to output data to see the data and also some pattern sample bottom to switch between template code so code editor is basically pressed to run the code it is basically JavaScript based syntax and of course we support some auto compression assist to access the model parameter and we have some two types of keyboard command execute command code and also interafter stopping audio command and another component is console view this one is notifying the UI status so when updating the code the user get from the result from the generate models and this console view basically shows the whole result from generate models so the audience can understand what can data get from this model and of course we have some template code to run at first so we are preparing some sample bottom to switch template code and another specific implementation is that we design pattern representations for making a rhythm which is not using the generate models but people can make the rhythm with using this pattern and right now we focus on the rhythm so as you can see they have two signatures one signature is time representation and the other one is middle number which correspond to the drum type, general middle type and this time signature is basically from ToneJS ToneJS is a web audio API for making interactive audio and current limitation is we can only define two measures and by using the ToneJS more easily to execute command without stopping audio or interacting metric from so here is how to use the pattern for this project it's quite simple just defining the rhythm and representation and following that point of action to create the pattern or not and also we did a couple of visual expressions first one is a JS background so we support some JS script to show some visualizations and also we specifically designed for some methods, similarity which is basically calculates the similarities of data from general models so the people can understand these data or similars before and after and these two dots we can see these correlations show the similarities so if the dot is more distances close it means similarity is high so yeah I can play some basics yeah I hope you can understand when the generated data sounds different these similarity dots is going to be more far distance so yeah how to use this pattern in practice for the live coding so basically we have a fundamental method to generate the data which is called gen functions so every model we are going to use in the following adding this gen function to get the data from general models and optionally we are preparing some output function so when we generate some data the live coder can generate data through logging from the output function and why we need to output functions so I'll play this video so it's not only about getting the result from general models but also transforming the new patterns to modifying your own rhythm so we're making the output function to get some result and modifying and transforming the new pattern and also some other methods will be available but not that these functions will be available in the models but in this example we have some input method which can fit into the data to generate the make the variation to the models and also interpretation method was combinations of different patterns to interpret the rhythm and so here's another example of how transforming the rhythm so the last part is we control the temperature on the property it means making the more different variations from the base on the input so the live coder can explore we want to make more similar variations or we want to make more not similar variations and they have some other type of general models I'll quickly play the video as well so steps how much do you want to make continuous melodies so if I put in 60 or 32 notes which means this model can generate 32 notes based on the input so this one is the last video but I will try to play it again until you got the idea of our work so this video is just the iterate process to how to make new patterns using getting from the general models and converting or transforming their own data so I will a little bit discuss about this work so first one is the transparency so we need to expose more substantial factor of general models and algorithm to make sure what kind of models we are using so we have in the configuration file we have some setting for describing checkpoints which means it's not mainly does not change in the happen in real time but we can show at first the checkpoints to show what kind of models we are using and output functions and console view have to understand what kind of data get from general models and even without output functions we make sure console view can see the result and there are reliability in the bias it's not address right now we think we need to add more supplement text such as what kind of data set we are using or size or license something like this and authorship we find out this design iterative process to improve the authorship as you can see in a lot of video we are making the variation based on the user's own data and then we can transform without live coding the human pattern making process so they can easily modify or combine other favorite data and even we can layer the model patterns at the same time so even we delegate the models too much but we have some still to see the data and current's limitation is we focusing on the rhythm patterns and we do not support training on data set and other discussions created expressions so yeah basically if we don't have any calls or before hand we can just start in the global expressions which means we can generate the model from the models and then if we found some favorite data sets we can just vocally in explorations and we still need to consider level of abstractions we currently very small number of the options we can support to generate but potentially general models have a lot of capable things so we can more applying lots of possibilities to challenge and as the data set or audio might be good for the exploration and yeah that's one visualization is even very simple like showing just only similarities but might be helpful for understanding so we want to keep and continue to investigate the other visualization method to to the effect for the live coders so what's next we want to more explore intuitive data representations and also we want to publish your components and more applying or combination of current live coding environments and also we want to know more adding some current general models so here's a summary so we introduce Jenny our live based live coding interface designed for working with general models and we show some how these patterns can afford super unique design patterns and for live coders and what kind of interactions can we use can we make and we discuss reflection our implementations, transparency authorship creative expression and visualization thank you so much thank you so much for your presentation yeah let's continue with the questions anybody has a question for Tunji regarding the Jenny system as an alternative to the output function would it be possible to pass a random C to the gen function to always get the same result could you say that again if I saw it correctly each time you evaluate you get a new generated pattern and then you use the output function to fix that pattern but could you also pass a random C like some number to the gen function to always get the same result yeah right that's a really good question it depends on I think the models current models does not support like random seed so yeah if you have more huge models I think it would be great to setting random seed or setting some even the random version any more questions do you see any hope to use these kind of models on languages which are only used by one person like obscure languages which do not maybe have a large body of code yet just only use one person okay okay so there's hope yeah maybe one more question if anybody has one of course yes so I saw this temperature so when you adjust the temperature do you think you can kind of guess how it will sound a bit so if it's increased the temperature value it sounds more I don't know caotic and if it's less it's like this so the temperature concept is how do you want to generate more randomly so if the temperature is the high the users or our expectation is get the data from more randomly and more different the melodies with data so but sometimes the general model does not understand the perceptual similarities or randomly so it might be so that is why we added some similarity visualizations to understand the difference between the general models and the human perceptions cool so another applause for Junishi Shimitsu and Jenny very promising system next on we have Matthew Keeney addressing accessibility in blind visually impaired life coders and while he is setting up I want to apologize to Mika because my own tiredness the delay in time and all the technical trouble I didn't give him proper thanks for his different notations so another applause for Mika sorry for that didn't get too much sleep last night so okay let's we had this started is just connected but I mean it recognizes it does something connected it's also sometimes a power problem that none of them I mean it recognizes it does something there yeah but it's a frequency problem that all the time it's connected sometimes it helps to put your computer on power especially with or on and off plug it into power plug it into power because it's USB-C having trouble with the projector just google yes really sure yeah one more time okay okay HDMI seems to work better yes maybe let's just that one works yeah just how difficult is the link okay then just send me the link to do you have telephone or okay that is your presentation okay problem solved let's continue with Matthew Kenney addressing accessibility for blind and visually impaired life coders I'll give the stage to him thank you so much for staying around and I'll be quick so I'll get back yes okay so yes this is our presentation on work that we've been doing around accessibility for blind and visually impaired life coders this project in particular has been with a group of blind and visually impaired students, learners so we are me I'm Matthew Kenney I'm a performer based out of New York I do music and visuals with a variety of different languages there with live code NYC and also my interests include development of live coding tools I've you know made some open source contributions to title cycles itself I've sort of built various versions of experiments with editors and collaborative systems and then I worked together with my friend Willie Payne who has just finished up his PhD at NYU where he was working with the NYU Ability Project which is a lab that deals with accessible technologies under the direction of Dr. Amy Hurst and most importantly over the last couple of years he's been working with the Philoman M. Dagestino Greenberg Music School which is a community music school in New York a pretty unique institution in that it's a school for blind and visually impaired musicians it pulls adults children kind of people from the community of all different ages levels of you know musical expertise and that's Willie here on the left getting some pizza with one of the students right before a recital and he worked with them for three years prior to this work kind of formed a relationship both with the institution and with these students and kind of helped do transcription of music into Braille helped lead various classes or teach one on one and he's been doing a kind of research with accessible music interfaces this was a previous kind of work that sort of precedes this called sound cells where it was a web based text editor because you know text is very easy to make accessible either through screen readers or Braille displays and essentially the editor it's pretty simple it uses ABC notation a sort of pre-existing music notation that's all ASCII based and a lot of music already exists in it and then this editor was designed as you can kind of see to be very customizable so that you could have different you know you can adapt the colors or the font sizes depending on exactly how much vision you have and then you could export either PDFs of a print score or a Braille music score and kind of based on that work he wanted to get into live coding because live coding as a text based tool for music making seems like an obvious kind of potential tool in the toolkit of you know blind and visually impaired musicians and so we set up Folork the Phil Laptop Orchestra right now it's a group of five all about high school sort of teenage music learners who have you know a lot of background in music I don't think any of them had any particular background in programming certainly not live code specifically and so with this we wanted to both figure out a curriculum and an environment for them to learn live coding in this case with title cycles as well as figure out how to make tools that work well with them and that they can interact with and you know the tools that they use kind of speaking to the diversity of different vision abilities are all different some use iPads with magnification in very large text others use MacBooks and the you know Apple voiceover software various of them use hardware braille displays which can you know refresh and generate the braille representation of a given line of text and so incorporating all of that we set up a learning environment built around a kind of in development experimental text editor that I've been building and and that I'll get to in a second but but basically we kind of structure this weekly class around initially sort of small prompts and lessons that would incorporate a couple of you know of like ideas from title in certain syntactic constructs or certain filters or different effects that you might use when manipulating samples and you know because because these are learners without you know with low or no vision ability of course we can't do some of the typical things you do in a classroom we can't project code for everyone to look at and so we really worked within the editor taking advantage of the fact that it was this collaborative document that everyone was in the same time and so we would start out with code that would have comments maybe describing okay this is a bit of code this is how it works or this is what it does and maybe offering a prompt to okay take this pattern and adjust it in this way or add you know add a sort of secondary effect on top of it and you know out of that a sort of process of developing you know developing the ensemble and developing the sort of things that they would perform kind of came up in addition we had some physical materials that we would give out and you know again trying to meet everyone's different needs we sort of designed these handouts so that at the same time they'll have braille representations of text they also have printed text again very large so that it's pretty readable also some of these particularly we've got a graphic here of various different you know common wave forms things like that that are typically expressed visually we used a variety of technologies that allow you to create tactile graphics so whether they're embossed or there's a technology called swell form where it's an ink that actually expands the paper when it's heated in an oven and so with that we kind of took that material did that process of you know running these classes kind of building up people's understanding and then at the same time working in this kind of in development text editor that I already sort of started and then have been continuing to develop alongside this course it's called text management which is both the name and URL and currently it just points to the GitHub there are a couple of pre-releases but that is going to be a sort of ongoing project to build a what will hopefully be a more general purpose live coding IDE it takes advantage of the fact that the recent release of the code mirror kind of JavaScript code editor has the best screen reader support so far which is to say it's still not perfect we've been going through the process of discovering which particular IDE features don't play nicely with the screen reader or on certain browsers or on certain platforms and trying to customize and adjust and like figure through problems as they come up and like I mentioned it has this collaborative structure and so here's the architecture that we used in these classes which will be very similar to Flock if you've ever used that basically there's an online centralized document and then each learner has their separate browser code editor and then the instructor was playing degenerated audio on a single kind of machine attached to speakers which was good because synchronizing code as opposed to synchronizing everyone's music simplifies the process of mixing all of these different musical streams together it also allows individuals to use headphones and listen to the voiceover of their particular code on their particular line that they're editing without it impacting the music in the classroom and you know here I can show you an example of a moment as they're sort of all adding things and kind of listening back and forth and there's a moment right here where it gets more and more chaotic and there was a sort of a rapport that built up and certain kind of norms around how do you do collaborative coding together that came both from the fact that all of these all of these learners have been at this school for a while they know each other and because of the particular way that they were working they figured out okay it's okay to silence other people if you ask them or perhaps you can silence everyone but only kind of in an emergency and it became this sort of balance of these relationships and how do you kind of add without going overboard in this case it turned out that someone had decided to I think create a pattern with a thousandth subdivision of a beat or something and it was just generating far too many events yeah and so our reflections on this and sort of issues that we're still working through is that of course coding is challenging to teach under any circumstances and we've had to figure out how to address confusion around syntax confusion around just all of the particulars of errors that will crop up and such in addition to the fact that there are many screen reader issues still browsers are not always consistent in how they you know kind of represent what's going on the screen and also there's a full universe of other collaborators or you know IDE pieces of feedback that you would expect you know syntax highlighting that sort of thing if you're looking at a visual presentation of an IDE but within the sort of one dimensional stream of you know audio description there are a lot of challenges to sort of design around but on the other hand there's a lot of potential here both for this population of musicians but also for the live coding community broadly like I mentioned text-based creative tools that are absolutely well suited to people who don't have much or any kind of vision ability and by building these tools or building these features increasingly into the tools that we use that kind of fulfills this desire of the live coding community to be a very open and inviting and accessible in sort of every sense community and this can then you know this brings new people, new musicians new developers, new feature requests and new ways of performing because in the same way that we want to make performing accessible to blind performers we also want to figure out okay then how do you represent what's going on to a blind audience member who can hear but wouldn't necessarily be able to see projected code are there other ways that you can be incorporating the material of live coding and then as you're sort of adding more complex or sort of more layers to the presentation these layers become things that can you know feed into the artistic work itself and I'll mention kind of going forward they this group had a their sort of first premiere concert was last December this is them rehearsing for that it was part of you know a larger kind of end of semester recital for the school and going forward from there we did a follow-up interview study talking to the students and then figuring out okay one semester on what worked what didn't about the curriculum about the tools about the way the classroom was set up and that fuller study has been accepted at 9 2023 and will be presented there and then this kind of this spring they have another performance scheduled for this Saturday actually they've got a further performance and they'll be visiting some of the research labs at New York University and then beyond that this summer we'll have more time to kind of pick up all the loose threads from the year and really start to identify and kind of sort through specific issues that came up you know the to design these sorts of things with the active input and involvement of the people who are actually using them but it's also particularly tricky to do that with you know with students who kind of don't necessarily yet know what it is that they need to be sort of designing so now that we've got this initial class of you know learners who will all be potentially you know can potentially continue on with the ensemble in future years now they have some expertise and can help sort of take a more active role in the kind of ongoing development and the school is very interested in expanding this ensemble we're trying to figure out how to make sure that there are more people who understand live coding understand how this works just build up the kind of momentum to keep things going and yeah I'll wrap up by saying thank you to the FMDG school all of our research assistants at NYU in particular yeah there was Shinran Shen and Eric Chu and also thank you so much to the five members of our ensemble for helping us figure this brand new thing out and thank you for joining okay thank you so much Matthew is there any more questions there's no more well we have a little bit of time after this so maybe two or three questions if anybody has any can be for that there it's fine what text to speech has that been something you've investigated and what limitations do you see within that also considering some of the recent developments that we've seen with things like Google co-pilot yes so text to speech is built in sort of natively a part of things like voice over things like JAWS is the popular window screen reader so it's I mean to some extent because like the accessible tools that people are using all the time we're not really dealing with things at the text to speech level much because you know they like a blind computer user gets really good at the exact you know sort of voice of voice over and figuring out how to pick up even like a very fast reading of a bunch of text on a screen how to sort of move forward you know that said there is the sort of future design that needs to be done around this is kind of how to augment just the basic reading of text with okay are there maybe audio cues or are there certain kind of things like that and with that I think there's yeah potential to sort of play perhaps with text to speech engines at a lower level to find ways that subtleties of voice could indicate that code is being actively edited or something you know things like that okay one more question over there oh yes how does a blind person read music I mean you know again it depends on exactly how much vision they have there are lots of musicians who will use large print scores which you know both are sort of physically larger but also might be more simplistic and there is also a full system of braille music notation that has been you know basically all of the relevant sort of pieces of music notation have been ported to use the the tech term over to a sort of braille representation and you know it's the sort of thing where they're going to be as many solutions or as many sort of variants of solutions as there are blind musicians and so you know anything you want to what you always want to think about when designing in this space is building things that are very customizable that are very easy for people to kind of own and use on their own one last question maybe yes so can the blind people I mean can monitor what they have written already so can they can they monitor the prescriptive part of the notation, the notation that actually does things yes you mean in terms of like the actual code so if you type like a line in title yes exactly can they trust this with the paper that you mentioned this technology the shape changing paper yes yes I mean you know the paper handouts are of course entirely instructional but they're using either they're using like a braille display which essentially has like little actuated dots so that it can you know you place your cursor in a line and you know just like a monitor it generates you know the dots rise up to show you that line of title and there'll be like a blinking you know a couple of dots that sort of jump up and down as a sort of indication of the cursor and also you know if you're using voiceover you'll you know enter in and it'll say okay you've entered a line of text the line says D1 dollar sign you know et cetera et cetera so you yeah so I mean there is this distinction between prescriptive notation the notation that does things descriptive so the cursor would be there that descriptive somewhere there which doesn't do anything really if right or the highlighting for for example in strudel the event highlighting this descriptive notation so yes there are any technologies for descriptive notation oh right descriptive versus prescriptive at this I mean at this point it's yeah getting the prescriptive notation is a lot of challenge on its own but certainly exactly those extra you know layers of descriptive understanding is you know in the same way that that IDEs for sighted people are still trying to kind of figure out how to implement those things then you know that's work for us to do as well okay thank you so much another great applause for Matthew applause and all our other amazing presenters of the day don't run away just yet because I think we have a short announcement unless we accrued too much delay sorry we apologize for all the technical problems we had is there still an announcement happening? dian verdank so dian verdank if I pronounce it correctly okay then there's no announcement about nine I'm sorry about that so this concludes the academic part of the day we'll have some time now and then I hope I'll see you all at a concert there's a concert tonight or am I completely off today no okay let's call it a day see you around bye


This is a video to show you how to terminate a screw terminal telephone socket. First remove the termination plate, and expose the termination screws. Insert the cable into the back box. Then strip back the outer PVC sleeve. Cutting off the internal drawer wire. Separate out the two blue cores. Strip off the plastic coating to expose the copper wires. Bend the copper wires to fit the screw terminals. Placing the cores securely under the screw terminals. Repeat the procedure for the second terminal. Once again securely tighten. Replace the termination plate and secure. Finally, fit the telephone socket faceplate and secure with screws. Further assistance or a free survey, please email or call on the details below, and click like and subscribe if you have enjoyed this video.


So, imagine, for example, a scholar in French literature who runs a blog and tweets where they talk about their interest. They're working on teen-century French fiction. And so they read a book, a scholar in the book, and they review it as a blog post. They come up with a beautiful quote from Malvernet, and they tweet it and so on. So they build a reputation through social media for their current interest in the project. And this eventually helps grow into their book. So they may publish an outline, they may publish a book proposal on their blog, and they'll get feedback. They'll be able to kind of think out loud about this. So over time, they've really built a kind of, if you will, a nimbus of social around their scholarly impetus. And then when they publish, say, a journal article, it may be that it appears behind closed doors through Elsevier and Jastore, it may be that it appears an open-access journal issue. Either way, readers will approach through different venues when they use it, or different paths. And maybe they make a monograph. In the monograph, they could have, say, film footage of an early dramatization of a play from the 1920s. It could be that they have an animation of Paris in the 1880s, a three-dimensional animation that they want to work through. And this will be part of the monograph that you could read online. So I mean, overall, you'll have something new. Each of these has a special promise. The material promise is that we can do more scholarship, and we can ask and answer more questions. So in the case, the hypothetical case I just mentioned, imagine a 19th century Paris visualization, a three-dimensional model where we could query the model and pull up data about settlement, about where people live, about sewage, about the roll out of streets over time. And then the promise, and maybe learn more than when you once knew, and the promise of the communication, well, in part is greater efficiency, but also greater conversation through open access, a larger audience, but also if they're connecting it to social media, a wider audience as well. So overall, taking together, in a sense, this is revolutionary. We're really taking a scar on the publication and transforming it into something new. Looked at another way, this is what the digital world is like. This is something that's not really that remarkable. Nothing that I've described is futuristic. It's simply the case of just implementing and doing it.