Hello, everyone, and welcome to the Active Inference Lab. This is Acton Flab Livestream, number 27.2 on August 31st, 2021. We are a participatory online lab that is communicating, learning, and practicing applied Active Inference. You can find us at the links here. This is a recorded and an archived Livestream, so please provide us with feedback so that we can improve our work. All backgrounds and perspectives are welcome here, and we'll follow video etiquette for Livestreams and see who joins us. The short link is the calendar of Livestreams of the past and future, and we did choose 28 paper, but didn't update this slide, so check the link for next week's paper. Today in 27.2, welcome to Steven's greetings. In 27.2, we're joined by the two Stevens, as it were, and we'll have a few questions that we can have slides for, the figures to go through, the paper itself, anything else that anyone wants to add, and then of course, any questions that people type in the live chat as well if they're watching live. So we can just say hello and then we can leave it to the author last just to maybe open it up or just bring what they were thinking about our second discussion here today. So I'm Daniel and I'm a postdoc in California. I'll pass to Steven Salette. Oh, hello there. I'm Steven. I'm based in Toronto and I'm working on social topographies and other spatial meaning making and hoping to find ways for active influence to shed light and all of that. And I'll pass it to the other Steven. Is that OK? Yeah, good day to you. I'm I'm in Finland, Southern Finland. Based at the Technical Research Centre of Finland, and I'm ready to answer questions. Cool. So again, anyone who is watching can post a question, but then there's a few that we have lined up a little bit. Just one theme that. Stood out as far as selecting this paper was that it was applied. It was applied to making the kinds of objects and supporting the ability to have the Internet and computers as well as just objects like pens and things like that. So how does this idea of helping us design physical systems, which ends up being our niche modification activity as an economy? How do we help design maybe if we're not an industrial engineer, but potentially people working on different kinds of projects that end up modifying the environment? Maybe even influencing the same systems that industrial engineers use or using some of the same techniques, but people who are just thinking along these lines and want to take what you translated one step towards application in your paper and then think about how it applies in their own work. So cyber physical systems are the latest trend in this is, for example, the in in the U.S. I think it's called smart manufacturing and in Europe industry 4.0 and a big thing in this is the idea of having digital twins or digital siblings or even avatars. And the idea of this is that. There's a physical. Machine. The physical process has got sensors on it, actuators on it. And the sensors bring live data to the digital equivalent of this physical process and through so and then it seeks to optimize by seeing what's going on. So and a basic reason basic level is although difficult to achieve is predictive maintenance. For example, so if nothing else, at least to stop the physical process from malfunctioning or even worse, stopping. And there's no. Um. There's no. Theoretical. Foundation for. For these industrial. Innovations for things like digital twins. Of course, there's a theoretical background of cyber physical systems, but there's nothing to. Anchor it with, in my opinion, that's very straightforward. So if we look at life sciences, generally. We know that evolution has had millennia of millions of years to iterate. To develop elegant solutions. So it seems sensible, at least to me, to look to the life sciences. To see how nature's evolved towards least action. And of course, a free energy principle and active inference framework or theory. From the life sciences and. In the essence, I believe very straightforward, as one would expect guiding principles in nature to be and applicable to human and artificial intelligent to artificial agents. So I believe it's useful to start from that basis. And with the free energy principle, this very simple, this very straightforward idea of. Of trying to minimize the information gap between a generative model. And the world itself. So that's a very straightforward idea. And I know there's the mass behind it is evolving all the time. And that some debate and courteous dispute about that. But the basic idea seems straightforward. So I believe it's therefore a good place to start. And because this is such a basic idea, isn't it? Minimizing the information gap between the generative model and the world itself. Is this what I've said helpful in any way? Thanks for this. I wrote a lot of it down. Stephen Slate. Yeah, some really good points there. One thing I'd be interesting to know what your thoughts are is, you know, we can end up kind of. Adapting our niche through action as a kind of consequence of. Like following a path and these types of things. And then we've got this kind of more constructed. Sort of deliberative. Organizational work in which, you know, in many ways, the point of that work can be to construct the environment in which tools will operate. You know, so in some ways we use the tools of our legs or the tools of being in in the world and it can sculpt that niche. And also we can be very much sculpting a bounded niche. So I wonder why question is thinking about whether you see those that kind of distinction as being significant and. How you see the idea of like the free energy or the implications of the free energy sort of rolling out in the niche versus the generative model for that niche being embodied in some way. So, you know, so anyway, so how much is there something when it's embodied? For instance, when you're talking about these total quality management environments where someone is. Trying to maintain, I don't know, ISO standards of manufacturing or whatever. So I wonder what your thoughts are about. That whether there's distinctions there in terms of active inference being particularly involved or, you know, the free energy principle more more generally. So what so I believe that that that the core of quality management. Is just an instantiation of. Free of realizing the free energy principle through active inference. That's why, you know, putting the paper. But there's a bigger issue about. That in nature, things are evolving and. As I put another one of my papers in entropy that. That. So. There are no. Things are evolving to some degree naturally in. So if you take predator and prey, they just it's evolved. And if you're a beaver trying to get back to the. Lodge that you've built your niche construction as a beaver. And there's a links in if you're if you see in between. Where you are in your beaver lodge and there's a links. A hungry link standing in your way. Then it's clear that okay, you know, this is a predator and I should. This is not my as a as a beaver. That's not my preferred observation. Whereas my beaver lodge is a preferred observation. But what goes on in human needs construction and. Because he's the man. Environmental engineering. Is not so straightforward. So for example. Highly refined food. It seems to be a naturally to us like a preferred observation because it tastes good and. So it makes life less strenuous and more stimulating. So if the quite if the if the if the company making. Hardly processed food, ready meals, so on it's it's acting within quality assurance so it's doing in my opinion active inference to realize the free energy principle, but what it's output is it's subvert. The free energy principle. The externality it subverts the free energy principle because. What it's offering to us as consumers, it seems to be consistent with what revolved to do so it makes life. Less strenuous and more more stimulating, but actually. It's not good for our health in the long term, of course. Now and then no problem, but if that's all you if you live in a food desert, if you live in a food desert, we know there are lots of studies that show that. Unfortunately, this isn't good for health. So if you take. One. Entity at a time. It can be conforming with the three energy principle through active inference, but it's output subverts the free energy principle, because it makes our preferred our preferred observations. Instead of being good for us are not good for us, but we only discover that 20, 30 years later, so that's quite a lot to I have a diagram in a simple diagram of this in papers called yeah, it's called multi scale free energy analysis human ecosystem engineering. And there's one diagram in it. Figure one. So it's an entropy as well. Yeah. I share my screen. Sure. Now remind me there it is. Okay. All right. Are you seeing the multi scale free energy analysis of human ecosystem engineering. Yep. All right. So here's this diagram then there it is. So what what these what active inference is leading us to is that from nature is our preferred observation support survival and unwanted surprise. So and unexpected difference between our prediction of what we're going to experience based on our internal generative multi scale free energy analysis. from nature is that our preferred observation supports survival an unwanted surprise. So an unexpected difference between a prediction of what we're going to experience based on our internal generative model. So an unwanted surprise. So if the beaver's going back to its lodge, it's thinking it's going to have an unhindered path back to its lodge and there's the unwanted surprise of a hungry lynx standing in the way and it threatens survival. So that's natural free energy principle but through human environmental engineering niche construction within that preferred observations can threaten survival. So if my preferred observation is a comfy sofa, a large TV screen and a device to change channels and do gaming and then I can just order some food in it comes and very tasty but perhaps not good for me. I could be sitting down for 16 hours and then go lie down for the remaining eight hours and maybe all my preferred observations but they're not good for me. Whereas the unwanted surprise that supports survival could be messages that okay we should not sit down for more than 20 minutes, we should take lots of physical activity, we should eat, make our own food and so on. So what's going on in human endeavours which may individually conform to realisation the free energy principle through active inference can end up subverting the free energy principle outside of that production activity. And here there's okay so so beaver could develop a new generative. All right so in nature for a beaver the preference distribution for sensory inputs is for example plus 10 for the protective pond, the large zero for woodland and minus 10 for a hungry predator but then if there's a bit of pollution you change like okay so now this is getting more complicated so there's plus 10 productive ponding clear water, plus 2 productive on polluted water, woodland zero, hungry predator still minus 10 but then or the example in this paper is in some island nations, very small island nations where there's a lot of important cheap calorie rich nutrient poor food and then the preference distribution could be plus 10 imported food, plus 2 local food, minus 10 no food even though there's clear evidence that the imported food is not good. Okay so there it is. Okay so did that um did that lit up the hands? I wrote down some stuff as well but blue welcome um go for it and then afterwards will be Steven Sillett. Thanks so I was really curious about that subverted um model and like where you're maybe not optimizing like where where what your preference is is not optimizing for survival and I think that um you know you used it in the human context but I can think of many contexts where animals do a similar thing like a moth will fly into a flame and like a cat like I have these two kittens now in my house and they're like such trouble right like their curiosity like had my cat trapped in a box earlier today like I mean trapped like under furniture in the box which was like so bizarre so I mean I can think of of models where it's always not optimal and so is this something that you frame as uniquely human because I can think of many instances of like non-human creatures that also have less than optimal preferences um no I'm not framing it as only human but what I'm pointing out is is that so for example the moth flies towards the flame but where did the flame come from and the cat's curiosity luckily did not kill your cats that's good um but they were going into boxes but where did the boxes come from so are these things um of course there are misjudgments so all living things can um they can be uh and of course with predators and prey that for sure the predators are trying to get the prey aren't they and people can have um and living things can misjudge the how far away the prey is and so on and signaling systems can can get messed up and things but with human beings the scale and the determination of um activities is is much larger so I don't there's a there's an American author who's his latest book and it's called Hooked um that uh he studied the the for example the food industry for many years and um he his view is that there's nothing at all malicious going on but it's just if you're if you're in the in the uh a certain business you want you're trying to get as many customers as possible and you're trying to retain them and um with no no bad intentions at all it's all good intentions but then the effects uh we discover it can be decades later that okay um this is what we thought was good is not so good whereas in nature if the the wherever the flame suppose it was a flame from somewhere in nature it was a natural fire well the the natural fire maybe came from bolt of lightning or something it wasn't a determined effort over many decades of ever increasing refinement of a market offering but uh so there's more so what I'm showing in the paper that you read that diagram about how the quality management system is like a general it's like an active inference system to reduce free free energy so it's optimum for the organization but it can be suboptimal outside the organization so I think a general word to them for this is externalities just uh two quick things one was trying to explore that distinction between what is human other than the fact that we care about us and each other what is human and I was just thinking of the ants foraging out there in the desert and they don't do margin trading or um staking of their seeds to try to get fractional seeds and lending of seeds it's like layer one seed gathering versus the higher order symbolic risks that the humans engage in so that it and the niche modification in the scale among other factors and then also what you said about misjudgment and the the ability to misjudge because a lot of people think oh well you know if um let's just say traditional understandings of evolution are about fitness then why why do organisms fail why do all organisms not just maximize their fitness or if active inference is about reduction of uncertainty or using action and inference to reduce uncertainty then why do organisms fail and that's where the engineering perspective comes into play like things just do fall apart and just because there is an imperative whether to maximize fitness or to minimize uncertainty the point is of the ensemble there's going to be a distribution of performance like a hundred engines they're going to have some range of failure times the ld50 for some pharmacological experiment so just because the framework has sort of a um imperative whether fitness or uncertainty reduction doesn't mean that it's utopian or like pangolossian it means that every single thing always works perfectly so just cool points that you raised but but also blue um my my uh i have imperfect knowledge that's for sure and i wouldn't make um i i i i i don't know enough but maybe nobody does um like what was just said about ants of course those words were coming for an expert and i don't i know next to nothing about that and many other behavior of many of all animals i don't know so for sure it's not it's not clear cut it's not one that humans are doing this and all other animals aren't doing it thanks steven select and then of course anyone who's watching live i think this raises some useful areas where we have spaces where it's very free energy minimizing as the main kind of way of getting a handle on things and the broader active inference so for instance when we're dealing with equilibrium chemistry we're dealing with things which are often boiled or heated or made to be effectively dead um which is what happens in most engineering processes in most food processes i mean the main thing that coca cola does by putting sugar in is it makes a preservative some of them are saying to me oh why can't they just use water it's an awful lot easier to to distribute coca cola than to distribute bottled water because coca cola doesn't go off because it's got so much acid and sugar in it it won't go off right so that what you've got is you've got these processes like you're talking about with tqm and i worked in the pigment manufacturing industry so we they were bringing in the iso standards there around pigment manufacturing because they're inheriting a lot of these formulas from like a hundred years previous and it was all about minimizing and to make things reproducible and within tolerance but you're always dealing with you never want things to be organic you don't want things to sort of to grow in any place in the in the process you want it to all be i suppose you could say dead right but then what's really interesting as you point out is okay well that might be what happens within the containment of the factory or the industrial process but it will go back into the world just like it will when it's out now as a product it will be used in an ecological niche and it will not be a dead niche you know that food that maybe isn't even biologically feasible or some of those chemicals are not there's no biological route to create them there's no way to create fluoride compounds biologically it just can't be done it has to only be done synthetically so what's your thoughts about that sticking with minimizing free energy more when you've only got dead equilibrium systems and then when it and what and how much active inference is involved in in that i suppose a first order is it only at a second or third order and then what how things pop out into the niche in the community or whatever um i think this is um open question i mean at one level maybe it's um in world systems theory this the the framing is that there are so now it's going about whole countries rather than just a company making something in one factory that but there are inevitably core i think the terminology is core countries that so it was the Netherlands then Britain then the USA now going towards China and then there are peripheral countries two levels of those and um the the argument is that the development of the core countries is dependent upon taking from the the peripheral countries and um is this entropy being shipped out of the uh so if if you think of like a a factory as a heat pump and it's uh you're getting all the entropy out of the factory operations but then it's where's it going is it being shipped out of the factory and then it is entropy being shipped out of the pumped out of the core countries into the rest of the world where things are confusing so i don't know i mean that's a bit of a a broad sweeping statement isn't it so but maybe maybe on one level that's what's going on if there's if you if you have supply chains value chains and you push all the physical disorder out of those by eliminating all the information uncertainty then does that mean that the physical disorder has inevitably got to go somewhere else i don't i don't know so uh but maybe and then but uh with but for sure there are externalities and the the interaction between levels is uh because in in human endeavors it's a it's a shifting matrix it's not just a simple hierarchy so i don't know can i just ask one thing just bouncing back off that i think it's an interesting in some ways by why things sort of being taken out of the ecological niche effectively by putting in a process a processing plant where it's effectively dead you know you heat up your sugar your sugarcane raw material and you know that with sulfuric acid and it you know it's all it's all dead it comes it's basically a chemical process that gets put into whatever foods that someone's eating um the consequence of having that dead equilibrium based process rather than non-equilibrium steady state process is that you're saying it puts a burden back on the ecosystem to try and reintegrate it in a way so in some ways is it pushing entropy out or is it just placing a burden on the system to try and find a way to cope with something that it can't naturally incorporate easily into its niche what do you think blue yeah i'm not sure really i'm not um i'm not uh entirely certain on this on this um transfer of disorder it's definitely very important like for example when people say well look at the miles per gallon of this car and then okay but where did the metal come from okay but you know everything is tied to everything and how we evaluate policies ultimately and choices are critical and we do have to understand the relationships of the world's supply chains but then i thought well it's not like there's a like we're in a bomb calorimeter or a heat bath where there's a constant um number or proportion of some kind of uh negative activity like there could be an ant colony with of in a niche with very low nesme aggression or bodies like there's organisms with low rates of cancer or in certain niches you know it's not like a statement of them across all patterns and times but um but then again it was like so things do fail and there's no perfect state but then at the same time um it's not zero sum in the informational game it's not like we're trying to because we're getting energy coming in so can we kind of transmute physical order with information disorder do we have to like offload something to another place physically does it bump up against any of the information limits or calculations or density of processing or storage will we not be able to remember enough and then there's a catastrophic failure i mean there's so many components to your question so i i hope that we can all keep thinking about that because it's important at the level of like human activity with a supply chain and it's also an important long-range question and if we don't have a long uh range preference then we're not going to get there could i have one more bit in there without yeah um i think what's interesting as well is there's this question of because i don't really believe there is strictly speaking neg entropy as such so in terms of um it's always kind of a relative thing but it's useful it can be useful to think about it that way sometimes but ultimately it's always a change so i suppose the question comes in in terms of how much is it around the ability to infer using entropy as much as it is about how you know how much is it to be able to use changing entropy or dynamics of entropy which can be a complex jungle niche you know it could be a lot of entropy changing a lot of stuff going on but it can be very healthy how much is that ability to dynamically get at the variational free energy really important for life whereas for us it's all about at times in an engineering context is it's just reducing free energy per se as a kind of an overall entropy term which could be actually not so much about variational change it's just an overall general reduction in entropy i wonder what what your thoughts are on that one thought on that on the biological side is what's an industrial engineering relationship ecosystem engineering relationship that has persisted for billions of years or millions of years or just long time however you want to see it the nitrogen cycle and some of the redox cycles in the bacterial relationships biofilms the mitochondria and the eukaryotes now that's a shifting interface but on the other hand eukaryotes can totally count on that interface the powerhouse of the cell as they say to persist but then there's this other sense in which there is still a game theory in a failure mode and probably an interface of maximal uncertainty and the requirement for active sensing and trade-off costs so it's again it's not like things i think work 99% of time or 100% or 20% of the time that is kind of how the chess board is set up for the for the um for it to even be a winning board at all there's just niches where it can't work like there's temperatures that just the population will just die at period so is our engineering condition such that there is a path that's going to lead to what standard of life with what precision and uncertainty for what distribution of people like did we way way overshoot and now it's going to change back down to a different way or is there some critical path to changing the distribution of industry because it's not going to be any way it was from zero to 2021 it's not going to be that way next year um are you do are you still staying my screen or i'm not could you re-share it okay all right this is a let me do that again this is a are you seeing now figure one meta ecosystems in the Anthropocene now see it yep okay yes this is a paper i'll go to the title um so also nothing but this is um uh not by me by others nothing to do with me i'm just a reader of it um and one thing that they're saying that you can just so here this this is the same these are showing the same things but a is where human impact on migration based meta ecosystem it's it's just sort of implicit and on this framing of it they say okay so we explicitly deal with the uh social ecological framework about okay so what are human beings what mental models of human beings got what regulations arise from that and then that inputs into it so let me go to the tops you see the title yeah embedding meta on embedding meta ecosystems into a social ecological framework and this is a response this is just a short paper it's a response to somebody who replied to that no it isn't actually no i'm mixing it up with another one there it is that's the title anyway so i'm wondering if um that may be that each each way of um making things has got its own entropy dynamics and that in this entropy and dynamics that there's physical disorder is reduced uh information insert is reduced but maybe that then pushes the entropy out elsewhere and then just doing more of the same thing it's just not possible there's a limit to it and with world um systems theory uh this is the biggest gist of it that there are these core countries and there are peripheral countries that serve them and maybe that if there's one way of doing things you can't address the entropy that's caused by that by trying to do more of the same or trying to tweak that way of doing things and maybe something quite different has to be done and maybe there's something about something to be learned from landscape restoration ecology in this about that about um habitat fragmentation that that uh so okay so here are these habitats and they've grown and they're thriving but these ones over here they fragmented they split and this there's nothing from them but dispersal so as people are dispersing from um parts of the world that are suffering from premature deindustrialization that maybe there's something in uh entropy dynamics that could be that well this doing this type of industrial production this it drives out the entropy but it's shipping it out it's going out elsewhere and the entropy that is then um in other places it can't be dealt with other than by some in another way of existing does that sound in any way sensible a very interesting area multiple levels of nested modeling and uncertainty and how informational and thermodynamic uh aspects transmute into one another and arbitrage with one another or relate to each other so it's really interesting area and uh one kind of thought on that and again to the fast food and things that might be unhealthy for us or policies that might be unhealthy in the long term for an organization it it made me think about being on a road trip and knowing that there was going to be not just the hyper stimulating experience which has been argued in many other frameworks like in reward based frameworks which is what you know dopamine the reward molecule reinforcement learning reward learning they'll already have a framing for how those products hyper stimulating lead to failure modes of complex societies via hyper stimulation okay so that is sort of a common stance with active inference we can also kind of highlight the fact that those products have lower variability and that there's actually very bland foods that for reason of blandness that or like a tv show or something that for reasons of narrative or aesthetic just similarity people will rewatch i mean even to videos that of types of people didn't expect why do people watch the log burning or the goop playing because they're sort of like the frontier of optimal foraging is about like how it's going to sound which is a very small thing to be resolving relative to the more um deeper unsettling levels of information that could be revealed like someone else's perspective of you or something like that so it's just interesting how we can actually as you point out like the supply chains are modeled on one hand by these cyber physical digital twin frameworks that need a unifying approach and so on one hand active inference does provide that but then looking at the other side of the equation so what you didn't discuss in our paper for these weeks but where you left off when you start introducing the biological and the survival imperatives then we can also have some new ways of thinking about how those industrial products through precision hacking rather than reward hacking end up leading to maladaptive behavior with or without reward incentivization are you um what are you seeing on are you seeing my screen now a different yep different diagrams you see birds yeah generalist bird gap specialist butterfly interior habitat but specialist beetle so this is another paper um i'll again i'll scroll up to the top she says tile um and it's making this point that landscape composition and configuration got a different species perceptions and so the first one's a human and then it's different species and as it's neatly illustrated there that the same the same place looks different to different species and so what i'm doing and the the kind of the terminology here is that there are patches habitat patches and they can be corridors between them or stepping stones between them and the stuff the other stuff is called matrix and i personally can't ever see that word or say it without thinking about the movie so that is a bit of a problem but uh but in the landscape ecology it's called the matrix so this i think this could be useful of illustrating that well okay that so here's the road to the mine these uh from country companies built the road to the mine they built the mine and they're taking the stuff out and they're taking it back to their country to um do highly efficient manufacturing with it and then they send the completed goods back and but then what's left um so if you just measure that in terms of GDP growth then the GDP goes up divided by the number of people and it appears that prosperity is rising but if if you look at it in this kind of view it shows that well there's a lot of blank spaces left to be left here and uh if we think about it in terms of this restoration or regeneration it it's looking at it all these because of course the perspective of the human the generalist bird the gap specialist butterfly and the the beetle they're all equally valid and um so let's see what this paper's called landscape ecology and restoration processes and even within the same physical overlap one bug might be underground or in the roots or on the ground in the canopy so even the physical micro niche that different strategies or individuals experience in the same spot in the matrix can be very different so there's just such richness to this ecological thinking and it's just challenging to bring it into play in some of our cognitive endeavors but i think that is also changing rapidly so i've taken um an ecological perspective in we're doing one paper on meme ecosystems and rhetoric ecosystems by the end of this year but it's not active direct but maybe it'll have some relationship it could well do um are you now seeing a table table one all right so what i've done here is i've done this from a uh ecological perspective and uh well you can see these terms but one thing in a restoration ecosystem restoration is mobile linking organisms that uh that like birds and animals that can travel quite a long way they can go so after this is a forest fire then animals that can travel quite a long way particularly birds they can come they can fly and drop seeds and and things like that so this is an established concept mobile linking organisms and i'm pointing out that this is something that could be done in mobile factories and then energetics another key issue is that if there's no if all the energy if all the matter and energy are flowing out then life isn't sustainable and uh of course there we are so i've done that relating to my favorite subject movable production technologies okay so um so that's like i think this is decentralized systems isn't it that uh on the second point that that was it that was the lead-in the segue okay nice how about oh go for it um Stephen yeah so i think this is the thing that if everything's so at the moment there's this centralization oh let's have a look at this i've got um i go again so let's have a look here are you what do you see now do you see a taxonomy uh taxonomy and manufacturing distributions and their comparative relations to sustainability right so here they are so there are distributed distributions and centralized um and as we see there's plenty of distributed options and uh then i compare them in terms of economic ecological social and institutional sustainability and where i think uh active inference i've just started to look into this um that this agent environment systems and uh at framing it as the the agent has a generative model that the world is a generative process the sensory inputs are coming from the generative process of the world and um i think that this enables um active influence to be linked to uh ecosystem dynamics and that that what we've been well what i've been saying before about this centralized distribution it's um it's super efficient and it's getting uh engineering out physical disorder and information uncertainty but then maybe that entropy is going somewhere else and uh maybe new information is needed to activate the more distributed options so do you think do you think that um that if if the only if only a certain um so if there if there is a predominant paradigm which in this case there is industrial manufacturing based on the 300 years old paradigm um that if the information about how to make things is all about that and how to improve that and then then does it become a zero some game that if if so if it is shipping out um entropy like a heat pump out to the peripheral nations do you think it's a zero some game if there is an alternative information so if nobody knows or there hasn't yet been uh conceived an alternative then then is it like a closed system so in a closed system the entropy has it's if it's pushed out of one place it goes to the other places so is it is a is it a closed system is it a closed system if all the information is about one paradigm so is this like for example does am I in any way coherent here it's an extremely important question and um you know socio-technical systems political systems tokenomics that last forever are these the the perpetual motion machines of the 2020s and just a new level of uh computational detail and flourish but the same kinds of fallacies and then um whatever the thermodynamics of the earth are our niche is finite and we've now made some potentially um whether or not the sum is zero or even negative or could be positive but is negative there may be informational and strategic games that um don't resolve like heat baths so they can't be understood with some of the at least most tangible physical metaphors that thankfully guide the evolution of theories that are first principle like FEP like Carl Friston with the woodlice seeing the trajectory of a real thing and that's based upon health and um longevity live yeah I had a question about the table that you pulled up earlier um specifically about ecosystem engineering and so I'm not sure if it's just like a term that I'm not familiar with or like in in terms of engineering I'm kind of naive so when you're thinking about ecosystem engineering um I I equate that to niche modification or like designing a system like even you know you could design like a world system so here it says it does not require overriding the natural evolutionary balance of equal equal ecological fitness but like I think about like the human paradigm could I just could I just have interrupted but that's movable production technologies so what I'm saying there is that if you've got a movable factory so a truck if you've got a second hand truck with a second hand container on the back of it it does not require overriding the naturally evolutionary balance so it's it's specifically in relation to if you look at the top it says movable production technologies so so how how is that ecosystem engineering though like I know it isn't yeah but the point that obviously this table is not clear and it is my responsibility it's not clear I am the responsible for this but so the what is meant by this is because further up in the paper it's pointing out that if you have fixed production technologies so if you've got a big fixed factory you build a load of roads huge roads going to it and so on that involves a great deal of ecosystem engineering but a mobile factory it can just go over rough terrain also to play devil's advocate against the author the enabling function of a realization of something something physical that can be an ecosystem engineering agent aka niche modifying agent that it's going to have to be a niche modifying agent so this is just sort of then connecting that to mobile organisms so you have quote plants aka factories that are planted and then those can be like they're not like trees simply but they're planted and so they have strategies not dissimilar from other rooted organisms but then you have mobile organic linking organisms so it sort of connects that all the things that mobile organisms like all animals do and can do with what a factory could do ecologically blue so it just kind of makes me think about like what kind of biological organisms take their niche with them like maybe a snail or a turtle like you know to take a large component of like their habitat essentially with them wherever they go the mobile technology made me think of that yep some people like to travel with light you know just the keys or something other people full backpack every time Stephen so that I like this idea of something coming out of the factory going into the space and then things being second hand especially the third fourth fifth hand because there is something there that is delivered is this I mean if you're going to use entropy I suppose you could say low entropy kind of new vehicle you know whatever it is and slowly it gets incorporated into the ecosystem it starts to get scratches it starts to get this and you see in India you know people start decorating the vehicle you know the all this sort of stuff will happen over time as it becomes somehow incorporated into the niche I suppose you could argue I think that's quite quite interesting how that maybe starts to change the dynamic I'm still not sure whether entropy is so useful term or whether it's some other way of looking at a metric maybe something to do with free energy minimization as a as a thing but that's yeah one other piece of that table that was interesting was there was multiple goals like sustainability and fitness so it is not a univariate optimization we're not just doing reward or utility maximization we can choose multiple goals and we can have multiple human goals which is especially important for things that you know we care about but then this one of mutualism is interesting because a lot of times mutualism could be seen as like an outcome like the outcome of these two bacteria's relationship in the niche is a commensalism or mutualism or symbiosis whatever it happens to be but actually using this idea of the the outputs like the observation states of a higher level Bayesian model being like the priors on a lower level nested model having a goal of mutualism is implicit in the lower level model enacting that mutualism because you're not accidentally going to metabolically coordinate through millions of generations and so this is kind of an interesting way where like we have to want to make it work because we're not going to accidentally make it work and get along but that can be a hyperpryor like a belief about how things ought to be or how we want or prefer things to be and then we enact policy along those lines by the way I do it's a by the way in um in another paper I do point out there are limitations to movable factories so I'm not uh saying that this is a panacea but it's sort of like like let's just say um cactus versus like a desert animal you know the desert animal they have different ways of getting water and so but when we can design the system we can do trade-offs like you could have something that's stationary but then isn't just holding all of its own money because its money can be like on a digital system so it can have some of the advantages of a stationary organism from an ecological perspective like a tree but then like the value of the tree um could be protected in some other way because we have some other affordance and or that forest could be de-risked at a higher level because it's like oh each one of these is we expect to fail this many percent of the time or we have these quality control checks on it yeah I think a lot of thought is required about about these things isn't it it's I don't think there's one big answer but um but at least I think one good thing about the energy principle active imprint so it ties into action and that ties into energy so this is good yep when you're doing the thought experiment pure uh g'denkin then it's like can be elucidating you know imagine you're going towards the speed of light or imagine you have a physical mass of this size but then there's no constraint which is again double edged sword on some of these attributes that require action and um so that's that's interesting idea that it ties us to um active states which is pragmatic and useful but also interesting and then also one thing on the affective system and then blue you can ask anything was just we've talked also about higher order variables in organisms being kind of like affective states whether they were the valence like positive or negative or the anxiety or the relative beliefs of an organism um so just how do we design for when there's higher order states maybe even ones that could be considered affective either for people or for organisms or I mean organizations that have this kind of organismal structure but blue so interestingly like this is going to touch on what you just said but also going back to what you said about mutualism and the goal like has to be mutualism so so I don't know um I really have to contact the author of the 2020 paper Sims is his name or how to count biological minds symbiosis the fvp and reciprocal multi-scale integration which I have found very fascinating but I would love to invite him on first stream someday uh the the whole point of that is that there's like nested cognition happening at the biological scale and specifically that paper looked at the squid and the um vibrio fissurei that like light up the squid in the in the light organ and um you know in that kind of context as in an industrial organization there's like this nested cognition happening um and where the the point of that was the goal wasn't necessarily mutualism or in that case symbiosis that that's not necessarily the goal but but a bigger grander organism was constructed out of the union between these two organisms and so perhaps in an industrial scale that's also something that happens right I mean there's different parts to the organization that'll keep it running and you factor in all the supply chain things and and so there's this nested hierarchical thing that happens and something bigger that has its own kind of cognition occurs as a result of this like mutual beneficial relationship we see the paper the paper we see the paper that Steven just put up yeah there's some um these um I thought it said in here so I thought it said in here somewhere about yeah these different types of relationships so mutualism is beneficial beneficial um and parasitism is beneficial harmful but in between those there's beneficial and neutral and there are several in between mutual and parasitism there are several other states and I just remember that the one of them is beneficial and neutral if I could give a kind of random thought on that and why the informational sphere might have a different entropy dynamic than the physical like the electromagnetic spectrum um that can have a lot of positive and neutral relationships like if someone's operating on two different wavelengths and one is only on one wavelength then they can be like a mutual service to each other or one could just be totally non-interacting because they can be overlapping in space and time at once and one person might not even have to be aware of it whereas in the tangible um I guess most embodied one can be you know only one thing per spot that's zero sum there just can't be two two atoms in the same spot or maybe you know two laws playing out in one territory but informationally there's probably a lot more multi-channeling and that might give enough room for practical success if not ultimate theoretical answering. Stephen Slut and then Blu. Yeah this brings up a lot of issues it's very fascinating um I'm curious um sort of coming back to this idea and I've been Daniel mentioned hyper priors are things that we kind of have fixed um I actually looked that up actually to make sure I understood it but things that we have is very distinctive. I'm again trying to compare these um these types of um environments where one where you've got an agent wanting or being able to just calibrate in amongst the noise ones where you want to just keep it as clean as possible you know so and and the challenges sometimes when you have a quality management system in those places so for instance I suppose one thing that comes to mind is the experiment with a rat that they used to have and they've had for years about having cocaine or addictive substances and how they would they only needed to have a couple of times and they would be addicted and it was seen as this kind of proof of principle about addiction and how it worked and then the more recent work where they instead of putting them in a bare box and just having the cocaine or whatever they were being um sort of exposed to they were in a more ecological box you know where there was things to do and they suddenly found that the behavior of the animal was very different they wouldn't necessarily become addicted in the same way because they would actually because they were in a better ecology now in theory the entropy has gone up there if we're looking entropically but for the animal they've been able to engage and dynamically process that that's very different to how you would try and create you know a medical production line right you you want to keep it clean you want to keep it sterile um you want to have those sorts of things so I'm just curious about that and the danger maybe in in biological system with this hyper prior say the some sort of kick or something that comes from food not having any other compliments around like like as Daniel you know so there isn't other things so that just naturally kicks in like the reward goal which may be always there as a default if there's not other things to compliment it just it starts running the show and that could be part of how we get into these maladjusted behaviors that you were talking about I wonder what your thoughts are on that just quick point of clarification that's not what a hyper prior it is that's not a hyper stimulating stimuli a hyper prior is a prior over another prior in a Bayesian multi-level framework so it doesn't have to be accentuating or differentiating or extreme or maladaptive it's a general framework for the relationship between variables yeah well I was thinking it was more the hyper prior could give a it's a it's a it's a fairly fixed prior isn't it's one which is a go-to solid prior and it's useful for being that doesn't just saying having that present hyper prior could be fixed or learned or learn okay okay but it tends to be hyper prior means it's quite accurate isn't it it's quite it's not fuzzy hyper is I think a spatial designation like supra not not in the extreme case hypers only referring to the fact that it's a parameter about another parameter so it doesn't tend to be hot or cold or anything it's just the general phrasing okay got you so but Stephen Fox if you want to add anything it otherwise blue okay we see the head yeah so in now the effective system so here here's my diagram so in in this saying that if it's a little bit so small I can't see it myself um um that if there's the same starting point but um of a generative model so if the human being in human robot interaction knows that um okay so I'm I'm in the predict I'm human I'm in the predictive processing business that's what I'm doing and um and um so I got prior beliefs and I have got generative models then the robot can be um what does that say the sonars and scenarios for robot based on the human's uh well that's a worldview but generative model and human readable model the robot mind that corresponds to that so that's um and this is in aligning human psychomotor characteristics with robot manual augmented reality in this so there's a bit of a model there that's also very related to the Felix Scholler's paper trust as extended control active inference and user feedback during human robot collaboration oh I'll put it in the youtube chat I'll put it in youtube chat great go for this one yeah so that's um it's all in the past experiences physiology so this is body memory in the personality gender culture emotion reasoning in bio cybernetics so this paper I'm uh relating that to what how this can affect interaction with robots exoskeletons and augmented reality we got here a few equations there it is but in the two as I point out in this that um you find yeah so in previous production literature the uncertainty of human behavior this is this is true so in previous production literature the uncertainty of human behavior been recognized as a source of productivity quality and safety problems and whether fundamental reasons for the uncertainty of human behavior have been framed previously as a black box so what I was trying to do there was relate it to to open up the black box and uh frame it as an open box and then fundamental reasons can be aligned with production time to facilitate improved production performance so that's as far as I've got with that it's almost like for the uh reward utility driven person you can say there's going to be an increased expectation of performance for the uncertainty minded you can say there'll be a reduced variance for the resilience minded or for the reconfiguration minded that there'd be an increased capacity to rapidly reorganize so I think that we'll have all or none blue so my thoughts kind of dated at this point but just going back to um touching back on to mutualism and um you know these these the electromagnetic spectrum when you're talking about how like you know the signal can amplify um there's also like the possibility for the signal to cancel right if if the you know wavelengths are are the same and they're heading in opposite directions it they just directly cancel each other out and it kind of made me think of like cancel culture and also like the relationship between um you know the parasite is is also like a nested biological relationship um even though it's not necessarily beneficial but there's still this nested layer like the the parasite is unable to function without the host of course but um the host embodying the parasite like is a completely different cognitive cognating creature and I think about like the zombie ants right that um I don't know if you guys have heard Daniel can probably unpack the zombie and a lot a lot better than I can but it's like a parasite that then drives the behavior of the ant due to some like rewiring of the cognitive process so even in the parasite instance there's still a greater cognizing structure and and going back to neutralism again the the other terms are so commensalism is where one gains benefit while the others uh it hasn't there's a new neutral no benefit no harm and then there's a mensalism where one is harmed while the other is unaffected so they're in between so there's first mutualism benefit benefit then commensalism where there is a benefit neutral and then there's a mensalism where there's um harm and neutral and then parasitism where there's one is harmed and the other's benefits I I wonder if this has to do with the movement from maybe just its semantic but from sustainability to regenerative because sustainability it's like the sustain on the guitar sound of the bell like let's sustain let's have this car and sustain it or make its it can be meant to mean more than that but it's about uh sustaining something that exists versus regenerating uh means like we're going to have to be doing more than just persisting or stabilizing it that it has to be also actively having this process of uh of regeneration which always involves like multiple parts like buckyfolder saying that unity is plural and at minimum two so then now thinking about how from polyculture of growing crops to monoculture green revolution and everything and then now oh maybe there should be the fish in between the roots of the plants and then the birds and then the mushrooms like this are these stable ecosystems that is what remains to be seen but some human modified niches like sourdough bread and things and beer have been really successful so maybe there's higher order combinations too that could be really successful and be very productive but um this is kind of a framework that helps us at least explore that so that's quite cool I was at some remotely of course there was a launch the South African grassroots innovation program they uh did um a launch yesterday with five uh lady innovators and one of them was doing um movable aquaponics and uh she's seemed to have addressed it's quite a sophisticated system but it's all uh it's all set up in buckets that can be carried one person can carry the buckets that's the size they are and then you build up the the farm by different just having the number of the different the more buckets you get the bigger the aquaponic farm so that was interesting and then again with the physical zero sum then the electromagnetic or informational or financial non-zero sum um if the overlapping is just given the tools that already exist in the ecosystem that you find yourself in like you're dropped on survivor island you still may be able to find some synergies most of them you won't know but they're over a long time traditional ecological knowledge can learn about some medicinal combinations and uh oh these two should be combined for the boat and these kinds of things but design and this is related to what Steven said about the product being plugged into another external niche and what you wrote about the cupboard being attached to the wall like we could design some pretty minimal pieces that might really scaffold like you build a trellis and then something grows within it so the engineering might not be huge to connect a few pieces that have a latent capacity to work together yeah the the word you use there scaffolding has that got any um as i've of course i've heard it before but does that have any basis in nature to your knowledge i would certainly say so um from the specular structures like in bones and the way that physical structures are scaffolded to um embryogenesis and the developmental scaffold scaffolding of the uterus and the core caretakers and different insects and humans and the family and so i think there's the cultural scaffolding and then all the way down to the microstructural cytoskeletal scaffold or scaffolding enzyme complexes that then can do very efficient things too yeah this is interesting because the problem with the the challenge with um so-called grassroots entrepreneurship and innovation is scaling it up that's the the challenge that's that's what was said yesterday that's what said in india there's this problem that solutions are developed and they they work but they at best stay where they've been in the village where it's been developed it sort of just stays there and so scaling up and it's an open source um would it's well there it is the technology is there isn't it open source ecology it's it's possible to put all engineering designs and bills and materials and methods of work and all on the internet where anybody can access it but still things are not scaling up thanks blue so i think about scaffolding in terms of like neural systems but also in societal systems in the brain like my phd work was focused on finding a scaffold that will promote the growth of neurons across um across a lesion and so when when you have a traumatic brain injury the glia which is like the natural scaffold like freak out and form a scar to prevent like the whole rest of the brain from dying because cell death can be like this kind of cascading process and so the glia and and it literally results in like a hole in the brain where the injury was so i was looking at these bioengineered scaffolds that promote the growth of neurons across that hole so like it'll form a bridge or a structure a scaffold for the neurons to grow and then you know in terms of the the societal setting i think about scaffolding as infrastructure and and i mean even in an organization like you have to have bridges and roads and internet connections and utilities and you know these things are it's the the infrastructure that's necessary for kind of new um branches or systems to start up and even in like the farming terms it's like you know that the um like i don't know here we just call them the ditches but like the irrigation canals i guess is probably the proper term like that you know the um where you can get a river to flow off into different um subsections and then irrigate the farmland with that water so so i think about that as as the scaffold awesome blue nice connection and sounds like cool phg research we see your screen steven so you see grassroots innovation program yes okay yeah so that was that was yesterday um all of the press release in july but it's not it's not detailing the but but there we are this is the um so in south africa there were the rate of unemployment is i think officially it's third it's more than 30 percent but um unofficially it could be as high as 50 percent and uh this is so they're now trying to address at least partly through this grassroots innovation problem program so one and there are there are similar and and we know that um the book decades ago small is beautiful and uh gandy was before that was promoting this but it does it's not scaling and so there are these um in between industrial production there's just this ever-increasing wasteland but efforts are going on around the world and grassroots is a good term because it's just growing from the growth evolving naturally doesn't it sounds like really interesting work thanks for sharing it steven select yeah actually that ties in actually we were running a project in south africa funny enough um and working on looking at how to create community driven innovation through strategic strategic conversations at the community level and there is there is an interest in difference between grassroots i think and community driven so i think that your grassroots is often kind of um tied to it's like the base level of the organizational structures you often hear it with political campaigns and stuff it's the grassroots but those grassroots are the grassroots of a campaign where people sort of get into there which is a little bit different to the kind of a community embedded community driven scale and that that comes back to this scaling question which is the same thing that had the problems that so the scaling there's now when you talk about scaling up and that's a big challenge is how does stuff get scaled up and they often are talking as well about um scaling out if you have a program or scaling deep so you know some things might scale up because they go up to a higher level and then they basically get dropped down again but how do these things get scaled out so there may be some i think you're you're you're tapping into a few multi-level issues which i think are quite quite important so i wonder what your thoughts are about going at uh going into these kind of embedded local context to create new seeds so to speak for like adaptability and innovation it's like a seed is a biological planting metaphor the scaffold is a um highlighting the industrial component of a biological process like you still need the tomato seed so it's not saying it's not community driven it's just like a soil metaphor like how these communicate what our vision is and they really do matter and so they're really um important points actually here's a comment from kevin in the chat who wrote is the scaffolding a schema or is this supposed to be anatomical that is is there some physiological analog in the human nervous systems or genome that corresponds to the scaffolding i think blue made an awesome point about the glia and about connective tissue being uh an immune tissue being scaffolding physically for like the synapse and for cells and then the genome oh yes the chromatin scaffolding goes deep in the nuclear pore structure and that's a whole cool area and maybe there'll be a day for transcription factor active inference soon enough so um what you're saying is i mean these hands this is a an organization um uh active in botzwana and everything they're doing there although we know you can see yeah yeah so this is led by uh thubbissa and he's um applying design methods in um communities in southern africa with great effect and this is all uh yeah and so the idea is that people are with their own hands are um developing innovations and as you say you can be scaled out oh yeah uh scaling this on the side so they're all the um so for example in in that part of africa that there are many movements like uh many initiatives going on but it would be very beneficial if there was um a framing a kind of unifying framing of what they're all doing which can be seen as an alternative an alternative form of socioeconomic development and rather than just as something that's uh well fundamentally inferior to industrialized to to the conventional model of industrialization so all suggestions are welcome i just put in the youtube chat a volume uh by caporel grissomur and wimpsat developing scaffolds in evolution culture and cognition and so the paradigm change as steven kind of hinted at very early is like from systems where we eradicate the life to where we welcome the life and all that it brings and we don't need to build that and we can't build it we're not going to be able to compose the lipids and the millions of years of fault tolerances into some sort of uh you know monster that helps our living systems we don't even have to though we just have to build an engineered scaffold and the scaffold can be radically different than what the final product looks like the scaffold can even disintegrate if it's biodegradable so it um it's a really interesting area that this discussion went which was like towards the engineering that makes us think of like foundries and some of the technical and precision processes tying them to some natural processes like this evolution culture cognition volume does and unifying frameworks those uh together almost point away towards engineering a scaffold not a booster rocket or just a bunker but bunkers might come into play of doing some scaffold design so that there could be reduction of expected free energy through deep time so pretty interesting um point steven thanks for sharing all these resources too blue just a quick comment um you were talking about the scaffolding disintegrating sometimes the scaffolding is designed to disintegrate like i think about in in biological you know terms we have like webbing between our fingers that enable like the growth of the digits and then you know it's designed to be eroded uh in in the developmental process um so sometimes that's that's part of it or even like when you get a wound sealed you know you have like the the stitches that scaffold that form the scaffolding so the healing process can occur but they dissolve inside the wound actually that's a good that's quite cool to think about that as well in the processes and uh when when we whether in areas that we live in or areas that we don't live in when we plan to intervene are we planning to make something that sustains itself or are we planning to deliver or co-create a scaffold that is planned to disappear those are really different views on local and distal intervention steven with a screen share what's on the screen steven fox yes so so now this is um this is an initiative called connected hubs so this this is coming from the southern african innovation support program and uh the idea is to um that there are hubs which support um innovation and entrepreneurship and these are this particular uh one is um so as you can see the connectivities here in this part of africa and what the idea is build capacities and support and so it's connecting uh different hubs together and so a hub it's what that means is it's just that usually they've got a physical location and they uh there are people you can contact there remotely and physically who've got can help with developing um ideas for entrepreneurship and social entrepreneurship so this is something that's going on it's connecting hub so maybe there's this is hopefully this is the intention is this persists after the southern african innovation support program it's on the second phase now after it finishes so this is one i suppose it's summed up in this uh diagram here this idea that there are more and more hubs in and the european commission is um trying this africa oh we'll see this and then i have to there we are so this is kind of things that are going on so i'm involved in this project sustainable um network of digital innovation hubs so that's something that's coming down from the african union and the european union and the hope the aim is that this is a somewhat new way of um facilitating socio-economic development and the idea is to try to get autocatalytic effects by you know more connections more um more dynamism well they're always welcome to come talk about active inference at whatever stage in the last few minutes we'll just have steven and then blue and yeah just give kind of last yeah thought or question here as we close out okay yeah now thanks very interesting and um i suppose with active inference i think very interested in these these developments it's actually how i ended up getting into this because i was trying to develop community-based hubs in south africa funny enough in rural rural naputa land um and i think what's quite interesting as daniel's this this idea of a unifying framework rather than trying to reduce down all the complexity which is what i was doing for far too many years is really amazing and the other thing that all of these aspects that you're taught you just shared and that ties in with this lab is it's it's about participation as soon as you bring in participatory processes you um you bring in these active inference dynamics you bring in this complexity you bring in this need but also the ability to do what mass manufacturing can't do but like i thought you said it's it's seen as being inferior or in our culture and i think seeing it as being vital but also challenging is really cool so i think that's something i'm going to take away because i can't i won't actually be able to stay too much longer i've got to jump on another call but that's something i'd like to take away so thanks very much brilliant really really enjoyed today thanks so much thank you steven see you later blue um so just a couple final thoughts uh john boyk and his idea of connected hubs i mean he was on like probably six different um live streams and uh he gave a very good overview of sustainability um and the you know how to build an architecture toward a sustainable future so the connected hubs kind of reminded me of that and then the one other thing was um just about scaffolding and you know how do we build kind of a scaffolding for our cognitive architecture and is this a scaffolding that's maybe designed to dissolve and so i think about you know scaffolding for our cognitive architecture as as education right like we send the kids to k12 and they you know form this like shared understanding of reality and and two plus two is four and how to read books and so forth and this kind of um education maybe daniel will remember at least in college when i started college um in biology many things are so complicated that they just give you like a very broad overview that doesn't make sense or i was always searching for like but that doesn't make sense but that doesn't make sense like in your entry level classes and and then as you get like deeper and deeper into studying biology or biochemistry you start to kind of understand electron transport and these different like things that are are very like micro but the macro level doesn't make sense without understanding that that micro and in my mind this this kind of educational or that way that i was taught in college was kind of like building a scaffold that was designed to disappear anyway um but anyway yeah thanks for the for the talks today and for coming on it was good if i can give my last comment and then to the author for the final comment i just love this idea of the biochemistry education as the removable scaffold like you learn it once that there's the essential amino acids and then you never wonder like why on the nutrition label is there essential and non-essential you forget the details the side chains you know the c-minus on the test that all disappears with something that's generalized but otherwise we would look at macroscopic phenomena like why can't i swim when there's this algae in the lake well there's some molecular answer we could know at least how to sense make in that area might even know the specific details but also it can be removed in a way but can it be removed intergenerationally a chemistry to alchemy reversion with similar or different tools that would be quite something so steven with the final notes but just wanted to say thanks for coming on these streams and it was just like super interesting discussions well thank you very much it was my pleasure and i look forward to further cooperation with you great if you or any of your collaborators on these projects or your contacts in these hubs wanted to present on this or facilitate in any way then we can talk another time very good thank you very much thanks again steven thank you blue bye


Music's my area, music and technology. And the Media Lab's been a fantastic place to work on music because it's not a music school, it's not an art school. It's a place where we can think about how far and in which directions we can push music in the future. So I've been particularly interested in seeing every possible way that we can use music to allow everybody to create, to connect people and to invent new kinds of collaboration. And I just want to give you a few glimpses of those areas today because we've been pushing this work in a lot of directions for a while now. I got into this field kind of naturally but my dad was one of the first people in the field of computer graphics. So he had a company that made some of the first graphics machines and I grew up seeing those be built and invented. And my mom also went to Juilliard. She's a pianist but she's a really great and pretty well-known piano teacher, very, very creative piano teacher. So after our piano lessons, she used to have all the kids, we were three, four years old, spent about five minutes going around the house. She'd asked each of us to find an object that made an interesting sound. We'd all come back together. She'd say, okay, what sound does that make? And that one, okay, great. What's the loudest sound you can make with that? Oh, great. What's the softest sound? What does it sound like if you play those two together? Oh, great, okay. And now we've heard all the sounds. Make some words. What are some adjectives that describe those sounds? And we'd make some words and then she'd say, okay, have these sounds. We have some words. Let's tell a story. What comes first? What do you want to say? So after 10 or 15 minutes, we'd have all these new instruments and we'd have made sounds and we'd make a story.  So when you come back next week, we can try it again.  Music's something that's live in our world. It's something that comes from the world around us. We can make our own systems of music. And even notation that hard writing down of music is just a way to remember what we did so that you can do it again. So that was very inspiring for me and I think for a bunch of other people too. I studied piano for a little with her, but then I studied the cello. That was my instrument. And played kind of normal cello, the repertoire for a little while. And then by the time I was in my teens, I was playing rock music on my cello and was inventing electronics to plug it into. And was making it sound really pretty odd, which I still do from time to time. Right about the time this picture was taken, a couple of years ago when I was probably 23, I was still at Juilliard and I got invited to come to India. And I had never been here before and I was quite lucky. I got invited by a pretty prominent family who live in Ahmedabad who have a large bit of land and have a Le Corbusier house, very splendid.  And I had the opportunity during that time to learn a great deal about Indian culture in general, but especially about Indian music. And I met all kinds of musicians, not in the south, but all over the north. And I took my cello around and would play for people and then they would play back. And I knew that the world of music was much bigger than I imagined and it was in full color. And so being here and listening to this country was one of the most transformative experiences I ever had. I've tried to do that since. Taking all those things, my love of performance, my love of classical music, rock music, music from around the world and technology, I started out by building instruments that would allow a performer to use his or her normal skills but to do much more than an instrument normally could do. So we invented this field called hyper instruments. That's Yo-Yo Ma playing one of our early hyper instruments. The idea is the instrument knows not just what notes are being played but how everything's being played, what the nuance is, what the expression is. So by changing your interpretation, the cello can turn into a flute or a violin or an electronic sound or a full orchestra and all under the control of the player. So this is a fairly recent example of a young English cellist playing a new version of hyper cello. It's just a normal cello with a special bow with all kinds of electronics embedded. But you'll hear that everything he's playing and all the lights he's controlling come from his normal performance on that instrument. So we do... Oh, thank you. So we build these kind of interactive systems that are quite smart and responsive in all kinds of contexts. And I do have robots around our lab and one of the most recent examples of these hyper instruments is for an opera that I wrote called Death and the Powers. And that's actually what we call a robot opera. It's about a guy who's rich, successful, powerful. He is actually, he wants to live forever and he's also kind of tired of the world. So he figures out a way of using all his smarts and all his money to allow himself to download himself into his environment and then to leave. So when the opera starts, he is about to turn on this system and he leaves the stage and for the rest of the opera, until the very, very end, the singer is down with the orchestra. You never see him again, but we wire everything about him. In fact, we invented this technology called disembodied performance, like a hyper instrument, but we also measure things that the performer isn't aware of like muscle tension and facial expression and breathing, things that are natural. And all of that information gives the whole opera stage a sense of how this performer is feeling. So everything you see, the set, the furniture, the objects, the sound, where it is, how it changes, all represents the inner spirit of this performer and the idea is that once he's gone, this environment is him and everybody else's family and his business associates and people who run the world have to decide if that's true and if they like that. So here's just a little tiny bit of Death and the Powers to give you a sense of how that works. The robots get the last word in this opera. If you want to see it, it's touring around, it's going to Dallas next, this beautiful new Dallas opera house in February. And another thing that interests us is how to take a great live experience. We're doing a live interactive simulcast in HD image and multiple channels around sound to a variety of cities around the world, but we're doing it with all kinds of layers of content like viewing from backstage, viewing what the robots see, viewing what it's like from inside the system, things that you don't see in the live house that gets beamed into the places remotely where people are watching and it also gets beamed to your mobile device. So you bring your mobile device to the theater. We're actually testing this just a couple of days ago at the Media Lab and content that you don't see in the live hall that gives you extra dimension to the story, that gives you extra images, comes up on your device and gives a whole extra dimension to the show. You can even send information back to the live opera house. There's a gigantic computer-controlled chandelier in Dallas. The next thing that interests me is not just building these crazy stage shows for some of the world's greatest performers, but finding a way to use similar concepts and technologies to make it possible for anybody who loves music, which is pretty much everybody, to be involved, to directly and actively participate in music in all kinds of ways. So we've done all kinds of things like make a whole orchestra of instruments that use video games and body gesture and voice and touch to control and make your own music. We made a censor chair that people like Bono from U2 plays, instruments that are squeezy toys for children so that they can play with professional musicians. You may have heard of these two things. They both came out of my lab based on all the things we built for that interactive orchestra. Thank you. We do a lot of work with music and health, both to use music to just promote general well-being but also to diagnose a variety of diseases and to give a voice, literally a voice to people who can't communicate in any other way. This is somebody named Dan Elsie, who was born with cerebral palsy. He came to one of our workshops. It turned out he was a very fine musician. We made a head tracking device that allowed him to use some of our software to compose his own music. Then we made a more sophisticated model of the device that allows him to perform his own music. So I thought I'd just show you this little clip of Dan playing his own piece, My Eagle Song. One thing with Dan that happened that we didn't expect is he's pretty popular as a musician now. So one thing that we do from time to time is go out and we're kind of his roadies for his shows. He'll go and perform in my students and I'll go and cheer him on and help him do that and try to put them together. How can we make a new kind of collaboration so that people who have a lot of musical background and a lot of experience can work as equals with people who just love music, the general public, students, anybody, and really make things together. Not a teaching model, not a mentorship model, but really let's build something together. I started this idea kind of in a big way. The Toronto Symphony Orchestra came to me about a year and a half ago and asked me if I wanted to write a symphony for them and I said, I'll write you a symphony, but only if I can invite everybody in the city of Toronto to compose it with me. And Toronto, they're great people up there. They're very nice. Unfortunately, they said yes. We did this big project in Toronto, figured out a model to collaborate with many, many people and have since done another one for the Edinburgh International Festival, which premiered about a month and a half ago. And the basic way that we do this is to start out with a picture, something that has no sound at all, to give an impression of what this particular piece and this particular city might be like. It's, you know, I usually go and take a look and just put it out there. So for Toronto, for instance, this is something that expresses that it might start soft and end soft and build up and there might be synchronized parts and fluid parts. And I also, at the very beginning, make a little story out of it, different ways of exploring the city. And I also say there are different ways that we're going to share information. Yours are things I'll ask from the community. You know, send me recordings that you make in the city. Send me stories. Send me pictures.  Mine are things that I make and send out, little bits of the piece, melodies, chords, harmonies, whatever they are. And ours within H are sessions that we do either in person or via Skype or whatever to take material from the piece and shape it into something that will actually end up in the final composition. By doing this in Toronto, for instance, I met people from all over the city, all different backgrounds, all different communities to listen and to make things. I got recordings about 15,000 recordings, pretty high quality ones, from places in the city, from people's houses, from conversations. A lot of people send mass transit systems where they sound kind of interesting. They're pretty different in each city, too. We worked with the school board in every city and Toronto, we worked with thousands and thousands of kids, grades three to six. So about age eight to 11 or 12. They were all given the task of writing short pieces with some theme of Toronto. So we brought kids together who had never met before to listen to each other's pieces. And they all used some software that we created called Hyperscore, which uses lines in color so people can make sophisticated pieces without having to know music theory. And this is just a little snippet of how I took hundreds of these pieces, put a little bit of my spice in them and wrote this for orchestra. It's about 30 seconds of some bits of Hyperscore. The next thing that we've done for these projects is to make a whole series of apps and there are a bunch of them online now. We keep adding to them that allow people to take music for the project as it develops and to make their own versions of things to post those again online so people can edit, comment, add to. This is called something called Constellation where I can take thousands of sounds recorded in a place, feed it into the software. The software breaks it up into little fragments, analyzes all the fragments according to their acoustical properties and you can sort them various ways. But then it sprays them on the screen like a Constellation and by moving my mouse and connecting them with a line I make my own composition based on these songs. So the line is my composition. I post that online and I can listen to different people's version of moving through a particular city. Media score, this allows me to take something like a melody, the fuzzy thing at the top. I can put a melody from the piece and you can take a series of virtual paint brushes and change the texture of the melody which then changes how it's orchestrated, how it feels and you can make your own version of different sections of the piece. Cauldron in Edinburgh, I made a kind of soup where if you stir the soup you change the overall sound of Edinburgh and if you pick the little spices, the little dots on the left you put ingredients into the soup and change the way they mix into the overall cities. That sounds like a tiny fragment.  We even did something where you can take fragments or bubbles that represent fragments of all the music in history and we did a live event with the Guardian newspaper where you could come online, listen to all these fragments. I had a young pianist at the Media Lab looking at the circles in which people preferred and Bach was getting bigger and Ornette Coleman was getting smaller and he made up the music depending on where the circles were and that result became something that we used for the final composition as well. Took all of that, wrote it into a piece, rehearsed with orchestras back and forth. This is just a little bit of working with people in Edinburgh on a kind of crazy bagpipe that's played by a whole orchestra, an imaginary bagpipe. So we've had really fantastic results with this project. It's been amazing to go into different cities, explore them, meet people, listen to the city with people, think about what the city feels like, maybe think about what the community wants to imagine together about their city now and in the future through music. I wanted to just throw out an idea. India is such a special place. I wanted to just suggest the idea that instead of doing a city, maybe we could think about doing a piece together about the whole country. So this is an invitation to you not just to start sending things, but since doing something for a country and especially this country that what I'd like to do is invite you to think about it, see if it's something that might be of interest. You can let me know if you think it's a good idea, if you might be interested in working on it, but I'd love to know if you would like to help. And I just wanted to end with one teeny little thing from Toronto. This is just the 30 seconds from the end of the piece that gives you an idea of what happens when you pull a whole city together in music. Thank you very much. Thank you.