There we go. This is the Fellowship of the Link call for Wednesday, March 1, 2023.  And just by means of check-in, I mentioned this on FreeJurisBin on Monday. I set up a new talk I'd like to give in places called My Life as a Cyborg. And we puzzled through whether Cyborg or Centaur or something else and a couple of other questions on Monday. But my wife, April, posted about it on LinkedIn. That got a bunch of questions in traffic. And so I'm sitting down thinking, well, OK, I'd love to do this. I'd love to get some attention. What does it mean to be a good cyborg? And how do we do something about that together? How does that fit into our different projects? Because I think I'm more motivated toward this question by other people's explorations with chat GPT than I am with my own extensions into the brain, extensions through the brain and stuff like that. But I think that the nexus of those things colliding is really juicy and important and interesting, like how does the agorah mix with massive, mix with chat GPT, mix with my brain, mix with other sorts of things. And I was just metaphorically using a bunch of pots simmering on the stove with Pete before you got on the call. It's like, how do we make a stew out of those things that is nourishing? That sounds, yes, very interesting and very much like what we want. And I just sort of like firstly believe that it will happen this year. And it will take more like more weeks than I usually think that you should like for everything. But like it just seems like it's coming. Yeah, somehow. I think so. Yeah, there is like we did some just some very short experiments with like the DBT API and by we I mean beta, who is also part of the other project with the Agora like a year ago or so. And it was really interesting how it just picked up on like our specific patterns. And you know, like you could even back then ask it to like essentially generate a node for an entity and it will follow the Agora conventions. And this is what we look before everything before the DBT and so on happened. I guess my my position here, I don't know, like I would love to like read that. I haven't that you have a link. I will check it out or or you tell me where to go for links in general, like export from your reign. But like I guess my I'm particularly interested in exploring one aspect of this, maybe, beyond the actual generation or like the experiments, which are definitely going to be fun and interesting. So for for me, massive weekies and Agoras to some extent are like a federation of brains or, you know, how are we going to go? Those for for me, they may solve the governance issue to some extent. In the sense of like, how groups of people can gather a corpus to like fund you like a language model and then agree to to to use it, right? Or to like rent it or, you know, whatever the the community wants is. In particular, even that, you know, it seems like one of the pitfalls here is like, you know, your copyright and how to share profits from like models, like, you know, like stable diffusion, how could stable diffusion like pay back to artists, for example? No, I think that we may like lacking like some distributed way of agreeing on what's OK to train and use and what's not. It seems like maybe like the biggest companies that can essentially influence copyright law may get away with it. And all the others may not, right? That's what we are a third-figure model, you know. So I guess to work around that, we could just pull together our own corpora. I guess I think I'm following you, but I'm not entirely sure I am. Whether this is a question about in ultra property rights or this question about commons and how to build commons or or how that proceeds. And I'm thinking here about hip hop culture where there was a lot of sampling done and sort of before hip hop culture, sampling was clamped down on heavily by the music industry, I think.  You know, here's a song that samples 30 different songs. Come and get come and catch me, et cetera, et cetera. And suddenly the sampling loosened or was more feasible or whatever. And suddenly you could see music everywhere. And is that what's happening for images? Or is this just the destruction of all creative talent for illustrators and artists and everything else? I think it would be interesting to find out my mind. I'm not sure that there was a lot of sampling before hip hop. There was certainly, but not necessarily a lot. And then hip hop basically started the sample culture. And then RIA or whoever got upset about that. The reason it happens nowadays is because they figured out how to do the cross billing for it. So everybody got spanked and their their hands got slapped. And then it's like, you can't sample. Well, we'll see your we'll see you if you sampled. But go ahead and sample and pass. And that's fine. So it was the the market mechanism to to make the copyright holders happy that that made it possible that you could have sampling kind of is similar. You know, it's it's funny to me now that I see non-official people putting up YouTube videos with soundtracks or even like whole chunks of video.  Now a lot of times it just gets monetized. Right.  And I'm, you know, we're monetizing the copyright holder. So it and that's that's actually even, I think, more automated than than the hip hop thing in the hip hop thing.  However, that connection is made in the YouTube thing. They just checked it and they they just, you know, do the right thing. Right. It's like, I guess that that's a pattern where like I guess in my mind, I have this file as like flow network. I guess maybe a coefficient because, you know, like networks used for like flow, you know, flow problems in algorithms, which I've always been in love with, I guess.  Once you see this, you can find them everywhere. I mean, society is flowing resources all the time by definition. I guess we do. And like, yeah, I believe that what YouTube does now is like, it's way more prosocial. And I'm sort of surprised at how and even like, you know, flowing profits for transcluded in like resources, I think was one of the core aspects of the Nelsons. You know, like, I guess a pitch for Sanadu, you know, and we are still in there. Like, essentially, you know, we don't have a way of like, including, you know, copyright content in some other platform and then using, you know, a YouTube infrastructure to like actually forward the monetization. I'm not aware of any, I mean, there's probably many, right? But I was at the top of my head. I'm not aware of any protocols for flowing, you know, like, essentially, like solving the sampling or the mashup problem. I guess it's, I think the big publishers suppress that kind of stuff. Right. They want to keep it to like where they can meet with on a table. Yeah. Yeah. And as soon as it, you know, if it was flowing through the network rather than being through the publishers, they would have less control. Other, you know, other smaller competitors would come up and essentially eat all their profits, or at least I'm sure that's what they worry about.  The image generators are a good example of it. And then the chatbots are similar. The image generators got into a problem because we realized that there's an asset that hadn't been being called an asset.  It's copying your style. So style used to be something that you couldn't steal. So it was never, you know, it was never recognized as an asset. And so the image generators started stealing styles. Maybe, you know, maybe one argument is that, well, you know, Pablo Casso said it, you know, good artist to steal. We've been stealing styles from one another for hundreds of years. That's nothing new. But then it became systematized. And at scale and anybody could steal, you know, it wasn't a fine artist who could steal your style. It was anybody, you know, running Dali or mid-journey. So so I think a chunk of a chunk of the what we and so chat, you PPP is the same thing, right? It's not that it's copying any of your copyrighted works.  And maybe they should be, maybe they shouldn't be, I don't know. Yeah, I mean, that's a great observation, Peter, I think. Yeah, like the, yeah, I think you're spot on with, like, style being an unrecognized asset. Yeah. So I guess there's, I will expect it to be like, I mean, I wonder if you can develop any kind of like water marking for style. It's doesn't, I'm not sure. I don't think I would. Okay. Although that's, that's an interesting tactic to take down. I just, the big problem is that we have a disagreement whether or not style is an asset, right? So I think where it kind of fell down or where it fell out, where it got to were, I haven't followed this very carefully, but the image generator, people who are having image generators said, okay, we'll back off. We won't, we'll try to make it so people can't copy the style of living, you know, living artists, at least. So I think that the fix is not so much water marking a style. The fix is just having style recognized as an asset and being able to go to intellectual property court and say, you know, look, there's a bunch of people making art that looks like mine.  Right. I guess this is where like, sorry, Jerry, go ahead. Just a couple brief things. Style is kind of what a mature artist develops. It's like, like, you know, you're getting somewhere where you when you have an identifiable style and people start asking for the thing that you do. So style is really kind of important, even if it's like not really reified or honored or otherwise sort of picked out the way Pete's describing. And then I don't know how you prevent one of these engines from imitating any any particular style that what they're what they're good at is essence. What they're good at is, oh, I know a maple leaf when I see a maple leaf and I can tell you that this is a maple and that's an that's an elm leaf because I know about the essence of what makes things maple leaf and what makes things sort of elmy or oaky. Hey, Chris. And and also even if you could parse out all the living artists to not interrupt their deals, there's there are a few entities worse on the planet than the inheritors of famous art who famously are basically shut down all attempts to do anything useful with their art for the most part. They're like errors. I don't I do not like errors of art and all that kind of stuff.  Yeah, I guess in general, oh, sorry, Chris, I can't hit that in general, I guess my first reaction is always like, it's also like, like, I guess I'm conservative in this sense, like, but depending on how you frame it, like, better not to limit creativity in any way, even assisted by, by models because of the risks, you know, like the killing effect that I can follow. And I guess your limiting style seems like something you wouldn't do before models, right? Because like, like you said, like it wasn't recognized as a particular like, like an attack vector or like something you could exploit, then we get this new tool and now we have to limit creativity in some sense more than before. So it seems like it could be like, like jumping like to fast on it and maybe like screw it up for future generations. And also like the living art is only, I mean, that's of course, like, I guess in a wood timeline, that yields renegotiation discussion on copyright law in general, which I think is completely unreasonable. Totally out of hand. Yeah, but like, yeah, it seems like it could be dangerous as well in some sense. And so there's the general question of, is this the end of traditional copyright? Is this just is are the copyright holders going to reassert some kind of control and try to crack down? Yeah, don't know. Maybe it's time for the copy of revolution. It's been it's been time for that for a very long time. Right. But in this case, you know, imagine like all these tools, a lot of young people, like every opinion, like a lot of people using mid-journey in this court and, you know, like getting a taste of what like cyber creativities to some extent, and then like a look, like a lot, make it comes over and says, like, well, actually, all these things you are enjoying, like are not illegal, you need to pay for, you know, like, and one person gets the right of like generating this, I maybe won't fly for a change, in particular, even that we are talking about a phenomenon that is global. I'm still like, actually, when you're like, when I think about like explaining to an alien, you know, like as a, as a framing device, I still, I'm sure if I could explain the current copyright state, even before this possible revolution, you know, like it seems like once it, once you have a global internet, I will have even copyright like fewer years than it has had so far to survive its current form. And somehow, like they conglomerate that whole copyright and the influence treaties are strong enough that they are keeping it upright. And I'm interested, has the, do any of us know of any efforts to rethink copyright thoroughly around the world that, that seem interesting and fruitful? One thing I thought of years ago was, I wanted to suggest to Google, for example, that they create attribution servers. And then one of the problems is the copyright conflates attribution with control, with ownership, with a couple other things. And I think attribution, this is just my personal take. Attribution is extremely important. You need to know that this person had an idea and then look at all these interesting riffs on it. But what I, what I was looking for was some kind of search engine where you could send it a snip of something, a piece of a tune, an image, some prose. And it would come back and say my, the earliest I can ever find anybody doing this is this thing over here. And then here is a tree of the apparent derivatives of that, the way you would do genetic gene mapping of COVID, like here are all the variants of that work. And all for the purpose of, oh, I'd like to reward the originator. And then you could send a tip, basically upstream to just the root or to the performance you just heard that was particularly good and the root and split it 50-50 or whatever. But, but, but if, if there was attribution that was well done, then you could have tipping. And I'm in favor of letting intellectual property like go in the world and touch people without paywalls and lockdowns. But then I'm in favor of making it really easy to reward creation and performance of all those things, like really, really, really transparently easy because the easier it is, the more likely anybody is to try to do that thing. And patronage tipping has gotten a little bit of traction, but not a ton of traction in, you know, online. Go ahead, Pete. I was totally, I think I almost agree. Except, except for the idea of paying somebody because they were the root node. Even tipping them. So not that you shouldn't be able to tip somebody. Right. But assuming that that's kind of the revenue model, I think it's still in the old world, right? How do you mean? Well, well, there might evolve several different norms or modes of tipping up the stream. And I and like that idea of tipping to the root node is just the first idea, the first thought that came to my head. I haven't really thought it through. I I it's it's not it's not bad in and of itself. But it it's a it's enough like the old model that it's dangerous. It's really dangerous to to think that way or to talk with other people that way. But we make it more likely to be adopted. This is really it may make it more likely to be adopted by the incumbents. Yeah, partly that and partly it's just the it's it's the wrong model to be paid for creativity. It it that's that's the root of the poison of the well kind of and it goes all the way back to that. So I I think you should be famous for being the originator. And people should come to you and say it's funny. The person I think of is Shepard Ferry. I don't know if he would be the root of the Shepard Ferry look. But he should get a lot more work. You know, the the person at the root should get a lot more work and a lot more attention. More people should want to hire them. More people should say could you art direct, you know, these other artists of mine that I want to have kind of the same look and feel that you did. But that. It's the I guess I guess I have a pretty strong feeling and maybe I'm not. I guess I still have it explored but I have a pretty strong feeling that creativity, human creativity is actually a shared thing. It's it's already a common. It's not a thing where, you know, Picasso was the guy who came up with the melted watch. So anything that has me walking in it, you know, should dolly. Sorry. Picasso is the guy who stole that. Dolly or all feeling for each other thing. So, you know, it's like, how do you how do you parse that into like, well, does that mean that a watch on a table that's not melted? Is that Dolly or is that somebody else, you know, the whole meltdown a little bit? You know, it's like all of that creativity stuff and look kind of like the Flancian's point. It's it it puts a priori limits on it when you think that's the that's the you know, that's so you know, I don't know, you should get more famous, but you shouldn't get more money for being at the root. So I like what you're saying a lot. And I agree. It's just that I have no employment opportunities for Shepherd Ferry, but I would love to give him a financial reward to make it easier for him to continue being creative in the whatever way that he might have like, like enjoyed doing. That's a great use case. Yeah. I think there should be societal mechanisms that enable that. Meaning a funding. You know, maybe so it wouldn't be this. But but just the off the top of my head. You know, there there should be there should be art councils and Shepherd Ferry should get a lot of money because he's doing a really cool thing that many people like.  Yes and no. I recognize the I agree with you. They're they're manipulable, they're corruptible. They're systems that want will want to self sustain themselves, whether or not they're being effective and all that kind of stuff. Right. And yet that I think that's not a reason not to do it. So we can argue mechanisms. Well, you're you're also putting all the onus and things on the original creator, which isn't always a single person. So let's say I'm I'm Disney and I help put together and finance and produce one hundred and fifty million dollar movie.  But what you don't see in the credits are the team of fifty people in the marketing department there, the forty five people in the distribution group, all the theater chain owners, all, you know, everybody from soup to nuts. And if that's a million people are touching that thing, which is in part as why it's as expensive as it is. So it's not just how do I renumerate the director and the actors or maybe the guy who shot the thing for a person of photography, but it's every interstitial person along the way. How do I how do I remunerate Flancian for being a Google engineer who has he may not even work anywhere remotely close to YouTube. But some of his work, I'm sure, impacts YouTube as a distribution channel. And it's not you don't see that you don't hear it. You don't know it. And it's all the it's all that invisible work that also needs to be supported as well. Yeah, so in a copyright case, so I knew the guy who in the late 80s and early 90s helped to push Disney into China. And it literally was a state actor level play. And Disney was the only studio whose material could be distributed at all in China for years. So they had a monopoly and they were taking and essentially a 90 percent haircut on what they would have been making otherwise, just to be in the market. And, you know, that doesn't even touch on the censorship issues and all the other work they had to do to be there. But they literally they were like, it's so valuable for us to be there and build a brand. We don't mind taking the loss. And whether we do it or not, it's all going to be pirated anyway. So we may as well take the loss and take the brand recognition for later. But it's taken decades for China to have distribution systems. And they're still probably only at 60 or 70 percent of what they would or could make out of it as a commercial thing. So it's you know, it's it's not just how do you do it and how easy is it to to tip someone or give someone? But how do they then in turn to shepherd fairy create a sweatshop for his marketing and distribution people for you to get his obey giant t-shirt and how do you hold him account to, you know, treating those people well? Or does he keep 99 percent of the money that comes in and, you know, pays nothing to the poor kid in Indonesia who actually sewed his t-shirt and you don't know, you have no way of knowing.  So at the origin end of this, I've got this thought distributed accounting for value flows, which is basically the many different platforms that are trying to figure out how do we reward people in a community for creating value of some sort? What does that even mean? And that's a big open question. There's co-maker, there's disco.coop.  And that feels to me like a really useful and interesting problem to solve because if that worked, then as you rewarded a movie, then you would drop your reward for the movie into the pool that would then just flow outward along this particular set of distributions of value if we wanted to do it that way. Yes, yeah, I've got a friend who is a ostensibly a movie producer who develops and puts together things and they use a fund based thing and it works for them in part because it's a you know, they're doing comic book, fandom type commercial material.  And the fan base, who is, you know, essentially they're pre-buying their movie tickets and saying, I'm going to invest in this project at the level of ten bucks so that when it comes out, I. I, you know, they're never going to see a piece of the pie. But as investors, they're, you know, essentially pre-buying the book or the ticket or the whatever. And the level of failure, even at that and in the issues and the problems that she has trying to sustain that model may actually be way worse than the corporate stream model of going through traditional Hollywood. And which and she's done both. So it's I don't necessarily know that those things save. Save people's bacon at the end of the day either. And if it does, it's sustainable, but they're literally they're making a fraction of what they otherwise would or could have. I guess I just wanted to point out that a lot of these I think the end game, the way I see it in a lot of like a project in this space is probably going to look like a pattern we keep running into, I think, which is. Things that could be solved if we could like read in a source of truth in a route for a resolution procedure in a commons, in a default common and so on. And in this context, I think about this in this way. I basically I wrote something about this in 2019 and I'm still looking for it. Wrote about something that sounds a lot like we're already discussing some super excited about the conversation by the way. And it goes back to a coordination problem as well, which is, you know, how to like, essentially agree on a different like like a value flow like a network or like a flow network and actually have it have any chance to replace it or to complement at first the existing distribution methods. And like the idea I played what we did back then, and I guess I wanted to like surface it to get the opinion is like actually latch on to piracy. Say like we're going to build on piracy, we start with piracy because I looked and a lot of people were like, oh, piracy is a problem or prices, you know, like I'm more like the policy solution can't buy it. But like, you know, like we have all these copyright holders, these big companies and so on that are controlling the distribution, precisely of like profits and so on. You know, assuming we could agree on like one route for doing this, we could do a following, right? For everything that is media, so say movies to be in with on topic, right? I read on an idea, right? So each MKV, MP4, any kind of like a format you can find on Parallel Day for a movie, you can match it to like, you know, like the actual cultural artifact that you want to like model. And then each of these could have like a decision tree attached or many actually. So, you know, like we were we discussed going back to discussion, we discussed should this should the root of the root node be the creator? Who is the creator in the case of a movie? Is it a director?  So that's a problem. In the case of a book, maybe easier is the offer if there's one. But, you know, like a default distribution policy. And then you can say, you know, by default, whoever is a root node can say what the default tree is. So whenever you inject whenever someone donates to this cultural artifact saying like, this is what the value it gave to me, it flows down or it flows in a network and doesn't need to be down. It doesn't need to be a tree. And by default, because someone has to set it, the creator receives probably a size all chunk and gets to say what the default policy is. If the person donating doesn't override, for example. Right. You could do around the system.  But it goes down to like the pitch of many air drops, which is, you know, you already have money here, essentially. So imagine, you know, people could donate through this piracy embracing mechanism and then authors, the book authors are, you know, are usually short changed by by the big publishing houses, say or that don't get the profits from, you know, that doesn't exist because people are not paying for piracy. They get donations this way. And the directors and the actors and, you know, the people that it may be the people that work in a movie that produce the movie could say, this is how we will distribute the any donations we get through the system. Right. All these people could claim this money directly, essentially, you know, using some identification method, which, again, is hugely complex, a priority, but trivial, if you agree on a source of truth, you know, just for the exercise. And then, essentially, this will be providing motivation for people to jump over to say the same goes for even for, like, for the big companies, like, Disney, surely, I mean, even though I actually don't like Disney because of copyright and so on, surely the people in Disney working on marketing and so on still deserve some part of the donations, right. So, but essentially, it will give power back to the people to say how they want to distribute the proceeds of the artwork. And this seems like a trivial attack on the current distribution systems. I essentially don't see why it won't work. Like, a lot of these things. Assuming we can agree on one such provider to be with Critical Match. So here's a good question for you, Flancy, and do you know if you're an author and you have a deal with a big book publisher for your first book? Of of the one dollar of every one dollar that comes in, how much money goes to the author at the end of the day? Do you know what that number is? I will guess it is widely, but I will expect it to be like what, 10 percent or something or something.  Samper's right. I get this. I get in zero for that chapter. Ninety three percent goes to the big publisher who then splits it up and is paying for the physical printing of the book, the material, the marketing, the editing, the type, type setting, the typography, the art, the art direction, all the, all the other bits and pieces. Right. And I think we tend to overvalue the contributor of the, you know, what one might call the O-Tour or the big name, the movie director, you know, even in Hollywood, the screenwriter, who really is, does the biggest chunk of the work and does it early and helps to get it off the ground. You know, they get no credit at all. And in terms of payment, they get even less. You know, writers get paid peanuts. You know, and at the same time, screenwriting is one of the most lucrative, well paid writing jobs on the planet. At least per word and at a level of writing, because the average movie screenplay is written at a fourth grade level. But it's, that's part of the problem is we always want to overlook the Floncians of the world who make the whole thing work, you know, but at the same time, as a platform, Google is taking generally the average movie studio, typically their fee is 30 percent off the top of almost everything as just a distribution fee, much less all the other pieces. But from that perspective, YouTube is essentially taking an 80 percent rake off the top of everything and then giving a pittance to all the creators. And everybody wants to be an influencer. They want to be famous for having, you know, a million followers on YouTube. But the number of people with over a million followers is a tiny, tiny fraction of the overall content. And they are making, you can't you can't make it's not a day job wage to try and do that. You have to have a, you know, it's a second hustle. So how do you reframe that? But it's Google is such a huge player in the space. They're literally taking all the money off the top. And for that thing, it may be useful to have the small creators do a thing. But you still Google still has to pay to keep the lights on and do the distribution on your behalf. So it's it's way more complicated than we're giving it any credit for. That's for sure. No, it's usually complicated. But I guess I'm still like thinking about it like in the long term, the system may have like a wish like some evolutionary advantage, right? Even they go back to the influencer thing, you know, like in between platforms, the influencers in wall gardens, this is a great example. That's sort of like tired, you know, join at the hip with a platform, right? They are not very often not able to like move, they are like active, like every use in a wall garden, but like more obviously so maybe because their livelihood is tied to that. And, you know, like building tools for like for like for that audience maybe could be like something that the comments could do better, essentially. And Sam, you're a little bit late, but we've sort of gone for quite a while here on on the theme of our dolly and all these sort of generative systems killing off creators, how do we reward them all of that? That's sort of that's kind of where we started. And we don't have to we don't have to stay on that thread for the whole call. We could we could wander or check in or whatever we'd like to do. Yeah, I mean, we just went straight in from the chicken, right? Yeah, yeah. Which is good. That's what we do. Yeah. It's a good time for the topic. It certainly is timely. I think so, yeah. I mean, I think that any good fellowship should include generative members. So I don't think the more interesting question for us is how do we how do we make good on the the opportunity to to have a much more robustly fleshed out world of. Of links. And structure. If I can riff on that, Sam. So I'm really interested in like one of the one of the dilemmas that I that that I chew on and that we're sort of bumping against here is, hey, intellectual ideas are should be used to help humanity and to help us sort our way out of all the problems we have. But we wind up locking ideas away in lots of different ways for lots of different reasons. And IP overprotection is sort of one of the big labels for some for some of that activity. How do we make it so that people can still work together and cooperate to build what we know into some some open space where we can share it out and then and then apply it, which means a whole bunch of other things as well. And so trying to figure out how to how to implement that could be. And a piece of this conversation is making me realize that I'm just trying to get money to people so that they can stay alive and pay for, you know, shelter and food. And there's another piece of it, if you bump up even higher, which is, hey, maybe we're moving into fully automated luxury communism and we won't need actual and the abundance society where we won't actually need money because things like shelter and food will be simple to get and accessible to most everybody in some other way. And that that's a revolution we will likely not live to see but a hundred years from now. Who knows, like larger, you know, changes that big have happened before, like the enclosure movements and the advent of industrialism and capitalism that that that changed the world at least as significantly as what I'm describing here. Go ahead. Yeah, that's a very interesting like a vision.  But they will still eventually remain as a coordination mechanism, right? Because even like in a postcard society, presumably, like you will still need this, for example, how do you attribute creativity, you know, like value to some extent, how do you attribute the authorship if we want to keep around the author as resurrected, right? So so it's it's it may may mutate. So I guess at that point, you need to ask, do we still want to call money something which is like a more generic coordination device? But, you know, it will be interesting to see and hopefully it will happen in our lifetimes.  What does that actually mean to be famous? And what is the value of being famous? And I would say that fame itself doesn't have any value, except in a marketplace to put value on the thing, which is why Steven Spielberg can make any movie he pleases, including horrible ones, if you really wanted to, primarily because he's famous for having made lots of good ones that made money in the past. So he trades his fame in for the ability to do those things. And sure, he'll make a big, you know, minority report every now and then, but he also has the ability to trade it out and force a studio to do something like Amistad or Schindler's List, because he feels like that's a thing that needs to be seen and done. And his fame then, in turn, allows him to put movie stars into those cultural objects that he, you know, if I wanted to direct them, there's no way in a million years I would be given that opportunity. And so I would I think if you talk to DeNiro, he would come act in your movie. But I'm just saying I he might, but only because I have a personal relationship with him. Damn it. OK. Can you invite him to the calls or or really, honestly, I probably would call Pacino first.  Brando's gone. But even at that, he's going to have an agent in his life who's going to say, no, you shouldn't go do your friends thing. Or even in both of their cases, although probably a little more Pacino, Pacino likes doing theater and plays and wants to do that. But it's incredibly hard even for him to do that because of the way the financial system works. And for him to do it, it means reaching into his own pocket to finance it himself, which is an incredibly hard thing for almost anybody to want to do because you know it's a money losing proposition, which is why Spielberg leverages his fame to get somebody else to pick up the ticket. So you get, you know, depending on how Hollywood's working today, you know, you go out and you try and find that the Ili Samaha or the Arnon Milchin, who is running gun money out of some foreign country and needs a way to expatriate that. So you take a piece of his finance stream and you use Hollywood finance to move that money around in a way that, you know, it's Paola and he's losing you know, 20 or 30 percent is a fee for doing it where they are or were. But you know, that's you know, the cost of doing business. And that's mostly how Hollywood works, honestly, is by financial schemes to move money from one place to another. That most people just generally have no idea that that's going on. But you know, there's a reason Dodie Fied was a movie producer. Now, you know, he liked movies, but there were other reasons, other things going on that that caused that type of material. So it's a great analysis, Chris. I also want to recognize a different part of fame, which is the respect of thousands or millions or billions of people over months or years or decades or centuries, which is, which is not fungible, doesn't feed you. But it means something.  He looks at it, you know, early indigenous societies and how they operated. And essentially, we stole the idea of freedom from them to create what is now our society that is kind of a fusion of that type of cultural tribal let's keep it all together and support each other. And we somehow have managed to overlay industrial industrialization and capitalization on top of it, which is a weird or crazy thing. And then Aflantian may appreciate this. His last book that came out, it was written in French in 2019, but it just came out earlier this month is called Pirate Enlightenment. And we have this mythology of pirates from the seven to late 1600s, early 1700s. But he posits that a lot of the Enlightenment movement was put on and subsumed by pirate ships and the structure that pirates actually use. So at the time, pirates were way more egalitarian and democratic in the structure of how their ships worked. And we don't we don't see that because we're working on the mythology of what came after 100 years later. But, you know, he looks at a case study of, you know, a hive of pirates in Madagascar and how they influenced the world with ideas of freedom and enlightenment in ways that we never would have had otherwise. But then they their marketing department was selling something else to foreign government. So that's what we get. That's the picture we maintain of pirates when, in fact, it was they were many democratic societies that were incredibly much more egalitarian than we give them credit for now. Thank you for that. So. So piracy may, you know, we give it a negative connotation, but piracy is, you know, maybe responsible for our current form of government. Two things. The book Under the Black Flag is really good about this. And there's a couple others about sort of insurance among pirates, democracy among pirates, a whole bunch of other really cool things. And then you said earlier that that capitalism or whatever, our modern systems kind of overlaid on top of this other way to do things. I really like my own take on this is that capitalism had to shove all the other things out and neutralize them because capitalism needs a labor force that is willing to kind of go do anything.  So it will do everything it can to destroy the other means of making a living. At least that's my perception, my jaded perception of it reading things like against the grain and dawn of everything and whatever else. So as and part of our problem right now is that our brains are way down the capitalist hole, witnessed the first 45 minutes of conversation for this call, trying to figure out how to reward artists to stay alive and put food on their families. So far down the capitalist hole that imagining other ways of living together is incredibly difficult for people like we can't do it. And you know, when people say well, what alternatives are there, I'm like, well, the Algonquins basically had matriarchal societies where the excess of the hunt and the gather went into long houses and the elder women of the tribe allocated it back out to whatever families needed food or whatever they had. That was the redistribution mechanism and it worked out just fine. And there's tons of other mechanisms and part of the message of dawn of everything is like humans were experimenting like crazy, trying to figure out how to make these things work. And then a couple models sort of lined up dominating. Sorry for the long monologue. That's OK. I mean, we definitely have there's something about capitalism that forces dramatic change in improvements. It's not and it's not just, you know, the industrial revolution, but it was the industrial revolution combined with capitalism that's gotten us much further, much faster. So the question is, how can you still have that but still have those other societies which moved along and didn't evolve nearly as rapidly? And, you know, if you suspect, well, other than the motivation of having more money, you know, having the shift in changes of improvement quickly over time would be a nice thing. We could also go back to those long house societies. Yeah, my hope there is that the interment. Well, I guess we are also on the internet in the field of the link, but like that may be the tool that we were missing. So essentially, like allowing communities to become so sustaining, so sort of governing to some extent in a way that previously could all be done by force or by use of money, right? So, you know, like in a authoritarian way or, you know, through capitalism to some extent. And, you know, I guess we will we are seeing that happen probably. So we are like within the storm. Sometimes I personally feel it's not going quickly enough for me. But here I go back to like some question, which is like, how can we actually, I guess, like, take the opportunity or like, or, you know, like just much, yeah, find a full group, like find the opportunity. And it's like, I don't know. In my mind, I always goes back to coordination point. You know, what is the minimum that we need to get like people to be able to coordinate on general, you know, projects? Yes, but that's that's the, you know, the glasses through which are you the whole thing? It's really cheap now to coordinate. So I don't think we need any more capabilities. We just need the coordination. It's like that it was doing an experiment being replicable in theory and actually being replicated. Right. We do the work. Yeah, we need and, you know, I think we need to get good at creating societies with new norms, which means we need to be creating hundreds and thousands of them. You have to do it a lot. So I.


There was I think one guy recycled due to a medical issue We had one person get kicked out because he lied about having asthma We had people that had to drop out of the Air Force completely because of mental stability So I think we lost about four or something people but beyond that everyone else passed then what for we had a couple people Who got actually sent home due to either a mental illness or some sort of panic attack from BMT And being stressed out. We had a lot of people recycled into our flight. They were washed back a couple weeks due to Zero-tolerant stuff and eventually were kicked out from that during basic training We only lost three people and every time we lost someone it was like losing your brother It was the worst thing ever so we had someone who just quit he just he was done He was tired exhausted. We had another person. He was suicidal That was the thing he would drink cologne and he just wasn't having a good time And then the third one he got out because of medical reasons He he got hurt and he pushed himself too hard We only lost three people and I know that's really low for an average flight to lose three people But every time we lost someone it was just terrible. We were really close as a flight three Two from our initial flight first guy. I think it was a medical discharge Nothing special second guy though ended up squaring up with our MTI and the chow line he was trying to do a right flank into like getting the food when dorm chief posted him and Kept on not being able to do it or MTI kept having redo it having read to it And he ended up squaring up with them and then just leaving the chow hall So he got recycled into a different flight and then from what I've heard I don't know personally tried to fight a guy in that flight And so he ended up getting like out of the Air Force third guy He got into our flight because he was on med hold for a while. I don't know what happened to him I've heard some things, but I don't actually know so I'm not gonna speak on that We did get a guy recycled into our flight though for on the Fourth week call going on snapchat someone busted him for that. So we got recycled into our flight He's a great guy now. He's like he's over there actually. Yeah, so you got recycled into our flight But three two from our initial flight and one came in so yeah So totally had three people get recycled in our flight and the first one was like second week because she Like was complaining about her ankle hurting and then went to med hold and then I think we saw her again like two weeks Later in some other flight. So she got recycled a couple times the other two girls Actually failed their PT test the last PT test six week and one of them actually Didn't get to go to be so this either she went to med hold because she is on a Sunday And she hit her knee on her security door and like cut her knee open and like the medical had to come up and like do Stitches in the dorm and she didn't get back to like 3 a.m. That night from the emergency room. So That sucks and then see how she didn't get to go abuse with us and then she graduated a week later The funny thing is they're both here with me at Keesler and they literally live like two doors away from me So that that's kind of funny, but also cool that they're here. We did have one training recycle He did a couple of things. He came out of bed one night and was looking joking around with one of his friends and slapped on the face and Turned out it was a different kid and he wasn't too happy about that He did a couple of other things like talking me on people's backs a couple of things that you shouldn't have been saying But in the end we all became really good friends with him We understood it wasn't that big of a deal to us, but he had to go unfortunately being recycled and being tease a really big thing There are people that do really dumb stuff something small that could add up or something big Personally my flight nobody got recycled which is a great thing. I graduated with my home flight as far as people getting recycled in my Flight we did have a lot of drama We had some people who were in serious trouble and they were about to get recycled They were passing us the brother flight and stuff like that They actually didn't end up getting recycled But we did have a girl get recycled over her PT test And then we had some girls go to med hold for like mental reasons and things like that We actually had one girl who was like really missing home And she didn't want to be there and then next thing you knew she was getting picked up by an ambulance So it's pretty crazy


Ok, merci, c'est un plaisir d'être ici, j'ai un petit peu de plaisir, si j'ai des choses stupides, pardonne-moi. Je vais parler de séries de travail que j'ai fait avec beaucoup de gens, surtout avec deux de mes étudiants, Samuel Joulin et Piotr Bojanovski, et quelques post-docs, Korean post-docs, il y a Min-Soo-Choo, Bum-Sum-Ham et Soo-Akwak. J'ai été un personne de vision, de vision. Aujourd'hui, la fashion est de faire beaucoup de benchmarking et de montrer beaucoup de tables, donc je ne vais pas faire beaucoup de ça, juste un petit peu. Je ne vais pas discuter des features visuelles en détail, et je ne vais pas prétendre que la façon dont nous implementons les fixations, que nos algorithmes sont les meilleurs, mais je pense que certains des étudiants sont intéressants. Je vais parler d'une vision supérieure, comme d'habitude. Dans les précédents étudiants, nous avons parlé de CNN, des choses comme ça. Je ne vais pas parler d'une vision supérieure tout à l'heure, mais peut-être dans les dernières quelques slides. Je vais parler de la vision supérieure. C'est un problème typique dans la vision d'incompteur. Vous voulez reconnaître les objectifs. Vous avez donné beaucoup d'exemples de entraînement, des images de beavers, de chairs, de trés, etc. Et donc quelqu'un a mis les speeches dans les sacs et a mis le label beaver, le label chair, le label tree. Et puis vous essayez de mettre des machines. Et puis, à la fin, vous vous donnez une nouvelle image, l'un de la droite. Et la machine va dire beaver. Et comme vous le savez, c'est généralement formulé comme un problème de machine d'entraînement. Vous êtes dans un espace. Et puis vous trainez votre machine par computer des fonctions de projection F qui va minimiser l'avantage de discrétion entre les labels qui sont données à votre temps de entraînement et la projection et le data. Et puis vous régularisez pour contrôler les types de fonctions que vous observez. Et puis vous avez l'illustration que vous avez pour les méthodes linéaires où vous travaillez dans des spaces fonctionnels sur le data, mais bien sûr, il y a aussi des méthodes linéaires comme vos networks. Mais pour toutes ces méthodes, ce que vous assumez c'est que vous avez donné les labels au CN. Et bien sûr, c'est expensif. Donc il y a un trend, je ne sais pas votre background, mais il y a un trend d'intervision de la communauté pour les label tout. Ok, donc après vous, vous avez les labels plus et plus de choses. Donc c'est un exemple de M.S.Coco et c'est un exemple que j'ai pris un an auparavant, donc probablement M.S.Coco est plus grand que ça. Donc il y a des gars qui label des images comme celui-ci et qui délinent tous les objectifs d'intérêt dans cette image.  Je n'ai jamais fait ça, je n'ai pas d'idée, mais j'ai fait ça pendant longtemps. Et ils ont fait ça pour 400.000 pictures. Ok, ce qui est bon parce que ça vous donne benchmarks, ça vous donne des problèmes difficiles, ça vous donne une manière rigoureuse d'évaluer les choses. Mais c'est mal aussi. Parce que c'est trop expensif. C'est un peu mal à faire ça. Il y a aussi des biases parce que très souvent, ces benchmarks sont créés pour des tasques réels. Je veux dire que les précédents stocks étaient des tasques réels. Mais très souvent, sur la télévision, il y a des tasques faibles, un bunch de gars qui vont dans leur chambre et qui décident des pictures qui vont mettre dans leurs bags. Et il y a un autre problème qui est assez intéressant pour la vision, c'est le niveau de granularité. Qu'est-ce que vous arrêtez ? Ces gars labèrent ce qu'ils imaginent, les chiens et des gars. Mais ils labèrent le gras quand ils labèrent l'individu gras. Donc cette idée de labèrer le monde n'est pas la façon dont il faut aller. Surtout si je vais parler de vidéo vous devez faire ça pour des millions des millions de pictures, des temps ou quelque chose. Donc ce qu'on parle de c'est de essayer d'utiliser moins d'informations moins d'informations manuelles. Et je vais vous donner un bunch d'exemples, c'est très simple, et on va bouger pas beaucoup d'informations à pas d'informations. Donc, pour commencer, let's look at the co-segmentation problem. So in the co-segmentation problem nobody delineates for you the objects in the training data. The only thing that you hear the task is to segment the image that is to separate it into regions more or less foreground regions and background regions or maybe sometimes multiple regions may correspond to interesting objects but to train in co-segmentation nobody delineates anything for you. The only supervision that you have, it's a weak form of supervision is that you are told these pictures belong to the same bag. They contain something in common may not tell you what. And you want to use that information to segment the images. And so these are some of the outputs of the algorithm which Julien did with Francis Beck and me a couple of years ago and as you can see, it gives reasonable results. In most of the examples I'm going to show you I'm going to show you examples where the method works. Of course, there are plenty of examples where it gives random, not random, it gives bad results. But here it works reasonably well. If you look at the stonehenge pictures for example I find it remarkable that it works so well and I find also strange that picture be the elephants but maybe I shouldn't spend time on them. All right, so that it works. The idea is very simple. It's just to generalise supervised classification. So this is the supervised classification scheme I was showing you before where you know the labels and there's something called discriminative clustering that was introduced in the mid-2000s and that exploited this idea. And the idea is very simple. There are two ingredients. One is that to do this clustering into categories that you don't know what you are going to do is not only optimise over the classifier but you are going to optimise over the labels. So this time you don't know the disease you are going to try to compute them. And since this is in general complicated what you are going to do is instead of taking a complex loss function you are going to take the square loss function. Because when you try to optimise over the classifier you get the classifier in close form it's a rich regression problem. So you knock off the classifier and all you are left is an optimisation over the labels. The labels which is a quadratic optimization problem and this for relaxation you can do efficiently. So at the first question session what we did is we just took that idea and generalise it a tiny bit. We changed the classifier from square loss to a softmax because we wanted to do multi-class things and it seemed a good idea. We added spectral clustering things to group together pieces of images that belong together and then this discriminative clustering method have a problem that they tend to put all the pixels into the same class. So we have done an entropy term at the end that kind of spreads out the wealth and prevents that. Otherwise you can see the same type, read the same type of technique. Of course then you need to see how you can optimise it so you relax it. The problem is non convex because of the entropy terms and so we are going to do block coordinate descent. You need to initialise that thing with reasonable approximation so you do a Taylor expansion near zero and you get a quadratic form that you can do in closed form and then you have to do running. With that you get the results I showed to you. This is an example of the same method applied to three pictures in the same shot of the same movie. As you can see I've got this kind of ideal setting because the emission conditions will be roughly the same they will be the same characters in a movie they will have the same appearance the background will change a bit. As you can see this thing succeeds in finding pretty well the main characters. It doesn't segment them exactly there's a shoulder patch that's missing but who cares. All we care about to extract information from an image is the rough shape of the object. That brings us to video. In the previous case I was relying on some amount of manual annotation that is there are people who come and say well these pictures I put them in a bag because they contain something in common. Here I don't want to have manual exploit metadata that may come with various types of images or videos. In the case of videos it's a remark that was done by a while ago I think first by Laptef and Perez is that you should be able to exploit the script data that comes in movies. So if you take any movie or any TV series this shot after a script and that script that takes content information that's correlated with the visual information. This is an example from Casablanca in the space of the script so as the head wetter takes them to a table and you can see it's correlated to the wetter. I don't remember the name of the actor to their table then she passes some and you can see it's also correlated with some keep his eye on the keyboard. And so the idea is that there is a correlation between a text and a visual content so can we use that text to inform us about the content of the video and figure out what's interesting in the video. So scripts are available for many many movies and many many series. First they always exist but the production companies don't always give them to you but many of them are on the web they enjoy typing scripts as they watch movies and TV series and they put them on the web. But the script is like a piece of paper it's not synchronized with the video. And the other end any film or any TV series comes with subtitles for people who have a hearing and the subtitles contain of course the dialogues but the script also contains the dialogues. The subtitles are timestamped and so you can align the script with the subtitles very easily using dynamic programming and then you timestamp your piece of script. And so that piece of this has the weather you know that the dialogue is at such time and so that what is described will happen in the first second slice of time. And then you can try to use that to figure out what is happening in that particular time. Here is an example you would like to identify characters in a movie and also what they do. And so here the the script tells you Rick's walk up BN Ilza you can imagine that you do some face detection and you get those yellow blocks and then you want to label them with the name of the person and you also want to say what is each person doing. So it's easy to formulate that using the same discriminatory formulation that we have. Again this time we are going to have two label variables, one for character names one for action names and so we have these two variables there will be linear constraints that tell you that the person is mentioned in that piece of the script if that action is mentioned in that piece of the script then that action and that person should appear at least once so you have these linear constraints about that but then you have linking constraints between the person and the actions that both of them should appear at the same time and so those are non linear again non convexities etc you need to be able to to to optimize that so again we relax to block coordinate descent we initialize one of them by taking a uniform distribution of our time and then we run them ok it's no big deal in doing all that so you see the kind of things that you get it takes a while to start again green is good red is bad so the face is found by some of the shear face detector the body is the larger rectangle it's just an offset and scale version of the face and you know it's not perfect but it's a relatively difficult task and sometimes of course it doesn't pick up the people ok alright so often people we deal with video treat video as a kind of three dimensional block of data it's not quite right because video as a time doesn't go backwards most cases and so we need to use that temporal information so that leads us to the next thing here the idea is you are going to get this piece of script we are going to do some keyword mining we decide on a vocabulary of actions so I don't know sit down, open door whatever is written there and then for a given piece of script we mine which ones of these keywords or synonyms of these keywords are going to happen and then we are going to make the assumption that the temporal order in the video and in the scripts are going to be the same that is that if in the scripts they open door before sit down then it should be the same in the video ok how can you formulate that we are again going to write it as again very close to discriminative clustering what we are going to do so here what everybody present thinks we have our dictionary of actions there is an empty action because there will be plenty of frames where nothing in the dictionary is happening then we have the meta data is given by that mapping A that tells you that the cave action in the script correspond to whatever entry in the dictionary and then you will have a mapping between time frames this is a small m here between time frames and your action and your meta data so what you want to do is again optimize your classifier but this time instead of being just the label you are going to have the entry in the table that corresponds to your assignment and the fact that the fact that the temporal order is maintained corresponds to the fact that the path that you have is momentous ok so it turns out that one possibility to the representation of the problem is to mix together your meta data and your non assignment into an indicator variable and then you will have complicated constraints on these indicator variables that are associated with the fact that you have a momentous path ok if you do that you get back to the usual setting and you recognize the square loss et cetera et cetera but the difficulty now is that when you try to do optimization of a discrete domain that is defined by very complicated and implicit constraints so doing that optimization is not obvious ok so when we relax it instead of relaxing it on the discrete domain we are going to relax it on its convexor and fortunately there is a nice way to the optimization of that convexor using the so-called Frankouf algorithm which is an old algorithm that has become popular in a few years in the machine learning community and what's the idea is that you have a convex function defined by a convex domain and that domain may be very complicated and you may not have an explicit description for that domain and yet the nice quality of Frankouf is that you can find the global optimum of your problem if you have an oracle function that allows you to do optimization of the tangent plane to that surface ok and since it's a relative algorithm basically you optimize over the tangent plane and then you take a point not midway but somewhere on the way between the optimum you have found and the current point there's a way to find the scale factor and this thing will iterate and the tangent plane can be done using dynamic programming so it can be done efficiently ok and this is the kind of things you can do we took a bunch of movies we decided on a vocabulary and then we let it fast forward and when the program thinks that it has found some action it slows down and it displays it and of course again very sometimes but I show you examples quite well and those are not easy what we did is that we wanted to evaluate these things they wanted to evaluate these things so we asked somebody to actually annotate by hand where they thought the actions were happening in the movie and so the wider bands at the bottom that's where the person annotated the narrow bands for the program found and I think this is the next example that I like so the yellow thing is drinking the white yellow thing is the person annotating it the thin yellow thing is the program the program finding the action drinking and I think both make sense so for the person this is drinking so drinking are you going to drink, are you going to drink blah blah blah it's a social thing for the computer and both are very reasonable and I think that's interesting because in most of computer research and visual recognition the semantics of the labels is absent so you have labels, you call it drink or seat or potato or banana but they should be one, two, three, four or ABC, they are pure symbols you never use, never exaggerating but by and large you don't use the semantics of the labels ok so it's quite reasonable that somebody would call drink or something like this and somebody would call drink something like this ok I'm going faster, that's good and so so this idea brought us to try to consider the problem totally unsupervised interpretation of image of the video this time no labels here you know it seems easy but it's really not that easy there was no manualization for that movie but still we are using the script here what I'm going to do is some people putting things in different bags I'm putting everything in the same bag figure out for yourself what's in there ok and that's actually the idea of Minsu Cho, a former political mind and and there are several key ideas the first one is this idea that if nobody re-uses the semantics labels are just a way to put things in bags they are just a way to put things together that people think belong together ok so the supervision by and large is used to say these three things belong together so some people may say those are horse pictures some people may say those are horse riding pictures some people may say those are banana pictures I don't care the main things that people think these things belong together ok if that is the case maybe you can replace that supervision process by an automated process that just say well these pictures they look like each other so they belong together so idea number one idea number two is that if you think that way if you take a collection of pictures say the internet and of course we don't do it at internet scale but say the internet this collection of pictures has some underlying implicit structure kind of graph structure two pictures are linked implicitly because they belong they contain things that look like each other ok and it would be nice to discover that structure and if you know that structure third idea then you can use that as supervisor racing ok so that's the whole idea and I think personally I think that's a very good idea now what I'm going to show you is how we do it and I'm not going to say that this is a good way to do it it's a way to do it the same way as before we do all these discriminatory the story which is very simple that may not be the best way to do it but it's a good idea to use metadata etc ah, there's another idea the thing is good is that the is this idea of parts but I will come back to that later so the way we are going to do it is we are going to represent pieces of images by regions so we pick random regions in an image and some of these random regions may enclose interesting parts of the images interesting objects in the images so in this case for example the red one could enclose the bike and some other of these regions may correspond to say salient parts define in some sense of these objects we are finding and then of course some other regions may correspond to random stuff ok so we are going to try to match images by finding regions that may correspond to objects or their parts ok so in reality me and so in the vision community object proposals have become popular so you have some bottom up process that puts a rectangle where it thinks that it might be something interesting personally I'm not so keen on that because I don't see why this should work without any prior information but to be pragmatic it works better by using those region proposals than by using random regions ok and so what we are going to do is we are going to try to bring in when we match these regions we are going to try to bring in some geometric, very simple geometric information to match region so imagine we take two images we have all these region proposals in the first image 2000 roughly we think we take between 2000 and 5000 region proposals in each image we take all these region proposals in the first image all region proposal in second image and then we look at a possible match so a possible match m is a pair of regions one in the left image one in the right image ok and what we are going to estimate is the probability that this match is correct given all the data that we observe so capital small d here is all the data that we observe ok so what we are going to do what we are going to do is try to marginalise I mean rather try to use Bayes rule and marginalising other possible configurations of the two images that is possible offsets between the region in left image region in right image and by offset I mean translation and scare ok if you want you could do rotation whatever but so the configuration C here is translation and scare ok so we marginalise over it and so we are going to probability we will decompose probability of the match given a configuration times the prior probability of the configuration given all the data now then what we do is say well you know appearance and geometry are decoupled they are independent from each other so we can factor the first term into two one that depends on your geometry so the appearance term can be any similarity measure between patches after normalisation of course that you want Euclidean distance you want or correction between sift or whatever the geometry term can be just for example the some thing based on the size of the region after you have applied if you have set those are simple and you can come up with anyone you want the complicated term is probably of the configuration given the data and that we don't know ok so we don't know it so when we don't know something we make up an approximation for it we use what's called probability of transform that's not a very nice term but we use a proxy for configuration given all the data what we are going to do is we are going to take all possible matches and we are going to make them vote for that particular configuration ok so we are going to see if given among all the match if a match is compatible with the configuration we will give a good vote for that configuration and a better vote even if the appearance is good this approximation but it's reasonable and it can be computed ok so that's for a match now let's say only 2 images if you want to have I'm sorry it's not only 2 images for a match now if you want to get confidence for a given region you want to get the maximum of the score of all possible matches that include that region ok c'est un exemple ici 2 images all the candidate regions and then if you color code each region in each image by its confidence you can see that blue is not confident red is very confident you can see that the red regions are going to concentrate on the salient objects we don't know it's a horse if they are 2 horses they look like each other and the 2 backgrounds don't look like each other that's why it concentrates on horses it concentrates on trees instead of horses if they are more structured there and at the top what we have done is just display the top 20 scores top 20 matches rather in the 2 images and as you see they rather concentrate on the score so that's encouraging if you don't use the geometry term you don't get any structure ok what is this one I'm not sure why I'm sorry I'm jet lag I guess it's just because that was a different example and the PHN again concentrates on the thing so this was just for 2 images if you do multiple images well you are going to take the average for each region or the sum of the score in that region in terms of the more images you have the better the more concentrated things become so at the bottom you have between 2 images and if you have multiple images you see that you have a lot more of these red regions ok again this assumes that the images that we are trying to match are linked that we assume that they contain some common thing that's interesting you think that we would like to capture this idea of part that you know you have an object it will be a box that more or less contains tightly contains it and then there will be bits within that correspond to object parts and we don't know how to do that I think I will come back to that a bit later I think but the I think one of the things people envision don't know how to do today is reason about parts in an intelligent manner and so shame because I think that's important I will show you a couple of examples later so here at least what we would like to do is say that you would like if the region is too big imagine you just want the tight box if the region is too big then it will contain more background than the tight box if it is too small then it will contain less foreground than the tight box so we should be able to use that the second one we don't know how to do it so far we have so many but we don't know how to do it so for the time being we just just use the the first one and we define a silences score as the difference between the score of a region and the max of all the regions that contain it and that tends to select regions that tightly bound objects but why not but this actually I think this is an important point trying to understand what's inside what's inside is important and now the algorithm this is it you take a bunch of images and then you initialize it by looking for the nearest neighbors of one image according to some criteria ok so at the beginning you don't have any region matching information you don't have anything so you have to take some global descriptor we use a descriptor introduced by Efros and what's her name and his wife, no not his wife not Efros, by Toralba and his former wife Oliva, here we go your friend, sorry don't film that anyway there's some global descriptor called GIST that's reasonable for finding two images more or less look like each other so we look for the nearest neighbors of course it's a toy example, we can make mistakes in that process of course and then we try to match those ok once we have done those match we have some hypothesis for the same object in our target image and then we can use that to refine the search while we do it all over and then we can do that to refine the search for nearest neighbors it's very simple, I'm not saying this is the way this should be done but it's just a way of implementing but I think it's a good idea so does it work does it work in general in general it works pretty well you get better matches after a few iterations this is after one iteration you get these big three regions up there after a few iterations it concentrates on the object of course it's anecdotal evidence just an example also the retrieval the search for neighbors improves with iterations at the first iterations you get all sorts of random things here after five iterations you get essentially cars which is what you want ok to wait for a few iterations and you are done so because I didn't put it in a slime going to forget if I don't see it now so of course this is ugly we are not even optimizing a function we have an algorithm it does stuff it doesn't lower error of whatever it does stuff this is ugly and of course we are not trying to formalize it as a proper optimization problem so it's interesting is to try so I said I would not show too many benchmarks but the idea is that we took there's a classical benchmark in vision called Pascal VOC07 so we took all the images put it in one bag, check it and give it to the program and then we we we report core lock which is the correct localization my memory is correct oh yeah this is unsupervised so it's always difficult to evaluate unsupervised methods because they are unsupervised so what we do to evaluate it we took Pascal VOC07 where people have put bounding boxes around objects of interest because it's a database designed for object detection and so when we say we are happy the candidate object that we find coincides that is overlaps enough one of the things that have been marked and core lock is aware that it overlaps kind of intersection of our union or something like that and so what's interesting if you like at the top table so this is per class so we don't have this thing or mixing the classes together but there is no supervision whatsoever except for the fact that it's the same class and all the other methods have essentially have some supervision with negative example even with some pre-trained CNN methods and without with very very little supervision we do quite well and then we can also check everything and that's the second table and it still does very well and none of these other methods works for that and so I think that's interesting now if you look at those are some examples of successes those are some examples of failures and these are anecdotal as well I'm not saying that the failures are always like that but a failure that's observed sometimes often which I think is natural we find the head of the cow or the head of the dog or the wheel of the bike what's wrong with that it's unsupervised and if you take mixed pictures and you have faces you know faces are salient things and the face of a dog looks like the face of a man looks like the face of a goat I mean pretty much not exactly you have two eyes, nose, mouth, ears and so it's a salient thing to find and again this has to do with this part thing and I think this failure mode of many detection algorithms what is magic about the body about the box of all the body and so but we don't have a good way of reasoning about parts in general I think nobody knows how to do that but I think it's a really important problem so since we could do it in in still images we tried to do it in videos as well and that's the work mostly of Sua Kwak we mean to show and I forget I should mention Francis Buck or Lee Ashmit and he's also involved in some most or some of this work and the idea we're going to do the same thing as before but we're going to introduce some temporal consistency so I'm not going to bore you with the details but we can do that and then you can do these things you take, we took clips from the Jason Bourne movies and we took clips that contains a lot of cartridges and we let the we let the computer figure out what was interesting and of course there are a lot of cartridges so it collects cars but you know it's moving, the camera is moving the the cars are they are translating but they are turning sometimes they don't move and it works really well and my favorite example is this one we gave the poem The Movie Babe so it's The Adventures of a Talking Pig I've never seen it but it's supposed to be pretty good and the and the poem got pretty good at finding Babe and in general animals and their faces but very good at Babe you know we didn't try to the New York Times to tell them we did that but it's totally unsupervised give it the movie so I'm pretty happy but just written and you find sheep's face and then I think there are monkeys later and if I had pigs' butts and maybe some dogs but you know ok and more babe but that doesn't bother me at all you know what's wrong with finding the monkey's face as Babe's face and of course and of course I don't think you'd want to if you want to deploy an object recognition system you don't want it to be fully unsupervised there will always be some metadata but I think it's interesting and important to look at this unsupervising and what you can do without any prior knowledge so these methods can be extended I'm not going to the technical details but the way we treat geometry here is very important to have geometry makes a huge difference but it's too global because we get everything to vote so we also introduce a local voting procedure and I'm not going to describe and that improves things quite a bit and we also use that to match images to obtain what's called sometimes scene flow you have optical flow optical flow you have two images of the same scene you get the same scene but the camera has moved a bit and you want to see how the other pixels have moved in the image optical flow scene flow is when you have two images of the same category same type of objects but they are not the same same object and you would like to find the deformation from one image to the other and you can use these methods basically because you can do this region matching and then you can interpolate the deformation between the regions and this is the example so what you get on the left you have two pairs of images and what we are trying to do is deform the first image in the pair into the second one and we overlay so here are the deformed versions overlaid on the image and we compare our method to a bunch of classical methods for that that problem and as you see as you see it does pretty well and interestingly enough I think it's essentially the only method that gives you a different image that looks like a real image maybe it's even more you can probably see that on Dalmatian and even on the hand and for example for Snoopy it actually overlaps the thing very well there's just the arms that can be well because they are not in the same configuration and you can even match drawing of a pizza to the pizza itself pretty well so I think that was interesting so what we are doing I'm almost done so it's funny because we are starting on service but now we want to go back to very supervised so we'd like to use the same kind of ideas but this time you know a priori that we measure in the same category you want to match them and we'd like to learn the matching process so it's not too complicated you can this was our semi-probabilistic approach now we are going to replace all the probabilities or proxies for matrices by similarity functions we can rewrite all these terms the same as this and then the W here will be the parameters that you are going to try to learn so I'm going to try to learn similarity functions while having some geometric geometric constraints over them to turn that we could have done it there as well it turns out that when you do it you have some kernel that appears in the middle but you have basically you get similarity function based on delivery your match similarity function some geometric kernel and some neighboring neighboring matches and then that's ugly what I don't like about CNN papers they are horrible you know where people put all these layers they have 5 confnets 3 max pooling 2 fully connected whatever 64 I don't know so for me there's a black box and this picture is a bit too complicated for me because I don't understand but apparently the dotted thing helps but the solid thing I understand basically all this is you have a black box that computes f ok and it's a black box that's differentiable so it's called net and then what this does is compute the the function I was showing you in the previous slide and then you optimize and then you get something that can match match match regions so we have to there are not many methods that do that it's an old idea to learn too much I've come back to 10 years ago paper by Yann Leucan we are trying to learn stereo matching there's paper by people from Stanford people from Stanford and NEC that's trying to be essentially the same idea but it doesn't have the geometric the geometric knowledge so as expected we do better what's interesting this is interesting I think this is interesting this is yeah we have I told you that we have this localized way of using geometry and we have not implemented it in this approach yet but we have seen that if you if use the features that we have on in in our previous approach instead of using sift or whatever then we get a lot better results with the localized thing so that means that when we implement the localized thing in this network we should do a lot better so I think that's kind of interesting and and I think I will stop there so it was just a short tour in our experience I think it's a key component of what we should do in vision today thank you questions yes I like deep learning but I am not obsessed by it and it doesn't work for unsupervised unsupervised so which is good and so I that's pretty much it but I like deep learning I think it's good so we have to convert ourselves to artificial intelligence for several reasons but I think it's very good that some of it has moved to industry if industry no no no but if industry believes that this was a done deal they wouldn't sponsor us they wouldn't hire our students they wouldn't try to work with us they wouldn't have research labs etc so I think most of the innovation still comes from academia and I think so again unsupervised and I don't know I'm not the useless follower of CNNs in the world and there are people who are much more knowledgeable than me but I don't know any work unsupervised that work in that area I think what I have told people a lot is if you look at the old CNN papers they are not just doing CNNs if you look at the the paper of the Tinture Use-Le-Net they have something called graph transformational network that are structured that are a lot more complicated where you have differential blocks that talk to each other and that's how they manage to read the checks not by using the net but using that stuff so I think the idea of having differentiable learning systems that you can learn N2N is good of course I think you need to look at I think more interesting block structure where you understand you understand a bit how you program that thing I think just doing this is not so interesting but so I'm not pessimistic at all I think we are in good shape I think it's good we are having an industry an impact at the same time that there are plenty of interesting problems left I think maybe I'm naive Thank you