2002 and so in a lot of ways this has kind of become you know sort of a regular and fairly important part of my calendar of a year and in fact it's far from the first time that my birthday has overlapped in fact a year that I will probably never forget in 2003 for lots of reasons in Perth we actually had the banquet and the keynote that I gave was on my birthday so LCA and my birthday all get tangled up and thank you very much for recognizing that this morning before we jump in to the Q&A though this week as I've talked to people I've realized that there are a couple of you know loops that I should try to close first of all for those of you who were at LCA last year how many of you sat in on the talk that I gave or have watched the video since I'm really pleased to report that a year makes a heck of a difference we're able to move into the new house built on our property at the end of September and the dust is literally starting to settle as is snow it's that time of year in Colorado also earlier this week my good friend Evan Moglen was asked about the status of freedom box and had this to say I'm pleased to report that trading a little sleep for other things last night I can tell you and to just be abundantly clear about this the only contribution I've made to the freedom box o.3 release other than talking about it this morning is one package uploaded to Debian a couple of days ago since I arrived at LCA there folks like Sunil Mohan and Nick Daly and Peta Reinholzen who've been carrying the torch over the last year or so and I'm really appreciative of that you know go take a look at this there's still lots of things to do if this is a project that you're interested in would like to help with I'm sure they still love to have your help so enough about those sorts of things and me and all of that well maybe not quite this is not the first time that I have shared the stage at LCA with these folks given the little you know ticket selling thing this morning if my beard is worth on the order of $40,000 Australian a 3d printer's got to be worth something more than that but in fact if we go all the way back to 2003 part of the reason that year is so significant to me is that at that LCA Linus and Tridge and I were asked to do something that hadn't been done until then at an LCA which was to just take questions from the audience for a while and I have said to the organizer several times since that's one of my personal favorite sessions at an LCA ever and part of the reason for that is that I actually like taking questions from people and I like having some sense of what other people want to hear about as opposed to just standing and talking for a really long time the other part was I think that was actually the first time that Linus and I or Tridge and I or indeed the three of us all together had actually shared a stage and that has sort of set the stage if you will for lots of things that have happened since of course there's some people who just couldn't leave well enough alone and I think within two questions yes that's my cue there so I was reviewing the audio from 2003 which how many of you here were at the 2003 Q&A are fantastic great so can anyone remember you know who interjected in the first two questions rusty right so the first two he interjected and then we realized that it was inevitable and we called him down to the front to join the panel and we thought we might as well just short-circuit that now because thank you Rusty you noticed we already set up the fourth chair ready for you so the other thing we had at the 2003 event was we had chopper chops and so there will be again chopper chop rewards for questions so if you want to chop a chop you have to ask a question and that also happened in 2003 although it was Linus that tried to grab one at the beginning and we told him that he had to earn it first so you'll get one at the end latest right so shall we start off then any more photos Peter no that was all for the photos but we also have an audio recording of the session from 2003 Tridge I think you thought there are a couple of questions from that that might be worth revisiting yes indeed so we had a few questions so how many of you actually listened to the link that I sent out to the chat list the audio right so there was a few pertinent questions I think that deserve an update and answer from you know 12 years later to see how it's going so the first question that was asked was when is 2.6 going to be released so do you remember when 2.6 was released I have no idea this is why I have source control these days so and there was a follow-up question which you know is 2.5 ready for users to try yet and I think we need an update on that do you think 2.5 is ready for users that was such a huge disaster that we changed the whole development model so I think that's a no that's a no right okay so there was a couple of other questions that I thought were worth worth revisiting one was about trusted computing and signed code and so remember the audience asked about what Lena's thought about trusted computing and signed code and so just just to give what you answered at the time you said that you're asked a vendor if there is an actual use case for a signed kernel and the answer was no and you thought it was just marketing so would you like to update that answer so these days will be to do it even without vendors asking for it we use it internally and I use it internally for example for signing all my modules and I think it's a great use especially the way I do it which is to generate random keys that you throw away after you've generated the modules which means that nobody can ever try to insert anything in your kernel so it's a great security measure it can obviously and I think that's something I've said for a long time although I have no idea what I said 12 years ago is that a lot of these security measures that are vilified in this community are actually can be very useful and sometimes they have really good reasons for being used even if some people tend to hate them intensely but but things have clearly changed we we do use it ourselves and I use it personally all the time so the final question I think is worth reprising from the 2003 event was about IPv6 and at the time if you recall your answer was that IPv4 is good enough for most people because you don't want to give you a shoe and IP address and so you also said that IPv6 is a decade away now 12 years later it's probably less than a decade away and most people here probably use it on your cell phones I don't actually know how the New Zealand telcos work it turns out I mean IPv4 clearly is still majorly useful and and while IPv6 now works I give it at least a decade before IPv4 is actually gone if ever we'll see it's I always thought that IPv6 was way over designed and it made it more complicated and more painful that it needed to be and private networks and things like that means that not everybody needs to use a public IP address anyway and a lot of people do for no good reason so your shoes if they do want an IP address maybe they could use one of your private network IP addresses and you can still use your shoes with IPv4 so I think they're the key relevant questions from 2003 so you may have noticed I sent out an email to the chat list yesterday asking for people to think up good questions for today's session and so how many people have got questions for today's session a few people right so I'd like you to come down and stand in front of the microphone down here to ask the question and if there's any applause at all you know at the end of answering your question you get a chopper chop so first person come please come down to the front and ask a question yeah so sort of come down the go round the back or up the front if you want to just and queue up in front of the microphone here morning so Linus nice easy question to start with over the years various people including myself have either reduced their involvement in the current community or steps away from its entirely due to the tone on LKML and especially your contribution to that why do you continue to argue that being really unpleasant to people's good leadership strategy part of it that's not actually my real argument my real argument is that people are different and I'm a really unpleasant person and that's what it really boils down to is that and some people think I'm nice and and some people are then shocked when they learn different and I'm not a nice person and I don't care about you really I care about I care about the technology and I care about the kernel and I really think that a lot of projects in the open-source community sometimes care about non-technical things too much and and the reason I say that is that the only thing we can actually agree on ever tend to be technical issues and if we start making a big deal about non- technical issues that are important don't get me wrong I'm not saying they're not important I'm saying that whenever we start making non- technical issues the primary issues that just guarantees will never agree right and I'm not trying to really make excuses I'm more trying to explain that this is where I come from and and I appreciate the diversity in open source but to me that diversity is not about gender it's not about skin color it's about people are different and people are different in what they are interested in people are different in what they're good at skin coloring gender and all these issues that get brought up as really important things those are details what is great about open source is that some people are unpleasant but they're technically really good right some people are pleasant and like bringing other people in and and I think that's one of maybe the most important part about open source is that you can do what you're good at so when you look at bringing in minorities bringing in females bringing in people who don't speak English my argument has been and is that we should look for people who are good at being the people who can be between other people right there are lots of good kernel developers who are great at working with people and they may not even be great technically and that doesn't matter because we all have strengths right and this is my argument and I refuse to change well that sounds bad it's not that I refuse to change it's that I don't think I want to try to bow down to what other people think my personal personality is pretty abrasive I love arguing but bearing at the Museum of Technology I was I spent the whole time just arguing with people right I mean and if you're that kind of personality when you are on LKML you will argue and I know there are other kernel developers that are like me and I know there are other kernel developers who not like me and that's all wonderful and that was a really long answer to a really simple question but the fact is that's how I feel and please introduce yourself at the start thank you good morning my name is Tommy Richards and Matthew you kind of stole my question well maybe I can I don't work in kernel development I'm not subscribed to the Linux kernel mailing list that I am in the community and I've noticed a worrying trend which is that over the last year there's been more hate there's been more abuse more vitriol in IOC and mailing lists on Google plus and speaking as a professional this really concerns me because I love my job I want to love it in 20 years time but if this trend continues I will not be in the industry in 20 years time so I'd like to know then this is already answered but maybe the other three of you could maybe tell us you all have a lot of influence in this room and in the wider community do you have any plans to try and improve the community for the I think we have other people who want to answer that I'll keep this quick I think basically what you're saying is not actually true I think the tone on LKML I don't know about IRC maybe people in IRC other masters and really it's nasty LKML I think has been getting better what has happened is that there's a lot of system day there's a there's a lot more talk and it's become a thing that is a newsworthy so I actually think LKML has in many ways calmed down and then people try to make a big deal out of it and and make news out of behavior that is not different but does it mean so you know Rusty mentioned system day those of you who have been paying attention will realize that I don't know my life and that of Keith and various other folks who are here got really tangled up in this whole and it's just a debate and it really came to a head in the Debian community in the last year in case you hadn't noticed the interesting thing about that community is that yes we ended up in a situation where we had a question that was partially technical and partially not so technical and emotions ended up running immensely hot particularly for some of the core participants in that debate slash discussion process and you know emotions got hot to the borderline nuclear level for a little while and it was immensely uncomfortable as you know one of the folks on the wrong end of some of the vitriol around that uncomfortable I'm being polite because you are sort of all my extended family here and I won't say things to you that I might say to other folks but the really intriguing thing is all of that vitriol represents what I have described before is the vocal minority and we do have a problem in the entire free software world where sometimes we are too willing to let the vocal minority represent us and to sort of set the tone or the agenda for some of the discussions I'm here to tell you that in the Debian community in particular the community is strong the community is healthy the community is recovering brilliantly from that whole ruckus around the init system today work is progressing the next stable release it's coming along pretty much on schedule you got a little extra time to help us chase down the remaining set of release critical bugs and at the end of the day part of what I think me and us might be trying to sort of get at around the edge a little bit is it's okay for us to have times where we all let our passions overflow and get in trouble with each other on sort of personal interaction and communication spaces as long as at the end of the day we somehow figure out how to make things right with all the folks that matter in the communities that we're working in and I really hope that if nothing else all of us here today can go forward into 2015 conscious of the fact that a lot of what we read and a lot of what gets amplified in the press and on blog posts and all of that sort of stuff is the opinion of the vocal minority and many of us would like to be in the silent majority are still just fine everything's good we're going to do it we can to support the friends and diverse friends around us and let's not get so distracted by all of this if we can help it that's the thing that would help me to feel better about some of the stuff we've had to deal with over the last while and I have to agree with Linus I think we have to be very very careful not to assume that the amount of noise we see in the press and elsewhere is truly indicative of what it's like to be part of most of the open source development communities that we know of yeah so as someone completely outside the system did debate I haven't seen any escalation in verbal but you know the trend has been pretty much downhill I think for the last at least five years right and by downhill he means better better yes I think we ought to give a shout out there been there been important folks in our community who've been doing really great things because it's what they're good at to try and help improve our openness to diversity candidates I'm really pleased by the diversity sponsorship at this and other recent LCAs and the fact that that's brought some interesting people into the LCA community a shout out to Karen Sandler and the other folks involved in the outreach program for women that has generated some really interesting new folks in the community who've done really useful technical work and are getting jobs on the basis of that and all those sorts of things we've seen the same thing you know we've seen a number of projects making concrete efforts to try and increase their openness to contributions from different and diverse folks it's not that there isn't a problem out there somewhere I think we just have to be careful about focusing on what the problem actually is and recognize and reward when things move in the right direction instead of spending so much time pointing fingers when somebody gets a motion next question so is it the year of the next stop it I should interject that we did have that question in 2003 and rusty you actually answered that and your answer was that by the end of 2004 it would be 50% Linux desktop in the world so how's that worked out for you rusty you know it's still so difficult to measure it's probably 50% of my desktops you are running extrapolating that it's all good and in in somebody letting's desktop on your desktop or excellent and on smart phones we are 80% okay something slightly different from the previous ones those of us have a certain age gone to computing mostly I think from the old AP computers and I remember like there were picture books for kids that taught advanced basic and even machine language now that the world's a different place now and there doesn't be anything like that some people are trying like you know Raspberry Pi anything but I'm just wondering what your thoughts are on getting the younger generations into our community because it's all about learning software literacy and understanding how our computers work and it doesn't seem to be there's much encouragement for the younger ones to join our community and basically fill the ranks as we sort of age so by younger ones you mean under 30 I wish I had the answer to that because I got three kids and none of them seem to be interested in technology except as users well that's not true but at least in computers in the sense I'm interested in computers and I think part of the problem is there's the technology of today is so complicated that for somebody like me who is interested in kernels it used to be easy to get in at at the hardware level and that's simply not true in computers anymore and the world has moved on and we're old farts for a reason that said it's much easier to get involved in many other ways maybe you're not into computers and you were into the whole 3d printing thing and maybe you're into doing these user application things and they're way easier to do than they used to be so I I don't have an answer I at the same time I don't really worry it's not the same thing anymore there's no point in teaching everybody machine language or basic or something like that I I'm looking at most young people tend to be pretty tech savvy right so I wouldn't really worry they may not be kernel tech savvy but we don't need everybody to do that yeah I think there's also this immense trend that we've been aware of in effect there've been some talks around edges of this here at this conference and that's towards accessibility of microcontroller platforms and other places where kids are really enthusiastic because they see a direct connection between little bits of programming and action that affects the real world I discovered a really long time ago well before I had kids of my own and it's something that I've enjoyed playing with them with that you know on some level computers are kind of boring I mean there's all sorts of things you can do with them and my kids are certainly far more literate and quick to go you know grab a computer to do something than even I am sometimes but there's this immense sort of feedback loop thing that works well if you even get to the point where they're turning an LED on or off or getting a servo to move or so my strong suggestion if you've got kids or you're trying to figure out how to get younger people enthusiastic is look for places where there's an intersection between simple computing devices and real world effects and feedback loops and you know if they figure out how to write a little bit of code and an Arduino thing or something then they might end up thinking trying to do kernel development is interesting someday I'm not sure whether would but hi I'm Simon my question is where do you see system D being in the next five years is it just a new is it just a new you dev or maybe dev is I don't know why the system big question comes up because I don't even care I am a system be user and I have not been involved in any of those arguments at all that I know apart from not liking one particular person and I'm pretty sure be Dale doesn't want to take that question so I use system D most of you probably do use system D it's all good I didn't care either until yesterday apparently they've got masquerading support and I looked at the code that the patch that actually put that in they don't fork out to IP tables because that would be so they use it directly and actually the code fairly sane so you know as long as the code is pretty good it's I don't really understand so just to be clear the real reason I don't want to take that question is I don't try to predict five years in the future about much of anything and you know the events in my life in the last couple of years have made it clear that trying to predict too far into the future is you know that way lies madness part of the reason that I ended up taking the decision that I made in the whole Debbie and system D thing is that it seemed to be the place where there was the most enthusiasm activity and good stuff being worked on for the future and one of the reasons that we worked hard to constrain the actual decision that we made in the Debbie and project to the next stable release cycle is we have no idea what will be the hot shiny right thing to be using another release cycle later my anticipation is that people continue to poke at it and do good things and and hopefully it ends up being you know code that we're all okay with okay my question is for Linus when you do like the current version control use road kit and you didn't like your doubt you don't find a good dive lock so you wrote your own dive lock what are you working on now that is not on the kernel are you hacking on something nice I'm not honestly I'm a lazy person which is why I like open source I get all these people to do all my job I don't really want to have other projects what I want to do is just sit sit on a beach and sip a foofy drink and just let you guys do the work and the fact that then occasionally I've had to say okay in order to be lazy for the next five years I have to do something that's why right now the thing I've always cared about has been the kernel and right now knock wood or something the kernel development process seems to be working really well and we don't have any huge looming concerns and I hope I'm not wrong about that so so we seem to have found a model that is working is getting us where we want to go and and we're I'm coasting well right now I'm coasting then the merge window comes along and it gets crazy for a few weeks again but so I don't have any projects I'm working on I'm hope to next week work on my dive log thing that I started but I gave away but that's even that hope is more like I hope things work so well that I don't need to thank you we've had a heretical question here I was just reading it the day about this new machine the machine that will have maybe petabytes of memristor so you don't have files because everything permanent and memory the same thing therefore Linux can't handle it you need a much better offering system that doesn't deal with files but there's many structures I wonder what you have to say about that Linus call me one so well known member of our communities just rejoined HP in order to work on the Linux port to the machine the one thing that might be worth I mean if Keith wants to come up and speak but I don't think he does one thing that might be worth mentioning is even when you have persistent memory and you can imagine a world where everything is in RAM and it never goes away that won't necessarily really change things as much as some people say because you will still want to have a file system just to organize this thing persistence also means that I mean one of the advantage of RAM and it goes away is that you do something and then you throw it away and that means you don't have to manage it as much because it the fact that it's ephemeral means that it's also something you don't need to care about long-term if you have persistent from or RAM memristors or any other kind of model you will still need to have a file system just to have a way to organize these things that stay around for years and years so it won't change that it will change the details in the file system when you have bite address ability you can do things that you can't do with disks but we already see that with SSDs that just the performance differences mean that you do different things in file systems than you used to do 20 years ago or 50 years ago so in some level one of the things I would suggest is that what's really interesting about the machine is that the intersection of a couple of parallel long-term technology development trends that are going to hit the market at about the same time are going to give us the opportunity sort of all at once to throw more existing assumptions up in the air possibly back them completely out the window then normally happens when we're watching sort of the steady evolution of technologies this is absolutely right the introduction of SSDs change the way we think about performance in a storage subsystem right I mean you know gone are the days when you know elevator algorithms for managing you know block transfer sequencing were fundamental to being able to get any kind of useful performance out of a storage says the needs the concerns the pinch points just moved it'll be different but it won't be it's not like the whole world magically got simpler suspended resumes going to be really fast there's been so many technical questions I want to ask a meta question instead so you can be purely opinionist given that Australia New Zealand technically share the same airspace do you think that they should come together and form a combined space agency given that we're the only G20 nation without one yeah I mean we technically all share the same airspace and it's expensive and I think we all need to get together to really go into space I'm really the wrong person to ask all the people in this room I'm probably pretty low down on the list to really answer that question right so we already had a question about the machine and Linux++ and things like that do you think there'll be any innovation in hardware in the next few years whether Linux kernel possibly couldn't make the transition very quickly and what do you think the implications would be for all of us more of our work as a whole if Linux loses the dominance it's achieved over the past decades I wouldn't lose sleepover yeah and I really don't lose sleepover in fact I think in the computer industry in general the big question is not what happens to software in the next few years but what happens to hardware and when scaling stops and it's not happening in a few years but it's happening in 10 and it's getting much more gradual and maybe somebody will come up with something really radically new and there are people who argue that hey we've been saying that scaling will stop for the last 50 years and it hasn't stopped for the last 50 years it will stop people who believe in quantum computers are before after the year is it going to stop before or after the year the Linux desktop I really think what happened is the Linux desktop scaled down and now we have it in our pockets so in that but but we're really at the point where in ten years five nanometers that's not a lot of atoms and the whole computer industry I mean software to don't get me wrong but the whole computer industry has for a long time lived on the notion that the computer five years from now will be fundamentally different and faster what happens when that's not true on a chip level anymore you can still improve right you can do clever things and you can improve but it's not going to be the same kind of improvement and we've seen some of it already but it's going to get much worse and that will impact software too but I think it will impact the hardware people much much more and that's going to be interesting in 10 to 15 years what happens I might mention there was a similar question actually in 2003 to that which I think I was worth mentioning at this point there was a question about somebody in the audience said that in that 20 years before so 1983 they had a computer on their desk that could write letters to their mum and print and now you know 20 years later 2003 they had a computer on the desk that could write letters to their mum and print you know what's changed that 20 years now the answer at the time from beta actually was that it's got slower that's what's changed but I asked a couple of people about that before this and they said well what's changed in the last you know 12 years and the answer for several people was it's got faster again it was two or three weeks ago there was this big surprise that it turned out that many current developers don't know how memory allocation works I think LW.net wrote too small to fail that problem so I thought to myself then well 10 15 years ago the answer to that problem would probably be to have RDFM the question today I would ask which fine manual we should have read do you think that documentation of the kernel is appropriate do you have any idea how to improve that well I clearly have no idea how to improve documentation the kernel this is where others sometimes step up and do a great job of explaining small details at the same time there was a talk yesterday about how programming should be provably correct and it's it would be a great thing if we could prove correctness and that we all suck at it and and I don't take that approach I'm more of a touchy-feely guy I actually I believe in biological processes and I believe in evolution and I believe in all these making mistakes and trying and we don't have documentation for how humans work but humans work really really well except when they break down and I actually think we're at the stage and we have been at the stage in kernel development and in a lot of technology where nobody really understands everything that's going on the VM is complicated deal with it and just realize that the answer to that is not to make the VM simpler because the VM is complicated for a reason it's doing all these crazy things and a lot of them are heuristics they don't have hard reasons for it there often is no this is what you need to do this is the correct answer a lot of the time it's like we want to go in roughly that direction and we don't even know how to get there but if we do this most of the time it works that's how a lot of real problems in real life are and that is hard to document when there is no real answer and we should continue to document the big rules we should continue to document the details like how memory ordering works which was another subject that came up yesterday but that still will never be in the situation where we can give a spec for the kernel and I don't think we should even try we should we should just realize that okay life is complex and we do the best we can and we need to have smart people but we also need to have a lot of people who test a lot of people who are willing to accept that I don't know what the solution is but I know what is better than what we do today in this area and if you continue to do that this is how biological systems work and it's a provably working process it doesn't get you to the perfect end result but it gets you to something that works better than anything else we've ever seen so when I wrote netfilter the first time I actually wrote a hacking how-to that had always details on how it all worked and how you could extend it and everything else and the first time I received a major patch I was really excited the patch was great and I said what did you think of the hacking how-to and of course he said what how to and so the lack of documentation does provide a filter for contributions it's terrible but it's true because they have to read the code anyway because the documentation will lie so why not skip the first point and make them read the code immediately because they'll have to deal with it so in practice I found that effort put into documentation in the kernel was not paid back I actually disagree I mean we have areas where we document small details like locking rules and and these kinds of very specific and very targeted areas we do need to have documentation because they are they are things that people can't keep in mind and they're kind of big enough picture that you can't look at one piece of code and see what you should do so we do need documentation but I think the argument is documentation is not a panacea it doesn't solve everything it it's not always the answer I I notice Tringe has not been answering a lot of questions you've been doing a lot of moderating and throwing out chup-a-chup so I'll crack out an old chestnut that everyone can can answer current desktops you know two three KDE Emax what do we run yeah all right okay that's checked from time to time and changes from time to time okay you don't want to start so you know this isn't my badge you know Emax get I run Ubuntu trustee and I've switched between a number of different you know window managers but actually you know a lot of my work is on embedded systems these days so most of the time my desktop is actually GNU screen and running GNU screen on a system that often the systems are on another continent and so really the closest thing to a desktop I have at the moment is GNU screen WN stable XFC e4 and a lot of terminal windows I'm on Fedora 21 right now I'm actually known for being a GNOME hater I don't like some of the development models they have but I am a GNOME 3 user with a couple of extensions to fix what I thought were big failures and I don't believe in Emax or VI so I use this one editor that nobody is saying would ever use but it's what I grew up with so I just fixed that a bug where it would crash if you had a big screen and more than 128 lines on the screen that's the kind of crap I run I run stock Ubuntu on my desktop but my build machine is arch and and the only way to manage that has is when I SSH in SSH-X otherwise it's all text I'm Dave Lane just have a question this probably applies to all of you but I guess primarily for Linus in this case do you all see yourselves being at some point in your leadership roles usurped by a computer and if so what kernel will it be running are you in your leadership role going to have that role you served by a computer and if so which kernel will it be running I don't know will it be heard yeah I think my mind will be going before that happens but the argument there is when that happens and I don't know I think there is actually a partly serious answer one gift that that question and it harks back a little bit to some of the social interactions something I've noticed in recent times is processes for doing software development are becoming more mechanized and so the so often these days you know you're on you know github or you know get orious or whatever and it's like a pull request and it's just a patch and you respond there there's less social interaction you might as well be you know talking to a computer because it is getting more and more mechanized that process and I think that feeds into you know I've never seen any you know really heated discussions in a pull request chat and so it is sort of moving to be a more mechanized process and you know being more computerized in a way so if I end up being replaced by a computer I don't actually sort of care on some level as long as it's running free software yeah hey my name's Laura Bell it's actually my first time at LCA security specialist while security bugs have always historically been found in Linux and open source products in the last 12 months the way they have reached the world through mainstream media has changed dramatically and you know our bash bugs of the last 12 months have kind of changed the way most people who don't think about security interact what are your predictions for security in the open source Linux space for the next 12 months and which areas of code if you were going to go and look at one right now for vulnerability research which would you start with Bitcoin if you knew which piece of code to look at for security issues we wouldn't have security issues it's going to be something very obscure and stupid and it's going to be obvious in hindsight and people will say how did we ever miss that and that's what it's always going to be the security is just the hard problem well I think you're absolutely right and this falls into another one of those I'm certainly not going to try and be very predictive categories I'm really pleased that at least part of the response to this sort of escalation or change in the way things are being reported and talked about over the last year so is that there are some activities attempting to sort of shine more light on put more resources on some of the core elements of technology that we all depend on I'm referring of course to things like the Linux Foundation core infrastructure initiative and so on I don't actually immediately know sort of how that's going but the idea that we're at least trying to figure out how to get more eyeballs on some of the things that we all agree are some of the most important elements of code in our infrastructure and that we found the corporate members of our extended community willing to empty up to help make that happen is is pretty cool and I hope that's a good stuff there's another thing that's been going on which I find personally very satisfying which is that people are less willing sometimes to kind of brush the problem under the mat and leave it up to vendors and have disclosure like infinitely long disclosure times I'm a huge believer in in just disclosing still somewhat responsibly but security problems need to be made public and there are people who argue and have argued for decades that you never want to talk about security problems because that only helps the black hats and the fact is that I think you absolutely need to report them and you need to report them in a reasonable time frame or reasonable for the carnal security list is admittedly five working days which some people think is a bit extreme and and in other projects it might be a month or a couple of months but that's still much better than the years and years of silence which we used to have so I'm gonna get the last question which is a real privilege as one of the organizing team as of now as of today and perhaps looking forward a little bit what do you feel is your biggest legacy to the open source movement that posting about the first release of Linux or the momentum that caused you to release get or something else me personally I actually think one of the things that I Linux has been really good at and I'm and this is going to raise that few hackles right I I like open source and I like this whole working together with commercial companies and this whole notion that you don't need to vilify people who also do closed source so for me personally one of the big things I'm happy about is that I was part of the group I mean and Linux was a big example for the group who who tried to take and now this is one tridge will stand up and give the other answer who tried to take this very you or us against the world approach of free software and made it more open not just in name but also in being acceptable to people who don't necessarily believe in our values but believe that our model is better and and that's to me something that that Linux was really instrumental in at the same time I'm really happy about get to because I think get has spread more than than the kernel in some respects and and maybe I'll be remembered more for gift and Linux we'll see if I would answer that on his behalf I think that the thing I think the long lens of history might find the most important is how both of those projects have positively impacted this notion that all of us as human kind working together collaboratively leads to better results than when we're all sort of hiding in separate corners and fighting and competing so we need to wrap up there but it's been a fantastic session and do you think that our panel has earned a chopper chopper


OK, I think if we should make a start. Good evening, ladies and gentlemen. My name is Peter Bruce. I'm the physical secretary of the Royal Society and one of the vice presidents. It's a great pleasure for me to welcome you all to Carlton House Terrace, the home of the Royal Society, and to this Bacariant award lecture. Now, let me just say a little bit about the Bacariant medal and lecture. It's one of our premier lectures. It is the premier lecture and medal in the physical sciences. It was established through a bequest of Henry Baker, who was a fellow of the Royal Society, who made a bequest of £100, and I shall read you the remit, if you like, for the Bacariant, as written at that time. For a narration or discourse on such part of natural history or experimental philosophy at such time and in such manner as the president and the council of the society for the time being shall please to order and appoint. Now, you'll gather that it's not a relatively recently introduced lecture and medal at the origin 1775, so it's been around for quite some time, and the medal as well as the medal, it's accompanied by a gift of £10,000. Now, it's a real pleasure for me to introduce the medal winner and our lecturer this evening, Andrew Zimmerman. Andrew is one of the principal architects of modern computer vision. His work in the 1980s on surface reconstruction and discontinuities is widely cited. He is the best known for his leading work in the 1990s establishing the computational theory of multiple view reconstruction and the development of practical algorithms that are widely used today. This culminated in the publication of a book in 2100, a turn of the century in other words, with Richard Hartley already regarded now as a standard text. His laboratory in Oxford is internationally renowned, and its work is currently shedding new light on the problems of objective detection, object detection and recognition. So without any further ado, you want to hear from him, I'm sure not me. Let me welcome Andrew to the stage to present his lecture as you see the title there, Computer Vision Learning to See the World. Andrew. Okay, good. Thank you for the introduction Peter, and thank you for the prize. It's a great honour for me, it's a great honour for the computer vision field as well, so very grateful. I'm going to talk on computer vision learning to see the world, and I'm going to start off by saying what computer vision is. The aim of computer vision is to extract visual information from images and videos so that the computer can understand images and videos much like a human would understand them. That's what we aim to do. So in this image here, we'd like to be able to answer questions like what is in the image, the objects in this case of person that's in the image, where are the things in the image, meaning the spatial layout of the scene, the pose of the person, and then what is happening. Nothing's much happening at the moment, but if I play the video, it's more interesting. A clip from Singing the Rain. The objective is to be able to carry out tasks like this to answer these questions. The field has made quite some progress. We can do quite a few things now, especially since you had learnt of deep learning. I'll show you some examples. The first one is I'm going to show you a video sequence in the top left and then various views of the 3D layout of the people and what they're doing as they collide with each other. That's the where task. The next one I'm going to show you is the what task. Here you're seeing object detection and tracking. These boxes are detecting the objects. Moving through the video, that's tracking it. There's also recognition going on. Maybe you can read the labels, but it's recognising various animals and objects. It recognises the tiger, deer, also some vehicles you're seeing at the moment. All of these are being recognised directly in the image and then tracked. That was sort of what is in the image, what is the video. The next one I want to show you is action. We can recognise human actions now. We can also recognise actions of humans and other animals. I'm going to show you animal examples. This is what you're going to see is chimpanzees. This is how computer vision can support other fields. Zoologists have hundreds of hours of videos that they want to analyse and annotate. With computer vision you can annotate the actions of the behaviour of chimpanzees automatically all through the video. Here you're going to see the behaviours of nutcracking and eating. Each of these tasks I've shown you, recognition tasks of actions or what's in the image, has been done by a deep learning model. A deep learning model for a visual task is trained in three steps. The first step is you construct a very, very large data set of images or videos that you label for the task you want, like recognising an image. Then you choose or you design a deep model. This is where the deep part comes, a neural network. The third step is to train the model's parameters on this data set. You train it by predicting the labels on the data set. I'll show you an example. For the recognition, I was showing you earlier, recognising the animals and the vehicles, we were using a network which had been trained to classify a thousand different object categories. This is a picture of the network. It takes in an image, in this case an elephant, and then the output is a choice of one of the thousand categories that's been trained to recognise. This network has 30 million parameters. To train it, you need a very large data set, and it was trained on a data set of a million images, a thousand images for each of a thousand categories. What that means is that somebody had to find a thousand images of monkeys like this, a thousand images of dogs, a thousand images of elephants, across an amount of work, and that's what the network was trained on. I'm going to the core of this lecture. This is not how a baby learns to see. A baby is not shown a thousand images and says, this is a dog, a thousand images, this is a cat. That's not how a baby learns to see. What I'm going to do in this lecture is explore how computers can learn to see more in a way that an infant learns to see, and that is to learn directly from the data. That's what I'm going to show. What this means is that we have these three steps for learning a visual task, and we're going to throw away the first step. We're not going to have a large data set that somebody has to construct. Instead, we're going to obtain the supervision to train the network directly from the data. Now, this is not a new idea. Turing in 1950 wrote a paper, and he said, instead of trying to produce a programme to simulate the adult mind, why not rather try to produce one which simulates a child? If this were then subject to an appropriate course of education, one would attain the adult brain. Just following what Turing suggested in 1950, it also ties in with what's been found by psychologists who studied cognitive development in infants. What they found is the importance of data in order to develop intelligence. Data from the physical world is needed. This paper, which is very good to read, gives six lessons in order to develop intelligence in children, and they're for us. Lesson number one is be multimodal. I'm going to be multimodal. For the next part, what I'm going to do next is I'm going to show how machines can learn directly from the data without having to have a labelled data set. I'm going to particularly pick out the theme of correspondence between modalities, and that's what we're going to learn from. I'm going to show three different examples. This method of learning from the data is called cell supervision. The first one I'm going to do, the modalities will be audio and visual, and I'm going to learn from the correspondence of those. Example number one. Here's an example, first of all, of what a synchronised signal is. This is what we have in the world. We have these synchronised signals coming in, the audio and visual are synchronised. We can, of course, make them unsynchronised by shifting the audio, and then it sounds like this. Just this difference, this synchronised signal for three, we can also make it unsynchronised. That's going to be the supervision we're going to use to train a network. I'm going to illustrate this with talking heads. The reason I'm doing talking heads for two reasons. One, if we think of the Turing baby, what is the baby going to see first? It's going to see its parents speaking. In terms of, that's one reason. The second is, we as humans are very, very sensitive to lack of synchronisation in talking heads. I'll show you an example. I was in the camps yesterday talking to people. There are 1.3 million earthquake survivors still living in those crowded camps. I hope you can see that. That was how to say, well, here, it's how to say, it's very annoying. Now to how we're going to train the network. We're going to train it to tell if the lip motion in a video sequence is synchronised with the audio or not. That will be its task. We're going to give it a video sequence, a video clip, and the audio and say, are these synchronised? That's going to be the training signal. The network itself is going to have a part which takes in a video clip, and this network is going to be called a visual encoder, takes in the video clip and produces a vector. We're also going to have an audio encoder that takes in an audio clip, produces a vector. These vectors, these are lists of numbers, maybe 512 numbers, but you can think of it just as a point in 3D. The visual encoder predicts a point. The audio encoder predicts a point. The training is going to be that if the audio and the visual is synchronised, we want these points to be closed together, and if the audio and visual are not synchronised, we want them to be far apart. It's very, very simple. Where do we get the data from? We get it from just natural signals coming in. We have some frames in the corresponding audio. That will be synchronised. We'll call that a positive sample. We can get any number of positive samples from videos of talking heads. If we take the frames and take a temporal displacement, the audio and visual will no longer be synchronised. That will be called a negative sample. We can generate these positive negative samples effortlessly from any video coming in of a talking head. We can get millions of these. That will be our training data. We take this network, which has a visual encoder producing a vector, audio encoder producing a vector, and train it on all these samples which we know are synchronised and not synchronised because we're creating them. We train it so that if it can tell, if it's synchronised, the points it produces are closed, and if they're not synchronised, they're not closed. Imagine we've done that and trained this on millions of examples. What we'll end up with is points which are close together when it's synchronised. What I'm showing here is called an embedding space. This is where these vectors we've produced live. This is what I'm showing you in 2D. When we have a synchronised signal, the points that produce the vectors produced by the video encoder and the audio encoder are closed like this. If they're not synchronised, then they won't be closed. That's what we've learnt. We've told us to do this. We've trained it just on that. Now I'm going to show you, once we've done this, what can we use it for? The first thing we can use it for is to synchronise audio and visual signals when they're not synchronised. The way this works is we know that when they're not synchronised, the audio and visual embeddings are going to be distant. We can then shift the audio and if we shift it and they become close, then they'll be synchronised. We start with something that's unsynchronised, shift the audio until these vectors become close, then it will be synchronised. What that means we can do is we have this annoying example of out of sync. We can now synchronise that to this. Of heavy rain and probably four or five hours of heavy rain ahead. I was in the camps yesterday talking to people. There are 1.3 million earthquake survivors. More importantly, we can also use this network that we've trained to find out where in the image or the video the person speaking is. This is going to be shown is we're going to have a video and an audio track and we can produce what's called a heat map where it's hottest, where the person speaking. We're going to use this for localisation. This is a bit more technical but the way this works is inside the network, there's a spatial grid of vectors. This spatial grid of vectors corresponds to the spatial grid of the pixels. We can take the audio encoding vector and we can pick out the vector in the spatial grid which is closest. The one which is closest will be the one which is most synchronised and that will be where the speaker is. I'll show you some examples. On the left you'll see the heat map and on the right you'll see a box around where the speaker is. It's the perfect place to come if you want to see old roses looking the absolute best and the very latest. It's about to take place and here's another clue. The next thing we can do is track somebody using their voice. We've got multiple people speaking on and off and maybe some people are moving their lips but they're not speaking for their awning or laughing. We can pick out the active speaker by this signal because we can find the pixels if you like which are synchronised with the voice we're hearing. We can do that by private concerns on US relations with those countries and also FIA freedom of information request. Finally, we're not tied to humans. We have ways to train networks which can pick out synchronised signals. This can work for cartoons where the mouth moves with the voice. The blue will be the active speaker and the red will be the inactive speaker. Dad, you said you were going to play catch with me tonight. I have to work but give the monitor a kiss. It's the end of the first example. What you've seen is we can take something which rises from the world, the physics of the world, which is synchronisation and then manipulate it slightly, train this network which has tens or hundreds of millions of parameters and then use this network to track the person who is speaking, for example. We're going to go on to the second example, which is audiovisual correspondence beyond just talking heads. Now we're going to consider more general objects, more general scenarios where we have various objects that make sounds or actions that make sounds. In terms of this Turing infant, we imagine that it's watching its parents talking, now it's sitting up, it's looking around at the world and looking and listening to objects around it. This is development. The idea here is if you see an image like this, an image of a drum, you know what it's going to sound like. If you hear a sound like this, you know the answer, so obviously this is a guitar. You have this semantic correspondence between what's in the image and the sound. This rises just again from the physical world. You look at the scene, if something's sounding, you can see it and you can hear it. It rises from the physics of the world. You use this semantic correspondence between the vision and the audio to learn from through the network. This is a weaker requirement of synchronisation. We can do it from a single image. We don't actually need temporal information for this, we need the audio signal and an image. The way to do it, I'm going to formulate it like it's a picking game. We're going to task the network to pick which of these images this sound corresponds to. Imagine that the sound is actually a guitar and it has to pick out which of these it corresponds to. It should pick this one. The way we're going to do this is, again, distances. We're going to find which of these embeddings have been embeddings. It has the smallest distance and pick that one. We'll have a similar network. We're going to train, as before, we're taking a video clip. It goes through a visual encoder, produces a vector, a list of numbers. We take an audio clip, it goes through an audio encoder, produces a vector. What we want is if the audio and visual correspond, then the distance between these vectors is small. These points are small. If they don't correspond, then the distance from the points should be large. That's it. Where do we get the data from? We get the data from any videos we have. Here's two videos. We don't need to know what's in them. They differ in this case. What we do know is that there's a correspondence between the sound and the frames. Now we can take samples from this for training. Take positive samples where we take a frame and the audio around it. We can take any number of these. How do we get negative samples? We simply take the audio from one video and the frame from another video. In general, they won't correspond. That's it. We can do this. We have videos which arise from the world, have this correspondence property naturally, and we can sample millions of positive and negative samples like this and train this network. That's it. Imagine we've done this and we'll look at this embedding space where these vectors live. What we'll have learnt is when the audio and visual correspond, the embeddings will be close together. If we have maybe another instrument like a drum, then the sound will be distant from the embedding from the guitar, but it will be close to the embedding of the image of the drum. We have an embedding space that we've learnt like this. What can we do with this? One thing we can do is what's called cross-modal retrieval. We can start with a sound and now we can find images which correspond to this sound. The way to do this is to populate the joint embedding space with frames from videos. That's what I'm showing you here. All these points are frames from videos. Now we can look at the neighbourhood of where the sound's been embedded and pick frames which are close by, and they must be corresponding. I'll show an example.  What is producing that sound? We can dive into this embedding space, look at nearby frames and find videos. Here are the videos that could have made that sound. Cross-modal. We start with audio and we find images. Now I'm going to show another use of this embedding space. We've seen that we've trained it so that when there's a correspondence between the audio and visual, the embedding vectors are close. Imagine here we have the sound of a guitar, then any other image of a guitar should be embedded close by to this. That's what we've trained it to do. By transitivity, what's happened with this network is it's learned to embed all of the objects of the same class near together. That's what it has to learn to do. This is in fact how it solves the problem, how else could it solve the problem of determining the correspondence between the audio and visual, and yes, it was doing this. We've actually learned to the visual network which learns to embed objects of the same class close together. Now we can use this for visual retrieval. We can start off with a frame of a video, put it into the embedding space, populate this with other videos, and now look at the neighbours of this. These will be other videos. We can start with a video, search for similar videos, or search for similar frames. Here's an example. We start with a frame of a guitar, search in a few hundred thousand images, and these are the ones that are nearby. We start from acoustic guitars, found acoustic guitars. Another query, we start with a drum, search inside the embedding space and we can find images of drums. The point is that all this has been learned simply from taking samples from videos where the audio and visual correspond. That's all we had to do. We've trained this visual network and now we can use this visual network for recognition. We can also use it for localisation. As we saw in the synchronisation case, we go inside the network. The network has this spatial grid of vectors to find out where the object is that's making the sound. We take the audio embedding, we look at the spatial grid of vectors, find the closest vector, and that will be where the object is that's making the sound. I'm going to show you an example. Now you're going to see a video and frame by frame, you're going to see the heat map in the centre and overlaid on the frame, and then on the right you'll see the heat map itself. As I said, this will all be done frame by frame. It's localising them. All of that learnt. Third example, we're going to change the mentality now. So far we have audio and visual, now we're going to change to language or text and visual. In terms of our infant, by about 10 or 12 months infants can start to understand words and speak words and eventually they'll learn to read like this. That's what we're going to do now. I've done it in this order. I've started with vision and audio visual and then moved on to language because an infant learns to speak after it's learnt to see. Let's go back to our cognitive psychology. I gave six lessons and lesson number six is learning language. We're following six lessons still. How to do it? We've seen that we can train networks like this. I showed you in the audio case where we have a visual encoder and an audio encoder, and then we have a contrastive loss and we minimise the distance when there's a correspondence between the outputs. Very simply, if we're changing the audio modality we can also just change this audio encoder to a text encoder. Now we have text which corresponds to the video.  We have a text encoder and we can use exactly the same idea, this correspondence idea, that if this text description and the sentence corresponds to this image, describes this image, then the output of these vectors should be close together and if it doesn't describe this image, it would describe some other image, then the output of these vectors should be far apart and that's it. We've got our network now. How do we train it? We need to train this, we need to have paired data between images and text, text which describes images. Where do we get that from? Unfortunately, on the internet there's something called alt text, which is, if you hover your mouse over an image you often see a sentence comes up and it's provided so that you don't have to download the image or for the visually impaired it can be run out. So this alt text is available at massive quantities, there are millions or billions of examples of alt text available. Now I'll put some examples on this slide, on the left hand side, far left, the alt text is trees in a winter snow storm, it's describing the image and one on the far right is a facade of an old shop. So this is available easily and we can train this network by getting millions of examples and it's paired visual text and as before, just taking positive ones where they correspond, distance should be small and when they don't correspond, so we pick a random image and a random text, they don't correspond, the distance should be large and that's it, we train the network. So again, imagine we train the network and we look at the embedding space where these vectors live, then what we'll have is if the text corresponds to the image as it does here, the embeddings will be close and if we have another text man playing at the passing down, that doesn't describe this image on the left so it will be far away, but it will be close to the actual image where a man is playing at the passing down. So we've got this embedding space again. Now how do we use this? So we're going to use this for searching and retrieval of images and videos using language. This is really useful. So once again, we have a joint embedding space. We can populate this with images. So these dots now represent images that have been encoded, the vectors from those and say we want to find a particular image and we want to find it using language. So we describe what we want. Here's a sentence. We embed that in this space and then we again look for neighbours of this. This will then correspond to an image of what we're looking for because of the way we've trained it. So now I'm going to show you a demonstration of this. The really remarkable thing about language in terms of communicating with a machine is you can keep on adding words in language. You can make queries even more complex. So you can keep on adding requirements. I'll show you how that works in the demo. So this will be a demo searching 35 million images from Wikimedia Commons. You'll see that text being typed in and the retrieval will come immediately. Rysau'r swath was something quite simple. The first one is a red car and there we are. Now we make it slightly more nuanced. So it's now a sports car and there it is. And now more interesting, several requirements. Person riding a bike. There we are. Change bike to horse. Fine. Now make it even more demanding. Riding a horse but jumping. And there it is. And so on. We can also search for animals. And we can search for animals doing particular things. Here's penguins raising their wings. So what you're seeing here is by this embedding it feels like you're communicating with a machine actually because you get this instance of response keep on making it more and more precise and search query you're looking for. So that's the three examples I wanted to show you of learning from data. So that's sort of the tutorial part of the talk. Now I'm going to finish by giving you two snapshots of research sort of more recent research that build again on this type of self-supervised learning of correspondence between modalities. We've done all this work. Now let's use it for some applications. So I'm going to do two applications. One is going to be recognising British Sign Language and then I'm going to do audio description of videos. OK. Number one, British Sign Language. So this is a visual language that the deaf community uses in Britain. And here's an example. I don't know how many of you can read British Sign Language. So I'll tell you what she was doing. She was interpreting the sentence every spring our planet is transformed. I'm going to play it again so you can look for the sign for planet. OK. But she actually does seven signs in that short sequence. And it's very challenging. So spot them all. We would like, of course, to have a machine that could understand British Sign Language for many reasons. One is so that then deaf people can communicate with a machine at the moment. We can speak to using a lecture. We can speak to machines and get them to do what we want. If a deaf person wants to do that, they have to type it much better if they could use their own language to communicate. And, of course, it would be very good if they could communicate with non-signers. The machine could help do that, could translate. So that's why we want to do this. How do we do it? Where do we get our data from? Where do we get our paired data from? And the answer is we get it by watching television. Because on television you would have seen signers overlaid with television programmes like this. And you have subtitles which correspond to what's being said. And the signer is also interpreting what's being said so you have a paired data correspondence between the subtitle and the sign sequence. And this correspondence is what we can learn from as we've been doing all the way through this talk. Now, the BBC have very generously made available 2,000 sign language, sign programmes together with subtitles to support academic research on recognising British Sign Language. I'm going to show you some work we've been doing on this large dataset that they released. And what I'm actually going to show you is how we can recognise signs using maw things. So, I'm going to use my maw things. When signers are signing sometimes they mawl the words that they're signing. Not always, but sometimes they do. That's what we'll pick up on. So, I'll show you some examples. On the left, you're going to see the sign for office, but also he maws office. On the right, he's going to do the sign for tree and he's going to maw tree. So, why is this useful? It's useful because we can pick up words that are being mawved on the lips. We know how to do that. So, how this is going to work is imagine we have a subtitle like this clip here. Are you happy with this application? So, each word in this subtitle a happy application and see if it's mawved. And this example, she does maw happy. So, we look at the lips and we find where happy is being mawved. And once we've done that and we know the temporal segment where she mawved happy then we know the sign. Because we know exactly where she's mawved and we know the sign. So, we have a way of automatically annotating the data on the signs. And the way we do the spotting on the lips uses this synchronisation network I showed you in the first example that's actually how we do it. Okay, so, that was one example and I imagine we do this at industrial scale. We scale up, we do it on the BBC 2000 programmes. So, we take the words that occur in the subtitles, we look at all the subtitles that occur and we see whether the person is mawhing that word and then we take that segment and the sign corresponds to the word. I'll show you some examples. So, first of all, for family. Very fast. Important. You see we're getting all these different examples. Before. So, actually if you look at this one before it's signed in two different ways in these examples. You can pick out some sign is doing it in one way, some sign is doing it in another way. Perfect. Now, we're getting these signs from mawhings, but of course once we have the sign we can recognise it just, we can learn to recognise it just from the hand movements, the hand gestures. Okay, so then we'll be able to recognise it whether they maw or not. And we can generate, for each word we can generate thousands of examples. Hundreds of thousands of examples here. You're seeing these perfect examples we're getting and more perfect. So really it's quite a perfect method in fact because we can generate signs for thousands of different words and hundreds of examples say for each one and now we have a way where we can learn all these signs and recognise them by computer. But this problem's not solved, but this is a way of generating the data. Now, second application I want to show you, a snapshot is audio description of video. Audio description is a soundtrack that's provided for the visually impaired. And it describes the visual elements of the television programme or the movie. So that they can understand what's going on. I'm going to show you a short example of audio description, a type of thing that's available. This is for the film Out of Sight. So you see how the audio description is complementary to the soundtrack. So the things that you couldn't tell from what's being said or the music, that's what it's providing and then someone who's blind can understand what's going on in the film. Okay so we'd like to be able to generate these automatically. So we'd like to have a machine that takes in the video and then produces the audio description probably as text and then we have a text to speech that will read it out so visually impaired can follow it. So how would we do that? So we obviously need to supply the video to a model we're going to train and learn. But we have to do more than that because audio descriptions have the names. You heard it, the names of the characters. So we also have to provide the names of the characters. We have to provide a character bank of the people who are in the film. So we need this auxiliary information. Given those two inputs, then we want to train a model to produce the audio description. So where are we going to get the training data and we need paired data between films and audio description. Fortunately volunteers have provided audio descriptions for thousands and thousands of films. So this paired data is readily available and as you've seen we can learn from these corresponding data. So we have films, we have the audio descriptions, we can learn a model which generates the audio descriptions. Okay and I'm going to show you two examples of audio descriptions that we've generated. Again this is still working focus, it's not finished. They're both from Harry Potter and the first one is a painful example. Concentrate Potter Focus Okay so that's the clip and this was the audio description that we predicted for that. So Snape correctly points at Harry. So we've got the characters right. Harry closes his eyes in horror. There's more pain I'd say but this is what's produced. Second example. A more pleasant example. So how are we going to get along? Hi of course. The audio description that was produced was Hermione, Ron and Luna's eyes are fixed on Harry who is standing the doorway, that's correct, that's what happened. And then Harry rides on horses back as a horse raised up in the air. So this model doesn't think this is a horse but of course it clearly is not if you're a Harry Potter but there's more to do here. Okay so that's the end of my snapshot. So I'm finishing now. This is what you've seen. You've seen that it's possible to learn visual encoders directly from data in various ways. There's no name for manual supervision which is a traditional way of doing this and I've gone through a learning curriculum for a virtual infant, audio visual synchronisation, audio visual correspondence and then language visual correspondence, that's what you've seen. I just mentioned that the field, computer vision field works on this problem a lot and even though I've shown cross-modal learning you can also learn visual encoders purely from the visual stream. So deaf people can see as well of course. I'd like to end by thanking people. So of course this work is not just not done by me, by any reason it's done by my students, my postdocs and I'm always inspired by talking to faculty Oxford in the UK, DeepMind internationally it wouldn't be possible without all these people and a lot of them are here in the audience so that's great, so thank you. Thank you very much Andrew for a super lecture, very stimulating. We have time for some questions so who would like to start us off. Now there are a couple of people with microphones so if you put your hand up someone will come to you with a microphone so let's start over here on my right to your left with the first question. We also have people who of course has been live streamed so we have people who can ask questions on Slido and I think one of my colleagues is hovering around with an iPad and will wave at me if we have some questions on Slido. Please go ahead. Hi, thanks for the lecture. What I was going to say is how far are you away from having real time live access to this system if you like really? So there are lots of systems I showed here. The real time demo I was showing that's real time. So you can type it and it will immediately retrieve images or videos from a large data set. Right, I think what I was trying to say is more the video. So the moment is trained on videos online I take it and what I was trying to say is that if you used to have a device that could see someone talking in real time like for instance the sign language could that be then translated? Sign language is not solved just to be clear. We can't do a sign language yet. At the moment most of these methods run on big machines, GPs etc. But there's lots of work on taking these big models and distilling them down to smaller models in various creative ways. So even though at the moment they run on GPs etc. some of these models already can run in your browser. You can do real time post detection of humans in your browser and you can see that ASR speech recognition can be done in your browser. So these models start large but then once they're ready they can be made smaller and more portable. Does that answer your question? Thank you. I want to say all of the models are too large. At the moment to do that of course. That's the way it goes. With that question just a few rules back I think. Congratulations on the price. Thanks for the lecture. It feels like for different tasks maybe you have to do a very specific data processing which you explain now. Do you see a way of doing self supervision with a fairly general type of data processing where you can apply later to very different tasks and maybe what ShadGBT does for texts. Thank you. Thank you for the question. I've concentrated on visual tasks here. We already know the answer. The answer is yes. I've shown some different types of self supervision here. There are many others as I said. Once networks have been trained in some way by using self-advised methods they can be used for multiple tasks by applying what are called different heads. They can be used for recognition object detection, tracking. Once you have a good feature a good network it can do multiple tasks and this is really the way that large scale networks are trained nowadays. But there's still an issue of you have to say what the tasks are. If we haven't it's still a research question to train network and then do all the tasks you want to do and predict depth, predict other things about images or videos. It's still work to be done here but we certainly have lots of evidence that training a good visual backbone enables lots of tasks after that. Okay, so I think we have a question there if you go one row back first I think you pass it back again. I think that's got to be long and correct. Hi, I just wanted to ask is there any dimensions the abennic space should be? Yes, thank you. It's usually a power of two and I think it's a good question but it's not a good answer. It's also storage vectors are very large it provides more storage but no I don't have a good answer. It's an empirical question really you try different sizes and you determine what works best empirically. Thank you for asking I don't have if anybody else here has got a better answer please. Any volunteers? Okay, go ahead please. Thank you very much for the talk Andrew. Knowing what you know now about this research project which parts were the hardest? Was it data collection, data prep was it designing the encoder neural net architecture like what would you for somebody else that's working on a similar research project that's knowing what you know, what do you give? Each stage of these projects you stumble across something there's always a way of project you have an idea or somebody has an idea and you start doing it and then unexpected things happen so it's difficult to answer really sometimes the networks are hard to train sometimes the data that you think is good is not good when you're putting together so many things like this unexpected things happen like one of my rules is when you have data sets there's always noisy it always happens there's always something wrong and you always have to look at your data and see what's going on so I can't give you a definite answer it's just that every stage always has problems. Okay, so we have a question at the front here. Thank you for a great lecture so with the first message you showed with temporal alignments and so on that could be how babies might learn to see as a cue but with the later work with things like text to object that still seems to require a huge amount of data. Is there any argument that that might be similar to ways that babies learn to see and correspond with language? It's good to ask. I think a huge amount of data in the text case is Achilles' heel. That's a problem at the moment but you need so much data to learn from and I think it's a research question how you can avoid needing so much data. By the way, I wasn't saying this is how babies necessarily learn in principle they can learn this way because we can see that just from the data you can learn these skills not saying necessarily that they do this at all it's just that the information is there and the order is that after they've learnt to see and hear cognitive development we know that it's later that they acquire language. That's the only point I was making there but how to avoid having to use such vast quantities of data in the text case I don't know if I have a good answer to that. In the audio and visual case it's readily available there's no cost to that. Question here. Hi, thank you. I am not in the field so my question might be stupid but I'm just wondering for the correspondence or synchronisation what if there's some false correspondence let's say I'm waving my hand but this actually corresponds with the sound which is not in the image at all will your model be able to pick out that? It's the large scale data that helps avoid problems like that but we can call it noise things which don't correspond to what's making the sound but when you see enough examples you can pick out the ones that matter and the ones that don't. That's the answer to that question really. If you think of the talking head there are lots of things going on the person who is talking their eyes might flutter their hair might blow in the wind but in order to solve the task of learning synchronisation it has to see that what really matters is the lips because it's the lips that are synchronised with the voice with the speech so it gets to ignore all these nuisance factors this noise and pick out what really matters otherwise it can't solve the task. So Andrew how many is enough? It's a good question. How many in different circumstances cannot be the same in all circumstances? It's another question that we always fight with. We keep on going empirically until things work well then you see if you can train more efficiently meaning less data I'm waving my hand and saying a million samples because that's what we use because it's so easy as well to get this data. Is there a statistical test you can apply generically to these things? To give you a more proper scaling laws where you can say for the number of parameters how much data do you need given a certain training budget is this published work for training models but again it starts off empirically it's not like physics or geometry where you can give counting arguments this field doesn't have things like that it's much more empirical and then generalising from that. I think it's more hands so yes, two over this side this one first Hi, I have two questions quick ones what do you think computer vision will look like in say 20 years and and the second is is superhuman vision possible? Do you mind if I don't answer the first question because I think these sort of predictions I know people always want us to make the value they're always wrong so whatever I say it's going to be wrong really because it could move so quickly superhuman yes what would be superhuman though because I think for sure we can do things that are superhuman being able to search through 35 million images in a fraction of a second surely that's superhuman and we're going to be able to for sure search enormous satellite images spanning the whole world and find something we already can do superhuman things for computer vision and it will go on in terms of temporal temporal resolution spotting things which exist over long time scales that human wouldn't notice or a short time scale that human wouldn't be able to see all of the things will happen yes once we can do a skill on a computer we can make it superhuman a question here thank you very much for the talk you showed us the multimodalities examples and then in the end you said that this also can be applied to the single modality but with the single modality obviously then you have to figure out the augmentations because it's less obvious how to compare positive and negative examples have you found that multimodal embedding and comparing the embeddings from multiple modalities learns better because you don't have to engineer those augmentations or can you you know or do they perform on par or do the single modality can perform actually better that's a good question and you're right that with images you have to do augmentations which to other people if you're going to do something from a single image the sort of things you might do is crop the image and you want the embeddings to match even though you've cropped so if an image has got a horse in it and you crop out parts of it it will still be a horse so you want the embeddings computed from these various crops to all match that will then train the network to understand that the contents of the image shouldn't be affected by these crops but to answer your question empirically learning from multimodalities generally works better than certainly in video than doing all these augmentations it's naturally doing augmentations so it works better but the other methods work very well as well you just have to work harder to make them work and some people in this room have done multimodal and unimodal learning they will work well in the end it's just you have to do more work to make the unimodal ones work you guys have a question here thank you for the lecture very interesting just because I'm curious I suppose on the on the topic of solving for finding a speaker in a crowd what if everyone is turned around how do you do and detect a speaker yes then it can't answer of course it has to see the lips but there are other cues if a person is speaking when I'm speaking I move my hands there's body language there are other ways you can do this as well I haven't shown that here but you can imagine if everybody was always speaking from behind then the network had to solve the problem it would do something like that okay I think general here we'll have the honour of the last question yeah I was just wondering in terms of there were several examples where you had for example there was a visual network would be working together with maybe an audio network or it could be a descriptive network and I'm wondering whether the sort of embedded space that you end up for the visual network can apply across multiple different problems you sort of alluded to that earlier saying that sometimes even if you trained it on one problem then it can actually be useful on other problems and I'm wondering whether these embedded spaces end up sort of encapsulating an overall description of the image which can be used in multiple different tasks and whether that's those embeddings are a good gestalt of the whole thing that's being presented the answer is yes you'll train them for one task but in order to solve that task they have to do something more than you've trained them for an example of all the drums being embedded close together all the guitars being close together and once it's done that then it's like a guitar or like a drum it will embed to a certain point so it's learnt the characteristics and that applies to thousands of different categories and then you can use it for tracking guitars or other properties you might want for guitars this is the answer is yes basically okay well that's super now before you applaud to thank Andrew again I'm going to combine that task with presenting him with his scroll a metal so Andrew if you want to come out here because it can make it easier I will hand you this if you want to hold that and if you can do a tick and pull me out at the same time I'm shaking my hand thank you all very much for coming and it was an excellent lecture thank you