And we were like, oh, that would be cool. But it turned out sort of better than expected. And that's thanks to all of you who made it here and made the workshop really special. So I'm going to talk about debiasing regularized estimators and high-dimensional linear regression. And we coined a term called spectrum over adjustment, and which will become clear why it sort of named this way. This is joint work with Yufan. So Zou already talked about Yufan a bit. The work he presented was joined with Yufan and Shubhuan Yihong. But Yufan is a joint PhD student with me and Shubhuan. He's just really exceptional. I wish he could have been here in person. But sometimes we have visa issues, which he'll be out of very soon. So hopefully you can see him at one of the youth workshops like in the future. OK, so the setting for my talk today is sort of the very traditional high-dimensional linear regression model. I apologize for using notations that a statistician is more familiar with. And I should have probably changed it to A's and X's, like Zou was very smart to do yesterday. But for me, the Y's are the outcome or the response variables. They're real-valued. The XI's are the covariates. They're p-dimensional. And the goal is to infer beta or say whatever you can say about beta. And the traditional approach, obviously in high dimensions, is looking at regularized estimators of this form where you look at some loss function. You look at its minimizer, but you add some penalty with some tuning parameter that is the H function here. Now, if we want to do inference in that, let's say, I look at a finite number of coordinates of the regression vector beta. And I want to recover them to the best possible ability. Sometimes that would mean that I want to create confidence intervals for them or create an estimator with proper uncertainty quantification guarantees. One of the starting points is looking at such regularized estimators. But what you soon realize is these estimators are biased in high dimensions. And they're biased for the simple reason being that you're not really minimizing a log likelihood or maximizing a log likelihood or something. You have this penalty term that makes your estimator, if you look at each coordinate, it is typically not centered around the corresponding true coordinate of your regression vector. So if I use sort of this high-dimensional scaling that we are all familiar with in this community that my number of features to samples diverges, both diverges and the ratio converges to some constant, then we know that for these kinds of estimators, I'm assuming convex likelihood and everything is convex. So this is roughly going to behave like your proximal operator of the penalty H, evaluated at some threshold theta. But you have this Prox operator, which is applied to your true regression vector plus some Gaussian fluctuation. So ideally, I would want this Prox to not be there. I want to have an estimator like that. But that's not what you get if you use regularization. OK. So now what I want for uncertainty quantification or confidence intervals is to find some denoise or eta, which when I apply to my beta hat, it's going to denoise this Prox function. In that, it's going to be centered around beta with some Gaussian fluctuations. So in other words, I wish to basically invert this proximal operator suitably. And this has been done in the literature. The way this goes or the way this evolved historically is roughly as follows. So there was an initial suggestion of how to construct such an eta. And the suggestion was as follows. So take your log likelihood or whatever was your loss function L in my previous slide and take a gradient step on that. OK. And evaluate it at beta hat.  So you take your regularized estimator. That's my beta hat. And then if I perturb it by this adjustment, this is basically coming from the score equations for the model. Then one observes that the resulting estimator is centered around your truth with some Gaussian fluctuations. Now the thing is, this was sort of the initial level of suggestions. But it turns out that if you just do this perturbation, so this is exactly the gradient of my log likelihood evaluated at beta hat. And I'm simplifying things by assuming Gaussian errors. This works, but this only works when the signal beta is highly, highly sparse. And whenever I say something is highly sparse, I mean it's at most small of n. Meaning it could be less than that, but it's not linear sparsity errors. It can't really account for high dimensional problems where maybe a fraction of the coordinates are non-zero. So this was an issue with this correction. Now, of course, people realize that. And the correction that was proposed to modify that takes a form like this. So you have the previous estimator, which had only the score step or the gradient of the log likelihood step. But in addition, you scale it by some correction factor. So I write it as one minus some correction factor. And then it turns out that this new estimator is also centered around your regression vector with Gaussian fluctuations. But the nice thing is that by adding this denominator here, this result of asymptotic normality roughly centered around the true regression vector, this result you get under much less assumptions on your regression vector. And the hope was this could be pushed to linear sparsity. It's not like it's really all the way up to there, but it's almost there. So it covers a much broader class of regression problems or regression models. All right. So I want to start the story basically by staring at this formula and try to give an intuition for if somebody tells me that I should add a denominator term here to correct my formula to get proper debiasing. That's called debiasing for obvious reasons because you get rid of the bias. Then what should be the right correction term? Now, okay, so I'm motivating this from like a statistical story, but a lot of us know approximate message passing theory and approximate message passing algorithms. And it turns out that if you just do some heuristics from those algorithms, it's very easy to predict what this correction term should be, okay? So I'm gonna do that heuristic on the slides in the least possible math that I can potentially write to present that. But it's done so that I can build some intuition for what should we expect here. Okay. So let's see how this goes. So what I'm presenting here is just under IID Gaussian designs or IID Gaussian covariates. If I were to look at the approximate message passing algorithm that tracks my regularized estimator, this is the form it takes. Okay, I have no one, okay, sorry, this pointer is here. Okay, so we have two iterates that we introduced. The beta hat T's are supposed to track my regression vector. They have a formula like this and this is an iterative algorithm. And the way there is another auxiliary iterates, the ZT's, the way you should think about it is this part, if you look at it, it's like a residual in your regression. And then there is a correction term that is added to that. And this correction term we know is our Onsager correction term. All right. Now, MP theory tells me that this guy tracks my estimator. So the MP iterates track my estimator. Now I'm going to take some jumps. All right. So the MP iterates track my estimator. And the jump I'm going to take is everything is nice and the algorithm has a unique fixed point. Okay. So if it has that fixed point, then that fixed point satisfies these two equations simultaneously. All right. Now let's stare at the second equation. I have these auxiliary iterates, their limit Z star here and I have a Z star here. So from here, I can write roughly a formula for Z star in terms of beta hat star. Okay. So that's just some algebra and I just write it this way. So I have Z star here, I have beta hat stars here. I have another Z star here but that can be simplified. I'll not present the simplification but roughly this is what you get. I have almost a closed form equation for my auxiliary iterates Z star. Nice. So I'm going to keep that equation here. I have a closed form for Z star. Now I'm going to leverage some insights we know from MP theory. So what we know from MP theory, if I could go over to these fixed points is that if I take my beta hat stars and add the following correction, which is X transpose of Z star. Z star is the auxiliary iterate. That guy is centered around my true regression vector with Gaussian fluctuations and this is just coming from state evolution results. All right. Nice. So what can I do? I have a closed form for Z star. I can plug that in here, right? I plug that in. That gives me a formula like this saying this whole object is centered around the truth with Gaussian fluctuations. Now we know that MP is supposed to track my regression regularize the estimator in nice problems. So if I could replace this beta hat star by my estimator of interest, what do I get? I basically get the following. Let me go back a bit. So what I'm doing here is replacing beta hat star by my regression vector and that would give me exactly a formula like I presented before because this guy is gonna give me that my beta hat with this perturbation has the right centering but that means I have a formula for this correction term. And this correction turns out to be something exactly in terms of the unsaggered term coming from MP. So basically there is like a one to one correspondence between approximate message passing algorithms which track your regularize estimators to an estimator that is centered around the signal that you want to recover. All right. Now, this was just a simple derivation in the case of IID Gaussian designs. And as there has been a theme of this workshop to study structured data problems, we know that IID Gaussian designs are things we don't like anymore. So we would like to go to structured data and the moment we go to structured data we will have more complicated approximate message passing algorithms. But hopefully this derivation tells you that if I can understand those algorithms really well, I should be able to construct these kinds of estimators which are de-biased versions of my regularized estimators with the right perturbations and the unsaggered terms in your approximate message passing algorithms should show up in the formula in one way or the other. So that's sort of the intuition. Cool. So the first thing I want to show you is examples where this Gaussian formula, this was really sort of the Gaussian unsaggered correction term, we have running examples of five design matrices that I'll present through the talk where this formula does not work well. Now, if I just told you this, that sort of, yeah, sure, we don't expect it to work well when things are not like IID Gaussians, but it actually took us some time to find sort of concrete examples where this de-biasing formula wasn't working well. Sometimes the MP results are not working well, but the de-biased estimator people have created, they were still working well, even in non-IID Gaussian settings. So it took us some work, and we have these running examples. I'm hoping that for all of you working on structured data, some of these matrices that we suggest here could also be taken as use cases to demonstrate where IID Gaussian things should really not work in practice. Okay, so let me first explain this picture and then I'll walk through what my five design matrices were. So in this picture, we are taking the previous de-biased formula I just presented, centering it by the true signal, scaling it by whatever its standard deviation is, and I'm, so this is a p-dimensional vector, and I'm plotting the histogram of its empirical distribution, okay? So those are the histograms, so these are the blue histograms. Now, when we say sample mean here, we mean whatever is roughly the center of that histogram. And this black curve over here is trying to fit a normal density to the histogram. Whereas everything in red is the standard Gaussian density. So this red curve is your standard Gaussian density and the X equals zero, and this red line is the X equals zero. So if my de-biasing theory from IID Gaussians were working well in these other designs, I would have observed that this black curve and the red curve, they just superimpose on each other and that the centers also match, that these two lines, they coincide. But as you could see clearly, for across all these examples, there is a shift in the mean. So this guy's mean is different from zero, and also it doesn't really fit the standard Gaussian, okay? So what are these designs? These are obviously five pictures for five different design matrices. And I'm going left to right. So the first one is just, it's a Gaussian object, but this is kind of a matrix normal distribution where I can have correlations between columns and rows. So this column covariance is, it's sort of like what we call this, they are one or something, the 0.5. So you have correlation that is 0.5 with the neighbor and it's decaying as the locations, as you're further off. And the row covariances, they're just drawn from some inverse wizard. This is the first one on the left. The second one is just a spiked matrix kind of design. I have some lower rank objects here. Alpha is some parameter and have Gaussian noise. So that's here. And this is something more familiar to all of you. The third one is essentially like a linear neural network type objects. I take a bunch of standard Gaussians, I just take a product of them. Here we took four of them and we get this plot. The fourth one is what we know in statistics as vector autoregressive models. And I'm presenting a stylized version of this. So here, the ith row of this design matrix, this pointer really does not go where I want it to go. The ith row of my design matrix is a linear combination of K many of the previous rows. K could be a small number. Let's say K equals three, K equals five, but basically it's a linear combination of K of the previous ones with some Gaussian fluctuations. That is the fourth one. And the last one is actually heavy tilt. So the last one is just multivariate T distribution with three degrees of freedom for this plot. So these were chosen in a very carefully crafted way because we are gonna change these examples later and each of them kind of, you can change in specific ways to demonstrate different phenomena later. Okay? So this is just to demonstrate that there are hosts of these objects, real stuff. You might observe them in practice or not. I don't know. I do more theory than practice. So probably I'm not the right person to judge that. But hopefully, these are use case examples where you can clearly see that the Gaussian based stuff does not work. Okay. So any questions so far? Yeah, so the ground truth is. Yeah, so the question was whether the ground truth is taken to be sparse. So the ground truth and all these problems, I'll just keep it the same model but I have mixture of two Gaussians and there is a spike at zero. And for the two Gaussians, one has a positive mean, the other has a negative mean. It's not symmetric even. It's like, could be different means. So mixture of Gaussians. Yeah, I mean, okay. For these plots, we did it fixed, but you could, you know, it's a sort of, you can think of it each coordinate as IID from this mixture Gaussian and there is a spike at zero. Yeah, other questions. What do we do? Therefore, we do something like rotational invariant ensembles. And by now, I think all of you have done that or there were many, many, many talks on this. And sort of before this work of ours, the most related is this wonderful work by Takahashi and Kabashima in 2018 where they suggest based on replica insides a debiasing formula for the lasso for rotational invariant ensembles. So when we looked at that, we were like, okay, we expect this Gaussian stuff to not work, but this is a much larger class of ensembles. So maybe that's what we should investigate. And we know this is a rich ensemble class. And by now, this ensemble class has already been motivated across very talks. So I don't need to motivate it much more. There has been very rich approximate message-passing developments by Manfred Doppel and collaborators. Zoo is here.  I'm not intentionally missing any of the others of you, but it's like this people here have contributed tremendously to this area. Marko is there. Okay. Now there are two works that are really related to ours. One is this non-rigorous debiasing formula suggestion, but the other is there was a work by Gerbalov et al. In 2022, it's a really nice paper where they characterize risk type functions of the regularized estimators under a rotational invariant ensembles. So there are some crucial differences between their work and ours. So they had a few assumptions of which there are some which may not be, you may not be able to check in these models. One is they actually assume that limits can be, the limit in N and the limit in T in AMP or vector AMP can be interchanged. So the state evolution part is not fully rigorous. And also they assume that limits of certain fundamental quantities sort of exist. So there isn't a formal proof that the limit goes to some things. Like assumption if the limit exists, then this should be the characterization. But these are the most relevant stuff. So what we'll do in today's talk is we'll leverage all of these insights and we'll propose a fully data-driven debiasing formula for right-rotation invariant designs. And I want to emphasize that the formula will be fully data-driven. And you will see that the way we construct this data-driven stuff, which actually leverages spectrum properties of the design very crucially, we know that that's what you need to do for these kinds of ensembles. And we are able to do that. And we have rigorous proofs of everything. We have a host of statistical applications. I'll present some of them. And the other is you can find in the paper, which is still baking in the oven, but it should be done in like two, three weeks, I promise. Okay, and what happened, we were really interested in debiasing, but as a byproduct of all of this, we actually ended up proving a bunch of new properties for the vector AMP stuff underlying these models and related quantities that arise in VAM. So I'm hoping that for other problems people study, maybe some of those proofs or lemmas or propositions could be useful. All right, now we'll get into our suggestion. Any more questions before I move on? So let me first present the suggestion. So the suggestion is, it's kind of simple in that, okay, we are gonna do debiasing with what we call spectrum error adjustment. It looks exactly the same. So I have my regularized estimator. I have my score kind of term, but now I have a correction and I'm just gonna tell you blanket that the way you find this correction is take the following equation. Let me explain what that is. So if I have a matrix X that is right rotational invariant, I'll formally present what that means. But if you look at the eigenvalues of X transpose X, those are my DI squares. H is my penalty. So I'm assuming that my penalty is twice continuously differentiable, at least on, except on a finite set of points. So adjustment had is the variable in this equation. And we're saying, just solve this equation system, you get a solution, that is your correction term, and you have formal proofs that are in our setting, this equation system always has a unique solution, okay? No, I'm getting there. So this is all under the high dimensional scaling limit. I'll present the formal theorems and our assumptions in a bit. But this is sort of like, what is the methodology? This is the methodology. So this equation, and note that there is nothing depending on population quantities here. So here I only need the sample, the eigenvalues of the sample covariance matrix. And H is something I know, okay? So this is fully data driven. That's the, yeah, yeah. So this one we prove there is a unique solution. So that needs a proof. So I write it this way, but we prove, and in our setting, it's unique. So I'll present the assumptions and we'll say that under that setting, yeah. Yeah, we can't prove lasso. Yeah, so for lasso, we can't prove this. So I'll present my assumptions and we can prove it for the lasso if you have like P less than N. So I think for the lasso, the right change obviously will be like some smooth approximation to H double prime and replaced by that. We tried really hard to prove the lasso and maybe we can prove it in three more months, but maybe now we just don't include lasso in the paper form. So what are the theorems? So under regularity conditions, which I'll present soon, I should have written this, but basically throughout the talk, I have number of features in samples diverging, the ratio converging to a constant. And the constant can have a wide range of values below one, above one. That doesn't really matter, that's okay. So what we show is this debased estimator centered around the right object with a certain standard deviation, which we completely, we can completely estimate this, goes to a Gaussian in a Wasserstein two sense. So this is sort of the usual name P type result, but the important thing is we have constructed an estimator, like a true estimator in that everything is data driven and we also have an estimator for the variance. Okay, so quick peek into how does this work? So this was the first row I have already presented to you. This was the Gaussian based formula and we were plotting empirical distribution of this guy on the left, with the Gaussian correction for the first row and for the row below its empirical distribution of this entire object. Now you see, remember that the red curves were my Gaussian densities and this was X equals, the red line is X equals zero. Now we have almost a perfect match. I mean, for some of them there's a little bit deviation, but our sample size and dimensions were actually relatively small for the simulation. And this is a range of different design matrices. Like for instance, I don't know if somebody gave me linear neural network type designs and multivariate T, it's not natural for me to immediately think that there's something that's gonna perform exactly the same on both of them. So we are trying to really demonstrate it across a range of different design matrices. So this is the first result. And as you can imagine, if you have a result like this, then from there you can immediately construct confidence intervals. If you assume that the entries of your underlying signal are all like, let's say, exchangeable random variables independent of your design and noise. Or if beta j is our IID from some pi and pi is nice, maybe has finite second moments, then you can give similar guarantees, but for each coordinate. So this is not as surprising once you have the Warshish-Tyn2 convergence. But note that, again, this is for each coordinate under an assumption of my underlying signal being exchangeable random variable. So it's not exactly a deterministic signal. The formula here are not important, but this is just to show that there is a way to estimate everything data-driven. So our goal was to estimate, let's go back here, my goal is to estimate the standard deviation. And the way we do that is we go through a bunch of intermediate quantities. So everything here is data-driven. And then there is a formula for this standard deviation, which kind of just works. And it seems if I just present that this, that this is coming out of the blue, turns out not quite. So it turns out that if you study the vector approximate message-passing algorithm underlying these problems, and if you look at the fixed point equation, whenever I say fixed point equation, I mean the scalar system of equations your state evolution gives, but let's have sent t to infinity. So I'm in settings where maybe that has a unique solution. We show where you can have unique solution or not, et cetera. But I'm in a setting like this. So it's a system of, I can write it as a system of four equations. It turns out that each of those parameters have a physical interpretation. And one of them is actually exactly our de-biasing correction term. And the other is also a physical interpretation, and I'll not go over that, but what you do is you try to construct estimators for all of these parameters by using your AMP insights, like what is the interpretation of which parameter. And from there, and using this on Sagar correction trick that I showed at the beginning, you can create a fully data-driven estimator for this correction term. And you also have a data-driven estimator for your variance, okay? And everything is function of the spectrum of X transpose X, which is nice, which is why we call this spectrum over adjustments. Any questions? Sorry, can you say that one? Right, so the thing is when I have a real data, I have, okay, so I'm trying to try to repeat, but I might have missed because I wanna hear as clearly, so I might ask you again. So I guess your point is, what is the purpose of data-driven quantities? Because we know in large dimensional limit, all of these concentrate, et cetera, but they concentrate to, let's say, things that are sort of population quantities. And if I have a real data, I have a single observation, right? And I don't know, so let's say these DIs, you know that, you know, there are functionals of these that are just gonna concentrate. But what is the concentrating measure? So if I've observed something in practice, I don't have a sequence of DIs for which I can take the limit. I don't know how to take the limit. I have one snapshot of the data. And theoretically, I can have a limit that I can analytically characterize, but it's not something I can compute from my data. You know, usually I don't have snapshots as my dimension is going to infinity. But the point is that's what we are trying to rigorously do here. Instead of having the expectation quantities of functionals of the spectrum, we are really trying to show that you can have things in terms of the empirical spectrum and functions of that work. Exactly, yeah. Other questions? Yeah, yeah, yeah, yeah, yeah, yeah, yeah. And the way we prove this is more like start from the art transforms, discover nice properties of it and go back and replace by empirical stuff and show that that works. Like you need to prove that these work, but that's how we construct these equations, yeah. Any other questions? So I should mention all the assumptions. So the first one is sort of usual right-rotational invariant assumption. So if you look at the SVD of my design, the O here is uniform from the orthogonal group independent of QD, that's our assumption. For the D here, as you may, empirical distribution converges. Variance has to be positive. It has a compact support. And the others are sort of usual in that this condition on the signal is the standard sort of, if you look at its empirical distribution, converges to something that has finite variance and maybe second moments converge. Now we have conditions on the penalty. So it's twice continuously differentiable except for a finite set of points. We can allow finite set of points where it's not, but actually this assumption sort of precludes the lasso in high dimensions in that we either need these eigenvalues bounded away from zero or each strongly convex for rigorous proofs. But in terms of the formula, we still expect some smooth approximation to each double prime replaced in the case of the lasso should work. Okay, so quick comparison. And if I'm wrong, please correct me. So I mean, the initial proposal for these problems was for the lasso using sort of the replica methods. And from my understanding, and I think I already clarified this in answer to US talk is we're trying to go to the empirical functions of the spectrum instead of the population functions. I think in the previous suggestion, the correction terms were in terms of these population functions of the spectrum. And we realized almost while finishing this is our insights actually generalized to non separable penalties. So we also have a conjecture for this, which we are not proving in this paper. It's for a future paper, but actually the whole construction could, if you have a non separable vamp analog, you can just do a similar construction. Okay, so some applications. The first couple of applications are just gonna be sort of, the first application is just gonna be a bit of a standard application, but it's just to show that this kind of works. So if you have all this Warshstein two convergence, you can estimate a lot of objects, right? So for instance, if I were to look at the mean square error, or if I want the norm of my underlying signal, we can construct, again, data-reven estimators for all of these quantities. Quick peek into how this works. It's probably not so visible from the back, but the left column is for the signal L2 norm and the right column is for the mean squared error. And across the five design matrices, let's maybe focus on the mean square error. The true value is let's say around close to 70 or between 60 and 70 roughly. If you do Gaussian stuff for these kinds of design matrices, you really, really underestimate. So you get like something between nine and 20, but then if you do this kind of debiasing, then you kind of recover and be in the right ballpark. So that's more like a first level sanity check that this is working. Now the application I want to spend some more time on is connected to something that we call principle component regression in statistics, and it's directly connected to PCA, which a lot of you presented. So principle component regression is sort of doing regression after dimension reduction, okay? So what it does is in regression problems which are really high dimensional, you take your sample covariance matrix and you perform its PCA. And usually what you do is you retain a subset K of the eigenvectors, let's say of the larger eigenvalues. And using this subset of eigenvectors, you transform your original data. So you take your original data X and you stand transform by okay transport transpose. It gives you a new data matrix. And now you want to run regression on this data matrix which has a reduced dimension, which is kind of nice. So it saves computation. And often this K is such that you can actually on the reduced space really just perform an ordinary least squares. So people typically do that, but that's not enough because my goal was to recover the original signal. And now I am in this reduced space. So one has to go back.  So this whole pipeline is known as principal component regression. And it's used a lot because it sort of dimension is really large, what do I do? This is sort of the first level thing that you do if you are in regression problems. All right, now if we stare at this equation at the very end, something that you'll observe is if my true signal is not in the span of the selected principal components, then this PCR estimator will suffer similar kind of shrinkage bias as you observe for regularized regression estimators. That's expected, right? But people still do it anyways because it's a nice method, it's a simple method. Okay, now we realize that actually using all these devising under spectrum aware approach, you can really correct this issue.  So I take these DKs, I scale the OLS estimator by these, I do a totally different transformation. And we realize that if you do this other transformation instead, if you started with the right rotation in variant design matrix, your eventual, this one is also right rotation in variant. And you can use our devising theory on this guy. And that's kind of nice because you also reduce dimension, so that's nice. So you can apply spectrum aware devising to this reduced or this transformed space. And that actually, we thought this is gonna work really well, okay? And the whole thing here can be made rigorous if the number of components that you retained is proportional to the dimension, okay? This should be the same K as here, but it is a different form, sorry about that. Okay, so why do I write almost works well? So we were very excited by this because we thought like, oh, okay, this is natural and one can do this. But what happened was when we use this, it worked much better than if you did something Gaussian based, which is obvious that it will work much better than Gaussian based stuff. But it started becoming pretty unstable numerically. And then we realized what's going on is often these are high dimensional scenarios. And often the matrices we were trying this on would have a few top eigenvalues, which are really large compared to the others. And that's why there is some unstable behavior. So we wanted to really fix this. And the way to fix this is again, very natural. So what do you do? So the main observation we had was that our estimator was okay, but its variance was very unstable, okay? So when I scale by that standard estimated variance, our standard deviation, then the thing becomes unstable. If I replace it by the population quantity, then it's stable. So we realized this instability is coming from having some eigenvalues, which are much larger than the rest for specific designs. So you can do the other thing. So for instance, you can basically do PCs to remove the top L. And that's something also done in high dimensional stuff. And after removal of the top L, you can just do spectrum error devising, no more transformations of the data. Now we found that this approach actually works very well for inferring the true signal, even in matrices where there are strong correlations or heterogeneity in the designs and some sort of serious heavy-tailed examples as well. So I'll show you in a picture in a little bit. So these are our examples. Again, I have the same five class of design matrices, but I will show you in a little bit that have completely changed their parameters to capture sort of challenging design matrix settings. So it's plotting the same object as before. So Gaussian debiasing, which we call degrees of freedom debiasing, that is much worse here. So okay, this is far from Gaussian. It's like the empirical distribution looks like this. It's almost bimodal. This is the Gaussian density. And the shift in the centers is much, much larger. If we don't remove the top L eigenvalues and just do our spectrum-aware stuff, you observe something like this, where I'm not yet Gaussian, but my means start to be okay. But you can see that my spread is not quite okay. And then if you remove the top few in this plot, we are removing top 20, then you do spectrum-aware, you exactly match the Gaussian density. Now I want you to maybe remember this picture a bit, the last one in your mind, because I'm gonna present sort of how we tweaked the parameters of these example design matrices to make the problem much more challenging so that spectrum-aware itself would not work anymore. But if you do this, it actually works. Okay, so what are these examples? So I had five running examples. First one was my matrix normal example. Previously I had a 0.5 here, so the column correlations were much lower. I'm just gonna increase that to 0.9, okay? So I'm giving it to test, I'm giving it challenges. Second one was a spiked object before, so you had a VW transpose before. What I'm going to do is I'm gonna introduce a diagonal matrix R in between, and make its entries really unbalanced, imbalanced, so that you're weighing rows columns very differently. So there's 500, 250, 50. So this is what we call maybe heterogeneity in the design matrix, that may not be the best naming, but if you have a better one, let me know. We had linear neural networks, we were multiplying four IID Gaussian matrices before. Now we are taking one matrix, X1, taking it to power 100 with itself, and something independent, sure. Same with the VAR, everything is a linear combination of previous three rows, but we are making this imbalanced again. So the third most back row is the most important, so. And the final one is probably the one I'm the most excited about. I mean, this is something completely my students discovery. I have no credit in this plot. I was telling him like, how much can we push this? How much can we push this? And he just did multivariate T with degrees of freedom one, and the way he constructed this gives you IID rows. Each entry is marginally actually a Cauchy distribution, and within the rows there are correlations. And that's this plot on the bottom right. So if you remove the top few eigenvalues to deal with the heavy tail, after that what you get, the spectrum of our stuff can actually still capture that. Of course I'm not claiming this works for every kind of design, but I kind of like that at least for these five classes we could put it to test, and it still gives something more reasonable than at least what I have seen before. Okay, so in the remaining five minutes, okay. Yeah, yeah, yeah. Right, so it depends. In the heavy-tailed cases, I mean, okay, we took the signals as still mixtures of Gaussian, so all the signal coordinates were kind of, they have similar contributions. Yeah, so we didn't try settings where there is additional complication coming from that. I'm sure all of this will break down, something to check whether this guy at the end breaks down less than the others. You could still, we also tried, remove top L and then do Gaussian stuff, right? And it works probably a bit better than this, but the Gaussian stuff doesn't capture these design matrices. So yeah, other questions? Okay, so I don't have as much more time for the proofs as I would have wished, but let me maybe give you a high-level pick. So there are two steps of the proof. One is characterize the empirical distribution of this device estimator in terms of the fixed points of the vector approximate message passing algorithm, and the other is provide consistent estimate of the fixed points from the observed data. Now, the first step is the kind of results that people are familiar in the AMP literature. And the idea is use VAMP to first track your regularized estimator, then you infer about your de-biased estimator. Now, in the 2022 paper, they tracked this regularized estimator, but there were two conditions there which were sort of non-rigorous. What we did was a different proof technique, which allowed us to make this step fully rigorous. And the main thing we proved was, if you take VAMP iterates in this linear regression problem, then it has nice Cauchy convergence property. And now we have a formal proof of that, maybe that can be useful for other settings. I want to comment that analogous Cauchy property for these kinds of algorithms have been proved in the literature for the Ising Spin glass by Fan and Hu and the base match setting in the high temperature case by, again, Zo, Shubhaprotha, Yehong and Yufan. And in this case, remember the talk, Zo's talk was in the high temperature case, whereas here it's sort of my ERM problem, I'm going beta goes to infinity limit, which is a low-temperature case. So I'm actually hoping that this step could be useful, maybe for all the people interested to extend this result to the low-temperature case or other temperature cases. I don't know. Okay, and the estimation of the fixed points, it's a bunch of analysis, but it's more like really understanding how the VAMP fixed-point equations behave and sort of developing insights from that. So I'll skip over this. Okay, so maybe I wrap up because we can just still take some questions. Then what we have is this data-driven devising pipeline under rotation invariant designs. And we tried hard to come up with these use case design matrices where you can see differences from the Gaussian to these objects nicely. I was surprised at least with the PC removal plus spectrum working in all those five cases, especially when we push the multivariate T degrees of freedom all the way to one. There are other statistical applications, immediately falls out of this, like in hypothesis testing questions, false positive rate, true positive rate characterization, et cetera, I did not present that because they sort of follow directly. And nutshell of what are proofs for VAMP properties we have, I told you about the Cauchy property, but we also prove things like the fixed-point solution fixed points have a unique solution if you have elastic net. So we have an actual proof for that. And for all these fixed points, whenever you have a solution, it is always gonna be unique in our regime. We first prove that, and then you have a consistent estimate for all of the members of your solution. So some of them are like signal to noise ratio parameters, some of them are other parameters. So that could be useful for other problems. Now obviously there's a lot we can't prove and I'll mention one and then others we can obviously chat offline, but we can't create confidence intervals for a prefixed coordinate. So every time we can create confidence intervals, it's assuming that all coordinates of my signal are exchangeable and I would love to relax this condition. And I think there are lots of interest in leave one out cavity for rotation in varying designs. So if anybody has idea how to do a prefixed coordinate here, I'll be very willing to hear. And thank you. I'll wrap up with that. Guys, thank you very much. Let me check it. I think this one works now. This works now. Yeah, this will switch it on. I think it does. Yeah, that one's on. Hello, hello. Wonderful. So now we can take some questions with microphone. We already had quite a few during the talk, but I think Marco was the first and then general come to you. Thank you. It's a really, really nice talk. I was wondering if you can foresee an extension of this spectrum-aware adjustments also to other sorts of algorithms. So here you mostly look at regularized regression. So I was thinking actually to stuff like A&P algorithm themselves. So the simplest case that comes to mind is A&P for rotational invariant PCA. So there you have to estimate all the on-zucker corrections, which depend on higher-order cumulons. So those are very, very bad to estimate in practice. Like essentially you cannot estimate anything over cumulon 30 or something like that. So this means that this limits effectively the number of iterations that you can run. So I'm wondering whether this strategy, whether you have thought about that and whether this strategy gives ideas on how to do this. Yeah, so thank you for the question. And I was emphasizing a lot that we do all this massaging with the fixed point equations because that's something we kind of wanted to solve and we realized that algorithm is really nasty. So it's not like we have something that immediately pops out that gives you a solution to that question, but that's something we'd be interested to think about. And that's sort of where we were trying to go generally as a message like, okay, what can we do with these kind of data-driven insights after this work? Thank you, Jean. I think you had another question. Thanks a lot for this wonderful talk.  It's not Bayesian sitting. So there is always this discussion around bias variance. Is it so clear that you should, do you always improve performance by the biasing and for which metrics and so? Yeah, so that was a major motivation for providing. So, okay, statisticians like devising because in some communities, okay, is it biased, then they're not happy, right? And I mean communities of practitioners, but not like the theory or methods that people, but that's why we were interested to estimate the mean squared error because now I have a host of maybe regularized estimators at my disposal and the hope is some of them even after debiasing in terms of mean squared error is actually good. And also there is a tuning parameter which you can really tune to obtain better mean squared error.  How that compares with Bayes optimal stuff unclear because this is traditionally a mismatch setting analog, but our motivation for the MSC estimator was just so that you can also do this and not just devise. So you can always improve the MSC, you say, no, so you can optimize that, where did it go? Yeah, so wait, hold on. So this guy is a function of the tuning parameter, right. So this MSC estimator, this you can compute and this is gonna be a function of the regularization you chose and the tuning parameter you chose in front. So among your class of estimators, you can optimize this and choose the best one so that makes people who want really debiles stuff also happy and the MSC is not gonna be that bad compared to because you can really optimize as a function of the tuning parameter and along the tuning parameter curve, usually the dip, like where it's lowest is actually pretty low, it's almost close to. Based off, I don't know the exact gap or whether there is, but that one is not that bad. Okay, thanks. Okay, do we have another question for, yes, we have two actually, how are we gonna do this? Thank you, very nice talk. So just to add a simple question, so is it very difficult to or not so to generalize it to the generalized linear model? So then we have to go to GAMP VAMP. I don't think that is conceptually very difficult. I didn't want to go to GAMP VAMP in the same paper but conceptually, these adjustment hat and all of these things, you change the L2 stuff with some M estimator kind of thing and it's qualitatively should be similar. The proof obviously needs GAMP VAMP, yeah. Okay, maybe last question before we go on to the break. Thank you, Prakia. I have a very simple question. I think I was just wondering for the sake of culture, maybe I remember that one of the earliest works on this devising stuff was by Zhang and Zhang, which- Yeah, the three concurrent works, yeah. Okay, I was wondering what is the relationship between these methods that you're developing and those methods there? And I remember in particular, there was some step in that series of works where you need to somehow estimate an inverse covariance or sort of orthogonalize a predictor against other predictors, which I'm not seeing appearing here. So I was just wondering if you could comment a bit. So the question I think has multiple parts. So one is this was the original Zhang and Zhang estimator roughly. This guy, the Zhang and Zhang. And they had sigmas because when they were often, they were not just looking at IID gautions, they had some covariance in the sigma, then you get a sigma inverse here in front. But then to make that data-driven, you need to estimate inverse covariance matrix, et cetera, et cetera. Now, if we introduce such a sigma in our model, you'll actually get a sigma pretty much. So this is bellic and Zhang. And then if you go to our stuff, the sigma appears in the same place if it's a right multiplication with sigma to the half in the right rotation invariant, right. So the interesting thing is, okay, for the sigma case, we have a full conjecture. And something that we observe is that in the Zhang and Zhang or bellic and Zhang stuff, you had to estimate really large dimensional objects, like the sigma inverse. You had to almost estimate all of it. For us, we were able to simplify it in terms of traces of objects if you have a strong convex penalty. So even if you have a sigma, what you need to estimate is not like your entire sigma inverse. Yeah, thanks. Okay, wonderful. Let's thank Pargya one last time.


Fortunately, the general public doesn't have to... Blegh, sorry. Lastly, we're ending... Ding! Oh, that's it for this week's Fort Worth Report. Fort Report. Ew, it's funny colored. Look at the TV. Wow. We center locations. Activities include sports, sneeze. Ding! Two. One. Three, two. Summer's coming up around the corner. Are on the city's website. Just go to www. Happy anniversary! All right, we got to do it again. Check out mayfest.org for... Ding! Ha! I know, right? This... What? I know, right? They're highlighting this weekend. I can't duck. It's evening clock. The nature center. Ah! That's a word transition. I'm sorry, I'm sorry, I couldn't do it. For a detailed schedule of him... Blegh! Ha! Design to highlight Frontier Era Life. Design 3 and Military Gear... Blegh! Design to highlight Frontier Era Life. Era Life, Frontier Error. Era, Frontier Fort's Day, Screams and Dings. Magnolia Park will host another First Friday on the green of... The title is Friday on the Green. First Friday on the Green. Know Your Math is My Dings. You can also get information... Daaah! Blegh, blegh, blegh, blegh, blegh, blegh. The 65th annual Invitational Golf Tournament at Colonial Zone... 3, 2... That's not what I'm talking about. And you seem to weigh... Ja, ja, ja, ja, ja! 9 a.m. Campers will prepare 19th century food with fellow campers with... Ding! 19th century campers. They're a little on the skeletal side. So they won't eat much. Well, I don't just go to screenings with B-movies. I've been to their performance of... Goop, goop, goop! It's gonna be so good! This event... Vee... Event... It's out. This event... Vee... Chh-chh-chh-chh-chh! This event features English... Super titles! Yeah, gotcha! To find out this summer's performance, you can go v... You know, here's the thing! It's time for another gas drilling and redistricting... 3... 2... And for another redistricting and g- Redistricting! This is yet another chance to have... Wrong! Hahahahahaha! Dun-dun-dun-dun! Dun! Dun-dun-dun-dun! Dun! Dun-dun-dun-dun! Dun! Hahahahaha!


Good morning Thank you very much to the hearty souls who have joined us nice and bright and early for the start of the sessions this morning I hope you're here because you're interested in the session called what if machines outsmart us all This promises to be a very intriguing discussion. I'm Mariette de Cristina. I'm the editor-in-chief of Scientific American I'm also the chair of the Meta Council for emerging technologies of the Global Agenda Councils and with me today our Tom Mitchell of Carnegie Mellon University, welcome You tear on sigh the president of Japan Society for Potion of Science welcome Stuart Russell of the University of California at Berkeley welcome and Dileep George co-founder and CTO vicarious welcome This session today Which I'm going to ask this the speakers to speak to you in a minute is part of a series of sessions on Emerging technologies and socially disruptive technologies that will be going on at the forum this this week This talks about deep learning algorithms and how they draw powerful insights from the massive amounts of data Many of which not those of us who are only merely human can't hope to to understand in completion and They we will take a look at some of the probable and possible Causes in effects on society and on the way we live today through this session And also, I hope you will be able to take a look at some of the sessions on disruptive technologies generally So I'd like to because it's early Well, I'm going to ask you in just a minute to think about Some some questions and maybe take we're going to take some audience votes But before then I'd like to ask each of the panelists to speak a little bit about how they're thinking of machines and deep learning and AI and The sort of issues that are on their minds. Can I start with you, please? Thanks So I just want to make a few Observations the first one really begins with the title of this session What if computers outsmart us all and it reminds me of some friends of mine who talk about The possibility of an upcoming Singularity when computers suddenly become smarter than us, but my point is this Computers are already smarter than us in many ways and dumber than us than many other ways. So they Certainly do arithmetic and multiplication better than I do they analyze brain image data better than a person can They helped me find Pages on the web better than I could if I had to go through them by hand so there are plenty of ways already that computers outsmart us and It's interesting observation that each of those examples and all the others I can think of so far Have been cases where once the computer became better than I am it helped me become smarter I make fewer arithmetic errors because I build on this computer expertise. So that's one point to think about Second is the Concern that I have for at least a high impact I think on society that will come from advances in AI and that is the economic impact on jobs so computers are Already replacing automating many things that people do more of that is coming Computers will be able to drive cars autonomously for example, and there are a lot of jobs that will be impacted by that so That's an issue that I do think is worth discussion consideration my one sentence summary of my position on that is Automation grows the wealth pie Who can be against that? But automation is also bad for wealth distribution if we keep the same Distribution scheme we have now because a smaller and smaller number of people will own the wealth They will be the ones who own the automation the final point I want to make is really about privacy and Think about how AI and intelligent computers Are going to change our whole notion of privacy? So right now? I Have probably the thing that has the most data about me on earth except for my wife is this and As it becomes smarter and smarter it will Probably also have more and more information about me because it'll be able to infer it So this will have very interesting impacts on our whole notion of privacy think about the difference between Whether you've ever been embarrassed that a person found out something about you Versus whether you've ever been embarrassed that a computer found out something about you I've never been embarrassed in front of my phone yet. Thank you. I Start at my work in machine learning around 40 years ago at Carnegie Mellon where Tom has working very earnestly right now and at that time the AI or you know machine learning was Kind of single purpose, you know just for automating people job and other things and in Japan now where shogi is surpassed by AI program and High-level professionals are beaten by Shogi in our software and that sort of single purpose AI is now Supposed to overcome by more general purpose AI programs that combine are Many kinds of Functions, I bet I started my work in human robot interaction around 20 years ago and To make a smooth interaction with robots that's a very difficult for machines Sensor humans take a very subtle actions or you know facial expressions and verbal expressions and it's not quite easy for AI programs to to you know, Understand the various subtle expressions and feelings of human beings, but the I believe that Technology will know the calm that sort of multiple Functions for machines and that's one point and then will it be you know succeeded I mean will be be successful either 2045 or 2035 we should discuss about that that but anyway, that's one point artificial general intentions would come around in a sense and second point is that That The learning I mean AI programs so to help us to learn more and or to design more beautiful things or to help us to create art and Or medical care various other things and that sort of feels particular for business Education medical care design and that those kind of Feels were not quite popular in business world But they would come around in a very near future and those fields are all concerned with the human you know internal cognitive processes and so I believe that cognitive science is cognitive computing or cognitive, you know computational or No Academia would be more combined with artificial intelligence business. That's a second point That one is the same ELSI, you know ethical legal and social issues and implications Would be the very important thing for us to discuss about and to realize Some laws for robots and that AI programs that should be done. Thank you Um So like dr. Anzai I wrote my first machine learning program about 40 years ago long before I was born and AI is is in my view by far the coolest thing you can possibly study With the possible exception of of the fundamental theories of physics It's an enormous mystery and it has incredible potential because Everything we have as human beings comes from from being intelligent and if we can Magnify that intelligence if we can make systems that we can use to solve problems that we can't solve unaided then The benefits for humanity are immeasurable and so that's one of the reasons why people are so excited today and why there's such a huge level of investment in industry and And partly as a result of that it's that excitement and that investment we're seeing a lot of Technological progress the the rate of things happening which leaves relatively experienced People like Tom and and dr. Anzai and myself rather surprised That we didn't expect that these things would be possible at this time so With this progress there's bound to be impact on the real world and because of that impact The general public are seeing stories in the media About robots that are going to kill us all about the end of it end of jobs as Tom mentioned and even the end of humanity So How real are those questions? On the question of robots that are going to kill us all extremely real Right now Defense departments governments around the world companies are Designing and creating robots that will be able to autonomously i.e. without human direct control Find and kill human targets The United Nations is working hard to prevent this And this is a very important subject right now. This is not for the future. This is today On the question of jobs. There was a meeting in Davos earlier this year of very distinguished economists Nobel Prize winners one after the other they got up and said that The disruption of employment by robotics is the biggest threat facing the world economy and Their solution was that we need more unemployment insurance So I think we need to put a bit of thought into that question And then on the the last question the end of humanity this really comes from The possibility of creating general purpose intelligence There is no real threat to humanity from better versions of Google from machine translation systems From medical diagnosis systems and so on but systems that are generally intelligent that can quickly learn about any Domain that they're faced with If they're able to make decisions better than humans that means taking into account more information looking further ahead into the future Then you can think of them as like the chess programs or the shogi programs that already defeat us You know with their eyes closed and the hands behind their back But instead of playing chess just for you know the glory of victory across the board We're playing chess with those machines for the world We don't want to be playing chess with those machines for the world We want to make sure that those machines are on our side and that's the big challenge if we look 30-50 years ahead into the future is how to make sure in a provable mathematical sense that the machines are 100% on our side Being the last speaker. I have to think about something original that others haven't covered already So I do think we should develop AI and The smarter than humans There it is smarter than humans in many specific tasks already and it is going to be it's going to get smarter and Exceed human intelligence at some point at the same time It is very clear that we have to do this carefully and this is not just about Intelligence whenever we give autonomy to systems even if those are not Generally intelligent if whenever we give autonomy to systems software systems or hardware systems or things that operate in the world We have to be extremely careful that the outcome is as we expect So for example when we make cars being driven autonomously We have to make sure that the outcome is what we are expecting and this becomes more challenging as intelligence becomes more general and You know as Stuart said We need to ensure that the outcome is Regresely tested and provable. That's one aspect Another thing that I want to address that others didn't touch upon is this metaphor of We being Slave to machines, you know, we hear this all the time, you know, will Humans be slave to machines? I want to turn that on its head in some sense. We are slave to machines now Look about think think about the cities that we create we create the cities To drive cars in them and we create infrastructure for cars to you know We for parking for driving etc. Think about agriculture We we make agricultural systems to fit what the machines can do today So if you if you fly over United States, you will see this circular fields because Irrigation equipment goes in circles and so we create our agriculture to fit the machines needs So when machines become smarter, this will change. They will be more like us and they will be true partners In what we are doing. They will start understanding what we need to what what our Desire is much more. Of course, we have to ensure that that's the way it is going to progress So that's my Thank you Thank you I hope you found that as interesting as I did hearing these different perspectives and now I think it'd be a great to hear the audience by way of voting since it's it's early and I think it's fun to get involved before the the conference there were poll questions put out by the forum and so we have some Public responses to share but I'd like to read to you the questions that were asked and Ask you all to just by hands to keep it easy Although I'm sure we could have done it with phones But just by hands to keep it easy see what the tenor of the room is on three quick questions The first is if you were brought to trial Falsely accused of committing a serious crime Would you rather be judged by a human judge or an AI judge? Put your hand up first if you'd rather be judged by a human judge That looks like a pretty strong majority How about the people who would trust in an AI judge Well, some Some hearty souls there How about the second question these are fun serious though If you were diagnosed with a life limiting illness And a human doctor prescribed you with treatment regime a And an AI doctor prescribed you with a second treatment regime B Which course of treatment would you follow? Put your hand up if you would follow the human doctors treatment course Not so many put your hand up if you'd follow the AI doctors treatment course Wow That was a majority Okay, we saved I think Very serious one for last Third question If your country was suddenly at war Would you want to be defended by the sons and daughters of your community or An autonomous AI weapon system Would you like to be defended by human soldiers? No AI soldiers Stuart, you had a good flip to this question. Would you ask that also just for Yeah, so the question that you were just asked was would you like to be defended by AI weapons? So let me ask if you'd like to be attacked by AI weapons The idea that that only your country will get to have the AI weapons is obviously fiction And then the second question is if both sides have AI weapons and one sides AI weapons win What happens next? Well, then the AI weapons attack the humans because the purpose of having a war is to get the humans to do what they don't want to do So there there is no end to a war after the AI systems have had that little fight That's when the real war starts I'd like to come back to that in just one minute Maybe the audience would like to see if we could how the public voted on those three questions. I think we have them summarized So the questions were the first one there If your country was suddenly This was the last one for us But it's the first one on this slide if your country was suddenly at war Would you want to be defended by the sons and daughters of your community or? by an autonomous AI weapons system and They're the majority of the public also agreed with the room in the second question if you were diagnosed with a life-limiting illness and A human doctor prescribed you with treatment regime a and An AI doctor prescribed you with treatment regime B Which course of treatment would you follow? The public was rather close in the vote there compared with the room, which is interesting Maybe we've self-selected people who are interested in AI here and The last question if you were brought to trial falsely accused of committing a serious crime Would you rather be judged by a human judge or an AI judge and hear the public agreed with the room here? vastly preferring a human judge I Think one thing that I'd like to just pose to the the panelists the speakers here is is Why we might see such different responses to those kinds of questions. I think maybe Well, I voted on question number two myself So the medical one I voted for the human doctor today The reason is I've never seen an AI program be able to debate a human doctor and win So I think explainability is a key part of the trust that we're going to need to develop if we're going to trust computers To to make those decisions and so computers today are just not yet at the point Where they can make good explanations even if they're making outstanding decisions On the law side I think AI systems can play a big role in ensuring that the laws are consistent So rather than on the application of the law side, it can be in checking the laws, you know When we when we write these volumes of you know, 300 pages new Laws there are you know AI systems can play a role in making sure that there are no loopholes They are consistent with each other so that when something is challenged Two years down the line three years down the line There is no inadvertence in advert and mistake that got into the text of the law Yeah, I think for the most part medicine involves a fairly constrained system Which is the human body? Whereas law in some ways incorporates everything about to understand whether the law applies in particular ways to the particular case as as Dileep said Laws can be analyzed if they're clearly written There was a case of the the citizenship laws of the United Kingdom Which were translated into the prologue language, which is the computer language that machines can understand And then the machine actually found that the law was was inconsistent with itself So it was possible for someone to be a citizen and a non-citizen at the same time So as a result, they actually had to fix the law to make it consistent I like to be diagnosed by a Very well trained web-trained doctor You know, I'd like to be judged by a very fair judge, you know and the humans are very distributed and the minds are distributed and There are some our studies in psychology that for Mines, you know are and mind-workings of judges and they are very distributed and so are and AI systems are now growing and being developed very of course very rapidly and I agree with that For medical diagnosis and something is I like to be judged by AI doctor and but for judges still Humans, you know, minds are very distributed, but still are you know AI is following human beings, I think So it's it's worth mentioning that I believe in Pennsylvania. They are using a machine learning algorithm to judge parole cases Based on indicators of the behavior of of the candidate for parole and the nature of the crime And other characteristics They've also shown that I think Judges in Israel who are deciding parole cases The percentage of time that they grant parole Declines dramatically from the time right after breakfast when they're feeling good To the time right before lunch when they're tired and hungry and hot So it goes from like sixty three percent down to something close to zero So I think the answer is if you're going to be judged by human judge have it right after breakfast Can I just riff on that one point I Think many of these Questions about would you have a computer or a person do this are? Missing the point that so far whenever computers get smart Us humans use that to make us smarter And so if we go back to the case, which I don't know the details of of Pennsylvania using computers to make parole decisions My guess is that there's a human judge there taking the advice of this Computer which is looked at a lot more data than that judge has and so I think that's another case of computers Outsmarting us, but then making us ourselves smarter. So many of these questions that we're thinking about except for the war one are really a question of not either a computer or person, but perhaps Person informed by computer One point I think Long being missed by your air research is as can trust as Tom mentioned, you know, how Human beings can trust each other it is of course concerned with intelligence, but Not just at all. I mean We I've been working in the field of human robot introduction and how humans can try bit can trust and robots That's a very interesting Research field and it really depends on the how robots and humans Converse each other, you know make compass conversations and how We can be or you know Absorbed into robots mind and that's sort of you know behavior or you know cognitive emotional processes are very important for future technology. I believe So so far this is maybe a good time to just synthesize just a bit so far When we ask the question, what if machines outsmart us? I think I think it's interesting how the panel has been addressing this so far I mean we've mentioned that so far machines are smarter than us in certain ways We've meant we've mentioned that we We see economic impacts from that social and privacy impacts Even questions about our humanity and our way forward and how do we trust them? I heard I heard we want to be sure that these machines are 100% on our on our side ultimately and also that they can help us To be better ourselves in many ways, so I think we've been gainfully Looking at things that are near or farther away at this point. I'd like to ask you What is the one if you could pick one Most exciting or urgent area that needs to be addressed in this and in this realm right now What's on your mind most heavily? Maybe I can guess yours So as I mentioned earlier the the question of autonomous weapons is is quite urgent So the United Nations has run three meetings already The last one in April I was invited to to address and explain how autonomous systems worked and what the likely future evolution of intelligent robots would be in the in the sphere of warfare and I think we are we are facing a very immediate choice We can we can have a treaty which would Which would put very strong constraints on the use of autonomous weapons or we can begin an arms race And if you look at the likely outcome of that arms race It seems to be a future where The capacity to kill millions of human beings Would be in the hands of essentially anybody who can afford You know, maybe 20 million dollars to buy a million microscopic robots So you can build a microscopic robot a small flying robot That carries a one gram high explosive charge Which is enough to blow a hole in a human being's head And there's really no serious way to defend yourself Against a large cloud of insect-sized robots. You can't shoot them down. You can't fight them off and you don't need You don't need a large number of supporters. You don't need a big civilian infrastructure You don't need a large military to run this at the moment the thing that prevents mass death Is the fact that in order to carry out mass death? You need an awful lot of people if you want to have a nuclear weapons capability. You need a huge Scientific industrial and military infrastructure to carry that out if you want to kill a million people with rifles You need a million soldiers You can't have a million soldiers without five million civilians to to support them and feed them and train them and look after them So nation states can do this kind of thing, but individuals can't when you have fully autonomous weapons Then you can launch an attack against the city of a million people just with money And weapons and you don't need all the rest of it So it creates a situation where the balance of power is tipped away from nations and towards non-state actors Or nations that are irresponsible and aggressive So it seems clear to me at least and to many other people in the community that this is a direction that we don't need to go And in fact the Financial Times had an editorial In support of the open letter which was written by 3000 AI and robotics researchers a title of that editorial is a nightmare that we don't need to invent. I think Russell is right and We should be prepared and we should discuss more about that issue of you know applying AI to wars and and my opinion my thought is to Move that That technology, you know advancement of technology more towards more consumer oriented businesses and industries and I think No, I said about human robot interaction and human interaction with artifacts and you know artificial things or Is sir is a field with You know great amount of future for not just for academia, but also for industries and Consumer oriented businesses and that is because you know if humans want to or You know communicate with Artificial and artifacts and You know our AI can support us To buy your big data analysis so machine learning and with any other things deep learning and other things are Since humans are very sensitive to very subtle No feelings or expressions and eye movements and other things and But still like AI is more oriented to kind of single or double with triple purpose Girl oriented or you know technology and human interaction with artifacts are more than that and I bet that of course learning medical care and judgment human judges are and us and design and creating or more kind of futuristic things are You know are waiting for us You know businesses and not just for academia and I like I'm you know are to turn that energy for you know AI war To that sort of you know or human oriented businesses and technology that would be the most important and ELSI issues social or an illegal Movement would support us for that. So that's a very serious topic Another issue I think we should all start discussing and addressing is the issue of jobs So I think the future of AI is bright And a lot of work will be done by machines a lot of wealth will be produced by the machines But there will be a transition period where jobs will be taken from humans and our Policy makers will need to start addressing this issue So Arthur Clark once said the future the goal of the future is full unemployment so that we can all play And I do believe in that it just that we have to rediscover the rules of play What it means to play that means everybody having a turn and I think policy makers will start start to Address this issue pretty soon Could you just just for one minute because I'm intrigued by the idea of play and nobody having to work But could you just speak to what it means for everybody to have a turn? Yes, so I do think in some future AI will do a lot of work for us and It could happen that humans don't need to work In do to make money or to to produce things for ourselves Humans do engage in things that are not work. For example, we play sports. That's not that's not because you know It is it is something to challenge us. It is not because you know putting a Ball in the basket gets us, you know, it doesn't produce anything But it is to challenge us so we do engage in activities which are challenging for us rewarding for us not necessarily to produce anything out of it just for the intellectual challenge just just to push ourselves and Our society can be taken, you know, we didn't have sports when we were hunters and gatherers We were we were all almost all the time we're spent in Searching for food technology enabled us to have more time for leisure entertainment, etc And I think that can happen in the future as AI does more of the produce protection for us. We just have to Like you know deal with it we just you know right now a lot of our economic systems are built around scarcity and Fighting for resources and that even that might go away. And so we don't we just have to manage that transition period well I guess I would Let me switch a little bit around When I'm thinking about the future impact of AI one of the questions I keep asking myself and my colleagues is What are the possible? Advances that could happen in the next let's say five to ten years that would really be game changers in terms of the economy and beyond and I just want to throw out a couple ideas that I think are Likely game changers in the next five to ten years. One is autonomous driving vehicles. We already have prototypes that demonstrate that Cars can be driven by computers more safely in many cases than they can be driven by people eventually society will get into that and I think that will be in the coming five to ten years the ripple effect of that It's huge. How many jobs today? are effectively jobs of Transportation that involve people Second one What if computers and I believe this will happen? Has a good chance of happening in the next five to ten years? What if computers get to the point where they can read really read like you and I read and Understand text in its full meaning. I'm not talking about a search engine that can find webpages Containing the words you type in talking about a computer that really can read and understand the document Reason about it summarize it for you answer test questions Decide whether it agrees with that document or not that would be another game changer and Once computers can read they'll read the whole web They'll be better read than we are by a factor of a million and that'll be a game changer. There'll be many jobs that we do today that are Based on knowledge that we've required by reading that'll be a game changer a third one I think is just the whole area of medicine and the thing that's going to make that a game a game changing economic situation is the huge volume of new data that is being collected on our bodies from smart watches and other devices so that coupled with machine learning is going to just change the whole game in terms of Effective medical treatments that are possible and there will be a huge economic ripple effect of that So I would just throw those three out as Among my top five Likely to have a huge impact on the economy and beyond in the next five to ten years So I think we've heard we've heard Very top of mind concerns about autonomous weapons which you know a Thing I think is is very present how we might shift to AI to support more consumer oriented businesses How we might need to deal with the jobs issues that are raised by you know imagining a future perhaps where We're looking for challenges like sports but we don't have to fight for survival in a way the way we do today with our work and These ideas of game changers autonomous driving vehicles and how much of work is involved around that what if computers really read and understood the way we do and How how they might Understand medicine through all the data that's now gathered on all of us every day I think maybe this is a good time to ask if the audience would like to ask a question or two of the panel as well Areas that are of concern to you. I think there's a microphone around There it is. Please. There's one right in the middle there, please Right there Just identify where you're from, please. I'm Martin Brunsk a company air mobile I just want to caution us against this vision of future where We don't need to actually works because the computers will do all the work for us and we You know, we can just have fun and have a good time because They're not just economic reasons, but there's also deeply psychological reasons why people have jobs And actually if you look at what country is feeding European country Most terrorism recruits for ISIS in the Middle East. It's probably France and these people are people who don't quote-unquote need jobs You know, they are young unemployed People of Originally Arab origin who don't have jobs because they're ostracized, but technically they don't eat they don't eat to work They can just you know, enjoy do sports yet They feel there's that there's something deeply wrong with their lives and are kind of feeding this type of problem So I think that a future where? We'll have this amazing AI will do all the job for us all that all that you know Searching from food to whatever and we just have a good time. I think it it kind of ignores a lot of these potential problems stemming from human psychology There's a question over here Hi MIT rethink robotics Stuart, I promised I wouldn't ask a question, but I want to give you your chance to redeem yourself By posing the question a little more crisply that you answered I think in a way that may mislead some people You you're asked about killer robots and you answered about Weaponized robots weaponized AI systems But I think a lot of the popular press when they talk about killer robots There's a myth I called the myth going on that we're soon gonna have AI systems Which are gonna get tired of humans are gonna decide to get rid of the humans and gonna eliminate humans Which is very different from a weapon system and and I just want to give you a little chance to Expand on your answer beyond that thing which is of great passion to you to the other way it may also be interpreted Yeah, no, I I don't think I misspoke. I don't think I implied that we're talking about Killer robots that will turn on the human race A particular aspect of it and I think that's where the press sometimes gets confused Okay, so just in case there wasn't a confusion and I don't think there was but in case there was we are talking about weapons that are At a high level controlled and directed by humans and used by humans against humans And this is not sort of us as in humanity versus them the robots This is us humanity versus us humanity And you know we have a long record of misusing weapons And this would just be a continuation in a perhaps much more severe form of that glorious or inglorious history So I just want to come back to something that delete said about That our policy makers are thinking hard about the future when there is no work I'm touched by deletes confidence in our policy makers But I'm a little skeptical that they really have an answer to the question and I think the gentleman said put his finger exactly on the point that That just left to ourselves to play We're not necessarily terribly good at Organizing ourselves to do that in a constructive way that we need to actually have Should we say a migration plan a transition plan from the kind of economy we have now to some future economy Where people still feel engaged and valued for what they're doing and part of a larger enterprise or effort That that gives a sense of value to the individual We have absolutely no idea Position plan I think that that raises a question for me anyway, or probably for a lot of us like what What are some of the you know, what is a future you'd like to start to see? What are some of the transition plans we need to put in place to get there? As Stu said, it is very hard to see Because so one thing that we don't know is what the timelines are so policy makers Are compelled to action only when they know what is going to happen and when so we we have only ideas about Especially this future stick things where people don't need to work anymore. Those are we don't know when exactly that is going to happen so that is that is definitely a challenge and And also how exactly is it is going to unfold which which jobs are going to go first? I think like you know in near term we have Good ideas on you know, which you know, for example as Tom mentioned driving jobs are going to go away and Some jobs about you know reading and understanding text and summarizing them. Those are going to go away We can plan for those but what's the next wave is harder to predict and that is that is the challenge? I've been working for a long time and fields of Cognitive so-called cognitive sciences including a psychological and brain Sciences and from that viewpoint there are so many unexplored topics and fields for human minds and We need to it is pretty urgent that we need to explore That sort of internal processes of human beings to apply those kind of things to create more, you know advanced technologies and also to create more Created create jobs, you know There are so many unexplored jobs are waiting for us and As I said design or you know, of course medical care and also education learning and other kind of things are Quite unexplored fields for businesses and also academia and I bet that We can you know explore and we can create more kinds of businesses and more kinds of jobs in future, so I Not quite pessimistic but that but we need to prefer that prepare for that and we need to invest Some money at least towards that future society I would just add to that First of all, I guess I just want to say that I agree with the speaker in the audience who suggested that People who want to do something with their lives They want to understand that they're moving forward the world and society in some way and so It's more than playing tennis Although tennis is fun But when it comes to policy as Stuart mentioned, we really don't understand What that future exactly is going to look like so it's hard to prescribe What those jobs will be given that situation the smart policy thing Smart policy move in my mind is to look at the education system and how to change that to prepare for the day when it won't be That you'll have one job throughout your lifetime, but instead will be that you will Progress will be happening quickly enough that you'll probably have multiple jobs through your lifetime. And so one policy Decision or topic that could be studied is the idea of changing the grain size of education from for example, four-year study to become an expert in some topic so that you can work for 40 years in that area to Perhaps instead just in time micro education and a Kind of educational system where we can Perhaps subsidized by the increasing wealth pie provided by automation Be able to dip into and learn new skills That allow us to move in and make new kinds of contributions that we couldn't foresee when we were a kid choosing our major I think this touches on an interesting area, too. I mean classically policy and regulations follow Technology and now you've mentioned investing I've mentioned a couple of ways for us to be adaptable. Maybe education is one Is there any advice we can give to to policy to help make that more adaptable so they can respond to the technologies as they Continue to shift around us One piece of policy advice that I think makes sense is measurement that is government should be measuring the progress of technologies Making predictions not just of what next year's GDP will be but what next year's and Five-year horizon Technical breakthroughs are likely to be you can imagine my friend. Yoav show him has suggested That just like we have a stock index and a futures index perhaps we should have an AI index that is a collection of quantitative measures of progress and Anticipated progress in the field that could then help governments track what's really happening and what's really anticipated to happen So that they could respond With policies that are appropriate so I think One of the problems perhaps with economists is they're extremely good at Analyzing an economic system and saying, you know, yes, this would work. This is incentive compatible The various participants would all play their role in such a system Or this other proposed economic system would fail because these this group would defect Or there would be imbalances that would diverge and so on But it's not part of the training of economists to invent new kinds of economic structures and systems new kinds of societies And in fact if you ask, you know, whose job is it to do that? You'd have to say science fiction writers So I think what policy makers should probably be doing is in some sense recruiting The best science fiction writers or people related to that people who were able to imagine different kinds of systems And have those people sit down with economists and and get some some real-world feedback on the realism of their of their various Ideas, but it seems to me that this ideation is absolutely central to what has to happen because Absent that absent a real goal towards which policy can move we're likely to drift in the direction That's happening right now simply Greater and greater automation greater and greater concentration of wealth And gradually discarding larger and larger parts of the human population And perhaps providing them with with a subsistence wage But no real meaning to their lives Probably if they were just being given a subsistence wage, then there's no need to educate them any more So we'd probably you know reduce the amount of money put into education as we are already doing So it seems like things are going in in sort of the wrong direction In this sense and it's because I think of lack of a goal Towards which people can make changes I agree with the rest of Russell that I like economists to do more things about the you know deal with the concept of information and you know our Economics are or no academic I mean our academic economics should more not be concerned with Information and which is dealt by AI technology and I don't think I'm of course a novice in economics, but I bet that Economists are you know more contribute to or hours or you know of heels by you know Consent be concentrated more on their on dealing with the concept of the information to be introduced to the economic field of economics and one more thing is that about policy and Our my organization is is just be a Japan society for the promotion of science Which is a represent the representative organizational funding agency for Japanese government and That sort of agency is located in Almost every country of course in the United States like NSF and Probably we need to Communicate each other or start to communicate each other about the you know what the impact The investment of AI technology for the future and that's I think when I was in problem I learned from this panel. Yeah Thank you so Is it Some you've let me with some real thoughtful things here So how you know the struggle for timelines although it's sort of interesting to me that AI has been something That's been coming for so long and some aspects are already here and some we don't we don't know where they where they are yet How important it is to develop the cognitive sciences alongside of it? How hard it is to Because it's hard to prescribe scenarios some ways where we might get more flexible With that's looking at the education systems and how they they work and also thinking about how we might ideate to look forward and to Have goals to prevent a really sort of dystopic scenario that I think It sounds like there's some concern on the panel that we would like to avoid and and Always important investing in the future that makes me want to just turn to the audience and ask if there are any Last question or two that people would like to I see there's one on the left-hand side there Yes, the honest precinct from manpower group. So what I'm curious to hear from the panel is The evolution of mankind and progress that has happened with the disruptive technologies from a historical perspective Has always ended up in the same place Which is greater wealth creation and after a reasonably sizable time of disruption Especially as it relates to jobs that were replaced or displaced more people eventually found more jobs because more categories were created So if I look at this change, why would AI be any different than what we've seen over the last 250 years because I'm sure that if you were in England in the 1860s And you're you're running your loom at home or you and your steam engine comes in you would say well This is a big change and the world is is really about to change and in the end it evolved You know for the benefit of all those markets and for the world at large So why would this time Things be so different that we shouldn't expect after a time of disruption wealth creation Be distributed roughly in the way that we've seen it Historically why is AI different from anything else? Why is AI different? Why would this time in history be different than all the other times when we've experienced disruption that have caused great wealth over over time of course so so look let's think back to the The agrarian revolution and the industrial revolution and The the the work that humans could do by virtue of their physical capabilities Has has largely disappeared And the same is true for horses the horses didn't have their minds to fall back on And so when you look at what happened to horses the number of horses in the US I believe went from 80 million in the mid 19th century to just about 2 million In the early 20th century so So they lost out big time They became cat food basically Because they couldn't fall back on their minds the human the human race fell back on its mind And most of the work going on now involves some application of the mind The difference about AI is that AI also to some extent replaces the mental work that humans do and what else? What's the third thing that humans are going to fall back on? Emotional emotional side. So there's the physical side the mental side and the emotional the empathy side Yeah, so so the question is is there an economy built out of empathy? And as yet. No, not really Empathy is is is not a terribly highly valued Thing there are there are some Job categories that involve mainly empathy some kind of social work Certain kinds of caring jobs, but these are actually very low pay You know personal therapy if you go to some parts of California. That's a very large Employment category personal therapy psychotherapy massage therapy But it's hard to imagine an entire economy built out of that. So that's the open question. Thank you I think it's a pity that I'm so sorry that we're actually out of time And I think this point on this high note actually of intelligence, you know I've been reflecting as we've been talking about how these machines are in many ways smarter than we are there are Our partners and we're theirs as we go forward to a future of you know And a more I'm going to put quotes around an intelligent world where as you'd said at the beginning Many things that come that are good for humanity come from our intelligence And I like the idea of the empathy economy. Maybe we can explore that in a future session. Thank you very much