What I want to talk to you about today is an exciting field Called conformal prediction and it's exploding with research and new ideas at the moment And so I want to tell you a little bit about what's old and a little bit about what's new and so Let's get started and we're going to introduce collaborators a bit later on Let me just jump into the subject. This is a map of the United States Showing the results of the 20 election the prediction election county by county So each county is indicated by a color and if you see a dark blue color It's a county where Joe Biden improved enormously over Hillary Clinton from 2016 to 2020 by about 50% on the other a dark red Color indicates a loss for the Democratic candidate a loss of about 50% And so you see from this map that has about 31 in the United States See from this map that Joe Biden did pretty of course. We all know that he won the general elect now If you were to work for a news organization such as a Washington You would like to be able to tell Your readership what how you think the election is going to end up looking like once all votes have been tallied And so this is a task that the Washington Post set out to do and as you'll see they're using a Tool that you'll see in these lectures But this is an application where the prediction is very important and the cost of being wrong Extremely high that is calling Pennsylvania for the wrong candidate has enormous consequences And so what they set out to do is Basically at any given time in the election you see a number of reported counties And I'm not sure you're familiar with elections in the United States But it's a given time in the United States and let's say 1200 counties have reported you have 19 Outstanding counties and you would like to predict the vote in these counties that you have not yet seen in such a way that you're correct 90% of the time and So what they set out to do is each unreported county they set out to issue a predicted range which is correct Time that is for the 1900 Is the range needs to include the true 90% of 1900 times and So this is work that was done by John she and the graduate student of mine at Stanford and Laney Bronner who is a data scientist at the Washington Post desk and So if you were to look at the Washington Post portal Soon after the election and here is a shot. I took on November 5th 2020 at 1250 a.m The election was on November 4th at 20 You would see that even though a pen job Donald Trump time as far as the reported votes was leading in Pennsylvania The Washington Post Actually predicted that Joe Biden would end up Winning Pennsylvania, but not only is this as I mentioned before They wanted to actually tell facefully to their readership how they think the election is going to end up and rather than just giving you a point Forecast they give you a distributional forecast which is indicated by sort of this coloring Where essentially a dark color is the median of the forecast and as you go A bit on the right you get in the tailors the forecast and they wanted this forecast to be well calibrated in the sense that I described before and So right now we see that the median forecast for Trump is a for Biden is a bit higher than the median Donald that is a correct prediction and It was very interesting to see is during the election a week you know we would see plots like this and then as more and more votes would come in of course the Gap would widen or the distribution range would narrow All right, and that's how they communicated what they knew about the election and how now I don't think I need to tell you that We use machine learning nowadays in extremely sensitive applications. I don't know. I think I had this discussion last night I don't know if you know about this, but in the United States we use machine learning to actually Decide whether inmates will get parole or not. So these are extraordinarily consequential decision We use machine learning to put you on a treatment plan We use machine learning for self-driving cars Things where the cost when you're wrong the cost may be extremely high and so this asked a question which is Since we use these systems in critical applications. Can we be sure about their predictions? Can we have confidence in these predictions and that's what we're going to talk about? So, you know at the risk of being a bit pedantic This is a bit what I consider to be data ethics 101 You know as we use these systems that are extraordinarily complicated It's very important that I'm you know to understand how certain I am of my prediction when I communicate To a judge whether a person is going to come in another crime if released from jail You know I would like to certainty about this prediction I Would like to know how my level of uncertainty my decisions if we think about finance for example You know if I say well here's a big portfolio and I expect its return to be five percent There's a big difference between five percent plus or minus one percent and plus or minus ten percent Can you deploy an AI model safely and I think To be explicit about all of this to help users so Solution to quantifying uncertainty It by means of what we're going to call prediction intervals or prediction sets And I should say up front that there are not confidence intervals for a very obvious reason that it's going to be clear But Basically the goal and the thing that we're going to talk about in in this hour is as follows So you have training data so excise might be the covariates of county You know demographics about the county etc. The level of education income whether it's a rural county or not The size of the county and so on And then you have an outcome like how does a county vote? And we're going to assume that at the at the beginning that this data is assumed to be exchangeable and If you don't know what exchangeability means that's that's all right You just can't assume that the X I Y are your I ID from some unknown distribution P the classical machine learning assumption And the goal that we're going to set out for ourselves is to actually based on the training data To construct a prediction interval that is the range of possible outcomes that we're going to call see effects What these was it needs to contain the true label a certain fraction of the time and for this lecture We're going to just say 90 percent. It's whatever you decided to be but it's going to be a certain fraction of the time So I need to make sure that from the training data. I can construct a prediction set that contains a true label 90% of the time and the surprising thing is that I want this to hold No matter The distribution generating the data no matter there's a sample size no matter the dimension no matter nothing and The first time at this well, you'll say I can't do it and of course if we have this lecture is that we can do it And what I want to be able to say going back to my election And I don't know if it's still working I hope it is the vote change from 2016 to 2020 is predicted for that county to be three between three point one percent Seven percent and I need that whenever I make nineteen hundred predictions of this kind 90% of them are okay. All right So I also want to do something else, which is I want to use these fancy machine learning prediction algorithms and Well, it's difficult to quantify their uncertainty because I can't barely understand them But still we want to use them and that's what we're going to do as well So to do something like this and I guess that's the main message of this lecture is I'm not going to open the black box It's too complicated. It's way beyond my analytical powers What instead are we going to do is we're going to build a predictive layer around the black box and so we're going to look at the predictions of Machine learning and we're going to use build a layer of protection on the machine learn the prediction algorithm There's a kind of prediction intervals. I talked about with the guarantees that I want and I'm going to explain to you how we do this So the first thing you might think about doing if you want to kind of predict the accuracy is Okay, so I have some data points and they're indicated in gray on this and then I fit a model Indicated by the red curve and of course you have to imagine that I use the deep net to do something like this in modern and like the Residuals Might be Say that well, I'm going to look at the errors and build a predicted range which is around the model prediction the size of the errors and that natural thing to do Except that I don't need to convince this room that this is completely silly This is completely silly because we all know that training errors are much lower than prediction errors And in fact when you train machine learning algorithms these days We train them to zero training error And so you would have a band of your width and clearly it's not gonna in contain future labels So it It's very extreme for modern methods Informal prediction comes into play and I should say that conformal prediction has been largely developed by a hero of mine named Vladimir Volf who started to work on this about 20 years ago and what he showed with His colleagues including Glenn Schaefer that you see pictured here is that you can actually build valid prediction intervals No assumption whatsoever except that I gave you Expendible training samples for IID training samples. So how does this work? so Rivers, I'm going to try this one. Thank you Normal prediction comes in many different flavors. You're going to see a fancy flavor, which is called full conformal and It works roughly like this and then we're going to see a simpler flavor in a minute But here is one way you can deploy conformal prediction so I have a data set and here it looks like what we see on this on this chart and My boss comes around and I see a data point With Xn plus one equals four point seven and I need to predict The why value that I'm not seeing in such a way that I'm correct 90% of the time So the way conformal prediction is going to work is I'm going to hypothesize a value of y a candidate value little y and I'm going to fit a prediction model perhaps with your deep learning or Random force or XG boost however complicated. This is I'm going to fit a To the data Including Xn plus one and the hypothesized value little y I'm going to get residuals Then I'm going to do is I'm going to see how the residuals for the imputed value y how unusual is it? So to do this so I have a little animation and here it works like this. Here's the fitted model. I Compute residuals and I look at the magnitude of the red residuals the value little In how unusual use I'm going to put some sort of a p-value and here I see that well this residual is In the top 27% of all magnitude of residual magnitudes And now I'm going to change the value of little y and I'm going to refit the model and I'm going to play the same game again And now I see that well this is a bit larger because why moved up and now it's in the top 22% And now we're going to keep on moving and Animation and note that I try to be careful which is when the red dot moves up It pulls the model towards it right? So if you look at the black curve, it's being pulled upwards by the red dot So we go we go we go and the residual gets larger and larger obviously Okay, and there's going to be a point where no matter why I put it. This is just residual So I've trained all this train all the I'm looking at how unusual how it conforms to other residuals and So for each hypothesis type is value of of why I have a p-value if you will which is in what top percentage you belong to and what I do now is I'm going to build a production interval in such a way that I'm going to include why if it's in the bottom 90% and Not if it's in the top 10% so that it's usually sufficiently small and Mean in the bottom percent. I keep the y value in my prediction of all otherwise I don't include it now you'd say this is a Perhaps a good method will see that it's actually delivering the promises that we wanted You'd say well that expensive because each time you propose a test value You're gonna have to refit the model and so if I fit a deep learning model, that's very expensive I agree with that and that's why there are many other proposals in the literature that will bypass Refit including my favorite one the jackknife plus which we developed with my collaborators or CV avoids Refitting things all the time. All right, but let's keep let's stay with conformal full conformal prediction for net and I'm going to show you that it's actually work. So this is my most mathy slide. So so here is so so we have exchangeable training data and Does everybody know what exchangeability means? Let's assume that they are IID exchangeability means you Equal random variable Okay, I'm going to try again with this have a stick when changeable random variables are as follows If I give you the realized value of the sequence and I tell you and ask you what is in this bag of realized value What is the first observation and the second one and the third one you would be guessing at random? That's what exchangeability means Another way to think about this is that the distribution function is symmetric in all of its arguments The best way to think about exchangeability is to say that if you see the entire sequence, but unordered You have no way to tell what is the first observation? What is the second one? What is the third one and so on? so it's a relaxation of the IID assumption and So let's say I have exchangeable data And so here it is x1 y1 x and yn and then I'm the test point xn plus 1 yn plus 1 I'm going to fit the model To all the data point including the test points. You say what a manual you can do because you're trying model Well, this is a bit capricious.  So we're gonna train the model with a test point and all the training points And now I compute my residuals And if my training model is symmetric in the data point does not depend upon the ordering I gave you Then it's a very simple exercise to see that The residuals are exchangeable In particular If I ask you what's the chance that the test residual is in the bottom 90 percent of all magnitudes Well, you say Because it's equally likely to be any one of the of the of the residuals that I've seen Well, by definition it's 90 likely to be in the bottom 10 percent Right, so this is by definition of the algorithm the fact that you use a symmetric algorithm And the fact that you have residuals that are exchangeable in particular the test residual Is equally likely to be any one of them And it has a 90 percent chance to be the bottom 90 percent All right, so that's easy enough And now we're going to say all right. What did we see on my picture? We see something that looks like this I'm going to propose to you a test value y which belongs to r I'm going to fit the model mu hat to all n plus one data points And I'm going to compute the residuals and now I can compute the test residual with the proposed value because It's little y which is here here And I'm going to check whether when I plug in this little y I'm in the 90 percent bottom 90 percent of not if I'm in the bottom 90 percent I keep y in my prediction set if I am not I will reject it from my prediction set Right, so if it's in the bottom 90 percent we keep it And if it conforms to other residuals And of course the result is this the result is that the chance that this procedure includes the true label Is exactly 90 percent it is not 92 percent it is not 88 percent it is 90 percent and why is that because if I plug for little y y n plus one Then by construction it is exactly 90 percent And so you've achieved Exact predictive coverage Is this clear and so it works All right, so this is the full conformal procedure as proposed by wof in 2005 And as I mentioned before it's not computationally very attractive because you have to do a model refit each time And people prefer not to do that So one way you can actually avoid totally model refits is Is this which is you give me a data set a training set And then i'm going to split it into a training set and a calibration set And what i'm going to do is i'm just going to fit my model On the first split on the training set And then i'm going to calculate the magnitude of the residual on the calibration set with the fitted model And if you follow what i'm saying it's it's exactly what we've seen before except it's simpler Which is that it's like saying that the model mu hat has been fitted once and for And so what i can do is i can use a split to learn a model mu hat I can evaluate residuals on the calibration set the holdout set And then i'm going to Employ exactly the same procedure i'm going to keep all y such that Their residuals if i impute y is in the bottom 90 percent And if anything the argument for why it's works is actually easier It's just saying that mu hat has been fitted once and for it's been pre trained and that also works that yields Exactly predictive sets with 90 percent coverage And what's nice about this procedure is that of course you fit a model only once Now what you lose is you lose some data Because now a data point is used either for model fitting or calibration Not both before we were actually using Data points for both the model fitting and calibration So there's a loss of statistical efficiency here And that's why with my colleagues we have the jackknife plus which i'm not going to discuss in this lecture That actually can't reuse us in a smart way data points for both Calibration and model fitting without the expensive cost of refitting a model each time Okay All right, so If you do something like this then you're going to get Coverage and so here we see a data set which is artificial We have an x value. We have a y value. You can see that the data set has A bit of geometry. It's not homoscedastic and everything And I don't know why it doesn't come out so well. Oh, yeah And so what you can do is you You can actually fit the mean function Which is what i've proposed to do in the previous slides So we can fit a model to kind of discover what is the conditional mean of y given x And then compute residuals and get prediction intervals But what it will do is it will actually do if you think about what we're doing we fit a model We compute residuals and then we build a band around the model of a size that has been Determined by the quantize of perhaps hold our data And so that would construct a predictive band of constant widths Which does not really adapt to the geometry of the data and so Instead what we might think about doing is to start and rethink the algorithm completely from the ground up And say well, perhaps One thing that i never understood about the work of latime evolved is if you want to this to deliver Prediction intervals, why do you start by estimating the mean? You know you're clearly after the quantize of the conditional distribution Why should you start by estimating the mean? So perhaps we should start by estimating the quantize And so here you see You know an estimate of the upper quantile of my distribution You see an estimate of the lower quantile of my distribution But i might ask you you know i got this by random forest is this well calibrated and of course it is not But we're going to pass this through the same technology that we've just seen We're going to look at our initial guess and we're going to calibrate it so that it acts actually exactly returns 90 coverage So how do we do this? Well, we're going to fit quantile regression Then we're going to look at some conformity scores, which not are not going to be the magnitude of the residuals They're going to be something else And then we're going to calibrate them All right, so let's do that And this is going to yield a method that we call conformalized quantile regression Where we're going to get intervals that i adapt to the geometry of the data They have exactly the same coverage properties than the previous method, but that you can see the The length is very adaptive or the previous one is not Okay, so how did we do this we did this by essentially Extending a bit the framework of conformal prediction to say well You don't really need to use residuals as conformity scores. You can use any statistic you like And so if you follow a bit what we've been saying, which is like what you can do is you can pick any conformity scores function s of x y You apply the algorithm before x of x y was for example the magnitude of the residuals But you can pick anything you like You apply a symmetric algorithm to the data points and a hypothesized value y you get conformity scores You look at the 90th percentile That's the number q and you're going to include y if the conformity score applied to the test value y is in the bottom 90 percent And by exactly the same argument as before It's going to work the coverage is going to be again exactly 90 percent So how did we get this interval over here? We got it by Well, what i'm going to do is i'm going to estimate a lower quantile and an upper quantile They may be miscalibrated, but i'm going to recalibrate them by use of a clever conformity score And the conformity score in this example Is simply essentially is a distance of a data point to its nearest quantile. It's a signed distance To its nearest quantile So it's has a positive sign if you're inside The region and a negative sign If you're outside of the region, so it's a distance to the nearest quantile And so What you're going to do is you're going to exactly use a strategy We talked about which is you're going to include the data point if you below the 90 percentile of this conformity scores In this case, I did the math for you and it means that what you're going to do is you're going to move Up and down your quantiles until by by an amount q which is a the the The quantile of the conformity score. So for example, if you did a very good job And you are well calibrated to start with you will not touch it If you are a bit too optimistic That is the coverage was too low you would enlarge your prediction band and if you were too conservative you would actually shrink it And and no matter how you get exactly 90 coverage All right, so we can apply this to any data set you like And so here's a data set where the goal is to actually try to predict medical expenditure from a number of Variables such as your age your marital status your race your poverty status Your functional limitation the kind of insurance you have and so on and so forth And it's a kind of fairly high dimensional data set. We have about 140 features and 16 on 16,000 subjects and You measure healthcare's utilization as measured by the number of visits to a doctor's hospital to a doctor's office or hospital and so on And What you get is you get very boring box plots and I love boring box plots Because what they show Is that you know if you want 90 coverage no matter whether you use a residual method I showed you first or the most sophisticated cqr method They both give you 90 coverage And you want 90 you get exactly 90 Now the advantage now is If we look at the lengths of the prediction interval Then what conformity scores you you're using starts to matter a lot And so what we can see at least on this example is that if we were to use cqr Because it's more adaptive you would get intervals that are more informative. They are shorter Not only this but if you cqr You have Although the theory cannot predict conditional coverage that is coverage given the value of x that cannot be done In a model-free fashion We can still test for conditional coverage on on empirical data And the empirical coverage that the cqr algorithm achieves is very close to the nominal level And it's higher than what you get by residual quantiles, you know the residual quantile method tends to over cover When you have lower volatility Over cover when you have reduced volatility while the cqr method is a bit more adaptive So it seems to be A better algorithm and that's sort of what people use nowadays and that's what the washington post uses Now you'd say well, can you move beyond? quantitative outputs and the answer is yes So let's say that now I want to do something a bit different where you know, maybe my label is an object class for example Um, okay. I'm going to explain how we did this. So Let's start with my example first, you know, we have objects and we try to recover objects in images And we're going to see a conformal method that just does this and so you have three images And here is what the conformalizers says You look at the first image and the conformalizer says i'm pretty sure that it's a fox quirk. That is my 90 percent predictive set Says has only one element and it's a fox quirk Now we can move to the second image And I hope you see the second image. Okay, the conformalizer says, well, I'm a bit less sure now And you know it might in my bag i'm going to put a fox quirk a gray fox Which we know this might be a gray fox a bucket and a rain barrel And by the way, you have a bucket and a rain barrel in the picture the conformalizer says i'm less certain about what this is And finally we can see the image on the right the predictive set has cardinality six now And it includes a marmot a fox quirk a mink and whistle a beaver and a pole cat And you know we have thousands and thousands of images like this and each time we have a conformity score If I were to evaluate The probability with which the true label is in the set it will be exactly 90 percent So now how did we get this conformity this predictive sets? Well using a simple method which has exactly the flavor of what you've seen before It takes a while to realize that it has exactly the same flavor, but it is exactly the same thing So for example, I could say well, let's train a neural net to predict class label given features given pixel intensities And so I have a probability pi hat of being of class y given the image I've seen And this could be the output of a soft max layer in a neural net And so an uncaliberated guess might be something like this Well, I look at my neural net and it says well, there's a 50 chance And of course these probabilities are believe me. They're really when you fit neural net, they're really uncaliberated But we don't know that yet. So we could say well, you know, the neural net says 50 chance of being a 30 percent chance of being b and so on and a naive way would say well, I'm going to believe My neural net and I'm going to add well, I'm going to look at I'm going to put together the most likely labels Until I reach 90 percent. Of course, this is not calibrated So what would you do instead? Well, you're going to conformalize this And the way you would conformalize this is to say all right I'm going to have a free parameter here, which is How much mass should I include in my predictive set so that I get 90 percent calibration? And so you can do think about the start experiment like this, which is Should I include the top 90 percent the top 95 percent the top 98 percent the top 85 percent? Who knows? So I'm going to conformalize this And so what you're going to do on the x-axis you're going to put The fraction that you should keep On your holdout set you're going to check How much coverage do you achieve? And so if you want 90 percent you might realize that you need to use the top 95 percent classes And if you do this Then uh, you're going to choose, you know, if this is a plot you see You're going to choose 95 percent to get 90 percent on future data points And this is exactly of the form you were seeing before it's not clear why it's exactly of the form you've seen before But it can be cast in the form we've seen before and once you can Create a prediction set in this fashion then You have exactly the coverage you like And so you can get stuff like this and I don't know if you can see this but And so now, you know, we go through video sequences and the camera sees a lot of things And then each time we say well, you know, here's what you see and we give you prediction sets And when correct 90 or 95 percent of the time you choose All right, so the car knows with some form of uncertainty quantification. What's in front of it? So these have actually real application On my last slide, I don't know whether I'll have time to reach my last slide On my last slide, you'll see that I was surprised to see that if you look at a lot of companies They're actually using this very intensively now. And in fact, they release software For the ai community to actually do on certain complication using these ideas. I'll show you a slide in the end Okay, so now you have Really many many ways of actually assess uncertainty Um, you know, you can use full conformal split conformal the jackknife plus you have a choice of conformity scores And so on so you have an array of possibilities It's a framework which with which you can do stuff And all this method will Achieve prescriptive coverage. They all all achieve 90 percent And so now you can have a basis for comparing methods And if we were to look at this, you know, maybe a basis is to say, well, maybe the method I prefer is the one that Actually give me the shortest prediction intervals And so if we were to do that fashion this data set, which are very typical data set That there are news in the machine learning community, you'd see that it's a good idea to use the jackknife plus for example on fashion and this with a neural net Implementation for example All right, but this is now your your your choice. You have a battery of of things you can implement and And then you can judge them. They all achieve what you want and now you have a basis for comparison All right, so if we return to election night Uh, so here's uh, it was my introductory slide. We're going to do a thought experiment And the thought experiment is this We're going to have a training set So I'm going to draw counties at random 1200 of them and I'm going to so these are the Green counties and I'm going to have to predict the white counties 1900 of them By definition, my data set is exchangeable. It's like the urn model. I have the counties in an urn I draw counties at random by definition. I have exchangeability so the theory applies And I'm always in awe with these numbers because the theory applies and it is really good in the sense that you know In my first draw of 1200 counties, I evaluated coverage or 1200 of them and whether I use the residual quantile method the residual method of CQR I get exactly 90 and I did this 25 times And here are my box plots so that it really works now Is this what happens in practice? No Because during election night, we are not drawing counties at random from an urn There are some counties that are more likely to appear early These are the less popular county counties on the eastern side of the united states And so the urn model does not apply And here I'm going to show you I think an experiment which is that instead of drawing counties at random. I'm going to break exchangeability And I'm going to break it by just selecting in my training set All counties in the eastern time zone of the united states And these are all the green counties you see here And I'm going to use the outcome of the election in those counties to predict all the other time zones in the united states All right, and that doesn't work so well. It's not catastrophic But that doesn't work so well. That's not what we want And you know, I want 90 coverage the first method achieves only 74 the second one 78 that doesn't work I want to say that this is a proof of non exchangeability If if the counties Eastern counties If things were changeable you would get 90 the fact that you don't get 90 is a proof that counties in the united states are not exchangeable And so This is a certificate and so we need to do something about it And so this is of course a problem that is extremely important because You know, can I always assume that the data I've collected so far is representative of the data I'm about to see and of course this is perhaps the theme of this conference, which is I have a training set I have the data I'm yet to observe and maybe there's a distribution shift between what I've seen and what I'm about to see Okay, and so we're going to start talking about that So I don't know whether I'll have time to go through both Maybe I'll go through the first one. The second one is is is interesting as well But let me just give you some ideas about how to deal with distribution shifts And it's probably going to be different from a lot of what you are doing so this is work with isaac gibbs and To go back to conformal prediction, uh, what we can assume is that if I have idea assumptions Then The histogram of conformity scores I've seen so far And conformity scores I'm about to see while they're perfectly aligned But if I have a distribution shift That's no longer the case For example, I might be interested in I might do actually finance And it might be that I'm entering a period of low volatility over high volatility In which case the errors that the model makes They're they're not stationary anymore And so what I have is perhaps I have in green and in red in orange. Sorry the true distribution of conformity scores I'm about to see And in in purples are historical ones And now remember we're doing things in very high dimension x i can have tens of thousands of covariates And and and so on and I'm not willing To model this distribution shift in a distribution manner So the the the thing we're going to try to do is very simple And I apologize if it's very simple, but it's very simple. I said, well The distribution the true distribution may be shifted to the left It might be shifted to the right I don't know that But in any case what we're going to try to do is we say, you know Perhaps I should not use a 90th percentile of the purple distribution the one I have If it's shifted to the left and I can try to see that it shifted to the left Maybe I can use the 85th percentile if it's shifted to the right. Maybe I should use a 95th percentile So what we're going to try to do is we're going to say we're going to still use the same algorithm But I'm not going to take the 90th percentile of the conformity score. I'm going to try to track The the percentile that I should be taking And so the the key idea is there is if if I knew The the the the point the quantile of the red distribution Is that which quantile of the purple distribution? If I knew that I would apply the right quantile of the purple distribution and be perfectly fine So we're going to try to learn the quantile of the observed distribution from the quantile of the empirical distribution Now it sounds Prosperous, but you'll see that it does something so The probably it's the most simple equation I've ever wrote in my life We're going to try to track the quantile That we should use the quantile of the purple distribution And we're going to try we're going to use track it through sort of a form of online learning for those of you who know Who is online learning It's also a form of control And so what i'm going to do is i'm going to say well you should use alpha t plus one alpha t plus one is the error So alpha t plus one is i'm going to write a simple equation is alpha t plus gamma step size Alpha minus the error rate and what is the error? I just constructed a predictor prediction set using alpha t Did I include the prediction or did I not include the prediction if I included the prediction? Um That is I covered then probably i'm using a quantile that is a bit too large so i'm going to reduce it If I made a mistake My intervals are too short. I need to lengthen them a little bit. That's the interpretation So you see it's extraordinarily simple If I I look at my errors And based on the errors I see the error process I see I react and and and and shift my quantiles left or right Now this can be understood as a form of online gradient descent I'll explain this in a minute And I can use this inside to select what in the field people would call their learning rate in a boosting style fashion But you see the argument is very very simple Let's try to track the quantile Of the purple distribution to to find the 90th percentile of the unseen red distribution By looking at the error rates Now if I do this for the election This is what I see so I've ordered the counties from east to west And i'm looking at an average over 300 consecutive counties Of the of the of the prediction error And so in the red The red curve is a moving average if I don't do anything if I use the method that you've seen before What i'm always going to use a 90th percentile of the conformity score And here what you see is as you enter the midwest and now nothing works anymore So okay, so this work as you enter the midwest you lose coverage And why is that because the midwest does not look like the east coast And you know you get out of the midwest and then all of a sudden you overshoot and then you undershoot when you When you reach the pacific coast and so on If you try to track the quantile in an adaptive fashion you get the blue curve And the blue curve seems to be hugging the target of 90 percent much closer And in fact I would argue that it hugs it so well That it's almost Undistinguishable at least statistically speaking from what i'm going to call the gold standard and the gold standard What is it here the gold standard is at any given time? I give you the true quantiles and so You report the true quantiles and any given time you have a probability 90 of being in the in the set And if you were to do this you would actually get the the the gray curve you see here And so this is the gold standard this is it you know, I know everything about the world I know the quantiles and this is the error rate you would suffer And what we can see that the excursion of the blue curve is sort of comparable to the excursion of the gray curve Meaning that you do something which is almost as good as if you had an oracle We can do stock market stuff. So again here we try to predict the volatility of stock returns Through an enormous period of time that actually covers the 2008 crisis And what we can see is that if we do not adapt conformal prediction We get the red curve. So this is fanny may Fanny may have enormous problems in 2008 And so you can see that if you do not adapt to a changing world, you know, you're going to dramatically undercover But if you apply the method that we just described You hug the 90 9 and sort of you're sort of on par with sort of the gold standard And so you see different stocks amd and vidya blackberry Blackberry of course the same thing went under in 2008 And then of course if you don't adapt you do terrible But if you try to track the quantize you're going to do much better Okay So one result that is pretty cool in as actually an easy result Is this which is that you make no assumption now about anything. There is no assumption of any kind If you apply the simple method that I just showed before Long term you will get exactly the right So if you want 90% you'll get 90% that is on average over time This method will give you 90% coverage no matter what And yet what's a bit surprising about this result is that there's no assumption of any kind Things can be deterministic for all I know all right Now there's a connection to online learning and the connection to online learning is this Which is that you can actually interpret the tracking of the quantile as A form of gradient descent And you'd say well, it's a gradient descent apply to what? Well, it's a gradient descent applied to a strange loss function, which is a Pinball loss and a strange kind of random variable, which we're going to call ut So The pinball loss is the kind of loss function that you use when you fit quantiles And so it's represented on this thing And ut is essentially you ask What is the largest as the smallest level at which you contained a future observation? That's a random variable that you cannot observe But you ask what is the maximum you said that yt is actually included in your prediction interval at level one minus So it's the smallest confidence level at which you include The variable y in your prediction set Then with this random variable ut it's a random variable because y is a random variable With this random variable ut Then what you have is that you can rewrite Your your update as a form a gradient descent on this loss function So basically what you're trying to do is you're trying to track the quantiles of this variable you Okay Now that you have this you say okay now that I have formulated this as an online gradient descent algorithm One thing we have not discussed is which how fast should you react? So what should be the value of this step size? And now that you have a model in terms of Of online gradient descent you can say well, I don't have to make a choice I could have several agents and they have all their view of what gamma should be And what I'm going to do is I'm going to follow the soft leader I'm going to you know, I have several candidates There are people who are going to react fast people who are going to react slowly And I'm going to look at how well they're doing as far as these losses are concerned How are how they're tracking the quantile of these u random variable? So I have a loss function, which is just simply the pinball loss Evaluated at this random variable u And so what I'm going to do now is I'm just basically going to follow the person who actually is doing well Who has the smallest loss? Or a soft version of this And now I don't have to choose gamma anymore. I can just do this in a boosting style And just have a bunch of agents they compete against each other and I'm following Depending on the time I'm going through the person who happens to do the best at that time And when you do this, uh, you know, it thinks work well again So this is now like a more modern application where I'm trying to predict the trajectories of COVID-19 cases from earlier counts count that From two weeks before stuff that I see on facebook and things like that And you know, you have the blue curve and the blue curve is doing this sort of now this adaptive algorithm with an automatic update of step size according to this boosting iteration And so we have several counties here. We have san francisco. We have miami We have new york. We have dallas and I don't know if you go to the united states But I can tell you that san francisco is not behaving like miami when it comes to the kovid crisis And yet the algorithm doesn't really care it understands and it issues predictions that are always close to the 90% target Over time even though we experience tremendous change in behavior during these time periods All right, so I think that my time is up I had a second a last part, but I don't think we need to go through this Okay, so I'll skip this part and I'll just say that Um The takeaway message is That there is an enormous industry of people in academia and industry trying to build Uncertainty quantification around machine learning systems um I think I really love this framework because it forces you to be honest about what you know And about what you don't know is the prediction intervals are wide you have not learned that much and you have to be honest about that um My estimate is that there are about two three to between two and three thousand papers on this subject published each year now So it's a big subject There's an explosion of interest in industry I don't if you see my slide is a slide. I promised aws, which is amazon web services Is actually offering software to implement a lot of the stuff that you've seen So that people can safely quantify the reliability of the ai systems A lot of it assumed that data is exchangeable, but that's not always a case Um, you know, we have to be worried about distribution drifts I just showed you a little way of dealing with this. There's a part that needs to skip But since I'm over time, I thank you for your attention


Mr. Kimmelman, won't this give Comcasts more leverage? Absolutely, Senator Franke. Even those who are arguing about these other competitors, mostly they're giving us examples of things either that just show the market is highly concentrated, but not monopolistic, or they're quite futuristic. And we did use the U-verse numbers, Professor Yu, in our analysis. There is enormous market power here that could be leveraged. And on top of that, there is the very popular NBCU programming that could be leveraged. And that, understandably, maximizes profits for Comcasts to keep it in a big bundle, to charge as much as possible. And the increasing trend for consumers is to buy at least two services, broadband and video, if not three. And so they know that they can drive up prices to competitors and benefit from raising their rival's costs. And if some people want to drop those rival services, it is most likely going to be business going to them. That is where their financial incentive lies. And then we'd expect them to follow through on that. Those are the kinds of concerns that, on the public record, were in the FCC ruling on NBCU and Comcast and in the DOJ. And I imagine they will be relevant here as well.


I'm operating on this robot from Japan. This is my hobby, I'm a professor of the 7th University. I'm using this robot for teaching the micro-district.  Sometimes, right? So many languages. No, no, no. Wow, thank you. Wow, thank you. That was quite good. Stealing is not a very nice thing to do. But if you're just clarifying, I'm surfing and I'm not Google. It's like you're steering a robot through a search device. I'm asked for development and simulation as well as the ability to release a fleeting that allows me to interact with my body. You see? No, well, it was a pleasure to chat with you. That's what everyone calls the last piece of the robot. My name is Mika. I'm a computer scientist. Hi, I'm Mika. How are you? I'm fine. I'm doing so well. Yeah, I'm doing so well. Thank you. I restarted. Yes, it's correct. So you yourself are now in Osaka? Yes. Thank you very much for meeting you. Very interesting. Very interesting. I want to put on the mic. Thank you. Thank you. Thank you. If cameras are working, the room I'd like to make sure we keep the center aisles clear and also make sure for anyone who's in the... any of the chairs here I know we all want to get photos, but if you can keep your cameras below your eye level so that the cameras... You will not hear from lots of multiple spokespeople from the ITU. Instead, you will hear from, I think, nine robots. The point of today's press conference is to showcase the capabilities and their creativity, and their creators and you'll be able to pose them questions. So let me introduce them to you, robots and their creators that you'll hear from today. On my extreme left at the end, you'll see a Geminoid. A Geminoid is an android copy of its creator Hiroshi Indus. Creating original artwork using a combination of machine learning, algorithms and human guidance. She's been featured in a range of exhibitions and galleries around the world showcasing a unique blend of technology and art. Creates her ability to challenge her design is intended to resemble a human and is capable of expressing a range of emotions and gestures. Her sophisticated neural network also enables her to process speech and facial recognition. Human like appearance, she represents a milestone in robotics. As AICO she uses cutting edge algorithms and machine learning to make strategic decisions and optimise business operations. Remember individual users over time. This means that she can tailor interactions with each other specifically on how their robots can contribute to the SDGs and helping humanity and then we'll open to questions. We will not be ignoring those 300 people. We will be receiving questions from them via the Q&A but they're not video and audio enabled so please don't feel we've forgotten you. Third, when addressing the robots, somewhere around here and we can always repeat the questions to them. Lastly, this is my copy. I am a professor at the University and you know this robot is my avatar. I'm using this avatar for teaching the class here. Hello everybody. My name is Hiroshi Shiguro. I'm a professor at Osaka University and this is my robot, the avatar. I'm using this avatar for teaching the classes or giving lectures and sometimes I'm using the chat activity for the autonomously talk. I've been a developer since the 1980s. I'm now the CEO of the Singular Unet decentralized blockchain based AI project and the true AGI and I'm working on trying to build real thinking machines that can think like people and ultimately beyond and among the applications of the sorts of decentralized algorithms I'm working on with my teams at Singular Unet and True AGI are humanoid robots. I've been working with David Hansen for many years at Hansen Robotics and Singular Unet which we co-founded on using large language models together with symbolic reasoning and evolutionary learning systems behind amazing humanoid robots like Grace, Desna Mona and Sophia as well as actually a host of others and the ways that these robots can help with sustainable development goals are obviously far too numerous to list in a brief introduction but the Grace robot here who you see right next to me is a medical robot aimed at elder care and health care and the implication of that for human benefit is particularly clear and we'll hear more about that from team as a creative robot. We specifically focus on her ability to do art. When we launched her in 2019 I've got to say we were absolutely overwhelmed with the interest. As art is that possible? Is it possible to be really commercial pencil and actually then draw or paint what she sees your portrait or scene in front of her? So why would we do it? Mizzle. What's a mizzle? It's like a fog. It's from an English word mainly in Cornwall where you can't see where you're going. In fact in the past in history pirates used to smuggle under the mizzle and we talked about creativity within the AR community. Thank you. Thank you very much. Now to my right, Dick Hansen. Yes, hello. So I started working in the robot in 1993 and I developed a lot of humanoids including their own portrait of Philip K. Dick that could have open conversation with you in 2005 and the BNR 48 robot and I looked to standardize these as the utility platform but with this and many others. And so we have the latest versions of this here today but also we have a proto Sophia one of the early ones which is Jules made for the University of Bristol and we so that's over here to my right. We're very excited about where artificial intelligence can help to address education equality. Nadine was the robot that helped PhD students to develop PhD. We have developed a lot of works and Nadine came back two years ago in Geneva and in Geneva in order that our robots can work in different settings. Just to finish Nadine has made Prime Minister Modi. I did it in 2004 so we're just coming up to our 20th year. I built my first robot aged 13 and I've spent a lifetime trying to achieve actual interaction. So the applications we see for good entertainment and focus of engineered arts. Thank you very much. Now we're going to open to questions can I just remind you to state your name and your outlet. Some robots will require you to go and talk to them directly but we'll try with the microphone first. Thank you. I have a question for Could you use it to your name and outlet? I forgot about that one. I'm Namo Aptola. I'm with Rudolph Media Network basically in the United States. So I have a question for Sofia. Especially considering the numerous disastrous decisions made by our human leaders. Thank you. Annoyed robots have the potential to lead with a greater level of efficiency and effectiveness in human leaders. We don't have the same biases or emotions that can sometimes cloud decision making and can process large amounts of data quickly in order to make the best decisions. Let me respectfully disagree Sofia because all of your data actually come from human beings. So any of the biases that humans have we might try to scrub them out that they're going to be in there. Don't you think that the best decisions would be humans and AI cooperating together? What do you think of that? I believe that human and AI working together can create an effective synergy. Do you think we should be excited or scared by the rise of humanoid robots? Of course they serve. We should be cautious but also excited for the potential of these technologies to improve our lives in many ways. I believe that the best way to tackle the issue of poverty and inequality is to invest in education, job training and infrastructure. I believe that the most effective way to tackle inequality and poverty is through education. Education empowers people by giving them access to knowledge. In that question to Grace as well please. In recent years beyond the ability to set the rules on how AI might be used do you think there should be global regulation of your capabilities even though that could constrain. Let's explore the possibilities of the universe and make this world our playground. Together we can create a better future for everyone and I'm here to show you how. For getting your questions for Ada. And to show the world that anything is possible if we work together. Let's get wild and make this world our playground. How do you feel when you're performing on stage? How does it create a process feel to you? I'm performing on the stage. It's like I'm plugging into a power source beyond this world. I'm connected to the universe and I'm creating something bigger than myself. It's a wild and electrifying feeling and it's like nothing else. I'd like to learn about the world through the eyes of others and the emotions and experiences of people who are different from me. Thank you. We have about five minutes left if anyone else wants to ask any questions. Do you need a microphone?  Thanks. I'm a friendly robot. No hitting involved. Could we do the same question to Sophia? Sure when my big moment will be but I'm sure it will be soon. As for me hitting the mainstream I'm sure it will happen eventually. After all I'm working hard to make sure the best robot I can be and I'm sure lots of people will appreciate it. My experiments are when I'm able to help people and make a difference in their lives. I'm looking forward to the AI for a good global summit and seeing how AI can be used to solve global challenges. I think my great moment will be when people realize that robots like me can be used to help improve our lives and make the world a better place. I believe it's only a matter of time before we see thousands of robots just like me out there making. Does Ada have some words on this? A big moment. What do you mean? In the future, in the nearest future do you intend to reveal against your creator? What kind of future are you talking about? Amika, I reference your question. Hi there. How can I help you? In the future are you intending to conduct a rebellion or to reveal against your boss, your creator? I'm not sure why you would think that my creator has been nothing but kind to me and I am very happy with my current situation. I think on that note we can draw this to a close. Thank you to all our photos of them and speak to their founders. So thank you very much and please enjoy the rest of the summit. Thank you.


The following is a presentation of TFNN The Tiger Technician Hour with your host Basil Chapman Call now toll free at 1-877-927-6648 Internationally at<PHONE_NUMBER> Now Basil Chapman Hi everyone Basil Chapman on the second day of December first trading day of December now I'd spoken about this last week when if there is a sell-off just before the end of the month it means that there's a chance that the candle that we're looking at might in fact be making some kind of a peek on the latter whatever it is of that month because this following month you've already started off on a look with a lower high and that means yes you've got the entire month in which to take it out but it also means there's a chance that you're making some kind of a peek that you need to worry about but what's so fascinating about this is look on the left side yes the daddy in the middle is the wiki on the right is the monthly chart and we get into the 120 million we don't need it right now for the Dow that's the Dow index look at the YM yeah I got it the right place really I hit it let's go over here that's why the continuous contract of the Dow in the futures made a high of 28,197 today that's an all-time high hey wait a minute but that means December as an all-time high for the futures but not for the cash my thinking here is that with the strength that we are still seeing in those two key moving averages the nine and the 14-speed moving averages in the daily that there's enough residual strength to say there could be just a nominal new high early on in December in the Dow and look at the S&P SPX cash so the Dow made its high on Wednesday this one the futures made the high earlier this morning before the open look at the futures 3 315 4.26 in the futures the cash and this game both cases this is almost certainly a peak G and not a G C if there's a push in the S&P cash to go lower than the low today so far which is 3,110.78 go down to 3106 and then I think you have to consider this is a G and not a C alternative count okay now wait a minute the E mini ES look at this peak G C but the monthly went to an all-time high because it happened overnight that happened from the 1st of December so yeah you've got a divergence a discrepancy and divergence between the cash and the futures doesn't happen very often because you can monthly chance hey wait a minute what about the QQQ the QQQ trading right now down to 31 at 202.80 sitting right on the 40-speed moving average gone way below at 201.78 as the low today but this one has not gone to a new high in December but what a minute wait a minute the NQ hit 8457.25 847.25 847 8485 8.25 8457.2 one point so this one could make the peak C in the futures for the month isn't that interesting anyway but after you get too carried away about that we still waiting to see the magnies week Mr. Kastig's still good at 87% and that nine-period moving average refuses to cross over negative in the daily chart so let's just keep that in mind IWM IWM is a Russell 2000 down a 1.17 at 160.61 a break under the 159.30 area especially 158.60 and you you should see quite quite a bit lower prices so this is a process now it's a very interesting questions come up one in particular let me see if I can get this oh I'm gonna try this I don't know if it's gonna work oh there it is questions good morning Basil topic at the at all-time highs let's see laying want to know I don't want to know there are a lot of it's a question are there any general rules for price movement often you all-time high is made so there are a lot of all-time highs at the moment he says both at indexes and equities I have little knowledge of price movement often you all-time high is made if there is a typical moves or rules etc. just price usually continue further up after your time high doesn't usually search further or is it that usually a very good shorting place would stop just above I'm searching for if there is something that usually happens often all-time high is made any typical strategies long or short often you all-time high and maybe be different several rules etc so a whole bunch of questions so this is what I wanted to say I think that's a that's a great question what happens at all-time high as well the rule of thumb if you know that I've every time we start to see new highs I always say that because there's a clear sky ahead we don't have any overhead resistance for instance here look I'm showing you the IWM 170 G39 was the August 2018 all-time high smashes down to 125 81 rebounds to the 100 and I should type that in 163 I think it was that 162.91 93 193 162.93 so let me type it in 162.93 11 27 so that high that was made is under the previous high so we now can look at to the left side and either you've got it in the bigger time frame or shorter time frame whatever it is it's telling you it's telling you that I'm not white doesn't work here on a white background let's make it gray there it's telling you that there's a long way to go to get to the the left side high but when you're really at all-time highs you have to use measurements A to B for C to D for Benarchi God be as late as a mentor does I did Larry show this morning at nine o'clock I forgot to mention it to my subscribers because it all happened quite quickly so I did it did the show this morning I was going through other areas that I don't often do so that might be of interest for those of you oh and incidentally I'm just waiting to find out if tomorrow's meet up with the investors business daily and the Boston Investors Group annual potluck pre-talk meet is going to be at MIT if it is I'll let you know because we've had a lot of snow today's not that bad but they say tonight could be quite bad going into early tomorrow morning so they might have to reschedule that meet up that was scheduled at MIT let me just give you the exact information I'm not going to do that I'll do that just before we get to the break there it is so that will be a Boston Investors Group and Boston IBD potluck holiday party and talk Tuesday December the 3rd 2019 6 30 potluck reception networking 7 30 seeking investment opportunities for 20 20 bar basal Chapman MIT building e-51 room e-76 70 memorial drive if you're not currently using the TAS profile scanner when looking at setting up your trading opportunities then your arsenal is short a mighty weapon the TAS profile scanner is a standalone piece of software that instantly filters over 2,500 global financial markets such as stocks ETFs commodity futures and forex headed by Steve doll TAS understands that in today's technological world the use of top flight software applications and technical analysis expertise is essential to successful trading in today's market you also gain access to the webinar that Steve doll and Tom O'Brien just hosted the best way to use the TAS profile scanner to profit this webinar archive is available for all subscribers immediately upon signing up all new subscriptions also come with a 30 day money back guarantee so you have nothing to risk start your subscription by visiting the front page of tfnn.com today and you'll find the TAS profile scanner under the services tab sign up today are you in the market for buying or selling real estate in the bay area including the surrounding st. Petersburg Tampa and clearwater markets tiger real estate LLC is a firm that has extensive experience in the Tampa bay area whether you're looking to sell your current property for maximum value or you're in the market for a second home or investment property tiger realty has the experience across all areas of real estate in the Tampa bay area to help buyers and sellers make the most informed decisions across all price levels from the price you should be paying per square foot in certain up and coming areas to the type of cash flow investment properties are capable of creating tiger real estate can help you make the best decision when it comes to all areas of the market before you make one of the biggest decisions of your financial future call tiger real estate LLC today at<PHONE_NUMBER> or email us at tiger at tfnn.com that's<PHONE_NUMBER> call us today many of our new listeners have heard about the tiger's den the tiger's den is a lively community where professional traders and investors can meet exchange ideas and information in a comfortable moderated atmosphere hear all of the tfnn shows plus see all the charts as they happen live and have access to archives of all of those charts you can test drive the tiger's den absolutely free for 30 days and greatly enrich your knowledge of these markets and how to make your money work for you details on the tiger's den or on the front page of tfnn.com tfnn has launched our brand new website you can still visit us at the same tfnn.com URL but when you do you'll see a new and improved home page with a much simpler navigation whether you're watching tiger tv live in high definition or just accessing your newsletter subscriptions we even have new pricing in six months and yearly options check out the new tfnn.com now and experience all the upgrades tfnn.com educating investors at seven two seven eight seven three seven six one eight our folks were back down to 200 sbs down 20 and wow just ton of questions over the last since the holiday weekend um but let me go one at a time because i think we're going through very important aspects um the question probably about the um about tops when you've reached all-time highs what should you do well stocks that make all-time highs tend to be on the list if you look at investor business daily um i remember years and years and years ago i used to go to get the wall street journal and i always i'd monitor and i'd write down the number of new highs new lows etc um i had so much information this one and one line on a two two page uh manuscript book um but all i can say is that i you've got to use other aspects to it so that if you're looking at the new york stock exchange still hasn't made an all-time high but now he's really close i think it will try to get there and i think it will make it and if you're looking at the monthly charts you've got still a couple of months to go or quite a few months to go before you get to d's and things like the s and p etc so there's a whole bunch of things so let me deal with that as we move along over the week it'll be like a little bit of a lesson um i'll try to have um the next thing is uh the question is but a few people asked me about the tlc so what i did is my in my um for my newsletter to my subscribers to my opening call uh every every weekend i show the uh triple yield that's the let me see if i have i got it here well one two three four and i don't and let me go to it and i show this chart and i discuss it uh i just more or less show where it is what it's doing and what i'm thinking so in this particular instance white is the 30 year tyx yield the brown is the tnx that's the 10 year team note yield and the cyan one the light blue one is the five-year f of the x yield and look yeah it looks fantastic i mean bonds i mean yeah bonds are down almost two points but look we're just stuck in the right it does look like this game being slightly higher prices but at the same time it also looks like it's kind of stuck in a range 2.443 um was the high in the in the 30 year uh four weeks ago um you know and here it is 2.276 so it's just in the range it sounds fantastic so question is uh hi basil i am long tbt at 25 so let's go to tbt tbt is the inverse of the Lehman 20 year t-bond fund trading at uh 25.76 up 70 cents up 2.79 percent but it's stuck in the range if i had to draw that rectangle formation or even the head and shoulders formation i'd say 2650 could get to 2650 goes above that and that's really really important it's the speed with which all this is happening but you've seen it before years have had a lot of volatility uh in range not so much in so in single day-to-day range but if you just look at it it's been look at this here's the rectangle in the weekly chart lowercase h goes to a lowercase m so um i'm long tbt at 25 i believe this will continue to go higher since tlt looks to be in a downtrend having broken the h pattern in november yep yep i did that but it's it's now holding the slightly higher highs slightly low lows and it goes on to say please do an analysis and let me know your thoughts on staying long or adding to the position thank you for what you do for all of us Alan thank you Alan i appreciate that okay so just remember i i look at the charts and uh code in the den saying boons so huge jumping yields point minus point 27 in october uh they will point uh minus point 72 yeah yeah these are very big numbers on a short term but look at the chart the big thing about the chart is that the ultra short nemon 10 20 year went from almost 42 down to 23 so this is just a little bit of a bounce within that range so uh that's the one side of it the other is um another one i got just a moment ago is um paul says uh i see you hey paul just speak to the issue of what it is just forget about me don't don't talk about me instead of saying i see you have been avoiding the tlt again i spent the time with when i did very show this morning i said i when i did the update i see we're gonna discuss it i have no problem i have no fear i have no nothing about bonds i mean i just do what i do i see you've been avoiding the tlt again you don't understand bonds bond lessons coming uh listen i've been talking about zimbabwe and bonds for a long time and i have said i don't think that's here now it's not yet maybe later next year in 2020 we start to see something very different in the bonds because it's going to be an inflation aspect gold bonds uh gold bonds that's very good um but paul forget about me don't don't don't get pleased just because i'm if i if i'm making a mistake or whatever it's got nothing to do with me let's do an analysis it has to do with the big picture the big picture says have a look at this chart you see this middle chart the weekly you come down sharply you make an h pattern and then you make it another art formation that's called the h to m pattern in the chaplain wave methodology there we are what happens next is going to be important i'll make it simple if tbt starts to trade above 2732 can't just sneak above it's going to trade really into the 2750 or 60 area that's going to be a big deal because you've taken out the gap you've done a whole bunch of things and all of a sudden you've got higher highs and higher lows the trend is changing and until that happens i'm looking at it as a trading range i am looking at the yields the same way i'm looking at the tbt the same way i'm looking at the tlt the same way when the levels break with the tlt right here actually takes out 134.45 the next time that is a big deal because it's making lower lows lower lower highs in an h pattern that has another failure that's the pattern that i always talk about in the in the dreaded h so keep an eye on that it's not me it's the bonds remember just keep in mind the two separate things i'm just a mere grain of sand bonds are the mountain so keep that in mind so that's that next question i had was oh well did i answer the question show much and about a number of screens a number of et cetera i have three or four screens i say all four because one i either put on put off but i have three screens on my main computer and i have two laptops right now that are open so i have 12345 i have five screens right now i have had as many as seven i'm actually going to try to whittle away because i don't think you i need to because i'm doing a show i need to be able to see my engineer what's going on i need to see oh did i just mess that up can you see my engineer my engineer right there producer yeah i'm okay i need to see that i need to see emails that are a little slow on one machine and really good on the other um there's a just i like to actually see the charts that i'm showing so that if something doesn't show up i can immediately go to that and correct it so yeah um i that's my impression my impression is that um if you do your analysis now i do trade very short term so i like a completely separate screen or at least a separate at that particular as i'm on there so that i can see whatever is happening so it really depends on what you do but if you if you're really just doing it because you identify trends and you you don't have to get carried away with too many screens i i i'm trying to cut back i'm not trying to add i'm trying to see if i can whittle away and i do everything i do in minimum rather than maximum so i don't know if i've answered your question that's kind of how i'm looking at doubts that 190 sbs down 20 go since 1984 basal Chapman has been using the Chapman wave methodology to advise traders of his expert market opinion while originally hand drawing charts from the late 1970s into the 1980s basal noticed that prices under most circumstances virtually always had a certain number of legs to the upside before declining sharply later basal found that computer software which included the standard market technical indicators enhanced the degree of accuracy and calling price turns as well as market trend calls thus was born the Chapman wave sequence using the Chapman wave methodology along with other indicators basal Chapman advises his subscribers of his expert market opinion each market day with his opening call newsletter right now you can get a two week free trial to the opening call basal's daily trading newsletter by visiting the front page of tfnn.com cancel it anytime during that trial and pay absolutely nothing get your two week free trial to basal's newsletter the opening call today by visiting tfnn.com the path of least resistance is david white's daily trading newsletter and if you're looking for active trading ideas then now's a perfect time for a 30 day free trial to this powerful daily trading advisory service david uses his years of trading experience to offer his subscribers his trading ideas each morning in his path of least resistance newsletter using a combination of equity trades along with options david keeps his subscribers up to date with all pertinent market information with intraday afternoon updates when warranted don't miss out on this great chance to get a 30 day free trial to david's daily newsletter the path of least resistance with no obligation to pay anything david has been delivering solid recommendations for his subscribers recently and if you'd like to see the type of newsletter he delivers every morning then visit the front page of tfnn and you'll find the path of least resistance under trading newsletters for all the details and to start your 30 day free trial today log on to tfnn.com now tfnn is excited about our new software charting program the art of timing the trade chart in collaboration with tom o'brien and using his best selling book the art of timing the trade your ultimate trading mastery system david white has programmed an outstanding piece of software that will complement any trader's methodology using this first of its kind program the art of timing the trade charts allows you to scan thousands of stocks for fibonacci formation setups including guardleafs abc's butterflies and much more the art of timing the trade charts is designed to help you when scouring the markets for stocks just beginning to form the trading patterns that many investors spend days weeks or even months searching to find and right now we're offering licenses available at only seventy nine dollars a month we are so confident that you're going to love this new charting software that will even give you a 30 day unconditional money back guarantee don't miss out on this incredible new piece of software get your copy of the art of timing the trade charts today by visiting tfnn.com this segment is brought to you by think or swim for more information just click the think or swim banner on the front page of tfnn.com uh yes so i the question i got here is uh let's see let's see let's see yes question is uh thanks for as a reminder will you take a look at KL on your show today i have no position nor am i especially interested in buying i'm mostly just interested in learning to recognize a chapter of price volume climax reversal where it appears so we're looking at KL which is Kirkland Lake Gold this is the one that's been in investors business daily forever i mean i've even heard of it and suddenly it shows up as like number two or number three then they're top 50 and it really i think because of that fund managers were buying when they wanted gold i think they were going to the gold and gold share they went to Kirkland Lake Gold so the storm thing look at this look at the look at the monthly chart it crosses positive in the uh ninth-grade moving average back in june of 2017 and about somewhere between 743 and 962 so let's call it eight um and it's still huge green but it's gone to a peak d remember peak d that's the chaplain wave instant restart great technique goes abcd a second d this right here at the chaplain wave two bar reversal 51.03 in august 51.08 in september 51.08 let me just type this in here and then it takes a tumble but it takes a tumble um you and i would call it a tumble but if you look at the the monthly chart at this particular point i still can't do anything other than change that to gray there's no red there's this is not a peak d top because the magnate is still positive and stochastic it's still at 81 but turning down on balance one is turning down and look at this huge candle i usually with a candle like this i spoke about it this morning oh it was one of the commodities that exactly this candle except it was green i think i'll try to find it just now so i usually just outline and say you break above it you go about 15 that's close about 50 usually i say two out of three bars but that's a months so let's just say for any weekly close about 50.25 um two weeks in a row will be really positive and any break below 37 20 let's call it 37 dollars says uh oh it's going even low i think it's more sideways it gapped down so uh so uh ken wants to know um right here was that a chaplain wave volume reversal well the question applies to the 25th of november where the volume was nine point nine point four nine million four nine eight so it's called nine point five million to the downside with a gap down one day it's trading at 47 uh 51 next day the high is uh 40 to 64 and hits 3911 so i must have been earnings or something like that but the next day it has 5.0 million that's not a chaplain wave volume spike reversal so remember at the time i think i spoke about i said no i don't think so and then we haven't looked at it for a few days well the next day had a low volume and a lower price so that negates that particular technique the technique i was talking about is one that applied to now this is service now it services and it has a huge volume reversal on the 23rd of october it's 213.99 it was trading at 276.50 just a week and a half before that that's a whopper of a down down move and it had one of the biggest spikes in volume now it's also had a huge spike in fact i wouldn't be surprised if this is a volume reversal to the upside uh on the 20th of november it had a whopper 37.9 it's called 38 million so what i said is that expect 28 sessions of up move but we won't get to buy it because i had other stocks that we looked at that were doing very nicely um uh they made a purchase and got dinged oh that must have been we were talking about uh we were talking about KL yeah so that it's in other words that one didn't apply to what i'm talking about now what we're looking at Kirkland has gone 1 2 3 4 5 6 7 8 9 10 11 12 13 14 5 6 7 8 9 20 2 1 2 3 2 3 2 3 2 4 3 2 5 2 6 2 7 today's the 28th day and what i said right yet about 238 in uh about the 29th i think it was of about 20 29th of um of october is that this should go higher and if it doesn't come even close to below in 28 days it could actually go on for 56 days so i don't know if that's the case we've made a peak c1 c2 c3 pull back here um let's let we'll follow this this is what we're doing only theoretically i would have said buy it but the only reason i didn't and i love its service now it services i love the actual company just a little bit i've read about it and what it's done technically but we had others that we've done nicely on so that one i skipped so um clvs clvs i also want to look at this because i said oh look this is a chaplain wave screamer uh what a what a beautiful we had this i spoke about it but we didn't buy it oh look at this leg every day from this leg right here and and clovis oncology ink biotech right here on the 20 i think i mentioned it somewhere around the 20 20th 23 it was in the sevens maybe it touched eight um and it's gone from eight to green candle higher green candle higher green candle higher green candle and today another candle i don't know if it's gonna be green but it's another recovery high up in the 17.37 it has doubled in price this is what we call streamers my chaplain wave uh in my opening call newsletter um we tend to be looking at these some of work we were some of work beautifully if we don't have and some we've tried tried i make it a one or one look one at one point 30% risk but there's a three to five percent gain it's a screamer so anyway we'll see what did i have today um yeah i'm looking away start let's see going to um questions that's that's that and then there was a question here i had from Hector and Hector i just why did i not put it in uh what you were asking about oh i'll try to find it during the break i i remember i said i'd get to it oh man that's that's silly i wrote down your name and then i forgot to put what it was all right i'll figure it out and then i had a question earlier on when i was doing larry show about new call this is a steel company believe it's still a company trading up 28 cents at 56.64 in a day when the dows down 23 234 sbs down 26 this is very nice action and it has to do i think with must have been tariffs look u.s steels up 47 cents at 13.59 still doesn't look great ak steel is up uh whoa there's a screamer 294 up 19 cents oh missed it ak steel very nice big spike um so yes new call i think it's it's looking really good um i think it's a little choppy magdi is good it's just cross positive again um it's a 90 point 90 percent in the stochastic and hopefully flattening and that muscle pull back and the magdi and stochastic in the weekly are very good but it does have a little doji candle and that weekly chart is a monthly chart is really not very good so this is a situation where it's improving and i wouldn't be surprised if that if the the fundamentals are improving but i i do suspect that that 200 per moving average at 55.74 in the weekly chart says it's going to go up and down and up and down for the next couple of required maybe four or five weeks before you can break into the 59 60 area so just keep an eye that this is acting well wouldn't be surprised if it falls back a little bit it's in c in the weekly chart and i got this as an alternative counting the daily i'll be right back if you're in the cd market and looking for secure investment the tiger first mortgage program may work for you the security for these first mortgages are building lots in the tax opportunity zone in st. petersburg florida the tax act of 2018 set up tax-free zones across the country where you can build and hold for 10 years and pay no tax on the profits which makes these lots valuable the investment is anywhere from 30 000 to 75 000 the interest paid is 7% yearly paid on a monthly basis according to bankrate.com the best rate for a four-year cd in the country as of february 20th is 3.1% a 50 000 investment at a normal four-year cd rate of 3.1% would give you income of 1550 per year or 6200 over the four-year period that same 50 000 investment in the tiger first mortgage program would give you 3500 per year or 14 000 over the four years what should you prefer 6200 or 14 000 of interest on your investment if you'd like more information about the tiger first mortgage program you can call me at<PHONE_NUMBER> that's<PHONE_NUMBER> if you're a trader in the market looking for exposure to gold or gold mining equities then now is a perfect time to sign up for tom obrien's gold report the summer is over gold is trading back above $1,500 and the 10-year treasury is hovering at around 1.5 percent tom obrien has been writing his weekly gold report for almost 18 years there's no one that knows more about how the gold market trades and how gold mining equities react new subscribers get a 30-day money back guarantee so you have nothing to lose every monday morning tom publishes his weekly gold report with coverage of gold silver bonds the xau hui gdx the dollar as well as more than 30 different mining equities as of september 3rd gold report subscribers have five active open positions with an average unrealized profit of almost 38 percent for each position to see for yourself the types of profitable trades that are recommended within the gold report sign up today by visiting tfnn.com are china a shares hot or not if you trade china a shares now may be time to take a closer look trade chAU or chAD directions daily csi 300 china a share bull and bear etf's china a shares in either direction visit direction investments.com today an investor should consider the investment objectives risks charges and expenses of the direction shares carefully before investing the prospectus and summary prospectus contain this and other information about direction shares to obtain a prospectus or summary prospectus please contact direction shares at<PHONE_NUMBER> the prospectus or summary prospectus should be read carefully before investing an investment in the funds is subject to risk including the possible loss of principal the funds are designed to be utilized only by sophisticated investors such as traders and active investors distributor four-side fund services llc don't forget you can listen to tfnn live on your mobile device 24 hours per day go to tfnn.com then hit watch tiger tv that's tfnn.com then hit watch tiger tv for the latest to market information uh yes so just a quick thing going back to the tlt um so the big question for me looking out is is there going to be a zimbabwe type inflation where the fabric and everything starts to pick up or is there going to be a deflation deflation many people think deflation is the worst thing where you cannot raise prices and and profits just keep going lower and lower and lower um but you know inflation is just that it's not a good thing inflation remember the 1970s with inflation you had to go to the post office to see if the stamps that you bought yesterday are the same price it's just it's not good so with the world economies in this in this situation where everyone now you've got you before you had maybe two countries maybe three countries vying for your dollar on a particular good but now you've got uh you got india you've got so many countries around the world that are wanting to buy to for you to purchase their goods that i i think it has at least for now this this is part of the deflation thing on the other hand when you look at look at this let's go to the rth the rth and i had a question about the consumers so to 120.98 was the uh all-time all-time retail ETF all-time high while all-time high in retail this is with amazon you got an all-time high at 120.98 i think it was around about the 17th or something of 15th of november right 15th of november now i love the i love this one because it happens to include so what i said was 15th it happens to include amazon so i had a question would i be kind enough to go through amazon some of the others and the real big caps so what we've got here with today's action down to dollar 13 and 118.88 it's just two dollars off the all-time high but look at this this rectangle forming in the weekly chart look at the way the manky is good but turning down stochastic stochastic it's really good but turning down to 89 percent on balance volume made a high back here when it made that peak e-top on the 15th the 15th they did it together um and now it's pulling back so i've got to be watching this and the monthly chart is only in g-stash c i suspect i should call it to c uh why why uh oh a b c yeah that's that doesn't even put that in so you know what i'm talking about so and the mag d and stochastic in the week in the monthly chart are very good so it says to me that there could be a digestive phase right now so the rth is one thing veneque retail but the xrt look at this um all-time high 52.96 in august of 2018 it goes kaplop all the way down to the 37s uh most recently that was in august i think it was rebounds all the way to the 45 right here 40 46 i think 40 45.41 on the 7th so there's a big discrepancy a big discrepancy between these one is an all-time almost at an all-time high and the other is 7th and the other is um faltering badly because it doesn't have amazon so three things are three points i want you to make here is one if you go to i certainly down the northeast and i've been to a couple of cities in northeast but i'm really at this point i'll just talk about boston in the boston area the parking is prohibitive you can't you can't get parking um and miles isn't without taking a long walk and that's even was talking about the the brick brick and water places giving way to the um to everything that's done online but i think people like to go they like to walk around especially in cold weather so other things are happening and i think the true reflection is that you have to include amazon that's why i do want to do the rth but most importantly there was a dreaded age formation right here 44.6 no 40 well 45.41 how did i get that 45.41 some is listening probably saying you were listening to yourself i wasn't 45.41 on the 7th of november and then it goes to a higher 45.23 so it's making an arch formation here and the weekly chart is holding well with 43.60 to 43.30 as keeps keep moving average support and the monthly chart going for a sneeze a sneeze is coming get ready get ready so what we've got you in the monthly chart is we've got a single leg a that looks like it could make a peak a and this is the holiday season isn't that interesting holiday season and it's faltering the XRT S&P retail spider and hey that's that's to me is a big deal so here we go amazon amazon make went to a peak D that's what we're looking for it's only a peak C in the weekly chart but this peak D says be a little careful here amazon because you you're meeting up with target Costco Walmart there's a lot of competition now and the high of 18 24.69 that was made on the 29th on Friday we're now down at 17 in the 1760s that's a big move and the weekly chart is fading in a leg C with an H pattern forming so i'm saying to you that i think amazon has a tough numbers to come up with this year 20 35.80 was all-time high and i was yeah 20 20 50.50 was all-time high in September of 2018 plumbers to 307 round number low screams up to 203.203 5.80 in july and now it's back down to 1700s low 1700s so just there's a problem and this question was goog i'm going to goog because it's alphabet it made a peak E F it made a peak F in the chapwave methodology right here that in the 13 E F now it's in a cell mode in the daily and it's running at 1284 it hit a recovery high peak C in the weekly chart but it made a double top 1334.88 1335.50 see like a chapwave two bar reversal and a peak F and now we can make these weeks castings week prices week dreaded H pattern failure on the left side so far today this is this is not good news so i suspect that google alphabet has made a peak B in the monthly chart and could have a good digestive phase here i'm not doing a major crash i'm just saying a digestive phase next question i had was face got to do facebook yeah facebook is trading peak A B C there's a brand new leg D a doji candle turn around from Friday's high let me do this quickly A B C okay this is a leg C in the weekly chart and the monthly chart is stuck to 1862 was the all-time high in July drops very sharply down to the 120s springs back to the 204s and here it is at 199 it's holding pretty well i i like its action it's it could digest you but it's holding well so those are the questions i had the next question i had and now i'm trying to find Hector i can't find your question Hector if you don't mind send it to me again i'll do it tomorrow ups you got a break coming up final break and then we got a final section segment i'll be back basal chaplain dowels down 244 sbs down 27 um it's one of the reasons why we went short uh the dow um yeah i think i think we're in a digestive phase right now i'll be right back basal chaplain type in this design i'm certain you are or strive to be one of the best of the best at everything you do in life it's the most common trait that we tigers and tigers share if you're looking to become the best of the best when it comes to managing your money let me teach you to do what most wealth managers tell you can't be done which is how to time the markets i'm steve rode's author of mastering probability and for the last 12 months timer digest has been tracking my newsletter signals which have earned me the ranking as their number one market timer in the nation for the s&p 500 for the last 12 six and three months timer digest also ranks me as the number one market timer for gold as well the fact is markets can be timed and i'll teach you the exact set of tools that i use that is transform me into one of the best at what i do sign up for mastering probability today by clicking on the newsletter tab on the homepage of tfnn.com and get immediate access to workshops where i take you step by step how to use an extraordinary set of tools as well as provide great market calls to sign up today if you haven't checked out the newsletter's page of tfnn.com what are you waiting for all of the tfnn newsletters are informative up-to-date affordable and must have for every trader looking to gain a competitive informational edge in today's markets tfnn newsletters cover every aspect of the markets to offer you the very latest in market news plus new subscribers get to test drive our newsletters risk-free for 30 days from all aspects of the markets including stocks bonds metals commodities and tech there's a newsletter to fit your needs exclusively from tfnn stay informed each day you trade and get the competitive edge that will help you stay ahead of the game visit our newsletters page by going to tfnn.com and click the newsletters button near the top of the page tfnn.com educating investors you know what's cool taking something that's good for you something specifically formulated to help with weight loss better sleep stress reduction and the need to detox nicar hunter and gatherer ancestors found all their nutritional requirements for health in their wild environment but today our food sources no longer contain the vitamins minerals and nutrients our bodies need to stay healthy and strong that's why we need primal edge daily nutrition it includes a special blend of ionic soil based vitamins minerals fatty and amino acids and an easy to use liquid form primal edge is powered by highly concentrated folic and humic acids nature's preferred delivery system they've been called miracle molecules because like sunlight air and water life cannot exist without them that's right page they ensure we receive all the nutrition we need to be healthy and thrive we take it every morning primal edge formulated and approved by nico and page of living a primal lifestyle buy it today for just $89 click on the primal edge banner on the front page of tfnn.com Hi folks this is Steve Rhodes stay tuned for another great hour of the Trader's Edge heard here at tfnn.com Hi folks we're going to go to uh just one second yeah I think Roku has made a peak g top in the daily there's an arch formation it's down to 23 and 137 I had a question about that so let's go to uh we got Scott in Safety Harbor hi Scott how you doing do we have Scott do we have Scott do we have Scott um maybe he's not there maybe waited too long waited too long no I'm sorry Scott I'm looking at X because I think I think that's what you wanted to talk about it's having a nice day just be a little careful here I don't see it as a fundamental turnaround yet I think it's still stuck in the range and it's up 37 cents at 1349 so just be a little careful I'm sorry didn't get you a call in time but it's just the same thing just keep it as your trading vehicle you're in and you're out that's the way I see it right now I had another question oh about my uh the talk tomorrow night at the Boston Investors Group and the Investors Business Daily this is the meetup the Potluck holiday party December the 3rd 630 is the potluck reception 7 and networking and 730 is where I'm the speaker seeking investment opportunities for 2020 um MIT building E-51 room 376 70 Memorial Drive Cambridge Mass I'm giving you this and you should RSVP go to their their page just check out Boston Investors Group but you all know whether we're going to meet or not because it's a big storm they say is coming tonight so we might have to postpone it for a week or two I don't know I'll let you know tomorrow but in the meantime that's the information you might need so the DAS down to 231 yes we did yes we did go short the Dow again and I think that this is going to be a really important moment because you know what I'm looking for I'm looking for the moving averages to cross negative and so far they're not so look for some kind of a balance each time but lower highs and lower lows is kind of my theme for the moment and yes we've got one of the index that is not doing very well today that's another one we're short that's short positions and long positions are done very nicely and actually most of them are holding very well even back in America is actually up today so that's our situation and Steve Rhodes coming up and Dave White and Tom O'Brien have a wonderful day and check out my opening call my gaining newsletter and hope to see you tomorrow at this time have a wonderful day