Awesome. All right, welcome. It's the 7th of June. This is documentation office hours. Remember, we abide by the Jenkins code of conduct. So, dear Raj, one of the things that I was hoping for was that you'd be willing to try another time to create the weekly changelog. Oh, I'm going to turn off my camera. We don't need to watch me. Are you interested in doing that again? Or is there something else that was more important to you for us to do tonight? From my side, I'm comfortable with anything that you think should be of more priority. Well, so I would love to do one more time to run the weekly changelog and have you be the person who submits it. Because that way, if I'm not available next week, or the weeks after that, we could still get a weekly changelog. I, when I brought this to, when I brought this to the Jenkins governing board, there was some skepticism while we're not sure that a, someone who's not an experienced Jenkins core developer can actually maintain the create the weekly changelog. And you were great proof last week that yes, somebody can. Yes, definitely. Very good. I'll drive a risk. Definitely. Okay. So if anybody could do it, somebody with some talent. Well, thanks. So should I share my screen? Yes, please. If you could. And is there any background noise? Even if there is, it's worth it. Let's just go ahead. That's great. You just leave your, your, your microphone on and we'll, we'll talk to you and ask you questions and let you know while you're sharing your screen. Awesome. So I think you can see my screen. I'll just find this right now. So we are at Jenkins core directory and we want to run the Docker command but before that last time we were. So are you, are you up to date with the current version that's remote on that's on the, on Jenkins? Exactly. So last time we did that, we ran into some problems. So you said that there's a file in which you need to change the version, the latest version. So from that, it will be fetching all the PRs. Right. Well, actually, even before that your Jenkins core, the, the, I think the directory in right now is the Jenkins directory. Right. So here, if you could do a Git log. Let's see what, where, where it's at and let me double check to compare that it's at the correct location. So when I look at mine, Jenkins core is. Okay, so you are. So you, it looks like your copy, your local copy of Jenkins core needs to be updated. And so what you're going to need to do is quit out of the, quit log and do a Git pull minus, minus all. Okay. Now you need to do a Git merge upstream slash master. And there's actually another command you could use instead, but this one will now bring your local copy current with what's on upstream. And now if you do the Git log, you can do something about SSHD core. Yes, it does. Good. Okay. So your, so Jenkins, Jenkins core is up to date. Now, if you'll, if you'll do it one more step, do a Git space push. This will push your merged master branch to your fork. You don't strictly have to do it, but it's, I like to do it just to make sure that you do it. Now you have to remember you're using the password. Super. Okay. So, so now your local copy of Jenkins core shows all the latest changes. So now you should be ready to run that Docker command. Let's see. This one. So this will take up all the PRs that has been. Right. It should. Oh, no. What's it saying? Okay. No. Interesting. Okay. That seems like the correct, that is the correct version. So why is it saying no URL specified. Docker run. Curl no URL specified. Okay, we may have to have you stop sharing. Let's have you stop sharing your screen so that you don't show us something we shouldn't see. And then show. Now on your local command line, instead of just the Docker run, put the word echo in front of it. And see what it's outputting as your GitHub underscore off. It should be a username. I think it usually a username, a colon and a passphrase or a token. You know, so it should look like it looked like a username password. Does it. Dash V and then a location of the workspace. Dash dash. And then location. Okay, so let me. You may not have a value assigned yet for GitHub underscore off. But we ran this last week. So I mean, it worked last week. Hmm. So you may need to define the GitHub underscore off environment variable. And let me double check while you're doing that. I'm going to double check just to be sure that GitHub isn't down.  So I think it may just be that you don't have a value for GitHub off. Okay, great. So now you can start sharing your screen again. I clear your screen so that we don't see your GitHub credentials. We don't want to record those. That's none of our business. So I cannot even go up now. Oh, okay. So, well, so the, if you look in the, if you look in the read me file of the, of the, the generator, it's got the exact Docker command you need. Yeah. In core changelog generator in the read me.md. Just exactly that one. Well, or, or the one above it, you need the one. That one is fine. You just have to remove the number off the end. The one that your icon is over right now is the one. Yeah. Yes, that's what we're looking for. Good. Okay. Okay, now that's a fun one. A parse error. Hey, could you do an echo? Dollar sign Lang. Space dollar sign LC underscore all it's capital. It's all uppercase. Dollar sign LC LC underscore all. And I think that's it. Yeah. Those two enter. Ah, yes. Okay. So, and they might be okay. But could you instead do. Export space Lang equals. En underscore. Capital I am dot. UTF dash eight, or the UTF dot eight days. That's eight is all caps. Yeah. And let's see if that helps. English has spoken in India using UTF eight characters. Let's try it. So I'll run the command once again. Yeah. And now your arrow key should work. Okay, that's better. So one of the problems I had found in the past. Was that that Docker image for some reason. Wanted did not have UTF eight as its character set. And so I think that's what you just did is you've now used the correct character character set. Good. Okay. So we have the template. Yes. So now we go to the site. Mm hmm. And open up there. Right. Let's edit here. What do you think using nano? We had some problems last week. Yeah, VS code really surprised us. So you're welcome to experiment with VS code, but I think, I think it's more aggressive with its formatting of YAML than we're ready for. So what do you suggest? Where should we get the files? Yeah. So, so nano is great if you're comfortable with nano, whatever editor you're comfortable with. This is change log file that we just generated. Right. We move forward. Yes. And this is the retail file that we want to generate. So, so, yes, please. Now, I am not, I am not a nano power user. So this will be a very interesting experience.  And then you go paste it into the other file and then we'll edit it in the other file.  Oh, no meta a, there it is. Yeah, look meta a mark text. Oh, this one or meta six. Yeah. So meta six is copy text. Right. So meta a to market and then meta six to copy it. What is this meta? Oh, meta means the alt key. Sorry. Yeah, that's a, that's an Emax damage. Yeah, but, but your technique that you're using now should be great. Now, if you can switch to the other, the other buffer and copy that with your editor. We're going to paste it. Yeah, I think so. Now, where are we in this editor? This, this is okay. So there should be a very long file. There should and there isn't. Okay. So maybe it's time to close this. Let's go back to VS code. You're comfortable with that. Let's use VS code and we'll figure it out. I mean, let's not put you into some strange and bizarre editor. Let's get you into, into something that's comfortable. We are not able to catch the difference. Yeah. Yes, good. Well, but, but I think let's just see if we can figure it out because ultimately you're, you're comfortable using VS code, right? And lots of lots, many, many users edit with VS code. So if VS code is going to do damage, we want to know how, what we need to do to make it work. This is the. Yeah, this looks great. Now we'll probably even after you get the file in the editor, we'll probably want to go check your Jenkins.io that it is current. And yeah, you can, yeah, you don't need to save that modified buffer. So here from your shell, we want to go to the Jenkins.io directory. I'm sorry. Yeah, so do a get pull minus minus all. And get space status, we'll probably want to just throw away local changes for the moment. Yeah. So I think what you want to do is say get space merge minus minus abort abort a B or T. All right. So now get space reset space upstream slash master. Oh, now that's interesting. Why would it have a stage change? Okay. Get get status. This is interesting. Okay. So get check out dash dash dot space dot. Okay. So I'm sorry. You need a space after the dash. Yeah. Enter. Now do the get status. So what we're trying to do is get this to, to match with upstream. Now we want to check out a different branch. Let's check out master. Get space check out space master. Okay. So get reset. Minus minus hard space upstream slash master. So what we're going to do here is make your local copy of the master branch exactly match upstream. And now do a get log. Okay. And yes, okay, that's good. So now. And now if you just say. What this is going to do is make your, your private, your, your personal copy, your fork match with upstream. Good. Okay. All right. So now we need a new working branch. So get check out minus B. And change log dash two dot 297. All right. Now we're ready to go back to VS code. Okay. So there's this is the prototype. This is the weekly dot YAML that's on its left also. Oh yeah. Very good. Yeah. So what we want to do now is take the, the text that's in the top and do the same edit we did last week to, to transform it into the bottom. Oops. That you just move the comments. You want to put it after those comments because it's a new one. Exactly. Right. Right. And because that looks like a number, it needs to be in quotes. So it needs to be either single or double quoted. Yeah, double quotes are great. And then the release date will be it's tomorrow is the eighth, isn't it? Yes. Yeah. So today is the eighth for you. Yeah. So, so the eighth is the release date. All that work is that now we need to start working on making these messages, right? Exactly. Right. Very good. Okay. So it says, is ignoring this. Post change one. Raise conditioning class loading. And for me, that's a good enough description. Yeah, I would take that. Again, this is me that, that admitting, I know who Jesse Glick is J Glick is a very, very frequent author and he writes very, very well. So whenever I see Jesse Glick and Daniel, I just need to close my eyes and just. Exactly. That's, that's been my approach anyway is when I see anything from Daniel or from Jesse, I just close my eyes and accept what they've got. Now this one does have something to teach us here. You see the characters surrounding the word linkage error? Yes. That sort of backwards quote. Those need to be transformed into an HTML code tag. So a less than code greater than. Similarly, how we did last week for more method. Yes. Yeah, you're right. I didn't remember we've done that last week. Yes. That's right. Hey, ready to go to the next entry. So this is a bug. And first one was also a bug. We should be at the top above RFA. Yeah, I don't remember which order. That's a good question. I'm going to have to look at the style guide to see which order the style guide says. You keep going and I'll look up the style guide. Yeah, okay. So this is one where we may want to beg for mega Kristen's help. I actually like the PR title better than the, better than the proposed changelog. It says do not change fonts when artifacts are shown as, I might say it as a tree. I almost like the PR title better too. Yeah. So, so I would keep the R title. It's more of a sentence. Yeah. And it's also, I think it's not. Yeah. Just like, it feels weird to say fix here, but yeah, I don't know. I kind of like it. Yeah. And now one of the things you need to do, there's be sure you end with a hard stop. With a period. Great. So there's two. The style guide. While we're picking at pros in the previous one, remove of comments, digester from core. I think the of could go. Oh, yes. Yes. Okay. Yes. So which one I suggest. She's, she's saying, keep the, keep the proposed changelog, remove comments, digester wrapper, and delete the PR title. Oh, now whoops, there was one item there. Did you see the developer colon? We want to retain the developer colon because this is a developer specific fix. And that's the end users won't perceive. Is this. Right. And that one you can tell because the category RFE. Or the category is developer. Now there's an indentation change there, dear eyes that you may need to correct. No, it's, oh, yes, it's different in your destination buffer than in the source. Interesting. Is that visual studio code doing that for us? Yes. Yes. Okay. And then. Ah, okay. So there you have a block indent and outdent facility in visual studio code. Nice. That looks right to me. Okay. So developer remove comments, digester wrapper from core and dependent. Yeah. So this is bumping. And for this, you suggested that some of the details regarding bumping from this version to that need not to be shown to the users. So we need to search in the file, whether this specific thing was previously added as a comment or not. If it was added as a comment, then we put it as a comment again as well. Right. Exactly. So look, look for this one. I think this one was mentioned. I think SSHD has been mentioned very prominently recently. And so I think this one will need to be mentioned. Yeah. It has been mentioned. Well, keep looking because that one is actually, I think different. Yeah. Bumping. No. This is the recent one. Oops there. Yeah. So, so we've done one case where we didn't mention it. And scroll down, down one. Oh, there it is. So this is, I would keep this one because it's been a source of a number of recent changes. So I think we should keep it. That's what I would take it as is just take the PR title. Let me double check just to be sure that. That's PR 5547. Yes. Okay. Let's take a look at it. Yes, we've got several approvals. So let's include it. Okay. Good. Okay. Okay. So I think that's a good one. So syntax. Would it be better to say bump SSH, or SSH decor from 2.5 to 1 to 2.70 in Jake and CLI? Oh yeah, good. Yes. I think in Jake and CLI should come after the release numbers. Right. Do you agree? This looks good, right? Yes. Okay. Just copy the comments. Now we have to do the investigation. It's, it's, they've used bump before that's a, a generated word used by dependabot. And so we've usually just left the word bump in there. Okay. All right. Now the challenge. I know some people that probably bothered by it. But I think that's a good one. Do you realize the challenge now is to figure out what the diffs look like to see if it has changed the file dramatically. Can you save that file and let's go out to a command line. And do some, some differencing. So do I get space diff? I think I need to do this. That looks good. Yeah. So we would, it would be nice to get rid of that trailing white space, but it doesn't. Yeah. That's a good file. So that's a, that looks like a big win. Suggesting to. There's no extra space. Interesting. There's no extra space, but it's, have you saved the file? Yes. Huh. Try to get diff again. Let's see. Oh, okay. Now it did the rewrite. Okay. So now how do we teach visual studio code. How do we do that kind of massive rewrite of the file? Any, let's see, maybe you do some searching. I'm going to do some searching. There's probably some configuration. VS code. Not rewrite YAML files. Oh, wait a sec. There was, it said something about turn off. Avoid reformatting. Save. Oh, oh, look, there it is. Command shift P save without. Command is probably alt save without formatting control KS. Okay. Now that it's saved it, let's try that diff again. Well, I guess I guess we could always save just the say is copy the file to a separate location, right? Okay. Reopen a fresh copy of the file and copy just the last block. And then save without form. Sorry, what about using what? Control instead of all. Control instead of all for command. I think it's usually control. Ah, could be. But I thought that when, when Deraj did it, it was just trying to do that save without formatting again. It may be that the files already been modified in, in memory. So it was. One shift. Right, that one. Because what did it say command alt P. Usually it's control. Well, and Deraj is running. Mark said to do all. Yeah, that's so am I. So my proposal. Let's have you copy the changelogs weekly.yaml to a temporary location. And reset it. And we'll just bring it in from the copy. Yeah, put it someplace safe. On your desktop or wherever's comfortable. And now you'll go to your command line and we're going to, or you could use VS. VS code as well. We're just going to ask gift to reset the file. So from the command line, it would be get checkout. Dash dash space dot. So do you want me to do it? Yeah, terminals easiest for me. I know it can be done with this VS code as well. I just don't know how to do it. So get checkout. Space dash dash. Space dot. So what that says is please check out any files that are currently modified and make them match what's on the room on the, in the repository. So this is basically a reset. Now, if you do a get status, it should says tell us that nothing has changed. Now we go back to VS code. Reopen that file. It may have already detected that it's changed. So you want to open the same file on VS code. Right. Open the copy and we'll paste its contents into the, into the original. And then, then when we save the original, we will be sure we have you save it with file save without formatting. Hey, now let's try that file save without formatting. Oh, yes. Okay. Now there's still maybe a formatting problem here because I think so now let's do a make run and see what it says. Because I think the indentation there is still a little surprising to me. Sorry D Raj that this is so complicated. Thank you for being willing to do it. Yes, that's my pleasure. Last time, she said to me that there should be some, there should be RFEs between major bug and another type of change, which was like there has to be some great colored changes between the two red color changes for visual easiness. So let's see if there are any available. Yes. This one, I'm not sure there will actually be any available. So that's, that's why we want to look at it visually here. So open it up in your browser and let's see what it shows. Okay. Go back to the terminal window. Let's see if it's reporting an error or some surprise. You can still hear me. Can hear you. Yes. Well, it's certainly, it's doing a lot of work generating this, the website. So, okay. And here comes the site. I was ready to go through the last week's video on YouTube, but I don't think you uploaded that, right? Oh, didn't I? I thought I had. I apologize. I will, I will get that uploaded. I should have done that. I uploaded two videos earlier today and must have missed the, that one. I will get it done. Yes. I was trying to practice on time. So that's why I was looking for it. Right. Well, and that would have been a great help if I'd done what I should have done and, and had it available for you. Oh, there it is. Nope. You're right. I had not uploaded it. I see it right there. Okay. And this one is showing us that we need to make that change to that file. Okay. Notice that it says 2.296 when you click the change log, I bet it will show us a change log for two, only 2.296 and we just wrote 2.297. Yeah. So now it's generating the change log. Or did it fail? So switch. Oh no, it's coming still. Okay. Right. So it's showing us. 2.296 and we need, because that's the most recent actual release. So we need to. Trick it by show having it show us the next one by changing one of the files on the disc. You remember which file that is there's a file named. Version something or other in the content underscore data. I think it's underscore temp directory. Yes. It's very slow. Right. Would it help if I just stop generating the website for now? It will because we'll need to stop it anyway. We'll have to regenerate it after we update that, that value. Time has also stopped. Well, but it's still, it's still delivering audio for you. It's still seeing the screen. So. No, the file that you're saying is. Yeah, I thought it was, let's see. So. Then data. Just a minute and I can, I can get you the exact name. It is content slash underscore temp, not underscore data. So content slash underscore TMP slash latest core. It's T X T and it's T M P rather than T E M P. And the core there will be, has a capital C in it. Well, and you could do, there's another way you could do it. We can, you can just say echo space. 2.206 202.297. Was that what it was? Was the number version we're working on? Yes. And it's not slash content, but dot slash content. And now if you press tab there, it should complete it for you. Okay. Now do the make run again. And let's see if it. If it will, let's show it to us this time. Go ahead and type make run. Oh, okay. It's very slow now. Maybe you want to stop sharing your screen. That may make your computer much faster. Why don't we, let's have you just stop sharing. Because you know what to do now. Yes. So let's just have you stop sharing your screen. Okay. Let's go ahead and type make run. I'm having some delay. Oh, okay. I'll type make them now. It's very slow now. Maybe you want to stop sharing your screen. That may make your computer much faster. Why don't we let's have you just stop sharing. Okay. I'm going to stop sharing your screen and. That way you get. Your computer performance back. Stop. Let's see. I'm going to stop your sharing. Yes. Okay. And then I'll stop my share. Oops. I still see your screen shared. So it didn't appear to have stopped your sharing. Okay. You should see a blank screen for me. That's weird. It keeps going back to showing your screen. Stop participant sharing. There we go. Yeah, I have a, I have a control on my side that tries to do it. Okay. Hopefully soon you'll get, you'll get your computer will be faster now. Do you Raj? It is a little bit slow. It tried. Okay. Okay. Right. So is it, is it not arriving yet? Okay. Well, let's, let's wait. Okay. I've got another meeting scheduled to start in just two minutes, but I hope we'll be able to finish this. And I may just tell the other people that I'm going to be a little late for their meeting. Because we're so close. Yes. But just in case it doesn't happen. It takes a long time. Should I submit the PR and. The photo. On the PR message. And then you can review it. Right. You could, you could submit the PR. If you submit the PR, even without us having seen it, you've already done quite a bit of help. And, and I can look at the PR. And do some reviews online after this other meeting finishes. So absolutely.  So I think you Raj, if you, if you're willing to do that submit, we could call this meeting done and I will. I'll review the PR asynchronously with you after, after you've submitted it. Mark, what about the minutes for this meeting? We didn't open those up. Oh, we did not. You're right. Meg, I should have made some notes. Yes. Would you be willing to do this? Good. We could just know who was here and what we did. Exactly. Yeah. If you could. Would you like me to do. If you could, that would be wonderful. The links not in the calendar. Yeah. I thought it was, but here I'll paste it into the, here is the, the, I'll paste the link to the minutes into the chat here. So there are the meeting notes. Can you open those notes? Okay. Yes. Great. There we go. I got it. Excellent. So even if all you can do is propose a revision there. I can then accept the change and, and we'll have notes. Okay. So Deraj, you're okay submitting the pull request on your own. Yes, definitely. That would be actually better considering everyone's time. All right. Thank you everyone. Deraj. Thanks again for being willing to work through this. You are heroic. Thank you very, very much. Great work, Deraj. My pleasure. Okay. I'm going to end the call and I'll upload. I promise this time I will open upload to meeting.


Good morning, cyber traders! Happy Monday, welcome back. Hopefully everyone's all excited. Big, big week, Thanksgiving week. Good to see you all, good morning, everybody. Good morning, everybody, on YouTube, Facebook Live with Steve Patrick. Good to see you all. This is the official kickoff of the holiday end of the year. So, we went, we got through September, October, early November, now we're gonna start seeing a couple of, we had a lot of big earnings that came out last week. We did really, really well with a bunch of stocks, specifically did really well with BJs, that did really well, Macy's did well, a lot of, a lot of retails came out with some good earnings. But we had some really, really good moves, we're gonna check them out. We got a good mover right now, this ISPC.  Since this morning, stock had a really, really, really big pop. It wasn't on the morning watch list because it didn't move, it came in around eight o'clock. You could see it right here that it was one of those eight AM movers. Remember, there's like three times, seven, 30, eight, eight, 30, and then there's also nine of another pop, which probably some of you gonna see some things move. But this thing moved really nicely, went from five to almost $10. Great little Fausto flag.  Big, big push. On level four, we're getting a lot of big buyers. Great pre-market. Once again, this is what the alert service is all about. So if you are subscribed to the alert service, you know, this is what you're gonna get. And not only that, there's gonna be a huge announcement that's gonna come in here in Cybertree University. Going into Black Friday, we got a lot of great promos that are coming on. And not only that, but we have some really, really good techniques and strategies that we're gonna add to our education, something that we've been working on and for, it's been actually been over, it's been a long time, almost like maybe a couple of years. But we're going to introduce a brand new app that we have with our alert service and it's gonna be better with the chat room, everything else. So you guys should be really, really excited about it. And once again, if you're a golden platinum, you know you're already locked in. That's just by nice about it. So we'll worry about that later, but let's talk about what happened on Friday because there were a couple of good stocks I wanna talk about. First of all, Sava was a really, really nice move. Sava was a stock that got crushed. You could see it right here, went from 48 all the way to about 56. Nasty stock had some nasty shakes right here around 10 third. You can see that thing went from 51, went back to 56. So, but that stock is a gap fill. Stock took a big hit from 100, went all the way down to 42. Stock's been not only a good day trade, but it trickled in into a swing trade. That sell also nice big pop right there. 940 all the way to 1020. Obviously you remember what happened on Friday. We had the bill that passed Congress and a lot of these social programs and the Green New Deal were part of that. So a lot of these stocks did make some big moves because of it. So that one worked out pretty well. I was really excited about BJ's though. BJ's, we did really, really well with that stock. Great news on that last week. That was a great swing trade for a lot of us. Listen, the longer it holds here, the better it is. That's what we want. This how you learn how to become a very good swing trader because you gotta know how to day trade first. That also did pretty well. Couple of the ones also made the watch list, but they're also taking big hits. I mean, LGVM was awesome last week. This thing was just ridiculous. This was some of you guys missed. This thing went from like $3 all the way to $15. This was probably our biggest winner of the week. Listen, you know what? This is what we do and this was great. PROG was also a nice one that we traded last week too. That one went from $3 to about $6. Starting to go up on Friday, it's pretty good actually. It did really well on Friday too. What else we had that they were, oh, LCID. That thing has just been an unbelievable swing trade too. Beginning of November, $25 up to $55. She moved really well on Friday too. You can see she went from $46 all the way to $55. Starting to get pretty volatile. You got to be a little more advanced trader to trade that, but that also did well. There are a couple of the ones, but you know what? Let's move on. I don't want to, it's always nice to remember, but I don't know about you. I always got short-term memory loss. I don't worry about what happened last week. All I know is had I finished out the week and it's time to move on, now let's move and see what's going on to this week. So once again, today's Monday. We're going into the, this is probably the biggest holiday of the year. Bigger than Christmas, bigger than everything. And I'll tell you why, because everybody celebrates Thanksgiving. So I know our Canadian friends had the Canada Day just recently, but here in the US, the markets closed. Not only that, Friday is a half day. So expect it to be a very kids come back every kid's coming back from school. My kids are already back early. They came back on Friday, but a lot of kids are going to be coming back this week. So don't expect it to be crazy. The only thing that's going to make certain stocks make big runs this week is that if they have to come out with news, you're not going to see a lot of earnings coming out this week. You're not going to see, unless there's some type of a catastrophe or some company had to come out with news, which does happen. Some people like to be quiet. A lot of these companies all of a sudden they like to push these things through to not make it too publicity, too much publicity going on, but you might see a little bit of that going on. But just don't get too cocky. Josh would always tell you, and Josh is back from vacation. Let's give Josh a very welcome back, Josh. Welcome back. I know a lot of people missed you, you know? But by the way, I got some great compliments on Rich. Rich did a great job also being a moderator in the room, do it. So let's thank Rich for doing a great job. Thanks, Rich, for everything you've done. Listen, this is the beauty about being part of C2U. You know, we've got a great staff. They love what they do here. And you know, that's what it's all about. So anyway, let's go about a couple of things that are moving and let's talk about some stocks. Once again, we got this big runner up here that's on the watch list. You know, we got, oh, look at that. Boom, what a great pop on it. You see that, that big buyer out there just ran that stock. ISPC is right now probably, you know, I mean, look at that gorgeous run right there. Is that unbelievable? So you know that one's on the big watch list. So you guys doing pretty well on that one. Look at that now. And look at it, she's just starting to make a big run right now. Which, you know what, I, where's the, oh, there it is right there. So you can see here on the level four that this seller right here just got filled right here at $10. You see that big spike? Let me tell you, we all know what's, what happens if this was during the market hours. This stock would have been halted, okay?  Unbelievable. Couple of the stocks, listen. You know what? There's a lot more out there. So we'll keep an eye on those. We got this warrant out here, which is obviously picking up from ASTR. I don't know what happened.  So this is gonna go on the B list. So not really too crazy about that one. VG, another one, gapped up pretty nicely. Great spread, beautiful spread. Level one stock, you know, stock trade 14 million shares. But she's starting to trend down. She was also pretty decent. She was at 2060s or a 23rd. She's not moving that much, but you know what? She's a little bit more expensive, but she trades like a level, she is a level one stock. Another one gapped up pretty big this morning. She started trending down. I see some support levels where it's holding right here. The stock could probably bounce, you know, if it holds here, but we'll check out the buyers and see how they're doing on level four. And there wasn't that much out there other than, and remember you only need one is the ISPC, okay? Yeah, ISPC, I'm watching it right now. Grant, you can see it's holding right now at 1136, but that's a hell of a pop. That thing that ran a dollar 20 since we actually started this meeting, which is hands down. You could just be done for the day. And you know what? Maybe you can start worrying about Thanksgiving shopping. This stock's getting crushed. And when we know what I got crushed, we would train this thing, look at that. This is awesome. Stock goes from a dollar to $3.00. We did really well with RETO. And what took about two weeks to get there takes about two days to come right back down to reality. Great short squeeze, but you know we're not trading it. Same thing with GTEC. Another big run up we traded on Friday. The stock is getting hammered right now. You know what? I don't know if it's on your short list, but that stock is not looking too well GTEC. So a lot of these stocks that, but once again, that make big run ups. And what you have to worry about is when they go up that fast, they come down crashing that fast. Couple of these spoon stocks right there. We all know about this stock CEI. She got the pretty big this morning. Just come back down. Be careful, we trade this stock. You remember we trade a 50 cents with the $5. You know what? I did really well with that, but I got suckered in a couple of times. I lost it. I refuse to trade anymore. It reminds me of dryers. Okay, I did it. It gave me a lot of money on dryers, the shipping stock, but then it plays these games, these teasers. It's just all algorithms. Don't even waste your time on it. This stock right here, good spread, good tier sizes, but it's another ETF. So what you talked about, a lot of ETFs are moving in the New York stock exchange, which most of them actually are trading in. But you could see on the list is really not that much out there other than what we're seeing over here on the NASDAQ. New York really doesn't have that much out there. Look at Apple. Apple's doing great. Wow, look at that. Apple finally broke all-time highs with Apple. I thought Apple was gonna lose that. It wasn't looking too happy right here and went from 155 to 140, but look at that. It's been, Apple made a big U-turn right there. So hopefully it turns the market. Market's up right now, right? How much we up? About 70, 78 points? Not too crazy. All right, guys. Anything I'm missing, I did not call out. Anything we're missing that we did not see? Anybody I knew, Julia? I see you chatting in the, you know, Julia said I can't believe that it's the 11-22. Anything we're missing I did not call out? Some news bumped up Apple, Norman.  It was going down. It is a big mover in the market, so everybody follows that along. I've been in, yeah. Oh wow, Mark, you've been into LGVN since this morning from 7.30. Just be very careful on that stock.  I would want it to break this resistance before it goes any higher, but I really don't like that stock.  I mean, I don't know if you're short, but long I'll be really, really careful on that. All right? All right, guys. So let's get back to the market opens up. We got about 15 minutes. So let's get ready for a market. Remember, it is Monday. It's a very, very, should be a, you know, I wouldn't be too over excited about trading. I'm going to take it pretty easy. You can see on my watch list that everything is pretty much green, but nothing is substantially green. So we're going into a big, big holiday Thanksgiving weekend. I know that I got to do some food shopping. So I'm just going to focus on the morning, do what I got to do in the middle of the day and get ready to put up my Christmas lights. So I'm expecting to have a lot of people in my house over Christmas. You know, we didn't spend Christmas. I usually spend Christmas day at my house. I get like, I don't know, 40 people. My wife, I think she, I think we're going to probably have almost, I don't know, maybe like 60, you know? So it's going to be exciting. I don't know if I can, I usually cook it all, but we'll see how it goes. But so enjoy your weekend.  But just be careful. It's Monday, don't get too cocky. Good luck everybody. And happy trading. We'll start commentary in 30 minutes. If you're watching YouTube Facebook Live, we'll be back here at 2.30 for the afternoon meeting. We'll see you back then.


All right. So for today's lecture, we are going to discuss basically the last two relevant items in the theory of Markov decision processes and how to optimize decision making. And so in the first half, we will basically discuss how to put at work all the theoretical results that we obtained in the last couple of lectures. That is how to construct algorithms to search for optimal strategies in policy space. And that would be that for the first half. And in the second half, we will address the problem of what can we do when the state space or the action space is just too large? So we cannot approach such a search procedure or directly the optimization guide, the solution of the Bellman equation, just because the sheer size of the state action space is so huge that we cannot address it. And therefore, we will discuss how to combine these algorithms with the function approximation. So that's the plan for today. But let's go one step at a time. So let me start by giving you a brief recap of what we have in our hands so far. So on one side, we have the solution of the Bellman's equation. Fine. We will use it explicitly tomorrow in the tutorial. So in the lecture tomorrow, we will see how to explicitly solve the Bellman's equation in two examples. One is the traveling assessment problem, in which the problem is explicitly time dependent. And then the second one will be solving the grid word in one or two instantiations, again, by Bellman's iteration, that is, by the value iteration algorithm. But now we are focusing on algorithms that take explicitly into account the policy, the structure of the policy. So summarizing our results so far, we have been somewhat painfully. But successfully, we've been able to derive the following expression. So remember, just recall that our goal, as always, is to find the policy which maximizes our total gain, which is more explicitly find the policy which maximizes the expected value of the sum of discounted rewards along a trajectory generated by our Markov decision process. And yes, what you have shown yesterday is that you can explicitly write the gradient with respect to any component of a policy pi in the following form. These eta and q are two functions, two, respectively, vector and a matrix that have a very specific interpretation. In terms of, basically, these can be interpreted as occupation times of state S discounted. And this is the state action value function. So let me just record you the basic definitions of these objects. So the eta as a vector was defined as 1 minus gamma p to the minus 1 times rho note. So this rho note is the initial distribution over states. This is where my system is at the initial time. It could be localized on one state. It could be dispersed across all states. And this capital P here is a matrix whose entries are defined to be just as the transition probability from one state to the next one given the policy pi. So this P explicitly depends on pi and eta implicitly depends on pi to the dependence on it. And then what is q? q is defined as the expected value under the policy pi. If we start from a certain state S and we pick action A, which means that if we are in state S and pick action A, we will end up in state S prime. And in this process, we will incur a reward as A S prime. And then from state S prime, what we will get under that policy is the value starting from state S prime. And what is the value? Well, the value you can also rewrite it in a simple form just as the following definition. This is just RT1 minus gamma capital P to the minus 1. This is the same capital P that is above. And what is this? R is a vector. It's a pro vector whose entries are just the expected rewards starting from state S. So if I'm in state S, I take action A according to my policy. I end up in state S prime according to my transition probability. And then I collect an average reward RSA S prime. So far, so good. This is just a very short summary of the definition. So in practice, so let's go back to our example when there are two states and two actions. Two states, two actions. In this case, the policy space is a square. So on this axis, we have the probability of taking action 1 given state 1. And here, we have the probability of taking action 1 given state 2. These are the only independent variables in terms of the policy. And my G is something that looks like something like this. This is the level lines of my G function. And then the gradient is some vector. So if I take any policy pi here in the middle, this is my gradient of G. So at this point here is any policy pi for my system. So how does that work? How do I compute this gradient in practice? Well, remember, we start with some things that we know. So the known things are the initial density, the initial distribution of our states, the policy, which is something we choose at any given point in this space of policies. Then we know the transition probability is small p. And we know the rewards are. These are parts of the model that we have. And we assume to be correct. And then from this, we can construct several things. So from pi and p, look at this formula here. Combining a linear combination of pi and p gives me capital P. And a linear combination of pi, p, and r, differently arranged. So I can combine this three together, p, pi, and r, to get my capital R. And then if I combine a row node with capital P, I can get peta. So how do I get this in practice? Well, one possibility is that now that I have my capital P metrics, I could just make this linear combination and then take the inverse. The inverse is relatively expensive as a procedure.  Whatever. The important thing is that this is a linear object. There's nothing nonlinear happening here. You just have to do inverses or you can use iteration methods. So you can deploy all the linear algebra that you know, just take off the shelf and the solver for a linear equation, and that will do.  OK? So you know, this can be written as a linear equation of the four metrics. Capital A times eta equals some rho node. So you just have to solve a linear problem here. And same for V. OK? V is also the solution of a linear problem, which is nothing but the recursion relation in the sense that V transpose 1 minus gamma P is equal to R transpose. Again, same structure only that now you're multiplying on the left. OK? So this is your unknown, and this is the matrix. This matrix you know, and this row of entries you know. So it's just an inversion. And you know that these metrics that you have to invert is well-behaved, so OK. And then this means that if I combine P and R, I can get my value function. And then from the value function, I obtain by combining it with the small R and P, I get my Q. OK? So these things are already in stock. This object here is also known. So the three of them together give me the Q. And eta and Q together give me my Vg in the pi. OK? So this is the flow of operation that you have to do. If I give you a policy, you will be able to compute the gradient. Given all the other side information, which are important, that is what is the initial distribution of states, what is the transition probability of small p, and what is the reward structure small R. OK? And all of this is just linear computation. So you can figure out what the complexity of these operations is. And as you can clearly imagine, this complexity raises growth very, very rapidly with the size of the system that you're considering. OK? But conceptually speaking, there's no mystery, no difficult step here. So now that we have our gradient, the question is, what can we do with that? OK? In this case, in this specific case, we know that the optimal solution is here. Just because I draw it like that. If you try out a different combination of rewards and transition probabilities for your two states to action, more or you might find different landscapes of your G function. OK? So let me repeat again. These are the level lines of G. So G here is increasing. And here is the maximum. So this one would be my optimal policy by star, which according to this sketch would be, this would mean every time that you are in state one, pick action one. And every time that you are in state two, pick action two, because the probability of picking action one is zero here in this point. OK? So the question is, how do we get from any initial policy pi to our optimal policy? So how do we search for this maximum of G in policy space? So in practice, what we are looking for is some sort of gradient ascent. We want to climb the gradient until we get to the maximum, which we know it sits on the boundary. So what we're going to do now in the next 20 or 30 minutes is to study different approaches to this problem of optimization when you know a gradient. So the things that I will describe now are in part are common to all kinds of optimization problems, and in part are specific of this optimization problem of an MDP. OK? So some things will be general, some others will be more specific of this problem. But these are nevertheless very, very broad concepts and ideas. OK? Fine. So the first algorithm that we will discuss goes under the name of policy iteration. So what is the idea? Well, the idea is very simple. It's sort of brutal in the sense that I rewrite here that my gradient is object here. So once more, I hand you a policy and then you can compute all these components of the gradients. For every pair A and S, you have a real number, which is the right hand side here. And this tells you basically how much your objective function depends on the various components of the policy. So policy iteration does the following thing. Suggests to pick as an action the one which maximizes the component of the gradient. So let's go back to our sketch here. So you have components for each direction. OK? So you have your component in this direction of the gradient. You have your component in this direction. And then for each of those, you pick the action which gives you the best. And so in this case, for instance, this gradient is telling you you should move to the right and you should move down. OK? So what this policy iteration is doing is basically is telling you believe in your current gradient, wherever you are, choose your gradient and go all the way to maximize within your policy space in that direction. OK? So you just compute the gradient and you go full throttle in that direction, which is what is expressed analytically by this expression here. So among the many components of your gradient, you pick the largest one. OK? So just to give you a further example. So suppose that now you are somewhere and you are told that you have to go, for instance, in direction north, northwest. So you have a compass and you say, OK, I have to go north, northwest. And what you're going to do with this is that you're going to go north, because north is the direction which is closest to your orientation of the gradient north to northwest. OK? So basically you ignore all the variations. You just fix on your relevant axes, which are just north, south, east, west. Let me say, OK, I'm going to take the direction, which is the largest component of my gradient. I hope this makes the idea clearer and not more confusing. But if you're not totally clear with this, just let me know. So by definition of the gradient, OK, if I have to pick the maximum over my A's, this means that this eta does not play any role here. And this is just the arc max over A of my Qs A. So this choice is also called the greedy optimization. OK? It's greedy because we, based on our current knowledge of what we're going to get in the future, according to this strategy, we go all the way with it. OK? So we do not allow for any other possibility than this being true. OK? So in graphically speaking, if we go to our example, again, with two states, two actions, this is my policy space, the graphical representation. So now, suppose that I have a different form of my Q function of my G function over that. OK? So that, for instance, locally, my optimum is here. But locally, my gradient points in this direction. OK? So it might well be, right? That is, it has some form like this. OK? So that eventually, your maximum is here. But if you start from some point here, maybe your gradient is pointing in that direction.  OK? This is what this is telling you. Pick a policy, your new policy, by prime. If you follow the gradient like this, will be a deterministic policy which assigns with probability 1 the action 8 yielded. And with probability 0, any other action. And remember, this is the characteristic function of this condition. So it means explicitly that this object is equal to 1 if a is equal to a star of s and 0 otherwise. So if I do this, then my local gradient here would tell me, OK, you have to go here. So my pi prime will be here. And then if I repeat, what I'm going to do is now I'm going to evaluate my gradient in this new point here. OK? So my new gradient will be pointing like this. So if I follow that and I repeat my operation, where will I go? I will hope here. OK? So this is at my time, my second guess now will be here. And if I compute the gradient here, now my third guess will send me, the gradient will send me here. And if I follow the gradient, I will end up here, which eventually will be my maximum. OK? So in this sketch, starting from a point pi, I have made the three steps to get the maximum. Because this operation of picking this greedy optimization sends me straight in the corners of my policy space. Remember, in general, the policy space is not a square. OK? It's a more complicated object. But it has corners. And these corners correspond to the deterministic policies. All those that are made of a single one for one action and all zeros. So this idea of policy saturation in a nutshell is compute the gradient, then go to the corner where your largest component of the gradient sends you. And then recompute the gradient from that corner, go to the corner in which you are directed to, and so on and so forth. So there are two questions here, of course. First of all, is this ever going to converge? I'm hoping through corners. So in general, my state space, suppose that I have s states and two actions. My policy space is an hypercube. OK? So policies belong to 0, 1. OK? Because if I have just two actions, there's just one single real variable from 0 to 1, which describes the policies. So here is, for instance, probability of taking action 1 in state s. And so this is power to ds. So this is a hypercube, an s-dimensional hypercube. So my algorithm, my policy improvement algorithm, as it is defined, would start somewhere in the middle of this cube, then jump to one corner, then jump to another. But there are exponentially many of them. OK? So will it converge, eventually, to the optimum? First question.  Could be exponentially long time to explore. OK? So first, we address the first question. OK, let's see. Now we're going to prove that this policy improvement algorithm does indeed converge to the optimal solution, which makes it sensible as an opportunity. And then in the second step, we will address this question. Can we do something smarter than this? Everything clear so far? Any question? So let me. This was a very sort of informal description, but now we can describe the algorithm in full. So this algorithm is called policy iteration. So the pseudo code for this is initialize a certain pi 0. This index 0 is just a counting index so it tells me what iteration I am in. So this is the first initial step. I choose one policy, whatever. Any admissible policy, anything that is inside my space of policies. Then I have a loop that goes as follows. From any pi k, I obtain a value function. OK? How do I do that? Well, let's go up a bit. You remember this flow here. So if I have a policy, I can construct all these terms and I can get my value function. So this step of giving a policy, computing the value function, is what is called a policy evaluation. It's pretty self-explanatory as a term. You have a policy and you compute a value function. Now from this value function, you can compute the corresponding queue, which is defined, you remember, here. So from this line, you have a value. All these things you knew from the beginning so you can combine them together and get your queue. And then from your queue, you get through the greedy choice, you get a new policy pi k plus 1. What does this arrow mean? Well, in practice, this means that your pi k plus 1 a s is just the arg max over actions e bar of queue k e bar. So this is what I just wrote up over here. It's actually this notion. So it's your a tilde's right here. This is also equal to 1 over a tilde k because this is taken from queue k of s. So you see the flow. Policy value, value queue function, queue function by the greedy choice new policy. Then k k plus 1 goes into k and you close the loop until some termination condition. For instance, the usual request is that your value function isn't changing much.  Or again, you might choose the relative change in v. So this is the algorithm. And what we're going to do now in a few lines is to prove that policy iteration converges to the solution of the Bellman equation. So this is not very complicated. So we're going to do this in a second. So the key step is to show that when we perform one step of this loop, basically we go from one gk to a new gk plus 1. So this g is my objective function. So this means that in my graph, every time that I make a jump like this, I'm actually going to a level lines, which is higher. So what we're going to show is that this object here is larger or equal than the previous one. So there is an ordering here. Actually, the proof goes into two steps. The first step will be that it's always less larger or equal to the previous one, the value of my objective function under one iteration of the policy. And the second thing will be that if it's equal, then it's the Bellman's equation. And this will close the matter. So by the way, this second step here, so this first step is policy evaluation, this second step is called policy improvement. So this policy iteration algorithm is an alternation of two steps. Start with a policy evaluated, which means compute the value function. Then from the value function through the greedy assignment, compute a new policy. Repeat. So the proof goes as follows. So remember that for any given policy, my objective function can be written in terms of the value function as the distribution of initial states times the values from those states. This is just another way of writing the expectation of sum over 0 to infinity of gamma t. Now let's concentrate on the value function. So the value function for my policy pi is just very simply connected to my Q function. So this first step is quite straightforward. Let's go up a little bit to the definition. Oh, sorry. So this object here is the definition of Q.  So this is the connection between the value and the function, or more directly. So what is the value function from state s? Is what you expect to gain starting from that point according to the policy pi, which in itself is nothing but peak action a. And then this will be what you expect starting from that Bayer state action. OK, so quite straightforward. All right. Now we are going to use the definition of my a tilde here. So what is a tilde? Is the maximum overall possible actions for my Q. So by definition, each of those is smaller or equal than Q pi s a tilde of s. This is by definition of a tilde. Because for any s, for any row s of my Q matrix, I pick the column which has the largest Q. So this is the first step. And the second step is to use the definition of Q pi by definition of Q. This is equal. Sorry, just one step before. So this object here, now notice, sorry, one more step before we go there. So we take all possible precautions. So now notice that this term here does not depend on a. So this means that our sum over a is only restricted to the pies, which are probabilities. And therefore, this object is equal to one. So my sum reduces just to a tilde of s. And then I can use the definition of Q pi, which is just sum over new states, S prime, S a, then a. Where here I have, instead of these a's in general, the present in general, I have my a tilde of s. So this means that at my current state, I'm picking the greedy action, a tilde. And then from that point on, I'm still sticking to my policy pie. Now look at what we have here. We have our policy and we have the same policy in a new point S prime. And these are linked by inequality. So what we're going to do here is just we're going to plug back this inequality back here. So this object here will be smaller than something on this side. So I'm going to apply again this inequality iteratively. B pi of s prime is going to be smaller than some other thing. So let me write it explicitly. I iterate the inequality. And then I get that this is smaller equal than, I have my first part here, sum of s prime, d of s prime, s a tilde of s, s prime, plus gamma. And now this object is smaller than again the right hand side. So we have another sum inside here. Now with respect to s second, p of s second, s prime, a tilde of s prime times r of s prime. And then I could do this again and again and again. So I can unroll all these inequalities. And what do I get? Well, look at each of these terms. So the first term here, which comes from this contribution, well, this is the expected reward under the choice of the greedy action. And the second term is gamma times the reward that I will get at the next step if I still keep on using the greedy action. So just with the moment of reflection, you realize that this object here is nothing but the expectation of the sum from t going to 0 to infinity of gamma t of the reward that you take if you are in state s t, you pick the action a tilde of s t and you end up in state s t plus 1. This condition on s 0 equals s. Just by repeating this unrolling of this inequality, eventually this will be less equal than the sum of everything. But this object here is the g taken according to the greedy policy. So this is g of pi prime by definition because I'm taking every time I'm picking a tilde, which is my choice according to the pi to the iteration pi prime. So if you look at this, sorry, this is not g pi. My bad, I'm rushing this through. This is v pi prime of s. So if you look at the term, the final term here and the initial term here, these are connected by a series of inequalities which go all in the same direction. So I can conclude that the value function of the new policy is larger than the value function of the previous policy during one of these steps of policy iteration. And this, of course, implies if I average over the initial distribution, this implies that my g pi is smaller than g pi prime. So this confirms the not so obvious intuition, actually, that every time that you make a step according to this greedy strategy which sends you in the corners, you're actually improving your g. You're moving up in the level lines every step. Final conclusion about this, what happens if there is an equality? If g pi prime is equal to g, then what do we have? Well, we have that. This means that v pi prime of s is equal to v pi of s. But let's go on step at a time. So v pi prime, by definition, is the maximum over a of q pi s a. So this is the greedy choice. But since pi is equal to pi prime, this is also equal to the maximum over a of q pi prime s a. But this, in turn, is equal to max a by definition of sum over x prime, again, using the definition of q. And now if you look at this object here, this equal to this. This is just the Bellman's equation. So this is, I mean, this is a sort of lengthy manipulation, nothing particularly interesting here, apart from the bottom line, which is that this process of using the gradient to visit all the corners of the policy space in order is guaranteed you to bring from one corner to a better corner and eventually reach, without exception, the optimal policy. So this motivates the use of this policy iteration approach, which you will find described also in the book by Satom and Bartow, but without making reference to gradients. But you have to realize that that's exactly the same thing that we're discussing here. OK, so is this a smart idea, after all? Well, there might be something smarter, right? So if you have a gradient, you might want to look for, for instance, gradient ascent. So rather than jumping on the corners, you might want to look for some way to go down the gradient, which will take you some sort of trajectory like this. Sorry, going up the gradient, because this is a maximization process. So how do we do gradient ascent for this problem? And that's what we're going to do after the break, OK? Sorry, can I make a question? Yes, please. Is a question about the notation. I didn't understand the difference between A tilde and the pk plus 1. Because in the policy improvement, we are using the formula of the A tilde. And then for saying that pk plus 1. You're making reference to this one? Yes, yes. Yeah. Because up we wrote that A tilde is equal to this formula. Yes, that's exactly the same thing. So there's no difference, except that you are repeating this operation at every step, which means that you have a different queue over which to optimize. So if you give me something more, I can explain more. But I'm not sure I understand the question. Because no, OK, OK. Is it clear, or you are just giving up? No, OK. I have to watch it maybe better later. So no problem. We can go through it. I mean, I think it's helpful for everyone now. So this choice corresponds to picking up the column in the queue matrix, which has the largest entry. So given a row labeled by a state, for each row, we pick up the column, which corresponds to an action, which has the largest entry. And this defines our A tilde of S. So in our queue matrix, which, for instance, is made of states and actions. So for every state, we run through the values of this queue. And then we identify the one which has the largest entry. So for every row, this dot means the ones which has the largest entry, which may be equal or different. And this defines my A tilde of S. Because for every S, I have an A tilde of S. So given a policy, I compute a queue. So given a policy pi, I can compute a queue. And from this queue, I can compute my alpha tilde. So this is the first step, which is called the greedy operation here. And I'm repeating this thing over different policies. So I take one, and then I use this new policy to compute a new queue function, which will give me a new policy by a degree of choice, and so on and so forth. Is that any clearer? Yeah. Thank you. Thank you. Any other question? OK. Then we take a break until 10 past 10. And then we resume the rest. Share my screen. So we've seen that there is one possible way of exploiting the knowledge of the gradient of our objective function in policy space. And it is through this policy iteration algorithm, which basically sends the policy in the corners of our policy space and then jumps from one corner to the other according to the direction of the local gradient. And this is guaranteed to terminate the search in the corner, which has the largest value of our objective function, which corresponds to the choice of the solution of the Bellman's equation. But like we said, there are other ways of using the gradient. And perhaps the most intuitive and metro one is to use it in a more local way, rather than jumping around in the policy space to follow locally the gradient and to climb the landscape of the objective function, which is depicted in this picture by this green path, which is described here. So the basic idea is to move to different kind of algorithms inspired by gradient ascent. So intuitively, how does gradient ascent work? Well, the idea is that if you have a gradient, you could say, OK, I'm going to move from my current policy using some little step beta in the direction of the gradient. This is a very intuitive idea, making us a step. And the length of this step in the general case is proportional to the intensity of the gradient. And this is called proportionality constant. It's called the step size beta here. And of course, there are immediately, and this is very general as approach. So it could be policies, but it could be any space, any domain, any function, any real function on a domain. Of course, the question is, how do I choose the step size? First thing. And the second thing is, what should I do if I end up outside of my domain? So let's address the second question first. So again, using, as an example, a policy space, my square. So if I am at a certain point, I suppose I am considering policy pi here, and my gradient is pointing in this direction, then clearly, if my beta is large enough, or if my pi is close enough to the boundary, I may end up with my new point pi prime, which is sitting outside of the domain. So what should I do in this case? Well, what all these kind of algorithms do is to prescribe some way to remap the point again inside the domain, or on the boundary of it. So all this kind of procedure, like gradient ascent, are usually combined with some projection back in the space of policies. So it's almost always necessary to do that, except in some cases. There are some cases in which if you do the gradient smart enough, well, you can happen to stay always inside the domain. And we will see an example later on. OK, of course, here there is an issue, if you want. Is that that the length of the step that you're doing in this process depends on the strength of the gradient if you keep the beta constant. So another opportunity that we have is to use a constant variation. So what is the area? Yeah, there is, for instance, you could do, OK, let's use the gradient. But now we use it in a different way. That is, we normalize the gradient by its length. So what is it the advantage of doing that? Well, is that now your parameter beta, you can interpret it as the distance between the previous and the following policy. OK, so in one case, if we go back to this scheme here. So in this first case, let's say that this is the one in yellow. No, I'm not managing to. OK, painful. So let's say this one. So what this algorithm is doing is that if it starts from a policy, then there is some gradient which is pointing like this. Here. And then it makes a step to a new policy by prime. And then in this new point, there is a gradient which is, for instance, much longer, much greater, much larger in magnitude. And then this new point will be by second. So the distance in policy space depends on how large the gradient is. This is the first option. What this other algorithm would do is think one. So what this algorithm does is that it moves along the same directions. So it starts always starts from five. And then it goes by the same length, always the same length. OK, so all steps are the same or the same length. It moves more uniformly in the space of policies, which is also a reasonable option. One can perform a lot of analysis on these algorithms and prove that they converge with trade that they converge and what are the properties that must be enforced, typically convexity of G. But this is just an overview to let you realize that there are actually many, many options on the table once you have a gradient at hand. And this requires also a lot of craftsmanship in choosing the proper ways to do it, depending on the problem and depending on the space. I have a question. Yeah, please. So how do we exactly know that we are outside the space of policing? Well, at every step you have to do the check. Let's go here. So the boundary of the space policy aren't known for us. The boundary, sorry, is? The boundary of this policy space. Yes. We don't know them. No, you actually know because the check that you have to do at every step. So check if all the pie. Must be. No negative and the sum. For every S. The sum of pie a has must be equal to one. So you realize immediately if you're violating these conditions and in which case you have to do some kind of projection. I'm not spending a lot of time about what kind of projection you can do, but there are several. And it's a piece of analysis in itself to define properly defined projections because if you define that them in the wrong way, maybe you get stuck into some bad place in the in your domain. There's a good way of doing it that preserves the, but I'm not spending time on this because actually we've never used this. Thank you. Okay. One thing that you should quite the, you need to realize is that when we reason in terms of the length of the steps that we make the distance between one policy and other, we are using the opinion distances here. But does it make sense, doesn't make sense to define the distance between two probability distributions in terms of equity and that distance. It's it's legitimate, but maybe there's something better that we can do here, something more meaningful. Okay. So in order to understand what we can do. Let's reconsider the gradient ascent from a different viewpoint, slightly different viewpoints. So you might have seen this before. So this is called gradient ascent. Pre-visit. So what I'm going to tell you is that you can see one step of gradient ascent as an optimization problem of the following sort. So suppose you want to take the maximum overall pies. And then I explain what this means of gradient of G times. By prime minus by this is computed in pi minus is there's a beta minus one half of the Euclidean norm square is the L2 norm. So what I'm what am I writing here so suppose that I have a policy pie. Okay, I have a gradient. G pi. This quantity here is approximately is a first order expansion. So it's approximately equal to G pi prime minus GPI. Okay, it's a local Taylor expansion of the difference between my value in the new point by prime. Okay, I agree with that just first order expansion of GPI prime around GPI. So this this term here inside the bracket means that we're trying to get the largest possible value of G. But but we are adding some cost for being too far. So I'm going to compromise between following the gradient in order to optimize over my objective function, which is the first part here. And the cost for going too far. And the parameter which threads off these two requirements is my beta here. So, if beta is equal to zero, then I don't move by prime will be equal to pi, because clearly the maximum of minus the distance squared is equal to the minus the minimum of the distance square, which is zero if the two points are equal. On the other hand, if beta goes to infinity. What is happening here is that I am ignoring this cost for going too far. And therefore I will be certainly projected very much out of my domain. So there is some beta in the middle which does the job of compromising between these two different requirements believing your local gradient and not believing too much. So the important thing here the problem messages that if we solve this optimization problem over the next by prime. We are actually doing gradient ascent. Why is that, well, because it's a you realize that the term in brackets here. This one is a quadratic object in by prime. This is quadratic. And since it's quadratic in pi prime, because this first term here is linear, and this one is just quadratic, then we can do the optimizations in a very straightforward manner. So we just have to take the derivative of what is inside the bracket with respect to pi prime. Okay, so if we do that. So let's let's introduce an object which we can call the Lagrangian, which is just this be right. So this product is not product here and this norm here is just I some overall possible a NS so explicitly, I can write this down. I'm a clearer this DG by s by prime. Now our case right of course this is valid for much larger. This is one possibility there are many of those right so this is the prime. Okay, so if notice that I have to add something here because I want to optimize but by primer must be a probability so I have to enforce the fact that by prime, the sum, the sum over by prime of as must be equal to one. So actually, I add to my Lagrangian multiplier, which enforces this condition this is the Lagrange multiplier that imposes the normalization of probabilities. And then I can take the derivatives of my Lagrange function in order to find the maximum. And how does it look like. How does that look like well. It's very simple because this is a linear the first term here is linear. So, the derivative gives me beta. I, then these are the second term here is quadratic, which gives me minus by prime minus pi. And then I have the Lagrange multiplier which enforces the normalization. Okay, but let's let's ignore this for a moment. The most important thing here is to. Okay, I could put the lambda and then the post normalization. Now this object here, the job that is doing is you realize this gives me pi prime equals to pi plus beta grad g pi. So this is the gradient descent step. And what this object here does is it enforces normalization. So this is the projection. So, I'm not going through the details you could also implement via Lagrange multipliers the condition that the pies must be the pi primes must also be positive. Okay, I'm not doing that but this is just to tell you that there is a deep connection between gradient ascent with projection and this optimization problem. So you can go from one description to another following the gradient also means making some compromise between the gradient itself and some notion of the distance. So, this, why do we care about this connection because this suggests different ways by which we can follow the gradient. And the basic idea is that we want to replace this object replace with some other notion of distance, because this notion of the gradient distance for probabilities is probably not the best one. This is also a very general idea that when you have to deal with complex spaces not necessarily do you pretty and measure is the is the right one. So what is the what is a better choice. So, what we want to do is want to repeat this kind of argument, but now with a different distance so we want to move in policy space, according to this. So we want to find by prime. My new policy by prime is going to be the arg max over by prime of the same thing, some beta, the gradient of G times by prime minus pi. And now I'm going to penalize this with some other notion of distance between the two policies. So do you know of any reasonable definition of distance between probabilities that you have encountered so far. Is anyone of you aware of. So if I have two probability distributions what is a way of measuring how far they are from each other. Exactly. Okay, so what we're going to do is going to repeat this argument by using kubak library divergence. So if you're not, if any of you is not familiar with the notion of kubak library divergence, this is something that you should fix, because it's a very widespread notion in machine learning and probably any information theory, but I assume that the level of knowledge that you need to be able to control this kind of calculation and operations is sort of very very basic so you can find it on every, on every manual, it's not very. So, now we're going to use KL divergence but in a way that is tailored to our problem at hand. Okay. So what we're going to do because we don't have a single policy we don't have a single probability distribution we have a collection of probability distributions. We have a probability distribution for every state. So we combine sort of this all these policies to make a global distance between two policies rather than between two policy property distributions. So, the way I'm going to define this distance between two policies is just as the sum overall states weighted with my occupation distribution is the same data that we've been using before for the gradient. So, my kubak library divergence between the policy, the prime in s and the policy pie in s. So it's actually it's a linear combination of kubak library divergences. Since these objects, it does are positive. It behaves pretty much as a ordinary kubak library divergence so it's a proper notion of distance is not strictly speaking a distance because you know it's not symmetric. It doesn't obey all the properties for to be defined properly as a distance but it's a divergence so it's a measure of separation between two policies. So, if now let's try and do the same thing that we did before now with this object clearly this this quantity here now is not anymore quadratic. It's more complicated. But the bottom line is that will be that if we use the kubak library divergence we are actually able to do this maximization. Exactly. Okay, so let's see how it works. Remember, always that we have to impose the conditions on the normalization of by prime and the positivity. Okay, so in this case as well this there will be a Lagrangian, which we can write as big right G dot by prime plus pi. Then we have minus the distance, and then we have plus my. Sorry, I have to put this inside. I have to impose normalization for each is a set of large multipliers I also multiply them by it as for reason that will be clear in a second. I have to impose the condition that my policies are normalized to one. Okay. These are large multipliers. And these are the constraints. Again, here I should also take care of the condition of positivity, but as we will see in a second in this case, it will turn out that we can enforce them automatically. It's a property. It's a good property of the kubak library divergences. So let's, let's take the derivatives. Okay, we're going to take the derivative of my Lagrange player with respect to policy by prime. Yes. And now we're going to use this explicit expression of the gradient here. So let's go one step at a time. So we have beta. DG in the pie. This is computed in pi. Okay, so this is when I take the derivative with respect to pi prime of this term, this is what I get. Then I have to take the derivative with respect to pi as of this D pi pi. So let's write it. And then if I take the other derivative I get minus lambda s. And all of this will have to be zero. Okay, now it's time to replace this quantities with the ones that I calculated. So this object here. I know it. So this is going to be beta of s q of SA. That's from the policy gradient theorem. So what about the derivative with respect to pi a s. Okay, let's, let's look at this. Okay, so first thing we have to. We are deriving the respect to the index s in the policy. Okay, so the sum over s goes away. And we are left with minus beta of s. And then we have to take the derivative with respect to pi as of the curve back library divergence between pi prime. Yes, and I'm going very slowly. It's just when I take the derivative I kill all the terms except the ones which have the s, and then I have to take the derivative of the curve back library and then I have minus on the s. So in this step, you see I have artfully introduced all this it as which are positive and also they can get rid of them here. Sorry, not this one. Now, last step I have to take the derivative of the cool back library divergence so let's remember that this is deep in deep by a s. So let's write the cool back library divergence explicitly. And with the sum overall possible actions let's write the a prime award of pi prime a prime s logarithm by prime prime s divided by a prime. I think the derivative is with respect to pi prime. Yes, thank you. Awesome here. Thank you very much. So this is the term that we are considering here. So now this is easy because I have just to derive this object and it says straightforward calculation. I get the logarithm of pi prime a s divided by pi as plus one, and then I can plug this back in. This I can plug back in here. What do I find? I find beta q SA minus log by prime. Yes, yes. Plus one minus lambda s equal to zero. And here is where the nice properties of the cool back library, the dangerous become ever just become fully fully appreciated because this allows me to write down explicitly by prime as a function of pi a s e to the beta. And then I can normalize that's where my large multiplier comes into the game I can normalize this. So before we discuss this result so let's make a few remarks. So first remark is that if pi is positive, as it should be, then also pi prime is positive, and by prime is normalized. So in this case you see that we don't need to make any projection. If you take, if you make one step. If we start with a policy which is within the admissible range, the next one will still be in. There's no risk of going outside the domain. This is a property of the cool back library. First remark. I want to recapitulate remember what I've been doing, we've been doing here is that we've been doing a sort of gradient ascent, but now with a different notion of distance between points between points in our space that is distance between policies. That's what we've been doing here. So as a result, if we start from a policy, we get to a new policy, which is obtained by multiplying each entry of my policy by exponential of beta of the queue function of that policy. Clearly this skews the property distribution towards the entries which have a larger queue. So where my metrics queue is larger there will be more probability. So this makes sense. In this policy, I construct an estimate of what this policy is worth. And then I get to a new policy, which is which has larger entries in the items in the in the actions that give the most. So let's have a look at what is happening in the string cases so clearly if beta equals zero. Then by prime is going to be equal to pi. And it goes to be expected because what does beta mean. Well, if you have a look at the optimization problem here. When beta is zero it means that you, you don't want to, you pay a cost for moving away from the point without getting no compensation for that because it is zero. On the contrary, now when beta goes to infinity something interesting happens here. You get outside the domain, because you're always inside your policy space. So what happens when beta goes to infinity, when beta goes to infinity, your new policy by prime. Look, if beta becomes very large. Here there is many exponentials that are competing with each other. But only the one which has the largest entry will win. And the project here on the right left hand side becomes the arc max over a of q pi s a. In fact, in fact you can think of this function here as a softmax as a so called softmax. It's a way to take the maximum but soft on it. Okay. So this is the greedy strategy. So in this limit better going to infinity. Our gradient ascent with kuba fiber divergence becomes the policy iteration. So in a sense we have forgotten the request that we have to stay close to our initial point, because the weight on the distances back getting very small. The importance of the distance with respect to the gradient is one over beta. So we forget about this notion of distance and then we go full throttle to optimize over the gradient. And we recover the original policy iteration that we have been discussing at the beginning of the lecture today. Okay. If you tune properly your beta, you will be making the right steps in the proper direction. And this is a very effective algorithm for policy improvement, because you don't go you don't visit all the corners so in a sense. Let's let's make a graphical summary of what is happening here. So this is our policy space. Suppose we start with a pie. If we, this is my gradient of G. Then if beta is infinity, I just go the corner here and then corner here and then corner here. So this is what happens when beta is infinity. When beta is zero, I don't move. I take a step somewhere in this direction, and then another in this direction, and then this length the length of the steps is variable. But I'm guaranteed that I will never go outside my policy space. So this case that I discussed here. So this algorithm here. Because you clearly see that you can turn this into an algorithm. Let's write it explicitly. This is the name, and it's called mirror ascent. Why it's mirror, it's a complicated issue but this is the name that comes in optimization theory works as follows. So initialization, your policy by zero, as usual. And then you have to iterate in a loop. And then by K compute your Q function Q K, as we did several times so far today. And then get your new by prime. Sorry, not by prime but by K plus one equals to by K exponential of beta Q K normalized. So what I mean here is a s as a, as a, as a, as a. Okay. And this goes on until, as always, there is, for instance, you can termination. This might be for instance, when you're distance between the new policy and the previous one is smaller than some website. This means that your steps are becoming smaller and smaller because you're close to the optimal. Or there might be other ways, you can also use your Q function as a terminator, whatever. And this is actually an instantiation of a much broader class of algorithms which are this mirror online mirror online online.  Which are of an almost importance in, in convex optimization theory, and find an application in this, in this context. Okay. And I said, more or less what everything I want to say about this. Algorithms in policy space so we did not have time to discuss the problems and answers that arise when you mix this algorithms, either value iteration or policy iteration with the function approximation. This will be the subject of the next lecture, not tomorrow because tomorrow we are going to have a tutorial on value iteration with practical applications. But let me just make up a short graphical summary of what we have achieved so far. We have two possible approaches, basically, for solving an MVP by value iteration. Okay, so graphically this means that you're working in the space of value functions. So this is abstractly the space of value functions this is the value for state one, but it was the two very first state three, and then there may be other many many access that don't make any sense on a on a picture but something like this. And then you start from a certain V note, and you use the bellman operator to send you to a V one, and this will send you to another V two, and all this process eventually converges to be star, which is the solution of the bellman equation be this star equals this star. Tomorrow we will see examples of this procedure of dynamic programming. Second possibility is policy search. Okay, now, the thing happens simultaneously, both in the space of values, and in the space of policies. Okay, so this is symbolic for both. Okay, don't take this literally. So what is the area here, the other is that you start with the policy by zero, and you use this to compute a value V zero. This new value allows you to compute a q zero function, which somehow by either algorithm, for instance by by this algorithm here, your new q will send you to a new by one here. This by one will send you to a new V one. And this V one computer q one which will send you to by two and so on and so forth. And this going back and forth between the policy space and value space will eventually lead you down here at your optimal policy by star, which will correspond to an optimal value V star. So back and forth, I can attend a stable game, eventually, this to this procedure stabilizes between the optimal value on the left and the optimal policy on the right. Both this kind of problems and architectures will be very important when we deal with the modern free problems, that is, we will be able to use the same kind of ideas, even if we don't know what the transition probabilities and rewards are. So that's why I'm insisting a lot on this. Okay. And I think that's a good point to stop. Any question. I suppose you have to digest this. Yes, can I see what you wrote just before the first example. Yes, perfect. Just one second. Again, if you want to play around with these two approaches with the two state to action model, then you can clearly visualize everything because the value space is two dimensional, the policy space is two dimensional. So you can really see these points moving around. And it gives a, I think a very, very neat interpretation, of course, this kind of exercises could be the basis for your final for your evaluation as a sort of exam. Okay, this is just a suggestion that would be many more many more things to come so this is one possible entry for your final assignment. Okay. Now no further question I stop sharing and stop recording.


talk about how many times Trump has lied. I've calculated that I think with your chiron with, you know, I don't know if there's any journalists left at CNN, but I know that, you know, if I were to estimate about 300 different, you know, distortions or misinformation that we get out of CNN, and you have to watch them in the airport, which is harsh. But if you added all that up to 46 months, it comes out to be 300,000 plus distortions of truth.  It's really just one of those things that is dividing our nation. And I don't believe in dividing our nation. It hurts our great nation. And CNN is really the enemy of the truth. And that's my opinion. Willie's next in Katie, Texas. Go ahead. Hello. Good morning. Hey, Mr. Stelter. Just a couple of questions.  And how can they be reliable? Which is the name of your show, Reliable Sources. How can they be reliable for your reader if your book is full of anonymous sources who you can't really track if that information is true? How did CNN spiral down to the absolute level of Trump hate that they are?  Who hates Trump? Who hates Trump?  Who do you think hates him? Oh, my God, that was great. Everything that that man said is accurate. Except for maybe I would wager that the number of lies told by CNN and the rest of the media is way higher than his count of 300,000. I'm not sure what metric he used there, but I'm pretty sure it's a lot bigger number than that. You got to understand that CNN does a lot of lying by omission. And by that, I mean, they leave out key facts and data that are inconvenient to the narrative that they're pushing. They'll have a story and they'll leave out key details of that story so that it doesn't hurt their political agenda. There's lots of very recent examples of this that I don't really want to talk about because if I do, the YouTube Matrix Sentinels will be sent after me and this video will be instantly demonetized. But I think you know what I'm talking about because I talk about it in just about every video. So trust me, if I ever get a chance to speak with Stelter, he's going to get an earful. Okay, so let's dig into Stelter's embolism inducing response to this caller. I'm grateful for the call and I know that you're not the old person that feels this way. There has been a process of radicalization that's happened in this country with media bashing that is absolutely unprecedented. What? I mean, you see what he did there, right?  Right, right, because the only people who would ever criticize the media are radicalized conservatives. What exactly makes them radical anyway?   He is a media basher. Day in and day out, that's what he does. He bashes his ideological and business competition over at Fox News. And he doesn't just bash Fox News, he accuses them of radicalizing and brainwashing half the country. And in fact, the Washington Free Beacon recently did a study that showed the CNN media team covers conservative news outlets over 20 times more than liberal ones. As you can see, the terms right wing, far right, and conservative news or conservative media pops up considerably more than left wing, far left, or liberal news media. And when they do mention left wing, it's typically in the context of criticizing the right. For example, the only mention of far left and Brian Stelter's newsletter was a quote from April 1st on Sean Hannity in which the Fox News host referred to red code co-founder Cara Swisher as a quote, far left media mob maniac. So you see, even when they do use left wing, it's only to attack Fox News and the right. And what about media bashing? Brian Stelter has basically adopted the exact same rhetoric that Barack Obama used in his war on Fox News. That's why it's so ridiculous when Brian Stelter acts like criticism of the media is something that's never been seen before. The Obama White House is firing back, charging that Fox News is different from all other news. That Fox News often operates almost as either the research arm or the communications arm of the Republican party. Media is operating basically as a talk radio format. Then that's one thing, and if it's operating as a news outlet, then that's another. And the White House has gone beyond words. Last September 20th, the president went on every Sunday news show except Chris Wallace's show on Fox. And on Thursday, the Treasury Department tried to exclude Fox News from pool coverage of interviews with one of its key officials. Look, the media is not above criticism. They do plenty to earn it. And trust in the media has been plummeting for the last 20 years. It doesn't have anything to do with Trump. In fact, I'd argue that Trump is president because of it. I challenge anyone to get on any of the major network news sites or any of the cable news websites and find a single negative story about Joe Biden. Instead, what you're gonna find is lots of stories defending Joe against attacks and criticism. Is that the job of the free press to act as campaign PR for one of the parties? Listen, when Brian Stelter calls you a radical for questioning the media's honesty, it means you're over the right target. 20, 30 years ago, conservatives talked about media bias, and there were some really valid points to that critique. It is absolutely true that lots of journalists are based in New York and Washington. They're based in big cities. They have liberal leadings. And mainstream newsrooms are built to make sure that that bias doesn't seep into the news coverage. And yet, sometimes it does. I absolutely acknowledge that. I think it's different to talk about things as if they're enemies. No American is an enemy of another American. No news outlet is an enemy of America. Wrong, wrong, wrong. Some of us still remember when the Democrats in the media demonized the Tea Party protests and really any opposition to Barack Obama in the worst possible terms. Even comparing them at times to domestic terrorists and suicide bombers. These people are crazy. There's eight to 15 to perhaps 20 members of the Tea Party influenced wing of the Republican Party in the house who are crazy. They are crazy. Yeah, these are people that don't believe in evolution of global warming. So why are they going to believe that default would hurt anything? I mean, you're not dealing with like rational people. Come on. They've strapped explosives to the Capitol and they think they are immune from it. These Tea Party guys are like strapped with dynamite standing in the middle of Times Square at rush hour and saying either you do it my way or we're going to blow you up ourselves up in the whole country up with us. Or how about Democrats telling the American people that Republicans want them to die? Come on now, Brian. You guys have been painting your political opposition as the enemy for a very long time. It's one of the things that motivated me to do what I do today.  Trump's odd behavior with Vladimir Putin is compelling so many people to ask, what does Putin have on Trump? Has Trump been compromised? Trump might have committed treason. Then on top of that, you paint Fox News as a propaganda machine that's radicalizing conservatives. I mean, how much more can you demonize us? Now, what I find real interesting is that Stelzer acknowledges that the media is filled with left-wingers and Democrats and can be bias, but somehow criticizing that is radical. They say that the best lies have an element of truth to them, and I think that's what we're seeing here with Stelzer's attempts to deny. Alright, that's all I can take of Stelzer for right now, but if you're on Twitter, I implore you to tweet this video to him. I'd love to get his response to some questions. If you enjoyed this video, please hit that like button, share and subscribe. I upload content pretty regularly, and if you would like to support this channel, you can do so by using one of the platforms that are listed in the description or the pinned comment. Thanks for watching. Keep coming back.