with a staff member and they work quite closely with the students that are in that group to ensure that the academic, the personal, the social, the mental well-being, if you will, are on the road to success and achieving the requirements as they move through the program. Each year they rotate to ensure that that tracking is kept in place. When there are any kinds of issues or concerns or challenges that are brought up, typically we red-flight the student and we try and work on a plan together so that we can achieve a level of success with them without leaving them behind or having them withdraw. We go out of our way to provide all kinds of supports for them, academic supports with tutoring for example, personal supports for referrals perhaps to another space on campus or off campus. Any kind of supports tied to their own well-being from their home life if you will or at community level we try and engage in assisting and guiding them through those processes because at the end of the day we want to ensure that we are reaching out the best way we can for them to do well and so by doing so we track. We have a system a couple of times during the year that I get a chance to look at to see what our numbers are like the successes and the retention if you will. Our attrition is very low and of course it happens typically in the first year. Those supports are really critical in terms of having them in place so that they can successfully achieve the first year and then continue to build and move on from there. Ultimately the retention that we have is very high because when students are done our program they feel proud and they feel good about who they are coming through the program and then getting into the classroom and sharing what they've learned and being a model if you will in the community and in the school for potential future candidates that will come into the program as well so we've got a good host of that between human resources and paper resources that work together in a good way. What are some of the services that you provide within the incentive program? Well one of the things that we do obviously is first have our students be part of our community. That's important is setting the foundations of being part of the community feeling comfortable with who they are knowing that they come from all different walks of life getting to engage in building a relationship with them that's foundational. After that relationship is built then we get to know the students a lot more closely and work with them more closely in a way that we begin to see the kind of supports that they might need so for example if we have a student that has a learning disability they don't find up to them well into the program well we're getting a hold of the Student Success Center to try and engage in the process of assisting how to deal with their disability if they never had an opportunity to go through that before. Another opportunity that we see success in engaging our students is various fundraisings throughout the year most recently student initiative with the verdict around the Colton Boucher case the GoFundMe page our students have taken it upon themselves to create t-shirts to get that awareness out there about how important social justice is and we promote and we support and we guide them through those processes they they look for us for support and in the hopes that we will engage in whatever opportunity that presents itself that we'll go forward with them in a good way in a very humbling way in a respectful way we encourage them to be a part of the university community in any way we know how whether it's through the Aboriginal Student Center engaging in UR talks having them present something that they may have learned for example we had a student last semester present on hunting and tracking and teaching his experiences around tracking when he's out hunting showing different different examples of what that looks like when you're out on the land so we take that into consideration we built into our classes the importance of land-based learning and community learning as well too right down to at the beginning of every year we take our entire student body down to the librette community and we spend the entire day out there and we look at different ways of looking at identity and culture and what that community has to offer family ties the history behind it so we don't have necessarily all on campus academic or personal related we extended out to the community for support this fall too and we find that students really get off on a good note when they're moving forward in their studies and we literally support everything that we that they do well can you tell me a little bit about how how much students are would be enrolled per year okay how much positions I guess we have a cap of 60 students that we can't go any higher with so we're small but mighty program I always tell everyone each year on average we would have anywhere between 15 to 18 in each year now typically what happens is we not only have the small numbers because I think that's the mandate of our success but what I really like to like seeing is how our students when they begin in the first year they travel together through every year of the program so they're the cohort model and they rely on one another for support and they get through first year they move to second year and so on until they're completely done now there are times of course when life dictates school and they have to leave early but they never leave us completely they end up coming back so currently we have 55 students out of 60 and that's a nice number for us we're very happy with that it can go up it can go down but currently we're feeling really good about where and how the students started each year in the program this year for graduation we'll be graduating 12 so we're looking forward to the graduation celebration as well two 10 female and two male and at this point I already have three of them that have job offers so we know that we're continuing to move in the direction of what our mandate is in terms of um of the of indigenous education on um on a larger scale what makes sun tep and the graveyard juban institute so important here you you've talked a little bit about here in Saskatchewan but important in indigenous education in general okay when you talk about indigenous education in general I'd like to thank the model of the work that we do here by having an indigenous education program can be expanded in any opportunity across this country in terms of the work that we do now what I what I mean by that is let's say something like land based learning the model that we follow meets the needs of our students in our program tied to their identity so we go out on the land and we go camping and we take them to various historical monuments or locations that the Métis presence was very important during that particular time and I'm not saying that it's always stuck in history those were the foundational teachings and from there moving forward to where we are today for let's say the history of what goes on for example in Winnipeg some of the Métis communities and places noteworthy in Winnipeg but then we also have a contemporary version of the Museum of Human Rights that the Métis voice in that museum as well too that continues to expand on the identity piece so I'd like to think that indigenous education what we do not only is it based on on what we see and perceive the importance of it being but it can be expanded in any type of model through the lens of an example of land based learning now where it becomes very unique and distinct for us is we offer a Métis language class Mitchif 100 the traditional language of the Métis people that's a very important language and it's also a dying language like most indigenous languages are now making that part of our program as a required language piece gives students that additional sense of identity and place of belonging of who they are and where they come from and we are very fortunate to have two staff members that can co-teach it together and have a good foundational history of one of them that being their first language so across Canada for a language class like that that has big implications there is no other post-secondary institution in this country that offers a Mitchif language class we are the only ones in this country that's a big achievement for us and we are very grateful to the faculty of education for the support in getting it through to make it part don't get me wrong we can send our students to a Cree class or a Cota class or a Sota class because they need to have that language component as part of their program but what a better way than learning about who they are and embracing and taking pride in their own identity by offering a language class that we were managed that we now have managed to do and we take a lot of pride in that so that on a larger scale I think has plenty of implications and just the basic things that we do we present at conferences guest speakers and events everything we do in indigenous education revolves around that piece of being maintained and the identity and the importance of culture and language that go along with it so there's all kinds of opportunities that are just some examples of what we do for sure what would be what would be your advice to give to other institutes or potential communities across the country in regards to increasing recognition and acknowledgement through programs education programs like this for the Métis populations well the first thing that comes to my mind and this is no disrespect but for a very very long time there's been a focus on the education indigenous education from a First Nations perspective unfortunately too often Métis education has been a bit of an add-on if you will an afterthought or after the fact it needs to be on equal footing the biggest advice that I can think of is how important that is for us you've got treaty education in the classrooms from K to 12 and it's mandated where Métis education where is it mandated it doesn't exist there's a bit of a disconnect there we need to ensure that the quality is there for all indigenous peoples and I can even extend it further to talk about the Inuit as well too we may be in the south here but that doesn't mean that we're not learning about the Inuit people either for that matter if you want to treat people fairly from an indigenous perspective First Nations Métis and Inuit has to be part of that conversation or part of that mix because even though the history of First Nations people has been thoroughly researched documented discussed if you will so has equally Métis education in addition to Inuit education but that lack of footing we want to make sure that we don't get all caught up in having it all speak for everybody if you will it's not all First Nations it's very distinctly key education very unique very much meets the needs of the community and those that were engaged in making some place like Gabriel de Mines to exist and continue exist and build for that matter that's my biggest message is making sure that we're all on the same footing and not forgetting because at one time we were closely tied and had strong relationships and kinships that somehow over time has gone non-existent if you will and it's reclaiming that back in terms of how important that is and that's the big thing that I think that would strike me as something that needs to be attended to in a good way and you're what do you say that's your future your future vision for indigenous education across the country then it would be the flip side of that obviously Heather knowing that the the equal footing of Métis education is out there in a way that showcases and highlights all the successes and the achievements the the great stories the history the language the land learning that goes on how it all comes together like everybody else whether it's in a way that that is through the oral tradition if you will or in the written format for that matter there are many old ones that we work with that share that knowledge with us so that we can in turn facilitate it to the students and enhance the program so that we're all we all come from the same place in that regard there's all that tie there that it's important that we maintain that as best as we can and continue to bridge it and move it forward we're not an afterthought for sure is there anything else that you'd like to add in our regards to the SunTep program some of the things that you have to offer currently or even how one can get involved in learning more about the SunTep program here with the Gabriel Duvall Institute sure Gabriel Duvall Institute because of the time we live in has a very active website if people are not familiar with who we are the website showcases all the various entities of the umbrella organization Gabriel Duvall Institute we are only one piece of the entire organization there is Dumont Technical Institute which is adult learning and education but also skills and traits so let's say you want to go get your safety tickets you want to do truck driver training you want to do office education that's Dumont Technical Institute and we're still under the same organization there's also training and employment training and employment helps individuals that are coming in that are underemployed and under-educated and need supports to get out there and get the resumes out there or need training or need work placements or what have you there's that piece as well too we also have libraries and the libraries which we house not too far from where we're sitting right now has a collection about half and half indigenous and non-indigenous in their collection and we have a very close relationship with library people because our students spend a lot of time in there and they support us by bringing in community people and hosting various events that we participate in that's really important we support one another that way and in Saskatoon we also have a publishing department so we publish a whole host of resources and these resources we use as part of our program it's called the virtual museum to get additional resources and there is a beautiful space in Saskatoon in the publishing department that showcases and highlights the importance of native education and how it can be achieved through different learning opportunities in their space and they even have a shopping site where you can purchase resources clothing kitchen things like just a whole host of things like everything that you possibly think of this is really important and it's updated annually we also have a piece tied to this in publishing with the language development that that in itself is something that we take a lot of pride in because revitalizing and reclaiming language is a critical piece in in the work that we do so there's that the other case that we have is we have scholarships for our students whether you're an adult education learner or a son tech learner or a skills and trades learner there's a whole host of scholarships and awards that are out there for our students that are part of this and they engage in getting a lot of support through our annual they are annual awards through the various community partners that we have that have set up a whole host of them last fall we have the release of our report is another item I want to draw your attention to gives you the whole history both with statistics background information and the importance of the program that we do since our program has been around since 1980 we've graduated 1238 grads almost a thousand of them are teachers in Saskatchewan schools that's pretty impressive that's really all over the province whether you're in Regina Saskatoon or Prince Albert and the others are in different capacities in education right so the report that was put together by Dr. Howell at the University of Saskatchewan highlights the importance of our program and the work that we do and why our program exists and it really is an investment not just from the past to today but also for the future as well too so this is another piece so we've got a whole range our applicants can also apply online when they're interested in learning about us we have all the information online I've shown you a hard copy we have our own handbook like we're very autonomous even though we have to follow guidelines and expectations and regulations of the University of Regina we're also very autonomous as well too so it works for us in you know both in the short term and in the long term perfect perfect is there anything else that you'd like to add I would like to highlight and showcase um the the students that we work with come from very high caliber places in terms of their academic learning they pretty much garner a good host of awards anywhere from in one semester because I received the lists of awards they could be anywhere from I don't know 20,000 to 100,000 even though we're a small population the academic achievement of the work that our students do is recognized locally nationally and regionally for that matter in a good way when I first came here six years ago one of our senior students who was getting ready to graduate was nominated and received the very first president's medal award for this program that's a big achievement walking out here with a 90 percent plus average and engaged in leadership and community involvement to be recognized at the institution for those achievements coming through our program and being awarded the president's medal it doesn't get any higher than that for the first time in a very you know since the inception of this program it's a big achievement for us knowing that we have the caliber of students I can't stop thinking about the students and and commend them for the commitment of the work that they do and the academic accolades that they get in various opportunities and I just don't mean monetary heather I think they get opportunities to present at conferences they get an opportunity to showcase the work that they do in classrooms or in schools with other institutions that are surrounding within the city of Regina we've got an event coming up at the end of the month where SAS Polytech instructors are coming to learn about Metis education here and the instructor is facilitating but the students are running the show for lack of warning those types of things are what continues to drive us and keep us going in a good way knowing that they're doing really well and the work that we do with them continues so I think that's important


Okay, shall we start? So today we are going to explore as much as we can in one hour and a half. What we can do when we are up there, so when we have perfect observations about the environment and we know exactly what the laws that govern the system are. Okay, these laws might well be stochastic, right? So it need not be deterministic, but we know the in the mark of chain language, we know the transition probability for this. But before getting into this, which would require a little bit of calculations and we would in the end write a mean result and basically summarize in one hour and a half the work of many many people over more than a decade around the 50s of the last century. Let's go back to examples first in order to get an idea of in a very simple case of where we are in this diagram what are the things we're talking about. So I redraw the two main concepts that I want to use to orient ourselves in the problem of reinforcement learning in general. So like I said yesterday, there is an agent. The agent continuously interacts with the environment, gets as per seps a couple of inputs. One of this is the reward, the immediate reward for some action that has been taken while being in some environmental state ST. The environmental state need not be accessible in general to the agent, but it might get some observation out of it, which hopefully conveys some information about the environment itself. And on the basis of this it just builds some policy which is a way of mapping actions, mapping sorry. The history of all experiences gathered up to the time into some action. Okay, and the goal is to find the action that maximizes some discounted form of the reward. I will write this again for you. So the goal is to maximize the policies the expected value of a sum. Okay, let's say we start conventionally at time t equals 0. Then this will be the sum over all the future of gamma t times the reward at the subsequent time rt plus 1. Okay. Where gamma is this discount factor which expresses how far in the future you want to look. So this is based at time t equals 0. If the environment is changing, of course, this might carry on another formula which I wrote yesterday, but that's just intended as a reminder here. Okay, so and then we were discussing this example, which is a very simple example in which there's just one state. Now, since there is just one state, there's little to be observed here, right? So we are actually moving up above here, but still there's all these axes which has to be explored. And we will see what these things mean in any case. But first, let's identify simply in this diagram what are the concepts that are written here. So the agent is the agent here which decides what to do. This is the state of the environment which is always the same. So the environment evolves just like state, given state, next time the same state, next time again, again the same state with probability 1. Nothing happens.  Then what the agent does is the opportunity of choosing between two actions and you can select one of these two actions with a given probability, which would be pi 1 or pi 0 in this case. So this probability distribution and of course, since the agent has to do something, they must sum up to 1. So they're just one independent number here running from 0 to 1. It chooses action 0. If it chooses that, then two things may happen. So there's some random process decided by the environment on which the agent has no control, which will give either an outcome of 1 as immediate reward with probability p 0 and an outcome of 1 minus p 0 with probability minus 1. Otherwise, if the agent selects action 1, the same thing will happen with different probabilities p 1. This is just one simple example of the decision process. So you might think, and we're going to take a poll about this, is this a mathematical oddity or does it have to do with any, with some particular decision process which is relevant to us humans? So who thinks that this stuff is just simple mathematical example? I mean, both statements are true. But who thinks that this is prevalently a mathematical toy or new thing? So no, this thing is very interesting because I have an example in mind. So who thinks that this is a mathematical toy? Who thinks that this is an example of decision making for real? One among you, what kind of thing do you have in mind? That's very abstract. See where the bus, fine, good enough, that's a nice example. Okay, that's okay. There's a little bit of other complication in this example is that you have to gauge the value for time that you're waiting, which is not immediately mapped into this. Just one more example, lottery is the example, very good. So we will go through that example because that's the example in which this kind of problems actually were born. So for this example, I will need a volunteer. Please come around the desk. So I took two coins here from my pocket and I'm putting here one here and one here. Right, you are the agent. You have to flip one of these coins.  And then if it's head, you will get one euro. Yeah, sure. Just not for me. And if it gets tail, you will have to give me one euro. That's for you. Speak up. Question? No, let's postpone, just for a second question. So do you recognize that this is the same thing? Do you agree? And it's the, okay? He completely knows the state. He says, he completely knows the state. Well, the state, of course he knows. There's no state to be known here. Ah, you mean he knows the model. He knows what probabilities are to be flipped. Do you know this? What are the probabilities? Half. Why? Why? This is even more surprising than this. Are the coins fair? You cannot tell. I just put two coins out of my pocket. You're just assuming a lot of things about me. You don't know me. You will get to know me soon. Okay? Nobody knows. You can make an assumption. You can say, okay, I think that the coins are, okay, in that case, what would be your decision? Yeah, exactly. Yeah, which one do you want to flip? This one. Why? Pardon? I have no preference. You have no preference. Yeah, right. Because if you have an expectation that these two probabilities would be the same, there's no point in choosing one or the other. But that's an assumption about how the word goes. Another thing is, if I told you, so this coin is biased because all my coins are biased. And this is 60% in favor of being, say, tail. And this one is 60% in favor of being head. Of being, yeah, head. Which one would you flip? This one. Given that information, you would choose that one. Okay? Does it make sense? Yeah. Now I recall you that not later than yesterday, at some time, some people propose that you have to choose proportionally to the probabilities. So does it make sense? Should I flip 60% this one and 40% the other? Okay? So that answers the question. And we will see how it comes from. So is it a general lesson that given the full knowledge of the environment, so given this P note and P1, the best strategy is always deterministic. Yes, I hear a resounding yes. So who thinks, yes, raise your hand. Given that you know the probabilities. Good. Who thinks the opposite? Okay, very good. Enough for spending half an hour on proving that. So, okay? If we don't have that, for these kind of problems, unless there is some degeneracy, that is, you get the same thing, in which case that's indifferent. But if there's, this degeneracy is lifted and there is actually is a way of getting the best result out of this, then it will be obtained by deterministic policy. Or at least it can be always obtained by the deterministic policy. Okay? That's true when you know everything. So when I tell you, this is 60%, this is 40%, we are staying. Thank you very much.  We are here, right? We said we are at the top here because there's nothing to be observed. And we are on the far right because we know the laws that govern our world. And the laws that govern our world here is the nature of the coins. How much they are, how much they are biased. Is it clear? Okay, then let's remove this kind of gracious information that I gave. And we are back one step and say, at the beginning, you don't know. You don't know.  Now you know something. You know that this is nonetheless the Bernoulli process. Because the outcomes can only be head or tail with a certain probability. So you might not know the values of the, sorry, the values of these probabilities, but you know that the structure of the process. You know that it will never come to, if I flip it, it will never be dragon. Okay? It can be head or tail. Okay? So you know something about the structure of the model. And you have to guess what the parameters of the model are. And if you get information about the parameters and these you do by experience, you can improve your way of making decisions. This is what happens when you are in the middle. You know something about the environment. That is, these are coins. They are not expected to fall exactly in this way. I will never be able to do this, of course. You don't expect any time to happen, this to happen, right? It might, in principle. But you are attributing zero probability to this event. So in which case the process is Bernoulli and it's just an assumption. And then you have to understand what the probabilities are, right? So you are here in the middle. And then there is, if you move further, further here on the right, there is the extreme. The extreme situation is the one in which, for instance, I give you the coins. You don't know anything about it. You flip the coin, you get some result. Then I put the coins again in my pocket. I pull out one pair of coins. Might be the same one. Maybe not. This is the case of a rigged lottery. Rigged. You know what rigged means? It's cheating. One way cheat, okay? I might pull out, if you win, I might pull out a different kind of coins and then adjust to your decision to make this against your will. This will be, so technically it's called an adversarial lottery. In this case, you know very little about the environment because the environment is changing as you do things against you. Can you still win at this game? Are there algorithms that do that? You can try. Yeah, exactly. That's one way. But you see the complication with that, right? So increasingly, if you move from this side and this, this, I hope it's clear for you. If I give you the probability, it's a matter of just computing. Right? You have to use a paper or intuition in this very simple case and then you will make a decision and the decision is correct. But there's no learning in that. What did you learn? You could have guessed this without flipping any coin. I told you, this is 60%, this is 40%. Nobody flips any coin. You know already what to do. There's no learning in that. When you know that it's Bernoulli, there has to be some learning because you don't know in advance what the probability will be. You might say, okay, I think that he looks like an honest person, so perhaps it's 50-50. Bad guess. Bad guess, but for you. But experience will show you that I'm not that honest and the coins are biased in some way. And then you go to the other case where you don't, not only you have to learn the probability with which these events comes, but the process which generates this ever-changing probability, which is an even more complicated task, and requires a lot of learning. Okay? You know something? Exactly. You know something? You still know that it's Bernoulli, but this probability would be changing every time, depending on the past history as well. Okay? So again, you know something. There's an even worse case in which, again, it could have, anything could happen. The coin could disappear while in flight. It could end on a edge. It could show dragons. If it's a, if this lottery is dematerialized and then it's just the machine that you find in a bar in which you press a button, well, you put a lot of confidence on that that it will always show you the same. Results, right? But you don't know. You don't know. Okay? So this is far on that side. Okay. So these kind of decision problems are actually extremely famous. And, and, and besides this very simple example that I gave you, which is the one a lot, are actually very, very serious and interesting applications. I would just mention a couple of them. Okay? So one of them is, you should be familiar with that. It's advertisement on, on internet. So in these days, if you click on a Wikipedia webpage, you will get a banner which says, please donate. But doesn't say always the same, doesn't give always the same text. Okay? There are several of these different ads with different texts. So this is exactly the same process. Suppose that there are just two ads. Now the agent is Mr. Wiles. Is that his name? No, I don't remember. Well, Mr. Wikipedia. Okay? He is the agent. And you have, the agent chooses whether to send out to you who are the environment now. You are clicking on the Wikipedia page.  You are sent banner zero or banner one. And then with certain probability, you will donate usually very, very small, okay? Otherwise it just, it will return zero reward for the agent to Wikipedia, right? Then what, there are algorithms that take very seriously this kind of process and decide online which kind of banners to propose in order to get the most out of you. So the one which is more efficient is discovered by trial and error. So you don't know in advance how much a banner will be effective. Okay? So it's the case like you don't know what the probabilities of the coins are. But you will repeat this game and you will rent out to do it. And of course it's small amounts multiplied by millions if not billions of clicks, which make out a substantial amount of money. You have to discover and click many times on Wikipedia page and notice that, okay? There are results about this, about past campaigns. And I have figures for that if you're interested. So typically the amount of money that you get for every click is minus Q. It's, I don't know, 100 percent of a dollar. It's software because there's not an immediate reward in that case. It's a little bit more complicated to map. So it's unclear how much YouTube gets advantage directly from that. But yeah, you could assimilate more generally to this, okay? This is a more direct example. The second example which is even more serious is how do you make clinical trials? Clinical trials. So you have some treatment for some illness and you have to check whether it's working or not. And you have to do this by comparing the treatment with the placebo. The placebo is something that looks like but doesn't do anything, right? So in that case, you have two options, placebo or treatment. And the outcome will be how effective it is on the patient. And you have repeated trials to do. And patient come in with time, okay? Sometimes you have a cohort or large cohort available, but for some rare disease, you just have one patient coming and then another one in three weeks and then another one in five months. So and as they come, you have to learn how effective it is. Because you don't want to wait 10 years before discovering that that treatment was actually extremely good or maybe had a negative outcome. So as this data come in, and this is again the same process. Now all these kind of process actually go under one name and the name is comes again from the lottery jargon. These are called multi-armed bandits. What does that mean? So this is the name of this kind of process in which there is one state and many actions to be taken. There is an obvious generalization of this with zero, one, two, capital A minus one actions. And then the rewards might be Bernoulli or whatever. Gautions, okay, it doesn't matter. All this class of process go under the name of multi-armed bandits. They are part of a class, bigger class, which is called sequential allocation problems, which is even wider and it's an enormous field of research for operations research. So for decision making in factories, in logistic, it's a huge field of mathematics. So what does that mean? What does this have to do with this simple model? So who knows what a one-armed bandit is? Have you ever heard about a one-armed bandit? No, sorry? A slot machine. Why is that? Yeah, slot machines have one arm and you have to pull the single arm, right? These are old slot machines, mechanical ones. You pull the arm and then if you were lucky, coins would pour into your basket. Why a bandit? It gets away with your money, yeah, right? That's the reason. Multi-armed bandits are just bandits with several arms, right? So this is a two-armed bandit. You can pull arm zero or arm one and then with a certain probability, you will get a coin or just, suppose you put, yeah, you get one euro or you lose the euro that you put. Whatever, okay? You can clearly see the analogy, fine? So these problems in their simplicity are extremely interesting. Still the subject of open research. Many counter-intuitive results on this. This will be our workforce number one. So we will come back to this problem repeatedly. Today we will treat it in this corner. Then we will discuss what happens here and then we will discuss over there. It will accompany us because it's very intuitive, it's simple, but it's very rich and not trivial at all in practice, okay? So before getting back again to this problem, which we'll discuss in greater detail, I would like to give you a few other examples of decision-making processes in which the exercise is to understand what the things are here. So what is the environment? What is the agent? Okay, what are the actions to be taken? So I don't have much time to do all the examples, so I will just focus on a couple of them. One is taken from, actually from physics. So this is example number one. Example number two is called the cart pole. So I'm discussing briefly qualitatively this example because this will be the one over which you will be able to train your deep Q-learning algorithm on next Tuesday and Wednesday in order to understand what is the best policy, okay? You will be running basically what is AlphaGo doing on a super trivial mechanical task like this one I'm describing, okay? This is just to avoid the complication of what you will discover. So what's the cart pole? The cart pole is a very simple task. It's, there is a cart and on top of this cart attached to a hinge, there is a pole, a stick. So the task is to keep the pole within a certain angle. Why is that difficult? It's only hinged here, right? There's a hinge. There's just a hinge at the bottom. So why is it difficult? It's unstable, right? It doesn't stay like this. It will fall down on one way or the other. So how can you keep it in place? Suppose it's falling this way. What can you do to keep it in place? You can move the cart and recover, right? So it's a balancing act. It's what you would do intuitively. This is pretty stable because water is on the bottom. But if it goes this way, I will try to compensate, right? This kind of thing. So it's possible to cast this cart pole problem in this framework. So what's the state of the system for this thing? What is the state of the system? Good. This is one degree of freedom. So the state is described by, let's say, the angle theta that the pole makes with respect to the vertical position and velocity of the cart. Position is enough. Someone proposes angular speed, right? Okay. So I would settle for this within the Newtonian framework, right? So it's just a system with two degrees of freedom, angle and position of the cart and the associated momenta, which are the angular momentum and the linear momentum. Of course, it's not this one. Of course it isn't. It's a situation where there are laws of motion. So we are not here at the top. I mean, I'm including it. Okay. That's a good question. That's a good question because this poses the question, what are the actions that you can do? So the point is making is right. If I could act directly on the pole, okay, if I could just decide what to do with this, then I would have just to control the angle. Okay. That is that what you have in mind, right? So I didn't specify that you can't do that. But what you can do is just act on this variables. I would say that in order to describe what will happen in the future, you would need position and momentum and linear momentum, momentum velocity, horizontal velocity. Why not position? Okay. Let's say, let's settle for something we all agree on. So is it okay for you to describe this mechanical system with these degrees of freedom? So there might be some situation. I will get to this comment in a second. There might be some situation where you might have a more compact description of the system, which might apply. Okay. I will ask directly to your question about position in a second. Then the actions that you can take, like I said, is the force, is a linear force which acts only on positions, okay? So the actions are force on x. So your dynamics will be x dot equals v, say n is equal to 1, and v dot equals f. And then there is the reaction of the pole on the cart. And this will be the equation for the cart, okay? You will see all this in detail in the example, right? So this is the kind of actions that you can take. Now, what's left still to do? So this is the state of the arm. You have to define the percept. So rewards. So rewards can be, for instance, you get a reward every time that you are within that angle. And when you get outside of that angle, you get some penalty, for instance. In addition to this, you might want to put boundaries on this. So every time that you move, when you hit on boundaries, you stop the system, for instance, right? And then you get a big penalty for that. In this case, position obviously arises. You see, there are many ways of shaping this reward function for this, okay? So what are observations in this case? What? You look at your system. You might be able to measure position, angular velocity, linear velocity. All these things, you might be able to measure them with arbitrary precision for all practical purposes. Or it might be a very coarse description. Or you might say, okay, I want to control this system, but I only look at the position of the angle. I don't care where the cart is. This might be good at some point, but you will hit on the boundary, for instance. So it's not, it's a very incomplete information for that part of the problem, which made us say, I have to keep the cart within a certain range of distance. You see? The only exercise I want to do now is just that you feel at ease with this description and say, okay, it includes all this. I think it makes sense. I can work out some specific instances, in one case or the other. Of course, if you change slightly the definition, the task will be slightly different. But what you would expect in practice, for this system to do, is just to stay very close to the vertical position. The percept is just a couple of the rewards that you take. The one that I was discussing. You get a positive reward if you are within that angle. And you get a negative one if you're out. You get a negative reward if you bump into the boundaries. And the percept is the outcome of your measurement of all this state, which might be coarse-grained observable of this. Say this plus Gaussian errors at every time. This might be the observables. Or it might be, as well, the state itself. Perfect observation. If the observation is perfect, yt coincides with st. You know everything about the system. Good. Any questions? Third example, it's called grid world. So this is a very simple example of a navigation problem. So there is some notion of space. Say a square grid, square two-dimensional grid. Like, can you see? Can you see? Yeah. So I'm drawing quite simple. It's a grid like this. Then there are some points on the grid which are sort of cancelled out. There is some point in the grid which is a starting point. Grid world, world, world. There is some starting point and there is some arrival point T. Capital T. So the agent is a worker which can jump on agents and sides of these lattice. It can make a move from here to here, or from here to here, or from here to here. So what is the state of the system in this case? Is the position of this agent on the chess board, if you wish, on this board. So if the agent is here, this is the position of the agent. So capital A is very bad because it looks like an action, but this is the position of the agent. So if I say this is ST, it means that the agent is in position two, four on my board. This is the state of the agent, which is also the state of the environment. It's the position of the agent relative to the environment. Then what are the actions? So the actions might be north, west, south, east. That is, the agent decides that it wants to go in up, left, down, or right, whatever you prefer. So there might be four actions in this system. Then the outcome of these actions might be random. So the agent might, for instance, say, I want to go north if there's a loud space. Suppose the agent is here and says, let's put it here. It's here, it wants to go south. It can, but it goes there just with the probability, say, one minus epsilon. And with probability epsilon over three, it chooses to go in the other direction. So there is an error in the actuation of the policy. If an action is taken, sometimes there's an error and this little robot cannot make a step in the direction he wished, and maybe it goes this way or this way or back. Or it stays in place, whatever. So this is how the environment changes as a result of the action. And this probability, this transition probability from one position to another is our model of the environment. These are the rules, the laws, which are obeyed by the environment under some action. Is that clear? In this case, what are the rewards? For instance, it might be that the rewards are no reward unless you sit on the state t. In which case, you get a positive reward. So the rewards depend on the state on which you are. Observations, well, the agent might know very well where it is. Suppose that it has some sort of GPS system that tells you, okay, I'm in position two four, or the agent might not know where it is in absolute space. So this is changing the quality of the observation. If he knows it very well, it will be up here. If he doesn't know it, it will be down there. So what is the final task? The final task with the reward that I just described is to get there as fast as possible and then stay there forever. This is the best thing you can do. So eventually in a problem like this, if you train well your algorithm, it will be able to find the shortest path, the shortest admissible path to reach the target and stay there. May know, may know in which case you're up here or may not know. It might discover it. So that's how the algorithm could work, right?  You will be already able to do this or you're in that case in the upper corner, you know everything. You know this, you know all the probabilities of jumping, given the actions that you decide. If you've given everything, again, it's a computational problem. Much more difficult than this one, of course, but still the goal is to understand how to compute these things. I hope this sort of covers many, many examples of this, of course. Actually, it's more difficult to find situations which you cannot really cast in this framework. The environment is the grid and the state is the position of the robot with respect to the grid. It doesn't know. It doesn't know. You're here below. The environment is still the same. It's just the observations that you get, the percepts that are very, very bad, but you will have to decide based on the percepts, not on the state because you don't know it. So that's why the problem is more difficult.  Good. So let's get back to this simple problem. And we just have to do some very simple. So we had this intuition with the coins that the best action to take actually is the one with the largest p. If p1 is larger than p0, then take action 1 always, then vice versa. So this example is so simple that you can actually work out explicitly the calculations. So that's what we will do. It just take a couple of lines. So we are going to evaluate this thing for this simple model. It's just a couple of lines. So let's start with the first step, t equals 0, first step. First step, gamma to the power 0, no matter what my discount factor is. Okay. It would be 1, the pre-factor. And then I have to take the expectation of my reward. So what do I expect in this case? I said with probability pi note, I will take action 0. So the first term in that sum, let me call this capital G for a second just for compactness. So with capital G, okay, first step, I take action 0 with probability pi note, which is my policy. And this is one thing I want to know how much this thing depends on the policy. So this is a function of the policy. And I want to optimize over the policy. So pi note, then what can happen? It can happen that with probability p note, I get plus 1.  Otherwise, I pick action pi 1. And with probability p1, I get 1. Plus 1 minus p1, I get minus 1. And this was the contribution from the first step. And then I'm back here again. So I am locked into this cycle where everything gets back again to the same starting point. Next step, well, first of all, now I have to discount. So time has elapsed. So the reward, the gain that I will get from this new round of action will be discounted, will be smaller by the gamma. But apart from that, it will repeat itself. Everything will be the same. I'm fresh on you, and I do another round. So this will be exactly the same. And I'm rewriting this, this minus this, it makes 2 pi note minus 1. And then I have this, but this pi 1 is nothing but 1 minus pi note. And this is again, sorry, what am I writing? This is pi note, 1 minus pi note, yes, 2p1 minus 1. And so on and so forth. Plus gamma squared, the same thing. Are you okay with this simple calculation? Yeah, pi note is the probability of picking action 0. And pi 1 is the probability of picking action 1. And they have to sum up to 1, which is what I used here. All of this. There's a whole single Markov process, which depends on probability distributions for these events to occur, and for the probability of actions to be taken. So every time there's a policy which I decide a probability distribution, I flip a coin, or I throw a random variable, and I decide, am I going to pick action 0 or 1 with probability pi 0 or 1 minus pi 0? And then the environment X randomly returns me plus 1 or minus 1 with probability pi 0 or 1 minus pi 0, and so on and so forth on the pool process. Okay, so all these terms will be the same. This is just, this happens just because you always end up in the same state. And the only thing that changes is this discounting for future. So when you sum up all these terms, it's 1 plus gamma plus gamma squared. It's a geometric sum. So that eventually your final result is that, let me, okay, I'll write it here. Your final result is that how much you can gain on average, given a certain choice for the policy, so given a certain probability distribution, is 1 over 1 minus gamma times. Now I'm again 2p1 minus 1 plus pi 0 2 pi 0 minus p1. So this is again the same thing. I just rearranged the terms here. You can check that that's exactly this. This clearly depends only on pi 0 because pi 1 is locked to be 1 minus pi 0. And now you can ask the question, what is the best policy? I have to find the maximum of this quantity. Overall, pi nodes that clearly, since these are probabilities are confined between 0 and 1. So what, where is the maximum? Where is the maximum? You make the derivative. Yes, it's a linear function. You can also go through for that, right, without making too much analysis. Where's the maximum? So this is a linear function as of pi, right? So it can be either like this or this. It depends on what? On the sine of pi 0 plus pi 0. If pi 0 is larger than pi 1, you will have to put pi 0 to 1. That's the best thing you can do. Otherwise, you will put pi 0 to 0. Are you okay with this? So max over pi of g pi is if p node is larger than p1, then I'm going to put a 1 here.  Otherwise, 2p1 minus 1, 1 minus gamma. In this case, the best action, the best policy is to choose pi 0 equals 1. That's the best policy. And in this case, the best policy is to choose pi 1 equals 1, okay? We're just computing the thing and showing that our intuition was correct. You always have to play the coin, which you know in advance has the best probability of giving you a positive reward. It's a pretty straightforward calculation and it's totally useless. Because as long as you go away from this simple example, it's a mess. But I'm showing this just because you can do it and then it's fair to show it. But I'm going to give you an example now of a slight change of our problem, which already makes it pretty cumbersome to calculate directly this optimal policy. And this variation is actually pretty simple. So it's a situation in which, so let's go back to the demonstration. So these are the two coins here, left and right. So this was the previous problem, this one. Now I have another pair of coins here. I had to buy many coffees this morning in order to get all the coins for this. I'm putting them on the other side, okay? So these two have different probabilities in general, right? So this might have p-node, p-1 and these two might have two different p's. Say p-prime-node and p-prime-1. Then the game is slightly changed and then if you keep these two systems decoupled, okay? Suppose that at each step you cannot play both of them. So you have to be either here and flip this coin or there. Right? And then the game is the same for each two. So you always get plus one or minus one depending on if it's head or tail. But now you have to pay a cost, you have to pay me 20 cents if you want to cross this. There's a bridge and I'm here. I want a 20 cent to move on the other side, okay? So now the complication is that if you are here you might as well know what to do. If you know in advance what the probabilities are there, you might also know that this is more advantageous to you to play over there. But you will have to pay a cost to get there. So is it worth? It depends on many things. So one thing you might have noticed from here is that this result does not depend on gamma. The outcome depends on gamma. But the policy does not depend on gamma, does not depend on the horizon. Why is that? Because you always back to time zero, right? It's just like a Bill Murray movie that is always in the same place every day, okay? Same time. But this is not generic. This is the extension. Now this system has two states. There is the state A and the state A prime, each of which has its own probabilities and in general rewards. And then there is a transition which has to be made between one state and the other. And there is a cost for doing that as well. So this is summarized in a simple diagram. So there is the state A, the state A prime, or left or right, whatever you want. And then there is the usual structure, like we said. So with a certain probability pi zero, I pick action zero here, which returns me here with probability p-node, I get one and one minus p-node, I get minus one. And then the same here, okay? Let me just, I'm not writing everything back again, like you know how to fill these things. And this would be just prime quantities, zero, p-zero prime, one, one minus p-zero prime minus one, and the same for these other policy, pi prime one, one, p-one prime one, one minus one, and et cetera. One minus p-one minus one, wow, p-one one. All right? And then there is this transition that can take place. So there is another action which is the action of switching. You're here, you must decide, should I flip this or I can take a third action that is I flip, I switch, I go the other way. So there is another action here, the action of switching. This action of switching, let's say you know, switching is not a good idea, let's call it action for switching, it's a mess. We'll send you here with probability one. So if I decide to switch, I will not get shot in the middle. I can go through to the other thing. It's just an assumption for simplification. And then in this case, you get a cost. There is a negative reward minus CS, where CS is positive. That's the cost for the fee that has to be paid in crossing the breach. And the same thing happens the other way around, right? You can switch back and the same thing occurs with probability one, you will end up here paying the same cost. Now challenge, can you do this calculation for that simple model? Give it a try. It's a mark of change. So give it a try. I can do this, okay? But it's painful. And clearly we are not covering much of what the real world gives to us. Remember, we started with the game of go. In the game of go there are, I don't know, 10 to the number of atoms of the universe stays available, right? There are huge numbers that we don't deal with. If we were switching, you think so. Then you have to optimize. You have to optimize over a space which is the space of all possible policies, which is a simplex in a highly dimensional space. Because for every state you will have a number of actions you can take. So this policy lives in a space which is a simplex in a space which is embedded in a number of states, numbers of action, dimensional space. It's not easy, okay? There are other ways of doing this. That's what I mean. And what I want to get to is to show you how these calculations are actually performed. How can you compute in systems which are much more complicated than this in general, the optimal policies? What are the challenges? Some things simplify, some others just don't go away. Some problems don't go away, okay? So in order to do that, we have a little bit to go away from example and abstract a little. So I mean, again, I know you might feel a little bit uneasy or talking about these things without having firm ground, but now the firm ground comes. We're going to give definitions of what the Markov decision process, which has disappeared was up above here, is and how to solve it, at least in principle, okay? And of course you will discover or recover that all this example, when you know exactly what these probabilities in general are and what the states are, are just Markov decision processes, okay? So it's possible to have situations in which your policy changes with time. Strictly speaking, it's not needed unless your PT change as well. You can do that. You can do that. Again, if you know the full future revolution of your PTs, P as a function of times, you will be able to derive an equation for your Pi. But again, it's more complicated because this is again something that happens in the space of states times actions times times, okay? So it's huge dimensional space. But in principle, yes. It's possible to extend all this to non-stationary environments, which I will not discuss, because it's just cumbersome from the viewpoint of, and you don't learn actually much more than what you learn in this specific case. So Markov decision processes, what are the ingredients? Yeah, because it's a cost. It's a negative reward. So I said that CS is positive, so I can interpret this as a cost. It's a minus reward, a negative reward. So you're penalized. You have to pay something if you do that. It's indifferent in neither way, okay? It's just an example, one possible way of implementing this. Clearly, if there was no cost in switching, and you knew everything immediately, then you would switch to the best option of the two and stay there forever. But if there is a cost, then there is a non-trivial decision to be made. That's a comment actually that I had to make before. So if gamma were zero, if you were totally myopic in this setup, and you get some reward out of them, so suppose that you always, the probabilities are such that both of them, both of these p's are larger than one-half, so you always get something out of it. And you're totally myopic, so gamma is equal to zero, you would never switch, never. Because you have to pay a cost, and you don't want to pay a cost. You can get something positive out of this without moving. This happens irrespectively of whether in this case, for instance, suppose in this case, you always get plus one. This is the super bias case in your favor, but if you're greedy, you will never discover it. You will stick to this. Now, if you allow for some horizon, your gamma becomes even slightly larger than zero, then it becomes a comparison, there's a comparison to be made. If I switch, I lose the opportunity of gaining something here, because I'm switching and I'm paying a price. Is it worth of whatever I will get here? I don't know, I have to go there. So you see, in this case already, if you just have two states, then the horizon becomes important. And your decision-making changes a lot, depending on your horizon. If you're farsighted or short, we will discuss about this later. It's not totally unrelated. You can get this question for the moment, because this will take us too far away, but we will discuss about this later. So, I was arising the board. Would you mean not coming? Yes, yes, it's in general unclear from a priori. So if you look at the problem, it's very clear that it's not a priori. So if you look at the problem, it's very difficult to say from the outset whether it's going to be gamma-dependent or not, except in these cases where there's just a single statement, where you know that you will be repeating your experience all the time, so that that really doesn't matter. It just scales the overall gain that you have. But otherwise, in multiple states, typically it's gamma-dependent. There might be instances in which it is not. So Markov decision processes. Shortly, MDP. What are they? So what do you need? If these are Markov chains. So first thing first, what do you need? States. Good. States. There is a space of states, right, which for us will be a discrete set of states. We will always be numbering them, okay, just for simplicity. It's not a restriction in general, but it's useful. Then there will be, it's a decision process, so there will be actions first. Remember the diagram. Okay, there are circles and then there are squares. And then there are transitions. Actions, we'll get to transitions, never mind. So there will be some action states. So again, there is a set of options. In general, this might depend on the state, right? There are some things that you can do in one state and you cannot do in other. For instance, think about grid world. If you're in a corner, not all actions are available. You cannot go up. So in general, this depends on the state, okay? But an easy way around this is just, suppose that all actions are available to all states, but the probability with which you would be able to take that action or this will change the environment is zero. So you can compensate for that, right? But that's a technical point. So states and actions. Then transition probabilities. So what are there? What are they? They are the probabilities of getting to a new state as prime, given the state S and the action A. So that's why I was waiting for A, because the way you go from S to S prime depends on the action you take. Think again about this two state, two-armed bandit. That's what it was. If you pick action zero or one, you would end up in the same state as before.  But if I decide another action, action I switch, I will be changing my state. So my final state as prime would be different from the initial one. So transitions between states depend on the choice of actions. Then rewards, yes, they are all over the place. Rewards, let's go for rewards. So this reward is something which is a function of what? Of the state, of the action, and possibly also the arrival state. So there is this triplet, which is always around, which is state action and new arrival state. And in doing this process, that is starting from a circle, going to a square and moving to another circle, to all these process you attribute some reward. So this might be random or might be deterministic, it depends. Here, in this case for market decision process, we assume that this is the average reward you get while taking this sequence of actions and state. So do you recognize all these actions in the previous examples? All these things that appeared? The just one thing more to add? Observation? No. No, not observation because here we're talking about something in which observations are perfect. Like I told you, we are up above over there. So the policy in general, the policy, which is the thing I'm looking for, and it's important because this is the thing over which we want to optimize, right? So this is the policy by of choosing an action on what must it depend? So in the full reinforcement learning problem, like we said, it depends on all the previous history. But now I said this is a mark of decision process and we know in which state we are. And this is the key property of Markov chains. Once you know where you are, you know what the future will be, at least in probability. So you don't need the full history here to make a decision. If you have access to the full state of the system, you don't need any knowledge of the past because everything is encoded in your actual position and the future will depend just on where you stand now. So the policy for Markov decision processes, it's a probability distribution of erections which depends only on the current state. No, the point is that if the system is Markovian and you have access to the state, all the history is already there. It's a sufficient, in the statistical terms, this is sufficient statistics for all the past history. At this stage, we are. That's why we say Markov decision process. The key point is that there might be an environment which itself is evolving in a Markovian way. So this might be true always. But the point is that we have access to the state. The key point is the observation. If the observation is the state of the system, then we can throw away all the history and we can decide just on the basis of the current state. This is completely unrealistic. In most situations, we do not have access to the full state because what is the full state of the system? Think about physics. You would have to specify all the degrees of freedom of the system in order to be able to predict. For any macroscopic system, this is not the case. Seriously. But this is very important because it helps us in shaping our concepts, our ideas, how we know what we can do in the case, in this ideal case where we were perfectly omniscient and then we just have to compute. It's like feeling like gods for 20 minutes. Being able to predict. Again, this is perfect knowledge because we know what the rules of law are. We know how our system, these things are given to us. We know that just before. We know the pinot and P1 for the one arm banded. Two arm banded, sorry. We know that the system, the observations are perfect, so we just need to map states into actions and not histories into actions. Hope that's clear. Fine. Then it, now it comes, I will try to keep it in 15 minutes so it's going to be rather quick. So the goal, like we said, is to optimize over, so there is one thing which is the return, which is the sum over all times starting from zero to infinity of gamma to the power t, and then there is reward that I get if I s t, if I'm in state s t, pick action at and go into state s t plus one. This is a long one trajectory. So I'm in state s, pick an action a, move that this would bring me to state s prime and then another action and do bring another place. So this is a random quantity and we want to optimize one quantity which is called the value, which is the expectation value with respect to the policy and the transition probability of r. That's the same thing as before, right? I'm just writing it down explicitly. This is the general definition and the goal is optimize this over pi. Now, this is a classical, what we will derive is a classical result in decision making theory, which we'll go under the name of the Bellman equation from Richard Bellman, the founding father of Markov decision processes. If you look into books, it's a bit convoluted in mathematical derivation. We will go for something which should be more intuitive for physicists, which takes a little bit of a long detour, but I think every step is clear than just an abstract mathematical derivation. So we'll try and go through all the steps of this derivation. So the first step is we have to write this thing explicitly. So, and one way to write this thing explicitly is we have to know after these steps where my agent will be. So this is ruled by what? This is ruled by, this is a Markov chain, right? This is ruled by the probability density over the states. And it obeys the Chapman-Comograph equation, which maps the distribution at one time into the distribution at later times. So what is this Chapman-Comograph equation? Let's call rho at time t plus one. This is the probability that my agent is in state s, let's say s prime, after t plus one step. How can I connect this to the probability of being in a state s at the previous time? Because that's what I want to do. Well, if I was in state s at the previous time, then what can happen? I could pick an action a according to my policy with that probability. According to this probability, I will be sent by the environment in a new state as prime. According to the probability p of s prime which depends on s and on a. And all this is some overall possible initial states and overall possible actions. So this is the equation which describes the evolution of the density under the policy pi, which I choose, and under the transition probability p, which is in the hands of the environment. And I cannot do anything about it, but I know it. So if I use this quantity, rho t, then I can write my expected reward as follows. This is going to be the sum overall of the future of gamma to the power t times what? Well, at time t, I will be in position s with that probability. I will pick an action with this probability. I will be sent into a new state with this probability. And I will get an average reward. And all this will have to be summed over states, action, and s prime, over triplets. Do we all agree that this is the expectation value of this? Again. So this thing is a triplet. It depends on where you are at time t, what action you will pick, and where you will be sent. The probability for each of these events are rho t, this is the probability of starting from s, pi a conditioned on s, which sends you here, and p of s prime conditioned on s and a, which sends you here. So these three make for the probability of the triplet s as prime. After these steps, the probability of seeing a s as prime is given by this. Object now, because it's Markov. If you really feel lost now, you better raise your hand immediately. This is what, this is what the expectation of something which is happening in the future. Starting from some initial distribution. At the beginning, your state is governed by some rho 0 s from which you pick one state. Then you have your policy here, you consult with your policy and say, what action should I take? And the policy in your program says, you should pick action a, 0. And then you consult with the environment, which says, where are you going to send me? Oh, give an action a, 0, and state as 0, I will send you to s1 with this probability, fair enough. And then you move to the second step, and so on and so forth. Just because we don't know in advance, we have to prove that the best policy, the one which maximizes this is deterministic. But in general, this thing is a function of any policy. Okay, so that's good, but not super good yet, because we have all these cumbersome times here. So what we're going to do now is going to define one quantity, which is, I call eta of s, which doesn't depend on time, and it's just sum over t going from 0 to infinity of gamma t times rho t of s. You see, this is the one that picks this object here. So we are going to rewrite this as sum over as s prime eta of s by a of s p of s prime s a, and then the word of s a s prime. What does this thing mean? It has a very simple interpretation. You remember, I told you, gamma can be interpreted as a probability of survival after each step. So this is nothing but the probability that I am in state s after t steps, and I'm still alive. This thing. And then I sum overall of them. And then what does that mean? That this object is the time that I spent in a given state s before being killed. It's the overall residence time in state s, average residence time, before being killed. Okay, do you agree on this? Okay, now we have come with an expression which is more compact. And what do we have to do? We have to optimize over v pi. So now we're going to do this as physicists do. So first thing, oh, some mathematicians, but they do it more accurately than we do. First thing, we have to find the stationary points of this function with respect to pi. So we want to take the derivatives of this object with respect to pi. What's the difficulty in this? Can you spot the difficulty? No? Okay, so let's start with doing this. I take the derivative with respect to pi. Does this depend on pi? No, this is the environment giving me the rewards. So pi has nothing to do with that. Does this depend on pi? No, this is how the environment evolves. It doesn't depend on how I choose my actions personally. Once an action is chosen, the environment will go that way, but it doesn't depend on the probability that I give. Does this depend on pi? I mean, it is pi. Does this depend on pi? And that's painful, because it depends on pi in a very, very intricate way. It depends on pi through all times, and this depends on pi through this. So do we want to take the gradient of this with respect to pi? Not sure. Yeah, I will argue that it's not the most efficient way of doing this calculation. Is there a workaround for this? So this is something, actually, it's a trick that you have to learn, okay? Because it pops out in so many different domains of physics, and you might have even used that without paying attention to this. So what do you do when you have such a nasty thing? What would be your ideal world? My ideal world would be the situation in which this is independent of that. So I could have to take the derivatives with respect to this, or with respect to that. But we said, okay, but this is not independent of that. And why it is not? Well, because it has to obey this. And then the idea pops up and says, well, perhaps I can try to optimize this thing regarding this object as an independent variable, and using this as a constraint just out of curiosity. Well, who has ever heard about Martin C. Jaros? Martin C. Jaros action? Nobody. Okay. Now you know how it works in a totally different setup. So if it pops up, that's the trick. Okay. So there's a price to pay for this, right? Because what we'll have to do is to introduce this dynamics as a constraint. And if you want to introduce a constraint in an optimization problem, what do you do? Lagrange multiplier. In order to do Lagrange multiplier, what do you have to put? The multiplier, which is an additional function coming into the game. So the price to pay is that you decouple the dependence of this from this, but you will have another additional field or vector. I will ask another question. Who has ever done the Hubbard-Stratonovich transformation in statistical physics? Isn't that the same? It is the same. And there are many other things in theoretical physics that you do the same way. You decouple nasty nonlinearities by expanding into constraints when you have done. Okay. I will never be able to make it in three minutes. And at one, I have to leave. So I will just outline what we will do first tomorrow before moving to something else. We will do exactly what I said. The only thing is that rather than using this equation for raw, we will use directly an equation for eta. That is the equation for the average residence time with killing probability. So what is the equivalent equation for this thing? I will write it down here. So eta obeys the following equation. Eta of s prime is equal to a row node of s prime plus gamma sum over s and a of p s prime condition on s and a by a s and eta. The average residence time with killing or without obeys a closed equation. And this is all we need. This is the constraint we need in order to use this formula. So we can get rid of all this row t around. It's a recursive formula. Okay. Ever seen that? Probably not in this form, but in fact you have. You don't know. Okay. So for tomorrow, if you want to do the exercise, try and prove this formula. We will start from this, from the combination of this expression and this formula tomorrow to derive the optimality equation and the result that we're looking for. See you tomorrow.


But this is the first romance book I actually like. Yes! You are my son! Welcome back to my channel. I hope you're well. I feel very enthusiastic today. I don't really know why. I'm just like in a good mood. I'm in the mood. It'd be a good idea in today's video to talk about my favorite short books. I always feel like around this time of year, everyone starts panicking. Shit, it starts hitting the fan because people are either on track for their Goodreads goal or they're way, way, way off. The ultimate shade of it all. I thought I would help out those people who are way, way, way, way, way off and talk about some of my favorite short books. These are also great if you're in a slump. I'm gonna be going from order of Shortest to not Shortest. No.  From Shortest to Longest. So in my trusty notebook, which I got from Fairy Lou, if you haven't seen my unboxing of Fairy Lou, I'll link it. I think it is that side. My first recommendation is the first in the series of Ghosted in L.A. So Ghosted in L.A. Volume 1 by Sina Grace. This is so short. This took me like three minutes to read. It's 28 pages. So like, bring it to me. You know, like bring me those short pages. This is about a girl who moves to L.A. because her boyfriend's there. She's like, I'm moving for the guy, you know? I'm going where he is. He breaks up with her as soon as she gets there and she runs away and comes across this house full of ghosts. I've basically told you the whole premise of the first volume. It's called Ghosted in L.A. Like, what am I supposed to tell you? I am so excited to read the rest in this series. This is all on script. Like, the whole thing. All the volumes are available for free on script. I will leave my link down below where you can get 30 days free of script in my link. It's actually 60 days free. So go get that coin, my guys, and gals xoxo. The reason I think this is so good is we all love those ghostly vibes, you know? I love a bit of ghostly taste. A little bit spooky. Scooby-Doo. Like, when ghosts are in this kind of like fun way, not scary way, I love it. I love a bit of ghost fun. My favorite ride at Hollywood Studios in Florida is the Tower of Terror, where there's all the ghosts. This kind of house that she stumbles upon has the same kind of like art deco vibes of the Tower of Terror, and I just live for it. So I can't wait to read the rest of this series. I might even read one today. Who knows? Kind of feel like it. Kind of in the mood, anyway. I really like the art style as well. So yeah, if you need like a super duper quick read, this is one I'd recommend because it would take you like four minutes. Next on my list is Evidence of the Affair by Taylor Jenkins. Read, this is 87 pages. Yeah, 87 pages. So I read this, I think Riley gave me this recommendation. I can't remember why I needed to read it. It was for some sort of rudathon I was doing, I think. It's about a man and woman who basically find out that their respective partners are having an affair with each other. This is awkward. The cheated on individuals are writing letters to one another about this affair, about how they feel and like, yo, okay. We all know Taylor Jenkins reads excellence from books like, I just realized it's here, Dave Jones and the Six, like we're all aware, but her older books are never spoken about. This book, I read it in literally like 20 minutes. It's so good. It's so good. It is really good. I love it. I love it. The stuff that happens, you're like, ooh, giving us a bit of, ooh, ah, you know. It's that kind of feeling. I'm getting older. I don't know if I want all this drama all the time, if I'm being honest. Most of us have read Taylor Jenkins' read, I think, and most of us love her stuff. So like what better to go and read a short book from an author you already know that you enjoy. I really like it in that letter format. It's a really interesting thing that I don't read enough of, like just a book told entirely through letters because you learn a lot about the characters in kind of the stuff that people will divulge through letters that they wouldn't necessarily say. However, it's still slightly unreliable because people don't fully tell the truth if they're not speaking face to face, I think, and like you don't tell everything through letters. So there's some stuff that is really cleverly revealed as time goes on that like our characters obviously knew about for a long time that we don't know.   I just think memory of being in my bed in Leeds, reading this book on my iPad, like just feeling so cozy and she's like, oh, ah, it was just a really, oh, ah. It was just a really nice reading experience. So I would definitely recommend that. Next is The Test by Sylvain Neuvel. So this is about the British citizenship test. It's about a man who wants to take the British citizenship test. That's what I'm gonna say. Like it takes a strange turn, it's very short, but it takes a very unusual unexpected turn. When I realized where this was going, I was shook. I was shook.  I kind of wish it had been longer but it's such a short read, it's really fun. It plays, I can't say what themes it necessarily plays without spoiling anything, but it plays with some really interesting themes. The idea of morality, right and wrong, and I think a bit of sci-fi often isn't short, you know? Like sci-fi, I think of sci-fi as being these long ass books that just kind of put you to sleep. Think about your boring next, you know? Whereas I think it's really refreshing to have a sci-fi book that is so short and is so readable and it's just like you fly through it. It's high stakes, you know? His family is relying on him. So that really keeps the momentum going all the way through, you know? This whole book is kind of tense rather than having ebbs and flows, like you're kind of just on edge the whole time. Next is actually a non-fiction. Oh, it is Toss and Light Shift Before Christmas by Adam Kaye. This is, by the author of This is Going to Hurt, this is 142 pages, so we're getting a little bit bigger. That's what she said, that's what she said, that's what she said. Good one.  And so it really has layers to it where it's hilarious but also very hard-hitting as well. And this book is all about Christmas. So it's taking you through, I think he has six Christmases and just taking you through what happens at Christmas and how craziest stuff happens at Christmas and he does work on the maternity ward for the majority of his career and so if childbirth and babies and pregnancy are triggers for you, I wouldn't recommend this because it's what it's predominantly about but it's hilarious. Just the way he views everything is so funny and he manages to bring so much light to some dark situations and also just getting a look into this profession that I'm never gonna go into because your girl ain't minded that way.  You. Are you dumb? It's really interesting and I can't wait to read everything he puts out in the future, he's such a funny guy and coming up to Christmas I feel like this is a great thing to read. The next is Meet Cute Club by Jack Harvan. I speak about this quite a lot so I won't go into too much detail because I read it quite recently and I've spoken about it a lot since but this is the first romance book I actually like. Yes. You may say I'm a doing. So this is about two individuals, one who works at a bookstore, one who runs a romance book club and the one who works at a bookstore makes one the other one for being into like, old, why don't I say two individuals, it's two guys, two guys. I'm sorry, two guys. Yeah, he makes fun of the other guy for liking like old women romances and then he joins the book club and it's their romance and it's so cute, it's so cute. I really, really love this. I love the characters. It's also a love of books. Like this book is all about the love of books and so reading DNF in a book just makes me so excited. If like me, you weren't or aren't super into romance, this is a great place to start. It is only 186 pages. It's a little bit steamy, there's like two sex scenes which I think is a good amount for like 180 page book. You get a lot of time with the two characters and them learning about each other and learning how to like love each other and it's just such a nice story. So yeah, everyone read Meet Cute Club. And then the last book that I wanted to mention, I am cheating a little bit because in my head short books are under 200 pages and this one is 214 but it does have entire pages that are illustrations. So this is a monster caused by Patrick Ness and this is about a young boy whose mom is dying of cancer. He's really struggling, his mom is getting worse and worse and then this monster appears outside of his house and promises to tell him three stories. If in return, Connor the boy tells him the truth, tells him his real truth. Let me tell you I sobbed. Like I cry a lot but I sobbed. If you're in need of a quick cry, this is it. I would really recommend the illustrated version. The illustrations in here are beautiful and really adds to the atmosphere. It's just heartbreaking. It's so sad but also so revealing and so insightful and just such a lovely story.  Like sometimes I just really say the stupidest stuff but it feels like you're really taking the word story down to its purest form. Sometimes you read books and they don't feel like a story. Do you know what I mean? Like that kind of old school childlike feeling of a story and that's what this feels like. Not only will it be a quick read but it will get your tears and emotion out and it's just a lovely, lovely tale. So yeah, I would definitely, definitely recommend this. There we have it. That is all of my short book recommendations. Let me know if you've read any of these or if you want to read any of these, which ones are gonna get to first. Yeah, I hope it was helpful if you're struggling with your Goodreads girl. Thank you so much for watching and I'll see you very, very soon in another video. Bye.


Hey everyone, I'm Bianca Renee, you're watching Bianca Renee Today and today we're going to deal with dandruff, itchy scalp, dry hair, and product buildup. We are going to make a home remedy using apple cider vinegar. We're going to make an apple cider vinegar rinse, also known as an ACV rinse, and this stuff can cure like everything. Besides even hair care, apple cider vinegar is also used to help with acne, with weight loss, like random stuff. It does everything. By doing this rinse, it'll also help with hair growth because we are now cleaning our scalp completely, getting rid of any buildup or bacteria, so that our scalp can breathe and grow more beautiful curls. Now you can use any brand of apple cider vinegar to make this, but it is best that you use an unfiltered, organic one. This is the brand that we usually see online, is this Bragg. It looks like this. It says organic, but the main thing that you want is the mother. The mother is all of this extra gooky stuff at the bottom, because that's supposed to have extra nutrients in it, so don't worry. It's not gross. It's supposed to be there. Now one thing that you will have to get used to when doing an ACV rinse is definitely the smell because this stuff, pretty potent. Oh man, I forgot how strong this is. I can't wait to put it all over my head. Now I know you've been walking around like this all day because you got that itchy scalp, but don't worry, we're going to cure it right now. Now all you're going to need is some apple cider vinegar, some water, a measuring cup, and a squirt bottle. So first you're going to get the apple cider vinegar and shake it up a little bit so that the mother gets all mixed around. So now we're going to pour two ounces or one fourth cup inside of our measuring cup. And now we're going to pour that into our water bottle. Now on some sites it's recommended to do twice as much water, but I'm just going to fill up the rest of my bottle just about that much. Shake it up and now we have our mix. Now I've been doing a lot of research on apple cider vinegar and I've watched a lot of videos and some girls like to do coconut oil first to detangle their hair and then apply the rinse. And I personally am 100% against detangling my hair when I'm not in the shower because it's just, it just seems like a horrible experience and nobody got time for that. So I'm not going to detangle now, I'm going to detangle after when I rinse it out and use conditioner and it's actually going to be easy.  Definitely don't wear any type of clothing or shirt that you care about. And now we're going to just start to apply this all over the hair and especially the scalp. Once you get in there you're going to massage, massage your scalp and get rid of any product or dandruff buildup. I'm trying until you just feel it all over your scalp wherever you think you need it. So after we think we got it all over our scalp it's also good to get it on the actual shaft of the hair because it will make your hair nice and shiny. Soaking wet with apple cider vinegar it's going to put a little cap on. Now I'm going to let this sit for about five minutes and then we're going to rinse it all out. And while we wait this is the perfect time for you to follow me on Instagram, Twitter and Snapchat. So now it's time to take off the shower cap and rinse out the apple cider vinegar. Today I'm going to be using this new decadence conditioner by Diva Curl. Now some girls use the apple cider vinegar rinse as a replacement for shampoo so you really could just skip to the conditioner part but make sure you really cleanse so you don't leave the shower smelling like vinegar. So now I'm going to grab my new best friend which is my scalp massager which I got from Vanity Planet and this is going to really cleanse our scalp even more and get rid of any dirt or dandruff.  So now I am fresh out the shower, my scalp is extra clean, no dandruff, no buildup and my curls are ready to be styled and no trace of apple cider vinegar. Now some girls do take this a step further and do a deep conditioning treatment after they use their regular conditioner. Totally up to you. If you feel like your hair needs some more moisture you could do that as well but this is the simple how to do a basic ACV rinse. I hope this video helped you guys if you're dealing with any dandruff product buildup itchy scalp. This really should work for you. Let me know if you guys try it. If you want to know how I style my hair after the shower I have plenty of videos showing you what products I use. I'll put all those videos down below in the description box and for the scalp massager as well. If you like this video please give it a thumbs up and remember to subscribe. I post new videos every Sunday so I'll see you guys next week. Thanks for watching Beyond Grinne today.


Hello, welcome to SS Unitec, succeed this site. And this is part one of Power BI tutorial. So today we are going to discuss introduction part of Power BI. So what is Power BI and what are the importance of Power BI, what are the types of Power BI, and how we can connect with the data sources. So let's start with the introduction of Power BI. So what is Power BI? So Power BI is a collection of software services, apps and connectors that work together to turn your unrelated sources of data into coherent, visually impressive and interactive insights. Your data may be Excel explicit or collection of cloud based or on-premises hybrid data warehouses. So Power BI lets you easily connect with your data sources, visualize and discover what is important and share that with anyone or everyone you want. So basically in Power BI, you can just understand as of now, this is a tool where we will get the data from different different sources and here we have a number of sources by which we can connect and get the data. After that we can create the visualizations and we can share that data with the management or any other in the organization. Next is types of Power BI. So basically we have three types of Power BI. First is Power BI desktop where we will build the reports and visualizations. After that we have Power BI services where we can publish the reports and visualizations. Then we have Power BI mobile. So in mobile we can if we want to access the reports in the mobile then we have an option for Power BI mobile. So these are the types of Power BI. And last thing in this video, what is the Power BI Pro and what is the Power BI premium and what are the differences between these two. Power BI Pro is an individual license that allow access to all content and capabilities in the Power BI services. You can even share content and collaboration with other Pro user. Next is only users can publish content to the work spaces, share dashboards and subscribe to dashboard and report. So this is the Power BI Pro. Next is Power BI premium. So it provides dedicated capacity to deliver more consistent performances. It also supports large data volume in Power BI for individual users. Premium enables wide spread distribution of content by Pro user and it doesn't require pre-user license for recipients viewing the content. So what it means. So as of now just try to understand we have two types of licenses available in the Power BI. So we can take those licenses from the Microsoft. First is second is Power BI premium. So once we purchase Power BI Pro then this is for the small organizations and here we can share the content with other Pro users in the organization and next is Power BI premium. So Power BI premium is used for large organizations and here if we want to see the report then there is no need to any Pro license for viewing the report. So these two types of licenses are available for the Power BI. So Power BI desktop is free and Power BI Pro and premium will be used once we will publish the report. So this is the introduction of Power BI. So in the next video we will see how we can download and install the Power BI and will connect with the data source. So thank you so much for watching this video. If you like this video please subscribe my channel and please share to others and don't forget to press the bell icon to get the notification of newly uploaded videos. Thank you so much.


uploading a new video. Mum must speak to you in a minute. Okay, babe. I'm going to give you the package. Hi. Hi, how are you? Hot. Grandma and Grandpa are here if you want to say hi. You're on speaker. Hi. Hi. What's up, guys? I'm in bed. I keep not sleeping at night. I have to do it in the morning. And by that point, there's nothing left to do. Like, for the day. Isn't that amazing? Hey, what's up, you guys? It's hot, hasn't been a good night. I can't sleep. I don't know what to do anymore. What? Say I'm not sleeping at night. You were not asleep last night. I guess I just like to sleep. I haven't done anything today. Um... Tomorrow I'm packing to go up to my mum's for Christmas. I'm going to film some videos to go up while I'm there because I don't know who I'll be able to film. But yeah, it's been a very slow day. I've been asleep a lot. I went to Tesco. I haven't done much today. But yeah, the end of this vlog here. If you're new, subscribe. Thanks for watching and thanks for supporting me. It's really appreciate it. Peace.