We gathered here at PlutusFest in December here at the University of Edebra. It's a great event that celebrates all the work we've done in smart contract research during 2018. Also a great opportunity for members of IOSK Research and the IOSK Development and Engineering team to come together and discuss not only things related to smart contract development, but also all the other research streams that we pursue at IOSK. Yes, so incentive and delegations is one of the major research efforts that we did within 2018. There is something that we've worked very intently into different parallel threads. On the one side what we were looking is that is it possible to decouple the protocol participation component in a protocol, in a protocol for stake protocol like Uroboros, with the stake holding part of the protocol which for instance involves a lot of end users that may not necessarily want to participate in the protocol. And this can be achieved via delegation, which is something that we invested a lot of research effort in 2018. And I'm very happy that we have reached a very, very good level now of a polished product that we're going to be releasing soon both in terms of a white paper, but as well as folding into Cardano. And this is something that we expect to happen very soon in 2019 and beginning of the year perhaps by the end of Q1. At the same time, the second parallel thread that investigates this is once you delegate what form would the delegation take in the blockchain. This is particularly critical. The reason is that once delegation is basically free and everybody can assign their stake to someone else what could happen conceivably in a worst case scenario is that everybody delegates to one, so the system becomes centralized. And in fact, we've seen in blockchain systems that existing blockchain incentive mechanisms can actually have such centralization tendencies. You can see it for example in the way Bitcoin operates, where there's just a few mining pools that are responsible for the whole Bitcoin ecosystem and that's certainly not a desirable feature for a decentralized system like the one that Cardano is. So basically what is important here is to design an incentive mechanism that enables delegation to operate in such a way that a natural ecosystem of multiple stake pool managers can emerge and be responsible collectively in a true decentralized fashion for the protocol. So this is something that we've put a lot of thinking in and I'm very happy that we have the white paper out. And we are now at the final stages of implementing that into Cardano production release, Shelley. So this was one of the major research thrusts in delegation incentives that we did in 2019 and very excited to see being rolled out in 2019. A new research topic that we started in 2018 and is very exciting from also technical but also applied point of view is Uroboros Kripsinus which is a protocol design which brings the Uroboros research into this privacy preserving setting. This was very exciting because privacy in blockchain systems is a very important question and it is something that still a lot of research needs to be done in order to have satisfactory systems that provide a good level of privacy. And at the same time it's very challenging from a technical point of view because somehow at the core of how a proof of stake system works you could think that it's inconsistent with privacy because by participating in the protocol itself you have to reveal some information about the stake you have in the system because that's exactly the core of how the protocol works. The proof of stake is basically a statement that one of the participants does that they are eligible to extend the ledger based on the fact that they have already some assets recorded on the ledger. So it sounds like intrinsically there is some privacy leakage that takes place during that process and that's why it's particularly challenging to develop a protocol that is capable of providing a proof of stake blockchain but at the same time do that in privacy enhancing fashion. So this is what we did or at least the first step was done and we completed Uroboros Kripsinus is the design of a proof of stake blockchain which is privacy preserving. We released this last month and this is also a research line that we will substantially be investing time within 2019 so that we can take this first protocol that we designed and continuously optimize it to the point that it will be feasible to implement it. So scalability in blockchain systems is one of the very important open questions right now. The reason is that most blockchain systems the way they operate they do not utilize the throughput that is available to the nodes that are participating the protocol in the optimal way and we can see that as it is in the basic blockchain protocols let's say that follow the Bitcoin blockchain algorithm in the way they operate you have to produce a block but you have to produce a block at regular intervals and the distance between the block production instances is essential for the security of the protocol because this provides the opportunity for nodes to converge to a unique view about what is the blockchain following the longest chain rule or the most difficult chain rule basic chain selection logic. So being capable of completely utilizing the network that is available to the nodes is a very interesting research question which we put some substantial amount of work in 2018 into. Parallel chains is a white paper we released in late 2018 and shows how is it possible to get a high performance high throughput blockchain protocols by combining basic block chains together. This is actually a very interesting operation also from a protocol design point of view because it enjoys modularity in the way you can design the final protocol. So essentially it says that you can take multiple block chains together and combine them into one unified distributed ledger that is going to enjoy the property that completely utilizes the network sources that are available to the nodes. So there's more research that is coming up in this direction which focuses on how to get a truly scalable blockchain system and this is something that it will be coming out in 2019. So side chains is another very critical component in the Cardano roadmap and also one that is going to facilitate smart contracts and also smart contracts that can operate with different programming interfaces. So one of the important aspect of side chains is the ability to transfer assets between block chains that operate in parallel but using different rules and different scripting languages as well as different mechanisms for example on how they're deployed on how they utilize different resources like the network. So being capable of transferring assets in a way which is secure from one blockchain to another is a concept that early on was held as one of the most important components for the blockchain ecosystem. So we've done in 2018 very, very significant work in this direction both in the proof-of-work space where our research was a non-interactive proof-of-work which was a work that was studying the basic component that is needed to transfer an asset from a proof-of-work blockchain to another proof-of-work blockchain but also we have done very substantial work on proof-of-stake side chains which is particularly important for Cardano and our work actually was submitted earlier in 2018 in the IEEE Security and Privacy Conference and that was accepted and will be presented within 2019. So this was another great success for our team because IEEE Security and Privacy is the top tier conference in cybersecurity and at the same time highlights the importance of designing secure side chain systems for distributed ledger technology in general. People ask me a lot why we have different Uroboros protocols. They say, well, there was this great piece of work that you did in 2017 but then there was another Uroboros version that came and what does that mean and is it something that, is it a good thing or is it something that it's something we have to worry about because a different protocol is being suddenly used which means perhaps that the old protocol there was something wrong with it. So in fact this is not the case at all. What happens here is a natural process from a research point of view of completely exploring the proof-of-stake design space. The fact of the matter is that different flavors of the protocol are more suitable for different use cases and it's very important from our point of view to completely understand all the different ways that we can fine-tune the Uroboros proof-of-stake protocol for all the use cases that could potentially the protocol can find itself useful and be deployed. So that's why for instance we have now Uroboros-Crypsinus which is a privacy-preserving proof-of-stake protocol. The core of the protocol still remains Uroboros and that's why the protocol is designed like that. So if you look at the protocol itself the core mathematical analysis remains the same as it was in the original Uroboros paper and in fact this also shows how useful is this design strategy that we have. We had a mathematical analysis which explained how is it possible to formally prove secure proof-of-stake protocols and that was with the original Uroboros protocol and now we're capable of using the same machinery to fine-tune the protocol in various high-value specific scenarios that we could actually deploy it in the future. Yeah so I'm very excited about 2019 actually it's the reason is that all this year so far we put so much research effort to understand how proof-of-stake protocols can actually work and how you can develop them and deploy them securely and 2019 is going to be the year that we're going to see actually this being launched. So it can't be this is really the most exciting moment in a researcher's sort of work line because we've done the work we understand the protocols. Whiteboards over whiteboards have been full with proofs and analysis and now the development is almost at its final stages so what's going to happen in 2019? We're going to have the pleasure of actually seeing these protocols work in practice and being deployed and be used by people so it's a very exciting time. 2019 for all the research team to see all the fruits of labor of these last few years come actually to the end users as part of Cardano.


ರವ್ತೆ ಟಿತಂು ಎೋומಂತ್ರ್ ಜವವಿಲತ猿ಲೈದ್ ಪೀವಿಕ inne percentbane problem between vapor absorption and vapor compression system. In my previous video riddle I have discussed simple vapor absorption system, also called practical vapor absorption , today we are going to compare the vapor absorption system & compression strength. at the end of this video session students will be able to compare the vapor absorption and vapor compression system on the basis of various criteria as well as they will be able to state the merits and demerits of each system that is vapor absorption and vapor compression system. the contents of this video session are comparison between vapor absorption and vapor compression system and what are the merits and demerits of vapor absorption system. now let us see what is the difference between vapor absorption and vapor compression system. now in this table as we have already seen the working of both vapor compression as well as vapor absorption system the major change as compared to the vapor compression and compared to the vapor compression system in vapor absorption system is that in vapor absorption system we do not use the compressor. whereas in case of vapor compression system we use the compressor for compressing the refrigerant coming out of the evaporator so the function of compressor is carried out by three devices namely absorber pump and generator so absorber pump generator assembly carried out the function of compressor in case of vapor absorption system. and therefore there are certain advantages in case of vapor absorption system so the first point is that if you see here the vapor compression system will have more wear and tear as well as it will produce more noise due to the moving parts of the compressor particularly when you use the reciprocating compressor. ok there are vibrations which produces the voice noise as well as there is going to be wear and tear between the piston cylinder so in case of vapor compression system due to the presence of compressor it will be noisy as you can experience particularly in case of our domestic refrigerator which works on the principle of vapor compression cycle. ok so every time we find that there are vibrations and it produces the noise whereas in case of vapor absorption system the compressor is absent so only moving part in this system is an aquapump but it is of very small size and therefore the vapor absorption system is quieter in operation compared to vapor compression as well as it will have less wear and tear. now second thing is that the vapor compression system in case of vapor compression system you supply the electric power to run the compressor because compressor is usually driven by electric motor and so you need electric power to be supplied in case of vapor compression system. whereas in vapor absorption system though we are using pump the pump work is usually very negligible and majorly what we supply our major input in case of vapor absorption system is the heat supplied in the generator so that heat which we supply in the generator for separating the refrigerant vapor from the strong solution the heat which is used can be waste heat which is low grade heat that is compared to the electric power heat is the low grade energy so waste heat low grade heat or even we can use solar power that is solar heat can be used so no need of electric power because some of the vapor absorption system they do not even use pump they are pump less so in absence of total electric power still the vapor absorption system can work now but the COP of vapor compression system is more whereas the COP of vapor absorption system is less the major reason here is that because your input which you are supplying is a low grade now COP already we know the definition is the refrigerating effect upon input so for vapor compression system in both the system the output desired is nothing but the cooling effect or refrigerating effect but the denominator is different in case of vapor compression system it is the compressor work whereas in case of vapor absorption system because pump work is negligible it is the heat supplied in the generator that is the denominator so COP in this case is refrigerating effect divided by heat supplied in the generator now because the heat is a low grade energy you require to supply the large amount of heat in case of generator therefore as compared to the vapor compression system COP of vapor absorption system is less usually the COP of vapor absorption system is less than one whereas in case of vapor compression it is more than one up till now the HHS COP of the order of 1.4 has been achieved with the help of vapor absorption system now the another disadvantage of vapor compression system is that at part load the performance is over means if it is not operated at full load condition then the problem is that the refrigerant doesn't get the required heat in the evaporator so there are chances that the vaporization of liquid refrigerant may not be complete in the evaporator and you need to compress the wet vapor so that is why the performance of this system deteriorates at part load whereas in case of vapor absorption system as we have seen already the refrigerant vapor coming out of the evaporator is absorbed in the absorber so even though the load is less and the vapor along with liquid enters into the absorber there is no much issue because in absorber this is going to be absorbed with the water so the performance of vapor absorption system is not much affected due to the load means partial load at part load also it will be performance will remain same but whereas here it will decrease now let us see the another comparison here you can see now in case of vapor compression system we are supplying the mechanical energy through the compressor means electric power and then it is given in the form of mechanical energy for running the compressor whereas in vapor absorption system we are giving the heat as the input heat energy is utilized but the major issue with vapor absorption system is that charging of refrigerant is difficult now in case of vapor compression system the charging of the refrigerant to the system is easy the reason is that in vapor compression system usually the refrigerant is charged into the compressor using the compressor compressor is provided with the charging line so once you open the charging line wall the refrigerant vapor from the refrigerant cylinder can be taken inside the compressor and it will pass into the system so charging is that way very easy but here because we do not have compressor we need to charge the refrigerant in the evaporator okay so liquid charging will be done in this case here it is vapor charging here it will be liquid charging so this charging of refrigerant is difficult particularly in vapor absorption system next here because as we have seen in the previous slide the input is electrical power so energy supplied is a high grade energy electric power so this energy supplied is one full to one half of the refrigerating effect so this energy supplied is much less so COP is higher performance is better whereas in case of vapor absorption system we are supplying low grade energy that is heat so energy supplied is heat energy in case of vapor absorption system it is more than about one and half times the refrigerating effect therefore the COP of this system is comparatively very less now in case of vapor absorption compression system because we have a compressor we require some preventive measure since liquid refrigerant accumulated in the cylinder may damage the cylinder cylinder so entry of liquid refrigerant particularly with reciprocating compressor is not desirable in vapor compression system whereas liquid refrigerant though it comes out of the evaporator and going to the absorber it has got no bad effect on the system now let us see compared with vapor compression system VARS that is vapor absorption refrigeration system have one major advantage we are compressing the liquid liquid is compressing instead of vapor and as a result work input is very small let us think for a while that why the work input will be less in case of vapor absorption system now already we know that liquid specific volume is very small compared to the vapor specific volume therefore the pump work is negligible as compared to the compressor work and that is why we get the advantage in case of vapor absorption system and the pump work is very small so work input in VARS is of the order of 1% of the heat supplied in the generator it is very negligible and so we it is often neglected in the cycle analysis VARS are more expensive than VCRS because absorption system they are complex and they occupy more space bulky so they are more expensive remember compared to vapor compression system which are compact in nature and also we have seen COP is much less so they are less efficient also and vapor absorption systems require much larger cooling tower to reject waste heat and they are more difficult to service since they are less common so large amount of heat is to be rejected in case of vapor absorption system in the condenser and absorber so you require larger cooling towers and servicing is also difficult because they are not used much widely compared to vapor compression system therefore vapor absorption systems are considered only when the unit cost of thermal energy is low means the waste heat is available in large quantity or it is projected that it will have low cost heat available in that low cost compared to the electricity these are the references thank you


Howdy, Moz fans, I'm Chris Thong with GoFish Digital, and today we're going to talk about advanced on-page optimizations. Commonly in SEO, when we think about on-page optimizations, we're typically thinking about core places to include the keywords such as the title, the H1, the URL within the content. But some people might be wondering, how can you take your on-page optimizations to the next level and get beyond some of those basic tactics? So that's what I want to cover today. So one of the best ways I found to shift away from the keyword mindset is actually to shift to more of an entity mindset. So for an example, if you were going to optimize a page for the term retire early, instead of using the term retire early a bunch of times on the page, you could use tools like IBM Watson or Google Natural Language. Both of those have public-facing tools that you can run any text document you want through. And if you ran it from a result like Investopedia, you might see that retire early comes up with entities such as Vicky Robin and Joe Dominguez to the top authors about retire early. As well, when you take a look at competitor pages and have more of an entity mindset in mind, instead of just thinking about how many times they're using a keyword on the page, you're thinking more strategically about common topics and themes you want to integrate within your own website content. Another great way to take your on-page optimizations to the next level is this concept of eat, expertise, authoritativeness and trustworthiness. One of the best ways to improve the expertise of your site content is just to simply look at your author biographies. A lot of sites still get this wrong. And when you look at author biographies, you should always be thinking about, hey, where can I highlight my years of experience, my education, my previous experience, my thought leadership directly within your author biographies to better highlight your expertise to both Google and users. As well, another thing I love to think about with on-page optimizations is this concept of information gain scores. It's one of my favorite patents analyzed by Bill Slosky, where he talks about the fact that Google looks to reward content that adds the search results and doesn't just repeat what's already out there. So think about where you can leverage your own unique expertise, data and insights to benefit from this concept of information gain scores. Another great way to improve the eat of your site's content is to actually cite sources, the Wirecutter is phenomenal at doing this. Anytime they cite an individual fact, actually cite where they got that fact and link to external trusted accredited sources to verify where they're finding that information from. Another great way to improve the trustworthiness of your content, take your on-page optimizations to the next level. Another strategy that I think is highly, highly underrated is this concept of freshness. We've actually run tests on our own site and we pretty consistently see that when we do things like update timestamps or just refresh content, we see noticeable upticks in both rankings and visibility and traffic. And I think this makes sense from kind of multiple perspectives when you really start to think about it. From a trustworthiness standpoint, if Google thinks the content is outdated, it's hard for it to trust that the information is actually accurate within the article. As well from a competitive standpoint, it's very hard for Google to compete in terms of real-time results, that's why users might go to platforms like Twitter instead of Google. However, in recent years, Google was making a push toward include live blog posting type URLs in top stories. I think they're trying to incentivize publishers to update their content in real-time to set the expectation that users can get real-time information on Google instead of just Twitter. Another great way of thinking about your on-page optimizations is this concept of historical competitor changes. Oftentimes when we think about our on-page optimizations, we're only thinking about what competitors are doing in the given moment, but we're not telling the story of how they've changed their on-page optimizations in order to get to that point. So you can do this type of analysis for really competitive queries. What I like to do is find a strong competitor that's actually improved in the rankings in recent years and then take that page and actually run it back through the wayback machine and see which on-page changes have they made over time, what content they're adding, what are they removing, and what are they keeping the same. And that can help you better isolate what the most prominent on-page changes competitors have made have been. Another great strategy to use is use a text-if-compare tool. You can actually take an old version of text and then compare that against the current version of text, run that through a tool, and the tool will actually highlight all of the changes competitors are making that makes it very easy for you to find what on-page strategies your competitors are utilizing. The final aspect of advanced on-page optimizations I want to talk about is this concept of keyword segmentation. We segment our traffic data in Google Analytics all the time, but we don't segment our keyword data in the same way. So using tools like stat, we can actually create keyword segments anytime we do some type of on-page optimization. If we update entities, if we update freshness, if we update eat, we can create keyword segments in all of those different instances and then over time we can compare the segments against each other and measure what the most important ones have been. That will actually give you better data about what type of on-page optimizations work best for your specific sites. So hopefully that's been helpful. Hopefully you'll walk out of here with some more strategies and concrete takeaways now you can improve your on-page optimizations and take them to the next level. Thanks a lot, Mosfans.


Hey guys, it's MJ the student's act tree and in this video. I want to look at some of the reasons on why the polls were wrong We were led to believe that Trump only had a 15% chance of winning. Oh Boy, how wrong were we? So what I want to do in this video is look at the five reasons why the polls were wrong So here they are at a glance, but what I'm going to do in this video is I'm going to look at each of these a little bit more detail So the first one is this idea of a biased sample We want to go out and we want to find the information By saying, you know, you're an American who would you vote for? But when we introduce the polling mechanism What we're basically doing is we're saying you are an American who's being exposed to this poll and Who are you going to be voting for? So what we've done is we've added in this extra condition Which alters the probabilities So from a graphical point of view, what we can see is that you've got your population And only a certain subset of your population Is exposed or more exposed to the polling mechanism Now the classic case in statistics was a few years ago Where not everybody had a telephone only the rich used to have the telephones And they conducted surveys via telephone Then the data that they were getting was distorted towards what the rich people thought and so it wasn't accurate Now with this election the polling mechanism could have been as slight as the type of website people visited Certain media websites that asked the polls Might have had a slight bias or the users that normally come onto this website Might have had a slight bias towards one of the candidates And hence all that polling data that we got from these websites Is basically kind of worthless Next I want to talk about this idea of the shy voter This is something that also popped up a lot with the brexit vote And the shy voter is someone who who wants to vote for trump But doesn't want to tell his friends that he's voting for trump Because voting for trump was kind of like associating yourself with racism and you know all these terrible things So trump supporters used to get you know told oh you're stupid or you're racist all these type of things So they would become shy they wouldn't tell people that they supported trump And even though polls were supposed to be anonymous these people may not have believed it or thought You know what let me not risk it in case somebody finds out and you know they share this poll Result on my facebook page, you know, and then i'm terribly embarrassed So once again what we're seeing is the poll mechanism Is separating these groups of people in not the right way Which means the polls are not getting the right results and we're not getting the truth But that's the problem with the polls. They also made some terrible statistical mistakes One of those is extrapolation abuse So this is another classic statistical mistake Is is that it's impossible to ask everybody, you know, who you're going to vote for, you know, America is massive There's millions and almost hundreds of millions of people voting in this election So they can't go and poll every single person So what they do is they go and they just poll a certain sample So they just ask a few individuals And then so in this case what i've got you on the screen is they ask four people Three of them are democrat one of them is saying they're going to vote for trump And then they say hey guys look at this. We've got our little sample data. This is what it's saying Let's just simply extrapolate the results And this is a dumb thing to do specifically with america where there's so many people voting That even if you poll a couple of millions of people your sample size is still going to be too small to be statistically significant Also voters tend to clump together. We kind of saw that with the states A lot of people in certain areas voted for the same party And that's something else that the polls would need to try and get around before they can just simply extrapolate Now another statistical mistake that they made Was that they said okay, we're going to take this poll data And we're also going to take historical data and we're going to build a fancy model And we are going to predict who the next president is going to be based on what the polls data is saying What economic data is saying what this data what this data, you know, they get all this information Funnily enough one of the pieces of information that they look at Is the height of the candidate and there's a study that has shown that the taller candidate Tends to win Now if this is accurate if this is a cause, okay, we're seeing that it's a correlation But if this is a cause it means next time the democrats if they want to win They just need to go out and find the tallest human being that they can find Okay, so that's if this correlation is accurate Or this correlation is merely a coincidence and it is just by coincidence that The taller candidate seems to get elected more often Now what these these guys doing who are building these models and everything like that They were looking at so much historical data that just by sheer chance they were finding factors that were correlated with the results Their big mistake though was that assuming that these factors were there for the cause So if I don't know if the length of ties had has indicated, you know, for republicans Just by chance in the past and they go and they measure the ties of people today and they say oh Well, because it's this certain measurement it must favor, you know, the democrats now You know, these are some of the silly things that we see in some of these prediction models around the elections But now there are some smart people and they did make a little bit, uh, you know more intelligent models But they were also making errors And one of them is the normal distribution error Now the normal distribution is an interesting thing I mean I spoke about it in my video back in february Where I said that we keep making this assumption that the population is normally distributed between, you know, democrats and republicans But if it is skewed then that's gonna heavily help trump and that seems to be the case So we see that the normal distribution has failed us in politics already But what these guys are also doing is they're saying, okay, we understand that the polls are not perfect And we're going to introduce something known as an error term to our model But then they go and they assume that the error term is randomly distributed on a normal distribution Okay, now why do they why do they do this? Well, because their school stats textbook says so it used to be It works nicely with mathematics. It's very nice for the formula, you know, so we just assume that it's normally distributed However, if you do a university course in statistics or, you know, you learn a little bit more about the subject You see that in reality very rarely are the error terms normally distributed They tend to have more fat in their tails And funnily enough, this is the exact same mistake bankers and credit agencies made back in 2007 When they assumed that market movements would be normal and thus they underestimated that whole risk of that financial recession So I guess the lesson here is that a little bit of statistics is dangerous You know, Alexander Pope said a little bit of learning is dangerous And we're seeing that with stats people think that they understand the basics and that they can then just, you know Blindly apply extrapolation and correlation and cause when in fact, they don't actually understand What they're doing Now this is bad. Okay, and actually we need to be We need to be aware of this and we need to you know, when we build models when we use stats We need to be aware that there are these dangers in being overconfident And we need to think through each thing before we apply them Because what's happening now is statistics is a powerful subject, but it's getting a very bad name You know, you're getting this these statements like there's three types of lies You know lies damn lies and statistics But you know, that's not true. The fact is is that people are abusing stats You know, this isn't the first time and I can show tell you that it's not going to be the last time And that's what's happened Abusing stats, we've got the wrong results and we've made the wrong predictions And now everybody's in shock. It happened with brexit. It's happened with trump And gosh, I really don't know when it's going to happen next But anyway, thanks so much for watching and I'll see you for the rest of my videos. Cheers guys


See you guys. I was at the library a couple of weeks ago, and I was walking around the science section, the public library, close to my house. And there was a book by a physicist. I think his last name is Weinberg, I'm not sure. There was a chapter on equations. And he had an interesting comment. He said that cathedrals are basically our link to the past when we think about architecture. So if you want to know something about what life was like three, four hundred, five hundred years ago, when you visit a remnant of life from that time, architecture is often one of the things that's left behind. So when you go to a cathedral or any other public building or palace or something that was built, you have a link to the time when that era expressed itself by building something. And he said that equations are kind of like that to scientists because great equations, what they do is that they provide you with a link to the past because at the time, when the person wrote that equation, some human being wrote some equation, they had some idea about what they were applying it to. And it may turn out that in the years and centuries later that it no longer applies to that because it's perhaps not really a part of that. Like for example, Maxwell's equations. Maxwell's equations were written, if you read the original paper, he's talking about ether. And so we don't talk about ether anymore when we think about physics, but Maxwell's equations were written in that timeframe when ether was the general knowledge of describing of how physical things interacted with each other through this thing called ether. But nevertheless, Maxwell's equations still apply today because they describe some physically real thing that Maxwell himself perhaps wasn't thinking about, but they apply to it. So yeah, what I liked about that chapter in his book was this concept of how great equations are a link to the past. And in science, great equations are rare things. So today we're gonna see another great equation. I think we've learned one great equation which has been the idea of the Coleman had in the very late part of the fifth decade of the 20th century, and that led to the concept of describing state estimation as a way to minimize posterior uncertainty. Today we're gonna see an equation that came about 10 years or so later from Bellman that describes the idea of how to find a policy, meaning how do you find commands that in the long term will minimize some cost, some maximization of some overall value. And the idea of coming up with a policy is based on the notion that it makes sense to have a way to respond to the states around you so that if you find yourself at this state, you do something differently than this state, and both of them are the best you could do in order to arrive at whatever goal that you have. So you have some long term goal and you don't wanna describe your actions from where you are now to the future because who knows, there might be perturbations, there may be uncertainties. So what you wanna do is to not describe all the actions but you wanna describe a policy. A policy is a way to think about how do you respond to a state. So if you find yourself here, what's the best command you could give? If you find yourself here, what's the best command you could give? So it's a feedback controller that we're gonna be describing. We have a long term goal, we find ourselves here, what's the best policy that we could do from here on in order to arrive at our destination? So a policy is more like a way, if you wanna think about it in the colloquial term, when they say honesty is the best policy, well what they mean isn't that honesty at this particular state is the best policy, is that it's the best policy in some general way of thinking no matter where you find yourself. So the way we're gonna be thinking about control in the framework that we've been describing is as follows. So we have, so today's lecture is gonna cover chapter 12.1 and 12.2 and in particular, I'm going to cover a document that I wrote to help you with understanding the Bellman equation that's on the webpage, it's just called I think Bellman equation or something like that, it's a four page or so document, you should look at it. It's unfortunately not in your book. So we have some cost that we wanna minimize. Let's see if I can find a good pen. No. So for example, we might say that by the end of the movement what we want is that from the beginning to the end from K is equal to zero, to K is equal to P, we wanna have minimization of some kind of a cost, maybe what I wanna do is to get there as effortlessly as possible, so U transpose L, U of K, some of all the commands that are produced, plus what matters to me is to be as close to the goal as possible. So the state at the end of my movement X at time point P should be as close to this goal as possible. So something like this, maybe this would be my effort cost, let's call it JU, this would be my state cost, let's call it JX, state cost. You know, I wanna be in Bahamas and the closer I am to Bahamas, the better it is, I wanna get there with as little effort as possible. So something like this. We begin with something that we wanna minimize, accumulation of some kind of a cost. And then what we have is we have a model of how our actions changes our state. So we believe that our state is gonna change based on some model that maybe looks like this, some kind of noise at the end of my model, maybe something like this, I could also have different kinds of noises here. And then I have some sensory system that measures these states. So this is what we call a forward model. So I can predict if I produce some action what's gonna happen. And then what I have is a way to estimate my state, where am I? So what I do with this is that I have a prior belief about where I am, I make a prediction about what should happen if I generate some input u and then I make some observation y. And so what I get is that I have a posterior belief that looks at the observation that I made minus my prediction about what should have been made. And this is my common gain.  And so my u is gonna be some function of my belief about where I am, my prior belief about where I am, and something that's gonna transform this into my motor commands g, sorry, u. Let's call this, in the case of this particular cost and this particular linear dynamical system, this is gonna be a linear transformation and this is gonna be my feedback control policy. And what we're after is this equation. What's the way to transform your belief about where you are to motor commands u in such a way that you minimize over the long term the cost? How should you do this? So what we have is the problem of having accumulation of some cost. We find ourselves at some state and we wanna minimize over the long term this cost. And the idea is to find a feedback control policy. Something that says you tell me where you think you are and I will tell you the best commands to produce in order to minimize this cost. So what we're after is this matrix g. Now that's for linear systems.  Today what I'm gonna do is that I'm gonna show you the problem in a nonlinear system but small enough so we can get our hands around it. So in principle, you're gonna understand basically how does this Bellman equation work? How do we find the transformation from the state to the commands u? So that's the basic idea. Any questions before I get into it? All right, so let me give you some examples of why this kind of thing is reasonable. We've been talking about saccades which are basically movements of your eye. So a typical saccade position of your eye versus time might look something like this where this is on the order of maybe 100 milliseconds.  So I'm looking, I move my eyes over there and if I do it while I'm blinking what happens is there's something that pushes down on my eye. So this action is like a perturbation. Your eyelids, when they fall on your eye they are pushing it down. And what one can do is we can put contact lenses that have a coil around them and you sit in this magnetic chamber and the position of your eye is sensed by this little wire on your eyes. And so when you move your eyes despite the fact that it's under your lid you can measure its position. So what happens then is that if you have a blink that looks like this where your eyelid is closed for a moment it pushes on your eyes. And if you look at what happens to your eye your eye goes up and does this. So it gets perturbed. But what's interesting is that it corrects some mechanism in your brain corrects for that perturbation and the reason we think is because this isn't some unexpected perturbation that's acting on your eye, you blinked. So it wasn't like me pushing down on your eyes it was you yourself.  And so this idea is that even on the course of 100 milliseconds, which is too short of a time to have sensory feedback the sensory feedback of all you had was sensory feedback from your eye muscles to tell you that your eye got perturbed. 100 milliseconds would be too short for you to respond to it and correct it. But if you yourself generate this action then some part of your brain says wait there's this motor commands that are being sent this is gonna have a consequence and I can respond to it, I can correct it because I can monitor these commands and say ah there's gonna be something that's acting on my eye. So here's an example of feedback control. It's not just some motor commands that are generated to your eye and they are blind to everything else that happens around you. If something perturbs the eye the nervous system responds to it. Another example of this is something that we did sort of like this. But instead of having people blink what we did is that we were using stimulation of the brain. So what you could do is that you could give a pulse to anywhere pretty much anywhere on the brain transcranial magnetic stimulation.  And if people are saccading right around the time that that zap takes place it appears to engage startle reflexes. And startle reflexes are complicated and if they're given considerably before a movement it makes it so that people jump but if you're giving a startle thing during a movement it seems to make that movement inhibit it. So it makes it look like basically your saccades are gonna have a motion that the eye begins moving and then it can come down almost to a stop and then it can correct itself. So this is with TMS. This is with eyelid closing. So I give you these examples because they're important from the point of view of demonstrating that the brain has mechanisms in place that monitors and corrects for actions that you're there and being produced. And it tells you that even the fastest of all movements saccades, they're under feedback control. Okay, so this feedback control is what we're gonna be studying. Feedback control, what that means is that actions that you produce depend on your estimates of state where they are. They're not just some sequence of actions that are programmed from beginning to end. It depends on where you believe you are and you generate a different feedback depending on where your body is. Final example of this that I wanna show you is something about the way you move your eyes naturally.  So when I'm looking from here to here I move both my eyes and my head. The way that works is as follows.  And this makes the sum of these two is called gaze is where you're looking at.  So when I'm looking to the left the eyes are almost centered in my head. When I'm looking to the right the eyes are centered in my head.  Okay so in the 70s my former mentor Emilio Bicci he did this experiment where in monkeys he looked at what would happen when the animal is about to make a movement and he wouldn't let the head rotate. So the eyes would rotate but the head on random trials was maintained in its position. So usually the monkey won't move its head but the head doesn't move in those cases. So you can imagine a little thing that holds the head down the break is applied the head doesn't move. And so what happens is that in that scenario the eye goes up and stays up until the head is released then the eyes come back.  No, it depends on where the head is. The commands to the head if they didn't cause any movement of the head then what happens is that the animal waits until the head moves and then the eyes come back. So the commands to the eyes are not just set in motion and then they go on by themselves. They depend on what's happening to the head. So that's the state here. This state is the state of the head and the eye. And so this command is only relevant is only gonna be the same basically. It's gonna let the eyes come back only if the head has been allowed to move. Okay, so what we're interested in is how does this gains look like? You know, how do we divide up these gains? And the way you can think about it is that when you have redundancy so we always have redundancy, right? For example, we have many muscles in our body we have many neurons. And one of the redundancies that we talked about last time was that suppose that you have two effectors, one arm, the other arm, these two arms. And so you can imagine that well, if you are doing something that depends on both arms, well then the feedback gain should make it so that when I need to respond to something if I can help with this arm I should respond with both arms. Because if you push my right arm and my objective is to maintain some location here then some of the response, say maybe the position of this thing that I'm holding is the sum of the right and the left arm. So if you just push on my right arm if I just respond with my right arm say you push me with one Newton. If I respond with one Newton back, that's fine. But if I respond half a Newton here, half a Newton here. If each arm does half a Newton and the together they produce one Newton then the effort is lower, right?  Does that make sense? So my feedback gain would be such that it would say despite the fact that this got perturbed I'm gonna respond with both of them. Because that's the best way to minimize that kind of a cost. So that's the nature of what we're after. What we're after is say that I have a state described by position of my head and position of my eye. The objective is to make the sum of these to equal to the gaze. That's the cost. I want the gaze to be on the target. And now if you hold the head in one trial what happens is that the eyes move there and they don't come back down. On the other hand, if you don't hold the head then one moves there and the other one moves there later and the two get aligned. So this thing tells us that if we could build a feedback control that does that then it is much more complicated of course but it sounds like it would be more interesting than something that says just generate these commands of the eye, generate these commands of the head and that's it. Our objective is to go after this feedback control. And what we wanna do is to minimize some kind of a cost that looks like this. Okay, any questions? All right, let me show you how to do it. So typically what we have is what's called cost per unit of, cost per step. And so for example, in the cost that I wrote up there what I have is, let's call this alpha, cost per step at time step K is equal to U at time step K times L. U of K plus in that case my cost is only at the endpoint. So it only gives me a cost as I've run out of time but in principle I could have something like this. X at time point K transpose T at time point K, X at time point K. So I took out the G because that doesn't really matter. It's just a constant. And really what you can do is you could incorporate G into X just to give you a sense of things. So for example my state, it could be my position, my velocity, my whatever you want but let's say if it's a second order state and let's second order system and then I could put the goal in here. My state could be the goal as well. That's the state. And so then if I'm interested in representing that equation I can just put T in such a way that it finds a subtraction between this and this. That matrix T. So G itself isn't very interesting. We could always incorporate that into the state. All right, so and this T has a superscript K because in some scenarios there's only a cost at the end. So all the other T's are zero and T is not zero at the end but in principle you could have a cost per step for the state. Okay, so this is a more general way of writing that cost. This is cost per step. Okay, all right. So what we wanna do is first of all assume a finite horizon. What this means is that our cost is gonna run total cost is the sum of K is equal to zero to P of alpha K. Meaning that P is the end. There's some time point at which we're gonna be done. That's called finite horizon. So the kind of problems that we're gonna be considering are called finite horizon optimal control. Now remember that on Wednesday I suggested to you that time itself is interesting. Why should something end at 100 milliseconds, right? So I suggested that there's another kind of a cost that's a cost of time. So that would be adding onto this cost and you would then get a cost for a particular P. So you say, all right, let's assume that my movement should be only 300 milliseconds. What's the optimum thing that I can do? And let's now find that for 400 milliseconds, let's find that for 300 milliseconds and you'll see what's the best time as well. So for now, let's assume that P is constant. We're not gonna change it. And we're gonna have two kinds of costs in our cost per step. There's gonna be an effort cost and there's gonna be a state cost. And the problem that we're considering is belongs to this class called finite horizon. So now what I wanna do is find the policy such that my policy is gonna be u of k is gonna be some policy pi of x of k. So you find yourself at some x at time point k.  And this pi could be non-linear. Up there I wrote it linearly, but in principle it could be non-linear function of this state. And the objective is to find this optimum policy that finds the u that minimizes this sum of alpha k, k is equal to zero to p. That's the problem statement. So how do we do it? Let me give you intuition about how this is gonna work. We're gonna begin at the end of our time, the last time point p. So we run out of time. We're gonna have a value associated with the states at the end. That's gonna be basically our cost. The farther away we are from our goal, the worse it is. That's pretty simple. Now we're gonna say step back one time point. Step back to time point p minus one. At that time point, whatever actions you take is gonna carry a cost. That cost is gonna be this cost per step. However, the result of that action is gonna be a state that you're gonna achieve. That state is gonna have a value associated with it. And the question is how do you find the policy which is the motor command that minimizes the sum of step that you've taken plus where you ended up? And so if you find that policy that minimizes that sum, which is the sum of the step cost plus where you ended up, that's the optimum policy. Then you step back two points and you say find the best motor command that minimizes this and so forth. And so the Bellman equation is a way to take our problem which is all of these time points and we're gonna begin at the end and then find for each step the optimum action. And we're gonna see that produces a sum that's the best policy. So let me see what do I, how am I gonna write it down for you? So all right, let's begin at time point P. So the last time point. And if we begin at the last time point, we've run out of time. So our only cost is gonna be this. Any action that we produce is irrelevant because there's nothing to be gained by our action. So the minimum of this function is gonna be produced no actions at all. And whatever state that I have is gonna be the cost that I'm gonna end up with. So suppose that I produce pi star of x of time point P is equal to zero.  And then what I end up with, the value of this policy at x of P is just gonna be the x of P transpose T x of P. So whatever state that I'm in, that's gonna be the value associated with it. So at time point K is equal to P minus one. What's the best thing that I can do? Well, whatever actions I perform, they're gonna carry a cost. This is gonna be the cost here. And so the U that I'm gonna produce is gonna be cost per step. It's gonna be my first thing that I'm gonna, it's gonna cost me.  And so that's gonna be the alpha at time point P minus one which is gonna be the action that I do at this time point plus whatever state I'm in. So at time point P minus one, I'm at some state. This is the action that I will produce is how much this is gonna cost me. Now that action results in a change in my state. I end up going to some other state. So I get, as a consequence of this action to a new state, as a consequence of U of P minus one, I go to X of P, right? Some state I'm gonna end up with. And this place that I end up at is gonna have a value. And that value is gonna be this. The goodness of this state, how good is it? That goodness of the state is associated with the value function. What is this value function? It is the best action that I could produce basically the minimum of this value. So the state is gonna have a value and that value is gonna be associated with the optimum policy. Yeah. It's a scalar quantity that's described by, we just saw what, so this value is associated with the policy and at this, so this is the policy to do nothing and at this state is gonna have this value. Just a scalar quantity. So, no, no, no, no, right, right. So for this policy, so what is the policy? The policy is do nothing. What is the value of that policy? For this state, it is this. Evaluated exactly, exactly. So a policy is something that tells us what we should do at any state. The value of the policy is at a given state what is the cost? So look at what I wrote. My policy at the last time point is to do nothing, but it could be something else. Whatever it is, the value of that policy is described as a number and that number tells us how good is that policy at this particular state? So this action is gonna take me anywhere and in each of those places, I'm gonna have a number. We'll do an example so you'll see what this looks like. Yeah, yeah. The one up there? Yeah. So the one up there is only at the end point. There's only one location for us so it doesn't have a superscript. Here I could have a different cost at different times. This one here? Yeah. Yeah, at the end it would be T or P. So you could have a cost that says the state at the end is gonna have some quadratic cost. I wanna be as close as possible to the center and the farther I am, but I could also have a cost associated with states at different times. In principle, it doesn't just have to be at the end. Okay, so all right. So as a consequence of generating this action, I'm gonna go to some state. And when I get to that state, what I have is this probability of X of P given that I was at P minus one and I produced action P minus one. So this is a probability that says where I'm gonna go given that I was here and I did that action.  All right, so now what we're gonna do is we're gonna multiply this times the value of the optimum policy at that position, X of P. So you generate action U, that alters your state, but the alteration of the state is probabilistic. You might go here, here, here, here, all these places. Each one of those places has a value. Think of it as a game board. Each position on the game board has a value and your actions changes your state to one of those places. Now, if it's a probabilistic thing, then it's 90% likely that when I move my piece from here to here, it's actually gonna go there but it's also possible that's go someplace neighboring states and so the value of each one of those states is gonna be weighted by the probability of actually getting to that state. So this is how good is the state to get to? How good is the state? Well, so obviously, if I get to the goal, that's great. That's a great state to be in, right? But there's neighboring states that I might have gotten to. How good are those? Well, there's a value associated with it. This is the probability of actually getting to those states given this action, right? Okay, so what we wanna do is we wanna say, this is the value of this state. This is the probability of getting to that state. You know, we wanna sum that up over all of these states, dx of p. And then what I wanna do is I wanna find the policy that minimizes alpha at p minus one plus this. So I wanna find the u argument u that minimizes the sum. This is my best action, my u star. So look what I did. I said, this is my cost per step. I produce some actions, and this is where I am. I'm at some state. I produce some action u. It's gonna take me to some other state with some probability. Each one of those states is gonna have a value. And I wanna find the action that minimizes this cost. Okay, let's do an example so you can get a sense of what I'm talking about. This is the expected value. This is the expected value of this function, right? Because it's multiplying the probability by this value function. This is the Bellman equation. It says the best action that you can do is a function of the cost per step plus the weighted sum of the values assuming that from then on, everything that you do is optimal. Let me do an example for you. Suppose that our cost per step we write it like this, jx plus ju. And suppose I have a world that has the following behavior. So suppose that I have a four by three grid. These are positions on a board. And what I can do at each one of these positions, I can perform following actions. I can move down, I can move up, I can move to the right, I can move to the left, I can move sideways diagonally and so forth. So there's a wall here. So that means that the only thing I can do here is to move down. This point I can move up or down. This point I can move up or down. This I can move to the right. I can move diagonally over to this position and so forth. So these are the actions that I can do. These are the states. So what are my costs? My costs are as follows. So this is my j of x. Regardless of the position I'm in, it's gonna cost me five, except this point, where it's gonna cost me nothing. So that's my goal. Okay? Yeah, it's just today's example. Yeah, so what about the cost of action, ju? That's equal to one, no matter what action I take, if I move zero, if I stay. What I want is a policy of x that minimizes the sum. And we're gonna use the Bellman equation to do it. So let's begin at the end. So we're gonna begin at k is equal to p at the last time point. And at the last time point, my best action is to do nothing since I've run out of time and I have, for each time point, some value function that I would like to describe. So I guess I should first write down what happens at time point k. All right, so. All right, let's write it like this. So my best action at time point p is gonna look like this. Do nothing. What's the value of this at time point p? Well, that value is gonna be jx plus ju. So for this point, I didn't move, it's just gonna be jx. It's just gonna be this. The value of this policy, here's my policy. What to do at every state? I'm not gonna do anything, I'm just gonna sit there. I'm not gonna move. The value of that policy is gonna be equal to this. jx plus ju, ju is zero, jx is just this. Okay? All right, time point p minus one. Let's consider one of these points. Let's consider this point here. So I'm gonna consider this point, this state. Let's consider some actions. Suppose I do nothing. Suppose I do this. What's gonna happen is that I'm gonna have a cost per step, which is gonna be equal to alpha of k, which is gonna be equal to the cost of u, which is zero. And then that's gonna have a cost of state, jx, which is gonna be equal to five. This cost here, right? jx plus ju, I chose to do nothing, so that's zero. I'm gonna have a cost of state five. Now, where is the state gonna take me? If I do nothing, it's gonna stay where it is. So what is the value of the optimal policy at that point? It's five. So alpha at p minus one plus value of the optimal policy at x of p is gonna be equal to five plus five, which is equal to 10. If I do nothing, that's the cost that I incur. Per step plus the optimal policy. That's if I do nothing. Okay, what if I move down? So if I do this, then what I get is alpha k is equal to, it's gonna cost me one to move plus the cost of state that I'm at, where I start from is five, right? So this, jx of k plus u of k, u k, ju of k, this is five. Now what I have is alpha of p minus one. Let me write this as p minus one as well. So where do I end up going? v of that's equal to six plus I end up at this state. If I go down, I end up at this state. What's the value of this state? It is zero. So that's six. Let me go over that again. I chose a time point p minus one to do nothing. What's the cost of that action? That action is gonna have a cost associated with the state that you start from and the action that you do. The state that you start from has a cost of five because that's where I'm starting from. The action that I do carries a cost of one. That's one. Five plus one is six. Where does it take you to? It takes me to this point. What's the value of that state under the optimal policy? The value of that state under optimal policy is zero. So that's six plus zero is equal to six. That's the value of the policy that I chose. So this v of pi at x of p minus one is this. This is the value of that policy. A time point p minus one. This is a better policy than this. Why? Because this cost here is more than this. What if I move to the right? Move right. Do this. The cost that I start from, alpha at p minus one, is I have a cost for my movement, one. I have a cost for my state, five. And then I end up to a new location. This new location is gonna have v of x of p is gonna be equal to five. So the value of my policy at time point x of p minus one is gonna be equal to 10. Okay. So, yeah. Oh, sorry, 11. Thank you. Right, okay. So if I stay, do nothing, the value of my policy is 10. If I move down, the value of my policy is six. If I move to the right, the value of my policy is 11. So what the Bellman equation says is pi star is the policy that minimizes alpha at time point p minus one plus the value of the policy, the optimum policy of the state that you go to at time point p. And in this case, the one that minimizes it is at time point k minus one. If I'm at this state, the best policy that I can have is the one that moves me down here. That's better than going to the right or better than going down. Better than staying still. Okay, so and what is the value of this policy at the state x of p minus one? So there's gonna be some number here. This number is gonna be six. There's gonna be some number there, some number there, some number there, some number there. So I will figure these out in a moment, but I'm just gonna put them as blank for now. I'm gonna have to compute these things. So the best policy at time point x p minus one has a value of six, which is a sum of the actions costs plus the optimal policy that it takes you to. So this is my pi star at x of p minus one. This is the value of the policy. All right, let me do another one for you. So suppose that I'm interested in this time point, this state there. What's the best action for that, this state over here? So I could do nothing, in which case my alpha at time point p minus one is gonna be equal to zero plus five, which is the state cost. Right, so that's just five. And where do I end up? Why end up where I stay? So I get alpha p minus one, so the value of under the optimal policy of x at time point p is gonna be equal to up there, it's five, so the sum is gonna be equal to 10. That's if I do nothing. What if I move down from that state? If I'm at this state, what if I move down? So here's what I'm gonna do, I'm gonna move down. So now what happens is that alpha at p minus one is gonna be one plus five, which is six, right? Five is the state value to begin with, one is the cost of the movement, and where do I end up? I end up in the state below it, and that state has a value of five. So the sum of these two is what? What's the value of moving down? 11. So what's better for me to stay or move down? It's to stay, it costs me more to move down. So this state is to stay, and if I do stay, the value at time point p minus one is gonna be 10, that's the optimal policy for that. Why is it that it's better for me to stay? Because it's hopeless, I cannot get to the goal. I might as well stay. Performing an action does not improve my state. See there's no difference between the value of this state and this state, there's no difference in there. So why should I move? It makes no difference anyway, I'm gonna run out of time. So the policy, the best policy for this point is to stay, not to move. It's gonna be the same for this. This is gonna be 10 as well. All of these far away places, it's best to do nothing. All right, let's consider, and it turns out, let's consider this one here, the goal state, what's the best policy there? So let's consider this point here. Suppose I decide, so here, I'm interested in this state. State row two column, column two. So let me label these. Say this is one, this is two, this is three, this is one, this is two, this is three, this is four. So I'm interested in the state associated with row two column two. So if I decide to stay, I have alpha P minus one is equal to zero plus the value associated with that state, which is zero. And then I have V where I end up under optimal policy. So if I stay, I'm just gonna stay where at this location. So that's gonna be equal to the value that state under optimal policy is zero. So alpha P minus one plus value under the optimal policy of X of P is zero. So the best thing to do is stay, obviously. If you're there, you don't want to move, no big deal. So, and if I do so, this is zero. These other states are gonna be very similar to this. So let me do this one here for you, for example. This state here, so let's do state row one column three. If I stay, I have the cost of staying alpha P minus one is gonna be zero plus the cost of the state is five. And then I have the value under the optimal policy of X of P where I end up, and that's gonna be equal to five as well. So this is 10 if I stay. If I move, in particular, if I move to the goal with a single act like this, my alpha at P minus one is gonna be one plus five. My value function at the state that I end up at is gonna be zero. So this is gonna be six. So similar to this, the optimum thing to do is to move to the goal, and if I do that, the value of that policy is six. And it turns out this is the case here as well. Six, six, six, and here's the action. So all right, if I'm near, if I'm one step away from the goal and I have one time point left, what I should do is move to the goal. What about out here? Let's pick this one here. So let's do row four, column three. Let's see what that turns out. So if I were to stay, I get alpha at P minus one is equal to zero plus five. The state is gonna have a cost five. Action is gonna have a cost zero. And where I end up with is V at pi star at X of P is the location that I'm at is gonna have a value of five. So the total cost is 10 for staying. What if I move toward the goal? So what if I do this? Say I'm considering this action. Is that like, I move from here to this state here. Is that a good thing or is that a bad thing? Well, alpha at P minus one is gonna be one plus five. The state that I end up at is this. What's the value of this state? The value of the state is up there, it's five. V of pi star at X of P is equal to five. So this is 11. So it's better for me to stay. And if I do, I end up with 10 and 10. So what did we do? We computed the optimum policy for every one of our points and my optimum policy is stay, stay, stay, stay, stay, stay, move toward goal, move toward goal, move toward goal. And the value of that policy is this. You notice that the value of the policy is not as good as the value at the last time point. So we're building up now our cost. The other thing that you notice is that whereas up there, look at the value of the policy there. It's uniform everywhere except at zero, right? Here we begin now to see a gradient. So now this says that these places are gonna have something better if you move toward the goal. Whereas these far away places are still gonna be irrelevant. You don't wanna move. So let's do it for one more time point. What is it with whiteboards? After a little while, they become like completely unusable. There's some kind of a technology flaw, it seems to me. It's terrible. All right, so let's begin. Let's write down our value function. So value out of the best policy at time point X of P is 55555 5555555, and this policy at this last time point is do nothing, do nothing, do nothing, do nothing. Do nothing, stay, stay, stay, stay, stay, stay, stay. The value of the policy at time point P minus one is equal to, at each state, I have a value<PHONE_NUMBER>, right? And my policy at each of those states is stay, stay, stay, stay, move down, move to the goal, stay, so these are my actions, right? You see it? Okay, let's do one more time point. So let's do time point P minus one, see what happens. Let's pick one state. So let me pick this point here. Let's pick an action, action. So if I do nothing, then what happens is that my cost at this time point is just gonna be five, because that's the cost of my state. And then my value of where I'm gonna end up with at time point P minus one is gonna be my current state, which is six. So this is gonna cost me 11 of doing nothing. What about the action of moving down? So if I were to move down, my cost per step is gonna be one plus five. And my value where I end up at is gonna be zero, because if I move down, that's the value of that state. That's six, this is better. So what this tells me is that at time point P minus two, my best action is gonna be, for this one it's gonna be moved down. And the value is gonna be equal to, I don't know what it is. This is gonna be six. We're gonna have to fill the rest of it in. The same way as we're gonna have to fill in the actions, we're gonna have to fill in the value of the optimal policy. Where should we do next? What should we do next? Pick a state. Which row, which column? What's rowing? All right, so row four, column three. So, this one here. Okay, so, give me the action. Diagonal. Alpha at P minus two is equal to one plus state five is equal to six, right? Value of the optimal policy at XP minus one where I end up at, so I end up at this state here, right? And that state has the value six, that's 12. So is that better than the other choices that I have? I could move up, I could move left, or I could stay where I am, right? Those are all the other options that I have. So let's take action, do nothing. Stay. Then I get alpha P minus two is five. This V is gonna be 15. Clearly it's better for me to move now, right? Now, is it better to move to the left or is it better to move up to this point here? Is this, should I move to this point or this point? This one. Either this or this, they're equal, equally good, right? So this is gonna be the best action to do. Say this one here. It's as good as to move up there and the value is gonna end up being 12. Do you see it? We break down the problem like this. And slowly, and it turns out that the best action for here is still is to stay because look what happens. If I stay at this point, I have 10 plus five, which is the value of the time per step, so I get 15. If I move, I get 10 plus five plus one, which is 16. So it's better for me to stay and the value is gonna end up being 15. Yeah. No, I decided to stay. So I didn't have any action. Yeah. I decided to stay with the other one. The other one is even worse. Yeah, it's like. Yeah, the sideways one is even worse. Yeah. Yeah. So I think this ends up looking like this. So I wanna stay here as well. This ends up costing me 15. Here, I think I wanna move because it's gonna cost me five, six, 16. So if I move, if I'm at this point, if I were to move up, it's just like this point. It makes it so that I go from one plus five plus six is 11 and I believe that six is six is 12. So I think this is gonna be 12 and this is gonna be like this. All these others are gonna be the same as before. This is gonna be a zero. This is gonna be six, six, six, six. No, no, no. So this is gonna be, at this point, if I move up, I have one plus five. Yeah, no, that's correct. Let me check my, yeah, okay. And then at this point here, this point here, the question is should I move? And I think if I move to this location, what I'm doing, let's consider that point. So this would be row four, column one. So if I stay, I have alpha P minus two is equal to five. If I, and then the value P minus one is gonna be equal to 10. That's the 15. So if I go to this state, it's gonna be six plus the value of the state that I end up at, which is six, which is 12. So it's better for me to move there. Okay, so look at how the value function changed. The value function at time point P was flat except at the goal, at time point P minus one, we have gradient that's one step away and we see that there's now this gradient forms and now this gradient has gotten bigger and it's beginning to show a difference here. So eventually what happens is that in the next time point, my best policy would be to move from here to here, stay, move from here to here, stay and finally move from the last time point down. All right, so what did we do? We said the best action that you can do is to find the action that minimizes a cost and that cost is composed of two things. The cost at the step that you're taking, the cost per step, the cost of state plus the state that you're at, it's gonna cost you something.  That location is gonna have a value. That value is defined by the optimum policy from that state. So we did a deterministic system. If it was a stochastic system, it would just be the expected value rather than the value itself.


Value theory within economics represents all theories that try to define what economic value is, where it comes from, why goods and services are priced the way they are and how to calculate some form of objective price if such a thing exists. In this video we'll be looking at two major paradigms to the theory of economic value, the standard extrinsic paradigm based around the idea of utility and the alternative non-linear paradigm of intrinsic value. The theory of extrinsic value, also called the subjective theory of value, is a theory which advances the idea that the value of a good is determined by the importance an acting individual places on a good for the achievement of his or her desired ends, what is called utility. The theory of extrinsic value posits that value cannot be measured or observed directly, so instead economists derived a way to infer underlining relative utilities from observing choices called revealed preferences. The economist Alfred Marshall put it like this, utility is taken to be correlative to desire or want, it has been already argued that desires cannot be measured directly but only indirectly by the outward phenomena to which they give rise and that in those cases with which economists is chiefly concerned the measurement is found in the price which a person is willing to pay for the fulfillment or satisfaction of his desire. An important component to this theory is what is called consistent choice, once an agent's preference have been revealed they are not allowed to change unless the properties of the good they are evaluating changes. If I prefer an apple to an orange now I cannot change my preference unless some properties of the goods changes but if all the properties stay the same and instead we add a banana to this set of choices and then I make the choice again I'm not allowed to choose the orange this time. Because in both cases I did not choose the banana it is considered not part of the optimization equation and thus cannot alter my choices. The banana is exogenous to the equation it is part of the context and within this model the context cannot alter what happens in the equation. What all of this is saying is that the value of something is in the properties of that entity and people's perception of them and you are not allowed to change your evaluation of it unless its properties change. If you suddenly notice that the apple has a little blemish on it then its properties have changed and you can change your evaluation of it and choose another option that's considered rational. Put more formally if an agent chooses x over y then whenever x and y are available she will always choose x over y. What this makes explicit is that the context cannot change the evaluation of things. Within this model the context cannot add or subtract any value as long as this condition of consistent choice holds we can then model agent's behavior in terms of an optimization over a set of choices and we will get a closed form solution. In our previous example because the banana should not have affected my choices we can abstract away from the particular context within which the choice is being made assuming that this context does not add or subtract any value to the properties of the good under evaluation and thus we don't need it in our model. Because utility is always defined through revealed preference it always exists with respect to someone or some organization. Objective value is then defined through the interplay between people's utility functions that is to say the interaction between producers and consumers trying to maximize their utility creates what is called a market and the market defines the economic value of something what we call its price. The benefits of this model to value are many, most importantly it gives us a very simple way to objectively evaluate something. This extrinsic value model gives us a very compact clean and robust mechanism for objectively evaluating things. Another important part of this idea of the market as defining price and value is the law of large numbers. It is assumed that the more people we have in this market the more information we're bringing into the equation and thus the greater the interplay between all these different subjective utility functions that will end up approximating some kind of real evaluation of the good and this is a central idea in the efficient markets hypothesis. The more people in the market and the more competitive they are the more information we will have and the closer we will approximate some kind of objective evaluation. This whole framework of extrinsic value theory is critical to achieving abstract mathematical models of markets as equilibrium points which is of course central to the whole enterprise of standard economics. Utility always defines value in relation to someone thus within this model something can't have value independent from someone desiring it which is essentially the definition of extrinsic value. When value is defined by the interaction between different well-defined utility functions this means that the thing does not have any intrinsic value. Thus the value of an entity cannot be dependent upon something else it cannot be dependent upon the context as we illustrated with our banana example previously. And this is where the theory and the data diverge. Many experiments from behavioral economics will tell us that value does in fact change depending upon the context. Not only this but value may in fact sometimes be fully defined by the context which is way outside the model that we have here. For example the value of a color is dependent upon what color is placed next to it. The perceived value of a product in a magazine is dependent upon other products in that magazine that they were never going to purchase but the price and quality of those other products did create the context for their evaluation of the item that they did purchase. With hyperbolic discounting our preferences for different payoffs is dependent upon the context of time. This phenomena that value is not just dependent upon the properties of something in isolation but also upon its context is everywhere and very intuitive to us. But incorporating context as a defining factor into value would lead us to a whole new paradigm called intrinsic value where the value of something can be at least partially independent from the value ascribed to it by some agent in isolation. With intrinsic value theory the change in value may derive from the items relations to other things and thus dependent upon those other things what we call the context. Thus this context can in fact have some value and it can add or subtract value to any individual item within that context. Allowing for this empirical fact will give us a very different model where value and preferences are dynamically changing depending upon the connections and context. Within this intrinsic value paradigm value is in how things are contextualized culturally socially environmentally and technologically and we'll take some examples of this to try and solidify it. I want to be a lawyer because it is a socially respected job. The job is giving me access to a culturally respected identity. In other words through the job I am gaining access to cultural capital. Every time another person joins this culture and believes that my job as a lawyer has prestige the value of my job goes up not because it got any better or I got paid any more but because the network that I'm getting value from has grown. I choose to wear a suit to work because that's what everyone else at my workplace does. The thing that is the suit is giving me access to social capital derived from being part of this in group. I buy a tablet computer because it will give me access to the internet. The value of these things is really in the fact that they give one access to a network that has value as a whole the internet, society, my culture, the natural environment etc. In all these examples the value is actually in the integrity of the entire network not a property of the thing. Every time another person joins Facebook it makes my account with Facebook more valuable. My account has not changed but the network's value has and because the value of my account is in its access to that network its value has gone up. This is what we call the network effect where the value comes from the network. The network is just another word for the context a set of connections that adds or subtracts value to the item. If things then have value in their connections the inverse is also true. By disconnecting them we will reduce their value. Many of the jobs that are considered undesirable are so because they only give economic value. Cleaning the street is such an activity. It has very low social status. It has no enduring value as it will be rubbish again tomorrow and you'll have to do it again.  All of these reduce the psychological or what we might call cultural capital received from the activity and due to this we will have to pay people a financial reward that is the only remuneration of value that they will get from this activity because they are disconnected from other networks that might add value to the activity. Because the value is in the connections it is both in your access to the network and the integrity of the network. If we were to disintegrate the internet by disconnecting servers or cables the value of all the devices connected would go down or if we believe the organization we form part of is without integrity the value to us of our social status within it will also be devalued and as we'll be discussing later the value of the whole system is often an emergent phenomena which requires all the parts to be working together in an integrated fashion out of this we get the emergent value of the whole system when we disintegrate the system this emergent value no longer exists and the value of all the components in the system is reduced this intrinsic conception of value is what we call well-being well-being is a very complex phenomena the value that it represents is a distributed one well-being is not the property of a thing it is what emerges out of forming part of having access to many different services ecosystem services industrial services social cultural etc but it is also in contributing to these systems co-creating value within them the value that emerges is a complex interaction of different forms of capital the value of something is not just contingent on its context within space but also its context within time the standard model is very much a static one time doesn't inherently change the value of things in the static model value just exists and people want it if they get it they will be happy it is assumed that agents will always try and achieve it at the lowest possible expense of resources and thus within this model there is no value in the process of achieving it but this is in contrast to empirical data where we often see that the value of the thing is created during the process that people go through in trying to achieve it for example in experiments that have been done with participants building things like Lego toys or origami figures the participants who engaged in this process valued those things more than those who did not engage in the process and those who had harder tasks valued the end product even more in experiments it turns out that people have to be paid more to do repetitive tasks that have no lasting outcome as soon as you add some elements of endurance to the finished product so that they feel like they are part of an enduring process this gives meaning to what they're doing and they will be more motivated being prepared to work at a lower cost this is an example of cultural capital it gives us a sense of direction and self-realization that adds value to what we're doing in cultural capital value is in fulfilling some motivation and this motivation plays out over time it is a process and the value of things is contextualized by that process the sense of achievement one gets when one goes through the full process of creating something might be an example of an emergent phenomena the integrity of that whole process adds some value to it in contrast when we divide the process up through specialization asking a worker to do repetitive tasks on a production line they become alienated from the end product and receive no value from it what I'm trying to illustrate here is that as with our previous example where value was added by the integrity of the context in space the same is true of time where the value is in the continuity of the process and that emergent phenomena adds value to any component within it like all emergent phenomena this is nonlinear the whole is greater than the sum of its constituent parts by performing or maintaining that whole process we get that emergent value value is added by the integrity of the network our standard theory of value was formalized within the context of the industrial age where the mass production of discrete tangible products was at the center of economic activity these discrete products could be easily given a homogeneous quantified price due to a lack of information this model of the market was an ideal mechanism for defining an objective value in the form of a single price metric today a number of different factors are converging to put stress on this traditional paradigm to value they include most of the major trends that we previously highlighted serviceization information technology the knowledge economy sustainability information technology is making it ever more explicit that the value is in the network of connections such as the internet or social networks in an interconnected world it is increasingly explicit that access and connections are what people want instead of the ownership of discrete tangible products information technology in the form of big data is giving us quantifiable values to all sorts of things that we never had previously how many people follow you on Twitter which is social capital how environmentally friendly a product is and how much people are prepared to pay for that which is ecological capital in big data we have the capacity to reveal the value of things and we're not just talking about aggregate values we may soon have individual personalized values associated with different individual goods and services Amazon the online retailer already uses sophisticated algorithms to adjust the price of products for different customers depending on information they have about your profile what device you're using what time of day it is and so on they may adjust the price accordingly we talk previously about serviceization we don't sell you a single product we sell you access to a service and that service is really a network that aggregates many different things in order to deliver a complex form of value to the right person in the right context added to this is the knowledge economy which requires a new skill set of intelligence self motivation meaning context creation all of which can't really be brought about with reference to extrinsic value they require the harnessing of motivations through intrinsic value in this module we've outlined two paradigms to economic value one extrinsic that is defined in terms of the value that an agent ascribes to an entity due to its properties in isolation what is called utility within this formalization value is a homogeneous variable it is derived from a single source the properties of the good or service and made manifest within prices that represent the interplay between different people's utility functions we've talked about an alternative formalization of value called intrinsic value that emerges out of the network connections that the entity is embedded within and this value changes depending on the context within this model value is heterogeneous it is derived from a network of interacting social cultural industrial and ecological capital we should maybe think about these two models as tools in our theoretical toolbox to be used when appropriate they both have their advantages and disadvantages that will make them applicable in different circumstances extrinsic models will often give us nice clean closed form solutions but we can't always use them as extrinsic models will run into the problem of market failures due to the fact that markets can't define intrinsic value the intrinsic model to value doesn't suffer market failures and it is much more representative of empirical data but it is also much more complex in many cases we don't really have the models to fully define it in order to instantiate them we're likely gonna need lots of data and again we often don't have this but with the rise of big data and advanced analytics it's very likely that we will increasingly have this in the future