From theCUBE studios in Palo Alto in Boston, connecting with thought leaders all around the world, this is a CUBE conversation. Hi everybody, welcome to this special digital presentation where we're covering the topic of data ops and specifically how IBM is really operationalizing and automating the data pipeline with data ops. And with me is Inderpal Bandari who is the global chief data officer at IBM. Inderpal is always great to see you. Thanks for coming on. My pleasure to answer your question. So, you know, the standard throwaway question from guys like me is, you know, what keeps the chief data officer up at night? Well, I know it's keeping you up at night. It's COVID-19. How are you doing? I think it's keeping all of us. Yeah, for sure. So how are you guys making out?  But what was your first move when you actually realized this was going to require a shift? Well, I think one of the first things that I did was to test the ability of my organization to work remotely. This was well before the recommendations came in from the government. But just so that we wanted to, you know, to be sure that this is something that we could pull off if there were extreme circumstances where even everybody was good. And so that was one of the first things we did. Along with that, I think another major activity that we embarked on is even that we had created this central data and AI platform for IBM using our hybrid multi-cloud approach. How could that be adapted very, very quickly to help with the COVID situation? Well, those were the two big items that my team embarked on very quickly. And again, like I said, this was well before there was any recommendations from the government or even internally with an IBM, we didn't have any recommendations, but we decided that we wanted to run ahead and make sure that we were ready to operate in that fashion. And I believe a lot of my colleagues did the same. You know, there's a conversation going on right now just around productivity hits that people may be taking because they really weren't prepared. It sounds like you're pretty comfortable with the productivity impact that you're achieving. Oh, I'm totally comfortable with the productivity. In fact, I will tell you that while we've gone down this path we've realized that in some cases the productivity is actually going to be better when people are working from home and they're able to focus a lot more on the work aspect.  Somebody like myself who had a long drive into work, you know, which I would use on phone calls, but that entire time can be used a lot more productivity in a lot more productive manner. So there is, we realized that that there's going to be some aspects of productivity that'll actually be helped by the situation, provided you're able to deliver the services that you deliver with the same level of quality and satisfaction that you've always done. Now, there was certain other aspects where, you know, the productivity is going to be effective. So, you know, my team, there's a lot of whiteboarding that gets done. There are lots of informal conversations that spark creativity, but those things are much harder to replicate in a remote environment. So we've got a sense of, you know, where we have to do some work to put things together versus where we're actually going to be more productive. But all in all, we are very comfortable that we can pull this off. No, that's great. I want to stay on COVID for a moment. And in the context of just data and data ops and, you know, why now? Obviously, with a crisis like this, it increases the imperative to really have your data act together. But I want to ask you both specifically as it relates to COVID, why data ops is so important? And then just generally, why at this point in our time?  When I joined, our data strategy centered around cloud data and AI, mainly because IBM's business strategy was around that. And because there wasn't the notion of AI in enterprise, right? Everybody understood what AI means for the consumer, but for the enterprise, people didn't really understand what it meant. So our data strategy became one of actually making IBM itself into an AI enterprise. And then using that as a showcase for our clients and customers who look a lot like us to make them into AI enterprise. And in a nutshell, what that translated to was that one had to infuse AI into the workflow of the key business processes of enterprise. So if you think about that, workflow is very demanding, right? You have to be able to deliver data and insights on time, just when it's needed. Otherwise you can essentially slow down the whole workflow of a major process within an enterprise. But to be able to pull all that off, you need to have your own data arcs, very, very streamlined so that a lot of it is automated and you're able to deliver those insights as the people who are involved in the workflow needed. So we've spent a lot of time while we were making IBM into an AI enterprise and infusing AI into our key business processes, into essentially a data ops pipeline that was very, very streamlined, which then allowed us to very quickly adapt to the COVID-19 situation. And I'll give you one specific example that will go to how one could essentially leverage that capability that I just talked about to do this. So one of the key business processes that we had taken aim at was our supply chain. You know, we're a global company and our supply chain is critical. We have lots of suppliers and they are all over the globe and we have different types of products so that has a multiplicative factors, because for each of those you have additional suppliers and you have events, you have weather events, you have calamities, you have political events. So we have to be able to very quickly understand the risk associated with any of those events with regard to our supply chain and make appropriate adjustments on the fly. So that was one of the key applications that we built on our central data and AI platform. And as part of our data ops pipeline, that meant the ingestion of those several hundred sources of data had to be blazingly fast and also refresh very, very quickly. Also we had to then aggregate data from the outside and from external sources that had to do with weather related events that had to do with political events, social media feeds, et cetera. And overlay that on top of our map of interest with regard to our supply chain sites and also where they were supposed to deliver. We'd also weaved in our capabilities here to track those shipments as they flowed and have that data flow back as well so that we would know exactly where things were. This was only possible because we had a streamlined data ops capability and we had built this central data and AI platform for IBM. Now you flip over to the COVID-19 situation. When COVID-19 emerged and we began to realize that this was going to be a significant, significant pandemic, what we were able to do very quickly was to overlay the COVID-19 incidents on top of our sites of interest as well as pick up what was being reported about those sites of interest and provide that over to our business continuity. So this became an immediate exercise that we embarked but it wouldn't have been possible if you didn't have the foundation of the data ops pipeline as well as that central data and AI platform in place to help you do that very, very quickly and adapt to it very quickly. So what I really like about this story and something that I want to drill into is that essentially a lot of organizations have a real tough time operationalizing AI and fusing it to use your word. And the fact that you're doing it is really a good proof point that I want to explore a little bit. So you're essentially, there was a number of aspects of what you just described. There was the data quality piece with your data quality in theory anyway is going to go up with more data if you can handle it. And the other was speed, time to insight so you can respond more quickly. If it's think about this COVID situation if your days behind or weeks behind which is not uncommon, sometimes even worse, you just can't respond. I mean, the things change daily, sometimes certainly within the day. So is that right? That's kind of the business outcome and objective that you guys were after. Yes. So from an infused AI into your business processing, the overarching outcome metric that one focuses on is end to end cycle time reduction. So you take that process, the end to end process and you're trying to reduce the end to end cycle time by several factors, several orders of magnitude. And there are some examples of things that we did. For instance, in my organization that had to do with the generation of metadata is data about data. And that's usually a very time consuming process. And we reduced that by over 95% by using AI to actually help in the metadata generation itself. And that's applied now across the board for many different business processes that IBM has. That's the same kind of principle that was you to be able to do that. So that foundation essentially enables you to go after that cycle time reduction right off the bat. So when you get to a situation like a COVID-19 situation which demands urgent action, your foundation is already geared to deliver on that. So I think actually we might have a graphic and then the second graphic guys, if you bring up the second one, I think this is, Inderpal, what you're talking about here, that sort of 95% reduction guys, if you could bring that up, we'll take a look at it. So this is maybe not a COVID use case. Yeah, here it is. So that 95% reduction in cycle time, improvement in data quality, what we talked about there's actually some productivity metrics, right? This is what you're talking about here in this metadata example, correct? Yeah, yes, the metadata, right? It's so central to everything that one does with data. I mean, it's basically data about data. And this is really the business metadata that we're talking about, which is once you have data in your data lake, if you don't have business metadata describing what that data is, then it's very hard for people who are trying to do things to determine whether they can even, whether they even have access to the right data on them. And typically this process is being done manually because somebody looks at the data, they looks at the fields and they describe it and it could easily take months. And what we did was we essentially used a deep learning and a natural language processing approach, looked at all the data that we've had historically over at IBM and we've automated the metadata generation. So whether it was, you know, you were talking about the data relevant for COVID-19 or for our supply chain or for our accounts receivable process, any one of our business processes, this is one of those fundamental steps that one must go through to be able to get your data ready for action. And if you were able to take that cycle time for that step and reduce it by 95%, you can imagine the acceleration across. Yeah, and I like what you were saying before, you're talking about the end-to-end concept. You're applying system thinking here, which is very, very important because, you know, a lot of clients that I talked to, they're so focused on one metric, maybe optimizing one component of that end-to-end, but it's really the overall outcome that you're trying to achieve. You may sometimes, you know, be optimizing one piece, but not the whole. So that systems thinking is very, very important, isn't it? The systems thinking is extremely important overall, no matter, you know, where you're involved in the process of designing the system. But if you're the data guy, it's incredibly important because not only does that give you an insight into the cycle time reduction, but it also, it clues you in into what standardization is necessary in the data so that you're able to support an eventual outcome. You know, a lot of people will go down the path of data governance and the creation of data standard, and you can easily boil the ocean trying to do that. But if you actually start with an end-to-end view of your key processes, and then by extension, the outcomes associated with those processes as well as the user experience at the end of those processes, and kind of then work backwards as to what are the standards that you need for the data that's going to feed into all that. That's how you arrive at, you know, a viable practical data standards effort that you can essentially push forward with. So there are multiple aspects when you take that end-to-end system view that helps the chief leader. One of the other tenets of data ops is really the ability across the organization for everybody to have visibility. Communications is very key. We've got another graphic that I want to show around the organizational, you know, in the right regime. And this is a complicated situation for a lot of people, but it's imperative. Guys, if you bring up the first graphic, it's imperative that organizations, you know, find, bring in the right stakeholders and actually identify those individuals that are going to participate so that there's full visibility. Everybody understands what their roles are. They're not in silo. So guys, if you could show us that first graphic, that would be great. But talk about the organization and the right regime there, Inderpal. Yes, yes. I believe you're going to, what you're going to show up is actually my organization, but I think it's very illustrative of what one has to set up to be able to pull off the kind of impact that I talked about, you know. So let's say we talked about that central data and AI platform that's driving the entire enterprise and you're infusing AI into key business processes like the supply chain to then create applications like the operational risk insights that we talked about and then extended over to a fast emerging and changing situation like the COVID-19. You need an organization that obviously reflects the technical aspects of the platform, right? So you have to have the data engineering arm and an AI arm as, you know, in my case, there's a lot of emphasis around deep learning because that's one of those skillset areas that's really quite rare and but also very, very powerful. So, you know, they're the major technology arms of that. There's also the governance arm that I talked about where you have to produce a set of standards and implement them and enforce them so that you're able to make this end-to-end impact. And then there's also, there's an adoption arm where there's a group that reports into me and a very, very, you know, empowered group which essentially has to convince the rest of the organization to adopt. But the key to their success has been empowered in the sense that they're empowered to find like-minded individuals in our key business processes who are also empowered. And if they agree, they just move forward and go ahead and do it because, you know, we've already provided the central capabilities. By central, I don't mean they're all in one location. In fact, we're completely global and, you know, it's a hybrid multi-cloud setup but it's central in the sense that it's one source to come for trusted data as well as the key expertise that you need from an AI standpoint to be able to move forward and deliver the business outcome. So when these business teams come together with the adoption team, that's where the magic happens. So that's another aspect of the organization that's critical. And then we've also got a data officer council that I chaired. And that has to do with the people who are the chief data officers of the individual business units that we have. And they're kind of my extended team into the rest of the organization. And we leverage them both from a adoption of the platform standpoint, but also in terms of defining an enforcing standard that helps us do both. I want to come back to COVID, talk a little bit about business resiliency. People, I think you've probably seen the news that IBM's providing supercomputer resources to the government to fight coronavirus. You've also just announced that some RTP folks are helping first responders and nonprofits and providing capabilities for no charge, which is awesome. I mean, it's the kind of thing, look, I'm sensitive to companies like IBM. You don't want to appear to be ambulance chasing in these times. However, IBM and other big tech companies, you're in a position to help. And that's what you're doing here. So maybe you could talk a little bit about what you're doing in this regard. And then we'll tie it up with just business resiliency and the importance of data. Right, right. So, you know, I'd explained the operational risk insights application that we had, which we were using internally when we COVID-19 even we were using it. We were using it primarily to assess the risk to our supply chain from various events and then essentially react very, very quickly to those events so you could manage the situation. Well, we realized that this is something that, you know, several non-government NGOs that they could essentially use the same capability because they have to manage many of these situations like natural disasters and so forth. And so we've given that same capability to the NGOs to you and to help them streamline their planning and their thinking. By the same token, when you talked about COVID-19, that same capability with the COVID-19 data overlaid on top of that essentially becomes a business continuity planning and resilience because let's say I'm a supply chain person, right?  This is likely to be affected. Let me move ahead and start making plans, backup plans, just in case it reaches a crisis level. On the other hand, if you're somebody in our revenue planning, on the finance side and you know where your key clients and customers are located, again, by having that information overlaid with those sites, you can make your own judgments and you can make your own assessment to do that. So that's how it translates over into a business continuity and resilience planning tool. We are internally doing that now to every department. You know, that's something that we are actually providing them this capability because we could build rapidly on what we had already done to be able to do that. And then as we get inside into what each of those departments do with that data, because, you know, once they see that data, once they overlay it to their sites of interest and this is, you know, anybody and everybody in IBM because no matter what department they're in, there are going to be sites of interest that are going to be affected and they have an understanding of what those sites of interest mean in the context of the planning that they're doing and so they'll be able to make judgments. But as we gain a better understanding of that, we will automate those capabilities more and more for each of those specific areas. And now you're talking about a comprehensive approach, an AI approach to business continuity and resilience planning in the context of a large, complicated organization like IBM, which obviously would be of great interest to our enterprise clients and customers as well. Right. One of the things that we're researching now is trying to understand, you know, what about this crisis is going to be permanent? You know, some things won't be, but we think many things will be. There's a lot of learnings. Do you think that organizations will rethink business resiliency in this context that they might sub-optimize profitability, for example, to be more prepared for crises like this with better business resiliency and what role would data play in that? So, no, it's a very good question and timely question, Dave. So, I mean, clearly people have understood that with regard to such a pandemic, the first line of defense, right, is not going to be so much on the medicine side because the vaccine is not even available. It won't be available for a period of time. It has to go into development. So, the first line of defense is actually to take a quarantine-like approach like we've seen play out across the world here. And then that, in effect, results in an impact on the businesses, right? In the economic climate and on the businesses, there's an impact. So, I think people have realized this now. They will obviously factor this into how they do business. It will become one of those things, this is the one talking about how this becomes permanent. I think it's going to become one of those things that if you're a responsible enterprise, you are going to be planning forward, you're going to know how to implement this on the second go round. So, obviously you'll put those frameworks and structures in place and there will be a certain cost associated with them. And one could argue that that could eat into the profitability. On the other hand, what I would say is because these are the two points really that these are fast emerging fluid situations, you have to respond very, very quickly to those. You will end up laying out a foundation pretty much like we did, which enables you to really accelerate your pipeline, right? So the data ops pipelines we talked about, there's a lot of automation so that you can react very quickly. Data ingestion very, very rapidly that you're able to do that kind of thing, the metadata generation, just the entire pipeline that we're talking about that you're able to respond and very quickly bring in new data and then aggregated at the right levels infuse it into the workflows and then deliver it to the right people at the right time. And that will become a must. Now, but once you do that, you could argue that there is a cost associated with doing that, but we know that the cycle time reductions on things like that, they can run, I mean, I gave you the example of 95%. You know, on average, we see like a 70% end to end cycle time reduction where we've implemented the approach and that's been pretty pervasive within IBM across this process. So that in a sense then actually becomes a driver for profitability. So yes, it might, you know, this might back people into doing that, but I would argue that that's probably something that's going to be very good long-term for the enterprises involved and they'll be able to leverage that in their business. And I think that just the competitive pressure of having to do that will force everybody down that path anyway. But I think it'll be eventually a good thing. That end to end cycle time compression is huge. And I like what you're saying because it's not just a reduction in the expected loss during a crisis. There's other residual benefits to the organization. Inderpal, thanks so much for coming on theCUBE and sharing this really interesting and deep case study. I know there's a lot more information out there so really appreciate your time. My pleasure. All right, take care everybody. Thanks for watching. And this is Dave Vellante for theCUBE and we will see you next time.


So, now, the question is fine I went from cardinality of log H to log to the log of cardinality of H, maybe I can do further small maybe like that just give you two algorithm right. Then if then we may ask the question is it possible like I will get something better maybe something better could be just like I can say ok instead of log to H is it like half of log to H if I can get that that algorithm is still better right that is giving that is making smaller mistake. Now, the natural question when we have this ok this algorithm better this algorithm seems to be still better still this algorithm to be still better. So, how much better we can do is there a limit to that.  If the environment become more and more and tougher right. So, as I said earlier if I have not put this visibility condition there is no way you could have gotten all these things right you would environment could have made you incur loss in every round. So, of course, what kind of bounds we are going to get it depends on how tough our environment is. We had restricted our environments capability by putting this reliability condition here ok ok fine. Even once we have put this reliability condition and we are restricted the power of environment we may want to say ok under this scenario what is the best I can do what is the best bound I can achieve is it that log to cardinality h is the best I can do or something I can go beyond this ok. So, to do this let us introduce some notion and notice that this bounds I am giving you is irrespective of how the environment is choosing x t's ok. Even though I have said that there is a fixed h star that is going to assign label I have not made any assumption that how this h star itself is chosen. The h star tuition could be very complicated and because of that you may not easy for you to figure out what is that h star ok. So, to understand that what is the best we can do maybe if we can give a lower bound on this. If we can say that ok no matter what algorithm you are going to use you will be you going to incur at least this much of loss. That means, that is like a dead end for me right like no matter how intelligent I am how best my algorithm is I will be going to suffer this much of loss. Then we will see that if at all we can derive that then we will see that whether whatever the upper bound I got was that how good it was compared to the lower bound ok. So, to do this we will just formally introduce the notion of mistake bounds and online learnability ok. Let us say this is one sequence from the environment you faced ok in round in the first round environment generated despair x 1 and associated label h star x 1 like that and in the nth round this was the one. So, notice that this is not the same sequence you are going to face every time you run a algorithm. If you are going to restart your algorithm the sequence seen by your algorithm could be altogether different ok. So, I am just saying let us say this is one sequence that to your algorithm faced and let me denote m a of s denote the maximum number of mistakes sorry the number of mistakes your algorithm made on this sequence s. So, for all my talk all the discussion we are going to fix this small n we are going to assume that my algorithm is for run for some some fixed number of rounds that is going to be n. What my interest is the number of mistakes I am going to make and now I do not want to I will be not interested in giving a bound on a particular sequence. I would like to give a bound which is irrespective of what sequence I am going to see right because I do not know what is the sequence my algorithm is going to face ok. So, then I am going to denote. So, this is the mistake my by my seen made by my algorithm a on a sequence s. Now, I am saying that I am putting it against all possible sequences and see what is the maximum number of mistakes it is going to make on any of these sequences. So, in a way I am going to take the worst case scenario here right I am looking at what is the the toughest sequence you face so far. So, toughest sequence is the one on which you made the largest number of mistakes. So, that I am going to denote by m a of h. So, h is my hypothesis class a is my algorithm. So, this is the number of the maximum number of mistakes made by my algorithm a while learning my hypothesis class h. So, then we are going to say that ok. Now, we are going to say that and I think this is ok let me rewind this. I said that taken s which has this n points in this and I am going to call this m a of s to be the number of mistakes made on sequence s and now I am going to take this supremum or s to be the maximum number of mistakes I made in any sequence and here I do not need to restrict my s to be of size n. This could be of arbitrary or arbitrary length you may have run it for n equals to 100 or you may be you might have run it for n equals to 1000 or may be n equals to 1 million. Now, we are going to say that this hypothesis class is online learnable if there exists some algorithm a. I do not know what algorithm is this for which I should be able to bound this guy what is this this guy this is the maximum number of mistakes I am going to make on any sequence if I can bound it by some b which is a constant and finite if that is the case then I am going to say online learnable ok. Now, my two algorithms like consistency algorithm and my halving algorithm were they online learning using them if I have a finite hypothesis class h can that final hypothesis class be made online learnable using these algorithms ok. So, like like like put alternatively if I have a hypothesis class which is finite in size is it online learnable according to this definition yeah let us assume this this right now we are in that reliability assumption is this online learningable according to this definition. So, what was b for my consistency algorithm? If h is finite then coordinate h is finite and I can take that value as to be my b right. So, I have a bound which is independent of what was my sequence length and for the halving algorithm I got log 2 cardinality of v which I can take it to be b. So, that is why it is ok. So, as long as I have a hypothesis class which is finite in size I am good according to this definition I am learnable. It is not like if you if I continue to use that algorithm forever I will make a large number of errors at some point you stop making errors because your number of mistakes is bounded by a finite number after that you are going to make no more mistake or just think of like if I have an algorithm you start applying on it and you keep on applying it forever after some point that algorithm will not make mistake right because you have ensured that it is mistake bound is finite ok fine. So, now we have to now answer the question what is the best we can do right is this whatever b we got earlier like cardinality h and log of cardinality h is that something better than we can we can do better or if that is the case what is the what is that we should aim for. So, for that we now look a lower bound on the number of minimum number of mistakes that the environment can force on me because see like environment is the one against whom we are learning right. If the environment becomes complicated maybe it can force more errors on me, but what is the maximum number of errors it can force on me and can I also ensure that what are the it can force maximum number of errors I will not make more errors than that. What are been enforced on me which is unavoidable ok fine that much errors I will make, but no more than that ok. So, then we will talk about what we call as any questions so far about this algorithms and the notion of whether hypothesis class is online learnable ok and we. So, see how you are stating this research and what assumption like if we do not have this reliability assumption this are all incorrect. We will in the next class we will relax that assumption like if we do not have that reliability assumption what is that we should look for ok. So, to understand this what is the lower bound we are going to get on this mistake we have to understand what is the power the environment has. So, the power environment has we we should think it in an adversarial manner. So, what is the best mistakes that the environment can force on you right. So, I mean sometimes we see it in some sports right and it is and all if you are playing against a very tough player he can force a lot of error on you right like I mean you can make self errors, but the opponent is very strong you can hit it in some spots which you which will make you error. So, it depends on what is the power of your enemy.  So, in that way we are always going to treat this interaction between environment and the learner as a game. One guys you are trying to play trying to learn your opponent and by making minimum number of mistakes and maybe like opponent let us say he is trying to enforce a more number of mistakes on you ok. Now, the question is you have ensured that through these bounds however tough is your enemy or however tough your opponent I will not going to make more than these mistakes you have ensured this, but now let us ask from the opponent's point of view or from your enemy point of view how much mistakes he can at least enforce on you that will be his question right. So, for that we are now going to understand what is the power of environment or what is the power of your opponent and remember we are going to look at the power of your opponent or environment still under the restriction that he has to follow the Relasability Assumption. We have a set up a rule for this game which is Relasability Assumption he has also has to show his power under this rule. So, to understand this we will consider little bit of this tree structure. So, consider a binary tree of depth n ok. So, if I have a binary tree of depth n how many nodes it is going to have? 2 to the power n plus 1 minus 1 right. So, let us consider for simplicity let us take the case of simply 3 nodes mean sorry at depth 2 I have a depth 2 tree here and I have 7 nodes here ok. Let us call this to be v 1. Now let us say I will come up with I am going to look at for this points I mean this v 1, v 2, v 3 they are basically sample points which we denoted as x t earlier ok these are the ones. So, v 1 is the first point, v 2 is the second point, v 3 is the third point like that. So, let us say the adversary tries to follow do the apply the following strategy. He will initially throw you v 1 let us forget what the learner is doing let us only think of from the adversary point of view or the environment point of view. He will start with v 1 this is the root node here. And then let us say he is going to apply assign some label to this. If he applies label while 0 label is going to move towards its right child and in the next round he is going to use this as his sample. If he gives label 1 he will move to this point and then he is going to use this as the next sample like this he keep on doing this. At this point after he suppose in this point he has assigned label 1 to this let us say he moved here and at this point he assigns label 0 he will move towards his left side and if we which is ok this is when I see this let us say this is my right and this is my left. So, when he is here when he applies 0 label he will going to move towards right and then he is going to take left he is going to assign label 1 and like that he keeps branching like this ok.  So, these are the node choice he has and using these node choice he has constructed a binary tree like this ok. So, now as I said his strategy is going to be he is going to choose x 1 to be v 1 to begin with ok. Now, right away let us at round t let us do at round t let us say at round t is at I t is this current node ok at something like let us say I t is this current node. Now from this node if he is going to assign a label 0 is going to go to the next child node which is towards his right and if he is going to assign value label 1 to the point at I t is going to move towards his left ok. So, let us say how to write. So, let us say if he choose x t equals to v t I think I have to use this slightly different label to be consistent ok. I will I will just flip the labels here.  So, let us try to define just this strategy of whatever this construct like this and then we will see that how the adversary is going to use this to inflict what kind of damage he can inflict on the learner ok. If this is the case at I t throughout he is going to either right or left depending on whether your label is going to be y t equals to 0 or y t 1 and I have labeled my nodes in this fashion v 1 v 2 v 3 if I have further down I will label it as v 4 v 5 v 6 v 7 then v 8 v 9 like this ok. Then if it is this case what is I t plus 1 is going to be it is going to be 2 I t. So, I am going to one stage down going to next stage. So, 2 into I t plus y t can I write this in an iteration fashion the node numberings. So, let us say let us take the simple case let us say I am in this node this is my I t let us say I made a label 1 and I came here what is this value is going to be. So, it is going to be 6 right. So, 3 it is 3 to the 6. So, when I said it is going to come this is going to be 0 right y t is 0 in this case it is going to be 3 into 2 6 we are going to get. So, like that I can iteratively like to write this. So, this is I t plus written expressed in terms of I t I can do the same thing right I can replace I t here by I t minus 1. If I keep on doing this repetition what I am going to get you can verify this. So, he can write I t plus 1 in terms of the labels. So, finally, what I have I t plus 1 express in terms of the labels I have seen till time t minus 1 right. Now, I am saying that let us say my environment is just going to use this strategy is going to start with this node we will assign a label to that based on whether you assign 0 or 1 we will go to the next one and we can continue to do this. Now, this is j running from 1 to t minus 1 right we are calculating the node at time t plus 1 that is expressed in terms of all the labels I have observed till t minus 1 ok. So, you are saying where t is missing right let me check that just to be concrete let us verify this one more time yeah let us do this. So, let us say from let us take this path can you verify this quickly. So, we are getting this to be 5 if I take this path where 0 and this is 1 this looks correct right ok. Just let me introduce one definition and then we will break. So, how many of you done again machine learning course in this rest properly hand. So, how many of you are already know the notion of VC dimension you know VC dimension which course ok anybody else. So, in machine learning especially in the supervised learning setting those of you know what is VC dimension I mean those who do not know about VC dimension kind of VC dimension tells you what is the complexity of your hypothesis class how tough it is and based on that we determine what is how many samples we need to get. So, that we can guarantee certain number of error on your hypothesis on your risk basically. We are going to use a similar notion of something VC dimension we will come to that, but as always like before we define the notion of VC dimension there is something called shattering right. So, we will just introduce the analog notion of that and we will just discuss that in the next class. So, those who do not know that notion of VC dimension. So, just read it before you come to the next class in the supervised learning setup when you look into basically if you look into the learning theory of supervised setting you will definitely come across this term shattering and the VC dimension. So, as I said we constructed basically this decision binary decision tree using this instances right. So, let us say you have the sequence of instances which forms a decision tree of depth D. We are going to say that this decision tree here whatever we have is going to be shattered if there exists a hypothesis class H such that you give any kind of sequence of labels. So, you have these points you give any arbitrary labels set of labels of D up to D then we should be able to find a hypothesis class H such that all these points here will be getting a label y t according to this strategy that we adopted that is the ith node will be selected in this fashion and when we select that ith node the corresponding node on that will get the label that you want. So, basically we are saying that this decision tree like this is going to be shattered if I can come up with a hypothesis such that you for any given you give me this sequence of labels label series right. So, this labels you are going to see I will reproduce that label by taking this path you understand this ok. So, we will discuss this more this notion of shattering it will be easier for you to digest this concept if you look into what we what is VC dimension in supervised learning ok we will revisit in the next class please read that notion of VC dimension.


Box 13, with the Star of Paramount Pictures, Alan Ladd as Dan Holliday. Box 13, care of the starred times. If you want adventure, plus a little payoff, register at the Morris Hotel as John Johnson. As soon as you've done that, take a seat in the lobby and wait to be paged. Don't ask any questions. Just follow instructions and you'll find five. Just follow instructions and you'll find five hundred dollars waiting for you. Don't ask any questions, let us set. I didn't. Not at first. But there were plenty that needed answering before this was wrapped up. And now, back to Box 13 and Dan Holliday's newest adventure, Round Robin. I'd like to make a reservation, please. Certainly double a single room. Oh, just a minute. Mr. Holliday. What's his name? Do you want a double or a single room at the Morris Hotel? Oh, make it a single. Okay. Hello? Yes, madam. That's a single room, please. It's for Mr. John Johnson. And he'll be in this afternoon before two. Thank you. We'll hold the reservation until that time. Thank you. Goodbye. Well, that's that, Mr. Holliday. Mm-hmm.  John Johnson. Why not Smith or Brown or Jones? You know, Mr. Holliday, it could just be that name is genuine. Five will get you ten. The name is a phony. Well, maybe. Anyway, there'll be a reservation for you. Oh, uh, do you want me to wait here till you get back? Hmm. Suzy, how many times have you said that? And how many times have you waited a long, long time? Lots of times. Oh, Mr. Holliday, I only hope that someday the wait isn't forever. That makes two of us. Okay, Suzy, I'm off to the Morris Hotel. Thinly disguised in name only as Mr. John Johnson. I registered at the Morris Hotel, signed the card as John Johnson, picked out a nice seat in the lobby and waited. Ten minutes went by, then fifteen. And it was easily a half hour before I became aware that a name was being called by a bellhop. Mr. John Johnson. Paging Mr. John Johnson. Mr. John Johnson, please.  Yes, that's the name I registered on our son. Thank you, sir. A message for you. Oh, thank you. Here you are. Thank you, Mr. Johnson. Want me to wait for an answer, sir? I don't know. I'll tell you in a minute. No, there's no answer. Very good, sir. Thank you again, Mr. Johnson. I was about to ask the boy who sent the message. But I figured that would be useless. What good would it do me? So I did what the note said, went to the fifth floor and knocked on the door of room 502. Yes? Who is it? Uh, John Johnson. It's Mr. Johnson. I heard you the first time. Come in. You didn't waste any time, did you? Punctual to the second. I always try to be prompt. It pays. Yes, especially in your profession. My profession? What you want is on the dresser.  On the dresser? This? Yes. Well, aren't you going to open it? Should I? You're very sure of yourself, aren't you? To tell you the truth, I'm not at all sure of anything at the moment. However, since we've apparently concluded our business, I may as well go. Yes, please do. No. Wait. Don't go out that door. Oh, the lady carries a luger. Isn't that rather large for your handbag? I'm not going to let you get away with it, no matter what the result. Well, look, maybe we'd better get something straight. I've just straightened things out in my own mind. Oh, how nice. Now, let's get around to my mind. What do we do about it? You're so sure of yourself, but I'm not going to let you do it.  You'd hold me up again and again. I'm not going to let you.  Maybe you'd better listen first. There's nothing you can say. Lady, look, with a luger muzzle putting a period right between my eyes, there's a lot I can say. I'd like to filibuster a little. You're insane. How can you stand there and speak like that? It's easy. And stay away. Please, lower the gun just a little. Don't cut any cl- You filthy! Don't do after it. Let it lie on the floor. I should have killed you right away. Please, sit down. Get out. Sit down. Now, here, take this package. Are you giving it back? Yes, I'm giving it back. Who are you? Here, take a look at this letter. While you're reading it, I'll get out a few proofs of who I am. Driver's license, auto club card, a couple of letters. I don't understand this. Obviously you don't. Here, now take the gun. I still want to use it. Who sent this letter to you? Apparently someone who wanted me to collect this package from you. Someone who didn't want to pick it up himself or herself. But why? I don't know. Maybe we'll take up an answer to that after you've explained a few things. For example, what's in the package? A diamond necklace. Yours? My husband's. It was his mother's. He gave it to me for a wedding present. Okay. Now who are you? Mrs. Therese Clemens. That doesn't mean anything to you, does it? No. But go on with your explanation. The necklace was blackmail payment. I have no money of my own. Blackmail, huh? And you thought I was the black sheep in the case? Yes, the last letter I received that I was to come here and wait, then deliver the necklace to a Mr. John Johnson. And I was to wait until someone asked me for the package. I don't understand why there are all of these elaborate precautions. Do you have any idea who's blackmailing me? Of course not. And a little delicate subject. Why? Does that matter? No, I suppose not. It's your business. Mr. Holliday, I have a 16-year-old daughter. I was willing to pay for her sake. Oh? You were also willing to kill me for her sake? I wouldn't have done it. I was trying to frighten you into letting me alone. Look, Mrs. Clemens, I've got an idea. An idea? What do you mean? No one knows what has happened in this room but you and me. Now, suppose I were to walk out of here with this package. The blackmailer would assume I had the necklace. He'd come after me, wouldn't he? You'd do that? Why not? I don't like blackmailers. They remind me of leeches and things that crawl. Why would you do this for me? Let's say it's the Luger. The Luger? Uh-huh. I also hate pearl-handled revolvers. This one is very business-like. What will you do? Walk out of here and out of the hotel without waiting for further instructions from our mysterious letter writer. Oh, meanwhile, give me your address. Yes, here. Here's my card. Okay.  So I walked out of the Morris Hotel with a fake package tucked under my arm. I hurried away, jumped into a cab, and had the driver double back and forth through the streets while I kept looking back. Yes, there was someone following another cab. And whoever it was thought I was trying the old trick of shaking a shadow. Well, it was fine with me because I was going to let him keep right on me. So I got out of my cab, ducked into a coffee shop, and sat at a table waiting. Not for long though, my boy got out of his cab and walked in. I waited until he passed my table, and... Sorry. I guess I stuck my feet out too far. Did you hurt yourself? No, no, not at all. It's okay. Forget it. But I don't want to forget it. I'm fully covered by insurance. You might have to be treated for shock. Look, mister, I said it was okay. Let's forget the whole thing. Have some coffee with me. No, thanks. But I insist. Coffee's a good stimulant.  Maybe, maybe you're hurt worse than you think. What are you, wise guy? No, you can think of something better than that. That crack went out with high-button shoes. Let go of my arm. Sit down. I'll call the cops. Please do. Right out the window, there's a large blue policeman. Tap on the window to get his attention.  I'll sit down. Oh, good. Cream and sugar? What's the idea? I was going to ask you the same thing. This, what you're looking for? I don't know what you're talking about. This package for Mrs. Clements. You're nuts. I gotta go now. No, I think I'll call a cop. Okay, mister, call a cop. That one right outside. You, you wouldn't mind? Not at all. Here, take a look at these. What are they? Credentials. I'm a licensed operative for a private detective agency. Are these on the level? Why don't you check and find out? I will. Okay. Now, if you don't mind, I'll take off. With me? Why don't you get lost? With you. Come on, lead me to your private detective agency. I'm a very curious man. This way, Holliday, we'll see the boss. What's his name? Barrett. Mr. Barrett. In here. Mr. Barrett. Yeah, Phil? Who's that? The guy you wanted me to pick up at the Morris Hotel. Well, why bring him here? All you had to do was pick up a package from him. Maybe you better ask him all about it. Okay, I'll take it from here. That suits me. See you later. You, close the door and come in. I assume you meant me. Mm-hmm. All right, let me have the package. First, did you write this letter to me? Yeah, I did. Now you want your 500. No. No, you keep it. What? I said you keep it, as payment for giving me an answer to something. Like what? Did you know what was going to be in this package? Going to be. This one is nice and empty. What's the double cross for him? For blackmail. What are you yammering about? Aren't you the nice, innocent boy? Cut it out, I was willing to pay you for a job. And I want to know why you had me pick up the package instead of one of your own men. Clients instructions. Confidential. You mean your client requested you to have the package picked up by me? Yeah. Who's your client? Sure, I tell you, just like that. Look, I didn't like the deal in the first place, but I was getting paid and I didn't ask questions. Can you prove what you're saying? To you? Yes. I don't have to. That's true. But you might have to prove it to the police. Look, I've got a license to run this agency, and I run it straight. So far, I've had no trouble with the force. So far. So far. What's that crack for? I don't believe you're telling the truth. No? Take a look at these, and then say that again. Here. A letter telling me to get you to pick up the package at the Morris Hotel. Your ad cut out of the start times.  No. Not even if I told you that that package is worth about a hundred thousand? Not even then. Aren't you even curious? I never get curious. Not even about blackmail? What? Now you're going to tell me you didn't know your client was set up to squeeze a blackmail payment? No, I didn't know. Well, that's true. There was a diamond necklace. It was blackmail payment, or it was to be. Oh. Now you get hysterical. Is blackmail so funny? No, no, no, it's not. But brother, it's going to be. What do you mean by that? Many a business.  Well, the rain's over, meaning I'm out. Exactly. I'll take it from here. On what? On what my client pays me. And something tells me the ante is going up. Now back to Round Robin, another Box 13 adventure with Alan Ladd as Dan Holliday. This was getting to be around Robin, me chasing the Robin and getting him away. It was obvious that Barrett's client didn't want to come out in the open. Barrett told me he'd never seen the client but worked through mailed instructions. Well, there was another way to go at this. See Mrs. Teresa Clements and try to find a lead from there. And if there was none, to bow out, let the police take over. So later that evening, I sat with Mrs. Clements in the library of her home. No, no, I don't know who it is. I've no idea who's doing it, Mr. Holliday.  I can't. For a few moments in the hotel, I had a little courage. We'll have some more. Don't you see Mrs. Clements a blackmail and never stops. Once he gets his hooks in, he keeps them in. What will I do?   I... I can't. It's the only way you'll have to risk it. Mr. Holliday, I... I want to tell you something. I... don't love my husband. But I do love my daughter. It's only for her sake. All right. But you're making a mistake. And I don't want to be an ar... Wait. Who's that? Charles, my husband. How would you explain me here? I'll take care of it. Now please don't say anything, please. All right. Therese. Therese. Why are you there? We're in here, the library, Charles. Therese, I want you to... Oh. Charles, this is Mr. Holliday. Mr. Holliday, my husband Charles. How do you do? My pleasure, Mr. Clements. Am I intruding? No. Mr. Holliday is an old friend. Oh, of course. By the way, Therese, there was a letter for you on the table in the hallway. I brought it for you. Thank you. Here you are. Would you stay for dinner, Mr. Holliday? Oh, no, thank you, Mr. Clements. I dropped in for only a few minutes just to say hello. Well, if you'll excuse me. Glad to have met you, Holliday. Thank you. I hope to see you again. Mrs. Clements, what's the letter? Look, this letter, it's from... The blackmailer. Yes, my last chance, he says. Mail the necklace to General Delivery, box 18. Your last chance. Look, I've got to do it. The envelopes postmarked 230 this afternoon, sent special delivery. What will I do? Mail the necklace if you want to. Is there anything left for me? Okay, that's the way you feel about it. It has to be. All right, Mrs. Clements. Goodbye. Things were beginning to get a little clear, but there was still a lot of muddy water in the way, and quite a bit of it was being stirred up by Mr. Barrett. I called his office, found out he was in, and went over to see him. What do you want now, Holliday? Can't you guess, Barrett? Look, I was just getting set to leave here when you called. I'm still set to leave, get it? Yeah, I get it. You, uh, said you ran this agency straight, didn't you? Yeah, why? Why don't you keep running it that way? What are you talking about? I think you know. You just think. Now, look, your part in this business is finished over. Why don't you back out and forget all about it? I'm only trying to keep you honest, Barrett. Meaning? I'll give you one more chance to tell me who your client is. That is confidential. That's good for you, isn't it? Yeah, it is. It gives you the chance to do a little squeezing yourself. They did, Holliday. But you guessed who your client was, didn't you? Look, I'm closing the office now, Holliday. If you want to stick around here and twiddle your thumbs, OK, but I'm going. All right. Mind if I use your phone? Help yourself. Mind handing me your phone directly? You want to split the rent for this place with me? Here. Is this a private call? You want me to leave my own office? No, stick around. Here we are. Are you sure you don't want me to leave? After all, a man has to have some privacy, or don't you know that? No, it's OK, Barrett. You'll enjoy this. Hello? Mrs. Clements, please. Hey, what are you doing? You'll find out. This is Mrs. Clements speaking. Oh, Mrs. Clements, this is Dan Holliday. Did you put that in the mail? Yes. When? Just a few minutes ago. Oh, thanks. Thanks. That's all I wanted to know. Is something wrong? Oh, no, no, no. Goodbye. Well, you've made your phone call. Now will you get out? Yeah. Now I'll get out. And Barrett? Yes. Here's hoping you don't end up in the red. Well, good morning, Mr. Holliday. This is the first time you've been to your office early in weeks. Mm-hmm. I had to make a trip to the post office in an early one. Well, I could have done it for you. I don't think so, Susie. I wanted to watch for someone. Watch for someone? Who? Nobody. Didn't you get up a little too early? Thank you, Susie. What did you mean? You watched for nobody. Just what I said. I watched a little box of general delivery, box 18. There was nothing in it and no one went to it. And why'd you watch it? For what I knew I wasn't going to see. I'll go out and get some coffee. It is early. Uh-huh. Just hand me the phone book. Oh, uh, which one? The yellow one. Does all this have to do with Mr. John Johnson? Mm-hmm. Mm-hmm. Here we are. Susie, I want you to do something for me. What, Mr. Holiday? Find out all you can about Mr. and Mrs. Charles Clements. Here's the card and their address. Well, how do I do it? Call that clinic clean at headquarters. Also, these numbers. Oh, what are they? Places where you can lose money heavily. Hello? Hello, Barrett? Yeah, who's this? This is Holiday. What do you want now? Look, Barrett, uh, you can lay off now. I told you to keep your nose out of this. You'd better do as I say.  What kind of a bluff do you think you're pulling? It's no bluff. Just see that you lay off. Goodbye. Want me to start calling these numbers, Mr. Holiday? Mm-hmm. Start calling them. And if they don't want to give you any information, get clean to do it. He, uh, he owes me a couple of favors. Where are you going now? I'll be at the number and address on that card. The Clements? The Clements. I don't like what I've got to do, but maybe it'll turn out for the best. Maybe I was wrong, but I was almost sure I was right. It all depended on what Susie found out. And if she found out what I had a hunch about she would, then I'd have the whole thing wrapped up neatly. So I went to the Clements' place and again sat with Mrs. Clements in the library. But I don't understand, Mr. Holiday. I sent the package. It should have been there. But it wasn't. I watched that box for nearly an hour. Every other box around it had mail in it. Number 18 didn't... Well, someone got there before you did. Oh, no, I got there before the post office opened. Well, then what happened? Where's your husband? Charles? He's upstairs. What do you want with him? You're not going to tell him. Well, you, uh, get him down here. But why? Please, please do us a favor. No, no, I won't. Then I'll go up to him. Wait. I'll... I'll get him down here. That's better. Oh, incidentally, don't worry about Mr. Barrett anymore. That's all cleaned up. Barrett? Yes, that's right. Now you go get your husband. Oh, if the phone rings, I'll answer it because it'll be for me. I... All right. I'll be back in just a moment. Please do. Hello? Mr. Holiday? Yeah. Oh, Susie? Uh-huh. Okay, let's have it. How did you know I'd find out all this? Just a hunch? What'd you learn? Well, there was a lot of money lost. Gambling. Then it was suddenly paid off. That's right. Okay, Susie, thanks. That's all. Oh, when will you be back? In a little while. Goodbye. Mr. Holiday? Oh, hello. Good morning, Mr. Clements. Terry said you wanted to see me. I do. And I hope you don't mind my coming here so early in the morning. No, of course not. What's this about, Holiday? Uh, diamond necklace. Yours, Terace? Yes. What about it, Holiday? Have you still got it? Well, of course. Why shouldn't we have it? Because you just don't. That's all.  Terace, who is this man? Let him talk, Charles. About what? Mr. Clements, the diamond necklace isn't here anymore. It's in the safe. You want a bet? Show you. No, Charles. It's not there. The box is empty. Empty? But you can get it back. From whom? Why is it gone? Do you want to tell him, Mrs. Clements? Yes, I'll tell him. Charles, I sold it to my gambling debts. You? What? What?  I was afraid to come to you for money. Yes, that's right. And what you said about your not loving your husband. It isn't true, Mrs. Clements. Of course not. Mind if I sit down? I seem to have lost track of things. Mrs. Clements, you had a beautiful thing worked out. Your husband is very much in love with you and you with him. But you were afraid to ask him for money to pay your gambling losses. So you were going to pretend the necklace was taken by a blackmailer. A what? Oh, good heavens. Let's have some sense. All right. I don't know what story Mrs. Clements was going to tell you about the blackmail. But she had written herself letters. Isn't that right, Mrs. Clements? Yes. The blackmail gimmick would explain the loss of the necklace.  Terace, for the love of heaven, why didn't you come to me? Well, I was afraid. I think I'll leave you two to settle this. I think you will. How about it? It's up to you, Charles. Yes, I guess it is. Let's talk it over, Terace. Good. Well, I'll see you around. Mr. Holliday, I... How did you find out? Mrs. Clements, I... I always save the explanations for my secretary. But how'd you find out, Mr. Holliday? Mrs. Clements told me herself. She didn't do anything of the kind. Well, not in so many words, Susie, but instead of getting me to pick up the necklace. That was phony. Why? Well, she was afraid of one of Barrett's men would recognize her. So she was going to have me pick up the package and then one of Barrett's men pick it up for me. Oh, and the package would be returned to her. Sure. And she'd have a witness that the necklace was picked up and taken from her.  But she wouldn't have used that gun. She had to make the whole thing look real. She was a pretty good actress. Only she didn't count on my coming back into the room and offering to help her. That surprised her. She had to think fast. And you guessed about the gambling thing? Yes, it was just a hunch. But I was sure of it when she got that special delivery letter. You see, it was postmarked 230, almost immediately after I'd seen it. Oh, what about Barrett? Oh, he guessed too. And he was going to hold Mrs. Clements up for a little deal of his own. So that's that, Susie, a nice round robin. Oh, yeah. I wonder how it'll all come out. Well, unless I miss my guess, Mrs. Clements will be very, very sorry. Mr. Clements will be very, very stern. You'll get the necklace back. She'll get it back. And I'll get back to some sleep. Good night, Susie. Listen in again next week when, through the courtesy of Paramount Pictures, Alan Ladd stars as Dan Holiday in Box 13. Box 13 is directed by Richard Sandville. And this week's original story was written by Arthur Bullen. Original music is composed and conducted by Rudy Schrager. And the part of Susie is played by Sylvia Picker. Byrne Carstensen is in charge of production. Box 13 is a Mayfair production from Hollywood. Watch for Alan Ladd in his latest Paramount Picture.


Okay, you don't need time of flight, you just need what asset, what they're going to get right to left, starting at the airfield and start the north to bottom. Consist deflection! Deflection! Two, four, five, three! Two, four, five, three! Charge two! Charge two! Elevation! Elevation! One, one, eight, seven! You're good, dude. Anything you got?  Yeah.


So I shall be discussing about principal components which probably many of you are aware of but in order to complete the things and as well as this probably I will try to give I mean a different way of looking at principal components. I think all of you may be knowing what a covariance matrix is if you have a vector x then its covariance matrix covariance matrix which are represented by sigma and suppose this vector x has here I am changing the notation slightly here see it has capital D dimensional vector let us just say then this matrix is going to be basically covariance of the first variable with itself covariance of x1 with x2 and covariance of x1 with xd then covariance of x2 with x1 covariance of x2 with x2 covariance of x2 with xd then in the last row covariance of xd with x1 covariance of xd with x2 covariance of xd with xd where this vector x curl is actually x1 to xd then this is the covariance matrix which is a capital D by capital D matrix which is a capital D by capital D matrix till now when we are talking about feature selection we had a criterion function criterion function is defined based upon some I mean some particular characteristics which we believe that the features should possess. Now in this principle components here we are not going to talk about classification here what we are going to do is we are given some capital D number of features we would like to see somehow where you have more I should say variance the places where you have more variance you are seemingly going to get I should say more variance provides there more of an idea about that particular variable or that particular I should say combination of variables basically you look at a data set like this say here you have two variables and you have a data set the variable x1 has some variance and the variable x2 also has some variance okay but probably if you take a linear combination and if the linear combination provides something like this this may have more variance than this or than this am I correct basically in principle components this is what we are trying to get that is let me explain it you have totally capital D number of dimensions there were number of dimensions is capital D then what we will see is that in this capital D number of dimensions number of dimensions is finite we are we would like to look at all possible directions for example here how many directions are there you have this direction you have this you have this like this you have directions right so basically your directions are 0 to uncountably many directions uncountably many directions you are going to have then we would like to find out that direction where it provides maximum variance what is the meaning of providing maximum variance the meaning of providing maximum variance is the following so this is your data set my direction is let us just say this axis my first direction then you take the projection of each point on to this you take the projection of each point on to this at this point when it is projected it falls here then you measure this length which is basically because this is already the x axis which is basically the x coordinate right so if it is this axis you are basically going to get the x coordinate values of each one of the points but if it is not this axis let us just say an axis like this this one then what are you going to do you take the projections and each one you measure the distance from the origin you measure the distance from the origin okay you measure the distance from the origin they will be the projected points the when this point is projected on to this then the corresponding value is going to be the distance from the origin to this one okay and similarly for this point the distance from the origin to this and something is positive another one is negative here some of this is negative and this is positive right if you are taking this as your positive side of axis this as your negative side then from here to here you are going to get negative distance from here to here you are going to get positive distance so the corresponding values are going to be that okay. So for any such direction say this is your direction then this point is projected here this is projected here then you take this thing so you take a direction and project the points on to that direction so then you will get single dimensional values for each of these points then you can calculate the variance of this are you understanding what I am trying to say you can calculate variance so for each direction you will get a value of the variance now you find out that direction for which the variance is maximum find out that direction where the variance is maximum okay how to find it out I will come to it later say suppose you have found it then you store the direction now look at all its perpendicular directions look at all its perpendicular directions now among them find the direction with maximum variance then you have two directions now look at all of all the all the directions perpendicular to these two directions all the directions perpendicular to these two directions then among them find the one which has the maximum variance like that you just go on and on and on doing it when you come to the last one that means from D you will find 1 2 3 up to say D-1 you have found D-1 directions always I mean you see you have found somehow D-1 directions then the dth direction is uniquely defined is it uniquely defined there are totally capital n orthogonal directions sorry capital D orthogonal directions you have already found D-1 orthogonal directions so capital D-1 is uniquely defined okay and there also you project the points on to that then you get the variance okay so now for each one of the points in this suppose one point is let us just say why it is a capital D dimensional point then corresponding to the first direction that is the one maximal variance you will get the corresponding coordinate value to that corresponding to the second direction for this point you will get a value to that like that corresponding to all these chosen D directions you will get a value so that is the transformed value of this vector why from the old set of axis that you are given to new set of axis we are basically going to get a new set of axis is it correct because all these are perpendicular directions and you have an origin and all these are perpendicular direction so you are basically going to get a new set of axis right these directions are called the components if you want to get small D number of components then 1 2 3 up to this small D you have to take they will be your small D components they are called principal components and there is also another term associated with this this comes from electrical engineering do you have any any one of your background in electrical engineering car honan low this is discrete car honan low expansion this is discrete car honan low expansion so why the variance is given importance variance is given importance because variance will tell you where you have the more variation in the data set and you do not want to lose the information on variances so if you have to represent the whole data set by a single component then you are choosing that principle that component with maximum variance that is why each time you are looking at the maximal variances okay now now the next question is how do you calculate this is the basic principle but then it looks to be extremely complicated you have to take a direction for which you have the maximal variance and then you take the perpendicular direction of you consider all the perpendicular directions to this chosen direction again find the one with maximal variance this seems to be a very highly cumbersome process so there is a simple way of doing it that simple way is you take the covariance matrix of this vector from which you have got all these observations and find its eigenvalues and eigenvectors find and write down the eigenvectors eigenvalues in decreasing order that is there is a very basic question here I just said write down the eigenvalues in decreasing order what happens if an eigenvalue becomes a complex number can I write it in decreasing order after all this is a matrix square matrix variance covenants matrix is a square matrix for every square matrix you can calculate eigenvalues and eigenvectors and it is not necessarily true that eigenvalues will be real they can be complex numbers also if they are complex numbers you cannot write down the eigenvalues in decreasing order or say something like that right now my question is that is it possible that for a covariance matrix the eigenvalues are complex numbers the answer is no for a covariance matrix eigenvalues can never be complex why covariance matrix satisfies several properties one of the properties is covariance matrix is a positive semi definite matrix or non-negative definite matrix positive semi definite or non-negative definite they mean the same thing it is if I read write down the covariance matrix as sigma then a prime sigma a is greater than or equal to 0 for all a if it is greater than or equal to 0 for all a not is equal to 0 vector here equality is introduced then this matrix is said to be positive semi definite or non-negative definite non-negative means it is not negative that mean it can be 0 or it can be greater than 0 non-negative is same as positive semi definite positive means strictly greater than 0 semi means you are including 0 okay so then sigma is said to be non-negative definite matrix or a positive semi definite matrix and for a positive semi definite matrix the eigenvalues and this matrix is also symmetric sigma is a symmetric matrix and positive semi definite matrix then the eigenvalues they are not only real they have to be also strictly greater than or equal to 0 this is a I mean this is a proven statement and from matrix algebra offering for sigma the eigenvalues are not only real but they are also strictly greater than or equal to 0 that happens because for a positive semi definite matrix the determinant is can you tell me what the determinant will be the determinant is actually for these matrices it is product of the eigenvalues the determinant is product of the eigenvalues so if the equality holds then there is at least one eigenvalue which is equal to 0 then that means the determinant is also equal to 0 there is at least one eigenvalue which is equal to 0 that means the determinant is also equal to 0 usually the covariance matrices are positive definite that is usually they satisfy this usually they satisfy this and if they satisfy this then this is true then that is true that means all the eigenvalues will be strictly greater than 0 okay. Now let me ask you a question corresponding to an eigenvalue how many different eigenvectors can you have we generally write corresponding to this eigenvalue you have this eigenvector my question to you is how many different eigenvectors you can have corresponding to a single eigenvalue do you have a unique eigenvector unique in the sense of the magnitude and the direction both have to be same or the direction is same the magnitudes are different can you say anything about it direction is same magnitude is different right that means suppose for the matrix sigma suppose lambda 1 is an eigenvalue then sigma x say eigen okay so x is an eigenvector so sigma x is equal to lambda 1 x and suppose I take some constant C times x then sigma of constant C times x is equal to C times sigma x this is C times lambda 1 x which is lambda 1 times Cx all right. So that means corresponding to an eigenvalue you are going to get vectors the same direction but different magnitude okay now suppose two eigenvalues are same okay before that let us just see suppose all the eigenvalues are different then can you say anything about the corresponding eigenvectors suppose all the eigenvalues are different then can you say anything about the corresponding eigenvectors here what is the meaning of corresponding eigenvectors I take only those vectors with magnitude as one I take only those vectors with magnitude as one so corresponding to an eigenvalue you are going to get basically two eigenvectors since you are going to take square root okay – 1 whole square is 1 and 1 whole square is also 1 right so plus I mean in two different directions you are I mean it is the same thing in the same axis okay so you might get two eigenvectors with magnitude as one but you take any one of them no problem similarly for lambda 2 you take one such eigenvector so for lambda d capital D also you are going to take one such eigenvector my assumption is all these lambdas are different then what can you say about the corresponding eigenvectors do you have an answer if eigenvectors are if I call them a1 a2 ad then ai prime aj is equal to 1 if i is equal to j if i is equal to j you are going to get ai prime ai which is actually the magnitude right square of the magnitude that is equal to 1 but if two eigenvalues are different here I am assuming all the eigenvalues are to be different then the corresponding eigenvectors they satisfy this property that means they are orthogonal they are orthogonal am I right ai prime aj is equal to 0 when i is not is equal to j and ai prime aj is equal to 1 if i is equal to j so if all the eigen if the if no two eigenvalues are same that means if all the eigenvalues are different this property is satisfied but if two eigenvalues are same can you say anything about eigenvectors the eigenvectors first you are going to have several problems about eigenvectors this is identity matrix what are the eigenvalues of this matrix this is identity matrix what are the eigenvalues of this matrix they are one and one they are same every vector is an eigenvector right am I correct so if eigenvalues are same then this sort of property may not hold are you understanding if eigenvalues are same this property may not hold but if I can if all the eigenvalues are different that means no two are same then eigenvectors are orthogonal and now you try to remember what I told you in the very beginning I said that you somehow find a direction now find all the directions perpendicular to that so each vector is perpendicular to all the others am I correct every if all the eigenvalues are different then eigenvector every eigenvector is perpendicular to all the other vectors because of this okay now if you take the first eigenvector a1 corresponding to this I said that you should get a real number what is that real number is a1 prime for that particular x you remember this diagram say this is the direction for this one you should take this for this you should take this again for this these are the values. So for a particular vector x the corresponding value is this corresponding to a1 corresponding to a2 the corresponding value is this corresponding to ad the corresponding to ad the value is this ad prime x these are the projected values and I was talking about variance the variance of all these values is actually for a1 it is lambda 1 for a2 it is lambda 2 for ad it is lambda d that is if I have to write it in mathematics variance of ai prime x is equal to ith 1 it is lambda i variance of ai prime x is equal to lambda i so these are all the eigenvalues and eigenvectors of this variance covariance matrix. So the eigenvalues are going to give you the variances in those directions and eigenvectors will provide you the directions since we are trying to look at the one with maximal variance so you take the eigenvalue which is the maximum and corresponding to this you find eigenvector this is your first component then I am assuming that the second one is strictly less than the first one the second eigenvalue is strictly less than the first one then I get a2 this is your second eigenvector so this corresponds to the second component now you take this and up to you take d lambda small d and corresponding to this you have the direction ad and your small d these are your d principal components these are your small d principal components and the corresponding variances are lambda 1 lambda 2 lambda d the corresponding variances are lambda 1 lambda 2 lambda d in fact principal component analysis it is used extensively because of these properties that I told you and there is also another property since I have been talking about variances is there any connection between these values and this diagonal note that every diagonal element is a variance term this is variance x1 this is variance x2 this is variance xd so is there any connection between these diagonal elements and those lambda 1 to lambda d the answer is s there is a connection what is the connection the connection is some of the variances this is the trace of the matrix sigma trace of sigma I hope you all remember the meaning of the word trace trace is the sum of the diagonal elements the main diagonal elements this is nothing but I want you to check these things I want you to check this I am not giving you any proofs for this thing but please check it summation i is equal to 1 to capital D of lambda I that means we are just summing up all the variances that we have got this is nothing but summation i is equal to 1 to D of variance of x i so basically what we are trying to do here this variance of x i the sum of variance of x i we are trying to make a partition where somehow we are just trying to keep the information about the larger variances and we are removing those things smaller variances and now what is the meaning of this this is a linear combination of the original variables right a1 x it is a linear combination of the original variables original variables are x1 x1 x2 xd and this is a linear combination of that we have taken here small d such linear combinations originally what we have is capital D such linear combinations that is the original setup where you have capital D such linear combination and these are all orthogonal to each other we have taken small d of them corresponding to the larger variances and the other capital D- small d they correspond to smaller variances and if the variance is small and if we remove those things would it create a problem or I would like to ask the question in another way variance is small how is it going to help you can you tell me if the variance is small we can replace all the values by the corresponding means are you understanding by the corresponding means because since the variance is small from the mean the distance will be very very small so we can actually replace the values by the corresponding means in that way we are losing some information I am not saying that we are not going to lose any information but the information loss is small so when the variance is small you can replace the values by the corresponding means okay so by keeping the larger variances and removing those things which smaller variances yes we are losing some information I am not denying that since the variances are small if we replace by the mean yes there will be some information loss but it is not really that much okay it is not really that much and the information loss or the loss in this procedure is actually measured by this the loss in this procedure is actually measured by this and some people may do this also that means maybe some people may take this ratio also you can measure the loss either by this quantity or this divided by this the ratio okay and so that whole theory that I was mentioning that can be easily done by looking at the eigenvalues and eigenvectors of the covariance matrix there is a theorem and proof for this relationship between those directions and the covariance matrix eigenvalues and eigenvectors of the covariance matrix that is generally available in many pattern recognition books it is also available in many electrical engineering books and I will not go into the details of that I will not go into the details of the proof of these statements if the people whosoever is interested in these things they can always go through the corresponding proofs in the books and that they can find very easily okay and this is really a popular procedure because of all these properties that I mentioned because of all these properties that I mentioned and it is used vast that just too many too many places it is used just too many places where principal components is used this also resulted in I mean in fact usually computer scientists are statisticians they have to go through this eigenvalues and eigenvectors because of principal components that is one of the reasons why statisticians are computer scientists they have to go through the literature on eigenvalues and eigenvectors because of this principal components and because of those wonderful properties of I mean the covariance matrix and when you see that there is a division that takes place the summation variance Xi is same as summation i is equal to 1 to d lambda I this is a very strong property this is a very strong property and so it just divides partitions that and that is very nice you see that is very very nice and there is a PCA LDA about which Dr. Sukhinder Das anyway he will teach okay and there are many other variations of PCA which are used at very many places and about one of them I shall take a lecture probably tomorrow okay where that is PCS are used for feature clustering where principal components are used for feature clustering in fact PCS have been used at I mean several several places one of the recent works is regarding principal components for sparse matrices a sparse matrix is a matrix where you have more 0 elements than non-0s you have more 0 elements than non-0s okay. And then that means basically your data set is such that you have too many dimensions and in those dimensions too many of them let us just say I was mentioning an example yesterday I will tell the same example today it is your your data set is something like a web mining data set that is say you have got a collection of web pages let us just say some documents let us just say you have 100 documents you have 100 documents okay. So in each document you have some sentences some words and some sentences let us just say the number of words per document is of the order of just give me some number let us just say 50 words are there of the order of 50 it may be 51 52 53 or it may be 47 48 49 or some of them may be even may be much more much smaller let us just say 50 so for each document on an average you have let us just say 50 words now you have 100 documents so so 100 x 50 let us just say 5000 words and for the sake of convenience let us assume that all these 5000 words are different even if some of them are same the number of words will be quite a bit okay so let us just say you have 5000 words and all these words are different now what we will do is that we represent a web page by a 5000 dimensional vector if word 1 is present you write 1 otherwise 0 if word 2 is present and that location write 1 otherwise 0 so your vector is going to be a 5000 dimensional vector where you have 0s or 1s like that you have 100 documents that means 100 vectors but your number of dimensions is 5000 number of dimensions is 5000 but the number of such vectors is 100 now if you have to look at the corresponding variance covariance matrix to find principal components you are going to have some problems if it is instead of 5000 in fact if you look at web pages you have too many words the number of words may be in lakhs then your variance matrix will be 1 lakh x 1 lakh such a big matrix right so but then most of the elements are 0s if you have to describe one web page where you have hardly 100 or 250 words 100, 150, 200, 250 let us just take 500 words but then your number of words that you have taken is 1 lakh you have 1 lakh words among which 500 are here so the rest all of them are 0s so your data matrix is basically a sparse matrix and nowadays many data sets are like this and for these sort of data sets if you have to do all these things then you may have to develop some new methods the reason is that whenever we do this sort of thing we assume there is an inherent assumption that the number of points is much more than the number of dimensions the number of data points is much more than the number of dimensions whereas for many real life problems that may not be true for many real life problems the number of data points may be much less than the number of dimensions I have mentioned for web mining data set okay and you have many many other data sets many data sets involving bioinformatics you have cancer patients you have many gene expression data sets where again the number of dimensions is of the order of 2, 3, 4,000 but the number of points may be of the order of 100 or 150 or 200 so these are some of the latest problems where when you are trying to apply principal components you may face some problems because of since the number of dimensions is much more than the number of data points then and your computer may not be able to support finding a finding Eigen vectors for a let us just say 5000 by 5000 matrix your computer may not be able to support it but the same thing for a 100 by 100 matrix probably your computer can support it so sparse data and sparse matrices are occurring many times and in many applications in real life so there are some papers where somehow people are trying to find the principal components for when you have sparse matrices there is one paper by Tipshirani in this regard I think that appeared in one of the statistics journals Tipshirani is a famous person working in machine learning and I hope by now you know that many of these things are we are calling it pattern recognition some people are calling it data mining some people are calling it machine learning and some people are calling it artificial intelligence so many of these things are actually occur in too many disciplines okay and Tipshirani and a few such others statisticians they call themselves as machine learning people so they are working on this thing some papers are already published and there are several problems related to principal components in very very high dimensional data sets because your computer may not be able to support such high dimensional I mean finding eigenvalues and eigenvalue vectors for such high dimensional matrices I am stopping it here if you have any questions please ask me no more questions