That couldn't get in, thank you outside. We begin by telling you what no other candidate for president will tell you. And that is that given the realities of the economy and the political life of America today, no president, not Bernie Sanders or anybody else, is capable of dealing with the many crises that we have. The only way that we transform this country is through a political revolution. What that means is that millions of people throughout our country, many of whom, by the way, have given up on the political process. They look at Washington, they look at their state capitals, and they do not see efforts there to address the real problems they face. They say, hey, I don't want to get involved in this charade. And young people, many of whom have never been involved in the political process, what we need to do now is to see millions of people jump into the political process and understand that when they stand up and fight back, they are very powerful people. You know, when we began this campaign almost a year ago, we had no political organization. We had no money. We had no name recognition out of Vermont. And we were considered to be a fringe candidacy by the media. They understood that my hair really looked great and that I had a GQ look which made me president. Despite all of that, they considered us to be a fringe candidacy. And not only that, we were taking on and are taking on Wall Street. We're taking on corporate America. We're taking on super PACs. We're taking on the corporate media. We are taking on virtually the entire democratic political establishment. And in the Clinton organization, we're taking on the most powerful political organization in the United States, an organization that has elected Bill Clinton twice, Hillary Clinton ran a very strong campaign in 2008. That is what we were taking on. Well, you know what, a year has come and gone. And as of today, we have now won 17 primary and caucus states. With your help, we're going to win here in Indiana next Tuesday, 1,350 delegates to the Democratic Convention. When we began this campaign, we were 60 points behind Secretary Clinton. Well, in the last few weeks, several of the national polls actually have us leading or just a few points behind. And most importantly and most interestingly, if you look at the last half a dozen national matchup polls between Secretary Clinton and Donald Trump, Bernie Sanders and Donald Trump, in every one of these polls, we are beating Trump badly and we are beating him by larger margins in every instance than Secretary Clinton. The political reason for that is pretty obvious and that is this campaign is getting the support of the vast majority of Democrats, but we are also reaching out to independence. And the truth is that Democrats will only win if we bring in a number of Democrats onto our side and many of them are prepared to do that. The reason that this campaign is going to win is because we are doing something very unusual in contemporary American politics. We're telling the truth. And let me just suggest to you that the truth is not always pleasant. It is not always easy to hear what we want to hear, not in our personal lives and not in our political lives. And I believe that if we continue to sweep under the rug the most important problems that we face, this country will never ever be able to go forward. And that's what this campaign is about and what the political revolution is about. It is having the courage to face up to the real problems that we face and then address them effectively. Let me mention to you some of those problems, problems that we don't talk about enough, very rarely talked about in the media. Problem number one is today we are living under a corrupt campaign finance system which is undermining American democracy. Democracy is not a complicated process. What it means is you have one vote, you got one vote, you got one vote. One person, one vote. That's democracy. Democracy is not about you having a vote and him spending several hundred million dollars in a campaign through a super PAC to elect candidates who represent the wealthy and the powerful. That is not democracy. That is called oligarchy. And one of the issues, one of the great issues that we have got to address is that politically, economically, and in terms of media ownership, we are seeing fewer and fewer wealthy and powerful entities controlling the United States of America. What democracy means to me is that our country has one of the largest voter turnout rates, one of the highest voter turnout rates of any major country on earth, not one of the lowest voter turnout rates.  Our job is to think big. Our job is to think outside of the status quo. Our job is to think of a democracy when 75, 80% of the people vote, not 40 or 50% of the people voting. Our job is to say that of any person here tonight, whether you're progressive or conservative or a moderate. If you want to run for public office, you should be able to do so without having to beg wealthy people for campaign contributions. So let me repeat what I have said many times before. If I'm elected president, no nominee of mine to the U.S. Supreme Court will get that nomination unless he or she is crystal clear in telling the American people they will vote to overturn Citizens United. So issue number one is to create a democracy now in which ordinary people have the power, not just a handful of billionaires. But that's not all that we have got to do. We have got to understand that today not only do we have a corrupt campaign finance system, we have a rigged economy. A rigged economy is when the top one-tenth of 1% now owns almost as much wealth as the bottom 90%. A rigged economy is when the 20 wealthiest people, 20 people, own more wealth than the bottom 150 million Americans, half of our population. A rigged economy is when one family, the Walton family who owns Walmart, one family owns more wealth than the bottom 40% of the American people. But it is worse than that because a rigged economy is when the wealthiest family in America pays its employees wages that are so low that many of these workers are forced to go on Medicaid and food stamps. And the middle class ends up paying higher taxes to subsidize the low wages paid by the wealthiest family in America. That is a rigged economy. I hear a lot of Republican senators and Congress people, they go all over the country and they talk about how people are ripping off the welfare system. Well, let me tell you that the major welfare beneficiary in this country is the wealthiest family, the Walton family. And I say to that family, I say to that family, the wealthiest family in America, get off of welfare, pay your workers a living wage. This campaign, again, is thinking outside of the box, thinking outside of the ways and the options that television or Congress wants you to think. Question, I want you to think about this one. 40 years ago in this country, before the explosion of technology and cell phones and space age technology and all that stuff, before the explosion of the global economy, one person in a family, one person could work 40 hours a week and earn enough money to take care of the whole family. The question that we have got to ask is how does it happen that with all of the technology and the increase in worker productivity, why today does it take mom working 40 hours a week, dad working 40 hours, the kids working, and they still don't have enough money to pay the bills? What we have been seeing in this country, again, not talked about terribly much in the media, is that for the last 30, 40 years we have been seeing the middle class shrinking and shrinking and almost all new wealth and income going to the people on top. In fact, in the last 25 years, the top one-tenth of 1% has seen a doubling of the percentage of wealth they own. There has been a massive transfer of trillions of dollars from the hands and the pocketbooks of the middle class to the very wealthiest people in this country. Here in the state of Indiana, for example, since 2007, real median income has gone down by more than $4,700. And Indiana is not the only state in our country where that is happening. Are you guys ready for a radical idea? Here is the radical idea. At a time when we have massive levels of income and wealth inequality. At a time when our people work the longest hours of any people in the industrialized world. At a time when we have the highest rate of childhood poverty of almost any major country on earth together, we are going to create an economy that works for all of us, not just the 1%. Now, if we were a poor country, and there are, God knows, too many impoverished countries all over the world, we would have a certain type of discussion. And the discussion would be sure we want great health care, sure we want great education, sure we want a great infrastructure. But we're a poor country. We just can't afford to do that. Brothers and sisters, we are not a poor country. We're the richest country in the history of the world. But nobody knows that because all of the wealth and income or almost all of it is going to the people on top. Now, that means that we've got to take a hard look at what goes on in America today. Question. Any psychologist here? Any psychologist? There we go, okay. Every psychologist who has studied the issue will tell you that the most important years of the human being's development are zero through fours, all right? All right. That is when we develop intellectually and when we develop emotionally. Think for a moment. Mom is out working. Dad's out working. Who's taking care of the kids? Well, all over this country, mom and dad are desperately trying to find quality, affordable childcare, and in many cases, they can't. Turns out that there are childcare workers, people who have the responsibility of providing the intellectual and emotional nourishment that the youngest Americans need. Their wages are lower than McDonald's workers. If we want the young generation to grow, to blossom, to be the kinds of young people that we know they can be, we owe them much more. We owe them a first-class childcare pre-K system. Is that, is that a radical demand to say that our young children, the future of this country, deserve quality childcare and their parents, deserve the right to know that when they go to work, their kids are well taken care of. I don't think that that is a radical idea. We need in America, when we think about our economy, if you go out to work and you work 40 hours a week, in America, you should not be living in poverty. But in America today, and again, this is something you don't see on the media, and the reasons I want you to be thinking about that as well. Why what gets on media and what doesn't get on media and who makes those determinations and for what reasons. But in America today, the wealthiest country in the history of the world, 47 million people are living in poverty. Here in your great state of Indiana, 330,000 children are living in poverty. 16,000 public school students in Indiana were homeless during the 2012-2013 school year. 16,000 children in this state were homeless. Why is that? Why do we tolerate that? Why do we accept that? And together, we are going to create an economy which will not accept that. But if you are working in this country and you're making eight, nine bucks an hour, you're not making it. There's nobody can make it on eight or nine dollars an hour. And that is why we have got to raise the minimum wage to $15 an hour. There is an issue of huge consequence that we have very little discussion about. And that is our disastrous trade policies that have cost us over the years millions of good-paying jobs. Since 2001, last 15 or so years, in our country, we have lost 60,000 factories and millions of good-paying jobs. Not all of that is attributable to trade policy, but a lot of it is. Policies like NAFTA and CAFTA and permanent normal trade relations with China were written by corporate America for one reason and one reason alone. They would determine not to pay people in Indiana or workers in Vermont 20 bucks an hour or 25 bucks an hour. They would determine to shut down plants in this country, move to Mexico, move to China, pay people their pennies an hour, and bring their products back into this country. Since the passage of NAFTA in 1994, Indiana has lost 113,000 good-paying manufacturing jobs. In 2014, Caterpillar threw 250 workers out on the street in Lafayette and moved its engine manufacturing plant to China. And in February, Carrier and United Technologies announced it would be shutting down two manufacturing plants in Indiana, throwing 2100 workers out on the street and moving to Monterey, Mexico, where they can pay workers $3 an hour. And that is a company that last year made a profit of $7 billion and a company that paid its CEO over $14 million in total compensation. But that is the type of corporate greed that is destroying the American middle class. And I say to corporate America that if I'm elected president, those days of greed are going to end. You're not going to throw, you are not going to throw American workers out on the street and then move to China or to Mexico. When we talk about our economy, we have to talk about the fact that our infrastructure is in a state of collapse. And again, I ask you, how does it happen that in the wealthiest country in the history of the world, our water systems, our wastewater plants, our roads, our bridges, our rail system, airports, levees, and dams are in massive need of repair, and in many cases are crumbling. Let me read you something, just a recent article. This is what it says. The United States is on the verge of a national crisis that could mean the end of clean, cheap water. Hundreds of cities and towns are at risk of sudden and severe shortages, either because available water is not safe to drink or because there simply isn't enough of it. I was in Flint, Michigan a few months ago, talked to the parents of children who have been poisoned through lead in the water. But it is not just Flint, Michigan, it is hundreds of communities throughout this country. And it's not just our water systems, it is other parts of our infrastructure. And that is why if I'm elected president, we are going to invest a trillion dollars in rebuilding our roads, our water systems, our wastewater plants. And when we do that, not only do we make America more productive and safer, we create 13 million good-paying jobs. This campaign is going to win because we are listening to the American people and not just wealthy campaign contributors. And I come before you with a great deal of pride in telling you that I am the only candidate out there for president who does not have a super PAC, does not want a super PAC. We made a decision early on in our campaign that we were not going to ask the drug companies or the fossil fuel industry or Wall Street for money. We don't want their money. We don't need their money. Instead, we did something very unusual. We said to the American people, if you want to support a campaign that is going to support the middle class, that is going to stand up to the big money interests, help us out. And what happened is over the last 11 and a half months, we have received more than 7 million individual campaign contributions. Anyone know what the average contribution is? All right. That's right, 27 bucks. And that in itself is revolutionary. Because what we are showing is we can run a winning campaign without being dependent on big money interests. Democratic opponent Secretary Clinton has chosen to raise money in a different way. She has several super PACs and has received $25 million in a recent reporting period from special interest, including $15 million from super PACs, from $15 million from Wall Street. And in addition to that, as some of you may know, she has given speeches on Wall Street for $225,000 a speech. I believe that if you give a speech for $225,000, it must be a pretty fantastic speech. Must be a speech that will probably solve all of the crises facing humanity. Must be a speech written in Shakespearean prose. So I kind of think that it would be great for the Secretary to share that speech with all of us so we can learn from it. Now, Secretary Clinton says, well, she is prepared to release the transcripts if other people do the same. So no, no, no, no. Are you ready for a major, major announcement? There it is. Hold your breath. It's coming. I am prepared tonight to release all of the transcripts of all of the speeches that I have given to Wall Street. Are you ready? Here they are. Of course, I am being, of course, I'm being a little bit silly. Wall Street has not yet invited me to give a speech for $225,000, but I've got to tell you I've got my cell phone on. I'm waiting. I'm waiting for Wall Street to call. And I don't want $225,000, I don't want $2,000, but I do want and would like the opportunity to tell Wall Street what they have done to the American people. In my view, as a result of the unquenchable greed of Wall Street, as a result of their recklessness and their illegal behavior, this country was plunged, as you all know, into the worst economic downturn since the Great Depression. And as a result of that, millions of our fellow Americans lost their jobs, and by the way, many have not yet recovered. Many have lost their homes, and some lost their life savings. That is what Wall Street greed did. And if I were to go before Wall Street, what I would tell them is they were bailed out by Congress against my vote, because the leading banks, because the major banks were too big to fail. In other words, the perception was that if they were allowed to go down, they would take half of the economy with them. Well, it turns out that today, three out of the four largest banks in this country are bigger than they were when they were bailed out because they were too big to fail. In my view, when you have a handful of giant financial institutions and the six largest banks in this country issue two-thirds of the credit cards and one-third of the mortgages, when you have a small number of people with so much economic and political power, we have got to break them up. We need a financial system that works for small and medium-sized businesses, that works for consumers, not just a financial system that is an island unto itself only concerned about their own profits. And I want to take a moment to tell you a short story, which really will kind of encapsulate everything that is wrong with our economic and political system. Some of you may have noticed that two or three weeks ago, Goldman Sachs, one of the major financial institutions on Wall Street, reached a settlement with the United States government and paid $5 billion. They reached that settlement because they were guilty of selling worthless packages of subprime mortgage loans. They were lying to investors about what was in those packages. They paid a fine of $5 billion. That's point number one. And you've got to think about that for a second. Wall Street is the most powerful economic entity in the United States of America, and virtually all of the major banks have a business model which is based on fraud. Number two, when we talk about the political power of big money interest, Goldman Sachs itself over the last 25 years has had two executives from their company, from Goldman Sachs, become secretaries of the Treasury. And many other people from Goldman Sachs have assumed very important governmental positions under Republican and Democratic administrations. That is called the revolving door. You know what the revolving door is? You go from Wall Street to government. You do Wall Street's business while you're in government, then you leave and you go back to Wall Street and you make a whole lot more money. Number three, like many other major Wall Street and powerful corporate interests, Goldman Sachs has made millions and millions of dollars of campaign contributions and in fact have paid certain speakers $225,000 for an hour's work. All right. Number four, point number four. So you're looking at a fraudulent Wall Street. You're looking at a political system in which power rests with powerful financial institutions. You're looking at their political power coming from major campaign contributions and speaker's fees. But the fourth point that I want to make probably gets the American people angriest of all. Some kid in Indianapolis tonight gets picked up for possession of marijuana.  And that's not a joke.  Have you ever been arrested? Well, yes, sir. I was arrested. Well, I got somebody else interested in that job. Not an insignificant thing, but I want you to consider this. If you are the executive of a major Wall Street firm whose illegal behavior destroyed the lives of millions of Americans, you don't get a criminal record. Not at all. You get an increase in your compensation package. And that is why we need to bring justice back to a broken criminal justice system. The word has got to go out. The word has got to go out from one end of this country to the other, that it is not just low income kids who are going to get criminal records. Even if you are powerful and a billionaire, if you break the law, you are going to pay the price. And while we are on the subject of criminal justice, let me say this. Every person in this room, in this state, in this country should be embarrassed that we have in this great country more people in jail than any other country on Earth. We've got to think that one through. Because we're talking about a corrupt campaign finance system, a rigged economy, and a broken criminal justice system in which today we are spending $80 billion every year to lock up 2.2 million Americans, disproportionately African American, Latino, and Native Americans. Together we are going to end that international embarrassment. I want this country not to have more people in jail. I want this country to have the best educated population in the world. Now, right now, and again, this is an issue, many of these issues I'm talking about, you don't see on TV, they're just not talked about, but they are the most important issues. Today, right now, throughout this country, youth unemployment, kids 17 to 20 who graduated high school, kids are white, 33% unemployed, underemployed. Latino, 36%, African American, 51%. Together what we are going to do is what makes sense from a human perspective and from a cost-effective perspective. We are going to invest in jobs and education for those kids, not jails or incarceration. Think of an America, not where half of our children, our kids, are hanging out on street corners, unemployed or underemployed, but an America where those young people are getting the job training they need to build the affordable housing and the quality schools that their communities deserve. When we talk about criminal justice, it is also important to talk about major police, local police department reform. I was a mayor for eight years in Burlington for a month, and I work very hard with my local police department, and I work hard with police offices and organizations all over the country. Truth is that the overwhelming majority of police officers in this country are honest, hardworking, and doing a very difficult job. But as in the case of any kind of public official, when a police officer breaks the law, that officer must be held accountable. We have got to demilitarize our local police departments. We have got to make police departments look like the diversity of the communities they serve. We have got to end private corporate ownership of jails and detention centers. It is a very perverse incentive to enable private corporations to make money by jailing fellow Americans. So we are going to end that. And the other thing that we have got to do, and this is a bit controversial, but we have got to take a hard look at the so-called war on drugs. For over the last 30 years, millions of Americans have received police records for possession of marijuana, and today, under the Federal Control Substance Act, marijuana is listed as a Schedule I drug right next to heroin. And that is why I have introduced legislation which will take marijuana out of the Federal Control Substance Act. Now, whether or not marijuana becomes legal is a statewide decision. Four states in D.C. have reached that decision, but possession should not be a federal crime. This campaign is listening to women, and women are tired of earning 79 cents on the dollar compared to men.  This campaign is listening to young people, and young people are asking me how does it happen that when they do the right thing, when they do what their parents tell them, what their teachers tell them, when they do what their community tells them, go out and get the best education you can. Why are they ending up $30, $50, $70,000 in debt? And the other point is, 40, 50 years ago, you had a high school degree. You were pretty well educated. You can go out and get a good middle class job. But the economy and technology have changed. People today need more education. And that is why, you know, when you think outside of the box and outside of the status quo, you come to the obvious conclusion that public education today in 2016 cannot simply be first grade through 12th grade. That was great 50 years ago, 30 years ago. Today we need to make public colleges and universities tuition free. Now think about it for a second. Is this a radical idea? It really isn't. The world has changed. People need more education than they used to. And if we believe in public education, it means we make public colleges and universities tuition free. Now, furthermore, all of you know that in Germany and Scandinavia and countries around the world, they're doing it today. Those countries are smart enough to understand that it makes a lot of sense to be investing in their young people and the future of their country. We also have to deal with the crisis of student debt. And that means that for the millions of people who have that debt, those debts today, we have got to allow them to refinance those loans at the lowest possible interest rates they can find. Now, I am criticized for thinking too big. This is just too radical an idea. Bernie, you're Santa Claus, you're giving away all these free things. How are you going to pay for it? Well, let me tell you exactly how we're going to pay for it. 2009, the United States Congress bailed out Wall Street. I believe that now, with Wall Street doing just fine, it is time to impose a tax on Wall Street speculation. And that will bring in more than enough money to make public colleges, universities, tuition free, and substantially lower student debt. The campaign is listening to the African American community. They're asking a very simple and intelligent question, and that is, how does it happen that we have enough money to spend trillions of dollars on a war in Iraq that we never should have gotten into? A war which I vigorously opposed, but how does it happen that we have trillions of dollars for war but we don't have sufficient funds to rebuild inner cities throughout this country? In Flint, Michigan, children were drinking poison water. In Detroit, their public school system is on the verge of a financial collapse. In Baltimore, Maryland, tens of thousands of people are heroin addicts and not getting the treatment that they need. And in inner cities all over this country, kids are living in extreme poverty, in homelessness, and certainly not getting the healthcare that they need. If this country is capable of investing in the infrastructure of Iraq and Afghanistan, we sure as hell can rebuild inner cities in this country. It is called getting our priorities right. Less money for war, no tax breaks for billionaires reinvesting in inner cities in America. This country is listening to the Latino community. And what they are telling me is there are 11 million people in this country who are now undocumented. Many of them are being exploited every single day by their employers because when you do not have legal status, your employer can cheat you and you have no recourse. And that is why I believe that we need comprehensive immigration reform and a path toward citizenship. This campaign is listening to a people whose pain is very, very rarely heard and that is the Native American community. Anyone who has studied American history knows that from before the time we were a country when the first settlers came here, the Native American people were lied to, they were cheated, and many of the treaties they negotiated and signed were broken. We owe the Native American people more than we can ever repay. Among so many other things, more than anyone else, they have taught us that we as human beings are part of nature, that we have got to live with nature.  That is what they have taught us. But yet, despite the enormous contributions they have made to our country on reservations throughout this country and in Native American communities, unemployment and poverty are sky high. The suicide rate of young people is beyond belief. If elected president, we will change our relationship to the Native American people of this country. I talked a moment ago about the greed and irresponsibility and illegal behavior of Wall Street and what they have done to our economy. Let me talk for a moment about the greed and irresponsibility of the fossil fuel industry. I am a member of the U.S. Senate Committee on the Environment. And in that capacity, I have had the privilege of talking to some of the great scientists in our country and around the world.  And what they have also told us is that we have a very short window of opportunity to move aggressively, to move boldly, to transform our energy system away from fossil fuel to energy efficiency and sustainable energies. This is no small thing. This is like the future of the planet. And we have, as custodians of this earth, we have a moral responsibility to work with China, Russia, India, countries all over the world in telling the fossil fuel industry that their short-term profits are not more important. Now, let me just connect a few dots here and tell you again something that the media, for obvious reasons, does not talk about. And that is how does it happen that we have a major political party, a party that controls the U.S. Senate today and the U.S. House, and yet not more than a handful of Republicans in Congress are prepared to tell you and the American people that virtually the entire scientific community has concluded. And that is that climate change is real and threatening the planet. Now, why does this happen? How does it happen? Is it because Republicans are dummies? The answer is no. I know you're going to say yes, it's no. Now, I've been on committees with Republicans. We talk about healthcare and Alzheimer's and diabetes, and they ask very sensible questions. The answer here ties into a corrupt campaign finance system. Because today, the day that a Republican comes here at IU and says, you know what, I've talked to the scientists, I've read the literature, this climate change is serious, we've got to do something about it. On that day, the Koch brothers and the other people in the fossil fuel industry cut their campaign funds. And that is why we have got to tell the Republican Party, listen to the scientists, not your campaign contributors. And by the way, when we, not if, but when we together transform our energy system, when we create a modern transportation system that takes cars and trucks off the roads, when we weatherize our homes and our buildings, when tens of millions of Americans have solar panels on their rooftops, and when wind energy becomes far more prevalent than it is today, when all of that happens, we're going to create millions of good paying jobs. And one of the differences between Secretary Clinton and myself is that I believe this crisis of climate change is so significant, is so threatening to our planet that we have got to act boldly. That is why I have introduced the most comprehensive climate change legislation in the history of the Congress, which includes a tax on carbon. And I would hope very much that Secretary Clinton would understand, given the severity of the crisis, that that is exactly what we have to do. Now this campaign is asking people again to think outside of the box, to ask kind of obvious questions that too often we just don't ask. Questions such as, why is it that the United States of America is the only major country on earth, the only one that does not guarantee health care to all people as a right? Go to the United Kingdom, health care for all, go to France, health care for all, Germany, Holland, Scandinavia, health care for all, go to Canada, 50 miles away from where I live, health care for all. We are the only country. And the reason again gets back to money and politics. Everybody, the vast majority of the American people understand that from a moral perspective, when a poor person or a working class person gets sick, that person should be entitled to the same type of health care as a wealthy person with good health insurance. The question is, why haven't we moved in the direction of every other major country? Why is it that today, despite the gains of the Affordable Care Act, why is it that 29 million people still have no health insurance? Why is it that many of you and millions of other Americans are underinsured with high deductibles and copayments? And you know what high deductibles and copayments mean? It means you may not go to the doctor when you get sick.  Why is it that in our dysfunctional health care system, we pay by far the highest prices in the world for prescription drugs? You want to hear crazy? Crazy is one out of five Americans going to their doctor, getting a prescription, and not being able to afford to fill that prescription. Crazy is senior citizens in Indiana cutting their pills in half because they don't have the money to buy the medicine they need. And crazy is the three major drug companies in this country last year made $45 billion in profits. Crazy is that despite all of these issues that we have, we end up spending far, far more per capita on health care than do the people of any other nation. We spend almost three times more than the British, 50% more than the French, far more than the Canadians. In my view, the time is now for us to join the rest of the industrialized world, guarantee health care to all people as a right, and do that through a Medicare for all single-payer program. My good friend Donald Trump, not just kidding, not my good friend. I myself and my wife did not even get invited to his wedding. Can you believe that? Some people did, but we didn't. And there is a lot of concern that Donald Trump might become president of the United States. Don't worry, it won't happen. It won't have reasons. I was just looking before I came here at some polling, the last half dozen polls taken in April of this year. Every one of them had us far, far, far ahead of Donald Trump. And I mean far ahead like 15 or 20 points. We're also much further ahead of him than Secretary Clinton. Donald Trump will not become president of the United States because he believes that when we have a minimum wage of $7.25 an hour, we should not raise that minimum wage. I think that's nuts. The American people want a minimum wage of $15 an hour, not keeping the minimum wage at 7 and a quarter. American people are not going to vote for Donald Trump for president because he thinks that climate change is a hoax. Now listen to this. This is pretty amazing that Republicans think it's a hoax, is sad but not surprising. But he has his own particular angle on this. He thinks it's a hoax created by the Chinese. Which was very confusing to me. I thought he would have thought it was a hoax created by Muslims or Mexicans. Donald Trump is not going to become president because he wants to throw millions of people off of the Affordable Care Act and he has no alternative. Donald Trump will not become president because in a Republican quote unquote debate, there I call them debates, he suggested that wages in America were too high. Donald Trump will not become president because he thinks it's a wonderful idea that we give hundreds of billions of dollars in tax breaks to billionaire families like his. But I'll tell you why else Donald Trump will not become president. Because the American people will not be voting for a candidate to become president who every single day is insulting Mexicans and Latinos.  That is not the type of candidate, that is not the type of candidate the American people will vote for for president. Which is why we can only hope and pray that he gets the Republican nomination. At a deeper level, the American people understand that what makes this country great is our diversity. The American people understand that when we come together, black and white people born in this country, people born abroad, Latino, Asian Americans, Native Americans, gay and straight, that when we come together as a people, that always is stronger and that always trumps dividing us up. The American people understand that no person in this world, in this country, survives alone. That in one way or another we are dependent on each other and that we need to support each other. That we are great when your family is in trouble, my family is there for you. When my family is in trouble, you are there for us. The American people understand that supporting each other always trumps selfishness and perhaps most importantly in a very troubled and violent world. The American people understand what every major religion on earth, whether it's Christianity, Judaism, Islam, Buddhism, whatever the religion may be, the American people understand in the depths of their souls that love always trumps hatred. In this country, the American people are looking at the status quo. They're looking around them and they're saying, why do we have so much income and wealth inequality? Why is it that the middle class continues to decline? Why is it that youth unemployment is 30, 40, 50 percent? Why is it that we have a dysfunctional child care system? Why is our infrastructure collapsing in front of us? Why is it that women are making 79 cents on the dollar? Why is it that kids are leaving school 30, 40, 50 thousand dollars in debt? Why aren't we doing more to transform our energy system in combat climate change? Why is it that we are the only major country on earth that doesn't guarantee paid family and medical leave or health care for all of our people? Now some people, including my Democratic opponent, say Bernie is thinking too big. You've got to be more incremental. You've got to think small. You can't really address those issues. And I believe I strongly disagree because I believe that the lesson of history, whether it is the fight to end worker exploitation, whether it's the civil rights movement, whether it is the women's movement, whether it is the gay movement, no matter what those movements are, what they always tell us is that real change in America never comes from the top on down, always from the bottom on up. And what this campaign is about is asking millions of Americans, from all walks of life, to begin the process of standing up and fighting back, of understanding that it is just too late for establishment politics and establishment economics to understand that when we stand together, we can create a government that represents all of us and not just wealthy campaign contributors. On Tuesday there is going to be a very, very important Democratic primary here in the state of Indiana. And what I have learned throughout this campaign is that we do well when the turnout is high. We do not do well when the turnout is low. Next Tuesday, let us have the highest voter turnout in Indiana history for Democratic primary. And let the great state of Indiana join the 17 other states have said it is time for a political revolution. Thank you all.


Hi, I'm Don McCrae, Minister of Social Development and Social Innovation. Congratulations to the Community Social Services Employers Association for 20 years of serving employers in British Columbia's social services sector. I'd also like to extend my greetings to all of you at the 20th annual AGM and Fall Conference in Inaimo. I recognize and appreciate that tremendous work being done around the province every day through agencies across the social services sector. This is a large and diverse sector that provides a range of services to thousands of British Columbians. You deliver vital social, health and educational services and help to provide a quality of life for those who rely on them. Your employers and staff are known for the dedication, passion and commitment to service. Since becoming Minister, I've had the opportunity to meet with some of the individuals who are delivering frontline services and see firsthand how you make a difference in all our communities. I understand that the social service sector faces many challenges and I look forward to finding ways to ensure that service continues to be a priority for all of us. This is a sector with a legacy of innovation and our government will continue to work with the sector to look for innovative ways to provide support and assistance to British Columbians. Congratulations again on 20 years of great service. Thank you.


Okay, good afternoon everybody. So today we cautiously tried the second microphone for the room. Hopefully everything works out so I don't need to yell that much in the camera's microphone if there's a problem we will go back to the old way. Okay, so today we have one more topic to cover before we jump into the actual material and hopefully next week we can go into the material. And that's a very important topic which is how to validate AI algorithm. So what you're asking is basically how do I use Turing test in practice? This is basically what you're doing. So we want to find out beyond the theoretical interests and the historical significance of Alan Turing. Okay, so he asked the question, can machine think, which clearly they cannot? And what Alan Turing suggested was, yeah, but they can imitate. And the degree to which they can imitate would be proportional to a machine intelligent quotient and you can measure it in the way that how long can the machine fool the human being? But that's still not, I cannot really do that for the test. How do we do that in practice of AI algorithm? So we usually get the data and then we train our algorithms, methods, whatever you want to call them. I want to use algorithm a little bit loosely here because a neural network is hardly an algorithm, so it's not really a predefined set of steps. And then the question is, the first question will arise after you train it. How do we know that it has learned what it was supposed to learn? It was supposed to learn. So you get a bunch of data, you select an algorithm which at the moment we don't, we have really sparse knowledge to do that. How do I know which algorithm is good for my application? And then I do some filtering and some dimensionality reduction and then I feed it into my some sort of AI agent. And how do I know it's good enough? How do I know it learned? Because I want to give it to somebody. People want to use it, people will rely on it. What about my AI agent is the fully cool autopilot in a 380? 500 people are sitting on board and flying, so I want to know that it has learned. So how do we do that? Well, I would say we should test it. We should test it. Well, of course, we test everything before we release it. If good enough, then release it. If good enough, then release it. Well, what is good enough? We cannot be vague about this, it has to be clear. We have to quantify it, we have to document it. People want even a simple thing like you want to put your website somewhere. The hosting company has to tell you, my server has an uptime of 99.9% of the time. They don't even give you 100% because, well, we can have power outage. But other than that, pretty much 100%. So, and if it cannot process certain type of inputs, you have to also convey it to the final user. So how do we make sure of this? How do I make sure that my AI agent, if I use that phrase, if the piece of AI software is good enough to leave the laboratory and go into the real world? Do serious stuff for different type of people, in the finance, in robotics, in medical, logistics, anything. So how do I know? How do I do that? Well, that's what we want to talk about because starting next week, we should be able to do this because we will learn the explicit methods that bring a little bit of intelligence into the way that we do things. So the first factor to make sure that things are good enough to make sure is to make sure that it is good enough is to have a target function, a target or objective function. I did not use the word objective or the phrase objective function alone because if I use that, people would say, oh, objective functions are reserved for optimization. AI is not optimization. Well, AI is optimization in a non-conventional way. But OK, so target slash objective function. So what do we have? Most common one is error. And if I have error, of course, I want to minimize it. So this is a big chunk of techniques that we have. We calculate the error for them, and then we have to minimize the error. That's the way that we make sure the model is good enough for the practice. Then there are techniques that use some sort of reward. And of course, if it is reward, I have to maximize it. Every time that you do something OK or good, so you get a reward. Sometimes it's about fitness. And of course, we want to have fit solutions. So again, here, we maximize it. Sometimes we get punishment. So of course, we want to minimize it. We don't want to get punished most of the time. And there are more. So perhaps most commonly used target function for us, if you look at normal networks, our error. So you calculate the error, which is the difference between the desired output and what you actually calculate. So when we get to reinforcement agent, you use reward and punishment. When you get to evolutionary concept, we use fitness. And there are others for other schemes. OK, now we usually are given the entire data that we have. We usually are given the entire data, which is the capital X being the set of all observations, Xt. And t goes from 1 to n, 1 to t, how much data you have. So usually, what you have are your inputs, which could be also your features.  Features are representative for data. Sometimes the data is just too raw to be processed by AI. So you have to give more expressive things to give it to an AI agent to do it. Now, this is your input, and you may or may not get output. So which is, in this case, the desired output. Again, this may be one row, and that gigantic table that we have, we get from somebody to start doing a project now. So if we only get this, we will be doing unsupervised learning. And if we get this, we will be doing supervised learning. Now, of course, just for emphasis, this is the desired output, not just A output or D output, is the desired output. For this type of measurement, I want to have this output. One of the critics that you're hearing, somebody sent me a video interview with Vladimir Vapnik, the inventor of SVM, and he, like many others, are skeptical to deep learning, which we have to be, to any method. No method is, there's no holy grail. There's science, we question everything. We got here because we question everything. So the question is, the actual intelligence, the pure intelligence, has to be unsupervised. So the human brain can detect and find patterns in data in a very small amount of data, unsupervised. We don't know what the outcome should be, but we just grasp it. But this is, as long as Homo sapiens and Homo erectus and Homo habilis are involved, at least 10 million years of evolution after we separated from the apes, at least. And how much evolution went into them over, maybe 100 million. So it's not easy to imitate that, Mr. Turek. But we thrive. You're at the very beginning. So if given the set of all hypotheses, capital H, which is set of all possible solutions, all possible solutions, find small h coming from the capital H, find one hypothesis, at least one, such that the sum of differences between my input vectors, xt, and the desired outputs for my xt coming from the superset x is small and goes towards 0. So this is xt of d. This is the desired output. Usually we get one, but it could be a small vector. Usually the vector that contains the inputs and features is very long. And then you get one or two or three desired outputs, which usually are classes. So for this measurement, this belongs to class A. For this measurement, it belongs to the class B, whatever. So another disappointing formulation or revelation about AI, this is AI. This is what we do. We minimize the error, the difference between the output that I calculate and the output that I measure. So how do we do this? Well, first of all, this is supervised. So be clear that I'm not, maybe I'm a little bit unclear about this. So this is, let's say, call it star. This is the output that you get using the xt, and this is the output that you should get because of xt, just to make it more clear. So the output that you calculate for xt and the output that you should get for xt. OK, this is supervised. This is what we do. And that's OK. So there is no shame in this. It's supervised. And basically everything that is out there right now is successfully supervised. Well, people don't talk about it. That this is very expensive. The xt is very expensive. You need people to give you xt. Somebody have to tell you what should be the output. There is a lot of intelligence in xt. If you don't have it, there is no supervised learning. So this constitutes a good fit for the model H into x. So one hypothesis H from many, many hypothesis capital H that fits my data. If the problem is big, chances are there are many solutions. There is not just one solution. And again, if you find one of them, you're happy, usually. It doesn't matter which one. So if there are 100 solutions distributed in this room and I want one of them, that's OK. I don't want to search for all of them. I just need one. So OK. Is there any concern with this concept? So I get data. And then for the data, I calculate an output. But then I compare my output with the desired output. And if there is a difference, I have to make sure that next time my difference decreases because I will guess the output again. That's fitting. That's regression. This is 60, 70 years old. Yeah, that's linear regression you have in mind. So we are talking about nonlinear regression. When I don't know the F of x, and F of x very, very difficult. So what type of scenarios do we have? So scenario number one, so what type of scenarios do you get in the real world? You have your data, and you have a solution. And proportions are intentional here. And then you train. What would happen if you have such a scenario? The data is gigantic, so I'm basically visualizing the complexity of data with the size of this irregular shape around x. So you have complex data, and you have a simple solution. Whatever simple means. Simple means for a rule-based decision tree something. It means for a reinforcement agent something, and it means for neural network something else. So if you do that, you will realize that it does not converge. It does not converge means you cannot push the error residual towards zero. Not going to happen. You stock at 5 milli. I want to go towards zero. So convergence means the difference between desired output and calculated output goes towards zero. That's convergence. So you converge to the desired output. But if you have big data, not just the amount of it, the complexity of it, and you have a simple solution, that may not happen. Which means what? The problem is small, is linear, is stationary. And the solution is too small to capture that. Oh, I'm writing for the next one. Nobody screams. Nobody screams. It just doesn't match with what I do. So the problem is big, non-linear, and non-stationary. Another disadvantage of having lectures after five. So the problem is big, non-linear, non-stationary, any bad word that you can imagine in data science. So big is big. You have 5,000 features. Not how many data points you have. What is the complexity of the feature space? It's non-linear. It means the relationship between an output cannot be described with a line, of course not, or a plane. And it's non-stationary. Which means it could be not that it's all of them at the same time, but it's some of it. Non-stationary means that the position of the solution changes every time. So if I want to use reinforcement learning to control elevators of this building, so the optimal solution is sometimes on the floor number 2, sometimes on floor number 5, sometimes on floor number 7. So the solution changes. The solution does not have a station. It's not somewhere. It's not x squared plus 2 is equal 0 that you can find and say, x is this. That's a stationary problem. Non-stationary problems are the nightmare of engineers, because you have to bring in some continuous adjustment. The solution, which we call the hypotheses, is not capable of capturing the complexity, of capturing the complexity. So you don't even get to test this. There is no touring test here necessary, because you will not get a solution. It goes, and goes, and goes, and goes, and goes. Because you said, go until epsilon is almost zero. We never say zero. That's too idealistic. We say, go until it is 0.001. And you see iteration number 2 million, and your error is still at 120,000. And it's staying like a constant line. So it's not changing. It's not going down anymore. So you're not converging. When you don't converge, you don't have a solution to test. That's one of my favorite problems, because I know, oh, OK, my model has a problem. OK, so let's make this bigger. What that means bigger? Add more layers. Add more branching. Add more states. Add more chromosomes. Whatever your solution is. OK, what is the second scenario? Scenario number 2. I get my data. I have a solution. This is nowadays a lot more the case. Then you train it. And you are really happy, because you get fast conversions. You run it in MATLAB on your laptop, and boom, after 20 seconds, boom, it gives you the result. Chances are that you are getting a crappy solution. When you get a fast solution, that's definitely useless, unless you have a very easy problem. If you have a very easy problem, what are you doing with AI? Well, I want to impress people. OK. As long as they don't have a degree, you may be able to do it. So here the problem is small, linear, and stationary. And the solution is too big, such that it completely owns the problem to use a non-AI terminology word for it, because there is a word for it in AI. So we don't say it owns the data. It completely swallows the data, basically. So you have a gigantic solution. And you have a small problem.  You push the epsilon right almost to zero, boom, within 10 seconds, or even 10 minutes, or even an hour. You converge fast. It's just too good to be true. When you get a solution so fast, something is wrong. Please double check everything. Go back. What is the problem again? How many features you have? How complicated this is? How people try to use it just with a linear regression, and they are reporting 92% accuracy done while you are using this AI? So here we say that H is memorizing X. This is what has happened. You have just too much memory. You just swallow the problem. And during training, you get 99% accuracy. It's just it cannot happen. If it is a difficult problem, it cannot happen that in the first attempts, you get high accuracy. Cannot be. Cannot be. There is no theory for that. There is no theorem that we can prove. Empirically, we know that you have to suffer for weeks to get to 85%. You cannot easily get to 80%. When you are in the 90s, oh, wait a minute. Well, if it is six months past the project and you are close to the deadline, maybe the 90% is actually something real. OK, so we decided to stay in the middle. Stay in the middle. Good. What does that mean? Now, you have to figure out how we find out. Because you could have a problem with 10 features and it's extremely hard. And you can have a problem with 5,000 features, and it's easy. Because you did not apply PCA, and most of them are redundant, and there's actually two features. So then, Mr. Turing, what is the ultimate sign, the algorithm has learned? So what is that? How do I know? Well, the ultimate sign is very simple. It can generalize the inherent XY relationship to unseen data. This is several generations, what several generations of AR researchers have figured out to practically implement the Turing test. You know that your algorithm has learned. If you say converge, convergence is a really tricky word. So how do I know? Maybe I converge fast. And my algorithm was nice. I was lucky to have the best design. Who knows? But we know that it has learned if it can generalize to unseen data. Data that was not part of the training set. If it has really learned that unknown, magical, mystical, f of x that nobody has seen, if it had really learned that, it can generate that relationship for any data. So the keywords here for us are generalization and unseen. From time to time, I encounter people, especially in conference, usually master students. PhD students don't make that mistake anymore. And they come and say, oh, I got 99%. And you see that they have used the training data for testing.  So we want to validate that. And the validation is fundamentally test for generalization. So we are getting close to become really explicit of running, touring tests for training of AI algorithms. So our idea is, the core idea is, keep one part of data for testing. Is that it? So I keep part of the data for testing. Wow. A puzzle is really easy after somebody has solved it. People work for several decades to find out, oh, OK. So take the data and keep a part of it. Don't use it for training and then test with it. But is that that simple? Well, maybe not. So if this is your data, if this is representing your data, most of the time, we break it down around 70%, 30%. These numbers are not magical. You can have 80%, 20%, 90%, 10%. Usually this part is larger because we use that for training. So we use this for training. And then we use this for testing. So we use a large part of the data to train and a smaller part of the data to test. And again, there is no magical threshold. But naturally, you need more data for training because you want to figure things out. So this has to be more than 50%. But there are other tricks we can use. Within the training, how do I know? Because here you will train one hypothesis, one model, one solution. Who says this is the best solution? You train in one iterated, one epoch, one stage, one phase, you train something. And you converge to 1.2. Is that the ultimate measure of generalization? I'm afraid not. OK, so what happens if I train for a second time, again from scratch? And this time, I get even slightly higher error what it could be a better solution. I may have slightly higher error, but my solution is better. It could be. So how do I know that? Well, you can keep a part of the data that is reserved for training, and you do validation here. So people mix that up. Newcomers to AI mix that up. So which means what?  So let's say from the 70%, you take 90% for training, 10% for validation. Which means what? I train, I validate. I train, I validate. I train, I validate. I get many, many hypotheses. But I choose the one that through the validation will give me the best. Why? Then I will use the testing data to give you my final number. So I have to give you two numbers, training accuracy and testing accuracy. But in order to be able to generalize, so keep in mind, this is outside. This is inside. Inside the company, outside of the company. Inside your lab, outside of your lab. When you give it here, you don't have any control anymore. It's too late.  So I test it to choose the best hypothesis. And then I do the actual test. Yes, yes, if your data is small, it will be biased. But depending on how you do this, you will get rid of the bias. But in the training, you will find h1 and h2. You can come up with n solutions hn. Which one of them is the solution? Almost 100% of AI techniques start from random configuration. So you can start here, and it takes you five minutes to get to the target. You start there, it takes two minutes to get to the target. But the quality of the target will be different. So the validation is part of training. Before I release a model, I validate because this number is my official accuracy. I want to have the best. Because this is unseen data. This is unseen. So whatever accuracy I get on unseen data is officially my accuracy. I want to do my best. So, OK. However, this may not be reliable. The split may be lucky or unfortunate. The split may be lucky or unfortunate. This split. So you do 70%, 30%, 90%, 10%. That may be, you may be lucky. And so you grab the difficult part for training, and you grab the easy part for testing. Just lucky. And then you release it to people. People test it with real-world data, it collapses. Or you may be unfortunate because you do the difficult split. So your training data is really, really easy. But the testing data is very difficult. Contains all that non-linearity and nasty stuff of non-stationary and complexity. So you are telling me, this is basically what we do to implement Turing test. But then you are telling me it's not reliable. Yeah, whenever things like that happen, we do a lot more of them of just one time. So we have to do this many, many times. So we do k-fold partitioning. So you get the data, you grab this 30%. You get the data, you grab this 30%. You get the data, you grab this 30%. This would be a three-fold. And for each fold, you do this. You grab any of them, 30%, put it aside, do training validation, training validation, training validation, test it. You get one number. Second time, third time, you get three testing accuracies. OK, now I know. It's not about lock, it's not about randomness. First of all, if we do things properly, those three numbers should be very close. The god of randomness is extremely fair, very fair. The only condition is you've got to do it all about. Otherwise, you don't have the probability to back it up. So talking about the god of randomness, we do random sampling, of course. If it is a real problem, nobody goes and takes from 1 to 200 as the 30%, nobody does that. Just don't grab the first 30%. No, the data may be ordered. The data may be ordered in the reverse order. The data people who collect the data may have some bias and preferences and put some stuff at the beginning. So we grab one and do random sampling, and then order it. This is the first 30%. Grab another one. Do random sampling. Order it. Another fold. Grab another one. Random sampling. And then order it to get another 30%. So we have to do a lot of random stuff to make sure that we are not relying on a bias in the data or we are not experimenting based on our bias. Yes. So when we get feedback from unseen data and the data is really unseen, depending on some other parameters, this is a real feedback with respect to the generality of our solution. We are assuming that you have enough data. We are assuming that data is complete. We are assuming that everything that we are talking about is in order. You are doing random sampling. There is no bias in the data. Then yes, whatever unseen data tells us is real unless you just grab 1% of the data as unseen. OK, then it cannot be representative. And you did the one fold. You did the k fold. So the level of experimentation is that it will give us comfort. Yes. They may have, but we don't go and correct it because otherwise it's not random. We just go random. And of course, then you have to rely on a good random number generator. Of course, yes. Yes. So each one of them, again, we get the 70% to use for training, 30%, let's say, for testing. Here, the 70% is split here. Every time is just a series of experiments separately. So I train with this and test. That will generate, first time, tells me 92% accuracy for unseen. What was in the training? We'll forget about it. You can report that I had 95% accuracy for training. You can do that for academic paper, but this is the important one. We do it for this one, and you get 91%. We do it for this one, and you get 89%. And they are relatively close. OK, that's a piece of comfort. I seem to be doing OK. OK, so what? All this would work if we had a lot of data. What if we don't? So what if we don't have enough data? We have other problems. How are you training something if you don't have enough data? But you may have enough data for training something, but you don't have enough data to do the K-fold cross validation. For high dimensional stuff, that happens a lot. So 1,000 measurements for many applications is not enough for AI, but it may be enough if your data points are videos. Having 1,000 videos, oh, that's more than enough. It's high dimensional. I can get a lot more information out of it. So 1,000 to just split it in this is not really enough for K-fold, unless you do, I don't know, 10-fold, 20-fold, 50-fold. Most experiments that I have seen are around or less than 10-fold. 3-fold, 5-fold, 10-fold, quite calm. Provided you have done the rest correctly, no big mixtase in other part. So OK. If that happens, we don't do K-fold cross validation. So this is K-fold cross validation. This is actually standard. Standard way of testing anything in AI. Yes. Yeah, we randomly sample, and then just for convenience of calculation, we push it in a certain, or you can keep the indices. It doesn't matter. Yes. So this, so A and B are not the same, and C and D are not the same, because they are a result of different random processes. So if we don't have enough data, we will not be able to use K-fold. We will use leave one out. So leave one out validation. So if you have 1,000 videos, you cannot use, OK, I use 700 for training, 300 for testing. Even 10-fold may not give you much, because the data is complicated. So the type of validation also is related to how difficult is the problem that you are trying to solve. So if I get 1,000 videos, I will do leave one out, which is I will train with 999 videos, and I test with one video. Then I repeat it again, grab another video, train with the remaining 999 and test again. So I have 1,000-fold. I do that 1,000 times. Doing 1,000-fold cross-validation, nobody can afford that, generally. And we don't. Why should I? I rely on randomness, 3-fold, 5-fold, 10-fold. Good enough. I'm good. I'm secure. Empirically. Not theoretically, but empirically. But if I don't have enough, so leave one out is our best bet. OK. Good. So if you have n data samples, we will use 1 for testing, and we will use n minus 1 for training. So because n is basically the cardinality of your set, how many, how many you have. So n-fold cross-validation. Very expensive. If I could, I would always do leave one out. But I cannot do it when I have 1 million data points. You want to test 1 million times? Who can afford to do that? It's a pure computational thing. But also, with everything that we know generally from conversion theory, if you do 10-fold, you see the pattern. You don't need a million-fold. So then leave one out category, which historically comes big time from clinical setting, because you do experiments on, study on, let's say, 50 patients in a hospital. You can never get a million people. So you get 20 patients, 50 volunteers. How much experiment can you do with 50? So we do leave one out, put one patient aside, test the 49, and then do something else. Very clever. Very smart to leave one out. It can easily boost the confidence in the AI, because again, so what do you have here? You grab this one, if this is just one. And then you grab this one, and you grab this one, and so on. So you do that n times. So very expensive, very expensive. Hence, suitable for small data. Suitable for small data, not a good idea for big data. If I have big data, again, random sampling is our best friend. The only thing that can happen if you are using a bad random number generator, which, if I'm using a major programming language that should not be the case, and I'm using standard packages that everybody has used, so if I'm using the random package of Python 3.2, so I'm fine. I don't need to be worried about, is the random number generator reliable or not? These are questions that you have to ask yourself. No, no, I want to use my own random number generator that last week and I created because I had so much fun. Well, I don't do that. OK, now, if I look at the model complexity and the error. So error as a function of model complexity. So again, model complexity is how big is your AI solution? Then you train, and you see, so the training error comes down when your model become more complex. You make your network bigger and bigger and bigger and bigger and bigger, and the error comes down, comes down, comes down, goes towards zero. And everybody in the lab is jumping around, yes.  So this difference is small. If the difference is small, that means well trained. So if the difference between what you train and what you test within one training session across different complexity, how you want to draw this curve is up to you. I can draw it just for one model. This could be just one model, and I'm just trying it just in one epoch. Or I'm using different models. Generally, when the model complexity goes up, the error comes down. Could be another case. Again, we have the error and the model complexity. And again, the training you see doesn't matter. The more complex it becomes, the bigger the solution H, the lower the error. So I start with three layers, and then five layers, and then 10 layers, and then 20 layers, and then the error goes down, down, down, down. And then I test it. So again, I want to have a bigger difference than we test. And we see that we have a big difference. So now I have a big difference. So it's considerable. So your model is not well-trained. It's not end of the world. So here you may get 90% and 88%. You may get 88% and 85%. Here you will get 90% and 82%. So the difference becomes bigger, becomes more considerable. What is not end of the world? End of the world comes when you see something like this. So again, model complexity, and you have your error. And then you train. And the world is beautiful, because the error goes towards zero for training. And then I test, and Jesus Christ, you get this. It's not even about difference. This guy is going crazy. It's just all over the place. So you make it bigger. In training, you were jumping around because you had 92% accuracy.  Wow. Dropped from 92% to 52%, 40%? Buddy, you didn't learn. You did not converge to a real solution. You converged computationally, but you did not converge in terms of optimal solution. So when this happens, so this is really big. And when this happens, we say you all have overfitted. First time that we used the overword, overfitting, number one enemy in deep learning. Because we tend to overfit most of the time. OK, what does that supposed to mean? Everybody had it? No? OK. So let's say we can put a number on model complexity and say this is the cardinality of set p, where p is the set of parameters of the hypothesis h coming from the universe of discourse of all hypotheses. So in terminology of modern AI, this is your hyperparameters. This is all parameters that make up your model. If you are doing a linear regression, you have five, six parameters. You are doing a deep learning of average size. You have 150,000 parameters. It's a lot of degree of freedom. So up to 10 years ago, it was considered impossible, intractable, to go after problems with 100,000 parameters. You simply have too much degree of freedom to fill it with meaningful numbers. So OK, go back to the roots. All comes razor. Keep it simple. Deep learning came from this door. OK comes razor escaped from the window. And OK comes razor is one of the oldest wisdom we have. And it's still valid. Keep it simple. The simplest solution tends to be the right solution. If you can't do it with 22 layers, instead of 50, do it with 22. Keep it simple for us has a word in the AI literature. It's called regularization. Regularize it. Don't let this beast run away with the data. It will run away so fast. Five minutes gives you 95% accuracy. But it doesn't pass the tourist. So we said that you look at the sum of whatever output my data gives minus the output that is desired for all data that I'm reading. And we should minimize this. It should go towards zero. That's what we said. So touring, Alan touring in action. Occam's razor in action. Human wisdom in action. OK, so with all this that you told me about, careful cross-validation, and leave one out, and testing, and validate. OK, so how should I do it in practice? Well, if this is the model complexity, you should give me the best result with the smallest solution. So let's go back to the 50s and 60s. If the solution was a decision tree, the size of the solution is the number of leaves and nodes in the tree. If the solution is of artificial neural networks, is the number of layers and the number of neurons in each layer. That's the size, the complexity of your model. You should minimize the error. But please, don't use a billion layers to do that. Can you reach 99% accuracy with six layers? No. OK, go to seven layers. No. Eight. No. 10. Yes. OK, give me 10. No, I want to do it with 250, dense net. Why? Because it's much cooler, and the graphic takes so much, and I have to draw the tensors and everything. I thought we are engineers. So we want the lowest error, and we want the smallest solution. So now we do this with k-fold cross-validation or leave one out, two-ring mission accomplished in action. Now we know it has learned. It's not fake. So the so-called augmented error function will be then, so the e prime will be your error total. Whatever your error is, however you calculate it, doesn't matter. Plus a factor, let's say lambda, and this is not the eigenvalue, call it alpha, beta, anything, lambda, times model complexity. Why I'm bringing in a lambda? Because I want to control it first. So this panelizes complex solutions with large variance. Which variance are we talking about? Training accuracy, testing accuracy, training accuracy, testing accuracy. 90%, 89%, 90%, 85%, 89%, 86%. That variance should be low. There should be agreement between training and testing if everything is working according to the plan. So we may need a lambda because in some application we want to put more or less emphasis on the model complexity, and we can do it. Yet another model. But please consider that things like that, they will become the driving force for learning. This is the way that we learn. So we will force the methodology to give me the minimum error by minimum size. So that's why we regularize it. So we regulate how it works. Now, if lambda is too large, you want to play with it, play with it. You get simple solutions, simple models. If you do simple models, what happens? We can force it, right? I can put a big number on lambda and say, I really like small solutions. Now you are exaggerating in another direction. What happens if lambda is too large? So I favor it small solutions. What does that mean? You increase the bias. You will be biased towards certain stuff. You will be biased towards certain pattern and their dominance in the data. It will collapse when you go online. So hence, we use cross-validation to optimize lambda. So it's not a bad idea to have a lambda. So the regularization part, its significance and its effect on the learning can vary. But it has to be part of the validation. Otherwise, nothing should leave that concept of that you are validating. OK. So this is one way of choosing the best configuration. Nobody says you could, but nobody used that to choose between what sort of you could do. But choosing whether I want to go decision trees, random forests, or I want to go reinforcement, or I want to go neural network, that's a different choice. That's a strategic choice. For simple problems, we may run all of them and then see how they do. And we do cross-validation of all of them. For this course, we will do. And it's expected, OK, run two, three different AI methods and see which one of them is the best. But this selects the configuration of one given AI solution. So you choose neural network. The question is, what type of neural network is good enough? So that's why we are doing this. But there are other ways of doing this. Was there a question? Yes. No. The model does not know about this. The cross-validation will enforce this. I mean the lambda. Yes, the lambda. So we can play with it. We can have different values. And then the cross-validation will bring us toward that. Any model, any AI model? We can use other AI models to adjust this, yes. We could use any optimization techniques to adjust this, yes. But we usually don't. Because it's embedded inside the cross-validation. But like any other parameter, it can be optimized in many different ways. So what about another method, another model selection approach? So basically, K-fold cross-validation and leave one out are a methodology that for us to select a model. Is the model good enough? Has it learned? Is it intelligent at all? So the other method is the Bayesian approach. Bayesian approach is used if we have prior knowledge. If you don't have prior knowledge, Bayesian cannot be used. And of course, we use the Bayes rule, which we will come back to it, but just for the sake of today. You will look at the probability that the model is a correct model given the data that you have. So that's another approach to model selection. What is the probability that the model that I'm using, the network that I'm using, the decision tree that I'm using, is the best model giving this data? What is that probability? And using the Bayes rule, we can calculate that. This is the probability of data given the model times the probability of model divided by probability of the data. So when we say you need prior knowledge, so do you have the probability of data, the frequency of the data? Do you have the probability of model? How many times that model was a good model? And how many of the experiments? Do you know something advanced? Or I have to run crazy amount of experiments to find that out. The Bayes rule is a very, very powerful rule, but it cannot be applied on all possible problems. Because not for every application, we have the prior knowledge to do it. So what are other? There are some other approaches, other methods. There are methods that are called structural risk minimization. And there is a really old one that is called minimum description length. Minimum description length comes from information theory, is closely related, inspired by OCOM's razor, and can also be used for our purposes. But with all of them, so we want to hit the good solution. But most of the time, we go toward bad solutions. So if you have a small solution or you have a big solution. So for small solution, the problem is that's nice. But when I say small solution, that means too small. So you get biased. And when you have a big solution, of course, you get very high variance. You get a lot of variance. So here you may get 1990, 1990, 1990, which is very stable because you are biased. And here you get 1985, 92, 82, 95, 72. Wow, a lot of going up and down. So we want to be here. A good solution is not biased and is not variational. And the only thing we can do to make sure of it is cross-validation with the unseen data. So any mistake that we do is very, very crucial that we do not make a mistake when we validate. So I train, validate, train, validate, train, validate with some regularization. And then I choose the best one and test it. You test once for every model. You cannot run the test again. You just did the test. But training validation, we can do many times. Train, validate, train, validate, test. Test your best model, the best foot that you can put forward with respect to everything that we talked about. Not too small, not too big. So that balance is give me something that is big enough to solve the problem, but is not small to just be biased toward patterns in the data. OK, so next week we will start with clustering. And we will talk about two methods, a very old one, K-means. And we will talk about self-organizing maps. And at the latest, by Tuesday next week, we should be able to basically use everything we talked about so far. Because suddenly now we have a method. We have two methods. One is a simple iterative algorithm. The other one is the first type of neural networks. But we don't call it neural networks at this moment because we don't know neural network. This is a clustering technique that has some structure. And today we have the tutorial that we will cover PCA. So we will take it. And I will upload some reading material and some videos for everything that we have covered this week.