Rep. Alexandria Ocasio-Cortez has recently talked about how during the January 6th insurrection of the Capitol, there was a moment where she actually thought she was legitimately going to die. She hasn't really talked much about that incident until now during an Instagram live stream where she discloses some more details and she shares how it really was the situation where she felt like that was the end of her life. So we're going to show some clips from her Instagram live stream and then we're going to talk about the response because the response to her sharing this traumatic experience, I don't even know what to say.  But nonetheless, let's listen to her story because I think this is important. I opened the door when all of a sudden I hear that whoever was trying to get inside got into my office. And then I realized that it's too late that it's too late for me to get into the closet.  And then I just start to hear these yells of where is she? Where is she? And I just thought to myself they got inside. And so I hide behind my door like this, like I'm here and the bathroom door starts going like this, like the bathroom doors behind me or rather in front of me and I'm like this and the door hinges right here. And I just hear where is she? Where is she? And this was the moment where I thought everything was over. And the weird thing about moments like these is that you lose all sense of time. In retrospect, maybe it was four seconds, maybe it was five seconds, maybe it was ten seconds. Maybe it was one second. I don't know. It felt like my brain was able to have so many thoughts in that moment between these screams and these yells of where is she? Where is she? And so I go down and I just, I mean I thought I was going to die and I had a lot of thoughts. You have a lot of thoughts. I think when you're in a situation like that and also one of those thoughts that I had was I just happened to be a spiritual person and be raised in that context and I really just felt like if this is the plan for me then people will be able to take it from here. I had a lot of thoughts but that was the thought that I had about you all. I felt that if this was the journey that my life was taking that I felt that things were going to be okay and that I had fulfilled my purpose. That is horrifying. It sounds like a nightmare. Now she later learned that the individual who was screaming where is she was a capital police officer and once he came in they still kind of got weird vibes because if you're a capital police officer shouldn't you announce that you are a capital police and after he was talking with her he was still super aggressive and he told her, he basically yelled at her, go to this area in the capital. He didn't tell her which room they should be in in this particular building and when they got there they ran there they weren't escorted there her and one of her staffers ran there and they were basically left wondering where do we go they heard the rioters outside so they were questioning whether or not they were in more danger after listening to this individual and even though in theory like you think okay this is someone who's supposed to protect me it's a capital police officer right well after seeing the way that they opened the gates for the insurrectionists I'm not so sure that that individual was on her side like if I were in that situation and capital police basically let this happen at least a substantial portion of them when they were taking selfies I wouldn't necessarily feel safe so the fact that they got a bad vibe from this individual after you know he was screaming at her I don't blame them I don't blame them at all now she explains that this situation was particularly traumatic because she experienced past trauma before and she came out as a sexual assault survivor which is something that is very very difficult to do. Folks who tell us to move on that it's not a big deal that we should forget what's happened or even telling us to apologize.  I give her credit that is something that's really difficult to do because in this day and age when we're supposed to be applauding sexual assault survivors and encouraging them to come out and share their stories like this is the Me Too era right you have so many people like laughing this off downplaying it oh she's just being melodramatic nobody wanted to kill her is that so because one of the individuals who was charged during the capital insurrection openly talked about wanting to assassinate AOC specifically is it really that shocking to think that white supremacists who are violent who were storming the capital wouldn't be targeting individuals like AOC Ilhan Omar Rashid Al-Talib I just you know the callousness is really what strikes me from this story like the response to her is what strikes me and I don't want to give you this false impression that most people dismissed this and mocked her because most people were supportive most people you know are standing in solidarity with AOC as she shares this story but usually you know the most negative things that we see are what kind of stay with us now one more clip that I want to play for you before I talk about the response from right wingers and some who claim to be on the left is from representative Katie Porter in an interview with Lawrence O'Donnell on MSNBC she talked about how when her and AOC connected the response that AOC had like the look on her face she was rattled well first she you know she saw me and we waved and went into my office and a couple seconds later she not and she said no could we could we come in and I said of course and she began to deal with her staffer was trying to describe what had happened and Alex is really usually like unfailingly polite and very personable and she wasn't even really talking to me she was opening up doors and I was like can I help you like what are you looking for and she said I'm looking for where I'm going to hide and the thing that will always stay with me the two memories that really you know especially as a mom I think we're just really powerful for me was when she said you know I I was saying well don't worry I'm a mom I'm calm I've got everything here we need we could live for like a month in this office and she said I just hope I get to be a mom I hope I don't die today and the second thing is she was wearing heels and I remember her saying to me I was wearing flats and I remember her saying to me I knew I shouldn't have worn heels how am I going to run and we went and we found her a pair of sneakers to wear from one of my staffers so that she could run if she needed to literally run for her life I can't imagine what she was going through that sounds absolutely horrible and this is going to be something that stays with her for the rest of her life like she was already dealing with existing trauma being a sexual assault survivor but that moment that's going to stay with her forever she's going to carry that with her for the rest of her life speaking from firsthand experience when you deal with psychological trauma it doesn't just like go away like you can't just get over it as easy as that sounds you can't it stays with you for years and decades but the response was just as I alluded to earlier I don't even know how to describe this it's just it's cold-hearted so right-wing conspiracy theorist Stephen Crowder tweeted out an image of Carl Marx with the shirt that says seize the means of reproduction and he wrote allegedly AOC's abuser was found I'm not necessarily sure what the joke is here apparently this is supposed to be funny but this is just his way of belittling her experience there Michael Tracy tweeted out good to know that any loopy delusion expounded by a politician must now be respected and quote believed under the aegis of trauma unreal and it's not just right-wingers there were some individuals who claim to be left-wingers which I don't think that they're left-wingers if they're downplaying this that's that you know they're just laughing at this oh well you know she's being melodramatic you know what about the folks who are dying because they don't have health care since she didn't agree to the strategy of force to vote therefore that means she doesn't support medicare for all in my twisted view of the world and therefore you know I don't care if she thought she was gonna die because what about the people who are dying now like the the mental gymnastics that people jump through in their minds to downplay this it just shows you that like there's there's no rationale there's no logic there's no human empathy it's just like in politics we've come to a point where we always work backwards from whatever conclusion we've arrived at and some folks have just chosen to make aoc the enemy demonized like one of the only left-wing members of congress who's a genuine ally demonize her and so whatever she says is invalid anything she says is wrong because she's the enemy she's the enemy this is the result of conditioning by folks with very large platforms who just do nothing but rail against aoc attack attack attack she's the enemy she's stopping you from having medicare for all she's the problem it's not corporate democrats it's not republicans it's her it's just sickening like the state of the left right now it it really worries me um it's just it's like a snake eating its own tail right demonizing aoc like i think that holding politicians accountable is absolutely necessary and warranted you have to hold everyone in congress accountable because they're in a position of power they're in a position of great influence but they can actually affect change but perhaps maybe when they're like sharing their story about how they believe that they were going to die that's not necessarily the right time to do it maybe maybe this is just me uh wait until a different instance maybe when it's more appropriate to bring up something like maybe when they're sharing the fact that they're a sexual assault survivor and dealt with existing trauma that might not necessarily be the best time to air your grievances with this individual who is an ally to us believe it or not it's just shocking to me and because she wasn't actually being hunted down by you know a far right extremist storming the capital because it was a capital police officer people are claiming that um she's just being melodramatic does that make her feelings any less real there are people who literally suffer from panic disorder who have what's known as amygdala hijack i think it's called where they just like inexplicably have panic attacks they uh have this like influx of adrenaline their heart starts racing they start sweating and they have that fight or flight response it could be like just the most random time ever you could be in the store grocery shopping and all of a sudden you get the instinct to get out run because you feel like you're gonna die there's no reason for it but people get this does it mean they're actually in danger no but are those feelings real yeah they're pretty fucking real they're pretty fucking terrifying i'm speaking from firsthand experience where you could just be doing the most random thing ever watching a movie you're relaxing you're having a great time and all of a sudden there it goes you're in full on panic mode out of nowhere and you feel like you're going to die you have that instinct to run away but there's nothing that's attacking you so you you kind of just like sit there and you try to play it off because you don't want to look like an idiot in front of other people um but in in your own mind in your body you're feeling all of this it's not just psychological it's physiological as well your heart's racing and it's horrifying is there actually danger no but that doesn't make the feeling any less real and horrifying and each time you suffer from a panic attack like this you know um you remember it you fear when the next one is going to come this is what aoc is trying to talk about because she had existing trauma like she's always in that mindset like where if something traumatizing or scary happens she immediately is going to think oh my god it's happening again i'm in great danger and it's not like she didn't have a good reason to believe that she was in danger during this insurrection because again there are violent white supremacists who want her dead and i just i don't understand the callousness i don't get it i just um yeah i'm very very uh discouraged at the state of politics currently uh on the left but thankfully you know we're kind of in this echo chamber where the dumbest folks usually are the loudest but they're not the majority and we still can move forward and progress um with the help of allies like aoc and so if we expect her to you know represent us and have our backs you know maybe at a time like this we should be having her back and uh defending her because what she went through is awful


Hey it's Missy! Welcome back! Today's tutorial is a Halloween tutorial for this galaxy girl, these messy buns, so I really hope you enjoy today's tutorial. If you're new to my channel, make sure to subscribe it down below and let's get started! The first thing I'm going to do is start by parting my hair in a zigzag part slightly off-center, and then continue the part down the back of my head to split my hair into a left and a right side. With the left section of hair, I'm going to tie this up and bring the right section up into a high ponytail near the crown of my head. Then I'm going to take a hair tie and wrap it once over the hair. Then I'm going to spread the hair around the hair band forming a donut shape and then wrap the hair band twice more over the hair to form the base of the bun. Now I'm going to pull on the edges of the bun to make it really big and full. And then with the ends of sticky now, I'm just going to wrap these towards the front of the bun and tuck them in to form a loop. Then I'm just going to grab some bobby pins and pin down the edges of the bun so it's flush with my head. Now I'm going to let down the left section of hair and repeat the same steps creating a second bun at the crown of my head next to the first one. Now I'm going to take some hair gel and mix it with some glitter. And then I'm just going to paint the glitter along the edges of my part to finish off the galaxy girl costume. I hope you enjoyed today's video. If you did, make sure to click the thumbs up. And if you haven't subscribed to my channel yet, make sure to subscribe. You can click my little picture right up here. Then you won't miss out on any future videos. Thank you so very much for watching and I will see you next time.


Hi, I'm Brian Maid. As we welcome in 2018, Mother Nature reminding us in her own special way that it was one year ago that Northern California was hit with several storms that would eventually last more than two months and cause widespread flooding across several counties. The first full week of 2018 will end with rain predicted for most of California. In the north, a winter storm is forecast for early next week. Snow levels of an inch are more expected in the Sierras. If you are traveling in the mountains, remember simple steps can make a huge difference. Watch your speeds, allow extra time, make sure you have chains or four-wheel drive and it's always good to have water, flashlights, blankets, and a few simple supplies with you. And Southern California may see significant rain for the first time in five months. A potentially strong storm with gusty winds expected for next week, there is the potential for flooded roads, downed trees, and power lines and traffic closures or delays. For the latest weather conditions, be sure to monitor the National Weather Service across their social media feeds or your local media.


Please subscribe to this YouTube channel mentor talk and do press bell button for notifications. India is contemplating a series of economic actions against Chinese companies, investors and interests in India as a retaliatory measure for the border conflicts. The process of economic onslaught started by India banning 59 Chinese apps. Further, the Indian Minister for Road and Transport and Highways has stated that Chinese companies would not be allowed to take part in the road projects. It is also being reported in some sections of the media that the government is considering trade and import restrictions targeting China in days to come. The enhanced scrutiny of Chinese investments in many sectors has taken place and the government might be evaluating decisions how to keep out the Chinese companies from 5G trials. As per reports, several public sector undertakings have by now begun to revoke contracts given to the Chinese firms regarding telecom, BSNL and railways. These unilateral actions from India may cause losses to the tune of billions of dollars in deals, bids, contracts and revenues to the Chinese firms and investors. I want to examine the legal implications of these economic sanctions which India may be subjected to in view of the World Trade Organization, that is, WTO obligations. These legal repercussions, if any, need to be examined because both India and China are part of the WTO agreements. It is a well-accepted proposition that one-sided acts, the unilateral acts, short of falling within the exceptions under the general agreement on tariffs and trade, that is, GATT, shall be believed to be in violation of the non-discrimination rule under the WTO legal mechanism. The basic foundation of the rules of the global trade is based on the premise that the transnational worldwide trade among countries should be non-discriminatory, whether India's actions in the past few days against Chinese economic interests in India would be considered as violative of the principles of non-discrimination or not is something which we need to examine. The doctrine of non-discrimination embodied in the WTO law is like a coin with two sides, one dealing with the most favored nation, that is, MFN treatment commitment, meaning reciprocity, when a WTO member gives specific favorable treatment to another member nation. Then it is required for that recipient member nation to reciprocate by giving the same favorable treatment to the grantor member nation. It has to be based on reciprocity, the MFN status. And the other side of the point of the WTO deals with the national treatment, that is, anti-treatment, N-T, not A-N-T-I, anti-treatment, which is national treatment. This means and obligates a WTO member nation to consider the foreign product services and the suppliers of services at par with the domestic product services and service providers. There should not be any disparity or discrimination between the domestic and foreign economic streams. One cannot give less favorable treatment to the foreign economic line or let's say one cannot give preference to domestic over the foreign investors. That is equal national treatment obligation. As I had touched upon earlier, that unilateral action can be branded as discriminatory unless it falls within a stipulated exception or exceptions. On the face of it, one could say that India's economic actions against China are one-sided and unilateral, hence discriminatory. But that may or may not be true. That is what I'm going to examine now. In the current context, I am of the premise you opinion that India's economic actions in the past few days against China should qualify Indian actions, should qualify as legitimate, valid and lawful for being squarely covered under an exception stipulated by Article 21, Roman 21, Article 21 of GATT, which deals with the security exception. As per the said exception, no member nation be prevented from taking any action which it considers necessary for the protection of its essential security interests. Yes, further such a nation can also not be required to furnish any information, be it obligatory, the disclosure of which it considers contrary to its essential security interests. So it may not disclose information. And such a nation is free to take any action, any action in pursuance of its obligations under the United Nations Charter for the Maintenance of International Peace and Security. Yes. So this exception squarely applies to India's actions. India's retaliatory economic actions are justified according to me, considering the overwhelming evidence of aggression by the Chinese forces on several border points with India. So consequently under Article 21 of GATT, any individual and independent actions taken by India cannot be deemed, cannot be deemed or judged as being a violative of non-discriminatory principles embodied in the WTO and GATT legal regime. Well, hope you found this video talk informative. I will see you again sometime soon with another video on law or public policy topic of your interest. Keep watching mentor talk and stay safe. Thank you. See you again. Bye for now.


So, let's continue. What else did I want to say? Oh, yeah. I don't know. I don't see people talking to each other. I'm bored in the evening. So, 8 o'clock in the lobby. If you want to come have a beer with me, I'll buy you beer. Alright? I don't know if I'm allowed to do that. So, if I'm not allowed to do that, I didn't say that. But I'll still be in the lobby at 8 o'clock for mysterious reasons. So, let's start. So, last time we started talking about machine learning and what we wanted to emphasize last time was the difference between predicting and fitting. And so, I hope, playing with the Python exercises, you saw that there's a subtlety in all this stuff that we want to predict on data we haven't seen yet. But we get to only fit on the data we've already seen. And that's going to come up again and again in what we do. And a lot of what you need to understand about machine learning right now is how we do this. And so, the plan for the class is basically for the lecture is I'm going to do it all on the blackboard. We'll go back to Python notebooks tomorrow in the morning. But to get you there, we have to understand the basic ingredients of how you do this kind of machine learning. And there's basically three things we have to understand. So, there's basically four parts to a machine learning model. There is the optimizer, because remember we have to minimize the training error in some way or actually minimize the generalization error. That part of that is choosing parameters and we do that by optimizing. And we'll talk about what that means. Then there is the cost function, which is how we measure how good we're doing. And then there is regularization, which is a little bit more subtle. But regularization is basically the idea that we get to see training data. And that's minimizing the error on the training data. But what we want to do is generalize well. What we want to minimize is the error on data we haven't seen yet. And in order to do that, remember what we saw last time is you can have overfitting. And so to take care of that difference in what I actually optimize and what I would like to optimize, you have to do something called regularization. That's the third component. And then the fourth component is basically how do we choose the model itself. And modern deep learning is basically based around neural networks. Jeff Hinton wanted to rebrand neural networks because they were extremely unpopular, so he started calling them deep learning in the mid-2000s. And then because as hard as it is to believe now, if you worked out, there were like five people left working on neural networks in the mid-2000s. And everyone thought they should stop working on them and do support vector machines until 2012, until there was this amazing result called AlexNet. And I basically had to do with the fact that they put these neural networks on GPUs. And if I have some time, I'll tell you about GPUs and computation, but we're not going to get into that right now. So let's go through all this together. And so as I said before, we have these two functions we care about. E-training, which is measured by some cost function, where you just basically look, oh wow, I'm making a lot of noise. And you have some other thing you care about, which we're going to say is test, but it's not really test, it's the error on a typical piece of data I see otherwise. And I have some parameters that I have to choose. And again, I want to emphasize the subtlety. I keep writing these things that are completely redundant over and over again, because basically saying is I have access to this, but I care about that. But we know that if this is terrible, this is also going to be terrible. This is necessary. You have to be at least pretty good at this, otherwise you're never going to be able to predict on anything new. So the question is how do we optimize and choose these parameters? And the trick of modern neural networks, what's really important, and I would argue is that there's two basic things that happened in the last 15 years that are completely surprising. One is that if you talked, read the old neural network literature when it was really unpopular, I would say 2010, 2009, back when I started as a faculty member. What they kept telling you is that we just don't have big enough computers or big enough data sets. And that actually turned out to be true. So the first thing that has made this easy is that GPUs and the size of the data sets. So this is computation. But the second thing that is kind of surprising, and I think I would argue that no one really still understands, is the surprising power of differentiability, especially in high dimensions. And by that I mean is that you would think it would be really hard to learn a map from a million dimensional space to another million dimensional space. But actually if you regularize, or if you use the fact, the thing you try to learn is essentially differentiable, it turns out you can do this quite easily. And so this kind of differentiable architecture is key to understanding what's going on. And so the kind of optimizations techniques I use really assume that these kind of models that I have, I can take derivatives with respect to the parameters easily. So this key to this, right, this is the key to all. I would say the real key to all this, and we don't really understand, the idea is that we want to optimize by taking derivatives. And so for that reason I'm going to focus on optimization, optimizers that are based on taking derivatives. And we'll see what we really want to be able to do is take derivatives and then use the chain rule. And derivatives plus the chain rule is back propagation. That's what the famous algorithm was discovered in the 80s, but no one really appreciated how powerful differentiability was. And I would even still don't really understand why, once you make things differentiable, you can learn these maps from super high dimensional spaces in a way that if you made me guess even a decade ago, it would have been really surprising. I mean to everyone, everyone. I think Jeff and Jan were the only people who actually believed that this was possible. So let's begin. So generally what we're going to be interested in is that we're going to have some cost function or some error function. And instead of using C, I'm going to use E for all this stuff. But the important point is that the error functions generally have a very special form, which is that they usually sums of individual errors for each of the training data points. So the training function, think about least squared regression. In least squared regression I do yi minus my prediction squared, but the important point is that it's a sum over individual training data points. It's the error on each data point, and then you sum it up. Is everyone okay with what's going on? So this is going to play an important role in what follows in a second. And now I just want to minimize this error function. So right now let's think of this as an optimization problem. So how do I optimize this with respect to some theta? Say I wanted to optimize parameters. So what I want to do is I want to iteratively update my parameters theta. So given some parameters at theta t minus one, I want to update them to theta t. And the basic idea is that I want to do this with gradient descent. So I want to do this with differentiable things, because some of our differentiability seems to be the key to all this. So how do I do this? Well, it's pretty easy. One thing I can do, and I'm going to try to establish some notation, is I can basically calculate, apparently that is not going to, I'm going to keep using this over and over again unless I move this over here. Oh, it sticks on. Oh, how clever. There's Velcro. Oh, then you can wash them? Oh wow, Korean technology. One thing I can do is just calculate the gradient, and I can update, let me see, minus Vg. So I have some function I'm trying to minimize. I take the gradient, and I just update the gradient equations. And I've introduced this very important parameter that's going to play a very important role in all this stuff. This a to a t, and this is called the learning rate. So this is the dumbest thing I could do. I just take my energy function, I take a gradient, I go in the direction that minimizes it, iteratively. So what's the learning rate mean? Why do I need this parameter called the learning rate? Why do you think? Yeah, so generically, I didn't have to do that. I'm confused about the boards. But you know, generically, I have a complex thing. The gradient only gives me local information. So I only want to make local moves in my parameter space. So the gradient in general, the learning rate controls how much do I move? How seriously do I take these things? So generically, the idea is that these landscapes are complicated, and they're also in high dimensions. So if I'm doing just optimization, generally, I don't do gradient descent. I do something called Newton's method. How many of you have heard of Newton's method? Raise your hand. How many people know Newton's method? Some of you. Okay. So what's Newton's method do? How's Newton's method different from this? Here I just use the first derivative. What does Newton's method do? Who wants to tell me what I use in the Newton's method? I don't just use the first derivative. What do I use? The what? This is also a linearization, but this is the first derivative, right? So what I'm saying here, maybe, is I'm saying e theta is approximately e theta plus delta theta is approximately e of theta plus delta theta times gradient of e. And what would be the next term in this thing? What would I have? Yeah, you'd have the Hessian. That just measures the curvature. So what Newton's method says is don't do this simple thing here. Instead, what I should do is I should now use this update. Vt is equal to h inverse theta t where h is the same Hessian that appeared up here. Del theta of e. And then I do the same update. Did I mess up some signs? Oh, I don't know why there's this minus sign here. Okay, so I do this thing, and instead of a learning rate, I've used a Hessian. What's the advantages of this Newton's method? What do I get? Why do I, if I'm just doing optimization and it's easy to calculate the Hessian, do I use this Newton's method? What's the intuition? Sorry? It's defining variance. Is that what you said? Yeah, in some sense it measures, you know, local curvature, right? And one of the things it does is that it shoots different directions in different ways, right? So the problem in high dimensions is I hear the learning rate, I take steps, and I take steps in all directions exactly the same. But generically in these high dimensional landscapes, you have some directions that are narrow, other directions that are really big. So what this does is you could think of this as adapting the step size, right? So this sets the step size, the learning rate. What the Newton's method does is it says, oh no, I'm gonna adapt how much I walk in these different directions based on second derivative information. So we'd love to do that, right? And in particular, you know, in the one-dimensional problem, so right? Let me see what I'll do. Yeah, I'll come back here. And erase this. If I have a quadratic potential, right? Then I know that the optimum learning rate, right? So imagine this is just a quadratic potential. So this is just what every local minimum looks like. It's a quadratic. Then I know that the optimum learning rate looks just like the second derivative to the inverse. And in particular, what happens is that if I have n is less than n optimum, you basically take little steps to reach the minimum, right? So if n is less than n optimum, if I do gradient descent, right? And everyone sees that this is just the one-dimensional version of this Newton's method up here. Because that's like the optimum thing. If n is equal to n optimum, then I just reach it in a single step, right? If n is greater than n optimum, right? I need two more colors. Maybe I have them. If n is greater than n optimum, but less than, but less, and then finally I'm going to have n is greater than 2n optimum, what happens is that you can convince yourself, in this case, you just kind of bounce back and forth across here, but you still converge. But in this final case, if I make my learning rate too big, what's going to happen is you actually start diverging. You never converge, alright? So you become really, really sensitive to this learning rate. That's something you'll see over and over again. This is going to be our first example of what you call a hyperparameter, which is a parameter that matters a lot whenever you're solving a hard machine learning problem and you have to sweep over it. And what's interesting is this is in one dimension, but I know I can just think about this in high dimensions, and in particular you see that if I go back to this picture, maybe here, here I want to show both the picture and this thing at the same time, but I'll just write it again. It's probably the best way to write it. If I look at what my gradient, if I look at what my update is for Newton's method, you know that you can always diagonalize this Hessian. It's a symmetric matrix. You diagonalize it. And the eigenvalues, now the eigenvectors and the eigenvalue directions basically look like independent directions. Everyone okay with that intuition, right? You diagonalize. But then each independent direction is going to look like this. If I go to the diagonal basis, it's just an independent problem. But now you get into a big problem, right? Generically, you're going to be limited. You want to choose your learning rate smaller than the smallest eigenvalue. Otherwise you're going to diverge in some directions and converge in other directions, right? So that's why you're so sensitive to the learning rate because it basically sets a resolution and in high dimensional spaces you're going to have a distribution of eigenvalues. And you don't want to, in some sense, you could always choose this smaller, but then it becomes computationally more expensive, right? That's why we don't choose it really small. So there's all of these trade-offs. That's why we're at the two of these hyperparameters. So much. All right. So I would love to do this, but I can't. All right? So let's think about what's going on. So this is gradient descent. And what we're going to see is that gradient descent itself is not what anyone uses because it has some fundamental limitations and some of them we can overcome with other things and other things we can't, right? So let's think about why we can't just use gradient descent or Newton's method to optimize stuff. So Newton's method is the first thing you should ask is if you're remotely computational, why don't I use Newton's method if I have millions of parameters? What's expensive? I consider myself like a pure theorist and even I know what's expensive. What's expensive in this method computationally? Yeah, inverse. Because at each step, I have to invert like a million by million. It's actually much bigger than that. It's not even computationally visible, but even if it was a few hundred thousand, which is a small neural network, you have to invert a few hundred thousand by few hundred thousand matrix at every step, too expensive. So this kind of Newton's method out from the beginning. Can't have big matrix inversions. But we still would like to adapt the learning rate, right? That's the intuition that you want to get from all this. On the other hand, we can think about gradient descent. And what's important is to understand the limitations of this gradient descent. So gradient descent seems great, but then you start... This is going to be the bane of my existence. All right. Gradient descent. So where are the limitations? In order to understand why people use the optimizers they do, it's good to understand what the limitations of gradient descent are. All right. So I've said that most of them throughout this talk. So who wants to tell me some of the limitations based on what we've been discussing so far? What we did this whole discussion, so someone tell me, what are the limitations? Yeah, I mean, even without the matrix inverse, it's actually computationally intensive. Why is it computationally intensive? Because look at the error. It scales like the number of training data points. Right? So you have to do a big sum that scales with a huge data set. So this is just computationally impossible. This is what I would call computationally expensive. Right? So the first thing we didn't really discuss it is actually quite computationally expensive because at each step I have to do a sum over the number of training data sets. And as the training data set gets big, it gets really expensive. What's another thing that is a limitation of gradient descent? Yeah, another thing we didn't talk about, but for sure, it's true, is that you can get tracked in local minima and not global minima. Right? And for a long time, people have been thinking about, well, I erased a very complicated thing, but in general, you know, you have a very complicated thing. I need to make the learning rate small, but if the learning rate is small, then I get trapped in local minima. It turns out the intuition is that most local minima that are reasonable are pretty good. That's one of the interesting things about deep learning and why modern statistics fails, but you do get trapped in local minima. Okay? What else? How about in high dimensions? What was the point that we were saying about this? What is one of the things Newton's method in principle fixes? What do I say about different directions? How many directions? How does gradient descent treat them all? Yeah, it treats them all the same. But we know that there's no reason to assume you have uniformity, but a lot of tricks are also about making this thing. So gradient descent also treats all directions the same. So that's kind of annoying. And the last thing is that that graph says is that it's also very sensitive to the learning rate. So in practice, it's true that if you have infinite amount of time, the best thing is really playing with the learning rate and doing, well, not gradient descent, but stochastic gradient descent, which we'll get to in a second. But in practice, what you want is some stuff that works off the shelf. So no one ever actually uses gradient descent or stochastic gradient descent for the first time to train something until they have some reasonable tuning of all the other hyperparameters, but we'll get to that, right? So here's my gradient descent that fixes as many of these problems as I can. This is never going to be able to be fixed by any local gradient derivative based function, so we can just store that out, right? And then we're going to use... try to fix all these things, right? So the first thing that turns out to be true, and at first, would be just a trick for fixing computational problems, but turns out to actually work much better than gradient descent in practice, is something called stochastic gradient descent, right? And this is the workhorse neither of these algorithms is ever used. What is used in practice is something called stochastic gradient descent. I can't spell. Or SGD, as it's commonly known. And SGD is something that has its origins as a computational trick slash something, another field of machine learning called online learning. But in the end, it turned out that this actually works much, much better for a reason we'll come back to. That has to do with the thing I keep emphasizing, that you have access to one thing, but you're actually trying to optimize another thing, right? Training test. So what you do in stochastic gradient descent is instead of taking, oh, that's hidden, isn't it? Okay, I'm not going to do it there. I need this board. So what I'll do is I'll erase this board. So what we do with SGD is you say, okay, here's all these training data points, right? And then you say, look, I'm working with e-train, but I really care about something else, which is, you know, e-generalization. So it doesn't really matter if I get this error function exactly right, because this is just an approximation for something I can't measure anyway. Everyone see that that's the fundamental thing? I can't measure what I want to put. So what I do is something dumb. Okay, instead of using all the training data, let me make something called a mini-batch, right? And the idea is that a mini-batch, so where do you should know? Just like you should know learning rate, right? A mini-batch is let me take this training data and let me just take some small amount of training data. Say, like, say, 128 of these. Order 100 data points. And actually the mini-batch size doesn't seem to depend on the amount of training data you have. It's not extensive, which I've spent way too much time thinking about why. It never came up with a good answer. All right? And then what I do is I divide my whole training data set. You know how big it is? Into little batches. If I have batch one, batch two, batch three, all the way to whatever batch, the total training data size divided by B, right? So I have whatever batch. How many ever batches you have? So you have 100, 1 to 100, 100 to 200, 200 to 300, whatever. And what I do is instead of taking the gradient over the whole thing, I take the gradient over a mini-batch. All right? So at each step, what I do is I just say, OK, I'm going to use the same algorithm here, but now I'm going to just replace it by a mini-batch. And not only am I going to do that, I'm just going to go through. At the first step, I'm going to use the first mini-batch. At the second step, I'm going to use the second mini-batch in the gradient descent. Next, I'll use this mini-batch, and I just go through. And once you go through a whole thing, it's usually called an epic. An epic is when you go through the whole data set once. So what is this mini-batch? What does this SGD do? First of all, you see that at each step, I'm taking a gradient of a different function. And if you think about it, this is really, really small. It's really weird because it's such a sparse, horrible, noisy sample. In some sense, I say, maybe this is what I really care about, but I'm going to estimate it with this kind of horrible, noisy thing. But actually, the noise helps you. That's one of the big lessons. Why does the noise help you, you think? Who said that? What? Who? Yeah, it's some kind of regularization. Again, the point is, I don't care about this. I care about that. And by making it noisier, I'm basically taking out these non-generic features of the training data. So at first, people thought it was just a computational trick, but in practice, it turns out to be way better than gradient descent, even if you could do the computation, even on small data sets. Because this is our first example of something that serves as a form of what we're calling this, which is regularization, which is that I need tricks, because I don't want to overly optimize for each train. What I care about is about each general. So SGD is the workhorse of modern machine learning, because I also need differentiability for reasons we'll talk about. Everyone okay with that? So learn these words, epic, learning rate, mini-batch. Everyone have those words? So that solved this problem. And we got a bonus. The special bonus, there was yellow chalk, but I can't find it. I got a bonus. Regularization generalizes better. Because I'm basically shaking. One way of thinking about it is that I'm shaking up the potential. Each time it's slightly different. And you're averaging out all the noise, right? And keeping everything. I can shake some more if it'll make people laugh. You guys are like a very somber crowd. Very, very, very, very, very subdued. Yeah, of course. Generally it turns out that you choose of order 100. So anywhere between 30 to 150, but you don't want it to be extensive in the size of the data set. No, no one knows. Open question. Seems to depend on the application. Depends very strongly on the learning rate. Too big is bad. So really you want to put the noise in. Because that regularizes. That seems to be true. So it's always chosen of order 100. It's also, you understand that machine learning is a lot like alchemy. For this audience, biology. If it works, you just keep doing it. You don't ask. It's like, I don't know if you've ever talked to experimental labs. You ask why things are done. They said because it's always done and it always works. So machine learning has a lot of dogma. A lot of dogma. We don't know why. We just know it works. It's not clear anything is the best solution, even though you'll see a 30 paper saying it's the best solution. Because like four years later, someone will come up with another solution. So it's really, it's empirical. I would say it's like alchemy. We have rules of thumb, a lot of guesses. We don't really understand very basic things, I would say. That's my guess. But it works. It's a great tool. Yeah? I'm sorry. Can you say louder? I'll use the first mini-batch for the first gradient. The next time you take the gradient, you go to the next mini-batch. Next time you take the gradient, you go to the next mini-batch. So you move it each time. You're actually taking the gradient with respect to a different potential. And then you just cycle through the whole data set. That's an epic. And you usually do multiple epics, 10, 20. We'll see this tomorrow when we run our Python notebook. Gaussian noise. Yeah, because Gaussian noise seems, It has a lot of physicists spend a lot of time on that. But it's because the nature of the noise is very different. This seems to have, you know, it's not white noise, right? And this noise seems to work better than just adding Langevin noise, which is uncolored. In some sense, the idea is that this noise knows about the correlation structure, right? Because it knows about the correlation structure of the energy landscape. Whereas white noise doesn't. So somehow I think it has something to do with the fact that we'll come to the next thing, which is that even in your noise, you don't want it to treat all directions in the same way. And you can see that this is going to be colored noise. And it's going to know something about the noise is going to be larger in some sense in the directions that are less constrained. Just intuitively.  That's basically the idea. Try to make that into a rigorous thing. It never works. I have tons of calculations I've done. They work on like three numerical experiments and fail on the next three. You know, it's like, it's very hard to make this into a rigorous theory. All right. Sorry, I have to see what I'm going next. Okay. So that's stochastic gradient descent. But in practice, often what people want to do is they want to actually, you know, they want, especially when you're doing initial explorations, you want an optimizer that kind of solves these things. I don't want it to be super sensitive to the learning rate because I'll never find it. I want to basically know, is my network pretty good before I start optimizing everything? Right? So in practice, what people do is they've come up with, what they'd like to do is up here in Newton's method, but it's too expensive. Too expensive. Don't want any matrix inverses. But I would still like to, you know, fix some of these things. I'd like to have something that's not that sensitive to what my hyperparameters are, that actually treats different directions differently. So people have come up with what are called essentially, you know, different, different, different things. And so there's two basic tricks that people use, right?  And the basic idea now is instead of, once again, you take the mini-batch, right? But now what you do is instead of just updating it all the gradient in this way, is that you give things inertia, right? So the basic idea is what you do is you give, let me see, did I write that right? I did. So what you do is you basically give things inertia, right? So they keep moving in the direction you were giving. You were moving it. Somehow that equation doesn't look right to me. Hold on.  Sometimes it's so confusing. Give me one second. I want to write something. I feel like I'm doing something silly.  The good part is it makes a better lecture. The bad part is when you get confused. Yeah, there's something wrong about what I wrote down, right? Oh, no, this is why I wrote it down wrong. It doesn't make any sense what I wrote. The momentum is not there. The momentum is here. What you do is you take the gradient and instead of changing it all at once, you, the gradient keeps on going in the direction it's going, and then you update a little bit, right? And alpha is a parameter between 0 and 1, right? Instead of just saying, oh, I'm moving the direction of the local gradient, I'm averaging the gradient over some time period that's set by alpha. Yeah? Sorry? It's just, it's usually chosen, there's some convention I forget. It's the default. It's the, it's the, it's the, it's the default. I don't actually, I've, again, let me write this. Let me use standard notation so I can do it. You do this usually, okay? And gamma is just the time step. It's usually chosen to be 0.7 or something like that. In every machine learning package that I tell you what gamma is, and there's a default, I think no one touches it. All right? There's some best rule of thumb. Who knows if it's the best, but no one touches it, right? But the important point is it's just, remember the gradient over some time period, right? Because you have the last time, and you see this one over gamma is basically setting a time period on which you remember things, right? And the point of this is that we know what momentum does in a ball.  All right? Everyone okay with, with that? Okay. So that's the first trick, but that still doesn't solve the problem of treating all the directions in the same, in the same, in the same way, right? And so what people call the, so this is SGD with momentum. And so what people do is what are called second-order moment, second-moment approximations. And what this, these are, they're a way to try to implement Hessians without calculating the full matrix of derivatives. And the most famous one, the one that is the default of everyone uses is something called atom. But there's also another one, which is very common, which is called RMS prep. I think much more likely is, much more nowadays I see much more atom use than everything else. So the default that people go to, if I, when you, you'll see that most people use is this atom. So what is the idea of these atom and RMS prop, right? So the basic idea is not only do I keep track of the mean, right? So here I'm just taking care, track, taking track of the gradient. I also take, I also keep track of the variance of the gradient. All right. And so the simplest version is something like this. So I have to write it. I forgot that I wouldn't have a printer here. So let me write down some gradient, which is del theta mini-batch, then I basically write down a moment m of t is equal to m beta m. These are standard notation, so that's why I'm trying to use standard notation. On minus beta, I am. And then I take care of s of t squared, this. And then I write, it's not going to be really important for all this stuff. And then, roughly speaking, spiritually speaking, what I do now is I say that theta t plus one is equal to theta t minus the moment, minus eta t over m of t over s of t plus epsilon. So this is basically what's going on. So let's walk through these equations. They look very annoying and confusing, but they're very straightforward to understand. So this was just the stochastic gradient descent that we already saw. There's one very clever trick in this and it doesn't work otherwise. This is just, again, just keeping track of the gradient with momentum. So this is saying, okay, my best estimate for the gradient is whatever gradient I got weighted by whatever momentum I had before, at the time step before. Does everyone understand that? This is just averaging. So now I average. I don't update it all at once. I keep track. I'm averaging over multiple time points, set by beta inverse. Then here, I'm not only keeping track of the mean, but I'm also keeping track of the gradient squared. So this is like the variance. This is how much things fluctuate. And then what I do is I basically now update my moment and my parameters by the gradient, which is the mean, divided by something like the standard deviation. This you can think of as trying to estimate the variance of the gradient because it's the second order fluctuation. This is trying to estimate the mean. And the basic idea is now I should step in an adaptive way such that I move the mean divided by the standard deviation because that's the dimensionless view that makes sense. So the standard deviation basically tells me what the natural unit is. And this mean basically allows you to step in that direction. So what MT should I take seriously? The ones that are much, much bigger than the variance. So it's almost like moving in Z-score. And this is another tricky machine learning all the time that you see over and over again. This is what batch normalization is. All these fancy words, a lot of them are because if you normalize things and work in Z-score so that you have natural scales for things, you do better. And this is called Adam. It's not quite this, there's another step that it's not really important for anything. You can look in the notes if you want to. But the idea now is I've taken these things and in particular you see that this is calculated for each direction separately. This variance and this mean. So in each direction I move in a different way and it also implements another form of regularization that seems to be important which is called parameter clipping which basically is the statement I never want to make really big moves in any direction because I don't trust my, I have new stochastic gradient descent I don't trust really big moves. So it allows me to set a natural scale on which to move and eclipse big, big, big moves. So that's why Adam will give you usually pretty good, not the best performance but pretty good off the shelf performance because it's kind of a risk averse algorithm. And I've gotten around, I would really like to use the Hessian but instead I just used the covariance. You know I used to use the second moment. So this is basically how we optimize. Yes, sorry I can't hear you. You have to louder.  It's just saying there's a minimum variance, you want to regularize. Nothing to become really small and blow up. It just says variances that are much smaller than this I don't take seriously. Second, right? Just a regularizer. Just as a scale, minimum scale on which I'm willing to take the variances seriously. Because sometimes especially when you're in optimization the variances can get very small, numbers can get very small and you just like ignore that. Just a regularizer. Cuts off small things. So that's the first section of all this which is the optimizer. And this is going to be really important because it's the differentiability that's really important for all this. And you see all these things are going to require us to be able to take derivatives with respect to parameters. This is all, I haven't said anything. I just told you if I could take a derivative how should I move in parameter space? Everyone see that? This is why this is a flexible part of every machine learning algorithm. The fun thing is, everything is modular. Yes, Antonio? Yeah. Yes. Yeah, I think what it allows you to do is it allows you to choose a slightly larger learning rate but it still suffers from the gradient descent. Because you're not, remember the problem, the biggest problem was that you were the smallest learning rate, right? By adaptively putting this thing, you do it like that. What's interesting is that you really need these to have the same dimensions. So a lot of people were saying it wasn't working and they were coding it up wrong or actually it turned out. So it's really the signal to noise ratio. But I think the way you should think about it is you're allowed to use much bigger learning rate. You become much less sensitive to learning rate because you're working in a dimensional ratio and you're doing that. You're not really solving the correlation problem. You're solving is the learning rate problem. Yeah, I mean, that's how I think about it. All right. So the second ingredient we need is the cost function, right? So the cost function, how much time do I have? 15 minutes. All right. This ends at 11. 12. 12. 12? 12. 12. Okay. Okay. No problem. All right. Let me clean some boards up. All right. So there's basically two common cost functions. The cost function is, again, whatever you want to do. We're just going to talk about the simplest classification problem. But almost at the top of all classification problems, right, is you have two kinds of data. You have categorical data, which means things are categories, right? Cat, not cat. MNIST 0 through 9. Or you have continuous data. I hope that's spelled right. I don't know. Can't tell. I'm slightly dyslexic and whenever I'm on a board, I just can't see anything. So categorical data for continuous data, the most common thing people use is to have some prediction that's continuous. And you just, again, the most common thing you use is least square error. All right? But you can use anything you want. The important point really is that you want whatever cost function, right, the cost function in general is going to be something that depends on y and something that depends on your predictions. So this y hat is your prediction. That's just a general standard thing. And you want to make sure this cost function, right, because of the chain rule, you want the cost function to be differentiable with respect to y hat, because that's how I take the derivative with respect to theta, from chain rule. So any differentiable cost function works. And people have clever, in more complicated things, people have more, you know, whatever, more complicated things, but generally these squares will give you most of the intuition you need. Categorical data is a little bit more fun. All right? And it's actually very physics inspired. So in categorical data, and this is, you know, the inspiration for this is, of course, regression. So who knows what, if, like, off the shelf statistics that I have to do categorical data, what kind of statistical model? Potts. Huh? Potts? That's very hard, not usually, but... It's a good guess. It's something like a potts model, actually, but it's logistic regression. How many of you have heard of logistic regression? Okay, you should go look up logistic regression. It's not going to be really important for what we do, because... Okay? So the standard thing is logistic regression. So let me tell you how logistic regression works. And in general, logistic regression slash softmax regression. So this is usually two variables, two categories, but this is general, many categories, say, M categories. I was going to explain to you the relationship between this and maximum likelihood, but I'm hoping it was already done. There's actually fun things that we're not going to get to, because it makes the regularizers make much more sense. But the basic idea is that imagine I have some data set, D, and now my data set kind of consists of, you know, some features that I measure, X, you know, the pixels in MNIST or whatever you want to say, and some variables, Y, and in general, it's useful to just say Y, you know, so I have some features, Xi, some predictions, and let's first start with the case where it's a binary variable, 0 or 1. And the basic idea of all this stuff is that I basically defined the probability of Yi equals 1 given some X to be equal to 1 plus e to the minus W dot X, right? Oh, let me use theta. Sorry, let me use these parameters. Right? And let me use, let me use it like this. Theta dot X. So my parameters are just take my theta dot X and just do 1 plus that. That's the probability of being 1. So instead of thinking about predicting either 0 or 1, generically you make this problem what's called soft. So instead you just try to predict the probability that I'm in each of these categories and many of you will recognize this as a Fermi function, right? It's a two level system and this theta is the difference in the energies of the two levels, right? What does this function look like? It kind of looks like this, right? It's a Fermi function. It's a sigmoid, right? And of course the probability of 1 is just 1 minus the probability of Yi equals 1 given X. Right? So this is how I define it. And now what I do is write now what I can do is something that Spinaurny talked about a lot in the lectures is I can do maximum likelihood estimation, right? I can say what's the probability of observing the data given my parameters? Well it's just going to turn out so if I call this sigma of sigma of X. X.theta like this, right? So sigma is just this sigmoid function. Then it's just going to be sigma to the product over all the data points that I have times sigma of theta.x to the Yi times sigma of 1 minus sigma of theta.x to the 1 minus Yi. Why? Because Yi is either 0 or 1 so the probability to absorb a data point is either theta or 1 minus theta and you see one of these terms is always 0 because if Yi is equal to 1 that I only get this thing which is the probability Yi equals 1 but if Yi is equal to 1 or 0 then this disappears and I only get this term and since the data points are all independent I can multiply the data point and the data probability of the data set is just a product for each data point. But what I usually do is I don't work with likelihoods I work with long likelihoods. Everyone okay with long likelihoods? So I take the log of this thing and if I take the log of this thing you see what I get is I get that this is equal to i equals 1 to n Yi log of sigma x dot theta plus 1 minus Yi log of 1 minus sigma x dot theta. All I did was take the log likelihood but now look if I look this is where I maximize but the cost function is minus the log likelihood because cost functions are minimized. So minus log likelihood which is the cost function looks like this and hopefully many of you recognize this as an entropy formula. This is like if I have a binary variable the entropy of a binary variable is just minus p log p minus 1 minus p log 1 minus p that's the entropy of a binary variable that can have probability heads. This is something called the cross entropy which is related to the Colbeck-Liebler divergence. Remember the Colbeck-Liebler divergence is pI log q over pI because I know you guys covered this and the part that depends on q is just pI log qI sum over i minus the entropy but this is the part that compares these things so this is called the cross entropy if qI is the only part that has parameter dependence if I'm going to take derivative with respect to it this is the same as taking the derivative with respect to the Colbeck-Liebler divergence right so I mean if qI is the only thing that depends on theta then if I want to take the derivative of the Colbeck-Liebler divergence that's the same as taking the derivative of this thing so this is called the cross entropy and you see that like q is basically I have the probability from the model and I compare that to the deterministic probability of the real data so in some sense it's calculating the KL divergence between the data labels and my labels and this is the standard categorical thing so I'm running out of time so I won't tell you how it generalizes in a very straightforward way to M categories instead of 2 using something called softmax so maybe I have time I can show you how it works so the basic idea again is this is this loss function and it comes from maximum likelihood but it also looks information theoretic which is not a coincidence because you can think of this as a variational free energy doesn't matter if you want all this subtlety you can go read the review for your purposes that's the whenever you see cross entropy it's maximum likelihood it's categorical data it'll show up tomorrow and what's fun is of course I can generalize this to M categories and the basic idea is I use what's basically a partition function so now I have data set again that's not what I wanted to do I wanted to read it like this Y i equals 0 1 2 all the way to M so if I have M categories the Y i can be what I do is generally I make what are called one hot vectors and the other thing you'll see a lot in machine learning but it's very dumb which is that instead of representing this as M numbers I represent this as a vector of length M and if something is in category 1 I put a 1 on the first category and everything else is 0 if something is in category 2 I just basically put a 1 at the second thing and everything else is 0 the fourth thing all the way to the Mth category and what's nice is that in terms of these Y i M so now everything gets 2 indices an index i that runs over data points and an index M that runs from 1 to the number of categories I have and what's nice is just like before now in soft map regression what you do is that you can write the probability then Y M prime equals 1 given X and now what you do is you put a set of parameters for each so now what you have is you have a set of rates for each of the categories so now if I have if I have M if I have NF features so these X's are I have M categories now I have NF times M parameters so you get a different parameter for each category and it's basically a Boltzmann distribution this is the probability of being in a given category it's just a Boltzmann distribution like this and you can go through and the same derivation we did right here you can basically do the same derivation again and you'll get another cross entropy so if we're running out of time maybe I'll just show you oh dang it I'll make it take longer since we're running out of time apparently you can't stop it in the middle oh is it not stopping uh oh I broke the screen anyway I'll just show you the formulas I hate showing bunches of formulas but it looks the same it's not really important for anything you do but I guess I should say in the last five minutes is what kind of regularizers do I do use so we already talked about one form of regularization is that we use SGD but often what people also do is that there's many kind of generic regularizers but one of the things people try to do is set as many parameters to zero as possible and the way you do that is you take your normal cost function that's where it is and it depends on some parameters data and your data and what people often do is they add a penalty for making the parameters too big so what you say is let's the most common functions are alpha equals well let's just start with weight decay so the most common thing is you put an L2 penalty on this thing so you put an extra parameter lambda that becomes another hyper parameter so this thing says the cost goes up whenever the parameters are big and so you try to shrink the parameters to zero and in general you can have the most common ones are L2 penalties or you can have what are called L1 penalties where you take the absolute value of these things these tend to give you sparse things but generally you put extra terms in the cost function to penalize the parameters alright and the basic idea again is because I don't really care about the training error I care about the test error so I want to put in extra terms that basically penalizing using too many parameters right and these are penalties on having too many parameters because if the parameter is big it costs a lot so I have to make sure every time I add an extra parameter the cost function goes down sufficiently and that ratio is set by this lambda which generally is another hyper parameter like the learning rate you have to choose right so this is for some reason these L2 and L1 penalties in the neural network literature are called weight decay so if you ever say if you read a neural network paper you'll see people say I use weight decay that's just adding these penalties alright and there's many more things that are very neural network specific you can get into drop out, batch norm right so I'll just tell you some names of things that are meant to regularize so another thing is drop out another thing is batch norm and these are just different tricks that people have for regularizing but the fundamental reason is that you need all this regularization is because you don't want to overfit the data alright in the review there's a long list anywhere you look up so that's basically we're at the point now where believe it or not you can write where we need about 10 more minutes to explain to you how neural networks work and how blackprop works about 30 minutes the first 30 minutes of the next class because we haven't said anything about this function f yet right except we know that we need the cost function to be differentiable as a function of f and we want f to be differentiable as a function of theta and we want to be computationally we want it to be really computationally easy to calculate these gradients because we have to calculate the gradients with respect to all the parameters all at once every time we take a gradient descent step so we'll start here tomorrow and we'll talk about why neural networks are really good the thing about modern neural networks is they're designed in such a way that it's really easy to take these gradients and calculate them quickly alright and then you plug it into this whole network and the regularizer is in you choose an optimizer and then you go let it run with some mini batches and stuff alright so we'll do that with the python notebooks tomorrow so I think 30 minutes more of explaining this thing then you can write some deep learning models and I promise you if you spend another couple of weeks you can actually understand what's going on in state of the art most of the state of the art stuff is not so hard conceptually you get to empirically play a lot there's a clever idea I don't mean to dismiss the ideas there's often clever ideas but it's the implementation that matters as much as the idea because you can't tell what's a good idea until you really play with it because there's so many things you have to tune you have to tune so many hyperparameters and there's so much hyperparameter tuning and we'll talk about that tomorrow that's what real machine learning is about it's a very empirical field it's numerical experimentation everyone's hungry I'm out of time thanks before we head towards the cafeteria do you have any questions? just one or two if not then we can have some more if you can ask a question during the lunch break


The study examined how changing the pH of a protein affected its ability to aggregate when it came into contact with a liquid gas interface. It found that lowering the pH caused more protein to aggregate at the interface, but only if the protein had been previously unfolded. This suggests that the unfolded protein is more likely to aggregate at the interface than the native protein. This article was authored by Ru-A-Li, Zhao Liang-Wu, Yan Ji-Wang, and others.


Let's go to the moon, right now. Hi everyone, we are in Paris here. J. Casano, CEO of Cointelegraph. And Christina Cornell, our editor-in-chief. And we are so happy that yesterday we announced the French version for our wonderful publication. This marks a very important step for us to give the voice to another beautiful community in this space. I'm very happy that we have a presence now in France.  I think France is a great community, a great location for us to have a presence now. For sure, and the French version actually joins a family of different versions that we have already, and we keep expanding and it's very important for us to not only talk, you know, a marriage-centric perspective about tech, about blockchain space, about communities, but also talk about those who are from different backgrounds who use different language because we are convinced that the more languages we have, the more representation we have, the more impact we can make on this world. Crypto has become and is becoming more and more mainstream. You know, we have to go beyond just telling the news at Cointelegraph because they're going to become more and more news outlets that tell those same stories. And what I think sets Cointelegraph apart is that we're made up of people like yourself and all of our journalists who actually use crypto themselves, who care about crypto, who have ideas about the future of crypto that they want to see, what they hope Web3 can accomplish. And stories, it's not only words, it's actually also visual element and I'm so happy to have a wonderful visual team and you can see some of our art. You know, as you're saying, the visual identity of Cointelegraph is, of course, I think one of the most unique things and it's one of the things that made me want to work here. We are operating in over 30 countries and our team is 24x7 here for you talking about what is our passion and not only technology but also you, the community. Yeah, and I think a great example of that. You know, if you're watching this on our YouTube channel right now, we've had this crypto stories series that's been coming out lately. I think that's a fantastic example of the kind of journalism we want to be doing that's not just about projects, as I was saying before, but also about people who are using this, who have had a difference in their lives because of crypto. Whoever watches us now, please feel free to share your stories with us. We're always happy to hear you, to read you and to meet you. Next big milestone for the company. We'll be launching a new mobile app very soon that will have a lot of really great features sort of like what I was talking about with being able to follow the projects that you're interested in and it's not just that, again, we're trying to create a feedback loop with our community here where it's not just that we're going to force feed you the content that you tell us you want to read but because you tell us the content you want to read, we're actually going to produce more of that kind of content. We also consider very important responsibility of ours to dissipate misconceptions about crypto and blockchain. Definitely, there are a lot of myths in the community outside blockchain and crypto and we are trying to show the positive side of the world of crypto that is an incredible, big, shining space that is keeping growing. Yeah, I think obviously people care about the space because of money and finance but people also have genuine ideas about wanting to do better things in the world wanting to take power away from web-to-platforms that centralize everything and put more power in the hands of users. Cointelegraph is a ten-year journey into dreams of blockchain, crypto and better world. Cointelegraph is us, Cointelegraph is you, Cointelegraph is the future of money.


Two years ago Ukraine initiated Black Poseidon operation which was extended for next year also because of its success and now we are trying to make stress on tobacco-licent products because geographical position of Ukraine is very favorable for smuggling and tobacco counterfeiting. This conference is a good opportunity to meet colleagues from private sector to share some intelligence and to share some experience and also to meet colleagues from other countries to learn their experience in combating particular IP crimes. Partnership are important because IP crime is transnational crime so therefore without support of our foreign law enforcement community we are not able to combat any crime.


I could see this taking a week, so I decided to just make it number one priority today. And so far, this morning I just came out and did this one, and by the time I got all the stuff cut for this one, that one was dry, so I did this one, and I'd just been going the whole time non-stop, except for snack time. Now, this is the first one that's still not cured. I've already got my fiberglass cut for it, my pole, half pole. I didn't show these yet. It's just a PVC pipe cut in half with some slots cut, so it'll conform to the curve I'm putting into this. So, I'm just waiting for this. This one's going slower because the sun is behind the clouds right now. What I was doing was I would put up the umbrella, so I'd be in the shade, get all the fiberglass on here, then take the umbrella away, and once this is in full sun, cure is really fast. That's why I've been able to do so many of these at once. Oh, and I've also been putting paper on the bottom, so I don't have to keep waxing this form. Getting solid, still a little too soft to move though. Right now it's in a phase where it is solid, but if I move it, it'll flex, and then when it hardens, it'll take that shape. So, I want to keep it in this little form until it's hard. Still have time to get the last three done, if it doesn't rain. Oh, I guess while I'm waiting, I can stuff this thing into the boat. Some prop shaft tube. I lathed this down to the right side to stuff it in. Right size to stuff it in. Oh, yeah, that was my goal for today. Two hours of light left. What should I do now? What are you guys doing? For doll wedding? Now, what are the chances I can get this shaft off without the sleeves inside moving? Ordin one stayed where it should be. Besides those holes a little bit. I need another person to grab this at the top. Oh, monkey arms. Looks like I'm getting a free propeller test. Well, that's not enough space to get the motor in. I have to cut this a little. That's enough space. Looks like I still have enough clearance there, barely. I think they're in the right places. So good. Alright, don't hit, don't hit. Ah, it hits the bottom. The motor's hitting just under here.  I think I'll shave it off the motor, because I don't want to put a hole through that. It wouldn't really, it probably wouldn't matter if I put a hole through that. But there's no reason to. I can just shave off a bit of the motor on the other side. Just like a tiny bit. Oh, that should be plenty. Ah, that's the stuff. Alright, that sucker's in. It's just resting on a piece of wood right now. I'll have to work on, you know, mounting the frame later. Totally functional right now, though. Oh, look at that. There's a prop shaft at the back. No propeller yet. That shaft is much heavier duty than that one. Ooh, almost out of paint. This is actually light gray. I bought white and then I put some black. You could tell if you put something really white next to it. Solar panels will cover all that middle part. It did paint the spaces between where the solar panels go. In case the strip of light gets through there.


How would you describe the energy difference second half yesterday and what you guys did tonight? Yeah, we came out ready. Early, got Garnet sets, executed early, played defense, physical. Then left them real comfortable, looked real comfortable up there or be comfortable up there. And that's how we need to play every night. We've seen your game for a while now. We know you try to play with attitude and physicality.  As I said, I'm going to energize a buddy and a team. Once I get going, everybody else gets going. Because then they got to worry about me rolling. And then Joe, his threes, Kev gets, I mean Kev's going to do what he do regardless. But yeah, I just try to bring energy, get easy ones out and transition just to get our orphans going. On the other side of the court, I mean considering the slow starts that you guys had had pretty much every single game. A, how much impetus did you kind of put on yourself? You're obviously the primary defender perimeter-wise.  And secondly, when you're guarding a guy like Bradley Beale, what are the things that you try to take away, the things that you try to do to him to make him at least work for his points? I didn't put too much emphasis on it. I think everybody knew once they see me in the starting line that when I was going to bring out there what I was going to do. And then Brad, just a tough guard. Really, I just had to know what he likes to do. He goes right, usually he goes to the basket or the pullback between the legs. If he goes left, step back usually every time. So I just had to know my personal.


for the Lorne Jean Chronoscope, a television journal of the important issues of the hour brought to you every Monday, Wednesday, and Friday. A presentation of the Lorne Jean Wittner Watch Company, maker of Lorne Jean, the world's most honored watch, and Wittner, distinguished companion to the world honored Lorne Jean. Good evening, this is Frank Knight. May I introduce our co-editors for this edition of the Lorne Jean Chronoscope. Mr. William Bradford Huey, editor and analyst, and Mr. Richard Waddell, management editor of Business Week Magazine. Our distinguished guest for this evening is Mr. Charles F. Brannon, former secretary of agriculture. Mr. Brannon, our viewers are particularly interested in the price of meat and butter this week, sir, and I don't know if any man better qualified to discuss prices of those two things. Because, sir, our viewers remember that you were the secretary of agriculture and in the Truman administration, and of course now the new administration is in trouble over falling meat prices. Now, sir, why are meat prices falling now? Mr. Huey, there are several reasons. One of the main reasons is that the range conditions in the country have been poor and have been deteriorating for some period of time, which means that the man who produces the first animal, the animal on his road to the butcher shop, has found it necessary to cut down his numbers because his range won't carry them. That set against the background of the fact that we have built our beef animal numbers, our livestock numbers in this country, to the highest point in, I guess, all our history. Well, that's that building them to the highest point in our history. Brings me to this point, sir. Was it your policy while you were secretary of agriculture to encourage abundance, meaning to build, to get as many beef cattle as possible in the country? Yes, it was. The philosophy of abundance, which I think is inherent here really in the present law, simply is founded upon the idea that if you produce a large quantity or have a substantial quantity of a given commodity, you thereby begin to certainly control price or prevent runaway prices. Can you give us an illustration of how this works and what is different, perhaps, between your philosophy of farm policy and Mr. Benson's? Well, I would say, let me say first of all this about the present administration and Mr. Benson's farm policy. I'm not going to be critical of him. It is as important to me as a citizen that he solved the problems much more important to me as a citizen, and to you and all the rest of our people. Then it is that we gain some political advantage by as a result of his failure to solve them. I hope he's able to solve them, and as far as I'm concerned, I'd like to be as useful as I can in the solution. But sir, is there some basic difference, as you understand it, between your philosophy of government and his philosophy of government and relation? I would say that his statement and to simplify it and bring it concise as we can, the use of the word disaster insurance indicates a very direct takeoff at attention from the existing philosophy under the existing law. The price support laws of the country have been designed to maintain purchasing power in American farmers to the end that they are able to take care of the basic resources without which this country cannot be strong. Now, therefore, we have we have tried to keep a fair return to the American farmer while encouraging him to produce abundantly. I understand from the from the disaster insurance idea that you wouldn't help the farmer until his prices had gotten so low that he'd lost his purchasing power to begin with, and about all you'd give him is enough to put him back in business the next year as a farmer. If that's true, why is it that the the American Farm Bureau and the National Grange both have indicated their support of Mr. Benson? Benson's policies is so far indicated. Well, first of all, let's represent farmers. I they are the hierarchy of the American Farm Bureau and of the Grange have a philosophy, which in my opinion is not the philosophy of the many of their members. Is that would you call that a low support philosophy? It's a low to no support philosophy. Low to no. And and and it's proper to identify your philosophy as a high support philosophy. Yes, there's a strong support of philosophy. Now you were explaining the idea of the of the hierarchy. Well, they the the existing law or or the strong price support philosophy, of course, was the philosophy of the membership and the hierarchy of the American Farm Bureau, until it changed leadership. Now, if the membership changed leadership knowing that they were going to a low to no price support philosophy, I'm I would be doubtful that they would have ever made the change they made. Do you think Mr. Benson's policy, although his actions, he's supporting the law, but his disaster statement would indicate something else. Are those policies his own? Or are they administration policy? Are they something that in other words, is he the man who must be blamed if there is any blame to be placed? Well, I'm confident that Mr. Benson is expressing an administration policy. And who are the men? Do you think who are fashioning that policy? The president, of course, must be the final architect or must say the final yes or no. I think advising him, of course, is his brother Milton is Mr. Allen Klein, President of the American Farm Bureau Federation, and a few people, all of whom are low or no price support people. Well, is it is it fair to say, sir, that there's a difference in degree? I mean, we use this high support and low support. You wanted the Department of Agriculture to assume more responsibility for assuring abundance or assuring the farmer a return. And whereas your successor now, Mr. Benson, wants to assume, wants the Department of Agriculture to assume less responsibility. Is that a... Mr. Hewitt, I wouldn't say that was quite the right description. I think it is the people's concern to maintain stability and high level production in farm commodities. Now, maybe price support isn't the mechanism, but it's the best that anybody has thought of. And it is essential. For example, take this question of milk that we were talking about a few minutes ago. We have an excess of milk at this season of the year always. Now, do we... The question is, do we allow the price of milk to go so low as to penalize the farmers who have produced the abundance now? Or do you find a device by means of which you can encourage them to produce this abundance, pass that the benefit of the abundance onto the American consumer without making the farmer share all of the financial responsibility? What happens to the dairy farmer if the price of milk goes too low? Well, he goes out of business. He sells his cows. And remember, it will take you as long to get a new cow back into production of milk as it takes to build the biggest battleship we've ever floated. It'll take them two and a half years. And if he sells his cows for beef or bologna, they are going to be producing next year. That's a low season. And it's your belief, sir, that the president administration, the Eisenhower administration is not evidencing enough concern for the preservation of those herds. I think so. I think if I were to say anything critical, I probably would would run along that line. Let me say it another way. I think the American people should always take their risk on having a little more than we need than then having taking their risk on having a little less than we need of our food commodities. That brings up a question. Do you think there is has been a decline in some some concern among the farm groups about in prices about prices? Do you think that this price decline in farm commodities is significant? Does it historically a drop in farm prices bodes a recession or a downturn? Do you think that's what we have the fear in this? Well, historically, in practically all of the depressions that have come to this country, at least the ones in modern times, the farmer prices have gone down first, gone down farthest and stayed down the longest. So there is something in the present indications to to watch with great great concern. Now, I don't want by any stretch of imagination to say that we've reached catastrophic you. Do you think that the farmers of America are likely to suffer in the next in the foreseeable future in the next year or two? It's it's much too early to say as far as I can tell. Do you think to our viewers, sir, do you think that the price of food is going to continue to go down? Do you think the price of milk will go down? The price of meat will go down? No, I'm not to consumers. If the if the drought continues out in the Western area, there may be further liquidations of cattle, which would cause a further reduction of return to farmers. But that isn't always reflected to consumers. Prices were coming down for almost 18 months to farmers for for beef animals. At the same time, actually, some prices of meat or prices of some cuts were going up. You bring up a point there. Senator Taft has said recently that this farm problem, if there is a problem, is not a Republican problem, but the one that they inherited from the Democrats because the prices have been falling for about two years at the farm level. Well, let me put it this way. Just just plain fact, the price of beef did not go to 90 percent of parity during the Democratic administration. And there was no law which would have let us do anything until it did. I'm sorry, Mr. Brown and our time is up. Thank you very much for being with us this evening, sir. The opinions that you've heard our speakers expressed tonight have been entirely their own. The editorial board for this edition of the Lone Gene Chronoscope was Mr. William Bradford Huey and Mr. Richard Waddell. Our distinguished guest was Mr. Charles F. Brannon, former secretary of agriculture. Do you know when Easter comes this year? Well, here is a rough rule. Easter is the first Sunday after the full moon after the first day of spring. Spring begins, here's the full moon, and here's Easter Sunday, April the fifth. Now, how about giving someone near and dear to you a magnificent new Lone Gene watch to wear on Easter Sunday? You know, giving a Lone Gene is almost like selecting a watch made to your special order because every year, Lone Gene produces literally hundreds of new and exclusive styles and models. This has always been a Lone Gene policy because we realize that those who buy Lone Gene watches expect such exclusiveness. Above and beyond faultless performance, above and beyond unparalleled accuracy, a Lone Gene watch offers pride of possession. For a Lone Gene is, in fact, the world's most honored watch, the only one of the world's finest watches to win so many honors for excellence, elegance, and accuracy at world fairs and in observatory accuracy competitions. For Easter, for an anniversary, for a birthday, or for any important occasion, throughout the world, no other name on a watch means so much as Lone Gene, the world's most honored watch, the world's most honored gift, premier product of the Lone Gene with Norwatch Company since 1866, maker of watches of the highest character. We invite you to join us every Monday, Wednesday and Friday evening at this same time for the Lone Gene Chronoscope, the television journal of the important issues of the hour broadcast on behalf of Lone Gene, the world's most honored watch, and Wet Nor, distinguished companion to the world honored Lone Gene. This is Frank Knight reminding you that Lone Gene and Wet Nor watches are sold and serviced from coast to coast by more than 4000 leading jewelers who proudly display this emblem, agency for Lone Gene Wet Nor watches. Sunday nights, it's Ed Sullivan's Toast of the Town on the CBS television network.