Welcome, in this lecture we will start our study of non-linear optimization techniques and in this topic, we will devote 3 to 4 lectures and in the current lecture I will first summarize the general methodology of optimization and briefly recapitulate the topic of single variable optimization which you are already conversant with and continue into developing the conceptual background of multivariate optimization. The actual multivariate optimization method will be taken up in the next lesson. First, in any typical situation where you encounter an optimization problem, to begin with you will have a number of variables which you can choose in order to minimize or maximize something, some function. That function which you try to minimize or maximize is called the objective function and among the variables which you can choose in order to have the minimum or maximum value of the objective function, those underlying variables are separated into two parts. One of them that is some of the variables you can treat as parameters which are kept constant for one particular study. In a particular problem, you may choose to keep all variables as design variables in hand and process all of them together. On the other hand, some of the variables at some situations are kept constant as parameters. After fixing the values of the parameters, the remaining variables which in a particular study you want to explore in to get the best possible value for the objective function, those only are treated as the variables of the optimization problem. And a typical statement of an optimization problem goes like this in which you say you want to minimize a function of x subject to certain constraints. So, there may be an optimization problem in which there are no constraints or there may be one in which there are constraints. Constraints are again of two kinds. One is inequality constraint, the other is equality constraint. Now, after formulating the problem, now this formulation part comes from the domain in which you are going to apply the optimization methods. Now, after studying the optimization methods, you would notice that in almost every branch of science and engineering and even humanities quite often, you come across situations where many problems can be solved through an optimization formulation. The problem may be one of explicit optimization where you actually want to minimize or maximize something or many times it happens that the actual problem is something else, but you can reformulate it in the form of an optimization problem. For example, in the last lecture, we formulated an equation solving problem in the form of an optimization problem. So, there are many such problems which can be formulated in the form of an optimization problem and then optimization methods can be used in those problems with advantage. Now, after the formulation is made from any given field, then you look for a suitable optimization method or algorithm to find a solution of this problem. Now, any point x, any variable, any set of values for the variables x that satisfies these constraints g x is less than equal to 0 and h x equal to 0, the given constraints or any such point is called a feasible solution that is it is allowable, it is permissible by the constraints imposed on the problem definition itself. And out of those feasible solutions, you try to find the one in which the function value is minimum if it is a minimization problem. The optimization problems can be of minimization or maximization type, but in most of the theory that we discussed, you will find that most of the time we are talking about minimization. Fixing our attention to minimization problem helps to keep the entire theory in one standard form. If a problem is of maximization, then we can always try to minimize the negative of the objective function. So, this is typically done in order to avoid the hassle in the notations itself. Now, after you apply some optimization method, a good number of them we will be studying in this course. So, after you apply that method, you get a solution which is the solution of this optimization problem. That means, it minimizes the function objective function and satisfies all these constraints. After getting that solution in hand, quite often you want to find out whether the parameters that is that subset of variables which you kept fixed, whether it was a nice idea, whether it was a wise idea to keep those values fixed. So, in that case, after you have got the solution in hand, then you conduct a sensitivity analysis. You try to find out that how sensitive is this solution that we have got in hand, how sensitive it is to the values of those parameters. If they are found to be very sensitive, then you try to see whether those parameters can be also changed in order to get a much better solution. On the other hand, if you find that the solution is quite insensitive to the parameters that you have fixed as parameters, then you say that fixing that is a wise idea. Unnecessarily, we need not conduct the optimization process with too many variables. Now, optimization problems as I told you just now can be unconstrained without these constraints in which case the entire space of x is feasible or they may be constrained. In that way, we classify optimization problems and for that matter optimization methods into unconstrained optimization and constrained optimization. Obviously, unconstrained optimization problem is easier to solve compared to constrained ones. Then, you also classify optimization problems as linear and non-linear problems. If both the objective function and the constrained functions g x and h x are all linear functions, then you call it all the optimization problem, a linear optimization problem or linear programming problem, LP problem. On the other hand, if either the objective function or any of the constrained functions is non-linear, then it is a non-linear optimization problem or non-linear programming problem, NLP problem. Now, you could also classify the optimization problem as single variable and multivariable problem. Single variable problem has a single variable and in multivariate problems, you have several variables in hand with which to play around to minimize the objective function. Now, you will notice that when we classify optimization problems as unconstrained and constrained and in another way, we classify them as linear and non-linear. In total, we do not get four kinds of problems. That is, any of the two above and any of the two below we cannot combine because in the case of a linear optimization problem, you cannot have it unconstrained because linear functions go on reducing in a certain direction, in certain directions. So, if there is no constraint on the variables in x, then they are unbounded on the lower side as well as on the upper side. So, linear unconstrained optimization problem does not exist. So, if it is a linear programming problem, then constraints will be there in any case. So, you get linear programming problem, which are linear constrained problems or unconstrained non-linear problems and constrained non-linear problems, which is the most difficult. Now, before going to the methods for multivariate optimization, which is going to be our main focus, let us quickly recapitulate the ideas of single variable optimization with which you are conversant already to a good extent. Say, for a function f of x, a single variable, a point x star is defined as a local minimum point. If there is some epsilon such that the value of f x at all other points that is all points in that neighborhood, in this epsilon neighborhood that is epsilon on this side and epsilon on that side, at all other points, the function value is greater than or equal to the function value at the current point x star. Then, you will define the point x star as a local minimum point. Now, schematically, let us have a look at this. There is a function, which is defined over the interval a to b. In this particular case, x 1 is a local minimum point and then x 2 is a local maximum point. x 3 is neither a local minimum point nor a local maximum point, because there is some point close to x 3 on the right side at which the function value is likely to be lower and this will not be satisfied. So, this is not a local minimum point. It is neither a local maximum point, because on the left side of it, the function value is likely to be higher. So, this is a point of infection. Now, here you find it is again a local minimum point. So, see the difference between this point, this point and this point. Here, it is a clear local minimum point. Here, it is not. Here, it is here the function profile comes from upward and then becomes constant for a while and moves up again. So, this is a minimum point. Here, the function profile comes downward, becomes constant for quite long and then goes down. But then, when we say that it becomes constant for quite long, it is not necessary that over an interval around x 3, it is constant. It is just that it touches its tangent more smoothly. That is possibly first order and second order derivative both are 0. Here, you find that a constancy kind of situation lingers for a little more smoother interval around it and then it goes up. So, it is a minimum point. You will notice that even b is a minimum point. Now, here the curve does not become flat, but since beyond b, the function is not defined. So, this is also a local minimum point. The function is defined only on the left side and all points on the left side are above the current point. So, b is also a minimum point. So, in this schematic x 1, x 4 and b are three minima. On the other hand, a, x 2 and x 5 are maxima. x 6 is neither a minimum nor a maximum. Similarly, x 3 is neither a minimum nor a maximum. Now, that is according to this definition and those points where the function is differentiable. There, you can find out certain optimality criteria based on the derivative and that is the first order necessary condition says if x star is a local minimum or maximum point and if the first derivative exists, then it must be 0. If the first derivative is non zero at that point, then if it is positive, then on the right side it will be, it will go up. The function value will go up. On the left side, the function value will go down. Similarly, if the derivative is negative, then reverse will happen. One side will be higher, other side will be lower. So, that way it can be, the point can be neither a minimum point nor a maximum point. Therefore, for being local minimum or maximum point, the first derivative can neither be positive nor negative and therefore, it must be 0. Now, note in this case what is happening? The first derivative is 0. The tangent is horizontal. What is the tangent before and what is the tangent after? That is what is the slope before that point and after that point. Before that point, the slope is negative going downwards. After that point, the slope is positive. At that point, the slope is 0. So, you find the first derivative is 0 and first derivative as x changes, as x increases, the first derivative is first negative, then 0, then positive. That means, the first derivative is an increasing function of x, which means that the second derivative is positive. So, that gives you the second order condition. Second order necessary condition is that the second order derivative is non-negative, positive or 0. That is why, this is also a minimum point. This is also a minimum point, but this is not. So, it is necessary, but not sufficient. Second order sufficient condition will be that the second derivative is positive. Now, if the second derivative is 0, then it satisfies the necessary condition, but not the sufficient condition. So, to resolve the situation, you need to go further. The way to go further is through Taylor series. So, if you write the Taylor series of f of x around a given point x star, the candidate point, then you get f of x star plus delta x is equal to f of x star plus first order term plus second order term plus third order term and so on. Now, keeping f x star that is transposing f x star on the other side, you talk of the change in function value. So, the change in function value from x star to x star plus delta x at that neighboring point. The change in the function value from Taylor series is given like this. Now, here you find that as long as the first derivative at that point is not 0, for small enough interval, for small enough delta x, this first term will dominate this series and the sign of this first order difference will depend on the sign of delta x. That is whether you are taking the other point on the positive side or on the negative side, whether delta x is positive or negative. Depending upon that, the first order term will change its sign and that term is going to dominate the series for sufficiently close points, for sufficiently small delta x. That will mean that on one side, this will be positive and the other side it will be negative and therefore the difference being positive on one side and negative on one side will preclude the possibility of the current point being a minimum or maximum point. Therefore, for the current point to be a minimum or maximum point, it is necessary that this derivative vanishes, the first order necessary condition as we saw just now. If this point vanishes, then for sufficiently small delta x values, this entire series will be dominated by the second order term and the sign of the second order term does not depend on the sign of delta x because delta x is appearing as a square. So, irrespective of whether you go this way or that way, delta x square is positive. It will depend upon the derivative sign. If the derivative is positive, then this will be positive for sufficiently close points. Now, if it is positive, that means the neighboring point both sides have higher function values. So, that will qualify the current point as a local minimum point. Similarly, for negative values of value of this, it will be a local maximum point and that is why the second order derivative being positive with the first order derivative 0 is sufficient condition for the current point to be a minimum point. If it is 0, if the second order derivative is also 0, then again the series will be dominated by the third order term, the sign of which will again depend on delta x because you see delta x cube appears odd power. So, this goes on. So, therefore, now looking at the pattern, you can say that for an extremum to occur at point x star, the lowest order derivative with nonzero value should be of even order. If up to third order, it is 0, fourth order is positive, then again it is a local minimum point. So, that gives you a working rule for determining candidate points and then classifying them as minimum, maximum and so on. So, first of all, you evaluate the first derivative and set that equal to 0 and solving that, you try to find out candidate points x star. So, such a candidate point is to begin with a stationary point. It may be a minimum point or a maximum point or it can be a saddle point, an inflection point. So, after we have captured certain candidates for further test, then at that point, at those points, we evaluate higher order derivatives. Still one of them is found to be nonzero. If we go on finding derivatives and several of them are found to be 0, second order, third order, fourth order, fifth order, then we stop at that point where first nonzero derivative is encountered. If its order is odd, then the current point x star is an inflection point, coming like this, going like this, or coming up and then going further up like this. If the order of that first nonzero derivative is even, that is either the second or the fourth or the sixth and so on, then that will be a local minimum point or a local maximum point depending upon whether that derivative is positive or negative. So, this much you have studied long back in 12 standard calculus itself and this was the working rule for finding maximum and minima at that stage. However, it requires the solution of an equation. Now, solution of an equation is always not an easy task. In the previous lecture, we also discussed the situation where for solving an equation, we formulate it as an optimization problem. So, equation solving is also not always extremely easy. There may be equations which are very difficult to solve. So, quite often, we do not try to rely on the equation solving process to capture the candidate points, but we follow an optimization based algorithm directly to find the minimum point. Now, for that, there are several methods. Now, there are some of these methods depend on gradient one way or the other and some other do not depend. That is, some of them depend on derivatives, some of them do not depend. Even those which depend on derivatives, some of them use derivatives explicitly and others do not. For example, Newton's method, which is reminiscent of the Newton-Raphson method of equation solving. So, in the case of equation solving, we had the, when we required the solution of an equation like this, then our typical iteration was x k plus 1 is equal to x k minus phi of x k divided by phi prime of x k. Now, here we are talking about minimizing the function f. Now, we know already that at the minimum value, at the minimum point, f prime is going to be 0. So, why not try to solve f prime x equal to 0? So, if we want to solve this equation, f prime x equal to 0, then in place of phi, if we put f prime, then we get f prime here, f double prime here, that is second derivative. That is the typical Newton's method for optimization of a single variable problem, single variable function. One difficulty of this is that, this formulation will not differentiate between a minimum point and a maximum point or an inflection point with 0 derivative, 0 first derivative. Now, here itself, in the case of second derivative, if we replace that with a finite difference kind of derivative formula, then we get this formula, which is the second method. Now, in the second method, you will notice that we do not need the second derivative, but we need the first derivative and the function value at two points. So, Newton's method works with a single point up to second derivative, which also means that the second derivative should exist. Second method works with two points at a time, that means in the initial guess, we need to give it two points and only up to first order derivative. Method of cubic estimation is another method, which uses function values and derivative values at two points. Starting with two points, it evaluates the function value and the derivative value at these points and then, that means that we have got four total number of four conditions, four conditions in total, two conditions at this point and two conditions at that point. Function value, derivative value, function value, derivative value and with this kind of four conditions, we can fit a cubic function in the local neighborhood. That is, for the local neighborhood, we can approximate the actual function by a function of this kind, a cubic fit. So, as we impose the conditions, that is at x 0 and x 1, we are prescribing the function value and the derivative value. As we prescribe these conditions on this, we essentially get four conditions, four equations, four linear equations for that matter in the coefficients a 0, a 1, a 2, a 3. From that, we can determine these coefficients and then, we say that we look for that point, where this cubic function is minimized or its derivative is 0. So, that we get in terms of a 1, a 2, a 3, etcetera and that becomes another point. Now, out of the two original points and this third point, we retain two points and drop one of the old points. Again, at two points, we evaluate the derivative and some derivatives and function values are already there. At the new point, we evaluate the function and derivative and continue. So, this is, this method is called the method of cubic estimation. Similarly, there is also a method of quadratic estimation. That operates not with derivative at all, that operates only with function values, but at three points. So, to begin with, you need to prescribe three points to this method and through those three points, only with the help of function values, the algorithm frames a quadratic a 0 plus a 1 x plus a 2 x, this is square, a 2 x square, only up to this and by fitting that quadratic with three coefficients, because of with the help of three function values and then asking for its gradient to vanish, it gets the new point. So, this method is method of quadratic estimation. So, in which only function values are used, no derivatives. Now, note that whether some of these methods use derivatives here up to second derivative, here and here up to first derivative, here no derivative, but then still all these four methods in an indirect manner refer to the vanishing of derivative, because that is the test, that is the requirement based on which the new point is generated. So, the disadvantage of all these methods is that, it treats all stationary points alike and does not differentiate between a minimum and maximum, that is a disadvantage in these methods. So, if the problem is such that it has a minimum and perhaps not a maximum, then any of these methods will work out nicely. On the other hand, for a problem which has lots of minima and maxima, this kind of a method runs the risk of reaching a maximum point. There are some other methods which first insist on a bracket and second, do not make any reference not even an indirect one to the derivative. First, what is this bracketing? In the case of equation solving or root finding problem, we refer to the continuity of a function and say that if there are two points x 0 and x 1 and the sign of f x 0 and sign of f x 1 are different. One is positive, the other is negative. That means due to continuity, it is necessary that at one point in between x 0 and x 1, the function is bound to cross the 0 line and that is the root. So, that was the way to bracket the solution of an equation. In the case of minimization problem, the bracketing has a slightly different meaning. If there are three points x 1, x 2, x 3, x 1 less than x 2, which is less than x 3, such that at x 2, the function value is lower than both x 1 and x 3. Then, we say that between x 1 and x 3, there is a solution. So, if we have a pattern of the function which is like this, then we can say that in between this, there must be a minimum point. Because it is known that from this point, the function value has gone down and then it is known that from this point, the function value has gone up. So, in between, what is the point here or here or somewhere where that actual trend is made? Now, these three could be like this, in which case the minimum is here or it could be like this, in which case the minimum is here. So, this is important. So, bracketing in the case of minimization problem requires three points and the pattern or trend of downward and then upward should be established to identify a bracket. Now, once such a bracket is there, some of the optimization methods, some of the single variable optimization methods try to continuously squeeze the bracket. Bisexuality is one possible way. For example, if we know that in between this, there is a minimum point, then in a similar manner of bi-section, we can try to see that whether this half is going to constitute a bracket or this half is going to constitute a bracket. There is which half of the complete interval is going to retain the nature of the bracket and then like that we can squeeze and find a solution. However, compared to bi-section, two other methods are found to be more efficient. That is, they conduct the same job with the same accuracy with less number of function evaluations. One of them is Fibonacci set, in which the interval reduction is not through half at every iteration, but in a variable fraction, the interval is reduced. The squeezing takes place at a variable rate and the subintervals are decided based on Fibonacci numbers. So, F n minus 1 by F n, in this way, you try to reduce the size and through this measure, what you ensure is that for bi-section, this interval, if you evaluate one point here, evaluate the function at one point here and at another point here, at same distance from the two end points of the interval and then either you retain this point and throw away this or you retain this and throw away this. Out of these, whichever is larger, whichever maintains the bracket, that is retained and the other is removed from here and that way, what happens is that in the next, the way the fractions are generated with the help of Fibonacci numbers, it becomes obvious that the in the next round in this interval, the two points that will be needed where the function will be evaluated out of that one will be this itself and the other will be symmetrically placed here. So, at every new iteration, the two new points, the two new internal points, interior points that will be needed, one of them is one of the old points. So, at every iteration, only one new function evaluation is made and every function evaluation is used twice on an average. Now, from the Fibonacci search method itself, one particular another search is developed, which is golden section search, in that the interval reduction fraction is not variable, but it is constant and it is equal to this golden section. So, in the golden section search, a similar operation is done, but at every iteration, the interval reduces by this fraction, which is the golden section ratio. Now, through this squeezing of the bracket, through this interval reduction at every step, there will be a stage where the interval is so small that is smaller than your requirement of accuracy. For example, if you wanted the solution up to an accuracy of 0.01, then by the time the size of the interval itself is less than 0.01, you say that any of these points is good enough as a solution. So, that is the way Fibonacci search and golden section search method follow, operate and they keep on squeezing the bracket and finally, make the bracket so small that any point in that bracket is good enough for the required accuracy. Now, with this much background of single variable optimization recapitulated, now we will go to discuss the actual problem of our focus, which is multivariate optimization. First unconstrained optimization in this lecture and the next and then we will study a little constrained optimization. Now, in an unconstrained minimization problem, the point a point x star is called a local minimum of the function, if there exists a delta, whatever small you like to choose, if there exists some delta such that within a ball centered at the current point x star and radius delta, all points have the function value, which is greater than or equal to the current point under question, then the current point x star is called a local minimum point. Now, this is the basic definition of a local minimum point and note that we are talking about local minimum point and most of the algorithms, which we will be discussing cater to the problem of finding a local minimum point only. Now, you can talk of finding all the local minima, several local minima and then out of them choose the smallest one and hope that that is the global minimum, that is the one option, if you want if your problem demands to find the global minimum. Now, if the function is differentiable, then you can work out some optimality criteria as we did in the case of first in the case of single variable problems based on derivatives. Say again making an approach to the Taylor series, we find that if x is a point neighboring x star, then the difference of function values f x minus f x star will be the first order change, which is gradient transpose delta x, where x minus x star is delta x plus half delta x transpose Hessian second derivative matrix into delta x plus the higher order terms. So, up to this is the truncated second order truncated Taylor series. Now, for x star to be a local minimum, again you argue in the same manner that as long as this gradient is non zero, there will be some directions delta x along which the function will increase and some directions along which the function will decrease. Now, a direction along which the function increases will ensure that in the opposite direction the function will decrease. So, the current point cannot be minimum or cannot be maximum. So, for minimum or maximum for any extrema, the first order term must vanish, which means the gradient as a vector the complete gradient vector must vanish all the partial derivatives should vanish. And then this second order term dominates the series for sufficiently small delta x and in that case the sufficient condition is the positive definiteness of this Hessian matrix at that point, which will ensure that for all delta x it is positive that is sufficient condition. Necessary will be that it is positive semi definite and indefinite Hessian matrix with some Eigen values positive and some Eigen values negative will characterize what is called a saddle point. Note that we can talk of the first order condition or second order condition, only when the function is first order disadvantageable and second order differentiable and so on. So, only for first order only for function which are differentiable up to the first order we can take off, you can talk of first order condition and only for functions which are differentiable twice, you can talk of the conditions. Now, with these optimality criteria, we proceed towards a few further issues, which will be found quite useful when later we consider multivariate optimization methods. And the most important issue in that direction is convexity. So, there are two aspects of convexity, a convex set or a convex domain and a convex function. Now, in the R n that is n dimensional real space, a set S or a region is called a convex set. If for all pair of points belonging to that set, the complete straight line segment joining them is also inside the in the set. That is, this region is not convex, because in this you can find two points, which are which belong to the set, but the straight line segment joining them does not completely lie within the set. Now, on the other hand, this region is convex, as for every two points inside the set, the straight line segment joining them will be completely inside the set, inside the region. Now, for unconstrained optimization problem, the region question will not arise, but it will arise in constrained optimization problems, but then it will be important, because further we will be defining a convex function, which can be sensibly defined only in a convex set or convex domain. Now, saying that the straight line segment joining the two points is saying this, that is for alpha belonging to 0 to 1 interval, alpha x 1 plus 1 minus alpha x 2. That is for alpha equal to 0, you get x 2, for alpha equal to 1, you get x 1 and for any intermediate value, you get a point in the line segment joining x 1 and x 2. So, such a set for which, in which for every two points, this will hold that the entire straight line segment joining the two points also will be, will belong to the set, will belong to the region, such a region is called a convex region or a convex domain. Now, over a convex set, over a convex domain, you can define a function f x, which will be a convex function, if for every two points belonging to that region and alpha again between 0 and 1, the function value at an internal point in that line segment is less than equal to the corresponding linear interpolation between the function values at the end point. Now, if this is not very clear, then think of it in this way, that is you have the function value at x 1, that is f x 1 and you have the function value at x 2, that is f x 2 and you want the function value at a point, which is intermediate between x 1 and x 2, say at 0.2 fraction of the distance from x 1 to x 2, that is from x 1 to x 2 in that line segment, 0.2 distance from x 1 and 0.8 distance from x 2. Now, 0.8 into x 1 plus 0.2 into x 2 is that point, you evaluate the function at that point and that is the function value here. Now, rather than evaluating the function value at that point, if you had tried to interpolate it from the function values at the 2 end points, then you would get this approximation, 0.8 into f x 1 plus 0.2 into f x 2. Now, the function is called a convex function, if in every situation, this interpolated function value is always an overestimate, that is chord approximation, interpolated approximation, if it is always an overestimate compared to the actual function value at all intermediate points, then it is called convex function. Equality is permissible, that is it can be equal. So, schematically seeing this is a convex domain, because any two points that you can take in this will ensure that the straight line join of those two points is completely inside. Now, this is an example, this graph of the function that is shown is a convex function, because you see that if you take two points x 1 and x 2 and the function value is here. Now, a linear interpolation between them, a chord approximation will be given like this. So, at this point, the function value through a chord approximation will be found to be this, whereas the actual function value is here, actual function value is lower and the chord approximation is higher, chord approximation is an overestimate. So, this kind of a function is called a convex function, that is for being a convex function such a thing must happen at every intermediate, every pair of points for every intermediate value. Now, you will see that other than the chord approximation, if you wanted to make a tangent based approximation, that is you know the function value here and you want to you know the derivative also at this point and based on the function value and the derivative through a first order Taylor series, you want to approximate the function value somewhere else, that is the tangent approximation. And the tangent approximation will be always an underestimate, it will be lower see the tangent is going lower on this side as well as on that side, compared to the actual curve actual graph the tangent approximation is lower on the left side as well as on the right side. So, such is the property of a convex function and the chord approximation is an overestimate that actually in a way implies that the tangent approximation will be an underestimate and that you can show through a few small steps which we will not going to the detail currently. The only thing that we need to stress at this point is that this gives you a first order characterization of convexity. This is the zeroth order characterization of convexity which is the definition in terms of only function values and this is equivalent to the first order characterization of convexity, which you can talk of if the function is first order differentiable, that is f x 1 is greater than equal to f x 2 plus gradient transpose x 1 minus x 2. You can work out a second order characterization also through another few small steps and that is actually quite straight forward. The second order characterization of convexity is that the Hessian matrix, the second order derivative matrix is positive semi definite. It is the function is strictly convex if it is positive definite. On the other hand if it is possibly semi definite then it is just convex. There is a certain class of problems in which the region the domain the feasible domain is convex and the function that we want to minimize is also convex. Such a problem is called a convex programming problem that is we try to minimize a convex function over a convex set and in that kind of a situation a local minimum is also a global minimum and all minima are connected together in a convex set. So, convexity is a very strong condition on a function. Further you know further nicely behaved function we will find in a quadratic function which could be convex or could be non convex also, but a convex quadratic function turns out to be a benchmark problem against which all optimization algorithms are qualified. A quadratic function is like this and if you try to find out its gradient and its Hessian then you will find very easily through first order and second order derivatives as a gradient is this a x plus b and the Hessian a is constant quadratic function. So, second order derivative should be constant and Hessian is this matrix a the second order derivative. Now, what kind of a matrix is this Hessian a if it is positive definite then you will say that it is a convex quadratic function and quite often when we use a quadratic function as a benchmark problem then we consider convex quadratic function in which the Hessian a is positive definite. Now, if a is positive definite then it is non singular as well which means that this equal to 0 will have a unique solution a x equal to minus b will have a unique solution and that unique solution will satisfy the first order condition gradient is 0 and the Hessian is positive definite anyway that together will satisfy the sufficient condition for that particular point to be a local minimum and that is the that since 0 gradient has that as the unique solution. So, that unique solution is the only minimum point of the function if a is positive semi definite then it is singular as well and in the case of singularity this system of equations this equal to 0 may be consistent or may not be. So, if it is consistent that is if minus b is in the range of a then in the case of singularity of a positive semi definiteness you will have infinite solution and all those points all those solutions all those infinite solutions of this are local minima and global minima as well and they are together connected that is they are distributed over an entire line or entire plane like that. So, that is again a convex set if a is positive semi definite, but it is this this system of equations this equal to 0 is inconsistent that is if minus b is not in the range of a that will mean that the convexity is not a problem, but 0 gradient condition is not satisfied anyway. So, in that case the function is unbounded and the minimization problem has no solution these second and third cases the first case is very simple the first case is like this this is the shape of the function. So, this is the minimum point 0 gradient 0 derivative condition is satisfied only here and the function is convex everywhere the second derivative matrix is positive definite everywhere it is constant. So, this is the unique minimum on the other hand the second case here a is positive definite positive semi definite that is singular and minus b is in the range of a that means that there are points where the gradient vanishes that is the function profile which is like the cylinder. In the cylinder you find that it is convex semi definite because you see along this direction it is straight and along this direction it is convex like this. So, this is a semi definite case and when you try to solve a x plus b equal to 0 0 gradient 0 slope then all these points satisfy the 0 gradient condition. And therefore, this entire line is the solution of the minimization problem all these are the points at which the level of this function is lowest. On the other hand the third case here this one is a same cylinder, but not placed horizontally but like this. In this case again it is semi definite because there are directions in which there is no convexity concavity and there are directions in which there is convexity. So, here 0 gradient condition this a x plus b equal to 0 that condition is not met that is there is no point which satisfies the 0 gradient condition as you can see on this surface there is no point at which the gradient is 0. So, that is why the function is unbounded that is along this direction you can go on going downward and there is no end to it compared to this case where you cannot go downward. So, 0 gradient condition is not met in this kind of a situation at any point. So, that is why in this case there is no solution now as benchmark problem we typically consider those quadratic functions for which the Hessian is positive definite non-singular case. Now, for an optimization problem algorithm we need to have a good picture a clear picture of how a typical optimization algorithm operates. Typical way to operate for an optimization algorithm is to start from a current point move to another point which is hopefully better than the first. Now, there are three questions that arise in this process first is which way to go second is how far to go and which decision is taken first. If we first decide the direction which way to go and then decide how far to go in that direction this gives us one strategy of optimization algorithm that is called the line search strategy. On the other hand if we first decide that within this much distance we are ready to go and then we decide that within this much distance in all directions which direction to take and how far to go that is how far we are ready to go if we take that decision first and this decision we take later then that is a strategy which is called the trust region strategy. Now, there are some algorithms which can be implemented in both the strategies some of them can be implemented in only one of the strategies. For any optimization algorithm there are two questions that arise one is the question of global convergence that is whether at every step the algorithm makes an improvement in the function value whether it approaches the optimum point and that is the issue of global convergence it is in terms of guarantee whether there is a guaranteed decrease of the function value whether there is a guaranteed approach towards the minimum point. The other issue that arises in terms of convergence of algorithms is the local convergence that is what is the speed of approach if we start sufficiently close. So, global convergence refers to the guarantee of approach from anywhere in the solution space local convergence refers to the speed of approach if started sufficiently close. So, some of the methods have linear convergence rate which are typically the slower method some have quadratic convergence rate which have which are typically fast method and there are algorithms which are in between. Now, with this much background in the next lecture we will try to study optimization methods and currently the points to note are here and quite a few exercises are there in this lesson in this chapter of the book and some of them you must attempt on your own to be very conversant with the idea behind this subject matter. Next lecture we continue into optimization methods. Thank you.


T.L.O. what's poppin? We are on kickkiscake.com We are live and you can join us. Link is down below. Don't forget to leave a like, comment, subscribe, turn on your post notification bells. Let's continue to grow the family from Chicago to the UK. If you don't want to join live for some odd reason and you faking like you got something to do when you really don't You can catch the highlights on the lit one live. Link to this also in the description. We also got merch. We also got patreon. What are we watching on patreon right now? We're watching line of duty. Salute. Appreciate the follow. Line of duty. Only fools and horses. Fresh meat. Sherlock. And afterlife. Those are the five active shows that we're watching right now. So if you're into any of those, you know, lock in. Don't forget we got a discord as well. The link to all of these are down in the description inside of the link tree. Link tree is a link that holds all of my socials and anything else you need. H&P Wormwood. Scrubs. British prison documentary. Apparently this is a well-known prison. A lot of rappers mention it. That's what the chat told me. Can we pull out the chat now? No, we're waiting another 45 seconds. Wormwood Scrubs. One of Britain's most famous prisons. From Pete Doherty to Keith Richards. It's locked up the country's ultimate rock star rebels. I really think that it was part of the Jewish issue. His hope that they might be gang raped, humiliated. And it's been home to some of the country's most dangerous inmates. Charles Bronson is one of the most violent. We're just a Bronson. An attack. Over its 140-year history, Wormwood Scrubs has gone from the best prison in the country to one of its most notoriously violent. When somebody takes their life in your care, you feel as if you're personally responsible. This is the untold story of what really happened inside Wormwood Scrubs. It's told by those who were there to witness it. Witness what? In the prison break of the century... George thinking of escaping. George said, I never feel anything else. To the most shocking scandal in prison history. We're going to put a noose around your neck.  We look like a suicide and we'll get away with it. We will actually really fright somebody who's going to get killed in that prison at the time. You two, I do not condone anything that's going on. I'm here for the simple education of myself. This is educational. I'm trying to learn. Today, Wormwood Scrubs is Britain's most recognizable prison. If you ask a population to name London prisons, I suspect the Scrubs would be right at the top. It would be the first one they'd say, the Scrubs. It's famous or it's notorious. When Michael Cain's character Charlie Crocker is released from jail in the Italian job, it's Wormwood Scrubs. And on many news reports about prisons, you guessed it. Wormwood Scrubs is in the... Never seen it. Never seen the Italian job. Background. And the prison's fame is not just from films and TV. Over the years, it's seen a steady intake of celebrity prisoners. In 2008, baby shambles frontman and serial drug offender Pete Doherty spent 29 days inside. I'd just like to say thank you to Mr. Lutcher for putting me in the company of the most dangerous criminals in this country. But Doherty wasn't Wormwood Scrubs' most famous rock star prisoner. In 1967, it locked up Rolling Stones guitarist Keith Richards. Oh, wow. It all started when Richards jagged. Never listened to a full Rolling Stones song. Can't name one. Sorry. And some friends were having a party at Richards Sussex Country Retreat, Redlands. Unsurprisingly, drugs, including LSD, were involved. Keith saw someone at the window. I thought it was a fan sort of nosing around to try to get an autograph. In fact, it was 18 police officers. Damn! What the rock stars didn't know was that the news of the world had got wind of the party and informed the police. They failed to find any acid. Wait, the news of the world? Is that what they just did? Like, so the news got informed of the party and then they tricked? That's hatin'. Did find small traces of cannabis. Enough for Richards and Jagger to appear before court on drug use and possession charges. And the judge wasn't inclined to let them off lightly. You have to try to imagine the climate of utter hatred of the Rolling Stones that existed among everybody over 30 in Britain and in America at that time for social offenses which now seem ridiculous. Should we watch a Rolling Stones documentary? I didn't know they was hated like that. It's surprising that I said it does hate, but I didn't know that. Their hair curled over their shirt collars, not wearing ties to the Savoy Hood, that sort of thing. And the establishment really set out to get them and get them they did. Despite the small amount of drugs found by the police, the judge sentenced Mick Jagger to three months inside Brixton Prison and Keith Richards to a whole year in Wormwood. Damn! It was a gasp of complete astonishment. It was the over the top brutality, the savagery of this obvious show trial. You know what's crazy? Just like then when you lock up a musician it does nothing but boost their career. The prize-all against pop musicians. These sort of offenses that were so minor that they were expecting a fine. The police who had actually busted them didn't think that it would be so severe and so savage. Richards was immediately driven off to the scrubs where Phillip Norman thinks the authorities hoped he'd have a hard time. I really think that it was part of the judiciary's hope that they might be gang-raped, humiliated, had their horrible long hair shaved off. But Richards' experience inside wasn't exactly the punishment the judge had been hoping for.  He was a true rock and roll rebel and in fact Keith was rather welcomed by the other inmates. He shouted out encouragement and one of them even asked if he wanted any hash. So he wasn't really going to a hash-free zone exactly. It was like rock and roll, you know, like anyone else. And Keith was a hero to young blokes, you know. The guitar hero had already emerged as sort of an archetype. You sent that man to a prison where he was one of them down there. People over 30 hate prisoners as well. You know what I'm saying? We sent them to his people. Of the rock business. But even for the Stones' hating establishment it seemed in this case the judge had gone a step too far. The judicial system seemed to realize at that moment this was going preposterously over the top. The next day by the middle of the next afternoon they were actually out. Oh, I thought they was in there serving real time. They got out the next day. Richards' brief stint at the Scrubs brought welcome relief for the prison. But two decades later Wormwood Scrubs welcomed one inmate who was feared by all. Charles Bronson, Britain's most dangerous prisoner. What was in every documentary? Charles Bronson is one of the most famous in the violent category who would just attack and attack. Therefore, had to be kept confined. This color suit means they're a flight risk, like for escaping, right? Because he would attack inmates. He couldn't be confined with anybody else. By the time he got to the Scrubs in 1986 Bronson had visited almost every prison in the country. But the Scrubs was unlike anything he had experienced before. His remarks on the Scrubs for the times he was there are very, very negative. He rates it way down the bottom. And he says, you know, that he was ripped apart and he was often in tears. Who care? He said Scrubs gave me no help. I had a miserable time there. I was like a broken little boy. And, you know, I put it as pretty near to hell really compared to some of the other nicks I've been in. Bronson's experience in the Scrubs was so bad that after two weeks he tried to strangle the governor. He tried to strangle... After that, I know he had a terrible time, for sure. Quickly moved on to another prison. Oh, okay. But Wormwood Scrubs' reputation as one of the worst prisons in the country was sealed. And it's a reputation that it still carries today. It's become famous for a succession of very poor inspection reports alleging everything from brutality to simply very poor conditions for the prisoners who are held there. Wormwood Scrubs is living from one crisis to another. We hear about the rise in self-harm and drug use in the prison and violence and other activities. We've seen increases in serious assaults on staff. We've seen increases in serious assaults by the prisoners. I don't think anybody would be happy with what you see in Wormwood Scrubs now. It's a damning reality for a prison that was originally built to be the very best in the country. One that was meant to treat its prisoners with humanity and transform them into better citizens. That's how it always starts, right? You know, make these state-of-the-art prisons at the time and they're supposed to do this, this and that but you get one bad apple from some other prison that comes in like I'm talking about like a guard or something and then they teach all the other guards this and that and, you know, downhill from there. How did it go so badly wrong? Wormwood Scrubs, Britain's most iconic prison. It has had rock star prisoners and featured in countless films and TV series. But today it's known as one of the worst prisons in the country. The complete opposite of how its founder, Edmund Duquesne, imagined it. Edmund Duquesne. In 1869, Duquesne, a military engineer by trade, was appointed the country's new prison chief. And he had a vision that prisons shouldn't just punish criminals but transform them into better citizens. The core of it is that you don't demand and hope. He had that attitude in the 1800s? That's a good attitude to have in the 1800s, I wouldn't think. In anybody that every life is worth something and that there is redemption possible for everybody. For a great public figure to believe that decent prisons and the possibility of reform was important was a significant thing. At the time of his appointment, Duquesne's vision was a radical one. 30 years before, Britain was in the midst of a crime wave. With the number of criminals at an all-time high, the Victorians embarked on the largest prison-building campaign the country had ever seen. Pentonville, Strange Ways, Wandsworth. These huge prisons were built to strike fear into the minds of the prisoners. Victorian. I always think with the bigger prisons, it's like, you can't catch everything that's going on, man. The bigger the prison, the more negative it's going to be in my mind. You got to make smaller prisons to concentrate their efforts specifically onto, you know what I'm saying? Local prisons, they tend to be big, intimidating buildings.  It was built to send a message to those going within the walls that you are now under the power of the state and this is to intimidate you. You know, it's crazy and some people see that and they're like, man, this don't scare me. Watch this. Watch this. And boom, there you have it, the worst prison. And inside the prisons, the Victorians made sure life was as punishing as possible for the inmates.  Looks like a little bit of slavery. When he was appointed as the new prison chief, Ducayne wanted to completely transform the system. And when an area of open fields in West London known as Wormwood Scrubs became available, Ducayne saw his opportunity. Although Sir Edmund Ducayne had responsibility for effectively every prison in England and Wales, the one he was particularly interested in was Wormwood Scrubs. This was the one where he was going to stamp his thinking on prison design. Ducayne's design would be revolutionary. He envisaged a series of parallel wings running in the North-South direction so that every single cell at Wormwood would get sunlight at some point throughout the day. That's thinking smart. Ducayne also designed what is said to be the country's largest prison chapel, big enough to accommodate every single inmate. It's a reflection of how important Sir Edmund Ducayne felt. Big prison chapel. I imagine on the way to the chapel or coming from the chapel or even in the chapel, there was a lot of fights then. Everybody could go and everybody was... What's the word I'm looking for? Everybody was probably... They promoted for everybody. They pushed for everybody to go. There we go. Christianity to be. He believed that reflecting on your crime and faith in God was central to improving people. You're not wrong. Ducayne also believed that to reform prisoners, it was crucial to take them out of isolation and silent labour treatment and put them to good use. So to make Wormwood Scrubs a reality, he came up with a radical idea. Have the prisoners build their own prison. A small team of convict workers begin by constructing temporary buildings. Bro, that sounds like the worst. Imagine a 2023 mindset prisoner going in to go build his own prison. Like, you got me messed up. I'm not funny to build this. I can't secretly build one room bigger and it'd be mine. Like, no. To house a hundred men. Once that's built, they move on to constructing another one and then once there's a couple of hundred men, they begin the process of building the permanent wings. By 1891, Ducayne's masterpiece was finally complete. Four wings now house almost 1,400 inmates and its huge entrance gate has become an iconic site. Inside the prison, Ducayne installed a strict reforming pathway for the inmates. Cells were provided. They would also work in workshops as they moved through the sentence. Make sure if you're in here watching it, you are not followed. Hit that heart. There's a heart. You should be able to hit a heart and it follows me. Appreciate everybody. And stated that they were effectively being reformed. So this is the new hope. We're going to lock people up in somewhere that is meant to be reasonable conditions to help change their lives magically. So that was the great optimism about Wormwood Scrubs. Ducayne's prison was a sensation and Wormwood Scrubs became the model for jails around the world. But it soon emerged that behind Wormwood Scrubs locked doors, life for the prisoners was still a living nightmare. In 1918, a young inmate named Arthur Wilkinson entered its gates. He was a conscientious objector, locked up for refusing to fight in the First World War. Oh yeah, definitely gone to jail. From inside Wormwood Scrubs, Wilkinson wrote letters to his family bringing to light some of the prison's hidden practices. At the start of a prison sentence, the procedures seem to be the same for everyone. First, a month solitary confinement and then, emphasizing that you are being punished, adding to the bleatness of the cell, your bed is at boards, raised a few inches off the floor by crossbeams. A mattress is provided after a month, but only if you have not incurred any punishment penalties for breaking the rules. Right, that seemed normal. As no rules are issued. You discover what these are only when introducing them. Along with suffragettes, Wilkinson was part of a growing number of middle-class people finding themselves inside the Scrubs and championing reform. Prison reform really grows out of people without experience of prison going into prison. When you take the middle classes into prison and you treat them like the lower orders and they experience the brutality, the reduction of liberty. That's when they start to champion the cause of penal reform. Thanks to people like Wilkinson bringing to light the plight of the prisoners, the Scrubs started introducing revolutionary practices. There were classes run by voluntary teachers who were cons and these were unsupervised by officers. What's fascinating to me in the 20s and into the 30s is that most of the people involved who are controlling the Scrubs, who have power in the Scrubs, seem to be open to trying new things. Imagine that today. Wouldn't happen today. What do you mean imagine that today? Unsupervised? Like they'd be fighting low-key, they'd be in their trade and spice, they'd be doing all type of... And soon, it wasn't just those in control of the Scrubs who wanted to reform the prison, but the entire world. In February 1938, four young men were facing the dock for stealing £13,000 worth of diamonds. They attacked... In February 1938, four young men were facing the dock for stealing £13,000 worth of diamonds. How much is that now? 1938 to 2023? That's probably a crazy amount. They attacked a jeweler and injured him very severely. Probably, if I remember rightly, they were lucky they didn't kill him. But these weren't just run-of-the-mill thieves. They were Mayfair socialites, sons of successful businessmen and high-ranking army officials. The tabloid press of the time had a field day. This went international. These days, we would say it went viral. But despite the accused's upbringing, the judge did not go like... Why is he talking like that? The leader of the gang, Robert Harley, received a seven-year prison sentence and, worst of all, flogging. He was sent to Wormwood Scrubs to endure 20 strokes of the dreaded, cat-a-nine tails. Hold on. Wait a minute. Heavier sentence of four, seven-year servitude and 20 strokes of the cat that Robert Paul Harley unmoved. Oh, wait. You was getting time plus beatings back in the day? You was getting beat and getting time back in the day. This is shocking. This is revolutionary to me. I never knew that. When you went to jail, they were spanking you. Strokes of the cat. It's crazy. Britain had for years a corporal punishment as a penalty, which could be inflicted by the courts. The cat-a-nine tails was a cane, but with lots of ends. So when you were beating somebody, he could be hit simultaneously by several lashes. It could be... It was a near-working smarter, not harder. Very, very brutally used. I actually believe in the availability of corporal punishment, but not, for heaven's sake, in the cat-a-nine tails.  And Robert Harley's flogging at Wormwood Scrubs got the whole world talking. It was generally considered that corporal punishment was something that was pretty well exclusively endured by the lower economic orders. It wasn't really anything that you did for gentlemen. Society was very divided. My bad. Hold on, do we? Quite right, too. Others said, oh, can't do it to people of that class. Jolly bad form, what? But there was a huge division. A petition was signed by leading MPs and clergymen calling for Harley's sentence to be commuted. And the case was debated in the Commons. What that in turn did was to actually focus attention on the use in any event of the cat-a-nine tails and what role it had to play in a society that regarded itself as civilized. Spurred by Robert Harley's case, reformers eventually won and flogging was abolished. Wormwood Scrubs was finally entering the modern era. I don't see how that was even a thing. I've never seen a man of 19, whatever, beating people was, you know, acceptable. But that's wild. When you get a whole whooping and prison time, it's tough. But it was not the end of Wormwood Scrubs' reputation for the inhuman treatment of its prisoners. In the early 1960s, one man walked through its gates to begin a sentence that would personify the overbearing power of the state. Then, was George Blake, Britain's greatest-ever traitor? Like, traitor as far as, like, espionage? Is that what the word is, espionage? Or what's the word I'm thinking of? Espionage? No, it's not espionage, is it?  We see in the late 1950s, early 1960s, an uncovering, if you like, of spy rings in the UK and in the West. And of course, as these agents are apprehended, a large number are given very long custodial sentences and end up in establishments like Wormwood Scrubs. In 1961, an MI6 spy by the name of George Blake was sensationally uncovered as a double agent. At the time, this is a man who is public enemy number one. It emerged that for the past decade, Blake had been feeding top-secret information to the Soviets. He had betrayed the names of hundreds of British spies, resulting in many being imprisoned and executed. Damn. So you've, you've planned both sides. You really got people, you know what I'm saying, unalived. Your fellow brethren, that's crazy. When Blake was uncovered as a double agent in May 1961, at his trial, the Lord Chief Justice Parker said that what he'd done was akin to the greatest treason that anyone... Treason! Treason. There we go. ...and he was determined that he should pay a full penalty for that. 42 years was a sentence. The longest sentence in British criminal history to that day. By a long way. I'm honestly not mad at that, though. That seemed about right. For treason, betraying your country, like, that's tough.  So it's like... Yeah, 46 years? That, that seemed good. There was no way that he was going to fulfill that sentence as far as he was concerned. He was beginning immediately to think about how he could get out. There's a small cohort who have some sympathy with him.  And people have been jailed for anti-governmental stances that are going to have sympathy with others that have taken an anti-governmental stance. One man who would find himself in wormwood scrubs for his stance against the government was anti-nuclear activist Michael Randall. Get your name and number to the governor. So I said, it's Michael Randall. Oh, he's still alive! Michael Randall, what? Just Michael Randall. I said, sir. I said, well, I like to talk man to man. The governor leaned forward and said, well, this is a matter of prison discipline. Now, are you going to call me, sir? And I said, no. And he said, right. Seven days, non-associated labor, which means solitary confinement. I was in prison for organizing a demonstration at an air base at Weathersfield in Essex where nuclear weapons were being stored and carried. This is Anne and myself. We had got married just a few days before, outside the old Bailey just before the start of the trial. That one is outside the old Bailey with myself, Anne, and the other people who were accused at the same time under the Official Secrets Act. While in wormwood scrubs, Michael Randall, along with fellow inmates Sean Burke and Pat Potter, became friends with George Blake. They were at the urinals together. And he, Pat, said to George, do you ever think of escaping? And George said, I never think of anything else. Together, Randall and the rest would hatch out a daring escape plan. And their actions were about to propel wormwood scrubs into a frightening new era. Think I heard about this, right? In the 1960s, ousted double agent George Blake was serving a 42-year sentence at wormwood scrubs. But with the help of fellow inmates Michael Randall, Sean Burke and Pat Potter, he decided to attempt an escape. Michael Randall's, you know what I'm saying, fully trained M6 agent or whatever. By 1966, his three fellow inmates had been released and they could finally put their daring plan into action. Wait a minute. Time out now. So they devised a plan to escape, got released and still executed the plan to get the one person out. And when I tell you that's another level of loyalty that I ain't never seen, I'm like, yo, I'm already free. What am I doing? So the escape came on a dank, drizzly evening on Saturday, October the 21st at about 6.30. It was done by throwing a rope ladder over the wall, George Blake climbed, Sean was on the other side holding the rope. I should add that the rungs of the rope were strengthened by knitting needles and this was a contribution of my wife Anne. The W wife. Blake managed to climb the ladder, but as he made it to the other side, he fell and broke his wrist. Sean had to bungle into the car. He drove off at speed. So this guy had to be before they discovered bar wire or something. There wasn't no bar wire on top or nothing. No double gate, no middle part with guards and dogs.  It was all on the news. It was on the telly. And then we suddenly had a feeling in our gut, my God, this is big news. The group hid out in various London safe houses. But Blake's broken wrist was becoming a problem and with his picture all over the news, he was unable to go to hospital. Can I go last? I don't know. So Randall had to look for a more willing type of doctor. I went to see this doctor. Who had been in the anarchist movement? He said, well, on the understanding that some people have an aversion to hospitals, I'll come and treat him. But the doctor did not have any bandages to make the cast. So to obtain some, Randall had to think outside the box. We had friends who worked at the BBC on the Doctor Who series in the makeup department. She looks like a movie. Is there a movie about this? So the BBC made their little contribution unknowingly to the whole escape. With Blake patched up, the group could now move on to stage two of the escape. We drove to Dover, across the Calais and then drove nonstop till we got to the border between East and West Germany, where we could see the lights of the checkpoint and let him out. Wave goodbye. And that was it. With Blake's safely-in-communist hands, Randall made it back to the UK. Oh, he stayed. It was not until 20 years later, following a newspaper investigation, that his identity and role in the escape was finally revealed. We were charged on three counts, helping a man out of prison, assisting an escaped prisoner and essentially getting an escaped prisoner out of the country. Damn. And how much time you give it at? 20 years later, at that point, I'm thinking I'm scot-free. 20 years later, I'm still looking over my back. And a paper, another paper did it. Amazing thing was that we were found not guilty. Amazing thing was that we were found not guilty. Oh, wow, okay. I don't have any sympathy with passing on information to the Russians, especially the names of agents. We didn't agree with that whatsoever, but we thought the sentence of 42 years was disproportionate. After he'd escaped, Wormwood scrumps became a more security-conscious prison. Restrictions were put on prison's movements and association and so forth. So the consequences of Blake's escape were not good for his fellow prisoners. You think Blake cared? He was gone. I'm out. Good luck. This is not just an escape, but this is a pretty spectacular escape. Now, what we've got to remember about prison escapes, although they're the stuff of film and legend, for the most part, they don't happen. Blake manages it. And certainly, having seen that happen, the spotlight is going to very, very firmly shine on prison security after that. Rehabilitation efforts gave way to dog patrols. They've got dogs. CCTV monitoring. And a new priority. No one can escape. They've got some barbed wire, finally. Prison in the 70s and 80s become very militaristic, very violent. They're very, very disciplinary. And scrubs for a while were sort of synonymous with that. Hard authoritarian style of imprisonment. And by the 1970s, a new breed of prisoners arrived at Wormwood Scrubs. People like DeMors murderer Ian Brady, Yorkshire Ripper, Peter Sutcliffe, violent, psychotic type offenders. It is weird. For their own protection, Brady and Sutcliffe were held in individual clothes. Who's the Brady guy? Did we do a documentary on him yet? I know we did one on the Ripper. Slee watched cells. There is going to know about their offenses. More often than not, they're also not going to like them. The real difficulty is protecting prisoners such as that and keeping them away from other prisoners. But the deadly serial killers would eventually come face to face for the very first time. There is a story that tells of the game of chess being played in Wormwood Scrubs between Ian Brady, the Morse murderer, and the Yorkshire Ripper, Peter Sutcliffe. I think Brady the Morse murderer. That sounds familiar. Sutcliffe says he's absolutely chilled. He finds him the sort of cold psychopath. Which when you think about their offenses, what Sutcliffe is essentially doing is he's saying, you know, Brady, he was wrong that his victims were just chilled. Who was Charles Savildomer? Richard.  So it's partly to do, I think, with his own individual psychology, which is placing... Oh, I do remember Sutcliffe. So it's partly to do, I think, with his own individual psychology, which is placing himself above Brady. I'm not a cold serial killing psychopath, such as him. Yeah, you are, buddy. Prisoners such as that, they create an atmosphere around the prisoner. Don't be delusional. The typical prisoner for somewhere like Wormwood Scrubs is going to be a Raman prisoner or it's going to be a short-term sentence prisoner. They're going to be burglars. They're going to be car thieves. They're going to be the housebreakers. They're not sophisticated offenders. When you get really sophisticated offenders in, their influence on the prison can be quite destabilizing. While psychotic killers were in its midst, life inside Wormwood Scrubs took a turn for the worse. Can imagine. There was something creepy about the Wormwood Scrubs and the way things were conducted. There was a Wormwood Scrubs way of treating prisoners and they couldn't understand it. But it just wasn't like they've treated conditions in other prisoners. Oh, okay. Yeah, yeah. We watched the documentary on Myra, for sure. And I remember the name, Ian. This is a phase where, for example, there is not in-cell sanitation. People are having to urinate and defecate into chamber pots and keep it in their cells overnight if they can't throw it out of the window. Tensions between... Yo.  Like, I gotta... I'm sad now. I couldn't do that. Old peeing water and doodle? Imagine that. In your cell? This is a, what, 5x5 cell? Man, look, man. Stay out of prison, people. Prisoners and staff soon reached boiling point. By the early 80s, Wormwood Scrubs prisoners were regularly rioting and in a blaze of publicity, the governor resigned, publicly calling the prison a penal dustbin. Wormwood Scrubs, 1970s and 1980s, goes through a really, really bad time. Bad inspection reports. A governor walks out. It's characterised as a brutal, difficult, violent place. The prison is really desperate for some positive PR. It was in 1986 that the positive PR finally came. I was in the enemy office and somebody said, here's a story for you. Do you want to do this band in Wormwood Scrubs? You know, it's funny. They lock up a lot of these artists and use their music against them as, you know what I'm saying, in court. But then music is the positive PR that they need in the trails. That's probably why they lock up so many. I used to promote bands inside the prison. Organised a few concerts inside the prison. I was told by one of the prison officers there's been a band formed in the prison. What were the instruments?  It's overcrowded in the prison. It's terrible. There's a band in this prison. It's going on a Friday night when they used to be here from six to seven. And I was very surprised. The whole concept was three convicted murderers and three prison officers. I've never had that before. Three convicted murderers and three prison officers. So they've joined two opposite sides of the fence, obviously, freedom and locked up for life joined together. That seems weird. It's not traditional. Hello. It was unique in one sense and there were very, very good musicians as well. My name is John McCarroll and I'm the drummer in the Scrubs band. Where are you on here? I'm the good looking one. That's you there? Oh, the one with the beard. Seriously? Yeah, it was a mistake. Basically, you've got Dave Bruce. Dave Bruce was the founder of the band. He was a prison officer. He was a prisoner. He was called Finn. Finn, that's the one. Very, very quiet man. He learned his English in prison. That's me. He was a singer, rhythm guitarist. These were people who had done murders and the people who were locking them up all together in some kind of heavy metal band.  All different types, walks of life. For a brief time, the Scrubs, as the band was known, hit the headlines. The good news story, the prison so desperately needed. What's that movie? Is that what this is? Yo, isn't there a movie made after this? It's a Jack Black in it or whatever his name is and it's the prisoners and they got a... I could be wrong. It must have affected the prisoners and it brought a positive light to the prison. We were getting a lot of media attention from all around the world and it was good. Good attention. It shows that not all prisoners are bad. I understand there's victims involved but whilst we're in my care, in the custody of members of the victims... Yeah, outfits and everything. But whilst we're in my care in the custody of members of the Scrubs during my time, they were alright. They're welcome. They're in a way. That's it? The man who says to the Conservative conference, prison works. It ensures that we are protected from murderers, muggers and rapists and it makes many who are tempted to commit crime think twice. Facing up to the new dynamic Shadow Home Secretary Tony Blair tough on crime, tough on the causes of crime. We will take a new approach to the whole of law and order. Both political parties wanted to be top dog in fighting crime and punishing the criminals and in Wormwood Scrubs it would be a game changer. From that point onwards you get a spike in the prison population and it begins to climb and climb and climb. Plus the 80s and 90s came along right? Was it School of Rock? Yeah, yeah, I think that's it. Weren't they in jail or what? I don't know. The fact of that practically is that a prison system had around 44,000 people in the early 1990s. Yeah, 80s, 90s came along drug and the drug movement. A quarter of prisoners were sharing cells designed for one person. It's now a prison system holding about 84,000 people. Wow. You can remember going around local prisons like Wormwood Scrubs that were very shocking. But no one was prepared for the revelations that came out of Wormwood Scrubs in the early years of the 21st century. Broken limbs, previous bodily harm.  As Wormwood Scrubs entered the 1990s political pressures were placing the entire prison service more stressed than ever before. But while the spotlight was on riots in prisons like Strangeways what was going on in the Scrubs was hidden and far, far more dangerous. Is the prison guards doing this now? I was Chief Inspector from 1995 to 2001. I was responsible for inspecting the conditions and treatment in every prison in England and Wales. We first went into Wormwood Scrubs in 1996. We were very disturbed because there were lots of reports of bullying in the segregation unit. But we couldn't get to the bottom of it. Everyone was clearly covering up. Yeah.  But there was a climate of fear. But in December 1997 human rights lawyer Daniel Jacoba got a tip-off about a prisoner being abused by guards inside the prison. He started lifting the lid on one of the greatest scandals to ever hit Wormwood Scrubs. We were approached by someone who worked inside the prison, effectively a kind of whistleblower and at the time he was concerned about a certain amount of violence directed against a particular prisoner. We didn't perceive this was some kind of big problem. It was a problem relating to a single prisoner because we knew what happened eventually was he began to see people they would say but I heard someone in the next cell get beaten and their name is X and they want to see you. We were learning of new cases as we were coming to the prison. Cases that had literally just happened. Makova uncovered shocking evidence that the Scrubs guards were regularly abusing, beating and even torturing inmates. How you think that was wrong. People coming to work taking their problems out on the prisoners and see this is what I be saying man, these prison guards man, they be wildin' out. Broken limbs assaulting a cage. Broken limbs? You can't even really hide that. Grievous bodily harm. Rape allegations. I think it locked kind of hard, didn't it? Shouldn't be on the floor. The video is almost over, let me just look at the allegations which were consistent of mock hanging. We're going to put a noose around your neck we're now going to kill you but it will look like a suicide and we'll get away with it. We were actually really frightened somebody was going to get killed at prison at the time. And worse than that, the Scrubs guards weren't just taking it out on their own prisoners. A prisoner was transferred from Bristol prison to one with Scrubs for a beating over the weekend so that he could appear at Bristol Crown Court on his case. Wait a minute, he had a drive-through beating? That's not funny. Like they was in cahoots with other guards at other places.  That's tough. You know how high up how high up that issue has to go? Like how high up in the chain of command for you to be able to send a prisoner from another prison to catch a beating and go back? Chief Inspector of Prisons, Sir David now Lord Ramsbottom We were a whore. What was his name? The Chief Inspector of Prisons, Sir David now Lord Ramsbottom We were a whore. Don't get no better than this, this is what I'm talking about. Lord Ramsbottom Ports to find that he's in a condition to prison. Get into it. Sir David to his credit immediately took seriously what we were saying had a strong ring of truth to it. He was very concerned and he took it immediately to the Home Secretary I think the very same day or the following day he took our dossier to the Home Secretary. The case, complied by McOver, eventually led to an unprecedented trial. 27 prison officers were charged with 13 different assaults. 6 prison officers were convicted. There had never been a mass prosecution of assaults by prison officers against prisoners. It was an unprecedented criminal investigation. G4S next. In my view it is one of the worst scandals to have hit a prison. The case led to sweeping changes inside Wormwood Scrubs. For example, unbelievably there was a drinking place inside Wormwood Scrubs prison and you could go and have a drink and go back to work. Oh, this was that prison? I remember that. I remember this. They had like a little bar inside for employees for screws. That was almost immediately done away with. But today, Wormwood Scrubs is plagued by another problem. Inmates hurting themselves and others. We are really in a situation where prison deaths have become an endemic feature of the prisons. When somebody takes their life in your care, you feel as if you're personally responsible. I don't believe what you're talking about, sir. It's not something that you ever get used to. It's not something that you ever take lightly. You're talking about very vulnerable prisoners held in awful conditions and for many of them self-harm, suicide is the reality of their experience. There's not many organizations where you go to work every day and you may be confronting somebody who's attempting to kill themselves. You may be confronting somebody who's armed and wants to harm you or you may be confronting somebody who's armed and wants to harm somebody else and you have to get yourself in between them. It's a day when you could influence people in terms of life and death. When Wormwood Scrubs was built its founder, Edmund Duquesne had a vision of a reformed and humane prison. But more than 140 years later that vision is still far from reality. I'm particularly concerned about the overall conditions that there are there, which is infestations, broken windows, to- Infestations, like what you mean by that? Like it's above infestations? Well, looks that don't work. It's really conditions that are appalling for anybody to live in. Almost 150 years from when it was built it seems that Duquesne's great vision for Wormwood Scrubs as a reforming prison is still waiting to be realized. It's never going to happen. Like, it's never going to happen. 150 years later, come on up. Stop. T.L.O., leave a like, comment, subscribe.