what is the best algorithm? How do we know if one algorithm is better than another? People say that a binary search is better than a brute force linear search, but in my hash table video I showed that a binary search was actually slower than a linear search. What gives? Today I'm going to show you software performance, big O notation, and algorithms. So here we have a Rust program, but inside the compiler this is represented as a string. We want the compiler to tell the user about this syntax error right here, where we use the single quote instead of a double quote. The problem is the compiler sees that this is index 23, but it doesn't know what line number it is. So we want to convert the number 23 into line number two. But how do we do that? Well, it's actually pretty simple. All we have to do is count how many new line characters there are before our error point. In this case, there's only one new line character, which means we're on the second line. So what code do we need to make this work? Well, all we need is a basic for loop. Our algorithm starts as if we're on line one. We loop through every character before the offset, which is that single quote that we talked about. If we see a new line character, we increment our current line. And otherwise we keep looping. Finally, when we reach the offending character, we return its line number. Now Rust program is love iterator. So let's write it in iterator style instead. So we tested this code, it works, ship our compiler, great, fantastic users are happy they have line numbers. So the first lesson today is to try it. I did the simplest thing that could possibly work. And of course it worked. Even if you can't think of a super elegant algorithm, just try the first thing that comes to mind, probably good enough. But is our algorithm good enough? Let's measure it. So I benchmark this algorithm with files of different sizes. And here were the results. Hey, wait a minute, if a file has like five lines, or if a file has a thousand lines, it still takes the same amount of time. What's going on here? The problem is our algorithm's performance does not depend on how big the file is. It actually depends on where the error is in the file. An error in the beginning of the file will take less time than an error at the end of the file because we have to do fewer loop iterations. So that graph I showed you was actually the best case where we find an error right at the beginning of the file. But let's measure the worst case where the error is at the end of the file. So as you would expect in the worst case, our algorithm takes longer. Notice that the worst case line is pretty much a straight line. If you remember in grade school math, you can write an equation for a line using y equals mx plus b. In this case, x is the file size, y is how long the algorithm takes, m represents the slope, and b represents some constant. When we talk about big O notation, we care about the shape of this chart, not about the specific numbers. Therefore, we're going to drop the constant factor. It doesn't matter what the slope of this line is either, we just care that it's a straight line. So we're going to drop the slope from our equation. So we end up with y equals x. And what we write in big O notation is O of x. Now actually, we don't normally use the term x, we use the term n. It's the same thing. So you would say that O of n is linear time complexity, linear because it makes a straight line. But that's in the worst case. What about the best case? In the best case, we also have a straight line. So we can use our straight line formula. However, in this case, our slope is zero. And because our slope is zero, the x term disappears. And we're just left with the b. But as I said, in big O notation, we don't care about this constant factor, we care about the shape. And we treat all constants as if they were one. So in the best case, our algorithm is 01, which means constant time complexity, which means the algorithm's time is constant, regardless of the input size. But only in the best case, which again, is if the error happens right at the beginning of the file. So the lesson number two is to check your loop bounds. When you're analyzing your algorithm, don't just see, oh, there's a loop. You have to see what is it looping to? Is it looping to the size of the input or is it looping to something else? That affects the time complexity of your algorithm. In our case, the performance is determined by where the error is in the file, not by the total file size. Lesson number three, stress test your algorithms. Don't just feed your algorithm the best possible case, try random inputs, try realistic inputs, and try to break your algorithm by giving it malicious inputs. In this case, I tested in the very beginning of the file and the very end of the file. And later we'll see testing random inputs in the middle and such. All right, here's a table of the algorithms we looked at so far. So we have our naive algorithm. And as we saw the time complexity for our algorithm in the worst case is of n, where n is the number of bytes in the file. Now, what's our real world performance? Well, we don't have a frame of reference. So we can't evaluate this yet. But let's try a different algorithm. But first, a pop quiz. What is the time complexity of the naive algorithm in the average case? We looked at the best case in the worst case, but what is it on average? Okay, let's talk about another algorithm. So we have our string. What if we pre compute some data? What if we look for where each line starts? So the first line starts here at offset zero. Second line starts here at offset 12. And the third line, offset 40. So we're going to put all these numbers into a vector. And then when we need the line number given an offset, we can search in our vector what line that offset would be on. So in this case, offset 23 is in between offsets 12 and 40, because it's in between those two offsets, it must be on this second line. Well, how do we know it's line number two, we can use the offset into this vector. Rust vectors are zero based line numbers are one base. So all we have to do is add one to the vector index, and then we get our line number. So how does a search work exactly? Let's look at a more complicated example. So we've made our table already, we're looking for the line number at offset 80. So we start at the left of a vector, we compare the number in the vector with our offset 80. And 80 is not less than zero. So we move on to the next one. 80 is not less than 12, we keep going. And we keep going until we find some number that is greater than our offset, in this case, 99. And all we do is look at the slot left of that. And that's where the line starts. Okay, so let's compare our naive solution with our new table based solution. The data is clear, our new line table solution is way faster. So let me zoom in the line table line on the y axis. And you'll see that they're both basically straight lines, which means they both have O of n time complexity. So even though both are O of n, one is clearly faster than the other. So the lesson here is that big O is not the full story. Remember that when you talk about time complexity, you're excluding constant factors like the slope. And in this case, the performance of a line table is a straight line, with a much lower slope than the naive solution. And because of this time complexity is not the same as time. So back to our chart, we show that line tables also O of n, and the performance of the line tables much faster than the performance of the naive solution. So let's look at a chart of our line table solution again. But this time instead of time, we're going to look at number of comparisons, just so the data is less noisy. And let's zoom in onto this little section of 500 bytes in less. Now, this doesn't look like a straight line number of comparisons should give us exact numbers. Why is it kind of jittery here? Well, the reason this line is jittery and not straight is because our x axis is file size, but actually our algorithm is based on number of lines, not the file size. And in real world text files, different lines don't have the same length. If we adjust our x axis to instead of being based on the file size, it's based on the number of lines, we end up with a straight line, which is exactly what we want. So really what we want to say is our algorithms time complexity is O of the number of lines, not the number of bytes. So we need a different variable than n to represent the number of lines. Typically in algorithms, you will use the letter m. However, I don't like the letter m because it sounds very similar to the letter n. So I'm going to pick capital L instead. So we'll say that our time complexity for our line table solution is O of L. So lesson number five, be careful with your variables that you use in your big O notation. Different variables have different meanings. The number of lines in a file is different than the number of bytes in a file. And in fact, you have fewer lines and you have bytes. And that affects our real world performance. Also note that if you see n, it doesn't mean number of bytes, it could be something else. So when you write big O notation, make sure you document what your variables mean so other people can understand. So you may have thought, Hey, wait a minute, our line table solution has this pre process step, but we're not measuring that pre process step, we're only measuring the lookup time. So that's not really a fair comparison between our naive solution with no prep time and our line table solution, which needs some prep time. So let's do a more fair comparison. So if you only do one lookup, the naive solution is clearly faster, because the naive solution only needs to look through it once, the line table needs to look through it twice, and keep some extra bookkeeping. But if we need to look up three times, the line table saves a lot of work. So it ends up being worth it. So let's include the time it takes to generate our table in the chart. So because our line table needs to scan the entire input, the time complexity is O of n. And because the naive solution doesn't need to do any work to prepare its tables, because it doesn't have a table, we would say that it has constant time complexity or O of one, it doesn't depend on the input size. But that's still not the full story. What about memory usage? A line table consumes memory, right? It's not free. So let's compare how much memory is needed for a line table solution versus the naive solution. And as you would expect, the naive solution basically uses no memory. And our line table solution consumes memory proportional to the number of lines in the file, because we store one entry per line. The lesson number six is to pre-process your data. You can get massive performance wins if you do some work ahead of time, instead of deferring the work to every time you look up. However, pre-computation comes at a cost. If you're only going to look up one line number, it's not worth it doing this table. But if you're going to do it many times, then the pre-computation becomes worth it. But you also have to keep in mind your memory usage. It may be faster, but maybe you don't have enough memory for it. Pop quiz. What's a different way to pre-process the data that can make lookups even faster? Let's look at our line table again. This time we're going to use a different algorithm to search through the line table. Instead of going left to right, we're going to use a binary search. So what a binary search does is it keeps track of a range of possible answers. When you start out the algorithm, all answers are possible, so our range is the full vector length. And in a binary search, we check the middle element. So we're going to see is 80 greater than, less than, or equal to 69? In this case, it's greater than, and greater than means nothing on the left could be the answer. So we set the left end of our search range to the middle element. So then we pick the middle of the new range, which is 110, and compare it to 80. 80 is less than 110. So we set the right end of the search range to before our element. Now our range only has two items. So we pick the middle, in this case, we're rounding up. We do the comparison, we determine that it's less than, and then we update our range. Now our range is spanning only one element, so that must be the answer. So that's the binary search algorithm. Now let's analyze it. So compared to our linear search using the line table, the binary search using line table does far, far fewer comparisons. If we blow up this line, we'll see that it's actually different than the linear algorithm. Instead of being a straight line, it's more of a curve. It's a jaggedy curve, but it's a curve. So the line jumps at 16, 32, 64, 128, and 256 lines. So when we go from 64 to 128 lines, which is a doubling, we do one more comparison. And when we go from 128 to 256 lines, which is another doubling, we also do one more comparison. That's different than a straight line where we would go over 10 and then up 10 or over 10 and up 8 every time. Here it's a multiplier on the x-axis. Now what is the time complexity of the binary search? Now it looks kind of like a logarithm, and that's actually the answer. But our binary search is all jaggedy. It's not a smooth line. What gives? Well the problem is you can't have half of a comparison or a quarter of a comparison. Where the log line would be say 7.3, you have to round that up to 8 comparisons. So here I'm going to round up the log line to the nearest integer, and you'll see it matches exactly to our binary search line. So we would say that the time complexity is log 2 of L. L is the number of lines. And typically we will not include the base of the log in time complexity. We just say it's a logarithm. So let's omit the 2. So lesson number 7. Big O is approximate.  It didn't follow the exact curve of a logarithm. Instead when we were talking about time complexity, we're looking at the trend, the shape of the curve. Big O is not precise. Pop quiz. In the binary search, when we doubled the number of lines, we got one more comparison. But why is it double and not triple? Or maybe 50% more? Why double specifically? So let's include the binary search solution in our chart. As we said, the time complexity for the lookup is log of L. And what about the table generation? Well, we're using the same table that we made for the line table. So it's the same time complexity. But what about real world performance? We looked at the number of comparisons, but what about time? Let's see how this binary search solution performs on real hardware. So here's a chart where I compared the two algorithms using random offsets. Instead of best case or worst case, this kind of represents average case. And we can see that the binary search performed way better than the line table solution. Pretty good. Our binary search is leaving the line table solution in the dust. But hold on a minute. I noticed something here. Let me zoom in. For very small files, the line table seems to be a bit faster actually than the binary search. Binary search is clearly better for big files. But for very small files, line tables a little bit faster. While looking at the linear approach, I noticed some optimizations I could do. And if I did those optimizations, it's way faster than binary search. So what's going on? I thought binary search was supposed to be faster. Well, if we zoom out again, we can see that the binary search does beat out the line table, even our optimized line table. But for small files, even up to 150 lines, our optimized line table solution, the linear solution is competitive. So lesson number eight, big O is for big inputs. Big O notation and time complexity tells you about how your algorithm will scale given huge inputs. And that matters for big data applications and distributed systems. So if you expect your file sizes to be really big, your input data sets to be really big, you really care about big O. But those constant factors that we talked about, like the slope of the line, definitely matter at smaller input sizes. Pop quiz. What is the time complexity of the SIMD approach? We know that the unoptimized line table is O of N. So what is the optimized version? Well, let's look at the implementation of the optimized version. Here's the unoptimized version, the line table solution that does a linear scan. And you can see that we're looping over the line offsets. And we're doing a comparison. If the offset is that we're looking for is less than the current lines offset, we return, I find it a lot easier to work with a normal full loop. Here, we're using a loop with enumerate. So let's refactor this into a while loop. First, you make our own counter variable, and then we turn the four into a while. Now we have this temporary line offsets variable. Let's inline that move the white space a little bit, cram this all on one line. And then what we're going to do is some copy paste. We're going to copy paste this line eight times. Now we're processing eight items per iteration, not one item, we basically unrolled the loop. So let's see how this performs. Here's the chart we showed before. And well, it's not super fast, but it's definitely faster than the original one. So we got some speed up just by copy, paste and code, probably not a good thing to copy paste code, but gotta do what you gotta do. So let's look at this code again, just for the slides. I'm going to condense this a little bit. And what we want to do is use SIMD SIMD is single instruction, multiple data.  So we need to separate the returns from the less than comparisons. So we're going to move all the returns below all of our ifs. So this code is functionally identical. Oh, forgot the commas. So let's measure it. And it looks like on average, this is a little bit slower, but don't worry, we'll make it faster soon. So let's condense this code a little bit for the slide, move some white space around. What we're going to do is start using SIMD. So what we need to do first is load the data from our vector. And to do that, we're going to use SIMD from slice to load eight items at a time. Now we have this less than comparison, but we want to do it with SIMD. So we need to use SIMD dot LT, but SIMD dot LT works on two SIMD values. We need to convert our offset number into a SIMD containing that offset number. So let's look at the performance of using the SIMD way. It's, it's still kind of bad what's going on. Well, the problem is our bottleneck is not the less than comparisons. We just refactor the less than comparisons to use SIMD, but the real problem is all these ifs down here at the bottom right now when we're doing the less than operation, we're immediately converting it into an array. Let's split that up into two separate steps. And now we have access to this mask variable. And the mask variable has a super handy function called any. So let's use that any function to say if any of them matched, then do all the if statements. If none of the match, we skip all of those if statements. Let's see how that performs. Now we're talking. We're still not faster than the binary search, but we're much faster than the unoptimized line table solution. So as I said, it's the zip statements that are the problem. So how do we get rid of all eight of these if statements? What we're trying to do is find the first true in our array and return its index. We could do that without converting our mask into an array first to bit mask followed by trailing zeros. Now what this does is count how many falses there were up until the first true. So then we use that answer instead of a bunch of if statements. And when we measure performance, now we're beating binary search, at least up until a hundred lines. So lesson number nine is SIMD is awesome. You can get major performance speedups with SIMD.  You're wasting over 80% of your CPU if you're only using one thread. SIMD is similar. You're wasting 80% of your CPU if you're only using scalar values. Now, compilers can sometimes auto optimize your code to use SIMD, but it's pretty rare. So it's worth it to check and see if you can apply SIMD to your own algorithms. So let's look at our table of algorithms again. Our optimized linear line table solution is faster than the unoptimized version, but for big file sizes, the binary search still wins out. The optimized version still uses the same line tables. So the time complexity for generating them is the same and the memory usage is the same. And we'll look at the time complexity in a bit. Now for the pop quiz answers. So pop quiz number one was what is the time complexity of our naive algorithm on average? Now, if we look at our naive algorithm again, we see that we're looping up until the offset, but on average, we're going to be looping up until maybe half of the length of the text. If you have a uniform distribution, the average number would be right in the middle. So looking at this loop, what is the time complexity? Because half of the text length is proportional to the text length, the time complexity of our algorithm is proportional to the text length. So the time complexity in the average case is still O of n, matching our worst case. Pop quiz number two was what's another way to pre-processor data that could give us a more efficient algorithm? Well, with our line table solution, we took the beginning of each line and put that in a vector. But what if instead of taking the beginning of each line and putting in the vector, we just pre-compute all of the answers? In that case, looking up the answer is a simple vector lookup. So if you remember, a vector lookup, in other words, an array lookup, takes a constant amount of time, there's no loops involved. So the time complexity for our pre-computed solution is O of one. And compared to all of our other algorithms, an array lookup just it beats the pants off any kind of loop. And because we need to create a table, we have the cost of generating a table, which takes O of n time. And unfortunately, the memory required for the table is O of n, which is much bigger than O of L. Our table has to contain one entry per byte in the input, not just per line. So our table ends up being maybe eight times bigger than the entire file. This is the classic memory time tradeoff that people talk about. You can make your algorithm super fast if you pre-compute all the answers, but pre-computing all the answers might take up a lot of memory. So our third pop quiz was when we have the binary search algorithm, why does the number of lines need to double in order for the number of comparisons to increase by one? So if you remember in the explanation of the binary search, once we make a comparison, the remainder of the algorithm only looks at half the data. When we do another comparison, we look at half of that data and so on. So if you look at this chart, once we do one comparison, we look at half the number of lines in the next iteration. When we do another comparison, we look at half the line still. The reciprocal of one half is 2x, explaining why it's double and not maybe 50%. Fourth pop quiz was what is the time complexity of the SIMD line table solution? And just as a reminder, here's what the chart looks like. We can make a straight line, but it doesn't look like it's a good match. But if we scale our data up to 50,000 lines and then draw a trend line, it's much more clearly a straight line. Because remember, big O notation is about how the algorithm scales, not how the algorithm performs at small data sizes. So the time complexity of our SIMD line table solution is also O of N. I hope you learned something about big O notation or SIMD or how compilers work or something. My last video was really popular, so I decided to make these rust stickers. You can buy them on my girlfriend's shop.  Class is missed.


This is the VOA Special English Health Report. In the past year, Burma has opened its political system and reached cease-fire agreements with some ethnic militias. The government has also eased media restrictions. But many aid groups say their jobs have not gotten any easier. Health workers are warning about the spread of a form of drug-resistant malaria. The malaria is resistant to treatment with artemiscinin. It was first seen several years ago in Cambodia. Frank Smithias started a group called Medical Action Myanmar. He says even with the recent political openings, aid organizations still find it difficult to gain access to areas most in need of aid. This is true, he says, especially in areas torn by many years of conflict along Burma's eastern border. He says there are now very good opportunities because of the cease-fire agreements with the Karen and the Mon. Larger areas should now be open for access to adjoint activity to stop this artemiscinin resistance spread. However, that has not happened yet. And we definitely need to have more openness and more activity and more international donor money, he says. In the past, many non-governmental organizations simply worked without official approval. Lower-level government officials largely ignored organizations that did not follow the rules. Until 2009, just three international non-governmental aid organizations had the required approvals to operate inside Burma. Yet more than 100 were operating there. Many were able to get a memorandum of understanding that allowed them to work without an official registration. But then came the recent political changes. Ends of aid organizations say lower-level officials are now unclear about their powers and responsibilities. In many cases, they are enforcing the procedures more. Save the Children is an aid organization whose work is mostly related to mother and child health in Burma. Kelland Stevenson with Save the Children says, in general, ministries and government workers have become much more cooperative. He agrees with those who say the barriers that humanitarian workers now face are largely bureaucratic. He says it shouldn't be a political discussion. There remains a bit of command and control from government, and it's not going to open up overnight. He says the discussion about getting access to new areas is underway. In his words, you have to remember, these changes have been extraordinarily dramatic in such a short time, a year. Kelland Stevenson says, things are changing so quickly. Still, aid groups warn that some issues, such as the drug-resistant malaria, may only get worse unless the government also takes action quickly. For VOA Special English, I'm Alex Villareal. For more health news and other information for people learning English, go to voaspecialenglish.com.


So I'm a mathematician, so like Dorothy's talk, this talk will be sort of about methods. And I've also benefited from Tetsuo's talk because he introduced some of the things that, sure, introduced some of the tools that I'm gonna talk about. So here's like the idea. Suppose we have one of our DNA knots, then the question that you wanna ask as a mathematician is, okay, what's the space of these curves, right?  What's the distribution? And an idea that's been around implicitly, at least for a while, is that, well, the knot is an equilateral polygon and it's got n-edges or something, and you're going to sample these polygons one way or another and you're gonna talk about polygonal knots. Or you're gonna talk about lattice knots, or you're gonna talk about knots made of chains of beads. And I'm kind of interested in those as mathematical models because they have a lot of interesting structure that lets you do computations and do theory in a really nice way. So here's what I wanna do. If you're a physicist, you might know of action angle coordinates. They're like this thing from mechanics, right? Where it's like, if you write things in action angle coordinates, everything is nice. And so really like my main point for this talk is that the space of polygons has action angle coordinates and they're quite simple. And I sort of suggest that we do computations in them as well as proof theorems. So let's see if this will work. Oh, no, it won't, of course. Okay, let's do it this way. Okay, so here's the next two things, right? So if you think about it, it's a very easy sort of classical thing to do to say I'm gonna take a random flight, which is to say I'll just sample a bunch of points on the sphere and stick them end to end to make a random walk in space. Now, if I want the walk to close, then it's weirder, right? Because it's some conditional probability thing where I have to condition on the probability zero event that it happens to come back to the same space. If you're a geometry, you would say I'm gonna restrict to the sub manifold of the space of end points on a sphere where the cloud of points has center of mass at the origin, exactly at the origin. But of course, an idea that goes back a long way is instead of conditioning on all of them summing to zero, condition on half of them summing to the same thing as the other half. And if you do that, then you can write down a very nice probability distribution on the segment that connects the origin to the place where the curves join. And let's see if my laser pointer will work. Okay, so this goes back to Raleigh, really. This is the probability density of the random flight in three dimensions. And it has this wonderful closed form, wonderfully awful maybe. But you know, the little plus means it's the thing if the thing is positive and it's zero if the thing is negative. So this is piecewise polynomial in the length of the end to end distance of degree n minus three. So you know, one thing you can work out from that is that the probability density of the segment joining two points on a random closed polygon is just sort of the product of these two PDFs, the one from the first arc and the one from the other arc. And this is some piecewise polynomial thing and it's degree n minus four. And I guess it's worth saying that there are a lot of exact computations you can do just from that page of formulas. Like for instance, here are some expectations for chord lengths, right? On a closed polygon. So, you know, they're exact, like this is not a sampling thing, this is a theorem.  I'll just draw it kind of flat. As two diagonals, let's call them D one and D two. And the computation up there is that the expected length of D one is 15 17ths. If, no, the other way, 17 15ths. If all these lengths are one. And of course, by symmetry, the expected length of D two is exactly the same. And using the PDFs, you know, you can plug this into Mathematica and there's the expectation of a, the length of the chord connecting vertex zero to vertex 37 and 112 gone. And it's that, exactly, exactly that. This is mathematics.  Okay, so, you know, and like you can do other functions of chord lengths. Like for instance, Tetsuo was interested in the approximation to hydrodynamic radius, which I think of as an average of one over the chord lengths. And you can do that too. And for instance, the expectation for the hydrodynamic radius of the 15 gone is exactly that with the weird, interesting series of descending log terms. Or about 0.768. But, okay, so, if you think about the chord length expectations, it's kind of interesting that they're rational numbers. And it's kind of interesting that the degree of the polynomials N minus four. So, now I'm gonna sort of, this is the part, I don't know, where we make the big leap. So, I'm going to say that those aren't accidents. Those are like actually really significant. And here's where we move from like classical old stuff to like very new stuff. So, here's an idea. If we built the polygon by joining like one arc to the other arc, we can fold the arcs around each other, right? This is like an old idea in random polygons and I don't know who the first person to do fold moves was, but Ken knows. Okay, Ken did it. So, that's cool. Okay, so, the thing about the fold moves that I like are the fold move is a continuous symmetry of the space, right? So, that's good. So, right, everybody who studies physics knows Noether's Theorem, right? Which is that if you have a continuous symmetry, you have what? That's right, a conservation law, right? Every continuous symmetry should have a conserved quantity. And it's actually not so hard to see what the conserved quantity ought to be. It's the length of the line connecting the two points. That's what's conserved when you rotate one around the other.  But in a very precise sense, it's supposed to be that length. So, here's the idea. There's like a ramped up version of Noether's Theorem called the Deuster-Math-Hekman Theorem. And it says a little more. So, if you have an even dimensional space, it says that if you have more than one symmetry and they commute, then the collection of D symmetries gives you D conserved quantities. Okay, that's really still sort of Noether's Theorem. But it says more. It says that the joint distribution of the conserved quantities is continuous, T-swise polynomial, and critically has dimension M minus D. So, half the dimension of the configuration space minus the number of symmetries. Okay, so let's just check. If we think about polygons, an N-edge polygon has N points sampled from the sphere. Those are the edges. And it's conditioned on three things adding up, right? The three coordinates of the end point have to be the three coordinates of the start point. So, we had two N things for the edges, minus three because it closes, and minus another three because there are three rotations in our three. So, if we consider the space up to rotation in our three, then we have a two N minus six dimensional space. And that is an even dimensional space. And it turns out to be like, that's like a key point, is that by modding out by rotations, you get back to an even dimensional space. Closed polygons before you modded out by rotation were two N minus three dimensional, which is odd and nothing works. Okay, so half of two N minus six is N minus three, and we had one symmetry. So, the chord length PDF should have been degree N minus four, and it was, and that was cool. Okay, but wait, there's more commuting symmetries than just one fold move, right? There are lots of fold moves. Now, they don't all commute. If you have a fold over D one and a fold over this guy, these are not commuting folds. And you can work out the exact failure to commute and so forth, but trust me that this rotation and this rotation are not commutative. But as long as the chords don't cross, then the fold moves sort of obviously commute. We have these rigid triangles and we're just folding, we're just pleading the thing up. Okay, great, so how many of these diagonals are there? Well, if we triangulate the polygon, there's only one, two, three vertices that aren't the end of a diagonal, right? The other N minus three are the ends of the diagonals and the triangulation. So there should be N minus three commuting symmetries, each of which has an angle and each of which has a conserved quantity. So here's the theorem, right? The theorem is that if you think of the joint distribution of these N minus three chord lengths and the joint distribution of the N minus three angles corresponding to them, both those distributions are uniform. After all, they have dimension, sorry, the space has dimension two N minus six, half was N minus three, we had N minus three symmetries. So the distribution is piecewise polynomial of degrees zero, but it's continuous, so it better be constant. So the only thing left is to say, oh well, constant on their domains, right? Because the D one and the D two aren't actually completely independent of one another. There are some constraints on the possible values, like this one can't be 10 and this one be two. Well, what are the constraints? They're just the triangle inequalities. There's the picture. So let me just show you what the inequalities are. If you look at the inequalities over here, what you see is there are these conditions that look like D two is less than two. Okay, well D two is part of a triangle with two edges of length one and D two is the third edge. So the most it could be is two of the triangles like flat. The least it could be is zero if they're crushed together. But in fact, it being zero says something about the next diagonal as well. Because D one, D two and the edge on the left are part of a triangle and the edge on the left has length one. So D one plus D two is at least one. On the other hand, D one minus D two can be no more than one because D two can't be too short. If I've gone out that far and I go back by an edge of length at most one, I can go back at most one unit towards the origin. So these linear inequalities determine an N minus three dimensional polytope. This is called the moment polytope. It's the polytope of momenta of conserved quantities. In general, it's like always a polytope. And on that polytope, the distribution is uniform. So if I took like a random five gone, then I would be picking a point randomly in that polygon in the plane and two random angles and I could put back together the random pentagon and it would be a perfect random pentagon. It would be sampled perfectly. Another way to say it is that if you wanna do these expectations, what you're doing is you're doing an integral. You're integrating over polygon space like with respect to volume and you're dividing by the volume of the whole thing. And the volume form in these coordinates is the simplest it could be.  And then you're good. And if you wanna go back to vertex coordinates, well, you know all the diagonals. So you've divided the polygon into a bunch of triangles and you know all three edge lengths of each triangle. So you can just build the triangles and then stick them together one after the other after the other until you build the whole thing. I should say that this is a lot like integrating over the sphere by using the coordinates and theta. So if you know this trick, right, the area form on the sphere in cylindrical coordinates is just D D D theta. There's no weird sign term or something. And the point is actually that the Deuster-Matthekmann theorem applies to the sphere as well. Z is the conserved quantity and rotation around theta is the continuous symmetry. Okay, so now what we're gonna do is we're gonna use this picture to sample equilateral closed polygons like quickly and perfectly and not a Markov chain and so on and so forth. But you realize that kind of the hard work's already done because we've already showed that what you're really sampling is a polytope. All you're sampling is this like region in Rn minus three cut out by linear inequalities that you can write down and know. Okay, so here's what it looks like. That's the picture for a hexagon. There are three diagonals and it cuts out some region in R3. It looks like this funny pointy box thing. So the triangle inequalities are the first and last guy are between zero and two and all the middle guys are within one of each other and some to at least one. That's not such a bad system of inequalities and in fact the trick is if I write things in terms of the differences between the consecutive Ds and I call those differences S for steps then as long as the step is between minus one and plus one then the condition that DI minus DI plus one is less than one is like automatically satisfied. So really I just have to pick the right steps but what's the condition on the steps? The condition on the steps is that they started zero they vary between plus and minus one and as I keep stepping I build up diagonals and the diagonals have to come back to zero for this thing to close. So it's like you're sampling a Brownian bridge or something. It's not quite a Brownian bridge it's really like an excursion because the sums have to stay positive as you go. Now it turns out that if you just sample SIs from the hypercube then the rebuilt system of diagonals will work about one over end of the three halves of the time. That was like a neat estimate that we had to do and it was way better than I thought it would be. You would think that if you're just trying to build points in a polytope from sampling some hypercube that contains the polytope the polytope's gonna occupy exponentially little of the space. So your chances of getting anything usable should be like exponentially small and it should take like exponential time to build the samples. But this polytope occupies tons of the hypercube. You know, polynomially like little of the hypercube and it's not even a bad power. So it's actually really fast to construct samples of diagonals that work. And it's really short too. So Tetsuo said I should convince people that this was not hard to program. So this is the whole thing in three lines. It's three lines of Mathematica because it had to fit on a slide. But it's not many more lines in C. And I should say that I'm not gonna read the code to you. I just wanted to point out that I could fit it on a slide. I should say first that this is not the first direct sampling algorithm. Yuanan's group came up with a direct sampling algorithm sort of based on the sync integral PDFs directly and so did Grossberg and more as well at around the same time. The difference maybe is that if you're dealing with the whole PDF you have to sample from that thing. And that's hard to do because these sync integral things are lengthy sums of alternating binomial coefficients, times polynomials and that's the density. And the point is there are these huge terms with hundreds of digits that almost but not quite cancel and what's left over is the PDF you're trying to sample from. If you do it this way you kind of automatically generate all that complexity without dealing with big numbers. But you can do more, right? So because I'm a differential geometer you know I like to really think of this as like integration over space. And if you want to integrate Monte Carlo is only one way to do it. And in fact for lower dimensional examples it's not the best way. So it'd be nice to have like some explicit grid system on polygon space so that you could like look in each box and see what kind of knot is in that box. And it turns out you can do this too. So if you look at the moment polytope and you round off every point to the one to the nearest point whose coordinates are all half integers then you divide the thing into this attractive system of boxes and triangles. It turns out that you want to subdivide those guys into right triangles or simplices depending on the signs of what you rounded off and also the size of the absolute value of what you rounded off. And when you do so you break up the space you're trying to sample into a collection of simplices that are all exactly the same size and exactly the same shape and they're all orthogonal and you can count them. And then the idea is that by picking one of these triangles at random you can then subdivide it however you want very centrically or whatever. It turns out that the number of these sort of half integer points that you're condensing to is exactly the Motskin number. The Motskin number counts a kind of excursion. It's a path that takes plus or minus one or zero steps starts at zero and comes back to zero. So in this sense it really is a random bridge. It's like explicitly a standard random bridge. But of course each attractive point doesn't attract the same amount of simplices. If you look at the one in the upper right hand corner it's got a whole square that goes to it whereas all the other guys have only like a large triangle. Now there are tons of these simplices. So this is only gonna work for so far but it really is gonna work until then. 12 guns or something, maybe 15 guns. Okay, so we did some experiments, right? So we could sample 60 guns for example and the sampling is perfect. And this is like 10 million samples. And what we did was we sampled them and then we took the Homfly polynomial of all the 10 million things. We saw about 6,000 Homfly polynomials and there were like 30 or 40 where it was too crumpled for us to resolve it. And then we made this plot which is the log of the probability of each Homfly type. They're not really not types versus the log of the rank, right? So the most common thing was rank one and the second most common thing is rank two and so on and so forth. And if you look at this plot it's really kind of suggestive, right? Because it's like a line. Now I should say this was first observed by Enzo's group and we didn't know this at the time we made this plot. We were really surprised. And Enzo's group observed it in a model of condensed globular polygons on the cubic lattice. So a really completely different model than this one. These again are equilateral polygons in space. It's an off lattice model with no repulsion whereas their model was an on lattice model with self avoidance. And yet nonetheless they both had this linear phenomenon. Now you'll notice that I said I observed 6,000 Homflies and if you can read the right hand edge of that plot you will see that I'm cheating, right? So where does my plot end? Okay so what happened to the other like five, six through the data? Well the point is that I claim those are under sampled because when you get out there what happens is that sure you see something one time in 10 million samples but that doesn't mean its probability is one 10 millionth or anything close to it. It could mean there were a billion guys each of which had probability one 100 millionth or something and you just picked one thing in that crowd and you're going to assign the whole cell to that one guy. So if you really do the plot all the way out you get something that looks like this. And Erika Yohara and Tetsuo's group made some plots like this for us and was very convinced that this log linear thing was maybe not so good because of this tail. So we decided to check it in a third model, right? And the advantage of the third model is it's a finite model and we can check every state so we can know everything. The question? Oh that would be cool. Well it also gets noisy. Okay, yeah it's just not just that. Yeah, yes that's true. But yeah our data gets kind of a little noisy at the end too. Okay so here's our finite model. Okay so the finite model is there's only so many ways to embed S1 into S2, the circle into the sphere, like up to diffeomorphism which is to say combinatorially. So if you want to see this as a division of the sphere into faces and edges and crossings there's a finite number of ways to do it and we listed them, all of them. Here's the first 40 or something on one screen. And they start with a diagram that kind of looks like the trefoil and then a diagram with some curly things in it and so on and so forth. And then to each of those pictures we associated plus or minuses randomly to the crossings uniformly. And then if the diagram had a symmetry like the trefoil diagram in the upper right hand corner, my right, I guess, your left, then we said well okay we're only gonna think of sine patterns as different if they're not related by a symmetry of the diagram. So like if you have two pluses and a minus on the trefoil it doesn't matter where they are like because there's a symmetry of the whole diagram. So with that count we could list all of the 10 crossing diagrams. There are about 1.6 billion of them and we classified all the knots in all the diagrams going all the way from the unknot which is about 70% of the space that's in the upper left there down to the 10 crossing knots which occur exactly once, some of them, and that's at the lower right hand edge. There's about 550 knot types that you see. Includes both primes and composites but of course there are no links because of the way we built the model that's only a single curve. And you see that kind of amazingly like these things look pretty linear at least to me and they even look like they kind of have the same slope. So it seems like, and there's no like bad thing at the end, right, it doesn't bend, it doesn't get noisy, like this goes through nine orders of magnitude from the unknot on the left to the single 10 crossing knots on the right. Eric Rodden is rechecking this data for us just to make sure we got all the knot types right but I think so far he hasn't done enough checking to tell whether we have any errors. So I claim that that means so far we're passing Eric's tests and it seems like this is really right. So, this is like my question for you, like should we have expected this? Is this some kind of universal feature of random knotting models? These log linear laws occur a lot, they occur in the frequency of words in large texts, they occur in social networks, they occur in random graphs and things. So why is a knot like a word? Okay. So let's see, I wanna talk a little bit more about the model that Tetsuo discussed earlier today. So here's a way to say it which is or in differential geometry language. So the idea is that there's an identification between closed framed curves in R3 and a Grasmanian of two planes in a complex vector space. So let me just like draw a picture, okay? So for instance, all I'm saying is that I have like CN for example and then I'm just gonna take two Hermitian orthonormal vectors A and B and this is Tetsuo's quaternion method picture where the entries in the A and the B combine to make quaternions. Those quaternions are pushed through the hopf map to become frames and those frames are the edges of the polygon. But notice that this is an amazing structure because it's not just a measurement of volume, it has a full Ramanian metric. So for instance, if you wanna compare one protein shape to another protein shape, you can ask what the geodesic distance is in the Grasmanian from that plane to the other two plane. And it's actually like an easy and fast and interesting computation to do it. If you want to think about smooth curves instead of polygons, you can do that too. All I need is a finite dimensional complex vector space like say generated by Fourier functions on the circle of degree no more than N or something. That's a nice L2 orthogonal basis for periodic functions on the circle which are complex valued. And two planes in that are the Fourier curves whose coefficients are not that big. Something that I find really interesting about this is that naturally framed space curves are not one Grasmanian but two Grasmanians. The first Grasmanian, the functions are periodic. The second Grasmanian, the functions are anti-periodic. That is to say like the function at zero is minus the function at two pi. Both of these map to curves downstairs with closed frames but which curves? Like the pretty theorem is that one Grasmanian is the curves with even self-linking number and the other one is the Grasmanian of curves with odd self-linking number. Because when you pass a curve through itself and do like link equals twist plus rive, the self-linking number of the frame changes by plus or minus two. So there can't be any way to get from one Grasmanian to the other one. A really pretty fact here is an interpretation of the total twist, right? So I told you once that every continuous symmetry has a conserved quantity, right? So there's a continuous symmetry of curve space where you change the base point. You push the base point around the curve. What's the conserved quantity? Twist. So twist is the Hamiltonian of that symmetry of the space of framed space curves. There's like more you can do but one thing that I'm really looking for co-authors for because I know how to do like the technical part but I don't really know what the interesting experiment is is that we can generate random framed space curves which are smooth and have frames from the beginning to the end. So there are these cool experiments that I saw once. I think it's like Steve Levine was doing them where you take curves that like get close to each other at the end and you say the alignment of frames relates to the probability that the polymer will cyclize and form a loop, right? If they're sort of matched up, they'll bond but if they're like rotated in some weird way they won't bond and this is a really natural way to generate ensembles for this but of course I don't know any of the constants so I don't know which experiment would be an interesting computation of cyclization probabilities but maybe one of you does. Okay, so it's late. So I will just say thank you for listening and these are the references.


So, last time we have discussed how clouds get generated in an atmosphere which is conditionally unstable that is to say which is stable to vertical displacements of dry air, but unstable with respect to vertical displacements of moist air. Now, today we are going to talk about organization of these individual clouds over different scales, mesoscales, synoptic scale and the planetary scales.  Now, such clouds have a typical horizontal extent of about 5 kilometers, they are organized into systems of larger spatial scales. Now, these important scales are mesoscale which is about 10 to 200 kilometers, so which will have several clouds of extent of 5 kilometers, then synoptic scale which is hundreds of kilometers and planetary scale which is thousands of kilometers. Let us see examples of this, now thanks to satellites we see examples of this very readily. This is a picture which shows nicely the mesoscale convection that you see here on the west coast of India here, a nice small blob, but notice see this is about 1000 kilometer, so this is only about few 100 kilometer maybe 200 kilometer or so in extent and it is isolated all around it is clear sky. So, this is a mesoscale convection you see here which is a scale of about 10 to 300 kilometers. This is another example of mesoscale convection, you see a cloud band stretching all along the west coast here, so this is another example of mesoscale convection. Now mesoscale convective cloud complexes could be studied only with the advent of satellites, but over the last decade or so there has been intensive study of this system and in all systems of every scale that spatial and temporal scales are intimately related ok. So, now if we look at mesoscale cloud complexes here is the size in kilometers and this is from a study done over the Pacific and this is the lifetime in hours. So, what you find is that the smaller clouds stay around only for a couple of hours or so as the cloud complexes become larger and larger then they stay almost to 18 hours or so. These are the big clouds and this is the frequency distribution. This is saying that the small clouds are much more frequent and larger cloud complexes are less frequent, but once they occur they last much longer. Now, next we see this is the mesoscale, next we see the synoptic scale. Now synoptic scale disturbances we have already come across before these are the like the monsoon disturbances we talked about last time. The synoptic scale disturbances are of the order of 100s of kilometers and you see here this is one synoptic scale disturbance this is another and there is another over by specific here ok. Now, in addition to that there is planetary scale disturbances as well and you can see that these synoptic scale disturbances or synoptic scale systems are organized over a much larger scale of 1000s of kilometers. This is the planetary scale which is 1000s of kilometers now it so happens that interactions between the different scales over which convection or rainfall is organized play a very important role in the variability of the convection on different time scales. Now, an interesting example of this interaction is what we will talk of now. Tropical cyclones are generated as loads or cloud blobs in the planetary scale cloud band associated with what is called a tropical convergence zone. Now, I will explain in detail in the next set of lectures what a tropical convergence zone is what are the dynamical characteristics and so on, but we have looked at already looked at the cloud band which is associated with a tropical convergence zone. This is an east west cloud band which occurs on the tropics so tropical cyclones are generated as cloud blobs in this cloud band as this low intensifies it moves away from the band and north south component of this track is always poleward. In other words in the northern hemisphere it would move to the north in the southern hemisphere it would move to the south and as it intensifies further the band begins to weaken the parent that gave birth to these cyclones begin to weaken and when these cyclones move away very far only then the band revives. A very interesting case of this occurred this month in October 2012 with one tropical cyclone emerging from each end of a band over the equatorial Indian Ocean. Now, this is a picture over the equatorial Indian Ocean on a date in October and it is 17th of October and what you see here is a cloud band. This is the TCG at either end of the cloud band are lows high clouding or intense clouding in blobs this designates the lows of the thing and they are at either end of this band. Now let us see what happens now this is the same band seen through an inside satellite and you see again see there is a cyclone here a cyclone here and this is the tropical conversion zone this is the band. So, at either end of the band there are cyclones that you see here now let us see what happens to them. Now, what happens is this cyclones intensify you see this one has intensified this one has also intensified and the band in between has weakened on the next day. So, the one over the western Pacific moved in fact see there were two cyclones that you saw this is the cyclone over the western Pacific this is the cyclone over South Indian Ocean and if we go back these were the two cyclones that you had and now you see this band has become weaker as the one over the west Pacific has moved northward the one over the South Indian Ocean is moving southward as I mentioned before they always move towards the pole. So, the one over the western Pacific moved northward whereas, the one over the South Indian Ocean moved southwards and now you see that the band has actually disappeared there is not much of a band here only one blob remains this is the typhoon over the west Pacific it has intensified to become a typhoon and this is the cyclone over the South Indian Ocean and you see the band structure that you had has very much weakened now you see the band has actually disappeared the TCG has actually disappeared and what you see here is a typhoon over west Pacific and a tropical cyclone over the South Indian Ocean you can see that they have both moved forward the one in the northern hemisphere has moved to the north the one in the southern hemisphere has moved to the south and this band which gave birth to both of them has actually disappeared what happens the next day next day now these cyclones have moved further off you can actually see the eye of the cyclone here and this is a typhoon here of which also you can see the eye. So, these have both become very very intense systems, but they have moved away from the original location they have moved very much northward. So, this is almost a 20 north this all began at the equator here this has moved to 20 north and this has come close to 20 south. So, the systems have moved northward and the band has revived has appeared again and you see that here as well in the global picture and notice by the way that the band I am talking about the TCG is seen all over the globe in the equatorial region you see it here in the Pacific you see it here across the Atlantic and here it has now weakened because of the cyclones, but what you see here are the two cyclones a typhoon in the West Pacific and a cyclone in the South Indian Ocean. And now this has very much weakened the South Indian Ocean one this one is still alive the West Pacific typhoon and meanwhile this band the TCG has flared up because these have moved away. You see the same story here in the global picture and you can see how the band is stretching right across the equatorial region right across the globe. So, the TCG has revived here and the two systems have moved far away. So, this is a very very interesting case of an interaction between a planetary scale and a synoptic scale. The synoptic scale systems were born in this planetary scale TCG then the intensified moved away and went forward which while they were close to it the band had disappeared once they moved sufficiently far the band has appeared again. See so now you see the band in all its glory very much intensified back in action whereas this is the remnant of that cyclone that was over the South Indian Ocean and this is the remnant of the typhoon over West Pacific both of them are about to die and this band has revived again. So, this is the same story you see here again this is almost reached coast of China now this has completely died and we have a revival of the planetary scale. So, as the cyclones died the cloud band revives. Now it is very important to understand you see now how the intense the cloud band has become nothing is left of the South Indian Ocean cyclone and this has also West Pacific typhoon has also disappeared. See these kind of interactions are very very important to study in the tropics and understanding these will have a major contribution to our understanding the variability of the tropics and now you see a totally revived planetary scale cloud band stretching again right across the globe. So, now we have to try and understand organization of these clouds over synoptic and larger scales you saw a nice example of organized clouds over synoptic and larger scales and the two systems interacting. Now why how come they get organized on synoptic and larger scales was a question that was very much a challenge in the mid 60s. Now why is it a challenge to understand because we have seen that ascent of the moisture at the surface at least up to a lifting condensation level is a necessary condition for generally getting clouds. This we have seen because without ascending to lifting condensation level water vapor cannot become water so we cannot get clouds. So, we need ascent of air to get clouds. Now we could have gotten individual clouds and in fact rarely, but we do experience them in the pre monsoon season of April and May. You know we get very heavy showers and some afternoons in April May when the surface air is forced to ascend because of the intense heating of the land. You know in the summer the land gets very hot and this causes a low pressure and ascent and this leads to formation of clouds and thunderstorms are a very familiar occurrence in April and May. The resulting thunderstorm associated with an isolated cumulonimbus or isolated clouds is often accompanied by hail why because the clouds are very very tall they go way above the freezing level. So, not only do you have liquid water in them, but you have ice as well. So, these thunderstorms are often accompanied by hail since the cloud is deep with the top above the freezing level. Now rain from such isolated clouds occurs over regions of very small scales because we have seen that the clouds have horizontal extent only a few kilometers. So, the rain from such clouds will also have typical scales of few kilometers and the demarcation between rainy and non-rainy areas is often very clear. In Maharashtra the saying goes that such thunder clouds give rain which can wet one horn of a bullock leaving the other horn dry. Now this is a something we experience all the time part of the road is wet the other part is not wet and so on and this is because the cloud the cloud that gives us rain has a relatively small spatial extent. Now the major question as I mentioned earlier is how do these clouds get organized over larger spatial scales. We have seen from satellite pictures that the clouds are very seldom individual clouds in fact one would not see them in the satellite picture at all if they were only few kilometers, but rather very often we see these clouds are organized over hundreds of kilometers and thousands of kilometers. Now how does this occur? Of course a critical feature is the ascent of moist surface air up to the level of lifting condensation without which you cannot get clouds at all. Now such an ascent can be forced by orography by orography one means topography mountains as we have seen earlier. Now we have seen that during the summer monsoon there is heavy rainfall over the west coast of the peninsula. This is the mean rainfall pattern for July and what you see here is that the rainfall is very heavy here over the west coast and western guards as well. So the rainfall is very heavy here in July and this large part of this comes because of the western guards which lie all along here. You see the winds surface winds in July want to go across here, but what do they meet? They western guards here and obviously the winds cannot penetrate the guards. So here upstream of the guards they have to ascend and this ascent leads to clouding because the air is very moist. So the air is forced to rise upstream of this topographic feature thus topography can provide the ascent necessary for cloud formation and rainfall. Now the heavy rainfall along the west coast of the peninsula has therefore been attributed to western guards. Topography is also believed to play an important role in the heavy rainfall over the northeast. However we do get a lot of organized rainfall over regions where topography is not important. So organized rainfall over regions where topography does not play an important role is generally associated with synoptic scale systems. Synoptic systems also contribute to the rainfall over the west coast. So it is not as if the rainfall over the west coast is entirely due to orography very often the most intense rainfall over the west coast occurs in association with synoptic scale systems over the Arabian Sea. Now it is important to note that the horizontal extent of the individual cloud is only around 5 kilometers. Now if we consider the conditional instability of the tropical atmosphere then the scale that grows the fastest will be this cumulus scale of 5 kilometers this we have seen in the last lecture. That is to say if we had a disturb the atmosphere if the perturbation had energy in all the scales what will happen with this conditional instability is that the smallest scale the scale of our few kilometers will grow the fastest. So eventually the perturbation will have will be dominated as the perturbation grows it will be dominated by that scale which grows the fastest in the instability which is actually the cloud scale. So how then can the synoptic scale cloud system get selected for in a conditionally unstable atmosphere can we conceive of a way by which of course clouds have to form because that is the scale which is most promoted by conditional instability the cloud scale. But how do we get scales of the higher like synoptic scales and planetary scales and so on. So how could convection of a larger scales be selected for since in a moist tropical atmosphere the cumulus scale must always been in the competition as I have said earlier. Now this is the question that was posed by Charney and Elyson for understanding why cyclones over why cyclones form and intensify in the conditionally unstable atmosphere. So note they are asking the question that how come in a conditionally unstable atmosphere systems of scales of hundreds of kilometers can get generated can form and intensify. So this is the question posed by Charney and Elyson and they suggested that the key to our understanding this was that the two scales the synoptic scale and the cumulus scale actually do not compete but cooperate with one another. Now this was a entirely original suggestion that they came up with that the interaction between the cumulus scale and the synoptic scale involves cooperation rather than competition. How do they cooperate now we have seen that ascent of the moist air surface air up to the level of lifting condensation is essential for clouds to form right. Now such an ascent occurs over the region of cyclonic vorticity because of convergence in the boundary layer. See this is again something we have seen in the earlier lectures that when there is cyclonic vorticity what do we mean by cyclonic by cyclonic we mean vorticity which is the same sign as the rotation of the earth. So when there is cyclonic vorticity above the boundary layer there is convergence in the boundary layer you may remember this picture we have seen earlier. When there is cyclonic vorticity above the boundary layer it is like the fluid is rotating faster and the earth itself is rotating with this velocity omega. So the air above the boundary layer is rotating with omega plus delta omega in this situation what you get is convergence in the boundary layer and ascent of air from the boundary layer. This is the special property of the Ekman layer or the boundary layer in a rotating fluid that we had seen earlier and this becomes very very critical in understanding this cooperative interaction. So what what how does the cooperative interaction occur we have a cyclonic vorticity over this region this is the vortex cyclonic vortex that you call a low or whatever low or a depression or whatever it is but Chen is thinking of in fact genesis of the low. So suppose you have a cyclonic vorticity here it leads to convergence in the boundary layer and ascent of air in the region of cyclonic vorticity. Now since the air is moist this kind of strong ascent of air will lead to clouds and deep convection of this kind. Now the this is what leads to organization of individual clouds on the scale of the vortex itself which is on the scale of the synaptic system. Now what is the cooperation involved the cooperation is the following. See initially for clouds to form you need ascent of air and this ascent of air is provided by the vortex by the cyclonic vorticity associated with the synaptic scale system. So the synaptic scale system by having the cyclonic vorticity triggers ascent and therefore clouds. Now what happens when we have clouds when we have clouds we get heating in there why is there heating because latent heat of condensation is released when water vapor it gets converted into water droplets for cloud in clouds. So we have heating within the clouds now what will happen with this heating when you have heating in this region the pressure will become lower in this region which would mean that the conversions will intensify which means that ascent will intensify which means you will get more clouds. So this is what we see here this is the second situation here that we have an Ekman layer or a boundary layer here this is the vertical velocity upward because there is a cyclonic vorticity and this releases a lot of heat and how much heat is released is actually proportional to this upward velocity. Now let me distinguish this from the earlier problem which was the birkenness or what we call the lily problem as well in which case the heating was proportional to the velocity of ascent and this is how clouds got selected. Now when we have synoptic scale then Ekman layer becomes very very important in creating the convergence which is required for the ascent and development of clouds within the synoptic scale. So now once the clouds form they heat the atmosphere due to the release of latent heat of condensation which is more than which more than compensates for the adiabatic cooling because you know as air passes rise up to higher and higher levels you know that they cool because of adiabatic processes but latent heat release more than compensates for the adiabatic cooling. So we have overall heating within the clouds this heating lowers the pressure of the synoptic system intensifies the convergence and leads to an increased convergence in the boundary layer increase the ascent this in turn leads to more clouds and more heating. So we have this kind of interaction in which for clouds the ascent required up to lifting condensation level is provided by the synoptic scale system and clouds in turn by heating all over the ascending region of the synoptic scale system intensify the synoptic scale system. So organization of clouds over synoptic and larger scales is possible because of the positive feedback between the low level convergence associated with the cyclonic vorticity above the boundary layer the heating of the atmosphere by the clouds and the intensity of the lower depression. So there is a positive feedback between the cloud scale and the synoptic scale because of the relationship of the convergence to the clouds. Now this is called conditional instability of the second kind to distinguish it from the first kind which gives cumulus clouds. So this was a major contribution of Charney and Elyson and what they said was that the organization of convection over synoptic scale as in a tropical cyclone or over the planetary scale as we have seen in the cloud band associated with the tropical conversion zone which is TCC which we have seen examples of earlier in this lecture. So it is a manifestation this organization over the larger scales is a manifestation of an instability conditional instability of the second kind of sisk in which these larger scales are selected for this is what Charney and Elyson said in 1964. Now this theory was extremely appealing and received very wide support for two decades after it was proposed and is actually now a part of many textbooks on tropical systems and tropical dynamics. However even in the Charney Elyson paper we find that the growth rest of sisk unstable modes are rather uniform over a broad range of horizontal scales rather than peaking at the synoptic scale. So that is to say when we first consider conditional instability of the first kind what we find that growth rate peaks at the cloud scale and so we expect clouds to be promoted by conditional instability of the first kind. However there is no such peaking at the synoptic scale when you do the mathematics of conditional instability of the second kind. So this created a problem because soon after Charney and Elyson suggested the hypothesis Saracen Israeli showed that there was no selection for large scale mode as suggested by Charney and Elyson. So although it sounded very good this kind of cooperation rather than competition and positive feedback between the scales and so on actually when the mathematics was done they found that there was no selection for synoptic or larger scale. Still it is only the cumulus scale which would get selected for if we looked at tropical cyclones as being generated in a conditionally unstable atmosphere. So what they are saying is that you will not get the systems or vortices of the scale of that we see synoptic scale generated in a conditionally unstable atmosphere by this instability. However soon after this the possibility of viewing synoptic scale cyclone as a manifestation of this instability was revived by the work of Srinivasan and Smith who showed that selection for a large scale mode occurs if there is a lag between convergence and rainfall as suggested by Emanuel. So it is possible under certain circumstances to actually get selection for a larger scale but you know there have been a lot of controversies about this concept of SISC. The concept however the concept of cooperative interaction between cumulus and larger scales which was proposed by Charney and Elyson is still very widely accepted. So it was an idea that has found favor throughout for four or five decades after Charney Elyson paper. In fact Uyama has done a very nice critical assessment of the whole thing and he says the spirit of SISC as the cooperative intensification theory is valid and alive. So although there have been many many objections raised to SISC because one does not quite see the selection for the synoptic scale that one expected a synoptic or larger scale the major concept introduced by Charney and Elyson. Namely the fact that these two scales can interact in a cooperative way is still considered is valid and alive. So the linear theory of SISC proposed by them has been widely criticized as I mentioned since 80s. It has been suggested and this is the most important criticism of that theory that because it is a linear theory it fails to take into account the non-linear feedbacks of processes which are needed to explain the dynamics of a mature tropical cyclone. So these feedbacks are inherently non-linear which a linear theory cannot really take into account. So actually a cooperative intensification theory was proposed independently by Uyama and what he did was not consider the synoptic scale vortex as being generated from the conditional instability as Charney and Elyson had done. Rather he considers what would happen to a vortex which is generated in a conditionally unstable atmosphere. So Uyama's formulation of a theory of cooperative interaction between cumulus clouds and an incipient vortex differs in certain respects to that of Charney and Elyson. Uyama considered a flow configuration with two layers of air above a shallow boundary layer of uniform thickness and he represented the heating effects of deep cumulus clouds in terms of mass flux from the boundary layer to the upper layer. So this is very similar to Charney and Elyson right. Mass flux from the boundary layer of moisture from the surface is very important. So boundary layer convergence is important in Uyama's theory as it was in Charney and Elyson's but the representation is based on the idea the deep cumulus clouds that form in such a region will also entrain ambient air from the middle of the layers of the atmosphere. So as the parcels of air rise from above the boundary layer through the atmosphere these moist parcels of air will actually entrain drier air from the surrounding region as they rise. So what happens then is for each mass each unit of mass transferred from the boundary layer into the upper layer, eta minus 1 units of mass are entrained from the middle layer and transferred to the upper layer also. So in a sense the net heating of the upper layer is proportional to eta and W. So the dilution of eta the dilution of the air has an impact in how much heat is released the more the dilution the less heat will be released because drier air has got mixed up. So he argued that the process of cooperative interaction between cumulus convection and a vortex must be intrinsically nonlinear this was Uyama's argument. So the process is represented in the nonlinear model from the basis of Uyama's concept of cooperative interaction or CISC which he viewed as a theory of vortex intensification from a state in which organization of convection by rotation was already present. So he says the vortex is already there and we talk of intensification this is a departure from the Charney Elyson formulation of the theory. Now numerical integrations of the nonlinear equations were able to produce hurricane like vortices with a considerable degree of realism including their growth rate, radial scale and mature strength. So this was a major achievement and Uyama pointed out that his model calculations demonstrated that latent and sensible heat transfer from a warm ocean were crucial to vortex intensification. So latent heat transfer of course Charney had talked about he said sensible heat transfer is also important and both lead to have contributions to make to the intensification of the vortex. Now Immanuel was one of the fiercest critiques of Charney Elyson theory and he said that actually intensification involves what he called wish which is a finite amplitude instability associated with wind induced surface heat exchange. So wish stands for W is for wind induced surface is S heat is H and exchange. So wish stands for wind induced surface heat exchange. Now I will consider all these theories in detail they cannot be considered without writing the equations and formulations and so on. So we will postpone that to a later date and just get an idea of the kind of physics they are suggesting. Now so there are these two streams of thought if you wish what happens how do we get selection for synoptic scale disturbances from the systems and Charney and Elyson think that in fact the conditional instability itself gives rise to synoptic scale disturbances whereas there has been a lot of controversy of that but Uyama suggests that incipient vortex intensifies due to the mechanism suggested by Charney and Elyson. Now we have seen once soon disturbances that occur in our system these are lows depressions and so on and we have also seen how over the Indian Ocean you get tropical cyclones now how do these systems get generated the last word on this is yet to be said. However what is very clear what people believe right now is that probably the vortex gets generated due to some kind of instabilities of shear flow in a rotating system horizontal shear corresponding to barotropic instability vertical shear of wind corresponding to baroclinic instability and because the atmosphere is unstable to both in terms of barotropic and baroclinic instabilities these vortices get generated. Now once they are generated the kind of cooperative interaction that people talked about actually takes place and then they intensify. So this is the present view of how these systems get generated and so on and so this is the present view of how the monsoon disturbances occur and now I think we have enough of a background to ask the question how do we get the monsoon what is the basic system responsible for the monsoon. So again I must remind you what is the critical background that we had we had we have seen that in a rotating system cyclonic vorticity leads to ascent of air. Now why is ascent of air important because our atmosphere is conditionally unstable right it is stable with respect to vertical displacement of dry air but unstable with respect to vertical displacement of moist air parcels upward displacement of moist air parcels it is the atmosphere is unstable. So if you can put up the atmosphere and make surface air which is moist rise to a certain level then particularly when you can make it rise to a level above the lifting condensation level. Now again to remind you what happens when we let a parcel of air rise actually if we do it adiabatically without adding any energy to the parcel then it will expand because the pressure at the higher level is lower and because it has to do it at its own cost it cools. So it cools and if that is all the process taking place because it cools it finds itself denser than the surrounding and returns to the original level. This is the stability this is the stability of dry parcels but if we can push the surface air up to a level at which condensation can begin because as it cools the amount of water vapor it can hold decreases so eventually as it cools it becomes saturated although it is never saturated at the surface. Now it becomes saturated at the lifting condensation level. Now beyond this level then latent heat of condensation gets released and it so happens that the tropical atmosphere the temperature variation with height is such or the lapse rate is such that in this situation when actually water vapor starts condensing to water liquid water then the moisture parcel that we began with at the surface becomes warmer than surrounding and keeps rising and this is the conditional instability we talked about which is in fact the critical thing required for getting clouds. So the clouds that we see in the tropical atmosphere are a result of the conditional instability and this conditional instability why is it called conditional because it tropical atmosphere is not stable to moist parcels of air but it is stable to dry parcels of air when lifted up. So the conditionally unstable atmosphere we showed in fact if we consider the instability problem of which scale gets selected for in a conditionally unstable atmosphere then we find that if we looked at a inviscid fluid right no viscosity at all then the thinnest cloud will get selected for that is the smallest horizontal scale gets selected for but in reality you cannot just ignore viscosity totally. So what happens is that viscosity comes into play and the thinnest clouds in fact will not be get selected for because the thinnest clouds have much larger area relative to volume and therefore you know the area relative to volume goes like one over the radius. So the thinnest clouds have much larger area relative to volume so due to entrainment of dry air the thinnest clouds will actually not be able to grow the fastest and in a real fluid then the clouds that will grow the fastest are roughly the scale of the troposphere which is about 10 kilometers or so. So the thinnest clouds the clouds that get selected for in a conditionally unstable atmosphere have horizontal extent of the order of 5 kilometers. So there were two things then we learned that we need ascent of surface air for clouds to form then how do we get ascent of air over scales for which rotation is important. Now these are the larger scales say from few hundred kilometers and so on and for these larger scales we noticed we noted that the effect of rotation of the earth becomes very important. So effect of rotation of the earth is important for these scales and so the boundary layer changes character in these rotating systems we have to consider rotation of the earth. So we looked at we look at boundary layers in rotating systems instead of non-rotating system. Now once you start looking at boundary layers in rotating systems the major difference comes in the fact that the boundary layer in a rotating system is a highly interactive boundary layer and when you have cyclonic vorticity above the boundary layer that is the atmosphere or ocean above the boundary layer is rotating in the way relative to the earth with the same sign as earth's rotation in that case we get convergence in the boundary layer and ascent of air above the boundary layer. So this mechanism is extremely important for giving ascent over special scales which are large enough for rotation of the earth to become important in their dynamics. So we need ascent of air up to the lifting condensation level for clouds to form ascent occurs in rotating systems provided there is cyclonic vorticity above the boundary layer. These were the first two lessons we learnt and the third lesson which is what we looked at this time was what how then can in this kind of an atmosphere how do systems of a larger scale which have clouds organized over that scale in fact get selected for or what is the basis for occurrence for organization of clouds over synoptic and larger scales and this is where an entirely new concept was introduced this concept that it is not that there is competition between scales whenever we looked at look at a problem as an instability problem we always consider what will happen to an unstable system when you have a perturbation which has energy in all kinds of scales. What happens is in that scenario when we have a perturbation with all kinds of scales the scale that grows the fastest which is determined by the nature of the instability itself is the one that gets selected for. So in that scenario you will get selection for that scale which is maximally efficient in tapping the instability of the system. So this is the approach we take and if we took the same approach for a conditionally unstable atmosphere then always in the competition with other scales the cumulus scale will win because that is what the mathematics of the instability of the conditionally unstable atmosphere tells us. So in that situation cumulus scale will always win but if we look at systems over the tropics we have very intense systems like tropical cyclones which have scales of few hundred to thousand kilometers and we have day after day organized clouds over planetary scale these are east west bands that we saw of clouds seen by satellite which stretch over thousands of kilometers and which are typically few hundred to thousand kilometers in latitudinal extent as well. So question is how then in this atmosphere we get organization of the clouds on these scales and as I mentioned that this is the problem that was addressed by Charney and Elishen and is actually at the heart of our understanding not only the monsoon but cloud systems over the tropics of which monsoon is a specialized case. So what we have now learnt today is that early attempts to look at conditional instability of the second kind by invoking one more phenomena namely the possibility of a cooperative interaction between the larger or the synoptic scale and the cloud scale and doing a linear stability analysis like we do for shear flow instability or like we did for the conditionally unstable atmosphere. So if we do that kind of an exercise the linear exercise then what was found is that in general there is no scale selection that is to say the so called conditional instability of the second kind will not promote the synoptic or the larger scale there is no scale selection from the cumulus scale to the synoptic scale. So there is no reason to believe that the synoptic scale disturbances will arise as a manifestation of conditional instability of the second kind even when we take into account the new phenomena that was proposed by Charney and Elishen which was cooperative interaction between the cumulus scale and the synoptic scale. So even when we assume that what occurs in real life is an interaction where by the synoptic scale system by creating convergence in the boundary layer and Ekman layer and ascent of moisture above the boundary layer can generate clouds provided the ascent is up to and beyond the lifting condensation level and the clouds by heating the region over which they are generated the region of the cyclonic vortex or region over which ascent is taking place from the boundary layer by heating that they actually make the pressure of associated with the vortex lower intensify the convergence and therefore intensify the ascent and clouds. So even if we take into account this kind of positive feedback between the scales with Charney and Elishen tried to do in their linear theory there is no selection for the synoptic scale with one exception and that as I mentioned that if one assumes that there is some time lag between the convergence and rainfall that occurs then you can get some selection for the broader scale. But on the whole this whole concept that you can get selection with that kind of formulation was criticized heavily then Uyama came up with this theory that in fact the key point of Ciske and Charney's work the cooperative interaction and it is now believed that the cooperative interaction is in fact the very very important part of tropical convection over synoptic and larger scales. In fact the strong link between vorticity and convection cyclonic vorticity above the boundary layer being essential for tropical convection over synoptic and larger scales which makes possible this positive feedback between the cumulus and the larger scales is an extremely important concept. And that concept is still very much alive today so it can help us understand how the two systems interact and just using the idea that this kind of cooperative interaction can play a very very important role in intensification of the vortex. So if you start with an incipient vortex in a conditionally unstable atmosphere this positive feedback between the two scales can lead to intensification of the vortex in fact was shown very nicely theoretically by Uyama who also pointed out that the non-linear feedbacks are very critical in getting a realistic intensification of the tropical cyclone. So at the end of the day then we believe that the key concept introduced by Charney and Elishan about actually the cooperative interaction between the cumulus scale and the synoptic scale is valid is considered valid today but it is not theoretically possible it has not been theoretically possible to show that one can conceive of the synoptic scale systems as being generated from a conditionally unstable atmosphere as a manifestation of the instability. I must also mention that the role of sensible heat. Now sensible heat is the heat transfer that occurs from a warmer surface to a colder surface and generally you know the atmosphere is overlying an ocean which is warmer. So sensible heat from land or ocean to the atmosphere is also supposed to play a very critical role. Now this was pointed out both by Uyama and Emmanuel and Emmanuel even coined a word special instability theory for it called the wind induced sensible heat flux, sensible heat energy exchange at the ocean atmosphere interface of the tropics. So be that as it may it is very clear that fluxes of latent heat and sensible heat from the ocean are very important for tropical convection and this kind of non-linear feedback between the larger scale systems and the cumulus cloud is also very critical for our understanding how tropical convection occurs on these scales. So now we have the background to actually pose the problem we started with. What is the basic system responsible for the monsoon? What is the physics of the monsoon? Because it is only when we identify what is the basic system responsible for the monsoon will we be able to propose hypothesis for understanding the nature of variability of the monsoon. And only when we can test these hypotheses that are proposed for nature of variability of the monsoon with models we will be able to incorporate the right physics in the models to make it possible to predict the variability of the monsoon. So now with all this background both in rotating fluids and clouds in tropical atmosphere we are ready to embark on the central problem of this lecture course. What is the monsoon? What is the basic system responsible for the monsoon? And that is what we will look at in the next lecture. Thank you.