We have been talking about the evolution of the ideas of the basic system responsible for the Indian monsoon. And at the end of the last lecture, we saw that in fact, there was a suggestion that the basic system may be the equatorial trough for ITCC, but there were serious objections to that view. And I mentioned the objection of Murakami, who said that if you look at the low OILA then the system responsible for the monsoon has to be considered distinct from the equatorial trough or ITCC associated with the rainfall over the Indian ocean in the northern winter. This is what he said in the other major book on monsoon, which came out also in 1987 by Chang and Krishnamurti. Now Murakami suggested that the system over the Indian ocean in the northern summer is different from the ITCC, because the latitudinal extent of the low OILA region which is the region of convection or rain over the Indian longitudes during the summer monsoon is much larger than that found elsewhere in the tropics, such as the Pacific where it is associated with the ITCC. So, that whereas the extent is about 10 degrees or so in the Pacific or Atlantic, it is much more like 30 or 40 degrees over the Indian longitudes in the summer. Now, it is interesting that Blanford also did not believe that the monsoon was a manifestation of the seasonal migration of the equatorial rain belt. He felt the monsoon was definitely associated with the equatorial rain belt, but he did not believe it was a migration. In fact, the way he expressed it is as follows. The equatorial belt of constant rainfall exists across the monsoon region and is not bodily transferred northward to India and southward to Australia with the annual march of the sun in declination is a well established fact is the way he puts it. He says rainfall registers of the Malay Archipelago remember he is talking in 1886. Sure that while in the neighborhood of the equator the season of the heaviest rain and the most frequent rainfall is from November to January, there is no month in which the precipitation does not amount to at least 3 to 4 percent of the annual total. In fact, during the monsoon the whole region between the equator and the Himalayas is more or less one of precipitation and may be regarded rather as an extension and broadening out of the normal equatorial rainy zone with a northward transfer of the of its maximum and a partial concentration in north India rather than a bodily transfer of the zone northward to southern Asia. Now, this it turns out to be a very, very perceptive remark and we will see that in many ways this is borne out by the later studies. So, there is a difference now between Murakami's approach and what Blanford said while Murakami questioned the assumption that the basic system is the ITCC. Blanford's view was that the same system meaning the basic system for the monsoon was the equatorial trough for the ITCC, but it had a larger latitudinal extent. He says the system is kind of spread from equator to north India. Thus the problem is to determine first whether the nature of the dynamical system responsible for the large scale summer monsoon rainfall over the Asian region is different from the ITCC with which organized rainfall over the non-monsoonal regions of the topics. This problem was addressed by Gadgil 1988 using insights provided by Sikha and Gadgil 1980 study which I have mentioned before. Now, it is worthwhile I think to look into the details of the results of the Sikha Gadgil study because it is very pertinent to the question at hand. This was a paper published in 1980 called on the maximum cloud zone and the ITCC over Indian longitude during the south-west monsoon. And it was the first systematic study of satellite imagery. What Sikha and Gadgil I will refer to them as SG from now on investigated was the daily variation of what they called the maximum cloudiness zone and I will define how they defined it over 70, 80 and 90 degrees east based on NOAA satellite imagery of the northern hemisphere and also the location of the 700 millibar equatorial trough. Now, SG showed that in satellite imagery the cloud band over the Indian region during the summer monsoon looks very similar to that associated with the classic ITCC over the pacific. And it is also similar to the cloud band over the equatorial Indian ocean in the pre monsoon that is April and we will see that in the slide. So, this is the kind of imagery that they looked at satellite imagery from NOAA this is the kind that used to be available in those days. This is for 8 July 1973 and you see the band is somewhat very similar in terms of appearance at least to the one in the pacific. This is the classical ITCC over the pacific with a tropical cyclone or a typhoon taking off from it. It is also similar in looks to this equatorial band here which you see this is in March this is an equatorial band. So, for as far as appearance goes the band that we see over India looks rather similar to these bands this is the one in spring again you see the ITCC stretched across the equatorial region here. So, in fact what they found was that the cloud band over the Indian region often extends eastward over the tropical pacific and on occasion as far as east pacific. So, it is not as if it is an isolated cloud band rather the same band is seen as being part of a band which is almost girdling more than half the earth. So, there is the this is the band that you saw earlier India is here, Ceylon is here, Sri Lanka is there. So, this is the band you saw earlier this is going towards the west this is west pacific now. Now, it is going slightly equator word here in this imagery this is still a hemispheric imagery and you see it as a this is over the pacific and you see it as a continuation almost up to up to about half the hemisphere as you can see. So, this is sort of giving even more support to the idea that it is the same beast that we see here we see a part of the same creature here and that is what is giving us monsoon rainfall. This is only of course imagery can only suggest. So, now question is we have to also see if dynamically you know the rain system over the monsoon region has the characteristics of an ITCZ. So, what are the important characteristics of an ITCZ? See dynamically the most important features of the ITCZ are large convergence in the boundary layer which as you know in rotating systems is associated with high cyclonic vorticity above the boundary layer. So, intense cyclonic vorticity above the boundary layer large convergence in the boundary layer and deep clouds and rainfall which means intense moist convection. So, these are the characteristics of the ITCZ. Now, the cloud band over the Indian region on 8th of July is associated with cyclonic vorticity at now let us go back and see that is the cloud band we are looking at this is the cloud band for 8th of July 1973 and let us see what is the circulation associated with this cloud band on 8th of July that is shown here this is from Sikha's paper in 1977. Now, what you see here is the story at 850 HPA this is just above the boundary layer this is about 1.5 kilometers above the surface of the earth. So, what you see here is cyclonic vorticity see in fact, the winds are westerlies here and easterlies to the north. So, the these are vortices which are anticlockwise or cyclonic they are going like this and so, you see a region of high cyclonic vorticity just where the cloud band was this is at 850. Now, see what is happening at 700 and you will come to realize the importance of this level 700 as we proceed in the lectures, but the important point is that when you have intense moist convection you also see high cyclonic vorticity at 700 HPA and the trough which would be here is in fact, just above the trough at 850 here. So, and same thing at 500 500 also you have the trough more or less at the same location. So, you have the system is essentially vertical with intense cyclonic vorticity and convergence up to a level beyond 700 HPA even higher than 700 HPA up to 500 and beyond this is 200 HPA. Now, this is another example that one was from the Sikha Gadgil paper that example of the cloud band and as I said the cyclonic vorticity pictures and so on were from Sikha's paper in 77. Now, this is another example you see the ITCG very good band coherent band here and let us see what is the vorticity of this thing. So, this is the from the IMD charts know this is at 850 HPA and you see a cyclonic vorticity where the cloud blob was and you see that it is also occurring at 700 HPA. So, the 700 HPA trough would also be over this region and 850 HPA trough would also be over this region. So, we have seen that on the daily scale if we look at the picture of a intense cloud band associated with the rainfall over the monsoon then it is also associated with the high cyclonic vorticity above the boundary layer and we see that the trough is almost vertical up to 500 HPA and it is associated of course with moist convection and heavy precipitation as is evident from the nature of the clouds it is a maximum cloudiness zone. Now, that was the daily scale let us see what happens in the July mean patterns also. Now, let us look at here we see relative vorticity at 850 HPA this is above the boundary layer at 1.5 kilometers all the blues are negative. So, this means anti cyclonic vorticity these yellows and reds yellows and oranges rather are all positive relative vorticity and you see that as far as 850 millibar vorticity is concerned over the entire monsoon zone the relative vorticity is positive and this is omega or the vertical velocity again what we have done is shown that the vertically upward velocity is the same colors this is orange and yellow and you see that at 850 HPA it is actually upward everywhere. Notice one more thing you know we had said when we looked at boundary layers in rotating system that if the vorticity is cyclonic above the boundary layer then you have upward velocity at the upper edge of the boundary layer if it is anti cyclonic it is downward. Now, notice that there is a blue blob here and here a very strong blue blob here this is over southeast coast of the peninsula around Chennai and so on during July mean you see a blob here and this means there is negative vorticity here or anti cyclonic vorticity and correspondingly you have descending air here into the boundary layer. So, in a way this confirms what our Ekman boundary layer theory had shown that if you have cyclonic vorticity only then you will get a scent of air if you have anti cyclonic vorticity the air will descend. So, this is the July picture now. So, now we look at what happens on a daily scale where is the axis of the MCG situated and they showed in fact that the axis of the MCG which is associated with deep moist convection in fact coincides with the 700 millibar trough. So, this means not only is it do we have cyclonic vorticity above the boundary layer further more at 700 millibar trough which in fact delineates the region of heavy rainfall axis of heavy rainfall region actually coincides with the axis of MCG. So, the structure of the dynamical system responsible for the organized rainfall over the Indian monsoon is similar in its critical features to the ITCG discussed by Chani. This is the conclusion we have come to. Now, how do we define the maximum cloud zone? They define maximum cloud zone as that cloud band which has the maximum brightness is predominantly zonal meaning in east-west direction and has a longitudinal extent of at least 10 degrees. So, this is their classic picture of course, this is the band and you can say by and large it is in the east-west direction it is zonal and it has it is coherent all the way from this is about 60 degrees east to beyond 100 degrees east. So, it is coherent over more than 40 degrees of longitude. So, it satisfies the condition they had that it has a longitudinal extent of at least 10 degrees. Now, they noted that on some occasions the MCG comprises cloud clusters linked by regions of less intense cloud brightness while on others it is equally bright at all longitudes at which it occurs. So, it is not necessary that the band is equally intense everywhere. In fact, this is a case where you have clusters of intense clouds joined by relatively less clouding in between, but they it is very clearly a coherent cloud band. So, then they documented the daily variation of the MCG during April and October and what they did was they took daily values of the latitudes of the northern limit the axis and the southern limit of the MCG at 70, 80 and 90. So, here we are the same picture. Now, this is 70 degrees east here. This is the longitude of 70 degrees I do hope you can see India quite clearly and you can see Sri Lanka there. So, that will be a good way to see where India is and 80 degrees is sort of considered the central longitude of India this is 80 degrees and this is 90 degrees the one that passes through head bay ok. So, at these 3 longitudes 70 degrees, 80 degrees and 90 degrees what did they read from imagery like this they would read what is for example, this is the southern limit of the MCG this is the northern limit of the MCG and they would be an axis in between. So, these 3 things were read off at 70, 80 and 90 from imagery of every day ok. So, daily values of the latitude of the northern limit the axis and the southern limit of the MCG at 70, 80 and 90 were read off from the cloud mosaics. They estimated that the expected error in their reading would be about 1 degree because this was a fairly large pictures good satellite imagery ok. The latitudinal position of 700 millibar trough at these longitudes they got from the daily weather charts at IMD. So, this is the basic data source then first of all the satellite imagery which gives the maximum cloudiness zone and then daily weather charts from which 700 millibar trough position was read off. Now, the first thing we want to do is how frequently does the MCG occur at different latitudes ok. So, the monthly averages we take of over 73 to 77 of the number of days on which the axis of the MCG occurred at different longitudes at 80 and 90. So, that is what we want to see here and this is the picture now. This is going all the way from April here, May, June, July, August, September, October. This is 80 degrees and this is 80 degrees here and this is 90 degrees on the right ok. So, now what we have plotted is number of days on which the axis occurred within a certain latitude. Now, this latitude is here, it goes only from 0 to about 30 north also ok, 0, 10, 20 and 30 north. Now, why does it start only from 0? In fact, the imagery that Sikha Gadgil had was that of the northern hemisphere only. So, the data began from 0 and went northward. So, data presented for the equatorial region is only for the part including a north of the equator. This is why we see here the latitude begins with 0 here. Later on when we look at what happens with OLR data, digital OLR data which became available much later then we will see that actually this equatorial band extends also to southern hemispheric equatorial region 0 to 10 south or so, but that we will see later. So, what they have done is plotted here and what you see? This distribution suggested the presence of two MCGs, one over the equatorial region and the other over the heated subcontinent. Let us see if that is suggested to you, you see the distribution first of July and August here. Now, you see very clearly that there are two modes, one is over the equatorial region and the other peak is here over land from 15 degrees and it is a fairly spread out mode here. So, you have, but there is a region here between 7 and 13 or so, where the chance of occurrence is very very low. So, you have two modes here, this is over the equatorial Indian Ocean and this is over the heated subcontinent in the north and you see a similar thing in August also. In September also you see it, but here at 80 degrees this looks more like a secondary and this looks more like a primary mode and that is also true here, but July, August the bimodal nature comes out very very clearly that you have one mode over the equator and another mode over the northern part here. Now, what have, how does this come about? In April actually most of the clouding is between 0 and 10. Most of the time MCG is over the equatorial region and this is more or less a unimodal distribution with a maximum somewhere in between. Now, this pattern will not be exactly same when we include data from southern hemisphere, but it will be very similar. May also it is unimodal, you have a lot of high number of cloudy days here and then it decreases as you go northward and in May already at 90 degrees you notice that the mode has shifted a bit from the equator region, but interestingly it is still unimodal. But come June it be you see it has begun to be bimodal here in June and that bimodality persists here. So, what is interesting is that it is a bimodal distribution and as I mentioned here in April and May the distribution is unimodal and the MCG fluctuates between 0 and 10 north. During June to September while the primary MCG is to the north of about 13 the secondary MCG persists over the equatorial region. In July and August the northern MCG fluctuates primarily in the region 15 to 25 north within which the monsoon trough is known to fluctuate now monsoon trough fluctuations have been documented for a very long time. So, it is well known that it fluctuates between 15 and 25 and you can see that this is about 15 here and this is 25 and that is where most of the fluctuations are occurring in July and August. So, in July and August the northern MCG fluctuates primarily where the monsoon trough is also known to fluctuate. Now since the frequency of occurrence is low from June to September in 7 to 13 degrees and we have seen this before here see it is very low here and low here what SG did was to look at where the frequency of occurrence is minimum and took that as the latitude of separation between what is considered a primary MCG and a secondary MCG. The primary one being the one which has a very high frequency of number of cloudy days whereas the secondary one has relatively lower, but there are clearly two distinct modes. So, this is how they took this the boundary of the two primary and secondary MCG were determined by taking where it was the lowest probability of having. So, since the frequency of occurrence is low from June to September in 7 to 13 SG took the latitudes of separation of the primary MCG from the equatorial one as 7 degrees in June, 11 degrees in July and August and 13 degrees in September. This was done simply by looking at the distribution. So, what did we learn then that in fact the distribution is bimodal there is a primary MCG which occurs north of about 15 or so and a secondary MCG which is the equatorial MCG. Now since we have separated the thing into two bands the primary and the secondary we can look at the variation of mean monthly location of each of those and that is what we see in the next slide. See this is the variation of the primary MCG that we looked at and this is the variation of the secondary MCG you have to remember in April and May there is only one MCG. So, this itself is the equatorial MCG which if you like continues as the equatorial MCG here in some sense, but what you can see very clearly is that this in fact we notice that in May itself at 90 degrees the MCG had begun to move northward. So, you see it in May and that continues. So, what you see here is if you focus on the primary band primary MCG then it moves northward up to July is more or less in the same location till August and after August it starts retreating southward. So, this if you like is the onset phase of the monsoon where it is going northward up to July and September onwards is the retreat phase of the monsoon. So, what you see here is a very clear seasonal migration of the band. Now what Sikhan Gadges did was they showed both the things one was at 90 which is the solid line and the dashed line is 80 and you can see that the movements are very very similar except as I mentioned that in May the one over Bay of Bengal seems to move northward before the one over the Indian region. So, this is the seasonal migration that you have seen. So, seasonal migration of the primary MCG is clearly seen in the variation of the monthly location of the axis. This means that Sikhan Gadges study has lent support to the hypothesis that the monsoon is a manifestation of the seasonal migration of the ITCC or equatorial trough in response to the seasonal variation in the solar radiation. So, this is what they have shown. Now all this was done with analysis of satellite imagery seeing where the band is by eye noting down where it is and doing the analysis. Now since 79 at that time we did not have at our disposal any digital data. From 79 now digital data on outgoing long wave radiation have become available and as you know low outgoing long wave radiation is a way to trace where the brightest clouds were in the visible imagery that we looked at where the deepest convection is occurring. So, just like we ask the question where does the MCG occur we can ask the question the number of days with low ALR at a particular latitude how many are they. So, we take a fairly stringent criteria of ALR less than 180 watts per meter square for June to September and what we see is here. Now, this is June and this is again you can see here this is latitude and this is average number of days. You see the bimodality spectacular actually the bimodality is very good in July August and September as you see, but in July and August the primary mode is definitely over the heated subcontinent by September and this is only at 80 degrees and that is why it is more. So, by September they become more or less even Stevens like they were in June, but July August the primary mode definitely dominates. So, the basic result that Sikha Gadgil obtained by looking at satellite imagery and plotting the frequency distribution of the occurrence of MCG at different latitudes is very similar to the result we now get with modern data with ALR less than 180 watts per meter square. When we ask what is the number of days on which at this latitude such a band occurs with lower ALR less than 180 watts per meter square. Now, so this means that although the Sikha Gadgil result was with images of clouds rather than digitized ALR data it has been proved to be valid even when we looked at digital data as far as the bimodal distribution is concerned as far as the primary and the secondary MCG that they found were concerned. So, what we have done so far is the following. We have asked the question what are the important dynamical characteristics of the ITCG listed them and shown that in fact the maximum cloudiness zone that we see by satellites which looks very much like an ITCG actually satisfies also the dynamical constraints dynamical characteristics associated with the ITCG. So, we saw that on the daily scale at least the ITCG does resemble the dynamical characteristics of the MCG are the same as those of the ITCG and so we can attribute monsoon rainfall to an ITCG type of dynamical system this we have seen. Now, this is all ok, but then what happens to Murakami's objection that you know the low ALR region is so broad why does that occur when we look at the monthly scale. Now, so relative to the same you know low ALR regions over at same longitudes over January or over Pacific and so on this is the one see just to remind you this is the low ALR now from more modern data than Murakami's and this is the winter situation you can see that over the Indian longitudes it is a very narrow region it is about 10 degrees in latitude, but when it comes to the monsoon it becomes so spread out here huge as compared to say the Pacific one or the Atlantic one. So, this is what Murakami's objection was and question is if it is the ITCG why why does it look so different when we look at monthly scales. Now, to understand this we have to look at few more results of the Sikha Ghatgir study and that is their detailed analysis of the daily variation of the location of these three parameters remember X is the northern limit and southern limit of the two MCGs and the 700 HPA trough at 90 degrees which you will see here this is from their paper this is 1977 1976 75 74 and 73 so 5 years are involved it is all at 90 degrees east this is the latitude going north for each of the years and these are the days April May June July up to October and what you see here is actually those three things and I will show simpler pictures of the same phenomena later, but it is always good to see first you know what the what the real picture is like and this one here is the 700 millibar trough and this is when these are the two limits of the MCG. So, what is the most prominent feature the most prominent feature of this is these northward propagations northward movements of the band you can see them here for example, this is in May and part of June you see this is the beginning of June and this is when the MCG has got established over the monsoon zone. So, this is the northward progression in the onset phase, but in addition to that you will also see other northward progression this is always typically occurs towards the last week of July and you see that also year after year. So, there are lot large number of northward progressions the first can occur even before the onset see here you can have a phase here in May for example, before the onset phase of the monsoon. So, there is a series of northward progression the time scale between these two varies from anywhere between 2 weeks to 4 weeks or 6 weeks, but the common element of this very complicated behavior is that bands tend to form in the equatorial region and move northward northward up to the monsoon zone and then hang around there this is the common feature. So, MCG is not present every day either over the equatorial or the northern region rather at any given longitude it appears at a particular latitude persists for a certain length of time with fluctuations in its position and then disappears. Each such spell is called an MCG epoch with given data and latitude of origin. So, let us go back to this what we are saying is if you look at any latitude or time you do not see suppose you look at this latitude here 15 degrees or whatever then you do not see MCG there present every day even near the monsoon zone around 20 north you will see MCG present on quite a few days, but not every day by any means. So, it appears and disappears that is the nature of the beast these cloud bands appear and disappear they appear at some latitude then they occur for a certain length of time and they may not move northward like in this case it appeared at this latitude and then died after 3 days here it appeared and died and then it appeared here and moved north. So, this is considered an epoch MCG epoch with a latitude of origin given here time of origin is the date here and in this case we would say latitude of origin is that wherever it was and time of origin we note and this is a case in which it moved northward you see that. So, this is a northward moving epoch and then its lifespan is actually just the time between its demise and birth. So, you can see that the best way to represent variation of this kind is by saying that at every longitude like 90 degrees east there are a series of MCG epochs which are born at a certain latitude on a certain date which die at a certain latitude at on a certain date. So, that determines both the birth latitude as well as the lifespan and within their lifespan they may also move northward like here very big northward movement here. So, you can think of the variation as comprising several MCG epochs which are characterized by latitude of birth date of birth date of demise and also whether they moved northward or not. So, because it is not something that is present day after day we should represent it in this way and each such spell we call an MCG epoch with a given date and latitude of original lifespan. Now, the variation of the MCG at any longitude is then described in terms of this unit as successive occurrences of MCG epochs with specific life history parameters. So, the most conspicuous feature of the variation as I pointed out is the series of northward propagations of the MCG and these are seen to occur every year. Now, this is a simpler picture of the MCG than the original Sikha Gargir picture what I have shown here is merely the northern limit and the southern limit of the MCG and between a hatch region showing where the cloudy zone is. Now, this is somewhat easier to look at and what you see here is again very nice northward propagations here one here one and yet another one here and not this is again at 90 degrees not that they occur year after year 75, 74, 73. So, it is a basic feature of the variation of the MCG over the Indian rangitudes that these propagations occur year after year it is a basic feature of the variation and further more they occur irrespective of whether it is a drought or a good monsoon season 74 was a severe drought 75 was a very good monsoon season even then in that good monsoon season it is not as if MCG hung around over the monsoons on every day no in fact it disappeared in between you see it disappeared here and it appeared again it disappeared here and then came an northward movement. So, even in good monsoon years you do have disappearances of the MCG again revival and you can actually see and I will preempt what will come in later lectures that the revival can occur either by generation within the same latitudinal band as here or by northward movement of the oceanic band as here. So, you see those two here here also you can see this is the revival by what we call in situ generation and this is the revival by northward propagation, but the important thing to see say see is that these northward propagation are basic to the variation and now here you see 75, but at 3 longitudes 90, 80 and 70 which we call the Indian longitudes and what you see is that these propagations that we saw are rather coherent across the longitudes see here this is the propagation here and this is the same propagation here. So, the propagations tend to be rather coherent what does that mean that means that the entire band is moving from extending from 70 to 90 is moving northward all the way from the equatorial region to north of 20 north also. So, this northward movement occurs either as a northward shift of the MCG or by generation of large cloud clusters in the region to the north of the MCG. So, what happens is either it moves as a band, but more often than not in fact what happens is to the north of the MCG some cloud clusters get generated one over the Arabian sea one over the Bay of Bengal and then it forms a continuous band there and the older MCG dies that is how it moves northward. The presence or absence of significant northward movement within the lifespan of an epoch is included as another feature of the MCG epoch. So, in addition to latitude of origin and latitude of birth and lifespan we have one more feature which we have used in the analysis. Now, again we have to see all this analysis was done by looking at the MCG from satellite imagery. We have to see if it is what we get from digital data which are available from 79 show this important feature discovered in this study namely northward propagation. So, what we will see is again now we plot instead of the maximum cloudiness zone it is the low OLR belt. It is the belt with OLR less than 180 and what you see this is again 70, 80 and 90 and this is for May to September 2007 and what you see here is again rather coherent northward propagation occurring across the region 70, 80 and 90. So, what we saw with the satellite imagery analysis of the satellite imagery we have also seen with the digital digitized OLR data. Now, since in the tropics high rainfall occurs from deep clouds with high tops OLR is used as proxy for rainfall with low values of OLR being associated with high rainfall. But now rainfall based on microwave measurements from satellite is also available. So, let us look at how the rain bands look if we use rainfall data from satellites and what we see is even more beautiful picture this is from Srinivasan's work and what you see this is at 90 degrees east and this is June to September 86 and what you see is really spectacular northward propagation you see this is the secondary ITCC. And again northward propagation notice it is good that the Sikha Gargay study detected the secondary ITCC, but you can see that quite often it occurs in the southern hemispheric equatorial region rather than in the north, but there are enough occurrences in the north. So, they could also see it and you see a very similar phenomena to what we saw earlier northward propagation then it hangs around here for a long time the MCG hangs around the monsoon zone disappears another northward propagation revives another northward propagation and so on and so forth. So, this is really the typical variation of the MCG or the ITCC whatever you wish to call it over the Indian region. So, now how does the large latitude and the extent occur it is very easy to explain it once we see this. First of all there are two locations right there is the primary MCG and the secondary MCG and so the distribution will be bimodal. But if it was a story that the primary MCG you know fluctuated within the monsoon zone and the equatorial MCG fluctuated within the equatorial region you would see still two bands of low oiler with nothing in between. But what you see is almost a continuous belt of low oiler which is what Murakami is talking about. Now, that occurs because it is not as if the equatorial ITCC or the MCG stays put in the equatorial region no. In fact, very often it moves northward on to the monsoon zone. So, it is the northward propagation that give rise to the occurrence of low oiler region in between the locations of the primary and the secondary zone. So, this is how the two features then because there is a series of northward propagation of the cloud bands from the equatorial ocean on to the Indian region which occur year after year irrespective of whether it is a good monsoon or a put drought. Because of these two features then what we will get is a large extent latitudinal extent of low oiler region on monthly and seasonal scale. You very seldom see it on the daily scale though. On the daily scale either you see distinct cloud bands one or two distinct cloud bands or none at all. But you never see a band stretching all the way from the equatorial region to the monsoon zone. So, that low oiler region that is so large in latitudinal extent arises because the band moves around so much. So, I think with this then we have given enough evidence to show that the monsoon is a manifestation of the seasonal migration of the ITCC over the on to the Indian monsoon zone in response to the seasonal variation of the incoming solar radiation. So, this is now where our understanding stands. There is something to now clarify because there is some confusion in literature about the term ITCC itself. Remember ITCC stands for intertropical convergence zone or where there is confluence of winds from the northern hemisphere and the southern hemisphere that is where the word intertropical comes from ok. So, while I have been denoting by ITCC the dynamical system that Charney talked about or that real described as equatorial trough in the sense of being a large scale convergence zone with intense precipitation. It is the same term is also used to denote a region of surface confluence in which winds from the two hemispheres converge ok. So, if winds from the northern and southern hemisphere converge on to a zone that is also referred to as ITCC even if it is not associated with intense rainfall large scale convergence and so on at higher levels. Now, such a confluence zone is not always a zone of organized moist convection also confluence of air from the two hemispheres is by no means a necessary condition for the occurrence of the dynamical system. So, happens that if the air happens to be from two hemispheres and is converging and all the other conditions are favorable it can become an ITCC of the kind we are discussing, but it does not have to be from two hemispheres if you have large scale convergence generated by cyclonic vorticity above the boundary layer even in the northern hemisphere northern hemispheric air itself converging can lead to a tropical convergence zone. So, in fact it is very clear that there are convergence in that kind of a dynamical system does not have to be intertropical and also we have to note that we have seen that over the Indian longitudes in the summer there are two such convergence zones over the same longitudes. So, if there are two convergence zones over the same longitudes obviously the convergence in both of them cannot be intertropical. So, it is a good idea not to harp too much on the intertropical instead of that I take the distinguish distinguishing attributes of the system to be the large convergence and organized precipitation and refer to this zone as a tropical convergence zone that is TCG. Now, we have seen that in the Indian monsoon is a manifestation of the seasonal migration of the TCG which occurs over the equatorial Indian ocean in spring on to the heated subcontinent. So, I will refer to this as TCG and thus the basic system responsible for the monsoon rainfall is the TCG which is the same as that which is responsible for the large scale rainfall over other tropical regions such as the Pacific. However, amplitude of the seasonal migration of the system is larger over the monsoonal region than over the oceanic region. So, we call this the large scale rainfall associated with the Indian monsoon which is Indian summer monsoon is associated with the TCG over the Indian region this TCG remember occurs on the Indian subcontinent. So, we call this TCG a continental TCG as it occurs over land to distinguish it from the more common TCG observed over the Atlantic and Pacific oceans as you have seen. So, this is what we call continental tropical convergence zone or CTCG and in fact, there is going to be a major there is a major program of the Indian climate research program which is focusing on understanding the processes that lead to the variability of CTCG and hence the variability of monsoon rainfall. So, CTCG is the creature is the beast that is of greatest interest to us if we are interested in large scale monsoon rainfall. It is a system with the dynamical characteristics of an ITCG, but which lies on land and that is what we will have to look at. Now, I just want to end with showing one more confusion in terminology which we should avoid that while the ITCG or TCG is associated with the trough at low levels it must be noted that a trough or a low pressure at the surface and cyclonic vorticity at 850 HPA are not always associated with moist convection and rainfall. In fact, in a trough in the surface pressure could be associated either with an ITCG or a heat low and what is the difference between a heat low and a tropical convergence zone in a heat low the there is also convergence in the heat low at the near the surface and this convergence persists till the boundary layer till about 2 kilometers or so from the surface, but above that there is no convergence. In fact, it reaches about 2 kilometers or so and then the air diverges and sinks. So, it is a very shallow cell with convergence restricted to about 2 kilometers, divergence above this level and subsidence throughout the lower 2 kilometers away from this loop. So, this is a shallow vertical cell. On the other hand, so this is what I have emphasized here that a heat low is characterized by ascent at the upper edge of the boundary layer and of course, 850 millibar vorticity being cyclonic. So, that it is consistent to have ascent at the upper edge, but only up to height about less than 3 kilometers. Therefore, at 700 HPA and above air is subsiding over a heat low and that is why there is no trough there is no low at 700 HPA above the heat low. This is why this 700 was a very critical level to look at the trough for the MCG study of Sikha and Gagya. So, above 700 HPA the air is subsiding at 700 HPA as well above a heat low and there is no rainfall. So, the circulation characterizing a heat low is a shallow cell with the air converging in the boundary layer ascending only up to 2 kilometers. Now, on the other hand the TCG is a very deep cell you have seen TCG or ITCG is the ascending limb of the Hadley cell and that is a very deep cell in which low level convergence leads to ascent almost throughout the entire troposphere and only at the upper tropospheric levels the air diverges and then sinks in the region around. So, there is very deep ascent of moist air and that results in intense clouding and rainfall. So, the TCG is the ascending limb of a deep cell, but what occurs over a heat low is a very shallow cell these are the differences. Now, heat low and dynamical low occurring side by side actually happens quite often you look at our July picture this is the mean pressure pattern for July and this is the low low pressure region the trough of which the lowest is here this is the this is the low that you see here. Now, in fact this is a heat low and this is what we call a dynamic low in contra distinction to a heat low. So, over this region you have only a shallow cell this is the heat low with the shallow cell and this is the one that has a characteristic of a TCG and you see this is the rainfall and you can see that over the region of the heat low which is here there is no rain at all and the rain occurs over this region primarily which is the trough here, but now you can see the same thing here this is the July 850 MB this is from the IMD Atlas and this is the entire low pressure region here and the lowest of that pressure is the heat low here, but if you go to 700 millibar which is 3 kilometers then the only low you see is over the dynamic low. The heat low does not appear at 700 at all it is descent is taking place here over the heat low and the low is only the dynamic low and this is again from the re-analysis this is the 850 where you see a clear low here and at 700 the low is entirely over the where it rains and these are streamlines this is July at 850 and you can see that at 850 you have a very clear cyclonic vorticity from of the west is here and the east is to the north but at 700 you see the circulation is entirely different and there is no cyclonic vorticity above the heat low. So, this is the difference and now what you can see here is from re-analysis this is showing vertical velocity again upward is oranges and so on and this is the vertical velocity at 850. So, this is just at the upper edge of the boundary layer and you find over almost the entire region the velocity is upward this is because over almost the entire region the vorticity is cyclonic, but you see here this is this was that hole that southeast peninsula hole in which it is sinking because there is negative vorticity, but if you go to 700 millibar then you see that over the heat low there is descent and rest of the region there is a sink. So, this is the dynamic low and this is the heat low corresponding to the Rajasthan desert and that is very clearly seen here this is the omega at 500 millibar or the vertical velocity at 500 millibar and vertical velocity at I am sorry this is the vertical velocity at 850 millibar which is upward everywhere, but you see the rainfall there is no rainfall over heat low even though the vertical velocity is upward at the edge of the boundary layer. So, I am going to stop here except to say that for seasonal variation of rainfall only the seasonal variation of the dynamic low or t c g is important. However, the heat low also contributes to the seasonal variation of the surface winds and now just another thing that we have seen that the large scale rainfall over the Indian region during the summer monsoon is associated with a c t c g over the region, but somehow the term monsoon trough has been extensively used for the moist convective system referred to here as c t c g. However, while we use the term c t c g to denote moist convective regime since the term monsoon trough is also often used for the entire surface trough comprising the heat low as well. So, there is a big confusion because monsoon trough very often this is from a very big review by Ding and Sikha and they say the eastern end of the monsoon trough is locked with the warm waters of the northern bay with dominantly moist processes operating the western end is situated in the predominantly dry convective area of the western India and Pakistan. So, this is the point the western end of the monsoon trough is a heat low. So, calling the same system as monsoon trough does create a lot of problems. Now, I am going to stop here and next we will consider the nature of variation of the c t c g. Thank you.


So it has taken me the best part of 500 eggs 500 attempts to try and get this perfect pal and I finally achieved it guys. In this video I'm going to showcase to you how you get this incredible OP dark pal known as Rostalia nut with the best passives this secret skill everything you need to know. How's it going people? My name is DPJ and before we go any further I'm giving away a couple of copies of this game or any game of your choice. To be the chance of winning simply drop a like on the video leave a comment down below make sure you are subbed and I will pick winners from the comment section and announce them Sunday evening good luck everybody. So I've already uploaded a video on the best shadow beak you can get possibly in the game with this amazing dark wisp hidden skill. In today's video guys I'll show you how you can get this exact setup on Rostalia nut. All the same passives the same amazing skill and how you breed it to this point. So let's start and firstly guys you actually need to get that perfect build of the shadow beak this is an absolute must this means we need a shadow beak with ferocious musclehead legend and lord of the underworld in regards to its passives but we also need to have that skill of dark wisp too. The process is long but it's definitely worth it just to even get shadow beak never mind Rostalia nut. So I'll drop a bit of footage from a video I already made which gives you the best rundown of how you get this incredible shadow beak which actually starts with us having to catch the tower boss so check out the footage from my older video on the screen now. But firstly we need to use that tower boss glitch to go and catch Victor and shadow beak.  Now as far as I'm aware this is the only place you can get this there's no way else in the game this drops it's only available on this boss. Firstly you want to catch this tower boss and if you don't know how this is done it's quite simple. You first need to attack a friendly NPC so you get that wanted status on your screen. This can be done anywhere where they are those friendly NPCs but I recommend you head into the small settlement found right here on the map. Once you load in here guys attack any of these NPCs and then simply just fast travel to that tower boss seeing where I go right here on the map.   Once they do guys 9 times out of 10 the tower boss glitches out and runs into the corner. From here guys you simply run up to it and throw a sphere at it. It doesn't matter what sphere it is it will always come up with a 0% capture rate but you'll capture it either or so it doesn't matter. So throw the sphere at Victor and Shadow Beats back I think this may be needed and then you have this in your collection. Again they come with this dark wisp ability which is incredible. Now that we do have it guys it's important to note when it comes to breeding tower bosses every combo you have seen on those online calculators is false besides one calculation. If you breed said tower boss with same said pal they will result in being the same said pal. For instance here if we breed victor and shadow beak with a normal shadow beak you get shadow beak. Any other form of breeding will result in I believe a chicka-pea. Which by the way I do recommend you do and after this setup is done because that dark wisp skill can pass down so we can indeed use this on chicka-pea at a later date and possibly add it on to other pals if needs be. But yeah breeding victor and shadow beak with another shadow beak will result in a shadow beak egg but it doesn't end there guys. The skill isn't a guarantee so numerous breeding between the two will possibly be needed unless you are super lucky and get it first time. But wait before you even do that guys before you breed these two shadow beaks together you want to make sure you have the best shadow beak first with the ultimate passives to make sure that this skill when it lands is as deadly as possible. This means in my opinion having a shadow beak with a passives of musclehead ferocious legend and lord of the underworld. What these do are as follows musclehead will give you a 30% increase to that attack power yes it takes away from work speed but that isn't needed here. We then want ferocious this gives you a further 20% attack damage. We then want legend which gives you a further 20% increase to that attack and a 20% increase to defense as well as a 15% increase to movement speed and then lastly lord of the underworld will give you a further 20% increase to that dark damage attack so yes these are the four passives you must have but this is where it may get a little confusing. The lord of the underworld passive only comes from necromus the legendary world boss there is no other way in getting this but necromus also comes with legend 2 that's two of the four passives we need straight away which is definitely a bonus when it comes to passing these on via breeding as you probably well know. So we will eventually need a necromus within our arsenal to even get this set up but firstly we worry about the other two passives first which actually can come on any power these are ferocious and musclehead now we need a mating pair of powers which have these passives on them now for me the fastest route to the offspring we want would be one parent with one passive each problem is passive seem really random and although they do and can pass down getting what you want exactly is harder than you may think. Okay so what we are looking for here guys is parents we need parents with one of these passives each the best route in dunes that I could find is a sparky and a relaxaurus so yes firstly you want to go and grab that sparky yes the little electric fellas easy to find easy to catch get as many as you can until you get one with either ferocious or musclehead on them or if you're lucky you may land one with both passives on well yeah that's extremely lucky once you've achieved this guys you want to go and get yourself a relaxaurus again you want to look for the opposite sex obviously and one which also has the passive and you need in regards to musclehead or ferocious once achieved guys read these together until you get a relaxaurus luxe with these two passives on it shouldn't take you long but if you can you want to try and land yourself a relaxaurus luxe with these two passives on it and nothing else maybe a long shot but if you've got the spare cakes you might as well try this now from here guys you need to get yourself a necromus if you don't have it already now this world boss is a part of a pair but it's only him we need to catch at this moment again he has the other two passives on him as standard legend and lord of the underworld two very very important passives to this setup okay so once you have this necromus guys you now want to breed these two together to relaxaurus luxe and the necromus this guarantees you an aster gun baby now breed these until you get the aster gun with all four passives on it no excuses here guys you need all four passives on the aster gun it's an absolute must now you have this guys i mean you can always try and breed a second of the opposite sex so you have a main pair with the same four passives that's perfect but yeah once you have the aster gun with all four passes on we now need to pass those passes onto a shadow beak and the only way to do this guys is to breed aster gun with a kitsun now the kitsun luckily enough isn't easy breed the parents of grizzbolts and lamble guarantee you a kitsun jolt hog and relaxaurus guarantee you a kitsun gob thin and surfend guarantee you a kitsun rush here and pinking guarantee you a kitsun lift monk and beacon also guarantee you a kitsun there are many many combinations you can use here but you need to get yourself a kitsun now once achieved people now you want to continuously breed that kitsun we just made with that aster gun until you get a shadow beak baby with all four passives eventually it will happen but it's an absolute must and don't worry about the other shadow beaks people that born and are useless because we'll use them for infusion material later on but eventually guys you will get that shadow beak with said four passives this again is an absolute must for this setup now when you eventually do you have the perfect shadow beak to now breed with victor and shadow beak to now hopefully get that baby shadow beak with all four passives and the dark wisp skill keep in mind if the dark wisp skill isn't there when that baby shadow beak is born i don't think you will ever learn it so keep reading away until you get this it's definitely worth it people and eventually you will get that shadow beak with all four passives of lord of the underworld legend musclehead ferocious and the dark wisp skill okay so back to frostalian nut so now you should have that shadow beak with the perfect rolls or the passives and that skill but we now need to pass all these onto the house sefa which means breeding our perfect build shadow beak with one of the following pals hoping to pass on all the passives and this hidden skill of shadow wisp so you can breed shadow beak with a following and you'll get a house sefa warsect elizabeth mama rest or mama rest christ reptile germantide or germantide igneous these guys you can burn with that shadow beak and it's a guaranteed house sefa baby now for me personally when it comes to me creating my shadow beak i made sure i bred both male and female the perfect breeding pair having all passives and a skill because i knew eventually something like this will come up but yeah guys take your pick of pals to breed a shadow beak what i would say is you want to go for the one that has the least passives attached to it i mean like i said it can be warsect elizabeth mama rest mama rest christ reptile germantide germantide igneous any of these guys you can breed with shadow beak and it guarantees you that house sefa but again you want to look for the one the one you've got or go out and get one catch one which has the least passives on it therefore when it comes to breeding it with shadow beak it means more times than not the baby's going to result in having more shadow beaks passive which is what we want here we want to pass on shadow beak and all his passives and that skill until that house sefa baby so once you do have this house sefa baby guys now guys you just breed this continuously with frost stallion yes the legendary world boss of frost stallion breeding these two together guarantees you that frost stallion not baby but obviously you want that frost stallion not baby to have all of house sefa's passives and that dark wisp skill you don't want it passing on anything from frost stallion obviously apart from legend which is something we already have now on house sefa so yeah guys do this until you get this frost stallion not i mean it will take you a while unless you're super super lucky i'm thinking probably a hundred eggs plus but that's okay because you'll end up using these hundred extra and spare frost stallion knocks in the condensation machine to make your now final frost stallion not the best version of it possible and don't forget also guys to use the statue power as well to level up that frost stallion not even further people and don't forget guys for even further damage on your frost stallion not to any other dark power use those who creates within your party the more the merrier and the more you have the more level these are the more damage output they will add on to the attack damage of your frost stallion not and shadow big so keep that in mind two people but yes they say get the best version of shadow big and frost stallion knocked in the entire game guys if you enjoyed the video leaving a like really helps out feel like what you're seeing once you more be sure to subscribe and hopefully guys i will see you on that next one


Hello, hello Hello All right. Hello everyone. My name is Flav Leitner. I work for the network services team and today I would like to talk about the challenges to provide networking for NFV environments Just for out of my curiosity how many of you know what NFV is all about Okay, half I would say so okay good excellent I will do some quick introduction moving from one step to another trying to Introduce to the idea the problems the challenges, so you will see that I will deep dive in the end into a specific solution But to get there I need to introduce some concepts and ideas So the agenda is concepts and The ideas for NFV what NFV is all about what's the goal what we want from it and Then we are looking for the requirements After that we are going to look quickly about the 10g networks see what the challenges are there how it works Then I'm going to talk about the physical to virtual to physical the so-called PVP scenario Which is the the basic before building the NFV Then some possible network solutions, especially the ones that we are more used to they are more commonly available But there are others so I'm not covering everything here all the possible solutions especially the newest ones But we can certainly talk about after the talk or even during the Q&A session Then we will dive deep into DP DK enable open the switch Because that's the the solution that we are working mostly with it right now I will try to explain how it works internally and not going to be it's invites But giving you an overview on the how it moves packets around and then possible improvements That we think it would Help to improve the current limitations This possible improvements. I'm not covering everything possible the most probably as you you know you fix one bottleneck you most probably will find another one and There are other work around solutions or solutions that they don't Have the same bottleneck so and there are efforts in upstream to resolve for other solutions as well But that will give you I hope that you'll after the stock you learn the difficulties for virtualizing network not only for NFV, but for other user cases as well Okay, so what is NFV all about NFV stands for network pollution virtualization and what we are trying to do is actually to move from the Physical device, you know, you have a firewall. It's usually this fire will be a physical device It has a custom software. It has a custom right hardware usually very specialized the hardware Accelerated higher so asics and supports one You have load balancer, it's another physical box you have the routers or depending on the region you are from routers And what do we want is we want to have this appliances Running inside of a virtual machine So this step here is actually we are decoupling software and hardware again. The same happened with the Other things as the end for instance is decoupling the control plane from The data plane here we are decoupling the software and the hardware And the idea is that Running these appliances on the virtual machine. We add this virtualization layer obviously running on top of a Also, so-called white box this is a It the idea is that it would be a commodity hardware. So nothing is specifically not you don't need to match these hardware with the specific appliances you are running it and you You in theory you should be able to run multiple appliances Regardless of the the function itself on the the same hardware So that's basically the idea So what's the goal most of you that's how we operate today you if you want to have a new Networking in the company or whatever You need to buy an equipment you need to Order something you need to talk with the hardware vendor need to negotiate prices Then you buy something maybe they have in stock or maybe they don't you have to wait to then a truck Stop by someone has to go deploy inside of your data center. Maybe you need more space So it's a slow process until it gets delivered and up and running It also has a high cost and Less flexibility by flexibility I mean you cannot automate most of this process after that so assuming that we have Succeeded to virtualize these appliances Then you have a fast process because if you want something you can just start a virtual machine It obviously should have lower cost because most probably you have the hardware already there you don't need to Have someone to come by and what everything look for more to space, you know buy more hacks or You already have a box there that's capable of growing as you need so you have a spare room You can just start new VMs and Obviously with the VMs with you plan you saw Some presentations about Open stack and how is how is he's becoming to manage a large number of virtual machines? So we will also need a way to manage This network of appliances as well So if we succeed if you need something like load balancer to how there's some pharaohs and etc You can just click a button and suddenly your devices will be there and what are the requirements? Well in an ideal world, we would like to have this appliance if you utilize it and don't give up on anything I mean we all love performance right so but yeah this This layer here impacts a lot So it has a cost and also the hardware is not that Heavily optimized it easy that's running there So we need to make some you know Pros and cons here they might not perform at the same level, but maybe for most use cases what we have today is good enough for instance, I was asking if for the Today we have Virtualization and some of our products. I ask around and they for some use cases They really don't care about the performance because today what is provided is good enough but for an FEV Let's see So the challenge you for an FEV is that we in order to compare with the hardware We need to go with these two things low latency and high throughput How much low latency I don't know it depends on the use cases The the market that the ones that are driving this effort mostly are telco companies So they have voice traffic and voices usually is small packets Not this modest one, but small packets especially for signaling and they also need high throughput High throughput means density, right? You can put more customers on the same equipment if you don't have high throughput You most probably will have more devices and then the cost benefit might not be interesting anymore and When we do this measurements the requirement is that zero packet loss And that really means that zero packet loss if you one packet is dropping then the measurement is not valid anymore And you have to see what is the the best number you can get with zero packet loss this is the way we we saw the vendors and hardware vendors especially comparing devices and see You know to evaluate to which one is best. So we are trying to do the same thing, but we're softer this time Okay, let's look about the 10g 10g is the most probably the The speed that most data centers are Moving or already at it and it will give you the Once you understand this then you can estimate what's going to happen when we move it to 25 40 or even 100 G so in the worst case scenario that we'll are going to use this wide speed with the smallest frame Here's some explanation what the smallest frame looks Looks like so the other nets we have two pieces mostly The other net specific That's not part of the packet the the packet that you were sending but specifically to the other net This appears in the wire so 20 bytes for the the whole thing Which is 12 for the interframe gap plus eight for the Mac preamble And then the other net frame itself which is the Mac header 14 bytes plus four six bytes as a payload The total is 84 bytes So if you multiply by eight you convert by two bits and then just divide to 10g you get this packet hate 14.88 million packets per second Why this number is important because if the software solution Can process packets at that speed it means it's Reach the line hate speed. So we are equivalent to the hardware solution Okay, so now that we have this Trafficking the The peak traffic hate how much time we have to process each packet So just do a simple math here. We found that is 67 nanoseconds Well, we know that CPUs are faster But how that how much that represents on a softer solution running on by metal So if you look if you consider a three gigahertz CPU, that's about 20 hundred cycles So basically we have 20 hundred cycles to do everything we need with a packet and that means receiving Processing and either drop or send someone else But that's all we got including the if there is any processing the application That's forwarding this case the appliance To put it also in perspective one cache means takes like 32 nanoseconds and Well, this depends on the platform you are looking in the CPU So if you have something specific, this is just an approximate number, but you can see it's pretty much a half of our budget So two cache misses. We already lost a packet L2 cache heat is like 10 cycles L3 cache heat 36 so in another words, it's a very small budget to deal with this It's The amount of the cycles you have and the work you need to do it's really a tight budget budget Okay, so Going to the software stack. This is the pvp scenario the physical to virtual and then physical Traffical generator will send traffic through a physical board Going to the virtual switch or some solution that Makes the packet appears magically inside of the virtual machine Then we have a forwarding solution there or I don't know some the the the virtual function And then back to the traffic generator to through another port. This is the basic Unit the how we are testing and comparing results with the other solutions as well And this is what I will be using in the next slide as To see what are the weak spots? So basically that 200 cycle budget needs to cover all this path Okay possible solutions we have a few available today one is Linux bridge Then open the switch SROV and DP DK enable of yes Linux bridge it uses the kernel data path It is a good thing because it's integrated with the kernel so you can most probably use The facilities available in the kernel But on the other hand it relies on nappy and has an unpredictable latency nappy means unit Napster is I don't know if you are familiar with it, but it's a mix of interrupt mode and pulley mode So basically you when you start it's in interrupt mode And then when the packet comes in it generates an interruption This is a schedule then a thread will come back and will fetch this packet But switch to pulley mode. It will pull mode either until a certain budget is reached Or there is no more packets then it switch back to interrupt mode So if there are more packets, let's say that you got 64 packets, and that's the budget It will reschedule itself and then continues and pulling mode as soon as you don't have any more packets It will go back to interrupt mode Well, the good thing about nappy is that if you have Not at the highest the speed possible You don't spend CPU cycles processing the network card, but on the other hand it impacts on latency And we are looking for low latency here. So I'm interrupting mode is really not an option at this point Another thing is some predictable latency One of the reasons is the interrupt mode and another reasons could be resource allocation The kernel does a pretty good job with a resource allocation But if you we for instance get a lock in the way, or there is a memory allocation Then our 200 cycles is gone. So we need to have the this predictable somehow and It's not a CDN ready. So Linux bridge doesn't Doesn't speak like for instance open flow protocol Or any other as the end of protocol And if we measure the throughput Very simple test the fee to fee so physical to physical. Let me go back a little bit This okay one I'm short-cutting here physical to physical Okay forward Physical to physical I get one million packets per quarter and that's a very low throughput. So What we can do with the bridge, right it is used for a lot many other things and it's very hard to change the way it works or the Holy kernel to make it fit in this use case and another thing to consider is that the keyboard runs in user spaces So since the Linux bridge is in the kernel space, we need to send, you know We need some context switches to send a packet through the kernel to Kimui in user space Open the switch is pretty much the same thing as Linux Linux bridge. It runs in the kernel data path also relies on api Unpredictable latency one good thing is that it is SDN ready at least it speaks one of main protocol But the throughput since it has pretty much it leverages the same infrastructure in the kernel The Troupled is pretty much the same one million packets per quarter Also, the Kmo is in user space. So there there is a solution for the for that to speed up Which is called V host, but it's still It it has an additional cost SR of E I guess most Most use cases that depend on low latency high throughput are using this solution Because with SR of E you can bypass the host you can literally move the device To inside the guest expose the device from the host to the guest. So you bypass completely SR of E It's interesting because one physical port It appears in the PCI bus as multiple virtual ones So you don't have only one port, but you might have some up to a 256 devices they are Sort of you don't have access for the whole PC PCI device You can change you cannot change for instance things that would impact on other devices So it's quite limit, but still you have everything you need to send the received packets And you can simply move bypass the host and moving this device directly to inside of yeah It's not SDN friendly because then you don't have the virtual switch running on the whole so you don't have control You don't you cannot tell what the guest is doing But if you want to stir traffic on Certain traffic to a certain guest or another a traffic to another guest. This is not possible anymore the devices Physically exposed it to the guest and that's also a problem because there is no obstruction then you move If you do migrations for instance, you have to make sure The host you are receiving the virtual machine supports the same hardware, you know the same Everything that you had before you need to have on the the other server Although 256 devices that's a limited for the that comes from the specification seems a lot That's not exactly what the hardware provides today if you get a card you might have only 16 supported maybe 32 I get I saw some supporting 64 virtual functions. So Just because this pack limited on 256 doesn't mean you will get there. So depends on the hardware also So good thing about the solution it offers low latency. It offers high high throughput But on the other hand you have all these Cons, right? Okay, so let's go back to software one thing that we could use is the data plane development kit We call DP DK and DP DK is a set of libraries and drivers Designed to one thing in mind do fast packet processing, right? So It's an open source project Has a BSD license if you there was a talk yesterday from Maxim and Victor They went a little bit deeper on what is the PDK so in this their talk are pretty much Related to this one as well. So I would recommend to watch their talk to see what they're doing in the v-host the virtual land So what's the user job DP DK receiving sending packets within a minimum number of CPU cycles That's what we are looking for but what the PDK is not and I'm putting this here because Quite a lot of people confuse that it's not a networking stack So it doesn't replace the car the Linux kernel if your application in the socket for instance You won't get the socket here. There is no TCP processing. There is no TCP Conception control algorithm and it just gets the packet deal with the buffers deal with rings and Send out a packet on the device Okay TPDK pull ball drivers so TPDK has this pull ball drivers And what is that is a way for you to get packets from the network card directly in the user space So the buffers are all mapped in user space The next thanks to the VF IO interface we can simply get packets there or send packets out on the network card very quickly Okay, so now speaking about open this switch plus the PDK open the switch It has a kernel module you probably saw that if you already played with open the switch You saw that there is a module there There is also a module in the open the switch repository, but the kernel part of it is just a cache So the user space does all the intelligence to the user space does all the processing For instance a packet comes in you match on it and then you accumulate all the actions You need to do and once you have this big giant Rule that says I need to match it on all these fields and do all these actions This is push it to the car. So when the next part comes in the car knows what to do, but What to get to that point user space did all the processing all the processing Well, the PDK provides libraries those libraries They work like a pre-allocated memory they use huge pages It's busy pulling now interrupts You have lockless rings you have rings with a single producer multiple consumers or single consumers a single producer So we can use these libraries and and the driver to have the v-switch doing all the work And that's what we are doing TPDK enable open the switch So remember the 14.8 million packets per second if you measure Physical to physical we can get around 16 million packets per second. So that's a huge bump There is one trick here that it's not the processing is not done per packet one of the features That the PDK bring us is batching. So we are moving batches of packets But it's still if you send put on the tracker generator and measure you will get around 60 million packets per second per core But that's not for free. We need one core 100% busy running in the PMD thread Okay, let's look at about OVS DPDK for an FV We need to provide network connectivity to virtual machines came who runs in user space But now we have DPDK we have OVS everything is running user space So there is this interface called the host user It is a shared memory between chemo and the virtual switch So we can simply copy packets on this ring and it will appear on the guest No kernel at all in the in the The whole side the throw puts about three and a half million packets per second per core Default features enable in the PVP scenario. The system is not heavily done it but since this is very CPU bound and It depends very much on the the platform you have the CPU you have So if you have a high-end system, most probably you get a better number than I have here also We also support multiple Streams multiple queues, which means if you enable more cores to process more queues This is k-waps quite linear linearly. So I would say with four cores We can get line hate speed even pushing to the virtual machine But yeah, this system needs to be carefully done it. We'll talk more about that in the next slides How does it work? Basically, we have this pull mode driver thread. It wants a CPU and that means it It's nothing else should be running on that CPU if you have something interrupting the CPU You are pretty much stealing CPU cycles from it and we already know that the budget is very tight. So Any interruptions any scheduling events or anything that happens that are locking a memory allocation something like this It doesn't work. So DP DK provides us all the resources pre-allocated. We already are working in busy pulley mode But still we need to make sure that these thread Wounds the the CPU The devices can be distributed between the PMD threads. I say the device here and but this actually queues By default device provides one queue, but you can enable more queues and then distribute this queues to different PMD threads each PMD thread will be easy looping and busy loop for packets and then do the processing and the model there there is run to completion Which means once you get the thread gets the packet it will go through the whole Processing until either the packet is sent out on another port or drop it. So we don't have any scheduling in the middle and one thing that I already mentioned one of the trick to Virtually expand our number of our budget for processing is to have batching so you move Instead of a packet by packets you move like 20 32 or 64 packets at a time Okay, so let's do an x-ray of the PMD thread to see how it works It's quite simple, but due to this simplicity sometimes you have some consequence of it So basically you have the any ports in this case It's a genetic scenario. You can have any ports as any ports as you want You have the queue is represented by this this box. So this is the receiving queue Which did this box here represents the PMD thread It will pull for packets for each received queue. Once it gets the received queue It will push the batch to the forwarding data plane That's the v-switch It will do all the processing with the packet. Let's say Modify the destination IP address I don't know remove a villain tag or add a villain tag Everything anything you can do in the SDN network and then either will drop the packet or send out on TX QE for another device That's basically how it works for every I mean for any use case you want to use the PMD thread as You can see as you add more ports or QE is it could be This actually could be any QE is from the same device, but it's the same thing So as you add more QE is you will spend more cycles here pulling each port and One impacts on another of course Okay, so going back to the just to fit things together, you know what better what I'm talking about We discussed this to simulate the NFV. So here I have I don't know a firewall running Where you can I don't know use Juno as a login on Juno s or something here to establish that And here you have two logical ports the v-switch in the middle to physical parts This is represented here So you have two physical ports here and the two logical ports and this weird our arrows simulate to represent the forwarding Just to show you the equivalent sort of what I am talking about I Represent as you can see this physical port It sends traffic to the virtual switch, but it doesn't send traffic out to the traffic generator So I didn't connect the TX QE here So you only have the RX when it's relevant and TX when it's relevant We'll talk more about the second picture, but this is just to put things into places Okay, this is now with a more realistic scenario for an FV Let's talk about the packet flow So this is the feed for the physical part that receives on traffic generator. It's called number 10 It will come here the thread will pull for that will match this Data plane rule open flow rule it says how coming from port and send out on port 21 Goes to the RX of the 21 Appears in the a guest the guest does the fort to the TX QE in Inside of the gas which becomes the RX QE outside of the gas and then the the packet goes back to the pooling Then it sent out on the for data plane. It will see another rule here Oh packets coming from port 20 sent out on the Port 11 and then it sent out here on the RX TX QE and back to the traffic generator So one thing to note here The packet goes twice through the PMD thread. So remember that budget we have we actually have to Divide by two because of this Okay, how to measure to put with zero packet loss what we expect is that we have a set up on the traffic Generator a constant hate then the systems is constantly dropping packets with decreased the traffic rate and repeat until we find something that works out Okay That's actually it happens We should look at where the packets are being dropped it because there is the are the bottlenecks we should improve So in this case, there are three red stars the three weak spots where we should look more in the detail The first week spot is the receive QE for the physical device What happens that? Basically, if the PMD thread is busy doing something else and it's not pulling this device fast enough this QE will overflow And it it's a physical device. So it has a limited number of resources and There you can some network cards allow you to extend a little bit, but still It is Limited so it's a classical producer consumer problem If this is not fast enough pulling packets out of this device the queue will overflow and you will drop packets So it's really important that this runs fast The other thing is this receive QE here For the same problem if this is not running fast enough this queue will start to build up and At some point you will overflow the difference between this and the previous one is that The the drops are reported in the port start so you look at the device You'll see the drops you can identify in this case. You don't see the drops stats in the host You see the drops the stats in the side of the guest because when the application is trying to push packets to this part The port is full so it's dropping inside of the guest and it's not visible in the hosts Then the other weak point is the TX QE here on the view host user But that's not a fault in the PMD anymore. What's happening is that the guest is not running fast enough So the guest is not fetching packets is not pulling packets from the this virtual device and that So the the PMD is trying to push more packets, but there is no space so what we can do There were again the same Producer consumer problem so as you can see it's like a clock, right? So if you something goes off a little bit, there will be consequences and depending on what's off one of these three points will blow up Okay, measure throughput with zero packet loss I told you about this but in reality. I saw this happening System is stable for a period of time. You can see traffic is running just fine You go out to lunch and then you come back. You have like two packets drop it Very low throughput and the goal is to understand the drops Here is We saw the 10 gigabit budget. I put the table here with Less because that's what the virtual machines are doing anyway So this is this is the budget for the three million four million five and six Not just calling attention for this number because it will appear later, but basically That's what we are looking at To see the how in how significant are the impacts of others Remember packets go twice to the PMD. So the real budget for packet processing is this one All right, I measure how much time I spend pulling each device and how much time I spend processing the device So physical device for a processing I spend three and a 3.1 micro seconds and the view user egress device which is coming back from the virtual machine I spend two micro two point one micro seconds Others are really low processing here is zero because we are not using that and the pulling is quite fast So these are the main ones, but the total if you sum up ever all the numbers you will get 6.2 Microseconds to loop a packet batch of packets between in and out All right batch. So you saw we saw the number is six point two microseconds is 24 times more The prepackage budget I showed you before that happens because of batching. So we are virtually extend our Budget doing batching. So batch is a fundamental thing Assuming that we would have like a perfect batch of 32 packets We could have five million packets per second instead of three and a half So something is happening trying to estimate the numbers with the three and a half zero packet loss the budget will be this We can calculate or estimate the batch size to two into one in average, this is something maybe we can improve but It depends on other things right if you go there and take out all the packets You have at that moment, but it's not 32. Should we wait for more? It might introduce latency So what is wasting time internal resources is external resources internal resources if you look at how it works There is no locking a more memory is pre-allocated There is no reason for it to stop so it runs 100% of CPU busy, but we should look at the external sources of problems What they are how much significant they are we look at the budget we know the time We have so we have a way to tell if they are significant or not For instance, I was you callbacks. This is inside of the kernel So just to remind BVD processing budgets three for three million packets per second is this value. I Use this cool to call F trace It shows you the Functions that are calling in the kernel for a specific CPU you can filter that and then it has a nice time stamp And you can calculate how much it takes to run the kernel are still come back and this is the number I got without the preemption preemption cost I mean moving from the PMD thread back to the kernel run everything The run the CPU callback and then back to the user space 50 Microsoft seconds. That's roughly eight batches one solution for that Disable the RS you callbacks on a specific CPU list and No CBE pool also But if we we need to do that, otherwise, there is no way to get rid of this Okay, next slide. Another thing is time interrupt. There is a periodic time interrupt that also causes scheduling issues and task accounting we can use this parameter here to Move to a no no Hertz Reduces a lot the time interrupt but doesn't solve the problem there still will be one interrupt per second and That's unfortunate. I know that there is a patch being working in upstream to get rid of it But at this point I couldn't have tried yet. So I don't know if we could do that Okay, all the resources Other sources of problems scheduling issues We have to take out the Iraqi balance because any hardware interrupt on that CPU running PMD thread can Impact we have to use ISO CPUs, which means take the CPUs out of this scheduler We don't want any tasks running on that CPU. We had to disable Watchdog watchdog will periodically check something this has some impact as well power management and that depends on your platform you can try to use this to Limited the state of the CPU to the maximum so at least in you are sure that your CPU are running at least at a certain level Hyper threading depending on what is running on the other thread. It can also impact in terms of cash trashing false sharing and so forth so on So this is important to look at either disable or know what you what you're running and Real-time kernel real-time kernel for other stuff because as you saw if the gas is not running fast enough So we need to take care of the key mood threads as well, especially the ones running the VCPU there We real-time kernel helps to make sure the kernel will not Introduce a high latency there Okay possible improvements you saw that It cycles between the devices. So if you want devices not working or is is lower It affect others one possible solution for this is using the DPDK altred subsystem, which is kind of a Scheduler of threads in user space highly optimize it, but it adds a cost So our budget for line H speed is 200 cycles. Does that worth it? Maybe our devices are good enough. It depends on the case for most hardware that I tried The impact the the old of yes when pushing packets to a virtual machine If there was no space in the ring it would retry for 100 microseconds Yeah, 100 microsecond is really too much then we change it for trying again if it succeeds try Send more packets if it succeeds try again up to eight times It's a lot better, but still you waste time doing that. Should we just drop the packet? Then we are not with the zero packet loss. Should we try again? Then we lose time So there is a compromise there Vhost user has a feature called measurable buffers, which is Buffer in direction to support buffers above the M2 If you disable that you pretty much simplifies how it works. So it runs faster This is changing as I mentioned there was another talk yesterday They did some nice work here Increase the bed size, but again, we saw the average was about 21 There were some tests in the past that show it 32 as a good number above that It doesn't work quite well below that. It's not that efficient So this is something we could look more with as other things change Increase the virtual ring size. This is being done in upstream Bio settings. This is very very very important each vendor each hardware vendor has some features there Some are not really clear if they are related or not. So we had to talk with Our hardware vendor to see the best possible options. If you look at the DP DK side, there will be Some chips there on what to enable especially for on Intel platforms one obviously way out is a faster platform is and CPUs, right and One thing that we need to do more is to improve the CPU resolution and this is How to make the PMD thread really wants the CPU and not having any Interruption, okay Questions, maybe I went too fast, but I can scroll back the slides Well, we are mission. We are trying to push as much as many packets as possible With the very small packets. So if we can get There is no queuing in or very small queues The what do we measure the latency? It's not what the hardware can provide to you, but it's a the question was Regarding to latency if if I have numbers to show about that we are still looking at the There are measurements about that but do it to other facts like packet drops We think that first we need to get rid of them to rise the the throughput and then we'll talk more about latency But so far latency The other team that measured that Told us that this is not a problem yet. Yes, this one. Yes Yes Okay, so the question is if I try to use multiple PMD threads Kind of having this better distribution. So I could improve this by adding more PMD threads. Yes, we try that Basically, you can do two things either you have more PMD threads and then you distribute the ports or you can increase the number of queues And then distribute the queues for instance, you could have four PMD threads and For each of it processing one queue for the from the same device So the host user would provide you for queues and the traffic will split equally between them It helps to improve the it's pretty much scales linearly So as you add more PMD threads, you have more power to process traffic But on the other hand you have one more busy core that you You cannot push it to idle for instance. Yeah Yes So the question is if I try to put the two PMD threads on the siblings Running on the same core. So basically we would Leverage the the the fact that those two were running on the same core and they could share caches It helps for some use cases, but might not help in the other use cases since it's a forging data plane here and Depending on the actions you are executing you might have some cash trashing in the middle So in this case of the my data pain programming were just two rules So it most probably would help but in the real-world scenario We need to try that. Yeah Yes, Nick you mean for the network card because right now the PDD Okay, so the question is if if I could name some of the Intel features that allows the PDK to work okay, so the PDK started as an Intel project and in the beginning was limited to Intel cards, but right now we have other vendors So most cards at least if you the most common ones are supported and the feature is pretty basic I mean it depends on the other feature which is called VFIO or UIO if you don't care much about careless about security, but you the common network cards So you can use yeah Yes, it was a good server, but not high-end one. This is a Young I Don't remember exactly the models we bought as well. No, it's not as well mine is a bit older than that So see it's not even as well. Yeah, most probably a bit. Yeah Yes, basically you have to the PMD trust needs to be ISO the question was sorry the question was if I have to What can I do or if there is any recommendation to isolate these threads from other things? Yes There are documents in the DP DK for instance, and there is a guide inside of the open the switch project Install dash DP DK and there is no it's not it changed So it's DP DK and there is a DP DK advanced that guide these advanced that guide will give you Good chips to how to get there We try to keep it updated because we also want other people to look into this and help us to improve the performance Yeah, and for the the chemo Libby virt has support for CPU pinning so you can tell where the V CPUs will be running the threads that Handlers the V CPU will be handling and then you can isolate those threads in the same way You will do for the PMD, but for the chemo the problems that chemo also And yet at some point we'll need to do these calls and other stuff So that's why I mentioned in the kernel with erity real-time support Make sure that it not it doesn't get stuck without locking for instance Red Hat does provide the erity kernel You need to talk with our Organization to see how to get that. It's not I don't think it's part of the standard row Yeah, I'm sure that's not part of her But you can get that package from the upstream. I'm not sure it's really I'm not looking into that But I believe the patches is our public and you cannot compile yourself I guess there is this guy here that knows about that. Okay, so Yes, that's on our to-do list. I haven't tried yet, but it's something we should look at yes Yes. Oh Sorry, I forgot to repeat the question if we look at into cache allocation technology for improved this Yeah, you you will have to look at your season and Say how many VMs you want to run and isolate them You cannot pre-isolate them and let the CPUs available So when you start the VMs the CPU will be there, but I agree with you It's something that you it's not really easy to isolate everything right now That's one my last bullet was to help improving the kernel isolation at least for that CPU Okay, I run out of time, but I will be in the highways if you need more. Thank you Sorry about that. Oh DJ