OK, so when I had to prepare this talk it was not easy to put all the slack together because I think yesterday we went a lot through a lot of the material already of seasonal forecasting. So there may be a bit of a repeat today. So, first of all, we start with seasonal forecasting. And I will go a bit to the history of seasonal forecasting. The history is quite recent, I mean relatively recent in work on seasonal forecasting come from the late 1970s and early 1980s. And particularly this person, which is Kikou Miyakoda from GFDL, one of the first ones we really opened the door for seasonal prediction. So there were several studies by Shoukla, by Sparita, 76 Miyakoda, which explored the predictability at the seasonal time scale. So they tried to go beyond the dynamic predictability, which was assumed to be around two weeks at that time. And they recognized that the seasonal prediction can be seen as an initial value problem with external forcing, boundary value problems. So as I mentioned yesterday, it's a bit a mixture of NWP and seasonal forecasting. So it's called predictability in the midst of chaos. That's Shoukla and Zopalmer. So one of the first case studies actually with the seasonal forecast, which was demonstrated by Kikou Miyakoda, was an event which took place in January 1977. That's the newspaper from the time, the Miami News, which was a snow in Miami, which is of course extremely rare event. And Miyakoda tried to predict this event. So the top panel shows the circulation, the climatological circulation for this period of time, for January, for all the months of January. And the bottom left is observations for January 1977, which is a very different circulation than normal, very abnormal situation with a strong trough, with a very cold hair coming towards the Florida here, and bringing the very cold hair and the snow in the area. And the bottom right is forecast from GFDL, Moodle, that Miyakoda was at GFDL at that time. We showed quite remarkably those forecasts for day 10 to 30, which shows remarkable similarities with observations. So it was deemed as a very successful forecast. And we met a lot of people very excited at that time, so we may be able to predict whether up to 32 days. At that time, actually, there was a lot of interest at SMDUF, too. I mean, at that time, Kuba, Shkibaldi, Molteni, tried also to, in the 1980s, make some experiment to extend the deterministic forecast to 30 days with the Global SMDUF spectral model. They published some reports on it. And they did some case studies. They used fixed SSDs. And the results were quite mixed. At the beginning, they were very excited because they found that the data from day 10 to day 30 were generally more skillful than climatology. But the problems then they found after that, in fact, if you persist the forecast at day 10, you tended to get actually more skillful forecast. So there was no point burning CPU time beyond day 10. And this shows that actually at this time range it's much more difficult to be persistent than to beat climatology. So for a long time, the seasonal prediction went down. And it's only in the 2000s that the interest for seasonal prediction came back. So all that to say is that seasonal prediction is still easy and fancy. It's much more recent, I would say, this seasonal forecast is much more recent than seasonal forecasting, which is 10 to 20 years, but it's 10 to 20 years earlier, in the early 1980s. And 10 years ago, a couple of professional centers were producing dynamical seasonal forecast. I mean, they were, that was ECMUF on JMA basically. Now, most of the global producing centers that were mentioned in the previous talks are producing and issuing or experimenting seasonal forecast. So this is, for example, an example I got from Aon Kumar from NCEP CPC, which is, I think, quite a nice slide. It shows the NWS National Weather Service seamless suite of forecast and on the left column you get a more, that's more the ready set go concept I mentioned yesterday. So for the long range, you get the outlook, then guidance, suite assessment when you get closer, set forecast, watches and then you alert and you take action at this time range. And then you have all the forecasts for minutes, hours, days, three weeks, so where NWS issue forecast. And then you have this time range two weeks, two months, where there is no official NWS outlook, no for weeks, three, four time period, for instance, now, whereas there is some outlook for season on years. So currently already some centers like NCEP are not issuing, operationally, official outlook for this time range. Now, this is a plot that didn't show well for some reason, but I have the equivalent here. And that's an experimental product from NCEP, where they start to produce forecasts for week three and four, which they present under the form of above average temperature on the left side, which is for the orange colors, or blue is below average temperature favored. So they present the information as a per side for week three plus four. And on the right is for precipitation, dream is above average precipitation and brown is below average precipitation. So if you are familiar with this, it is very similar for week two already. And this plot is already produced not directly from the dynamical model, but by using MGO forecast and INU forecast. And basically it is a canonical response of those events. They plot this type of forecast. They produce this type of forecast. But they have plans also not to use directly the outputs of the dynamical model to do this. So all that to say that some operational centers are just starting to issue forecasts at this time range. And if I come back to the table of S2S models I showed already yesterday, most participants have developed seasonal forecasts. So we have already over 10 models which have dedicated seasonal forecasting systems. Only METO Francier doesn't have of the 12 GTCs that was mentioned in the previous lecture. Only two are not doing seasonal forecasts. One is the CPTEC, the second one is METO France which are only seasonal forecasts. Some of them use the same model as their seasonal forecasts but with more ensemble members. So the strategies, as I mentioned, are very different between models. There is no consensus there. Some models, seasonal forecasts are used the same forecasting system as their seasonal forecasting systems. What they do is they use the exact same system. For example, it is the same CFS S2 system for seasonal forecasts but you have much more ensemble member or much more frequent start dates up to day 45. For UKMO, it is the same system as their Zaglo C5, same system as their seasonal forecasting system but they have twice more ensemble member up to day 60. Some of the centers, seasonal forecasts is an extension of the medium range weather forecast, for example, the RTCMDUF under the Environment Canada. The RTCMDUF, for instance, the 46th, it is not 32, it should be 46, 46th forecast, the medium range EPS sensible system is extended to 46 days twice a week so that the exact same system as we have for medium range forecast is a different system from system 5 which is used for seasonal forecasting. And in other centers is a separate system which contains characteristics for seasonal forecasting. That is the case at CEMDUF before 2008. That is the case also at GMA which is a separate system. As I mentioned already yesterday, there are very different configurations for forecasting systems. There is no consensus on the optimal configurations.  The reason for those choices four times a day with four Ansible members is because they have a chunk of their machine allocated permanently to seasonal forecasting. So that's a way to optimize the use of the machine. Whereas at CEMDUF we can allocate a large, almost half of the machine to seasonal forecasting system for a period, for a short period of time. The model resolution can vary from 250 kilometers to typically 50 kilometers. Most of the Ansible members are less than 100 kilometers or typically less than seasonal forecasting. The time range goes from 32 to 60 days on different model setup. And that's something I didn't discuss yesterday. But some models have a coupled and ocean atmosphere. Some also have an active sea ice. And this difference comes mostly from the fact that some models, the models which usually are, as I mentioned, using the same system as the seasonal forecast system, usually have a coupled ocean atmosphere system. And they use also an active sea ice model. For the models we are always an extension of medium range forecast. Often the medium range forecast are not coupled, as was discussed in the previous talk. And they often don't have active sea ice. So usually the seasonal forecast produced from those models are usually no coupling or no sea ice. So if you look at the list here, I put in red the one which is not in the system yet. And in terms of ocean atmosphere coupling, we can see the majority of the model are coupled to an ocean. Only three of them are not. And a few of them, one, two, three, four, five, have an active sea ice model. And those are, as I mentioned, seasonal forecasts which are similar, which use the same system as the seasonal forecast systems. ECMWF and BOM are active sea ice models, which is planned for the next year. OK, so the next step is, as we showed, the ECMWF Ansible Forecasting System, which is based on the IFS version 41a1, which is at a running at a 32 and 64 km resolution in a single edged two-table. And it uses a couple to name, which is a one-by-one degree ocean model at a zero to three degree near the equator with a 42 vertical level. And we use HTSOL, which is for the land surface model. So IFS is coupled to HTSOL. And the initial condition we use a four diva variational data simulation for the initial condition for the atmosphere and three diva for the ocean. So the perturbations are EDA perturbations with singular vectors. So we come to that more in today's time. We use the five ocean analysis for the ocean. And the model, the couple GCM, so we run for 51 times at a once again resolution T639, which is about 32 km or 64 km up to day 46. And that creates the Ansible forecast. OK, so the IFS will change the resolution up to half day 10. So we run at 32 km for the first 10 days, then we lower the resolution and then we go today to 64 km afterwards because it's too costly to go to 32 km for 46 days. So we change the resolution. Between day 10 and day 46, it will be at 64 km. That's correct. I mean, this is a bit of a problem for some users because for precipitation it can create some inconsistency when you pass day 10. So we have to be very careful in the way you retrieve actually precipitation. So we have plans to extend actually this 32 km to 15 km to day 15, but that won't stop the problem. That puts the problem a bit further in time, but there's still this. So this was necessary to be able to use the same system for seasonal forecast because as I said, it will be way too costly for us to run at 32 km all along. OK, sub-seasonal forecast product. So for short or medium-range forecast, usually the forecast are issued for instantaneous or daily values. So basically a typical medium-range forecast is to tell you what would be the temperature in three days' time over three years. For seasonal forecasting, products are generally on a seasonal mean. At this time range, you cannot distinguish the weather from one day to another. There is some work no one is going to produce a monthly means, but it seems that after months one and two, there is very little skill in this kind of issue, the forecast from one month to another. So in general, the products and season time scale are more on the seasonal means. Sub-seasonal forecast, it will be on two weeks. Once again, there is little predictability in the day-to-day variability. We cannot really predict that the day 22 will be different from day 21 weather. But there is some skill in predicting weekly mean anomalies. So most of the products at this time range are on a weekly mean period, which is a good compromise between a period which is long enough so that you have some skill to predict the change from one week to another. And it's short enough to give you useful information. You could also issue a monthly seasonal forecast from day zero to day 30, for instance, but then you lose a bit of resolution. I mean, the two first weeks of the month may be very warm. Two second weeks may be two following weeks, very cold. On an average, you will have a normal month, whereas it was very unusual. So most of the products, and this is an example, similar to the one we did this morning, where we look at a weekly mean anomalies of two meter temperatures. So that's the typical product from ECM2F model. So here we take the sensible mean of the return forecast minus sensible mean of the return forecast. And we had those statistical tests on top of it to see if the sensible distribution of the return forecast is statistically different from the sensible distribution of the return forecast. And we shed the area where the difference is statistically significant. On the area which is white, on the area where there is no significance. And, of course, the longer we do in the time range, the more blank area you are, because your model is doing more and more towards climate and there is less and less skill. So there was a bit of distribution yesterday, or is it a useful product or not? Because here you lose a bit of information on the sensible distribution in one hand. But the fact that you also apply these statistical tests means you still keep some information. And it tells you basically where the area, where the distribution is really shifted from the return forecast. And the reason people like this type of slide is that in comparison to the probability maps, as I will show later on, it gives you an idea of the amplitude of the signal, the amplitude of the anomalies. Another type of plot is to produce a forecast as probability of precision, for example, anomaly in the upper teresile. So we define, of the model climatology, the limits of an upper and lower teresile. And you simply count the number of model and several members which are in the upper or lower teresile. So here it tells you, for example, that there is a low probability to be in the upper teresile. But there is high probability here to be in the upper teresile for precipitation. The high probability of precipitation is those areas. But compared to the products, this one doesn't tell you much about the exact intensity of the anomalies. So we also have a series of more sophisticated products. For example, here we have a six predefined weather regime over Europe. And we project each sensible member onto each of those regimes. And we simply contain the number of sensible member which can be associated to one regime. So, for example, in this example, we can see that two sensible member only over 50 go into this regime. 13 sensible member go to this one, 11 to this one, 9 to this one, and 11 to this one, only four. So you can, the model predicts that those two weather regime are more or less likely to happen than the others. And once again, if you are in the first week of the forecast, you tend to have more sensible member cluster into one weather regime. And when you go to the week three, week four, you tend to have a more uniform repartition of the ensemble across those six weather regimes. Yes? For clusters, it typically is Z500, geoprotection 500 hectopascal. Well, this is the old product. Now we are already finding with the four weather regime, classic weather regimes, bloatings, NEO plus, NEO minus on the Atlantis ridge. But this, one of the products is a tropical cyclone, which I already mentioned yesterday. So, we tried the tropical cyclone in the ECMDF of Z500 forecast. And we compute from that the probability of a tropical storm within 300 kilometers. And by the number of tropical, number of sensible member which predicts the strike, we can deduce the probability. And this tells you there's more than 50, 60 percent chance there will be a tropical cyclone strike in north of Australia. So, that's another type of product we can define. And one important is also to monitor the sources of predictability. I mean, highly, we'll do much more in details into that. But we also have products, there is a prediction of the MGO, evolution of the MGO, over the next 30 days. So, this uses a wheeler and end-on diagram. I think we see quite a lot of them in the next two weeks where the MGO can be represented as a circle going from the Indian Ocean, Maitam continent, western Pacific, or west hemisphere. And the distance from the center of the circle represents the amplitude of the MGO. So, a strong MGO would be a strong circle around here. So, here we show the forecast that in here, we are holding the west Pacific. And each dot represents one sensible member. So, we can see they are all very cluster, very small spread during the first day of the forecast at day one. Then this one, magenta, is day five. You start to have a bit more spread. Day ten is a bit higher spread. On the day 20 is a green one where a lot of sensible member predicts that the MGO will die. OK? So, model seasonal forecast on the forecast. So, once again, seasonal prediction is in between medium range forecasting and seasonal forecasting. For seasonal forecasting, as was mentioned in the previous talk, you need to calibrate your model because the model bias is way too large. And so you cannot issue seasonal seasonal forecast directly from your real-time model output. For medium range forecasting, it's very rare, actually, to have a re-forecast. For the TD database, for example, there is no re-forecast there. Most medium range forecasts are really produced by looking directly at the output of your model because at this time range is a bias but it's small enough. It's a way very, very small compared to the signal you are predicting at this time range, which is very large, usually. So, a system that has been, we are starting to use actually a re-forecast to calibrate medium range forecasts. And there has been some papers showing that there is some advantage in doing it but it's not entirely necessary. So, what's about seasonal forecast? Well, the seasonal forecast is more like a seasonal forecast in that regard. So, the model systemic system will grow very quickly during the model integration. So, the model will grow very quickly towards its own climate. And after two weeks, it can be as big as the signal we want to predict. So, we show here an example of bias, a bias of two meter temperature for the time range 600, which is the day 26 to 32. So, the last week of the whole system, the seasonal forecasting system, we show that in some areas, actually, the bias, we have a one bias of two meter temperature for this, that's in August, which can exceed of four degrees, which can exceed between two and four degrees.  This is just a model. This is, sorry, that's a bias with respect to error in theory, yes, that's right, yes, yes. And, yes, error is of the 2.4 degrees, which is as large as the signal we want to predict. I mean, often we see a forecast that week four anomalies will look a bit like that. So, we need to remove this bias. And, there's two options. There is an option which is to make correction during the model integrations, which is a flux bias correction. So, you know your flux is a systemically wrong, for instance. Then, you adjust them to, usually, a small for coproduction atmosphere system. And, this system, which is very popular in the climate simulations, but it tends to mess up a bit of the physics for the short-term, medium-range forecast, which means that for sub-sino-sino forecasting system, almost everybody uses the apostere corrections where you basically run your same model over a certain number of years, previous years, as a handcast or reforcast. So, you produce a large set of reforcasts, which will tell you what is a model climate. And then, from this model climate, you can see what is a model systemically wrong. You can measure them and correct your model. So, it's not a perfect methodology because you implicitly assume that the shift in the model forecast related to the model climate corresponds to the true shift in the true forecast related to this old climate. Basically, you assume that there is a linearity in your error, which sometimes doesn't happen. Sometimes your bias can be very strongly flow dependent. So, once again, for the sub-sino-prediction systems, there is not much, there is a lot of the differences in the way the reforcas are set up. Some are using a very long, as I mentioned, reforcast set, some a very short one. Some, we have talked a lot about it yesterday, around the fly, some are fixed. Some have a large and stable size, some a lower one. So, I won't go too much to that. We have been, we have discussed that yesterday, but once again, to remind you, it's important to mention that fixed reforcast, for example, NCEB, BOM, GMA, those are reforcasts which are produced once for all because it is very important to understand this. So, advantage is more user friendly. It's usually application like it because you can measure for, you can calculate your down-scaling coefficients once for all whereas for the on-the-fly forecast, reforcast, you have to constantly monitor because the model changes physics. Yeah? Why is that the variety? So, for example, we have NCEB which is a reforcast is daily. So, if you issue a forecast on the 8th of August, you will have the exact corresponding date in the reforcast data set. But some of the reforcasts which are fixed like Australia, they have a reforcast is the first, the 6th, 11th, 16th, 21st, 26th of each month. So, if you have the 8th of August, you usually have to take the two closest dates in the reforcast. So, you will be on average, I would guess, between the 6th and the one on the 11th of the month. Okay? Yeah? Yeah, but I will say you will try to do that for each date in advance, I think. That shouldn't be too painful for you. Yeah? Yeah? Yeah, but I will say you will try to do that for each date in advance, I think. That shouldn't be too painful. Yes? Questions? So, that's another example. So, some of the reforcasts are for really fixed dates. The JMA2 is a three dates per month. And then, yes, you have to figure out which are the closest dates in the reforcast. It's a bit more complicated. Yes? Sorry? Yes? Yes? Yes? For best reaction, the second one. Nobody does the first one. Is that what you mentioned? Nobody does only in climate simulation. In long-climate simulation, they do that because they don't like to do a reforcast. But for sub-zero prediction, everybody does apostereal corrections, which means to run a large set of reforcasts to do that. Okay? And then, you have on the fly, every reforcast, like this M2FUKMO, which guarantees usually that you have the same dates as the real-time forecast, which is not always true because UKMO is also three fixed dates of the month. That's another story. And the advantage of this methodology is that it ensures you have the best, substantial, best version of your model at a given time. The disadvantage is this methodology is maybe more difficult for applications because each time you change model, well, your statistical model may have to be adjusted. Okay? So here, I will give an example of the ECM2FUKMO. So we read this, which is on the fly, reforcast. So we have this real-time forecast. That's an old example, sorry, from 2014. We produce our forecast on March 27, 2014. So we run 51 ensemble members at T639 for 10 days. And then T319, which is 64 kilometers up to day number 46, at that time was 32 days. And at that time, we have the reforcast consisted of five ensemble members starting the same day and same month over the past 20 years. So from 1994 to 2013, with the same resolution, though exactly the same model, so 32 kilometers, 64 kilometers for between day 10 and day 32. And we produce those reforcasts two weeks in advance. So that we have always a five-weeks window. Because for our medium-range forecast, we use a five-weeks window for some products, like the eight-week forecast index. So for a substantial forecast, those reforcasts should be as consistent as possible with the real-time forecast. The initial conditions start from era interim with the real-analysis, ocean real-analysis, ocean real-analysis under those soil real-analysis. And we go to that more in detail in two days. Perturbation Army, making what we are doing for real-time forecast. We're using our vectors. We use EDA for 2015 because we don't have EDA in the past. And we do use those basket of schemes, such as in physics. Now, a question we have often is, all our products, the system of forecast product at CMDF, use the same reforcast, the only one set of reforcasts, the one we start at the exact same day and same month as the real-time forecast. And the question we often have is, why don't you use a five-week window to calibrate your most substantial forecast? Because with a five-week window, you get a much larger ensemble. And if you have a much larger ensemble, it means you have a much better way to measure the limit, the boundary of the tersile or the boundary of the d-siles. So you can get a much more accurate calibration. But the problem we have is that, for example, if I take the example of two-meter temperature, you have a seasonal cycle. So if your real-time forecast is at the time where the seasonal cycle is at the maximum of the season, so that will be my week zero, the climate of week minus one, week minus two, and week plus one, week plus two will be systematically cooler than at week zero. So if you look at the difference between the climate from week zero, which is the one we use now, and the climate we will use if you did a five-week windows, you can get warm anomalies over the northern hemisphere and cold anomalies over southern estratropics. It's not very, very big, large, but the anomaly can reach one degree, which is almost the same order of magnitude once again as the signal you want to predict. So if you use this five-week window, you can mess up seriously your forecast. That's why it's very strongly recommended to use the exact data to calibrate your forecast, your real-time forecast. OK, so reforcas are very useful for two things. One is to calibrate the models, which is also very useful for skill assessments. And a large reforca database is very useful for calibration, once again to distinguish between a random error and systematic errors, and also to estimate flow-dependent errors. Some calibration are using like Bayesian techniques, a very large set of number of years to be able to not only correct systematic bias, but also the spread of the ensemble depending on the flow. And a large reforca database is also needed for verification and for flow-dependent assessment, like assessing the concurrent impact of ENSO on a specific phase of the MGO. If you want to see if your model really simulates well the impact of MGO when there is an N in your event, then you need a very large number of reforcasts, and also for forecast scores. The signal-to-noise ratio is also improving long-reforecast data sets. The large ensemble size is also important for skill assessments since some proboscis scores are impacted by the ensemble size. So, if you verify your five ensemble members, your proboscis scores are usually much lower. Some can be very sensitive to the ensemble size, although there can be some corrections. Ideally, we should have a very large reforcast, or a set of handcasts. The problem is that a large reforca dataset with large ensemble size is not affordable. It's too expensive. If you have a 30-year reforcast with a 51 ensemble member that will turn forecast, you make a system that will be a 30 times more costly reforcast than your return forecast. So, it means you will not be able to run 32 kilometers or 64 kilometers subsystem forecast. So, usually actually in the S2S database, there is a model which has a huge reforcast dataset. It's a bomb, the one from Bureau of Meteorology, and which has 32 ensemble members, six months for 30 years, six times a month for 30 years. But that's the one which has a very low resolution, the 47. The models which are very expensive usually are very short reforcasts. So, that's the trade-off. What factor? The high signal-to-noise ratio. Well, it depends. I mean, if you have a week before you're in the long range, your model forecast is very close to the climatology. So, you need really a large set of reforcasts to be really able to distinguish between the two. If you look at the maps, you have a lot. The way I wish I shaded are very small. So, that's where actually a large set of reforcasts can be useful. The issue is very hard. Yes, over a region like, for example, if you have a Nino, it's much lower. But a region that's over Europe, over summer, for instance, then you can have a lot of noise. On the for Nino, for instance, I've been sure with Adam Scaff, actually, the signal-to-noise ratio can be quite huge. So, however, I went to this one. The second point is that the long reforcasts are also from inconsistency in quality in the initial conditions, pre-satellite period, and they also start from initial conditions like from here and here, which are not as low quality as the one you use for your time forecast. So, you have to be aware that the skill assessment of reforcasts is often lower boundary of the real skill of your system. So, yeah, another point which is important is there has been some institution about what is more important. If you want to, if you have a lot of resources for a reforcast, what is more important? To have more years, a very large reforcast here, 50 years of reforcast with a small and stable size. Or is it better to have maybe just 10 years, but with a 30-member ensemble? Unfortunately, our forecast always says it's an open question. We don't know yet the answer. For medium-wide forecast, people have looked at this issue and they found that it's better to have a lot of a large number of years on small and stable size, because in medium-short or medium-wide forecast, the ensemble spread is very small. So, if you have very few years of reforcasts, also a lot of the cluster of the distribution will be cluster around a few points. But for stable forecast, it's not clear. And that's one thing we plan to investigate in S2S, actually. So, verification. So, it's not only important to produce forecast. We need to verify them. So, one simple way to do verifications on something we have in a web server, which is open to the public, is to verify each individual forecast this morning. So, you have the analysis from ERA Interim, which shows the anomalies here of two meter temperature for a given week. And here, we show the forecast that was issued for a different time range, but verifying on the same week. So, here is a forecast starting here on 19th February 2015 for day five to 11. And that's the forecast that we started one week earlier on 12th of February. So, different start dates is not the same forecast. But for the time range, day 12 to 18, verifying on the same week on the run. And then the last one is forecast starting on 29th of January for the time range, day 2632. So, it's just a case. It doesn't tell you if your model is really good or not. But, when you are doing case studies, it's always useful to see how your model perform at different time range and at which time range the model captures a particular signal. So, this show, for example, that this cold anomaly was relatively well captured, at least up to two weeks in advance. And it was a bit more to the east, to the west, three weeks in advance. So, you can do that, for example, as I was an important case, which was the Pakistan flooding since 2010. So, here, we verify the worst time was between 26th of July, 2010 and 1st of August. And this is an analysis from Iran Teri, which shows those big blobs of precipitation over Pakistan. And that's a forecast from day 5 to 11, which has issued on the 22nd of July showing that the event was fairly well predicted at this time range. All those amplitude is a bit lower, but that's a sensible mean. And that's a forecast from one week earlier, 15th of July, shows those same pattern of precipitation, and even three weeks in advance. For the forecast starting on 8th of July, we get those very strong signals. So, it's useful to verify some case studies, but if you want to have a more statistically significant assessment of your model scale, then you have to use a large number of cases to measure the scale. So, here we are using probabilistic scale score, which is called the rock area. So, this is quite a popular scale score in the NWP community, which is based on the heat on the full salamow rate. So, basically, the closer to 1, it is better. The perfect forecast would be at one.  So, here is a scale score of the same reference, one substantial forecast since the beginning. So, we put all the cases, all the real-time forecasts in 2004. So, we have about 600 cases. And, it shows that for this time range, day 5 to 11, there is generally quite a relatively good scale. And interestingly, you can see that the tropical regions, are the regions where the scale is lower. So, at this time range, you have more scale in the stratropics than in the tropics, which is well known actually at this time range. When you go to day 12 to 18, so you start to go to the 600 time range, the scale, of course, you have a big jump in scale. You have very few areas where you are more than 0.8 rock area. But, most of the time, you are more than 0.6 between 0.6 and 0.8.   But, overall, there seems to be a cool scale in the tropics, the stratropics. And, when you go to the next week, which is day 1925, here you start to have another big jump in scale scores to be between 0.5 and 0.6, some area between 0.6 and 0.7. But, here, it's in tropical area. Actually, that will get the strongest scale, like north-west Brazil and north-west Australia. And, if you go to day 2532, the last week here, you have very little scale in the north and extra tropics. You are always, I mean, that's a good news. Actually, you are always in the red color, which means you are always doing better than climatology. So, you are very close to 0.5, 0.6. So, you are very close to the future of climatology. But, this is slightly better. So, you are not losing anything. But, you still have some scale here in the tropical region. So, here, you start to be really like what you get in seasonal time range. So, it seems that day 5 to 11 is really behaving really like medium range forecast. 3532 is more really like seasonal forecast. And, between here is between is a transition between, you see, medium range and seasonal forecast thing. This is the wall here, yes. Yes, wall here. For winter, we say the patterns are the same in winter and summer. But, summer, you have more scale in winter than in summer. And, the opposite in the southern hemisphere, yes. So, as I mentioned earlier, there was an early attempt to do 60 to 40 years ago. And, as I mentioned, the scale score at that time were better than climatology. So, people were quite happy. But, it was doing worse than persistence. So, here, we see that the model is outperforming climatology. The question is, is it doing better than persistence? So, for that, we use rock area again, which is this curve. So, the closer it is to the top corner, the better. And, this diagonal is for when there is no scale, on the below that, when there is no scale. So, the rock area is actually the area below this curve. So, if it's at one, it's perfect. If it's a point five, then you are like climatology. So, the closer you are to the top left corner, the better it is. And, here, we are comparing the forecast of the real-time signal forecast with the forecast you would obtain if you persisted in the previous week. For persistence, you could persist day zero, the initial conditions. But, usually, it's much tougher actually to beat the previous forecast. So, here, we persist day five, 11 forecast. And, we compare it to the signal forecast of day two to 18. And, we see that the signal forecast actually significantly beats the persistence of day five, 11. So, we are doing better than persistence, which means that we are not wasting CPU time in pushing our system up to the 18, at least. For day 1932, again, here, we have put two last week together. We compare with the two week period from day five to 18. And, here, again, it's not much, but we still beat persistence at this time range. So, the white panel represents the time which was a bit explained by Adrian yesterday. It's a quite visual way to look at the reliability of forecast. So, I'll show you for forecast probability how many times, for given forecast probability, verified with observation. For example, if your model predicts 0% chance of an event to happen, it should never happen. If your model is perfect, it should never happen. Each time your model predicts 50% chance of an event to happen, it should verify 100% of the time. So, if your model is perfectly reliable, it should be along the diagonal. If your model is not reliable, it should be along the horizontal line. So, the closer to the diagonal, you are the better. And we can see that for the forecast at day two to 18, actually, we are quite close to the diagonal and definitely much better than the persistence of day five, 11. On the same thing, on the same day, 1932, we have a much more reliable forecast than the persistence of the two previous weeks. So, this shows that subsequent forecast, even at this time, range, which is 3, 8, 4, the still is not very high. Don't expect too much, but it's better than, I'm not saying exactly that climatology. For precipitation forecast, we get much lower stills course, but the same conclusion, we get the same conclusion. I mean, this still, even the still is low. For precipitation, we get more still, more still full than climatology at week 3, 8, 4. And generally, still, though, more still full than persistence. But the still vary from parameter to parameter. To make up the temperature like here is much more predictable than precipitation. OK, and another point I want to make is that the still can be very strongly flow dependent. And, for example, during a period where you have an MGO, we tend to get a much, much still full forecast over the next three days. And, for example, there is no MGO. So, this is, for example, here we are looking at the still, the relabity diagram for T815, the upper tertile, for date 1925, over week three. And the red is when you have an MGO in the initial conditions, which shows very reliable forecast at this time when for North American or whether if you have no MGO in the initial conditions, you tend to be more flat, much flatter, much less reliable. And that's something useful for the users. I mean, you may know that if you have an MGO in the initial conditions, you can trust more the forecast than if there is no MGO. So, it's a bit of a difference with the medium-range forecast where you have, I would say, always normally some still. You may have some flow dependence, still maybe higher in some circumstances than others. But there is a rarely period where you have no still at all, on day five or day six. Whereas here, for substantial forecast, we have really a window of opportunities. And that's where you have to be very careful with where your model simulates the MGO and simulates its impact, for instance. Because, well, if we come to the tool by Hylian, the MGO is one main source of predictability at this time range. And as I showed in the plot yesterday, here we are not looking at the phase of the MGO. We are looking more at when we have a strong amplitude. We are looking more at the amplitude, yes. Significant more amplitude outside the circle. Yeah, exactly, yes. Yes? Sorry? In both categories? About, yes. Yes, about, yes. It happens about 50% of the time during this period of time. Yes? Yes. And here was a recent paper by Rome Tripati, a work with him, where we are doing something similar from stratocytodon warmings. So, as I explained yesterday, those stratocytodon warmings can be also an important source of predictability at this time range. And we look at the skill scores over North Russia, for example, East Canada. When you have SSW in the initial conditions, which is a red line scores, and blue is a score when you have no SSW in the initial conditions. Those are higher, the better. Those are for week one, week two, week three, four. Of course, the skill scores are going down from week to week. But you can see that the red bars are always statistically significant higher than the blue bars. And same over Canada. We show that again, when you have a stratocytodon warming, you tend to get a higher skill score over those areas. OK, so last 10 minutes, I want to go to model development quickly. One point I want to make is that the models are evolving very quickly. This is the evolution of the models from the JMA model from 1996 to now. So, they have changed resolution three, four times. They have no resolution of about 50 kilometers now, whereas they were around 150 kilometers in 1996. The models were vertically evolving, the models are getting much, much finer resolution quite quickly. This is a change from the ECMDF substantial forecast over the last 10 years. 10 years ago, we were running the model at T159, which is about 21 degrees, 100 kilometers, 110 kilometers for 32 days with only 42, 40 vertical levels on the top at TENETO Pascal. We were re-forecast for only 12 years and we were running re-forecast with five members per week. Now, in terms of resolution, we have gone to T639, T319, those 32 kilometers, so we have gone from 110 kilometers to 64 kilometers. We have 91 vertical levels instead of 40, top at 1, 0.01 ET Pascal instead of 10 ET Pascal, sorry, 1 Pascal. Coupling, that's what I'm interested in. Originally, it was a couple system, then we merged with EPS, which was no couple from day zero, and then now it's a coupling, a couple really from day one. Re-forecast has been extended to 20 years, 11 members, and we are using error interring. So, the models are really getting much finer. I mean, we are changing resolution next year. We will go to around 30 kilometers next year. And, on the skill score, the model is also improving in terms of physics, not only has the resolution improved, increased, but the model physics is also improving. So, this shows, for example, the evolution of the amplitude of the MGO, the amplitude error of the MGO relative to error interring. So, if you are below zero, means that your MGO is way, way weaker, statistically weaker than in error interring. And we see that in the early age of the monthly forecast system, student forecast system, the MGO tended to be very, very weak. The model was not able to maintain an MGO for more than a few days. It was quite a considerable improvement. Some point that your MGO was a bit too strong. And now, the MGO is only about 10%, within 10% of the amplitude, in general, of the error interring MGO. And the skill score of the MGO has also improved considerably. This shows the day we reach MGO correlation of the index at the correlation of 0.5, 0.6, 0.8. And 10 years ago, in 2002, the model has some skills. If we take 0.6 as a reference, the model has some skills to produce MGO for only about two weeks in advance. Now, we are around 26, 27 days in advance. So, there has been a gain of about 10 days of predicted skills over the last 10 years. The MGO prediction has been really one of the big success stories of weather forecasting, I would say, over the last 10 years. So, I think it's very important here to understand that models are really getting used to be 10 years ago, and they are really improving very quickly. And, here is actually an impact of the MGO on the teleconnection over the mercenaries' attributes. So, 10 days after an MGO in phase three over the active phase of the MGO over the Indian Ocean, we tend to get this very strong positive NEO projection. And, in 2002, where the MGO was very, very weak, the impact of the MGO was extremely weak. Whereas, in 2011, in the recent version of the models, we get the patterns which are much more consistent with the Grand Theorem and with the same much similar amplitude. So, not only is MGO improving, but also the teleconnections on the model are much more able now to exploit the predictability associated to those sources of predictability. And, the other skills are also improving as the evolution of two meter temperature skills course showing that for day 12 to 18, there has been almost a doubling of the APSS skills course over the last 10 years. And, the skill for week four now is better actually than the skill for week three 10 years ago. So, there has been quite a very nice, much more nice improvement now of this sort of prediction. And, this represents the next grid. We are going to alter the dual grid, and then we are going to go to the next system, which has a resolution of around 32 kilometers, which will be probably next April. OK, so I would do just quickly to say that the impact of resolution is also positive on the skills course. We go quickly to that. So, when we improve the resolution, we tend to get a better forecast of precipitation on Z 500. And, we tend to get also better prediction of some extreme events like tropical cyclones. So, that's the current operational system, that forecast that I showed yesterday for tropical cyclone PAM. And, with a higher resolution, we get much higher probabilities over the right area. If we go to 32 kilometers, to 16 kilometers all along, then we get even a much better forecast. So, there is quite a sensitivity. Some extreme events like tropical cyclones, heavy precipitation events can be very sensitive to higher resolution and usually benefit from much finer grids. OK, so the model is evolving, not only in having better physics and improved parameterizations and also finer resolution. We also try to make the model more complex. And, the current system now, we have an atmosphere coupled to an ocean wave model, land surface, vegetation, snow ice. And, in the future, we plan also to have chemistry in the atmosphere, to introduce chemistry, as was mentioned in the previous talk. For the ocean, we plan also to have a triosphere. We plan also to have a river, a modizational river outflow to use it to close actually to link with the ocean model. So, to do a much more complex system or to have the wave on the ocean model completely coupled now. So, one question often is do we need really an ocean coupled ocean atmosphere system for substantial prediction? As was mentioned earlier for signal forecasting, you need it because you need to predict a linear, which is a coupled ocean atmosphere mode. For medium-range forecasting, as I mentioned, a lot of models are not coupled for the first 15 days. The reason for that is that there are very little benefits to couple them because the SST anomalies don't vary much during the first 15 days. And when you couple, you tend to introduce model errors. The fluxes of the ocean seas are often worn by a few watts per meter squared, which means that the ocean is doing it very quickly. And you end up to really mess up more your still scores when you have a couple system than you have not coupled, which is why it took a long, long time at SMWF to finally be able to couple an ensemble system from day zero. And we are the only one to do it now, so we follow certainly. But it's a very, very difficult situation. But for substantial prediction, one of the main source of predictability is the MGO. And there has been a lot of paper showing that actually ocean atmosphere coupling is quite important. It's not necessary to get a good prediction of the MGO, but it generally improves the propagation of the MGO on the still score. So, for example, here, blue is a still score with the old couple model. If you perceive the SST anomalies, green is with the observed SSTs, and orange is with the coupled model. So, even the coupled model beats even the observed SST. And this is the reason that you need this interaction between the upper layer of the ocean with the atmosphere. And what is particularly important for substantive forecast, more than substantive forecast, is actually the resolution of the top layers of your ocean. You need to have a very good diagonal circle of SSTs. And for that, you need a resolution of about at least one meter in the top layers of your ocean. Yes? Well, we did some experiment with the Miss Liar model. First, actually, attempt was with the Miss Liar model, which did as well as the coupled model, actually. For the MGO, for the MGO. But the East Pacific, however, already at this time range of one month, you start still to see the development of an incomplete Miss from that. So, for the MGO, it's the same. But for some regions where advection is quite important, like in the East Pacific, you are losing. And now we are going to 45, 60 days, which is even more crucial to get that. So, I think everybody is going towards actually a full, a 3D ocean model. But it's an ocean model. You don't need to have a big resolution in the bottom of the ocean, which doesn't matter at this time range. That's the main difference with the climate models, maybe. And we are also experimenting with a new sea ice model, like the sea ice model. Currently, we are just persisting sea ice. And we can find quite a better skill for summer and winter than the current system to predict the evolution of sea ice compared to the current system. So, I go almost to the last slide. And what is quite interesting is that we also get some improvement in the still score of Europe when we have a sea ice than when we have a control run. So, there seems to be some positive impact of this active sea ice on the prediction of Europe. And we are testing, the ocean is one degree, whereas the testing next year, we should go to a quarter of a degree with 75 vertical levels model. And when we run it, we still don't understand exactly why, but we get quite a dramatic improvement. This is over the north enemy sphere for width 3, width 4 in winter time, which is probably linked to a much better resolution of the Gulf Stream, which can have some impact. I mean, as shown by the method. The annual sea ice is also quite improved with a quarter of a degree. OK, so the conclusions of this talk is that, once again, I cannot stress long enough that we are still at the end of forecasting. There is still a lot, a lot of open questions. There is no consensus on the optimal forecasting system. On S2S database, it would be quite helpful to compare the various configurations and maybe help to answer some of those questions. That's one of the goals of the S2S project. So, S2S forecasts, you need calibration. Flow-dependent calibration, however, we need much more free forecasts than actually produced by most of the models. So, that's something we have to keep in mind. All forecasts, although another important message, have been proved over the last 10 years. But still, the width 4 is marginally better than chemotherapy. And the models are getting more complex, with a higher resolution, more component on the Earth system. And we hope that to keep this forecasting curve to continue to improve. So, that's all. Thank you.


Ssssssssssssssssssssssssssssss Nass drunk! If you're around my age, and I'm 40, then when it comes to retro gaming, you're probably nostalgic for not one, but two different eras. One is for the games you grew up with, obviously, but the other is that golden period in the early-to-mid 2000s, before prices for picking up those same games weren't completely bonkers yet. I can remember getting stuff like Contra 3 complete in the box for less than $30, or games like Shadowrun or Legend of Mystical Ninja loose for like 20 bucks each.  It goes for a few thousand these days, loose, so it's become a status symbol more than anything else. But that's another discussion entirely. The conversation that I'm more interested in is, what about the actual game? Is Little Samson any good? Boy, it should be funny if it wasn't, but nah, it's a pretty dang good game. It's got some of the same folks that worked on arcade games like Strider and Ghouls and Ghosts, and while it may not share a lot in common thematically with those titles, it does share the same kind of polish those games have. That's probably the biggest reason this game became so popular. It's got that same glossy, high quality, AAA level of workmanship. Little Samson was made by people who were not only big fans of games like this, but it was made by people who clearly knew what they were doing. And they took great care in the details, as opposed to a game where a dev team was merely just assigned a project. Sometimes when you get recommended hidden gems, there's usually something missing or something off that you're just willing to overlook.  What's really interesting, however, is that in a lot of ways, this game is extremely simple. There really isn't a whole lot in Little Samson that brought something new or innovative. In fact, in a lot of ways it's just rehashing stuff you're probably already very familiar with. If you've played Mario 3 or DuckTales or any number of action platformers, and you switch characters and weapons just like you do in Ninja Turtles and Mega Man, but it's just that this game executes the basics extremely well while providing a ton of variety and it makes for a game that's not only well worth playing, but well worth revisiting once in a while. The structure is late out Mega Man style. Where to start out with, you get four different levels to choose from, only each of the four has you playing as a different character. The first has you play as Little Samson himself, the most balanced character who shoots a projectile and can climb walls and ceilings. By the way, I should mention quickly, the game's story has nothing to do with the actual story of Samson or the ancient Israelites or anything like that. It's just a weird Taito localization change they made for whatever reason. Anyway, Kikira the Dragon is your hero for the second level, who can fly for five seconds at a time while shooting a fire projectile that arcs upward. There's Gam, who looks like a chibi version of The Thing. He's the slowest and strongest with the largest health meter, but a really slow punching attack, but at least he can walk on spikes no problem. And finally, there's K.O. the Mouse, who has a tiny life bar to the point of one hit deaths at first, but he's also the quickest, sporting a bomb similar to what Samus drops in her morph ball form. The Mouse can also climb walls and ceilings, and get through small passages that Samson cannot. Once you clear all four stages, you have to defeat Kikiri the Dragon in a one-on-one fight for some reason. The game doesn't really explain why, and it's actually a pretty tough boss battle. But once you get past that, the game finally opens up to a series of levels laid out in this overworld, and you've got all four characters at your disposal, each with their own health meters, and you can switch between them at any time by pressing start. And the level design here does an excellent job of laying out certain obstacles that only certain characters can get past, whether it's making yourself a smaller target by choosing the Mouse for really tough levels like this, or switching to the Golem so you can punch downward against ledges to fight these annoying skeleton dudes. At first, it seems like the Dragon is by far the most overpowered character. I mean, that's always my first instinct, just fly past everything. But the game puts the kibosh on that by eventually rolling out these really tough Mega Man-style levels with spikes, and enemies with Herald Miner's vertical leap popping up instantly out of nowhere, or this really tough part here with falling rocks and spikes where you have to constantly be switching between the Mouse and the Golem. That's the other thing Little Sampson has going for it. This game is a long playthrough, and it's pretty dang tough too. But this game is just so dang fun, flipping between characters and using different weapons that it all seems very manageable. There are really tough levels like this one, with disappearing platforms, there's your token, ice, and fire worlds. Seems kind of generic at a glance, but man, oh man, the graphics in this one look just awesome. This Dragon boss here looks badass. A lot of the backgrounds and settings look fantastic as well. Since I'm here, I should mention the music, and the music is fine, but the levels themselves don't have any tunes, only the character you're playing has, has their theme playing. Kind of a bummer since it's easy to get sick of hearing those themes, but it's not like the music is bad or anything. There's a couple other gameplay elements I should mention. You can upgrade your health for each character, as well as stash away an item that replenishes your health meter, kind of like an e-tank, so that's pretty cool.  Another interesting thing about this game is that there is a substantial difference between easy and normal difficulty. Normal puts a cap on how much you can upgrade your health meter, and you've also got more enemies to deal with that pop-up all over the screen. There's also permadeath in normal mode. If one of your other three characters die, you can't select them again the rest of your playthrough. However, Samson will always trudge ahead, even if he dies. One quick aside here, and bear in mind this is mostly conjecture on my part, but there may be an actual reason this game is kind of rare. It was developed by Takeru in October 1992, a tiny company started by former Capcom developers Shinichi Yoshimoto, who directed the aforementioned Strider and Ghouls and Ghosts arcade games, and he was joined by Akira Kitamura, who of course created Mega Man, and Koichi Yatsui, who worked with Yoshimoto on developing Ghouls and Ghosts. Takeru made a couple games in the early 90s, one with the title Nostalgia 1907, a text adventure made for the X68000 computer and Mega Drive CD, as well as a few other computer systems, but it was a flop, and suddenly Takeru needed help. Turns out that having strong publishing muscle like Capcom on your side really does matter, so for Little Samson, they turned to Taito, and even then this game just didn't sell very well. So while we don't have any actual hard data, or sales figures or anything like that, the circumstantial evidence of a struggling developer selling a game for a console that a lot of people had already left behind, then well, yeah, it makes sense why there wouldn't be very many copies. Again, I really wish I had some hard evidence to go on like sales figures or something, so if anyone has anything, let me know in the comments. So yeah, Little Samson is mostly known for its comically overinflated price tag these days, but it should still be remembered for being a really good frickin' game. Again, there's not much new or innovative here, but there doesn't need to be, because the basics are done so well. It's really fun flipping between characters, especially since there's not usually an exact way to proceed through certain sections of the game. You can get creative with it, and that provides a ton of replay value. And again, the graphics here are just fantastic, some of the very best on the NES, so that helps a lot too. Little Samson has every bit of the same polish as some of the best NES games have, so for that reason, I'm always gonna recommend that you play this one any way you can. All right, I wanna thank you for watching, and I hope you have a great rest of your day.


Hi everyone My talk will target an Audience of just system administrator or network administrator. I'm not a developer that will I believe everyone should understand this talk Not like the preview previously with lots of strange language on the screen. Okay, then I'm Olivia Koshara be I'm network engineer at orange telco French company One of my second role in orange is I'm doing the support for about 300 the firewall PF based on Second job I've got is I'm a visiting lecturer at Ren University during few days by Earth when I give Network lesson, but all based on free BSD On my spare time I am the author of funas, but I'm no more involved to this project since I've given to each system And I'm free BSD port commuter on my current playground is a BSD water project with just a simple free BSD tunes for routine stuff Then working on this project I had to to tune it for networking when I had to build some network Performance ladder I reuse The concept of my colleague at orange and I try to do it But just with software or not with very expensive XR stuff like that Then doing my talk I will present you lots of benchmark stuff benchmark result on about benchmark. There is Very interesting talk from random reg In his talk I didn't attend but I read the slides as one slide which says that all benchmark are wrong And I'm quite agree with this one on this will include all my talk to and I will explain you why First before to speaking about benchmark on tuning. We need to be agree by the unit we will use I'm a network guy. I'm working with networking stuff then the Router have two main job. The first job is forwarding. It's mean he need to Forward packets between these interface the second one is maintaining his voting table with voting protocol But I will not speak about about this Then our main unit is a packet a second unit. It's the only unit you have to use when you are a network guy Second stuff there is an RFC explaining how to correctly the methodology of benchmarking a router. I Try to follow but not perfectly this RFC and I will explain you why later Now that we are agree about the units You need to understand some reference units on a gigabit interface The maximum line rate is the one hundred forty eight frame per second We use the The term frame because frame is for a standard layer on packet just on the IP layer But in all my bench, I'm using small packets and one packet is one frame. I will use the term packet per second on another big detail that sits almost everywhere forgotten. It's a gigabit Since gigabit is a net there is two wire one wire for transmitting and one wire for receiving It's a full-duplex media then Terrorically you need to do transmit and receive at the same time This means when you are speaking about line rate gigabit router Your device need to be able to forward at three million about three million packet per second when this is But this is about a line rate router But on 10 more for 10 gigabit router, but who needs technically speaking line rate Then I need to introduce you another concept Which is how to translate packet per second with boundaries It's quite easy You just multiply your pps with the packet size, but on the internet Each packet has the same size Then for simplify Long years ago, they choose distribution a standard distribution which is for one big packet You've got Four medium packet and seven small this called a simple internet emix on using this Distribution from your packet rate. You can Estimate or give some value using a standard reference about the equivalent in bond with rate But I'm network guy. I'm not using the One with the IP layer then I'm using Bonn with as a recent layer I need to add the internet overlay only in my slide if you will see some table Referencing one with it will be as an internet level Now about who needs line rate gigabit or 10 gigabit router with these two concept packet per second on this distribution We can Estimate what is a minimum rate using the simple emix distribution? For reaching for filling your gigabit link or your 10 gigabit link at full duplex And this is the seven key recall 700,000 packet per second for a gigabit water on seven million for a 10 gigabit water on my lab I Need that all my device reach at minimum this value for be called just a simple gigabit water or 10 gigabit Not line rate, but a simple gigabit Now how to I how I did my bench We are a telco at a ranch we don't we are not we don't sales hardware then I Need to take care of my customer. We don't have any problem We have a telco. We don't care about what the customer will do with our link Then if I install a gigabit water I need to take care that this water Teorically can receive the worst case possible. It's been a line rate traffic using smallest packet size on this Why my bench are different from the earthy? My bench. I'm generating the worst case ever In one side on on the other side, I'm waiting and I'm measuring how what's the rate during this worst case It's very different of the of the unit you will see and for some material of the manufacturer Now on my lab a little more details on my hardware lives. I will use for all my talk I'm using I'm reusing all stuff I get in orange and I will do lots of test my biggest server is a Xeon a core with Hombedded a mulex card 10 gigabit card on the Chelsea o-card to give it I've got two super micro with eight core atom One with Chelsea on one with until this is great because I can compare the behavior between Chelsea one until on two small One that gate on one PC and Chinese IPU this for my gigabit World I don't have melanox. I want to test them. They are very good review I don't have for them a month sex career one circuit CPU and I don't have multi-circuit CPU Because but this is can be interesting too. I'm about to detail all the SFP 10 gigabit SFP are the same everywhere It's not a matter No before tuning a freeBSD or every kind of server There's a first concept that you need to understand is how Today a standard Network card is working on a freeBSD Today when your driver loads the kernel give the information to the network card how many CPU you've got And with this information The network card will create a multiple queue with the maximum value depending of the driver Then each time the card will receive a packet. He will use Hushing in this packet on each packet belonging to one flow will end into The same queue each time. It's a very important concept in network world none too For the whole packet belonging to the one floor need to use to cross the same path Then from this first basic concept now We can have two very important points Like I see when I read the freeBSD network mailing list that lots of problems come from here the first one is you need multiple flows if you are If you are using just for example IPR for just one IP source or one destination IP source with the same TCP It's it will not work. So you cannot use this correctly On if you are using some stuff like IPsec or GRE You will break this concept to has endpoint on your router or firewall. You will break this concept to and lots of people forgot that Has example in France. I've got lots of people using a PC Engine's IPU with a BF sense on their home fever, but the high SP which is orange For them to use PPPOE and once I did that They don't have any more multiple flow They just using one core on the performance of a very big impact. This is a very important step You need to to understand So second is what today is a CPU is a physical core logical thread or what about new map? This are some question I need to answer Then for helping you to before to try to tune You need to ticker need to check how many what the current Q How many Q's the traffic are spread between all your Q you've got this information if you check We see settle and free busy if you get the network statistical information They are not very easy to pass then how is the script from young Lex? Which gave me which a piton script who allow you to to in real time to see how many the cure are It's very important stuff for for network guy And then you need to check this step once you have you are sure that you are using multiple flows We can start to to work together second step is about What is the core today? How is an example just on my Xeon Hacker with a Shelshaw card when I boot my server is default mode You see 16 CPU eight peer to trade The Shelshaw by default driver choose to create only 16 on maximum of 16 Transmit Q one for its CPU right, but only eight with you of Q. I don't know why eight. Okay. You choose to eight then here I said, okay. Here is my first bench now How I'm doing bench I'm creating three configuration sets one of the default value a second one where I will force my Shelshaw card to use the 16 Q available CPU available and the latest one where I just simply disable it per threading My test I'm using of all my talk. I will use the freebies the 11 at one release All then I all my script are benched. I know you mind interaction then I will push a configuration set I will do a benchmark with picket again at the end of the bench I will reboot again, and I will do it five time with a reboot between each time All these benchmarks is very important for statistical analysis At the end I will have three text file for each configuration set on each Each this three five I will have five five value for my file value of my benchmark on now a little bit much With this three configuration file and put them into mini stat mini stat is a great tools If you need to do benchmark, it's a mandatory tool to use, but it's very monetary and he will analyze Your different input sets and you will check if okay if it's very you really have an impact or not On this example on the right part for example, you can see that by default. I've got the worst result It's on my left side and I When you can see with a distribution, okay, there is no arguing about it's very worst I reach about four to six million packet per second if I force my car to use a 16 CPU I've got a little improvement, but if I just these other hyper threading. I've got the biggest improvement then Okay, the first thing is okay You need to do some hyper threading because it does help you anymore on this case On this use case because you could have lots of interrupt on threading don't help you to manage all this interrupt The first the second news. It's it's a bad news This device is my biggest device when I sell to my boss. I will do I will create you a 10 gigabit water This means seven million packet per second and I've got only five that seven Okay, I need to find a solution I Got a second question once I get this result was Perhaps if I will had some core a bigger server I will solve my problem But before what's the relationships between the number of Q then of core assigned on the performance on my server then I've generated eight prefixate I forcing my Shell so far to use just for the first just one Q and one CPU on up to eight CPU On surprise with a reference blue line rate. It's a linear reference We've got a problem. I can't had CPU, but you will not solve a lot of my problem There is some stuff some big difference is does the scale today Okay, I'm a simple and users how on free BSD do you dig this kind of problem It's quite easy. You just need four common line on this kind of four common line. You generate some stuff called from graph plan graph it's Give me the How to say the hot function? How to spend how to consume the CPU time? And from this graph I can okay I see first for interesting point the first is Okay, that is a lot contention about the first function with this code IRP resolve This function is used just for adding the destination MAC address to my packet once there are forwarded So second one is a routing table Okay, I've got only two static route on my bench I try to do some stuff using a true No change then but this first two I have and users I can do nothing about it There was a second one called a random RFS queue Hmm, I will dig about it later on the last one is you see the all the part building to the Isolate pass the kernel is an ad pass on the next driver Okay, what can I do? I will try has a end user just to play with a random RF source on the free BSD as Entropy source by default interrupt on or isn't it packet I read some data are taking from this packet For using but at this very high to write this impact a little bit your performance Then I've just choose to disable Interrupt on isn't it packet frame from my configuration I've got it's just with this command line You change the input and then I generate a new bench, but in all my server in my lab And this is a result with all my server Bad news on my big one. I've got no difference at all I've got some improvement with my eight core atom a little bit my Small gigabit router see some improvement to First conclusion here is that about okay. I've got two small gigabit router Who are correctly able to do gigabit forwarding for my point of view? Second one is okay. I still have a big problem regarding my 10 gigabit water Then how can I do as a simple end user? It's very easy just call Google on you will found that this are known problem. There are already fix since few years by a crazy team from beyond X There are already have patches about that on you just have I just have to take their patches on to apply it to my Free BSD. I just backported these patches one for the two look contention. Okay, let's try Then I keep my harvest mask modified. I apply this patch to my labs. I know her is a new result. Oh surprise Almost twice twice more Performance on my router which now okay my manager will be happy Second impact is go. There is a 10 gigabit Nick into this same server, but it just reached 1.33. It's even not a gigabit line rate This card. What's this card? Okay? If you dig a little bit you see that these cards you can configure the number of q you cannot configure flow control. It's a very Bad nick Then you need really to take care about the new card you will use Shells you are very good until to I Didn't have Melanox for proving it, but they are very very good review by the Melanox one Then really try to to use good network round Oh, no with these patches. What about my linear performance rate? I did this bench again and you can see that the green line is Is lots more better than previously Now perhaps if I can add some core and perhaps I can have more performance the second one is quite funny because Between four core on a curve is a six core, but this does not scale It's like we need to have a power of two number of core. I don't know why I didn't dig this But it's a funny value. I was waiting more for my six core in this example Now, let's how can I try to improve more my informer? I need to to works of improving the performance perhaps Regarding the Network card on the driver of the network cards themselves today driver are using lots of feature Then I try to to look the main page of the driver. I said, okay, I am a bit lots of checks and stuff On there a bit Two feature that are very great for server for endpoint server Which is called TCP cementation off-load on large receive off-load But these two features are very bad concept in a routine world. I believe even in the firewall world Then This is not technical for a plus technical but more philosophical choice I don't want to have these two features Unable it in my lab when I try to read the main page then I choose to disable this feature in all my lab and I try to Generate my bench again for checking if I don't have a performance. Okay, this confirm There are no performance impact at all About this this challenge then really under firewall or router you should disable these two feature the second one on this slide You can see that's on my hate core Atom I've got a quite difference between my shelsio card on my entail card About one million packet per second Then I try to look a little more how I work entail card. What can I do in about the entail card? I did a lot of tests. I died long document about Unterip management about entail with lots of contours I'll try lots of things but Like I say all benchmark are wrong on my use case my bench is very specific With my line rate of packet when I don't want to impact just my very specific benchmark Or not a standard use case then I didn't Works more on the entail driver But at the end I found just one parameter that that matter This parameter is by default Gigabit entail card or 10 gigabit have a maximum number of packet the process Per per what per time per I need to dig this one And if you remove this limitation if you just say okay, don't Don't keep a high value about the number of packet. You can manage it in one times. You can you found a little improvement And this is okay. This is only by checking how the user point of view and the driver This is the only stuff at all I can find Then I speak about with other on the same event what about Queuing painting your multiple queue to each CPU. It's mean you you prevent the scheduler to change the CPU of your IRQ I Check the driver, but all almost lots of driver already by default Pins the queue to the to each CPU with an exception of Chelsea, which is interesting. I'm using Chelsea Is this function, but it's not by default on the generic kernel is not enabled then Again, you indexed give me a script or was there already did all this stuff And I say, oh, you can perhaps try this script. It's a versus script when you force the binding. I try it There is not lots of difference It's just you start to see benefit just with very high throughput more than 10 million packet per second For example, I might a core. I've got a little little benefit 1% It's why it's not enough to to take care about on my Atom a core. I don't see any any difference at all. Thanks. Okay. It's not Francis use case is not very important Then, okay, I believe just for rooting point of view I which has a simple user the maximum I can reach this mean, okay If you want to use freebies the other router, you really need to check the patches from the index or for this team There are online or there are people that freebies the page if you know the name or you need to disable hyper threading The harvest mask need to be fixed to And there are working progress in this direction. And if you've got yes on TSU LRO, you can do it There's no impact on the performance On an Intel card you try you can try to play a little bit Then if I raise you just for forwarding Before the tuning on my patch on after we can found some okay a Good improvement on the Xeon 8 core with a service you a little less with the other hardware On this mean, but okay. I reach my goal, which is okay I've got a Xeon 8 core which I can call 10 KB router My 8 core Atom don't reach this statues. Okay.  This is it's a pretty small code machine But I speak only you but IPv4 But in the wrong you are doing IPv6 then let's just Again Yondex made for me a patches to packaging it back packaging for adding IPv6 Support on I can generate IPv6 packet in place of IPv4 surprise ouch Okay I've got a quite big impact Hopefully I'm still Over by seven million packet per second, but the impact is quite It's quite large on funny things here on the shelves in the 8 core atom There is no more difference between the shell see on the Intel because it's a kernel who who lose your all your Performance is on the canal not on the price on the driver anymore Okay, this is the impact with just IPv6 We need to work a little bit on this part Then bad customer have Use standard course setup Difference and mine one of these for setup is up lots of them are using valentagging Um, we are using at orange another setup, which I'm calling multi tenant rotor Which is using vnet jail then I want to check the impact of these two configuration set on my configuration For valent, I'm just generating okay my standard reference config without the land on the second one with valentagging It's just a very simple difference On today is a network interface if you check them they unbeat lots of valent accelerating stuff Then terrifically They should be have no impact with my pinch surprise again 17% of degradation Okay, that's a lot quite large impact on this it's but Again, it's a known problem and again young dex use already a patch. It's a long time on their server by this problem, it's it's regarding the Other free with this tack when he's got a target frame. He used a long pass for They are encapsulating this frame and once you remove the valentagging He takes a frame on he sends them back to the beginning of the ethernet for for a standard It's quite not very efficient We've got a big impact on here then okay lots of customers who are using the land tagging perhaps would not be happy But I'm still over my seven million packet per second. It's still okay The second one is my multi tenant router. I need for that by default you need Vmage It's for the virtual network stack into free BSD. It's not enabled by default Then before to create a new configuration set I need to add this option in my kernel Then if I had to add an option I bench it on the result with this option enabled is here in I lose some performance in IPv4 and I ring a little on IPv6 But it's very small if you compare to to the other bench with the valent then it's it's okay It's quite okay 6% is okay now The net jail On my server, I will move all the IP configuration When I move them into a venet jail This means has no more IP address for my on my hardware my host all the IP configuration Into the jail one I can play now I check I play on the PC on giants IP u2. I play to generate I don't remember it's between 100 on 150 jails venet jail on this small hardware. It's very fun Then what's about the impact of this setup? improvement degradation It's quite No impact at all It's quite very very very good surprise Venet now I Tried with firewall IPFW on pf because it's supported to On the in the same as no impact at all. It's quite once you enable the image. It's quite a very good good use Venet jail works very Okay, no, okay. I'm Finish all my forwarding bench. Let's start playing a little bit with firewall Okay, I will show you a graph After all this slide my first step is just to enable firewall that's we have three firewalls in free busy and I want to check How hard just enabling this firewall Unpacked my performance The next graph will I will not compare the firewall. It's very important Destination I will just compare how by the impact my routing firewalling Then okay, all bench are wrong then don't interpret this benchmark by Saying that's Pfsx in front of IPFW. It's not what this bench mean. It just means that okay On my Xeon 8 core the reference on the left is my forwarding rate at EPV4 and EPV6 I generated Stateful on stateless setup is a minimum number of rules one or perhaps two for IPF the old one And I just try okay. How about this impact? Bad news, okay If I want to sell a 10 gigabit firewall to my manager, it's start very bad Perhaps IPFW in stateless mode. Okay, but about the all the others It's the impact. I don't compare fire or just the impact is quite a bag No Let's dig a little bit about what is technically speaking a firewall Let's do a next bench with in front of my Hello all rules my unique hello rules I've had lots of denied rules with an unexisting IP and I love that try to play Okay, let's try it with one ten one thousand one hundred Rules You can see Still on Now I switch my hardware. I don't anymore use my big Xeon 8 core. I use my smallest One of my most it was a negate. It's a four core atom Why because like I said, I'll do a reboot between each my bench on the reboot and a big HP server Takes a very long time about five minutes on my bench is about one minute on its It's take weeks to do all this bench But the negate on IP are very fast to boot on I'm was I'm more interested by the impact or not by the full value Then Harry's we can see that IPFW on EPF Our linear relation between the number of rules and that break their performance very very easy Then the first rule is you need to keep the minimum number of rule, but very small at 10 You start to do to hit your performance just 10 rules The second one what's it's my pf. What are you doing? He don't care about to find my rules Then I dig a little on this tricky guys Covert all my rules into one table then all bench are wrong. This is a wrong bench I'm comparing April on pitch because once is using table or not the other one. Okay, you want to play with table Let's play with table For this next bench. I just I remove all my deny rules I just have one denied table rules on one allow all rules when I try to increase The number of IP into this into this table on okay using the same Feature we can see that we are agree Really use table if you can really it's very important. The impact is is huge Really really choose Other stuff is a good thing exact pay F It's lots of lots martyre by default because if you will do that in your place Then you can save your brain for doing other stuff Now I play a lot of with stateless Now let's play with stateful setup on my FW stateful. I try to dig a little about the default configuration setup of stateful IPF and if you check if you my packaging generate only UDP unidirectional flow and if you found one flow IPF create one Dynamic rules. It's one states on IPFW If you check this by default he accept a maximum number of 16,000 rules When you've got done it's for this size is related to the hash table size It was not very clear in the beginning when I start this I discover what is a hash table size and Then I try to increase this hash size tab to the maximum allowed by IPFW with this 65 Thousand it's a power of two and then I estimate the number of potential Keeping the same relationships The perhaps with this value. I can push my IPFW to manage four millions UDP flow without any problem Then let's try I generate lots of multiple flow by using a large source range or large destination range of my IP packet and we can see that the you can hit starting at about 10,000 you can you can hurt a little more of your performance on the one million It's totally out of Then I cannot reach four million with a IPFW Okay Just for information when I show this diagram to the young next team the next day I had three patches luck less The crazy guy, but okay, I need to do it. I didn't have to didn't have time to do it for this for today now with PF PF have a different behavior for one floor PF consume two states one for each direction On There is a linear relationships and PF have the same number of state limit like pay on the same hash table size Then on my small net gate device. I've got 8 gigabit of RAM then I try to Increase lots my hash table For using the maximum value I can which is about a 2.5 gigabit of RAM use for this hash table on this mean about okay, I can I can reach 10 million states then 5 million of UDP flow in my previous with my bench on Here is the impact with PF Okay, he does not care much about the number of states Then really if you want to use the state food firewall with complex rule, okay This why you know now we perhaps use PF It's quite a very good a behavior for this for this kind of stuff No, let's back in 2002 When I present for the first time my BSD water project I Complain that we don't have MPLS feature and has a telco In a run you are using free BSD as a firewall because it's a no go if you don't have MPLS support on Netmap was existing in 2002 and I was waiting for okay Where are the software the user on the software who on the free BSD using net map for routing stuff? now today still no MPLS but the concept of user space forwarding is Is a way to go today if you want if you check today At orange for example Preview this solution when I speak with my network Colleague if I want to install a software router, it was it were no go I cannot talk with them about a software based on Linux or free BSD. It's a no go But today they are removing all the Cisco or Johnny Perth stuff and they are using software router but based on Linux because it's Cisco fdao and dpdk server and on this if you want to compare Then we need to reach 12 million packet per second per core. I didn't even reach this with eight core on this is It's quite Very interesting gap To to reach on this why why not of people ask me? Hey, can you add to your benchmark? Can you comply to append BSD or can you comply to a Linux? It's I'm not really interested by that once because Like you see I will never compare a standard Non-tint free BSD with another stuff because it's really crazy Then I need to learn how to optimize a pen BSD and a BSD a dragonfly and I haven't half time to the for that because You need to compare April with April then if on Second tough if I want to add a Linux in my lab. What kind of Linux I will have what distribution? Then I know my my only interest is okay now We've got a value a reference value to reach on I prefer spend my time to help people to reach this unique value We will see after our free BSD there were from net gate Software called net map forward. I Don't think this is maintained in your own Github. I don't found I don't know it I didn't reach it to to use it On I don't see lots of core of lots of code updated to it This way I'm I'm saying that no production ready software on free BSD This is a shame because we are losing a role really it's At last I'm very happy to see my Cisco or Johnny Perth stuff replaced by software But it's a Linux. Okay But all is not lost if you Can help space warding is still improved and you say young decks have a lot of longer lists of patch very interesting It's an old project called project routines lots of old ideas that we need really to refresh The real talk last year about how to improve IP for adding fast past an interesting talk because reading this I discovered that User space forwarding have some problem. I was not aware about and Yes, perhaps Okay, I believe we can still perhaps place this this game again on about my All my bench script my all the script I've used for all this stuff all the configuration step I use all my result all the Flamm graph about when I generate my graph After I generate a new bench just for generating Flamm graph then for each difference It can help developer on the lots of the script you can it found on them on the busy Web page or get a page are all all online Why because I'm very afraid when a free BSD developer send you a patch Force ask you can you test this patch, please? I believe it's improved I'm very afraid of doing a mistake in my lab or giving false value to a developer Then this why I want to lots of more people reading my script on my configuration stuff or my way of doing bench because it's quite stressful to To give false information of bad information Okay, now do you have any question? Yes Is it possible that this is because of the association between course and I guess by design you will have even number of cues So if you use like an odd number of course, you will have a mismatch Is there a developer who can answer to this question? Can you bring the microphone? I Know just the very same and is actually any power of two Yeah, but until selling some six core CPU then try to avoid them on your water This is an advice I can give but I don't understand this. I need to work with the developer for that. I'm not Okay Ah It's mr. Nedgate We basically stop work on that map or because There is work underway If he's Then you are working on the user space forwarding using the PDK on free BSD Great Great, I'm waiting this very do you have a schedule? I Are the patches from your decks will be merged I Don't know if it's a Russian culture or not, but they are very Each time you ask that I say my patch is not ready. It's not clean enough There are perfect tonists person, you know then because I'm not a developer My role is to push them on to do some Marketing stuff for them for trying to push this patch. Then yes, I'm using is their patch in my protein with DRP I'm testing it on the real field or now I can push them for saying, okay, I Can give I can say okay. I use them. They're working then you need I push them to the free BSD project You can comment about what you said about TSO and our for your workload They should not matter at all because you should never receive a TCP flow in a router. You never go above the IP Yes Yeah, but you can turn them off But if you don't turn them off if you keep them It's a TCO TCP all your stuff the firewall have IPFW for example the libalias the net library does not work Then it's it's it's in the main page on user of problem with that Then my advice if you don't have it if you don't want to have any problem. Yes on the router Yes, it's it's logical. We don't have to Have negative effects on the rest of the system The other thing is that What Olivier is looking at is performance, but not always correctness So they have a cash is often need a bit of they're right They need to be fixed so that they don't just remove locking they still need the correct Oh, no, not you The problem is when I did my bench I did a third one by disabling the acceleration stuff But it's not a correct bench between before We need to fix this villain and we need to fix this villain Problem into the candle pass on the ones. I will test this patch I can now generate a new bench without the hardware acceleration for Real point of view about with or without Currently because the kernel pass already slowed on the rate You've got a worst result with this feature of this a blade but But it's not very worse because it's a kernel who already break your performance Then I will do this. Yes. I will check with the feature on with your feature off Once I will check the new patch from beyond X. Let's fix this problem Yes, and the villain and you had The number of the villain into the packet On this number when you receive a ticket from then the network have to remove this This tag for example, and this is how to on just the number of villain Is that you need to remove this this tag on use this tag for the virtual interface Holding it's a very small Action really technically bad. It's it's matter Yes, hello, hello Packets of what out of order delivery No, I'm using for the moment. I'm using that map picket a gen which is a very dumb Very simple tools just it's Daniel of service tools if you want if I want to measure Out of order packet. I need to have a more smarter tools that Will read all the Index in each package generated or we'll compare for that at all You've got X here, which is very expensive hardware But I don't have it for the moment in my lab But I need to check as I the Cisco stuff based on DPG key called T-rex and I need to check this software for For for this kind of more advanced performance test. Yes, really I need to do it Just behind you Really From the kind of space it's No, because I don't have Jim perhaps Can you get a micro to Yeah, but we need to fight against this solution Thanks for the talk, I just wanted to know about the Simplicity of begin a gen and you mentioned fighting It's actually surprising to say but you can use fighting and fighting findings for a net map to give some smartness into that map and and for example like for checking off the Of the ordering stuff like that multiply with that by site and you will give You will have a decent performance right now. Oh, what did you find this? Well, I'm from Intel and we're doing you I'm a network software engineer. I'm the guy that a zone drivers. Okay, we're talking about and yeah, we do use net map for different things like for for example for by the way flow control testing. Okay, and Yeah, we use that API to integrate into our Python written tools and they well I just you know, I thought it would be a huge performance drop when you use fighting to manipulate the reins But wasn't really it was it was decent You're creepy does your script is online? No, it's internal open source. That means that it's shame shame on you Anyway, you can here's your sleepy Thanks


My family upbringing really did influence my whole career journey, even though I didn't know it at the time. I'm very lucky to have had an education. Things are different today. I see that with my own children. But I still think those essential elements are true. Move forward and pay it forward. I got there through perseverance. I have to tell you that when I got my PhD, I didn't really think about a research career at that stage. So my first specialised job was as a dietitian to the Hunter Region Cystic Fibrosis Clinic. I really wondered whether what I did was working, whether it really made a difference. And my boss said, you can answer that if you do research. I was pretty determined that the best way I could make a difference would be in this area of research. The work in Cystic Fibrosis has had a lasting impact, both in Australia and around the world. We kept those kids well enough to benefit. Many children have lived long enough to receive these therapies. I can't remember whether it was four or five times before I got my first NH and MRC fellowship. The key thing is not that you didn't get it. It's actually that you get back up off the canvas when you've been knocked out. I've graduated now 38 research higher degree students. And for me, I think that's the best way. I can not only build an impact, but leave a legacy. So my research in precision and personalised nutrition is marrying the science of individual metabolism and using technologies to personalise nutrition advice for you. It won't get finished in my lifetime, but there's a lot of attention in this area globally and I can't wait to see what unfolds. We don't know all the answers yet, but we will eventually. And that's why doing the science right now is so important. I'm Laurie Professor Claire Collins and I'm a very proud University of Newcastle alumna.