You probably already saw on the program. But tonight we have the reception in the area here. So I think we've got it down for 6. There'll be some drinks and snacks and food from 6 onwards tonight. So we can all get to know each other a little bit more, a little bit of a icebreaker. Why I want to do now for the next hour is, I'm going to, almost take a step back, this is going to be a very basic introduction to forecast systems and their set up. So, in a way it's almost a kind of, I do say a pre-introduction to some of the things that Frederick was introducing this morning. Mae'r bwysig o'r adroddau yn rhan o'r rhan o'r cyffredinol. Mae'r bwysig o'r adroddau yn gwneud o'r gweithio'r dda. Yn mynd i'n cael ei fod y ffordd o'r ysgrifennu i'w ddweud i'r hyn o'r masygar yn ddweud i'r cyffredinol o'r ddod i'r pwysig o'r pwysig. Felly mae'n holl o'r ddweud i ddweud o'r ddataeth. Mae hynny'n gwneud o'r cyffredinol o'r syddai'r systemau. siwydd y gallwn gweld o hyncyffwys, ond y llai, mae'n cael ei ffix, mae'n cael ei ffix, mae'n cael bod yn cael bod yn gweithio'r cael dŵr. Felly dyna'r parwysydd gwaith yw'n cael ei ddaeth i chi, ac mae'n sgwr i'n fawr i'r ddaeth i'r ddaeth i'r ddaeth i chi. Mae'r rhaid i chi wedi'i'r rymlu yn ysgol yn ysgol iawn, i'n gofym i'n cael y rhaid i'r rhaid i'r ddaeth i'r rhaid. I'm a native speaker, as you probably noticed, and I sometimes speak too quickly. I will try and keep slow and clear. If I get too fast because I get excited sometimes, as you saw this morning, put your hand up and slow me down. I tell my diploma students to do the same because they're used to me now. Sometimes, don't be afraid. Tom King's, calm, deep breath, slow down. I already talked about some of our programmes, but I've left the slide in so it will be on there for reference as well for shortcuts about some of our programmes and some of the things that we do. Frederick already showed a plot like this this morning, but I'm going to start there again to emphasise where we are. This is one of these. If you Google, you find 100 different million versions of this kind of charts. They use them in all sorts of fields. You find them for ecology. It's amazing. Evolutionary plant studies have them. They run from paleo, climate and so on. This is one that's applicable for our field for metrology. We have a spatial scale across the X and a time scale across the Y. Essentially, what the chart is showing is different kinds of... The colours are not showing up very well here, but different kinds of basically phenomena on the time scale in which they occur. Some of the things I was working on in my own early days were the dynamics of thunderstorms and so on. The point we have to remember, and Frederick already mentioned that in passing quickly this morning, is that all of these with their time scales are associated with a time scale of predictability as well. For example, for evolution of thunderstorms, we're really talking about short time scales and now casting. When I'm on my bike like this morning and I want to cycle home, what do I do? I look at... It doesn't rain so often in Triestas where I come from in England, but I look at the radar and I use my brain to make a now cast forecast to extrapolate the rain to see when exactly I should leave my office to make sure I don't get wet. That's a forecast. You can also purchase software that does it for you, takes the images, moves them forward in time and works out when they're going to pass over where you are. That's now casting. Of course, if we start to look at the forecasting systems like ESMWF for the next few days ahead, we're not going to be able to predict where the thunderstorms will occur, even if we have the statistics correct. If you look at the ESMWF forecast for three days ahead, you can tell if there's going to be a chance of thunderstorms, but we don't know if they're going to arrive here or up the road in Montfacony. Because then all you have is the smaller scale processes essentially are, shall we say, stochastic and you can only predict some of these statistics in the envelope. If you go up to longer times on the days, you're basically now the larger scales such as fronts. Three days ahead, as Frederick showed very nicely, we can predict the passage of fronts but the smaller scale phenomenon, we're only really able to predict the statistics. What is this workshop about? This is these next two tranches, so basically sub-season out to this interface here. Frederick was talking about some of these phenomenon this morning where basically on the month to multiple months timescale, then you can predict larger scale, for example, evolution of planetary waves like in the tropics of Madden-Julian oscillation, so large scale waves, but some of the things such as cyclones, you're not going to be able to predict their precise position and evolution and genesis on those longer monthly timescales. So that's again why he was picking up these phenomena. So during this week, we are going to have lectures, for example, from Frederick Cicharski introducing the basics of planetary waves. We were going to have lectures on the MGO and moving out towards ENSO. Because, as Frederick said, we're really concerned with the sub-seasonal out to the borderline season, the first season. And then, of course, we moved beyond that to the decadal initialised prediction, which was the subject of last week's workshop. So if you're interested in that, you're in the wrong week. You arrived seven days too late. So, as I said, that was just to emphasise what the target of the workshop is. So the things I'm going to very give you a basic overview of are the things that we need to consider. So ensembles, why we need ensembles, initialisation and hindcasts. And there's going to be more detailed lectures during the week, so this is only an introduction. So Frederick, for example, is going to talk about forecast initialisation in more detail. So that's just an overview again of the aim of this course. So to introduce a sub-seasonal phenomenon that's in week one, such as the MGO, give an overview of the NWP systems that we're going to be using to emphasise the use of the new S2S database, how to look at the web interface. I think we'll show this to you briefly in a demonstration this afternoon. And then tomorrow in the lab we'll be looking at this. And then with Powler especially in the labs, we're going to be showing you how to retrieve with the Python scripts. So this enables you to do it from the command line. It gives you a lot more power, but it is a little bit more complicated. Most of you are familiar with using a Linux interface, maybe I think, no. For those of you who haven't had that much experience, don't worry, we will go over slowly and we'll have the chance to actually perhaps have some small breakout groups for those that need a little bit more instruction on that. Powler is going to introduce the database, as I said, and we'll have a little bit of introduction to the reanalysis, an error interim in the afternoon. And then week two is the application. So now I'm going to talk about in particular uncertainty and how that relates to the way these systems are set up. So as I said, I'm starting from something really basic. So NWP systems, they're basically made up, underlying these systems are a set of fundamental equations. So we have the equations of motion, equations of state, thermodynamic mass balance and water balance equations. So it's important to realise that these underlying equations are basically derived from first principles and they're well known. And we solve those equations on a grid. So when we talk about model resolution, what we're essentially doing is dividing the atmosphere up into a series of boxes. So we can imagine this room, we want to model the atmosphere in this room. We divide it up into a grid, a grille. And in each box we assume that the atmosphere is essentially a continuum. And so we have just one number that describes the properties of the atmosphere in that box, such as the temperature, the water vapour, the velocity and so on. So the key thing to realise is that in each box these properties are considered to be basically uniform. We have neglect the sub-grid scale variability in most models. That's not true for all models. One of the works I was doing in the ICAM model was to also introduce prognostic predictive equations for higher order moments, such as the variability of the water vapour within the boxes. And these equations are integrated numerically forwards in time. Step by step over a time step. So, for example, the ECMWF system now, the highest resolution, is probably using around a 10-minute time step, is that right? So it's 10-minute time step. So basically you step forward 10 minutes in time, integrating the thermodynamic and dynamic equations. So how large are these boxes? Well, it depends on your problem. Of course, if you have a global model, then you have to have a lot more boxes to cover your world compared to a small-scale regional model. It depends on your problem. So if you're only forecasting for the next three days or you're forecasting for the next 10 years, again, that takes computer part-time on your need to change your grid box size. When we start to talk about sub-olds, if you want to have many forecasts, again, it's going to affect the affordability. So these were some of the choices that Frederick was alluding to earlier that we're going to talk in a little bit more detail about. So just to give you some examples from the two extremes, this on the top is showing you the progress of the climate model resolution at the various IPCC reports, starting from the first report right through to this is the fourth report. I haven't got the fifth report graphic here, but in fact, the resolution between AR4 and AR5 wasn't that much different. There was a small improvement. It tended to be that AR5 increased the ensemble size. So you can see, for example, in Europe, we're starting to resolve the difference between France and Britain. That's Frederick's from France. We can make lots of jokes about the English Channel and so on there. Okay. And if we then go to the NWP end, so this is kind of the recent upgrade about, I think, four years ago at ECMWF, moved from T799 to 1279 over here. And this is showing how basically the state-of-the-art centre like ECMWF resolves the topography. So most of the S2S systems are kind of sitting in between these ranges, as we said. So Frederick had a table of the resolutions. Of course his S2S is here. The finest S2S is bordering on this kind of resolution on the left. Okay. So what's the issue concerning this finite grid scale? Well, of course, as we said, in each of these boxes, everything's considered uniform. So if we have a box size that's on the order of 50 or 100 kilometres wide, you're not resolving any of the fluctuations that occur on smaller scales. Okay. But of course we know that the weather that we experience from day to day, thunderstorms and so on, is essentially on scales which are much smaller.  Mm? Yes, exactly. So on the order of magnitude of a kilometre or so. So you've got these fast updrafts on the order of magnitude of a kilometre. If you really want to resolve the process as well, you have to, for example, the mixing into the updrafts, you need to have resolutions even much finer than that. Otherwise, all of your mixing between the updrafts and the environment is going to be handled anyway by subgrid scale schemes for diffusion and diffusion mixing. Okay. So if you can't afford that, you have to represent these kinds of scales of motions in convection, cloud micro-physics processes, and the interaction with radiation by essentially what's called parameterisations. So these are small simplified models that take the information that you know on the large scale, so it might be the vertical stability profile, and uses that to describe the statistics of the small scale that you can't represent. Okay. So a lot of my work at ESMWF was focused on developing those kind of parameterisation schemes. How can you represent the statistics of what's going on on the small scale by what you know on the larger scales? And it's very important because what happens on those smaller scales feeds back onto the larger scales. Okay. So just to summarise the kind of things that you have to parameterise, deep convection, shallow convection, cloud micro-physics, and interaction. And you're never going to get away from this. No matter what resolution, there are always going to be processes such as micro-physics that are going to occur on smaller scales. So even using one of my PhD students at the moment, we're looking at a convection organisation in the tropics with a model using two kilometre resolutions, but the way that the convection organises is extremely sensitive on the assumptions that you make concerning the sub-grid-scale mixing, so the parameterisations. Even though that's considered to be a cloud-resolving model and it's the same for the ocean. Again, you've got sub-grid-scale processes that are not represented. Okay. So why am I worried about parameterisations while they're not always derivable from theory? They may contain ad hoc assumptions, particularly to close the equation set, and these parameters might be difficult to measure from observations or to derive from theory. Okay. So the result, which I'm highlighting in red, is model uncertainty. Okay. So a lot of the uncertainty that we attribute to models is not on their fundamental equation set. The equations of motion and basically the momentum equations. Okay. The way that you solve them are our differences. The way you solve the advection equation is accuracy and its stability. There will be differences between models, but a far greater source of differences between models is often associated with the representation of the physics. So just taking the climate models, for example, cloud parameterisation was a big source of differences. This just emphasises that these are all different models that contributed to the latest IPCC report, for example. And the top line here is showing basically the net and the short and the long-wave cloud sensitivity. And then these are the clear sky feedback. So you can see there's a lot of agreement in the clear sky processes. If you look at the cloud, they're all over the place. And this is the result, not just of the cloud micro-physics, but also the convection, the turbulence processes, shallow, deep and so on. The radiation assumptions, the assumptions about cloud particle properties. Okay. So we have basically an uncertain model. So why or how is that going to impact our forecast? So let's imagine in this schematic that we have some state x of t and we're trying to predict the future at some time in the future at x t plus t. So this might be a two-week forecast. We're one of the S2S models. So if the observed state evolved in this way, we might conduct a forecast and the forecast might diverge over time from the observed state. So what are the reasons why we're going to have this divergence? It's because we have an imperfect model. Our representation of the physics is not perfect. But we're also going to have our divergence, of course, because there's uncertainty in the initial conditions. So how do we account for that uncertainty? I mean, we can try, over the long term, to improve our models and try and bring that down. You said the word. You run the ensembles. So this is why we run many, many forecasts. It's just basically a sledgehammer approach to try, it's basically the only way to really know of trying to assess our uncertainty. So rather than simply running a single forecast over the two-week or four-week or two-month period, we run a whole set of forecasts. So we might perturb our initial conditions and we might perturb our model during the actual forecast. I'm going to talk about how we do that in a second. The result of that is rather than just one forecast, we'll end up with 50 or 100, depends how big our ensemble size is, a cluster of solutions. So if we have a very tight cluster of solutions, we might think that we're quite confident about our prediction or if we have a very large range in our solutions. We might think that we're not very confident about our prediction. But how do we know what that is? Well, let's show a couple of examples first before I ask that question. So this is to try and account for both initial condition uncertainty and model uncertainty. So here's an example just from a recent Nature paper where we start with lots of perturbations and schematically and these are all different precipitation forecasts for the UK and you can overlay them and put a probability map together of where you think it's most likely to rain in a certain time in advance. This is another example back from over a decade ago, but again, it's showing basically three-day forecasts for a severe storm in the United States. I quite like this storm because I was actually there at the time and I got in one day earlier, luckily, because all the airports were closed down and the whole place was snowed off and it was very poorly predicted. I was very surprised at the chance at the time you found that basically you can look at different members and some of them had an extremely low pressure system and some of them were basically failing completely to have any kind of prediction at all of the event. So it was very, very sensitive to small perturbations in the initial condition. This is another example on a longer time scale. So this is looking at seasonal predictions on that borderline sub-seasonal to seasonal in Ethiopia and you can see again that you can have huge uncertainty between the different members. So this is just nine members out of a larger ensemble showing predictions for rainfall over Ethiopia three months in advance. So how many of you have seen plots like this? This is a spaghetti plot of Nino. So this is the Nino 3 forecast. This was done a few months ago. This was back in August. So each of these red lines is again one model forecast to the future of the anomaly of the SSTs and the Eastern Pacific. So you can see at the time that most of the models were forecasting an amplification with a peak towards the end of the year and then scaling off. So this is a tremendous variety between these different model integrations. So some of them have the ENSO tailing off straight away. Some of them have a very strong amplification. So this is an example more close to home. So this is today's forecast of Trieste's weather this week. I thought if everybody was completely asleep by this point maybe this would wake some of you up at least because it's quite relevant for your week ahead. We have precipitation. In the middle we have wind speed. On the bottom we have the two meter temperature. So the boxes and whiskers are showing again the uncertainty, their statistics, about the ensemble for the prediction ahead. So you can see that as the lead time this is the forecast was made yesterday on Sunday. As we go to the right time is increasing so it's looking further and further ahead into the future. So you see what you would expect that the box whisker size that shows you the uncertainty so the larger boxes is going from the 25% to the 75% tile. And then the thinner boxes are going from the 10% to the 90%. They're widening as the uncertainty increases the further ahead you go. So you can see that we have a borah predicted for tomorrow. It's going to get pretty windy. It calms off and then it gets even stronger so Thursday, Friday. So you're going to have an experience of a borah. Those of you that haven't experienced it before this is going to be about medium strength. I'm scaling up here when I say medium strength because the wind speed here says 8 but we've got box sizes which even though this is a world leading centre on the order of 16km box sizes I know from experience that when you downscale this there's a very fine topography over to Ester and I've got to scale this up by a factor of 2-3 to get the true wind speed here. So even then you have to do some post-processing which is something else we're going to talk about during the week post-calibration which I do in my head. The same is with the temperature. Topography is very complicated. You only have to go in the winter just up to Al Pacino two or three kilometres away and it can be 7 degrees warmer at night because again of the topography holding the cold air over the lip and here we're down by the sea. So again it's this issue of resolution and not being able to represent those small scale fluctuations so I know that this thing tends to be by so the temperature at the bottom here due to that effect to the cold side. Even though this is adjusted to station height for the location I'm using a lapse rate adjustment. Question? So the deterministic let me answer that a little bit later because I come back to this slide later when I've introduced the different systems at the different ranges because if I start to answer the difference between these now it might confuse some people. I have a slide in a minute where I show the different range systems and then I come back to this so let me come back to that question in a moment. So how do we actually introduce these perturbations? So initial conditions we could do a variety of things. We could just apply random perturbations we can target them we can even have ensemble data assimilation so what's the difference between those? Well if we look at just short range for example the median range the idea there is one looks at a window normally a range of around two days it's still two days now the targeting for the singular vectors so 48 hours and that 48 hour window one assumes that the evolution of the dynamics is approximately linear and then one uses a targeted technique based on singular vectors to try and work out where your perturbations will lead to the most sensitive growth and divergence between the forecasts so if you're building an ensemble and you only have 50 shots you don't want all of those 50 forecasts to look exactly the same maybe they might be very similar because the atmospheric state is very predictable but it might just be that you haven't found the locations where it's very sensitive for error growth because at the end of the day butterflies have a very bad reputation we all seem to think that they're destroying half the world through their flapping of their wings but that doesn't happen the butterfly really has to be in the wrong place at the wrong time to cause carnage through tropical cyclones so the atmosphere will be sensitive in particular locations depending on the stability and in others that butterfly can flap as much as it likes those perturbations will just dissipate away so you can try and target that's what we do at ESMWF that's what ESMWF does it targets those to try and work out to have the atmosphere to have the fastest growth now when you go to longer time scales we don't really know how to do that yet so when you look at seasonal forecasts we don't really know yet and it's one of the areas that's under active research how to do the equivalent for a one or a three month forecast so for S2S systems or so on or a seasonal forecast system tends to be the perturbations that are just putting randomly well that's fine anyway we still have the growth do you really want to target it's a much more open research question of how you perturb your ensemble to have the right spread over longer time scales and ESMWF now have introduced the system as well where they have the assemble data assimilation what does that mean well if we're trying to I'm gonna talk about data assimilation again in a very introductory talk after lunch but if you want to assimilate information okay and you have some sources of information to be quite sparse you have an observational error attached to different kinds of observations whether they're station data or satellite so your data assimilation answer it's not just one correct answer you can have a whole set of shall we say a simulated pictures of the atmosphere that are consistent with the observations that you have so you might want to try and incorporate that by actually having an ensemble of assimilations okay so ESMWF now have an ensemble technique which I think was that operational last year no two years ago now time flies so these are all ways of actually having different perturbations so these are quite different this is really to account for your lack of precise information about the observations whereas this is to try and in fact that you want to really sample the entire space perturbations to model physics well we can change the parameter settings if you have for example a convection scheme you might want to just change one parameter in that scheme so it might be the entrainment rate so you might run five forecasts that have all slightly different parameters for the entrainment rate so climateprediction.net that's the other extreme for climate prediction that's how they created their ensemble they had each climate forecast model was basically using a different set of parameters for the different parameterization schemes so the actual model was the same it's just the actual parameters all you might want to use stochastic physics so at its very simplest level stochastic physics merely takes the output from the parameterizations, the convection scheme the radiation scheme and multiplies it by a random number on a patchwork just simply multiplies it by a random number between a half and one and a half your radiation says I should be cooling one k per day you multiply it by a random number it becomes one and a half k per day over a patch for a certain period of time again there's a lot of research into ways of actually increasing the complexity and should we say justification you do this or you can combine both of these and actually introduce multi-model systems so you have multiple centres of multiple models and you combine them now I've put this in red because sometimes when people talk about multi-models especially for the longer time scales especially for climate but also for the seasonal prediction often people interpret differences between multi-model systems as being the uncertainty due to the model physics that happens in climate a lot you compare 10 different CMIT5 models and you say this is the uncertainty they're different this is the uncertainty due to the model physics it's also the initialization because all of these systems have different systems and different ways of setting up their initial conditions different data assimilation systems so you have to remember that when you look at systems it's accounting for both initial condition uncertainty but also model physics not just the model physics so it's important to emphasise so on the top I showed you an example of the basically the ENSO plumes just using the UCNWF system on the bottom here this is a much thicker plume because this now contains four systems it's from the EuroSIP basically multi-model system ok so here I've basically listed and we're going to talk in a little bit more detail about some of the databases out there which have basically all been put together over the last, should you say one to five years so in a way I would say that the NWP and sub-seasonal and seasonal prediction has lagged a little bit behind the climate change problem in a way but the databases for the CMIT process have been around for quite a long time now and with CMIT 5 there was a big effort with CMOR to bring them into a standard format to make them much easier to use now we're catching up in the atmosphere so for example for the short to medium range there's TIGI that's been around for a number of years now based at UCNWF that Frederick mentioned already ok so in the final four casts there's EuroSIP which is a combination of four modelling systems ESMWF, Meta France, Met Office and NSEP recently joined them at the moment it's only available online as graphical products but I believe under Copernicus it will become open source with digital products as well so the reason why I've included this is though it's not digitally open at the moment it will be in the near future so watch this space there's a funded Copernicus program which is the climate services side is actually based at ESMWF in Reading there's also the North American multi-model ensemble, NMME how many of you have heard of this or used this just out of interest so there's quite a few six or seven of you so again this is focused on the seasonal forecasting timescale and I have a slide in the moment showing some of the centres that contribute to that and then there's CHFP how many of you have heard of CHFP those are your IRI involved to less, okay that's interesting so that's something that's come out of one of the working groups WG SIP that's focusing on the basically sub-seasonal to inter-annual predictability and this is a database which actually tries to accumulate the hindcast suite which I'm going to talk about in a moment in one location so one can document how seasonal forecasting systems have improved over time it's supposed to be a long-term archive and then here in the middle as I said we have the sub-seasonal so the S2S database at ESMWF which is the one we're really going to focus on that's just come out this year so NMME I think went operational in 2014 CHFP has been around for a couple of years I said EUROSIP is not online yet so this is all really in the last few years I think TIGI was perhaps a bit long how long is TIGI going around for now four or five or yeah about five five years, okay oh 2005, okay a bit longer then so that's been around for research and then there is actually plans to actually supplement the NMME with additional higher resolution forecasts on the sub-seasonal timescale as well which I believe is penciled in for 2016 as well so there will be a second database that complements the S2S database at ESMWF which is focused more on the American systems that's coming up in 2016 okay now I hope I haven't forgotten anything from that list that I should have included feel free to interject if I have okay so it's quite exciting at the moment because we really in the last couple of years we've got all of these sources of information where we can look at how these systems are working, the predictability just weren't there just a few years ago essentially so this is an example from the NMME from Ben Kirkman's paper that came out just last year and BAMS showing the ENSO plumes and the red line is how the model ensemble as a whole is doing compared to the black line which is the observations and these are the modelling systems that contribute to that just to mark the IRI we've got some people from IRI of course that are helping us out here their system have got a star because it's no longer operational and that's one of the things about these databases is do you have it that's purely operational groups or do you open it to a wider community of research models as well so the slight difference in strategy between for example the S2S database and perhaps the NMME and the others is that this has a mix of operational and research models whereas the S2S is purely operational centres as you saw which does mean that you've got perhaps a wider ensemble here but you have systems that might not be sustained it would change over time so for example these two systems now are no longer operational ok so if we go back to my simple schematic of this forecast we have this cluster of solutions so we could look at a root mean square error difference between all of these different forecasts and we refer to that usually as the ensemble spread so it's not what you put on your toast at breakfast time it is basically a root mean squared error difference ok between all of these forecasts now how big should your spread be remember your spread is telling you your confidence so if you have a very small spread your forecasts are very similar so maybe you're very confident tomorrow I think it's going to be 10 degrees plus or minus a half maybe the next day your forecast will be after is 10 degrees plus or minus 5 so you're not very confident but if you set up your ensemble how big should this spread be rhetorical question maybe but what should it be roughly equal to should be less does everybody agree with him I would say you don't want it less than the error what do you think I mean I would say I kind of agree with you but I think it should be on the same order of magnitude as the error so I mean essentially again you want roughly the spread of the ensemble is roughly the same as your your RMS error so if this is my observation you want that observation to kind of be in that cluster ok so if your observed state always lies outside your cluster that means that you think your modelling system is much better than it is so you say ok I think in 3 weeks time it's going to be 10 plus or minus 2 or 15 so you don't want your observed state to be out here you don't want the spread to be less than the RMS because that means you have an overconfident system ok likewise if your observed state is always near the ensemble mean and you've got this huge spread that means that you're perturbing your initial condition perturbations and your model physics perturbations are agitating the system too much ok ok so then you have an underconfident system you're giving people error bars that are too large you're saying well more or less I think it's 10 plus or minus 5 degrees tomorrow but in fact your system is always within 0.5 or so ok so you don't want to be doing that either because you're basically throwing information away ok so you want your spread to be roughly of the same order of magnitude and that's your RMS error now that sounds simple but it's not as simple as it seems why not say again well if you have an NWP system the nice thing about S2S time scales and so on is that you can look at your handcast which I'm going to go into a moment over the past and you can evaluate your model so why is this not as simple as it seems because you could say ok I'm looking at tomorrow's weather and say my spread is too large always well I've just tuned down my perturbations making my perturbations a bit smaller and then not perturb the model and I reduce the spread until I get it just right everyone's happy yes but why is it not as simple as that ok so very good so you can't do this just for one particular day you have to average it over the statistics of a lot of forecast which may be regime dependent so you might find in certain regimes you're always over confident and others you're under confident also your spread behaviour will be a function of lead time location and variable ok so you might be looking at for example now that I'm going to show an example in a moment where the Met Office I've been trying to predict ENSO for the season ahead at lead times of 3-4 months or even longer when the system is under confident but they may be able to adjust that but they might find then that their predictions for other variables are over confident so it's not that you can just tune the system for one particular kind of forecast it depends on what variable you're focused on what your location is so one of the I should have put a slide in to demonstrate it and now these days it's improved but when I was at ESOM WF 10 years ago one of the key problems was the spread in the mid-latitudes looked nice in the medium range out to the month so they had the forecast ensemble nicely spread out but in the tropics the system was over confident and it was very difficult to make changes to the system that made the tropics forecasts diverge enough to account for the error without having the mid-latitudes spread out too quickly so you had this disparity between behaviour in the tropics and the mid-latitudes so it's not that simple it sounds like a very simple thing to fix and tune but it really depends on your lead time on your area that you're focusing on and so on so this was that example I wanted to show from the Met Office where essentially they're looking at ensemble skill at predicting the NAO and this is this black line here and this is just of the number of members in the ensemble but don't worry about that for the moment the thing I wanted to show this was shown in a talk last week by Doug Smith was when they actually looked at the model predicting itself so that means if they pretend that one of the 20 members is actually the truth and they look at how well the other members predict that member they find that the prediction is much less skillful so their interpretation he said was well they were thinking maybe that's a sign that the system is basically um underconfident and again I was surprised that they hadn't gone away and just looked at this kind of diagnostic just simply compared the RMS error against the ensemble spread which again I think is why it's important to emphasise this they hadn't done that but I think this is just another way of showing that their ensemble for that particular metric is actually underconfident there's too much variation between the ensemble members so we're going to talk a little bit about how one uses the forecasts and the variability so a very simple example we imagine we start from our perturbations initial conditions and we look at basically our different forecasts for a certain future time we might be looking at a certain event so I've just got Ray no Ryan to keep it simple for S2S next week we're going to be looking at for example prediction of drought occurrence or flood and extreme flood so the time range is basically should we say generic here how would we actually validate this so if I say tomorrow I think so 70% of my forecasts say it's going to rain and 30% say it's not going to rain if it rains is that a good forecast if I say tomorrow 70% chance of rain how many of you bring your umbrella quite a few of you that's because the cost of bringing the umbrella is basically zero it's a little bit of weight now if you had to go and buy an umbrella that starts to change it a little bit you've got to buy an umbrella for 50 euros how many of you would buy an umbrella for 50 euros if I gave you a 70% chance of rain forecast no not many because your loss in that case is quite small you get a little bit wet so most of you would rather keep the 50 euros and spend it on something else so this is why so how would we actually validate that if it doesn't rain tomorrow how many of you would be crossed with me I say it's 70% chance of rain and it doesn't rain am I right or am I wrong how many of you think I'm right nobody on one thank you my friend in the front row you're wrong nobody you're all very shy well I just gave you a probability so I mean I can't be wrong as long as I don't say zero I can hedge my bets I could just say every day 50% chance of rain and I'm always going to be right but it's not very useful is it so how do you test that system well basically you have to look at many of my forecasts so you say okay let's look at Tonkin's forecast forecast's over the last year let's look at all of the times he said 70% chance of rain how often should it have rained 70% of the time okay if it rained all of the time I've been under confident it comes back to that cluster of points okay means I've been under confident so when we talk about how to validate this we're not going to do just thinking about ensemble means this week we have to start to also think probabilistically and that's important when you get along beyond the deterministic range okay so I know it's a little bit of a silly example in a way because it's very simplistic but it's important to start thinking about that when we look at these products okay on the longer ranges okay so I've got a few more slides I'm in danger of overrunning here to start to introduce some of these systems so I just wanted to put a little bit of framework into place so I'm going to show the ECMWF system and the Met Office system not all of the systems because it would be tedious and I'm not so familiar anyway with the other systems if you're interested we can find out this week about all the nitty gritty and so on so at the top end we have this deterministic run which is the highest resolution system and we have a single forecast okay now ECMWF for quite a long while has been supplementing that they started with an ensemble run that was running out for 15 days and that was many different forecasts to try and account for this model and initial condition uncertainty by perturbing the initial conditions and perturbing the model physics then basically almost two decades ago they started to supplement that with a system which was basically looking at seasonal forecasting time scales okay so we have the highest resolution the ensemble was lower resolution because you're running 50 or 51 of these things so it's a lot of computing power so they have to run at lower resolution the seasonal forecast was running at even lower resolution because this thing is running out for many months in advance okay some of a subset of those were extended is this still four times a year now yeah so it's out to 13 months okay so they run each month for seven months but four times a year they run out for 13 months ahead but you can see we've got a little bit of a gap here okay that's what that bridge in the gap is all about that Frederick was talking about this morning so it started off Frederick was basically if I understand correctly you pretty much put the whole thing together yourself back in 2003 so Frederick when he had a spare moment here and a spare moment there I remember at the time it was like on top of everything else he was doing started to put this system together which was basically sitting at an intermediate resolution between these two and at that time the research mode basically run once per week for 32 days if I get any of this wrong you step in and tell me yeah so it was running 32 days and it was run the system once per week so it had this like burst ensemble okay once that went operational there was a decision to basically try and bring these together a little bit in a similar way that mimics what's done with the seasonal system okay so the ensemble now is essentially extended out to 48 days not 32 days okay and it's actually operated now twice per week okay so you essentially have this and it's split resolutions so you have a higher resolution at first and it switches to a lower resolution for the extended run okay one of the key differences and I'm going to talk about this and in a couple of slides time is the hind casts I'm going to talk about why we need hind casts in a moment first so just a flag there's a big difference between these systems in the fact that here we have dynamic hind casts here we have fixed hind casts so I'm going to come back to that in a moment I just wanted to flag it now and both of these systems at the moment have 51 members okay so this is what now I wanted to come back to this forecast so this blue line is showing this very high resolution for example system run here these bars are showing the ensemble for the EPS so you can see that this lead time only shown here just basically goes out to 10 days but these barred forecasts are carrying on now twice per week right out across the blackboard to day 48 which is somewhere over there so that's how that fits in the red line is the control which essentially what does that mean it's essentially the basic forecast system without any perturbations okay now I actually personally think that if I became director of ESMWF I wouldn't do this anymore because I don't like personally the way that you have a system where you have perturbations and then you have one forecast system that doesn't have perturbations because the modelling system is not the same and that makes it very very difficult it's quite a constraint on your stochastic physics because what it means is anything you do in a stochastic sense has to have a zero mean perturbation you can't have if you try and do anything that doesn't have a zero mean perturbation it means then you start getting into all sorts of trouble because your unperturb system has a different set of physics to your perturb system and it's very difficult to keep the control inside your pdf so I said the talk was a bit basic that's sitting brackets for the slightly more into stochastic physics and perturbations don't worry if that's confusing just ignore me switch off switch back on again now okay so you can combine these two so next week if I show you the health system what we've done there for example of our health modelling system is we've actually taken and we need to upgrade this now to the 48 day we take the 32 day because we were looking when the older system once per week we take the 32 day system for month one and then we combine that supplement it for months two to four with the seasonal forecasting system calibrate and then use that to run the applications model so if we find that there's more interest for that I will show that at the end of next week okay so why would we do this one of the reasons of course is the resolution we've talked about the higher resolution and if we look what I'm showing here is just a simple correlation map of the temperatures I'm interested in Africa so I'm showing Africa on the left the right panel is showing the increase in skill of using the sub seasonal system the extended EPS for 32 days compared to the seasonal system and this is exactly like for like so what I'm doing is if the if the because of course the one thing that's difficult comparing these systems is they start at different times the seasonal system starts on the first day of each month okay and the as we said the sub seasonal system starts twice per week so I think it's Monday and Thursday yes if I remember rightly so every Monday and Thursday so I'm comparing exactly like with like if I have a forecast that starts on Tuesday the 10th of March runs for 32 days I take those exact 32 days predicted from the seasonal forecast that starts on the 1st of March so it's exactly the same days for a whole year's set of forecast and hindcast okay so where does this skill advantage come from let's see that's right you will have some variability in some areas I mean just from chance you wouldn't expect to have zero everywhere so there will be some sampling issue there and you might find that there are some aspects of the system that actually gives you detriments or so on but on the whole if you average it you have an advantage in skill yeah so where would the advantage or difference because there is actually a point no it's an important question you made because there are some things that may not necessarily be an improvement let's see how many of you have been listening so far to my basic lecture so where is this going to come from I've told you one already resolution what else do we expect to have a skill advantage when I'm looking at this is a year's worth of forecast plus all the hindcasts of the S2S system compared to the seasonal system the way you're initialising is different actually and that's one of the reasons where it could be better but sometimes you could get worse because the seasonal system actually they have a longer time before the issue of the forecast whereas the twice per week forecast you don't have the time to do for example all of the things of the ocean so Frederick would be talking about the ocean's initialised but you have to have a system that's a little bit faster so you blow winds across is that right to get the ocean perturbed state which could be worse could be better you have to look at that systematically so because of your fast lead time there could be things that are actually detrimental with the monthly system because you've got to get the forecast out quicker it's no good waiting 15 days like you do with the seasonal forecast because it's twice per week it would be too late very good any other ideas a different timescale but these are exactly the same dates I think I want to know what you're getting at so you're talking about the different start times or the different timescales because I'm looking at the same 32 days so the seasonal is the first 32 days of the seasonal so I'm not looking at different should we say I'm not looking at the whole four months of the seasonal but you are the answer is the right answer but for a different reasons I was kind of sedating to earlier it's the lead time advantage it sounds like a really stupid thing but your seasonal forecast is once per month your S2S is updated twice per week so if on the first of March you make a forecast well they both start at the same time but say I'm running operationally for my case malaria in Uganda I can continue to update it so when it gets to the 28th of March it's the most recent S2S and it's almost a whole month newer than the most recent seasonal forecasting system so it sounds a complete banality but it's something that's very important filling that gap is not really so much about the resolution the biggest advantage you've got is that you have a system that's much more frequently updated but goes out to 48 days so you have a lead time advantage so you have a lead time advantage that's your biggest key step in fact you have a framework advantage you've got the ability to have more computing power spent so higher resolution also there are other differences in initialisation and then there's a middle one which I'm going to come to now you have basically your physics now what do I mean by that and it comes back to these fixed hind cuts so I'm going to come back to this point in just a second so I have a few more slides and then I'm going to wrap up so there's just hind cuts just in case I've confused somebody why do we need hind cuts well imagine this is evolution of some variable forward in time and this is the real world so this could just be temperature and this is my forecast so maybe I have a forecast here normally it's 10 degrees I forecast 8 so that means we're 2 degrees colder than usual but of course these models have biases so this might be my model world ok so my forecast is warmer than the model world so we have to basically translate this anomaly onto the real world this is just a very simple case of bias correction of course I've actually drawn both lines identical I mean the real model world we had the real world, the model world and then the real model world is that all your variance and everything else is going to be different as well which means that bias correction as well it's not as easy as it seems we could have a whole workshop on bias correction techniques so you need to have hind casts for the past to be able to calibrate your model ok so how's that done let's introduce first of all another modelling system this is quite a complicated diagram it's from a paper that just came out this year in QJ from the Met Office system so they have a slightly different way of doing things they actually have each day four forecasts two which extend right out for seasonal timescales and two which only extend for sub-seasonal timescales and they do this every day so in a week they've got seven sets of these seven times four 28 ok and then for a month three weeks worth of these forecasts so and they combine that together to get their basically their forecast set so their S2S product uses seven days of these four forecasts to give you an ensemble of 28 days ok so the difference in that how is it different well you can see again it comes back also to this lead time ok if ECMWF has a burst of 50 forecasts on a Monday then your information for Tuesday is going to be nice you've got a lot of ensemble members ok for that Tuesday with the the Met Office forecast four of those forecasts will have been made six days earlier ok so there will be much older information but of course if you then look a couple of days later for the ECMWF if you look on the Wednesday then you won't have any information newer than two days whereas the Met Office will always have some of these which are just one day old so their system has a very continuous should we say lead in time whereas the ECMWF is a little bit like a sawtooth you know each twice per week you have a burst of information which then decays in terms of how new it is so again it's different strategies of how to set this up and they also have a dynamic hindcast suite so just to make sure I put this in just before because after Frederick talked about on the fly and fixed I just wanted to make sure that everybody understood what that meant so I just added this very quickly it's why it's a little bit of a rubbish picture that's all I could manage over coffee break this morning because I wanted to make sure that was clear so what do we mean by on the fly what it means is ok this is my forecast I'll assemble 50 members which I'm doing now today so the date today is 22nd 23rd now you can see why I arrived 20 seconds before the start so today is 23rd of November and we make a forecast so the dynamical on the fly hindcast is ok let's also make an ensemble for the same day for the last 20 years excuse me it used to be 18 and now it's extended to 20 years am I right so it's now recently changed last 20 years so you say ok let's also start for the 23rd of November 2014 23rd of November 2013 12 all the way back for the last 20 years so you don't just run the forecast now you run that every single time also for the last 20 years for exactly the same date ok so that's a kind of dynamical system by contrast you might have a fixed system that says ok we have our new system, new model let's run a whole load of hindcasts and then we'll forget about it that's my database of hindcasts so no matter where my forecast starts you might run hindcasts or the last 18 years maybe starting every single day I don't know but you run this hindcast suite just in time why would you use one or the other well it depends on the number of factors and the key one is how often are you updating your system ok now for example the seasonal forecast system is not updated as often as the others on the order of every four to five years a little bit like climate modelling systems each modelling centre tends to have a new system for each IPCC report but then they use that system for the next four or five years if your modelling system stays the same then you might go for something like this because for the next four or five years your model physics are not changing so you can run all these and then freeze them what's the advantage of doing that well because you just do it once you can perhaps have your hindcast suite with more members ok so you have a large ensemble to sample your uncertainty however if you have an NWP system that's updating quite frequently so on average it's been a little bit less often recently but ECMWF will update their system two or three times a year why, well there might be new satellites that come online that you want to incorporate into your data assimilation system there might be new model physics updates if your forecast quality and so on and you want to bring them online as quickly as possible you don't want to wait five years before you start using a satellite probably the satellite won't be there anymore in five years time they have a five year lifetime so if your system's updating all the time you don't have that much advantage to do this so you really want to basically each time you have a forecast redo those hindcasts to make sure you're always using the same modelling system there's no point having a hindcast with a different model because it'll have different bias characteristics the bad thing of course though is it makes it more expensive to run which means it hits your ensemble size if you didn't have to do this each time instead of 50 members you might be able to afford 100 members here or even more so the system used to have just five members per year which has been increased now but it used to be just five members per year so as Frederick showed you different centres have different methods a lot of the centres that perhaps update the model less frequency are using this fixed methodology rather than the dynamic it also means that the database construction was a little bit more complicated due to this disparity between the centres it made it more difficult to have a generic interface it also means when you want to look at the NAO predictability how do you do it we could take the operational forecast the system at ESMWF has been running from 2008 so if you wanted to look for a few years you could look at the inter-annual changes in the predictions using the actual operational forecast ensemble but the problem is that that model is changing all the time there are updates to the system so what one tends to do is for any particular start date is actually use the hindcast so this is year minus one, year minus two, three, four down to 20 years or beyond to look at the inter-annual variability so I just wanted to emphasise that cos that can also sometimes cause some confusion that you actually have two time axes cos you can look at the operational forecasts and how they change in time but then the model version is changing or you can look at the hindcast sweep but then you have a smaller ensemble size so the malaria work I was showing was just based on five members so also by using the hindcast sweep again you're restricted to the length of the database range and as I said this particular work we only had five members in the hindcast sweep whereas the operational system of course has 51 members as time goes on you're adding more and more operational cycles you're going forward in time so as I said it's been around since 2008 so it does make it a little bit more tricky to set up the database you'll notice for example for the NNMME that the hindcasts are all fixed and they're all roughly covering the same period which does make it easier to inter-compare these systems that's not necessary to the case for things such as S2S where you have dynamic hindcasts are moving forward in time so your oldest years are actually dropping off if you look at the system in 2015 you won't have the same set of years as the system had for 2012 so I'm going to stop there so I've overrun actually by 10 minutes apologies and as I said in the afternoon Frederick is going to show just a quick demonstration of some of the aspects of the S2S and I will be showing a similar thing in parallel for the reanalysis and so on ok, I'll stop there, thank you


We have rods of different metals attached to a central hub. On each of the rods is a small amount of butter. The metal rods have different specific heats and will heat up at different rates. We will watch the order in which the butter melts. The aluminum melts first, the brass melts second, the iron melts third, the nickel melts fourth, the copper melts fifth, the stainless steel melts sixth.


So welcome everyone to data preprocessing, this is the first talk, and today we are going to talk about the some introduction about this concept and something about data integration. My name is Ann Wren and I'm a researcher associate at Kathy Marsh Institute and UK data service. I'm based in Manchester and my email is attached here. If you have any questions, you can come to the Q&A session later or you can just send me an email after this talk and I'm very willing to help you. So this is the table of content, there are two talks and one code demonstrations on this topic, one talk on today and another one on next Thursday. Today we are going to cover the definition, context and collecting data and also data integration meaning on joining tables. And next Thursday we are going to cover more on data cleaning, data reduction and data transformation. On the Thursday after next Thursday we are going to deliver a live streaming code demonstration on YouTube. We are going to provide some sample Python codes and perform these tasks. If you are interested in this, you're very welcome to join the next two sessions. Also if you want to know more about this topic or any specific part that I didn't mention, we are going to send a questionnaire after the talk and please leave your comments there to help us to improve. And also although we cannot hear you during the talk, we will have a Q&A session later and answer the questions then. But meanwhile you are welcome to leave your questions in the Q&A box throughout the talk. My colleague Julia and Joe will answer them directly there if it's not too complicated or they might leave it to the Q&A session so don't worry. So without further ado, let me give you a definition of data preprocessing. Data preprocessing or known as data preparation is the process of manipulating or preprocessing raw data from one or more sources into a structured and clean data set for analysis. It is an important part of data analytics. When the data has been fully prepared, it is then ready for further analysis like classification, association, prediction and clustering of the data. There's no clear recipe or standard for data preprocessing. Usually it includes various tasks and considerations. For example, selecting and acquiring data to use, integrating different data sources together, conducting exploratory analysis, and also cleaning and repairing data like dealing with missing data, inconsistent data, and also data reduction, transformation, etc. So generally speaking, it can be regarded as everything you do before the actual modeling.  It seems like in these days everyone was a data scientist and for good reason. According to LinkedIn, the career has seen exponential growth and becoming the second fastest growing profession. A huge amount of data is generated each day, but many companies are still facing obstacles when it comes to utilizing their data correctly. And some are not even sure of what to do with their data science team. Here's a graph of one day in data. You can see that around 500 million tweets are generated each day. And on Facebook, around four parabytes of data generated, including 350 million photos and 100 million hours of video watching time. So it is fair to say that the popular phrase, big data is not an exaggeration, we are facing a real data deluge now, which means that the amount of data being generated is overwhelming the capacity of organizations to use them. And the organizations all know that the raw data itself is not a treasure. Only if the data is attached with good preparation and proper analytics, it can then provide actionable knowledge so that we can get insights or findings that we want. The companies in real world, especially the big international ones, put much effort in thinking about how to use data to provide more service around the data. For example, you may have heard of Google Trends. Some scholars use it to track the change of popularity of something that they want to study. And also digital banking is something that where you can find a personalized page or reports within the bank apps in your cell phones. And also YouTube analysis, many YouTubers use it to see what kind of content will make them lose their audience so that they can avoid these kind of contents next time. And also a typical example is Amazon Recommendation. When you log into your Amazon account, you can always see a lot of automatic recommendations for you that allure you to make more orders. They are all based on your past searches and purchases. So in some sense, we can see that the world nowadays is full of pitfalls. As your data is almost recorded every second, the businesses will collect the data, analyze them, and then create more strategic presentations or products to attract your attention. And you find yourself spend more time and money on them. And without doubts, this phenomenon is going to be more and more severe in the following decades. Do everyone needs to be prepared for this big data change? Well, it should be minded that although all these applications sounds pretty direct and easy, there's a big gap between the expectation and the reality of this work. I think this picture is a really good presentation. Real-world data analytics is full of challenges, and some challenges might be very time-consuming or even impossible to overcome. Rather than developing the fancy algorithms or plot the eye-catching figures, much time and effort are spent on data collection and preparation. Forbes found that data preparation accounts for about 80% of the work of data scientists. In particular, 19% is spent on collecting data sets, and 69% is spent on cleaning and organizing data. And other tasks like building training set, collecting and mining data for patterns and refining algorithms and others, they only account for a very little in comparison. It is not unfair to say that data preparation is crucial for getting meaningful results from data analytics, not enough time. So it's sufficient to prepare the data list to garbage in and garbage out, which we also call it as GIGO. It is a computer-sized acronym that implies bad input will result in bad output, which is the message of this picture. Well, it says that if your business is a house, the data is its foundation. So before cleanly seeing, the data quality might be only 40%, and if you do anything on it, you might get wrong prediction, so you don't have any meaningful results or any meaningful message from it. However, if you do proper data cleanly seeing, then data quality can improve to 90%. Then you can build your house on it, you can get right prediction on it, then you get the meaningful insights and findings. So although data preprocessing may not appear in your research plan or in your methodology plan, but please don't forget it or underestimate it. It is very crucial in your research or in your data analysis. How to check data quality? There are many ways to check the quality of your data, and here I'd like to cover two common ways. One is to use metadata. Metadata is the data about the data. The motivation is to better understand the data, including its availability, types, quantity, complexity, and other things. Another one is exploratory data analysis, or EDA. EDA is widely used in data analytics, and one of its functions is to check data quality. It is an approach to analyzing data sets to summarize their main characteristics and often with visual methods. EDA is primarily for seeing what the data can tell us before formal modeling or hypothesis testing tasks. For example, if I have cells at hand and I want to build a model to predict the performance of next month, then maybe I will start with EDA to do some random explorations, such as creating histogram to see whether there are any regions out from the rest, or if maybe we can have a clear higher income. Among others, the source of data is very crucial to determine the data quality from the very beginning. There are many ways to collect the data. If you run your own business, which generates data itself, then you might think about building a data warehouse to store the data and organize it as you wish, and the data can be easily retrieved when needed, but that usually costs a lot. Or maybe you can script data from a website to collect the raw data yourself, but that usually takes longer time to pre-process it, as the data tends to be very dirty. For example, if I scrape the data from a hotel page, then the price may be shown as a piece of text rather than a number. Then I need to change it later. But the good point is that if I do web scraping, then I can customize the data structure while collecting the data. Alternatively, you can use the ready-made data sources, for example, Statista, which is usually for business data, and also WordBank's UKDS, which is UK Data Service. You can find a lot of open source data on the UK Data Service website, including census data, international macro data, qualitative data, and survey data. You can also access data from other providers and use the UKDS API to customize the layout as you want and then export the data. The data is usually clean, and this may save much time from data reduction integration later. And there are detailed instructions on how to access and use the data. So that's it for the context and for collecting data. And then you may ask again, what do we do in the typical data pre-processing after we have the data at hand? Well, major tasks in data pre-processing include data integration. That means the integration of multiple databases, data cubes, or files, and data description summarization and visualization, which is also included in the EDA part. And also data cleaning, which means a fill in the missing values, a smooth noisy values, and identify or remove outliers and noisy data, and resolve inconsistencies. And data reduction, which is usually to reduce the representation in volume, but produce the same or similar analytical results. And data transformation, which is usually the normalization aggregation. And last but not least is data discretization and data generalization. That means that you maybe categorize the continuous data into different categories so that we can build a model on that. But it should be very in mind that, as I said, there's no recipe or standard for data pre-processing at the moment. So there's no certain order of doing these steps. All these tasks are just the most common ones. In fact, it's quite often that you will have to repeat some of them multiple times. For example, after reading the metadata, you clean the data to deal with the missing values at the beginning. And then after joining the tables, you find some new missing values are created, and they need to clean it again. So in the rest of today's session, I'm going to talk about the first entry, data integration. And for the next Thursday, I'm going to cover how to clean, reduce, and transform the data. By definition, data integration is combining data from multiple sources into a unified view. The benefits include to improve data quality, to enrich data with additional information, and to allow reliable data analytics and beyond. Data integration is a kind of unavoidable step to some extent, as it is not common that all information will be stored in one spreadsheet. For example, I have a data set on hotel information, including the hotel's name, city, star level, etc., which is very static. And also I have another data set of the reservation, including the customer's name, hotel name, payment methods, price, and others, which is always updated. And if I want to find which features of the hotel determine the hotel's popularity, then I will first need to link the two tables based on the hotel name or hotel ID. So this is a very typical data integration. Usually integrating in-house data within data warehouse together is relatively straightforward, with common attributes and structures across the schemas. For example, joining the tables, as the name indicates, joining the tables enable you to extract and simultaneously process data from more than one table. There are four types of joining, inner drawing, full drawing, left drawing, and right drawing. Notice that the methods are quite common in various applications and languages, so SQL and Python may have different joining commands, but you can always select which join type you want to use. And also, you should remember that recheck data quality after the joining, because as sometimes new missing values or unexpected roles might be created during the joining process. And if we do not recheck the data quality, then maybe after the joining, the quality of the data might even be reduced. So the first type is inner join. By default, the joining query performs an inner join, which includes matching roles only in the results. So for example, we have employee payroll here and employee organization here. And we can join these two tables on the column employee ID. And only the roles that are presented in both tables will be kept in the merged data set. So this is the inner join. If a role, if the employee ID only appears in one table, then it will be dropped in the final data set. By contrast, a full outer join includes all roles from both tables, but it will create a lot of missing values. For example, the role fifth in the merged data set, it doesn't have the department on the second table. So that field is missing. And similarly, the last role in the merged data set, it doesn't have the salary in the first table. So that is another missing value. A left join is a bit different. A left join includes all roles from the left table. And by left table here, I mean the table that you put first in your command. And similarly, a right join includes all the roles from the right table, which is the table you put on the second place in your command. So these two kinds of joining is very sensitive to the order of the tables in the command. So we need to be careful on which table we put first. Data integration is not an easy job, as there are many difficulties. First is the database heterogeneity, including system heterogeneity and schematic heterogeneity. System heterogeneity occurs when using different operation system, hardware platforms. And the schematic or structural heterogeneity is the cases where the native model or structure to store data differs. So for example, the SQL database versus the NoSQL database, if we want to combine the data sets from the two different database, then we need to think about how to make them compatible with each other as they have different structure. The second issue is about detecting and resolving data value conflicts. For the same real-world entity attributes values from different sources at different, including different representations, different scales, for example, metrics versus British units. Last but not least is the entity identification. We're integrating data, we may need to identify entities from multiple data sources, but the same entity or attributes may have different names in different databases. For example, Bill Clinton and William Clinton actually refers to the same person, but they may have different presentations in different databases. We will discuss this later in the data linkage. A very vivid example is the health surveillance system, which is quite related to our situation now. Preventing the outbreak of epidemics requires monitoring of occurrence of unusual patterns of symptoms in real time. Data from many different sources need to be collected, including the travel and immigration records, doctors emergency and hospital admissions, drug purchases in pharmacies, animal health data, etc. Such databases can have different formats and codings, and they usually contain data of different quality. Volume is another issue, as they can contain millions of records. And another issue is the privacy and confidentiality. If such data are stored and linked at the central location and without proper regulation, then there might be some privacy issues. Another two parts of data integration is data linkage and data enrichment. They're relatively more complicated and not always needed, so I won't go too deep into it, but I think it's good to know about them in case that you may encounter them in some days. Data linkage is the process of bringing together information from two different records that I believed to belong to the same entity based on matching variables. It is also known as record linkage data matching and entity identification or others. It is a challenging task if there are arrows in their key variables that are used to link the data. There are two kinds of linkage. First is a deterministic linkage. Records must agree exactly on the key attributes in order to conclude that they correspond to the same entity. It can be used when high-quality identifiers such as ID numbers are available. Another kind is probabilistic linkage. It doesn't require that all key variables match. Rather, it involves frequency analysis of the data values, which helps to calculate the weight that indicates how likely that the two fields refer to the same entity. Sometimes there are some uncommon value agreement or very strong evidence for linkage, and then they need to be included into consideration manually. For example, here we have two datasets and we need to think about which of these records represent the same person. Based on the deterministic linkage, we can set the SSN number as the reference and then the row 1 in set B will be assigned to the row 3 in set A because they have the same SSN number. However, the row 2 in set B will fail to find a match because the SSN number is missing in that row. But if we are adopting the probabilistic linkage, we can find the row 2 in set B, a replication of the row 1 in set A, as they share the same date of birth, sex, and zip code. Although the name is not exactly the same. Another thing is the enriching data. So data enrichment is the process of introducing more data. It can be disparate data from other internal sources or third-party data from external sources. The benefit is that first, it can improve the contextualization with additional data. And second, it helps enrich or validate our data. For example, to get a better understanding of advertising effectiveness, a company can enrich its internal sales data with a third-party advertisement data. Enriched data is a valuable asset for any organization because it becomes more useful and in set 4. A majority of brands conduct data enrichment on their raw data so that they can use it to make informed decisions. In 2018, data enrichment grew by 80%. So that's it for today's talk. And at last, I want to say that some of the content based on this book, Data Pre-Processing in Data Mining, which is published in 2015. So if you are interested in this and want to find more material, you're welcome to consult this book.


Hi everybody, Joe for jaskiescasebrakes.com coming at you with 2013 Panini limited football one box random team break number one. It's with EJ manual on the front of the box I don't know who else is in this 2013 class, but we're gonna do the break itself first Then we're gonna give away the Handful of spots in that big that big high-end football mixer that we have in the site in that mixer You can win a Super Bowl mixer spot a six a six thousand dollars spot, which is kind of crazy Also, if we this is a tonight only promo check the details in the schedule But if we get the requisite number of breaks done this break is involved for a big break credit promo as well So there's a lot going on on a Sunday, so thanks for joining us folks. Hopefully we can give away some break credit and a bunch of spots Let's do the break itself, and then the second dice roll will be for the giveaway part Let's roll it randomized the first list first two lists four and a six ten times one two three four Niko down to deco No, no no low-end mixers today. I'm going with a going going bigger going going home Four and a six ten times for the teams one two three four five Six all right. Here's the first half of the list right here Here's the second half of the list right over here No trade windows and filler breaks. Let's print that out And we'll re-randomize that list after we see what's in this box What are we looking for three autographs or memorabilia cards per box on average? It's a good point right there, too. Here's the final print out hot off the presses Thanks for hanging with me on a Sunday. All right. We got a Kenyon Barnard and a dual autograph coming up Here's Kenyon Barnard 299 for Carolina. That'll be for Kevin and the dual auto both Both Rams Ten pals autographs Steadman Bailey and Tavon Austin Nice, and then we got 84 out of 99 limited threads Tony Romo I'm not sure Martin. Let me answer that question after I finish this break In the middle something right yet. See you. Hey, sirs. Give me a second here 47 out of 49 Marcus Wheaton to color dual relic and autograph. I think no I think we got to still have to do a couple more things But I'll I'll give you a confirmation on that in a little bit Steelers that's gonna be for Eric. Hey the Marcus Wheaton. We'll blast in the past here 45 out of 49 Jordy Nelson Got Greg little to 399 and Sam Bradford to 399 some blast from the past it's here. All right Stephen flat had the Rams by the way had that do a lot of graph. Let's flip back over here It's gather everybody's names and we're gonna see who's gonna win some spots in that high-end NFL mixer We're gonna roll it and randomize it Four and a five nine times top five after nine. Good luck one two three four five six seven eight and Ninth and final time After nine times top six so from seven on down sad times, but if we accomplish or If we accomplish all the filling the requisite number of boxes today You'll still be part of that promo that we have going on the break credit promo So it could be still be some kind of nice consolation prize at the end top six happy times You're in that mixer Brandon Eric Niko Alex Joe and Stephen and you kind of get double spots in that break credit promo Because you'll be in the other list too for the mixer if we get it all done tonight. Jasper's case breaks comm I'll see you for all of that Throughout the day. Bye. Bye


Mexican news channel accidentally airs a man's testicles the eclipse happened and they were like send in your videos of the eclipse he positioned his balls In front of the Sun. Yeah, the Sun was blocked out, but it wasn't the moon It was just his nuts you could see some hair on and then yeah, and then the camera moves a little bit clearly his nuts Just deep lengths to get his nuts on TV for it to get on TV I don't blame them putting on TV like at the very start. You're like, yeah, this is another one You don't blame them for putting it on TV. I don't once you see that it's testicles. It's very clearly testicles Yeah, but you don't think someone's gonna do that