Hello everyone, the course of business forecasting. Today we will enter into the module of ARIMA, but before we go in details of ARIMA, we will understand the basic concept of SEF and PSEF. These two terms are very important to understand the steps of ARIMA. In ARIMA, we have auto regressive model, we have moving average process and then we have ARIMA model and auto regressive integrated moving average process that is called ARIMA model. All these three four models we will learn in detail, but before that we need to understand the concept of autocorrelation function and partial autocorrelation function. In time series models, you have the same variables data right with time. So therefore when you develop a auto regressive model, you need the correlation between the data of current lag and the past lag and their features. In order to understand that lags and the correlation between the current data and the old data, how they are interdependent among each others, you need SEF and partial autocorrelation function. So you need to study that today and then we will enter in the next day into the ARIMA model. Now even before we go to SEF and PSEF, let us understand or require the basic concept of correlation coefficient and partial correlation. If you know these two concepts effectively for basic regression analysis between any two variables, that concept can be extended to time series data and we can understand easily the SEF and PSEF functions. So first what is correlation coefficient? We all know correlation coefficient is the correlation, linear correlation between or linear relationship between two variables. If you have a variable say x and y, x and y and if you have a data set, you can find the correlation coefficient between them using excel right. So this is the formula covariance by variance of x and variance of y. And then using that formula, you can calculate the correlation coefficient of the two variable x and y. This is the linear relationship between the or how they are correlated that can be calculated. The range of correlation coefficient generally lie between minus one and one. Now the question is that if there are say three variables and you need to find the correlation between x1 and x2 and x3 is involved over there also which has an impact on x1 and x2. In that case, look at the definition the correlation between two variables may be sometimes be affected by a third variable or other variables. In that case, you need to eliminate the impact of the other variable to find the correlation between the main two variables. That means if you have three variables say x1, x2 and x3. Suppose three variables you have and you need to find the correlation between x1 and x2 or xyz whatever but x3 will have an impact between x1 and x2 on. In that case, the impact of x3 has to be eliminated. Then only the correlation of x1 and x2 can be calculated otherwise there will be a interdependency among the variables. Now that correlation is not the actual correlation. So therefore we need to calculate the partial correlation. So let us see what is partial correlation. Here I have mentioned few more examples to get a clear picture of the idea of what is partial correlation. Correlation coefficient is simple the relationship between the two variables. But in partial correlation what happens you know the third variables or the other variables might have an impact. So you have to eliminate that. Look at the example the correlation between prior experience suppose you are a candidate for job or whatever prior experience and job performance may be affected by the educational qualification. Imagine if educational qualification is another variable which might have an impact on your prior experience as well as the experience for in your profile in your CV as well as say you know job performance. Similarly you can think about that you know the written test and interview might have an impact from prior academic performance. So your academic performance will have an impact on your written test and interview. In that case if you want to find the correlation correlation between written test and interview your prior academic data performance is having an impact on that. So therefore you remove the impact of prior academic performance and then find the correlation between written test and interview completely you know as an independent variable no impact of academic performance. In that case if you find this correlation between the written test and interview after removing the impact of performance academic performance as another variable say then this correlation coefficient between written test and interview will be called as a partial correlation coefficient. This into understand and in time series we required this part this partial correlation coefficient rather than correlation coefficient both are required but these two are major concept in understanding SCF and PSF I will come to that now and then the ARIMA models. This is the basic recap of correlation and the partial correlation right now how to calculate the partial correlation the concept of correlation coefficient is easy and you can calculate it in basic Excel but for partial correlation you need some steps because the concept that example that I have given you can understand but how to execute it in Excel say or in data that I have mentioned here the steps look at the steps here how to calculate the partial correlation between the data. Suppose you have three variables as I mentioned X1 capital X1 say X2 and X3 three variables and you want to remove the eliminate the impact of X3 among X1, X2 say. So now what we have done first you take the center of the data like you take the calculate the average of each data average of each data and then like you have a data say you have the data say you take the average of these data this is a bar average of the data same way for all three variables all three parameters you know in Excel you might have you can understand that calculate the average of all of them and then take the differences you will get the center of all of them take that data okay after differencing from the mean and then let us note them with new variables say small x1 x2 x3 these are the new representative now because we have not taken that we have scaled down the data now the variation has come down only the gap or variation we are calculating now we are counting now so these variations of the data will consider of each variables now this will be replaced with small x1 small x2 small x3 now say because we have taken the differences only the differences among them from their mean this data will take forward to calculate the partial correlation among the actual variable X1 X2 and X3 now we have to remove the impact of X3 from X1 and X2 right so you regress next step you regress X1 upon X3 and X2 upon or against X3 so now first you regress X1 against X3 so we have done that remember here you do not need since we have already done the centered of the data you do not need the intercept part right only the slope part is sufficient so you do not need the intercept part force the intercept equals to 0 in your excel and then you do the regression you will find the you know direct slope which will pass through origin not a matter so you found the relationship between the X3 and X1 how X3 is explaining on X1 or how X1 is explained the new variable the scale variables after taking the center of the data how the X1 which is represented of capital X on main variable your objective is to calculate the correlation or partial correlation between capital X1 and capital X2 but X3 is coming into the picture which is impacting X1 and X2 so you have to remove eliminate the impact of that that you are going to do now so now what you do you regress the new scale data X1 center data X1 with X3 that means how X1 is explained by X3 right the variations this we have developed similarly we calculate regress X2 against X3 that means how X2 small x2 is explained by X3 that you can calculate take the difference of them you will find a new component say U1 and U2 I have noted here so these are the new component the differences between the scale data of X1 and its regress data with X3 so this values you will get suppose this is component one between X1 and X3 of regress data and this is say X2 and X3 so these two data if you collect and if you calculate the correlation coefficient whatever this will be called as a partial correlation coefficient actual correlation coefficient partial means actual without having any impact of X3 so that we have removed now and this is nothing but the partial correlation of two variable where the third variable impact has been removed look at here therefore the correlation between U1 and U2 represent the partial correlation clear now I will explain that using excel later so these are the two basic concept correlation coefficient here which we have discussed in this slide and partial correlation which we have explained the steps here the calculation process the concept is mentioned here but the steps of calculation is mentioned here so these two will help you in calculating SCF and PSF now which will go to Arrimer model now look at this formulas here you can see no term this particular formula in many books this will be mentioned here using this formula you can calculate in excel you can calculate the PSA what is that the partial correlation between 1 and 2 X1 and X2 given X3 equals to nothing but the correlation coefficient between 1 and 2 minus 1 and 3 into 2 and 3 and this this so this formula Arrimer popular formula you can use this formula to calculate the partial correlation so these steps you can get otherwise you can follow these particular steps that I have mentioned or direct this formula anyone you can follow we will follow these particular steps that we have discussed today now look at the auto correlation now we are going to go to the time series data now we have entered into the time series data right now in time series data we have same data say y right so y1 y2 only one variable you do not have any you know independent variable here like this you do not have any independent variable and you need to find the correlation among the data what does it mean it means that you want to see how your current data are being impacted by the past data that means how say you are at this position say how your y6 is impacting by y5 how your y5 is been impacted by y4 or how y4 is explaining y5 that you want to calculate that we call it is a lag past like 1 lag 2 lag 3 lag like that this way how like this is stock price or say you know temperature how yesterday's temperature has a impact or how much impact it has in today's temperature so that correlation that linear relations we want to find this relationship we call it is a auto correlation why auto auto correlation coefficient because this correlation coefficient are not from 2 independent variable 2 different variable not it is not like that x and y and you are finding correlation coefficient between them here effectively you are having the same data sets and among them you are trying to find the correlation therefore we call it is a auto correlation more detail I will explain this is the formula you can see covariance between current data and say k pure lag right and that correlation you want to find say y4 so y6 and y5 or y6 and y4 y6 and y3 so the impact of the older lags or older data you can also calculate through correlation coefficient we will understand the first case like only one period lag and then you can extend that concept to the older like like multi period lag also so this is same logic covariance of current and the older data that current corresponding period you want to consider so suppose this is y6 and this is suppose y3 so you want to calculate the covariance correlation coefficient auto correlation among them so you calculate the covariance and the variance among them also and this will come as a to some extent say you know square because it is the same data so ultimately it will be the same almost and then you can calculate using this formula right so now we will find the plot as well as the lag impact on them right so here let us take one practical example and we will plot the SCF as well as the concept calculations we will illustrate here now suppose we have taken 3 months data of TCS from NSC nesla stock exchange we have taken the TCS data actual data we have taken and we will understand the SCF the basic correlation coefficient you know right basic correlation coefficient you know which I have shown you in the previous slide and the basic calculation formula you can calculate between two variables first let us understand the SCF and then we will go to the TACF now remember here it is a time series data it is a time series data you do not have any you know kind of two variable three variable same variable here and the other say stock price of TCS how will calculate the SCF what is the concept let me repeat it again suppose here you have the data say y this actual data suppose here y 1, y 2, y 3, y 4, y 5, y 6, y 7, y 8 dot dot dot then what you do in excel you take a new variable say suppose this is y t this is y t minus 1 just copy this data and paste one row below in your excel y 1, y 2, y 3, y 4, y 5, y 6, y 7, y 8 like this remember this data we have copied this data we have copied and pasted one row below now you delete the first row and delete the last row say delete the last row now you have a pair of data look at this pair of data let me put a highlight you will get a letter clarity look at you have this data and you have these data sets right we have the pair of data now same pair of data and if you ask the computer or your excel that find a relationship between this data set and this data sets so excel will consider this as one variable say and this is another variable say so this suppose excel will consider let me open the pane again so excel will consider this say you know one variable say say x 1 and say this is to say say x 2 or you can say excel will consider this as a y t and this is y t minus 1 so that means say you know or say as I mentioned x 1 and x 2 say so it is a yesterday data say this is nothing but yesterday data because you have taken the t c s stock price so here this yesterday this this y 1 has come here you can think then y 2 has come here y 3 has come here y 4 has come here it means that y 5 now you have taken a pair right you have not taken a pair so y 5 will be dependent on y 4 this is your say y this is your x or x and y whatever your x 1 x 2 you can consider so now you have a pair of data say just all yes these are all yesterday these are all yesterday actually right as compared to this first column and these are all today these are all today so wherever you are you are today you appear at y 7 y 7 here today stock price y 6 y 0 yesterday stock price but why y 6 you have kept in your excel in the same row of y 7 so this pair of data if you give it to excel and if you ask excel to calculate the correlation coefficient excel will calculate the correlation coefficient suppose 0.8 but this correlation coefficient are not simple correlation coefficient this is auto correlation coefficient why because this is the auto data same data done we understood the correlation auto correlation coefficient or auto correlation function now we have to plot it right so what do you do you take the data and copy in the next column but one row below then drop delete the first row and the last I will show you in excel and last row then you have a pair of same data sets you ask the excel to calculate the correlation coefficient simple correlation coefficient formula that I have shown in the previous ppt you can slide you can calculate so this is nothing but correlation it is but actually auto correlation coefficient right so now you draw that so here your a cf auto correlation function and this is your say lag right time say or you can say lag so now with the same data if you draw the same your correlation coefficient will be always one right if you copy the same data y1 y1 y2 y2 it will be always one so we are not interested about that we what would like to study from the first leg first period so leg one you consider leg one so this this correlation now yt and yt minus one or today and yesterday you suppose you found 0.8 suppose you found 0.8 you draw it 0.8 now you calculate you might say the rates are of my stock price might depend on debut for yesterday also so today's back also so why you are not considering that correlation among them it might have an impact of today's stock price also so you copy their data again and then paste from here so y1 y2 y3 y4 y5 y6 like this this and then you drop the first two row just delete the first two row and the last two row you will have a pair of data now here you can see y1 has come here sorry it will start from here y1 y2 y3 y4 y5 like this so you can see look at this data set now look at this let me put a different color highlight with green so look at this this data set now even here you can you can see this so here effectively your y3 will depend on yesterday y2 and y1 so current stock price will depend on yesterday's stock price and debut for yesterday stock price now you want to find the correlation among them this correlation are called the autocorrelation also so now this will give you the SCF like you know AR model auto regressive model of order 1 or order 2 I will discuss in the next class but now let us focus about simple SCF and PSA so here you can see this relationship between current and debut for yesterday one day back if you find the correlation you might find it here same maybe say 0.6 so it is 2 days like and then again if you put one more data you may find a lack correlation between them say you know 0.7 say 0.5 so these are nothing but the SCF plot or correlogram one part of correlogram that is called SCF plot PSA we will discuss later now so this is what the SCF and plot here I have shown you the calculation process here you can see we have the actual data then copy them and paste one row below one row below one row below one row below and now suppose you want to calculate the SCF of order 1 SCF of order 1 that means you just take current and the 1 lakh below yesterday and you calculate the drop first delete first row and the last row you will get a pair of data you calculate your first correlation coefficient auto correlation coefficient because at the same time series data therefore we have to use the word auto auto correlation coefficient same formula simple we are using the word auto because you are same data you are copying and you are considering as a dummy independent variable and therefore we call it a SCF of lag 1 then if you calculate current and the 1 lakh older so it will be a 2 days back data so in that case the correlation between them will be called SCF of lag 2 and you can calculate here look at here you can take that data and directly all the data you can take and you can if you have a larger amount of data directly you can calculate your SCF graph function through this formula of correlation and you found the lag and the corresponding correlation you can see so here you know with the immediate data you have a maximum impact the correlation is very strong the next day correlation is little lower then it is reducing actually the other means older data of TCS are not impacting to the current data much but there is a relationship there is a relationship this is called the SCF plot drawing in correlogram clear everybody I believe now let us go to the next level that is called PACF calculation same logic just you need to understand the partial auto correlation not the actual correlation because the auto part are being there with the same data but the point here is that here in this case in this case you have calculated the direct correlation right so here you can see how much in the first case it is 95% the correlation the yesterday stock price is impacting to the stock price with almost 95% that it is explaining look at the day if you rest it is also 90% 90% but the practical case is that this is the direct correlation but the practical case since it is auto data the second day data like day before yesterday data will not have this much of impact 90% impact with your current stock price the reason is that there is an intermediate data called yesterday which was in between so when you take the data of yesterday in your current data and calculate the correlation this is fine but when you go one luck back or one day back day before yesterday in that case your the impact of day before yesterday data in the current data should not be the same you have to take the partial impact of that because yesterday some day was a today then day before yesterday was a yesterday and then yesterday was a today so that impact also is there already so you cannot take direct value of day before yesterday into your current you have to see the remove the impact of yesterday between the current calculation correlation between today and day before yesterday this is called the partial correlation let's see how we can calculate it this is the formula same formula PSA with the same luck now now we are not taking the x1 x2 etc we are taking the same auto data so you can take YT the not a general formula people use like this YT with how many luck like 1 lakh 2 lakh 3 lakh if you take 1 lakh then YT and YT minus 1 if you take 2 lakh then YT and YT minus 2 so the relationship is said to be direct and exclude the relationship with the intermediate lag intermediate lag has to be removed suppose that for example if you want to calculate the impact of say today and yesterday today and yesterday is fine but today and day before yesterday it may not be like this it may be very less amount the reason is that it is a partial impact you cannot take the full input the intermediate lags impact has to be removed right so that we are going to do through PSA now what are the first steps I have already mentioned that simple partial correlation calculation steps same calculations we will be using here but in that case we have talked about x1 x2 and x3 and we have seen the impact of x3 on x1 and xx1 and x2 that we are not going to do here what we will do we will consider here now YT current I would say YT and then YT minus 1 day before yesterday and day before yesterday right we want to see the calculation the PSA between these two YT and YT minus 2 these you want to see YT and YT minus 2 we want to see this rather I will write the arrow here this side so the impact of this YT minus 2 on YT the correlation we want to calculate but this correlation should be called as a partial correlation or partial auto correlation because it's auto data same data same time series data you have put in a different column only but it's the same data but Excel will understand these as a independent some variables but we know it is auto data so therefore we need to find these two are same same simple partial will be the same as it is auto correlation because it's just yesterday direct impact you have to take but for debut yesterday you cannot take the direct full impact you have to take partial impact because because intermediate yesterday is already been involved over there so you cannot take the direct impact because it will be very less impact because you are taking full impact of yesterday so only partial impact of debut yesterday you have to consider how to calculate it same logic here what you have to do in this case here you have to remove the impact of YT minus 1 from YT and YT minus 2 and then center the data and calculate the actual partial correlation between YT and YT minus 2 so here steps are center the data of all these three data you might have some you might say so these are the same data only one row below etc so how to calculate centers and everybody will have the same center average data everybody will have the same average because one data if you delete it will not have much impact if you have a longer amount of data so everybody will have the same average and then you calculate the center center value will be different so these data this the scale data will consider and then we'll calculate the partial correlation so here first to the center data then calculate the slope of YT and YT because we want to calculate the PSCA between YT and YT minus 2 debut yesterday capital Y say we want to calculate that that is our objective the partial between partial autocorrelation function or partial autocorrelation value between YT and YT minus 2 we want to calculate that right debut yesterday impact on current the partial impact of that actual impact of that in today's stock price so that you want to calculate so center the data first for every three all three today yesterday and debut yesterday then calculate the slope like the regress you have to calculate the slope and regress and slope of YT against YT minus 1 slope of YT minus 2 against YT minus 1 because here we want to remove the impact of yesterday from both from YT as per the formula PSCA from YT and YT minus 2 and then once we'll get the you know impact less data between YT and YT minus 2 and then calculate your correlation that will be a partial correlation then calculate the component of YT and you know YT minus 2 component you can say error not error but it's a gap differences right and then calculate the correlation between them this correlation are nothing but the partial correlation let's see and how we can draw you'll see the actual impact it will be very less actually not the actual partial the actual correlation here you can see the actual correlation 90% 85% but it will first time will be the as it is but after that you will see the partial will be very less maybe these these these it will be fall down quickly because older data will not have a much impact because it is a partial impact we are considering let's see how it works now the steps center the data then regress after taking the same center data regress the scale data or the impact of current we have taken a notation of YT minus white small YT in the previous PA partial correlation slight I have shown that so same notation we'll consider so YT the impact of YT minus 1 yesterday scale data yesterday on YT and impact of YT minus 1 on YT minus 2 here we are calculating the correlation between the partial correlation between these two so we will consider these two variables and we will remove the impact of YT minus 1 remember this differences this concept here now if you regress now this we are considering as independent variable now after scaling scaling down the data so this impact we want to calculate right and you want to remove so what do you do regression and calculate the slope and the corresponding regress regression data regress data and then you will find the difference between them as your new component the gap similarly YT minus 1 on YT minus 2 and you will get the component so these two component you will get ET and ET minus 2 calculate their correlation simple correlation that will be a partial correlation so now you take these two and calculate your partial you can either use this formula or that formula or the steps that I have mentioned let's see so here you take the actual data then one day back look at YT minus 1 and 2 days back just copy and paste and you can get to know like this I have removed all this I'll show you in Excel and then take the average of this data average of this data average of this data now you are in Excel now Excel have three variables this variable this variable this variable and you want to calculate the PSN now Excel will not understand the same data but you know it is auto data so they everywhere will use the auto part auto work otherwise it is just simple partial correlation we are calculating the steps that I have mentioned earlier so now we calculate the center of each of them average of all of them all this data to some extent average value will be the same and then you calculate the center so this minus this this minus this this minus this for IT current data now you calculate the center this minus the average data will get the center of second data second you have the yesterday data t minus 1 data and center of depth if you yesterday data that is t minus 2 data so this is the scaled down data so this is we are using small yt small yt minus 1 note as a notation small yt minus 2 right so these notations we are considering now the capital yt yt minus 1 and yt minus 2 for the time being we are not considering scaled down data we have taken now we call it is center data take the center data calculate the slope between these two slope between these and then once you get the slope you calculate your integration value and that will give you the component the ET and ET minus 2 these two components the current like you know partial data will give you the partial correlation or the difference data will give you the new partial correlation or PSEM value so now what you do you can think about say you know center data you have already calculated now calculate the slope between these and this look at here similarly calculate the slope between these and this look at here you have calculated the two slope note down once you get the slope you can get the regress line y equals to mx y equals to because intercept we have considered 0 so that and this the additional parts say the component but none to add the difference this minus this you will get the component here we have done that look at two slope we found corresponding regression line you will get here we have calculate the regression line you can see and if you subtract that from your actual data now center data is your actual data right to some extent though it is a scaled data we are considering actual data so you take the difference sorry you take the difference among them you will get the component new component now you have a new data fresh data after subtracting the you know impact of the intermediate data and then calculate the correlation between these two data set this will be nothing but or we consider auto auto partial correlation coefficient clear here you can see you can calculate this this data sets are nothing but this and you calculate your correlation between them this is nothing but effectively found the partial correlation between capital yt and yt minus 2 so yesterday today and day before yesterday not yesterday this correlation found this is called partial correlation earlier it was 90 percent now you found only minus 10 10 percent imagine that you can see here earlier it was how much here you can see 90 percent look at 90 percent there if you restart the impact on current data but now you can see even here also 86 83 you can remember it will fall down suddenly because partial impact you are considering this huge impact will not be there so therefore you cannot take in Arima models I will discuss in the next class you cannot take a shape graph and corresponding impact you have to take the PSA graph in calculating your lag in developing your regression model auto regressive model in Arima I will discuss that detail so here you can see it was currently 9 ACF value is 90 percent auto correlation but partial autocorrelation was minus 0.1 you can see here it is minus 0.1 here you can see the further graph minus 0.1 and then you can see 0.3 only 0.03 3 percent only here it was 86 percent earlier here it was 90 percent here it was 90 percent you can see so it is drastically falling down look at here so this is what the first one immediate data will have a almost same data line 95 percent whatever then onwards you will see the impact of it this is what the partial autocorrelation coefficient or partial autocorrelation function and this is called the correlogram of PSA graph the previous one the correlogram of SCF put together we call it is a correlogram of time series data now this part we understand in excel and then we will wind up today's session let's go to excel so here we have taken the TCS data of last three months say three months so October to October 2023 to December 2023 three month data we have taken so here the data we have taken from the NSE site and you calculate the SCF first you can understand the SCF look at the tag here in the sheet SCF calculation simple you copy the data and paste one row below like copy and paste one row below here I do not have to copy because my data will change calculation will change but you can do that here right so you can see 2 lakh 3 lakh just you have copied and paste now you want to calculate the SCF between current data YT and YT-1 simple calculate the correlation coefficient you will find 99% this one will find this one as the correlation between current and yesterday right or one day lack data one day period old period data so it's a TCS data for stock price you can consider as I said Dow Jones or Nasdaq data also you can consider you can also consider say you know gold price you can also use the temperature or say crude oil wherever you want to apply the Arima model these concepts will help you in developing the Arima model today we are discussing the background SCF and PSF concept once you understand this Arima model will be very easy to understand just you know maybe 10 minutes will be taken to understand the AR process and then MA process then ARMA process one by one we will discuss in the next one one or two session so now if you want to calculate the lack between current and yesterday one lakh one or lack of time series data to take this data and this data calculate the correlation coefficient now you want to calculate 2 lakh 2 period land data so consider this data and 2 days back data like I mean the lack 2 of day before yesterday data you consider and you calculate your correlation coefficient like in that case you take data from here and so here so what you do correlation between say these data let me say today and then we will consider say so we take from here till this last data we will delete calculate so you have calculated the correlation between yesterday today and day before yesterday here I believe it is clear to everybody now look at this this and this so this is nothing but the correlation between current data and day before yesterday so this is what your SCF calculation similarly here I have drawn the graph correlogram graph here you can see 0.95 then 90% then 86% all this data we have plotted here they are SCF calculation now you go to PSEF five steps first step is that center the data first you calculate the average so we have taken the average of the data so what we have done we have suppose only three period we have taken today yesterday and day before yesterday more detail you can develop in different software say you know Ithon etc but the basic steps of PSA calculation I have shown here so now this is your current data this is your yesterday data this is your day before yesterday right so now you have to take the center of the data the scale data you have to take so first you calculate the average of all of them everybody will have the same average as I mentioned in the PPT take the center of the data current minus average for everybody current minus average for everybody current minus average now this data look at this column number f and g this data is your center data right now what you do you calculate the slope between current and you want to see we see the partial correlation between yt and yt-2 right and you want to remove the impact of yt-1 so this day before yesterday how it is impacting to your today's data that you want to calculate right so therefore what you do we calculate the equation between current and yesterday with scale data center data and between day before yesterday and yesterday yesterday will be your with scale data will be your independent variable here to calculate the regression so now here you can calculate the you can calculate the you know slope between current and yesterday with center data and slope between day before yesterday and yesterday look at day before yesterday and look at h is first and then g is coming second. So calculate the correlation or slope between them also. Now using these slopes you can calculate your components. What is that? The current minus the regress data. Current minus regress data so you know slope into this value minus actual minus the current minus slope into this value. So you can calculate using this formula you can use this formula look at my mouse and you can calculate the center new component data for everybody look at here. So you are taking first h and then you are subtracting the slope into yt minus 1 look at slope into yt minus 1. You are subtracting that from yt minus 2. So yt minus 2 is your main data variable data and yt minus 1 is your independent data for the sake of understanding. So you are seeing the impact of that. So now you found these two components k column k and column l after eliminating the impact of yesterday. Now these are fresh data. Like you know the impact or the dependency of yesterday is removed from both from today's data and day before yesterday's data. Now you have the fresh data today and day before yesterday. Find the correlation among them. Simple correlation among them. This correlation is nothing but the partial correlation between actual stock price of today and day before yesterday. Between the actual temperature of today and day before yesterday. Between the actual kudal price of today and day before yesterday. This partial correlation coefficient you have to consider in your algorithm model. How? Why? That we will discuss in the next class. We can see the PSA here. So this if you come back to your PPT you can see this I have mentioned here. Here you can see this 0.1 minus 0.1 and 0.03 are coming here into your PSA calculation. Not the actual correlation it is a partial correlation. But in PSA because 90% 86% but here it is just 0.1 0.3 kind of thing. So imagine this is what the PSA. This we need to carry forward to the algorithm In the next class we will discuss the introduction of auto regressive model and the moving average process and how the SCF and PSA concept are being used there to evaluate to calculate the AR model and MA model or ARMA model that we will discuss in detail. Thank you.


OK, Shoshanim, another interview between the long-suffering Scott Douglas Jacobson and the insufferable ostentatious Sam Wagner and today's topic is in sheep's clothing, victims who are victims just in name. Like rhinos, you know, republicans in name only, vynos, victims in name only. Today's interview was published yesterday in news intervention. In the description area you will find a link, click on it and it will take you where you have never gone before. And this is a threat, not a promise. OK, OK, enough nonsense, let's get to business. My name is Sam Wagner and I'm the author of Malignant Self-Love, Narcissism Revisited. What else? And a professor of psychology. And I just gave an interview to news intervention about the issue of victimization, victim movements and so on and so forth. I read to you the interview. Scott Douglas Jacobson, what defines victimization, he asks, what defines a real victim in contrast to a fake victim? The ineffable Sam Wagner response, victimization involves the denial of the self-determination, identity, self-actualization, rights and boundaries of a person without their express consent and collaboration. Dazzle Jacobson, what makes victim identity movements, in fact, movements? Wagner, when victimhood becomes an organizing and explanatory hermeneutic principle, when victimhood becomes a determinant of the victim's identity and a socially binding force centered around grievances, prosocial or communal grandiosity, entitlement, conspiracism, paranoid or persecutory delusions, aggressive engagement or, on the other end of the spectrum, schizoid withdrawal, dis-empathy, lack of empathy or declining empathy, defiance, reactance, and contumatiousness, a rejection of expertise and authority. When you have all these together, we have on our hands a victim identity movements.  These are actually narcissists with grandiosity, aggression, paranoid ideation and so on and so forth. So that's a prime example of a movement in the making, coalescing as we speak in front of our eyes. And I say in the interview, no one is a victim. I repeat, no one is a victim. We may end up being victimized, but it doesn't render us victims for life. It doesn't brand us. Jacobson, some studies in British Columbia, as you have noted, found some victimhood movements have been hijacked by narcissists and psychopaths. How does this muddy the waters of the real justice movements and make them ineffectual? Wagner, this was not the only study to have unearthed this very disconcerting undertow. We are beginning to wake up to the reality of what Gaby and others called the tendency for interpersonal victimhood, or TIV. These are professional or career victims with emphasized narcissistic and psychopathic tendencies. And they find new homes, pathological narcissistic spaces in these social justice upswills. It makes it difficult to tell apart legitimate evidence-based grievances from entitlement fueled manipulative and counterfactual claims. One helpful way to distinguish the two, in other words, one helpful way to distinguish between narcissistic, psychopathic controlled movements and real social justice grievance fueled movements. One way to tell the two apart is by noting that narcissists and psychopaths are destructive. They are not solutions oriented. Narcissists and psychopaths thrive on negative effects such as anger and envy. They're loath to invest in the routine and tedious chores attendant upon rectifying wrongs and building a better world. And I refer Jacobson to a video of me titled Victimhood movements hijacked by narcissists and psychopaths. I want to summarize this segment for you. It's a bit difficult to grasp at first sight or upon first hearing. So there is growing evidence that social justice movements and victimhood movements, grievance fueled movements are being taken over and hijacked by narcissists and psychopaths. This makes it very difficult to tell the difference between legitimate evidence-based grievances and entitlement fueled manipulative and counterfactual claims. So how can we tell the difference? Narcissists and psychopaths are destructive. They don't seek solutions. They don't work in the long term. They don't try to build a better world. They don't try to rectify wrongs. They are into the public display. They are into the spectacle. They are ostentatious. Okay, Shoshanim, we move on. Jacobson asks, what have been extreme historical cases of this going awry, as this phenomenon has been historically cyclical, including one close to home in 2004?  And to a lesser degree, of course, the white man's grievance movement implausibly headed by Trump, Donald Trump, is a more recent example of such subversive dynamics. Jacobson, what is the typical art of development of victim movements? Vaknin, the sociologist Bradley Campbell suggested that we have transitioned from a culture centered around dignity to one based on victimhood. You can learn more by reading Habermas, Fukuyama and Fukuo. All justice-seeking movements start with grievances, injustices. They decry and seek to remedy and reverse individual transgressions. And this was typical of the narcissistic abuse movement online as well. They seek to somehow address societal and cultural biases, implicit or explicit, discrimination, suppression. These are the victimhood, these are the movements, the justice-seeking movements, the societal activism movements at the inception at the very beginning. The victims organize themselves around exclusionary identity politics and intersectionality. And it is disorientation, disorientation for, I'm a victim, I'm the exclusively victim, I'm the only victim, I'm the most victim. No one has been victimized before like me, and I'm a victim on multiple levels.  So this, exactly this orientation, results in grandiosity and entitlement. In other words, in growing narcissism, increasingly more aggressive, these movements often become psychopathic, defined and consummations. They demonize the other. Left-cleaning victimhood movements center around entitlement and reparations claims on the majority, on social institutions and on history. This is a grievance focus or a grievance-centered approach on the left. Right-wing movements, because they are right-wing victimhood movements, of course, ask any white supremacists. Right-wing movements are conspiracy-minded and they are avoidant. They don't engage society in an attempt at reform, but they're avoidant. They're also more violent than left-leaning movements. Narcissists and psychopaths gravitate towards such movements in order to obtain narcissistic supply, power, money and sex. So these movements become the watering holes of narcissists and psychopaths. They see a new victimhood movement, they're all over it. They become the public faces and the media darlings of these helpless victims. Having hijacked the legitimate complaints and demands of the victims. Jacobson, how much of the online content of narcissism and psychopathy is garbage, worthless or worse now? Vaknin, about 90%, 90%. It is not only worthless, wrong. It is dangerously misleading and entrenched as a lifelong, self-defeating and self-aggrandizing victimhood stance, even as it demonizes and mythologizes the abusers. Jacobson, what is the tendency for interpersonal victimhood, or TIV? The vaknin, a series of two studies by Israeli scholar Gabbai and others published in 2020. And in these studies, the authors provided this abstract. In the present research, say the authors, we introduce a conceptualization of the tendency for interpersonal victimhood, TIV, which we define, say the authors of the studies, which we define as an enduring feeling that the self is a victim across different kinds of interpersonal relationships. So I'll read to you again the definition of tendency for interpersonal victimhood, TIV, an enduring feeling that the self is a victim across different kinds of interpersonal relationships. The authors continue to say, then in a comprehensive set of eight studies, we develop a measure for this novel personality trait, TIV, and examine its correlates, as well as its affective, cognitive and behavioral consequences. In part one, we establish the construct of TIV with its four dimensions. In other words, need for recognition, moral elitism, lack of empathy, and rumination. And then we assess TIV's internal consistency, stability of a time, and its effect on the interpretation of ambiguous situations. In part two, we examine TIV's convergent and discriminant validities, using several personality dimensions, and the role of attachment styles as conceptual antecedents. In part three, we explore the cognitive and behavioral consequences of TIV, specifically we examine the relationships between TIV, negative attribution, and recall biases, and the desire for revenge, and the effects of TIV on behavioral revenge. The findings highlight the importance of understanding, conceptualizing, and empirically testing TIV, and suggest that victimhood is a stable and meaningful personality tendency. I'm going to read to this to you again. Victimhood is a stable and meaningful personality tendency. Again in the description, you will find relevant links. I refer Jacobson to another interesting study. It's a study titled, Signalling Virtuous Victimhood, as indicators of dark triad personalities. It was authored by Ekin Ock, Yikian, and others. It was published in the Journal of Personality and Social Psychology of the American Psychological Association in May 2020. And here's the sentence and excerpt from this study. New research provides evidence that narcissism, psychopathy, and Machiavellianism, maladaptive personality traits known as the dark triad, are associated with overt displays of virtue and victimhood. I'm going to repeat this mind-boggling sentence. Evidence shows that narcissism, psychopathy, and Machiavellianism, manipulativeness, maladaptive personality traits known as the dark triad, are associated with overt displays of virtue and victimhood. The study suggests that people with dark personalities use these signals of virtuous victimhood to deceptively extract resources from other people. It's pretty straightforward, I would say. Stand, Jacobson. What are the primary signifiers of narcissists and psychopaths who have or might hijack legitimate victimhood justice movements, looking for money, power, and sex, sex, vacuum? Ironically, these narcissists and psychopaths who latch onto social justice movements, they're usually pro-social or communal narcissists and psychopaths. They often control from the bottom. They emotionally blackmail other people by being self-sacrificial or by posing to be victims. So the infestation of victimhood activism by narcissists and psychopaths is the tip of a submerged iceberg of ersatz altruism. Some narcissists, for example, are ostentatiously generous. They dedicate time and other resources to social justice movements and to activism. They donate to charity, lavish gifts on their closest, abundantly provide for their nearest and dearest, and in general are open-handed and unstintingly benevolent. But all this is a facade, it's fake. It's a form of virtual signaling. How can this be reconciled with a pronounced lack of empathy and with a pernicious self-preoccupation that is so typical of narcissists? And so we need to understand that ostentatious giving is a way to extract narcissistic supply. Posing as a victim is a way to garner attention. The act of giving enhances the narcissist's sense of omnipotence, his fantastic grandiosity, and the contempt that he holds for others. It is easy to feel superior to the supplicating recipients of one's largesse. Narcissistic altruism is about exerting control and maintaining control by fostering dependence in the beneficiaries. But narcissists give for other reasons as well. The narcissist flaunts his charitable nature as a bait. He impresses others with his wealth or selflessness and kindness, and in this way he lures other people into his lair. He entraps other people, he manipulates other people and brainwashes them into subservient compliance and obsequious collaboration. People are attracted to the narcissist's larger-than-life persona and posture, only to discover his true personality traits when it's far too late. So it's a kind of give a little to take a lot. This is the narcissist's creed and motto. This does not prevent the narcissist from assuming the role of the exploited victim. Narcissists always complain that life and people are unfair to them in justice. Narcissists complain that they invest far more than their share of the profit.  Narcissists are likely to say, she's getting out of our marriage far more than I do, or I do all the work around here and they are getting all the perks and benefits, or I'm a perennial victim, I should be representing the victimhood movement. Some narcissists are actually compulsive about all this. They're compulsive givers, for example. And to all appearances, the compulsive giver is an altruistic, empathic and caring person. Giving is not only about money. You could give time, you could give energy, you could give resources, you could give your place to be used, you could give... But if it is done compulsively, it's a problem. Again, on the surface, the compulsive giver is altruistic, empathic, charitable, caring person. But actually, he or she is a people pleaser and a codependent. The compulsive giver is trapped in a narrative of his own confabulation. How his nearest and dearest need him because they are poor or young or inexperienced or lacking in intelligence and good looks and are otherwise inferior to him. The act of giving makes him feel superior. Compulsive giving therefore involves pathological narcissism. In reality, it is the compulsive giver who coerces cajoles and tempts people around him to avail themselves of his services or money. The narcissist forces himself on the recipients of his ostentatious largesse and coerces the beneficiaries of his generosity or magnanimity in full public view. The narcissist is unable to deny anyone their wishes or requests even when these are not explicit or expressed and are mere figments of his own neediness and grandiose imagination. Some narcissists, as I said, are ostentatiously generous. I don't know, they donate to charity, they volunteer, they lavish gifts on their closest, abundantly provide for their nearest and dearest and so on and so forth. And so, again, it makes them feel superior. There is a subgroup of narcissists, co-dependent narcissists and so on. These are people-pleasers and they also gravitate towards victim movement. People-pleasers dread conflicts and they wish to avoid them. They are conflict-averse. And so they need to believe that they are universally liked. They are always pleasant, well-mannered, civil, conflict-averse, but at the same time they are evasive, vague, hard to pin down, sometimes obsequious and generally a spineless non-entity. This is the profile of the covert of vulnerable narcissists. These qualities are sub-defeating as they tend to antagonize people rather than please them. But conflict-aversion is only one of several psychodynamic backgrounds for the behavior known as people-pleasing. And these other components or elements in people-pleasing, in conjunction with the conflict-aversion, drive these covert and vulnerable narcissists into victim-hood movements. There's kind of power in numbers. Some people-pleasers cater to the needs and demands of other people as a form of penance or self-sacrifice. Many people-pleasers are codependence and they strive to gratify their nearest and dearest in order to allay their own abandonment anxiety and ensuing, intense and at times life-threatening dysphoria. It's kind of like, I am nice to him so he won't break up with me or if I cater to her needs she will never leave me and so on. Then there are people-pleasers who are narcissists. Pleasing people enhances their sense of omnipotence, grandiosity. These kind of narcissistic people-pleasers seek to control and disempower their charges, the people who benefit from their services. And so they say, she so depends on me and she so looks up at me, to me. She so looks up to me that I cannot abandon. Even the pity of these self-pleaser narcissists, even their pity, is a form of self-aggrandisement. They say, only I can make her life so much better, she needs me.  These narcissistic people-pleasers are misanthropic altruists and compulsive givers. All people-pleasers, narcissistic or not, use common coping strategies which they import into victim-mood movements. Number one, dishonesty in order to avoid conflicts and unpleasant situations. Number two, manipulation to ensure desired outcomes such as an intimate partner's continued presence. Number three, fostering dependence. Co-dependent people-pleasers leverage their ostentatious helplessness. They manifest weaknesses in order to elicit the kind of behaviors and solicit the benefits that they are angling for, while narcissistic people-pleasers aim to habituate their targets by bribing them with gifts, monopolizing their time, and isolating them socially. And all these strategies are leveraged and imported into victim-mood movements. As they grow, look what's happening with Black Lives Matter, with the mid-to-movement, and so on.  And finally, the self-punishment, self-defeat and self-sacrifice to signal self-annulment in the pursuit of people-pleasing. Some of these strategies are not typical of narcissists and psychopaths, but all of them are typical of many social activists. Exasperated Jacobson, what historically speaking can be done to combat these cluster B bad behaviors connected to some social movements? Vaknin. As the grievances of these movements are addressed, these movements become a part of the establishment. This is when the hard work begins, the labors of writing laws, regulatory oversight, politics, negotiations, and compromise, and the tedium of perseverance and routine. These newfangled demands on the psychological and logistical resources of the movement and its adherents drive narcissists and psychopaths away. They are unaccustomed to and they reject the hard slog and the often Sisyphian undertakings of public policy. In other words, they're lazy. And then Jacobson, thank you so much for the time and opportunity, Dr. V. Dr. V, is it V for victim or V for Vaknin? Just kidding. Thank you for suffering me yet again. And to you too.


Welcome to Toffy TV. It is the Everton News Daily. Everton have today made their second signing of the January transfer window, bringing in Nathan Patterson from Rangers, 20-year-old right back Scottish International for a fee of a region of £11 million, which looks set to rise to £16 million, with all things considered. The player we were after in the summer, obviously we didn't reach the valuation of Rangers, but now we have, and that right back that we've all been desperate for for so long has joined the club, and fantastic news only a couple of days after we got Michael Enco signing over the line as well, so a left back and a right back in January, just what we needed with being in such a poor run. And hopefully both can be involved in Saturday's game against Hull in the FA Cup. Patterson will wear the number three shirt, the shirt left vacant by Leighton Baines. Not a fan of that myself, not a fan of right backs, weird and left backs numbers, but hey ho, hopefully in the summer he can take the number two shirt up, or I don't know, just maybe make the number three shirt his own. But not a massive fan of that as I say, but fantastic news to get him in, as I said, just in 20, you know, six foot player can get up and down that right side, and we were so used to Seamus Coleman for so many years, bumming up on the right side, you know, it's not been the same obviously for a while, but hopefully he can come in, get used to the Premier League and do that job for us. He signed a five and a half year deal as well, so big commitment from the club to the player, and hopefully he settles in well, and you know, there's been some really good links between Rangers and Everton, you know, players coming into Rangers like to Duncan, from Rangers like to Duncan Fakes and Paul Rideout, Mo Johnson, and going the other way, they like to Trevor Steven, Gary Stevens, we've had the likes of Richard Gough and David Ware play for both clubs as well, there's been a lot of plays, we've played for both clubs and had a lot of success on both sides. So, let's hope Nathan Patterson continues that trend, so good luck to him. Everton have also been linked with Sean Longstaff and Ross Barclays today, you know, Rafa Benithas did reveal that he wanted, you know, two or three, and I think he said on Friday he's got a couple in, and I think the next one Everton will probably be looking at is a midfielder, a centre midfielder, you know, you know, whether it's a number eight type play, you can drive us forward, or maybe you're slightly more defensive player, you can help Abdelhaid, the court, they start driving us forward, then we'll have to wait and see, but obviously a lot of links this time of year, but Sean Longstaff does seem to be someone that Everton have been linked with for a while, and Rafa Benithas has worked on before he is out of contract in the summer at Newcastle, so once keep an eye on, definitely might just be someone to bolster the squad, someone young and upcoming to bolster the squad, we'll have to wait and see. Ross Barclays obviously slightly different with the history and baggage between him and Everton Football Club, and he's also, I think he's 28 now as well, his careers just seems to have passed by in a flash, so we'll have to see what happens. Mason Holgate has been linked with a move away from Everton today with Burnley, linked to them, and Newcastle being linked to them. I think this is someone that all Everton fans had real hope for a couple of years ago, but his career just seems to have drifted away and he just seems to be making the same mistakes, hasn't really progressed, and if Everton could get maybe 12 to 15 million for him, maybe with that eye of bringing in a real classy centre back in the summer, then I think Everton would probably look to do that business, so we'll have to keep an eye on that one as well. Everton have obviously got at the moment five centre backs, but with Yeti Mina continuing to struggle with absence, but Anthony obviously very young and inexperienced and really no one in those centre back places really making their own this season. It's going to be interesting to see what Rafa Benitez, those from what the club do in terms of the centre back position. Chelsea apparently are stepping up their interest in Lluca Dean. Player seems to be available for a fee of around 25 million. Talk today that personal terms have been agreed between the player and the club, but you've got to put your money on the table, haven't you? You've really got to put your money on the table if you want to. I'm sure he's available. If the money's right to be available, we've just brought in a left back, just got a new right back. These need to be paid for, so be interesting to see what happens with Lluca Dean going forward, but it feels to me he'll be gone in the next few weeks. As Everton is trying to settle the whole situation down, settle the squad down and get some kind of continuity on the pitch with players coming back as well. Dominic Arvenloon obviously came back at the weekend with Charleston hopefully in the next few weeks, Mina hopefully in the next few weeks as well with some really big games, Hull, Leicester and Norwich ahead, which could really have a big burden on our season. So there you go. This has been the Everton News Daily assigning through the door. It's always lovely to say those words, especially someone that people really want to see Everton sign. So let us know your thoughts in the comments. Make sure to give this video a thumbs up. Subscribe if you haven't already. If you want more great daily live videos, join us over on Patreon. The link is in the description and on the screen now. See you later.


Okay, hi everyone. I'm Christy Hegramiller. I'm an oceanographer and research scientist at SOFR ocean, which is a relatively small startup that occupies the blue tech or climate tech space. And I'll be talking about some of the tools and workflows that I've learned since being with SOFR, and how we can translate those and adapt them for our academic workflows. This talk was really motivated by the fact that I'll be transitioning back to academia at the end of this year as well. And so I'm looking to basically make my work more efficient and more collaborative. And I will admit that I had pulled my colleagues at SOFR for some ideas for this talk as well. So their names are down here. And they include other oceanographers and some of the very patient software engineers who work very closely with our ocean science team. Okay, so SOFR ocean is on a mission to address data sparsity across the ocean surface and provide insights to communities and industries operating on the world's oceans. In particular, we're really interested in characterizing and observing the sea state, so wave conditions across the global oceans. And we do that through a full stack approach, and that means that we're interested in not only building the instruments and tools that will help us observe wave conditions across the global oceans, but then also building you know software and tooling on top of that that can make use of those observations more effectively. And then finally, building more tools and more software that can directly interact with customers. In particular, the bottom of our stack is this small buoy called the spotter wave buoy which some of you I think are familiar with. It's a small basketball sized solar powered wave buoy that measures the directional wave characteristics so it's basically observing how energy is distributed across frequencies and directions. It's free drifting and we have about 500 of these in the global oceans right now, drifting around with different ocean currents, and sending data back to us over a really iridium connection hourly. In addition to measuring wave characteristics we're also observing sea surface temperature, barometric pressure and wind speed and direction through proxy and also rain, whether it's raining or not through proxy as well. So really trying to characterize the air sea interface. On top of these observations we maintain and develop a global operational wave forecast. So this is really predicting wave conditions over medium timescale so the next 10 days or so. And that's the colors that you're seeing on the right here. That's our operational wave forecasting system and we'll be talking extensively about next. And lastly, you can see one of these kind of customer facing tools that we've built on top of this way forecasting infrastructure which is a ship routing service called Wayfinder. That helps container vessels will carriers optimize their routes across the global ocean for fuel efficiency. And for time. So we're really making use of the observations that we collected the ocean surface better forecast that we're able to provide because of those observations, and then knowledge about the shipping industry or whatever other commercial partners we might be working with. I am an oceanographer as I said my expertise is in global ocean and coastal wave dynamics and so I really sit in the operational wave forecasting part of this company and don't have as much insights into the other spots. So I'll be talking a lot about that way of forecast that we've built and that we're maintaining and actively developing and the tools that we use to be able to engage with that that forecast analyze it and develop it as rapidly as we can. Okay, this is our overarching way of forecast infrastructure. So there are two components one is an analysis track and that is basically combining our forecast with the observations that we're making hourly and producing the best initial conditions for a longer term forecast that we have so the two tracks that we have here are this analysis track, and then the long or 10 day forecast medium range forecast I understand we're thinking about a range of time scales at this conference. This operational way of forecasting system is built on wavewatch three which is a global or a spectral wave model maintained and developed by Noah. We have a data assimilation methodology that we've been working on. This system is forced by wind forcing and surface current fields and see ice concentrations from other operational centers, namely the European Center for medium range weather forecasting and the Gulf's 3.1 high com currents for the ocean surface. And then we make use as I said of our own observations across this network. That's split up here and satellite observations as well. And I just want to point out what each one of these little modules in this forecasting infrastructure actually is so that when we talk about the tools that we use at so far you can maybe link it back to what your own repositories are doing some of this work. So, of course wavewatch three is an open source community developed for trans based model so this might look like lots of other numerical models that you use in your own work. Our data assimilation repository is Python based so this is code that we've written that we maintain and we develop and manage through GitHub. The repositories that pull in forcing from external data sources, whether those are other operational forecasting centers or observation platforms like satellite altimeter observations or something like that those are all repositories of bash NCO and CDO tools. And then lastly I really want to point out that we've built in some automated assessment into our forecasting infrastructure and that helps us really focus in on the science questions that we're interested in. And these repositories for data analysis of both, you know this automated workflow but then also any other exploratory work that we do is all Python based as well. And lastly we do a tremendous amount of development and experimentation with different parts of this system whether or not that's the entire infrastructure, and we want to run a side by side experiment that looks exactly like our production forecast. Or if it's you know I want to run a reanalysis for one month using just wavewatch three and maybe some of these external fields. So the modularity of this system and having kind of known inputs and outputs to each one of these components allows us to break it up into individual pieces, and together into our operational system as effectively as possible. And that enables us to have this like very rapid iteration between research and operations, which you may be familiar with from like other types of operational forecasting centers there's always this transition period where you do a tremendous amount of research, and then it's not smooth to get that back into your production infrastructure so we've tried to eliminate some of those challenges. And then just to talk at a high level about some of our infrastructure. We are using cloud based infrastructure from end to end. I won't talk about all of this in detail. And, but I will mention that we use an orchestration layer that allows us to kind of take the human out of the loop for dependencies and individual tasks of this forecast that need to happen concurrently or you know sequentially. We use containerized builds which I will be talking about a lot. Next, to help us modularize our code and know exactly what we're working with. We run everything on cloud compute through AWS batch, which is a parallelization tool that AWS has. And then everything is stored in final form on AWS S3 and served to customers or internally to us within the company as well via API's. Okay. So, right, this is all wonderful but why, why are why am I here. I think that a lot of the challenges that we face in this type of environment look very similar to what we face in academic science environments as well and so we have these grand challenges that we need to face especially as we incorporate more modeling and larger data sets into our work. Those challenges, I think are you know reproducibility of experiments and results barriers to entry for model development and use and barriers to entry for big data analysis and then maintaining and growing community engagement development and sustainability and so that sustainability part is really making sure that, you know, not only one person has all of the knowledge about a particular modeling system or a particular tool that we're using within the community and allowing as many people as possible to contribute work with and improve different systems that we are using. So the tools that I'm going to talk about are not unfamiliar probably to most of the community here, but the three that have been most effective for me to actually make the transition to using these tools seriously are get for code management containerization of all our code through Docker, and really investing in the upfront work to have automatic data approximate analysis tools. This is not perfect I am not a software engineer right, but while I working with software engineers I was able to build and you know, rapidly scale up to using some of these which allows now any other person who joins the company to more quickly spin up with the tools that we're using in the infrastructure that we have. Okay, I'm going to fly through get because I'm pretty sure probably a lot of people here use get or get hub to manage their code but just at a high level. Get is version control for your code. It's a framework for collaboration amongst multiple developers and get hub is a platform that allows us to host get repositories and really fosters collaboration. The practices that I found to be really helpful with implementing get in my own workflows are creating repositories for really modular discrete parts of code that can be reused in different places. Commit early and commit often so you leave a paper trail behind you of what you've done. Code reviews, that's what any level of formulas so you know whether that's casually bringing someone in and saying hey I want you to take a look at this one thing or whether that's like properly not being able to merge anything in until you have someone approve it. And that's, you know, a good opportunity to catch bugs and get feedback on large scale code structure. And then what can get enable for me and enables really clean clear tracking of code changes, and the ability which is really important to revert back to any version of that code that I've had in the past. And that might be because I found a bug or I might want to reproduce my results from a previous paper that, you know, used one version of this code that I had. Right. And then I think really importantly it also gives us the opportunity to mentor on code design and development and so that's really beneficial for students who might be joining your group. Okay, the tool that I'm going to spend the most time talking about is Docker. I love Docker. I have found it to be extremely effective in my workflow. So Docker it's a platform that delivers software in packages which are called containers, and it basically is like a recipe for an automated setup of a compute or analysis environment. We use this integrated with GitHub so that it links the exact commits that we've made to build a new image of that code. So what will it do for us so far and what will it do in my academic work it provides a tremendous amount of portability for when hardware goes through major changes. It also provides an exact record of what model setup and compilation was used to create a particular set of results or simulations, and most importantly and I think that the biggest thing that this has the potential to do for us as a broader modeling community is that it alleviates the barriers to entry for code and model quick start so any model that you might jump into has some quick start guide right which build has like some steps for compiling that code. But if you go to any model forum there are a million questions about code violation issues and lots of people spend months trying to figure out how to properly compile their code. So Docker kind of removes some of that work and if you were to provide a Docker file with your model development you might be able to just set someone on their way because you've provided this recipe for compiling it in a particular space. So I had heard all of this from rich signal, I'm in my previous job at us yes, and I still wasn't able to like actually visualize that what that would look like in my work so I'm going to provide a tangible example of how we do this at so far so that hopefully it. You can see how it might look for your own work. This might be a little serious bear with me. Okay, so a Docker image or Docker container is built from a Docker file which is just this recipe as I mentioned. You probably can't see all this so I'm just going to tell you what some of these parts are right at the top of this Docker file actually the first thing I want you to notice that this is actually not that big and this is the entire implementation for wavewatch three. So, if I run this and build a Docker image off of off of these lines of code, I now have a wavewatch the implementation that I can hand to someone else and they can run it wherever they are as long as they have Docker running. The first part of this is I define my like kind of base package for the, the hardware infrastructure and that I'll be running on.   I point to a very specific tag commit tag at in the wavewatch three repository. That will allow me to reproduce this no matter how many times I build it right even if they've made changes in that repository. And all of this stuff looks like your normal compilation so it's just pointing to particular libraries and compiling the code down here. And as I said, this now builds a container that has all of the necessary information for running wavewatch three within it, and just has an executable that you can call. The origin of this is a Docker file to build a Python based repository. This is even shorter right and it basically is just a pip install of your requirements that you might need, and copying all of your Python code into that container. And so again I can pass this now to someone else. And without any work on their end they can just run the Python code or whatever it is that we're working with a little bit more about this what does this look like in practice here's kind of you know, our Docker hub repository for our wavewatch three implementation. Each image name that is here is tagged for a specific build that we've done and it has a you know it has a semver tag associated with it so it's telling me something about how big the change was that I implemented some of them have development tags and so I can point directly back to one of these and use it to rerun a model, knowing exactly what I've done or you know link it back to some results that I've generated or a paper. And then just again one line, anyone can launch one of these containers and run an executable on their own machine. So this is the one line now that with the image that I generated. And I could pass to someone else and they could run wavewatch three. So this really has been effective for us in terms of spinning new people up in terms of our forecasting infrastructure so I can pass this to a colleague who might be working on a very different level of so far as mission. And they can without you know having to spend days figuring out all the libraries that they might need or you know what does it mean to compile wavewatch three, they can actually just run it and get the output that they need. This has been extremely effective. It's not, you know, this is this is I think a shared vision across a lot of academia. So this was a very recent, you know, three weeks ago dear colleague letter from physical and dynamical meteorology and NSF, and they explicitly call out a new effort to supply a containerized version of warf which is an atmospheric model which will allow you know us to lower the barrier to entry.  I'm going to, I'm going to, I'm good. I'm just going to talk quickly about some one other thing that we've done so we've really worked hard to have data approximate analysis, which gave a really nice clinic on this, and we're not implementing all of the things that he talked about in that clinic, but what it has done for us is allowed us to monitor or have some intermediate analysis of model simulations. And so I'll just talk briefly and I'm happy to take questions about this but we've built dashboards basically that produce results for us on the fly, and have slack integrations that give us a bunch of plots that we can monitor every day for how our forecast is performing. In conclusion, we've talked a lot yesterday about you know academia agencies and community collaborations and I think industry can be a really effective partner in that collaboration as well. I'd love to talk more about that, you know with you all over this conference, software engineers teach and enforce really healthy development and maintenance habits for code and that's been super helpful for me and some of the tools and practices that we use in the industry are portable back to academia. And then lastly, you know if I can do it I think you can too and so I would love to talk with you more about what this looks like in practice and. Yeah, thank you so much. Thank you Chris, that was a really inspiring talk for like best practices in software development for like many of us here. There is a question in the back there. Wonderful talk. Thank you. I was wondering since your, your last slide was about you know partnering and partnering and having the commercial world with academia, etc. So, you're using way for three and at the same time you have those many buoys in the ocean.  Any feedback from your company to know for example who runs that operational way for three version. So you can indicate. Well there is really a bias in this part of the ocean that you can, you know, yeah, come out with a new version that is a great question so we, we do collaborate really closely with Noah, and we have, I think like several proposals that work to either develop infrastructure for them that can be easily pulled into their own operational systems, or like you mentioned with our historical data from our observation network to validate or calibrate their models and so you know we work with, I think, like some of the European space agencies as well to calibrate satellite data. So there is a lot of feedback between agency centers so far and the academic community as well the last thing I did want to point out is that all of our historical data is freely available within academic license. So if you're interested please reach out and I'd be happy to share that. Thank you. I'll allow one more question. Yeah, sort of a science question. What data from these buoys is most important to improve the forecast compared satellite data and other data. Yeah, that's a great question I didn't get a chance to talk much about the interesting science that we're doing. But I think the most valuable thing that we've done with the data is really be able to assimilate the directional wave observations into the model. So these models are spectral wave models so they're solving again for the wave energy distributed in frequency and directional space. And instead of assimilating just a significant wave height which is only available from satellite altimeters we're able to actually assimilate the directional moments that are observed by the buoys and that variance density or how energy is distributed in frequency and we see that that gives us the ability to really allow waves to propagate further away with better skill than just pulling the entire wave height fields to some value. I'm happy to talk more about that is the best part.


This is something that we've been talking about for a long time on here There's possibility But it's important that we remain realistic about what it could mean and actually what it could not mean But anyway, Greg Roman Ravens current offensive coordinator There's been a report that's been going around that has been getting so many Ravens fans hyped and happy and excited and jubilated and just They they ready for it to happen. I've seen so many people talk about oh, we're gonna throw a party like project X Everybody's invited. I don't see so many people talk about oh, it's gonna be a cookout. Everybody come through I don't have people ask oh, how you gonna celebrate when Greg Roman gets this job And I'm like, hey, that's cool. I understand the excitement, but at the same time You got a temper expectations Not even tempering expectations like he won't get the job, but temper expectations for what could possibly come next now Reading the report is from SL Mandale Stuart Mandale. He said source Stanford has talked with Baltimore Ravens offensive coordinator Greg Roman about its coaching vacancy Roman was on Jim Harbaugh Stanford staff in 2009 and 2010 I didn't know that I Didn't you know that that's funny. I had no clue about that. I didn't know about that But that makes these hardballs would it when Harbaugh if a hardball loves you. Oh boy, you you in you always got a job I never knew that because again with Jim Harbaugh, he was on Stanford staff in 2009 and 2010 But then Jim Harbaugh, he went to the NFL and he went to the 49ers and you know his offensive coordinator like 40s And then of course Years later, but I think before what was it 2007 with Greg Roman was I think he's like an offensive line coach or something for the Ravens So it was a while back where Greg Roman was actually on the Ravens before But then yeah, I guess he was on Stanford and he went to 49ers and then uh, yeah He went to bills and whatever else he went and then he made his way to the Ravens So if these Ravens, I mean, excuse me at the hardballs if the hardballs love you You will have a job forever You will always be taken care of so hey, you can say a lot about them hardballs But you can never say they don't put their people on but anyway Not knowing how many other candidates there are at this time for the Stanford vacancy And I know again once a lot of Ravens fans saw that they jumping for joy like, oh, that's gonna be it Greg Roman could finally possibly be on up out of here So many people excited and I get the excitement, but at the same time I I just I've been saying on here Greg Roman could go He could go tomorrow Ravens could be like, all right Greg Roman you're relieved of your duties Or if they waited till the end of the year or whatever his contract expiring anyway if he leaves, okay, cool great, but These Ravens will still have the same philosophy So whatever offensive coordinator they bring in They're gonna tell that offensive as long as they have the same leadership the same owner the same GM the same head coach They're gonna tell that offensive coordinator. Hey, look, this is what our plan is. This is what we want to do on offense This is how we want to operate on offense. This is how we want our offense to look This is what we want you to run. This is how we want you to run it now. We ain't gonna get in your way But these are our priorities on offense You either get with it or you can get lost Because we've seen this story time and time again, I could again I ain't trying to burst nobody bubble But we got to be honest about it like we remember back people complained about complain about Cam Cameron for the longest now when he got five courses ended up being Jim Carwell than a Super Bowl But then after that After that Remember the Mark Trestman's and I know a lot of people it's funny because and I forget it myself So I don't blame anybody who forgets a lot of people be like, oh Remember the last time that we fired an offensive coordinator in the middle of the season Remember what happened Super Bowl, baby? And I had to tell somebody none of the way hold up there, buddy That's wrong because the last time Ravens fired an offensive coordinator in the middle of the season There was not a Super Bowl that followed. I believe that was in 2016 when they did that to Mark Trestman They let him go in the middle of the year. So anyway But yeah, there were the complaints about Mark Trestman. There were the complaints about Marty Morniwake They were the complaints about Greg Roman. They were the complaints about who am I missing? I know. Oh now Gary Kubiak They weren't any complaints because I think Gary Kubiak was like a breath of fresh air. I think Ravens feels like oh my goodness It's a breath of fresh air. I can't believe this and He just it seemed like he brought life to this Baltimore Ravens here and and Ravens were like They were in a dog dog place because that was around the time with that whole whole right stuff and all that stuff So yeah, it was a lot of a lot of stuff going on back then So with Gary Kubiak, just the fact that the Ravens were operating how they were it was like whoo But um, but anyway, um, I just Yeah, a new offensive coordinator great but change it offensive coordinator great But I just my expectations would be like these are still the Ravens They still have the philosophy that's in place. They still have the way that they want to run this football team in place So what should I really expect to change? What can I really what what is a realistic expectation of what's next on offense for these Baltimore Ravens? Ravens do not value offense like other teams value offense They don't value the valuable positions on offense like other teams value the valuable positions on offense And I feel like that that's why their offense is limited. That's why their offense has so many hiccups Now could the offense be better? Yeah, it certainly could with a new offensive coordinator. It could be better Could be worse too, but it could be better and I just don't like what Greg Roman, we know Yeah, he's a special specializes in the the run game. He got this strong run game Da-da-da. I know that's been one of a lot of people's reason Oh, no, we don't want to let Greg Roman go because what about the run game? Look man, it's a copycat league as we all know you can take concepts from what Greg Roman does Well, you can take you can take that stuff. You can borrow that stuff I mean, look at the bears. Look at the bears. You can build bar that stuff without And you could leave out the bad stuff. You know, I take that I take that too. I take that ooh that part Yeah, I don't want no pieces of that So you can do that like and the Ravens have always been a team that can't can run the ball That was both before Greg Roman and it'll be after Greg Roman too. That's not a shot at Greg Roman But that's just who the Ravens been they've been running team always. I mean they did have had some years Here and there with ugly especially like last year, but Overall, they've been pretty successful at running the ball So I wouldn't expect that to change but I know as far as passing the ball That's where there's been the biggest hiccups and mistakes and just been bad But think about this all fans of coordinator changes. Okay, cool You still got the same GM you still got the same head coach You already know how they are and you know who they are What would change from them if they got a new offensive coordinator Would they all of a sudden be like, all right, we by the place real emphasis on the wire receivers We buy the place real emphasis on this passing game. Do you think you would really change? I don't I really don't Because deep that's not even that that's just who the Ravens have been Ravens they never been that team that put that emphasis on a passing game like that and again You gotta say gotta say this out loud cuz I know a lot of people that they try to take stuff and run with it Oh, they try to think in these extremes Well, it's like I if you say that the Ravens need to up the passing game They need to put more emphasis on it and more emphasis on receivers and stuff that he'd be like Oh, what about the run game? No, it's not saying that you get rid of the run game No, you still want to be able to run the ball But you just want to be able to do both and do both well You want to be able to do both and do a good job at both It's like on defense you don't want to have this just this amazing run defense But your passing defense is garbage. No, you want to be able to do both and do both well Same way with offense Same with all with offense because again with with with with the comparisons and stuff a lot of Ravens fan If you mentioned something on defense, though If you talk about something on defense the way that you talk about it on offense a lot of Ravens fans for defense They'll understand it. Oh, yeah. Yeah for defense. Okay, we get it Yeah, we want to have the best of the best players at this position that position that position We want to have all these playmakers here there everywhere. Yeah. Oh, we want to be able to stop the run We want to be able to stop the pass we want to be able to do both of it We want to be able to rush to pass her as well We want to be able to tackle well We want to be able to do all these things well on defense and we want to have the best of the best players at all these positions But then when you have that conversation with them on offense It's crickets Silence It's all no, no, we don't need it. Oh How can we afford to pay that? Oh Lamar should take a home down discount because we ain't gonna be able to But you see what I'm saying So, yeah, hey Greg Roman, I hope it goes well Because the weather he gets that head coaching job, but he get a head coaching job in NFL So I don't expect him to be with the Ravens next year And I don't expect it to be like any nasty breakup or anything like that. No, it's contract expiring Like hey, we wish you well all that on your future and that was thanks for everything you did I expect him to get one of those But he's gonna get another job and that's great for him. It's great for him But it's it's probably been time with the Ravens again like always say Greg Roman Excellent introductory offensive coordinator for a young quarterback excellent introductory offensive coordinator But it just it peaks Lamar been talking about all season. Hey, we don't want to peak too soon His offense has peaked years ago It's peaked years ago and It's it just hasn't been going anywhere. We haven't seen these these significant Advancements with the Ravens offense It's just we haven't seen it. We see no small stuff here and there, but it's like really Yeah, it's not enough so I hope Greg Roman gets a job. Whatever. I mean he gonna get one regardless. He gonna get a job somewhere But again my expectations for the Ravens and their offense and their philosophy and their values and stuff That doesn't really change for me