<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Step 1
Step1:  Red dots are interpolated values.
Step2:  We want to obtain a stationary "feature" from the data, firt differences are an easy place to start.
Step3:  It appears that early temperature sensors had rather imprecise readings.
It also appears as though the interpolation introduces some systematic errors.  I used pchip interpolation, which tries to avoid overshoot, so we may be seeing the effects of clipping.  This would particularly make sense if missing data was from regular periods, e.g. at night when the temperature was reaching a minimum.
Step4:  It looks like there is still some nonstationarity in the first differences.
<ASSISTANT_TASK:>
Python Code:
print('Original bounds: ', t[0], t[-1])
t_obs = t[D['T_flag'] != -1]
D = D[t_obs[0]:t_obs[-1]]  # Truncate dataframe so it is sandwiched between observed values
t = D.index
T = D['T']
print('New bounds: ', t[0], t[-1])
t_obs = D.index[D['T_flag'] != -1]
t_interp = D.index[D['T_flag'] == -1]
T_obs = D.loc[t_obs, 'T']
T_interp = D.loc[t_interp, 'T']
c = ['b' if flag != -1 else 'orange' for flag in D['T_flag']
plt.scatter(t, T, c = c, alpha = 0.5, s = 0.5)
plt.title('T')
#obs = plt.scatter(t_obs, T_obs, marker = '.', alpha = 0.5, s = 0.5, color = 'blue');
#interp = plt.scatter(t_interp, T_interp, marker = '.', alpha = 0.5, s = 0.5, color = 'red');
     #  If I plot one after the other, the red is much more prominant...  Very annoying
#plt.legend((obs, interp), ('Observed', 'Interpolated'), markerscale = 15);
Explanation: Step 1: Truncate the series to the interval that has observations.  Outside this interval the interpolation blows up.
End of explanation
# Centre the data
mu = D['T'].mean()
D.loc[:, 'T'] = D.loc[:, 'T'] - mu
T = D['T']
print('E[T] = ', mu)
Explanation: Red dots are interpolated values.
End of explanation
T0 = T[0]
dT = T.diff()
dT = dT - dT.mean()  # Center the differences
dT_obs = dT[t_obs]
dT_interp = dT[t_interp]
plt.scatter(t, dT, marker = '.', alpha = 0.5, s = 0.5, c = c)
#obs = plt.scatter(t_obs, dT_obs, marker = '.', alpha = 0.5, s = 0.5, color = 'blue');
#interp = plt.scatter(t_interp, dT_interp, marker = '.', alpha = 0.5, s = 0.5, color = 'red');
#plt.legend((obs, interp), ('Observed', 'Interpolated'), markerscale = 15);
plt.title('dT')
Explanation: We want to obtain a stationary "feature" from the data, firt differences are an easy place to start.
End of explanation
rolling1w_dT = dT.rolling(window = 7*24)  # 1 week rolling window of dT
rolling1m_dT = dT.rolling(window = 30*24)  # 1 month rolling window of dT
rolling1y_dT = dT.rolling(window = 365*24)  # 1 year rolling dindow of dT
fig, axes = plt.subplots(3, 1)
axes[0].plot(rolling1w_dT.var())
axes[1].plot(rolling1m_dT.var())
axes[2].plot(rolling1y_dT.var())
Explanation: It appears that early temperature sensors had rather imprecise readings.
It also appears as though the interpolation introduces some systematic errors.  I used pchip interpolation, which tries to avoid overshoot, so we may be seeing the effects of clipping.  This would particularly make sense if missing data was from regular periods, e.g. at night when the temperature was reaching a minimum.
End of explanation
from itertools import product
t_days = [t[np.logical_and(t.month == m, t.day == d)] for m, d in product(range(1,13), range(1, 32))]
day_vars = pd.Series(dT[ti].var() for ti in t_days)
day_vars = day_vars.dropna()
plt.scatter(day_vars.index, day_vars)
r = day_vars.rolling(window = 20, center = True)
plt.plot(day_vars.index, r.mean(), color = 'red', linewidth = 2)
plt.title('Variance of dT, folded by days')
Explanation: It looks like there is still some nonstationarity in the first differences.
End of explanation
<END_TASK>