<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Exploring alert data for cities within the system
In this notebook, we will illustrate how to access the data from the system and run some basic models.
Step1:  Loading The Data
for our exploration let's pick the city of Rio de Janeiro.
Step2:  Let's look at the tables
Step3:  Let's try to join the tables by date. To align them, we must downsample each one to a weekly time frame
Step4:  Note que as datas para as datas mais antigas os dados faltantes de Temperatura e Tweets, foram substituídos por NaN. Podemos remover estas datas, ficando com uma tabela sem dados faltantes. Mas perde-se mais de dois anos de dados.
Step5:  Calculando uma previsão
Step6:  Splitting the dataset for out-of-sample prediction validation
Step7:  Looking at state-wide data
Step8:  Let's keep only the columns we want to use
Step9:  Converting dataframe from long format to wide format
The dataframe currently have all cities stacked on top of each other. In order to use this data in a predictive model, we need this table in wide format, that is, have only time along the rows and have cities variable listed as columns.
Step10:  Now we have a multi-level column index. It may be preferable to flatten it.
<ASSISTANT_TASK:>
Python Code:
import pandas as pd
import getpass, os
os.environ['PSQL_USER']='dengueadmin'
os.environ['PSQL_HOST']='localhost'
os.environ['PSQL_DB']='dengue'
os.environ['PSQL_PASSWORD']=getpass.getpass("Enter the database password: ")
os.chdir('..')
from infodenguepredict.data.infodengue import get_temperature_data, get_alerta_table, get_tweet_data
%pylab inline
Explanation: Exploring alert data for cities within the system
In this notebook, we will illustrate how to access the data from the system and run some basic models.
End of explanation
A = get_alerta_table(3304557)#(3303500)
T = get_temperature_data(3304557)#(3303500)
Tw = get_tweet_data(3304557)#(3303500)
Explanation: Loading The Data
for our exploration let's pick the city of Rio de Janeiro.
End of explanation
A.head()
T = T[~T.index.duplicated()]
T.to_csv('temperature_rio.csv', header=True, sep=',')
T.head()
Tw = Tw[~Tw.index.duplicated()]
Tw.head()
Explanation: Let's look at the tables
End of explanation
T.resample('W-SUN').mean().tail()
Full = A.join(T.resample('W-SUN').mean()).join(Tw.resample('W-SUN').sum())
Full.head()
Explanation: Let's try to join the tables by date. To align them, we must downsample each one to a weekly time frame
End of explanation
Short = Full.dropna()
Short.head()
Short[['casos_est', 'temp_min', 'umid_min', 'numero']].plot(subplots=True, figsize=(15,10),grid=True);
Explanation: Note que as datas para as datas mais antigas os dados faltantes de Temperatura e Tweets, foram substituídos por NaN. Podemos remover estas datas, ficando com uma tabela sem dados faltantes. Mas perde-se mais de dois anos de dados.
End of explanation
from infodenguepredict.models import sarimax,GAS,GASX
import statsmodels.api as sm
fig, axes = plt.subplots(1, 2, figsize=(15, 4))
fig = sm.graphics.tsa.plot_acf(Full.ix[1:, 'casos'], lags=52, ax=axes[0])
fig = sm.graphics.tsa.plot_pacf(Full.ix[1:, 'casos'], lags=52, ax=axes[1])
# Short.casos = Short.casos.apply(pd.np.log) 
model_1 = sarimax.build_model(Full, 'casos', [])
fit_1 = model_1.fit()
fit_1.summary()
def plot_pred(fit):
    plt.Figure(figsize=(10,7))
    predict = fit.get_prediction(start='2017-01-01', dynamic=False)
    predict_ci = predict.conf_int()
    Full.casos.plot(style='o',label='obs')
    predict.predicted_mean.plot(style='r--', label='In sample')
    plt.fill_between(predict_ci.index, predict_ci.ix[:, 0], predict_ci.ix[:, 1], color='r', alpha=0.1)
    forecast = fit.get_prediction(start='2017-03-05', end='2017-06-21', dynamic=False)
    forecast_ci = forecast.conf_int()
    forecast.predicted_mean.plot(style='b--', label='Out of Sample')
    plt.fill_between(forecast_ci.index, forecast_ci.ix[:, 0], forecast_ci.ix[:, 1], color='b', alpha=0.1)
    plt.legend(loc=0)
plot_pred(fit_1)
model_2 = GAS.build_model(Full, ar=2, sc=6, target='casos')
fit_2 = model_2.fit()
fit_2.summary()
model_2.plot_fit()
plt.savefig('GAS_in_sample.png')
Full.casos.plot(style='ko')
model_2.plot_predict(h=10, past_values=52)
model_2.plot_z(figsize=(15,5))
Explanation: Calculando uma previsão
End of explanation
plt.figure()
ax = plt.gca()
train = Full.loc[Full.index<'2015-01-01']
model_3 = GAS.build_model(train, ar=2, sc=6, target='casos')
fit_3 = model_3.fit()
Full.casos.plot(style='ko', ax=ax, figsize=(15,10))
plt.hold(True)
model_3.plot_predict(h=10, past_values=20, ax=ax, intervals=True, figsize=(15,10))
model_4 = GASX.build_model(Full.dropna(), ar=4, sc=6, formula='casos~1+temp_min')
fit_4 = model_4.fit()
fit_4.summary()
model_4.plot_fit()
model_4.plot_predict(h=10, past_values=15)
Explanation: Splitting the dataset for out-of-sample prediction validation
End of explanation
rio  = get_alerta_table(state='RJ')
rio.head()
Explanation: Looking at state-wide data
End of explanation
for col in ['casos_est_min', 'casos_est_max', 'Localidade_id', 'versao_modelo', 'municipio_nome']:
    del rio[col]
rio.head()
Explanation: Let's keep only the columns we want to use
End of explanation
riopiv = rio.pivot(index=rio.index, columns='municipio_geocodigo')
riopiv.head()
riopiv['SE'].head()
Explanation: Converting dataframe from long format to wide format
The dataframe currently have all cities stacked on top of each other. In order to use this data in a predictive model, we need this table in wide format, that is, have only time along the rows and have cities variable listed as columns.
End of explanation
riopiv.columns = ['{}_{}'.format(*col).strip() for col in riopiv.columns.values]
riopiv.head()
riopiv.shape
Explanation: Now we have a multi-level column index. It may be preferable to flatten it.
End of explanation
<END_TASK>