<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Plotly Setup
Step1:  Load Cleaned Data from S3
Step2:  Number of Torrent Titles by Release Year
Step3:  Trim Dataset by Years of Interest/Relevance
Due to the low number of titles for the years below 1995, these torrents were removed from the dataset. Also, since the current year (2016) is only partially completed, films released in 2016 were removed from the dataset as well.
Step4:  Quantity Genre Classifications
Step5:  Most Dominant Genre out of Genres Given per Title
Step6:  Dominant Genre Quantities per Year
Step7:  Remove Films Not Rated - PG-13, PG, G, or R
Step8:  Log Transform Scatter Matrix
Step9:  Drama Only
Step10:  Log Transform
<ASSISTANT_TASK:>
Python Code:
# plotly validate with credentials
with open('../_credentials/plotly.txt', 'r') as infile:
    user, pw = infile.read().strip().split(', ')
    
plotly.tools.set_credentials_file(username=user, api_key=pw)
text_color = 'rgb(107, 107, 107)'
colors_dict = {'grey':'rgb(189, 195, 199)', 'aqua':'rgb( 54, 215, 183)', 'navy':'rgb( 31,  58, 147)',
            'purple':'rgb(142,  68, 173)', 'blue':'rgb( 25, 181, 254)', 'green':'rgb( 46, 204, 113)',
            'yellow':'rgb(253, 231,  76)', 'orange':'rgb(250, 121,  33)', 'red':'rgb(242,  38,  19)'}
colors_lst = [colors_dict['yellow'], colors_dict['orange'], colors_dict['red'], 
              colors_dict['green'], colors_dict['blue'], colors_dict['purple'], 
              colors_dict['navy'], colors_dict['aqua'], colors_dict['grey']]
Explanation: Plotly Setup
End of explanation
# aws keys stored in ini file in same path
os.environ['AWS_CONFIG_FILE'] = 'aws_config.ini'
s3 = S3FileSystem(anon=False)
key = 'data.csv'
bucket = 'luther-02'
df = pd.read_csv(s3.open('{}/{}'.format(bucket, key),mode='rb'))
# update dates to datetime objects
df['Released'] = pd.to_datetime(df['Released'])
df['Year'] = pd.DatetimeIndex(df['Released']).year
df['Year_Int'] = pd.to_numeric(df['Year'])
df['Month'] = pd.DatetimeIndex(df['Released']).month
# year extremities
yr_start = df['Year'].min(axis=0)
yr_stop = df['Year'].max(axis=0)
Explanation: Load Cleaned Data from S3
End of explanation
# number of titles per year in dataset
df_yr = df['Year'].value_counts().reset_index()
df_yr.columns = ['Year','Count']
# create plotly data trace
trace = go.Bar(x=df_yr['Year'], y=df_yr['Count'], marker=dict(color=colors_dict['red']))
def bar_plot_data(_dataframe, _label, color):
    df_temp = _dataframe[_label].value_counts().reset_index()
    df_temp.columns = [_label,'Count']
    # create plotly data trace
    trace = go.Bar(x=df_temp[_label], y=df_temp['Count'], marker=dict(color=colors_dict[color]))
    
    data = [trace]
    layout = go.Layout(
        title='Quantity of Torrent Titles by Year Released ({0}-{1})'.format(yr_start, yr_stop),
        xaxis=dict(
            title='Release Year',
            tickfont=dict(size=14, color=text_color)),
        yaxis=dict(
            title='Number of Titles',
            titlefont=dict(size=16, color=text_color),
            tickfont=dict(size=14, color=text_color)),
        barmode='group',
        bargap=0.15,
        bargroupgap=0.1)
    fig = go.Figure(data=data, layout=layout)
    return py.iplot(fig, filename='luther_titles_annually({0}-{1})'.format(yr_start, yr_stop))
bar_plot_data(df, 'Year', 'red')
data = [trace]
layout = go.Layout(
    title='Quantity of Torrent Titles by Year Released ({0}-{1})'.format(yr_start, yr_stop),
    xaxis=dict(
        title='Release Year',
        tickfont=dict(size=14, color=text_color)),
    yaxis=dict(
        title='Number of Titles',
        titlefont=dict(size=16, color=text_color),
        tickfont=dict(size=14, color=text_color)),
    barmode='group',
    bargap=0.15,
    bargroupgap=0.1)
fig = go.Figure(data=data, layout=layout)
py.iplot(fig, filename='luther_titles_annually({0}-{1})'.format(yr_start, yr_stop))
Explanation: Number of Torrent Titles by Release Year
End of explanation
def df_year_limit(start, stop, df):
    mask = (df['Year'] >= start) & (df['Year'] <= stop)
    df = df.loc[mask]
    return df
# get count of records before trimming by year cutoff
yr_before = len(df)
print('{0} records in dataframe before trimming by year cutoff'.format(yr_before))
yr_start, yr_stop = (1995, 2015)
# trim by year cutoff
df = df_year_limit(yr_start, yr_stop, df)
yr_after = len(df)
print('{0} entries lost ({1}%) due to date cutoff between {2} and {3}'.format(yr_before-yr_after, 
                                round((yr_before - yr_after)/yr_before *100, 2), yr_start, yr_stop))
# number of titles per year in dataset 
df_yr = df['Year'].value_counts().reset_index()
df_yr.columns = ['Year','Count']
trace = go.Bar(x=df_yr['Year'], y=df_yr['Count'], marker=dict(color=colors_dict['blue']))
data = [trace]
layout = go.Layout(
    title='Number of Torrent Titles by Release Year ({0}-{1})'.format(yr_start, yr_stop),
    xaxis=dict(
        title='Release Year',
        tickfont=dict(size=14,color=text_color)),
    yaxis=dict(
        title='Number of Titles',
        titlefont=dict(size=16, color=text_color),
        tickfont=dict(size=14, color=text_color)),
    barmode='group',
    bargap=0.15,
    bargroupgap=0.1)
fig = go.Figure(data=data, layout=layout)
py.iplot(fig, filename='luther_films_annually({0}-{1})'.format(yr_start, yr_stop))
Explanation: Trim Dataset by Years of Interest/Relevance
Due to the low number of titles for the years below 1995, these torrents were removed from the dataset. Also, since the current year (2016) is only partially completed, films released in 2016 were removed from the dataset as well.
End of explanation
# split genre strings into a numpy array
def split_to_array(ser):
    split_array = np.array(ser.strip().replace(',','').split(' '))
    return pd.Series(split_array)
# turn numpy array into count of genre occurances
genres = df['Genre'].apply(split_to_array)
genres = pd.Series(genres.values.ravel()).dropna()
genres = genres.value_counts().sort_values(ascending=False)
# convert series to dataframe for plotting
genre_ser = genres.reset_index()
genre_ser.columns = ['Genre', 'Count']
# bar chart of each genre in dataset
trace = go.Bar(x=genre_ser['Genre'], y=genre_ser['Count'], marker=dict(color=colors_dict['yellow']))
data = [trace]
layout = go.Layout(
    title='Count of Genre Classifications ({0}-{1})'.format(yr_start, yr_stop),
    xaxis=dict(
        title='Genre',
        tickfont=dict(size=14, color=text_color)),
    yaxis=dict(
        title='Number of Classifications',
        titlefont=dict(size=16, color=text_color),
        tickfont=dict(size=14, color=text_color)),
    barmode='group',
    bargap=0.15,
    bargroupgap=0.1)
fig = go.Figure(data=data, layout=layout)
py.iplot(fig, filename='luther_genre_quantity({0}-{1})'.format(yr_start, yr_stop))
Explanation: Quantity Genre Classifications
End of explanation
def convert_frequency(ser, genres=genres):
    split_array = np.array(ser.strip().replace(',','').split(' '))
    genre = genres.loc[split_array].argmax()
    return genre
# add new column to dataframe classifying genre list as single genre of significance
df['Genre_Single'] = df['Genre'].apply(convert_frequency)
# look at number of single genre counts after extraction
df_count = df['Genre_Single'].value_counts().reset_index()
df_count.columns = ['Genre_Single', 'Count']
# bar chart of significant single genre in dataset
trace = go.Bar(x=df_count['Genre_Single'], y=df_count['Count'], marker=dict(color=colors_dict['yellow']))
data = [trace]
layout = go.Layout(
    title='Quantity of Dominant Genre Classifications ({0}-{1})'.format(yr_start, yr_stop),
    xaxis=dict(
        title='Genre',
        tickfont=dict(size=14, color=text_color)),
    yaxis=dict(
        title='Quantity of Classifications',
        titlefont=dict(size=16, color=text_color),
        tickfont=dict(size=14, color=text_color)),
    barmode='group',
    bargap=0.15,
    bargroupgap=0.1)
fig = go.Figure(data=data, layout=layout)
py.iplot(fig, filename='luther_dominant_genres({0}-{1})'.format(yr_start, yr_stop))
def df_genre_limit(df, genres):
    df = df[~df['Genre_Single'].isin(genres)]
    return df
# get count of records before trimming by dominant genre cutoff
genre_before = len(df)
print('{0} records in dataframe before trimming by genres'.format(genre_before))
# trim by dominant genres
cut_genres = ['Romance', 'Western']
df = df_genre_limit(df, cut_genres)
genre_after = len(df)
str_genres = ', '.join(cut_genres)
print('{0} entries lost ({1}%) due to droppping dominant genres {2}'.format(genre_before-genre_after, 
                                round((genre_before - genre_after)/genre_before *100, 2), str_genres))
Explanation: Most Dominant Genre out of Genres Given per Title
End of explanation
def get_stackedBar_trace(x_category, y_counts, _name, ind):
    '''
    x_category -- category from feature set
    y_counts -- count of x_category in feature set
    _name -- _name of x_category
    ind -- number indices for color list
    Return: Plotly data trace for bar chart
    '''
    return go.Bar(x=x_category, y=y_counts, name=_name, marker=dict(color=colors_lst[ind]), opacity=0.8)
def get_stackedBar_traces(df, feature, count_label):
    traces = []
    
    for ind, _feat in enumerate(df[feature].unique().tolist()):
        temp_df = df[df[feature] == _feat]
        _value_counts = temp_df[count_label].value_counts()
        temp_dict = _value_counts.to_dict()
        temp_dict = sorted(temp_dict.items())
        feature_lst = [ft for ft, ct in temp_dict]
        count_lst = [ct for ft,ct in temp_dict]
        traces.append(get_stackedBar_trace(feature_lst, count_lst, _feat, ind))
    
    return traces
def get_stackedBar(_dataframe, feature, count_label, _title, _x_title, _y_title, _filename='stackedBar'):
    
    date = get_stackedBar_traces(_dataframe, feature, count_label)
    
    layout = go.Layout(
        title=_title,
        xaxis=dict(
            title=_x_title,
            tickfont=dict(size=14, color='rgb(107, 107, 107)')
        ),
        yaxis=dict(
            title=_y_title,
            titlefont=dict(size=16, color='rgb(107, 107, 107)'),
            tickfont=dict(size=14, color='rgb(107, 107, 107)'),
            dtick=20,
        ),
        barmode='stack',)
    fig = go.Figure(data=data, layout=layout)
    return py.iplot(fig, filename=_filename)
_title = 'Genres Annually ({0}-{1})'.format(yr_start, yr_stop)
_x_title = 'Year'
_y_title = 'Number of Films'
_filename = 'luther_stackedGenres_years({0}-{1})'.format(yr_start, yr_stop)
get_stackedBar(df, 'Genre_Single', 'Year', _title, _x_title, _y_title, _filename='stackedBar')
traces = []
for i,genre in enumerate(df['Genre_Single'].unique().tolist()):
    _genre_df = df[df['Genre_Single'] == genre]
    _value_counts = _genre_df['Year'].value_counts()
    gen = _value_counts.to_dict()
    gen = sorted(gen.items())
    
    year_lst = [yr for yr,ct in gen]
    count_lst = [ct for yr,ct in gen]
    
    traces.append(make_bar_trace(year_lst, count_lst, genre, i))
data = traces[::-1]
layout = go.Layout(
    title='Genres Annually ({0}-{1})'.format(yr_start, yr_stop),
    xaxis=dict(
        title='Year',
        tickfont=dict(size=14, color='rgb(107, 107, 107)')
    ),
    yaxis=dict(
        title='Number of Films',
        titlefont=dict(size=16, color='rgb(107, 107, 107)'),
        tickfont=dict(size=14, color='rgb(107, 107, 107)'),
        dtick=20,
    ),
    barmode='stack',)
fig = go.Figure(data=data, layout=layout)
py.iplot(fig, filename='stacked-bar')
# ratings
df_rated = df['Rated'].value_counts().reset_index()
df_rated.columns = ['Rated', 'Count']
df_rated
def df_genre_limit(df, ratings):
    df = df[~df['Rated'].isin(ratings)]
    return df
ratings_remove = ['NOT RATED', 'X', 'TV-14', 'NC-17']
df = df_genre_limit(df, ratings_remove)
df_rated = df['Rated'].value_counts().reset_index()
df_rated.columns = ['Rated', 'Count']
# bar chart of ratings
rated_traces = go.Bar(x=df_rated['Rated'], y=df_count['Count'], marker=dict(color=colors_dict['blue']))
data = [rated_traces]
layout = go.Layout(
    title='Quantity of Dominant Genre Classifications ({0}-{1})'.format(yr_start, yr_stop),
    xaxis=dict(
        title='Genre',
        tickfont=dict(size=14, color=text_color)),
    yaxis=dict(
        title='Quantity of Classifications',
        titlefont=dict(size=16, color=text_color),
        tickfont=dict(size=14, color=text_color)),
    barmode='group',
    bargap=0.15,
    bargroupgap=0.1)
fig = go.Figure(data=data, layout=layout)
py.iplot(fig, filename='luther_dominant_genres({0}-{1})'.format(yr_start, yr_stop))
Explanation: Dominant Genre Quantities per Year
End of explanation
# get count of records before trimming by year cutoff
rated_before = len(df)
print('{0} records in dataframe before trimming by rating'.format(rated_before))
ratings = ['PG-13', 'PG', 'G', 'R']
df = df.loc[df['Rated'].isin(ratings)]
rated_after = len(df)
print('{0} entries lost ({1}%) due to limiting to only {2} ratings'.format(rated_before-rated_after, 
                        round((rated_before-rated_after)/rated_before *100, 2), ', '.join(ratings)))
print('{0} entries lost total ({1}%)'.format(yr_before-rated_after,
                                            round((yr_before-rated_after)/yr_before *100, 2)))
# Combine Genre_Single and Rating as a new label
df['Genre_Rated'] = df['Genre_Single'] + ' ' + df['Rated']
df['Gen_Rat_Run'] = df['Genre_Rated'] + ' ' + df['Runtime'].apply(lambda x: str(x))
df['Gen_Rat_Bud'] = df['Genre_Rated'] + ' ' + df['Prod_Budget'].apply(lambda x: str(x))
df['Gen_Sin'] = df['Genre_Single']
df.columns
colors_scat = colors_lst[:-2][::-1]
df_scat = df[['Prod_Budget', 'Runtime', 'Gen_Rat_Bud', 'Gen_Rat_Run', 'Gen_Sin']]
fig = FF.create_scatterplotmatrix(df_scat, diag='histogram', index='Gen_Sin', 
                                    height=1000, width=1000, colormap=colors_scat[::-1])
py.iplot(fig, filename='Luther Scatterplot Matrix')
Explanation: Remove Films Not Rated - PG-13, PG, G, or R
End of explanation
df['Log_Prod_Bud'] = np.log(df['Prod_Budget'])
df['Log_Runtime'] = np.log(df['Runtime'])
df['Log_Ttl_Tor'] = np.log(df['Total_Torrents'])
colors_scat = colors_lst[:-2][::-1]
df_scat = df[['Log_Ttl_Tor', 'Log_Prod_Bud', 'Log_Runtime', 'Gen_Rat_Bud', 'Gen_Rat_Run', 'Gen_Sin']]
fig = FF.create_scatterplotmatrix(df_scat, diag='histogram', index='Gen_Sin', 
                                    height=1000, width=1000, colormap=colors_scat[::-1])
_ = py.iplot(fig, filename='Log Luther Scatterplot Matrix')
Explanation: Log Transform Scatter Matrix
End of explanation
df_drama = df[df['Genre_Single'] == 'Drama'].reset_index()
df_drama = df_drama.drop('index',axis=1)
df_drama['Log_Bud_Rated'] = df['Log_Prod_Bud'].apply(lambda x: str(x)) + ' ' + df['Rated']
#df_scat = df_drama[['Log_Ttl_Tor', 'Log_Prod_Bud', 'Log_Runtime', 'Log_Bud_Rated', 'Gen_Sin']]
df_scat = df_drama[['Total_Torrents', 'Prod_Budget', 'Runtime', 'Rated', 'Gen_Sin']]
fig = FF.create_scatterplotmatrix(df_scat, diag='histogram', index='Gen_Sin', 
                                    height=1000, width=1000, colormap=colors_scat[::-1])
_ = py.iplot(fig, filename='Log Drama Luther Scatterplot Matrix')
from patsy import dmatrices
patsy_formula = 'Total_Torrents ~ Prod_Budget + Year + Month + Runtime + Genre_Single'
y, x = dmatrices(patsy_formula, data=df_sub, return_type='dataframe')
import statsmodels.api as sm
model = sm.OLS(y, x)
results = model.fit()
results.summary()
from sklearn.linear_model import LinearRegression
model = LinearRegression()
model.fit(x, y)
mod_lr_score = model.score(x, y)
mod_lr_coef = model.coef_
model.results
from sklearn import cross_validation as cv
from sklearn import metrics
x_train, x_test, y_train, y_test = cv.train_test_split(x,y,test_size=0.20,random_state=1234)
model = LinearRegression().fit(x_train, y_train)
# store results
mean_sq_err = metrics.mean_squared_error(y_train,model.predict(x_train))
cv_mod_score = model.score(x_train, y_train)
# reset x, y otherwise errors occur
y, x = dmatrices(patsy_formula, data=df_sub, return_type='dataframe')
from sklearn.cross_validation import KFold
kf = KFold(len(df_sub), n_folds=10, shuffle=True)
for train_index, test_index in kf:
    x_train, x_test = x.iloc[train_index], x.iloc[test_index]
    y_train, y_test = y.iloc[train_index], y.iloc[test_index]
    clf2 = LinearRegression().fit(x.iloc[train_index], y.iloc[train_index])
# store results
mean_sq_errKf = metrics.mean_squared_error(y_train,model.predict(x_train))
cvKf_mod_score = clf2.score(x,y)
#NORMAL RESULTS
print('Model Linear Regression Score = {0}'.format(mod_lr_score))
print('            Mean Square Error = {0}'.format(mean_sq_err))
print(' Cross Validation Model Score = {0}'.format(cv_mod_score))
print('     Mean Squred Error K-Fold = {0}'.format(mean_sq_errKf))
print('Cross Val. K-Fold Model Score = {0}'.format(cvKf_mod_score))
fig = plt.figure(figsize=(12,8))
fig = sm.graphics.plot_regress_exog(results,'Prod_Budget', fig=fig)
Explanation: Drama Only
End of explanation
df.columns
df_sub['log_budg']=np.log(df_sub.Prod_Budget)
#df_sub['log_year']=np.log(df_sub.Year)
#df_sub['log_run']=np.log(df_sub.Runtime)
df_sub['log_tor']=np.log(df_sub.Total_Torrents)
trans = df_sub[['log_budg', 'Year', 'log_tor']]
plt.rcParams['figure.figsize'] = (15, 15)
pd.tools.plotting.scatter_matrix(trans)
log_patsy_formula = 'log_tor ~ log_budg + Year + Month'
y, x = dmatrices(log_patsy_formula, data=df_sub, return_type='dataframe')
import plotly.plotly as py
from plotly.tools import FigureFactory as FF
df_a = df_sub[['log_budg', 'Year', 'Month', 'log_tor']]
fig = FF.create_scatterplotmatrix(df_a, diag='histogram', index='Month',
                                  height=800, width=800)
py.iplot(fig, filename='Histograms along Diagonal Subplots')
import statsmodels.formula.api as smf
results = smf.ols(formula=log_patsy_formula, data=df_sub,).fit()
results.summary()
from sklearn.linear_model import LinearRegression
model = LinearRegression()
model.fit(x, y)
# store results
log_mod_lr_score = model.score(x,y)
from sklearn import cross_validation as cv
from sklearn import metrics
x_train, x_test, y_train, y_test = cv.train_test_split(x,y,test_size=0.20,random_state=1234)
model = LinearRegression().fit(x_train, y_train)
# store results
log_mean_sq_err = metrics.mean_squared_error(y_train,model.predict(x_train))
log_cv_mod_score = model.score(x_train, y_train)
# reset x, y otherwise errors occur
y, x = dmatrices(log_patsy_formula, data=df_sub, return_type='dataframe')
from sklearn.cross_validation import KFold
kf = KFold(len(df_sub), n_folds=10, shuffle=True)
for train_index, test_index in kf:
    x_train, x_test = x.iloc[train_index], x.iloc[test_index]
    y_train, y_test = y.iloc[train_index], y.iloc[test_index]
    clf2 = LinearRegression().fit(x.iloc[train_index], y.iloc[train_index])
# store results
log_mean_sq_errKf = metrics.mean_squared_error(y_train,model.predict(x_train))
log_cvKf_mod_score = clf2.score(x,y)
#LOG RESULTS
print('Log Model Linear Regression Score = {0}'.format(log_mod_lr_score))
print('            Log Mean Square Error = {0}'.format(log_mean_sq_err))
print(' Log Cross Validation Model Score = {0}'.format(log_cv_mod_score))
print('     Log Mean Squred Error K-Fold = {0}'.format(log_mean_sq_errKf))
print('Log Cross Val. K-Fold Model Score = {0}'.format(log_cvKf_mod_score))
df_TEST = pd.read_csv('data/test_data2.csv', encoding='latin-1')
df_TEST['log_budg']=np.log(df_TEST.Prod_Budget)
df_TEST['log_run']=np.log(df_TEST.Runtime)
df_TEST['log_tor']=np.log(df_TEST.Total_Torrents)
def split_to_array(ser):
    split_array = np.array(ser.strip().replace(',','').split(' '))
    return pd.Series(split_array)
genres = df_yr.Genre.apply(split_to_array)
genres = pd.Series(genres.values.ravel()).dropna()
genres = genres.value_counts().sort_values(ascending=False)
def convert_frequency(ser, genres=genres):
    split_array = np.array(ser.strip().replace(',','').split(' '))
    genre = genres.loc[split_array].argmax()
    return genre
df_TEST['Genre_Single'] = df_TEST.Genre.apply(convert_frequency)
log_patsy_formula_test = 'log_tor ~ log_budg + Year + Month + log_run + Genre_Single'
y, x = dmatrices(log_patsy_formula_test, data=df_TEST, return_type='dataframe')
print(clf2.score(x_test, y_test))
print(metrics.mean_squared_error(y_test,model.predict(x_test)))
_ = plt.plot(y, model.predict(x), 'bo')
plt.figure(figsize=(25,10))
ind = np.arange(len(yr_dict))
width = 0.35
bar_year = [year for year, count in yr_lst]
bar_count = [count for year, count in yr_lst]
plt.bar(ind, bar_count, width, color='r')
plt.ylabel('Count')
plt.xlabel('Year')
plt.title('Number of Torrents per Year')
plt.xticks(ind + width/2., (bar_year), rotation='vertical')
plt.yticks(np.arange(0, 91, 5))
plt.show()
#log_tor ~ log_budg + Year + Month + log_run + Genre_Single'
fig = plt.figure(figsize=(12,8))
fig = sm.graphics.plot_regress_exog(results,'log_budg', fig=fig)
fig = plt.figure(figsize=(12,8))
fig = sm.graphics.plot_regress_exog(results,'Year', fig=fig)
fig = plt.figure(figsize=(12,8))
fig = sm.graphics.plot_regress_exog(results,'Month', fig=fig)
Explanation: Log Transform
End of explanation
<END_TASK>