<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 0. Data Science 101
An journey through data wrangling, visualization and modeling with some real world data.
All too common question
Step1:  1. Meet the data
2015 NFL Regular Season Fanduel data
Dataset contains a row for each player in the league, for each week
Several Categorical variables (full_name, team, week, home_team, away_team)
Several Numeric lag variables (mean_passing_att across all prior weeks, prev_passing_att for one prior week)
Step2:  What columns do we have?
Step3:  A quick look at our data
Step4:  Where do we begin?
We know some Python. Let's use that.
First thing to figure out is how to get the data files on your machine into Python in the first place.
We can use Pandas.
Pandas
Pandas is a Python library you may have heard of that is great for exploring data in real time. Let's look at what we already did with Pandas and what else we can do!
Reading data from disk
Step5:  What did that just do?
We called "read_csv", which presumably reads CSV files... and does what with them? 
pd.read_csv does a magical thing
It reads a CSV file into a DataFrame. 
DataFrames are mystical creatures in Data Science. 
Popularized by R, they provide a standardized matrix-style format for interacting with your data. Most data can fit into this row and column format
Step6:  Get subset of columns (DataFrame)
Step7:  Get a subset of rows using a boolean array
Step8:  So now we know about DataFrames
...What's next?
Exploratory Data Analysis
Does it look like test data should? 
Is it completely empty? Full? Lots of missing values and NaNs?
What are in the rows? columns?
Does it have appropriate features? (characteristics common to records belonging to a dataset)
It's impossible to make good decisions moving forward until we know more
We can just output the entire dataframe to the console, but that doesn't scale beyond a couple hundred rows.
<pre>
In [1]
Step9:  What about relationships between variables?
Should we compute covariance?
Nah, let's make some plots!
A quick aside.
IPython Notebooks (like this one) are great.
Plots in your notebook!
Two plotting weapons
Matplotlib
The historical go-to for plotting
allows lots of fine-grained control
built with numpy in mind (Numpy and its cousin Scipy are the number-crunching go-to's in Python)
Seaborn
Expressive power
built with pandas in mind
trendy newcomer, but gaining a loyal following
We will mainly use seaborn examples in this presentation. It's very intuitive and powerful to use.
What is the distribution of points earned?
Step10:  Target is skewed --- let's transform it so it's more normal
Step11:  I see lots of zeroes --- Where are these bad performances coming from?
Step12:  Modeling
Scikit-Learn
Widely used machine learning package
Classification Models
Regression Models
Clustering techniques
Dimensionality Reduction
Preprocessing
...
Experimental Design
Build a model that is useful
Can only build model on data that has a known output
If we had missing values in our target (output variable) we would want to be careful
Avoiding overfitting
Need to keep data we train on from data we validate on
Otherwise the results will be overly optimistic
... and ultimately, the model will perform poorly on new data
We will setup our training and test set in a bit
Step13:  Today we will use simple train/test split
Recommend using multiple cross-validation folds
Variable Preprocessing
Most statistical models require numeric encoding
Many will choke on missing values
Need some massaging before fitting a statistical learner
Step14:  Missing values in numeric columns
We will impute with the median value
<ASSISTANT_TASK:>
Python Code:
%pylab inline
import pandas as pd
import seaborn as sns
Explanation: 0. Data Science 101
An journey through data wrangling, visualization and modeling with some real world data.
All too common question: Where do I start?
Should I get a graduate degree in Statistics or Machine Learning?
NO
The world's data science problems will never be solved by this approach
You are what you repeatedly do/practice.
Mindset is more important than any particular technique
Most important tools:
Curiosity, Patience, Common Sense, Healthy Skepticism
Let's go on a first pass through a dataset, with a focus on exploration.
End of explanation
raw_df = pd.read_csv("/home/brianb/Downloads/odsc_football_modeling_data_2.csv")
df_no_week_1 = raw_df[raw_df.week > 1]
Explanation: 1. Meet the data
2015 NFL Regular Season Fanduel data
Dataset contains a row for each player in the league, for each week
Several Categorical variables (full_name, team, week, home_team, away_team)
Several Numeric lag variables (mean_passing_att across all prior weeks, prev_passing_att for one prior week)
End of explanation
ff_cols = raw_df.columns
sort(ff_cols.values)
Explanation: What columns do we have?
End of explanation
raw_df[ff_cols].head()
df_no_week_1[ff_cols].head()
Explanation: A quick look at our data
End of explanation
raw_df = pd.read_csv("/home/brianb/Downloads/odsc_football_modeling_data_2.csv")
Explanation: Where do we begin?
We know some Python. Let's use that.
First thing to figure out is how to get the data files on your machine into Python in the first place.
We can use Pandas.
Pandas
Pandas is a Python library you may have heard of that is great for exploring data in real time. Let's look at what we already did with Pandas and what else we can do!
Reading data from disk
End of explanation
raw_df['full_name'].head()
Explanation: What did that just do?
We called "read_csv", which presumably reads CSV files... and does what with them? 
pd.read_csv does a magical thing
It reads a CSV file into a DataFrame. 
DataFrames are mystical creatures in Data Science. 
Popularized by R, they provide a standardized matrix-style format for interacting with your data. Most data can fit into this row and column format: financial transactions, iPhone app user records, medical histories, etc.
Since you were wondering
Pandas has support for many formats
CSV, Text (tab separated, pipe separated, etc.), Excel, JSON, HTML, SQL, Stuff copied to your clipboard, HDFS...
Hold up. What's really going on with these DataFrames?
Two data structures: Series and DataFrame
Series
Think of this as one column of your data - one data type.
DataFrame
All of the columns in your data. Mixed data types. 
Many Series can be combined and represented as a DataFrame object.
A DataFrame can be represented as many Series objects.
Pandas provides tons of functions to:
slice, dice, merge, join, group by, select, append, find, transform, sort, reverse, pivot and anything else you want to do
... for both Series and DataFrames.
Most functions are designed to work with either type or even combinations of the two, just like you would intuitively expect:
i.e. A concat function can contatenate arbitrary combinations of 0 to n Series and DataFrames.
Accessing data in a DataFrame
Get one column (Series)
End of explanation
raw_df[['full_name', 'position', 'team']].head()
Explanation: Get subset of columns (DataFrame)
End of explanation
raw_df[raw_df.week > 1].head()
Explanation: Get a subset of rows using a boolean array
End of explanation
df_no_week_1 = raw_df[raw_df.week > 1]
df_no_week_1.isnull().sum()
Explanation: So now we know about DataFrames
...What's next?
Exploratory Data Analysis
Does it look like test data should? 
Is it completely empty? Full? Lots of missing values and NaNs?
What are in the rows? columns?
Does it have appropriate features? (characteristics common to records belonging to a dataset)
It's impossible to make good decisions moving forward until we know more
We can just output the entire dataframe to the console, but that doesn't scale beyond a couple hundred rows.
<pre>
In [1]: df = DataFrame(np.random.randn(10, 4))
In [2]: df
Out[2]: 
          0         1         2         3
0  0.469112 -0.282863 -1.509059 -1.135632
1  1.212112 -0.173215  0.119209 -1.044236
2 -0.861849 -2.104569 -0.494929  1.071804
3  0.721555 -0.706771 -1.039575  0.271860
4 -0.424972  0.567020  0.276232 -1.087401
5 -0.673690  0.113648 -1.478427  0.524988
6  0.404705  0.577046 -1.715002 -1.039268
7 -0.370647 -1.157892 -1.344312  0.844885
8  1.075770 -0.109050  1.643563 -1.469388
9  0.357021 -0.674600 -1.776904 -0.968914
</pre>
Look for missing values
End of explanation
pylab.hist(raw_df['fanduel_points'],
               normed=True,
               bins=np.linspace(-1, 35, 12),
               alpha=0.35,
               label='fanduel_points')
pylab.legend()
pylab.figure(figsize=(15,15))
Explanation: What about relationships between variables?
Should we compute covariance?
Nah, let's make some plots!
A quick aside.
IPython Notebooks (like this one) are great.
Plots in your notebook!
Two plotting weapons
Matplotlib
The historical go-to for plotting
allows lots of fine-grained control
built with numpy in mind (Numpy and its cousin Scipy are the number-crunching go-to's in Python)
Seaborn
Expressive power
built with pandas in mind
trendy newcomer, but gaining a loyal following
We will mainly use seaborn examples in this presentation. It's very intuitive and powerful to use.
What is the distribution of points earned?
End of explanation
transformed_target = pd.DataFrame.copy(raw_df[raw_df.fanduel_points > 1])
transformed_target['fanduel_points'] = np.log(raw_df[raw_df.fanduel_points > 0]['fanduel_points']-1)
pylab.figure(figsize=(15,15))
pylab.hist(transformed_target['fanduel_points'],
               normed=True,
               bins=np.linspace(-1, 4, 100),
               alpha=0.35,
               label='fanduel_points')
pylab.legend()
Explanation: Target is skewed --- let's transform it so it's more normal
End of explanation
no_nans = raw_df[raw_df[raw_df.fanduel_points< 3].notnull()]
no_nans = raw_df[raw_df.position != 'UNK'] #remove Unknowns from this dataframe
no_nans.groupby('position').size()
bad_performances = pd.DataFrame({'count' : no_nans.groupby('position').size()}).reset_index()
bad_performances = bad_performances.sort(['count'], ascending=[0])
print bad_performances
g = sns.factorplot("position", "count",
                    data=bad_performances, kind="bar",
                    size=15, palette="pastel", dropna=True, x_order=bad_performances.position.values)
#Let's look at an individual player
raw_df[(raw_df.full_name =='Tom Brady') & (raw_df.week == 2)]
#All positions are not created equal -- some score many more points than others
raw_df[raw_df[raw_df.fanduel_points> 1].notnull()].groupby('position')['fanduel_points'].sum()
#Since a few positions seem to score all the points, let's zoom in on those
plot_order = ['TE', 'WR', 'RB', 
              'K', 'QB']
top_positions_only = raw_df[raw_df.position.isin(plot_order)]
top_positions_only.groupby('position')['fanduel_points'].mean()
# Violin plots are a nice alternative to boxplots that also show interesting detail about
# the shape of the distribution
nonnull_subset = top_positions_only['fanduel_points'].notnull()
plt.figure(figsize=(12, 6))
sns.violinplot(top_positions_only['fanduel_points'][nonnull_subset], 
               top_positions_only['position'][nonnull_subset], 
               inner='box',
               order=plot_order,
               bw=1,
               size=16)
# QB's and kickers score the most points. Let's look into those using a Histogram
qb_k = ['K', 'QB']
qb_k_data = raw_df[raw_df.position.isin(qb_k)]
groups = qb_k_data.groupby('position').groups
pylab.figure(figsize=(15,5))
for key, row_ids in groups.iteritems():
    pylab.hist(qb_k_data['fanduel_points'][row_ids].values,
               normed=True,
               bins=np.linspace(-10, 50, 50),
               alpha=0.35,
               label=str(key))
pylab.legend()
# We can use a Facetgrid to analyze teams
def vertical_mean_line(x, **kwargs):
    plt.axvline(np.percentile(x, 95), **kwargs)
    
teams = ['NE', 'CHI']
team_data = raw_df[raw_df.team.isin(teams)]
team_data = team_data[team_data.week < 4]
g = sns.FacetGrid(team_data, row="team", col="week", 
                  margin_titles=True, dropna=True, size=4)
bins = np.linspace(-3, 30, 30)
g.map(plt.hist, "fanduel_points", color="black", bins=bins, 
      lw=0, normed=True)
g.map(vertical_mean_line, 'fanduel_points')
# We can use a Heatmap to analyze teams kicker and qb performance
teams = ['NE', 'CHI', 'NYG', 'DET', 'NYJ']
team_data = qb_k_data[qb_k_data.team.isin(teams)]
team_data = team_data[team_data.week < 10]
ptable = pd.pivot_table(
    team_data, 
    values='fanduel_points', 
    index=["team"], 
    columns='week')
reorder_teams = ptable.reindex(teams).fillna(0)
pylab.figure(figsize=(15,5))
sns.heatmap(reorder_teams.astype(int), annot=True, fmt="d", cmap="YlGnBu")
# Zero values are bye weeks
# Are previous week's points a good predictor of current week's points? 
# Let's consider only kicker and QB data for these teams
# We have to exclude week 1 here since there is no previous weeks' mean
team_data_no_week_1 = team_data[team_data.week > 1]
grid = sns.JointGrid(team_data_no_week_1['mean_fanduel_points'], 
              team_data_no_week_1['fanduel_points'],space=0, size=10, ratio=50)
grid.plot_joint(plt.scatter, color="g")
grid.plot_marginals(sns.rugplot, height=1, color="g")
# We can use jointplot (uses JointGrid internally) to get a quick regression line for this
sns.jointplot('mean_fanduel_points', 'fanduel_points', data=team_data_no_week_1, 
              kind="reg", color=sns.color_palette()[1], size=9)
# QB's are significantly more important than any other position. Let's dig in
qb_df = raw_df[raw_df.position == 'QB']
# Passing attempts from previous weeks --- is there a trend with next week's performance? No
sns.jointplot(qb_df['mean_passing_att'], 
              qb_df['fanduel_points'], kind="reg", size=9)
Explanation: I see lots of zeroes --- Where are these bad performances coming from?
End of explanation
import sklearn
# Let's prep for modeling
exclude_week_1 = top_positions_only[top_positions_only.week > 1]
model_data = pd.DataFrame.copy(exclude_week_1)
model_data = model_data[model_data.fanduel_points > 0]
print np.isnan(model_data['fanduel_points']).sum()
# Let's cut our target out so we don't train on it
target = model_data.pop('fanduel_points')
# We don't need player id's --- let's throw this away
throw_away = model_data.pop('player_id')
import sklearn.cross_validation
(train_data, 
 test_data, 
 train_target, 
 test_target) = sklearn.cross_validation.train_test_split(
    model_data, target, test_size=0.2, random_state=1337
)
Explanation: Modeling
Scikit-Learn
Widely used machine learning package
Classification Models
Regression Models
Clustering techniques
Dimensionality Reduction
Preprocessing
...
Experimental Design
Build a model that is useful
Can only build model on data that has a known output
If we had missing values in our target (output variable) we would want to be careful
Avoiding overfitting
Need to keep data we train on from data we validate on
Otherwise the results will be overly optimistic
... and ultimately, the model will perform poorly on new data
We will setup our training and test set in a bit
End of explanation
#Handle categorical vars
import sklearn.preprocessing
import sklearn.feature_extraction
from sklearn.feature_extraction import DictVectorizer
encoder = DictVectorizer(sparse=False)
#Let's do one-hot encoding in sklearn using DictVectorizer
categorical_vars = ['full_name', 'position', 'team', 'week', 'opponent', 'home_team', 'away_team']
vardata = train_data[categorical_vars].fillna('MISSING')
encoder.fit(vardata.to_dict(orient='records'))
train_catdata = encoder.transform(vardata.to_dict(orient='records'))
test_vardata = test_data[categorical_vars].fillna('MISSING')
test_catdata = encoder.transform(
    test_vardata[categorical_vars].to_dict(orient='records'))
pd.DataFrame(train_catdata).describe()
Explanation: Today we will use simple train/test split
Recommend using multiple cross-validation folds
Variable Preprocessing
Most statistical models require numeric encoding
Many will choke on missing values
Need some massaging before fitting a statistical learner
End of explanation
#Handle numeric vars
from sklearn.preprocessing import Imputer
imputer = Imputer(strategy='median')
numeric_vars = list(set(train_data.columns.tolist()) - set(categorical_vars))
numdata = train_data[numeric_vars]
imputer.fit(numdata)
train_numdata = imputer.transform(numdata)
test_numdata = imputer.transform(test_data[numeric_vars])
train_this = np.hstack([train_numdata, train_catdata])
test_this = np.hstack([test_numdata, test_catdata])
import sklearn
from sklearn.linear_model import LinearRegression
print np.any(isnan(train_numdata))
print np.all(np.isfinite(train_numdata))
lr = LinearRegression(fit_intercept=False)
lr.fit(train_numdata, train_target)
lr_predictions = pd.Series(lr.predict(test_numdata),
                           name='Linear Regression')
p_df = pd.DataFrame({'Prediction': lr_predictions,
                     'Actual': test_target.values})
pylab.figure(figsize=(10, 10))
sns.jointplot('Actual', 'Prediction', data=p_df, 
              kind="hex", color=sns.color_palette()[1])
#Let's take a look at our residuals using using just the categorical vars
from sklearn import metrics
test_metrics = {
    'Explained Variance': metrics.explained_variance_score,
    'MAE': metrics.mean_absolute_error,
    'MSE': metrics.mean_squared_error,
    'MedAE': metrics.median_absolute_error,
    'R2': metrics.r2_score
}
def metrics_report(*predictions):
    records = []
    for prediction_set in predictions:
        record = {'name': prediction_set.name}
        for metric_name in sorted(test_metrics.keys()):
            metric_func = test_metrics[metric_name]
            record[metric_name] = metric_func(test_target, prediction_set)
        records.append(record)
    frame = pd.DataFrame.from_records(records).set_index('name')
    return frame
        
metrics_report(lr_predictions)
# We need to add reference models to track a baseline performance that we can compare our other models to
mean_response = np.mean(train_target)
mean_predictions = pd.Series(np.ones_like(test_target) * mean_response,
                             name='Mean Response')
median_response = np.median(train_target)
median_predictions = pd.Series(np.ones_like(test_target) * median_response,
                               name='Median Response')
metrics_report(mean_predictions, 
               median_predictions, 
               lr_predictions)
#Time for ElasticNet
from sklearn.grid_search import GridSearchCV
from sklearn.linear_model import ElasticNet
estimator = ElasticNet()
parameters = {
    'alpha': np.linspace(0.1, 2, 10, endpoint=True),
    'l1_ratio': np.linspace(0, 1, 10, endpoint=True)
}
enet = GridSearchCV(estimator, parameters)
enet.fit(train_numdata, train_target)
print(enet.best_params_, enet.best_score_)
print()
print("Grid scores on development set:")
print()
for params, mean_score, scores in enet.grid_scores_:
    print("%0.3f (+/-%0.03f) for %r" % (mean_score, scores.std() * 2, params))
print()
estimator2 = ElasticNet()
parameters2 = {
    'alpha': np.linspace(0.4, 0.6, 10, endpoint=True),
    'l1_ratio': np.linspace(0.4, 0.6, 10, endpoint=True)
}
enet2 = GridSearchCV(estimator2, parameters2)
enet2.fit(train_numdata, train_target)
print(enet2.best_params_, enet2.best_score_)
print()
print("Grid scores on development set:")
print()
for params, mean_score, scores in enet2.grid_scores_:
    print("%0.3f (+/-%0.03f) for %r" % (mean_score, scores.std() * 2, params))
print()
enet_predictions = pd.Series(enet.predict(test_numdata),
                             name='Elastic Net')
p_df = pd.DataFrame({'Enet Prediction': enet_predictions,
                     'Actual': test_target.values})
pylab.figure(figsize=(10, 10))
sns.jointplot('Actual', 'Enet Prediction', data=p_df, kind="hex",
              color=sns.color_palette()[2])
from sklearn.grid_search import GridSearchCV
from sklearn.ensemble import RandomForestRegressor
estimator = RandomForestRegressor()
parameters = {'n_estimators': (5, 10, 15, 20, 25, 30, 35),
              'max_depth': (3, 5, 7, 9, 11),
             }
rfr = GridSearchCV(estimator, parameters, n_jobs=3)
rfr.fit(train_this, train_target)
rfr_predictions = pd.Series(rfr.predict(test_this),
                            name='Random Forest')
p_df = pd.DataFrame({'RF Prediction': rfr_predictions,
                     'Actual': test_target.values})
pylab.figure(figsize=(10, 10))
sns.jointplot('Actual', 'RF Prediction', data=p_df, kind="hex",
              color=sns.color_palette()[3])
metrics_report(mean_predictions,
               median_predictions,
               lr_predictions,
               enet_predictions,
               rfr_predictions)
lr_diffs = lr_predictions - test_target
lr_diffs.name = 'LinearRegression Error'
rfr_diffs = rfr_predictions - test_target
rfr_diffs.name = 'RandomForest Error'
sns.jointplot(lr_predictions, rfr_predictions, kind='resid', color=sns.color_palette()[4])
from sklearn.linear_model import Lasso
from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import make_pipeline
degree = 2
model = make_pipeline(PolynomialFeatures(degree), Lasso())
model.fit(train_numdata, train_target)
poly_preds = pd.Series(model.predict(test_numdata),
                       name='Polynomial Lasso',
                       index=test_target.index)
sns.jointplot(test_target, 
              poly_preds,
              kind='resid',
              color=sns.color_palette()[5])   
metrics_report(mean_predictions,
               median_predictions,
               lr_predictions,
               enet_predictions,
               rfr_predictions,
               poly_preds)
Explanation: Missing values in numeric columns
We will impute with the median value
End of explanation
<END_TASK>