<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Question 3
The accompanying CSV files ‘pitchclassificationtrain.csv’ and ‘pitchclassificationtest.csv’ contain information from nearly 20,000 pitches from five different pitchers over three years. There are 13 columns
Step1:  Extract columns to be used for prediction. Pitcher and year are probably not predictive, so I am leaving them out.
Step2:  Relabel pitch types using the scikit-learn label encoder (XGboost requires sequential labels).
Step3:  XGBoost runs faster using its own binary data structures
Step4:  We can specify the hyperparameters of the model, to use as a starting point for the analysis.
Step5:  Using arbitrarily-chosen hyperparemeters, the model achieves nearly perfect accuracy on the training set.
Step6:  However, this model may be overfit to the training dataset, so I am going to use 5-fold cross-validation to select hyperparameters that result in the best fit.
Step7:  I can select the best parameters from the cross-validation procedure to use to predict on the test data (I could do a more refined search of the hyperparameter space, but the multiclass errors appear to be broadly similar across the range of values that I used, so I will stick with these).
Step8:  The accuracy score for the training data is nominally lower than the original model, but not much; moreover this model should perform better in out-of-sample prediction.
Step9:  Below is a plot of feature importances, using the F-score. This quantifies how many times a particular variable is used as a splitting variable across all the trees. This ranking makes intiutive sense, with movement and velocity being the most relevant factors.
Step10:  Generate predictions on the test set using fitted classifier.
Step11:  Back-transform label encoding, and export to predicted_pitches_fonnesbeck.csv.
<ASSISTANT_TASK:>
Python Code:
from pybaseball import statcast
pitch_data = statcast(start_dt='2017-04-01', end_dt='2017-04-30')
pitch_data.shape
pitch_data.pitch_type.value_counts()
pitch_type = pitch_data.pop('pitch_type')
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
                    pitch_data, pitch_type, test_size=0.33, random_state=42)
Explanation: Question 3
The accompanying CSV files ‘pitchclassificationtrain.csv’ and ‘pitchclassificationtest.csv’ contain information from nearly 20,000 pitches from five different pitchers over three years. There are 13 columns:
pitchid: a unique identifier for each pitch.
pitcherid: identity of the pitcher (1-5). The identities are the same in both datasets.         Pitcher 3 in the training set is the same pitcher as Pitcher 3 in the test set.
yearid: year in which the pitch occurred (1-3).
throws: handedness of the pitcher, where 1 = right-handed and 2 = left-handed
height: height in inches of the pitcher.
initspeed: initial speed of the pitch as it leaves the pitcher's hand, reported in MPH
breakx: horizontal distance in inches between where a pitch crossed the plate and where a hypothetical spinless pitch would have, where negative is inside to a right-handed hitter.
breakz: vertical distance in inches between where a pitch crossed the plate and where a hypothetical spinless pitch would have, where negative is closer to the ground.
initposx: horizontal position of the release point of the pitch. The position is measured in feet from the center of the rubber when the pitch is released, where negative is towards the third-base side of the rubber.
initposz: vertical position of the release point of the pitch.  The position is measured in feet above the ground.
extension: distance in feet in front of the pitching rubber from which the pitcher releases the ball.
spinrate: how fast the ball is spinning as it leaves the pitcher's hand, reported in RPM
type: type of pitch that was thrown (will only appear in the training dataset).
Your goal is to give the most likely pitch type for all of the pitches in the test dataset using information from the training dataset. Note that the pitchers in the datasets do not correspond with any specific real pitchers but are meant to be representative of real data. Please include the following with your final submission:
CSV with two columns: the pitchid and the corresponding predicted pitch type
write-up of your method and results, including any tables or figures that help communicate your findings
all code used to solve the problem
This is a multi-class classification problem. One approach is to use a parametric statistical model, and express the pitch class as a categorical outcome. However, this requires the explicit specification of the combinations of variables thought to be predictive of pitch type a priori. A more flexible approach, and an effective one when plenty of data are available (as we have here), is to use an ensemble machine learning algorithm. A key advantage of such methods is that higher-order interactions among variables that may be powerful predictors can be discovered automatically, without pre-specification. 
An effective ensemble method for classification is gradient boosting. Boosting combines a set of "weak" learners (ones that, individually, perform only slightly better than chance) to a yield a system with very high predictive performance. The idea is that by sequentially applying very fast, simple models, we can get a total model error which is better than any of the individual pieces. Each successive fit to a new weak learner yields and estimate of residual error, which is used to weight the remaining observations such that the next learner emphasizes the classification of observations that have not yet been correctly classified. The process is stagewise, meaning that existing trees are left unchanged as the model is enlarged. Only the fitted value for each observation is re-estimated at each step to reflect the contribution of the newly added tree. The final model is a linear combination of many trees (usually hundreds to thousands) that can be thought of as a classification/regression model where each term itself is a tree.
On average, boosting outperforms competing algorithms like random forests or support vector machines, but requires less data than deep neural networks, which are often applied to large classification tasks.
XGBoost is an open-source, high-performance implementation of gradient boosting methods for decision trees. It features natively-parallel tree construction, out-of-core computation, and cache optimization, and is readily deployable on clusters. It typically offers better speed and performance relative to scikit-learn and other comparable libraries.
Import data from csv files.
End of explanation
prediction_cols = ['p_throws', 'release_spin_rate', 'effective_speed', 'release_extension',
                   'vx0', 'vy0', 'vz0', 'ax', 'ay', 'az']
Explanation: Extract columns to be used for prediction. Pitcher and year are probably not predictive, so I am leaving them out.
End of explanation
from sklearn.preprocessing import LabelEncoder
encoder = LabelEncoder().fit(y_train)
y_train_encoded = encoder.transform(y_train)
Explanation: Relabel pitch types using the scikit-learn label encoder (XGboost requires sequential labels).
End of explanation
import xgboost as xgb
dtrain = xgb.DMatrix(X_train[prediction_cols], label=y_train_encoded)
dtest = xgb.DMatrix(X_test[prediction_cols])
Explanation: XGBoost runs faster using its own binary data structures:
End of explanation
xgb_params = {
    'max_depth': 12,  
    'eta': 0.2, 
    'nthread': 4,                # Use 4 cores for multiprocessing
    'num_class': 6,              # pitch types
    'objective': 'multi:softmax' # use softmax multi-class classification
}
boosted_classifier = xgb.train(xgb_params, dtrain, num_boost_round=30)
Explanation: We can specify the hyperparameters of the model, to use as a starting point for the analysis.
End of explanation
from sklearn.metrics import accuracy_score
predictions = boosted_classifier.predict(dtrain)
accuracy_score(y_train_encoded, predictions)
Explanation: Using arbitrarily-chosen hyperparemeters, the model achieves nearly perfect accuracy on the training set.
End of explanation
param_grid = [(max_depth, min_child_weight, eta) for max_depth in (6, 8, 10, 12) 
                                                 for min_child_weight in (7, 9, 11, 13)
                                                 for eta in (0.15, 0.2, 0.25)]
min_merror = np.inf
best_params = None
for max_depth, min_child_weight, eta in param_grid:
    print("CV with max_depth={}, min_child_weight={}, eta={}".format(
                             max_depth,
                             min_child_weight,
                             eta))
    # Update our parameters
    xgb_params['max_depth'] = max_depth
    xgb_params['min_child_weight'] = min_child_weight
    xgb_params['eta'] = eta
    # Run CV
    cv_results = xgb.cv(
        xgb_params,
        dtrain,
        num_boost_round=50,
        nfold=5,
        metrics={'merror'}, 
        early_stopping_rounds=3
    )
    # Update best score
    mean_merror = cv_results['test-merror-mean'].min()
    boost_rounds = cv_results['test-merror-mean'].idxmin()
    print("\tmerror {} for {} rounds".format(mean_merror, boost_rounds))
    if mean_merror < min_merror:
        min_merror = mean_merror
        best_params = (max_depth, min_child_weight, eta)
print("Best params: {}, {}, {}, merror: {}".format(best_params[0], best_params[1], best_params[2], min_merror))
Explanation: However, this model may be overfit to the training dataset, so I am going to use 5-fold cross-validation to select hyperparameters that result in the best fit.
End of explanation
xgb_params = {
    'max_depth': 10,  
    'eta': 0.2, 
    'min_child_weight': 9,       # minimum sum of instance weight needed in a child 
    'nthread': 4,                # use 4 cores for multiprocessing
    'num_class': 6,              # pitch types
    'objective': 'multi:softmax' # use softmax multi-class classification
}
boosted_classifier = xgb.train(xgb_params, dtrain, num_boost_round=30)
Explanation: I can select the best parameters from the cross-validation procedure to use to predict on the test data (I could do a more refined search of the hyperparameter space, but the multiclass errors appear to be broadly similar across the range of values that I used, so I will stick with these).
End of explanation
from sklearn.metrics import accuracy_score
predictions = boosted_classifier.predict(dtrain)
accuracy_score(y_train_encoded, predictions)
Explanation: The accuracy score for the training data is nominally lower than the original model, but not much; moreover this model should perform better in out-of-sample prediction.
End of explanation
xgb.plot_importance(boosted_classifier)
Explanation: Below is a plot of feature importances, using the F-score. This quantifies how many times a particular variable is used as a splitting variable across all the trees. This ranking makes intiutive sense, with movement and velocity being the most relevant factors.
End of explanation
test_predictions = boosted_classifier.predict(dtest)
test_predictions
Explanation: Generate predictions on the test set using fitted classifier.
End of explanation
predicted_pitches = pd.Series(encoder.inverse_transform(test_predictions.astype(int)), index=X_test.index)
predicted_pitches.name = 'pitch_type'
predicted_pitches.index.name = 'pitchid'
predicted_pitches.to_csv('predicted_pitches_fonnesbeck.csv', header=True)
Explanation: Back-transform label encoding, and export to predicted_pitches_fonnesbeck.csv.
End of explanation
<END_TASK>