<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Project 2
Movie Genre Classification
Classify a movie genre based on its plot.
<img src="moviegenre.png"
     style="float
Step1:  Create count vectorizer
Step2:  Create y
Step3:  Train multi-class multi-label model
Step4:  Predict the testing dataset
<ASSISTANT_TASK:>
Python Code:
import pandas as pd
import os
import numpy as np
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.preprocessing import MultiLabelBinarizer
from sklearn.multiclass import OneVsRestClassifier
from sklearn.ensemble import RandomForestRegressor, RandomForestClassifier
from sklearn.metrics import r2_score, roc_auc_score
from sklearn.model_selection import train_test_split
dataTraining = pd.read_csv('https://github.com/albahnsen/PracticalMachineLearningClass/raw/master/datasets/dataTraining.zip', encoding='UTF-8', index_col=0)
dataTesting = pd.read_csv('https://github.com/albahnsen/PracticalMachineLearningClass/raw/master/datasets/dataTesting.zip', encoding='UTF-8', index_col=0)
dataTraining.head()
dataTesting.head()
Explanation: Project 2
Movie Genre Classification
Classify a movie genre based on its plot.
<img src="moviegenre.png"
     style="float: left; margin-right: 10px;" />
https://www.kaggle.com/c/miia4200-20191-p2-moviegenreclassification/overview
Data
Input:
- movie plot
Output:
Probability of the movie belong to each genre
Evaluation
20% API
30% Create a solution using with a Machine Learning algorithm - Presentation (5 slides)
50% Performance in the Kaggle competition (Normalized acording to class performance in the private leaderboard)
Acknowledgements
We thank Professor Fabio Gonzalez, Ph.D. and his student John Arevalo for providing this dataset.
See https://arxiv.org/abs/1702.01992
Sample Submission
End of explanation
vect = CountVectorizer(max_features=1000)
X_dtm = vect.fit_transform(dataTraining['plot'])
X_dtm.shape
print(vect.get_feature_names()[:50])
Explanation: Create count vectorizer
End of explanation
dataTraining['genres'] = dataTraining['genres'].map(lambda x: eval(x))
le = MultiLabelBinarizer()
y_genres = le.fit_transform(dataTraining['genres'])
y_genres
X_train, X_test, y_train_genres, y_test_genres = train_test_split(X_dtm, y_genres, test_size=0.33, random_state=42)
Explanation: Create y
End of explanation
clf = OneVsRestClassifier(RandomForestClassifier(n_jobs=-1, n_estimators=100, max_depth=10, random_state=42))
clf.fit(X_train, y_train_genres)
y_pred_genres = clf.predict_proba(X_test)
roc_auc_score(y_test_genres, y_pred_genres, average='macro')
Explanation: Train multi-class multi-label model
End of explanation
X_test_dtm = vect.transform(dataTesting['plot'])
cols = ['p_Action', 'p_Adventure', 'p_Animation', 'p_Biography', 'p_Comedy', 'p_Crime', 'p_Documentary', 'p_Drama', 'p_Family',
        'p_Fantasy', 'p_Film-Noir', 'p_History', 'p_Horror', 'p_Music', 'p_Musical', 'p_Mystery', 'p_News', 'p_Romance',
        'p_Sci-Fi', 'p_Short', 'p_Sport', 'p_Thriller', 'p_War', 'p_Western']
y_pred_test_genres = clf.predict_proba(X_test_dtm)
res = pd.DataFrame(y_pred_test_genres, index=dataTesting.index, columns=cols)
res.head()
res.to_csv('pred_genres_text_RF.csv', index_label='ID')
Explanation: Predict the testing dataset
End of explanation
<END_TASK>