<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Prediction Failed Movies
Loading the dataset
Step1:  Feature Generation
Generating some additional basic features
Step2:  The number of null values per column
Step3:  Keeping all genre dummy variables
Step4:  Dropping non-feature columns
Step5:  Dropping all rows that still have null values
Step6:  Now, making sure we have no null values
Step7:  We end up with a dataset of size
Step8:  Prediction
Step9:  Predicting failed movies
We define a failed movie as a movie whose ROI (Return On Investment) is below zero, meaning the investors actually lost money making it
Step10:  Running logistic regression over 5 folds of our dataset
<ASSISTANT_TASK:>
Python Code:
import os
import pandas as pd
import sklearn as skl
import holcrawl.shared
dataset_dir = holcrawl.shared._get_dataset_dir_path()
dataset_path = os.path.join(dataset_dir, 'movies_dataset.csv')
df = pd.read_csv(dataset_path)
Explanation: Prediction Failed Movies
Loading the dataset
End of explanation
df['ROI'] = (df['gross_income'] - df['budget']) / df['budget']
df['name_length'] = df['name'].map(lambda name: len(name))
len(df)
Explanation: Feature Generation
Generating some additional basic features:
End of explanation
df.isnull().sum()
BASE_FEAT_TO_KEEP = [
    'duration', 'budget', 'opening_month', 'opening_day', 'opening_day_of_year', 'year',
    'avg_mc_critic_by_opening', 'num_mc_critic_by_opening', 'name_length', 'opening_weekend_income',
    'num_imdb_user_by_opening', 'avg_imdb_user_by_opening', 'opening_weekend_screens'# 'avg_mc_user_by_opening'
]
Explanation: The number of null values per column:
End of explanation
FEAT_TO_KEEP = BASE_FEAT_TO_KEEP + [col for col in df.columns if 'genres' in col]
features = df.drop([col for col in df.columns if col not in BASE_FEAT_TO_KEEP], axis=1)
Explanation: Keeping all genre dummy variables:
End of explanation
dataset = df.drop([col for col in df.columns if col not in FEAT_TO_KEEP], axis=1)
Explanation: Dropping non-feature columns:
End of explanation
dataset = dataset.dropna(axis=0)
Explanation: Dropping all rows that still have null values:
End of explanation
dataset.isnull().sum().sum()
Explanation: Now, making sure we have no null values:
End of explanation
len(dataset)
Explanation: We end up with a dataset of size:
End of explanation
import numpy as np
from sklearn import linear_model
from sklearn.model_selection import cross_val_score
Explanation: Prediction
End of explanation
failed = df['ROI'].ix[dataset.index] < 0
X = dataset
Y = failed
Explanation: Predicting failed movies
We define a failed movie as a movie whose ROI (Return On Investment) is below zero, meaning the investors actually lost money making it:
End of explanation
logreg = linear_model.LogisticRegression()
acc_scores = cross_val_score(logreg, X, Y, cv=5, n_jobs=1)
mean_accuracy = np.mean(acc_scores)
accuracy_std = np.std(acc_scores)
print("Accuracy is {:.2f}% ± {:.2f}%.".format(mean_accuracy*100, accuracy_std*100))
recall_scores = cross_val_score(logreg, X, Y, cv=5, n_jobs=1, scoring='recall')
mean_recall = np.mean(recall_scores)
recall_std = np.std(recall_scores)
print("Recall = {:.2f}% ± {:.2f}".format(mean_recall*100, recall_std*100))
Explanation: Running logistic regression over 5 folds of our dataset:
End of explanation
<END_TASK>