<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 This notebook provides a very basic info about the Kaggle ASAP Dataset  https
Step1:  Essay 5 prompt text and passage refer to the word document in data folder
A sample response from essay type 1
Note
Step2:  Let us this data for features and model building
1. Before features building, we will divide the dataset into train and test dataset arrays. We can use lambda functions to build features
Step3:  2. Lets us Build Features
Let us basic CountVectorizer() and tf-idf (refer here https
Step4:  3. Model building
Step5:  4. Model Prediction
Step6:  This is a very bad model, I will work on this again tonight and build a model. A very bad prediction
<ASSISTANT_TASK:>
Python Code:
import pandas as pd
import numpy as np
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
# Load data
dataset_essay_1 = pd.read_csv("/data/data/automated_scoring_public_dataset.csv")
dataset_essay_1.shape
Explanation: This notebook provides a very basic info about the Kaggle ASAP Dataset  https://www.kaggle.com/c/asap-aes
End of explanation
dataset_essay_1['essay'][0]
print ("Mean word count: ", dataset_essay_1['word_count'].mean())
print ("Max word count: ", dataset_essay_1['word_count'].max())
print ("Min word count: ", dataset_essay_1['word_count'].min())
print ("STD word count: ", dataset_essay_1['word_count'].std())
dataset_essay_1_dropped_NaN_columns = dataset_essay_1.dropna(axis=1, how='all')
dataset_essay_1_dropped_NaN_columns.shape
dataset_essay_1_dropped_NaN_columns.head(2)
Explanation: Essay 5 prompt text and passage refer to the word document in data folder
A sample response from essay type 1
Note: The Kaggle dataset organizers replaced sensitive information like person names and phone numbers with entity types
    You may see the words like @PERSON in student text
End of explanation
# we are interested only in two columns
# data (X) is essay text and truth value being (Y) 'rater1_domain1'
dataset = dataset_essay_1_dropped_NaN_columns[['essay', 'rater1_domain1']]
dataset.rater1_domain1.value_counts() # this is the rater 1 human score distribution
def convert_dataframe_to_arrays(dataset):
    essay_array = np.array(dataset['essay'].tolist()) # data
    essay_rater1 = np.array(dataset['rater1_domain1'].tolist()) # truth value
    return essay_array, essay_rater1
from sklearn.model_selection import train_test_split
def split_train_test_X_Y(dataset):
    X, y = convert_dataframe_to_arrays(dataset)
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, random_state=42)
    print("X_train shape: ", X_train.shape)
    print("X_test shape: ", X_test.shape)
    print("y_train shape: ", y_train.shape)
    print("y_test shape: ", y_test.shape)
    return X_train, X_test, y_train, y_test
# Split it in to train and test arrays
X_train, X_test, y_train, y_test = split_train_test_X_Y(dataset)
X_train[0]
y_train[0]
X_test[0]
y_test[0]
Explanation: Let us this data for features and model building
1. Before features building, we will divide the dataset into train and test dataset arrays. We can use lambda functions to build features
End of explanation
count_vect = CountVectorizer()
X_train_counts = count_vect.fit_transform(X_train)
X_train_counts.shape
X_train_counts[0]
tf_transformer = TfidfTransformer(use_idf=False).fit(X_train_counts)
X_train_tf = tf_transformer.transform(X_train_counts)
X_train_tf.shape
tfidf_transformer = TfidfTransformer()
X_train_tfidf = tfidf_transformer.fit_transform(X_train_counts)
X_train_tfidf.shape
Explanation: 2. Lets us Build Features
Let us basic CountVectorizer() and tf-idf (refer here https://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html), We can also pre-trained embeddings and build features from them
End of explanation
from sklearn.naive_bayes import MultinomialNB
clf = MultinomialNB().fit(X_train_tfidf, y_train)
Explanation: 3. Model building
End of explanation
X_test_counts = count_vect.transform(X_test)
X_test_counts.shape
X_test_tfidf = tfidf_transformer.transform(X_test_counts)
X_test_tfidf.shape
predicted = clf.predict(X_test_tfidf)
Explanation: 4. Model Prediction
End of explanation
predicted
y_test
Explanation: This is a very bad model, I will work on this again tonight and build a model. A very bad prediction
End of explanation
<END_TASK>