<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Working with Text Data
<img src="figures/bag_of_words.svg" width=100%>
Step1:  Exercises
Create a pipeine using the count vectorizer and SVM (see 07). Train and score using the pipeline.
Vary the n_gram_range in the count vectorizer, visualize the changed coefficients.
Grid search the C in the LinearSVC using the pipeline.
Grid search the C in the LinearSVC together with the n_gram_range (try (1,1), (1, 2), (2, 2))
<ASSISTANT_TASK:>
Python Code:
import pandas as pd
import os
data = pd.read_csv(os.path.join("data", "train.csv"))
len(data)
data
y_train = np.array(data.Insult)
y_train
text_train = data.Comment.tolist()
text_train[6]
data_test = pd.read_csv(os.path.join("data", "test_with_solutions.csv"))
text_test, y_test = data_test.Comment.tolist(), np.array(data_test.Insult)
from sklearn.feature_extraction.text import CountVectorizer
cv = CountVectorizer()
cv.fit(text_train)
len(cv.vocabulary_)
print(cv.get_feature_names()[:50])
print(cv.get_feature_names()[-50:])
X_train = cv.transform(text_train)
X_train
text_train[6]
X_train[6, :].nonzero()[1]
X_test = cv.transform(text_test)
from sklearn.svm import LinearSVC
svm = LinearSVC()
svm.fit(X_train, y_train)
svm.score(X_train, y_train)
svm.score(X_test, y_test)
def visualize_coefficients(classifier, feature_names, n_top_features=25):
    # get coefficients with large absolute values 
    coef = classifier.coef_.ravel()
    positive_coefficients = np.argsort(coef)[-n_top_features:]
    negative_coefficients = np.argsort(coef)[:n_top_features]
    interesting_coefficients = np.hstack([negative_coefficients, positive_coefficients])
    # plot them
    plt.figure(figsize=(15, 5))
    colors = ["red" if c < 0 else "blue" for c in coef[interesting_coefficients]]
    plt.bar(np.arange(50), coef[interesting_coefficients], color=colors)
    feature_names = np.array(feature_names)
    plt.xticks(np.arange(1, 51), feature_names[interesting_coefficients], rotation=60, ha="right");
visualize_coefficients(svm, cv.get_feature_names())
Explanation: Working with Text Data
<img src="figures/bag_of_words.svg" width=100%>
End of explanation
# %load solutions/text_pipeline.py
Explanation: Exercises
Create a pipeine using the count vectorizer and SVM (see 07). Train and score using the pipeline.
Vary the n_gram_range in the count vectorizer, visualize the changed coefficients.
Grid search the C in the LinearSVC using the pipeline.
Grid search the C in the LinearSVC together with the n_gram_range (try (1,1), (1, 2), (2, 2))
End of explanation
<END_TASK>