<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 I've seen a couple of nice kernels here, but no one explained the importance of a morphological pre-processing of the data. So I decided to compare two approaches of a morphological normalization
Step1:  A visible example of how do they work
Step2:  So, what approach will be better for the given task? Let's see.
First of all, we need to load modules for linear algebra and data analysis as well as gensim (for training a Word2Vec, a classic algorithm for obtaining word embeddings). We also need some stuff from scikit-learn to teach and evaluate the classifier and pyplot to draw plots. seaborn will make the plots more beautiful.
Step3:  And a little bit more of the linguistic tools! We will use a tokenization( breaking a stream of text up into meaningful elements called tokens, for instance, words) and a stop-word dictionary for English.
Step4:  And check if the .csv-files with the data are okay.
Step5:  So let's write some code. First of all, let's train a Word2Vec model. We will use the training set as a training corpus (Previously I used the test set, but it uses much more memory while the model trained on it has the same efficiency; thanks to @Gian12 for the notion). This set contains some NaN values, but we can just drop them since in our task their lack is not meaningful.
Step6:  Let's make a list of sentences by merging the questions.
Step7:  Okay, now we are up to the key method of preprocessing comparation.  It provides lemmatization or stemming depending on the given flag.
Step8:  And then we can make two different corpora to train the model
Step9:  Now let's train the models. I've pre-defined these hyperparameters since models on them have the best performance. You can also try to play with them yourself.
Step10:  Let's check the result of one of the models.
Step11:  Great! The most similar words seem to be pretty meaningful. So, we have three trained models, we can encode the text data with the vectors - let's make some experiments! Let's make data sets from the loaded data frame. I take a chunk of the traning data because the run of the script on the full data takes too much time.
Step12:  A little bit modified preprocess. Now it returns only words which model's vocabulary contains.
Step13:  This method will help to obtaining a bag of means by vectorising the messages.
Step14:  And now we can obtain the features matrices.
Step15:  That's almost all! Now we can train the classifier and evaluate it's performance. It's better to use a metric classifier because we are performing operations in the vector space, so I choose a Logistic Regression. But of course you can try a something different and see what can change. 
I also use cross-validation to train and to evaluate on the same data set.
Step16:  So, the lemmatized model outperformed the "clear" model! And the stemmed model showed the worst result. Why does it happen?
Well, any morphological pre-processing of the training data for the model reduces the amount of information that model can obtain from the corpus. Some of the information, like the difference in morphological roots of the same words, seems to be not necessary, so it is better to remove it. This removal is a mush-have in synthetic languages (languages with high morpheme-per-word ratio, like Russian), and, as we can see, it is also pretty helpful in our task. 
The same thing about stemming. Stemming further reduces the amount of information, making one stem for the different word forms. Sometimes this is helpful, but sometimes this can bring noise to the model since some stems of the different words can be ambiguous, and the model can't be able to separate "playstation" and, say, "play".
In other words, there is no silver bullet, and you should always check various option of pre-processing if you want to reach the best performance. However, lemmatisation nine times out of ten will increase the performance of your model.
However, the logarithmic loss of my approach is not very high, but you can use this notebook as a baseline and try to beat it's score yourself! Just download it and uncomment the commented strings (because Kaggle doesn't allow to use so much memory)
<ASSISTANT_TASK:>
Python Code:
from nltk.stem.wordnet import WordNetLemmatizer
from nltk.stem import LancasterStemmer
stemmer =  LancasterStemmer()
lemmer = WordNetLemmatizer()
Explanation: I've seen a couple of nice kernels here, but no one explained the importance of a morphological pre-processing of the data. So I decided to compare two approaches of a morphological normalization: stemming and lemmatization. Both of them reduce the word to the regularized form, but a stemming reduces the word to the word stem, and a lemmatization reduces the word to it's morphological root with the help of dictionary lookup. 
I evaluate the efficiency of these approaches by comparison their performance with the naive Bag of Means method: every word is encoded with a word embedding vector, and then the common vector of two messages is computed as a mean vector of these vectors. Some of the researches proved that such approach can be a very strong baseline (Faruqui et al., 2014; Yu et al., 2014; Gershman and Tenenbaum, 2015; Kenter and de Rijke, 2015). Then I use obtained vectors as feature vectors to train the classifiers.
I will also make a comparison with a default approach (no morphological pre-processing). 
Okay, let's load NLTK and try to implement these two approaches with a Lancaster Stemmer (one of the most popular stemming algorithms) and a WordNet Lemmatizer (based on WordNet’s built-in morphy function):
End of explanation
print(stemmer.stem('dictionaries'))
print(lemmer.lemmatize('dictionaries'))
Explanation: A visible example of how do they work:
End of explanation
from gensim import models
import numpy as np
from pandas import DataFrame, Series
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import learning_curve
from sklearn.model_selection import ShuffleSplit
from gensim import models
import matplotlib.pyplot as plt
import seaborn
Explanation: So, what approach will be better for the given task? Let's see.
First of all, we need to load modules for linear algebra and data analysis as well as gensim (for training a Word2Vec, a classic algorithm for obtaining word embeddings). We also need some stuff from scikit-learn to teach and evaluate the classifier and pyplot to draw plots. seaborn will make the plots more beautiful.
End of explanation
from nltk.corpus import stopwords
from nltk.tokenize import wordpunct_tokenize, RegexpTokenizer
stop = stopwords.words('english')
alpha_tokenizer = RegexpTokenizer('[A-Za-z]\w+')
Explanation: And a little bit more of the linguistic tools! We will use a tokenization( breaking a stream of text up into meaningful elements called tokens, for instance, words) and a stop-word dictionary for English.
End of explanation
from subprocess import check_output
print(check_output(["ls", "../input"]).decode("utf8"))
Explanation: And check if the .csv-files with the data are okay.
End of explanation
df_train = DataFrame.from_csv('../input/train.csv').dropna()
Explanation: So let's write some code. First of all, let's train a Word2Vec model. We will use the training set as a training corpus (Previously I used the test set, but it uses much more memory while the model trained on it has the same efficiency; thanks to @Gian12 for the notion). This set contains some NaN values, but we can just drop them since in our task their lack is not meaningful.
End of explanation
texts = np.concatenate([df_train.question1.values, df_train.question2.values])
Explanation: Let's make a list of sentences by merging the questions.
End of explanation
def process_sent(words, lemmatize=False, stem=False):
    words = words.lower()
    tokens = alpha_tokenizer.tokenize(words)
    for index, word in enumerate(tokens):
        if lemmatize:
            tokens[index] = lemmer.lemmatize(word)
        elif stem:
            tokens[index] = stemmer.stem(word)
        else:
            tokens[index] = word
    return tokens
Explanation: Okay, now we are up to the key method of preprocessing comparation.  It provides lemmatization or stemming depending on the given flag.
End of explanation
corpus_lemmatized = [process_sent(sent, lemmatize=True, stem=False) for sent in texts]
corpus_stemmed = [process_sent(sent, lemmatize=False, stem=True) for sent in texts]
corpus = [process_sent(sent) for sent in texts]
Explanation: And then we can make two different corpora to train the model: stemmed corpus and lemmatized corpus. We will also make a "clean" corpus for sure.
End of explanation
VECTOR_SIZE = 100
min_count = 10
size = VECTOR_SIZE
window = 10
model_lemmatized = models.Word2Vec(corpus_lemmatized, min_count=min_count, 
                                   size=size, window=window)
model_stemmed = models.Word2Vec(corpus_stemmed, min_count=min_count, 
                                size=size, window=window)
model = models.Word2Vec(corpus, min_count=min_count, 
                                size=size, window=window)
Explanation: Now let's train the models. I've pre-defined these hyperparameters since models on them have the best performance. You can also try to play with them yourself.
End of explanation
model_lemmatized.most_similar('playstation')
Explanation: Let's check the result of one of the models.
End of explanation
q1 = df_train.question1.values[200000:]
q2 = df_train.question2.values[200000:]
Y = np.array(df_train.is_duplicate.values)[200000:]
Explanation: Great! The most similar words seem to be pretty meaningful. So, we have three trained models, we can encode the text data with the vectors - let's make some experiments! Let's make data sets from the loaded data frame. I take a chunk of the traning data because the run of the script on the full data takes too much time.
End of explanation
def preprocess_check(words, lemmatize=False, stem=False):
    words = words.lower()
    tokens = alpha_tokenizer.tokenize(words)
    model_tokens = []
    for index, word in enumerate(tokens):
        if lemmatize:
            lem_word = lemmer.lemmatize(word)
            if lem_word in model_lemmatized.wv.vocab:
                model_tokens.append(lem_word)
        elif stem:
            stem_word = stemmer.stem(word)
            if stem_word in model_stemmed.wv.vocab:
                model_tokens.append(stem_word)
        else:
            if word in model.wv.vocab:
                model_tokens.append(word)
    return model_tokens
Explanation: A little bit modified preprocess. Now it returns only words which model's vocabulary contains.
End of explanation
old_err_state = np.seterr(all='raise')
def vectorize(words, words_2, model, num_features, lemmatize=False, stem=False):
    features = np.zeros((num_features), dtype='float32')
    words_amount = 0
    
    words = preprocess_check(words, lemmatize, stem)
    words_2 = preprocess_check(words_2, lemmatize, stem)
    for word in words: 
            words_amount = words_amount + 1
            features = np.add(features, model[word])
    for word in words_2: 
            words_amount = words_amount + 1
            features = np.add(features, model[word])
    try:
        features = np.divide(features, words_amount)
    except FloatingPointError:
        features = np.zeros(num_features, dtype='float32')
    return features
Explanation: This method will help to obtaining a bag of means by vectorising the messages.
End of explanation
X_lem = []
for index, sentence in enumerate(q1):
    X_lem.append(vectorize(sentence, q2[index], model_lemmatized, VECTOR_SIZE, True, False))
X_lem = np.array(X_lem)
X_stem = []
for index, sentence in enumerate(q1):
    X_stem.append(vectorize(sentence, q2[index], model_stemmed, VECTOR_SIZE, False, True))
X_stem = np.array(X_stem)
X = []
for index, sentence in enumerate(q1):
    X.append(vectorize(sentence, q2[index], model, VECTOR_SIZE))
X = np.array(X)
Explanation: And now we can obtain the features matrices.
End of explanation
results = []
title_font = {'size':'10', 'color':'black', 'weight':'normal',
                  'verticalalignment':'bottom'} 
axis_font = {'size':'10'}
plt.figure(figsize=(10, 5))
plt.xlabel('Training examples', **axis_font)
plt.ylabel('Accuracy',  **axis_font)
plt.tick_params(labelsize=10)
for X_set, name, lstyle in [(X_lem, 'Lemmatizaton', 'dotted'),
            (X_stem, 'Stemming', 'dashed'),
            (X, 'Default', 'dashdot'),
            ]:
    estimator = LogisticRegression(C = 1)
    cv = ShuffleSplit(n_splits=6, test_size=0.01, random_state=0)
    train_sizes=np.linspace(0.01, 0.99, 6)
    train_sizes, train_scores, test_scores = learning_curve(estimator, X_set, Y, cv=cv, train_sizes=train_sizes)
    train_scores_mean = np.mean(train_scores, axis=1)
    results.append({'preprocessing' : name, 'score' : train_scores_mean[-1]})
    plt.plot(train_sizes, train_scores_mean, label=name, linewidth=5, linestyle=lstyle)
   
plt.legend(loc='best')
Explanation: That's almost all! Now we can train the classifier and evaluate it's performance. It's better to use a metric classifier because we are performing operations in the vector space, so I choose a Logistic Regression. But of course you can try a something different and see what can change. 
I also use cross-validation to train and to evaluate on the same data set.
End of explanation
clf = LogisticRegression(C = 1)
clf.fit(X, Y)
#df_test = DataFrame.from_csv('../input/test.csv').fillna('None')
q1 = df_train.question1.values[:100]
q2 = df_train.question2.values[:100]
#q1 = df_test.question1.values
#q2 = df_test.question2.values
X_test = []
for index, sentence in enumerate(q1):
    X_test.append(vectorize(sentence, q2[index], model, VECTOR_SIZE))
X_test = np.array(X_test)
result = clf.predict(X_test)
sub = DataFrame()
sub['is_duplicate'] = result
sub.to_csv('submission.csv', index=False)
Explanation: So, the lemmatized model outperformed the "clear" model! And the stemmed model showed the worst result. Why does it happen?
Well, any morphological pre-processing of the training data for the model reduces the amount of information that model can obtain from the corpus. Some of the information, like the difference in morphological roots of the same words, seems to be not necessary, so it is better to remove it. This removal is a mush-have in synthetic languages (languages with high morpheme-per-word ratio, like Russian), and, as we can see, it is also pretty helpful in our task. 
The same thing about stemming. Stemming further reduces the amount of information, making one stem for the different word forms. Sometimes this is helpful, but sometimes this can bring noise to the model since some stems of the different words can be ambiguous, and the model can't be able to separate "playstation" and, say, "play".
In other words, there is no silver bullet, and you should always check various option of pre-processing if you want to reach the best performance. However, lemmatisation nine times out of ten will increase the performance of your model.
However, the logarithmic loss of my approach is not very high, but you can use this notebook as a baseline and try to beat it's score yourself! Just download it and uncomment the commented strings (because Kaggle doesn't allow to use so much memory)
End of explanation
<END_TASK>