<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 There are a thousand movie reviews for both
positive and
negetive
reviews
Step1:  Now I need to store it as
python
documents = [
    ('pos', ['good', 'awesome', ....]), 
    ('neg', ['ridiculous', 'horrible', ...])
]
OR 
Storing it in a dictionary would also be a better idea, will try out with both
python
documents = {
    'pos'
Step2:  Getting the list of all words to store the most frequently occuring ones
Step3:  Making a frequency distribution of the words
Step4:  will train only for the first 5000 top words in the list
Step5:  Finding these feature words in documents, making our function would ease it out!
Step6:  What the below one does is, before hand we had only words and its category. But not we have the feature set (along with a boolean value of whether it is one of the most frequently used words or not)of the same word and then the category.
Step7:  Training the classifier
Step8:  We won't be telling the machine the category i.e. whether the document is a postive one or a negative one. We ask it to tell that to us. Then we compare it to the known category that we have and calculate how accurate it is.
Naive bayes algorithm
It states that
\begin{equation}
posterior = \frac{PriorOccurences \times likelihood}{CurrentEvidence}
\end{equation}
Here posterior is likelihood of occurence
<ASSISTANT_TASK:>
Python Code:
movie_reviews.categories()
Explanation: There are a thousand movie reviews for both
positive and
negetive
reviews
End of explanation
documents = [(list(word for word in movie_reviews.words(fileid) if word not in stop_words), category)
            for category in movie_reviews.categories()
             for fileid in movie_reviews.fileids(category)
            ]
random.shuffle(documents)
Explanation: Now I need to store it as
python
documents = [
    ('pos', ['good', 'awesome', ....]), 
    ('neg', ['ridiculous', 'horrible', ...])
]
OR 
Storing it in a dictionary would also be a better idea, will try out with both
python
documents = {
    'pos': ['good', 'awesome', ....],
    'neg': ['ridiculous', 'horrible', ...]
}
End of explanation
all_words = []
for w in movie_reviews.words():
    all_words.append(w.lower())
Explanation: Getting the list of all words to store the most frequently occuring ones
End of explanation
all_words = nltk.FreqDist(all_words)
all_words.most_common(20)
all_words["hate"]  ## counting the occurences of a single word
Explanation: Making a frequency distribution of the words
End of explanation
feature_words = list(all_words.keys())[:5000]
Explanation: will train only for the first 5000 top words in the list
End of explanation
def find_features(document):
    words = set(document)
    feature = {}
    for w in feature_words:
        feature[w] = (w in words)
    return feature
Explanation: Finding these feature words in documents, making our function would ease it out!
End of explanation
feature_sets = [(find_features(rev), category) for (rev, category) in documents]
feature_sets[:1]
Explanation: What the below one does is, before hand we had only words and its category. But not we have the feature set (along with a boolean value of whether it is one of the most frequently used words or not)of the same word and then the category.
End of explanation
training_set = feature_sets[:1900]
testing_set = feature_sets[1900:]
Explanation: Training the classifier
End of explanation
## TO-DO: To build own naive bais algorithm
# classifier = nltk.NaiveBayesClassifier.train(training_set)
## saving the classifier
# save_classifier = open("naive_bayes.pickle", "wb")
# pickle.dump(classifier, save_classifier)
# save_classifier.close()
## Now that the picke is saved we will use that.
## Using the pickle file now 
pickle_classifier = open("naive_bayes.pickle", "rb")
classifier = pickle.load(pickle_classifier)
pickle_classifier.close()
## Testing it's accuracy
print("Naive bayes classifier accuracy percentage : ", (nltk.classify.accuracy(classifier, testing_set))*100)
classifier.show_most_informative_features(20)
Explanation: We won't be telling the machine the category i.e. whether the document is a postive one or a negative one. We ask it to tell that to us. Then we compare it to the known category that we have and calculate how accurate it is.
Naive bayes algorithm
It states that
\begin{equation}
posterior = \frac{PriorOccurences \times likelihood}{CurrentEvidence}
\end{equation}
Here posterior is likelihood of occurence
End of explanation
<END_TASK>