<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Natural Language Processing (NLP)
Overview
corpus - collection of texts
lexicon - collection of words (or sequences) we put into our index
bag-of-words - take each word and count how many times it appears
n-gram - count how often each set of n words appears
Tokenization
break text into tokens based on characters, words, sentences, etc.
Tokenizing Sentences
Step1:  Tokenizing Social Media
Step2:  N-grams
Step3:  Stop-words
Step4:  Sentiment
VADER (Valence Aware Dictionary for sEntiment Reasoning)
<ASSISTANT_TASK:>
Python Code:
from nltk.tokenize import TreebankWordTokenizer
sentence = "How does nltk tokenize this sentence?"
tokenizer = TreebankWordTokenizer()
tokenizer.tokenize(sentence)
Explanation: Natural Language Processing (NLP)
Overview
corpus - collection of texts
lexicon - collection of words (or sequences) we put into our index
bag-of-words - take each word and count how many times it appears
n-gram - count how often each set of n words appears
Tokenization
break text into tokens based on characters, words, sentences, etc.
Tokenizing Sentences
End of explanation
from nltk.tokenize.casual import casual_tokenize
tweet = "OMG @twitterguy that was sooooooooo cool :D :D :D!!!!"
print(casual_tokenize(tweet))
casual_tokenize(tweet, reduce_len=True, strip_handles=True)
Explanation: Tokenizing Social Media
End of explanation
from nltk.util import ngrams
list(ngrams(sentence.split(), 2))
Explanation: N-grams
End of explanation
import nltk
nltk.download("stopwords")
stop_words = nltk.corpus.stopwords.words("english")
stop_words[:10]
Explanation: Stop-words
End of explanation
nltk.download('vader_lexicon')
from nltk.sentiment.vader import SentimentIntensityAnalyzer
sia = SentimentIntensityAnalyzer()
negative_sentence = "This is the worst!!! I hate it so much :( :("
sia.polarity_scores(negative_sentence)
sia.polarity_scores(tweet)
Explanation: Sentiment
VADER (Valence Aware Dictionary for sEntiment Reasoning)
End of explanation
<END_TASK>