<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 SFrame -- Scalable Dataframe
Powerful unstructured data processing
Step1:  Unpack to extract structure
Step2:  Votes are still crammed in a dictionary. Let's unpack it.
Step3:  Quick data visualization
Step4:  Represent datetime
Step5:  Munge votes and add a new column
Step6:  Filter rows to remove reviews with no votes
Step7:  Classifiction task
Predict which reviews will be voted "funny," based on review text.
First, the labels. Reviews with at least one vote for "funny" is funny.
Step8:  To save time, take just a small subset
Step9:  Create bag-of-words representation of text
Step10:  Create tf-idf representation of the bag of words
Step11:  Create a train-test split
Returns immediately because SFrame operations are lazily evaluated.
Step12:  Train classifiers on bow and tf-idf
Dictionaries are automatically interpreted as sparse features.
Not demonstrated here, but any string/categorical columns are automatically interpreted as sparse features as well.
Step13:  Evaluate on validation set and compare performance
Step14:  Baseline accuracy (what if we classify everything as the majority class)
Percentage of 'funny' reviews
Step15:  Percentage of not funny reviews
<ASSISTANT_TASK:>
Python Code:
reviews = gl.SFrame.read_csv('../data/yelp/yelp_training_set_review.json', header=False)
reviews
reviews[0]
Explanation: SFrame -- Scalable Dataframe
Powerful unstructured data processing: read straight up json
End of explanation
reviews=reviews.unpack('X1','')
reviews
Explanation: Unpack to extract structure
End of explanation
reviews = reviews.unpack('votes', '')
reviews
Explanation: Votes are still crammed in a dictionary. Let's unpack it.
End of explanation
reviews.show()
Explanation: Quick data visualization
End of explanation
reviews['date'] = reviews['date'].str_to_datetime(str_format='%Y-%m-%d')
Explanation: Represent datetime
End of explanation
reviews['total_votes'] = reviews['funny'] + reviews['cool'] + reviews['useful']
reviews
Explanation: Munge votes and add a new column
End of explanation
reviews['total_votes'] > 0
reviews = reviews[reviews['total_votes'] > 0]
reviews
Explanation: Filter rows to remove reviews with no votes
End of explanation
reviews['funny'] = reviews['funny'] > 0
reviews = reviews[['text','funny']]
reviews
Explanation: Classifiction task
Predict which reviews will be voted "funny," based on review text.
First, the labels. Reviews with at least one vote for "funny" is funny.
End of explanation
reviews = reviews[:10000]
Explanation: To save time, take just a small subset
End of explanation
word_delims = ["\r", "\v", "\n", "\f", "\t", " ", 
               '~', '`', '!', '@', '#', '$', '%', '^', '&', '*', '-', '_', '+', '=', 
               ',', '.', ';', ':', '\"', '?', '|', '\\', '/', 
               '<', '>', '(', ')', '[', ']', '{', '}']
reviews['bow'] = gl.text_analytics.count_words(reviews['text'], delimiters=word_delims)
Explanation: Create bag-of-words representation of text
End of explanation
reviews['tf_idf'] = gl.text_analytics.tf_idf(reviews['bow'])
reviews['tf_idf'] = reviews['tf_idf'].apply(lambda x: x['docs'])
reviews
Explanation: Create tf-idf representation of the bag of words
End of explanation
train_sf, test_sf = reviews.random_split(0.8)
Explanation: Create a train-test split
Returns immediately because SFrame operations are lazily evaluated.
End of explanation
m1 = gl.logistic_classifier.create(train_sf, 
                                   'funny', 
                                   features=['bow'], 
                                   validation_set=None, 
                                   feature_rescaling=False)
m2 = gl.logistic_classifier.create(train_sf, 
                                   'funny', 
                                   features=['tf_idf'], 
                                   validation_set=None, 
                                   feature_rescaling=False)
Explanation: Train classifiers on bow and tf-idf
Dictionaries are automatically interpreted as sparse features.
Not demonstrated here, but any string/categorical columns are automatically interpreted as sparse features as well.
End of explanation
m1_res = m1.evaluate(test_sf)
m1_res
m2_res = m2.evaluate(test_sf)
m2_res
Explanation: Evaluate on validation set and compare performance
End of explanation
float(test_sf['funny'].sum())/test_sf.num_rows()
Explanation: Baseline accuracy (what if we classify everything as the majority class)
Percentage of 'funny' reviews:
End of explanation
1.0 - float(test_sf['funny'].sum())/test_sf.num_rows()
Explanation: Percentage of not funny reviews:
End of explanation
<END_TASK>