<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Getting Meta with Big Data Malaysia
Scraping the Big Data Malaysia Facebook group for fun. Profit unlikely.
Hello World
This is an introductory-level notebook demonstrating how to deal with a small, but meaty dataset. Things we will do here include
Step1:  Is it big enough?
Now we have all our data loaded into variable big_data, but can we really say it's Big Data?
Step2:  Wow! So data! Very big!
Seriously though... it's not big. In fact it's rather small. How small is small? Here's a clue...
Step3:  At the time this was written, the file was just about 3MB, and there were fewer than 2k posts... note that excludes comments made on posts, but still, this stuff is small. It is small enough that at no point do we need to do anything clever from a data indexing/caching/storage perspective, so to start we will take the simplistic but often appropriate approach of slicing and dicing our big_data object directly. Later on we'll get into pandas DataFrame objects.
Anyway, size doesn't matter. It's variety that counts.
Fields of gold
Now we know how many elements (rows I guess?) we have, but how much variety do we have in this data? One measure of this may be to look at the number of fields in each of those items
Step4:  Are we missing anything? A good way to sanity check things is to actually inspect the data, so let's look at a random item
Step5:  From that you should be able to sense that we are missing some things - it isn't simply that there are some number of fields that describe each item, because some of those fields have data hierarchies beneath them, for example
Step6:  From that we can see some fields have hierarchies within them, e.g. likes have a list of id dictionaries, which happen to be relatively trivial (names and ids... I wonder why Facebook didn't just post the id and make you look up the name?) but the comment field is a bit more complex, wherein it contains a list of dictionaries with each field potentially being a dictionary of its own, e.g. we can see that the second comment on that post tagged Teuku Faruq
Step7:  Data quality annoyances
Actually I'm not even sure why the comments field is a single entry list. Is that always the case?
Step8:  Apparently that's not always the case, sometimes there are 2 items in the list, let's see what that looks like...
Step9:  Skimming the above it looks as though very long comment threads are split into multiple "pages" in the comments list. This may be an artifact of the paging code in pull_feed.py, which is not ideal. At some point we may fix it there, but for the time being we'll just consider it a data quality inconvenience that we will have to deal with.
Here's a function to work around this annoyance
Step10:  Start plotting things already dammit
Now that we're counting comments, it's natural to ask
Step11:  This sort of adds up intuitively; posts with long comment threads will be rare, though from experience with this forum it does not seem right to conclude that there is a lot of posting going on with no interaction... the community is a bit more engaged than that. 
But since this is Facebook, comments aren't the only way of interacting with a post. There's also the wonderful 'Like'.
Step12:  Note that the above does not include Likes on Comments made on posts; only Likes made on posts themselves are counted.
While this paints the picture of a more engaged community, it still doesn't feel quite right. It seems unusual these days to find a post go by without a Like or two.
I have a hunch that the zero-like posts are skewed a bit to the earlier days of the group. To dig into that we'll need to start playing with timestamps. Personally I prefer to deal with time as UTC epoch seconds, and surprisingly it seems I need to write my own helper function for this.
Step13:  In general it seems my hunch may have been right, but it will be clearer if we plot it.
Step14:  This is looking pretty legit now. We can see that lately there's been a significant uptick in the number of posts, and an uptick in the ratio of posts that receive at least one Like.
As another sanity check, we can revisit the Likes-per-post Histogram, but only include recent posts. While we're at it we might as well do the same for the Comments-per-post Histogram.
<ASSISTANT_TASK:>
Python Code:
# we need this for later:
%matplotlib inline
import json
INPUT_FILE = "all_the_data.json"
with open(INPUT_FILE, "r") as big_data_fd:
	big_data = json.load(big_data_fd)
Explanation: Getting Meta with Big Data Malaysia
Scraping the Big Data Malaysia Facebook group for fun. Profit unlikely.
Hello World
This is an introductory-level notebook demonstrating how to deal with a small, but meaty dataset. Things we will do here include:
Loading a JSON dataset.
Dealing with a minor data quality issue.
Handling timestamps.
Dataset slicing and dicing.
Plotting histograms.
A "follow the data" approach will be taken. This notebook may appear quite long, but a good portion of the length is pretty-printing of raw data which noone is expected to read in entirety, but it's there for one to skim to get an idea of the structure of our data.
Get all the data
This notebook assumes you have already prepared a flattened JSON file into all_the_data.json, which you would have done by:
* Writing your oauth token into oauth_file according to the instructions in pull_feed.py.
* Running python pull_feed.py to pull down the feed pages into the BigDataMyData directory.
* Running python flatten_saved_data.py &gt; all_the_data.json.
End of explanation
print "We have {} posts".format(len(big_data))
Explanation: Is it big enough?
Now we have all our data loaded into variable big_data, but can we really say it's Big Data?
End of explanation
import os
print "The source file is {} bytes. Pathetic.".format(os.stat(INPUT_FILE).st_size)
Explanation: Wow! So data! Very big!
Seriously though... it's not big. In fact it's rather small. How small is small? Here's a clue...
End of explanation
import itertools
all_the_fields = set(itertools.chain.from_iterable(big_data))
print "We have {} different field names:".format(len(all_the_fields))
print all_the_fields
Explanation: At the time this was written, the file was just about 3MB, and there were fewer than 2k posts... note that excludes comments made on posts, but still, this stuff is small. It is small enough that at no point do we need to do anything clever from a data indexing/caching/storage perspective, so to start we will take the simplistic but often appropriate approach of slicing and dicing our big_data object directly. Later on we'll get into pandas DataFrame objects.
Anyway, size doesn't matter. It's variety that counts.
Fields of gold
Now we know how many elements (rows I guess?) we have, but how much variety do we have in this data? One measure of this may be to look at the number of fields in each of those items:
End of explanation
import random
import pprint
# re-run this as much as you like to inspect different items
pprint.pprint(random.choice(big_data))
Explanation: Are we missing anything? A good way to sanity check things is to actually inspect the data, so let's look at a random item:
End of explanation
pprint.pprint(big_data[234])
Explanation: From that you should be able to sense that we are missing some things - it isn't simply that there are some number of fields that describe each item, because some of those fields have data hierarchies beneath them, for example:
End of explanation
pprint.pprint(big_data[234]['comments'][0]['data'][1]['message_tags'])
Explanation: From that we can see some fields have hierarchies within them, e.g. likes have a list of id dictionaries, which happen to be relatively trivial (names and ids... I wonder why Facebook didn't just post the id and make you look up the name?) but the comment field is a bit more complex, wherein it contains a list of dictionaries with each field potentially being a dictionary of its own, e.g. we can see that the second comment on that post tagged Teuku Faruq:
End of explanation
set([len(data['comments']) for data in big_data if 'comments' in data])
Explanation: Data quality annoyances
Actually I'm not even sure why the comments field is a single entry list. Is that always the case?
End of explanation
multi_item_comment_lists = [data['comments'] for data in big_data if ('comments' in data) and (len(data['comments']) > 1)]
print len(multi_item_comment_lists)
pprint.pprint(multi_item_comment_lists[0])
Explanation: Apparently that's not always the case, sometimes there are 2 items in the list, let's see what that looks like...
End of explanation
def flatten_comments_pages(post):
    flattened_comments = []
    for page in post:
        flattened_comments += page['data']
    return flattened_comments
post_comments_paged = multi_item_comment_lists[0]
print "Post has {} comments".format(len(flatten_comments_pages(post_comments_paged)))
Explanation: Skimming the above it looks as though very long comment threads are split into multiple "pages" in the comments list. This may be an artifact of the paging code in pull_feed.py, which is not ideal. At some point we may fix it there, but for the time being we'll just consider it a data quality inconvenience that we will have to deal with.
Here's a function to work around this annoyance:
End of explanation
comments_threads = [data['comments'] for data in big_data if 'comments' in data]
count_of_posts_with_no_comments = len(big_data) - len(comments_threads)
comments_counts = [0] * count_of_posts_with_no_comments
comments_counts += [len(flatten_comments_pages(thread)) for thread in comments_threads]
import matplotlib.pyplot as plt
plt.hist(comments_counts, bins=max(comments_counts))
plt.title("Comments-per-post Histogram")
plt.xlabel("Comments per post")
plt.ylabel("Frequency")
plt.show()
Explanation: Start plotting things already dammit
Now that we're counting comments, it's natural to ask: what does the number-of-comments-per-post distribution look like? 
IMPORTANT NOTE: Beyond this point, we start to "follow the data" as we analyse things, and we do so in a time-relative way (e.g. comparing the last N days of posts to historical data). As Big Data Malaysia is a living breathing group, the data set is a living breathing thing, so things may change, and the conclusions informing the analysis here may suffer logic rot.
End of explanation
likes_threads = [data['likes']['data'] for data in big_data if 'likes' in data]
count_of_posts_with_no_likes = len(big_data) - len(likes_threads)
likes_counts = [0] * count_of_posts_with_no_likes
likes_counts += [len(thread) for thread in likes_threads]
plt.hist(likes_counts, bins=max(likes_counts))
plt.title("Likes-per-post Histogram")
plt.xlabel("Likes per post")
plt.ylabel("Frequency")
plt.show()
Explanation: This sort of adds up intuitively; posts with long comment threads will be rare, though from experience with this forum it does not seem right to conclude that there is a lot of posting going on with no interaction... the community is a bit more engaged than that. 
But since this is Facebook, comments aren't the only way of interacting with a post. There's also the wonderful 'Like'.
End of explanation
import datetime
import dateutil
import pytz
def epoch_utc_s(date_string):
    dt_local = dateutil.parser.parse(str(date_string))
    dt_utc = dt_local.astimezone(pytz.utc)
    nineteenseventy = datetime.datetime(1970,1,1)
    epoch_utc = dt_utc.replace(tzinfo=None) - nineteenseventy
    return int(epoch_utc.total_seconds())
posts_without_likes = [data for data in big_data if 'likes' not in data]
posts_with_likes = [data for data in big_data if 'likes' in data]
timestamps_of_posts_without_likes = [epoch_utc_s(post['created_time']) for post in posts_without_likes]
timestamps_of_posts_with_likes = [epoch_utc_s(post['created_time']) for post in posts_with_likes]
import numpy
median_epoch_liked = int(numpy.median(timestamps_of_posts_with_likes))
median_epoch_non_liked = int(numpy.median(timestamps_of_posts_without_likes))
print "Median timestamp of posts without likes: {} ({})".format(datetime.datetime.fromtimestamp(median_epoch_non_liked),
                                                                median_epoch_non_liked)
print "Median timestamp of posts with likes: {} ({})".format(datetime.datetime.fromtimestamp(median_epoch_liked),
                                                             median_epoch_liked)
Explanation: Note that the above does not include Likes on Comments made on posts; only Likes made on posts themselves are counted.
While this paints the picture of a more engaged community, it still doesn't feel quite right. It seems unusual these days to find a post go by without a Like or two.
I have a hunch that the zero-like posts are skewed a bit to the earlier days of the group. To dig into that we'll need to start playing with timestamps. Personally I prefer to deal with time as UTC epoch seconds, and surprisingly it seems I need to write my own helper function for this.
End of explanation
plt.hist(timestamps_of_posts_without_likes, alpha=0.5, label='non-Liked posts')
plt.hist(timestamps_of_posts_with_likes, alpha=0.5, label='Liked posts')
plt.title("Liked vs non-Liked posts")
plt.xlabel("Time (epoch UTC s)")
plt.ylabel("Count of posts")
plt.legend(loc='upper left')
plt.show()
Explanation: In general it seems my hunch may have been right, but it will be clearer if we plot it.
End of explanation
def less_than_n_days_ago(date_string, n):
    query_date = epoch_utc_s(date_string)
    today_a_year_ago = epoch_utc_s(datetime.datetime.now(pytz.utc) - datetime.timedelta(days=n))
    return query_date > today_a_year_ago
# try changing this variable then re-running this cell...
days_ago = 30
# create a slice of our big_data containing only posts created n days ago
recent_data = [data for data in big_data if less_than_n_days_ago(data['created_time'], days_ago)]
# plot the Likes-per-post Histogram for recent_data
recent_likes_threads = [data['likes']['data'] for data in recent_data if 'likes' in data]
recent_count_of_posts_with_no_likes = len(recent_data) - len(recent_likes_threads)
recent_likes_counts = [0] * recent_count_of_posts_with_no_likes
recent_likes_counts += [len(thread) for thread in recent_likes_threads]
plt.hist(recent_likes_counts, bins=max(recent_likes_counts))
plt.title("Likes-per-post Histogram (last {} days)".format(days_ago))
plt.xlabel("Likes per post")
plt.ylabel("Frequency")
plt.show()
# plot the Comment-per-post Histogram for recent_data
recent_comments_threads = [data['comments'] for data in recent_data if 'comments' in data]
recent_count_of_posts_with_no_comments = len(recent_data) - len(comments_threads)
recent_comments_counts = [0] * recent_count_of_posts_with_no_comments
recent_comments_counts += [len(flatten_comments_pages(thread)) for thread in recent_comments_threads]
plt.hist(recent_comments_counts, bins=max(recent_comments_counts))
plt.title("Comments-per-post Histogram (last {} days)".format(days_ago))
plt.xlabel("Comments per post")
plt.ylabel("Frequency")
plt.show()
Explanation: This is looking pretty legit now. We can see that lately there's been a significant uptick in the number of posts, and an uptick in the ratio of posts that receive at least one Like.
As another sanity check, we can revisit the Likes-per-post Histogram, but only include recent posts. While we're at it we might as well do the same for the Comments-per-post Histogram.
End of explanation
<END_TASK>