<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 1. The data
Step1:  Inspect the data visually
Step2:  A small bit of data cleaning
Step3:  2. Convert to a TimeSeries object
Format the timestamp
Step4:  Convert from SFrame to TimeSeries
Step5:  3. Basic TimeSeries operations
Many operations are just like SFrame
Step6:  Some operations are little different
Column subsets retain the time index
Step7:  Some operations are unique to TimeSeries
Row slicing with a datetime interval
Step8:  Working with the time index
Step9:  We can always go back to an SFrame
Step10:  4. Appending more data
Read in new data and preprocess
Step11:  Get the union of the two datasets
If the indexes don't overlap, this is equivalent to SFrame.append.
If there is an overlap, TimeSeries.union enforces time order.
Step12:  5. Grouping observations by value
Step13:  The group_info SFrame tells us what the group names are and how many observations are in each group.
Step14:  The get_group method lets us isolate just the observations for any group.
Step15:  6. Grouping observations by time component
The date_part of a TimeSeries object let's us specify components of a datetime value.
Step16:  7. Resampling
Four things happen with the resample method
Step17:  8. Setting up the next notebooks
For the modeling notebook, we'll use the global earthquake data, downsampled to daily statistics.
For the anomaly detection notebook, we'll use just the Oklahoma data, downsampled to daily statistics.
<ASSISTANT_TASK:>
Python Code:
import datetime as dt
import graphlab as gl
sf = gl.SFrame.read_csv('raw_data/global_earthquakes.csv', verbose=False)
Explanation: 1. The data: global earthquake events
I pulled the data from the USGS Advanced National Seismic System Comprehensive Earthquake Catalog.
- http://earthquake.usgs.gov/data/
- http://earthquake.usgs.gov/earthquakes/search/
- Global seismic events
- Magnitude 2.5+
- Real-time data starts February 2, 2013.
- Main pull goes through March 15, 2016.
- Update pull goes from March 16 - March 22, 2016.
Load the data
End of explanation
sf.show()
useful_columns = ['time', 'latitude', 'longitude', 'mag', 'type', 'location']
sf = sf[useful_columns]
mask = sf['type'] == 'nuclear explosion'
sf[mask]
Explanation: Inspect the data visually
End of explanation
mask = sf['type'] == 'earthquake'
sf = sf[mask]
sf = sf.remove_column('type')
print "Number of earthquake events:", sf.num_rows()
Explanation: A small bit of data cleaning
End of explanation
sf['time'] = sf['time'].str_to_datetime(str_format='%Y-%m-%dT%H:%M:%s%ZP')
sf['time'] = sf['time'].apply(lambda x: x.replace(tzinfo=None))
Explanation: 2. Convert to a TimeSeries object
Format the timestamp
End of explanation
quakes = gl.TimeSeries(sf, index='time')
Explanation: Convert from SFrame to TimeSeries
End of explanation
quakes.print_rows(3)
quakes[4:7].print_rows()
Explanation: 3. Basic TimeSeries operations
Many operations are just like SFrame
End of explanation
quakes[['latitude', 'longitude']].print_rows(3)
Explanation: Some operations are little different
Column subsets retain the time index
End of explanation
start = dt.datetime(2014, 5, 1)
end = dt.datetime(2014, 5, 2)
quakes.slice(start, end).print_rows(3)
Explanation: Some operations are unique to TimeSeries
Row slicing with a datetime interval
End of explanation
print "Earliest timestamp:", quakes.min_time
print "Latest timestamp:", quakes.max_time
print "Timestamp range:", quakes.range
print "Index column:", quakes.index_col_name
print "Value columns:", quakes.value_col_names
print quakes.index[:3]
big_one = quakes.argmax('mag')
quakes[big_one]
Explanation: Working with the time index
End of explanation
sf2 = quakes.to_sframe()
print type(sf2)
Explanation: We can always go back to an SFrame
End of explanation
sf_recent = gl.SFrame.read_csv('raw_data/global_earthquakes_recent.csv', verbose=False)
# Trim away the columns we're not interested in.
sf_recent = sf_recent[useful_columns]
# Remove any non-earthquake events.
mask = sf_recent['type'] == 'earthquake'
sf_recent = sf_recent[mask]
sf_recent = sf_recent.remove_column('type')
# Convert the timestamp to a `datetime` type.
sf_recent['time'] = sf_recent['time'].str_to_datetime(str_format='%Y-%m-%dT%H:%M:%s%ZP')
sf_recent['time'] = sf_recent['time'].apply(lambda x: x.replace(tzinfo=None))
# Convert to a `TimeSeries` object.
recent_quakes = gl.TimeSeries(sf_recent, index='time')
recent_quakes.print_rows(3)
Explanation: 4. Appending more data
Read in new data and preprocess
End of explanation
all_quakes = quakes.union(recent_quakes)
print all_quakes.min_time
print all_quakes.max_time
Explanation: Get the union of the two datasets
If the indexes don't overlap, this is equivalent to SFrame.append.
If there is an overlap, TimeSeries.union enforces time order.
End of explanation
grp = quakes.group('location')
print grp
Explanation: 5. Grouping observations by value
End of explanation
grp.group_info().topk('group_size', k=8)
Explanation: The group_info SFrame tells us what the group names are and how many observations are in each group.
End of explanation
oklahoma_quakes = grp.get_group('Oklahoma')
oklahoma_quakes.print_rows(3)
Explanation: The get_group method lets us isolate just the observations for any group.
End of explanation
grp = quakes.group(quakes.date_part.HOUR)
hour_counts = grp.group_info()
hour_counts.print_rows(5)
import matplotlib.pyplot as plt
%matplotlib notebook
plt.style.use('ggplot')
fig, ax = plt.subplots()
ax.bar(hour_counts['time.hour'], hour_counts['group_size'], color='dodgerblue')
ax.set_xlabel('Hour of the day')
ax.set_ylabel('Number of earthquakes')
fig.show()
Explanation: 6. Grouping observations by time component
The date_part of a TimeSeries object let's us specify components of a datetime value.
End of explanation
import graphlab.aggregate as agg
daily_stats = quakes.resample(period=dt.timedelta(days=1),
                        upsample_method='none',
                        downsample_method={'count': agg.COUNT('latitude'),
                                           'avg_mag': agg.MEAN('mag'),
                                           'max_mag': agg.MAX('mag')})
daily_stats['count'] = daily_stats['count'].fillna(0)
daily_stats.print_rows(5)
Explanation: 7. Resampling
Four things happen with the resample method:
1. A new time index is created with uniformly spaced intervals.
2. Each observation is mapped to an interval.
3. Downsampling: aggregate statistics are computed within each interval.
4. Upsampling: values are interpolated for empty intervals.
End of explanation
def compute_daily_stats(data):
    daily = data.resample(period=dt.timedelta(days=1),
                                upsample_method='none',
                                downsample_method={'count': agg.COUNT('latitude'),
                                                   'avg_mag': agg.MEAN('mag'),
                                                   'max_mag': agg.MAX('mag')})
    daily['count'] = daily['count'].fillna(0)
    return daily
# Save the daily counts and recent daily counts.
daily_stats.save('working_data/global_daily_stats.ts')
compute_daily_stats(recent_quakes).save('working_data/global_daily_update.ts')
# Filter just the Oklahoma data from the recent events.
grp = recent_quakes.group('location')
recent_oklahoma_quakes = grp.get_group('Oklahoma')
# Compute daily stats for the Oklahoma quake events.
compute_daily_stats(oklahoma_quakes).save('working_data/ok_daily_stats.ts')
compute_daily_stats(recent_oklahoma_quakes).save('working_data/ok_daily_update.ts')
Explanation: 8. Setting up the next notebooks
For the modeling notebook, we'll use the global earthquake data, downsampled to daily statistics.
For the anomaly detection notebook, we'll use just the Oklahoma data, downsampled to daily statistics.
End of explanation
<END_TASK>