<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 To-Do
Step1:  geo
Step2:  1. 'city'
Step3:  we only consider Champaign and Urbana as our target in this dataset
Step4:  2. 'is_open'
Step5:  we only consider those with is_open == 1
Step6:  3. 'Hour'
Step7:  Drop all records with missing 'Hour'
Step8:  4. 'postal_code'
Step9:  5. 'review_count'
Step10:  6. 'categories'
Step11:  Here we should set the list of categories we are interested.
7. 'Neighborhood'
Step12:  All 'Neighborhood' == None
8. 'Attribute'
First of all, remove records with None 'Attribute'
Step13:  We need a threshold to decide which attributes we use to build model.
Step14:  There are 2 kinds of attributes
Step15:  If we would like to build a binary classification model, here we could set threshold 'star' as '<4.0'
Step16:  output df_uc_open
<ASSISTANT_TASK:>
Python Code:
import seaborn
import pandas as pd
import pylab as pl
import yaml
%pylab inline
df = pd.read_pickle("../yelp-challenge/data_urbana_champaign/business_urbana_champaign.p")
df.reset_index(drop=True, inplace=True)
print df.shape
print df.columns.values
len(df.business_id.unique())
Explanation: To-Do:
* category [list]
* attribute 
    * transform 'attribute' to col
    * threshold (value/proportion) -> missing value
* hours
End of explanation
df.head(2)
Explanation: geo: u'address', u'latitude',  u'longitude'
No None geo-data <br/>
No None name
End of explanation
print df.city.unique(), '\n', len(df.city.unique())
city_count = df.city.groupby(df.city).apply(lambda x:x.count())
uc_sum = city_count[city_count.keys().isin(['Champaign', 'Urbana'])].sum()
100.0 * uc_sum / city_count.sum()
Explanation: 1. 'city'
End of explanation
df_uc = df[df.city.isin(['Champaign', 'Urbana'])]
df_uc.shape
Explanation: we only consider Champaign and Urbana as our target in this dataset
End of explanation
print "{}% ({}/{}) business_id(s) are open".format(100.0 * sum(df_uc.is_open == 1) / len(df_uc), sum(df_uc.is_open == 1), len(df_uc))
Explanation: 2. 'is_open'
End of explanation
df_uc_open = df_uc[df_uc.is_open == 1]
df_uc_open.reset_index(drop=True, inplace=True)
df_uc_open.shape
Explanation: we only consider those with is_open == 1
End of explanation
100.0 * df_uc_open.hours.dropna().shape[0] / df_uc_open.shape[0]
Explanation: 3. 'Hour'
End of explanation
df_uc_open = df_uc_open.dropna(subset=['hours'])
df_uc_open.reset_index(drop=True, inplace=True)
df_uc_open.shape
Explanation: Drop all records with missing 'Hour'
End of explanation
len(df_uc_open.postal_code.unique())
Explanation: 4. 'postal_code'
End of explanation
df.review_count.describe()
df_uc_open.review_count.describe()
df_uc_open.ix[df_uc_open.review_count.argmax()]
Explanation: 5. 'review_count'
End of explanation
cat = {}
none_num = 0
for i in df_uc_open.categories:
    if not i:
        none_num += 1
        continue
    for j in i:
        cat[j] = cat.get(j,0) + 1
none_num
sorted(cat.iteritems(), key=lambda x: x[1], reverse=True)
Explanation: 6. 'categories'
End of explanation
df_uc_open.neighborhood.unique()
Explanation: Here we should set the list of categories we are interested.
7. 'Neighborhood'
End of explanation
df_uc_open = df_uc_open.dropna(subset=['attributes'])
df_uc_open.reset_index(drop=True, inplace=True)
df_uc_open.shape
# 814 -> ?
# df_uc_open.attributes
# import yaml
att_count = {}
for i in df_uc_open.attributes:
    data = yaml.load('['+','.join(i)+']')
    for d in data:
        att_count[d.keys()[0]] = att_count.get(d.keys()[0], 0) + 1
sorted(att_count.iteritems(), key=lambda x: x[1], reverse=True)
df_uc_open.shape[0]
Explanation: All 'Neighborhood' == None
8. 'Attribute'
First of all, remove records with None 'Attribute'
End of explanation
data = yaml.load('['+','.join(df_uc_open.attributes[452])+']')
data
Explanation: We need a threshold to decide which attributes we use to build model.
End of explanation
pl.hist(df_uc_open.stars)
df_uc_open.stars.describe()
s = 3.9
1.0 * sum(df_uc_open.stars > s) / 729
Explanation: There are 2 kinds of attributes:
    * Binary 
        * 0 or 1
        * True or False
    * Multiple
        * dict eg. 'GoodForMeal'
            * if ALL False, ?
            * if more than ONE True, ? eg. 'BestNights'
        * multi-value eg. 'RestaurantsPriceRange2' 1,2,3... or 'Alcohol'
    * Anything Else
* How to deal with missing values?
9. 'Stars'
End of explanation
df_uc_open.stars.value_counts() /729
pl.hist?
c = ['Champaign', 'Urbana']
print c[0], '\n',df_uc_open[df_uc_open.city == c[0]].stars.describe()
pl.hist(df_uc_open[df_uc_open.city == c[0]].stars, normed=True)
print c[1], '\n',df_uc_open[df_uc_open.city == c[1]].stars.describe()
pl.hist(df_uc_open[df_uc_open.city == c[1]].stars, normed=True)
Explanation: If we would like to build a binary classification model, here we could set threshold 'star' as '<4.0'
End of explanation
print df.shape, df_uc.shape, df_uc_open.shape
df_uc_open.to_pickle("UC01_df_uc_open.p")
Explanation: output df_uc_open
End of explanation
<END_TASK>