<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Sale price distribution
First step is to look at the target sale price for the training data set, i.e. the column we're trying to predict.
Step1:  The sale price is in hte hundreds of thousands, so let's divide the price by 1000 to get more manageable numbers.
Step3:  Merge the training and test datasets for data preparation
Step4:  Categorical features
We've got 42 categorical features. We can use the describe method to get some statistics
Step5:  Number of categories per feature
Step6:  Number of nulls per feature
Step7:  But that's a lot of numbers to digest. Better get started plotting! To help with plotting, but also to improve linear regression models, we're going to standardize our data. But before that we must deal with the NaN values.
http
Step8:  Replace true nulls with mode 
(work in progress)
Step9:  Change dtypes to category
And order categories by the mean price, so the plots are easier to read.
Step10:  Order columns by uniques
That way we see plots with more categories first.
Step11:  Explore categories
See http
Step13:  Plot sale price distribution for each category
Step14:  Use samlib.featureplots to plot the distribution and value count of each category. 
Gives an idea of the distribution of values for each categorical variable. We can see that some categories, such as 'Condition2', are almost constant so are unlikely to have a large impact on predicting the sale xprice.
Step15:  Let's create an indicator to detect the bad features, that
Step16:  The medians of the logprice within each categories should have maximum variance (the less the variance, the less we can distinguish the price within each category).
Step17:  The entropy of the value counts should be large
Step18:  Good features have a high variability and more uniform counts.
Step19:  Let's plot our top features and check that they indeed appear helpful (good variability in the medians and high entropy in the counts).
Step20:  Sort columns by goodness and save
<ASSISTANT_TASK:>
Python Code:
target = pd.read_csv('../data/train_target.csv')
target.describe()
Explanation: Sale price distribution
First step is to look at the target sale price for the training data set, i.e. the column we're trying to predict.
End of explanation
target = target / 1000
logtarget = np.log1p(target)
Explanation: The sale price is in hte hundreds of thousands, so let's divide the price by 1000 to get more manageable numbers.
End of explanation
def read():
    Read training and test data and return a dataframe with ['Dataset','Id'] multi-index
    
    raw_train = pd.read_csv('../data/train_prepared_light.csv')
    raw_test = pd.read_csv('../data/test_prepared_light.csv')
    df = pd.concat([raw_train, raw_test], keys=['train', 'test'])
    df.index.names = 'Dataset', 'Id'
    return df
    
df = read()
pp = samlib.Pipeline(df.copy())  
assert pp == df  # the pipeline output equals df
def select_categorical_features(df):
    return df.loc[:, df.dtypes == object]
pp.append(select_categorical_features)
Explanation: Merge the training and test datasets for data preparation
End of explanation
pp().shape
pp().describe()
pp().isnull().sum()
Explanation: Categorical features
We've got 42 categorical features. We can use the describe method to get some statistics:
End of explanation
plt.figure(figsize=(12, 10))
pp().describe().loc['unique'].sort_values(ascending=False).plot(kind='barh')
plt.title('Number of categories per feature')
Explanation: Number of categories per feature
End of explanation
nulls = pp().isnull().sum()
plt.figure(figsize=(12, 10))
ax = nulls[nulls > 0].sort_values(ascending=False).plot(kind='barh')
plt.title('Number of nulls per feature')
Explanation: Number of nulls per feature
End of explanation
def replace_bad_nulls(df):
    nulls = df.isnull().sum()
    bad_nulls_colz = nulls[nulls > 0].sort_values()['BsmtFinType1':].index
    return samlib.fillna(df, lambda x: 'none', bad_nulls_colz)
pp.append(replace_bad_nulls)
Explanation: But that's a lot of numbers to digest. Better get started plotting! To help with plotting, but also to improve linear regression models, we're going to standardize our data. But before that we must deal with the NaN values.
http://sebastianraschka.com/Articles/2014_about_feature_scaling.html
Deal with NaN values
Based on the descriptions, most of the null values are not missing values but an extra "none" category (e.g. no pool, or no alley etc). In the above graph, this is true for all features starting from 'BsmtFinType1'.
End of explanation
def fill_nulls_with_mode(df):
    return samlib.fillna(df, lambda dg: dg.mode().loc[0])
pp.append(fill_nulls_with_mode)
Explanation: Replace true nulls with mode 
(work in progress)
End of explanation
pp()['LotShape'].head()
def ordered_categories(series):
    dg = pd.DataFrame(series).copy()
    dg.loc['train','LogSalePrice'] = logtarget.values
    return dg.groupby(dg.columns[0]).median().sort_values('LogSalePrice', ascending=False)
ordered_categories(pp()['LotShape'])
def categorize(feature):
    feature = feature.copy()
    return pd.Categorical(feature, ordered_categories(feature).index)
categorize(pp()['LotShape'])
def objects_to_categories(df):
    return df.apply(categorize)
objects_to_categories(pp())['LotShape'].head()
pp.append(objects_to_categories)
pp()['LotShape'].head()
Explanation: Change dtypes to category
And order categories by the mean price, so the plots are easier to read.
End of explanation
def order_columns_by_uniques(df):
    colz_ordered_by_unique = df.describe().loc['unique'].sort_values(ascending=False).index
    return df.reindex_axis(colz_ordered_by_unique, axis=1)
pp.append(order_columns_by_uniques)
pp().head()
Explanation: Order columns by uniques
That way we see plots with more categories first.
End of explanation
df = pp()
df.shape
Explanation: Explore categories
See http://seaborn.pydata.org/tutorial/categorical.html for some ideas
End of explanation
train = pp().loc['train']
def plot_price_dist(y='LotShape', data=train, logtarget=logtarget, **kwargs):
    Plot the price distribution for each category
    dg = data[[y]].copy()
    dg['LogSalePrice'] = logtarget
    # Order categories by mean value
    sns.violinplot(x="LogSalePrice", y=y, data=dg, scale='width', **kwargs)
    #sns.swarmplot(x="LogSalePrice", y=y, data=dg, color="w", alpha=.5);
def plot_value_counts(y=None, data=df, **kwargs):
    if y is None:
        if data.shape[1] == 1:
            y = data.columns[0]
        else:
            raise ValueError('Must pass y or a dataframe with a single column')
    return sns.countplot(y=y, data=data, **kwargs)
train.shape
Explanation: Plot sale price distribution for each category
End of explanation
samlib.featureplots(train, nrows=21, ncols=4, figsize=(2, 8), plotfuncs=(plot_price_dist, plot_value_counts), axis=1)
Explanation: Use samlib.featureplots to plot the distribution and value count of each category. 
Gives an idea of the distribution of values for each categorical variable. We can see that some categories, such as 'Condition2', are almost constant so are unlikely to have a large impact on predicting the sale xprice.
End of explanation
looks_good = 'Neighborhood'
looks_bad = 'Condition1'
Explanation: Let's create an indicator to detect the bad features, that:
- are nearly constant (almost all values are in the same category)
- the medians of the categories is nearly constant (unlikely to help predict the price)
End of explanation
# medians
med_price = logtarget.median().values
df = pp()
sharps = df.apply(lambda col: ordered_categories(col).std().values).iloc[0] / med_price
fig, ax = plt.subplots(1,1, figsize=(12, 12))
sharps.sort_values().plot(kind='barh', ax=ax)
plt.title('Std of median logprices for each category (high is better)')
Explanation: The medians of the logprice within each categories should have maximum variance (the less the variance, the less we can distinguish the price within each category).
End of explanation
from scipy.stats import entropy
df = pp()
unicounts = df.apply(lambda ser: entropy(ser.value_counts()))
fig, ax = plt.subplots(1,1, figsize=(12, 12))
unicounts.sort_values().plot(kind='barh', ax=ax)
plt.title('Entropy of value counts for each category (high is better)')
Explanation: The entropy of the value counts should be large: the higher the entropy, the more uniformly distributed the value counts.
End of explanation
goodness = sharps * unicounts    
fig, ax = plt.subplots(1,1, figsize=(12, 12))
goodness.sort_values().plot(kind='barh', ax=ax)
plt.title('Goodness of category (high is better)')
topcolz = goodness.sort_values(ascending=False)[:12].index
topcolz
Explanation: Good features have a high variability and more uniform counts.
End of explanation
samlib.featureplots(train[topcolz], nrows=6, ncols=4, figsize=(2, 8), plotfuncs=(plot_price_dist, plot_value_counts), axis=1)
Explanation: Let's plot our top features and check that they indeed appear helpful (good variability in the medians and high entropy in the counts).
End of explanation
df = pp()
df.columns = goodness.sort_values(ascending=False).index
df.to_csv('cleaned_categorical_vars_with_colz_sorted_by_goodness.csv', index=True)
Explanation: Sort columns by goodness and save
End of explanation
<END_TASK>