<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Bridging Schema
This notebook demonstrates the Bridging Schema design pattern
Create dataset
Let's create a dataset by simulating the scenario described in the book
Step1:  Bridging
Create dataset of 5000 new transactions and 1 million old transactions
Step2:  How many samples do we need to evaluate properly?
Train a model on the old dataset and evaluate it on different subsets
Step3:  Looking at this, it is clear that (on this problem) 3500 eval samples would be ideal, but 2000 is tolerable.
So, let's keep 2000 new examples aside for evaluation.
Take 3000 new examples for training
Step4:  Bridging schema using static approach
Estimate the probability of the different card types from the new data to find how to encode the
Step5:  Prepare new data
Because we explicitly encoded the fields for the bridged data, let's make sure to repeat this for the new data also.
Step6:  Augmented training dataset
Start with the 3000 new training examples, then add in bridged examples
from the old data as long as performance improves. This is a hyperparameter tuning operation.
Here's we'll simply do grid-search.
<ASSISTANT_TASK:>
Python Code:
import pandas as pd
import numpy as np
import seaborn as sns
sns.set(rc={'figure.figsize':(10, 10)})
def set_payment_type(prob):
    # 30% of transactions are cash
    if prob < 0.3:
        return 'Cash'
    # stretch the remaining 0.3-1.0 to 0-1
    prob = (prob-0.3)/0.7
    if prob < 0.1:
        return 'Gift'
    if prob < 0.4:
        return 'Debit'
    return 'Credit'
def create_dataset(N, is_old_format):
    order_amount = 5 + np.abs(np.random.normal(scale=10, size=N)) # 5+, peaks at 15
    order_distance = 1 + np.random.rand(N)*4  # 1-5
    payment_type = np.random.rand(N)
    
    df = pd.DataFrame.from_dict({
        'order_amount' : np.round(order_amount, 2),
        'order_distance': np.round(order_distance, 1),
        'payment_type': np.vectorize(set_payment_type)(payment_type),
        'tip': np.zeros_like(order_amount)
    })
    
    df.loc[df['payment_type'] == 'Cash', 'tip'] = 0.2
    df.loc[df['payment_type'] == 'Gift', 'tip'] = 0.1
    df.loc[df['payment_type'] == 'Debit',  'tip'] = 0.15
    df.loc[df['payment_type'] == 'Credit', 'tip'] = 0.25
    
    df['tip'] = np.round(df['tip'] * df['order_amount'] + 0.5*df['order_distance'], 1)
    
    if is_old_format:
        df.loc[df['payment_type'] != 'Cash', 'payment_type'] = 'Card'
    
    return df
df = create_dataset(5, False)
df
df = create_dataset(5, True)
df
Explanation: Bridging Schema
This notebook demonstrates the Bridging Schema design pattern
Create dataset
Let's create a dataset by simulating the scenario described in the book
End of explanation
df_new = create_dataset(5000, False)
df_old = create_dataset(1000*1000, True)
Explanation: Bridging
Create dataset of 5000 new transactions and 1 million old transactions
End of explanation
N_train = (int) (len(df_old) * 0.5)
N_eval = len(df_old) - N_train
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import OneHotEncoder
model = LinearRegression()
enc = OneHotEncoder(handle_unknown='ignore')
df_train = enc.fit_transform( df_old.loc[:N_train, df.columns != 'tip'] )
model.fit(df_train, df_old.loc[:N_train, 'tip'])
df_eval = enc.transform( df_old.loc[N_train:, df.columns != 'tip'] )
model.score(df_eval, df_old.loc[N_train:, 'tip'])
sizes = []
score_mean = []
score_stddev = []
for subset_size in range(100, 5000, 100): # 5000 here because that is the number of new examples we have
    sizes.append(subset_size)
    # compute variability of the eval metric at this subset size
    scores = []
    for x in range(1, 25):
        indices = np.random.choice(N_eval, size=subset_size, replace=False)
        scores.append(
            model.score(df_eval[indices], df_old.loc[N_train+indices, 'tip'])
        )
    score_mean.append(np.mean(scores))
    score_stddev.append(np.std(scores))
scores = pd.DataFrame.from_dict({
    'eval_size': sizes,
    'score': score_mean,
    'stddev': score_stddev
})
scores.plot(x='eval_size', y='stddev', linewidth=2);
Explanation: How many samples do we need to evaluate properly?
Train a model on the old dataset and evaluate it on different subsets
End of explanation
N_NEW = 2000
Explanation: Looking at this, it is clear that (on this problem) 3500 eval samples would be ideal, but 2000 is tolerable.
So, let's keep 2000 new examples aside for evaluation.
Take 3000 new examples for training
End of explanation
df_train_new = df_new.loc[N_NEW:, :]
counts = df_train_new.groupby('payment_type').count()
counts
num_credit = counts.loc['Credit', 'tip']
num_debit = counts.loc['Debit', 'tip']
num_gift = counts.loc['Gift', 'tip']
num_total = num_gift + num_debit + num_credit
imputed_value = [
    0,
    num_gift / num_total,
    num_debit / num_total,
    num_credit / num_total
]
print(imputed_value)
bridged_old = df_old.copy()
bridged_old['encoded_cash'] = 1  # default to cash
bridged_old['encoded_gift'] = 0  # default to cash
bridged_old['encoded_debit'] = 0  # default to cash
bridged_old['encoded_credit'] = 0  # default to cash
bridged_old.loc[bridged_old['payment_type'] == 'Card', 'encoded_cash'] = imputed_value[0]
bridged_old.loc[bridged_old['payment_type'] == 'Card', 'encoded_gift'] = imputed_value[1]
bridged_old.loc[bridged_old['payment_type'] == 'Card', 'encoded_debit'] = imputed_value[2]
bridged_old.loc[bridged_old['payment_type'] == 'Card', 'encoded_credit'] = imputed_value[3]
bridged_old.drop(columns='payment_type', inplace=True)
bridged_old[:10]
Explanation: Bridging schema using static approach
Estimate the probability of the different card types from the new data to find how to encode the
End of explanation
encoded_new = df_new.copy()
encoded_new['encoded_cash'] = 0  # default to zero
encoded_new['encoded_gift'] = 0  # default to zero
encoded_new['encoded_debit'] = 0  # default to zero
encoded_new['encoded_credit'] = 0  # default to zero
encoded_new.loc[encoded_new['payment_type'] == 'Cash', 'encoded_cash'] = 1 # one-hot encode
encoded_new.loc[encoded_new['payment_type'] == 'Gift', 'encoded_gift'] = 1 # one-hot encode
encoded_new.loc[encoded_new['payment_type'] == 'Debit', 'encoded_debit'] = 1 # one-hot encode
encoded_new.loc[encoded_new['payment_type'] == 'Credit', 'encoded_credit'] = 1 # one-hot encode
encoded_new.drop(columns='payment_type', inplace=True)
encoded_new[:5]
encoded_new_eval = encoded_new.loc[:N_NEW, :]
evalX = encoded_new_eval.loc[:, encoded_new_eval.columns != 'tip']
evalY = encoded_new_eval.loc[:, 'tip']
encoded_new_train = encoded_new.loc[N_NEW:, :]
Explanation: Prepare new data
Because we explicitly encoded the fields for the bridged data, let's make sure to repeat this for the new data also.
End of explanation
# add N bridged examples from old and train.
bridged_examples = []
scores = []
for num_bridged in range(0, 1000*100, 1000):
    bridged_examples.append(num_bridged)
    df_train = pd.concat([encoded_new_train, bridged_old[:num_bridged]], ignore_index=True)
    X = df_train.loc[:, df_train.columns != 'tip']
    Y = df_train.loc[:, 'tip']
    model.fit(X, Y)
    scores.append(model.score(evalX, evalY))
scores = pd.DataFrame.from_dict({
    'number_bridged': bridged_examples,
    'eval_metric': scores
})
scores.plot(x='number_bridged', y='eval_metric', linewidth=2);
Explanation: Augmented training dataset
Start with the 3000 new training examples, then add in bridged examples
from the old data as long as performance improves. This is a hyperparameter tuning operation.
Here's we'll simply do grid-search.
End of explanation
<END_TASK>