<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Step 0 - hyperparams
vocab_size is all the potential words you could have (classification for translation case)
and max sequence length are the SAME thing
decoder RNN hidden units are usually same size as encoder RNN hidden units in translation but for our case it does not seem really to be a relationship there but we can experiment and find out later, not a priority thing right now
Step1:  Include relevant deals
Step2:  So for the same date window if we find data from the relevant deal we are good to go
If we do NOT find them ... we could go and search the next deal within some limit of course
So if we exceed this limit, what could we do?  
We could pass zeros is one solution
We could drop this item completely is another solution
Step3:  This is taking longer than expected but ok
Step4:  Combine Data
Step5:  Train - Test Split
<ASSISTANT_TASK:>
Python Code:
num_units = 400 #state size
input_len = 60
target_len = 30
batch_size = 50
with_EOS = False
total_train_size = 57994
Explanation: Step 0 - hyperparams
vocab_size is all the potential words you could have (classification for translation case)
and max sequence length are the SAME thing
decoder RNN hidden units are usually same size as encoder RNN hidden units in translation but for our case it does not seem really to be a relationship there but we can experiment and find out later, not a priority thing right now
End of explanation
from time import sleep
data_path = '../../../../Dropbox/data'
ph_data_path = data_path + '/price_history'
npz_full = ph_data_path + '/price_history_mobattrs_date_dp_60to30_62020.npz'
assert path.isfile(npz_full)
csv_in = '../price_history_03_seq_start_suddens_trimmed.csv'
#npz_train = ph_data_path + '/price_history_dp_60to30_63548_46400_train.npz'
#npz_train_mobattrs = ph_data_path + '/price_history_mobattrs_dp_60to30_57994_train.npz'
# npz_test = ph_data_path + '/price_history_dp_60to30_57994_11584_test.npz'
# npz_test_mobattrs = ph_data_path + '/price_history_mobattrs_dp_60to30_57994_test.npz'
obj = PriceHistWithRelevantDeals(npz_path=npz_full, price_history_csv_path=csv_in, random_state=random_state,
                                verbose=False)
dic = obj.execute(relevancy_count=2)
npz_augmented = ph_data_path + '/price_history_mobattrs_date_deals_dp_60to30.npz'
dic.keys()
dic['inputs'][0].shape
for key, val in dic.iteritems():
    print key, len(val)
len(dic['targets'])
len(dic['inputs'])
args = np.argwhere([curin.shape != (60, 9) for curin in dic['inputs']]).flatten()
len(args)
args = list(args)
# for cur in dic['inputs'][args]:
#     print cur.shape
for arg in args:
    print dic['inputs'][arg].shape
keep_args = set(range(len(dic['inputs']))).difference(args)
assert len(keep_args) == len(dic['inputs']) - len(args)
keep_args = list(keep_args)
newdic = {
    'inputs': np.array([dic[key][keep_arg] for keep_arg in keep_args])
}
newdic['inputs'].shape
for key, val in dic.iteritems():
    if key == 'inputs':
        continue
    newdic[key] = dic[key][keep_args]
    print newdic[key].shape
#dic['inputs'] = np.array(dic['inputs'])
np.array(newdic['inputs']).shape
np.savez(npz_augmented, **newdic)
npz = np.load(npz_full)
for key, val in npz.iteritems():
    print key,val.shape
my_current_ind = 100 #just because
target_ind = npz['sku_ids'][my_current_ind]
target_ind #this is the SKU ID we are interested in now
from relevant_deals import RelevantDeals
rd = RelevantDeals()
all_deals = rd.getSome(target_ind)
relevancy_order = 2 #2 extra sku ids to keep
relevant_sku_ids = all_deals[:relevancy_order]
relevant_sku_ids
#get everything normalized globally
df = PriceHistory27DatasetGenerator(random_state=random_state).global_norm_scale(
    pd.read_csv(csv_in, index_col=0, quoting=csv.QUOTE_ALL, encoding='utf-8')
)
yearday = 1
month_ind = 2
weekday_ind = 3
year_ind = 4
yearweek_ind = 5
day_ind = 6
year_ind, month_ind, day_ind
print np.unique(npz['inputs'][my_current_ind][:, 1]) #<--- this is year day
print np.unique(npz['inputs'][my_current_ind][:, 2]) #<--- this is month
print np.unique(npz['inputs'][my_current_ind][:, 3]) #<--- this is weekday
print np.unique(npz['inputs'][my_current_ind][:, 4]) #<---- this is the year
print np.unique(npz['inputs'][my_current_ind][:, 5]) #<--- this is year week
print np.unique(npz['inputs'][my_current_ind][:, 6]) #<--- this is month day
the_input = npz['inputs'][my_current_ind]
start_item = the_input[0].astype(np.int)
start_item.shape
start_date = "{}-{:02d}-{:02d}".format(start_item[year_ind], start_item[month_ind], start_item[day_ind])
#this format is useful because we can compare them as strings without conversion
start_date
end_item = npz['inputs'][my_current_ind][-1].astype(np.int)
end_date = "{}-{:02d}-{:02d}".format(end_item[year_ind], end_item[month_ind], end_item[day_ind])
end_date
Explanation: Include relevant deals
End of explanation
# for one sku id
cur_sku_id = relevant_sku_ids[0]
cur_sku_id
seq = PriceHistory27DatasetGenerator.extractSequence(df.loc[cur_sku_id])
len(seq)
check = seq.index[0] <= start_date and end_date <= seq.index[-1]
check
#extract the sequence of interest
begin_ind = np.argwhere(seq.index == start_date).flatten()[0]
begin_ind
ending_ind = np.argwhere(seq.index == end_date).flatten()[0]
ending_ind
seq_of_interest = seq[begin_ind:ending_ind+1]
seq_of_interest.shape
the_input.shape
sns.tsplot(seq_of_interest)
plt.show()
unbiased = PriceHistory27DatasetGenerator.removeBiasFromSeq(seq_of_interest)
sns.tsplot(unbiased)
plt.show()
ready_deal = unbiased.values[np.newaxis].T
ready_deal.shape
newinput = np.hstack((the_input, ready_deal))
newinput.shape
np.array([unbiased, unbiased]).T
aa = np.array([unbiased, unbiased]).T
ee = np.array([]).T
ee.shape
np.hstack((the_input, ee)).shape
mylist = []
for ii, jj in zip(range(-3, 0), range(0, 3)):
    mylist.append((ii, jj))
mylist
map(list, zip(*mylist))[1]
sns.tsplot(aa[:, 1])
Explanation: So for the same date window if we find data from the relevant deal we are good to go
If we do NOT find them ... we could go and search the next deal within some limit of course
So if we exceed this limit, what could we do?  
We could pass zeros is one solution
We could drop this item completely is another solution
End of explanation
%%time
dic = PriceHistory27DatasetGenerator.merge_date_info(npz_path=npz_full)
for key, val in dic.iteritems():
    print val.shape
# npz_full_with_date = ph_data_path + '/price_history_dp_60to30_63548_date_info.npz'
# np.savez(npz_full_with_date, **dic)
Explanation: This is taking longer than expected but ok
End of explanation
combinator = PriceHistoryMobileAttrsCombinator()
%%time
dic, inds, count_key_errors, key_errors = combinator.combine(npz_in=npz_full_with_date)
for key, val in dic.iteritems():
    print val.shape
npz_full_mobattrs_date = ph_data_path + '/price_history_mobattrs_date_dp_60to30_62020.npz'
np.savez(npz_full_mobattrs_date, **dic)
count_key_errors#, key_errors
Explanation: Combine Data
End of explanation
npz_train_mobattrs_date = ph_data_path + '/price_history_mobattrs_date_dp_60to30_62020_train.npz'
npz_test_mobattrs_date = ph_data_path + '/price_history_mobattrs_date_dp_60to30_62020_test.npz'
PriceHistory27DatasetGenerator.train_test_split(fullpath=npz_full_mobattrs_date, test_size=6200,
                                                train_path=npz_train_mobattrs_date,
                                                test_path=npz_test_mobattrs_date, random_state=random_state)
npz_train_mobattrs_date_small = ph_data_path + '/price_history_mobattrs_date_dp_60to30_62020_6000_train.npz'
PriceHistory27DatasetGenerator.create_subsampled(inpath=npz_train_mobattrs_date, target_size=6000,
                                               outpath=npz_train_mobattrs_date_small, random_state=random_state)
Explanation: Train - Test Split
End of explanation
<END_TASK>