<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Setup
Step1:  Running the examples
Example 1
Step2:  Example 2
Step3:  Example 3
Step4:  His Code
Batch 3
python ex3_batch.py $brand
runs this on each brand
Step5:  Creating User Vecs
Step6:  Exploring the Data
requires the virt-env to be set-up WTF!
Step7:  candiates are items
Step8:  These features are from the images of the previous users. These are the average of the previous images viewed.
Step9:  These features are from the images of the previous users. These are clusters assigned to each image. In an algorithm later, it's easier to just compare similarities of items within the same cluster (rather than every pairwise image).
Step10:  Expanded view list does something with the 'expand browses' and uses formar14_pv. 
For instances 2597.. had formar14 =4 so the expanded list repeated that row 4 times. 
Descriptive Statistics
Step11:  Same number of goods in candidates list as in the buylist
Maybe these are the ones used for recommendations?
Step12:  He has features for all 9863 goods. Does he have the images for those? 
Sessions?
Also where are the person's locations in the database? 
- are these individual sessions for the same user?
Step13:  What are the item views for each person?
Step14:  Dates
<ASSISTANT_TASK:>
Python Code:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import pickle
from IPython.core.debugger import Tracer
import seaborn as sns
%matplotlib inline
Explanation: Setup
End of explanation
import tensorflow as tf
import sklearn
import h5py
import keras
from keras.preprocessing import image
from resnet50 import ResNet50
from imagenet_utils import preprocess_input, decode_predictions
model = ResNet50(weights='imagenet')
from IPython.display import SVG
from keras.utils.vis_utils import model_to_dot
SVG(model_to_dot(model).create(prog='dot', format='svg'))
#model.
img_path = 'img/euro/EUROMODA-U125256-39-5.jpg'
img = image.load_img(img_path, target_size=(224, 224))
img
x = image.img_to_array(img)
x.shape
x = np.expand_dims(x, axis=0)
x = preprocess_input(x)
x.shape
plt.imshow(x[0,:,:,0])
preds = model.predict(x)
plt.plot(preds.T)
preds.shape
print('Predicted:', decode_predictions(preds))
# print: [[u'n02504458', u'African_elephant']]
Explanation: Running the examples
Example 1
End of explanation
from vgg16 import VGG16
from keras.preprocessing import image
from imagenet_utils import preprocess_input
model = VGG16(weights='imagenet', include_top=False)
img_path = '1360x.jpeg'
img = image.load_img(img_path, target_size=(224, 224))
x = image.img_to_array(img)
x = np.expand_dims(x, axis=0)
x = preprocess_input(x)
x.shape
features = model.predict(x)
print features
features.shape
## save out instead 
# from keras.utils import plot_model
# plot_model(model, to_file='model.png')
from IPython.display import SVG
from keras.utils.vis_utils import model_to_dot
SVG(model_to_dot(model).create(prog='dot', format='svg'))
Explanation: Example 2
End of explanation
from vgg19 import VGG19
from keras.preprocessing import image
from imagenet_utils import preprocess_input
from keras.models import Model
base_model.input
base_model.get_layer('block5_pool')
# the layers appear to be keras objects. 
base_model.get_layer('block5_pool').output
# input and output appear to be tensorflow tensors. 
base_model = VGG19(include_top=False, weights='imagenet')
model = Model(input=base_model.input, output=base_model.get_layer('block5_pool').output) 
# this Model creates a model based on some input tensor and some output tensor. 
# here we've taken the base_model, and grabbed it's input layer, and it's 5th output layer, 
# and created a new model with just those layers or is it all the layers inbetween 
#img_path = '1360x.jpeg'
img_path = 'img/euro/EUROMODA-U125256-39-5.jpg'
img = image.load_img(img_path, target_size=(224, 224))
x = image.img_to_array(img)
x = np.expand_dims(x, axis=0)
x = preprocess_input(x)
img
block4_pool_features = model.predict(x) # now we predict based on those layers
block4_pool_features.shape # same shape as before. 
np.shape(block4_pool_features.tolist()[0][0])
import itertools
flattened_list  = list(itertools.chain(*block4_pool_features.tolist()[0][0])) # * will unpack 
for item in flattened_list: print item
Explanation: Example 3
End of explanation
import numpy as np
from vgg19 import VGG19
from resnet50 import ResNet50
from xception import Xception
from keras.preprocessing import image
from imagenet_utils import preprocess_input
from keras.models import Model
import itertools
def get_middle_layer(img_path):
	img = image.load_img(img_path, target_size=(224, 224))
	x = image.img_to_array(img)
	x = np.expand_dims(x, axis=0)
	x = preprocess_input(x)
	block4_pool_features = model.predict(x)
	flattened_list  = list(itertools.chain(*block4_pool_features.tolist()[0][0]))
	return flattened_list
def dot(K, L):
   if len(K) != len(L):
      return 0
   return sum(i[0] * i[1] for i in zip(K, L))
def similarity(item_1, item_2): 
	return dot(item_1, item_2) / np.sqrt(dot(item_1, item_1) * dot(item_2, item_2))
import os
import sys
base_model = ResNet50(include_top=False, weights='imagenet')
model = Model(input=base_model.input, output=base_model.get_layer('avg_pool').output)
path = 'img/baiyi'
features = dict()
for filename in os.listdir(path): # loop through all images in the folder
	img_path = path + '/' + filename
	features[filename] = get_middle_layer(img_path) # get the features from the middle layer 
len(features['Baiyixiuzi-B978N340-5.jpg'])
import itertools
similarities = {item: similarity(features[item[0]], features[item[1]]) for item in itertools.product(features,features)}
for key, item in similarities.items():
	print key[0] + '|' + key[1] + '|' + str(item)
similarities
Explanation: His Code
Batch 3
python ex3_batch.py $brand
runs this on each brand
End of explanation
import math
import numpy as np
import pandas as pd
import pickle
import sys
sys.path.append('4.personalization/')
import utils.kmeans as kmeans
from collections import Counter
def average(lists):
    #Tracer()()
    return [np.mean(i) for i in zip(*[l for l in lists])]
def cluster(lists, model):
    #Tracer()()
    user_cluster = kmeans.predict(np.array([l for l in lists]), model)
    user_vec = [0] * model.n_clusters
    for i in user_cluster: user_vec[i] += 1
    return [elem / float(sum(user_vec)) for elem in user_vec]
user_log = pd.read_pickle('4.personalization/data/viewlist_imagef.pkl')
user_vec = user_log.groupby(['user_id', 'dt'])['features'].apply(lambda x: average(x))
user_log.groupby(['user_id', 'dt'])
user_log[0:100].groupby(['user_id', 'dt'])['features'].apply(lambda x: x)
f = lambda y: y**2
f(7)
#user_log[0:100].groupby(['user_id', 'dt'])['goods_no'].apply(lambda x: x)
model_path='4.personalization/utils/model.pkl'
model = kmeans.load_model(model_path) 
type(model)
model.cluster_centers_.shape
user_vec = user_log.groupby(['user_id', 'dt'])['features'].apply(lambda x: cluster(x, model))
len(user_vec[0])
Explanation: Creating User Vecs
End of explanation
buylist = pd.read_pickle('4.personalization/data/buylist_imagef2.pkl')
buylist.head(20)
candidates = pd.read_pickle('4.personalization/data/candidates.pkl')
candidates.head()
Explanation: Exploring the Data
requires the virt-env to be set-up WTF!
End of explanation
candidates_cluster = pd.read_pickle('4.personalization/data/candidates_cluster.pkl')
candidates_cluster.head()
user_vec_average = pd.read_pickle('4.personalization/data/user_vec_average.pkl')
user_vec_average.head()
Explanation: candiates are items
End of explanation
user_vec_average_no_pv = pd.read_pickle('4.personalization/data/user_vec_average_no_pv.pkl')
user_vec_average_no_pv.head()
user_vec_cluster = pd.read_pickle('4.personalization/data/user_vec_cluster.pkl')
user_vec_cluster.head()
Explanation: These features are from the images of the previous users. These are the average of the previous images viewed.
End of explanation
len(user_vec_cluster['features'][0]) 
viewlist = pd.read_pickle('4.personalization/data/viewlist_imagef.pkl')
viewlist.head(100)
viewlist_exp = pd.read_pickle('4.personalization/data/viewlist_imagef_exp.pkl')
viewlist_exp.head(100)
Explanation: These features are from the images of the previous users. These are clusters assigned to each image. In an algorithm later, it's easier to just compare similarities of items within the same cluster (rather than every pairwise image).
End of explanation
print('rows in buylist: {0}').format(str(len(buylist)))
print('rows in  viewlsit: {0}').format(str(len(viewlist)))
print('rows in  viewlsit expanded: {0}').format(str(len(viewlist_exp)))
print('rows in  candidates: {0}').format(str(len(candidates)))
print('rows in  candidates cluster: {0}').format(str(len(candidates)))
print('rows in  user_vec_average: {0}').format(str(len(user_vec_average)))
print('number of users in buylist: {0}').format(str(len(set(buylist.user_id.unique()))))
print('number of users in viewlist: {0}').format(str(len(set(viewlist.user_id.unique()))))
print('number of users in both sets: {0}').format(str(len(set(buylist.user_id.unique()).intersection(set(viewlist.user_id.unique())))))
print('number of goods in buylist: {0}').format(str(len(set(buylist.goods_no.unique()))))
print('number of goods in viewlist: {0}').format(str(len(set(viewlist.goods_no.unique()))))
print('number of goods in both sets: {0}').format(str(len(set(buylist.goods_no.unique()).intersection(set(viewlist.goods_no.unique())))))
Explanation: Expanded view list does something with the 'expand browses' and uses formar14_pv. 
For instances 2597.. had formar14 =4 so the expanded list repeated that row 4 times. 
Descriptive Statistics
End of explanation
viewlist.features[3000]
Explanation: Same number of goods in candidates list as in the buylist
Maybe these are the ones used for recommendations?
End of explanation
for uid in viewlist_exp.user_id.unique():
    print('user: {0}').format(uid)
    indices = viewlist_exp.loc[viewlist_exp.user_id==uid].index.tolist()
    print('places in database: {0}').format(indices)
    print('')
#viewlist.dt
Explanation: He has features for all 9863 goods. Does he have the images for those? 
Sessions?
Also where are the person's locations in the database? 
- are these individual sessions for the same user?
End of explanation
uid = 18318014
indices = viewlist_exp.loc[viewlist_exp.user_id==uid].index.tolist()
indexlast = -1
print('single user: {0}').format(uid)
for index in indices:
    # find product 
    if index-indexlast>1:
        print('')
        #print('new session')
    print('row {0}, good number {1}  date {2}').format(index,viewlist_exp.loc[index,'goods_no'],viewlist_exp.loc[index,'dt'])
    indexlast = index.copy()
    
Explanation: What are the item views for each person?
End of explanation
dates_per_user = np.array([])
for uid in viewlist_exp.user_id.unique():
    dates_per_user = np.append(dates_per_user,len(viewlist_exp.loc[viewlist_exp.user_id==uid,'dt'].unique()))
plt.hist(dates_per_user)
sns.despine()
plt.xlabel('number of dates per user')
dates_per_user
viewlist_exp.dt.unique()
Explanation: Dates
End of explanation
<END_TASK>