<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 How to use
maximum_superleave_length indicates the maximum length of superleaves to consider. Right now, maximum runnable on my local machine is 5.
ev_calculator_max_length indicates the maximum length of superleave that we want to calculate the EV for based on a log file of games (log_file).
To-do
Have creation of duplicates go in reverse so that all necessary columns are present.
Changelog
1/27/19 - Determined that the speed of creation of the rack dataframes is a function of the length of the dataframe. From that, realized that we should organize leaves by least-frequent to most-frequent letter, such that sub-dataframes are created from the shortest racks possible.
Step1:  Create a dictionary of all possible 1 to 6-tile leaves. Also, add functionality for sorting by an arbitrary key - allowing us to put rarest letters first
Step2:  The bottom creates the full set of leaves for all lengths from 1-5 (6 breaks on my local machine)
Step3:  Order rack (originally alphabetical, but now custom key with rarest letters first for maximum efficiency). Note that this is slower than alphabetical organization because it has to use the index function, but should be rewarded with subsequent performance enhancements.
Step4:  Set up dataframe for storing EV of all leaves.
Step5:  To find all of the racks corresponding to a particular leave, we have added columns to the dataframe of plays df marking each letter (A, B, C...) and also for duplicates (AA, BB, CC...) and triplicates where possible (AAA, DDD, EEE...).
If the letters in a given leave are all different, we can look for rows by using df['A']&df['B']. However, if there are duplicates involved, we have to look for df['AA']. The following function gives the correct dataframe columns to be looked up.
Step6:  Benchmark figures
With 2M racks, following amount of time was taken
Step7:  Calculate leave "synergy", in other words the difference between the EV of the rack and what we'd expect just from adding the individual values of the tiles
<ASSISTANT_TASK:>
Python Code:
from itertools import combinations
import numpy as np
import pandas as pd
import seaborn as sns
from string import ascii_uppercase
import time as time
%matplotlib inline
maximum_superleave_length = 5
ev_calculator_max_length = 5
log_file = 'log_games.csv'
Explanation: How to use
maximum_superleave_length indicates the maximum length of superleaves to consider. Right now, maximum runnable on my local machine is 5.
ev_calculator_max_length indicates the maximum length of superleave that we want to calculate the EV for based on a log file of games (log_file).
To-do
Have creation of duplicates go in reverse so that all necessary columns are present.
Changelog
1/27/19 - Determined that the speed of creation of the rack dataframes is a function of the length of the dataframe. From that, realized that we should organize leaves by least-frequent to most-frequent letter, such that sub-dataframes are created from the shortest racks possible.
End of explanation
tilebag = ['A']*9+['B']*2+['C']*2+['D']*4+['E']*12+\
          ['F']*2+['G']*3+['H']*2+['I']*9+['J']*1+\
          ['K']*1+['L']*4+['M']*2+['N']*6+['O']*8+\
          ['P']*2+['Q']*1+['R']*6+['S']*4+['T']*6+\
          ['U']*4+['V']*2+['W']*2+['X']*1+['Y']*2+\
          ['Z']*1+['?']*2
            
tiles = [x for x in ascii_uppercase] + ['?']
# potential future improvement: calculate optimal order of letters on the fly
rarity_key = 'ZXKJQ?HYMFPWBCVSGDLURTNAOIE'
# alphabetical_key = '?ABCDEFGHIJKLMNOPQRSTUVWXYZ'
sort_func = lambda x: rarity_key.index(x)
t0 = time.time()
leaves = {i:sorted(list(set(list(combinations(tilebag,i))))) for i in 
          range(1,maximum_superleave_length+1)}
# turn leaves from lists of letters into strings
# algorith runs faster if leaves non-alphabetical!
for i in range(1,maximum_superleave_length+1):
    leaves[i] = [''.join(sorted(leave, key=sort_func))
                 for leave in leaves[i]]
    
t1 = time.time()
print('Calculated superleaves up to length {} in {} seconds'.format(
    maximum_superleave_length,t1-t0))
Explanation: Create a dictionary of all possible 1 to 6-tile leaves. Also, add functionality for sorting by an arbitrary key - allowing us to put rarest letters first
End of explanation
for i in range(1,maximum_superleave_length+1):
    print(i,len(leaves[i]))
column_dict = {
    0:'rack',
    1:'score',
    2:'tiles_remaining'
}
df = pd.read_csv(log_file, header=None, keep_default_na=False)
df.rename(columns=column_dict,inplace=True)
tile_limit = 1
df = df.loc[df['tiles_remaining']>=tile_limit]
df = df.iloc[:2000000]
Explanation: The bottom creates the full set of leaves for all lengths from 1-5 (6 breaks on my local machine)
End of explanation
t0 = time.time()
df['rack'] = df['rack'].apply(lambda x: ''.join(sorted(x, key=sort_func)))
t1 = time.time()
print(t1-t0)
tb = time.time()
df_dict = {'': df}
for multiple in range(1,maximum_superleave_length+1):
    t0 = time.time()
    # iterate through all 27 tiles
    for c in leaves[1]:
        if multiple*c in leaves[multiple]:
            condition = df_dict[(multiple-1)*c]['rack'].apply(lambda x: multiple*c in x)
            df_dict[multiple*c] = df_dict[(multiple-1)*c].loc[condition]
            df[multiple*c] = condition
            df[multiple*c].fillna(False, inplace=True)
                        
    t1 = time.time()
    print('Added columns for all duplicates up to length {} in {} seconds'.format(multiple,t1-t0))
    
te = time.time()
print('Added all necessary columns in {} seconds'.format(te-tb))
Explanation: Order rack (originally alphabetical, but now custom key with rarest letters first for maximum efficiency). Note that this is slower than alphabetical organization because it has to use the index function, but should be rewarded with subsequent performance enhancements.
End of explanation
all_leaves = []
for i in range(1,ev_calculator_max_length+1):
    all_leaves += leaves[i]
    
df_dict = {leave: pd.DataFrame() for leave in all_leaves}
df_dict[''] = df
ev_df = pd.DataFrame(columns=['mean','std','count','ev','synergy'],
                     index=all_leaves)
Explanation: Set up dataframe for storing EV of all leaves.
End of explanation
def get_columns(leave):
    letters=list(set(leave))
    tags = []
    
    for l in letters:
        tags += [sum([l==letter for letter in leave])*l]
    
    return tags
for leave_length in range(3,5):
    print(leave_length)
    t0 = time.time()
    
    for leave in leaves[leave_length]:
        print(leave)
        print(len(df_dict[leave[:-1]]))
        t2 = time.time()
        condition = df_dict[leave[:-1]][get_columns(leave)].all(axis=1)
        t3 = time.time()
        df_dict[leave] = df_dict[leave[:-1]].loc[condition]
        t4 = time.time()
        ev_df.loc[leave]['mean'] = df_dict[leave]['score'].mean()
        t5 = time.time()
        ev_df.loc[leave]['std'] = df_dict[leave]['score'].std()
        t6 = time.time()
        ev_df.loc[leave]['count'] = len(df_dict[leave])
        t7 = time.time()
        print('condition calc time (ms): {:.5f} ({})'.format(1000*(t3-t2),100*(t3-t2)/(t7-t2)))
        print('condition calc time (ms): {:.5f} ({})'.format(1000*(t4-t3),100*(t4-t3)/(t7-t2)))
        print('condition calc time (ms): {:.5f} ({})'.format(1000*(t5-t4),100*(t5-t4)/(t7-t2)))
        print('condition calc time (ms): {:.5f} ({})'.format(1000*(t6-t5),100*(t6-t5)/(t7-t2)))
        print('condition calc time (ms): {:.5f} ({})'.format(1000*(t7-t6),100*(t7-t6)/(t7-t2)))
        
    t1 = time.time()
    print('Calculated mean, std and count in {} seconds'.format(t1-t0))
Explanation: To find all of the racks corresponding to a particular leave, we have added columns to the dataframe of plays df marking each letter (A, B, C...) and also for duplicates (AA, BB, CC...) and triplicates where possible (AAA, DDD, EEE...).
If the letters in a given leave are all different, we can look for rows by using df['A']&df['B']. However, if there are duplicates involved, we have to look for df['AA']. The following function gives the correct dataframe columns to be looked up.
End of explanation
for leave_length in range(1,ev_calculator_max_length+1):
    print(leave_length)
    t0 = time.time()
    
    for leave in leaves[leave_length]:
        condition = df_dict[leave[:-1]][get_columns(leave)].all(axis=1)
        df_dict[leave] = df_dict[leave[:-1]].loc[condition]
        ev_df.loc[leave]['mean'] = df_dict[leave]['score'].mean()
        ev_df.loc[leave]['std'] = df_dict[leave]['score'].std()
        ev_df.loc[leave]['count'] = len(df_dict[leave])
        
    t1 = time.time()
    print('Calculated mean, std and count in {} seconds'.format(t1-t0))
ev_df['pct'] = 100*ev_df['count']/len(df)
ev_df['ev'] = ev_df['mean']-df['score'].mean()
Explanation: Benchmark figures
With 2M racks, following amount of time was taken:
* 1 - 3s (.11s/leave)
* 2 - 15s (.04s/leave)
* 3 - 38s (.011s/leave)
* 4 - 84s (.0033s/leave)
* 5 - 244s (.0016s/leave)
-> 383s total
Using improvement of non-alphabetical leaves, performance was as follows:
* 1 - 3s (.11s/leave)
* 2 - 10s (.027s/leave)
* 3 - 23s (.0066s/leave)
* 4 - 64s (.0025s/leave)
* 5 - 226s (.0015s/leave)
-> 326 total (15% faster than previous version)
End of explanation
for leave_length in range(2,ev_calculator_max_length+1):
    for leave in leaves[leave_length]:
        ev_df.loc[leave]['synergy'] = ev_df.loc[leave]['ev']-\
                                      sum([ev_df.loc[c]['ev'] for c in leave])
ev_df
ev_df.to_csv('leave_values_011219_v7.csv')
ev_df.sort_values('synergy')
Explanation: Calculate leave "synergy", in other words the difference between the EV of the rack and what we'd expect just from adding the individual values of the tiles
End of explanation
<END_TASK>