<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Association Mining
The goal in this problem set is to design and code an algorithm for generating association rules based on the apriori algorithm. The dataset that you will use to mine for association rules is a synthetic dataset containing over 1500 transactions. The format of the data is as follows
Step1:  2. For a bit of EDA
Step2:  3. Create a function to compute support for a given itemset from a transaction
Step3:  4. Write an apriori function that handles the pruning and generation of frequent itemsets
should return a dict of frequent itemsets as keys, and their support as values
must have a method for setting a minimum support value to determine frequent itemsets
hint
Step4:  5. Create a function that uses the frequent itemsets to generate association rules
must have a min confidence level as a parameter
do not keep rules that fall under this threshold
return dict of rules with string for keys
string key will take a form like
Step5:  6. Create a scatter plot to analyze the association rules
one axis for antecedent and one for consequent
set marker size based on support (use a multiplier to scale these up)
set marker color based on lift
hint
<ASSISTANT_TASK:>
Python Code:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline
with open('itemsets.dat') as f:
    transactions = []
    for row in f:
        transactions.append(row.strip().split(','))
transactions[0:5]
Explanation: Association Mining
The goal in this problem set is to design and code an algorithm for generating association rules based on the apriori algorithm. The dataset that you will use to mine for association rules is a synthetic dataset containing over 1500 transactions. The format of the data is as follows:
Each row in the file represents a single transaction
Each transaction contains one or more letters representing items
The number of each item is immaterial for mining, so each unique letter will appear at most once in a transaction
Note: this means you cannot just read the data into a dataframe, because the data is not columnar
Example:
A
A, C, E
B, D
The above represents 3 transactions, where the third transaction contained at least one of each for items B and D.
Problems
Write a scipt to read the data file 'itemsets.dat' into a list of lists
one inner list of items for each transaction
the .dat file is essentially just a text file
For a bit of EDA: 
create a script that plots the proportions of single items sold
plot the counts for the different itemset sizes
Create a function to compute support for a given itemset from a transaction
Write an apriori function that handles the pruning and generation of frequent itemsets
should return a dict of frequent itemsets as keys, and their support as values
must have a method for setting a minimum support value to determine frequent itemsets
hint: use sets and frozensets <- dict keys cant be mutable objects
Create a function that uses the frequent itemsets to generate association rules
must have a min confidence level as a parameter
do not keep rules that fall under this threshold
return dict of rules with string for keys
string key will take a form like: "{'A', 'B'} -> {'C'}"
values must be a dict containing metric:value pairs where metrics include support, confidence, and lift
return dataframe of rules with the following columns
antecedent, consequent, support, confidence, lift
Note: the support is for the entire set: antecedent union consequent
Create a scatter plot to analyze the association rules
one axis for antecedent and one for consequent
set marker size based on support (use a multiplier to scale these up)
set marker color based on lift
hint: convert axes columns to numeric codes using .astype('category').cat.codes
How should you interpret the lift values that are less than 1?
1. Write a scipt to read the data file 'itemsets.dat' into a list of lists
one inner list of items for each transaction
the .dat file is essentially just a text file
End of explanation
items_counts = {}
for tran in transactions:
    for item in tran:
        if item in items_counts:
            items_counts[item] += 1
        else:
            items_counts[item] = 1
plt.bar(items_counts.keys(), items_counts.values());
itemsets_sizes = {}
for tran in transactions:
    tran = frozenset(tran)
    if tran in itemsets_sizes:
        itemsets_sizes[tran] += 1
    else:
        itemsets_sizes[tran] = 1
fig, ax = plt.subplots(figsize=(16, 8))
ax.bar(itemsets_sizes.keys(), itemsets_sizes.values())
ax.set_xticklabels([list(itemset) for itemset in itemsets_sizes.keys()], rotation='vertical');
Explanation: 2. For a bit of EDA:
create a script that plots the proportions of single items sold
plot the counts for the different itemset sizes
End of explanation
def support_count(trans, itemset):
    count = 0
    for tran in trans:
        if set(itemset).issubset(set(tran)):
            count += 1
    return count
print(support_count(transactions, ['A']))
print(support_count(transactions, ['A', 'C']))
print(support_count(transactions, ['F']))
def support(trans, itemset):
    return support_count(trans, itemset) / len(trans)
print(support(transactions, ['A']))
print(support(transactions, ['A', 'C']))
print(support(transactions, ['F']))
Explanation: 3. Create a function to compute support for a given itemset from a transaction
End of explanation
def generate_supsets(items, sets):
    sups = {}
    for s in sets:
        for item in items:
            sup = frozenset(s.union(item))
            if sup != s and sup not in sups:
                sups[sup] = 1
    return list(sups.keys())
def apriori(trans, minsupp):
    frequent_itemsets = {}
    # get items list
    items = list(set([item for tran in trans for item in tran]))
    # initialize list of itemsets to check
    curr_sets = items
    # iterate till max itemset length
    for i in range(len(items)):
        # print(i, curr_sets)
        # initialize candidates itemsets for generation of supsets
        next_sets = []
        # initialize current iteration unfrequent itemsets list
        unfrequent = []
        for s in curr_sets:
            supp = support(trans, s)
            # print(s, supp)
            # if we are over minsupp add itemset to frequent list and to supsets generation candidates
            if supp >= minsupp:
                frequent_itemsets[frozenset(s)] = supp
                next_sets.append(frozenset(s))
            #else add to unfrequent list
            else:
                unfrequent.append(frozenset(s))
        # if this is the first iteration update items list in order to optimize supsets generation
        if i == 0:
            items = [item for item in items if item not in unfrequent]
        # generate supsets and exclude those that contain an unfrequent itemset
        curr_sets = generate_supsets(items, next_sets)
        for unfr in unfrequent:
            curr_sets = [s for s in curr_sets if not unfr.issubset(s)]
        # print(next_sets)
        # print(unfrequent)
        # print(curr_sets)
        if len(curr_sets) == 0:
            break
    return frequent_itemsets
apriori(transactions, 0.1)
Explanation: 4. Write an apriori function that handles the pruning and generation of frequent itemsets
should return a dict of frequent itemsets as keys, and their support as values
must have a method for setting a minimum support value to determine frequent itemsets
hint: use sets and frozensets <- dict keys cant be mutable objects
End of explanation
def calc_metrics(trans, X, Y):
    supp = support(trans, X.union(Y))
    supp_X = support(trans, X)
    supp_Y = support(trans, Y)
    conf = supp / supp_X
    lift = supp / (supp_X*supp_Y)
    return conf, lift, supp
def generate_subsets(items, sets):
    subs = {}
    for s in sets:
        for item in items:
            sup = frozenset(s.difference(item))
            if sup != s and sup not in subs:
                subs[sup] = 1
    return list(subs.keys())
def association_rules(trans, frequent, minconf):
    rules = {}
    # get items list
    items = list(frequent)
    # initialize list of antecedents to check
    curr_antecedents = generate_subsets(items, [frequent])
    # iterate till itemset length - 1
    for i in range(len(frequent)-1):
        # print(i, curr_rules)
        # initialize candidates itemsets for generation of subsets
        next_antecedents = []
        # initialize current iteration unfrequent itemsets list
        unconfident = []
        for ant in curr_antecedents:
            cons = set(items).difference(ant)
            conf, lift, supp = calc_metrics(trans, ant, cons)
            # print(ant, conf)
            # if we are over minconf add rule to rules list and to subsets generation candidates
            if conf >= minconf:
                rule_ant = ', '.join('{}'.format(a) for a in (list(ant)))
                rule_cons = ', '.join('{}'.format(c) for c in (list(cons)))
                rule = '{{{}}}->{{{}}}'.format(rule_ant, rule_cons)
                metrics = {}
                metrics['confidence'] = conf
                metrics['lift'] = lift
                metrics['support'] = supp
                rules[rule] = metrics
                next_antecedents.append(frozenset(ant))
            #else add to unconfident list
            else:
                unconfident.append(frozenset(ant))
        # generate subsets and exclude those that are contained in an unconfident rule
        curr_antecedents = generate_subsets(items, next_antecedents)
        for uncf in unconfident:
            curr_antecedents = [ant for ant in curr_antecedents if not uncf.issuperset(ant)]
        # print(next_antecedents)
        # print(unconfident)
        # print(curr_antecedents)
        if len(curr_antecedents) == 0:
            break
    return rules
association_rules(transactions, frozenset({'A', 'B', 'C'}), 0.2)
def get_all_rules(trans, frequent_itemsets, minconf):
    rules_dict = {}
    for frequent in frequent_itemsets:
        rules = association_rules(trans, frequent, minconf)
        rules_dict.update(rules)
    rules_df = pd.DataFrame(rules_dict).T.reset_index()
    rules_df['antecedent'] = rules_df['index'].apply(lambda x: x.split('->')[0])
    rules_df['consequent'] = rules_df['index'].apply(lambda x: x.split('->')[1])
    rules_df.drop('index', axis=1, inplace=True)
    return rules_dict, rules_df
frequent_itemsets = apriori(transactions, 0.1)
rules_dict, rules_df = get_all_rules(transactions, frequent_itemsets, 0.3)
rules_dict
rules_df
Explanation: 5. Create a function that uses the frequent itemsets to generate association rules
must have a min confidence level as a parameter
do not keep rules that fall under this threshold
return dict of rules with string for keys
string key will take a form like: "{'A', 'B'} -> {'C'}"
values must be a dict containing metric:value pairs where metrics include support, confidence, and lift
return dataframe of rules with the following columns
antecedent, consequent, support, confidence, lift
Note: the support is for the entire set: antecedent union consequent
End of explanation
x = rules_df['antecedent'].astype('category')
y = rules_df['consequent'].astype('category')
fig, ax = plt.subplots(figsize=(12, 10))
sct = ax.scatter(x.cat.codes, y.cat.codes, s=rules_df['support']*10000, c=rules_df['lift'])
ax.xaxis.set_major_locator(plt.MaxNLocator(len(x.unique())))
ax.xaxis.set_ticklabels(np.append([''], x.cat.categories))
ax.yaxis.set_major_locator(plt.MaxNLocator(len(y.unique())))
ax.yaxis.set_ticklabels(np.append([''], y.cat.categories))
fig.colorbar(sct);
Explanation: 6. Create a scatter plot to analyze the association rules
one axis for antecedent and one for consequent
set marker size based on support (use a multiplier to scale these up)
set marker color based on lift
hint: convert axes columns to numeric codes using .astype('category').cat.codes
End of explanation
<END_TASK>