<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Basic EM workflow 2 (Restaurants data set)
Introduction
This IPython notebook explains a basic workflow two tables using py_entitymatching. Our goal is to come up with a workflow to match restaurants from Fodors and Zagat sites. Specifically, we want to achieve precision and recall above 96%. The datasets contain information about the restaurants.
First, we need to import py_entitymatching package and other libraries as follows
Step1:  Matching two tables typically consists of the following three steps
Step2:  Block Tables To Get Candidate Set
Before we do the matching, we would like to remove the obviously non-matching tuple pairs from the input tables. This would reduce the number of tuple pairs considered for matching.
py_entitymatching provides four different blockers
Step3:  Match tuple pairs in candidate set
In this step, we would want to match the tuple pairs in the candidate set. Specifically, we use learning-based method for matching purposes.
This typically involves the following four steps
Step4:  Next, we label the sampled candidate set. Specify we would enter 1 for a match and 0 for a non-match.
Step5:  For the purposes of this guide, we will load in a pre-labeled dataset (of 450 tuple pairs) included in this package.
Step6:  Splitting the labeled data into development and evaluation set
In this step, we split the labeled data into two sets
Step7:  Selecting the best learning-based matcher
Selecting the best learning-based matcher typically involves the following steps
Step8:  Creating features
Next, we need to create a set of features for the development set. py_entitymatching provides a way to automatically generate features based on the attributes in the input tables. For the purposes of this guide, we use the automatically generated features.
Step9:  Converting the development set to feature vectors
Step10:  Selecting the best matcher using cross-validation
Now, we select the best matcher using k-fold cross-validation. For the purposes of this guide, we use five fold cross validation and use 'precision' and 'recall' metric to select the best matcher.
Step11:  We observe that the best matcher (RF) is  getting us the best precision and recall. So, we select this matcher and now we can proceed on to evaluating the best matcher on the unseen data (the evaluation set).
Evaluating the matching output
Evaluating the matching outputs for the evaluation set typically involves the following four steps
Step12:  Training the selected matcher
Now, we train the matcher using all of the feature vectors from the development set. For the purposes of this guide we use random forest as the selected matcher.
Step13:  Predicting the matches
Next, we predict the matches for the evaluation set (using the feature vectors extracted from it).
Step14:  Evaluating the predictions
Finally, we evaluate the accuracy of predicted outputs
<ASSISTANT_TASK:>
Python Code:
import sys
sys.path.append('/Users/pradap/Documents/Research/Python-Package/anhaid/py_entitymatching/')
import py_entitymatching as em
import pandas as pd
import os
# Display the versions
print('python version: ' + sys.version )
print('pandas version: ' + pd.__version__ )
print('magellan version: ' + em.__version__ )
Explanation: Basic EM workflow 2 (Restaurants data set)
Introduction
This IPython notebook explains a basic workflow two tables using py_entitymatching. Our goal is to come up with a workflow to match restaurants from Fodors and Zagat sites. Specifically, we want to achieve precision and recall above 96%. The datasets contain information about the restaurants.
First, we need to import py_entitymatching package and other libraries as follows:
End of explanation
# Get the paths
path_A = em.get_install_path() + os.sep + 'datasets' + os.sep + 'end-to-end' + os.sep + 'restaurants/fodors.csv'
path_B = em.get_install_path() + os.sep + 'datasets' + os.sep + 'end-to-end' + os.sep + 'restaurants/zagats.csv'
# Load csv files as dataframes and set the key attribute in the dataframe
A = em.read_csv_metadata(path_A, key='id')
B = em.read_csv_metadata(path_B, key='id')
print('Number of tuples in A: ' + str(len(A)))
print('Number of tuples in B: ' + str(len(B)))
print('Number of tuples in A X B (i.e the cartesian product): ' + str(len(A)*len(B)))
A.head(2)
B.head(2)
# Display the keys of the input tables
em.get_key(A), em.get_key(B)
# If the tables are large we can downsample the tables like this
A1, B1 = em.down_sample(A, B, 200, 1, show_progress=False)
len(A1), len(B1)
# But for the purposes of this notebook, we will use the entire table A and B
Explanation: Matching two tables typically consists of the following three steps:
 1. Reading the input tables 
 2. Blocking the input tables to get a candidate set 
 3. Matching the tuple pairs in the candidate set 
Read input tables
We begin by loading the input tables. For the purpose of this guide, we use the datasets that are included with the package.
End of explanation
# Blocking plan
# A, B -- Overlap blocker [name] --------------------|---> candidate set
# Create overlap blocker
ob = em.OverlapBlocker()
# Block tables using 'name' attribute 
C = ob.block_tables(A, B, 'name', 'name', 
                    l_output_attrs=['name', 'addr', 'city', 'phone'], 
                    r_output_attrs=['name', 'addr', 'city', 'phone'],
                    overlap_size=1, show_progress=False
                    )
len(C)
Explanation: Block Tables To Get Candidate Set
Before we do the matching, we would like to remove the obviously non-matching tuple pairs from the input tables. This would reduce the number of tuple pairs considered for matching.
py_entitymatching provides four different blockers: (1) attribute equivalence, (2) overlap, (3) rule-based, and (4) black-box. The user can mix and match these blockers to form a blocking sequence applied to input tables.
For the matching problem at hand, we know that two restaurants with no overlap between the names will not match. So we decide the apply blocking over names:
End of explanation
# Sample  candidate set
S = em.sample_table(C, 450)
Explanation: Match tuple pairs in candidate set
In this step, we would want to match the tuple pairs in the candidate set. Specifically, we use learning-based method for matching purposes.
This typically involves the following four steps:
Sampling and labeling the candidate set
Splitting the labeled data into development and evaluation set
Selecting the best learning based matcher using the development set
Evaluating the selected matcher using the evaluation set
Sampling and labeling the candidate set
First, we randomly sample 450 tuple pairs for labeling purposes.
End of explanation
# Label S
G = em.label_table(S, 'gold')
Explanation: Next, we label the sampled candidate set. Specify we would enter 1 for a match and 0 for a non-match.
End of explanation
# Load the pre-labeled data
path_G = em.get_install_path() + os.sep + 'datasets' + os.sep + 'end-to-end' + os.sep + 'restaurants/lbl_restnt_wf1.csv'
G = em.read_csv_metadata(path_G, 
                         key='_id',
                         ltable=A, rtable=B, 
                         fk_ltable='ltable_id', fk_rtable='rtable_id')
len(G)
Explanation: For the purposes of this guide, we will load in a pre-labeled dataset (of 450 tuple pairs) included in this package.
End of explanation
# Split S into development set (I) and evaluation set (J)
IJ = em.split_train_test(G, train_proportion=0.7, random_state=0)
I = IJ['train']
J = IJ['test']
Explanation: Splitting the labeled data into development and evaluation set
In this step, we split the labeled data into two sets: development (I) and evaluation (J). Specifically, the development set is used to come up with the best learning-based matcher and the evaluation set used to evaluate the selected matcher on unseen data.
End of explanation
# Create a set of ML-matchers
dt = em.DTMatcher(name='DecisionTree', random_state=0)
svm = em.SVMMatcher(name='SVM', random_state=0)
rf = em.RFMatcher(name='RF', random_state=0)
lg = em.LogRegMatcher(name='LogReg', random_state=0)
ln = em.LinRegMatcher(name='LinReg')
nb = em.NBMatcher(name='NaiveBayes')
Explanation: Selecting the best learning-based matcher
Selecting the best learning-based matcher typically involves the following steps:
Creating a set of learning-based matchers
Creating features
Converting the development set into feature vectors
Selecting the best learning-based matcher using k-fold cross validation
Creating a set of learning-based matchers
End of explanation
# Generate features
feature_table = em.get_features_for_matching(A, B)
Explanation: Creating features
Next, we need to create a set of features for the development set. py_entitymatching provides a way to automatically generate features based on the attributes in the input tables. For the purposes of this guide, we use the automatically generated features.
End of explanation
# Convert the I into a set of feature vectors using F
H = em.extract_feature_vecs(I, 
                            feature_table=feature_table, 
                            attrs_after='gold',
                            show_progress=False) 
# Display first few rows
H.head(3)
Explanation: Converting the development set to feature vectors
End of explanation
# Select the best ML matcher using CV
result = em.select_matcher([dt, rf, svm, ln, lg, nb], table=H, 
        exclude_attrs=['_id', 'ltable_id', 'rtable_id', 'gold'],
        k=5,
        target_attr='gold', metric='precision', random_state=0)
result['cv_stats']
result = em.select_matcher([dt, rf, svm, ln, lg, nb], table=H, 
        exclude_attrs=['_id', 'ltable_id', 'rtable_id', 'gold'],
        k=5,
        target_attr='gold', metric='recall', random_state=0)
result['cv_stats']
Explanation: Selecting the best matcher using cross-validation
Now, we select the best matcher using k-fold cross-validation. For the purposes of this guide, we use five fold cross validation and use 'precision' and 'recall' metric to select the best matcher.
End of explanation
# Convert J into a set of feature vectors using feature table
L = em.extract_feature_vecs(J, feature_table=feature_table,
                            attrs_after='gold', show_progress=False)
Explanation: We observe that the best matcher (RF) is  getting us the best precision and recall. So, we select this matcher and now we can proceed on to evaluating the best matcher on the unseen data (the evaluation set).
Evaluating the matching output
Evaluating the matching outputs for the evaluation set typically involves the following four steps:
1. Converting the evaluation set to feature vectors
2. Training matcher using the feature vectors extracted from the development set
3. Predicting the evaluation set using the trained matcher
4. Evaluating the predicted matches
Converting the evaluation set to  feature vectors
As before, we convert to the feature vectors (using the feature table and the evaluation set)
End of explanation
# Train using feature vectors from I 
dt.fit(table=H, 
       exclude_attrs=['_id', 'ltable_id', 'rtable_id', 'gold'], 
       target_attr='gold')
Explanation: Training the selected matcher
Now, we train the matcher using all of the feature vectors from the development set. For the purposes of this guide we use random forest as the selected matcher.
End of explanation
# Predict on L 
predictions = dt.predict(table=L, exclude_attrs=['_id', 'ltable_id', 'rtable_id', 'gold'], 
              append=True, target_attr='predicted', inplace=False)
Explanation: Predicting the matches
Next, we predict the matches for the evaluation set (using the feature vectors extracted from it).
End of explanation
# Evaluate the predictions
eval_result = em.eval_matches(predictions, 'gold', 'predicted')
em.print_eval_summary(eval_result)
Explanation: Evaluating the predictions
Finally, we evaluate the accuracy of predicted outputs
End of explanation
<END_TASK>