<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 |S.No|  Package | Comments  |
|---|---|---|
|1|  pandas  | provides data structures (such as DataFrame) to <span style="color
Step1:  Read input tables
Step2:  Block tables to get candidate set
Step3:  From the plot we can see that 20003 is definitely an error. We will replace 20003 by 2003.
Step4:  Debug blocking output
Step5:  Match tuple pairs in candidate set
1. Sample candidate set  --> S
2. Label S
3. Split S into development set (I) and evaluation set (J)
4. Select best learning-based matcher X, using I
5. Compute accuracy of X on J
Step6:  Selecting the best learning-based matcher using I
1. Create a set of ML-matchers
2. Generate a set of features (F)
3. Convert I into a set of feature vectors (H) using F
4. Select best learning-based matcher (X) using k-fold cross validation over H
5. Debug X (and repeat the above steps)
Step8:  Debug X (Random Forest)
Step9:  Compute accuracy of X (Decision Tree) on J
1. Train X using  H
2. Convert J into a set of feature vectors (L)
3. Predict on L using X
4. Evaluate the predictions
<ASSISTANT_TASK:>
Python Code:
import py_entitymatching as em
import profiler
import pandas as pd
Explanation: |S.No|  Package | Comments  |
|---|---|---|
|1|  pandas  | provides data structures (such as DataFrame) to <span style="color:red;">store and manage relational data</span>. Specifically, DataFrame is used to represent input tables.  |
|2| scikit-learn   | provides implementations for common <span style="color:red;">machine learning</span> algorithms. Specifically, this is used in ML-based matchers.  |
|3| joblib  | provides <span style="color:red;">multiprocessing</span> capabilities. Specifically, this is used to parallelize blockers across multiple processors.|
|4| PyQt4  | provides tools to <span style="color:red;">build GUI</span>. Specifically, this is used to build GUI for labeling data  and debugging matchers.|
|5|  py_stringsimjoin |  provides <span style="color:red;">scalable</span> implementations for <span style="color:red;">string similarity joins</span> over two tables. Specifically,   this is used to scale blockers. |
|6| py_stringmatching | provides a comprehensive set of <span style="color:red;"> tokenizers and string similarity functions</span>. Specifically, this is to create features for blocking and matching.|
|7|  cloudpickle |provides functions to <span style="color:red;"> serialize Python constructs</span>. Specifically, this is used to load/save objects from/to disk. |
|8| pyprind  | library to display <span style="color:red;"> progress indicators</span>. Specifically, this is used to display progress of blocking functions, matching functions, etc.  |
|9| pyparsing  | library to <span style="color:red;">parse strings</span>. Specifically, this is used to parse rules/features that are declaratively written by the user.  |
|10|  six |  provides functions to <span style="color:red;">write compatible code across Python 2 and 3</span>. |
End of explanation
## Read input tables
A = em.read_csv_metadata('dblp_demo.csv', key='id')
B = em.read_csv_metadata('acm_demo.csv', key='id')
len(A), len(B), len(A) * len(B)
A.head(2)
B.head(2)
# If the tables are large we can downsample the tables like this
A1, B1 = em.down_sample(A, B, 500, 1, show_progress=False)
len(A1), len(B1)
# But for the demo, we will use the entire table A and B
Explanation: Read input tables
End of explanation
profiler.profile_table(A, 'paper year')
profiler.profile_table(B, 'paper year')
Explanation: Block tables to get candidate set
End of explanation
B.replace({'paper year':{
            20003:2003
        }}, inplace=True)
### Blocking plan
### A, B -- AttrEquivalence blocker [year]--------------------------| Candidate set
# Create attribute equivalence blocker
ab = em.AttrEquivalenceBlocker()
# Block tables using 'year' attribute : same year include in candidate set
C1 = ab.block_tables(A, B, 'paper year', 'paper year', 
                     l_output_attrs=['title', 'authors', 'paper year'],
                     r_output_attrs=['title', 'authors', 'paper year']
                    )
len(C1)
C1.head(2)
Explanation: From the plot we can see that 20003 is definitely an error. We will replace 20003 by 2003.
End of explanation
# check whether the current blocking method has dropped a lot of potential matches
dbg = em.debug_blocker(C1, A, B)
dbg.head()
# em.view_table(dbg)
# Revised blocking plan
# A, B -- AttrEquivalence blocker [year] --------------------|
#                                                           |---> candidate set
# A, B -- Overlap blocker [title]---------------------------|
profiler.profile_table(A, 'title', plot=False)
profiler.profile_table(B, 'title', plot=False)
# Initialize overlap blocker
ob = em.OverlapBlocker()
# Block over title attribute
C2 = ob.block_tables(A, B, 'title', 'title', show_progress=False, overlap_size=1)
len(C2)
# Combine the outputs from attr. equivalence blocker and overlap blocker
C = em.combine_blocker_outputs_via_union([C1, C2])
len(C)
# Check again to see if we are dropping any potential matches
dbg = em.debug_blocker(C, A, B)
dbg.head()
Explanation: Debug blocking output
End of explanation
# Sample  candidate set
S = em.sample_table(C, 450)
# Label S
S = em.label_table(S, 'label')
# Load the pre-labeled data
S = em.read_csv_metadata('labeled_data_demo.csv', 
                         key='_id',
                         ltable=A, rtable=B, 
                         fk_ltable='ltable_id', fk_rtable='rtable_id')
len(S)
# Split S into I an J
IJ = em.split_train_test(S, train_proportion=0.5, random_state=0)
I = IJ['train']
J = IJ['test']
Explanation: Match tuple pairs in candidate set
1. Sample candidate set  --> S
2. Label S
3. Split S into development set (I) and evaluation set (J)
4. Select best learning-based matcher X, using I
5. Compute accuracy of X on J
End of explanation
# Create a set of ML-matchers
dt = em.DTMatcher(name='DecisionTree', random_state=0)
svm = em.SVMMatcher(name='SVM', random_state=0)
rf = em.RFMatcher(name='RF', random_state=0)
lg = em.LogRegMatcher(name='LogReg', random_state=0)
ln = em.LinRegMatcher(name='LinReg')
# Generate a set of features
F = em.get_features_for_matching(A, B)
# List the feature names generated
F['feature_name']
# Convert the I into a set of feature vectors using F
H = em.extract_feature_vecs(I, 
                            feature_table=F, 
                            attrs_after='label',
                            show_progress=False) 
# Select the best ML matcher using CV
result = em.select_matcher([dt, rf, svm, ln, lg], table=H, 
        exclude_attrs=['_id', 'ltable_id', 'rtable_id', 'label'],
        k=5,
        target_attr='label', metric='f1', random_state=0)
result['cv_stats']
Explanation: Selecting the best learning-based matcher using I
1. Create a set of ML-matchers
2. Generate a set of features (F)
3. Convert I into a set of feature vectors (H) using F
4. Select best learning-based matcher (X) using k-fold cross validation over H
5. Debug X (and repeat the above steps)
End of explanation
# Split H into P and Q
PQ = em.split_train_test(H, train_proportion=0.5, random_state=0)
P = PQ['train']
Q = PQ['test']
# Debug RF matcher using GUI
em.vis_debug_rf(rf, P, Q, 
        exclude_attrs=['_id', 'ltable_id', 'rtable_id', 'label'],
        target_attr='label')
# Add a feature to do Jaccard on title + authors and add it to F
# Create a feature declaratively
sim = em.get_sim_funs_for_matching()
tok = em.get_tokenizers_for_matching()
feature_string = jaccard(wspace((ltuple['title'] + ' ' + ltuple['authors']).lower()), 
                            wspace((rtuple['title'] + ' ' + rtuple['authors']).lower()))
feature = em.get_feature_fn(feature_string, sim, tok)
# Add feature to F
em.add_feature(F, 'jac_ws_title_authors', feature)
# Print supported sim. functions
pd.DataFrame({'simfunctions':sorted(sim.keys())})
# Print supported tokenizers
pd.DataFrame({'tokenizers':sorted(tok.keys())})
F['feature_name']
# Convert I into feature vectors using updated F
H = em.extract_feature_vecs(I, 
                            feature_table=F, 
                            attrs_after='label',
                            show_progress=False) 
# Check whether the updated F improves X (Random Forest)
result = em.select_matcher([rf], table=H, 
        exclude_attrs=['_id', 'ltable_id', 'rtable_id', 'label'],
        k=5,
        target_attr='label', metric='f1', random_state=0)
result['cv_stats']
# Select the best matcher again using CV
result = em.select_matcher([dt, rf, svm, ln, lg], table=H, 
        exclude_attrs=['_id', 'ltable_id', 'rtable_id', 'label'],
        k=5,
        target_attr='label', metric='f1', random_state=0)
result['cv_stats']
Explanation: Debug X (Random Forest)
End of explanation
# Train using feature vectors from I 
dt.fit(table=H, 
       exclude_attrs=['_id', 'ltable_id', 'rtable_id', 'label'], 
       target_attr='label')
# Convert J into a set of feature vectors using F
L = em.extract_feature_vecs(J, feature_table=F,
                            attrs_after='label', show_progress=False)
# Predict on L 
predictions = dt.predict(table=L, exclude_attrs=['_id', 'ltable_id', 'rtable_id', 'label'], 
              append=True, target_attr='predicted', inplace=False)
# Evaluate the predictions
eval_result = em.eval_matches(predictions, 'label', 'predicted')
em.print_eval_summary(eval_result)
Explanation: Compute accuracy of X (Decision Tree) on J
1. Train X using  H
2. Convert J into a set of feature vectors (L)
3. Predict on L using X
4. Evaluate the predictions
End of explanation
<END_TASK>