<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 The Data Science Pilot Action Set
The dataSciencePilot action set consists of actions that implement a policy-based, configurable, and scalable approach to automating data science workflows. This action set can be used to automate an end-to-end workflow or to automate steps in the  workflow such as data preparation, feature preprocessing, feature engineering, feature selection, and hyperparameter tuning.  More information about this action set is available on its documentation page.
Table of Contents
Today we will set up the notebook and go through each of the seven actions. 
Setting Up the Notebook
Explore Data
Explore Correlations
Analyze Missing Patterns
Detect Interactions
Screen Variables
Feature Machine
Generate Shadow Features
Select Features
Data Science Automated Machine Learning Pipeline
Conclusion
Setting Up the Notebook
First, we must import the Scripting Wrapper for Analytics Transfer (SWAT) package and use the package to connect to out Cloud Analytics Service (CAS).
Step1:  Now we will load the dataSciencePilot action set and the decisionTree action set.
Step2:  Next, we must  connect to our data source. We are using a data set for predicting home equity loan defaults.
Step3:  Our target is “BAD” meaning that it was a bad loan. I am setting up a variable to hold our target information as well as our policy information. Each policy is applicable to specific actions and I will provide more information about each policy later in the notebook.
Step4:  Explore Data
The exploreData action calculates various statistical measures for each column in your data set such as Minimum, Maximum, Mean, Median, Mode, Number Missing, Standard Deviation, and more. The exploreData action also creates a hierarchical variable grouping with two levels. The first level groups variables according to their data type (interval, nominal, data, time, or datetime). The second level uses the following statistical metrics to group the interval and nominal data
Step5:  Explore Correlations
If a target is specified, the exploreCorrelation action performs a linear and nonlinear correlation analysis of the input variables and the target. If a target is not specified, the exploreCorrelation action performs a linear and nonlinear correlation analysis for all pairwise combinations of the input variables. The correlation statistics available depend on the data type of each input variable in the pair. 
- Nominal-nominal correlation pairs have the following statistics available
Step6:  Analyze Missing Patterns
If the target is specified, the analyzeMissingPatterns action performs a missing pattern analysis of the input variables and the target. If a target is not specified, the analyzeMissingPatterns action performs a missing pattern analysis for all pairwise combinations of the input variables. This analysis provides the correlation strength between missing patterns across variable pairs and dependencies of missingness in one variable and the values of the other variable. This action returns a CAS table listing all the missing variable pairs and the statistics around missingness.
Step7:  Detect Interactions
The detectInteractions action will assess the interactions between pairs of predictor variables and the correlation of that interaction on the response variable. Specially, it will see if the product of the pair of predictor variables correlate with the response variable. Since checking the correlation between the product of every predictor pair and the response variable can be computationally intensive, this action relies on the XYZ algorithm to search for these interactions efficiently in a high-dimensional space.   
The detectInteractions Action requires that all predictor variables be in a binary format, but the response variable can be numeric, binary, or multi-class.  Additionally, the detectInteractions Action can handle data in a sparse format, such as when predictor variables are encoded using an one-hot-encoding scheme.  In the example below, we will specify that our inputs are sparse. The output tables shows the gamma value for each pair of variables.
Step8:  Screen Variables
The screenVariables action makes one of the following recommendations for each input variable
Step9:  Feature Machine
The featureMachine action creates an automated and parallel generation of features. The featureMachine action first explores the data and groups the input variables into categories with the same statistical profile, like the exploreData action. Next the featureMachine action screens variables to identify noise variables to exclude from further analysis, like the screenVariables action.  Finally, the featureMachine action generates new features by using the available structured pipelines
Step10:  Generate Shadow Features
The generateShadowFeatures Action performs a scalable random permutation of input features to create shadow features. The shadow features are randomly selected from a matching distribution of each input feature. These shadow features can be used for all-relevant feature selection which removes the inputs whose variable importance is lower than the shadow feature’s variable importance. The shadow features can also be used in a post-fit analysis using Permutation Feature Importance (PFI). By replacing each input with its shadow feature one-by-one and measuring the change on model performance, one can determine that features importance based on relative size of the model’s performance change.  
In the example below, I will use the outputs of the feature machine for all-relevant feature selection. This involves getting the variable metadata from my feature machine table, generating my shadow features, finding the variable importance for my features and shadow features using a random forest, and comparing each variable's performance to its shadow features. In the end, I will only keep variables with a higher importance than its shadow feature for the next phase.
Step11:  Select Features
The selectFeatures action performs a filter-based selection by the criterion selected in the selectionPolicy (default is the best ten input variables according to the Mutual Information statistic). The criterion available for selection include Chi-Square, Cramer’s V, F-test, G2, Information Value, Mutual Information, Normalized Mutual Information statistic, Pearson correlation, and the Symmetric Uncertainty statistic. This action returns a CAS table listing the variables, their rank according to the selected criterion, and the value of the selected criterion.
Step12:  Data Science Automated Machine Learning Pipeline
The dsAutoMl action creates a policy-based, scalable, end-to-end automated machine learning pipeline for both regression and classification problems. The only input required from the user is the input data set and the target variable, but optional parameters include the policy parameters for data exploration, variable screening, feature selection, and feature transformation.  Overriding the default policy parameters allow a data scientist to configure their pipeline in their data science workflow. In addition, a data scientist may also select additional models to consider. By default, only a decision tree model is included in the pipeline, but neural networks, random forest models, and gradient boosting models are also available. 
The dsAutoMl action first explores the data and groups the input variables into categories with the same statistical profile, like the exploreData action. Next the dsAutoMl action screens variables to identify noise variables to exclude from further analysis, like the screenVariables action.  Then, the dsAutoMl action generates several new features for the input variables, like the featureMachine action. After there are various new cleaned features, the dsAutoMl action will select features based on selected criterion, like the selectFeatures action. 
From here, various pipelines are created using subsets of the selected features, chosen for each pipeline using a feature-representation algorithm. Then the chosen models are added to each pipeline and the hyperparameters for the selected models are optimized, like the modelComposer action of the Autotune action set. These hyperparameters are optimized for the selected objective parameter when cross-validated. By default, classification problems are optimized to have the smallest Misclassification Error Rate (MCE) and regression problems are optimized to have the smallest Average Square Error (ASR).  Data scientists can then select their champion and challenger models from the pipelines. 
This action returns several CAS tables
Step13:  Conclusion
The dataSciencePilot action set consists of actions that implement a policy-based, configurable, and scalable approach to automating data science workflows. This action set can be used to automate and end-to-end workflow or to automate steps in the  workflow such as data preparation, feature preprocessing, feature engineering, feature selection, and hyperparameter tuning.  In this notebook, we demonstrated how to use each step of the dataSciencePilot Action set using a Python interface.
<ASSISTANT_TASK:>
Python Code:
import swat
import numpy as np
import pandas as pd
conn = swat.CAS('localhost', 5570, authinfo='~/.authinfo', caslib="CASUSER")
Explanation: The Data Science Pilot Action Set
The dataSciencePilot action set consists of actions that implement a policy-based, configurable, and scalable approach to automating data science workflows. This action set can be used to automate an end-to-end workflow or to automate steps in the  workflow such as data preparation, feature preprocessing, feature engineering, feature selection, and hyperparameter tuning.  More information about this action set is available on its documentation page.
Table of Contents
Today we will set up the notebook and go through each of the seven actions. 
Setting Up the Notebook
Explore Data
Explore Correlations
Analyze Missing Patterns
Detect Interactions
Screen Variables
Feature Machine
Generate Shadow Features
Select Features
Data Science Automated Machine Learning Pipeline
Conclusion
Setting Up the Notebook
First, we must import the Scripting Wrapper for Analytics Transfer (SWAT) package and use the package to connect to out Cloud Analytics Service (CAS).
End of explanation
conn.builtins.loadactionset('dataSciencePilot')
conn.builtins.loadactionset('decisionTree')
Explanation: Now we will load the dataSciencePilot action set and the decisionTree action set.
End of explanation
tbl = 'hmeq'
hmeq = conn.read_csv("./data/hmeq.csv", casout=dict(name=tbl, replace=True))
hmeq.head()
Explanation: Next, we must  connect to our data source. We are using a data set for predicting home equity loan defaults.
End of explanation
# Target Name 
trt='BAD'
# Exploration Policy 
expo = {'cardinality': {'lowMediumCutoff':40}}
# Screen Policy 
scpo = {'missingPercentThreshold':35}
# Selection Policy 
sepo = {'criterion': 'SU', 'topk':4}
# Transformation Policy 
trpo = {'entropy': True, 'iqv': True, 'kurtosis': True, 'outlier': True}
Explanation: Our target is “BAD” meaning that it was a bad loan. I am setting up a variable to hold our target information as well as our policy information. Each policy is applicable to specific actions and I will provide more information about each policy later in the notebook.
End of explanation
conn.dataSciencePilot.exploreData(   
        table  = tbl,
        target = trt,     
        casOut = {'name': 'EXPLORE_DATA_OUT_PY', 'replace' : True},
        explorationPolicy = expo
    )
conn.fetch(table = {'name': 'EXPLORE_DATA_OUT_PY'})
Explanation: Explore Data
The exploreData action calculates various statistical measures for each column in your data set such as Minimum, Maximum, Mean, Median, Mode, Number Missing, Standard Deviation, and more. The exploreData action also creates a hierarchical variable grouping with two levels. The first level groups variables according to their data type (interval, nominal, data, time, or datetime). The second level uses the following statistical metrics to group the interval and nominal data:
- Missing rate (interval and nominal).
- Cardinality (nominal). 
- Entropy (nominal). 
- Index of Qualitative Variation(IQV; interval and nominal). 
- Skewness (interval).
- Kurtosis (interval).
- Outliers (interval).
- Coefficient of Variation (CV; interval).
This action returns a CAS table listing all the variables, the variable groupings, and the summary statistics. These groupings allow for a pipelined approach to data transformation and cleaning.
End of explanation
conn.dataSciencePilot.exploreCorrelation(
        table = tbl, 
        casOut = {'name':'CORR_PY', 'replace':True},
        target = trt
)
conn.fetch(table = {"name" : "CORR_PY"})
Explanation: Explore Correlations
If a target is specified, the exploreCorrelation action performs a linear and nonlinear correlation analysis of the input variables and the target. If a target is not specified, the exploreCorrelation action performs a linear and nonlinear correlation analysis for all pairwise combinations of the input variables. The correlation statistics available depend on the data type of each input variable in the pair. 
- Nominal-nominal correlation pairs have the following statistics available: Mutual Information (MI), Symmetric Uncertainty (SU), Information Value (IV; for binary target), Entropy, chi-square, G test (G2), and Cramer’s V. 
- Nominal-interval correlation pairs have the following statistics available: Mutual Information (MI), Symmetric Uncertainty (SU), Entropy, and F-test. 
- Interval-interval correlation pairs have the following statistics available: Mutual Information (MI), Symmetric Uncertainty (SU), Entropy, and Pearson correlation. 
This action returns a CAS table listing all the variable pairs and the correlation statistics.
End of explanation
conn.dataSciencePilot.analyzeMissingPatterns(
        table = tbl, 
        target = trt, 
        casOut = {'name':'MISS_PATTERN_PY', 'replace':True}
)
conn.fetch(table = {'name': 'MISS_PATTERN_PY'})
Explanation: Analyze Missing Patterns
If the target is specified, the analyzeMissingPatterns action performs a missing pattern analysis of the input variables and the target. If a target is not specified, the analyzeMissingPatterns action performs a missing pattern analysis for all pairwise combinations of the input variables. This analysis provides the correlation strength between missing patterns across variable pairs and dependencies of missingness in one variable and the values of the other variable. This action returns a CAS table listing all the missing variable pairs and the statistics around missingness.
End of explanation
# Tranform data for binary format
conn.dataPreprocess.transform(
    table = hmeq, 
    copyVars = ["BAD"], 
    casOut = {"name": "hmeq_transform", "replace": True}, 
    requestPackages = [{"inputs":["JOB", "REASON"], 
                        "catTrans":{"method": "label", "arguments":{"overrides":{"binMissing": True}}}}, 
                      {"inputs":["MORTDUE", "DEBTINC", "LOAN"], 
                       "discretize": {"method": "quantile", "arguments":{"overrides":{"binMissing": True}}} }])
conn.fetch(table = {'name': 'hmeq_transform'})
conn.dataSciencePilot.detectInteractions(
    table ='hmeq_transform', 
    target = trt, 
    event = '1', 
    sparse = True, 
    inputs = ["_TR1_JOB", "_TR1_REASON", "_TR2_MORTDUE", "_TR2_DEBTINC", "_TR2_LOAN"], 
    inputLevels = [7, 3, 6, 6, 6], 
    casOut = {'name': 'DETECT_INT_OUT_PY', 'replace': True})
conn.fetch(table={'name':'DETECT_INT_OUT_PY'})
Explanation: Detect Interactions
The detectInteractions action will assess the interactions between pairs of predictor variables and the correlation of that interaction on the response variable. Specially, it will see if the product of the pair of predictor variables correlate with the response variable. Since checking the correlation between the product of every predictor pair and the response variable can be computationally intensive, this action relies on the XYZ algorithm to search for these interactions efficiently in a high-dimensional space.   
The detectInteractions Action requires that all predictor variables be in a binary format, but the response variable can be numeric, binary, or multi-class.  Additionally, the detectInteractions Action can handle data in a sparse format, such as when predictor variables are encoded using an one-hot-encoding scheme.  In the example below, we will specify that our inputs are sparse. The output tables shows the gamma value for each pair of variables.
End of explanation
conn.dataSciencePilot.screenVariables(
    table = tbl, 
    target = trt, 
    casOut = {'name': 'SCREEN_VARIABLES_OUT_PY', 'replace': True}, 
    screenPolicy = {}
)
conn.fetch(table = {'name': 'SCREEN_VARIABLES_OUT_PY'})
Explanation: Screen Variables
The screenVariables action makes one of the following recommendations for each input variable:
-   Remove variable if there are significant data-quality issues. 
-   Transform and keep variable if there are some data-quality issues. 
-   Keep variable if there are no data quality issues. 
The screenVariables action considers the following features of the input variables to make its recommendation:
-   Missing rate exceeds  threshold in screenPolicy (default is 90). 
-   Constant value across input variable.
-   Mutual Information (MI) about the target is below the threshold in the screenPolicy (default is 0.05)
-   Entropy across levels. 
-   Entropy reduction of target exceeds threshold in screenPolicy (default is 90); also referred to as leakage. 
-   Symmetric Uncertainty (SU) of two variables exceed threshold in screenPolicy (default is 1); also referred to as redundancy. 
This action returns a CAS table listing all the input variables, the recommended action, and the reason for the recommended action.
End of explanation
conn.dataSciencePilot.featureMachine(
    table = tbl, 
    target = trt, 
    copyVars = trt, 
    explorationPolicy = expo, 
    screenPolicy = scpo, 
    transformationPolicy = trpo, 
    transformationOut       = {"name" : "TRANSFORMATION_OUT", "replace" : True},
    featureOut              = {"name" : "FEATURE_OUT", "replace" : True},
    casOut                  = {"name" : "CAS_OUT", "replace" : True},
    saveState               = {"name" : "ASTORE_OUT", "replace" : True}  
)
conn.fetch(table = {'name': 'TRANSFORMATION_OUT'})
conn.fetch(table = {'name': 'FEATURE_OUT'})
conn.fetch(table = {'name': 'CAS_OUT'})
Explanation: Feature Machine
The featureMachine action creates an automated and parallel generation of features. The featureMachine action first explores the data and groups the input variables into categories with the same statistical profile, like the exploreData action. Next the featureMachine action screens variables to identify noise variables to exclude from further analysis, like the screenVariables action.  Finally, the featureMachine action generates new features by using the available structured pipelines:
-   Missing indicator addition. 
-   Mode imputation and rare value grouping. 
-   Missing level and rare value grouping. 
-   Median imputation. 
-   Mode imputation and label encoding. 
-   Missing level and label encoding. 
-   Yeo-Johnson transformation and median imputation. 
-   Box-Cox transformation. 
-   Quantile binning with missing bins.
-   Regression tree binning.
-   Decision tree binning. 
-   MDLP binning. 
-   Target encoding. 
-   Date, time, and datetime transformations. 
Depending on the parameters specified in the transformationPolicy, the featureMachine action can generate several features for each input variable. This action returns four CAS tables: the first lists information around the transformation pipelines, the second lists information around the transformed features, the third is the input table scored with the transformed features, and the fourth is an analytical store for scoring any additional input tables.
End of explanation
# Getting variable names and metadata from feature machine output
fm = conn.CASTable('FEATURE_OUT').to_frame()
inputs = fm['Name'].to_list()
nom = fm.loc[fm['IsNominal'] == 1]
nom = nom['Name'].to_list()
# Generating Shadow Features
conn.dataSciencePilot.generateShadowFeatures(
    table = 'CAS_OUT', 
    nProbes = 2, 
    inputs = inputs, 
    nominals = nom,
    casout={"name" : "SHADOW_FEATURES_OUT", "replace" : True},
    copyVars = trt
)
conn.fetch(table = {"name" : "SHADOW_FEATURES_OUT"})
# Getting Feature Importance for Orginal Features
feats = conn.decisionTree.forestTrain(
    table = 'CAS_OUT', 
    inputs = inputs, 
    target = trt, 
    varImp = True)
real_features = feats.DTreeVarImpInfo
# Getting Feature Importance for Shadow Features
inp = conn.CASTable('SHADOW_FEATURES_OUT').axes[1].to_list()
shadow_feats = conn.decisionTree.forestTrain(
    table = 'SHADOW_FEATURES_OUT', 
    inputs = inp, 
    target = trt, 
    varImp = True)
sf = shadow_feats.DTreeVarImpInfo
# Building dataframe for easy comparison 
feat_comp = pd.DataFrame(columns=['Variable', 'Real_Imp', 'SF_Imp1', 'SF_Imp2'])
# Filling Variable Column of Data Frame from Feature
feat_comp['Variable'] = real_features['Variable']
# Filling Importance Column of Data Frame from Feature
feat_comp['Real_Imp'] = real_features['Importance']
# Finding each Feature's Shadow Feature
for index, row in sf.iterrows():
    temp_name = row['Variable']
    temp_num = int(temp_name[-1:])
    temp_name = temp_name[5:-2]
    temp_imp = row['Importance']
    for ind, ro in feat_comp.iterrows():
        if temp_name == ro['Variable']:
            if temp_num == 1:
                # Filling First Shadow Feature's Importance
                feat_comp.at[ind, 'SF_Imp1'] = temp_imp
            else:
                # Filling First Shadow Feature's Importance
                feat_comp.at[ind, 'SF_Imp2'] = temp_imp
feat_comp.head()
# Determining which features have an importance smaller than their shadow feature's importance
to_drop = list()
for ind, ro in feat_comp.iterrows():
    if ro['Real_Imp'] <= ro['SF_Imp1'] or ro['Real_Imp'] <= ro['SF_Imp2']:
        to_drop.append(ro['Variable'])
to_drop
# Dropping Columns from CAS_OUT
CAS_OUT=conn.CASTable('CAS_OUT')
CAS_OUT = CAS_OUT.drop(to_drop, axis=1)
Explanation: Generate Shadow Features
The generateShadowFeatures Action performs a scalable random permutation of input features to create shadow features. The shadow features are randomly selected from a matching distribution of each input feature. These shadow features can be used for all-relevant feature selection which removes the inputs whose variable importance is lower than the shadow feature’s variable importance. The shadow features can also be used in a post-fit analysis using Permutation Feature Importance (PFI). By replacing each input with its shadow feature one-by-one and measuring the change on model performance, one can determine that features importance based on relative size of the model’s performance change.  
In the example below, I will use the outputs of the feature machine for all-relevant feature selection. This involves getting the variable metadata from my feature machine table, generating my shadow features, finding the variable importance for my features and shadow features using a random forest, and comparing each variable's performance to its shadow features. In the end, I will only keep variables with a higher importance than its shadow feature for the next phase.
End of explanation
conn.dataSciencePilot.screenVariables(
    table='CAS_OUT', 
    target=trt, 
    screenPolicy=scpo, 
    casout={"name" : "SCREEN_VARIABLES_OUT", "replace" : True}
)
conn.fetch(table = {"name" : "SCREEN_VARIABLES_OUT"})
Explanation: Select Features
The selectFeatures action performs a filter-based selection by the criterion selected in the selectionPolicy (default is the best ten input variables according to the Mutual Information statistic). The criterion available for selection include Chi-Square, Cramer’s V, F-test, G2, Information Value, Mutual Information, Normalized Mutual Information statistic, Pearson correlation, and the Symmetric Uncertainty statistic. This action returns a CAS table listing the variables, their rank according to the selected criterion, and the value of the selected criterion.
End of explanation
conn.dataSciencePilot.dsAutoMl(
    table = tbl,
    target = trt, 
    explorationPolicy = expo, 
    screenPolicy = scpo, 
    selectionPolicy = sepo,
    transformationPolicy = trpo,
     modelTypes              = ["decisionTree", "gradboost"],
        objective               = "ASE",
        sampleSize              = 10,
        topKPipelines           = 10,
        kFolds                  = 5,
        transformationOut       = {"name" : "TRANSFORMATION_OUT_PY", "replace" : True},
        featureOut              = {"name" : "FEATURE_OUT_PY", "replace" : True},
        pipelineOut             = {"name" : "PIPELINE_OUT_PY", "replace" : True},
        saveState               = {"modelNamePrefix" : "ASTORE_OUT_PY", "replace" : True, "topK":1} 
)
conn.fetch(table = {"name" : "TRANSFORMATION_OUT_PY"})
conn.fetch(table = {"name" : "FEATURE_OUT_PY"})
conn.fetch(table = {"name" : "PIPELINE_OUT_PY"})
Explanation: Data Science Automated Machine Learning Pipeline
The dsAutoMl action creates a policy-based, scalable, end-to-end automated machine learning pipeline for both regression and classification problems. The only input required from the user is the input data set and the target variable, but optional parameters include the policy parameters for data exploration, variable screening, feature selection, and feature transformation.  Overriding the default policy parameters allow a data scientist to configure their pipeline in their data science workflow. In addition, a data scientist may also select additional models to consider. By default, only a decision tree model is included in the pipeline, but neural networks, random forest models, and gradient boosting models are also available. 
The dsAutoMl action first explores the data and groups the input variables into categories with the same statistical profile, like the exploreData action. Next the dsAutoMl action screens variables to identify noise variables to exclude from further analysis, like the screenVariables action.  Then, the dsAutoMl action generates several new features for the input variables, like the featureMachine action. After there are various new cleaned features, the dsAutoMl action will select features based on selected criterion, like the selectFeatures action. 
From here, various pipelines are created using subsets of the selected features, chosen for each pipeline using a feature-representation algorithm. Then the chosen models are added to each pipeline and the hyperparameters for the selected models are optimized, like the modelComposer action of the Autotune action set. These hyperparameters are optimized for the selected objective parameter when cross-validated. By default, classification problems are optimized to have the smallest Misclassification Error Rate (MCE) and regression problems are optimized to have the smallest Average Square Error (ASR).  Data scientists can then select their champion and challenger models from the pipelines. 
This action returns several CAS tables: the first lists information around the transformation pipelines, the second lists information around the transformed features, the third lists pipeline performance according to the objective parameter and the last tables are analytical stores for creating the feature set and scoring  with our model when new data is available.
End of explanation
conn.close()
Explanation: Conclusion
The dataSciencePilot action set consists of actions that implement a policy-based, configurable, and scalable approach to automating data science workflows. This action set can be used to automate and end-to-end workflow or to automate steps in the  workflow such as data preparation, feature preprocessing, feature engineering, feature selection, and hyperparameter tuning.  In this notebook, we demonstrated how to use each step of the dataSciencePilot Action set using a Python interface.
End of explanation
<END_TASK>