<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Feature Selection
Featuretools provides users with the ability to remove features that are unlikely to be useful in building an effective machine learning model. Reducing the number of features in the feature matrix can both produce better results in the model as well as reduce the computational cost involved in prediction.
Featuretools enables users to perform feature selection on the results of Deep Feature Synthesis with three functions
Step1:  Remove Highly Null Features
We might have a dataset with columns that have many null values. Deep Feature Synthesis might build features off of those null columns, creating even more highly null features. In this case, we might want to remove any features whose null values pass a certain threshold. Below is our feature matrix with such a case
Step2:  We look at the above feature matrix and decide to remove the highly null features
Step3:  Notice that calling remove_highly_null_features didn't remove every feature that contains a null value. By default, we only remove features where the percentage of null values in the calculated feature matrix is above 95%. If we want to lower that threshold, we can set the pct_null_threshold paramter ourselves.
Step4:  Remove Single Value Features
Another situation we might run into is one where our calculated features don't have any variance. In those cases, we are likely to want to remove the uninteresting features. For that, we use remove_single_value_features.
Let's see what happens when we remove the single value features of the feature matrix below.
Step5:  Now that we have the features definitions for the updated feature matrix, we can see that the features that were removed are
Step6:  With the function used as it is above, null values are not considered when counting a feature's unique values. If we'd like to consider NaN its own value, we can set count_nan_as_value to True and we'll see flights.carrier and flights.flight_num back in the matrix.
Step7:  The features that were removed are
Step8:  Remove Highly Correlated Features
The last feature selection function we have allows us to remove features that would likely be redundant to the model we're attempting to build by considering the correlation between pairs of calculated features.
When two features are determined to be highly correlated, we remove the more complex of the two. For example, say we have two features
Step9:  Note that we have some pretty clear correlations here between all the features and their negations.
Now, using remove_highly_correlated_features, our default threshold for correlation is 95% correlated, and we get all of the obviously correlated features removed, leaving just the less complex features.
Step10:  The features that were removed are
Step11:  Change the correlation threshold
We can lower the threshold at which to remove correlated features if we'd like to be more restrictive by using the pct_corr_threshold parameter.
Step12:  The features that were removed are
Step13:  Check a Subset of Features
If we only want to check a subset of features, we can set features_to_check to the list of features whose correlation we'd like to check, and no features outside of that list will be removed.
Step14:  The features that were removed are
Step15:  Protect Features from Removal
To protect specific features from being removed from the feature matrix, we can include a list of features_to_keep, and these features will not be removed
Step16:  The features that were removed are
<ASSISTANT_TASK:>
Python Code:
import pandas as pd
import featuretools as ft
from featuretools.selection import (
    remove_highly_correlated_features,
    remove_highly_null_features,
    remove_single_value_features,
)
from featuretools.primitives import NaturalLanguage
from featuretools.demo.flight import load_flight
es = load_flight(nrows=50)
es
Explanation: Feature Selection
Featuretools provides users with the ability to remove features that are unlikely to be useful in building an effective machine learning model. Reducing the number of features in the feature matrix can both produce better results in the model as well as reduce the computational cost involved in prediction.
Featuretools enables users to perform feature selection on the results of Deep Feature Synthesis with three functions:
ft.selection.remove_highly_null_features
ft.selection.remove_single_value_features
ft.selection.remove_highly_correlated_features
We will describe each of these functions in depth, but first we must create an entity set with which we can run ft.dfs.
End of explanation
fm, features = ft.dfs(entityset=es,
                          target_dataframe_name="trip_logs",
                          cutoff_time=pd.DataFrame({
                              'trip_log_id':[30, 1, 2, 3, 4],
                              'time':pd.to_datetime(['2016-09-22 00:00:00']*5)
                                                  }),
                          trans_primitives=[],
                          agg_primitives=[],
                          max_depth=2)
fm
Explanation: Remove Highly Null Features
We might have a dataset with columns that have many null values. Deep Feature Synthesis might build features off of those null columns, creating even more highly null features. In this case, we might want to remove any features whose null values pass a certain threshold. Below is our feature matrix with such a case:
End of explanation
ft.selection.remove_highly_null_features(fm)
Explanation: We look at the above feature matrix and decide to remove the highly null features
End of explanation
remove_highly_null_features(fm, pct_null_threshold=.2)
Explanation: Notice that calling remove_highly_null_features didn't remove every feature that contains a null value. By default, we only remove features where the percentage of null values in the calculated feature matrix is above 95%. If we want to lower that threshold, we can set the pct_null_threshold paramter ourselves.
End of explanation
fm
new_fm, new_features = remove_single_value_features(fm, features=features)
new_fm
Explanation: Remove Single Value Features
Another situation we might run into is one where our calculated features don't have any variance. In those cases, we are likely to want to remove the uninteresting features. For that, we use remove_single_value_features.
Let's see what happens when we remove the single value features of the feature matrix below.
End of explanation
set(features) - set(new_features)
Explanation: Now that we have the features definitions for the updated feature matrix, we can see that the features that were removed are:
End of explanation
new_fm, new_features = remove_single_value_features(fm, features=features, count_nan_as_value=True)
new_fm
Explanation: With the function used as it is above, null values are not considered when counting a feature's unique values. If we'd like to consider NaN its own value, we can set count_nan_as_value to True and we'll see flights.carrier and flights.flight_num back in the matrix.
End of explanation
set(features) - set(new_features)
Explanation: The features that were removed are:
End of explanation
fm, features = ft.dfs(entityset=es,
                          target_dataframe_name="trip_logs",
                          trans_primitives=['negate'],
                          agg_primitives=[],
                          max_depth=3)
fm.head()
Explanation: Remove Highly Correlated Features
The last feature selection function we have allows us to remove features that would likely be redundant to the model we're attempting to build by considering the correlation between pairs of calculated features.
When two features are determined to be highly correlated, we remove the more complex of the two. For example, say we have two features: col and -(col).
We can see that -(col) is just the negation of col, and so we can guess those features are going to be highly correlated. -(col) has has the Negate primitive applied to it, so it is more complex than the identity feature col. Therefore, if we only want one of col and -(col), we should keep the identity feature. For features that don't have an obvious difference in complexity, we discard the feature that comes later in the feature matrix. 
Let's try this out on our data:
End of explanation
new_fm, new_features = remove_highly_correlated_features(fm, features=features)
new_fm.head()
Explanation: Note that we have some pretty clear correlations here between all the features and their negations.
Now, using remove_highly_correlated_features, our default threshold for correlation is 95% correlated, and we get all of the obviously correlated features removed, leaving just the less complex features.
End of explanation
set(features) - set(new_features)
Explanation: The features that were removed are:
End of explanation
new_fm , new_features = remove_highly_correlated_features(fm, features=features, pct_corr_threshold=.9)
new_fm.head()
Explanation: Change the correlation threshold
We can lower the threshold at which to remove correlated features if we'd like to be more restrictive by using the pct_corr_threshold parameter.
End of explanation
set(features) - set(new_features)
Explanation: The features that were removed are:
End of explanation
new_fm, new_features = remove_highly_correlated_features(fm, features=features, features_to_check=['air_time', 'distance', 'flights.distance_group'])
new_fm.head()
Explanation: Check a Subset of Features
If we only want to check a subset of features, we can set features_to_check to the list of features whose correlation we'd like to check, and no features outside of that list will be removed.
End of explanation
set(features) - set(new_features)
Explanation: The features that were removed are:
End of explanation
new_fm, new_features = remove_highly_correlated_features(fm, features=features, features_to_keep=['air_time', 'distance', 'flights.distance_group'])
new_fm.head()
Explanation: Protect Features from Removal
To protect specific features from being removed from the feature matrix, we can include a list of features_to_keep, and these features will not be removed
End of explanation
set(features) - set(new_features)
Explanation: The features that were removed are:
End of explanation
<END_TASK>