<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Now it's your turn to test your new knowledge of missing values handling. You'll probably find it makes a big difference.
Setup
The questions will give you feedback on your work. Run the following cell to set up the feedback system.
Step1:  In this exercise, you will work with data from the Housing Prices Competition for Kaggle Learn Users. 
Run the next code cell without changes to load the training and validation sets in X_train, X_valid, y_train, and y_valid.  The test set is loaded in X_test.
Step2:  Use the next code cell to print the first five rows of the data.
Step3:  You can already see a few missing values in the first several rows.  In the next step, you'll obtain a more comprehensive understanding of the missing values in the dataset.
Step 1
Step4:  Part A
Use the above output to answer the questions below.
Step5:  Part B
Considering your answers above, what do you think is likely the best approach to dealing with the missing values?
Step6:  To compare different approaches to dealing with missing values, you'll use the same score_dataset() function from the tutorial.  This function reports the mean absolute error (MAE) from a random forest model.
Step7:  Step 2
Step8:  Run the next code cell without changes to obtain the MAE for this approach.
Step9:  Step 3
Step10:  Run the next code cell without changes to obtain the MAE for this approach.
Step11:  Part B
Compare the MAE from each approach.  Does anything surprise you about the results?  Why do you think one approach performed better than the other?
Step12:  Step 4
Step13:  Run the next code cell to train and evaluate a random forest model.  (Note that we don't use the score_dataset() function above, because we will soon use the trained model to generate test predictions!)
Step14:  Part B
Use the next code cell to preprocess your test data.  Make sure that you use a method that agrees with how you preprocessed the training and validation data, and set the preprocessed test features to final_X_test.
Then, use the preprocessed test features and the trained model to generate test predictions in preds_test.
In order for this step to be marked correct, you need only ensure
Step15:  Run the next code cell without changes to save your results to a CSV file that can be submitted directly to the competition.
<ASSISTANT_TASK:>
Python Code:
# Set up code checking
import os
if not os.path.exists("../input/train.csv"):
    os.symlink("../input/home-data-for-ml-course/train.csv", "../input/train.csv")  
    os.symlink("../input/home-data-for-ml-course/test.csv", "../input/test.csv") 
from learntools.core import binder
binder.bind(globals())
from learntools.ml_intermediate.ex2 import *
print("Setup Complete")
Explanation: Now it's your turn to test your new knowledge of missing values handling. You'll probably find it makes a big difference.
Setup
The questions will give you feedback on your work. Run the following cell to set up the feedback system.
End of explanation
import pandas as pd
from sklearn.model_selection import train_test_split
# Read the data
X_full = pd.read_csv('../input/train.csv', index_col='Id')
X_test_full = pd.read_csv('../input/test.csv', index_col='Id')
# Remove rows with missing target, separate target from predictors
X_full.dropna(axis=0, subset=['SalePrice'], inplace=True)
y = X_full.SalePrice
X_full.drop(['SalePrice'], axis=1, inplace=True)
# To keep things simple, we'll use only numerical predictors
X = X_full.select_dtypes(exclude=['object'])
X_test = X_test_full.select_dtypes(exclude=['object'])
# Break off validation set from training data
X_train, X_valid, y_train, y_valid = train_test_split(X, y, train_size=0.8, test_size=0.2,
                                                      random_state=0)
Explanation: In this exercise, you will work with data from the Housing Prices Competition for Kaggle Learn Users. 
Run the next code cell without changes to load the training and validation sets in X_train, X_valid, y_train, and y_valid.  The test set is loaded in X_test.
End of explanation
X_train.head()
Explanation: Use the next code cell to print the first five rows of the data.
End of explanation
# Shape of training data (num_rows, num_columns)
print(X_train.shape)
# Number of missing values in each column of training data
missing_val_count_by_column = (X_train.isnull().sum())
print(missing_val_count_by_column[missing_val_count_by_column > 0])
Explanation: You can already see a few missing values in the first several rows.  In the next step, you'll obtain a more comprehensive understanding of the missing values in the dataset.
Step 1: Preliminary investigation
Run the code cell below without changes.
End of explanation
# Fill in the line below: How many rows are in the training data?
num_rows = ____
# Fill in the line below: How many columns in the training data
# have missing values?
num_cols_with_missing = ____
# Fill in the line below: How many missing entries are contained in 
# all of the training data?
tot_missing = ____
# Check your answers
step_1.a.check()
#%%RM_IF(PROD)%%
num_rows = 1168
num_cols_with_missing = 3
tot_missing = 212 + 6 + 58
step_1.a.assert_check_passed()
# Lines below will give you a hint or solution code
#_COMMENT_IF(PROD)_
step_1.a.hint()
#_COMMENT_IF(PROD)_
step_1.a.solution()
Explanation: Part A
Use the above output to answer the questions below.
End of explanation
# Check your answer (Run this code cell to receive credit!)
step_1.b.check()
#_COMMENT_IF(PROD)_
step_1.b.hint()
Explanation: Part B
Considering your answers above, what do you think is likely the best approach to dealing with the missing values?
End of explanation
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_absolute_error
# Function for comparing different approaches
def score_dataset(X_train, X_valid, y_train, y_valid):
    model = RandomForestRegressor(n_estimators=100, random_state=0)
    model.fit(X_train, y_train)
    preds = model.predict(X_valid)
    return mean_absolute_error(y_valid, preds)
Explanation: To compare different approaches to dealing with missing values, you'll use the same score_dataset() function from the tutorial.  This function reports the mean absolute error (MAE) from a random forest model.
End of explanation
# Fill in the line below: get names of columns with missing values
____ # Your code here
# Fill in the lines below: drop columns in training and validation data
reduced_X_train = ____
reduced_X_valid = ____
# Check your answers
step_2.check()
#%%RM_IF(PROD)%%
# Get names of columns with missing values
cols_with_missing = [col for col in X_train.columns 
                     if X_train[col].isnull().any()]
# Drop columns in training and validation data
reduced_X_train = X_train.drop(cols_with_missing, axis=1)
reduced_X_valid = X_valid.drop(cols_with_missing, axis=1)
step_2.assert_check_passed()
# Lines below will give you a hint or solution code
#_COMMENT_IF(PROD)_
step_2.hint()
#_COMMENT_IF(PROD)_
step_2.solution()
Explanation: Step 2: Drop columns with missing values
In this step, you'll preprocess the data in X_train and X_valid to remove columns with missing values.  Set the preprocessed DataFrames to reduced_X_train and reduced_X_valid, respectively.
End of explanation
print("MAE (Drop columns with missing values):")
print(score_dataset(reduced_X_train, reduced_X_valid, y_train, y_valid))
Explanation: Run the next code cell without changes to obtain the MAE for this approach.
End of explanation
from sklearn.impute import SimpleImputer
# Fill in the lines below: imputation
____ # Your code here
imputed_X_train = ____
imputed_X_valid = ____
# Fill in the lines below: imputation removed column names; put them back
imputed_X_train.columns = ____
imputed_X_valid.columns = ____
# Check your answers
step_3.a.check()
#%%RM_IF(PROD)%%
# Imputation
my_imputer = SimpleImputer()
imputed_X_train = pd.DataFrame(my_imputer.fit_transform(X_train))
imputed_X_valid = pd.DataFrame(my_imputer.transform(X_valid))
step_3.a.assert_check_failed()
#%%RM_IF(PROD)%%
# Imputation
my_imputer = SimpleImputer()
imputed_X_train = pd.DataFrame(my_imputer.fit_transform(X_train))
imputed_X_valid = pd.DataFrame(my_imputer.fit_transform(X_valid))
# Imputation removed column names; put them back
imputed_X_train.columns = X_train.columns
imputed_X_valid.columns = X_valid.columns
step_3.a.assert_check_failed()
#%%RM_IF(PROD)%%
# Imputation
my_imputer = SimpleImputer()
imputed_X_train = pd.DataFrame(my_imputer.fit_transform(X_train))
imputed_X_valid = pd.DataFrame(my_imputer.transform(X_valid))
# Imputation removed column names; put them back
imputed_X_train.columns = X_train.columns
imputed_X_valid.columns = X_valid.columns
step_3.a.assert_check_passed()
# Lines below will give you a hint or solution code
#_COMMENT_IF(PROD)_
step_3.a.hint()
#_COMMENT_IF(PROD)_
step_3.a.solution()
Explanation: Step 3: Imputation
Part A
Use the next code cell to impute missing values with the mean value along each column.  Set the preprocessed DataFrames to imputed_X_train and imputed_X_valid.  Make sure that the column names match those in X_train and X_valid.
End of explanation
print("MAE (Imputation):")
print(score_dataset(imputed_X_train, imputed_X_valid, y_train, y_valid))
Explanation: Run the next code cell without changes to obtain the MAE for this approach.
End of explanation
# Check your answer (Run this code cell to receive credit!)
step_3.b.check()
#_COMMENT_IF(PROD)_
step_3.b.hint()
Explanation: Part B
Compare the MAE from each approach.  Does anything surprise you about the results?  Why do you think one approach performed better than the other?
End of explanation
# Preprocessed training and validation features
final_X_train = ____
final_X_valid = ____
# Check your answers
step_4.a.check()
#%%RM_IF(PROD)%%
# Imputation
final_imputer = SimpleImputer(strategy='median')
final_X_train = pd.DataFrame(final_imputer.fit_transform(X_train))
final_X_valid = pd.DataFrame(final_imputer.transform(X_valid))
# Imputation removed column names; put them back
final_X_train.columns = X_train.columns
final_X_valid.columns = X_valid.columns
step_4.a.assert_check_passed()
# Lines below will give you a hint or solution code
#_COMMENT_IF(PROD)_
step_4.a.hint()
#_COMMENT_IF(PROD)_
step_4.a.solution()
Explanation: Step 4: Generate test predictions
In this final step, you'll use any approach of your choosing to deal with missing values.  Once you've preprocessed the training and validation features, you'll train and evaluate a random forest model.  Then, you'll preprocess the test data before generating predictions that can be submitted to the competition!
Part A
Use the next code cell to preprocess the training and validation data.  Set the preprocessed DataFrames to final_X_train and final_X_valid.  You can use any approach of your choosing here!  in order for this step to be marked as correct, you need only ensure:
- the preprocessed DataFrames have the same number of columns,
- the preprocessed DataFrames have no missing values, 
- final_X_train and y_train have the same number of rows, and
- final_X_valid and y_valid have the same number of rows.
End of explanation
# Define and fit model
model = RandomForestRegressor(n_estimators=100, random_state=0)
model.fit(final_X_train, y_train)
# Get validation predictions and MAE
preds_valid = model.predict(final_X_valid)
print("MAE (Your approach):")
print(mean_absolute_error(y_valid, preds_valid))
Explanation: Run the next code cell to train and evaluate a random forest model.  (Note that we don't use the score_dataset() function above, because we will soon use the trained model to generate test predictions!)
End of explanation
# Fill in the line below: preprocess test data
final_X_test = ____
# Fill in the line below: get test predictions
preds_test = ____
# Check your answers
step_4.b.check()
#%%RM_IF(PROD)%%
# Preprocess test data
final_X_test = pd.DataFrame(final_imputer.transform(X_test))
# Get test predictions
preds_test = model.predict(final_X_test)
step_4.b.assert_check_passed()
# Lines below will give you a hint or solution code
#_COMMENT_IF(PROD)_
step_4.b.hint()
#_COMMENT_IF(PROD)_
step_4.b.solution()
Explanation: Part B
Use the next code cell to preprocess your test data.  Make sure that you use a method that agrees with how you preprocessed the training and validation data, and set the preprocessed test features to final_X_test.
Then, use the preprocessed test features and the trained model to generate test predictions in preds_test.
In order for this step to be marked correct, you need only ensure:
- the preprocessed test DataFrame has no missing values, and
- final_X_test has the same number of rows as X_test.
End of explanation
# Save test predictions to file
output = pd.DataFrame({'Id': X_test.index,
                       'SalePrice': preds_test})
output.to_csv('submission.csv', index=False)
Explanation: Run the next code cell without changes to save your results to a CSV file that can be submitted directly to the competition.
End of explanation
<END_TASK>