<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 <a href="https
Step1:  Binary Classification
In this unit we will explore binary classification using logistic regression.
Some of these terms might be new, so let's explore them a bit more.
Classification is the process of mapping a set of data points to a finite set of labels. From our regression labs, you likely remember that regression models such as linear regression map input variables to a range of continuous values. In the domain of machine learning, models that predict continuous values are considered regression models. Models that predict a known finite set of values are considered classification models.
So what does binary mean?
Binary means there are only two values to predict. Binary classification is used to predict one of two values. These can be true/false, malignant/benign, yes/no, or any possible this-or-that options. For simplicity, these options are usually encoded as 1 and 0.
And what about logistic regression?
You've already seen linear regression, which attempts to fit a line to a set of data in order to predict continuous values. Logistic regression similarly attempts to fit a line to data. However, the line is typically a logistic/sigmoid curve. Instead of predicting a continuous value, the model uses the logistic curve to split the data into two classes. One class falls to one side of the line, and the other class falls to the other side of the line.
Framing the Problem
Cindy's Produce For Good has a problem. Its business model revolves around collecting unsold fruit and vegetables from local growers and distributing them to families in need so that they can consume it or resell it at local farmer's markets and roadside stands.
Quite a few complaints have come in lately from families and customers who have had a bitter surprise. They've peeled what they thought was an orange only to bite in and find out that they are eating a grapefruit!
Cindy's growers give her truckloads of mixed citrus
Step2:  There should now be an oranges-vs-grapefruit.zip file in the virtual machine for this lab. Let's unzip it so we can access the data.
Step3:  There is now a citrus.csv file in our virtual machine. Let's start digging into the data next.
Basic Analysis
First and foremost, we need to load the data. For that we'll rely on Pandas and use the read_csv function since the data was provided to us as a CSV file.
After we load the data, let's sample it to get an idea of what we are working with.
Step4:  It looks like we have a mixed bag of fruit containing oranges and grapefruit, just as expected.
How many do we have of each?
Exercise 2
Step5:  Interpreting Our Histogram
The histogram shows the data evenly distributed across different types of fruit. This distribution makes the dataset very balanced for building a model for our classifier.
Describing Our Dataset
Next let's do a simple describe of our dataset to get some more detailed information.
Step6:  Since every count is 10,000, we don't seem to have missing values.
Also, every min value is a positive number. This is good since it would be really odd to have negative diameters, weights, or colors.
Do the values themselves look reasonable? The diameter is measured in centimeters. Is a 2 cm piece of fruit believable? What about a 16 cm piece of fruit?
Similarly, do the weights seem within ranges that we'd expect?
It is actually difficult to tell since we have different kinds of fruit in this bag. It would be easier to inspect summary statistics for each type of fruit.
Exercise 3
Step7:  Visualizing With Box Plots
Now that we've sanity checked our data, let's visualize it to see if we can gather more insight. Above we gathered the min, max, mean, etc. for each numeric column for each type of fruit in a tabular form. Let's now visualize that data using a box plot and the Altair visualization library.
To start using Altair, we simply import it.
Step8:  Next we will use the mark_boxplot method of the Chart class to create our boxplot.
Let's start by plotting the diameter by name.
To do this we must first sample a subset of our data. We have 20,000 rows of data, and Altair cannot visualize that much data in a boxplot. The row limit is 5,000 rows, so we'll create a 5,000 row sample and then pass that sample to Altair.
Step9:  What insights can we glean from this graphic?
As expected, the diameter of a grapefruit trends larger than that of an orange, but there is some overlap.
Let's now add in weight to our boxplot.
Step10:  Correlation
Notice that relative weight and diameter seem pretty similar. These two columns might be closely correlated enough that we only need to use one of them. Let's check the correlation coefficient.
Exercise 4
Step11:  Understanding the Correlation
The correlation between diameter and weight is over 99%. That is a very high value.
This shouldn't come as a big surprise. We should expect that the weight of a piece of fruit grows as its diameter grows.
For now we can leave the data as is, but remember this correlation. We might be able to use it to remove a column from our training data without negatively affecting our model.
Let's take another look at height and weight. They are definitely correlated, but how do they relate to each other for each fruit type?
One way to see this is to use a scatter plot chart to plot the diameter versus the weight, segmented by fruit type.
We'll use Altair to do this.
Step12:  We can see that oranges and grapefruit have very similar rates of weight gain as their diameter increases. This shouldn't be too surprising since they are very similar fruits.
In this chart we can also see that there are some fruits that are clearly oranges because of their small size and weight, as well as some that are clearly grapefruit due to their large size and weight. However, we have a large number of fruits that will be difficult to classify using diameter and weight alone.
Checking Color Values
We've looked pretty closely at the diameter and weight values, but we haven't done much with the color (RGB) values.
Let's first see if boxplots are helpful.
Step13:  There doesn't seem to be a lot of value there, at least examining each element of color separately. There is quite a bit of overlap between each color element, with grapefruit displaying a little less red and green typically.
It would also be nice to "sanity check" the color values, similar to how we checked to make sure that our diameters and weights were within reason. We could see if the values fall within a reasonable range, but then we'd need to know reasonable RGB values for oranges and grapefruit.
Since we are dealing with color data, we can just create an image for each piece of fruit that contains a sampling (or all) of the colors that we have and we can see if it looks reasonable.
First, let's get an exact count of the number of samples of each fruit type.
Step14:  As expected, we have 5000 samples each. We can create a 100x50 image for each fruit type and visualize the data.
We'll use PIL's Image class to create a white 100x50 image. Then we'll get the editable pixel map from the image and assign the color value for each orange in our data to a different pixel in the image.
Once we have the image filled out with color, we'll use PyPlot to display the image.
Step15:  That looks like a pretty reasonable orange color. What about the grapefruit?
Exercise 5
Step16:  Data Analysis Summary
We've done a lot of data analysis and have a pretty good feel for our data. We have
Step17:  We'll use 'diameter', 'weight', 'red', 'green', and 'blue' as feature columns. Using 'name' for our target column is tempting, but remember that it contains fruit names for values, and for this exercise, we are only interested in determining if a piece of fruit is an orange or not an orange. Let's create a new column called 'is_orange' that contains the value True if the datum is an orange and False otherwise.
Exercise 6
Step18:  Examining Our New Target Column
Now that we've created a new target column, we should do some checking to make sure that it was created correctly.
First we'll simply see the count per value.
Step19:  There should be 5,000 True values and 5,000 False values.
Now check to see that all 5,000 of the True values have the name "orange".
Step20:  We should only see a single value in the unique list ('orange') since all rows with 'is_orange' set to True should have a 'name' of 'orange'.
Train/Test Split
We can now split our data for training and testing. First we will create variables to hold our training and target column names.
Step21:  We need to split the data into a training and testing set. In this case we'll split 20% of the data off for testing and train on the other 80%. We can use scikit-learn's train_test_split function to do this. It is also a really good idea to shuffle our data, and train_test_split allows us to do this too.
After we make the split, we can see how many data points we will train off of for each class.
Step22:  Hmm. It looks like our training set has become a little uneven. Ideally, we would maintain the same ratio of oranges to non-oranges in our training and testing groups as the ratio in the whole set (50/50). But after splitting the data, we've ended up with a training set that skews towards non-oranges, and a test set that skews the opposite way, towards oranges.
Step23:  Luckily, there's a solution for this problem
Step24:  Examining The Split Data
We can now verify that we have 80% of the data in training...
Step25:  And 20% in testing.
Step26:  Let's look at the training data and see if it stratified correctly.
Step27:  From this output we can see that there are 8,000 pieces of data with 2 unique values. The top value is True, and it occurs 4,000 times. That would leave us with 4,000 other values that are False.
We can do the same for the y_test data.
Step28:  Another alternative is to use groupby on the series. Notice that the by argument contains the series once again and not a column name.
Step29:  Create and Train the Model
It is finally time to build and train our model. As a reminder, we are using sklearn.linear_model.LogisticRegression.
First, we'll build a baseline model with default arguments, and see how well it does. To build the model we import LogisticRegression, create a class instance, and then fit the model.
Step30:  Measure Model Performance
We now have a model ready to use to make predictions. Let's first make predictions on the test data that we held out of our training set and see how well we did.
The first step is to actually make the predictions.
Step31:  Now we can use metrics functions from scikit-learn to see how well our model performed. We'll check the accuracy, precision, recall, and F1 scores.
Step32:  Numbers for most of the metrics are above 90%, which is better than Cindy was sorting!
Let's see how this looks in a confusion matrix.
Step33:  We have just under 100 falsely identified fruit. There are about twice as many false negatives as there are false positives. Let's take a few minutes to think about what this confusion matrix means.
Exercise 8
Step34:  We can see that there is a steep increase in false positives as the true positive rate crosses into the 90% range.
Precision Recall Curve
We can also get a feel for how precision and recall relate for this model by plotting the precision recall curve.
Step35:  This shows the balance between precision and recall as the model adjusts classification thresholds.
Improving Our Model
Our initial model was actually pretty good. But can it be even better?
In the next exercise we'll attempt to improve our model by exploring hyperparameters and manipulating features.
Exercise 9
Step36:  Question 2
Step37:  Question 3
<ASSISTANT_TASK:>
Python Code:
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
Explanation: <a href="https://colab.research.google.com/github/google/applied-machine-learning-intensive/blob/master/content/04_classification/01_binary_classification/colab.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/></a>
Copyright 2020 Google LLC.
End of explanation
!KAGGLE_CONFIG_DIR=`pwd` kaggle datasets download joshmcadams/oranges-vs-grapefruit
!ls
Explanation: Binary Classification
In this unit we will explore binary classification using logistic regression.
Some of these terms might be new, so let's explore them a bit more.
Classification is the process of mapping a set of data points to a finite set of labels. From our regression labs, you likely remember that regression models such as linear regression map input variables to a range of continuous values. In the domain of machine learning, models that predict continuous values are considered regression models. Models that predict a known finite set of values are considered classification models.
So what does binary mean?
Binary means there are only two values to predict. Binary classification is used to predict one of two values. These can be true/false, malignant/benign, yes/no, or any possible this-or-that options. For simplicity, these options are usually encoded as 1 and 0.
And what about logistic regression?
You've already seen linear regression, which attempts to fit a line to a set of data in order to predict continuous values. Logistic regression similarly attempts to fit a line to data. However, the line is typically a logistic/sigmoid curve. Instead of predicting a continuous value, the model uses the logistic curve to split the data into two classes. One class falls to one side of the line, and the other class falls to the other side of the line.
Framing the Problem
Cindy's Produce For Good has a problem. Its business model revolves around collecting unsold fruit and vegetables from local growers and distributing them to families in need so that they can consume it or resell it at local farmer's markets and roadside stands.
Quite a few complaints have come in lately from families and customers who have had a bitter surprise. They've peeled what they thought was an orange only to bite in and find out that they are eating a grapefruit!
Cindy's growers give her truckloads of mixed citrus: lemons, limes, oranges, and grapefruit. A volunteer crew sorts the fruit. They are really good at sorting lemons and limes, but they falsely identify grapefruit as oranges about 5% of the time.
In order to ensure customers get the oranges they expect, Cindy has created a machine that measures the weight, color, and largest diameter of fruit. She wants to create some software that can use this information and tell her workers if the fruit is an orange or not.
She put a few thousand pieces of orange-looking fruit from one of her shipments through the sensors and manually labelled them as oranges or grapefruit. Looking at the data, she couldn't see an obvious pattern. Her best performance was about 90% accuracy. Her human sorters can do at least 95%. She's requested our help to see if we can solve the orange vs. grapefruit problem.
In this lab we'll examine Cindy's citrus data and try to build a model to help her reliably sort her fruit as well as or better than human sorters.
Exercise 1: Thinking About the Data
Before we dive in to looking closely at the data, let's think about the problem space and the dataset. Consider the questions below.
Question 1
Is this problem actually a good fit for machine learning? Why or why not?
Student Solution
Please Put Your Answer Here
Question 2
If we do build Cindy a machine learning model, what biases might exist in the data? Is there anything that might cause her model to have trouble generalizing to other data? If so, how might she make the model more resilient?
Student Solution
Please Put Your Answer Here
Question 3
We've been asked to create a system that determines if a piece of fruit is an orange or not an orange. But aside from that, we haven't gotten much information about how the system would work as a whole.
Describe how you would design the system from end-to-end. Things to consider:
Would the input fruit be all of the fruit that Cindy receives? Only the fruit suspected of being an orange? Only questionable fruit? Anything suspected of being an orange or a grapefruit?
What happens to fruit classified as "not orange". Is it automatically considered a grapefruit? Is it thrown away? Put in a mixed fruit bag?
Justify the inputs and the output actions for the system. What are the trade-offs?
Student Solution
Please Put Your Answer Here
Exploratory Data Analysis
Acquire the Data
We have some idea about the problem that we are trying to solve, so let's take a look at what has been collected. The data is hosted on Kaggle. You can download the dataset and then upload it to this lab or use the code blocks below to fetch the data directly.
Direct Kaggle Download
Follow the API Credentials instructions and get a kaggle.json file (if you don't already have one), and upload it to this lab.
Then run the code block below to download the oranges vs. grapefruit dataset.
End of explanation
!unzip -o oranges-vs-grapefruit.zip
!ls
Explanation: There should now be an oranges-vs-grapefruit.zip file in the virtual machine for this lab. Let's unzip it so we can access the data.
End of explanation
import pandas as pd
citrus_df = pd.read_csv('citrus.csv', header=0)
citrus_df.sample(10, random_state=2020)
Explanation: There is now a citrus.csv file in our virtual machine. Let's start digging into the data next.
Basic Analysis
First and foremost, we need to load the data. For that we'll rely on Pandas and use the read_csv function since the data was provided to us as a CSV file.
After we load the data, let's sample it to get an idea of what we are working with.
End of explanation
# Your Solution Goes Here
Explanation: It looks like we have a mixed bag of fruit containing oranges and grapefruit, just as expected.
How many do we have of each?
Exercise 2: Basic Statistics
Let's take a moment to determine the distribution of fruit in our dataset. Use pyplot to create a histogram of the values in the name column of our DataFrame.
Student Solution
End of explanation
citrus_df.describe()
Explanation: Interpreting Our Histogram
The histogram shows the data evenly distributed across different types of fruit. This distribution makes the dataset very balanced for building a model for our classifier.
Describing Our Dataset
Next let's do a simple describe of our dataset to get some more detailed information.
End of explanation
# Your Solution Goes Here
Explanation: Since every count is 10,000, we don't seem to have missing values.
Also, every min value is a positive number. This is good since it would be really odd to have negative diameters, weights, or colors.
Do the values themselves look reasonable? The diameter is measured in centimeters. Is a 2 cm piece of fruit believable? What about a 16 cm piece of fruit?
Similarly, do the weights seem within ranges that we'd expect?
It is actually difficult to tell since we have different kinds of fruit in this bag. It would be easier to inspect summary statistics for each type of fruit.
Exercise 3: More Focused Description
We have used describe() to get statistics about the entire dataset, but there isn't a lot of information in the data. Write Python code to print describe() statistics for each type of fruit in the dataset. Use the percentiles argument to the describe method to not print the 25th and 75th percentile.
Your output should look similar to:
```
orange
          diameter       weight          red        green         blue
count  5000.000000  5000.000000  5000.000000  5000.000000  5000.000000
mean      8.474424   152.804920   156.832800    81.988200     7.115200
std       1.260665    18.669021     9.890258    10.090789     6.493779
min       2.960000    86.760000   123.000000    49.000000     2.000000
50%       8.470000   152.665000   157.000000    82.000000     4.000000
max      12.870000   231.090000   192.000000   116.000000    38.000000
grapefruit
          diameter       weight          red        green         blue
count  5000.000000  5000.000000  5000.000000  5000.000000  5000.000000
mean     11.476946   197.296664   150.862800    70.033000    15.611200
std       1.221148    19.193190    10.103148    10.044924     9.271592
min       7.630000   126.790000   115.000000    31.000000     2.000000
50%      11.450000   197.430000   151.000000    70.000000    15.000000
max      16.450000   261.510000   187.000000   103.000000    56.000000
```
Student Solution
End of explanation
import altair as alt
Explanation: Visualizing With Box Plots
Now that we've sanity checked our data, let's visualize it to see if we can gather more insight. Above we gathered the min, max, mean, etc. for each numeric column for each type of fruit in a tabular form. Let's now visualize that data using a box plot and the Altair visualization library.
To start using Altair, we simply import it.
End of explanation
citrus_df_sample = citrus_df.sample(n=5000, random_state=2020)
alt.Chart(citrus_df_sample, width=400).mark_boxplot().encode(
    x='name',
    y='diameter'
)
Explanation: Next we will use the mark_boxplot method of the Chart class to create our boxplot.
Let's start by plotting the diameter by name.
To do this we must first sample a subset of our data. We have 20,000 rows of data, and Altair cannot visualize that much data in a boxplot. The row limit is 5,000 rows, so we'll create a 5,000 row sample and then pass that sample to Altair.
End of explanation
alt.Chart(citrus_df_sample, width=400).mark_boxplot().encode(
    x='name',
    y='diameter'
) | alt.Chart(citrus_df_sample, width=400).mark_boxplot().encode(
    x='name',
    y='weight'
)
Explanation: What insights can we glean from this graphic?
As expected, the diameter of a grapefruit trends larger than that of an orange, but there is some overlap.
Let's now add in weight to our boxplot.
End of explanation
# Your Solution Goes Here
Explanation: Correlation
Notice that relative weight and diameter seem pretty similar. These two columns might be closely correlated enough that we only need to use one of them. Let's check the correlation coefficient.
Exercise 4: Correlation Coefficient
Based on our visualization above, we suspect that diameter and weight are highly correlated. Write code to find the correlation coefficient between the diameter and weight columns in our DataFrame.
Hint: Check out the corr documentation.
Student Solution
End of explanation
alt.Chart(citrus_df_sample).mark_circle().encode(
    x='diameter',
    y='weight',
    color='name'
)
Explanation: Understanding the Correlation
The correlation between diameter and weight is over 99%. That is a very high value.
This shouldn't come as a big surprise. We should expect that the weight of a piece of fruit grows as its diameter grows.
For now we can leave the data as is, but remember this correlation. We might be able to use it to remove a column from our training data without negatively affecting our model.
Let's take another look at height and weight. They are definitely correlated, but how do they relate to each other for each fruit type?
One way to see this is to use a scatter plot chart to plot the diameter versus the weight, segmented by fruit type.
We'll use Altair to do this.
End of explanation
alt.Chart(citrus_df_sample, width=400).mark_boxplot().encode(
    x='name',
    y='red'
) | alt.Chart(citrus_df_sample, width=400).mark_boxplot().encode(
    x='name',
    y='green'
) | alt.Chart(citrus_df_sample, width=400).mark_boxplot().encode(
    x='name',
    y='blue'
)
Explanation: We can see that oranges and grapefruit have very similar rates of weight gain as their diameter increases. This shouldn't be too surprising since they are very similar fruits.
In this chart we can also see that there are some fruits that are clearly oranges because of their small size and weight, as well as some that are clearly grapefruit due to their large size and weight. However, we have a large number of fruits that will be difficult to classify using diameter and weight alone.
Checking Color Values
We've looked pretty closely at the diameter and weight values, but we haven't done much with the color (RGB) values.
Let's first see if boxplots are helpful.
End of explanation
citrus_df.groupby('name')['name'].count()
Explanation: There doesn't seem to be a lot of value there, at least examining each element of color separately. There is quite a bit of overlap between each color element, with grapefruit displaying a little less red and green typically.
It would also be nice to "sanity check" the color values, similar to how we checked to make sure that our diameters and weights were within reason. We could see if the values fall within a reasonable range, but then we'd need to know reasonable RGB values for oranges and grapefruit.
Since we are dealing with color data, we can just create an image for each piece of fruit that contains a sampling (or all) of the colors that we have and we can see if it looks reasonable.
First, let's get an exact count of the number of samples of each fruit type.
End of explanation
from PIL import Image
from matplotlib.pyplot import imshow
import numpy as np
height, width = 50, 100
img = Image.new('RGB', (width, height), color=(255, 255, 255))
pixels = img.load()
row_i, col_i = 0, 0
for _, fruit in citrus_df[citrus_df['name'] == 'orange'].iterrows():
  pixels[col_i, row_i] = (fruit['red'], fruit['green'], fruit['blue'])
  col_i += 1
  if col_i >= width:
    col_i = 0
    row_i += 1
imshow(img)
Explanation: As expected, we have 5000 samples each. We can create a 100x50 image for each fruit type and visualize the data.
We'll use PIL's Image class to create a white 100x50 image. Then we'll get the editable pixel map from the image and assign the color value for each orange in our data to a different pixel in the image.
Once we have the image filled out with color, we'll use PyPlot to display the image.
End of explanation
# Your Solution Goes Here
Explanation: That looks like a pretty reasonable orange color. What about the grapefruit?
Exercise 5: Create a Color Map Image
We only visualized data from oranges. We'd really like to see the colors of all of the fruit. Create and show a 100x100 image that contains the colors for all of the oranges in the first 100x50 block. This should be followed with the colors for all of the grapefruit in the next 100x50 block. Visually inspect your image to see if the colors are believable as oranges and grapefruit.
Student Solution
End of explanation
citrus_df.columns
Explanation: Data Analysis Summary
We've done a lot of data analysis and have a pretty good feel for our data. We have:
Examined the distribution of our dataset and seen we have an equal distribution of fruit types
Determined that no data is missing
Determined that our weight, diameter, and color values are all within reason
Found a strong correlation between weight and diameter
Let's see if we can build a model to classify our oranges!
Simple Logistic Model
It is now time to build and iterate on a model. We'll start with a simple logistic regression model using scikit-learn's sklearn.linear_model.LogisticRegression class and the feature columns already in our training data.
Let's first remind ourselves of the columns we have at our disposal.
End of explanation
# Your Solution Goes Here
Explanation: We'll use 'diameter', 'weight', 'red', 'green', and 'blue' as feature columns. Using 'name' for our target column is tempting, but remember that it contains fruit names for values, and for this exercise, we are only interested in determining if a piece of fruit is an orange or not an orange. Let's create a new column called 'is_orange' that contains the value True if the datum is an orange and False otherwise.
Exercise 6: Is Orange?
Create a new column in citrus_df called is_orange. The column should be a boolean column and should contain the value True if a given row is labeled as an orange and False otherwise.
Student Solution
End of explanation
citrus_df.groupby('is_orange')['is_orange'].count()
Explanation: Examining Our New Target Column
Now that we've created a new target column, we should do some checking to make sure that it was created correctly.
First we'll simply see the count per value.
End of explanation
citrus_df[citrus_df['is_orange']]['name'].unique()
Explanation: There should be 5,000 True values and 5,000 False values.
Now check to see that all 5,000 of the True values have the name "orange".
End of explanation
target_column = 'is_orange'
feature_columns = ['diameter', 'weight', 'red', 'green', 'blue']
target_column, feature_columns
Explanation: We should only see a single value in the unique list ('orange') since all rows with 'is_orange' set to True should have a 'name' of 'orange'.
Train/Test Split
We can now split our data for training and testing. First we will create variables to hold our training and target column names.
End of explanation
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
    citrus_df[feature_columns],
    citrus_df[target_column],
    test_size=0.2,
    random_state=180,
    shuffle=True)
y_train.groupby(y_train).count()
Explanation: We need to split the data into a training and testing set. In this case we'll split 20% of the data off for testing and train on the other 80%. We can use scikit-learn's train_test_split function to do this. It is also a really good idea to shuffle our data, and train_test_split allows us to do this too.
After we make the split, we can see how many data points we will train off of for each class.
End of explanation
y_test.groupby(y_test).count()
Explanation: Hmm. It looks like our training set has become a little uneven. Ideally, we would maintain the same ratio of oranges to non-oranges in our training and testing groups as the ratio in the whole set (50/50). But after splitting the data, we've ended up with a training set that skews towards non-oranges, and a test set that skews the opposite way, towards oranges.
End of explanation
# Your Code Goes Here
Explanation: Luckily, there's a solution for this problem: stratified sampling. Stratifying our data ensures that the ratio of distinct values in the given column remains the same in our training and test sets as it is in the whole set (half orange and half non-orange, in this case).
Exercise 7: Stratified Train Test Split
Look at the documentation for train_test_split and find the argument that can be used to stratify the data. Rewrite the split above to create a stratified split. When you are done, there should be 4,000 True values and 4,000 False values in the training data and 1,000 of each in the testing data. Print the counts to verify.
Student Solution
End of explanation
X_train.shape, y_train.shape
Explanation: Examining The Split Data
We can now verify that we have 80% of the data in training...
End of explanation
X_test.shape, y_test.shape
Explanation: And 20% in testing.
End of explanation
y_train.describe()
Explanation: Let's look at the training data and see if it stratified correctly.
End of explanation
y_test.describe()
Explanation: From this output we can see that there are 8,000 pieces of data with 2 unique values. The top value is True, and it occurs 4,000 times. That would leave us with 4,000 other values that are False.
We can do the same for the y_test data.
End of explanation
y_test.groupby(by=y_test).count()
Explanation: Another alternative is to use groupby on the series. Notice that the by argument contains the series once again and not a column name.
End of explanation
from sklearn.linear_model import LogisticRegression
model = LogisticRegression(random_state=2020)
model.fit(X_train, y_train)
Explanation: Create and Train the Model
It is finally time to build and train our model. As a reminder, we are using sklearn.linear_model.LogisticRegression.
First, we'll build a baseline model with default arguments, and see how well it does. To build the model we import LogisticRegression, create a class instance, and then fit the model.
End of explanation
predictions = model.predict(X_test)
Explanation: Measure Model Performance
We now have a model ready to use to make predictions. Let's first make predictions on the test data that we held out of our training set and see how well we did.
The first step is to actually make the predictions.
End of explanation
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score
print('Accuracy: ', round(accuracy_score(predictions, y_test), 3))
print('Precision: ', round(precision_score(predictions, y_test), 3))
print('Recall: ', round(recall_score(predictions, y_test), 3))
print('F1: ', round(f1_score(predictions, y_test), 3))
Explanation: Now we can use metrics functions from scikit-learn to see how well our model performed. We'll check the accuracy, precision, recall, and F1 scores.
End of explanation
from sklearn.metrics import confusion_matrix
tn, fp, fn, tp = confusion_matrix(y_test, predictions).ravel()
print(f'True Positive: {tp}\nTrue Negative: {tn}\nFalse Positive: {fp}\nFalse Negative: {fn}')
Explanation: Numbers for most of the metrics are above 90%, which is better than Cindy was sorting!
Let's see how this looks in a confusion matrix.
End of explanation
from sklearn.metrics import roc_curve
from sklearn.metrics import roc_auc_score
scores = model.decision_function(X_test)
fpr, tpr, _ = roc_curve(y_test, scores, pos_label=True)
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate')
plt.plot(fpr, tpr)
plt.show()
Explanation: We have just under 100 falsely identified fruit. There are about twice as many false negatives as there are false positives. Let's take a few minutes to think about what this confusion matrix means.
Exercise 8: Interpreting a Confusion Matrix
In the text cell below, explain what a false positive and false negative represent in our dataset: which is an orange classified as a grapefruit and which is a grapefruit classified as an orange?
Student Solution
Please Put Your Answer Here
ROC Curve
We can visualize this in another way using the Receiver-Operator Curve. This graph plots the true positive rate on the y-axis against the false positive rate on the x-axis.
End of explanation
from sklearn.metrics import precision_recall_curve
scores = model.decision_function(X_test)
precision, recall, _ = precision_recall_curve(y_test, scores)
plt.xlabel('Recall')
plt.ylabel('Precision')
plt.plot(recall, precision)
plt.show()
Explanation: We can see that there is a steep increase in false positives as the true positive rate crosses into the 90% range.
Precision Recall Curve
We can also get a feel for how precision and recall relate for this model by plotting the precision recall curve.
End of explanation
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score
from sklearn.model_selection import train_test_split, GridSearchCV
citrus_df = pd.read_csv('citrus.csv', header=0)
citrus_df['is_orange'] = citrus_df['name'].apply(lambda name: name == 'orange')
target_column = 'is_orange'
feature_columns = ['diameter', 'weight', 'red', 'green', 'blue']
X_train, X_validate, y_train, y_validate = train_test_split(
    citrus_df[feature_columns],
    citrus_df[target_column],
    test_size=0.2,
    random_state=42,
    shuffle=True,
    stratify=citrus_df[target_column])
model = LogisticRegression(
    random_state=2020,
)
search = GridSearchCV(model, {
  # Your Solution Goes Here
})
search.fit(X_train, y_train)
print(search.best_estimator_)
Explanation: This shows the balance between precision and recall as the model adjusts classification thresholds.
Improving Our Model
Our initial model was actually pretty good. But can it be even better?
In the next exercise we'll attempt to improve our model by exploring hyperparameters and manipulating features.
Exercise 9: Using GridSearchCV
We will now experiment with different hyperparameters to see if we can tune the model to increase our scores. To do this we will use the GridSearchCV class to tune hyperparameters of the scikit-learn LogisticRegressor.
GridSearchCV is a class used to test different hyperparameters for a model. The search accepts a dictionary containing keys that map to model parameters. The values are lists for hyperparameters that you want to experiment with or single values for parameters that you want to keep constant.
Question 1: Performing the Search
Below is some code that imports the necessary functions and classes and sets up a logistic regression model for grid search. Add code to the grid search to test different hyperparameters such as tol, C, solver, and max_iter.
The best estimator will be displayed after running the code block.
Student Solution
End of explanation
# Your Solution Goes Here
Explanation: Question 2: Validate the Model
Now that we have found a model that scored the highest in a cross-validation grid search, let's validate the model to see if it generalizes well on our validation data.
We held out validation data in the X_validate and y_validate variables. Use that data to calculate the accuracy, precision, recall, and F1 scores for the model.
Student Solution
End of explanation
# Your code goes here
Explanation: Question 3: Relative Model Quality
Now that we have the scores for our model on our validation set, is the version found by grid search notably better? Discuss the difference in scores, if any, between our base model and the model selected by grid search.
Student Solution
Please Put Your Answer Here
Exercise 10: Final Model Assessment
Given our model performance, is this machine learning model a good fit for the problem?
Student Solution
Please Put Your Answer Here
Challenge
Question 1
Normalization and standardization of data is not strictly required for performing logistic regression. It is, however, suggested in some cases. Research reasons why you might want (or not want) to normalize or standardize your input data to a logistic regression.
Explain your findings and link to any relevant articles.
Student Solution
Please Put Your Answer Here
Question 2
Use the sklearn.preprocessing.StandardScaler to scale the feature data before training a logistic model on our oranges dataset. Use sklearn.model_selection.GridSearchCV to iterate through hyperparameters to find an optimal model.
Student Solution
End of explanation
<END_TASK>