<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
Step3:  <a href="https
Step4:  We've succesfully loaded our data, but there are still a couple preprocessing steps to go through first. Specifically, we're going to
Step5:  Now that we have our features and labels set, it's time to start modeling!
1) Model Selection
The results of our models are only good if our models are correct in the first place. "Good" here can mean different things depending on your application -- we'll talk more about that later in this assignment.
What's important to know for now is that the conclusions we draw are subject to the assumptions and limitations of our underlying model. Thus, making sure we choose the right models to analyze is important! But how does one choose the right model?
1.1) Building Intuition -- Which model do you think is best?
To build some intuition, let's start off with a simple example. We'll use a subset of our data. For ease of visualization, we'll start with just two features, and just 10 cities.
Step6:  The following blocks of code will generate 2 candidate classifiers, for labeling the datapoints as either label 0 (high obesity rate), or label 1 (high obesity rate).
The code will also output an accuracy score, which for this section is defined as
Step7:  1A) Which model do you think is better, Classifier 1 or Classifier 2? Explain your reasoning.
1B) Classifier 2 has a higher accuracy than Classifier 1, but has a more complicated decision boundary. Which do you think would generalize best to new data?
1.2) The Importance of Generalizability
So, how did we do? Let's see what happens when we add back the rest of the cities (we'll keep using just the 2 features for ease of visualization.)
Step8:  2A) In light of all the new datapoints, now which classifier do you think is better, Classifer 1 or Classifier 2? Explain your reasoning.
2B) In Question 1, Classifier 1 had a lower accuracy than Classifier 2. After adding more datapoints, we now see the reverse, with Classifier 1 having a higher accuracy than Classifier 2. What happened? Give an explanation (or at least your best guess) for why this is.
2) Evaluation Metrics
In question 1, we were able to visualize how well our models performed by plotting our data and decision boundaries. However, this was only possible because we limited ourselves to just 2 features. Unfortunately for us, humans are only good at visualizing up to 3 dimesnions. As you increase the number of features and/or complexity of your models, creating meaningful visualizations quickly become intractable.
Thus, we'll need other methods to measure how well our models perform. In this section, we'll cover some common strategies for evaluating models.
To start, let's finally fit a model to all of our available data (e.g. 500 cities and 8 features). Because the features have different scales, we'll also take care to standardize their values.
Step9:  2.1) Accuracy
2.1.1) Classification Accuracy
We've seen an example of an evaluation metric already -- accuracy! The accuracy score used question 1 is more commonly known as classification accuracy, and is the most common metric used in classification problems.
As a refresher, the classification accuracy is the ratio of number of correct predictions to the total number of datapoints.
classification accuracy
Step10:  2.2) Train/Test Splits
The ability of a model to perform well on new, previously unseen data (drawn from the same distribution as the data used the create the model) is called Generalization. For most applications, we prefer models that generalize well over those that don't.
One way to check the generalizability of a model is to perform an analysis similar to what we did in question 1. We'll take our data, and randomly split it into two subsets, a training set that we'll use to build our model, and a test set, which we'll hold out until the model is complete and use it to evaluate how well our model can generalize to simulate new, previously unseen data.
2.2.1) Choosing Split Sizes
But what percentage of our datapoints should go into our training and test sets respectively? There are no hard and fast rules for this, the right split often depends on the application and how much data we have. The next few questions explore the key tradeoffs
Step11:  2.2.3) Training vs. Test Accuracy
As you may have noticed from 2.2.2, we can calculate two different accuracies after performing a train/test split. A training accuracy based on how well the model performs on the data it was trained on, and a test accuracy based on how well the model performs on held out data. Typically, we select models based on test accuracy. Afterall, a model's performance on new data after being deployed is usually more important than how well that model performed on the training data.
So why measure training accuracy at all? It turns out training accuracy is often useful for diagnosing some common issues with models.
For example, consider the following scenario
Step12:  2.3A) Play around with the code box above to find a good value of $k$. What happens if $k$ is very large or very small?
2.2B) How does the average score across all folds change with $k$?
2.4) Other Metrics Worth Knowing
2.4.1) What about Regression? -- Mean Squared Error
Different models and different problems often use different accuracy metrics. You may have noticed classification accuracy doesn't make much sense for regression problems, where instead of predicting a label, the model predicts a numeric value. In regression, a common accuracy metric is the Mean Squared Error, or MSE.
$ MSE = \frac{1}{\text{# total data points}}\sum_{\text{all data points}}(\text{predicted value} - \text{actual value})^2$
It is a measure of the average difference between the predicted value and the actual value. The square ($^2$) can seem counterintuitive at first, but offers some nice mathematical properties.
2.4.2) More Classification Metrics
Accuracy alone never tells the full story.There are a number of other metrics borrowed from statistics that are commonly used on classification models.
It's possible for a model to have a high accuracy, but score very low on some of the following metrics
Step13:  3A) Run the code boxes above and select which model you would choose to deploy. Justify your answer.
3B) Consider a new Classifier D. Its results look like this
<ASSISTANT_TASK:>
Python Code:
Install Data Commons API
We need to install the Data Commons API, since they don't ship natively with
most python installations.
In Colab, we'll be installing the Data Commons python and pandas APIs 
through pip.
!pip install datacommons --upgrade --quiet
!pip install datacommons_pandas --upgrade --quiet
Imports
This is where we'll load all the libraries we need for this assignment.
# Data Commons Python and Pandas APIs
import datacommons
import datacommons_pandas
# For manipulating data
import numpy as np
import pandas as pd
# For implementing models and evaluation methods
from sklearn import linear_model, svm, tree
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.preprocessing import StandardScaler
# For plotting
from matplotlib import pyplot as plt
from mlxtend.plotting import plot_decision_regions, category_scatter
Loading the Data
We'll query data using the Data Commons API, storing it in a Pandas data frame
city_dcids = datacommons.get_property_values(["CDC500_City"],
                                             "member",
                                             limit=500)["CDC500_City"]
# We've compiled a list of some nice Data Commons Statistical Variables
# to use as features for you
stat_vars_to_query = [
                      "Count_Person",
                      "Median_Income_Person",
                      "Count_Person_NoHealthInsurance",
                      "Percent_Person_PhysicalInactivity",
                      "Percent_Person_SleepLessThan7Hours",
                      "dc/e9gftzl2hm8h9", # Commute Time, this has a weird DCID
                      "Percent_Person_WithHighBloodPressure",
                      "Percent_Person_WithMentalHealthNotGood",
                      "Percent_Person_WithHighCholesterol",
                      "Percent_Person_Obesity"
                      
]
# Query Data Commons for the data and display the data
raw_features_df = datacommons_pandas.build_multivariate_dataframe(city_dcids,stat_vars_to_query)
display(raw_features_df)
Explanation: <a href="https://colab.research.google.com/github/datacommonsorg/api-python/blob/master/notebooks/intro_data_science/Classification_and_Model_Evaluation.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/></a>
Copyright 2022 Google LLC.
SPDX-License-Identifier: Apache-2.0
Classification and Model Evaluation
So you've built a machine learning model, or perhaps multiple models... now what? How do you know if those models are any good? And if you have multiple candidate models, how should you choose which one to utimately deploy?
The answer: model evaluation. Understanding how to evaluate your models is an essential skill not just to check how well your models perform, but also to diagnose issues and find areas for improvement. Most importantly, we need to understand whether or not we can trust our model's predictions.
Learning Objectives
In this lesson, we'll be covering:
Model Selection
Generalization and Overfitting
Train/Test Splits
Cross Validation
Statistical Evaulation Metrics:
How do we know a model is "good"?
Tradeoffs between evaluation metrics
Need extra help?
If you're new to Google Colab, take a look at this getting started tutorial.
To build more familiarity with the Data Commons API, check out these Data Commons Tutorials.
And for help with Pandas and manipulating data frames, take a look at the Pandas Documentation.
We'll be using the scikit-learn library for implementing our models today. Documentation can be found here. 
As usual, if you have any other questions, please reach out to your course staff!
Part 0: Introduction and Setup
The obesity epidemic in the United States is a major public health issue. Obesity rates vary across the nation by geographic location. In this colab, we'll be exploring how obesity rates vary with different health or societal factors across US cities.
Our Data Science Question: Can we predict which cities have high (>30%) or low (<30%) obesity rates based on other health or lifestyle factors?
End of explanation
# Make Row Names More Readable
# --- First, we'll copy the dcids into their own column
# --- Next, we'll get the "name" property of each dcid
# --- Then add the returned dictionary to our data frame as a new column
# --- Finally, we'll set this column as the new index
df = raw_features_df.copy(deep=True)
df['DCID'] = df.index
city_name_dict = datacommons.get_property_values(city_dcids, 'name')
city_name_dict = {key:value[0] for key, value in city_name_dict.items()}
df['City'] = pd.Series(city_name_dict)
df.set_index('City', inplace=True)
# Rename column "dc/e9gftzl2jm8h9" to "Commute_Time"
df.rename(columns={"dc/e9gftzl2hm8h9":"Commute_Time"}, inplace=True)
# Convert commute_time value
avg_commute_time = df["Commute_Time"]/df["Count_Person"]
df["Commute_Time"] = avg_commute_time
# Convert Count of No Health Insurance to Percentage
percent_noHealthInsurance = df["Count_Person_NoHealthInsurance"]/df["Count_Person"]
df["Percent_NoHealthInsurance"] = percent_noHealthInsurance
# Create labels based on the Obesity rate of each city
# --- Percent_Person_Obesity < 30 will be Label 0
# --- Percent_Person_Obesity >= 30 will be label 1
df["Label"] = df['Percent_Person_Obesity'] >= 30.0
df["Label"] = df["Label"].astype(int)
# Display results
display(df)
Explanation: We've succesfully loaded our data, but there are still a couple preprocessing steps to go through first. Specifically, we're going to:
Change the row labels from dcids to names for readability.
Change the column name "dc/e9gftzl2hm8h9" to the more human readable "Commute_Time"
The raw commute time values from Data Commons shows the total amount of minutes spent for everyone in the city. Let's instead look at the average commute time for a single person, which we'll get by dividing the raw commute time (Commute_Time) by population size (Count_Person)
Similarly, we'll get a Percent_NoHealthInsurance by dividing the the count of people without health insurance (Count_Person_NoHealthInsurance) by population size.
To perform classification, we need to convert our obesity rate data into labels. In this lesson, we'll look at binary classification, and will split our cities into  "Low obesity rate" (label 0, obesity% < 30%) and "High obesity rate" (label 1, obesity% >= 30%) categories.
End of explanation
# For ease of visualization, we'll focus on just a few cities
subset_city_dcids = ["geoId/0667000", # San Francisco, CA
                     "geoId/3651000", # NYC, NY
                     "geoId/1304000", # Atlanta, GA
                     "geoId/2404000", # Baltimore, MD
                     "geoId/3050200", # Missoula, MT
                     "geoId/4835000", # Houston, TX
                     "geoId/2622000", # Detroit, MI
                     "geoId/5363000", # Seattle, WA
                     "geoId/2938000", # Kansas City, MO
                     "geoId/4752006"  # Nashville, TN
                    ]
# Create a subset data frame with just those cities
subset_df = df.loc[df['DCID'].isin(subset_city_dcids)]
# We'll just use 2 features for ease of visualization
X = subset_df[["Percent_Person_PhysicalInactivity",
                 "Percent_Person_SleepLessThan7Hours"]]
Y = subset_df[['Label']]
# Visualize the data
colors = ['#1f77b4', '#ff7f0e']
markers = ['s', '^']
fig, ax = plt.subplots()
ax.set_title('Original Data')
ax.set_ylabel('Percent_Person_SleepLessThan7Hours')
ax.set_xlabel('Percent_Person_PhysicalInactivity')
for i in range(X.shape[0]):
  ax.scatter(X["Percent_Person_PhysicalInactivity"][i],
             X["Percent_Person_SleepLessThan7Hours"][i],
             c=colors[Y['Label'][i]],
             marker=markers[Y['Label'][i]],
            )
ax.legend([0, 1])
plt.show()
Explanation: Now that we have our features and labels set, it's time to start modeling!
1) Model Selection
The results of our models are only good if our models are correct in the first place. "Good" here can mean different things depending on your application -- we'll talk more about that later in this assignment.
What's important to know for now is that the conclusions we draw are subject to the assumptions and limitations of our underlying model. Thus, making sure we choose the right models to analyze is important! But how does one choose the right model?
1.1) Building Intuition -- Which model do you think is best?
To build some intuition, let's start off with a simple example. We'll use a subset of our data. For ease of visualization, we'll start with just two features, and just 10 cities.
End of explanation
# Classifier 1
classifier1 = svm.SVC()
classifier1.fit(X, Y["Label"])
fig, ax = plt.subplots()
ax.set_title('Classifier 1')
ax.set_ylabel('Percent_Person_SleepLessThan7Hours')
ax.set_xlabel('Percent_Person_PhysicalInactivity')
plot_decision_regions(X.to_numpy(),
                      Y["Label"].to_numpy(),
                      clf=classifier1,
                      legend=2)
plt.show()
print('Accuracy of this classifier is:', classifier1.score(X,Y["Label"]))
# Classifier 2
classifier2 = tree.DecisionTreeClassifier(random_state=0)
classifier2.fit(X, Y["Label"])
fig, ax = plt.subplots()
ax.set_title('Classifier 2')
ax.set_ylabel('Percent_Person_SleepLessThan7Hours')
ax.set_xlabel('Percent_Person_PhysicalInactivity')
plot_decision_regions(X.to_numpy(),
                      Y["Label"].to_numpy(),
                      clf=classifier2,
                      legend=2)
plt.show()
print('Accuracy of this classifier is:', classifier2.score(X,Y["Label"]))
Explanation: The following blocks of code will generate 2 candidate classifiers, for labeling the datapoints as either label 0 (high obesity rate), or label 1 (high obesity rate).
The code will also output an accuracy score, which for this section is defined as: 
$\text{Accuracy} = \frac{\text{# correctly labeled}}{\text{# total datapoints}}$
End of explanation
# Original Data
X_full = df[["Percent_Person_PhysicalInactivity",
                 "Percent_Person_SleepLessThan7Hours"]]
Y_full = df[['Label']]
# Visualize the data
cCycle = ['#1f77b4', '#ff7f0e']
mCycle = ['s', '^']
fig, ax = plt.subplots()
ax.set_title('Original Data')
ax.set_ylabel('Percent_Person_SleepLessThan7Hours')
ax.set_xlabel('Percent_Person_PhysicalInactivity')
for i in range(X_full.shape[0]):
  ax.scatter(X_full["Percent_Person_PhysicalInactivity"][i],
             X_full["Percent_Person_SleepLessThan7Hours"][i],
             c=cCycle[Y_full['Label'][i]],
             marker=mCycle[Y_full['Label'][i]],
            )
ax.legend([0, 1])
plt.show()
# Classifier 1
fig, ax = plt.subplots()
ax.set_title('Classifier 1')
ax.set_ylabel('Percent_Person_SleepLessThan7Hours')
ax.set_xlabel('Percent_Person_PhysicalInactivity')
plot_decision_regions(X_full.to_numpy(),
                      Y_full["Label"].to_numpy(),
                      clf=classifier1,
                      legend=2)
plt.show()
print('Accuracy of this classifier is: %.2f' % classifier1.score(X_full,Y_full["Label"]))
# Classifier 2
fig, ax = plt.subplots()
ax.set_title('Classifier 2')
ax.set_ylabel('Percent_Person_SleepLessThan7Hours')
ax.set_xlabel('Percent_Person_PhysicalInactivity')
plot_decision_regions(X_full.to_numpy(),
                      Y_full["Label"].to_numpy(),
                      clf=classifier2,
                      legend=2)
plt.show()
print('Accuracy of this classifier is: %.2f' % classifier2.score(X_full,Y_full["Label"]))
Explanation: 1A) Which model do you think is better, Classifier 1 or Classifier 2? Explain your reasoning.
1B) Classifier 2 has a higher accuracy than Classifier 1, but has a more complicated decision boundary. Which do you think would generalize best to new data?
1.2) The Importance of Generalizability
So, how did we do? Let's see what happens when we add back the rest of the cities (we'll keep using just the 2 features for ease of visualization.)
End of explanation
# Use all features that aren't obesity
X_large = df.dropna()[[
              "Median_Income_Person",
              "Percent_NoHealthInsurance",
              "Percent_Person_PhysicalInactivity",
              "Percent_Person_SleepLessThan7Hours",
              "Percent_Person_WithHighBloodPressure",
              "Percent_Person_WithMentalHealthNotGood",
              "Percent_Person_WithHighCholesterol",
              "Commute_Time"
              ]]
Y_large = df.dropna()["Label"]
# Standardize the data
scaler = StandardScaler().fit(X_large)
X_large = scaler.transform(X_large)
# Create a model
large_model = linear_model.Perceptron()
large_model.fit(X_large, Y_large)
Explanation: 2A) In light of all the new datapoints, now which classifier do you think is better, Classifer 1 or Classifier 2? Explain your reasoning.
2B) In Question 1, Classifier 1 had a lower accuracy than Classifier 2. After adding more datapoints, we now see the reverse, with Classifier 1 having a higher accuracy than Classifier 2. What happened? Give an explanation (or at least your best guess) for why this is.
2) Evaluation Metrics
In question 1, we were able to visualize how well our models performed by plotting our data and decision boundaries. However, this was only possible because we limited ourselves to just 2 features. Unfortunately for us, humans are only good at visualizing up to 3 dimesnions. As you increase the number of features and/or complexity of your models, creating meaningful visualizations quickly become intractable.
Thus, we'll need other methods to measure how well our models perform. In this section, we'll cover some common strategies for evaluating models.
To start, let's finally fit a model to all of our available data (e.g. 500 cities and 8 features). Because the features have different scales, we'll also take care to standardize their values.
End of explanation
print('Accuracy of the large model is: %.2f' % large_model.score(X_large,Y_large))
Explanation: 2.1) Accuracy
2.1.1) Classification Accuracy
We've seen an example of an evaluation metric already -- accuracy! The accuracy score used question 1 is more commonly known as classification accuracy, and is the most common metric used in classification problems.
As a refresher, the classification accuracy is the ratio of number of correct predictions to the total number of datapoints.
classification accuracy:
$Accuracy = \frac{\text{# correctly labeled}}{\text{# total datapoints}}$
Note that sometimes the classification accuracy can be misleading! Consider the following scenario:
There are two classes, A and B. We have 100 data points in our dataset. Of these 100 data points, 99 points are labeled class A, while only 1 of the data points are labeled class B.
2.1A) Consider a model that always predicts class A. What is the accuracy of this always-A model?
2.1B) How well do you expect the always-A model to perform on new, previously unseen data? Assume the new data follows the same distribution as the original 100 data points.
2.1C) Run the following code block to calculate the classification accuracy of our large model. Is the accuracy higher or lower than you expected?
End of explanation
'''
Try a variety of different splits by changing the test_size variable, which
represents the ratio of points to use in the test set.
For example, for a 75% Training, 25% Test split, use test_size=0.25
'''
test_size = 0.25   # Change me! Enter a value between 0 and 1
print(f'{np.round((1-test_size)*100)}% Training, {(test_size)*100}% Test Split' )
# Randomly split data into Train and Test Sets
x_train, x_test, y_train, y_test = train_test_split(X_large, Y_large, test_size=test_size)
# Fit a model on the training set
large_model.fit(x_train, y_train)
print('The TRAINING accuracy is: %.2f' % large_model.score(x_train, y_train))
# Evaluate on the test Set
print('The TEST accuracy is: %.2f' % large_model.score(x_test, y_test))
Explanation: 2.2) Train/Test Splits
The ability of a model to perform well on new, previously unseen data (drawn from the same distribution as the data used the create the model) is called Generalization. For most applications, we prefer models that generalize well over those that don't.
One way to check the generalizability of a model is to perform an analysis similar to what we did in question 1. We'll take our data, and randomly split it into two subsets, a training set that we'll use to build our model, and a test set, which we'll hold out until the model is complete and use it to evaluate how well our model can generalize to simulate new, previously unseen data.
2.2.1) Choosing Split Sizes
But what percentage of our datapoints should go into our training and test sets respectively? There are no hard and fast rules for this, the right split often depends on the application and how much data we have. The next few questions explore the key tradeoffs:
2.2A) Consider a scenario with 5 data points in the training set and 95 data points in the test set. How accurate of a model do you think we're likely to train? 
2.2B) Does your answer to 2.2A change if we have 500 training and 9500 test points instead?
2.2C) Consider a scenario with 95 data points in the training set and 5 data points in the test set. Is the test accuracy still a good measure of generalizability?
2.2D) Does your answer to 2.2C change if we have 9500 training and 500 test points instead?
2.2.2) Try for yourself!
2.2E) Play around with a couple values of test_size in the code box below. Find a split ratio that seems to work well, and report what that ratio is.
End of explanation
'''
Set the number of folds by changing k.
'''
k = 5 # Enter an integer >=2. Number of folds.
print(f'Test accuracies for {k} splits:')
scores = cross_val_score(large_model, X_large, Y_large, cv=k)
for i in range(k):
  print('\tFold %d: %.2f' % (i+1, scores[i]))
print('Average score across all folds: %.2f' % np.mean(scores))
Explanation: 2.2.3) Training vs. Test Accuracy
As you may have noticed from 2.2.2, we can calculate two different accuracies after performing a train/test split. A training accuracy based on how well the model performs on the data it was trained on, and a test accuracy based on how well the model performs on held out data. Typically, we select models based on test accuracy. Afterall, a model's performance on new data after being deployed is usually more important than how well that model performed on the training data.
So why measure training accuracy at all? It turns out training accuracy is often useful for diagnosing some common issues with models.
For example, consider the following scenario:
After performing a train/test split, a model is found to have 100% training accuracy, but only 33% test accuracy.
2.2F) What's going on with the model in the scenario? Come up with a hypothetical setup that could result in these train and test accuracies.
Hint: This situation is called overfitting. If you're stuck, feel free to look it up!
2.3) Cross-Validation
If you haven't already, run the code box in 2.2.2 multiple times without changing the test_size variable. Notice how the accuracies can be different between runs?
The problem is that each time we randomly select a train/test split, sometimes we'll get luckier or unlucky with a particular distribution of training or test data. To borrow a term from statistics, a sample size of $n=1$ is too small! We can do better.
To get a better estimate of test accuracy, a common strategy is to use k-fold cross-validation. The general proceedure is:
Split the data into $k$ groups.
Then for each group (called a fold):
hold that group out as the test set, and use the remaining groups as a training set.
Fit a new model on the training set and record the resulting accuracy on the test set.
Take the average of all test accuracies.
A Note on Choosing k
The number of folds to use depends on your data. Setting the number of folds implicitly also sets your train/test split ratio. For example, using 10 folds implies 10 (90% train, 10% test) splits. Common choices are $k=10$ or $k=5$.
End of explanation
# Classifier A
x_A = df[["Count_Person",
          "Median_Income_Person"]]
y_A = df["Label"]
classifierA = linear_model.Perceptron()
classifierA.fit(x_A, y_A)
scores = cross_val_score(classifierA, x_A, y_A, cv=5)
print('Classifier A')
print('-------------')
print('Number of Data Points:', x_A.shape[0])
print('Number of Features:', x_A.shape[1])
print('Classification Accuracy: %.2f' % classifierA.score(x_A, y_A))
print('5-Fold Cross Validation Accuracy: %.2f' % np.mean(scores))
# Classifier A
x_A = df[["Percent_Person_PhysicalInactivity",
          "Median_Income_Person"]]
y_A = df["Label"]
classifierA = svm.SVC()
classifierA.fit(x_A, y_A)
scores = cross_val_score(classifierA, x_A, y_A, cv=5)
print('Classifier A')
print('-------------')
print('Number of Data Points:', x_A.shape[0])
print('Number of Features:', x_A.shape[1])
print('Training Classification Accuracy: %.2f' % classifierA.score(x_A, y_A))
print('5-Fold Cross Validation Accuracy: %.2f' % np.mean(scores))
# Classifier B
x_B = df.dropna()[[
          "Percent_NoHealthInsurance",
          "Percent_Person_PhysicalInactivity",
          "Percent_Person_SleepLessThan7Hours",
          "Percent_Person_WithHighBloodPressure",
          "Percent_Person_WithMentalHealthNotGood",
          "Percent_Person_WithHighCholesterol"
]]
y_B = df.dropna()["Label"]
classifierB = tree.DecisionTreeClassifier()
classifierB.fit(x_B, y_B)
scores = cross_val_score(classifierB, x_B, y_B, cv=5)
print('Classifier B')
print('-------------')
print('Number of Data Points:', x_B.shape[0])
print('Number of Features:', x_B.shape[1])
print('Training Classification Accuracy: %.2f' % classifierB.score(x_B, y_B))
print('5-Fold Cross Validation Accuracy: %.2f' % np.mean(scores))
# Classifier C
x_C = df.dropna()[[
          "Percent_NoHealthInsurance",
          "Percent_Person_PhysicalInactivity",
          "Percent_Person_SleepLessThan7Hours",
          "Percent_Person_WithHighBloodPressure",
          "Percent_Person_WithMentalHealthNotGood",
          "Percent_Person_WithHighCholesterol"
]]
y_C = df.dropna()["Label"]
classifierC = linear_model.Perceptron()
classifierC.fit(x_C, y_C)
scores = cross_val_score(classifierC, x_C, y_C, cv=5)
print('Classifier C')
print('-------------')
print('Number of Data Points:', x_C.shape[0])
print('Number of Features:', x_C.shape[1])
print('Training Classification Accuracy: %.2f' % classifierC.score(x_C, y_C))
print('5-Fold Cross Validation Accuracy: %.2f' % np.mean(scores))
Explanation: 2.3A) Play around with the code box above to find a good value of $k$. What happens if $k$ is very large or very small?
2.2B) How does the average score across all folds change with $k$?
2.4) Other Metrics Worth Knowing
2.4.1) What about Regression? -- Mean Squared Error
Different models and different problems often use different accuracy metrics. You may have noticed classification accuracy doesn't make much sense for regression problems, where instead of predicting a label, the model predicts a numeric value. In regression, a common accuracy metric is the Mean Squared Error, or MSE.
$ MSE = \frac{1}{\text{# total data points}}\sum_{\text{all data points}}(\text{predicted value} - \text{actual value})^2$
It is a measure of the average difference between the predicted value and the actual value. The square ($^2$) can seem counterintuitive at first, but offers some nice mathematical properties.
2.4.2) More Classification Metrics
Accuracy alone never tells the full story.There are a number of other metrics borrowed from statistics that are commonly used on classification models.
It's possible for a model to have a high accuracy, but score very low on some of the following metrics:
True Positives: The cases where we predicted positively, and the actual label was positive.
True Negatives: The cases where we predicted negatively, and the actual label was negative.
False Positives: The cases where we predicted positively, but the actual label was negative.
False Negatives: The cases where we predicted negatively, but the actual label was positive.
False Positive Rate: Corresponds to the proportion of negative datapoints incorrectly considered positive relative to all negative points.
$FPR = \frac{FP}{TN + FP}$
Sensitivity: (Also known as True Positive Rate) corresponds to the proportion of positive datapoints correctly considered as positive relative to all positive points.
$TPR = \frac{TP}{TP + FN}$
Specificity: (Also known as True Negative Rate) corresponds to the proportion of negative datapoints correctly considered negative relative to all negative points.
$TNR = \frac{TN}{TN + FP}$
Precision: Proportion of correctly labeled positive points relative to the number of positive predictions
$Precision = \frac{TP}{TP+FP}$
Recall: Proportion of correctly labeled positive points relative to all points that were actually positive.
$Recall = \frac{TP}{TP+FN}$
F1 score: Measure of a balance between precision and recall.
$F1 = 2 \cdot \frac{1}{\frac{1}{precision} + \frac{1}{recall}}$
2.4.3) Tradeoffs Between Metrics
Oftentimes, our definition of a "good" model varies by situation or application case. In some cases, we might prefer a different tradeoff between accuracy, false positive rate, and false negative rate.
2.4) Read through the following scenarios. For each case, state which metrics you would prioritize, and why.
Scenario 1:
Doctors have identified a new extremely rare, but also very deadly disease. Fortunately, they also discover a simple medication, that if taken early enough, can prevent the disease. The doctors plan to use a machine learning model to predict which of their patients are at high-risk for getting the disease. A positively labeled patient is high-risk, while a negatively labeled patient is low-risk.
Scenario 2:
Data Is Cool Inc. is a company that attracts many highly (and equally) qualified applicants to its job posting. They are overwhelmed with the number of applications received, so the company implements a machine learning model to sort all the incoming resumes. A positively labeled resume gets passed to a recruiter for a very thorough, but time-costly review. Negatively labeled resumes are held for future job openings.
3) Tying It All Together -- Choosing A Model to Deploy
Now that we've seen many different evaluation metrics, let's put what we've learned into practice!
One of the most common problems you'll encounter as a data scientist is to decide between a set of candidate models.
Each of the following code boxes below generates a candidate classifier for predicting high vs low obesity rates in cities. The models can differ in different ways: number of features, learning algorithm used, number of datapoints, etc.
End of explanation
your_local_dcid = "geoId/0649670" # Replace with your own!
# Get your local data from data commons
local_data = datacommons_pandas.build_multivariate_dataframe(your_local_dcid,stat_vars_to_query)
# Cleaning and Preprocessing
local_data['DCID'] = local_data.index
city_name_dict = datacommons.get_property_values(city_dcids, 'name')
city_name_dict = {key:value[0] for key, value in city_name_dict.items()}
local_data['City'] = pd.Series(city_name_dict)
local_data.set_index('City', inplace=True)
local_data.rename(columns={"dc/e9gftzl2hm8h9":"Commute_Time"}, inplace=True)
avg_commute_time = local_data["Commute_Time"]/local_data["Count_Person"]
local_data["Commute_Time"] = avg_commute_time
percent_noHealthInsurance = local_data["Count_Person_NoHealthInsurance"]/local_data["Count_Person"]
local_data["Percent_NoHealthInsurance"] = percent_noHealthInsurance
local_data["Label"] = local_data['Percent_Person_Obesity'] >= 30.0
local_data["Label"] = local_data["Label"].astype(int)
# Build data to feed into model
x_local = local_data[[
              "Median_Income_Person",
              "Percent_NoHealthInsurance",
              "Percent_Person_PhysicalInactivity",
              "Percent_Person_SleepLessThan7Hours",
              "Percent_Person_WithHighBloodPressure",
              "Percent_Person_WithMentalHealthNotGood",
              "Percent_Person_WithHighCholesterol",
              "Commute_Time"          
]]
x_local = scaler.transform(x_local)
y_local = local_data["Label"]
# Make Prediction
prediction = large_model.predict(x_local)
# Report Results
print(f'Prediction for {local_data.index[0]}:')
print(f'\tThe predicted label was {prediction[0]}')
print(f'\tThe actual label was {y_local[0]}')
Explanation: 3A) Run the code boxes above and select which model you would choose to deploy. Justify your answer.
3B) Consider a new Classifier D. Its results look like this:
Number of Data Points: 5,000 \
Number of Features: 10,000 \
Training Classification Accuracy: 98% \
5-Fold Cross Validation Accuracy: 95%. 
Would you deploy classifier D? Name one advantage and one disadvantage of such a model.
4) Extension: What about YOUR city?
Now that we've got a model trained up, let's play with it! 
Use the Data Commons Place Explorer to find the DCID of a town or city local to you. 
Use the code box below to add your local town or city's data, and run the model that data.
Note: Data may not be available for all locations. If you encounter errors, please try a different location!
End of explanation
<END_TASK>