rest of the process of building the model. Remember that you will most often be building models for a company or an organization. Because of that, you'll need to ensure that throughout the other stages of PACE that your data, modeling, metrics, and optimizing strategies stay focused on what you developed during the plan stage. Let's explore an example. Suppose that you're working in the finance industry and your firm is trying to predict housing prices. You have a massive data set about houses in a certain area. This data set contains information about the houses, such as square footage, the number of bedrooms and bathrooms, and location. Most importantly, the data set also contains the most recent sale price for each house. As the second part of the plan stage of your PACE workflow, you use the context and requirements of the business need to consider what type of machine learning model would be best suited for the problem of predicting housing prices. Based on what you know of machine learning models so far, you need to create a supervised continuous model to get the desired numerical result. Continuous models are the only ones that can help you predict housing prices for this example. Now that you have your plan for the business problem and the scope for the machine learning model, you're ready for the next step. Analyze. See you soon. I'm Ganesh. I'm a group data science manager at Waymo. So Waymo is a self-driving car part of Alphabet. Our goal is to automate driving for users so that they can just hail a car from anywhere, become pick them up and go them drop off wherever we want to. I manage a team of data scientists whose main goal is to figure out opportunities for optimization in improving the processes at Waymo. Every day is a new challenge because today you're working on one problem, tomorrow it's completely different. And part of the charter that my team does is run experiments for Waymo. So whenever someone thinks of a process change or whenever someone wants to launch a new feature, we use experiments or A-B test as you call it to figure out is this change truly helpful? Is it working? What is the expected impact we're gonna get from these changes? So there was this time, wherein we were conducting an A-B test to measure the improvements that a process is going to have. We had a control group who was doing the old process and then we had a test group who was doing the new process. We were expecting a lot of improvements to come from that because we were super sure, we did some demos and we thought that the new process is significantly better than the old process. But when we launched it, nothing really happened. We waited, nothing happened again. This was hard because a lot of program managers, a lot of product managers have put their effort into this. We did some field studies to go and see why it was wrong. The users were so accustomed to the old process that they weren't willing to try the new process at all. They were going back to the old ways and that's why our test wasn't performing as expected. That was a learning moment for us and for me especially, right? Because this is a big part of launching an A-B test. And that's when I realized that you don't wanna launch process changes with existing users or customers. It's always good to use your new users to try these new processes because that way you get an unbiased opinion of what is happening and that way you can truly measure the impact of your new process. I mean, had the problem not been resolved, we would have thought that the new process is bad and not launched a thing that is truly helping the company. So resolving this, figuring out what the problem was, made sure that we didn't throw away that and we launched something that was truly important for the company. It's okay to make mistakes. You're encountering situations where many people are in for the first time. So it's okay. It's okay to ask for help. It's okay to make those mistakes. But the big part is we shouldn't be repeating those mistakes again and again. So having this learning mentality of trying and doing is gonna be the most important thing that any data scientist has to build. Earlier, we explored the planned stage of our PACE framework. Now that you've finished preliminary planning, it's time for the analyzed stage of PACE. While keeping the business need in focus is particularly true for the planned stage, it's also essential throughout the process of developing a machine learning model. The business need informs a data professional on what the model needs to produce. The result indicates what type of model is required. The business need also informs data professionals about the data that's necessary to train the model to achieve the desired result. The main focus of the analyzed stage is to develop a deeper understanding of the data while keeping in mind what the model needs to eventually predict. For example, if you're creating a supervised learning model, the first thing you'll need to know is what your model is trying to predict. In other words, you'll need to understand your response variable. You've already done this earlier in the course when you were deciding which type of supervised learning model to use, continuous or categorical. Let's use weather as an example. If you need to predict the exact amount of precipitation in inches, you'll probably go with a continuous model. But if the business needs a model that can predict whether it will be rainy, cloudy or sunny, that requires a categorical model. Often as a data professional, your data isn't structured exactly the way you need it to be. This is where you can take advantage of many of the exploratory data analysis principles you learned earlier. You'll be using all the techniques you learned so far to develop an understanding of what data you have available and how it's structured. Let's return to the continuous model example about predicting precipitation in inches. In your data set, the rainfall amounts that were recorded might not be in the exact units needed. Or the data could be split between rain, snow and other types of precipitation. These are all details that a data professional needs to analyze before building the model itself. For the categorical model example, where you're predicting whether it's rainy, cloudy or sunny, your data set might not be labeled with the categories you need. The individual days in the data set might be labeled with only cloud cover metrics, which is something you'll have to change to be able to analyze the results effectively. After getting a solid understanding of what your response variables are and how they're structured, the next step is exploring your predictor variables. Understanding the relationships that exist between variables in your data set is essential to building a model that will produce valuable results. Similar to the response variables, the predictor variables might not be in the format or style that you want. In those cases, the same considerations apply. You'll need to figure out how you want your data structured before building your model. This process of carefully considering the variables you have and what you need leads into the next part of the analyze stage, feature engineering. Coming up, you'll learn about feature engineering, various techniques that are available, which scenarios to use them in and what they can do for your data. Welcome back. Previously, you learned that the main focus of the analyze stage is to develop a deeper understanding of the data. Carefully considering the variables you have and what you need leads into the next part of the analyze stage, feature engineering. Feature engineering techniques solve the problems in how your data is structured and if done well, can improve your model's performance. In this video, you'll learn more about feature engineering, how it works and when to use it. Feature engineering is the process of using practical, statistical and data science knowledge to select, transform or extract characteristics, properties and attributes from raw data. This definition has quite a bit to consider. First, the process of feature engineering is highly dependent on the type of data you're working with. Before we go much further, let's check out some examples. Earlier, you learned about types of features or variables called continuous and categorical. Remember, continuous variables are variables with values obtained by measurement. As a result, they can take on an infinite and uncountable set of values. Categorical variables on the other hand are variables that contain a finite number of groups, categories or countable numerical values. Your process for feature engineering will be changing and altering these variables with the end goal of using them to train a model. This can often be a challenging process. Data sets used in the workplace can sometimes require multiple rounds of EDA and feature engineering to get everything in a suitable format to train a model. This process highlights one reason why the PACE framework is so beneficial to data professionals. The analyze stage builds directly from the plan stage or put more simply, the plan informs how you analyze. Without a strategically aligned business and technical plan, the analyze stage and feature engineering process would be like trying to build a skyscraper without having blueprints. The three general categories of feature engineering are feature selection, transformation and extraction. Let's learn more about those categories. We'll start with the feature selection. The goal of this type of feature engineering is to select the features in the data that contribute the most to predicting your response variable. In other words, you drop features that do not help in making a prediction. This can be done either manually or algorithmically. Let's use a simple weather data set as an example. It contains five different variables and 14 data points. The variable on the far right represents whether or not you would wanna place soccer or football outside based on the other data. In our example, whether or not it's windy outside might not affect our playing soccer. If that's the case, then we would want to select outlook, temperature and humidity and drop windy from our data set. Features selection would mean selecting the variables that help the most in making a prediction. The next category is feature transformation. In feature transformation, data professionals take the raw data in the data set and create features that are suitable for modeling. This process is done by modifying the existing features in a way that improves accuracy when training the model. In our weather data set example, your data might include exact temperatures, but you might only need a feature that indicates if it's hot, cold or temperate. To make that transformation, you could define some cutoff points for the data and create a new categorical feature from the numerical data. You might define anything above 80 degrees Fahrenheit as hot, anything below 70 as cold and anything in between as temperate. Feature transformation would mean transforming the degrees into the temperature categories you've defined. Finally, let's discuss feature extraction. This type of feature engineering involves taking multiple features to create a new one that would improve the accuracy of the algorithm. For example, imagine we want to create a new variable called muggy that could be used to model whether or not we play soccer. If the temperature is warm and the humidity is high, the variable muggy would be true. If either temperature or humidity is low, then muggy would be false. Remember, the feature engineering techniques are designed to improve your model's performance. While you can make a lot of improvements by tweaking and optimizing the model, the most sizable performance increases often come from developing your variables into a format that will best work for the model. In my work, this often appears as needing to make an outcome variable binary. For example, we may get survey ratings from users that are on a scale of one to five stars, but we need to predict whether a piece of content is good or bad. In this case, we need to change our response variable by mapping the star ratings to either the good or bad label. This is one example of how data professionals better understand their data in the analyzed stage. During EDA, you're just beginning to develop an understanding of your data. Feature engineering is a step beyond EDA. You're selecting, extracting, or transforming variables or features from data sets for the construction of machine learning models. Now that you have a better idea of the concept of feature engineering, you're ready to perform some of your own feature engineering in Python, which you'll do in a later video. Hello again. In this video, we'll continue our exploration of the analyzed stage of pace. Understanding what your variables are and how they're structured is only part of the process. It's also essential to understand the frequency in which the variables exist. For classification problems, you need to specifically understand the frequencies of the response variable. As a data professional, you might encounter data sets that are unequal in terms of their response variables. One example of unequal data sets is in the context of fraud detection. You could have millions of examples of non-fraudulent transactions and only a few thousand examples of actual fraudulent transactions. How can a model be built to detect fraud with such limited data to train the model? This issue is known as class imbalance. A class imbalance is when a data set has a predictor variable that contains more instances of one outcome than another. The class with more instances is called the majority class, while the class with fewer instances is called the minority class. It's extremely rare for a data set to have a perfect 50-50 split of the outcomes. There is normally some degree of imbalance. However, this isn't necessarily a problem. Believe it or not, a 70-30 or 80-20 split can be fine. Major issues only arise when the majority class makes up 90% or more of the data set. You'll only know if there's an imbalance issue after the model is built. There are two techniques that allow us to fix any potential issues, up sampling and down sampling. Both of them involve altering the data in a way that preserves the information contained in the data while removing the imbalance. Down sampling involves altering the majority class by using less of the original data set to produce a split that's more even. The number of entries of the majority class decreases, leading to more of a balance. You can use different techniques to achieve this, but generally they are all based on this concept. One technique is to do this randomly by selecting entries to remove, or you can follow a formula. For example, you can take the mean of two data points in the majority class, remove those data points and add the average data point. Up sampling is the opposite of down sampling. Instead of reducing the frequency of the majority class, you artificially increase the frequency of the minority class. Similar to down sampling, there are multiple ways you can achieve this. The simplest technique is called random oversampling, where random copies of data points in the minority class are copied and added back to the data set. Or mathematical techniques can be used to generate non-identical copies, which are then also added to the data set. So if both up sampling and down sampling achieve the same result, you might be wondering which one to use. Most of the time, you won't know which one is preferred until you've built the model and observed how it performs. However, there are some general rules you can follow regarding when to up sample and when to down sample. Down sampling is normally more effective when working with extremely large data sets. If you have a data set that has 100 million points, but has a class imbalance, you don't need all of that data to build a good model. You definitely don't need the additional data that would come from up sampling. Alternatively, up sampling can be better when working with a small data set. If you're working with a data set that only has 10,000 entries, removing any of that data will more than likely have a negative impact on the model's performance. Keep in mind, class balancing is a fickle process and may require some trial and error. Building models with both up sampled data and down sampled data will determine which technique is better in any given situation. Additionally, you'll have to experiment with what sort of split you're rebalancing achieves. Balancing the data so the classes are split 50-50 might not always be optimal. On the other hand, turning a 99-1 split into a 70-30 split might be fine and that is something to consider as you develop your model. Welcome back. In this video, you'll code in Python to wrap up the analyze stage of the PACE framework. In the analyze stage, you gain a deeper understanding of your data. This is the stage where you prepare your data so it can be used to train the model. Then you'll move on to the construct stage of the PACE workflow. For the rest of this part of the course, we'll build models that will predict customer churn at a bank. Customer churn is the business term that describes how many and at what rate customers stop using a product or service or stop doing business with a company altogether. The models will be supervised classification models because they'll each predict a categorical target. In this case, it's a binary one, whether or not each customer churned or stayed. Before we get into the data set, we need to import the packages that we'll use. All we'll need for this notebook are NumPy and Pandas because we're only preparing the data for modeling. Now, we'll read in the data set from a CSV file to a Pandas data frame. We'll call the data frame DF original and inspect it using the head function. The data set that we'll be using to solve this problem contains customer data, where each entry in the data set represents one customer. For each customer, there are several features that describe the customer's relationship with the bank and information about the customer's finances. Additionally, there's metadata for each customer, like name, gender, and customer identification number. The two features that might not be completely intuitive are tenure, which represents how many years the customer has used the bank, and geography, which identifies which country the customer lives in. Additionally, we have the feature labeled exited. This indicates whether the customer left the bank. A one signifies that they stopped doing business with the bank and a zero indicates that they are still a customer. The variable exited will be the response variable or the variable that our model will attempt to predict. When modeling, the best practice is to perform a rigorous examination of your data before beginning feature engineering and feature selection. This process is important because not only does it help you understand your data, what it's telling you and what it's not telling you, but it can also give you clues about what new features to create. You've already learned the fundamentals of exploratory data analysis or EDA. So this notebook will skip that essential part of the modeling process. Just remember that a good data science project will always include EDA. Let's get a quick overview of our data. We'll use the info function to inspect the data frame. From this table, we can confirm that the data has 14 features and 10,000 observations. We also know that nine features are integers, two are floats, and three are strings. Finally, we can tell that there are no null values because there are 10,000 observations and each column has 10,000 non-null values. Next, we'll prepare this data set to be used to train the model. The first thing we're going to do is feature selection. If you recall, this is the process of picking out the features that we want the model to use to predict the outcome. And we drop any features that aren't useful to the model. In our bank data, notice that the first column is called row number. This just enumerates the rows. Since a row number shouldn't have any intrinsic correlation with our response variable, we'll remove this feature from our data set. The same is true for customer ID, which appears to be a number assigned to the customer for administrative purposes and surname, which is the customer's last name. We'll drop these two. As you complete feature selection, keep the ethical concerns you learned earlier in the course in mind. Consider the implications of your data and the resultant model. For example, in this modeling exercise, we will not include the gender column. This data feature raises a set of complex issues, technically, culturally, and ethically. We recognize that the most rigorous approach would be to model both with and without this feature and examine how it influences predictions. Whatever the approach, it should be driven by an aim for equitable outcomes and for your particular use case. We'll remove these columns by calling the drop function on our data frame. We pass to it a list of names of the columns that we want to remove, and we indicate that we're dropping columns, not rows, by including axis equals one. We'll assign the results to a new data frame called churnDF. The resultant data frame is shown by calling the head method. However, there's still more to do before we start training the model. Next, let's practice feature extraction. This is the process of taking two or more features and using them to create a brand new feature that will make the model more accurate. Normally, feature extraction is done using statistics to analyze how predictive each variable is and whether the new feature that is extracted is more predictive than the original variables on their own. For now, we're going to extract a feature as an example. Without conducting the analysis, we would normally perform if we were trying to build a production-ready model. Let's create a new variable and call it loyalty. We'll do this by taking the tenure of a customer and dividing it by their age. The logic behind using tenure and dividing it by the customer's age is that it represents the percentage of a person's life that they've been customers of the bank. People with greater percentages may be more loyal customers. Now we have a new column called loyalty, which we can verify by inspecting the data frame. Let's move on to feature transformation. There are some features in this data set that need to be transformed. Remember, feature transformation is the process of changing how a single feature is represented in the data set with the goal of improving the accuracy of the model. Many classification models require you to convert categorical features to make them numeric. Our data set has one categorical feature called geography. Let's check how many classes appear in the data for this feature by using the unique function on the series. There are three unique values, France, Spain, and Germany. Let's encode this data so it can be represented using Boolean features. We'll use a pandas function called get dummies to do this. When we call pd.getdummies on this feature, it will replace the geography column with three new Boolean columns, one for each possible category contained in the column being dummied. When we specify drop first equals true in the function call, it means that instead of replacing geography with three new columns, it will instead replace it with two columns. We can do this because no information is lost and the data set is shorter and simpler. In this case, we end up with two new columns called geography Germany and geography Spain. We don't need a geography France column. Why not? Because of a customer's values in geography Germany and geography Spain are both zero, we'll know they're from France. After feature selection, extraction, and transformation, the next step is modeling. We'll be using this data set for most of the modeling you'll do in the remainder of this part of the course. And you now have a solid foundation for the construct and execute stages of pace. Hello again. You're now about halfway through the pace workflow. During the planning stage, you developed a better understanding of the business need and the data available. And in the analyze stage, you investigated the data using exploratory data analysis practices. You applied feature engineering techniques to select, transform, and extract data into a format that was suitable for training. This video continues on to the construct stage where you'll bring the model to life. You'll do this by building a model called naive Bayes. Naive Bayes is a supervised classification technique based on Bayes theorem with an assumption of independence among predictors. The effect of the value of a predictor variable on a given class is not affected by the values of other predictors. Let's break it down. Bayes theorem gives us a method of calculating the posterior probability, which is the likelihood of an event occurring after taking into consideration new information. In other words, when you calculate the probability of something happening, you take relevant observations into account. It can be represented with this equation, which calculates the posterior probability of C given X. Probability of X given C is the probability of the predictor given the class, which is multiplied by P of C, the prior probability of the class. The product of these two terms is then divided by P of X, which is the prior probability of the predictor. The posterior probability equation can be rewritten to reveal what's going on behind the variables. The probability of a class given the first predictor variable P of X1 given C is multiplied by the probability of a class given the second variable P of X2 given C, and so on for all predictor variables used in the model. This is complex. So let's return to the weather-based example that we discussed in an earlier video and apply naive Bayes to gain a better understanding. The weather data set will help you build a model to decide whether to go outside and play soccer. This data set has five columns. The first four are the predictor variables, and the final column is the label of the data set. It shows whether we should play soccer. The outlook variable identifies if it is rainy, cloudy, or sunny. The humidity variable indicates the relative humidity, and of course, the windy variable determines if there's wind. Start with the outlook variable. Calculate the posterior probability of one of the features in the data set. To do this, construct a frequency table for each attribute against the target by tallying the number of times soccer is and isn't played for a given attribute. Then, transform the frequency tables into likelihood tables by calculating the number of times soccer is and isn't played for each attribute. Use this information to find the probability of the predictor given the class P of X given C, the probability of the class P of C, and the probability of the predictor P of X. In other words, everything needed to calculate the posterior probability. As a reminder, you may pause and review the video if needed. The process of finding the posterior probability needs to be done for every possible class that is potentially being predicted. In this case, there are only two outcomes, play or don't play. Once these values are found, the prediction is made based on the class with the highest posterior probability. When you repeat the process for the second class, don't play, you're only required to modify the calculations after finding the likelihood table. Observe that the posterior probability of playing while it is sunny is higher than the posterior probability of not playing. So if it's sunny outside, a naive Bayes model would predict that the conditions are right to play soccer. Later, you'll explore how multiple predictor variables can be used to make a prediction. All the same concepts will apply. No matter the number of variables that are used, naive Bayes calculates posterior probabilities and makes predictions based on which outcome has the highest probability. You're doing great work. Keep up the momentum and I'll be with you again soon. Now that we've discussed how naive Bayes works, it's time to implement it using Python. We're going to continue our work with the bank churn data frame that we prepared in the feature engineering notebook. Remember, we dropped the row number, customer ID, surname and gender columns. dummy encoded the geography column to convert from categorical to Boolean and engineered a new feature called loyalty by dividing each customer's tenure by their age. Recall that the predictor variables of this data set are of different types. For example, balance and estimated salary are continuous while geography is categorical. Also, remember that Scikit-learn has a few different implementations of the naive Bayes algorithm. And each assumes that all of your predictor variables are of a single type. As a data professional, one of the first things you'll learn on the job is that real world data is never perfect. Sometimes the data violates the assumptions of your model. In practice, you'll have to do the best you can with what you have. For this lesson, we're going to use the Gaussian NB classifier. This implementation assumes that all of your variables are continuous and that they have a Gaussian or normal distribution. Our data doesn't perfectly adhere to these assumptions, but a Gaussian model may still give us usable results, even with imperfect data. Let's get started. As always, the first thing to do is import any packages and libraries that you'll need. We'll begin by importing NumPy, Pandas, and Matplotlib. We'll also import train test split to help us split our data into training and test sets. The model we'll be using is called Gaussian NB, which we'll import from scikit-learn's naive Bayes module. Next, we'll import functions that we'll use to calculate our model's accuracy, precision, recall, and F1 scores. Finally, we'll import confusion matrix and confusion matrix display, which will help us calculate and plot a confusion matrix of our model's results. Let's read in the data frame and call it churnDF. Before we begin modeling, let's do a couple more things. First, we'll check the class balance of the exited column, which is our target variable. We can do this by calling value counts on the Pandas series. The class is split roughly 80-20. In other words, about 20% of the people in this data set churned. This is an unbalanced data set, but it's not extreme. So we'll proceed without doing any class rebalancing of our target variable. Secondly, naive Bayes models operate best when the predictor variables are conditionally independent from each other. When we prepared our data, we engineered a feature called loyalty by dividing tenure by age. Because this new feature is just the quotient of two existing variables, it's no longer conditionally independent. So we're going to drop tenure and age. This step may or may not be beneficial, but we'll do it to help adhere to the assumptions of our model. We've prepared our data and we're ready to model. Now we need to split the data first into features and target variable, and then into training data and test data. Let's assign our predictive features to a variable called x, and the exited column, our target, to a variable called y. Then we can split into training and test data. We do this using the train test split function. We'll put 25% of the data into our test set and use the remaining 75% to train the model. Notice that we include the argument stratify equals y. If our master data has a class split of 80-20, stratifying ensures that this proportion is maintained in both the training and test data. Equals y tells the function that it should use the class ratio found in the y variable, which is our target. The less data you have overall and the greater your class imbalance, the more important it is to stratify when you split the data. If we didn't stratify, then the function would split the data randomly, and we could get an unlucky split that doesn't get any of the minority class in the test data. In that case, we wouldn't be able to effectively evaluate our model. Worst of all, we might not even realize what went wrong without doing some detective work. Finally, we set a random seed, so we and others can reproduce our work. Now it's time to build the model. Just as with linear and logistic regression, our modeling process will begin with fitting our model to the training data and then using the model to make predictions on the test data. First, we'll instantiate the Gaussian NB model, assigning it to a variable called GNB. Then, we'll fit it to the X and Y training data. Lastly, we'll use the predict method to use the model to make predictions on the X test data, assigning the results to a variable called Y PREDS. Now we can check how our model performs using the evaluation metrics we imported. For each one, we pass to it first the actual Y test data, and then the predictions. This isn't very good. Our precision, recall, and F1 scores are all zero. What's going on? Well, let's consider our precision formula. There are two ways for the model to have a precision of zero. The first is if the numerator is zero, which would mean that our model didn't predict any true positives. The second is if the denominator is also zero, which would mean that our model didn't predict any positives at all. Dividing by zero results in an undefined value, but scikit-learn will return a value of zero in this case. Depending on your modeling environment, you may get a warning that tells you there's a denominator of zero. We don't have a warning, so let's check which situation is occurring here. To do this, we'll call NumPy's unique function on the predictions. Okay, the model predicted zero, or not churned, for every sample in the test data. Both the numerator and the denominator are zero. Consider why this might be. Perhaps we did something wrong in our modeling process, or maybe using Gaussian NB on predictor variables of different types and distributions just doesn't make a good model. Maybe there were problems with the data. Before we give up, maybe the data can give us some insight into what might be happening, or what further steps we can take. Let's use describe to inspect the X data. Something that stands out is that the loyalty variable we created is on a vastly different scale than some of the other variables we have, such as balance or estimated salary. The maximum value of loyalty is 0.56, while the maximum value for balance is over 250,000, almost six orders of magnitude greater. One thing that you can try when modeling is scaling your predictor variables. Some models require you to scale the data in order for them to operate as expected, while others don't. Naive Bayes does not require data scaling. However, sometimes packages and libraries need to make assumptions and approximations in their calculations. We're already breaking some of these assumptions by using the Gaussian NB classifier on this data set, and it may not be helping that some of our predictor variables are on very different scales. In general, scaling might not improve the model, but it probably won't make it worse. Let's try it. We'll use a function called minmax scalar, which we'll import from the sklearn pre-processing module. Minmax scalar normalizes each column, so every value falls in the range of zero to one. The column's maximum value would scale to one, and its minimum value would scale to zero. Everything else would fall somewhere in between. This is the formula. To use a scalar, you must fit it to the training data and transform both the training data and the test data using that same scalar. Let's apply this and retrain the model. First, import the scalar. Then, we'll instantiate it and assign it to a variable called scalar. Now, we fit the scalar by passing our X-train data to it. Next, we use the transform method to scale the X-training data. Finally, we transform the X-test data.  When we calculate the performance metrics for this model, we don't get an error. The model isn't perfect, but at least it's now predicting customers who churn. Let's examine more closely how our model classified the test data. We'll do this with a confusion matrix. Remember that a confusion matrix is a graphic that shows you your model's true and false positives and true and false negatives. We can plot this using the confusion matrix display and confusion matrix functions that we imported. Here's a helper function that will allow us to plot a confusion matrix for our model. All of our model metrics can be derived from the confusion matrix and each metric tells its own part of the story. What stands out most in the confusion matrix is that the model misses a lot of customers who will churn. In other words, there are a lot of false negatives, 355 to be exact. This is why our recall score is only 0.303. Coming up, you'll investigate the various model evaluation metrics and when to use them. You'll explore the use of several metrics to evaluate model performance. Then you will determine which of the models best satisfies the business requirements for the data project. Meet you there. Wow, you've reached the last stage of the PACE workflow. Execute. This is where model analysis happens and it's finally production ready. You've already learned a lot about model metrics, the options available to you and what they can demonstrate to a data professional about the model that has been built. And you've built supervised learning models, including a categorical model in the form of logistic regression. The metrics you used to evaluate those models will also make it possible to evaluate a naive Bayes model. As a review, accuracy reflects the number of correct predictions divided by the total number of predictions. However, accuracy doesn't always tell the full story. Some data sets will feature a strong class imbalance which occurs when the majority of items belong to only one class. Then the data set is considered imbalanced. Here's an example using a binary classification problem. An IT professional wants to use a model to detect malware in the computers at their company. Perhaps there are 5,000 instances in their data set but only 500 positive instances where malware was actually present in a computer. This person would have an imbalanced data set as the chances of finding malware among all the checks that happen is actually comparatively low. This is where the precision and recall metrics can help. Precision measures what proportion of positive predictions were correct. In other words, if the model predicted that malware was going to be present, how many times was it actually on a computer? Precision is calculated by dividing the number of true positives by the sum of the true and false positives. And the recall metric, on the other hand, finds the proportion of actual positives that were identified correctly. In the context of our example, recall indicates how many would-be malware threats were classified. Recall is calculated by dividing the number of true positives by the sum of the true positives and false negatives. F1 score combines both precision and recall in one metric. Accuracy, precision, recall, and F1 score are top metrics in classification techniques.  In any case, data professionals use all four metrics to evaluate categorical supervised learning models. As you've begun to discover, each model performs differently and some algorithms work better than others. When building any model intended for production, it's essential to improve the results. You might change specific parameters to discover how the performance improves. So, you should always keep in mind that model building is an inherently iterative process. The first model that you produce will almost never be the one that gets deployed. The iterative process provides the information needed to get the model working optimally. After tweaking the parameters or changing how features are engineered in each model, the performance metrics provide a basis for comparing the models to each other and against themselves. Coming up, we'll find out what these metrics reveal about our model, use them to evaluate other models we've built, and examine how to improve model performance. Continuous improvement is a key part of being a data professional, so these exercises are preparing you to keep advancing all kinds of data processes. Developing a machine learning model is a complex process, but having a solid framework to rely on helps set you up for success, no matter the business need. In this part of the course, the PACE workflow provided support to help you navigate the different stages of addressing an example business scenario. During the plan stage, you assessed the business need to determine what type of model is best suited for predicting bank customer churn. This decision was based on the available data. Next, in the analyze stage, you examined the data using EDA practices and feature engineering techniques. This process revealed more details about the data to help inform your plans for building the model. From there, you went on to construct the first iteration of your naive base model. You then tested the model with preliminary evaluation metrics to determine its performance against the test data. And finally, in the execute stage, you closely evaluated the model's performance and considered how it could be improved. And that is the PACE workflow for machine learning models. Having this process in your tool belt will allow you to solve many business problems. While the models may very well get more complex as you continue your journey, sticking within the framework will help you achieve the results you need. You've come a long way on your journey into the world of data and you've been building a strong foundation for data modeling. So far, you've learned about linear regression, logistic regression, and naive base models. All supervised learning techniques that make predictions on labeled data. Most machine learning applications today are based on supervised learning. But when we consider all the available data in the world, the vast majority of it is unlabeled. Photographs, voice recordings, videos, social media posts, these are all examples of unlabeled data. You may be familiar with this concept in the context of data analytics. If you earned your Google Data Analytics career certificate, you learned that any data that's not organized in an easily identifiable manner is known as unstructured. In this program, we'll sometimes refer to it as unlabeled, but the meaning is the same. So how do we make sense of all that unlabeled data? We use unsupervised learning techniques. When our data is unlabeled, these methods make it possible for data professionals to learn about the data's underlying structure and find out how different features relate to each other. Earlier in the course, you explored one very common type of unsupervised learning, recommendation systems. You learned that they're a subclass of machine learning algorithms which offer relevant suggestions to users, such as new songs for your playlist or a new coat for winter. Now, you'll get to know many other exciting methodologies and applications of unsupervised learning. In this section of the course, you'll first learn about K-means, an unsupervised modeling technique. You'll investigate how it clusters data based on each observation's similarity to others in the data. You'll also build a K-means model and learn how to evaluate it using metrics called inertia and silhouette score. I'm thrilled to have you with me as we explore unsupervised learning. There's so much great potential for future development. We've only just begun tapping into the vast amount of unstructured data in the world. Let's start modeling. In this video, we'll introduce you to the K-means algorithm. K-means is an unsupervised partitioning algorithm. It's used to organize unlabeled data into groups or clusters. It does this by creating a logical scheme to make sense of the data. With K-means, each cluster is defined by a central point or a centroid. Its position represents the center of the cluster, also known as the mathematical mean, hence the name K-means. There are four steps to building a K-means model. Let's examine them one at a time. In step one, you choose the number of centroids and place them in the data space. K represents the number of centroids in your model, which is how many clusters you'll have. This is a decision that you make. Sometimes you'll have an idea about the number of clusters necessary for a project. For example, if your company manufactures five different products, you might want to set your K value to five. Other times, you won't know how many clusters your data should be split in two. So try different values for K and determine what provides the best results. Here, it's apparent that our data is grouped into two clusters, one on top and the other on bottom. At step one, we'll randomly initiate two centroids represented by the blue and red Xs. Step two is to assign each data point to its nearest centroid. The nearest centroid is the one that's closest in space. In this example, the top two observations are assigned to the blue centroid and the bottom two observations are assigned to the red centroid. As a quick refresher, in this context, an observation is simply any data point being observed. Step three is to recalculate the centroid of each cluster. Again, the centroid's location is calculated by taking the mean of all of the points in its cluster. Note that the centroids move to the midpoint of their clusters. This will happen each iteration until the algorithm reaches convergence. This is the stable point found at the end of a sequence of solutions. Step four is to repeat steps two and three until the algorithm converges. In this case, we have very little data, so the model is simple and has already converged. If you had a lot more data, those centroids would get increasingly closer to their related clusters. You also might find that the cluster assignment of each data point changes as the centroid locations move within each iteration. Something to be mindful of is that it's important to run the model with different starting positions for the centroids. This helps avoid poor clustering caused by local minima. In other words, not having an appropriate distance between clusters. Let's explore this concept using our example. What if this had been the initial positions of our centroids? Notice what happens. In step two, we assign the points to their nearest centroid. With these particular starting positions, the two observations on the left are assigned to the red cluster, and the two observations on the right are assigned to the blue cluster. For step three, we recalculate the position of each cluster centroid. The model has converged, but the clusters aren't what we'd expect. We know that the most intuitive clustering would be for the top two observations to be in one cluster and the bottom two in another. But that's not what happened. And further iterations will not change this. This is why it's important to run the model with different centroid initializations and to avoid poor clustering due to the model converging in local minima. Note that this clustering isn't wrong. It's still a valid resolution of the model. It just doesn't make much sense in this context. After all, the goal is to find a clustering scheme that makes sense of your data. Finally, note that even though K means is a partitioning algorithm, data professionals typically talk about it as a clustering algorithm. The difference is that outlying points in clustering algorithms can exist outside of the clusters. However, for partitioning algorithms, all points must be assigned to a cluster. In other words, K means does not allow unassigned outliers. So to recap, K means is an unsupervised learning technique that groups unlabeled data into K clusters based on similarity. The clustering process has four steps that repeat until the model converges. The value for K is a decision that the modeler makes. And finally, it's important to build multiple models to avoid poor clustering. Later in this section, you'll learn how to determine the best value for K. You'll also discover some of the limitations of K means models. Lots more coming up. At this point, you're familiar with the basic intuition behind the K means algorithm. In this video, we're going to demonstrate how to apply your knowledge of K means to an actual example. We'll use the K means algorithm to compress colors in a photographic image. This demonstration is intended to lead you through an application of the K means theory to give you a deeper understanding of how it works. So for this video, focus less on the mechanics of the code itself and more on the results. Let's get started. We're going to use a photograph of some tulips as our data, which we'll read in as an array using Matplotlib's I am read function and display using its I am show function. This is the image that we're going to manipulate using K means. When we check the shape of the image in pixels, we're told that it's 320 by 240 by three. We can interpret these numbers as pixel information. Each dot on the screen is a pixel. This photograph has 320 vertical pixels and 240 horizontal pixels. But what is the dimension of three? This dimension refers to the values that encode the color of each pixel. Each pixel has three parameters, red or R, green or G, and blue or B. Together, these values are known as RGB values. The value for each color, R, G and B, can range from zero to 255. This means that there are 256 cubed or more than 16 million different possible combinations of RGB, each resulting in a unique color. To prepare this data for modeling, we'll reshape it into an array where each row represents a single pixels RGB color values. Now we have an array that is 76,800 by three. Each row is a single pixels color values. Let's create a panda's data frame to help us understand and visualize this data. Each row of the data frame represents a single pixel and the three columns are its R, G and B values. Because we have only three columns, we can visualize this data in three-dimensional space. This graph plots each of the photograph's pixels in a 3D coordinate space. Each axis ranges from zero to 255, just like each value in RGB and each dot in the graph is the color specified by its RGB values, just like in the original photograph. The more intense the color, the more dots are concentrated in that area. The most represented colors here are the most abundant colors in the photograph, mostly reds, greens and yellows. We can examine this graph from different angles and even zoom in and out. We can also train a K-means model on this data. The algorithm would create K clusters by minimizing the squared distances from each point to its nearest centroid. Here's an experiment. What do you expect to happen if we built a K-means model with just a single centroid? In other words, with K equal to one. Let's find out. We'll first instantiate the model. As a refresher, instantiation involves creating a copy of the class, which inherits all class variables and methods. So, let's set the number of clusters to one and fit it to our data. Now we're going to copy the original image, replace each of its rows with the values of its closest cluster center and reshape the image so we can display it. The image we get back doesn't resemble tulips at all. So what happened? Well, let's run through the K-means steps. First, the algorithm randomly placed a centroid in the 255 by 255 by 255 color space. Then it assigned each point to its nearest centroid because there was only one centroid, all points were assigned to it and therefore to the same cluster. Next, the algorithm updated the centroid's location to the mean location of all of its points. Again, there's only a single centroid, so it updated to the mean location of every point in the image. Usually these steps would repeat until the model converges, but in this case, it took only one iteration. We updated each pixel's RGB values to be the same as those of our centroid. The result is the image of our tulips where every pixel is replaced with the average color. We can verify this for ourselves by manually calculating the average for each column in the array. This will return the mean R value, G value and B value. Let's compare this to what the K-means model calculated as the final location of its one centroid. We'll do this by using the cluster centers attribute of the fit K-means model object. They're the same. Now let's return to the 3D rendering of our color space. Only this time, we'll add the centroid. The centroid is a large circle in the middle of the color space. Notice that this is the center of gravity, so to speak, of all the points in the graph. Okay, now let's refit another K-means model to the data, only this time using K equals 3. Take a moment now to consider what you might expect to result from this. Go through the steps of what the model is doing like we did above. What colors are you likely to see? So we refit the model setting our number of clusters to three and we get three centroid locations, which are the RGB values we can use to display the colors of each centroid. We'll use a helper function to display our color swatches and there they are. You might have hypothesized that there would be similar colors as a result of the three cluster models and that's correct. The photos dominant colors of red, green and yellow are present here. Again, we can replace each pixel in the original image with the RGB values of the centroid to which it was assigned by the new K-means model. This is a function that will display the photo for any value of K that we choose. We'll call this function with three as its argument. We now have a photo with just three colors, the same three colors from the swatches above. Each color's RGB values correspond to the values of the location of its nearest centroid. We can return once more to our 3D coordinate space. This time, we'll recolor each dot to correspond with the color of its centroid. This will allow us to see how the K-means algorithm clustered our data spatially. Check it out. Each pixel is now colored according to the RGB value of its centroid and the clusters are at the vertices of the space. This whole process can be applied for any value of K. Here's the output of each photo for K equals two through 10. Notice that it becomes increasingly difficult to see the difference between the images each time a color is added. This is a visual example of something that happens with all clustering models, even if the data is not an image that you can see. As you group the data into more and more clusters, additional clusters beyond a certain point contribute less and less to your understanding of your data. This demonstration has deepened your understanding of how the K-means algorithm works. Soon, we'll explore methods for numerically determining which K value is best for particular data. As always, feel free to explore the notebook more on your own to keep building your skill set. You now have some familiarity with the intuition behind the K-means methodology. In some of the examples we presented, we plotted points in two-dimensional space. In those cases, it was clear if the model was correctly assigning points to clusters. We also studied an example where we were able to visualize the data in three dimensions. Unfortunately, most cases data professionals encounter on the job are not so easy. Your data will have many more than three dimensions, so you'll not be able to visualize how each observation relates to those around it. You might not even know how many clusters there should be. So how do you decide the value for K? And once you do, how do you know if your model is working as intended? In linear and linguistic regression, you used metrics such as r squared, mean squared error, area under the ROC curve, precision, and recall to evaluate the effectiveness of your model. But in unsupervised learning, you don't have any labeled data to compare your model against. The metrics aren't applicable. In fact, your model isn't predicting anything. Instead, it's grouping observations based on their similarities. It's up to you to investigate and understand the different clusters. Remember, if you have some domain knowledge or the problem you're trying to solve has its own constraints, use these things to your advantage. For example, maybe you're investigating customer segmentation for a service that offers four different subscription levels. Then you'd probably want to use four as your value for K. But if you have no way of knowing in advance what value to use for K, don't worry. There are other ways to figure it out. Before we get started with evaluation metrics, consider what makes for a good clustering model. Basically, you want clearly identifiable clusters. This means that within each cluster, or intra cluster, the points are close to each other. It also means that between the clusters themselves, or inter cluster, you want lots of empty space. One way to evaluate the intra cluster space in a K-means model is to identify its inertia. This is a different concept from inertia as it's defined in physics. Here, inertia is defined as the sum of the squared distances between each observation and its nearest centroid. Essentially, this is a measurement of how closely related observations are to other observations within the same cluster. That information is then aggregated across all the clusters to produce a single score for the particular metric being measured. Another important metric for evaluating your K-means model is the silhouette score. This is a more precise evaluation metric than inertia because it also takes into account the separation between clusters. Silhouette score is defined as the mean of the silhouette coefficients of all the observations in the model. We'll cover this in more depth later. For now, just know that the silhouette score helps evaluate your model, provides insight as to what the optimal value for K should be, and uses both intra-cluster and inter-cluster measurements in its calculation. Nice work. You now have a better understanding of what makes a good clustering model. You also understand more about some of the metrics to determine your model's effectiveness. Coming up, we'll further explore these metrics and learn how data professionals put them to use on the job. Previously, you were introduced to inertia and silhouette scores as metrics to evaluate K-means models. These are indispensable tools for data professionals who work with K-means models and any other model in the clustering family. Now let's expand on these concepts. Consider, again, what makes a good clustering model. Ideally, you'd have tight clusters of closely related observations. And each cluster is well separated from other clusters. Remember that inertia is the sum of the squared distances between each observation and its closest centroid. It's a measurement of intra-cluster distance. So it gauges how closely related each observation is to the other observations in its own cluster. Inertia can be represented by this formula, where n is the total number of observations in the data and C sub k is the centroid of the cluster that the observation x sub i is in. The more compact the clusters, the lower the inertia because there's less distance between each observation and its nearest centroid. Therefore, it's important for inertia to be as close to 0 as possible. Can inertia ever be 0? Well, it's possible, but this scenario wouldn't offer any new insight into the data. Here's why. In one case, if all observations were identical, this would mean all data points are in the same location. Then, inertia equals 0 for all values of k. And the second case is when the number of clusters is equal to the number of observations. If each observation is in its own cluster, then its centroid is itself. Inertia is a great metric because it helps us to decide on the optimal k value. We do this by using the elbow method. In the elbow method, we first build models with different values of k. Then, we plot the inertia for each k value. Here's an example. Notice that the greater the value is for k, the lower the inertia. So should you always select high k values? Well, no. A low inertia is great, but if it results in meaningless or inexplicable clusters, it doesn't help you at all. A good way of choosing an optimal k value is to find the elbow of the curve. This is the value of k at which the decrease in inertia starts to level off. In this example, that occurs when we use three clusters. Sometimes, it might be difficult to choose between two consecutive values of k. In that case, it's up to you to determine which is best for your particular project. The second important metric for evaluating your k means model is the silhouette score. This is a more precise evaluation metric than inertia because it takes into account the separation between the clusters. Silhouette score is defined as the mean of the silhouette coefficients of all the observations in the model. Each observation has its own silhouette coefficient, which is calculated as b minus a over whichever value is greater, a or b, where a is the mean distance from that observation to all other observations in the same cluster, and b is the mean distance from that observation to each observation in the next closest cluster. The silhouette coefficient can be anywhere between negative 1 and 1. Consider this schematic. If an observation has a silhouette coefficient close to 1, it means that it's both nicely within its own cluster and well separated from other clusters. A value of 0 indicates that the observation is on the boundary between clusters. If your observation has a silhouette coefficient close to negative 1, it may be in the wrong cluster. So as you've just experienced, when using silhouette score to help determine how many clusters your model should have, you'll generally want to opt for the k value that maximizes your silhouette score. Inertia and silhouette score are important metrics to help you determine the most appropriate number of clusters for your k-means model. Now that you're familiar with how they're derived, you'll be well prepared to continue working with them. Previously, you learned about inertia and silhouette scores. You understand that their metrics used to help decide an effective value for k in a k-means model. And because k-means is an unsupervised learning model, it's used to find structure and relationships within data, but there are no right answers. Therefore, data professionals who use these models have to rely on these metrics to help them determine whether their model is identifying characteristics of their data that are useful for their needs. In this video, we're going to build a k-means model and evaluate it using inertia and silhouette score. We'll go over which packages to import, how to scale data, instantiating, and fitting the model, and of course, using the labels and inertia attributes and silhouette score function to determine a final value for k. Once again, we'll return to Jupyter Notebooks as the platform where we'll build our models. In a new notebook, the first step, as always, is to begin with import statements. This will create the computing environment with the necessary packages and tools for your project. In this case, we'll import NumPy and Pandas as our operational packages. We'll also import the following task-specific items from scikit-learn, k-means, silhouette score, and standard scaler. Note the syntax, as each item is imported from a different package. k-means comes from sklearn.cluster, silhouette score comes from sklearn.metrics, and standard scaler comes from sklearn.preprocessing. The function make blobs is something we'll use just for this demonstration to help us create synthetic data. We'll use Seaborn for graphing. In practice, you'd have a real dataset and you'd read in this data and perform EDA, data cleaning, and other manipulations to prepare it for modeling. For simplicity and to help us focus on modeling and analysis, we're going to use synthetic data for this demonstration. We'll start by creating a random number generator. This is to help create reproducible synthetic data. We'll use it to generate clustered data. For now, we won't know how many clusters there are. By calling the random number generator and assigning the result to a variable, we can avoid viewing the true number of clusters our data has. This will let us use inertia and silhouette coefficients to determine it. This next step uses make blobs and our random number generator to create data that has an unknown number of clusters, at least to us. These steps return a NumPy array, but it's usually helpful to view your data as a pandas data frame. This is often how your data will be organized when modeling on the job. So we'll convert our data to a pandas data frame. In the six columns, we find that our data has six features. This is too many dimensions for us to visualize in 2D or 3D space. We can't observe how many clusters there are, so we'll need to use our detective skills to figure it out. Because K-means uses distance between observations as its measure of similarity, it's important to scale our numerical data before modeling if it's not already scaled. For this, we'll use scikit-learn's standard scaler. Standard scaler scales each point x sub i by subtracting the mean for that feature and dividing by the feature's standard deviation. This ensures that all the variables have a mean of zero and a variance standard deviation of one. There are a number of scaling techniques available in scikit-learn's pre-processing package, including standard scaler, min-max scaler, normalizer, and others. There's no firm rule for determining which method will work best, but with K-means models, using any scaler will almost always lead to better results than not scaling at all. We can instantiate standard scaler and transform our data in a single step by using the dot fit transform method and passing our data to it as an argument. Here's a tip. If your computer has enough memory, it's helpful to keep an unscaled copy of your data to use later, so we'll assign the scaled data to a new variable called x underscore scaled. Now that the data is scaled, we can start modeling. Because we don't know how many clusters exist in the data, we'll begin by examining the inertia values for different values of K. Let's start with K equals three, an arbitrary number in this case. One thing to note is that by default, scikit-learn implements an optimized version of the K-means algorithm called K-means plus plus. This helps ensure optimal model convergence by initializing centroids far away from each other. Because we're using K-means plus plus, we will not rerun the model multiple times. Now, let's instantiate the model. Because we want to build a model that puts our data into three clusters, we set the n underscore clusters parameter to three. We'll also set the random underscore state to an arbitrary number. This is only so others can reproduce our results. If we left this value blank, it's possible others could replicate our code exactly and still get different results due to the random initial placement of centroids. The next step is to fit the model to the data. We do this by using the fit method and passing in our scaled data. This returns a model object that has learned your data. You can now call its different attributes to view inertia, location of centroids and class labels, among others. We can get the cluster assignments by using the labels attribute. Similarly, find the inertia by using the inertia attribute. Let's find out what happens when we check the cluster assignments and inertia for this model. The labels attribute returns a list of values that is the same length as the training data. Each value corresponds to the number of the cluster to which that point is assigned. Because our k-means model clustered the data into three clusters, the value assigned to each observation will be zero, one, or two. The inertia attribute returns the sum of the square distances from each sample to its closest cluster center. This inertia value isn't helpful by itself. We need to compare the inertias of multiple k-values. To do this, create a function called k-means inertia that fits a k-means model for multiple values of k, in our case, two through 10. The function calculates the inertia for each value, appends it to a list, and returns that list. Then, we plot it. The x-axis is the number of clusters, and the y-axis is the inertia. This plot contains an unambiguous elbow at five clusters. Models with more than five clusters don't appear to reduce inertia at all. Right now, it seems like a five cluster model might be optimal. But let's check the silhouette scores. Hopefully the results will corroborate our findings. To get a silhouette score, we call the function and pass to it two required parameters, the training data, and its assigned labels. Let's check this out for the k-means three model we created earlier. It worked! However, this value isn't very useful if we have nothing to compare it to. Just as we did for inertia, we'll write a function that compares the silhouette score of each value of k from two through 10. Now, plot these silhouette scores. This plot indicates that the silhouette score is closest to one when our data is partitioned into five clusters. It confirms the inertia analysis. In this case, because we used synthetic data, we can review how many clusters actually existed in our data. This is the variable created by the random number generator at the beginning of the video. We called it centers. We were right! We were able to use inertia and silhouette score to correctly deduce that our data has five clusters. At this point, we'll want to do some further analysis to determine whether we can understand our clusters and if they're appropriate for our use case. We'll instantiate a new k-means model with n underscore clusters equals five and fit it to our scaled data. Okay, now we can confirm that there are five unique labels ranging from zero through four. So, we can use them to create a new column in the unscaled data frame. Next, we could perform analysis on the different clusters to identify what makes them different from one another. This would not have been possible with the previously scaled data because the numbers wouldn't make a lot of sense. Note that in many cases, it's not always clear what differentiates one cluster from another and it can take a fair bit of effort to determine whether it makes sense to cluster your data a given way. This is where domain knowledge and expertise through practice are very valuable. Congratulations! You've reached the end of another section of the course. Along the way, you've discovered that unsupervised learning is a vast field with many different applications. First, you learned about k-means models for deriving structure from your data. You were introduced to the concept of clustering similar groups of data around centroids and building a k-means model. We explored the importance of running a k-means model multiple times with different values for k. We also explained the issues with local minima and how to build models with different centroid initializations to make sure you're getting the most accurate results. You're now much more familiar with inertia and silhouette score, methods for choosing the best number of clusters and evaluating the effectiveness of your model. And you can identify the elbow of an inertia curve and use it to help determine an optimal k-value. There's a whole world of unsupervised learning models and methodologies out there. This is just the beginning, but now you're empowered with key tools for navigating the landscape and developing your talents as a data professional. Hello, and welcome back. You've come so far building models of your own using the tools and skills you've been learning throughout this program. Now, in this final part of the course, you'll revisit supervised machine learning by investigating some more advanced classification techniques. These advancements are very exciting for data professionals because they enable us to overcome some typical modeling limitations. One such method is tree-based learning. Tree-based learning is a type of supervised machine learning that performs classification and regression tasks. It uses a decision tree as a predictive model to go from observations about an item, represented by the branches, to conclusions about the item's target value, represented by the leaves. Soon, you'll learn how single decision trees provide a foundation for more advanced approaches to all kinds of data work. Then, you'll move on to ensemble learning techniques, which enable you to use multiple decision trees simultaneously in order to produce very powerful models. In addition to learning how these new models work and their use cases, you'll be introduced to hyperparameter tuning. Knowing how and when to adjust or tune a model can help a data professional significantly increase performance. Together, we're going to build models that can be very impactful for tons of different business applications. What you're about to learn can really make you stand out to employers in the industry. Let's get started. In the world of supervised learning, there are tons of techniques that can help you make predictions. One popular tool for classification and prediction is the decision tree. It serves as a foundation for some of the most effective models used in industry today. A decision tree is a flowchart-like supervised classification model and a representation of various solutions that are available to solve a given problem based on the possible outcomes of related choices. Like all supervised learning classification techniques, decision trees enable data professionals to make predictions about future events based on the information that is currently available. They also have some very specific advantages in certain areas over other supervised learning models. Decision trees require no assumptions regarding the distribution of underlying data. Unlike the models we've covered previously, they can handle collinearity easily. Additionally, preparing data to train a decision tree can be a much less complex process, requiring little pre-processing, if any at all. However, decision trees are not perfect. No model is. Decision trees can be particularly susceptible to overfitting. The model might get extremely good at predicting scene data, but as soon as new data is introduced, it may not work nearly as well. This is something that you'll need to keep in mind while building these types of models. A decision tree consists of nodes and edges. The edges connect together the nodes, essentially directing from one node to the next along the tree. Decisions are made at each node. At each, a single feature of the data is considered and decided on. By the end, any relevant features will have been resolved, resulting in the classification prediction. Let's explore this a little further. Here's a decision tree that will help you decide whether or not to go outside and play soccer or football on any given day. The first decision that will be made relates to the weather outlook. For this tree, there are three options, sunny, cloudy, or rainy. This node, where the first decision is made, is called the root node. It's the first node in the tree and all decisions needed to make the prediction will stem from it. It's a special type of decision node because it has no predecessors. The nodes where a decision is made are decision nodes. Decision nodes always point to a leaf node or other decision nodes within the tree. So for our example, if it's supposed to be sunny or rainy, the tree will continue making more decisions to arrive at a final prediction. However, if it's cloudy, the tree arrives at a prediction. Soccer will be played. This brings us to a leaf node. Leaf nodes are where a final prediction is made. The whole process ends here, so no further decisions are required after this point. Now, view where the decision tree would have gone if the outlook had been sunny. We're not at a leaf node yet. There are still decisions to be made. This time, the consideration is about the humidity. If the humidity is above 75%, the tree ends at a leaf node that says don't play soccer. However, if the humidity is below 75%, the decision tree will say play soccer. The nodes that are pointed to, whether leaf nodes or other decision nodes, are child nodes. The node that is pointing to them is a parent node. The algorithm decides what and where variables are split based on what will provide the most predictive power. So for example, if 90% of the time that it rains, soccer is not played, this variable would be very predictive. Splitting the data on outlook would give new groups, each of which has a majority of play and don't play. Now you know the basics of decision trees. This foundation will be helpful as you continue learning about tree-based modeling. Coming up, you'll cover the aspects of building the tree and using your training data to develop the nodes and edges. Then, you'll learn how to optimize tree-based models and what you, as a data professional, can do to maximize their capabilities. Now that you've developed a solid understanding of classification models, it's time to build a model that's a bit more advanced. In this video, you'll examine how to create a single standard decision tree in Python. We're going to use a decision tree to approach the same business need as with the naive Bayes model from earlier, modeling customer bank churn. The first thing, as always, is to import any necessary packages and libraries into our notebook. You've experienced most of these before, but the packages for the decision tree itself are new. Let's import decision tree classifier, the scikit-learn implementation of a single decision tree. Additionally, import the plot tree function to produce a visual of the decision tree after it's built. We also import the confusion matrix and confusion matrix display functions to help us calculate and plot a confusion matrix for our model. And lastly, we have our four evaluation metrics. We're going to read in the original dataset as a panda's data frame as usual. Remember, this is where you'd normally do exploratory data analysis or EDA. Then you would use what you learned from EDA and what you know about the use case of your model to decide on an appropriate evaluation metric. For our bank churn models, we're going to assume that a metric that balances precision and recall is best. The metric that helps us achieve this balance is F1 score, which is defined as the harmonic mean of precision and recall, or their product divided by their sum. Again, there are many metrics to choose from. The important thing is that you make an informed decision that is based on your use case. Now that we've decided on an evaluation metric, let's prepare the data for modeling. Just as before, we'll drop the unpredictive features and the gender column, so our model doesn't predict based on gender. Then we'll dummy encode the geography column, creating Boolean columns from the categorical column. Our last preparation is to separate our target variable from the rest of the data and then split the data into training and test sets using the train test split function. Don't forget to stratify based on the target. The first thing we'll do is train a baseline decision tree model. We won't tune it. It's just to give us scores that we can use as points of reference. We do this by instantiating the classifier and setting the random state. We'll assign it to a variable called decision tree. Next, we'll fit it to the training data. This grows a decision tree on our data. It all happens behind the scenes. Finally, we'll use the predict method to use the tree we just grew to make predictions on the X test data, assigning the results to a variable called dt-pred. Now we can get the results by using the different evaluation metric functions we imported. This model's F1 score is better than what we got from the naive Bayes model we built. Let's inspect the confusion matrix of our decision tree's predictions. First, we'll write a short helper function to help us display the matrix. Notice from this confusion matrix that the model correctly predicts many true negatives. This is to be expected because the data set is imbalanced in favor of negatives. When the model makes an error, it appears slightly more likely to predict a false positive than a false negative, but it's generally balanced. This is reflected in the precision and recall scores, both being very close to each other. Next, let's examine the splits of the tree. We'll do this by using the plot tree function that we imported. We pass to it our fit model as well as some additional parameters. Note that if we did not set max depth equals two, the function would return a plot of the entire tree all the way down to the leaf nodes, but we are most interested in the splits nearest to the root because these tell us the most predictive features. Class names displays what the majority class of each node is and filled colors the nodes according to their majority class. How do we read this plot? The first line of information in each node is the feature and split point that the model identified as being the most predictive. In other words, this is the question that's being asked at that split. For our root node, the question was, is the customer less than or equal to 42 and a half years old? At each node, if the answer to the question it asks is yes, the sample would move to the child node on the left. If the answer is no, the sample would go to the child node on the right. Genie refers to the node's genie impurity. This is a way of measuring how pure a node is. The value can range from zero to 0.5. A genie score of zero means there's no impurity. The node is a leaf and all of its samples are of a single class. A score of 0.5 means the classes are all equally represented in that node. Samples is how many samples are in that node and value indicates how many of each class are in the node. Returning to the root node, we have value equals 5,972 and 1,528. Notice that these numbers sum to 7,500, which is the number of samples in the node. This tells us that 5,972 customers in this node stayed and 1,528 customers churned. Lastly, we have class. This tells us the majority class of the samples in each node. If we look at the top of the tree, this plot tells us that if we could only do a single split on a single variable, the one that would most help us predict whether a customer will churn is their age. If we look at the nodes at depth one, we notice that the number of products and whether or not the customer is an active member also are both strong predictors of whether or not they will churn. This is a good indication that it might be worthwhile to return to your EDA and examine these features more closely. Now that you have a basic understanding of how tree-based modeling works in Python, you have two more techniques to learn before moving on to some more powerful optimization techniques. Hyperparameter tuning and cross-validation. Using these, we can optimize single decision trees even further and you'll use those concepts to supercharge the models you'll learn later on. Meet you again soon. Recently, you've been exploring how to build a decision tree classifier model. For many of the models you've worked with throughout this course, you used evaluation metrics such as F1 score to gauge their performance. But throughout this section of the course, you'll be taking an extra step to gain some additional performance increases from your models. A very popular and widely used technique to improve performance after creation is known as hyperparameter tuning. Hyperparameters are parameters that can be set before the model is trained. They can be tuned to improve model performance directly affecting how the model is fit to the data. Hyperparameter tuning is the process of adjusting the parameters to find the best values that will result in the most optimal model. Just like a musician tuning the strings on their guitar, the idea is to achieve balance and a beautiful result. For tree-based modeling, there are many hyperparameters that can be tuned and they can have a big impact on the model itself. You've actually already used one previously in this course when you worked with K-means. As you'll recall, when building a K-means model, you set the value of K to produce different cluster results. But when you changed its value, you performed hyperparameter tuning. One of the more basic hyperparameters for a decision tree is called max depth. Setting this hyperparameter defines a limit of how long a decision tree can get. The depth of a decision tree is the number of levels between the root node and the farthest node from the root node with the root node itself being level zero. Consider our previous example. This tree has three levels. The root node is level zero. The nodes in the middle are level one and the leaf nodes all the way at the bottom are level two. So this tree has a depth of two. However, this decision tree has a depth of four. Even though this decision tree isn't as filled out as the previous example, what matters is the distance of the farthest node from the root and whether it's a leaf node or a decision node. This leads us back to max depth. When working with very large data sets, you could potentially create massive trees that are very deep. But this isn't necessarily what you want for your model. So setting a value for max depth can help reduce overfitting problems by limiting how deep the tree will go. Additionally, it can reduce the computational complexity of training and using the model in the first place. For example, if you're finding that a decision tree has the same performance with a depth of 10 versus a depth of 100, you can set max depth to 10 and achieve the desired performance more quickly. Another very commonly used hyperparameter is called minSamplesLeaf. This hyperparameter defines the minimum number of samples that must be contained in a leaf node. It means that split will only happen if there are enough samples in each of the result nodes to satisfy the required value. For example, maybe part of the way down your tree, there's a decision node that currently has 10 samples. However, the minSamplesLeaf hyperparameter is set to six. There would be no way to split the data so that each leaf node has six samples and therefore no further split can take place. There are other hyperparameters for decision trees that you'll learn, but first let's explore finding the optimal values for the parameters. And here's where something called grid search is useful. Grid search is a tool to confirm that a model achieves its intended purpose by systematically checking every combination of hyperparameters to identify which set produces the best results based on the selected metric. So at the end, you'll have values that produce optimal results for your model. When performing a grid search, the first step is to specify which parameters you want to tune and the set of values that you want to search over. For example, maybe we want to tune maxDep and minSamplesLeaf. We would define potential values for each of these. For maxDep, we could check depths of four, eight, 12, 20, and 30. For minSamplesLeaf, we could try 10, 50, and 100. The algorithm will check every combination of values to see which pair has the best evaluation metrics. It would first check maxDepth of four with minSamplesLeaf of 10, then 50, then 100. The algorithm would then check maxDepth of eight with minSamplesLeaf of 10, then 50, then 100. This continues until every combination has been analyzed. Remember, you can try any values and any number of values during grid search if you believe the benefits are worth the cost of your computing time. Coming up, we'll put into practice many of the tree-based modeling concepts you've learned so far, all the way from constructing the tree to optimizing it and using it to make some classification predictions. Looking forward to it. You now know about many of the tools involved with building some pretty powerful models. Now in this video, you'll explore model validation. Model validation is the set of processes and activities intended to verify that models are performing as expected. This is achieved with a validation data set, which is a sample of data that's held back during training. The validation data set is instead used to give an unbiased estimate of the skill of the final tuned model. Note that validation data is different from test data and must remain unseen until the very end of the process. In some of the models we've built so far, we haven't been doing this, but that's okay. We were just using those models to understand the process of building an evaluation. Previously, before training the model, we took our data and split it into two sets, one training set and one testing set. These sets were used to train and test the model respectively. With validation, the data is actually split into three sets. The first two are training and testing sets as before, but now there's an additional validation set. This validation set is used instead of the test set to evaluate the model, leaving the test set untouched. In addition, another popular method is cross-validation. Cross-validation is a process that uses different portions of the data to test and train a model across several iterations. It works like validation, but with a slight twist. Instead of having one validation set to evaluate the model, the training data is split into multiple sections, known as folds. Then the model is trained on different combinations of these folds. For example, perhaps we want five folds. First, the data would be split into the training and test data. Then the training data would be split into the five folds. The first model iteration will train with folds one, two, three and four, using the fifth fold to get metrics for the model. The next will train with folds one, two, three and five using the fourth fold to get metrics. This process repeats until every combination is done and the evaluation metrics are averaged to get final validation scores. Which validation technique you choose mainly depends on the data set you're working with. Cross-validation is particularly useful when working with smaller data sets as it maximizes the utility of the data available, more so than standard validation. However, cross-validation is not necessary when working with very large data sets. There's so much data that maximizing the utility is not required and actually can be problematic depending on the computing resources at your disposal. However, if limited computing resources or constraints in the data are not issues, then cross-validation is almost always applied. Validation schemes are essential to building and selecting effective models. Data professionals working on these types of business projects are responsible for determining the best scheme to use. This comes with experience, along with an understanding of the data and the available tools. You're well on your way to developing these important skills. Hello and welcome back. In this video, we'll be taking the foundation we built when creating a decision tree classifier in Python and expanding on it to fine-tune our models. If you recall, hyperparameter tuning involves changing parameters that directly affect how the model trains before the learning process begins. Different models have different types of hyperparameters that are available for you to adjust. You've learned about two that apply to tree-based models, max depth and min samples leaf. As a reminder, max depth defines how long a decision tree can get and min samples leaf defines the minimum number of samples for a leaf node. These are the hyperparameters that will be tuned on a single decision tree. When originally exploring hyperparameter tuning, we also considered the steps for finding the optimal values for hyperparameters. Randomly entering values won't produce the best results, which is why we use grid search. As a refresher, grid search specifies a series of values for each hyperparameter to be tuned. It systematically checks every combination of those values to determine which set produces the best results based on the selected metric. Think of it as brute-forcing the different hyperparameter values. Imagine forgetting your pin and trying every single number between 0000 and 9999. Sure, it would take time, but eventually you'd find it. This is how grid search works. Okay, now let's get into the code. You'll work within the same framework as the other classification models you've created. We'll begin where we left off in the decision tree notebook. Remember, we've already performed feature engineering and the data has already been split into X and Y data, as well as training and test sets. But now, we're going to add a new function. Grid Search CV is imported from the model selection package of Scikit Learn, enabling the hyperparameters to be tuned. The CV in Grid Search CV stands for cross-validation. Each time a set of hyperparameters is used, it's scored against a validation set, keeping the test data unseen. You'll use the validation scores when comparing models moving forward. Note that you won't be comparing this tuned decision tree to the existing models. All the other models were scored and compared using test data, which is actually an improper practice. When data professionals perform model selection in the workplace, the test data must always remain unseen to the models being worked on. That data is only used at the very end of the model development process. As mentioned before, the parameters you'll tune will be max-depth and min-samples-leaf. A dictionary is defined where the key is the name of the hyperparameter and the value is a list of numbers that will be tried as that hyperparameter. While the grid search is based on F1 score, you still want to find out what the other scores are. So create a set called scoring with the names of each of the desired metrics. Next, create an instance of a decision tree classifier named tuned decision tree. The grid search CV function is then called as arguments pass in the decision tree classifier object, the parameters, the scoring methods, the number of cross-validation folds, and specify the metric the search will focus on. Finally, fit the model to the data. Check which hyperparameters the grid search identified. By getting the best estimator attribute from the grid search object, you can observe the values it found. So, a max-depth of eight and a min-samples-leaf of 20 was best when using the F1 score as a measure. Getting the best score attribute confirms the best average F1 score across the different folds among all the combinations of hyperparameters. Note that this model achieved a score of about 0.5607. This final code block is a helper function to extract scores for the model. It produces a data frame that has the name of the model along with the four scores you've been using. Call this function at the very end and save the resulting data frame as a CSV file for later use. Right now, there's nothing to compare this score with. However, note the model has an F1 score of 0.5605. And soon, you'll go on to create other, more advanced tree-based models and find scores to compare with this one. With those numbers, you'll be able to determine which model is not only the best performer, but also the best in the context of the business needs. At this point, you've learned that decision trees are useful because they're easy to understand and interpret, flexible with regard to the data they use and highly versatile. Decision trees can be good predictors, but you also know that they're prone to overfitting and they're very sensitive to variations in the training data. How do we solve these problems? The answer is by using the wisdom of the crowd. Perhaps you're familiar with this concept because it can apply to everyday situations as well. If I have a jar filled with jelly beans and I ask a spatial math expert to examine it and guess how many jelly beans there are, their estimate will typically be less accurate than if I ask a thousand ordinary people to do the same thing and then take the average of their guesses. We can apply this same concept to modeling using a process called ensemble learning or simply, ensembling. Ensemble learning involves building multiple models and then aggregating their outputs to make a final prediction. Just like in our jelly bean example, predictions using an ensemble of models are very accurate even when the individual models themselves are barely more accurate than a random guess. A best practice when building an ensemble is to use very different methodologies for each model it contains, such as a logistic regression, a naive Bayes model and a decision tree classifier. In this way, when the models make errors and they always will, the errors will be uncorrelated. The goal is for them to not all make the same errors for the same reasons. You could build an ensemble using the three models I just mentioned. You'd train each model on the same data, then use each model's individual predictions to make a final prediction, say by taking the majority vote if it's a classification task or averaging the results if it's a regression task. But there's another way to build an ensemble, a way that uses the same methodology for every contributing model. In this kind of ensemble, each individual model that comprises it is called a base learner. For this method to work, you usually need a lot of base learners and each is trained on a unique, random subset of the training data. If the base learners were all trained on the exact same data, there would be too much correlation between the errors. This would affect the strength of the base learners. And if a base learner's prediction is only slightly better than a random guess, it becomes a weak learner. So to address this, data professionals do something called bagging in order to ensure random subsets of the data and strong learners. The word bagging comes from bootstrap aggregating. Let's break this down. Remember from statistics that bootstrapping refers to sampling with replacement. That's what happens during bagging too. Each base learner samples from the data with replacement. For bagging, this means the various base learners all sample the same observation and a single learner can sample that observation multiple times during training. The aggregation part of bagging refers to the fact that the predictions of all the individual models are aggregated to produce a final prediction. For regression models, this is typically the average of all the predictions. For classification models, it's often whichever class receives the most predictions, which is the mode. When bagging is used with decision trees, we get a random forest. A random forest is an ensemble of decision tree base learners that are trained on bootstrapped data. The base learners predictions are all aggregated to determine a final prediction. Random forest takes the randomization from bagging one step further. A regular decision tree model will seek the best feature to use to split a node. A random forest model will grow each of its trees by taking a random subset of the available features in the training data and then splitting each node at the best feature available to that tree. This means that each base learner in a random forest model has different combinations of features available to it, which helps to prevent the problem of correlated errors between learners in the ensemble. Each individual base learner is a decision tree. It may be fully grown, so each leaf is a single observation or it may be very shallow, depending on how you choose to tune your model. Ensembling many base learners helps reduce the high variance that you typically get from a single decision tree. Ensemble learning is powerful because it combines the results of many models to help make more reliable final predictions. Plus, these predictions have less bias and lower variance than other standalone models. Coming up, we'll explore random forests in more detail. Lots to come. Now that you've been introduced to random forests, let's examine a little more closely what they are and how they function. It's important to understand this methodology because it's commonly used in data work and many of its component steps are used by other more advanced modeling strategies. As a refresher, a random forest is an ensemble of learning trees whose predictions are all aggregated to determine a final prediction. Each tree in a random forest model uses bootstrapping to randomly sample the observations in the training data with replacement. Remember, this means that any tree in the model can use the same observation and the same observation can be sampled more than once by the same tree. Bootstrapping is a critical component of random forest models. It ensures that every base learner in the ensemble is trained on different data while allowing each learner to train on a dataset that's the same size as the original training data. Because there are duplicated observations in the tree's training data, each one will be missing some of the observations from the original training dataset. One more important principle of random forest models is that all trees in the ensemble are trained on a random subset of the available features in the dataset. No single tree sees all the features. Again, this is to introduce another element of randomness and ensure that each tree is as different from the others as possible. You learned that one of the main weaknesses of decision trees is that they are very sensitive to new data, so they're prone to overfitting. Therefore, randomizing both the data and the features used by each base learner means that no single tree can overfit all the data. This is because no single tree sees all the data. In fact, the trees underfit the data. They are high bias, but together they can be very powerful predictors that are more stable than a regular single decision tree. In addition, random forests are very scalable. All the base learners they rely on can be trained in parallel using different processing units, even across many different servers. Finally, just like decision trees, random forest models need to be tuned to find the combination of hyperparameter settings that results in the best predictions. After all, random forests are made up of many decision trees, and data professionals want them all to be as effective as possible. Hey, welcome back. In this video, we'll build on your understanding of how decision trees grow. This will be the basis on which we'll tune a random forest model. You've learned that random forests make predictions by sampling features and observations to grow trees. With decision trees, splits are decided by which variables and which cutoff values offer the most predictive power. Now, let's consider that decision trees continue to split until one of a certain set of conditions is met. The first condition has to do with the observations that a leaf contains. When all of the observations belong to the same class, this means the leaf node is pure. The second condition affecting where a tree splits is whether the minimum leaf size or maximum depth is reached. Also, a decision tree may stop growing if it achieves a certain performance threshold. The value and metric for this threshold can both be specified by the modeler. You'll recall that settings such as these are known as hyperparameters, and they can be tuned to improve model performance, directly affecting how the model is fit to the data. We demonstrated that one of the most important hyperparameters in a decision tree is its max depth. This specifies how many levels the tree can have, and ultimately determines how many splits it can make. Remember, every time a node splits, your data gets separated into smaller subsets. The model is drawing another decision boundary. We also introduced you to minSamplesLeaf, which defines the minimum number of samples for a leaf node. With minSamplesLeaf, a split can only occur if it guarantees a minimum number of observations in the resulting nodes. Now, a new concept, minSamplesSplit, can be used to control the threshold below which nodes become leaves. Random forest models have these same hyperparameters because they are ensembles of decision trees. These hyperparameters control how the learner trees are grown, but random forests also have some other hyperparameters which control the ensemble itself. This first hyperparameter controls the randomness of the trees, and it's called maxFeatures. This setting specifies the number of features that each tree selects randomly from the training data to determine its splits. For example, if you have a data set with features a, b, c, d, and e, and you build a random forest model with maxFeatures set to three, your first tree might use features a, c, and e to determine its splits, and the next tree might use features b, d, and e, and so on. The second hyperparameter, number of estimators, controls how many decision trees your model will build for its ensemble. For example, if you set your number of estimators to 300, your model will train 300 individual trees. If you're building regression trees, then the model's final prediction would average the predictions of all 300 trees. If you're building classification trees, the final prediction would be determined by whichever class received the majority vote from the 300 individual trees. For random forest models, performance will typically increase as trees are added to the ensemble, but only to a certain point. After this point, improvement will level off and adding new trees will only increase your computing time. This happens because the new trees will become very similar to existing trees, so they won't contribute anything new to the model. As a final point, many data professionals build models without hand setting each hyperparameter. In fact, when using scikit-learn, the model might perform well with no hyperparameters at all. That's because it has an effective default setting. And remember to make the most of grid search to help you iterate. Data professionals know how to experiment with combinations of hyperparameters in order to build the model that makes the very best predictions. Now that you're familiar with the logic behind random forests and some of its most important hyperparameters, you're ready to build a model. In this video, we'll build a random forest model that uses grid search to cross validate and tune the hyperparameters. Let's open up a Jupyter notebook and get started. Recall that in this scenario, we're trying to predict which customers will close their bank accounts. Import NumPy and Pandas, Matplotlib, GridSearchCV, and TrainTestSplit. Then all the evaluation metrics. Also import random forest classifier. Note that we're using the classifier because we're trying to solve a classification problem. But we could also import the random forest regressor if we were predicting on continuous data. Remember that we've prepared our data by dropping the row number, customer ID, and surname columns because they don't have predictive value. We've also dropped the gender column because we don't want the model to predict on the basis of gender. Then we dummy encoded our categorical variables to prepare for modeling. The last step before modeling is to split the data into the training and test sets using TrainTestSplit. We're going to compare our models F1 score to what we got from our naive Bayes and decision tree models. So it's important to split the data in the same exact way to ensure that all models train and test on the same data. Therefore, we'll make sure that our test data is 25% of all the data. We'll also stratify based on our target column and set the random seed to 42, just as we did previously. This splits our X and Y data into XTrain, XTest, YTrain, and YTest. Okay, we're ready to model. Let's find out what happens if we tune our model using cross-validation. One thing you'll notice when we build ensemble models is that the training time will usually be much longer than what you've experienced so far. That's because instead of building a single tree, we're now building from 75 up to 150 trees for each combination of hyperparameters we specify. It's useful to know how long it takes a model to train. You can get a cell's runtime by entering percent percent time at the top. This is called a magic command. Magic commands, often just called magics, are commands that are built into IPython to simplify common tasks. They always begin with either percent or percent percent. Now, define the hyperparameter grid. Tune five hyperparameters. Max depth, min samples leaf, min samples split, max features, and number of estimators. Notice that for max depth, none is included. This means that one of the options allows the trees to grow without a specific limit on their depth. Next, instantiate our classifier and assign it a random state for reproducibility. And to specify the metrics, the model will capture. Finally, instantiate the grid search object. It has two positional arguments, the classifier and the parameter grid. Tell it to use the scoring metrics specified above and set CV to five. This means the model will be cross validated using five folds. Lastly, specify refit equals F1. This is necessary when we've given multiple scoring metrics because it tells grid search that even though we want to check a few different metrics, the one we care most about is the F1 score. As a quick refresher, F1 score is a combination of precision and recall, combining the two into a single metric. In this instance, when we call the best estimator, it's the one with the highest average F1 score across all five validation folds. Now, fit the model to the training data. Depending on the processing power available, the number of hyperparameter combinations specified in the grid search, the size of the data set, and the number of folds used to cross validate, this could take a long time. In this example scenario, the time magic tells us it took about 20 minutes to fit. There's always a trade-off between searching over a large hyperparameter space and good runtime. The more hyperparameters you search, the better your model will be, but the longer it will take to fit. When models take a long time to fit, you don't want to have to run them again. If your kernel disconnects or you shut down the notebook and lose the cells output, you'll have to refit the model, which can be frustrating and time consuming. The good news is that there's a method that enables you to save the fit model object to a specified location and then quickly read it back in. And in the next video, we'll discover how that works. Meet you there. In the last video, we began creating a random forest model that used grid search to cross validate and tune hyperparameters. Now, we'll build on that by using a separate validation dataset to validate a model. But first, recall where we left off. We discovered a common issue in the data realm, the trade-off between searching over a large hyperparameter space and a good runtime. As we observed, the more hyperparameters searched, the better the model, but the longer it takes to fit. When models take a long time to fit, it's inefficient to have to keep running and refitting them again. Once you find a model you're happy with, you don't wanna start from scratch every time you open your notebook. And that's where pickling comes in. Pickle is a tool that saves the fit model object to a specified location and then quickly reads it back in. It also allows you to use models that were fit somewhere else without having to train them yourself. So let's pick up where we left off and pickle the model. First, specify a file path to the directory where the model will be saved. Then create a with open statement passing to it the file path plus the name you want to use to save this model followed by dot pickle. This creates an empty pickle file. The second argument, WB, gives permission to write to the file in binary, which is how pickling works. Use as to assign the return value of open to a local variable named to write. Call pickle.dump and pass the fit model object to it. Then the to write variable. In the next cell, read back in the pickled model from where it saved. The only difference in syntax is using RB to specify that we'll be reading binary and using pickle.load to assign a new variable which points to the fit model. Make sure you call this new variable by the same name you used for your fit model above. In this case, RF underscore CV. If you comment out the line of code where you fit the model and the cell where you pickle the model, you can close the notebook, reopen it, and rerun all the cells without having to wait for the model to fit. You can also send the pre-fit model to other people to use. Now, use the best params attribute to identify the hyperparameter values of the model that had the best average F1 score across all the cross validation folds. To find the average F1 score of the best model, use the best score attribute. Then use the make results function to generate a table of all the results and concatenate that with the overall table to compare all the models. Interesting, the cross validated random forest model has an average F1 score of 0.58 across all five validation folds which is a little better than the single tuned decision tree. It also has better recall, precision, and accuracy. Nice. Okay, now let's use a separate validation data set to validate the model. To do this, split the training data into a new training set and validation set. Use train test split. Remember to stratify the wide data. Use an 80-20 split. Don't forget, this is only splitting the training data which itself is 75% of all data. This means that our new training set will be 80% of 75% of the data. And the new validation set will be 20% of 75% of the data. The test data remains untouched. This next part is a little tricky. Grid search CV wants to cross validate the data. In fact, if the CV parameter was left blank, it would split the data into five folds for cross validation by default. Because you're using a separate validation set, it's important to explicitly tell the function how to perform the validation. This includes telling it every row in the training and testing sets. Use a list comprehension to generate a list of the same length as our XTR data where each value is either a negative one or a zero. Use this list to indicate to grid search CV that each row labeled negative one is in the training set and each row labeled as zero is in the validation set. Call this list split index. Now, import a new function called predefined split from scikit-learn's model selection package. Predefined split provides trained test indices to split data into training and testing sets using a predefined scheme. The next step is almost identical to what you did before. Search over all the same hyper parameters and keep the syntax the same as when cross validating. But now, pass the new split index list to predefined split and assign it to a variable. We'll call the variable custom split. Finally, set the CV parameter of the grid search equal to custom split. Now it's time to fit the model. Use time magic to get the time it takes the model to train. Wow, in this example scenario, the model took only about four minutes to train. During cross validation, the training data was divided into five folds. An ensemble of trees was grown with a particular combination of hyper parameters on four folds of the data, validating it against the fifth fold that was held out. This whole process happened for each of five holdout folds. Then another ensemble was trained with the next combination of hyper parameters, repeating the whole process. This continued until there were no more combinations of hyper parameters to run. But now, there's a separate holdout set for validation. An ensemble is built for each combination of hyper parameters. Each ensemble is trained on the new training set and validated on the validation set. But this only happens one time for each combination of hyper parameters instead of five times with cross validation. That's why the training time was only a fifth as long. All right, now pickle the model again. Run the cell where the pickle is written. Then go back and comment out the line of code as well as the call to write the pickle. Remember to have a cell where the pickled model can be read back in. Check the results. When you call best params, notice that the ensemble with the best F1 score used slightly different hyper parameters than the cross validated model. Now the F1 score is 0.576, better than the single decision tree model, but not as good as the cross validated model. Both would likely produce similar results. Just keep in mind that the cross validated model is a little more reliable because it was more rigorously validated. In fact, if a different random seed had been used to create the validation set, it's possible that we might have gotten lucky and even had a model that performed a little better than our cross validated model. But note that this doesn't mean it would be expected to perform better on the test data. This notebook's purpose was to demonstrate the different processes involved in validating a random forest model using cross validation and validation with a separate data set. In practice, it's unlikely that you'd do both. Instead, it would be more effective to choose an approach based on time requirements, the amount of data, and the number of different hyper parameter combinations to be explored. As always, the work that data professionals do requires thoughtful tradeoffs and adaptability. Random forest is one methodology for building tree-based ensemble models. Now, I'm gonna introduce you to another related methodology called boosting. Boosting is one of the most powerful modeling methodologies in the field. It's used in nearly every industry that relies on predictive modeling. Many winning models from Kaggle and other competitions use boosting. It's an essential tool in any modelers tool belt. Boosting is a supervised learning technique where you build an ensemble of weak learners. This is done sequentially with each consecutive base learner trying to correct the errors of the one before. Remember, a weak learner is a model whose prediction is only slightly better than a random guess, and a base learner is any individual model in an ensemble. This practice is similar to random forest and bagging. Like random forest, boosting is an ensembling technique, and it also builds many weak learners then aggregates their predictions. But there are some key differences. Unlike random forest, which builds base learners in parallel, boosting builds learners sequentially. This is because each new base learner in the sequence focuses on what the preceding learner got wrong. Another difference from random forest is that for boosting models, the methodology you choose for the weak learner isn't limited to tree-based methods. However, we will use tree-based implementations in this course because these are common and effective ways of building boosting models. There are various different boosting methods available, but throughout this part of the program, we'll explore two of the most commonly used methodologies. The first is called adaptive boosting, or adaboosting. Adaboost is a tree-based boosting methodology where each consecutive base learner assigns greater weight to the observations incorrectly predicted by the preceding learner. Here's a demonstration. Adaboost builds its first tree on training data that gives equal weight to each observation. Then, the algorithm evaluates which observations were incorrectly predicted by this first tree. It increases the weights for the observations that the first tree got wrong and decreases the weights for those that it got right. This process repeats until either a tree makes a perfect prediction or the ensemble reaches the maximum number of trees, which is a hyperparameter that is specified by the data professional. Once all the trees have been built, the ensemble makes predictions by aggregating the predictions of every model in the ensemble. Because adaboost can be used for both classification and regression problems, this final step is a little different depending on which type is being addressed. For classification, the ensemble uses a voting process that places weight on each vote. Base learners that make more accurate predictions are weighted more heavily in the final aggregation. For regression, the model calculates a weighted mean prediction for all the trees in the ensemble. There is one disadvantage to note about boosting. You can't train your modeling parallel across many different servers because each model in the ensemble is dependent on the one that preceded it. This means that in terms of computational efficiency, it doesn't scale well to very large data sets when compared to bagging. However, this generally isn't a concern unless you're working with particularly large data sets. But there are many noteworthy advantages, including being one of the more accurate methodologies available today. Also, just like random forest, the fact that it's based on an ensemble of weak learners means that the problem of high variance is reduced. This is because no single tree weighs too heavily in the ensemble. Boosting has a few key advantages. First, unlike random forest, it reduces bias. It's also easy to understand and doesn't require the data to be scaled or normalized. Boosting can handle both numeric and categorical features and it can still function well, even with multicollinearity among the features. Plus, it's robust to outliers. Now, note that resilience to outliers is a major strength of all tree-based methodologies. This is because the model splits the data the same way regardless of how extreme a value is. Here's an example. Suppose you have six elephants. Three are females that weigh 2,000, 2,500, and 3,000 kilograms. And three are males that weigh 4,000, 4,500, and 5,000 kilograms. If you grew a decision tree using this data, it would draw a decision boundary between males and females at 3,500 kilos, the midpoint between the weights of the heaviest female and the lightest male. Now, suppose that instead of weighing 5,000 kilos, the last male elephant weighed 10,000 kilos. Your model would still divide males and females at 3,500 kilos. It doesn't matter that the last elephant doubled in size. Speaking of doubling in size, your experience and skills are growing at an enormous rate. I'm really proud to be taking this journey with you and can't wait to keep it up. Previously, you learned that boosting is an ensembling technique that builds models sequentially, with each model in the sequence focusing on the mistakes of the previous one. And you discovered that add-a-boost works by assigning greater weight in each model to the incorrect predictions of the model that preceded it. Now, we're going to explore gradient boosting. Gradient boosting is different from adaptive boosting because instead of assigning weights to incorrect predictions, each base learner in the sequence is built to predict the residual errors of the model that preceded it. Here's a demonstration. This example uses a decision tree regressor, so imagine that the target is a continuous variable. Let's start with a set of features, x, and a target variable, y. We'll train the first base learner decision tree on this data and call it learner one. Learner one makes its predictions, which we'll call y hat sub one. The residual errors of learner one's prediction are found by subtracting the predicted values from the actual values. Call the set of residual errors error one. Now, train a new base learner using the same x data, but instead of the original y data, use error one as the target. That's because this learner is predicting the error made by learner one. Call this new base learner learner two. Learner two's predictions are assigned to y hat sub two. Then compare learner two's predictions to the actual values and assign the difference to error two. In this case, the actual values are the errors made by learner one. This process will continue for as many base learners as we specify. For now, repeat it just once more. Stopping here results in an ensemble that contains three base learners. To get the final prediction for any new x, add together the predictions of all three learners. If you like, pause the video now and repeat the process to review how it works. Ensembles that use gradient boosting are called gradient boosting machines or GBMs. GBMs are among the most widely used modeling techniques today because of their many advantages. One of these is high accuracy. As we mentioned earlier, many machine learning competition winners succeeded largely because of the accuracy of their boosting models. Another advantage is that GBMs are scalable. Even though they can't be trained in parallel, like random forests, because their base learners are developed sequentially, they still scale well to large data sets. GBMs also work well with missing data. The fact that a value is missing is viewed as valuable information. So GBMs treat missing values just like any other value when determining how to split a feature. This makes gradient boosting relatively easy to use with messy data. Also, because they're tree-based, GBMs don't require the data to be scaled and they can handle outliers easily. Gradient boosting also has its drawbacks. One is that GBMs have a lot of hyperparameters and tuning them can be a time-consuming process. Another drawback is that they can be difficult to interpret. GBMs can provide feature importance, but unlike linear models, they do not have coefficients or directionality. They only show how important each feature is relative to the other features. Because of this, they're often called black box models. This is a model whose predictions cannot be precisely explained. In some industries, such as medicine and banking, it's essential that your model's predictions be explainable. Therefore, GBMs are not well-suited for some applications. GBMs can also have difficulty with extrapolation. Extrapolation is a model's ability to predict new values that fall outside of the range of values in the training data. For instance, if one loaf of bread c