eld. What we're telling the notebook here is to create a data frame with weekday the mean number of strikes for that day. To understand what this data is telling us let's plot a box plot chart. A box plot is a data visualization that depicts the locality spread and skew of groups of values within quartiles. For this data set and notebook a box plot visualization will be the most helpful because it will tell us a lot about the distribution of lightning strike values. Most of the lightning strike values will be shown as grouped into colored boxes which is why this visualization is called a box plot. The rest of the values will string out to either side with a straight line that ends in a T. We will discuss more about box plots in an upcoming video. Now before we plot let's set the weekday order to start with Monday. Now to code that input G equal sign SNS dot box plot. Next in the argument field let's have X equal weekday and Y equal number of strikes. For order let's do weekday order and for the show flyers field let's input false. Show flyers refers to outliers that may or may not be included in the box plot. If you input true outliers are included. If you input false outliers are left off the box plot chart. Keep in mind we aren't deleting any outliers from the data set when we create this chart we're only excluding them from our visualization to get a good sense of the distribution of strikes across the days of the week. Lastly we will plug in our visualization title lightning distribution per weekday for 2018 and click run cell. Now you'll discover something really interesting. The median indicated by these horizontal black lines remains the same on all of the days of the week. As for Saturday and Sunday however the distributions are both lower than the rest of the week. Let's consider why that is. What do you think is more likely that lightning strikes across the United States take a break on the weekends? Or that people do not report as many lightning strikes on weekends? While we don't know for sure we have clear data suggesting the total quantity of weekend lightning strikes is lower than weekdays. We've also learned a story about our data set that we didn't know before we tried grouping it in this way. Let's get back into our notebook and learn some more about our lightning data. One common structuring method we learned about in another video was merging which you'll remember means combining two different data sources into one. We'll need to know how to perform this method in Python if we want to learn more about our data across multiple years. Let's add two more years to our data 2016 and 2017. To merge three years of data together we need to make sure each data set is formatted the same. The new data sets do not have the extra columns week and weekday that we created earlier. So to merge them successfully we need to either remove the new columns or add them to the new data sets. There's an easy way to merge the three years of data and remove the extra columns at the same time. Let's call our new data frame union underscore df. We'll use the pandas function concat to merge or more accurately concatenate the three years of data. Inside the concat argument field we'll type in df.drop to pull the week day and week columns out. We also input the axis we want to concatenate which is one. Lastly and most essentially we add the data frame name we are concatenating to df underscore two. We also input true for ignore index because the two data frames will already align along their first columns. And now you've just learned to merge three years of data. To help us with the next part of structuring create three date columns following the same steps you used previously. We've already added the columns for year month and month underscore text to the code. Now let's add all the lightning strikes together by year so we can compare them. We can do this by simply taking the two columns we want to look at year and number of strikes and group them by year with the function.sum on the end. You'll find that 2017 did have fewer total strikes than 2016 and 2018. Because the totals are different it might be interesting as part of our analysis to see lightning strike percentages for each month of each year. So let's call this lightning by month grouping our union data frame by month text and year. Additionally, let's aggregate the number of strikes column by using the pandas function named ag. In the argument field we place our column name and our aggregate function equal to sum so that we get the totals for each of the months in all three years. When we input the head function we have the months in alphabetical order along with the sums of each month. We can do the same aggregation for year and year strikes to review those same numbers we saw before with 2017 having fewer strikes than the two other years. We created those two data frames lightning by month by year in order to derive our percentages of lightning strikes by month and year. We can get those percentages by typing lightning by month dot merge with lightning by year comma on equal sign year in the argument field. You'll find that the merge function is merging lightning by year into our lightning by month data frame according to the year. Lastly, we can create a percentage lightning per month column by dividing the percentage lightning dot number of strikes by percentage lightning after which we'll add the asterisk 100 to give us percentage. Now, when we use our head function we have a restructured data frame. To more easily review our percentages by month, let's plot a data visualization. For this one, a simple grouped bar graph will work well. We'll adjust our figure size to 10 and 6 first. Then we use the Seaborn library bar plot with our X axis as month text and our Y axis as percentage lightning per month. For some color, we'll have our hue change according to the year column with the data following the month order column. Finally, let's input our X and Y labels and our title and run the cell. When you analyze the bar chart, August 2018 really stands out. In fact, more than one third of the lightning strikes for 2018 occurred in August of that year. The next step for a data professional trying to understand these findings might be to research storm and hurricane data to learn whether those factors contributed to a greater number of lightning strikes for this particular month. Now that you've learned some of the Python code for the EDA practice of structuring, you'll have time to try them out yourself. Good luck finding those stories about your data. You know that moment in the puzzle process when you're starting to see the full picture take shape? Like, maybe you have the frame pieces connected but still have a way to go. So far, you've learned about some practices of EDA. You've learned how to gather, analyze, organize, and structure data. Coming up, you will continue to put these pieces together and the picture will become clear. You're making great progress. You've learned about the data sources, data types and data formats and the importance of knowing the basics about your data. We've worked through how to use Python to uncover big picture understandings of your data sets like column headers, d-types, size, and shapes as well as basic visualizations. Along with these parts of EDA and discovering, we've learned about date and time transformations in Python. As for the EDA practice of structuring, you've learned to order from chaos with functions like sorting, extracting, filtering, slicing, joining, merging, and grouping. In Python, you practice applying these functions to data sets that are similar to those you might work with in your career as a data professional. Data professionals commonly use each of these concepts. You will continue to build your skills with them as you search for the stories hidden in data. As a data professional, I can say that cleaning and organizing your data set is 90% of the battle. And once you've structured your tables, uncovering insights and trends can be like a walk in the park. I mentioned earlier how I used to work as a data analyst in healthcare consulting and would analyze vast amounts of medical record data to help recommend treatments for patients with severe illnesses. Typically, data is hosted across multiple different sources and tables, making them difficult to merge together. Plus, medical records may be organized so that medications a patient took and their corresponding conditions are separated. So, in order for me to understand what type of treatment a patient took for which illness, the side effects they experienced and whether the treatment worked, I had to merge hundreds of tables together. Once I did, it was incredibly easy for me to compare and contrast the different types of treatments taken and its impact on each patient. You started to learn the skills needed to complete similar tasks. Along the way, you also learned some key workplace skills like the timing for communicating updates and posing questions to project stakeholders, managers and subject matter experts. We also talked about making and testing hypotheses on your data sets, which narrows the hopes and sharpens the detail of your data-driven stories. In short, you are understanding more and more what it means to perform EDA on a data set. Great work! Later in the program, we'll discuss how to put together the rest of the data story by learning what to do with missing data and outliers, as well as making and using visualizations to help tell that story. I'll meet you there. Let's imagine you work at a board game manufacturer. As the Quality Assurance Manager, your job is to make sure every game is put together correctly. One day, you discover that the machine that produces the games is having technical issues, causing misprinted cards and incorrect counts of game pieces. Your manager asks you and the rest of the Quality Assurance team to go through the affected boxes and replace misprinted cards and missing pieces. Your goal is to make sure each box is usable. Searching through game boxes for missing and incorrect game pieces is similar to cleaning, joining, and validating your data sets as part of exploratory data analysis, or EDA. Earlier, you learned how to discover and structure data in order to understand and tell its stories in impactful ways. In this part of the course, we will cover three of the other practices of EDA, cleaning, joining, and validating data. Although there are many different ways to clean, join, and validate data, we will focus on missing values and outliers, the need for transforming categorical into numerical data and the importance of input validation. Of course, we won't just talk about the concepts of cleaning, joining, and validating. You will learn how to apply them in a Python notebook setting. We'll use data sets that are comparable to what you will see on the job. Along the way, we'll go through some important tips for improving your workplace skills like communication. We will discuss when to communicate with stakeholders and engineers about missing or outlier values and about the ethical implications you must consider when dealing with missing and outlier data values. These practices of EDA, along with effective communication and Python coding, are all essential to preparing a data set for the next steps in the PACE workflow. By following PACE while performing the six practices of EDA, you will not only save time and energy on future processes, but you will be more effective in finding and telling the data story. One of my first jobs at Google was a quantitative analyst on Google Translate. It was my responsibility to help identify areas that we could improve to help users. I recall a project where I was overly eager to dive into the data. I spent countless hours identifying opportunities ranging from when a user first downloads the app to their second, third, and twentieth time using the product. I was proud of my findings and I developed a long presentation to share with my executives. Now, while my insights were well received, the team was reluctant to invest in any of my ideas because there were so many and they weren't sure which ones to prioritize. I realized after that meeting that I needed a plan. I had spent so much time mining for insights that I was suffering from what my manager called analysis paralysis. Instead, what I needed to do was find out what issues were affecting the most number of users and the severity of it. By organizing my thoughts using the strategy, I was then able to hone my analysis into one specific issue and then recommend clear actions to help. Like your job as a quality assurance manager that we imagined at the beginning of this video, as a data professional you may be responsible for cleaning up data sets so that they are ready to use. You won't always be sure what obstacles are in cleaning data, so let's get started on some of the most common. Remember earlier when you imagined yourself as a quality assurance manager at a company assembling a board game? When you discovered that the machine that produces the games was having technical issues and causing game boxes to have pieces missing, you and your company could have chosen a different number of paths forward. One, you could have thrown away the impacted game boxes. Two, you could have left them as is and pulled them at a discounted rate. Or three, and the one you chose in the example, you rummaged through the boxes and corrected the mistakes. There was a plan, strategy or protocol for dealing with missing pieces. The same should be true for any missing values you find in a data set. When you have missing entries in your data set you have to decide on a plan for dealing with them. Even more than that, you must be ready to communicate with stakeholders if missing values impact your analysis. Missing data, which is often encoded as a NA, NAN, or BLANK is defined as a value that is not stored for a variable in a set of data. This is different from a data point of zero. I'll explain more on that in a bit. Data professionals often experience the challenge of missing data. Every data set is different, so there are a wide variety of reasons for values to be missing. Everything from a computer error during a data upload to someone simply forgetting to input it. Depending on the number of entries in your data set and the quantity of missing values, the impact of data fields with a not a number or NANs can range from negligible to substantial. The impact will also determine how you communicate to stakeholders or clients, ranging from a note at the bottom of a data visualization about the fact that there's missing data to a face-to-face meeting about how analyses cannot be completed due to the large number of missing values. Here's an example of the impact missing data has on a data set and the ethical questions missing data creates. Imagine you're a data professional trying to learn more about sleep habits. You email a questionnaire to say 100 people and all of them complete it. One of the questions asks do you sleep exactly 8 hours each night? The answer choices are yes, no, or unsure. 9 people answer yes, 9 answer no, and 15 answer unsure. Because only 33 people responded to that question, it's hard to draw a definitive conclusion about the sleep habits of all 100 students. Since 100 people completed the questionnaire and only 33 responded to this question, 67% of the replies would be considered missing. These missing values greatly impact the questions resulting data. It would be unwise for a data professional to try to use the 33% response rate in order to draw conclusions about the entire population. If you can counter this level of missing values while working as a data professional, you should communicate the inability to complete any analysis and suggest possible solutions. Another challenge that can come up during EDA is with the value zero. In some data sets a zero could be considered a missing value, but in other data sets could be a legitimate data point. In some data sets a nan or blank space might be a mistake. So one forgot to fill it in. Or the data may have been left blank because the column may not apply to that data point. Data professionals unsure of whether or not the blank space is intentional should ask a stakeholder or the owner of the data to confirm the space's legitimacy. Whenever you find missing data, you have a choice to make on how to handle it. As a data professional, you should consider how the missing data might impact stakeholders and who should be made aware. Here are four common ways to handle missing data. 1. You requested the missing values be filled in by the owner of the data. 2. You delete the missing columns, rows or values, which would work best if the total count of missing data is relatively low. 3. You create a nan category or 4. You derive new representative values such as taking the median or average of the values that aren't missing. First, we'll talk about filling in the missing data. If there are large quantities of missing values in the data you receive, the best method to handle the missing data would be to contact the owners of the data and request that that data be filled in. For example, in our sleep example, you could send a follow up request for people to log a response to that particular question. Or, you could rephrase the question, how many hours do you sleep each night? However, in many data sets you'd see while working as a data professional, the option to get missing data filled in may not be feasible for a variety of reasons. For example, if you're working on a project that is time sensitive, you might not have time to gather more data before you're deadline. Or, the data cannot be retrieved because the study happened too long ago. If filling in missing data isn't an option, you can also choose the option of deleting the missing data. Though some might assume it's not appropriate to delete data, removing rows or whole columns of data is ideal if there isn't a large percentage of man's or the values are not going to impact the business plan for the data. One thing to watch out for, though, is discarding missing data that is not missing at random. Deleting values that have been left blank intentionally can skew the results of your analysis. For example, in the sleep questionnaire, if you were to remove all the people that left that question blank, you'd delete a majority of the data. A third option is to make the nans their own category, which is a good strategy if the missing data itself is categorical rather than numerical. For example, in the sleep questionnaire, if you were to put all the non-responses into their own category called answer not recorded, you would be creating a category for the missing data. Finally, there's the strategy of filling in the missing data by creating a representative value. This strategy can be more useful with business plans that call for a predicted value or forecast. There are multiple methods within this filling in strategy to choose from, including the foremost common forward filling, backward filling, deriving mean values, and deriving median values. We will talk about how to do all these missing data operations using Python in another video. These are the most common methods for handling missing data. Choosing which method to use comes with experience, intuition, and reasoning. Sometimes you'll have the opportunity to use more than one option with a data set. Every data set and project plan will be different, so you will be making this determination each time you encounter new data. Sometimes you'll want to confer with peers, managers, or stakeholders in order to make a decision. The impact missing values has on your analysis should be the determining factor in whom you should contact. As a data professional, it is essential that you are thoughtful and intentional about how missing data are addressed. Consider the quantity of NANDs and their importance in relation to the project plan. Ask yourself how will this approach impact this data set? And what are the ethical consideration? Just like the board game scenario, it is essential to determine a strategy and decide on a plan. Now that we reviewed the importance of having a plan to address missing data, let's talk about how to identify and decide how to work with missing data in a Python notebook. We'll use the data you're familiar with, the group of NOAA Lightning Strike data sets. The goal of this video is to learn how to identify missing data. You'll be provided two different slices of data from August 2018. One slice will have the columns date, center point geom, latitude, longitude, and number of strikes. The second slice of the data will include the same columns as the first slice, along with the zip code, state, city, and state code. You will learn through comparisons of these two data sets how to find missing data. Before we look for any missing values, we need to import some Python libraries and packages. We'll use pandas, numpy, seaborn, datetime, and pyplot from matplotlib. We will begin by looking at only the number of lightning strikes for August 2018. Using less data in a Python notebook allows you to spend more time coding and less time waiting for the code to run. For this exercise, our first dataset will have two additional columns called latitude and longitude, which were points pulled from the center point geom column. Let's explore the column headers and the overall size of this first dataset using the functions that you're already familiar with, head and shape. We'll find the two additional columns we just mentioned for latitude as well as a shape of 717,530 rows and five columns. Let's put this aside as our first data frame, saved as df. Next, we'll use the second dataset for August 2018. This one has columns for zip code, which is a postcode delivery area number in the US as well as city, state, and a column titled state code that has two letter abbreviation of states in the US. We'll call the second data frame df zip, so we don't get confused with the first dataset we created earlier. Let's run the same two functions, head and shape, on df zip. The head function returns what we expected out of the dataset. The data columns zip code, city, state, state code, centerpoint geom, and number of strikes. The shape function, however, returned 323,700 rows and seven columns. Given that this data frame is also from August 2018, we expected to see the same number of rows as the first data frame, 717,530. To further explore this, let's put the two data frames together with the merge function. We'll need to create a new data frame. We'll call it df joined followed by an equal sign and df.merge. This df.merge indicates that we want to merge the first data frame we saved named df with another data frame. For the merge, put df underscore zip in the argument field, which is the data frame we want to merge with df. Next, fill in the parameters for the merge, including how equals left followed by a comma. Finally, input the last parameter on equal sign followed by two column names date and centerpoint geom. These two parameters how we want to merge the first data frame on tell python which way to join the data. When we run the head function on our new data frame, we find nans listed for the columns zip code, city, state, state code and number of strikes y. Now that the data is merged, let's use a basic function we learned in a previous lesson to search for nans. When we run the cell, we find that the total count of lightning strikes has been divided into two. Number of strikes x and number of strikes y. Once we merge the two data frames, pandas automatically separated number of strikes into data entries with full data rows and data entries that have missing entries. We can find the total amount of data that's missing using this next piece of code. We'll create a data frame called df null geo by taking our join data frame, df joined and then adding the pandas function is null all one word. The term null indicates that the data is missing. This function pulls all of the missing values from the df joined data frame. We already know which of the columns have missing values. Since we're interested in finding the total, we can use any one of those columns. For this code, we selected state code. After that, we'll use the shape function to give us the total rows and columns which comes to 393,830 and 10 respectively. For even more detail, use the info function which we used in another video. When we input df joined .info with the argument field left blank, we get the column names and another column called non-null count. Non-null count is the total number of data entries for a data column that are not blank. A quick check of the non-null count column helps us confirm which columns have missing data. Zip code, city, state, and state code. In fact, if we subtract the total number of strikes x and the number of strikes y, the result is the number we identified earlier. 393,830 Let's take a look at the top portion of the new data frame we created, df0 using the head function. The output tells us exactly what we expected to find in the data frame we created. In the top five lines of data, the columns zip code, city, state, state code, and number of strikes y have the nan values in the cells. Now that we've pinpointed exactly what values are missing in our data frame, the last thing we'll do in this notebook is learn how these missing values impact our data. The best way to do that is to create a data visualization. A plotted map will help us see where the majority of the missing values are located geographically. To design the map, let's start by creating another data frame. This one we'll call top underscore missing. We are gathering only the data comms we will need in order to plot a geographic visualization. Latitude, longitude, and number of strikes x. As you'll recall from earlier in this video, number of strikes x includes all of the 717,530 data rows while number of strikes y are a segment of those 717,530 with missing data in the zip code, state and state code columns. You'll find that it's helpful to group the columns by latitude first, then longitude. This will make it easier for most data processors to plot. Lastly, we'll sort the lightning strikes x column by sum of values. In the ascending field, we'll input x because we want the largest sums of lightning strikes at the top of the data frame. Finally, let's take a look at what we built using the head function again. After running the cell, the first 10 rows show the most strikes are falling on and around the same latitudes and longitudes. Let me give you a helpful hint before we plot a visualization using this data frame. You should import the express package from plot leaf first. Express is a helpful package that speeds up coding by doing a lot of the back end work for you. If we don't use express for this particular data set, which has hundreds of thousands of points to plot, your run cell times could be long or the code could even break. Now it's time to create the map. Our format for the graph will be from plot leaf express called scatter geo. As indicated by its name, this graph type is used for scatter plots on a geographic map. In the argument field, we'll input the data frame we just created top missing. For the parameters in the argument field, we want to filter the number of values to be only those latitudes and longitudes with lightning strikes of more than 300. Naturally, we plug latitude and longitude into their appropriate parameter spots. Then, for the size parameter, we plug in the final column number of strikes X. Lastly, let's use the update layout function within the figure to give the plot title missing data. Now, we create it. It's a nice geographic visualization but we really don't need the global scale. Let's scale it down to only the geographic area that we are interested in, the United States. So, let's copy the same code we just wrote. This time, though, we add the parameter geo underscore scope equals USA. This will limit the geographic scope to just the United States of America. Then, we can plot it again and check the output. The resulting map shows a majority of these missing values cropping up along the borders or in spots over bodies of water, like lakes, the ocean, or the Gulf of Mexico. Given that the missing data were in the state abbreviations and zip code columns, it does make sense why those data points were left blank. There are no zip codes for bodies of water. You'll also notice some other locations with missing data that are not over bodies of water. These types of missing data, latitude and longitude on land are the kind of missing data you would want to reach out to the NOAA about. What's interesting about cleaning the lightning strike data and looking for missing data values is that we learn something new in the process. We found one of the stories hidden in the data. The data values were missing because the lightning strikes were over bodies of water. It goes to show you that data cleaning may sound tedious, but you'll never know what you learn. Hi, my name is Remi, and I'm a customer engineer for Google Cloud. A customer engineer is someone who focuses in some technical field. For me, it's data analytics and machine learning, and is the expert in that field and helps their customer answer questions and build things related to data analytics. I had an exaggeration around 30 different jobs before I got into the field that I'm currently in. I was a cashier, a hostess, a cocktail waitress, an office assistant, and quite frankly, I wasn't very good at any of those jobs. If something's like repetitive, I can't concentrate, and then I end up making silly mistakes. So that's kind of what's so great about data science is a lot of the minutiae and repetitive tasks is software does it for you where you can write a script to do it. There is definitely no typical day in the life of a customer engineer. Usually I will start the day by looking at emails. I'll usually have some kind of questions from a customer, another colleague related to data that I can help answer. There's a lot of informational emails about new features, products, or techniques. I have to read those and learn about them because it's my job to be the person who knows about all of these things for our customers. I'll also typically have a few meetings where I'm working with our account and customer teams where we kind of try to strategize how we're going to help customers and get them to do new things with our tools. A new feature that I've been working with is related to our data warehouse big query. You can actually create machine learning models within BigQuery so you don't have to move the data anywhere. I have a variety of customers that use BigQuery machine learning so I can now come to them and share with them this new feature and show them how it actually works and if while they're using it they might have an issue that comes up, they get some kind of error I can help troubleshoot that problem for them. There's so much going on in this field of data and it's also new it's going to feel overwhelming I've been doing it for almost a decade and I still get overwhelmed on a daily basis. The best thing you can do is pick a software, pick a language maybe a specific area and just focus on that so you know get really good at one thing and then you can expand from there. As a child did you ever play a puzzle game in which you were supposed to look at a picture and find an object that doesn't belong? Maybe sometimes the object was obvious but other times it may have been more difficult for example if you were given a picture of a bustling marketplace full of people selling fruits, vegetables, and grains it would probably take some time to find the item for sale that's not in the right place. Similar to visiting a busy market when you encounter a data set that hasn't been cleaned it can be really challenging. Data may at first seem structured but it often can be hard to determine whether there are any data points that are measurably different from others these extreme data observations, the ones that stand out from others are known as outliers. Outliers are observations that are an abnormal distance from other values or an overall pattern in a data population. In this video we will discuss the three main types of outliers and why it is important for data professionals to identify them in the data we analyze. As a data professional you should be fully aware of the meanings and ends, highs and lows and extreme points of your data across every variable in the data set. These values will often be your outliers. It is essential that you recognize them what their value is and where they are. Otherwise they will likely skew any conclusions you draw or models you build from it. There are three different types of outliers we will discuss in this video. Global outliers, contextual and collective outliers. Later you will learn how to find outliers and work with them in Python. The first outliers we will discuss tend to be the easiest data points to detect. Global outliers are values that are completely different from the overall data group and have no association with any other outliers. They may be inaccuracies, typographical errors or just extreme values that you will not see in a data set.  Typically, global outliers should be thrown out to create a predictive model. Contextual outliers can be trickier to spot. Contextual outliers are normal data points under certain conditions but become anomalies under most other conditions. As an example, movie sales are expected to be much larger when a film is first released. If there is a huge spike in sales a decade later, that would typically be considered abnormal or a contextual outlier. These outliers are more common in time series data. Another example might be an outlier only in a specific single category of data. For instance, 2.5 meters may be a normal enough size for a category called mammal heights but if the mammal is found out to be a mouse we would likely consider this an outlier despite it fitting in with other mammal heights. Lastly, we have collective outliers. Collective outliers are a group of abnormal points that follow similar patterns and are isolated from the rest of the population. Think of a parking lot at a store. It is not uncommon to have cars or scooters coming and leaving consistently during the hours a store is open but to have a full parking lot after the store is closed would be considered a collective outlier. It could be that there is a company party or a local event nearby which would explain the outlier of cars parked in the store parking lot after hours. One useful way to find these different types of outliers in our data is something we've already discussed quite a bit, visualization. It will be easier to see any big dips or giant blips in the data when we plot it on a line graph or a bar chart. No matter how you discover that your data contains global, contextual or collective outliers, it is essential that your EDA includes a strategy for dealing with them. It will be up to you to decide how outliers need to be represented or whether they need to be removed completely. The decision on what to do must always be done in the context to the data set and the business plan for the data. As mentioned in other videos, always consider the ethical implications of any decision you make about outliers. When you're working as a data professional, it may be tempting to remove outliers to improve results, predictions or forecasts but you will need to ask yourself does that change the data story? For example, let's imagine that you are a data professional at a retail business. You review two years of sales data and find that there's one month where sales of a typically popular item are down dramatically. You might at first assume that there's a typo in the reported data, but as a diligent data professional you ask questions about it. You discover that several top sales people were on leave during that month and your company advertised a new product which slowed sales on the existing item. It is likely that both of these issues led to a drop in sales. Rather than dismissing the drop in sales as an outlier this information will be helpful for the business team. Using this information managers can prevent a future drop in sales numbers ahead of product launches by hiring additional sales team members limiting requested employee leave during those critical times. Later you'll learn how to find outliers and work with them in Python. Earlier you learned about three types of outliers global, contextual and collective. Now that we discussed outliers on a conceptual level I'll show you how to identify them and analyze their impact in a Python notebook. Our goal will be to identify outliers across a 33 year span of total lightning strike counts in the United States. As usual, let's start by importing our libraries and packages. Pandas, NumPy, Seaborn and Pyplot from Matplotlib. We'll continue using our NOAA Lightning Strike data set. This time we'll group the total sum of the lightning strikes in the United States by a year from 1987 to 2020. Let's first reveal the top 10 rows of data using the head function. The data set has two numeric columns year and number of strikes. As you'll notice we're dealing with some fairly large numbers for the lightning strike totals. It would be helpful to make them a bit shorter and easier to read in visualizations. To do that let's write a readable numbers function. Below the function there is a long explanatory text inside a triple set of quotation marks. This is called a documentation string or doc string. A documentation string or doc string is a line of text following a method or function that is used to explain to others using your code what this method or function does. A doc string represents good documentation practice in Python. It makes the code easier to understand and can be easily exported to create library documentation. A doc string was already provided for this readable numbers function. Next we'll form an if else statement. We want to code any number above six digits to be formatted to one decimal place followed by an M. Under the else statement we are formatting numbers with more than three digits to the same one decimal place followed by the letter K. Under these statements we define the new data frame we want titled number of strikes readable. We then use the apply function to set our readable numbers statement to only the number of strikes column in the data frame. Using the head function we'll see the output of our newest code. The new column has readable values of 15.6 million, 209,000 and 44.6 million in its first three rows where before there were just long strings of numeric digits. This indicates that we coded the read numbers function correctly. Now that we have more readable values let's plot our data. As you'll recall our goal is to find any outliers among the total number of lightning strikes in 33 years of NOAA data. One way to find any outliers would be to investigate the mean and median of this data. The mean is the average number of the given values and the median is the middle point of the given values. When we code for these two values we use the numpy functions of mean and median. We're using the readable numbers for simplicity. The resulting output is the mean 26.8 million and the median 28.3 million. With the mean being 2 million strikes less than the median we suspect the data distribution is likely skewed to the left. The left side of the distribution will be a good place to investigate. One effective way to visualize outliers is a box plot. As we discussed in a previous video a box plot divides the distribution of data points into four main quadrants or quartiles. The box plot is helpful in visualizing and confirming the outliers we've already found. Let's plot the box plot first in more detail. To design a box plot using Python we'll use the seaborn box plot function. We'll use the number of strikes column from our data frame as the basis for the plot. Next we'll use set x tick labels based on the readable numbers statement we created already. X tick labels simply allow us to give names to the x marks so that they are read more easily. Lastly we input the x and y labels and the title. You'll recall that the purpose of a box plot is to show the distribution of values separated into quadrants or quartiles. What we are most focused on with our plot are that the two points beyond the far left of the line are below 10 million. This data visualization is showing us very clearly there are outliers included in our group of data points. The blue rectangles in your box plot are called the interquartile range. This range represents the difference between the third quartile and the first quartile of a set of data values. A standard rule statistics which you'll learn more about in another course is that any data point that falls beyond 1.5 times the blue boxes are considered outliers. This next piece of code will help us define what value is at 1.5 times below our interquartile range. We'll use the quantile function to define what we'll call percentile 25 and percentile 75. We enter our data column number of strikes for our data frame and then 0.25 0.75 in the argument fields. The values that occur between the 75th and 25th percentile are the interquartile range. We create a statement for interquartile range IQR defining percentile 75 minus percentile 25 is equal to IQR. Now let's define two statements, upper limit and lower limit. Upper limit is equal to percentile 75 plus 1.5 times IQR. Lower limit is percentile 25 minus 1.5 times the IQR. Lastly, let's have Python provide the exact value of the lower limit. We'll input print followed by the text lower limit is colon plus sign, readable numbers and lower limit in the argument fields. After running the cell we learn that the lower limit is 8.6 million. Don't worry if you're still trying to understand interquartile range. This concept will be explored further in depth later in the program. Next, let's plot the values that lie below our lower limit of 8.6 million. Place the number of strikes column inside the brackets next to DF. Then, use the less than symbol followed by the lower limit we just derived. And now, here are the outliers. One great way of seeing these outliers in relation to the rest of the data points is a data visualization. For this plot, let's do a scatter plot. You may recall that a scatter plot represents relationships between different variables with individual data points without a connecting line. To start the scatter plot, let's first add labels to each point on the plot. We'll define them using the add labels function for X and Y points. The plt.text function allows us to define how we want the text of each data point to appear. In this case, we tell the text to start 0.5 pixels to the left of the data point and to start 0.5 pixels above the data point. We also define the number of values using our readable number statement from the beginning of the video. Next, let's add color to the data points. Let's make the outlier points red and the other points blue. To do that, we'll define colors by the lower limit. For those that fall below the previously defined lower limit, we'll code that they should be R for red, and everything else will code as B for blue. Our next line of code should feel familiar. We start by configuring our visualization size. In this case, we code 16 and 8 inches, which are the default units for this function. Next is our code for scatter plots. ax.scatter The ax refers to axis, which tells the notebook to plot the data on an x versus y data graphic. Inside the argument field, we first input the x-axis and then the y-axis, which is year and number of strikes, respectively. For the rest of the parameters, we fill in the x and y label and the title. We finish the parameters of our scatter plot by defining the x-axis tick labels and setting the rotation of the x-axis ticks to a rotation of 45 degrees. The resulting plot shows us the years 1987 and 2019 are the two values in red are two outliers. Now that we've narrowed our scope of the outliers to just two years, 1987 and 2019, we can do a little more investigating. Let's start with the 2019 Lightning Strike data. If we import just the 2019 data, the first thing we'll want to do as we learned in a previous video is convert our date column to date time. Next, we'll create some new columns in our data frame, month and month text. You'll notice for the date, we use the str.slice function to cut the month names down to just the first three letters. Finally, we create a data frame that groups the total number of lightning strikes for each month. The result explains why the year is an outlier. For the 2019 data set, only lightning strikes from the month of December have been inputted. If you were a data professional tasked with calculating lightning strike risk, you would first research the documentation available to see if there's a reason given. If you don't find anything based on your research, you would then ask the NOAA why lightning strikes have been recorded for the other 11 months of 2019. When we do the same code for 1987, the resulting output is different from 2019. We find there are lightning strikes recorded for each month in 1987. This difference that 2019 does not have data for all months and that 1987 does have data for all months helps us to know how to handle these two outliers. For 2019, it would make sense to exclude it from our analysis. As for 1987, we recognize that it is indeed an outlier, but it should not be excluded from our analysis because it does have lightning strike totals included for each month of the year. Now before we end, let's return to the mean and median values that we calculated earlier. First, let's create a new data frame including only data points that exclude the outliers we identified. Our goal here is to show the mean and median of the data set excluding the outliers. To do this, we'll use a data frame without outliers we just created and our readable numbers statement to only include data points that are greater than our lower limit. When we exclude those outliers, the mean and median are much closer together suggesting a fairly evenly distributed data set. Now, let's revisit the goal we set earlier. To identify outliers across the 33 year span of total lightning strike counts in the United States. We discussed a lot of new data sets and we achieved this goal. Have you ever tried to charge your phone with the wrong type of plug? It is a frustrating problem but definitely not an insurmountable one. You will probably experience a similar frustration when you work with categorical data. Categorical data is data that is divided into a limited number of qualitative groups. For example, demographic information tends to be categorical like occupation, ethnicity and educational attainment. Another way to think about categorical data is data that uses words or qualities rather than numbers. As a data professional, you will likely work with data sets that contain categorical data. Categorical data entries can typically be identified quickly because they are often represented in words and have a limited number of possible values. Many data models and algorithms don't work as well with categorical data as they do with numerical data. Assigning numerical representative values to categorical data is often the quickest and most effective way to learn about the distribution of categorical values in a data set. There are some algorithms that work well with categorical variables in word form like decision trees, which you'll learn about in another course. However with many data sets, the categorical data will need to be turned into numerical data. This conversion is often essential for predicting, classifying, forecasting and more. There are several ways to change categorical data to numerical. In this video, we will focus on two common methods creating dummy variables and label encoding. One way to work with categorical data is to create dummy variables to represent those categories. Dummy variables are variables with values of 0 or 1 which indicate the presence or absence of something. It helps to open a data set with dummy variables already created to understand what their function is. You'll find in this data set that for any value that has been determined to be mild A1 has been input. All other values in the mild column are labeled with 0s. The same goes for other categories and values. You can think of it as the 1 representing a yes and the 0s representing a no. Dummy variables are especially useful in statistical models and machine learning algorithms. Another way to work with categorical data is called label encoding. Label encoding is a data transformation technique where each category is assigned a unique number instead of a qualitative value. For instance let's imagine you had a data set about mushrooms in which you wanted to understand the distribution among different types of mushrooms. Imagine you've been given a data set about mushrooms with a column titled type and the options of black truffle crimini, king trumpet, button, hedgehog, morel, portobello, tollstool, or shiitake. You can use label encoding to transform each mushroom type into a number. Black truffle would be changed to a 0. Button would become a 1, crimini a 2, hedgehog a 3, king trumpet a 4, morel 5, portobello 6, shiitake 7, and lastly tollstool would be 8. So why do data professionals use label encoding? Data is much simpler to clean, join, and group when it is all numbers. It also takes up less storage space. An algorithm or model typically runs smoother when we take the time to transform our categorical data into numerical data. For example, let's say you try to run a prediction model using the mushroom data set, which will try to anticipate the percent chance a new mushroom introduced to the data is a portobello. If we try to create a model without first performing the label encoding, the prediction model will not function. Of course, there will be models and algorithms in which you may not want to perform label encoding. Let the business need be your guide on whether or not you'll need to perform label encoding. The type of model or algorithm you choose will determine whether or not you encode labels. You'll learn more about algorithms and models in an upcoming course. Much like making sure you have the right phone charger to plug into your phone as a data professional, you need to make sure your data is ready to use with models or algorithms. In a previous video, you learned about the importance of label encoding, also known as transforming categorical data into numerical data. Now it's time to open our Python notebooks and learn how to do it. Let's continue with the same NOAA data set we've been using for lightning strike counts. For this notebook, we'll look at data from 2016, 2017, and 2018. Let's start by importing Python libraries and packages. For this notebook, let's import Datetime, Pandas, Seaborn, and Pyplot from Matplotlib. As you learned previously, start by converting the Date column into DateTime, which makes it easier to manipulate. Next, create a column called Month. Like we did earlier, we will use str.slice to cut the month names down to only the first three letters. When working with this data, it is helpful to make sure the month names stay in chronological order. Our next line of code then is months, which simply defines the group of months in order. You'll find this helpful in the next line of code, where we use the pandas function, categorical, to group the number of strikes column by the month column. Because we have three years' worth of data, let's also create a column listing the year. To do this, we name the column year using the DateTime function strftime. This function will return a string representing only the year in the original Date column. Inside the argument field, we input %y indicating we only want the year from the DateTime string. Finally, let's create a data frame called df by month, which groups the number of strikes first by year and then month. The last part of the code here is making sure that the number of strikes is added together for each month using the sum method. We'll tack on the reset index function to tell the notebook to recount the rows from zero without using its initial order. When we look at the top of the data frame using the head function, we get the first five rows of data with columns of year, month and number of strikes. Let's create a column for categorical variables called strike level. In this column, we will group or bucket the total number of strikes into categories, mild, scattered, heavy and severe. This is how we perform label encoding in Python. To do this, we'll create a new column in our df by month data frame called strike level. This new column will be coded using the pandas function, q cut. The q in this function refers to quantile, meaning that when using this pandas function, you can cut the data into four equal quantiles. First parameters, we first input the desired column we want to cut into quantiles, which will be the column called number of strikes. Then we input the number of quantiles we want, which is four, one for each classification. We type in those classifications under the label parameter mild, scattered, heavy and severe. Let's use the head function to find out how our code modified the data frame df by month. It may seem counterintuitive to create categorical data in order to then transform it to numerical data, but this process will allow us to segment the data into useful chunks, which can then be plotted on a graph in interesting ways. Not to mention, string categories are easier to understand in a data visualization than numbers. One other column which will help us later is to assign numerical values to the strike levels we just defined as mild, scattered, heavy and severe. We'll call this new column strike level code. We'll define it by taking the column strike level and adding dot, cat, dot, codes. Cat codes is a pandas function that takes categories and assigns them a numeric code. In this case, when we run the cell, we find that mild has been assigned a zero, scattered assigned a one, heavy a two and severe a three. And that's how we perform labeling coding. Having both the categorical terms for different levels of lightning strikes and the numeric codes to go with them, we can create data visualizations or models that are straightforward to interpret. One other helpful way to work with categorical data is to create dummy variables to represent those categories. As you'll recall from a previous video, dummy variables are variables with values of zero or one which indicate the presence or absence of something. The pandas function we'll use to achieve this in our python notebooks is called get dummies. To clarify, new columns of ones and zeroes are also known as dummies. The get underscore dummies function converts these categorical variables mild, scattered, heavy and severe into numerical or dummy variables from the column called strike level. If we run the cell you'll find this function creates four new columns and puts a one in any row where the number of lightning strikes falls into the labeled category and a zero for anything else. Now that we have our new columns of categorical data, numeric data and dummy variables, let's discover what you can potentially learn from these groupings. To do that, let's plot our data. We'll first create a data frame called df5 month plot. Let's reshape the data by using the pandas function pivot. Pivot allows us to reshape our data frame into any given index or set of values we want based on the parameters inside the pivot function. Those parameters are index, columns and values in that order. To help us with the visualization, let's use year in the index field, month in the columns field and strike level code in the values field. If we use our head function, we find all three years in the rows, the months in the columns and the strike level categories in each month just like we coded. Next let's visualize our data using what is called a heat map so we have a better idea of where most of the lightning strike values are. A heat map is a type of data visualization that depicts the magnitude of an instance or set of values based on two colors. It is a very valuable chart for showing the concentration of values between two different points. Let's create a heat map using Python. Using the heat map plot from Seaborn we can place df5 month plot in the parameter field for the data along with cmap equals blues. This map definition will give us a preset color gradient from dark blue to white. Next we fill in some of the parameters to help with understanding the visualization. In this case, we'll use the standard collection 0 from Seaborn for our color bar. This will just give us a preset gradient color bar which saves us a lot of time from having to code our own colors on a color bar. Next, we'll set the color bar ticks to 0 through 3 and label those ticks according to our 4 strike levels severe, heavy, scattered and mild. Lastly when we code the show function we are given a beautiful heat map. Using the heat map we can find the most severe lightning strike months across all three years. Thanks to Pandas and Seaborn transforming data into categories and back into numerical data is possible with just a few lines of code. As we discussed, it's easier to plot a visualization by using strike level code as the base values. The categories we created of mild, scattered, heavy and severe became the labels on our heat map plot. Label encoding can be as simple as that and it is an essential skill for data professionals. When I think about input validation and the EDA practice of joining I like to think about vegetables. Allow me to explain. When you're at the market picking out leafy greens and root vegetables you check that they're fresh first, right? And not just at the store you also check on them at home when they're in the refrigerator.  Lastly, if for some reason the vegetables have gone bad or you didn't buy enough for a recipe you would need to add more. Performing EDA on a data set is similar to checking for the freshness of vegetables. You are searching, exploring and checking that the data is as error free as possible. You should check and recheck your data sets to make sure that they are correct. As I mentioned before as a data professional you should know your data set thoroughly. One way to make sure you know a data set is validating it. Which is, as you recall, one of the six practices of EDA. There are many different ways to validate a set of data but here we'll discuss input validation. Input validation is the practice of thoroughly analyzing and double checking to make sure data is complete, error free and high quality. Input validation is intended to be an iterative practice. Meaning you should perform it again and again in between and during the other five practices of EDA which are discovering structuring, cleaning joining and presenting. Most often as a data professional you will perform input validation when starting a new analysis project or getting familiar with a new data source. We will discuss more about the how of performing input validation later in the program. Now we will focus on the why and the what. Why should we take the time to validate data? What exactly are we looking for? When we validate data we help make more accurate business decisions and we improve complex model performance. Think about it like this. If a gourmet chef doesn't double check the vegetables for freshness before cooking a dish the food could taste horrible or worse. Make people sick. It is much the same with data. The more careful we are in checking and rechecking our data after each manipulation of the data during EDA the less likely we are to have problems later. Clean and validated data can help prevent future system crashes coding issues or wrong predictions. So while we are performing our validation work what is it that we are searching for? No two data sets are alike. There will be different things to check for based on the type. Here are some questions to consider when validating data. Are all the data entries in the same format? For example are people's ages expressed as solitary numbers like 23 and 47 or within a range like 18 to 35 and 35 to 50? Are all entries in the same range? For instance in finance are some of the values expressed in thousands of euros and others in millions of euros? Finally are the applicable data entries expressed in the same data type? For example are all the data entries expressed in the same format of month, day or year? While asking these questions or performing EDA in general, you may find that the data you have been given is not sufficient to answer the business questions that you have been tasked with. For example let's go back to the comparison we made earlier in the video to a gourmet chef and his vegetables. Imagine that the gourmet chef is introducing a new recipe to their restaurant. They are not sure how many vegetables will be needed for the new dish. In the end the chef realizes half way through that day that they did not account for additional vegetables to be used in vegan and vegetarian dishes. It is at that point the chef would buy more vegetables. The EDA practice of joining is much the same principle. As you've learned joining is different from the structuring technique merging. While asking these questions or while performing EDA in general, you may find that the data you have been given is not sufficient to answer the business question. Joining is the process of augmenting data by adding values from other data sets. The practice of joining will be most useful if we validate the data to ensure formatting and data entries align and are the same data type. For example while adding the new vegetable the chef will want to make sure the consistency and taste are similar to the vegetables that you're buying all day. Otherwise the dish may not turn out the same. You will need to use your own logic, common sense and experiences to understand the ways in which you should join or validate each particular data set you work on. There won't be a rigid process to show you down to the detail how to handle each data file. Data science takes a lot of analytical thinking and shifting of perspectives to be thorough in your EDA validation. Experience and effort will be the best way to improve your performance in analytical thinking. We'll also be exploring examples in our Python notebooks as well. Keeping your validation practices in line with the pace workflow will also help you keep a focus on ethics. Validation should be about cleaning and correcting data for the sake of quality and correctness. You probably won't be surprised to learn that the EDA practice validating fits squarely in the analyze phase of our pace workflow but is also a practice you should use for all four phases. Remember, pace stands for plan, analyze, construct and execute. This means that what and how we join and validate data should be an alignment with the plan phase of the workflow. For example if your task is to discover which week within a given month is profitable for a business, it will be important that you check and recheck that you've grouped the revenue dates into weeks correctly. Your analysis won't be effective if you think you've grouped revenue by week but actually done it by day or month. When you look to validate your data to this level of detail it will help you toward meeting your pace goals, specifically for the construct and execute phases. Before we end there's one more thing that can help you with validating and joining aspects of EDA. Look to your peers and managers to help you. A peer reviewed data set is one of the best ways to ensure that you check yourself on bias keep ethics as a focus and ensure you are on track with the pace workflow. Remember validating data is like checking your vegetables to find if they're the right choice for the recipe. It may take extra time on the front but it may save you from headaches and stomach aches down the road. In this video you'll perform input validation in Python. Remember input validation is the practice of thoroughly analyzing and double checking to make sure data is complete, error free and high quality.  With that knowledge fresh in your mind it's time to discuss how to perform input validation in Python. We'll focus our validation on the NOAA data. The goal will be to check for errors and prepare this data set for publication. Let's open our Python notebooks and get started. First step, as usual we will import the Python libraries and packages that we need for input validation. You'll notice we have Pi plot from Matplotlib, Pandas plotly express and Seabourn. Next with our lightning strike data libraries and packages all imported in our notebook, let's use the head function to take a look at what we have in terms of columns and the top five rows. You'll find our familiar lightning strike data set also includes a fourth and fifth column longitude and latitude respectively. If we were to check the data types now using df.d types we would discover the date column is a string data type. As we did previously let's convert the column to date time. One thing we should include in our input validation is a check for missing data. You'll recall from earlier that we likely will have already done this check for missing entries during the cleaning part of EDA. The definition of validate assumes an action or process has already been completed. So if we input is null.sum into our notebook in order to give us missing values for the second time it is a practical and not redundant thing to do. After running the code we find that there are indeed no missing values. Next, for this data set let's review the ranges of all the variables. In this review we will check the highest and the lowest values of each column and the overall distribution of values. To do this we'll use the describe function which you may recall from previous video. In the argument field we'll use the parameter includes equal this will tell the notebook to include every parameter covered by the describe function. You'll notice NAND values in some of the describe fields like unique, last, and max. Remember that these are not fields in the data set but rather a summary table of attributes. For example, you'll notice that there are NANDs in the date column for mean, 50% and max. These are to be expected because dates are not data types that would be averaged. Speaking of dates, let's double check our date column. First are there any calendar days missing? We know from prior EDA work on this data set that the date column should include every day of each year. Let's confirm each day is listed by designing a calendar index. We'll call it full date range. We'll then use the pandas date range function with the first and last dates of 2018 in the argument field under the start and end parameters. After that let's add full date range dot difference then in the argument field we'll add df and in the brackets the column header date which will limit this function to checking only the date column. The result is interesting. There are four consecutive days missing in June of 2018 then two consecutive days in September and two consecutive days in December. This finding would be something to investigate or to question the owner of the data to find out the reason why. Given that the number of missing days is relatively small, you could complete your analysis by making a note of the missing days in the presentation. This will ensure that anyone who analyzes your visualization or presentation will know that the data depicted doesn't include those missing dates. Now that we've done some initial validation, let's get a better sense of the range of the total numbers of lightning strikes. If we use a simple box plot data visualization with the number of strikes as our data parameter in the argument field, we'll find that the distribution is very skewed. Most days of the year have less than five lightning strikes but some days have more than 2,000. As you recall from a previous video we can remove the outliers from the box plot visualization to help us understand where the majority of our data is distributed. We can do that by adding show fliers equals false into the argument field. The result is much easier to interpret. From what we know of lightning strikes in the North America regions of the United States, Mexico and the Caribbean, this distribution is conceivable. If the highest distribution of strikes were all in the 2,000s we might be a little more skeptical. The last bit of input validation we will do is to verify that all of the latitudes and longitudes included in this data set are in the United States. This will ensure we don't have any mistakes in our geographic values. To do that we'll first create a new data frame that removes all the duplicate values. We'll type df underscore points equals df then latitude and longitude in double brackets. Following that we'll add dot drop underscore duplicates. The reason we want to drop duplicate points for the latitude and longitude is that we don't need to check a particular location on a map twice. After running that code we have a new data frame that only includes two columns latitude and longitude. Within those two columns we find that there are no duplicate data points. Finally, we will plot these points on a map. Here's a tip. When you're working with hundreds of thousands of data points plotted on a map it takes a lot of computing power. To make it a shorter runtime we'll use the plotly express package. The package is designed to keep run times as low as possible. To plot these latitude and longitude points on a map we use the scatter geoplot from plotly express. Within the argument field we'll fill in the proper parameters for a scatter geoplot. In this case we'll first input the data frame name df points followed by first lat for latitude then lon for longitude. And the last code is to show the scatter plot. After running the cell you'll notice the runtime is still slow because of the sheer volume of the data. Most data professionals use input validation regularly so these are important concepts to learn and important technical skills to practice. As you learn data sets can be busy and chaotic. There are many different things happening in them it can be difficult to figure out where to focus and how to uncover those stores buried beneath the surface. In this section of the course you learn some important strategies to help you correct mistakes and discover hidden data stories. We focused our story finding efforts by learning the EDA practices of cleaning, joining, and validating. We discussed a lot of ways to engage in those practices. We focused on working with missing values and outliers, transforming categorical into numerical data and input validation. And we reminded ourselves not to forget to visualize our data in Python to help further our understanding. We discussed identifying missing data and outliers in Python and why it's important both from an ethical perspective and a business sense to find them. We considered the difference between categorical and numerical data and why transformation using Python is important. As for input validation you learned what it is, why it's important and how to perform it using Python. We also discussed some workplace skills along the way, such as understanding when to communicate with stakeholders or other subject matter experts about missing values. We also reviewed ethical considerations you should make when performing cleaning, joining, and validating work. You learned why these practices are vital when performing EDA on a data set as part of the PACE workflow. EDA can be an exciting process. Thorough EDA not only saves time and energy on future processes, but it also helps you find the trends, patterns, and stories in the data. Later, you'll learn how to use Tableau to design and present data visualizations to key stakeholders and business managers in a business setting. In the meantime, you've learned some amazing skills that data professionals use almost every day. The ability to perform careful EDA on a data set in Python is an essential building block in a data professional's career. Great job on your work so far. We've reached the final practice of EDA, presenting. As we discussed, visualizing and presenting don't necessarily come at the end of your data exploration. You may create visuals of your own data throughout the data analysis process. The practice of visualizing your data will help you discover insights and further your own understanding. The presenting practice of EDA is an important part of both the analyze and execute phases of the PACE framework. Sometimes you'll create visualizations to analyze the data during the analyze phase. At other times, you'll utilize data visualizations as part of executing an algorithm. In this video, we will focus on designing data visualizations for the purpose of presenting. In this course, we'll discuss concepts such as accessibility, tablo basics, dashboards, and data visualizations. You may recall some of these concepts from the Google Data Analytics certificate. If you'd like, take a few minutes to review that content before moving ahead. Coming up, you will learn how to improve your data visualization skills. The accurate representation of data is one of the most important aspects of a successful presentation. You'll learn specific techniques for making data visualizations that represent the data with precision. Another aspect of designing successful data visualizations is ensuring they are inclusive. We will talk about tips and techniques for making your data visualizations accessible to a diverse audience. You will also learn about the data visualization platform, Tablo. Tablo is a versatile data visualization software primarily used for presenting data to inform and improve businesses. Using the software, you'll learn how to create data visualizations that tell stories and explain technical concepts to non-technical audiences. You'll learn to create dynamic visualizations with interactive and motion elements, and to alter visualizations based on different audiences' needs. We will also discuss how to choose the right graphic or chart, give context during presentations, and select the proper order and timing for your data visualizations while presenting. Data professionals often need to create data visualizations and deliver presentations. That's why we'll discuss the most important skills to help you thrive in this area.  As a data professional, you will have an opportunity to tell data-driven stories that change your team, department, company, industry, or even the world. Now let's get started. Often, creating a data visualization is an iterative process. The first graph you design may not be the one you share with stakeholders. The colors, text, labels, scale, and areas of emphasis are all aspects of a graph that may change to meet different business needs. But how do we get to that final product? What decision should we make along the way to create a successful visualization? To answer these questions, we'll take a sample data set and create a data graphic. Then, we'll discuss how to alter the visualization along the way. This will illustrate the types of decisions data professionals make when creating graphs and charts for presentations. Now let's get started. Imagine you're working on a new app which will gather a list of the top-rated rental homes and departments in Europe in one easy-to-access place. Your company asks you to start by gathering the best rental listings in Athens, Greece. You want to know the locations of rental units where the owners have 40 or more property listings. You only want properties with many good reviews and a rental listing price between 90 and 250 euros. This helps to clarify the type of visualization you need. Since you want the locations of rental property units where owners have 40 or more property listings, you need a geographic map. Additionally, you limit the count of listings based on the criteria of your analysis. To create a map with the locations of rental properties, a program like Tableau will be the most effective. If you're familiar with Tableau, feel free to add this data set into the public Tableau and try it yourself. First, we'll start with a map of Athens. We'll plot all the rental listings on that map. We'll use filters to remove rental properties that don't meet the analysis criteria such as prices outside our defined range. Finally, we'll adjust the formatting to make our map easily understandable and accessible. Let's consider the dimensions of the graphic. We know that the end result needs to be a map of listings in Athens, Greece. Let's input the latitude and the longitude from the data set and plot those listings with the host name as the labeled spot on the map. Now that we've added those data points to Tableau, let's see what it looks like. We have a map of Greece, but we need to refine it so that it's easier to read. Next, let's filter the listings we don't want based on our analysis criteria. Narrow the results to only those owners who have more than 40 rental listings. We can do that with the column in the data set called calculated host listings count. We limit the amount to only values over 40. This narrows the fields. Next, we limit the entries by price. Our criteria includes listings between 90 and 250 euros, which gives us a much more manageable number of listings. We want to keep only the listings with a lot of reviews. The word a lot, of course, is relative. So we'll use a percentile filter to just give us the top 50% of total reviews. This is the result. Now we have a reasonable amount of data points. But what's next? We need to make it easier to comprehend. To do so, we add a title and add a color coding for the owners of each listing. We also add the price for each listing and this is the result. At this point, we have a workable visualization that meets our criteria. This is a successful first version. However, if we want to make this visualization part of a formal presentation, we need to make it accessible. We do that by including a descriptive caption beneath it and making the marks slightly smaller so more of the prices display. We also want to be sure the colors are friendly to individuals who might have difficulty seeing color. This is the result. The goal with visualizations is to meet an audience's needs. You want to share the geolocation of rental property owners. You accomplish that with your map of Athens. But what if the app's business team decides they need different data display? Or if you want to review property listings by neighborhood, you need to update the visualization. The main point here is that the resulting visualization may not always fit the business need. Designing visualizations as you'll remember is a multi-step process. Even after you've completed a visualization you will sometimes find you need to update it. You may also get new information that requires you to change the original criteria. Making changes and fixing mistakes is all part of the process. We talked a lot about the power and usefulness of Tableau.  Great. We'll discuss how to access a free online version of Public Tableau, build basic data visualizations and understand when to use a variety of charts. You may remember some of these concepts from the Google Data Analytics Certificate and you can revisit that program for a quick refresher. When you access Tableau Public it may appear different from this video. Keep in mind that Tableau Public may have updated its user interface. This should not be a problem because the steps you follow are almost the same. To begin go to the Public Tableau website which is the free online version of Tableau you can access from your browser. Sign in to your Tableau Public account. Next, you'll need to access a new data source from the Tableau Public. We'll use the NOAA Lightning Strike data set again for this video. On the Tableau Public home page select Create a Viz. Upload your data source from your computer using the file provided for you. Once the data is uploaded it will be redirected to Tableau's data source screen. The data set is divided into four columns Date, Number of Strikes, X coordinate and Y coordinate. There are symbols above or to the side of these fields names representing the types of data that are included in the data set. There is a calendar icon for the date, a pound sign for the numeric column, number of strikes, and globe icons for the latitudes and longitudes or X and Y coordinates. There are also tabs representing a new worksheet, new dashboard and story. Create a new worksheet. A worksheet in Tableau Public is a data page that contains a single view of a data visualization. The new worksheet is blank other than the data column. The data source field of the worksheet is pre-loaded with the column headers we identified on the data source page. There is a thin line separating the list into two sections. This indicates the data type and whether the data is discrete or continuous. Tableau divides all the data fields into two broad data types, dimensions and measures. You can tell them apart based on the icons Tableau assigns to each field. Dimensions are qualitative data values used to categorize and group data to reveal details about it. Measures are numeric values that can be aggregated based on calculations. The green and blue colors in Tableau indicate another aspect of the list of dimensions and measures in the Tableau worksheet data tab. Green indicates that the data field is continuous. Blue indicates that the field is discrete. The term continuous is a mathematical concept indicating that a measure or dimension has an infinite and uncountable number of outcomes. Discrete is a mathematical concept indicating that a measure or dimension has a finite and countable number of outcomes. With these definitions we can learn about our list of data fields even before we plot them on a chart or graph. Keep in mind that Tableau may assign the wrong data type. You can always change data from measure to dimension or discrete by right clicking on the data field if the data type and its label do not match. Next, let's begin our visualization by plotting a line graph. Line graphs are useful for presenting time series data or tracking changes in data values over different periods of time. To start, drag date into the column field. In the pop up that displays you can select different segments of time. Select year for this chart. Next, drag and drop number of strikes into the row field. You now have a line graph. Now, you know how to create line graphs. Next, we'll create a bar chart. Bar charts are useful when you want to compare things like data from two different time periods. To get started use the toolbar to duplicate your worksheet. In the new sheet, open the drop down menu in the marks field. Select bar. Changing data visualization types is really that simple. Next, let's create a bar chart to compare two data sets.  Check 2009 and 2018. Leave all the other years unchecked. You'll find a comparison of the total number of lightning strikes in the years 2009 and 2018. Next, add a label to each bar by dragging number of strikes to the square in the marks field titled label. Now, we know the exact difference between the number of lightning strikes in 2009 and 2018. 30.1 million and 44.6 million, respectively. For the last visualization let's compare 2009 to 2018 by quarter. To start duplicate the worksheet again. Drag date to the column field. Tableau will automatically add quarters to the stacked bars sectioning out Q1, Q2, Q3 and Q4 for both years. To make the side by side comparison of each quarter stand out, drag date to the color in the marks field. The colors will adjust based on the year with 2009 in the original color and 2018 in a different color. You can keep the chart like this or click the drop down arrow and select quarter instead. Now the quarters are segmented by different colors. To determine the color scheme decide if you want to highlight the differences between quarters or between years. You're starting to discover how useful Tableau can really be. Often data professionals use Tableau for their EDA work because it helps them quickly create visuals of their data. Soon you'll be able to do that too. So far in this program you've learned the basics of Tableau public like how to upload a data source and use discrete or continuous measures and dimensions to plot data visualizations. Next you'll design more complex visualizations including heat maps, box plots and histograms. You'll also learn how to input calculations and write code within the design elements of a visualization. Let's get started. Go to Tableau public and create a new data visualization. We'll use another data set from the NOAA lightning strike data. Once you've uploaded the data source you'll start your data visualization on the data source page with the NOAA lightning strike data. The date, longitude, latitude and number of strike columns are all there. Let's open a new worksheet so that we can create a geographic map. This requires latitude and longitude data that Tableau can use to plot the points. First, drag longitude from the data list to our columns field. Next, drag latitude into the rows field. Now, we have a map. This is a great start. Next, filter the data so that you have fewer data points to work with. Drag date into the filters field and select year from the dropdown. Then, configure the filter to select only the year 2018. Next, use color gradients to help differentiate the denser locations on the map from the more scattered lightning strike locations. Drag number of strikes to the box labeled color. In the dropdown menu under marks, make sure that density is selected. Now that the map is much better, the default color for this visualization is blue. The lighter blues indicate more scattered lightning strikes and the darker blues are heavier lightning strike areas for the year of 2018. To change the color, click on the color square and change it from automatic to just something else. Try one or two. Consider which will make the map most accessible for yourself and for people who might have visual disabilities. Next, let's create a heat map. As discussed in a previous video, a heat map is a type of data visualization that depicts the magnitude of an instance or set of values based on two colors. To create a heat map in Tableau Public, you need one or more dimensions or one or two measures. To start, drag date to the row field and select year. Next, you need to create a calculation that derives the month from the date string. In the date dropdown menu, click create and select calculated field. A pop-up will open. Let's give the calculator a title. For the calculated field name input month, use the left function to pull the month names from the string in the column date. Pull the first three letters from the month listed in the string type left, parentheses, date name, parentheses, month, in single quotes, a comma, date and brackets, parentheses, a comma, and three. Then add a closing parentheses. Click OK. Now, there is a new field on the list of data on the left. Drag month to the columns field. Then drag number of strikes to the color square under the marks field. Be sure that in the marks dropdown, that square is selected. Now, you have a heat map. Just like the density map, the default color for this heat map is blue. You can adjust the heat map to a color range that fits your needs. Remember to consider accessibility as you create your visualizations. Now, let's create a box plot. You'll recall from a previous video that a box plot is a data visualization that depicts the locality, spread, and skew of groups of values within quartiles. You've already learned how to create a box plot chart in Python. To create a box button tab below public, first drag date to the row field. Then drag number of strikes to the columns field and select year. If tab below doesn't default to box plot, select circle from the dropdown menu under the marks field and scroll to the show me tab to select box and whisker plot. What you have so far is not much to look at. All you're seeing is a thin line of data points in a blank field of white background. However, once you drag date into the detail square under the marks field and select day, you'll see box plots appear in full strength. With box plots, most data pros will include a legend or annotation of the exact numbers of median and mean for presentation. You can, of course, change the colors and size of the circle in this box plot using the color and size squares under the marks field. Our last complex plot is a histogram. A histogram is a data visualization that depicts an approximate representation of the distribution of values in a data set. To create a histogram, first, you need to create bins. A bin is a tab below term that describes the custom segments of data that values can be grouped into. Bins are an important part of histograms and the groupings in heat maps determine how the data is segmented and compared. For a histogram of lightning strike data, group the total number of lightning strikes into bins. Do this by clicking on the number of strikes drop down. Select create and then bins. You can give the bin a name and then determine the size. If you select 50 for the size of bins, there would be one bin for every 50 lightning strikes. Any value between one and 50 would be included in that bin. Because there are millions of values of lightning strikes, you should select a number between 5 and 10. This will display the distribution of values in much more detail. Next, drag the new bin to the columns field. Then drag number of strikes to the row field. Make sure the number of strikes in rows is the count or C and T rather than sum or average. This presents the actual count of each bin. To give us more detail, drag number of strikes to the filter field and limit the field numbers between 1 and 200. Drag number of strikes to the label square under the marks field. Make sure it is set to count. You can adjust the color to make the data visualization more accessible. Coming up next, you'll learn how to create a series of data visualizations to use in a presentation. I'll meet you there. Hi, my name is Drew and I am a predictive modeling specialist here at Google. So originally I actually studied to be an accountant. I worked as an accountant for about 7 years. Most of what I did was working in finance. So most people who work in finance become incredibly good at Excel. And so I became better and better and better at Excel all the time and started working with different formulas. Formulas led me to figure out coding and then coding kind of led me towards data analytics. Exploratory data analysis is everything. You can't do anything without understanding the data first. You need to first go in and you need to really understand what every single variable is, what the distributions look like, what the initial correlations look like. You could start to go deeper than that building explanatory models to start to look at longer correlations and multi-correlations between multiple variables together. It's really it's everything. That is data science. I can remember a project we were working with a client who is spending tons of money on takeover media. Takeover media requires a large investment and it basically blocks out all other media spend from other people during a day so that you're concentrated on that day. What we found is that there wasn't a reaction between takeover media and conversions on the other side. What we had to do when framing it to the client is first figure out what are all the potential objections the client might have. Once we got those inputs we were able to put together a story that wasn't so destructive to what they're doing currently but was constructive and showed them the way forward. When delivering news about data it's important to shift from a data mindset to a client mindset. You need to marry the ideas that you learned during your data analysis to how it can actually be used by the client. A lot of the ways that we do this is through visuals. We try and incorporate visuals that are very simple yet communicative. We use humor. Humor is a great way of engaging people and it makes it very memorable for people. You use personal stories. Personal stories help get your point across to others very quickly. I once had a professor who told me that you have 30 seconds to actually communicate with someone until they stop paying attention so you need to be able to get in there make sure they remember it know what the takeaways are and have various succinct next steps of what to do with the data that you have presented to them. Data professionals often need to present their data. Creating, sharing and discussing data visualizations is an important part of telling data stories. In this video you will create a series of data visualizations that work together to tell a story. This will allow you to lead an audience from one concept to the next and build additional context as you present each visualization. But first, let's consider a scenario. Imagine you're working for an organization that wants to learn more about lightning strikes in the United States. They request a series of data visualizations that illustrate the increase in lightning strikes over time and detail lightning strike data in three states. Texas, Oklahoma and Kansas. After you create your visualizations you will need to share them with the organization's directors. With the audience in mind you can get started. You need to consider three strategies for organizing and presenting data visualizations in a series. Chronological generic to specific and specific to generic. A chronological approach to data visualizations is useful for data that is best understood in a time series. A generic to specific approach helps an audience consider an issue before describing how it affects them. And a specific to generic approach is useful to highlight impacts the data can have on a broader scale. Let's select an approach. A chronological approach or time series would be most helpful to consider data over time. This does not address an important aspect of the request. Visualizing strike data in three specific states. With a specific to generic approach the climax of the presentation would focus on the United States as a whole rather than lightning strikes in Texas, Oklahoma and Kansas. This does not meet the needs of the organization. A generic to specific approach would allow you to illustrate the country wide increase in lightning strikes followed by specific data on each state. You can highlight national trend before targeting the lightning strike data in Texas, Oklahoma and Kansas. This meets the organization's needs so a generic to specific approach is the best choice. Before you design your visualizations you need to create three different Tableau public worksheets. These will form a story in Tableau public. A story is a Tableau term for a group of dashboards or worksheets built into a presentation. For the first worksheet you will create a line graph showing an upward trend in lightning strikes over three decades starting with 2009. For the second worksheet you will show that the greatest amounts of lightning strikes have shifted from the east coast of the United States to the south central mainland over three decades. For the third worksheet you will show the number of lightning strikes in Texas Oklahoma and Kansas in 2018. These three sheets will form the generic to specific organization for the story. Let's open Tableau public and start designing. As always start by uploading your data source. For the column field drag in date which is a discrete dimension and select year for it. And for the row add the number of strikes measure into that field. A line graph should appear. If it doesn't open the show me tab and find the lines discrete selection. You'll notice the line moves steadily upward from 2009. Next we will make the data visualization easier to interpret. For example add two annotations to the line graph. First add 30.1m strikes in 2009 to the beginning of the graph to give the audience a clear starting point. Then add 45 m strikes in 2018 to its applicable position. Next let's create a geographic data visualization. Since we want to progress from generic to specific it is important to show the trend of lightning strikes moving west over the decades. To do that we need to create a time series map that shows the location of lightning strikes in the US for each year since 2009. In the new worksheet place the x coordinate in the column field and the y coordinate in the row field. This will create a map focused on the eastern and central US. If you're still seeing a line chart make sure that both your x and y coordinates are marked as continuous. Once you have the geographic map click on the drop down for both the x and y coordinates and select measure then average. This will give us a manageable view of the average number of lightning strikes in the US for each year. Next add the date and the discrete dimension to the filters and the pages fields. Select year for the filter method. The filter field allows us to create a dynamic filter which can be used to select just one year of lightning strikes. The pages field can create a distinct snapshot or page of lightning strikes for each year. Finally add the number of strikes to the detail field. The map showing lightning strike locations is very responsive. We just need to add the interactive part to it. With the year dimension in the pages and filters fields you can format interactive legends to the side of the data visualization. These allow users to self-select the years they wish to see. We have two of the three worksheets done that we need for our presentation. The last thing to create is the most specific visualization 2018 lightning strikes for Texas, Oklahoma and Kansas. To do this we'll create three new worksheets one for each state. Then we'll place the three snapshots into one dashboards. We'll work together on the Texas visualization. Then you can create the Oklahoma and Kansas visualizations all on your own. Our map of lightning strikes in the state of Texas starts in almost the same way as the geographic chart we just made. We put the X coordinate in the column field and the Y coordinate in the row field. We put the number of strikes measure in the detail field. This time we add number of strikes to the color field and make the color coincide with the number of strikes. Use red for most strikes and yellows for fewest. Because we will limit these state snapshots to just 18, we drop the date into the filters field, select year, then uncheck all the years except 2018. Finally, we need to create a set in Tableau Public. A set is a Tableau term for a custom field of data created from a larger data set based on custom conditions. We can create a set by first selecting all the data points in Texas. To do this we will use the lasso tool. You will follow the state line as best as you can all the way around Texas. Once complete, you will find a popup window. Select keep only, then click create set. Drag your newly created set to the filter field. Now, only the average lightning strikes for Texas will show up on our map. We can now assemble them into one story. First, add the two other state worksheets for Oklahoma and Kansas which were created in the same way as the Texas worksheet. Then create a dashboard for the three states illustrated in the locations of lightning strikes in the U.S. for the year 2018. You should also create dashboards for the other two worksheets created earlier the line chart and the interactive map of the U.S. Now that we have three dashboards we can create a story in Tableau. On the first page of the story insert the dashboard with the line chart. Fill in the caption to give details about what the chart is showing. On pages two and three of the story, put the interactive map of the U.S. followed by the dashboard that includes all three states, Texas, Oklahoma and Kansas. With the three captions filled in, we have a complete story. Coming up, you'll learn how to create an interactive dashboard in Tableau. I'll meet you soon. So far in this program, you've been practicing essential Tableau skills. Now, you will learn how to put these skills together and create an interactive dashboard. The interactive dashboard you create will display the quantity and the location of all lightning strikes in the United States from 2009 to 2018. The dashboard will be color coded based on the quantity of lightning strikes and it will allow viewers to select any year or location to learn the count of the strikes. Let's get started. Begin by opening Tableau public and uploading the data source provided. To build the dashboard, you will need to create three different worksheets. First, you'll create an interactive U.S. map with locations of lightning strikes for each year. Then you'll create a dynamic stat for each year. Finally, you'll design a way to pull and display the lightning strike metrics for the individual locations on each year's map. Let's start with the interactive lightning strike map. To begin, add the X coordinate and the Y coordinate in the column in the row field. Next, drag date into the filters field and select year. Make sure the show filter option is checked. For the last part of this map, drag number of strikes into both the color square and detail square in the marks field. The color square will categorize the dots on the map with a color code based on the quantity of strikes. The detail square will plot the locations for each of the lightning strikes for each year. Make sure when you do this that both number of strikes variables are set to continuous. For the next worksheet, you will create a list of three metrics that will change on the dashboard whenever a year is selected. The three metrics are total number of strikes, average number of strikes for each location for that year, and maximum number of strikes in any location. This worksheet will differ from our other visualizations. It will be text rather than a bar chart or line graph. It will sit to the side of the interactive map and update whenever a new year is selected. First, create a calculation from the number of strikes dimension. In the blank field space, type number of strikes and put it in brackets. Next, drag the new calculation into the tool tip square in the marks field. In the drop down select measure and average. Repeat this process, but instead of selecting average you should select sum and maximum. This creates dynamic fields for three metrics. The total number of strikes, the average number of strikes for each location for that year and the maximum number of strikes in any location. Finally, create a place to display these calculations. To keep it simple go to worksheet and select show title. Input the title, metrics for the year. Under that you will include the three calculations in the tool tip. To do this, put an asterisk followed by total colon and then sum and number of strikes inside the left and right angle brackets. Repeat this step for the maximum and average tool tips you created. And there you have it. For the last worksheet, drag number of strikes to the text square in the marks field. In the drop down, select measure and average. This creates a dynamic field for the dashboard we'll create next. Now, select new dashboard. You should find all your worksheets in the list on this page. Drag over the worksheet with the interactive map in it. It may take a few seconds to load. Next, drag your other two worksheets into the dashboard. Before we clean these up, we need to create an action for the dashboard. An action is a tab load tool to help a user interact with a visualization or dashboard by allowing control of a selection. When a year is selected from the filter bar, all the other fields will update with it. We will connect the three worksheets together with an action. Click on the map in the dashboard. Select worksheet and actions. In the actions pop up, select add action and filter. In the add filter action menu in the source worksheet section, select your dashboard from the drop down. Check the box for your map. And for the run action on list, click select. For target sheets, select your dashboard from the drop down. Select the map and the recently completed worksheet that will gather metrics for individual locations. From the list of clearing the selection will, select show the values. Lastly, select all fields for the filter section. Click OK. And now you have an interactive dashboard. Update the text and titles before you're done. Add a title to the filter section. You should also title the location average variable, location metrics and write brief instructions to clarify what the tool does. For example, you might type choose a point on the map to view location specific metrics. Take a few minutes to test your dashboard. Select different years and locations on the map to make sure the tools function properly. What a collection of tools and techniques you've learned in this video will help you communicate and present ideas more effectively throughout your career as a data professional. Good luck and happy designing. If you were to design a data visualization of what you've learned in this program, what might it look like? Would you plot your engagement level over time on a line graph? Or would you create a heat map showing the tasks you completed with different colors representing the tasks that took the least and most time? No matter what visualization you choose you've learned the skills you need to create an effective graphic or visualization that an audience can understand. You've learned to use Tableau Public to tell your data stories and to adjust the content depending on the audience. We also discussed the best ways to share technical concepts with non-technical audiences. You've learned to use Tableau at a high level and you've applied the technique data professionals use to create visualizations. No data visualization discussion would be complete without considering ethics and accessibility. You've learned that as a data professional it is your responsibility to ensure visualizations accurately represent the data. You also learned how to make data visualizations accessible for people who have visual impairments. You learned a lot. Congratulations on completing this section and good luck on the rest of your journey. Hi, I'm Tiffany and it's great to be with you again. You've made a lot of progress in the program so I'm back to tell you more about your portfolio projects and how you can use them in your future job search. Remember, your portfolio will be a collection of materials you can use to showcase your approach to solving data-oriented problems. In the portfolio project for this course you will demonstrate your knowledge of telling stories using data. This is an incredibly important skill for a data professional to know on the job but it's also critical for success in an interview. As potential employers assess you as a candidate they might ask you for specific examples of how you approach cleaning, structuring and validating data in the past. You can use your portfolio as a way to discuss actual data challenges you have resolved and real stories you have presented. Additionally, some employers might ask you to create a presentation based on what they provide. The skills you've learned in Python and Tableau to create data visualizations will help you feel more comfortable and prepared for those interviews as well as build out your portfolio. You already learned about experiential learning which is when people gain understanding through doing. Watching an instructor create a visualization is one thing. Creating a visualization yourself to expand your understanding of the concepts and improve your skills at presenting to the project is also a great opportunity to discover how organizations are using data analytics every day and show off your knowledge of how to tell data driven stories. To complete the portfolio project you'll be working with a database and an accompanying business scenario. You'll use instructions to complete a Jupyter notebook showing your EDA work and three to five visualizations in Tableau in response to the scenario. By the time you complete this project you can add to your data professional portfolio. In your Pay Strategy document you'll also have documentation of the steps you took along the way which you can use to explain your work to future hiring managers.  Now it's time to demonstrate what you've learned. This portfolio project will help you practice and demonstrate the skills you've learned throughout the course. For example, you'll be able to show how to form databases of EDA in Python. Using Tableau you'll demonstrate how to create data visualizations that accurately detail a data set story. And you'll be able to demonstrate how to prepare and document a comprehensive workflow strategy using PACE. Ready? Let's go. In this course you've been learning about using the six steps of the EDA process to tell stories with data. Now it's time for an exciting next step. Putting all this to work for your portfolio project. In the previous course you learned the basic formatting and structure for writing code in Python. These coding skills will be critical to the completion of your next portfolio project which will require you to clean, analyze and present data to technical and non-technical audiences. Now that you have some practice completing portfolio projects think about how to reframe the data you're analyzing and cleaning it into a well thought out story. In this part of the course you'll take a data set and apply the six steps of EDA to formulate a useful document that can help you present key findings to stakeholders. In other sections of this program you will work to develop additional skills to help you thrive in the data career space. There's so much more to learn about telling stories with data. As a data professional a large part of your job is focused on transforming messy data into an organized, clear story that meets business goals and helps stakeholders understand important details needed for making business decisions. This portfolio project is a great opportunity to demonstrate to potential employers that you can do exactly that turn messy data into a logical story. And remember developing your skills as a data professional is an iterative process so you can continue to improve as you have new ideas or learn new things. At this point in the program you've done a lot of work toward better understanding data and how it can be useful for enacting change in a business. You completed a Jupiter notebook created visualizations to support your work and refined your presentation to meet the needs of your particular audience. As you continue to make progress in this program remember that documenting your learning process and skills will help you communicate what you've done to potential employers and hiring managers in future interviews. You may recall from previous sections of this course that audience awareness is essential. During the interview process knowing how to talk about your work process transferable skills and other achievements will lead to much greater success. In this course you learn the importance of following the pace structure in a data career. You practice using Python to manipulate data and you demonstrated how to organize and analyze a set of data to tell a compelling story. The portfolio projects were designed to help you thrive on the job market and the transferable skills you applied contributed to the tangible artifacts you created. As you begin preparing for future interviews you should be ready to answer questions like what is your process for cleaning data? What tool do you use for creating data visualizations? How and why do data visualizations enhance the stories data tells? And what considerations are top of mind when sharing data stories with non-technical stakeholders? Of course there may be many other questions as you are asked as you interview for a data professional role. Each portfolio project will help you prepare responses. For example in the portfolio project you just completed you use the EDA process to clean organize and analyze a data set. Then you turn your data into a presentation full of visualizations that will help stakeholders understand insights from the data story you discovered. Don't forget that you recorded all of your considerations questions, process notes and more in your pay strategy document as well. Coming up you are going to learn all about the power of statistics and data driven work. Then you will have an opportunity to use statistical analysis to simulate an A-B test. By the end of these courses you will have lots of artifacts in your portfolio.