s that are available to it. One of the most widely used and important of these is NumPy. Recall that NumPy contains multi-dimensional array and matrix data structures, as well as functions to manipulate them. You'll learn more about these data structures and functions soon, but for now, let's just consider NumPy at a high level and learn more about it. NumPy's power comes from vectorization. Vectorization enables operations to be performed on multiple components of a data object at the same time. This is particularly useful for data professionals, because their jobs often involve working with very large quantities of data, and vectorized code saves a lot of time, because it computes more efficiently. Let's explore this a little more. Suppose we have list A and list B, both the same length, and we want to create a new list, C, that's the element-wise product of each list. In other words, we want to multiply the first element of list A by the first element of list B, then multiply the second element of list A by the second element of list B, etc. If we try to multiply list A and B together, the computer throws an error. To perform this operation, we could write a for loop. We'd start by defining an empty list, list C, then make a range of indices to loop over, and depend the product of list A and list B at each index to list C. This gets the job done, but if you're thinking there's got to be an easier way, you're right. We can use NumPy to perform this operation as a vectorized computation. Simply convert each list to a NumPy array, and multiply the two arrays together using the product operator. The results are the same, but the vectorized approach is simpler, easier to read, and faster to execute, because while loops iterate over one element at a time, vector operations compute simultaneously in a single statement. The efficiency of this might not be noticeable now, but when working with large datasets it will be. Vectors also take up less memory space, which is another factor that becomes important when working with a lot of data. You might have noticed that when we use NumPy, we first had to import it. This is called an import statement. An import statement uses the import keyword to load an external library, package, module, or function into your computing environment. Once you import something into your notebook and run that cell, you don't need to import it again unless you restart your notebook. When we import NumPy, we import it as NP. This is known as aliasing. Aliasing lets you assign an alternate name or alias by which you can refer to something. In this case, we're abbreviating NumPy to NP. Notice the NP's in the code below the import statement where we create our arrays. If we didn't give NumPy an alias of NP, we'd have to type out NumPy here in order to access its array function. Aliasing as NP makes the code shorter and easier to read. Note that NP is the standard alias. If you use something else, other people might get confused when reading your code. In addition to being highly useful in its own right, NumPy powers a lot of other Python libraries, like pandas. So understanding how NumPy works will help you use these other packages. There's a lot more to discover about NumPy. Coming up, you'll learn about its core data structures and functionalities. See you soon. Welcome back. You recently learned that the NumPy library uses vectorization to make working with data faster and more efficient, which makes your job easier. I demonstrated how NumPy performed an element-wise multiplication of two lists by converting the lists to arrays and then simply multiplying them together. Now we're going to continue learning about arrays and how to work with them. The array is the core data structure of NumPy. The data object itself is known as an n-dimensional array, or nd array for short. The nd array is a vector. Recall that vectors enable many operations to be performed together when the code is executed, resulting in faster run times that require less computer memory. You can create an nd array from a Python object by passing the object to the nd array function. nd arrays are mutable, so you can change the values they contain. If I want to change this last value from a 4 to a 5, I can do that by identifying the index number. Since we're dealing with the last value here, it's necessary to use a negative one. But I can't change the size of the array without reassigning it. If I try to add a number to the end of this array, the computer throws an error. So if you want to change the size of an array, you have to reassign it. Another requirement of the array is that all of its elements be of the same data type. If I create an array with the integers 1, 2, and then a string of coconut, NumPy will create an array that forces everything to the same data type, if possible. In this case, everything becomes a string represented here by U21, meaning Unicode21. So be careful when creating your arrays that they all contain data of the same type, or if they don't, that this is intentional and useful to what you're doing. You previously learned that calling the type function on an object will return the data type of the object. If we do that with an array, as you might expect, we get a NumPy array. We can use the dtype attribute if we want to check the data type of the contents of an array. Here, the dtype attribute indicates that this array consists of integers. As the name implies, nd arrays can be multidimensional. For a one-dimensional array, NumPy takes an array-like object of length x, like a list, and creates an nd array in the shape of x. A one-dimensional array is neither a row nor a column. We can use the shape attribute to confirm the shape of an array. We can also use the ndim attribute to confirm the number of dimensions the array has. Data professionals will often need to confirm the shape and number of dimensions of their array. For example, if they're trying to attach it to another existing array. These methods are also commonly used to help understand what's going wrong when your code throws an error. A two-dimensional array can be created from a list of lists, where each internal list is the same length. You can think of these internal lists as individual rows, so the final array is like a table. Notice that this array has a shape of four rows by two columns and is two dimensions. If a two-dimensional array is a list of lists, then a three-dimensional array is a list that contains two of these, so a list of two lists of lists. This array can be thought of as two tables, each with two rows and three columns. Thus, it has three dimensions. This can go on indefinitely. Thankfully, there are ways to help simplify working with multi-dimensional arrays, which you'll learn about later. And unless you're doing very advanced scientific computations, you typically won't be working directly with NumPy arrays that are more than three dimensions. NumPy also lets us reshape an array, using the reshape method. Our two-dimensional array was four rows and two columns. But what if we wanted this data to be two rows by four columns? We just plug these values into the reshape method and reassign the result back to the array 2D variable. Reshaping data is a common task in data analysis, so it's important for you to be familiar with what it means and how it works. There are many other operations that can be performed with arrays, and you'll surely learn those as the need for them arises in your projects. But there are other helpful functions and methods in NumPy that you'll use regularly. These include functions to calculate things like mean and natural logarithm and floor and ceiling operations, which round numbers to the nearest, lesser and greater whole number, respectively. And many other frequently used mathematical and statistical operations. NumPy is very robust. There are so many things you can do with it that we can only briefly consider them here. As you know, NumPy powers many other useful libraries and packages. In this certificate program, we won't be working a lot directly with NumPy, but we will be working a lot with the library that depends on it, pandas. It's important that you understand the basics of NumPy, because it will help you when you start working with pandas. As you develop your skills as a data professional, you'll find yourself returning to NumPy time and time again, because it's such an integral part of advanced data analysis. Bye for now. Hi there. We've previously discussed NumPy and how it's an important tool for data professionals and anyone else whose job requires high-performance computational power. We've investigated how other libraries and packages use NumPy because of the efficiencies that come with vectorization. One of these libraries is pandas, a quintessential tool both in this certificate program and in the world of data analytics. In this lesson, you're going to learn more about pandas and why it's so useful. Because pandas is a library that adds functionality to Python's core toolset, you have to import it. When we imported NumPy as NP, pandas has its own standard alias of PD. Typically, when using pandas, you import both NumPy and pandas together. This is just for convenience, given that numpy is often used in conjunction with pandas. Strictly speaking, you don't have to import numpy to work in pandas. Pandas is fully operational on its own. Pandas' key functionality is the manipulation and analysis of tabular data. That is, data that's in the form of a table with rows and columns. A spreadsheet is a common example of tabular data. While numpy is capable of many of the same functions and operations as pandas, it's not always as easy to work with because it requires you to work more abstractly with the data and keep track of what's being done to it even if you can't see it. Pandas, on the other hand, provides a simple interface that allows you to display your data as rows and columns. This means that you can always follow exactly what's happening to your data as you manipulate it. In this video, I'll give you a demonstration of pandas and what it's like to use it. Later, we'll go into greater detail on its unique classes, processes and functions. First of all, you can load data into pandas easily from different formats like separated value files, or CSVs, Excel, and other spreadsheets, databases, and more. Here, I'm loading a CSV file that I'm accessing via a web URL. The file contains information for some of the passengers from the Titanic, including their names, what class ticket they had, their age, ticket price, and cabin number. By the way, this table of data is called a data frame. The data frame is a core data structure in pandas. Notice that the data frame is made up of rows and columns, and it can contain data of many different data types, including integers, floats, strings, bullions, and more. If I want to calculate the average age of the passengers, we do so by selecting the age column and calling the mean method on it. I can also get the max, min, and standard deviation with minimal effort. I can also quickly check how many passengers were in each class. Checking summary statistics of the entire dataset only requires one line of code. This method gives me the number of rows, as well as the mean, standard deviation, minimum, and maximum values, along with the core tiles for every numeric column. These concepts are all covered in greater depth elsewhere in the program. I just want you to pay close attention to the power of pandas and all that you can accomplish with it. Pandas also allows me to filter based on simple or complex logic. For example, here I'm selecting only the third class passengers who were older than 60. In addition to all of these data analysis tools, pandas also gives us ways to manipulate and change the data. For example, I can add a column that represents the inflation adjusted price of a ticket from 1912 to 2023. Florence Briggs-Feyer paid 71.28 pounds for her first class ticket. Today, that ticket would have cost her 10,417 pounds sterling. If you're wondering how I knew her name was Florence Briggs-Feyer, it's because I can also select rows, columns, or individual cells from the data using indexing. Her name is in row 1, column 3. I can also do more complex data groupings and aggregations. For example, here I'm grouping the passengers by class and sex and then calculating the mean cost of a ticket for each group. Hopefully you're excited to start working with pandas. I know I'm looking forward to guiding you as you learn more about this powerful and fun data analysis tool. Now that you have a good understanding of the core structures and routines of Python and some of the basics of NumPy, you're ready to start working with pandas. Pandas is one of the primary tools that you'll use throughout the rest of the certificate program, as well as in a large and growing number of data professions. In this video, you'll learn about the main classes in the pandas library and some important ways to work with them. Pandas has two core object classes, data frames and series. Let's begin with the review of data frames. A data frame is a two-dimensional labeled data structure with rows and columns. You can think of a data frame like a spreadsheet or a sequel table. It can contain many different kinds of data. Data professionals use data frames to structure, manipulate and analyze data in pandas. Just like we did in the previous video with the Titanic example. We can create a data frame using the pandas data frame function. This function has a lot of flexibility and can convert numerous data formats to a data frame object. In this example, we created a data frame from a dictionary where each key of the dictionary represents a column name and the values for that key are in a list. Each element in the list represents a value for a different row at that column. We can also create one from a NumPy array resembling a list of lists where each sub-list represents a row of the table. Notice that in this example we included separate keyword arguments for columns and index. This approach lets us name the columns and rows of the data frame. These are just a couple of the many different ways to create a data frame with the data frame function. For examples of some others, be sure to review the available pandas documentation on this topic. Often, data professionals need to be able to create a data frame from existing data that's not written in Python syntax. For example, maybe we want to take an existing spreadsheet and manipulate it in pandas. Spreadsheets can be saved as CSV files which can then be read into pandas as a data frame. CSV stands for comma separated values and it refers to a plain text file that uses commas to separate distinct values from one another. Here is a sample of the first few lines of a source data from the Titanic data set that we used previously. This is what a CSV file looks like. In this file, you'll find values for passenger name, age, sex, fare, and more. Notice that a comma is used to separate each value from the next. To create a data frame from a CSV file, pandas has the read CSV function. Here's the same Titanic data rendered as a data frame. For the sake of an example, it's defined here as df3. The read CSV function can read files from a URL, like in this example, and it can also read files directly from your hard drive. Instead of a URL, you just provide the file path to your file. Now, let's discuss the other main class in pandas, series. A series is a one-dimensional labeled array. Series objects are most often used to represent individual columns or rows of a data frame. So, if we select a row or a column from this Titanic data frame file type on it, it will return as a pandas series object. Like data frames, individual series can be created from various data objects, including from NumPy arrays, dictionaries, and even scholars. Again, refer to the pandas documentation for examples. Now, let's use the Titanic data set to review some of the basics of working with data frames in series. The data frame in series classes have many super useful methods and attributes that make common tasks easier. Remember, a method is a function that belongs to a class. It performs an action on the object. An attribute is a value associated with a class instance. It typically represents a characteristic of the instance. Both methods and attributes are accessed using dot notation, but methods use parentheses, while attributes do not. Earlier in the video, we named the Titanic data set df3, but let's change the name to Titanic for clarity. We can do this by simple reassignment. If we want to access the columns of the data frame, we can use the columns attribute. This returns an index of all of the column names. We can use the shape attribute to check the number of rows and columns contained in the data frame. This data frame has 891 rows and 12 columns. And we can get some summary information about the data frame by calling the info method. This tells us that there are 891 rows and 12 columns and it also gives us the column names, the data type contained in each column, the number of non null values in each column, and the amount of memory the data frame uses. By the way, I want to address a couple of points about terminology in Pandas. First, null values and Pandas are represented by NAN, which stands for not a number. And second, if a series object contains mixed or string data types, when you check its data type, it will come back as an object. This is an example of how Pandas is built on NumPy, but the details of this are beyond the scope of this video. One of the most common tasks when working in Pandas is selecting or referencing parts of a data frame. This has many similarities with indexing and slicing. For example, if you want to select a single column, you can type the name of the data frame, followed by brackets, and within the brackets enter the name of the column as a string. This returns a series object of that column. You can also use dot notation, but this only works if the column name does not contain any white spaces. Using dot notation is faster to type, so for very simple lines of code, you may prefer to do this, but if the code begins to get more complex, it's generally better to use bracket notation because it makes the code easier to read. To select multiple columns of a data frame by name, use bracket notation. Within the brackets, enter a list of column names. This returns a view of your data frame as a new data frame object. If you want to select rows or columns by index, you'll need to use iLoc. iLoc is a way to indicate in Pandas that you want to select by integer location based condition. If you enter a single integer into the iLoc brackets, you'll get a series object representing a single row of your data frame at that index. Because I entered zero here, I got the very first row in my data frame as a series. If you enter a list of a single integer in the iLoc brackets, you'll get a data frame object of a single row of the data frame at that index. You can access a range of rows by entering the indices of the beginning and ending rows separated by a colon. Pandas will return every index starting with the beginning index up to but not including the last index. So, 0 colon 3 returns row indices 0, 1, and 2. You can select subsets of rows and columns together too. This returns a data frame view of rows 0, 1, and 2 at columns 3 and 4 only. So, if you want a single column in its entirety, you select all rows and then enter the index of the column you want. And you can even get a single value at a particular row in a particular column by using two indices separated by a comma. This is similar to iLoc, but instead of selecting by index location, loc is used to select Pandas rows and columns by name. Let's investigate loc with the Titanic data frame. In this example, I'm selecting rows 1, 2, and 3 at just the name column. Note that in this example, we're referring to the rows with numbers even though we're using loc to select. This is because our rows are indexed by number. If we had a named index, however, we'd have to use row names, like what we're doing for columns. And one more thing, if you want to add a new column to a data frame, you can do that with a simple assignment statement. Now, we have a new column at the end here. There are so many things you can do in Pandas, and I can only share so much in the time we have. In this case, the documentation is your friend. There will inevitably be times where you need to do something that wasn't explicitly covered here. In those cases, the documentation almost always has simple examples that demonstrate how to do the thing you need to do. There's still more to come though, so I'll meet you soon. Previously, we investigated several different ways of selecting data in a data frame, like selection, row selection, and selection of combinations of rows and columns using name-based and integer-based indexing. In this video, you'll learn how to filter the data in a data frame based on value-based conditions. You know that Boolean is used to describe any binary variable with two possible values, true or false. With Pandas, you can use a powerful technique known as Boolean masking. Boolean masking is a filtering technique that overlays a Boolean grid onto a data frame in order to select only the values in the data frame that align with the true values of the grid. Data professionals use Boolean masking all the time, so it's important that you understand how it works. Here's an example. Suppose you have a data frame of planets, the radii, and the number of moons each planet has. Now suppose that you want to keep the rows of any planets that have fewer than 20 moons and filter out the rest. A Boolean mask is a Pandas series object indicating whether this condition is true or false for each value in the moons column. The data contained in this series is type BOOL. Boolean masking effectively overlays this Boolean series onto the data frames index. The result is that any rows in the data frame that are indicated as true in the Boolean mask remain in the data frame, and any rows that are indicated as false get filtered out. Here's how to perform this operation in Pandas. We'll begin by creating a data frame from a pre-defined dictionary using the Pandas data frame function. The data frame is called planets. The next step is to create the Boolean mask by writing a logical statement. Remember, the objective is to keep planets that have fewer than 20 moons and to filter out the rest, so we define the mask by writing planets at the moons column is less than 20. This results in a series object that consists of the row and disease where each index contains a true or false value, depending on whether that row satisfies the given condition. This is the Boolean mask. To apply this mask to the data frame insert it into selector brackets and apply it to the data frame. It's also possible to apply the conditional logic directly to the data frame, skipping the part where we assign it to a variable named mask, but breaking out the steps individually can make the code easier to follow. Note that we haven't permanently modified the data frame. Applying the Boolean mask using the conditional logic only gives a filtered view of it, so when you call the planets variable again it returns the full data frame. However, we can assign the result to a named variable. This may be useful if you'll need to reference the list of planets with moons under 20 again later. Sometimes you'll need to filter data based on multiple conditions. Pandas uses logical operators to indicate which data to keep and which to filter out in statements that use multiple conditions. These operators are the ampersand for and, the vertical bar for or, and the tilde for not. Let's review how this works. Here's how to create a Boolean mask that selects all planets that have fewer than 10 moons or greater than 50 moons. Notice that each condition is self-contained and a set of parentheses and the two conditions are separated by a vertical bar, which is the logical operator that represents or. It's very important that each component of the logical statement be in the parentheses. Otherwise, the statement will throw an error, or worse, return something that isn't what you intended. To apply the mask, call the data frame and put the statement or the variable it's assigned to in selector brackets. Here's an example of how to select all planets that have more than 20 moons, but not planets with 80 moons and not planets with a radius less than 50,000 kilometers. Let's break it into pieces. First is the statement for planets. In parentheses, we put planets at the moons column must be greater than 20 and close the parentheses. Then we use the and not operators before parentheses. Planets at the moons column equals 80, close parentheses. Then we again use the and not operators before parentheses. Planets at the radius column less than 50,000, close parentheses. When we apply the mask to the data frame, we're left with just one planet, Saturn with 83 moons and a radius of 58,232 kilometers. There are a near infinite number of ways to select and filter data using the basic tools that you've learned so far. As always it takes a lot of practice before you know exactly how to execute every selection statement that your work requires. So make sure to bookmark any and all resources that you find helpful to reference. Keep up the good work and I'll meet you in the next video. Now that you've learned how to select and filter data using name and location based indexing, as well as Boolean masking, it's time to take the next step. In this video, you'll learn how to group your data, aggregate it and perform calculations on these groupings to help you discover what the data is telling you. One of the most important and commonly used tools to group data in pandas is the group by method. Group by is a pandas data frame method that groups rows of the data frame together based on their values at one or more columns, which allows further analysis of the groups. Let's return to our planets data set to demonstrate some different ways to use the group by method. This time we'll add a little more data including the type of planet whether or not it has a ring system average temperature in degrees Celsius and whether or not it has a global magnetic field. As always, when learning a new data tool, it's helpful to begin by applying it to a small example. This will better enable you to understand exactly what's happening. First, let's examine the mechanics of what happens when you use group by. When you call the group by method on a data frame, it creates a group by object. If you do nothing else, the group by object isn't very helpful. You'll basically get a statement saying, here's your object. It's stored at this address in the computer's memory. But once you have this object, there are all kinds of things you can do with it. For example, if we group the data frame by the type column and then apply the sum method to the group by object, the computer returns a data frame object with three columns, one for each planet type and three columns, one for each numerical column. Only the numerical columns are returned because the sum method only works on numerical data. The type column is an index of this data frame. This information can be interpreted as the sum of all the values in each group at these respective columns. So for example, radii of all the gas giant planets, sum to 128,143 kilometers. That information probably isn't very useful in those cases, but the total number of moons could definitely be something we want to calculate. If you want to isolate the information at particular columns, just insert the columns as a list in selector brackets following the group by statement. You can also use other methods instead of sum. For example, min, max, mean, median, count and many others. Group by will work on multiple columns too. When we pass a list containing the type and magnetic field columns to the group by method and then apply the mean method to the result, we get a data frame that contains a row for each unique combination of planet type and magnetic field. Again, the columns contain the mean calculated values for each numerical column for each group. Group by is very useful because it helps you to better understand your data. It's also useful to organize data that you want to plot on a graph. You'll learn more about this later. Another important method to use on the group by objects is the ag method. Ag is short for aggregate. Ag method allows you to apply multiple calculations to groups of data. Let's start simple. What if we want to group the planets by their type and then calculate both the mean and the median values of the numeric columns for each group? We call the ag method after the group by statement. In its argument field, we enter a list of the calculations we want to apply to the data. If these calculations are existing methods of group by objects, they can just be entered as strings. We can group by multiple columns and apply multiple aggregation functions to each group. For instance, we can group the planets by type and whether or not they have a magnetic field, and then use the ag method to calculate the mean and max values of each group. And we can even define our own functions and apply them. For example, suppose we want to calculate the 90th percentile of each group. We can define a function called percentile 90 that uses the quantile method on the array and returns the value at the 90th percentile. Then we can call this custom function in our aggregation. Notice that we can enter mean as a string because it's an existing method of group by objects, but we type the percentile 90 function as an object because it's custom defined. Group by an aggregate are two tools that together can give deep insight to the story that your data is telling you. The types of calculations that we just viewed are daily tasks of data professionals in nearly every field. Even though we only applied them to a very tiny dataset, the same exact operations would work on a dataset of every planet in the galaxy. If we knew them all and if we had enough computing power to perform the aggregations. There's a lot more we can do with group by an aggregate and as always, I encourage you to explore more on your own, but you should now have a solid understanding of how and when to apply these tools. Hello again. You've learned a lot about pandas, that it's a powerful library that makes working with tabular data easier and more efficient. How to select an index in a data frame, how to filter data using Boolean masks and how to group and aggregate data to derive insights from the story it's telling. In this video, you'll learn how to add new data to existing data frames. This is a common task for data professionals, but it's not as simple as just adding two data frames together. There are important considerations to be aware of. By the end of this lesson, you'll have a good understanding of what these considerations are so you can make informed decisions about how best to add data to your project. We're going to learn about two pandas functions, can cat and merge. There's considerable overlap between the capabilities of these functions, but it's most important that you learn the basics of each because you will encounter them regularly as a data professional. We'll start with the can cat function. Recall that to concatenate means to link or join together. The pandas can cat function combines data either by adding it horizontally as new columns for existing rows or vertically as new rows for existing columns. It's also capable of handling many data specific complexities that arise, which allows for a high degree of user control. In this video, I'll demonstrate how to use the can cat function to add new rows to existing columns. But remember, there's plenty of support documentation if you'd like additional information. Pandas has a specific way to indicate which way we want the data to be concatenated. We do this by referring to axes. In fact, many pandas and numpy functions have an axes keyword, so you can specify whether you want to apply the function across rows or down columns. The two axes of a data frame are 0 which runs vertically over rows and 1 which runs horizontally across columns. We'll use our basic planets data set to demonstrate how can cat works. This data has four planets, their radii, and the number of moons, but it's missing the data for Jupiter, Saturn, Uranus, and Neptune. Now, suppose we want to add this data which exists as a separate data frame. Let's examine this second data set with information about Jupiter, Saturn, Uranus, and Neptune before joining them. Notice that this data is in the same format as the data in the DF1 data frame. It has the same columns for planet, radius, and moons. To combine the two data frames we'll want to add DF2 as new rows below DF1. To concatenate the first data set with information about Mercury, Venus, Earth, and Mars with the second which has information about Jupiter, Saturn, Uranus, and Neptune, we call PD-Concat and insert a list of the data frames we want to concatenate. Then, we need to include an axes keyword argument. This instructs the function to combine the data either side by side or one on top of the other. We want our resulting data frame to have eight rows and three columns which means we want to combine the data vertically. In other words, we want to add new data by extending axes zero, the vertical axes. Perfect! The data was added as new rows. Notice that each row retains its index number from its original data frame. If you want the numbering to restart, just reset the index. We can include the drop equals true argument because otherwise a new index column will be added to the data frame which we don't want in this case. Now, the enumeration of the row indices goes from zero to seven. The concat function is great for when you have data frames containing identically formatted data that simply needs to be combined vertically. If you want to add data horizontally, consider the merge function. The merge function is a pandas function that joins two data frames together. It only combines data by extending along axes one horizontally. Let's return to the planets. Now, we have the radius and number of moons for all eight planets. But suppose we want to add the data for the planet type, whether it has rings, its average temperature, whether it has a magnetic field and whether it has life on it. Perhaps this data exists as a separate data frame, but it's missing Mercury and Venus and it has some recently discovered planets from other star systems, Janssen and Tadmore. That's okay. We can still work with this. First, let's conceptualize how data joins work. For two data sets to connect, they need to share a common point of reference. In other words, both data sets must have some aspect of them that is the same in each one. These are known as keys. Keys are the shared points of reference between different data frames, what to match on. In our case, the keys are the planets. Each data frame contains planets for us to match on. Now let's consider the different ways that we can join this data. We can join it so only the keys that are in both data frames get included in the merge. This is known as an inner join. Alternatively, we can join the data so all of the keys from both data frames get included in the merge. This is known as an outer join. We can also join the data so all of the keys in the left data frame are included, even if they aren't in the right data frame. This is called a left join. Finally, we can join the data so all the keys in the right data frame are included, even if they aren't in the left data frame. This is called a right join. Let's examine how each type of join affects our planet data. We can use the function and enter df3 and df4 as the left and right positional arguments respectively. Then we include the keyword argument on which lets us specify what our keys to match on should be. In this case, we want to use the planet column. Now we have the how keyword argument. This is where we enter the kind of join we want. Let's try inner first. We only kept the planets that appeared in both data frames. This means we're missing data from Mercury and Venus from the left data frame, as well as for Janssen and Tadmore from the right data frame. Now, let's try an outer join. Our function call will remain the same, except for the how keyword argument, which we'll set to outer. As expected, this results in a data frame that contains all the keys from both initial data frames. Notice that because Janssen and Tadmore aren't in the left data frame, they don't have information for radius and moons. So these columns get filled with NaNs. Similarly, because Mercury and Venus aren't in the right data frame, they too are missing some information in the final table, which is represented by NaNs. Next, we'll do a left join. Again, the function gets the same syntax except for the how argument, which is set to left. This results in a data frame that retains all the keys from the left data frame and only the keys from the right data frame that exists in the left data frame too. So Janssen and Tadmore are excluded. Finally, we'll perform a right join. As expected, the result is a data frame that has all the keys from the right data frame, but none of the keys from the left that weren't also in the right. So Mercury and Venus are excluded. Nice job. Now that you know the fundamentals, you can use these pandas tools to do the most common kinds of data joins, which will be useful for a wide variety of data projects. And as you advance in your career, you'll discover even more about joining data and how it can get very complex. These tools will be a big help as you do. We're in a long way and we are now ready to start using pandas to explore your data like a true data professional. See you soon. This is the end of the fourth section of the Python course. You now have a strong foundation of Python skills that you can continue to build on throughout your future career as a data professional. In this section of the course, you learned how data professionals use data structures to store, access and organize data. Understanding which data structures fit your specific task is a key part of data work and will help you analyze your data with speed and efficiency. We reviewed fundamental data structures that are super useful for data professionals. Lists, tuples, dictionaries, sets and arrays. We also discussed two of the most widely used and important Python tools for advanced data analysis. The first was NumPy, which data professionals use for computational power. You learn how NumPy can help you rapidly process large amounts of data and perform useful calculations. The second Python tool you learned about was pandas, which is a powerful tool for analyzing tabular data. You learn how pandas can help you perform key tasks such as filtering, grouping and merging data. Data professionals often work with tabular data. You'll use pandas throughout the rest of the certificate program and your future career. Coming up, you have a greeted assessment. To prepare, review the reading that lists all the new terms you've learned and feel free to revisit videos, readings and other resources that cover key concepts. Congratulations on all your progress. Way to go. Hi, it's great to be with you again. You might recognize me from the last course. I'm Tiffany and I lead the responsible AI program management team here at Google. I'm back to talk more with you about your portfolio projects and how you can use them in your job search. Now that we've had some time to explore Python, I'm excited to help you work on a project that you can add to your professional portfolio. As we complete this segment of the program, you'll have the opportunity to begin showcasing your coding skills. This portfolio project is a really valuable opportunity to develop your interview skills. When potential employers assess you as a candidate, they might ask for specific examples of how you tackled coding challenges in the past. You could use your portfolio as a way to discuss a real problem you have solved. Additionally, some employers might ask you to load, clean and structure data during an interview to prove your proficiency. Getting some practice creating a database structure to address data-driven projects means you'll be prepared for that type of situation. You have already learned about experiential learning or the idea of understanding you're doing. This portfolio project is a great opportunity to really discover how organizations manage data with Python and practice the skills you've been learning in this course. To complete the portfolio project, you'll be presented with details about some business cases and some unstructured data files. Choose one business scenario and use the instructions to complete a new entry in your energy document. Based on the scenario, your task is to load, clean and structure the data so that your end product is a tidy data set. Data tidying is structuring data sets to facilitate analysis. Tidy data sets are easy to manipulate, model and visualize and have a specific structure. Each variable is a column, each observation is a row and each type of observational unit is a table. By the time you complete this project, you'll have a structure data set that you'll use in the next course portfolio project. In your pay strategy document, you'll also have documentation of the steps you took along the way, which you can use to explain your work and thought process to future hiring managers. At this point, you're almost finished with this course, which means you've learned everything you need to keep advancing as a data professional. This part of the project will focus on demonstrating mastery of data manipulation and understanding how data professionals use Python to explore and extract information through custom functions. Ready? Then let's get started. In this course, you've been learning about the advantages and simplicity of Python, as well as basic Python syntax, loops, strings, data structures and object oriented programming. Now it's time for an exciting next step, putting all this to work for your portfolio project. In the previous course, you learned about the flexibility of a data professional career and the ways communication has a direct impact on data driven work. You also practice thinking like a data professional, as you assess a business scenario and recorded project considerations in your pay strategy document. These skills will also be applicable to this new project. In this part of the course, you'll be presented with some unstructured data files. Your goal is to load, clean and structure their data in a tidy data set that is targeted towards a specific business scenario. Coming up, you'll begin to explore what it means to be a data professional. In other sections of this program, you'll work on developing additional skills to help you succeed in the data career space. There's so much more to learn about data visualization, statistics, models and machine learning. The skills you learn and strengthen through this program will help you be a better collaborator when completing future data projects. Learning how to use and navigate Python will also make you an ideal candidate for data professional roles. As a data professional, a large part of your job involves engaging with data to help your team and others in your organization develop critical insights that ultimately drive business decisions. Often, there is so much data that tools like Python are needed to successfully complete daily work. This part of the portfolio project is a great opportunity to demonstrate to potential employers that you can do exactly that. Take unstructured data and clean, organize and manage it to achieve an actionable goal. And remember, developing your skills as a data professional is an iterative process. So you can continue to improve as you have new ideas or learn new things. I appreciate how far you've come at this point in the program. You've done a lot of work already. You completed two entries in your pay strategy document and began writing your own code. As you continue to work on your portfolio projects, you will want to consider how you can document your process and explain what you've done to potential employers and hiring managers in future interviews. First, it's important to recognize that as a data professional, you may be asked to learn new tools. There are a lot of great options out there and different businesses have their own preference depending on their needs. As you apply for jobs, keep in mind that you have learned a lot of transferable skills that can be applied across organizations and industries. For example, in the part of your portfolio project you just completed, you use Python to build a tidy database, focus on solving a data-focused business scenario. Python is a great tool and knowing how to use it is a tremendous skill. But even more importantly, you've learned to consider how a data professional's work contributes to business decisions and strategic insights. You've learned the importance of communication, the value of tools available to you and how to use Python to manage large datasets. These are skills worth highlighting in job interviews, no matter what tools the position requires. This portfolio project is a great way to showcase these transferable skills and give interviewers insight on your approach to problems, your thought processes, and why you made certain decisions. In addition to making sure you're highlighting transferable skills when talking about your portfolio project, you'll also want to make sure you're considering your audience. As you have been learning throughout these courses, you will often work with different kinds of stakeholders who have different levels of technical knowledge. When you're communicating with them about technical processes, you'll want to keep in mind who your audience is, what their goals are, what they already know, and what they need to know. This is just as true when you're discussing your portfolio project with interviewers. Often, there will be people conducting or joining your interview who aren't necessarily data professionals. For example, hiring managers may not have the same detailed understanding of data processes as you do. In order to keep your presentation relevant to them, try to remember any questions about your audience. Your interviewers have a business challenge, just like stakeholders on data projects. They have an open job position they need to fill. Think about what they need to know about you to make a decision that solves that challenge. Coming up, you're going to learn all about how to tell stories with data. Then you'll have an opportunity to perform some exploratory data analysis and create data visualizations. By the end of the program, you'll have a strong portfolio. Congratulations! You have completed the end-of-course project. You now have a tangible product you can present to future employers that demonstrate your Python proficiency. Wow, you've learned so many new Python skills. First, you learned how to use variables to store and label your data. And how to convert and combine different data types such as integers and floats. Next, you learned how to call functions to perform useful actions on your data and use operators to compare values. You also learned how to write conditional statements to tell the computer how to make decisions based on your instructions. And you practiced writing clean code that can be easily understood and reused by other data professionals. Then you discovered how to use loops to automate repetitive tasks. You also learned how to manipulate strings by slicing, indexing, and formatting them. After that, you explored fundamental data structures such as lists, tuples, dictionaries, sets, and arrays. Lastly, you learned about two of the most widely used and important Python tools for advanced data analysis. NumPy and Pandas. Coming up, you have even more exciting discoveries to make. Now that you understand how to create systems to prepare data for stakeholders, it's time to start thinking about how to present that data and make it useful for decision making. You now have a strong foundation of Python skills that you can continue to build on in your future career as a data professional. So go get ready and continue your learning journey. Imagine you're an archaeologist, someone who unearths artifacts to study ancient civilizations and preserve the stories of history.  The rising sun creates a warm glow on the orange and yellow rock of the ancient riverbed in front of you. You forget your early morning tiredness and yesterday's 15-hour flight as you breathe in the crisp, clean air. You pause as you remember the words of your site leader the previous evening. This spot has never been studied, but it's absolutely perfect for preservation. You're guaranteed to find something, anything we find will be able to present at the International Archaeological Research Institute next summer. What could be hidden under that rock? What ancient mysteries will be uncovered? What stories will be unearthed? The possibilities seem endless. Hi there. My name is Rob. I'm a consumer product marketing leader. I work on marketing projects here at Google. And every time I get the opportunity to analyze data it feels like I could be an archaeologist on the verge of an incredible discovery. Some people see a data sheet or a table of disorganized numbers and they think, ugh, this is so lame. But data professionals know better, don't we? We know that hidden inside the numbers, columns and rows are golden nuggets of information never before seen insights or compelling trends. These interesting bits of hidden knowledge are stories waiting to be shared. And stories are one of the most impactful ways to communicate ideas. The amazing truth for us data professionals is that all data has stories to tell. So whether you're an aspiring data professional or you want to learn to tell a good story, preferably both, welcome to this course. By now you have a fairly good idea of the scope of this program and the basics of Python coding. Now it's time to dig deep into unexplored data and try to make sense of it. Are you ready to explore? We will begin with how to find and sculpt stories using a six part process called exploratory data analysis. Then we'll discuss how PACE applies to telling stories using data. Do you remember the data professional workflow acronym PACE? That is plan, analyze, construct and execute. We'll talk about how PACE applies to exploring data and learn about the necessity of visualizations in understanding the data. Finally, I'll show you how to perform exploratory data analysis on a data set in Python expanding on those Python skills you learned previously. Later in this course, you'll learn about data sources, data types, data structuring and data cleaning. Using Python notebooks, we'll discuss how to work with missing values, outliers and categorical data. We'll talk about the ethics of exploring and cleaning raw data and how to communicate your questions and findings to various audiences. Later in this course, you'll learn more about the visual analytics platform tab below. We'll talk about how to enhance your data story with visuals and presentations. Along the way, I'll share tips on shaping your data stories for different audiences like how to build data visualizations that target an audience's needs. Data professionals at Google identify the essential skills that are fundamental to working in data. Throughout this course, there will be opportunities to practice finding and telling stories with data. In short, stories can change lives. And data-driven stories are particularly compelling because they are based in numbers. They can also communicate principles, concepts, cautions and new ideas to others. To give you an example for my own life, I actually started my career as a data analyst for a healthcare consulting firm. I was responsible for identifying and recommending the most effective treatments for patients with serious medical additions ranging from asthma to diabetes to cancer. To develop my recommendations, I analyzed the medical records of millions of patients and compared and contrasted the outcomes, side effects and medical costs of each treatment. Through my findings, we were able to recommend treatments that would not only help these patients heal but also help improve their quality of life and reduce their medical bills. The stories are out there just waiting to be found. So get out your shovels and magnifying glasses and let's start exploring. I'm Rob. I'm a product marketing manager at Google. I kind of grew up in the late 90s and the early 2000s in Boston. As an Asian American, I was often really ridiculed for all the standard stereotypes of being Asian and among those were being good or passionate about math or science. When I got into high school, I really found myself rejecting math and science and really trying to avoid those classes, trying to avoid studying them and leaning more into things like humanities or even sports just because I just wanted to fit in so badly. I was lucky enough to be hired as essentially a literature review specialist at an economic healthcare consulting firm. What I found when I was first hired there was that there was this whole other arm of the company that was devoted to analyzing millions and millions of medical records data to really understand and help test the efficacy and safety of medications or pharmaceutical medications to help people with really severe issues and they would do this again by analyzing data for I think it was like over 30 million users and when I heard about this I was so excited. I said, wow, that is a really cool field to potentially get into. I had a chat with my manager who was really open to me taking on maybe some side projects of course after I finished my day to day job and I remember I would spend time after work after hours. I had this textbook that was reading up on just standard data analytics and statistics but also something called SAS programming which is a statistical analysis software that was used by the analysts at our company and so I dove straight and head on into that and eventually I became so fluent at it that I was able to transition to becoming a data analyst and during my time as a data analyst I was so passionate about understanding more and more about statistics and math in general. I actually ended up taking night courses at a local community college because it was my goal to really try to study more about statistics and eventually I was lucky enough to apply to a lot of masters programs in statistics and given my background, my experience I was fortunate enough to get into one. I really wish that I hadn't maybe listened to the other people kind of making fun of me and I just did what I wanted to do and I do to a degree regret that right and I didn't have that foundation but what makes me almost proud to a degree is that like I was able as an adult to really transition into this field and like it really just highlights in my mind that you can really do whatever you want and accomplish whatever you want at any time in your life. Take a step back and just realize that we're all different people on different paths at the end of the day. Take your time each person learns in so many different ways whether it be through these awesome online courses that we offer or enrolling in a university program or you know talking to their friends and working with them it's okay whichever way you are best at learning you know the point is take your time and do what you need to do to learn. You shouldn't put yourself under pressure and compare yourself to anybody else you know just be you and it's okay. Hello again welcome to the first section of this course. In the next few lessons we'll be exploring the foundations of data analysis and how the six practices of exploratory data analysis help us find and tell stories using data. We'll begin by discussing each of the six practices of exploratory data analysis. You'll learn about what they are and why they're important. Then we'll learn about how our data professional workflow PACE or plan, analyze, construct and execute fits into the process of exploratory data analysis. Finally you'll learn the importance of data visualizations in the data exploration process. We'll consider what data visualizations are and how data professionals use them to learn about and share data. The next few lessons will lay the foundation for how to tell stories using data. I hope you're as excited as I am to get started.  Your manager says no one has used the building for decades and doesn't know what the owner's used it for over the years. You've been assigned to clean it out and create a detailed inventory of what you find inside so that the items can be sold at auction. The manager suspects there'll be interesting antiques inside. Since many items are covered by blankets, you don't know what the room contains. You start cleaning from one side of the room to the other and keep a digital inventory of the items you find. As you explore, you find a pile of metal gears, chains and connection parts of all different sizes and shapes. You count them and write Metal Parts 47. You also find 5 large metal sheets, 12 large tricycles and 7 metal poles, among other things. As you write these new items you start to wonder if they're related or not. This simple example demonstrates how data professionals explore data and learn more about the stories and trends along the way. As you uncover and clean up what is in front of you the individual parts and pieces within the data will start to tell a larger story. Of course, data requires a few extra techniques and practices than in the warehouse example, but the general idea is the same. You'll need to make sense of raw data content by reordering, categorizing and reshaping it. You'll find a lot of different terms in the world for this process. Data wrangling, data remediation, data munging data cleaning are all the most common. We combine all of these practices into a familiar term that most data professionals are familiar with exploratory data analysis or EDA. Exploratory data analysis or EDA is the process of investigating organizing and analyzing data sets and summarizing their main characteristics often employing data wrangling and visualization methods. The 6 main practices of EDA are discovering structuring cleaning, joining validating and presenting. These practices do not necessarily have to go in this order and depending on the needs of the data team and the type of data they study, they may perform EDA in different ways. You'll also find that often the EDA process is iterative which means you'll go through the 6 practices multiple times in no particular order to prepare the data for further use. Let's spend time on each of the 6 practices so that you have a better idea of what I mean. Discovering is typically EDA's first practice. During this practice, data professionals familiarize themselves with the data so they can start conceptualizing how to use it. They review the data and ask questions about it. During this phase, data professionals may ask, what are the column headers and what do they mean? How many total data points are there? In the example of the old warehouse beginning of the video, the discovering practice might involve walking through the room and removing coverings to get an idea of the amount and types of items. After some initial discovering, the next step is to start organizing. This EDA practice is called structuring. Structuring is the process of taking raw data and organizing or transforming it to be more easily visualized, explained, or modeled. Structuring refers to categorizing data columns based on the data already in the data set. In terms of the calendar data, for example, it might look like categorizing data into months or quarters rather than years. From the old warehouse analogy, structuring could be categorizing the items into metal and non-metal categories and getting a total count for each type. Before we move to the next practice of EDA, let's take a moment to talk about bias. Bias in the context of data structuring is organizing data in groupings, categories, or variables that don't accurately represent the whole data set. Most experts would agree that eliminating all bias from how data is structured is almost impossible because each person's individual ideas, training, and experiences are different. As professionals, however, it's important to try to avoid bias while structuring data. For example, imagine you want to know what percent of the population has a college degree in the UK, but your model data only contains London residents. The data would be considered bias until you add data from the rest of the UK. In our warehouse example, trying to avoid bias might involve remaining flexible with the categories, potentially creating new categories as you uncover more and more items. The next EDA practice is data cleaning. Cleaning is the process of removing errors that may distort your data or make it less useful. Missing values, misspellings, duplicate entries, or extreme outliers are all fairly common issues that need to be addressed during the data set cleaning. In the warehouse example, you might decide to put broken or unusable items in a separate box away from other items. We'll move to another practice called joining. Joining is the process of augmenting or adjusting data by adding values from other data sets. In other words, you might add more value or context to the data by adding more information from other data sources. For example, you might find during the discovering, structuring, or cleaning processes that the data set doesn't have enough data for you to complete a specific project. In that case, you should enrich the data by adding more to it. As an example, remember when I talked about the UK and college degrees to help understand bias? In that instance, joining would be adding the data from the rest of the UK rather than just including data on London residents. Going back to our warehouse analogy, imagine a museum manager sorts through the items and gives you the dates of when each was made. The information from the museum manager acts as a different group of data you can join with your own, as you can. Next on the list of EDA practices is Validating. Validating refers to the process of verifying that the data is consistent and high quality. Validating data is the process for checking for misspellings, an inconsistent number or date formats, and checking that the data cleaning process didn't introduce more errors. Data professionals typically use digital tools such as R, JavaScript, or Python to check consistency and errors in a data set and its data types. As for our warehouse analogy, Validating could be like using the museum manager's knowledge to get an idea of how old the items are. The last EDA practice is Presenting. Presenting involves making your cleaned data set or data visualizations available to others for analysis or further modeling. In other words, Presenting practice is sharing what you've learned through EDA and asking for feedback whether in the form of a clean data set or data visualization. We will be using the term data visualization a lot. To be clear, a data visualization is a graph, chart, diagram, or dashboard that is created as a representation of information. You might think that Presenting always comes at the end of the EDA process. However, Presenting can come at any point in EDA. Data visualizations aren't exclusive to the Presenting practice. They should be used throughout EDA. They help you understand data and point out trends and insights to others. In the warehouse analogy, Presenting could mean showing your manager the progress on the warehouse and how many different items were found. In the workplace, as a data professional, Presenting might look like preparing visuals and a slide presentation to share with your team. As you begin to plan your presentation, you should consider people with visual or auditory impairments by providing robust descriptions of the data. You can use things like alt text, descriptive text, or captioned recording of the data so that your audience can explore the data themselves. We will cover each of the EDA practices in detail later on, but one of the most important things about the process is to ensure your EDA work does not misrepresent the data itself. The story you uncover should come from the data, not from your mind or biases in the data. It is your duty to convey your data in both an ethical and accessible way. Consider the warehouse analogy. To ensure you communicate your data ethically, you should give an accurate count of the materials and not overestimate the quantities of the antiques. In the workplace, communicating data ethically would be presenting sales numbers in context, year over year so that rises and falls don't appear exaggerated in data visualizations.  You find a dusty pair of signs that advertise food for sale. Suddenly the items you found start to make sense as a whole. These antiques appear to be a collection of supplies of food vendors. How about that for discovering a story? Of course, you should confirm this information using historical records before sharing. Hi, I'm Benj. I'm a product analyst at Google. I lead a team of analysts attempting to understand Google Chrome through real-world usage and insights. Exploratory data analysis is a key piece of my work and in particular when you get a new piece of data or when you have a new question, the first thing you have to do is take a deep look through the data that's there and understand what's in front of you. You have to understand who the data is from. You have to understand why it was created and what isn't there and what important caveats are sitting on top of it. Storytelling is the way that your insights make it to other people and really make change. It's oftentimes the case that we'll make reports that are tens of pages long, but what really changes the way people think is when you can tell a short and clear story about what's going on. A really good way to tell a story with data is to think about categories of users, categories of devices or categories of use cases. When you can tell people that a lot of people are using Chrome on low-end phones in India, it tells a story about who your user base is and what they're doing and knowing who they are and knowing what that looks like allows them to think differently about who the audience is and what product they should build. One thing I try to do to make sure that I'm really staying true to data is trying to check my own biases and make sure that I am giving back what I actually see. One way that I attempt to make sure that my reporting isn't biased is to try to come into new analyses with as few preconceptions as I can. You need to know some things to be true and to verify them against the data set to make sure that users aren't spending 27 hours a day on their phone because there aren't 27 hours in a day. To not assume that everyone is going to look the same. To not assume that all places in the world are going to act the same when it comes to any data. It's very important ethically to make sure that your data is working for everyone. To make sure that if you try to remove identifiers and try to take data and de-personalize it, that you've really done that well and effectively. The simplest methods aren't always the best. It isn't enough to just say, oh I tried something and I need to stop. You have to explore a little bit more and make sure you're really de-identifying data. Curiosity is one of the most important traits for an analyst in general and by trying to learn something new you're showing that curiosity already. Have you ever walked into a room to grab a specific item and then become distracted and forget why you were there? As data professionals, our curiosity and excitement for finding stories and data might cause us to forget the original purpose for our data exploration. Obviously we want to maintain our natural curiosity but most importantly we want to focus that curiosity on what questions need to be answered or what problems need to be solved. We seek a balanced mindset, one of targeted curiosity. This balance can be achieved by using pace. As you learned, pace or plan, analyze, construct and execute is the workflow some data professionals use to remain focused on the end goal of any given data set. Imagine you work for a multinational company that manufactures office equipment. The finance department asks you to use the last 10 years of data to predict sales for the next 6 months. How might you approach your EDA of the data given this task? In this video you'll learn to combine pace and EDA practices. Remember, EDA stands for exploratory data analysis. You might think that because analysis is in its name that EDA falls only in the analyze part of the pace workflow. The truth is EDA applies to every part of pace. You'll find the 6 practices of EDA all intersect with the other parts of pace. Discovering for example is in line with the planning part of pace and presenting can be a major part of the executing part of pace. As you will learn, data professionals don't just find and tell any random story from the data. Data insights are guided by a project's purpose and goals. The plan may come from a stakeholder or manager. You can think of it as the project plan or the stated goal of the company. For example, let's consider the sales prediction from the office equipment company earlier in the video. As you recall, your task is to predict the next 6 months of sales performance based on 10 years of sales data. When you start your EDA, you realize the data sets you've been given contain a lot more data than you need. It would benefit you and your company to extract only the columns you need to predict sales which would be date of sale item, price and sales rep for a total of 4 columns. While cleaning, joining and validating, you can exclude data on material cost, item ID numbers, database cost center numbers and vendor names. Cleaning 10 years of data sets with 4 columns is much more manageable than 8 columns. Of course, in a typical workplace, it shouldn't surprise you that after you've done a lot of the work to complete your EDA based on the plan, the finance department says they actually wanted the predicted profit margin, not the sales targets. Miscommunication happens in every workplace, even in data analysis. If stakeholders, engineers and data specialists are not clear on the plan, the results won't tell a cohesive or effective story. Instead, the results will likely lead to confusion, disagreement and wasted time. One example of good communication would be sharing the pace plan with anyone who might be involved. Another example could be sharing the analysis with a working group to get feedback before sharing the analysis more broadly. And finally, it's important to understand stakeholders' most important goals for the company before presenting them. We will address details about communication strategies in upcoming videos. The point is when you have a data set in front of you, remind yourself of the reasons for your analysis. Once again, consider the office equipment company. You've been given two data sets, one with transaction ID number and date, the other with transaction ID number and total cost. You're asked to forecast the next six month sales date. If you consider the purpose of this task, what is something you might do with the two data sets? You might consider the EDA joining practice and include all data from the two data sets together, transaction ID, date and total cost. For another example, let's say you have a hospital's data on treatments performed over the last year. If the goal is to prepare data to make a purchase order for the materials and supplies for the upcoming year, what type of EDA might you consider? It would be a good idea to start by using the EDA practice of structuring to group the data by the type of items needed for each treatment. Data professionals are set up for success when they follow a framework like PACE. When a professional applies that framework to the way they perform the six practices of EDA, they keep priorities in order and they focus on achieving the project's purpose. Of course, as data professionals, our first priority is to accurately represent the data itself. If your company's project plan does not align with what the data is telling you, it is your responsibility to communicate that to stakeholders. Let me return to the sales forecast request at the Office Equipment Company. Imagine you're given data on sales revenue specific from two geographic regions but stakeholders are requesting a global forecast. As a data professional, it is your responsibility to ask for globally representative data to complete the task. Working with data from only two regions would be inadequate to make a global projection. Timelines, stakeholder pressure, or client needs should never cause a data professional to bypass what is required by the data. Misrepresentation of data is never warranted. Let's review. Data professionals should strive to align their work with the plan part of PACE. Keeping the focus on PACE helps determine the most effective ways to perform EDA practices while maintaining ethical representations of data. Coming up, we'll explore how PACE can help guide the development of data visualizations. I'll meet you there. Let's review a scenario, shall we? You're given a giant table of numbers and a data visualization, which would you choose to most quickly communicate data insights. Data visualizations are far more effective at quickly communicating complex information than data tables. Now, let's consider how this relates to EDA. Data visualizations are important tools that data professionals use to tell stories with data throughout their workflow, particularly the EDA practice of presenting. Plotting parts or all of your data set on a bar graph, a scatter plot, a pie chart, or a histogram will help you and others to understand the data no matter where you are in the EDA process. As a data professional, you won't be working with simple, clean data sets that have only a few hundred lines of data. You'll be working with data frames that have thousands or hundreds of thousands of data points spanning months, years, or decades even. The more data you have, the more visualizations you'll need to create to understand how each variable impacts the other. When you start looking at new data, a popular and valuable tactic is to visualize it, like time series data, online charts to understand periodicity, or scatter plots to get a good idea of data distribution. Data visualization can also help you explain your data set to stakeholders and other data professionals. It is your job to discover the important points, trends, biases, and stories that need to be shared than design data visualizations in an effective way for different audiences. For example, imagine you're a data professional who works for an appliance manufacturer. Let's say you perform an analysis for the manufacturing team. During the analysis, you discover a delay in the manufacturing process. You need to communicate these findings to two different audiences, the manufacturing supervisors, and the executive leadership team. When you share your findings, some stakeholders may not understand the data insights without the help of data visualizations. The ways data visualizations are designed will communicate different messages to stakeholders with vested interest in different parts of the project. For example, the manufacturing supervisors might need to review time series data plotted out over time to identify the manufacturing delays. Meanwhile, the executive leadership team would likely be more interested in financial impact analysis. Those two data visualizations would need to be designed differently based on the needs of your audience. We will go into detail on exactly how to do that in the upcoming videos, but in the meantime, understand that the balance of words and data visuals in a presentation impacts the business decisions made from data insights. A carefully prepared data visualization can mean the difference between changing the mind of a stakeholder and the data being ignored. However, visualizations can also cause confusion or even misrepresent the data. For instance, imagine a data professional develops a visualization and changes the scale of the axes or the ratio of the graph height and width to make the line chart look flat or steep. This example of skewing is the opposite of what data analysis is about. Because you will be the person most familiar with the data in its story, it is essential that your visualizations not mislead your audiences. Returning again to the manufacturing equipment example, if you provide a data visualization showing a sharp increase in sales over the last six months, your audience might assume the company is doing very well. But if data from the last two years shows the six month increase comes right after a long 18 month decline, the last six months has a different context. Showing only the last six months as opposed to the two years is misrepresenting the sales data. Your data driven storytelling is an opportunity to present facts and visualizations that are ethical, accessible and representative of the data. Being an ethical presenter of the data you analyze means being honest and very clear about what is and isn't in the data. Along with that, remember to design your data visualizations in a way that is accessible to everyone. For example, avoid pairing red and green in data visualizations as it can be difficult for people with color blindness to read. Generally blue and orange are the better choice to use in data visualizations. We'll go into more detail on ethics and accessibility in upcoming videos. In other videos, we'll use digital visualization tools like Tablo and Python packages like Matplotlib and Seaborn and Plotly to understand how to use your data and graphs and charts in ethical and accessible ways. These tools will be part of your everyday work as a data professional. Remember, creating visualizations about your data sets will help you throughout the entire EDA process. There's often no better tool for telling a good data-driven story than a well-made visualization. We've talked a lot about finding stories within data and about how it is your job to tell those stories to the best of your ability. There is, however, another story weaving its way through the readings, videos and quizzes and is still continuing to unfold as you listen to this video. And if it isn't clear, it's the story of coming this far in the program shows you are determined to become a data professional. I hope you feel inspired by the idea of defining not only the stories of your data, but your own story as well. I'm sure you noticed that this part of the course had very little coding instruction in it. That's by design. The job of data professionals is about more than coding and statistical modeling. The first part of the course reflects that. It is also important to recognize how exploratory data analysis or EDA applies to finding, sculpting and telling data-driven stories. If you talk to data professionals in the career space, many will share how beneficial EDA is in day-to-day work. So I took some time to talk about the practices of EDA. Discovering, structuring, cleaning, joining, validating and presenting. And review the value of these practices. We talked about the pace workflow and how it can apply to EDA in our search for stories within the data. You also learned about the ethics of working with data and the importance of representing data accurately when we tell data stories. Lastly, you learned how data visualizations are essential for understanding, forming and presenting data-driven stories. You learned that telling a data-driven story that accurately represents the data and is inclusive of its audience is an example of exceptional data analysis. The story doesn't end here though. We have many more concepts to learn in this course, including using Python tools and coding blocks to help you in your EDA practices of discovering, structuring and cleaning raw data sets. In this program you will work with data sets that represent the type of raw data you'll likely see every day in your career as a data professional. Remember, all data has stories to tell. These stories are often hidden well and it will take an especially curious and determined professional to discover them. The data-driven stories you'll be discovering have the potential to change an entire company or even the world. I am looking forward to continuing our journey together through the rest of this course as you discover questions solve problems and learn new concepts. I hope with each new concept you learn you discover a little more about your own story as well. I'll see you soon. Have you ever tried to explain something to a friend that you didn't understand very well? Maybe it was a difficult math assignment, a complex news story, or the way a family member cooks a favorite dish. When you're trying to teach a topic you're not an expert on, it can be challenging to explain the details and give clear instructions. As a data professional, you never want to be in this position with the data you analyze. In fact, your goal should be to know your data very well. When you're reviewing a table of data, it's important to understand where the data is from, what the column headers mean, what the data will be used for, what the imperfections are, and the small details in between. Making sense of raw data is why we're here. Welcome. I'm excited for you to build on the knowledge that you've learned so far to perform exploratory data analysis or EDA on the kind of data sets you'll see on the job. You'll start by learning about the many different data types and data sources that you will encounter in your work and how to study them. After that, you will return to those Python notebooks to start coding the concepts you've learned. We'll also go into more details on the first two practices of EDA, discovering and structuring. While learning about the discovering practice, we will use popular Python functions to get to know the information contained in the data sets. You'll learn to use different visualization techniques to uncover hidden correlations and connections in the data. As for structuring, you will learn to apply Python functions to large data sets for many different operations, including sorting, filtering, extracting, slicing, joining and indexing large data sets. You'll learn to make basic corrections or formatting improvements to hold data columns or entire data sets. All of these techniques and practices will help you learn about the data and find the story that needs to be told. Along with all the Python functions and coding scripts, we will talk about what to do when questions about the data set arise that you can't answer, like why they're missing data fields, for example. We'll also make sure the EDA you perform aligns with the pace workflow that we have set. At its core, this section of the course is about digging through a data set for the first time and investigating it as meticulously as you can. It is up to you to find the stories in the data. Most data professionals will tell you that comprehensive EDA is the key to useful visualizations and models. If there are questions left unanswered or misunderstanding still present in the data after EDA, any presentation or machine learning model based on it will not be particularly useful. It might feel like trying to explain a math assignment you don't fully understand. Remember, all data sets have stories to tell. Let's get to work and find them. My name is Alassair and I'm a senior financial analyst at Google. I was born in Nicaragua and I immigrated to the United States when I was two years old. I grew up in inner city Philadelphia. After high school, I joined the United States Marine Corps, spent time in Japan, South Africa, Saudi Arabia and Panama. One of the big reasons I got interested in finance is because when I was in my first duty station in South Africa, I befriended the ambassador who had worked in Wall Street. He took me under his wings and mentored me and taught me a lot of things about personal finance corporate finance and the global economy and my mind shifted from I'm going to be a Marine forever to I'm going to get out of the Marine Corps after my four years and go back to school and it worked out. I learned about advanced data analytics on my own actually between random YouTube channels just Googling stuff. I got into big data analytics partly because to understand and drive value in your organization, you have to really understand underlying data. As a senior financial analyst, the typical day in the life really is focused around a couple of things working with strategic business partners. I spend a lot of time trying to figure out what the right financial view is so they can have the insights into their business to influence their decision making. Completing an online course like this is difficult. When you look at that course schedule and see that it's three months or six months long just focus on that next day and stick to your schedule, show up every day. It's like running a marathon you know you're not running 26.2 miles, you're running one mile 26.2 times just focus on the very next thing. I believe in you, you can do it. Imagine for a moment you're preparing a meal for friends. You have a recipe and raw ingredients but this is the first time you'll cook the dish. Of course you want your friends to enjoy it. If you're like me the idea of being judged on perfectly preparing a recipe the first time you've made it is scary. Believe it or not this cooking example is what it's like to be a data professional during the discovery practice in EDA. The recipe you're trying to follow is equivalent to your company's project plan. You have everything you need at your disposal, a kitchen or a digital workspace with dedicated servers for data analysis computing, raw ingredients or a data set and it's up to you on how to best mix, blend and cook the ingredients in order to make a winning dish that your friends will enjoy. In this video, let's focus on raw ingredients. As a data professional, you will be handling all kinds of different data in a number of different formats and file types. I will share with you the most common data sources data formats, data types and a few Python functions. Once you're familiar with a data set source, format and the data types within it you'll be ready to handle questions or challenges as they arise during the analysis. First when you're given data you'll want to know the source. The term data source can have many different meanings in different contexts, but for our purposes we'll be defining data source as the location where the data originates. One good thing to know about a data source is how and when to contact subject matter experts such as engineers or database owners. These are the people who either generate the data or are in charge of delivering the data sets. When you have questions about the data, as you do your discovery they will be the ones you reach out to. Knowing the ownership and source of the data is critical because by understanding the data source and who is responsible for it, you can determine its reliability. Does the data owner have experience in storing data? Does the data owner have any financial stake in the data's output? Understanding the source of the data will help you in telling the story of the data and make ethical decisions about its use. Another important part of determining the data source is understanding how it's collected. Whether the data was collected through a report from a computer system a custom selection from a large online database or a data table that has been manually entered knowing about how the data was gathered will help you understand questions that may come up during EDA. For example missing values could mean many different things. Maybe the database owners either didn't know or didn't want to disclose the data for manual entry or there might be lagging data or system bug from an online database. The next thing you'll need to know about your data sources is the data file format. The main data formats you'll experience as a data professional are tabular files, XML files CSV or comma separated value files spreadsheets, database files or JSON files which stands for JavaScript object notation. Here are a few examples of what these file types look like. If you've gotten this far into the program you'll be familiar with tabular and Excel files by now. As you know they organize data in tables with data variables organized by rows and columns. Rows representing the objects and columns representing aspects of the objects in the data set. The advantage of this file type is a clear identification of patterns between variables. CSV files are a simple text file which can be easy to import or store in other softwares, platforms and databases. They look like rows of text and numbers separated by commas or other separators. The rows of data are broken up by commas rather than strict columns. The advantage of CSV files is more from a computer science aspect in that it is really a file type which is easily read even in a text editor. It is also easy to create and manipulate. In Python you can use the read CSV function to read, write and work with tabular data that is in the CSV format. Database or DB for short is another way of storing data often in tables, indexes or fields. Database files are great for searching and storage. They often require some basic knowledge of search query language or SQL. We will explore how to query data from databases in an upcoming section. Lastly, JSON files are data storage files that are saved in a JavaScript format. You'll find the information in these files better resembles with different language, function and format. JSON files may contain nested objects within them. You can think of nested objects as expandable file folders or drop down menus within the code itself. For example, a JSON file might have the ingredients for a recipe listed and under each ingredient you would have included nested information like weight, calories and price objects that define the ingredients. There are a few advantages of JSON files. They have a small message size. They are readable by almost any programming language and they help coders easily distinguish between strings and numbers. There are Python libraries and functions built for working with JSON files. As you learned previously, you can import the JSON Python module into your Python notebook to use as an encoder and decoder of JSON files. There are also tools in the pandas called read JSON and to JSON which will translate JSON files and convert an object to JSON format type, respectively. As a data professional you may also be tasked to find data stories in other formats like HTML, audio files, photos, email text, images or text files. In every case there are Python libraries or functions that can help you discover and structure the data for your research project. There's no best format for the data. It will depend on the project and storage type. You're just looking for the format that best fits that particular data set. Finally, the last raw ingredient in understanding data is types of data. You may already be familiar with the many different categories of data. As a reminder there is a first, second and third party data. First party data is data that was gathered from inside your own organization. Second party data is data that was gathered outside your organization but directly from the original source. Finally there's third party data which is data gathered outside your organization and aggregated. Knowing the types of data whether first, second or third party will help you be able to more efficiently answer or seek answers to the questions you may have that arise during analysis. For example, if there are missing values in first party data then someone in your organization can help you determine whether the missing data can be recovered. For third party data you'll likely have to reach out to a separate organization. You'll also be familiar with different types of data like geographic, demographic numeric, time-based financial and qualitative data. Your job as a data professional will require that you understand and work with all these types of data. Knowing the data source format and type you're working with will help you to answer two very important questions. First, given what you know of the data so far does it align with the plan as defined by your pace workflow? And second, do you have enough data to follow through with the plan in the pace workflow? If you've answered either of these questions with a no or not sure it is your job to reach out to the owners of the data and project stakeholders to inform them of the issue you found. For example, let's say you're assigned to the project manager and the project manager will predict the number of customers a retailer will expect to see next month. Unfortunately, you're only given profit margin data and only two months worth of customer purchase data. Because the profit margin data won't help you with returning customers and only two months of data won't give you much confidence in prediction in the last few years to accurately predict an upcoming month. Returning to the data source and requesting more data will keep everyone in the process including you, focused on the plan that was established as part of your pace workflow. When you're working as a data professional, keeping this focus will be essential in identifying and carrying out high priority and value add tasks. As a data professional, you're going to know what the data means and how to work with it to find solutions to business problems like cooking a meal. You'll be given the tools and ingredients you need to work on the problem. It is up to you to work with your team if there aren't enough ingredients or if what you've been given won't make a great dish. Now that you have a better understanding of exploratory data analysis and why it's important, let's do some data sets similar to one you might work with in your career as a data professional. The National Oceanic and Atmospheric Administration or NOAA keeps a daily record of lightning strikes across much of North America. Let's imagine we've been tasked with performing EDA on this data set so that it can be used to predict future lightning strikes in this region. In this video, we will use Python to do the EDA practice of discovering on data gathered by the NOAA in 2018. We will talk through the first steps data professionals will typically do when encountering a data set for the first time. Let's open up a Jupiter Python notebook to begin. First, we'll want to import the data set we wish to analyze and the applicable Python libraries we want to use. This preparation step is similar to the way a painter gathers paint brushes, paints and an easel before they start painting. Similarly, we want to collect all our tools and data. The NOAA data set we will use is in the public domain, but we have provided a file to download so you can follow along. To begin, import the Python libraries and packages you plan to use. In our case, we'll use Pandas, NumPy and Matplotlib.Pyplot To save time and increase efficiency, rename each library or package with a two-letter abbreviation. PD, NP and PLT, DT, respectively. Another quick tip for saving time, instead of clicking Run Cell, you can always use the keyboard shortcut Control or Command Enter. You may recall that Python packages like Pandas and NumPy are open source code packages that have been pre-designed and coded to help analyze and manipulate data more efficiently. Pyplot is a package focused on plotting charts and visualizations. Most data professionals import all their commonly used libraries, packages and modules at the beginning of any coding session. But you can import them at any time while working with data in a Python notebook. The Pandas conversion function to date-time is incredibly helpful when working with dates and times. We will first convert our date column to date-time which will split up the date objects into separate data points of year, month and date. This is good practice because throughout your career you'll be grouping data by various time segments. We'll get to that later. After running these initial functions you are ready to begin your EDA discovering practice with this data set. Let's begin with the head method or function. Head will return as many rows of data as you input in the argument field. We'll look first at the top 10 rows. When I input head with a 10 in the argument field and click run the first 10 rows of the data set are now in our notebook. This data set contains three columns of data date, number of strikes and center point geom. At this point in the exercise we will need to clearly understand what each of these columns means. Date and number of strikes are fairly straightforward. You'll want to pay attention to any dates and their format which in this case is year month date. You'll also find when looking at the date column that lightning strike data is recorded almost every day for at least one location in the year 2018. Some column headers are obvious but some like the column header center point geom aren't. If you are not sure about what the column headers mean while you're working on a project you could go back to the public documentation available from the NOAA to confirm its meaning. For this video the information is provided. The center point geom column refers to the longitude and latitude of the recorded strikes. We know the number of columns but we also want to know how many rows of data we'll be working with. Another great EDA discovering tool is the info function. The method called info gives the total number and data types of individual entries. Keep in mind that data types are called d-types in Pandas. If we type df.info into our notebook and run it we will get this output. Our range index tells us our total number of entries. Nearly 3.5 million. We also find that the data values in our date and center point geom columns are classified as objects and the data in our number of strikes data column is classified as int64. int64 refers to integer 64. It means that the data type contains integers or numbers between 2 to the power of negative and positive 63. Simply put, int64 is a standard integer somewhere between negative 9 quintillion and positive 9 quintillion. There is one more d-type you might see when performing the info function. S-T-R which refers to a string. Like objects you'll likely be familiar with strings already. Strings are sequence of characters or integers that are unchangeable. Getting back to the data set we have a pretty good idea about its size and scope. We know it has 3 columns and we know it has 3 million, 401 and 12 rows. We know what the columns mean. There are other methods or functions that we could use during our practice for discovery. For example the methods describe sample, size and shape are all useful for learning about a data set. For this data set though we have what we need in order to understand what is going on in the data set from a discovery standpoint. So next, let's determine which months have the most lightning strikes by plotting our first visualization. Remember your data set has over 3 million rows. If we don't do any categorizing or grouping we can end up with a notebook that is stuck endlessly trying to run the code. We've already converted our date column to date time which makes it easy for us to manipulate the data in the column. We want to group our daily strikes into something more manageable like months for example. So let's make our date reference the number each month corresponds to in a year. That is January is 1, February is 2 and so on. We'll do that by creating the column month using the code dfdate.dt.month dt in this instance stands for date time. Next, we'll make it easier to interpret on a chart. Let's convert the month data which are currently numbers to month name abbreviations. We'll do this by creating another column month underscore text. This code will take the string of characters or month names and slice to include only the first three letters. Before we try to plot this data on a chart we need to sum the numbers of strikes in all locations by month. We can do this by creating a data frame using the group by function in which we will include the columns month and month text because these are the columns we will use to group and order the lightning strikes. We'll use the sum function to add the lightning strikes for each month together. For our last lines of code in this video let's plot the number of strikes by month for the year 2018 into a bar chart and then total number of strikes by state on a map. This will help us get a good sense of the data's story. You'll find that this looks like a big block of code but because we are using the matplotlib.py plot the actual code for the visualization is not too difficult to follow. First, we have the plt.bar function which will have us enter the data columns we wish to plot with x-axis first month underscore txt followed by height or y-axis which we will fill in with number underscore of underscore strikes. Next, we will give the bars a legend or label as it's called in Python which we will input as number of strikes. Lastly we'll fill in our bar graph details with the x- and y-axis labels and the visualization title. Leave the legend and show argument field blank for now. After we run the code you'll see that August 2018 had the most lightning strikes while November and December 2018 had the least amount of strikes. Now, you've completed your first experience coding for an EDA practice in Python. Very well done. We have a lot more to talk about and this is a great start. I've often found that stepping away from a challenging project and taking a break helps me find new perspectives on my work. I like to get a can of seltzer from the fridge, take a walk and reflect on what I know and what I don't know about the project I'm working on. The EDA discovering process requires a similar shift in perspective. After I perform an initial discovering of a dataset I've usually learned enough about the project to make a hypothesis based on that data. A hypothesis is a theory or an explanation based on evidence that has not yet proved true. Data professionals often use hypotheses as a starting point for continued investigation or testing. Once I form my hypothesis I'm in a better position to discover more about the data and achieve my ultimate goal telling a story. So far in this course we've discussed how to begin the discovering practice of EDA. You learned how to examine data sources, data formats and data types. You've considered column header information and averages and you've made some initial visualizations to represent your data. You use Python to determine the size and scope of the dataset and learned when you need to ask the owner of the data clarifying questions. After you've made sense of the raw data you're ready for the next step of the process, drafting a list of questions and forming hypothesis. In this video you'll learn how to ask meaningful questions about the goal you've outlined in the PACE workflow to better understand what is missing from your dataset and what you still need to find out. One way I like to do this is by breaking the original problem into smaller chunks. Some questions you might ask include how can I break this data into smaller groups so I can understand it better? How can I prove my hypothesis or in its current form can this data give me the answers I need? Let's consider these questions in context. For example, imagine you work as a data analyst for an international airline and you need to determine whether lowering the prices of tickets will attract more customers in certain days of the week. To solve this problem you might ask which months have the most passenger traffic? Which weeks, dates or known holidays have the highest number of passengers? When are tickets typically purchased? Then you'd form your hypothesis. In this case, your hypothesis might be I predict that Tuesdays and Wednesdays of a normal business week have the fewest number of passengers and flights. So, if the airline lowers prices for Tuesdays and Wednesdays during non-holiday weeks, then they will sell more tickets. Eventually, you would test your hypothesis by analyzing the data to understand whether the airline would attract more customers by lowering the prices on those specific flights. The purpose of asking questions and forming a hypothesis is to better understand what you want to learn from the data and what the results of your testing might show. Later, when you're performing other practices of EDA, you can refer to these questions and your hypothesis to determine whether you've supported or refuted your original theory. To answer the questions and test the hypothesis you or your team formed, you will need a plan. For instance, you might need to contact the subject matter expert who owns the database or is more familiar with the data source. Or, you may need to do your own research. In other words, leave no stone unturned. It's an old saying about an ancient Greek legend and it means to search everything you can to think of to find what you're looking for. If you discover that your search for answers only brings more questions, that's a good thing. You're eliminating the possibility of misinterpreting or misrepresenting the data each time you learn more. At some point, you may need to decide you need to organize or alter the data to find the answers to your questions. For example, you may need to regroup entries into months or years rather than days or weeks. Or, you might want to group customer ages into age ranges to help you understand trends more effectively. Sometimes combining or splitting data columns will be necessary for creating models to answer questions. Other times, changing date formats or time zones in time bound data may be all that's required. For example, in your work with the international airline, you were tasked with finding days to enact lower ticket prices so the airline could attract new passengers. Imagine the data you were given listed ticket prices in US dollars, but the original request was to lower ticket prices for passengers departing from Europe. One change you would need to make immediately is to convert US dollars to euros. Making small changes to your data, like formatting the time, changing a unit, or converting the currency is all part of the discovering process. However, with every change you make stay focused on the problem you are assigned or the plan that you established as part of the PACE framework. As we discussed, every data set is different. Asking questions and forming a hypothesis will take time and effort but ultimately answering questions and testing your hypotheses will be the way you find the stories hidden in your data. If you get stuck, it might help to step away from your initial discovering work and think through your questions and hypotheses again. Any visualization rendered, conversion made, questions answered or hypothesis tested must be true to that data set story. Who knows, maybe you'll find the answer on your walk break and wouldn't that be refreshing. As a data professional, you can expect to work through time objects and date strings. In this video we'll continue coding in Python and practice converting, manipulating and grouping data. By the end of this video, we'll create a widely used data visualization, a bar graph that tells a story with your data. Working with date strings will often require breaking them down into smaller pieces. Breaking date strings into days, months, and years allows you to group and order the other data in different ways so that you can analyze it. Manipulating date and time strings is a foundational skill in EDA. In this video, you will learn to convert date strings in the NOAA lightning strike data set into date time objects. We will discuss how to combine these data objects into different groups by segments of time, such as quarters and weeks. Let's open a Python notebook and I'll show you what I mean. Let's begin by importing Python libraries and packages. To start, import Matplotlib and Pandas, which you've used before. To review, Pyplot is a very helpful package for creating visualizations like bar, line, and pie charts. Pandas is one of the more popular packages in data science because its specific focus is a series of functions and commands that help you work with data sets. The last package, Seaborn may be new to you. Seaborn is a visualization library that is easier to use and produces nicer looking charts. For this video, we'll use the NOAA lightning strike data for the years 2016, 2017, and 2018 to group lightning strikes by different time frames. This will help us understand total lightning strikes by week and quarter. As I mentioned at the beginning of this video, when manipulating date strings, the best thing to do is to break down the date information like day, month, and year into parts. This allows us to group the data into any time series grouping we want. Luckily, there's an easy way to do that, which is to create a date time object. Now, as you'll recall from a previous video, this NOAA data set has three columns giving us the date, number of lightning strikes, and latitude and longitude of the strike. For us to manipulate the date column and its data, we'll first need to convert it into a date time data type. We do that by simply coding DF date and making that equal to PD dot to date time with DF date input in parentheses. Doing this conversion gives us the quickest and most direct path to manipulating the date string in the date column, which currently is in the format of a four-digit year, followed by a dash, then the two-digit month, a dash, and lastly, the two-digit day. Okay. This is the exciting part. Because our dates are converted into pandas date time objects, we can create any sort of date grouping we want. Let's say, for example, we want to group the lightning strike data by both week and quarter. All we would need to do is create some new columns. You'll see here we're creating four new columns, week, month, quarter, and year. With the first line of code, we're creating a column week by taking the data in the date column and using the function S-T-R-F time. This function from the date time package formats are date time data into new strings. In this case, we want the year followed by a dash and the number of weeks out of 52. If we want that string, we need to code it as %Y-W %V. The % is the command which tells the date time format to use the year data in the string. The W implies this is a week and the V stands for value. As in a sequential number running from 1 to 52. The final string output for the column data will be in this format 2016 dash W 27. The next line of code gives us the new month column. The argument is then written as %Sign Y-%Sign M. This will output the four-digit year by a dash then the two-digit month. Essentially, we're removing the last two-digit date from the original date string. Next, we will create a column for quarters. In this case, a quarter is three months. Many corporations divide their financial year into quarters so knowing how to divide data into quarter years is a very useful skill. In this case, it only takes one month to write a column on a code. We'll call the new column quarter and we'll use our date column with 2 underscore period to create the quarter column. The date time package has a pre-made code for dividing date time into quarters. In the two underscore period argument field, we only need to place the letter and we can use the function S-T-R-F time to complete the string. For the argument, we put percentSign Y-Q percentSign lowercaseQ The first Q is placed into the string to indicate we are talking about quarters. The percentSign followed by the lowercaseQ indicates the pandas that we want the date formatted into quarters. Our final column will be the easiest to code them all. The year column is created by taking our original date column data and creating a string that includes only the argument percentSign Y. This creates a column of data with it with only the year in it. Now that we have formatted some strings, let's quickly review our work by using the head function we learned in the previous video. When we run this code our four new columns are there week, month, quarter, and year. They are all formatted just as we discussed. We can use these new strings to learn more about the data. For example, let's say we want to group the number of lightning strikes by weeks. An organization whose employees primarily work outdoors might be interested in knowing the week-to-week likelihood of dealing with lightning strikes. In order to do that, we'll want to plot a chart. We've reviewed a couple of charts coded in Python by now. Next, let's code a chart with the lightning strike data. For plotting the number of lightning strikes per week, let's use a bar chart. Our graph would be a bit confusing using all three years of data. Let's just use the 2018 data and limit our chart to 52 weeks rather than 156 weeks. We can do this by creating a column that groups the data by year and then orders it by week. We will then learn more about the structuring function in another video. For now, let's focus on plotting this bar chart. We'll use the plt.bar function to plot. Within our argument field, we select the x-axis, which is our week column. Then the y-axis, or height, which we input as number of strikes. Next, we'll fill in some of the details of our chart. Using plt.plot, we will place arguments in the x-label, y-label and title functions. The arguments are week number, number of lightning strikes and number of lightning strikes per week 2018, respectively. This renders a graph, but the x-axis labels are all joined together. So, we have a chart, but the x-axis is difficult to read, so let's fix that. We can do that with plt.xtix function. For the rotation, we can put 45, and for the font size, let's scale it down to 8. After we use plt.show, the x-axis labels are much cleaner. Given our bar chart illustrating lightning strikes per month in 2018, you could conclude that a group planning outdoor activities for weeks 32 to 34 might want a backup plan to move indoors. Of course, this is a broad generalization to make on behalf of every North American location in the dataset. But, for our purposes and in general, it is a good understanding of our dataset to have. For our last visualization, let's plot lightning strikes by quarter. For a visualization, it will be far easier to work with numbers in millions, such as 25.2 million rather than 25 million, 154,365, for example. Let's create a column that divides the total number of strikes by 1 million. We do this by typing df by quarter, and entering the relevant column in the arguments field. In this case, we want number of strikes. Next, we add on .div to get our division function. Lastly, for the argument field, we enter 1 million. When we run this cell, we have a column that provides the number of lightning strikes in millions. Next, we'll group the number of strikes by quarter using the group by and reset index functions. This code divides the number of strikes into quarters for all three years. Each number is rounded to the first decimal. The letter m represents 1 million. As you'll soon discover, this calculation will help with the visualization. You'll learn more about these functions in another video. We will plot our chart using the same format as before. We'll use the plt.bar with our x being from our df by quarter data frame with quarter in the argument field. For the height, we put the number of strikes column in the argument field. It would be helpful if each quarter had the total lightning strike count at the top of each bar. To do that, we need to define our own function which we will call add labels. Let's type add labels then input our two column axes quarter and number of strikes separated by columns and brackets. At the end, we use the format we created earlier number of strikes formatted to label the number of strikes by quarter. To finish the bar chart, we label the x and y axes and add the title. Before we show the data visualization, there are a few small things we want to add just to make it more friendly to read. Let's set our length and height to 15 by 5. Next, let's make the bar labels cleaner by defining those numbers and centering the text. Our bar chart now gives us the number of strikes by quarter from 2016 to 2018. To make the information easier to digest, let's do one more visualization. Here is the code for plotting a bar chart that groups the total number of strikes year over year by quarter. Review the code carefully and consider what each function and argument does in order to create this final polished bar chart. Each year is assigned its own color to highlight the differences in quarters. And now we have our chart. Coming up, you'll learn more about the different methods for structuring data. I'll see you there. As a data analytics professional, you will often need to learn more about your data sets. This is where the structuring practice of EDA can help. Let's explore the valuable methods that are part of this practice. As you'll recall from earlier in the program, structuring helps you to organize, gather, separate, group, and filter your data in different ways to learn more about it. Next, we'll talk about the methods involved in structuring, and later you'll practice these concepts using Python. First on the list of structuring methods is sorting. Sorting is the process of arranging data into meaningful order. Imagine that you're given a data set and these furry creatures are native to Australia and Papua New Guinea. They're known for their strong tails and belly pouch they use to cradle their babies called joeys. The kangaroo data set contains information about kangaroo characteristics like pouch size, tail length, total body length, and much, much more. The first data we'll consider is a data column measuring the volume of the kangaroo pouches in cubic centimeters. We can sort those values in ascending or descending order from biggest to smallest or smallest to biggest. Another useful structuring tool is extraction. Extracting is the process of retrieving data from a data set or source for further processing. You can think of extraction as retrieving whole columns of data. An example of extraction is to take the kangaroo data from before, then evaluate just two of the columns from the data set such as pouch volume and tail length. You can use the resulting data for analysis comparisons or visualization. Another structuring method is filtering. Filtering is the process of selecting a smaller part of your data set based on specified parameters and using it for viewing or analysis. You can think of filtering as selecting rows of a data set. In the case of our kangaroo data set, filtering can look like viewing only the kangaroo pouches of kangaroos that also have tails shorter than one meter. This is useful in finding meaningful groups or trends in the data. Next on the list of structuring methods is slicing. Slicing breaks information down into smaller parts to facilitate efficient examination and analysis from different viewpoints. Think of slicing as an either or both options for columns and rows. A combination of extraction and filtering. In the kangaroo data set, let's say you have a column of their body length called total body length. In another column, you have the kangaroos identified as one of the three different regional populations. If you were to take the body length of only one of the three regional populations, you would be pulling a slice of the data. Grouping is our next structuring method. Grouping, sometimes called bucketizing, is aggregating individual observations of a variable into groups. An example of grouping is to add a new column called total body length next to the kangaroo tail length column. Then group all the tail lengths into three types. Long, average, and short based on the measurements in the tail length column. You can now find and organize the total body length values based on the kangaroo tail length groups. The last structuring method is merging. Merging is a method to combine two different data frames along a specified starting column. For example, imagine we had an additional data set of kangaroo information from a different field study but with the same parameters and variables. We might use the merge or join functions to align the columns and combine the new data into one data set. It's essential that you do not change the meaning of the data while performing your filtering, sorting, slicing, joining, and merging operations. For example, we did not merge the kangaroo pouch measurements correctly with their matching kangaroo name and ID. The data would not be representative and our analysis would be far less than useful. Being true to the data is being true to its story. Hopefully, you are beginning to understand the value of organizing and structuring data in order to analyze it. Coming up, we will practice structuring Python. You have learned about how structuring data can help professionals analyze, understand, and learn more about their data. Now, let's use a Python notebook and discover how it works in practice. We will continue using our NOAA lightning strike data set. For this video, we will consider the data for 2018 and use our structuring tools to learn more about whether lightning strikes are more prevalent on some days than others. Before we do anything else, let's import our Python packages and libraries. These are all packages and libraries you're familiar with. Pandas, NumPy, Seaborn, DateTime, and Matplotlib.pyplot. For a quick refresher, let's convert our Date column to DateTime and take a look at our column headers. We do this to get our dates ready for any future string manipulation we may want to do and to remind us of what is in our data. As you remember, there are three columns in the data set. Date, number of strikes, and center point geom, which you'll find after running the head function. Next, let's learn about the shape of our data by using the df shape function. When we run the cell, we get 3,401,000 and 12 comma 3. Take a moment to picture the shape of this data set. We're talking about only three columns wide and nearly three and a half million cells vertically. That's incredibly long and thin. We'll use a function for finding any duplicates. When we enter df.drop underscore duplicates with an empty argument field followed by dot shape, the notebook will return the number of rows and columns remaining after duplicates are removed. Because this returns the same exact number as our shape function, we know there are no duplicate values. Let's discuss some of those structuring concepts we learned about earlier. Let's start with sorting. We'll sort the number of strikes column in descending value or most to least. While we do this, let's consider the dates with the highest number of strikes. We'll input df sort underscore values. Then in the argument field type by, then the equal sign. Next, we input the column we want to sort, number of strikes followed by ascending equal sign false. If we add the head function to the end, the notebook outputs the top 10 cells for us to analyze. We find that the highest number of strikes are in the lower 2000s. That does seem like a lot of lightning strikes in just one day, but given that it happened in August when storms are likely, it is probable these 2000 plus strikes were counted during a storm. Next, let's look at the number of strikes based on the geographic coordinates latitude and longitude. We can do this by using the value underscore counts function. We type in df followed by the center point geom. Then we type in dot value underscore counts with an empty argument field. Based on our result, we learn that the locations with the most strikes have lightning on average every one in three days with numbers in the low 100s. Meanwhile, some locations are reporting only one lightning strike for the entire year of 2018. We also want to learn if we have an even distribution of values or whether 108 is a notably high value for lightning strikes in the U.S. To do this, copy the same value counts function but input a colon 20 in the brackets so that you can see the first 20 lines. The rest of the coding here is to help present the data clearly. We rename the access an index to unique values and counts, respectively. Lastly, we'll add a gradient background to the counts column for visual effect. After running the cell, we discover zero notable drops in lightning strike counts among the top 20 locations. This suggests that there are zero notably high lightning strike data points and that the data values are evenly distributed. Next, let's use another structuring method grouping. You'll often find stories hidden among different groups in your data like the most profitable times of day for a retail store, for instance. For this data set, one useful grouping is categorizing lightning strikes by day of week which will tell us whether any particular day has fewer or more lightning strikes than others. Let's first create some new data columns. We create a column called week by inputting df. date. dt. ISO calendar. Let's leave the argument field blank and add a .week at the end. This will create a column assigning numbers 1 to 52 for each of the days in the year 2018. Let's also add a column that names the weekday. Type in df. date. dt. day underscore name, leaving the argument field blank. For this last part, let's input df.head again. For the dates now have week numbers and assigned weekdays. We have some new columns, so let's group the number of strikes by weekday to determine whether any particular day of week has more lightning strikes than others. Let's create a data frame with just the weekday and number of lightning strikes. We'll do this by inputting df. weekday. We'll add a number of strikes by weekday and number of strikes by day of the week. Both in single quotes followed by more double brackets. Next, we'll add one of our structuring functions grouped by, followed by weekday. mean within the argument fi