is interviews. And that's not because it's the most common, but because it's the one you would do for the most basic problem. Now, basically, an interview is nothing more than a conversation with another person or a group of people. And the fundamental question is, why do interviews as opposed to doing a survey or something else? Well, there's a few good reasons to do that. Number one, you're working with a new topic, and you don't know what people's responses will be how they'll react. And so you need something very open ended. Number two, you're working with a new audience, you don't know how they will react in particular to what it is you're trying to do. And number three, something's going on with the current situation, it's not working anymore. And you need to find what's going on and you need to find ways to improve the open ended information where you get past your existing categories and boundaries can be one of the most useful method for getting that data. If you want to put it another way, you want to do interviews when you don't want to constrain responses. Now, when it comes to interviews, you have one very basic choice. And that's whether you do a structured interview. And with a structured interview, you have a predetermined set of questions. And everyone gets the same questions in the same order, it gives a lot of consistency, even though the responses are open ended. And then you can also have what's called an unstructured interview. And this is a whole lot more like a conversation where you as the interviewer and the person you're talking to, your questions arise in response to their answers. Consequently, an unstructured interview can be different for each person that you talk to. Also, interviews are usually done in person, but not surprisingly, they can be done over the phone or often online. Now, a couple of things to keep in mind about interviews. Number one is time interviews can range from just a few minutes to several hours per person. Second is training interviewing to special skill that usually requires specific training. Now, asking the questions is not necessarily the hard part. The really tricky part is the analysis. The hardest part of interviews by far is analyzing the answers for themes and way of extracting the new categories and the dimensions that you need for your further research. The beautiful thing about interviews is that you allow you to learn things that you never expected. So in some interviews are best for new situations or new audiences. On the other hand, they can be time consuming. And they also require special training, both to conduct the interview, but even more to analyze the highly qualitative data that you get from them. An interesting topic in data sourcing when you're making data is card sorting. Now, this isn't something that comes up very often in academic research, but in web research, this can be a really important method. Think of it as what you're trying to do is like building a model of a molecule here, you're trying to build a mental model or a model of people's mental structures, put more specifically, how do people organize information intuitively? And also, how does that relate to the things that you're doing online?  And you can do this physically with like three by five cards, or there's a lot of programs that allow you to do a digital version of it. Then what you do is you give this information to a group of respondents and the people sort those cards, so they put similar topics with each other, different topics over here and so on. And then you take that information and from that you're able to calculate what's called dissimilarity data. Think of it as like the distance or the difference between various topics. And that gives you the raw data to analyze how things are structured. Now, there are two very general kinds of card sorting tasks. They're generative and there's a evaluative. A generative card sorting task is one in which respondents create their own sets, their own piles of cards, using any number of groupings they like. And this might be used, for instance, to design a website. If people are going to be looking for one kind of information next to another one, then you want to put that together on the website so they know where to expect it. On the other hand, if you've already created a website, then you can do an evaluative card sorting. This is where you have a fixed number or fixed names of categories, like, for instance, the way you've set up your menus already. And then what you do is you see if people naturally put the cards into these various categories that you've created. That's a way of verifying that your hierarchical structure makes sense to people. Now, whichever method you do generative or evaluative, what you end up with when you do a card structure is an interesting kind of visualization. It's called a dendrogram that actually means branches. And what we have here is actually 150 data points. If you're familiar with the Fisher's Iris data, that's what's going on here. And it groups it from one giant group on the left and then splits it in pieces and pieces and pieces until you end up with lots of different observed, well, actually individual level observations at the end, but you can cut things off into two or three groups or wherever it's most useful for you here as a way of visualizing the entire collection of similarity or dissimilarity between the individual pieces of information that you had people sort. Now, I'll just mention very quickly, if you want to do digital card sorting, which makes your life infinitely easier because keeping track of physical cards is really hard. You can use something like Optimal Workshop or User Zoom or UX Suite. These are some of the most common choices. Now, let's just sum up what we've learned about card sorting in this extremely brief overview. Number one, card sorting allows you to see intuitive organization of information in a hierarchical format. You can do it with physical cards or you also have digital choices for doing the same thing.  When you're doing your data sourcing and you're making data, sometimes you can't get what you want through the easy ways and you got to take the hard way. And you can do what I'm calling laboratory experiments. Now, of course, when I mentioned laboratory experiments, people start to think of stuff like, you know, Dr. Frankenstein in his lab, but lab experiments are less like this. And in fact, they're a little more like this. Nearly every experiment I have done in my career has been a paper and pencil one with people in a well-lighted room. And it's not been the threatening kind. Now, the reason you do a lab experiment is because you want to determine cause and effect. And this is the single most theoretically viable way of getting that information. Now, what makes an experiment an experiment is the fact that researchers play active roles in experiments with manipulations.  All that means is you are manipulating the situation you're causing something to be different for one group of people or one situation than another. It's a benign thing. But it allows you to see how people react to those different variations. Now, you're going to want to do an experiment, you're going to want to have focused research, it's usually done to test one thing or one variation at a time. And it's usually hypothesis driven. Usually don't do an experiment until you've done enough background research to say, I expect people to react this way to the situation and this way to the other. A key component of all of this is that experiments almost always have random assignments. So regardless of how you got your sample, when they're in your study, you randomly assign them to one condition or another. And what that does is it balances out the preexisting differences between groups. And that's a great way of taking care of confounds and artifacts, the things that are unintentionally associated with differences between groups that provide alternate explanations for your data. If you've done good random assignment and you have a large enough people, then those confounds and artifacts are basically minimized. Now, some places where you're likely to see a laboratory experiments in this version are for instance, eye tracking and web design.  That's how we know for instance, that people don't really look at ads on the site of web pages. Another very common place is research and medicine and education. And in my field psychology, and in all of these, what you find is that experimental research is considered the gold standard for reliable valid information about cause and effect. On the other hand, while it's a wonderful thing to have, it does come at a cost. Here's how that works. Number one, experimentation requires extensive specialized training. It's not a simple thing to pick up. Two, experiments are often very time consuming and labor intensive. I've known some that take hours per person. And number three, experiments can be very expensive. So what that all means is you want to make sure that you've done enough background research, and you need to have a situation where it's sufficiently important to get really reliable cause and effect information to justify these costs for experimentation. In some laboratory experimentation is generally considered the best method for causality or assessing causality. That's because it allows you to control for confounds through randomization. On the other hand, it can be difficult to do. So be careful and thoughtful when considering whether you need to do an experiment and how to actually go about doing it. There's one final procedure I want to talk about in terms of data sourcing and making new data. It's a form of experimentation and it's simply called AB testing. And it's extremely common in the web world. So for instance, I just barely grabbed a screenshot of amazon.com's homepage. And you've got these various elements on the homepage. And I just noticed, by the way, when I did this, that this woman is actually a animated gift. So she moves around. That was kind of weird, never seen that before. But the thing about this is this entire layout, how things are organized and how they're on there will have been determined by variations on AB testing by Amazon. Here's how it works. For your web page, you pick one element like, what's the headline or what are the colors or what's the organization or how do you word something? And you create multiple versions, maybe just to version A and version B, which is why he called AB testing. And then when people visit your web page, you randomly assign those visitors to one version or another, you have software that does that for you automatically. And then you compare the response rates on some response, I'll show you those in a second. And then once you have enough data, you implement the best version, you sort of set that one solid and then you go on to something else. Now, in terms of response rates, there's a lot of different outcomes you can look at. You can look at how long a person's on a page, you can actually do mouse tracking if you want to. You can look at click throughs, you can also look at shopping cart value and or abandonment, a lot of possible outcomes. All of these contribute through AB testing to the general concept of website optimization, to make your website as effective as it can possibly be. Now, the idea also is that this is something you're going to do a lot. You can perform AB tests continually. In fact, I've seen one person say that what AB testing really stands for is always be testing, kind of cute. But it does give you the idea that improvement is a constant process. Now if you want some software to do AB testing to the most common choices are optimistically, and VWO which stands for visual web optimizer. Now, many others are available, but these are especially common. And when you get the data, you're going to use statistical hypothesis testing to compare the differences or really the software does it for you automatically. But you may want to adjust the parameters because most software packages cut off testing a little too soon and the information is not quite as reliable as it should be. But in some, here's what we can say about AB testing is a version of website experimentation done online, which makes it really easy to get a lot of data very quickly. It allows you to optimize the design of your website for whatever outcome is important to you. And it can be done as a series of continual assessments, testing and development to make sure that you're accomplishing what you want to as effectively as possible for as many people as possible. The next logical step in data sourcing and making data is surveys. Now think of this, if you want to know something, just ask, that's the easy way. And you want to do a survey under certain situations. The real question is, do you know your topic and your audience well enough to anticipate their answers to know what the range of their answers and the dimensions and the categories that are going to be important? If you do, then a survey might be a good approach. Now, just as there were a few dimensions for interviews, there's a few dimensions for surveys, you can do what's called a closed ended survey. That's also called a forced choice. It's where you give people just particular options like a multiple choice. You can have an open ended survey where you have the same questions for everybody, but you allow them to ride in a free form response. You can do surveys in person. And you can also do them online or over the mail or phone or however. And now it's very common to use software when doing surveys, some really common applications for online surveys are survey monkey and Qualtrics or at the very simple end, there's Google Forms, and at the simple and pretty end, there's type form. There's a lot more choices, but these are some of the major players and how you can get data from online participants in survey format. Now, the nice thing about surveys is, you know, they're really easy to do, they're very easy to set up, and they're really easy to send out to large groups of people, you can get tons of data really fast. On the other hand, the same way that they're easy to do, they're also really easy to do badly. The problem is that the questions you ask, they can be ambiguous, they can be double barreled, they can be loaded. And the response scales can be confusing. So if you say, I never think this particular way in a person put strongly disagree, they may not know exactly what you're trying to get at. So you have to take special effort to make sure that the meaning is clear, unambiguous, and that the rating scale, the way that people respond is very clear, and they know where their answer falls, which gets us into one of the things about people behaving badly. And that is, beware the push pull. Now, especially during election time, like we're in right now, a push pull is something that sounds like a survey. But really what it is, is a very biased attempt to get data just fodder for social media campaigns, or I'm going to make a chart that says that 98% of people agree with me. A push pull is one that's so biased, there's really only one way to answer to the questions. This is considered extremely irresponsible and unethical from a research point of view. Just hang up on them. Now, aside from that egregious violation of research ethics, you do need to do other things like watch out for bias in the question wording, in the response options, and also in the sample selection, because any one of those can push your responses off one way or another, without you really being aware that it's happening. So in some let's say this about surveys, you can get lots of data quickly. On the other hand, requires familiarity with the possible answers in your audience. So you know sort of what to expect. And no matter what you do, you need to watch for bias to make sure that your answers are going to be representative of the group that you're really concerned about understanding. The very last thing I want to talk about in terms of data sourcing is to talk about the next steps. And probably the most important thing is, you know, don't just sit there. I want you to go and see what you already have, try to explore some open data sources. And if it helps, check with a few data vendors. And if those don't give you what you need to do your project, then consider making new data. Again, the idea here is get what you need and get going. Thanks for joining me and good luck on your own projects. Welcome to coding and data science. I'm Bart Polson. And what we're going to do in this series of videos is we're going to take a little look at the tools of data science. So I'm inviting you to know your tools, but probably even more important than that is to know their proper place. Now, I mention that, because a lot of the times when people talk about data tools, they talk about it as though that were the same thing as data science as though they were the same set. But I think if you look at it for just a second, that's not really the case. Data tools are simply one element of data science, because data science is made up of a lot more than the tools that you use. It includes things like business knowledge, it includes the meaning making and interpretation includes social factors. And so there's much more than just the tools involved. That being said, you will need at least a few tools. And so we're going to talk about some of the things that you can use in data science, if it works well for you. In terms of getting started, the basic things. Number one is spreadsheets, it's the universal data tool. And I'll talk about how they play an important role in data science. Number two is a visualization program called Tableau, there's Tableau public, which is free. And there's Tableau desktop, and there's also something called Tableau server. But Tableau is a fabulous program for data visualization. And I'm convinced for most people provides the great majority of what they need. And though, while it's not a tool, I do need to talk about the formats used in web data, because you have to be able to navigate that when doing a lot of data science work. Then we can talk about some of the essential tools for data science, those include the programming language R, which is specifically for data. There's the general purpose programming language Python, which has been well adapted to data. And there's the database language SQL or SQL for a structured query language. Then if you want to go beyond that, there are some other things that you can do. They're the general purpose programming languages C, C++ and Java, which are very frequently used to form the foundation of data science and sort of high level production code is going to rely on those as well. There's the command lined interface language bash, which is very common as a very quick tool for manipulating data. And then there's the sort of wild card supercharged, regular expressions or regex. We'll talk about all of these in separate courses. But as you consider all the tools that you can use, don't forget the 80 20 rule also known as the Pareto principle. And the idea here is that you're going to get a lot of bang for your buck out of a small number of things. And I'm going to show you a little sample graph here. Imagine that you have 10 different tools and we'll call them A through B. A does a lot for you B does a little bit less and it kind of tapers down to you've got a bunch of tools that do just a little bit of stuff that you need. Now, instead of looking at the individual effectiveness, look at the cumulative effectiveness, how much are you able to accomplish with a combination of tools? Well, the first ones right here at 60% where the tool started, then you add on the 20% from B and it goes up and then you add on C and D and you add up little smaller smaller pieces. And by the time you get to the end, you've got 100% of effectiveness from your 10 tools combined. The important thing about this is you only have to go to the second tool that's two out of 10. So that's B that's 20% of your tools. And in this made up example, you've got 80% of your output. So 80% of the output from 20% of the tools. That's a that's a fictional example of the Pareto principle. But I find in real life, it tends to work something approximately like that. And so you don't necessarily have to learn everything and you don't have to learn how to do everything in everything. Instead, you want to focus on the tools that will be most productive and specifically most productive for you. So in sum, let's say these three things. Number one, coding or simply the ability to manipulate data with programs and computers, coding is important. But data science is much greater than the collection of tools that's used in it. And then finally, as you're trying to decide what tools to use and what you need to learn and how to work, remember the 8020 rule, you're going to get a lot of bang from a small set of tools. So focus on the things are going to be most useful for you in conducting your own data science projects. As we begin our discussion of coding and data science actually want to begin with something that's not coding, I want to talk about applications or programs that are already created that allow you to manipulate data. And we're going to begin with the most basic of these spreadsheets. We're going to do the rows and columns and cells of Excel. And the reason for this is you need spreadsheets. Now, you may be saying to yourself, no, no, no, not me, because you know what, I'm fancy. I'm working in my big set of servers, I got fancy things going on. But you know what, you two fancy people, you need spreadsheets as well. There's a few reasons for this. Most importantly, spreadsheets can be the right tool for data science in a lot of circumstances. There are a few reasons for that. Number one, spreadsheets, they're everywhere, they're ubiquitous, they're installed on a billion machines around the world. And everybody uses them, they probably have more data sets in spreadsheets than anything else. And so it's a very common format. Importantly, it's probably your clients format. A lot of your clients are going to be using spreadsheets for their own data. I've worked with billion dollar companies that keep all of their data in spreadsheets. And so when you're working with them, you need to know how to manipulate that and how to work with it. Also, regardless of what you're doing, spreadsheets or specifically CSV comma separated value files, are sort of the lingua franca, the universal interchange format for data transfer to allow you to take it from one program to another. And then truthfully, in a lot of situations, they're really easy to use. And if you want a second opinion on this, let's take a look at this ranking. There's a survey of data mining experts, the Katie Nuggets data mining poll. And these are the tools they most use in their own work. And look at this lowly Excel is fifth on the list. And in fact, what's interesting about it, it's above Hadoop and Spark two of the major big data fancy tools. And so Excel really does have place of pride in a toolkit for data analysts. Now, since we're going to go into sort of the low tech end of things, let's talk about some of the things that you can do with a spreadsheet. Number one, they're really good for data browsing, you actually get to see all the data in front of you, which isn't true if you're doing something like our Python, they're really good for sorting data, sort by this column than this column than this column, they're really good for rearranging columns and cells and moving things around they're good for finding and replacing and seeing what happens so you know that it worked right. Some more uses they're really good for formatting, especially conditional formatting, they're good for transposing data switching the rows and the columns, they make that really easy. They're good for tracking changes. Now it's true if you're a big fancy data scientist, you're probably using GitHub, but for everybody else in the world, spreadsheets and the tracking changes is a wonderful way to do it. You can make pivot tables that allows you to explore the data in a very hands on way, in a very intuitive way. And they're also really good for arranging the output for consumption. Now, when you're working with spreadsheets, however, there's one thing you need to be aware of they're really flexible, but that flexibility can be a problem. And that when you're working in data science, you specifically want to be concerned about something called tidy data. That's a term I borrowed from Hadley Wickham, very well known developer in the our world. Tidy data is for transferring data and making it work well. There's a few rules here that undo some of the flexibility inherent in spreadsheets. Number one, what you want to do is have a column be equivalent to the same thing as a variable columns variables, they are the same thing. And then rows are equal, exactly the same thing as cases. And then you have one sheet per file, and then you have one level of measurement, say individual, then organization, then state per file. Again, this is undoing some of the flexibility that's inherent in spreadsheets. But it makes it really easy to move the data from one program to another. Let me show you how all this works. You can try this in Excel. If you've downloaded the files for this course, we simply want to open up this spreadsheet. Let me go to Excel and show you how it works. So when you open up the spreadsheet, what you get is totally fictional data here that I made up, but it's showing sales over time of several products at two locations, like if you're selling stuff at a baseball field. And this is the way spreadsheets often appear, we got blank rows and columns, we got stuff arranged in a way that makes it easy for the person to process it. And we got totals here. And with formulas, putting them all together. And that's fine. That works well for the person who made it. And then that's for one month. And then we have another month right here and we have another month right here. And then we combine them all for the first quarter of 2014. We've got some headers here, we've got some conditional formatting and changes. And if we come to the bottom, we've got a very busy line graphic that eventually loads. It's not a good graphic, by the way. But similar to what you will often find. So this is the stuff that well, it may be useful for the client's own personal use, you know, you can't feed this into our Python, it'll just choke and it won't know what to do with it. And so you need to go through a process of tidying up the data. And what this involves is undoing some of this stuff. So for instance, here's data that is almost tidy. Here we have a single column for the date, a single column for the day, a column for the site that so we have two locations, A and B. And then we have six columns for the six different things that are sold and how many were sold on each day. Now, in certain situations, you would want the data laid out exactly like this. If you're doing, for instance, the time series, you'll do something vaguely similar to this. But for true tidy stuff, we're going to collapse it even further. Let me come here to the tidy data. And now what I've done is I've created a new column that says what is the item being sold. And so by the way, what this means is that we've got a really long data set. Now it's got over 1000 rows, come back up to the top here. But what that shows you is that now it's a in a format that's really easy to import from one program to another. That makes it tidy. And you can reminipulate it however you want, once you get to each of those. So let's sum up our little presentation here in a few lines. Number one, no matter who you are, no matter what you're doing in data science, you need spreadsheets. And the reason for that is that spreadsheets are often the right tool for data science. Keep one thing in mind, though, and that is as you're moving back and forth from one language to another, tidy data or well formatted data is going to be important for exporting data into your analytical program or language of choice. As we move through coding and data science and specifically the applications that can be used, there's one that stands out for me more than almost anything else. And that's Tableau and Tableau public. Now, if you're not familiar with these, these are visualization programs. The idea here is that when you have data, the most important thing you can do is to first look and see what you have and work with it from there. And in fact, I'm convinced that for many organizations, Tableau might be all that they really need, it will give them the level of insight that they need to work constructively with data. So let's take a quick look by going to Tableau.com. Now there are a few different versions of Tableau. Right here, we have Tableau desktop and Tableau server. And these are the paid versions of Tableau. They actually cost a lot of money unless you work for a nonprofit organization, in which case you can get them for free, which is a beautiful thing. What we're usually looking for, however, is not this paid version, but we're looking for something called Tableau public. And if you come in here and go to products, and we've got these three paid ones over here to Tableau public, when we click on that, it brings us to this page, it's public tableau.com. And this is the one that has what we want. It's a free version of Tableau with one major caveat, you don't save files locally to your computer, which is why I didn't give you a file to open. Instead, it saves them to the web in a public form. So if you're willing to trade privacy, you can get an immensely powerful application for data visualization. That's a catch for a lot of people, which is why people are willing to pay a lot of money for the desktop version. And again, if you work for a nonprofit, you can get the desktop version for free. But I'm going to show you how things work in Tableau public. So that's something that you can work with personally. The first thing you want to do is you want to download it. And so you put in your email address, you download, it's going to know what you're on to pretty big download. And once it's downloaded, you can install and open up the application. And here I am in Tableau public right here, this is the blank version. By the way, you also need to create an account with Tableau in order to save your stuff online and to see it. We'll show you what that looks like. But you're presented with a blank thing right here. And the first thing you need to do is you need to bring in some data. I'm going to bring in an Excel file. Now if you've downloaded the files for the course, you'll see that there's this one right here, DSO three, two, two Tableau public dot Excel sx, it's a Excel file. And in fact, it's the one that I used in talking about spreadsheets in the first video on this course. I'm going to select that one. And I'm going to open it. And a lot of programs don't like bringing in Excel because it's got all the worksheets and all the weirdness in it. This one works better with it. But what I'm going to do is I'm going to take the tidy data. By the way, you see that it put them in alphabetical order here. And I'm going to take tiny data, I'm just going to drag it over to let it know that it's the one that I want. And now what it does is it shows me a version of the data set, along with things that you can do here, you can rename it, you can I like you can create bin groups, there's a lot of things that you can do here. I'm going to do something very, very quick with this particular one. Now, I've got the data set right here, what I'm going to do now is I'm going to go to a worksheet. That's where you actually create so cancel that and go to worksheet one. Okay, this is a drag and drop interface. And so what we're going to do is we're going to pull the bits and pieces of information we want to make graphics. There's immense flexibility here. I'm going to show you two very basic ones. I'm going to look at the sales of my fictional ballpark items. So I'm going to grab sales right here. And I'm going to put that as the field that we're going to measure. Okay, and you see it put it down right here. And this is our total sales. We're going to break it down by item and by time. So let me take item right here, and you can drag it over here or I can put it right up here into rows. Those will be my rows. And that's how many we've sold total of each of the items. Fine, that's really easy. And then let's take date and we'll put that here in columns to spread it across. Now by default, this unit by year, I don't want to do that only have three months of data. And so what I can do is I can click right here, and I can choose a different timeframe. I can go to quarter, but that's not going to help because I only have one quarter's worth of data. That's three months. I'm going to come down to week. Actually, let me go to day.  So that's no good. So I'm going to back up to week. And I've got a lot of numbers there. But what I want is a graph. And so to get that, I'm going to come over here and click on this and tell it that I want to graph. And so we're seeing the information, except it lost item. So I'm going to bring an item and I'm going to put it back up into this graph to say this is a row for the data. And now I've got rows for sales by week for each of my items. That's great. I want to break it down one more by putting in the site, the place that it sold. So I'm going to grab that. And I'm going to put it right over here. And now you see I've got it broken down by the item that is sold and the different sites. And I'm going to color the sites and all I got to do to that is I'm going to grab site and drag it onto color. Now I've got two different colors for my sites. And this makes it a lot easier to tell what's going on. And in fact, there's some other cool stuff you can do.  And I can tell it for instance to put an average line through everything. So I'll just drag this over here. Say, now we have the average for each line. That's good. And I can even do forecasting. Let me get a little bit of a forecast right here. I'll drag this on. And if you go over here, I can get this out of the way for a second. Now I have a forecast for the next few weeks. And that's a really convenient, quick and easy thing. And again, for some organizations, that might be all that they really need. And so what I'm showing you here is the absolute basic operation of Tableau, which allows you to do an incredible range of visualizations and manipulate the data and create interactive dashboards. There's so much to it. We'll show that in another course. But for right now, I want to show you one last thing about Tableau public. And that is saving the files. So now when I come here and save it, it's going to ask me to sign into Tableau public. Now I sign in and ask me how I want to save this same name as the video. There we go. And I'm going to hit save. And then that opens up a web browser. And since I'm already logged into my account, see, here's my account, my profile. Here's the page that I created. And it's got everything I need there. I'm going to edit just a few details. I'm going to say, for instance, I'm going to leave its name like that I could put more of a description in there if I wanted. I can allow people to download the workbook and its data. I'm going to leave that there so you can download it if you need to. If I had more than one tab, I would do this thing that says show the different sheets as tabs, hit save. And there's my data set. And also, it's published online and people can now find it. And so what you have here is an incredible tool for creating interactive visualizations. You can create them with drop down menus and you can rearrange things and you can make an entire dashboard. It's a fabulous way of presenting information. And as I said before, I think that for some organizations, this may be as much as they need to get really good useful information out of their data. And so I strongly recommend that you take some time to explore with Tableau, either the paid desktop version, or the public version, and see what you can do to get some really compelling and insightful visualizations out of your work in data science. For many people, their first experience of coding and data science is with the application SPSS. Now, I think of SPSS. And the first thing that comes my mind is sort of life in the ivory tower, though this looks more like, you know, Harry Potter. But if you think about it, the package name SPSS comes from statistical package for the social sciences, although if you ask IBM about it now, they'll act like it doesn't stand for anything. But it has its background in social science research, which is generally academic and truthfully, I'm a social psychologist, and that's where I first learned how to use SPSS. But let's take a quick look at their webpage, IBM.com slash SPSS. If you type that in, that'll just be a alias that'll take you to IBM's main webpage. Now IBM didn't create SPSS, but they bought it around version 16. And it was very briefly known as PASW predictive analytics software that only lasted briefly. And now it's back to SPSS, which is where it's been for a long time. SPSS is a desktop program. It's pretty big. It does a lot of things. It's very powerful. It's used in a lot of academic research. It's also used in a lot of business consulting, management, and even some medical research. And the thing about SPSS is it looks like a spreadsheet, but it has dropdown menus to make your life a little bit easier compared to some of the programming languages that you can use. Now, you can get a free temporary version.  Otherwise, SPSS costs a lot of money. But if you have it one way or the other, when you open it up, this is what it's going to look like. I'm showing SPSS version 22. Now it's currently on 24. And the thing about SPSS versioning is in any other software package, these would be point updates. So I sort of feel like we should be on 17.3 as opposed to 22 or 24 because the variations are so small that anything you learn from the earlier ones is going to work in the later ones. And there's a lot of backwards and forwards compatibility. So I'd almost say that this one, the version you have practically doesn't matter. You get this little welcome splash screen. And if you don't want to see it anymore, you can get rid of it. I'm just going to hit cancel here. And this is our main interface looks a lot like a spreadsheet. The difference is you have a separate pane for looking at variable information. And then you have separate windows for output and then an optional one for something called syntax. But let me show you how this works by first opening up a data set. SPSS has a lot of sample data sets in them, but they're not easy to get to and they're really well hidden. On my Mac, for instance, let me go to where they are. In my Mac, I go to the finder, I have to go Mac to applications to the folder IBM to SPSS to statistics to 22 the version number to samples. Then I have to say I want the ones that are in English. And then it brings them up. The dot SAV files are the actual data files. There are different kinds in here. So dot SPS is a different kind of file. And then we have a different one about planning analysis. So there are versions of it. I'm going to open up a file here called market values dot SAV, the small data set in SPSS format. And if you don't have that, you can open up something else. It really doesn't matter for now. By the way, in case you haven't noticed SPSS tends to be really, really slow when it opens. It also, despite being a version 24, tends to be kind of buggy and crashes. And so when you work with SPSS, you want to get in the habit of saving your work constantly, and also being patient when it's time to open the program. So here's a data set that just shows addresses and house values for and square feet for some information. This I don't even know if this is real information, it looks it looks artificial to me. But SPSS lets you do point and click analysis, which is unusual for a lot of things.  So I simply click values, put that right there and I'll put a normal curve on top of it and hit OK. And then it's going to open up a new window and it opened up microscopic version of it here. So I'm going to make that bigger. This is the output window. And so this is a separate window and it has a navigation pane here on the side. It tells me where the data came from and it saves the command here. And then, you know, there's my default histogram. And so we see most of the houses were right around one hundred and twenty five thousand dollars. And then they went up to at least four hundred thousand. I have a mean of two hundred and fifty six thousand the standard deviation of about eighty thousand and there's ninety four houses in the dataset. Fine, that's great. The other thing I can do is if I want to do some analyses, let me go back to the data just for a moment. For instance, I can come here to analyze and I can do descriptives. I'm actually going to do excuse me. I'm going to do one here called explore. And I'll take the purchase price and I'll put it right here and I'm going to get a whole bunch of stuff just by default. I'm going to hit OK. And it goes back to the output window. Once again, made it tiny. And so now you see beneath my chart, I now have a table and I've got a bunch of information to stem and leaf plot and I've got a box plot to great way of checking for outliers. And so this is a really convenient way to save things. You can export this information as images. You can export the entire file as an HTML. You can do it as a PDF or a PowerPoint. There's a lot of options here and you can customize everything that's on here. Now, I just want to show you one more thing that makes your life so much easier in SPSS. You see right here that it's putting down these commands. It's actually saying graph and then histogram normal equals value. And then down here, we've got this little command right here. Most people don't know how to save their work in SPSS and they kind of just have to do it over again every time. But there's a very simple way to do this.  I'm going to go to new syntax. And this is just a blank window that's a programming window. It's for saving code. And let me go back to my analysis I did a moment ago. I'll go back to analyze. I can still get at it right here. And descriptives and Explorer and my information is still there. And what happens here is even though I set it up with drop down menus and point of click if I do this thing paste then what it does is it takes the code that creates that command and it saves it to the syntax window. And this is just a text file. It saves it as dot SPS. But it's a text file that can be open in anything. And what's beautiful about this is it's really easy to copy and paste. And you can even take this into like Word and do find and replace on it. And it's really easy to replicate the analysis. And so for me, SPSS is a good program. But until you use syntax, you don't know the true power of it and it makes your life so much easier as a way of operating it. Anyhow, this is my extremely brief introduction to SPSS. All I want to say is that it's a very common program. Kind of looks like a spreadsheet, but it gives you a lot more power and options and you can use both drop down menus and text based syntax commands as well to automate your work and make it easier to replicate it in the future. I want to take a look at one more application for coding and data science that's called JASP. This is a new application, not very familiar to a lot of people and still in beta, but with amazing promise. You can basically think of it as a free version of SPSS. And you know what? We love free. But JASP is not just free. It's also open source and it's intuitive and it makes analyses replicable. And it even includes Bayesian approaches. And so take that all together. You know, we're pretty happy and we're jumping for joy. So before we move on, you just may be asking yourself, you know, JASP, what is that? Well, the creators emphatically deny that it stands for just another statistics program. But be that as it may. We'll just go ahead and call it JASP and use it very happily. You can get to it by going to JASP dash stats dot org. And let's take a look at that right now. JASP is a new program.  You're going to want to download it. Best of find your platform. It even comes in Linux format, which is beautiful. And again, it's beta. So stay posted. Things are updating regularly. And if you're on Mac, you're going to need to use X Quartz. But that's an easy thing to install. It makes a lot of things work better. And it's a wonderful way to do analyses. When you open up JASP, it's going to look like this. It's a pretty blank interface, but it's really easy to get going with it. So for instance, you can come over here to file and you can even choose some example data sets. So for instance, here's one called Big Five. That's personality factors. And you've got data here. That's really easy to work with. Let me scroll this over here for a moment. So there's our five variables. And let's do some quick analyses with these. Say, for instance, we want to get descriptives. We can pick a few variables. Now, if you're familiar with SPSS, the layout feels very much the same and the output looks a lot the same.  And then I can choose additional statistics. I can get quartiles. I can get the median. And you can choose plots. Let's get some plots. All I do is click on it and they show up. And that's a really beautiful thing. And you can modify these things a little bit. So for instance, I can take the plots and let's see if I can drag that down. And if I make it small enough, I can see the five plots. Well, I went a little too far on that one. Anyhow, you can do a lot of things here and I can hide this. I can collapse that and I can go on and do other analyses. Now, what you really need, though, is when you navigate away from it. So I just clicked in the blank area of the results pane or back to the data here. But if I click on one of these tables like this one right here, it immediately brings up the commands that produced it and I can just modify it some more if I want. Say I want skewness and kurtosis. Boom, they're in there. It's an amazing thing. And then I can come back out here. I can click away from that and I can come down to the plots, expand those. And if I click on that, it brings up the commands that made them. It's an amazingly easy and intuitive way to do things. Now, there's another really nice thing about JASP, and that is that you can share the information online really well through a program called osf.io. That stands for the Open Science Foundation. That's its web address osf.io. So let's take a quick look at what that's like. Here's the Open Science Framework website. And it's a wonderful service. It's free and it's designed to support open, transparent, accessible, accountable, collaborative research. And I really can't say enough nice things about it. What's neat about this is once you sign up for OSF, you can create your own area. And I've got one of my own. I'll go to that right now. So for instance, here's the data lab page in Open Science Framework. And what I've done is I created a version of this JASP analysis and I've saved it here. In fact, let's open up my JASP analysis in JASP. And then I'll show you what it looks like in OSF. So let's first go back to JASP. And when we're here, we can come over to File and click Computer. And I just saved this file to the desktop. Look on desktop. And you should have been able to download this with all the other files. DSO324JASP. I'm going to double click on that to open it. And now it's going to open up a new window. And you see I was working with the same data set, but I did a lot more analyses. I've got these graphs. I have correlation scatter plots. Come down here. I did a linear regression. And we just click on that and you can see the commands that produced it as well as the options. Didn't do anything special for that, but I did do some confidence intervals and specified that. And it's really a great way to work with all this. I'll click back in an empty area and you can see the commands go away. And so I've got my output here in JASP. When I saved it, though, I had the option of saving it to OSF. In fact, if you go to this web page, osf.io slash 3T2JG, you'll actually be able to go to a page where you can see and download the analysis that I conducted. Let's take a look. This is that page. There's the address I just barely gave you. And what you see here is the same analysis that I conducted. It's all right here. So if you're collaborating with people or if you want to show things to people, this is a wonderful way to do it. Everything's right there. Now this is a static image. But up at the top, people have the option of downloading the original file and working with it on their own. So in case you can't tell, I'm really enthusiastic about JASP and about its potential, still in beta, still growing rapidly. I see it really as an open source, free and collaborative replacement SPSS. And I think it's going to make data science work so much easier for so many people. I strongly recommend you give JASP a close look. Let's finish up our discussion of coding and data science, the applications part of it by just briefly looking at some other software choices. And I'll have to admit, it gets kind of overwhelming because there are just so many choices. Now, there's this in addition to the spreadsheets and Tableau and SPSS and JASP that we've already talked about. I mean, there's so much more than that. I'm going to give you a range of things that I'm aware of. And I'm sure I've left out some important ones or things that other people like really well. But these are some common choices and some less common but interesting ones. Number one, in terms of things that I did not mention is SAS. SAS is an extremely common analytical program, very powerful, used for a lot of things. It's actually the first program that I learned. And on the other hand, it can be kind of hard to use and it can be expensive. But there's a couple of interesting alternatives. SAS also has something called the SAS University Edition. If you're a student, this is free. And it's slightly reduced in what it does. But the fact that it's free and also it runs in a virtual machine, which makes it an enormous download. But it's a good way to learn SAS if it's something that you want to do. SAS also makes a program that I really love where it's not so extraordinarily expensive. And that is called jump. And it's a visualization software. Think a little bit of like Tableau, how we saw you work with it visually. And this one you can drag things around. It's a really wonderful program. I personally find it prohibitively expensive. Another very common choice among working analysts is Stata. And some people use Minitab. Now for mathematical people, there's Matlab. And then of course there's Mathematica itself. But that's really more a language than a program. On the other hand, Wolfram, who makes Mathematica, is also the people who give us Wolfram Alpha. Most people don't think of this as a stats application because you can run it on your iPhone. But Wolfram Alpha is in fact, incredibly capable. And especially if you pay for the pro account, you can do amazing things in this, including analyses, regression models, visualizations. And so it's worth taking a little closer look at that also, because it actually provides a lot of the data that you need. So Wolfram Alpha is an interesting one. Now, several applications that are more specifically geared towards data mining.  But there's rapid miner. And there's nine and orange. And those are all really nice to use because they are control languages where you drag nodes onto a screen and you connect them with lines and you can see how things run through. All three of them are free or have free versions and all three of them work in pretty similar manners. There's also big ML, which is for machine learning. And this is unusual because it's a browser based, it runs on their servers. There's a free version, although you can't download a whole lot, doesn't cost a lot to use big ML, and actually is a very friendly, very accessible program. Then in terms of programs, you can actually install for free on your own computer. There's one called SOFA statistics. That means statistics open for all kind of a cheesy title, but it's a good program. And then one with a web page straight out of 1990 is past three. This is paleontological software. On the other hand, does do very general stuff. It runs on many platforms and it's a really powerful thing and it's free, but it is relatively unknown. And then speaking of relatively unknown, one that's near and dear to my heart is a web application called Stat Crunch. It costs but it costs like six or 12 bucks a year. It's it's really cheap and it's very good, especially for basic statistics and for learning. I used it in some of the classes that I was teaching. And then if you're deeply wedded to Excel and you just can't stand to leave that environment, you can purchase add-ins like Excel stat, which give you a lot of statistical functions within the Excel environment itself. That's a lot of choices. And the most important thing here is don't get overwhelmed. There's a lot of choices, but you don't even have to try all of them. Really, the important question is what works best for you and the projects that you're working on. There's a few things you might want to consider in that regard. First off is functionality. Does it actually do what you want? Or does it even run on your machine? You don't need everything that a program can do. I mean, think about all the stuff that Excel can do. People probably use one 5% of what it's available. Then there's also ease of use. Some of these programs are a lot easier to use than the others. And I personally find that the ones that are easy to use, I like them. And so you might say, no, I need to program because I need to do custom stuff. But I'm willing to bet that 95% on what people do does not require anything custom. Also, the existence of a community constantly, when you're working, you come across problems, don't know how to solve it and being able to simply get online and do a search for an answer and have enough of a community that there are people there who have put answers up and discuss these things. Those are wonderful. Some of these programs have very substantial communities. Some of them it's practically non-existent. And you get to decide how important that is to you. And then finally, of course, there's the issue of cost. Many of these programs I mentioned are free. Some of them are very cheap. Some of them run on sort of a freemium model and some of them are outrageously expensive. So you don't buy them unless somebody else is paying for it. So these are some of the things that you want to keep in mind when you're trying to look at various programs. Also, let's mention this, don't forget the 80 20 rule, you're going to be able to do most of the stuff that you need to do with only a small number of tools. One or two, maybe three will probably be all that you ever need. So you don't need to explore the range of every possible tool. Find something that does what you need, find something you're comfortable with, and really try to extract as much value as you can out of that. So in some in our discussion of available applications for coding and data science.  And the single most important thing is remember, what works for you may work well for somebody else. If you're not comfortable with it, if it's not the questions you address, then it's more important to think about what works for you and the projects that you're working on, as you make your own choices for tools for working in data science. When you're coding in data science, one of the most important things you can do is be able to work with web data. And if you work with web data, you're going to be working with HTML. Now, in case you're not familiar with it, HTML is what makes the worldwide web go round. What it stands for is hyper text markup language. And if you've never dealt with web pages before, here's a little secret. Web pages are just text. It's just a text document, but it uses tags to define the structure of the document. And a web browser knows what those tags are and it displays them in the right way. So for instance, some of the tags, they look like this, they're in angle brackets, and you have angle bracket and then a beginning tag, so body, then you have the body, the main part of your text. And then you have an angle brackets and a backslash body to let the computer know that you're done with that part. You also have P and backslash P for paragraphs. H one is for header one, and you put it in between that text. TD is for table data or the cell in a table. And you mark it off that way. If you want to see what it looks like, just go to this document DSO three, three one HTML dot text. I'm going to go to that one right now. Now, depending on what text editor you open this up, it may actually give you the web preview. I've opened it up and textmate. And so it actually is showing the text the way I typed it. I typed this manually, just typed it all in there. And I have HTML to see what a document is, I have an empty header, but that sort of needs to be there. This I say what the body is. And then I have some texts, allies for list items. I have headers. This is for a link to a web page. Then I have a small table. And if you want to see what this looks like when it's actually displayed as a web page, we'll just go up here to window and show web preview. This is the same document, but now it's in a browser. And that's how you make a web page. Now, I know this is very fundamental stuff. But the reason this is important is because if you're going to be extracting data from the web, you have to understand how that information is encoded in the web and is going to be in HTML, most of the time for a regular web page. Now I will mention something that there's another thing called CSS and web pages use CSS to define the appearance of a document. HTML is theoretically there to give the content and CSS gives the appearance. And that stands for cascading style sheets. I'm not going to worry about that right now because we're really interested in the content. And now you have the key to being able to read web pages and pull data from web pages for your data science projects. So in some, first, the web runs on HTML, that's what makes the web pages that are there. HTML defines the page structure and the content that's in the page. And you need to learn how to navigate the tags and the structure in order to get data from the web pages for your data science projects. The next step in coding and data science when you're working with web data is to understand a little bit about XML. I like to think of this as the part of web data that follows the imperative data define thyself XML stands for extensible markup language. And what it is XML is semi structured data.  But unlike HTML, the tags are free to be defined any way you want. And so you have this enormous flexibility in there, but you're still able to specify so the computer can read it. Now there's a couple of places where you're going to see XML files. Number one is in web data. HTML defines the structure of a web page, but if they're feeding data into it, then that will often come in the form of an XML file. Interestingly, Microsoft Office files, if you have doc x or xls x, x part at the end stands for a version of XML that's used to create these documents. If you use iTunes, the library information that has all of your artists and your genres and your ratings and stuff, that's all stored in an XML file. And then finally data files that often go with particular programs can be saved as XML as a way of representing the structure of the data to the program. And for XML tags use opening and closing angle brackets just like HTML did. Again, the major difference is that you're free to define the tags however you want. So for instance, thinking about iTunes, you can define a tag that's genre, and you have the angle brackets and genre to begin that information. And then you have the angle brackets with the backslash to let it know you're done with that piece of information. Or you can do it for composer, or you can do it for rating, or you can do it for comments. And you can create any tags you want and you put the information in between those two things. Now let's take an example of how this works. I'm going to show you a quick data set that comes from the web. It's at airgast.com and API. This is a website that stores information about automobile Formula One racing. Let's go to this web page and take a quick look at what it's like. So here we are at airgast.com and it's the API for Formula One. And what I'm bringing up is the results of the 1957 season in Formula One racing. And here you can see who the competitors were in each race and how they finished and so on. So this is a data set that's being displayed in a web page. If you want to see what it looks like in XML, all you have to do is type XML onto the end of this dot XML. I've done that already. So I'm just going to go to that one. You see is only this little bit that I've added dot XML. Now it looks exactly the same because the web page is structuring XML data by default. But if you want to see what it looks like in its raw format, just do an option click on the web page and go to view page source. At least that's how it works in Chrome. And this is the structured XML page. And you can see we've got tags here. It says race name, circuit name, location. And obviously these are not standard HTML tags that are defined for the purposes of this particular data set. But we begin with one, we have circuit name right there. And then we close it using the backslash right there. And so this is structured data, the computer knows how to read it, which is exactly this is how it displays it by default. And so it's a really good way of displaying data. And it's a good way to know how to pull data from the web. You can actually use what's called an API, an application programming interface to access this XML data. And it pulls it in along with its structure, which makes working with it really easy. What's even more interesting is how easy it is to take XML data and convert it between different formats, because it's structured and the computer knows what you're dealing with. So for example, one, it's really easy to convert XML to CSV or comma separated value files, that's a spreadsheet format, because it knows exactly what the headings are, and what piece of information goes in each column. Example two, it's really easy to convert HTML documents to XML, because you can think of HTML with its restricted set of tags, that's sort of a subset of the much freer XML. And three, you can convert CSV or your spreadsheet comma separated value to XML vice versa, you can bounce them all back and forth because the structure is made clear to the programs that you're working with. So in some, here's what we can say. Number one, XML is semi structured data. What that means is it has tags to tell the computer what the piece of information is, but you can make the tags whatever you want them to be. And XML is very common for web data. And it's really easy to translate the formats XML, HTML, CSV, so on and so forth. It's really easy to translate them back and forth, which gives you a lot of flexibility in manipulating data. So you can get into the format you need for your own analysis. The last thing I want to mention about coding and data science and web data is something called JSON. And I like to think of it as a version of smaller is better. Now what JSON stands for is JavaScript object notation, although JavaScript is supposed to be one word. And what it is is that like XML, JSON is semi structured data. That is, you have tags that define the data.  But like XML, the tags can vary freely. And so there's a lot in common between XML and JSON. So XML is a markup language, that's what the ML stands for. And that gives meaning to the text that lets the computer know what each piece of information is. Also, XML allows you to make comments in the document. And it allows you to put metadata in the tags, you can actually put some information there in the angle brackets to provide additional context. JSON, on the other hand, is specifically designed for data interchange. And so it's got that special focus. And the structure JSON corresponds with data structures, you know, it directly represents objects and arrays and numbers and strings and booleans. And that works really well with the programs that used to analyze data. Also, JSON is typically shorter than XML because it does not require the closing tags. Now there are ways to do that with XML, but that's not typically how it's done. As a result of these differences, JSON is basically taking XML's place in web data. XML still exists, still used for a lot of things. But JSON is slowly replacing it. And we'll take a look at the comparison between the three by going back to the example we used in XML. This is data about Formula One car races in 1957 from airgas.com. And so you can just go to the first web page here and then we'll navigate to the others from that. So this is the general page. This is if you just type in without the dot XML or dot JSON or anything. So it's a table of information about races in 1957. And we saw earlier that if you add just dot XML to the end of this, it looks exactly the same. That's because this browser is displaying XML properly by default. But if you were to right click on it and go to view page source, you would get this instead. And you can see the structure. This is still XML. And so everything has an opening tag and a closing tag and some extra information in there. But if you type in dot JSON, what you really get is this jumbled mess. Now, that's unfortunate because there is a lot of structure to this. So what I'm going to do is I'm actually going to copy all of this data. Then I'm going to go to a little web page. There's a lot of things you can do here. And seek you phrases called JSON pretty print. And that is make it look structured. So it's easier to read. I just paste that in there and hit pretty print JSON. And now you can see the hierarchical structure of the data. The interesting thing is that the JSON tags only have tags at the beginning. So it says series in quotes and then a colon. And then it gives the piece of information in quotes and a comma and it moves on to the next one. And this is a lot more similar to the way that data would be represented in something like our or Python. And so it's also more compact. Again, there's things you can do with XML. But this is one of the reasons that JSON is generally becoming preferred as a data carrier for websites. And as you might have guessed, it's really easy to convert between the formats. It's easy to convert between XML, JSON, CSV, etc. And so you can get a webpage where you just paste one version in and you get the other version out. There are some differences, but for the vast majority of situations, they're just kind of interchangeable. So in some, what do we get from this? Like XML, JSON is semi structured data, where there are tags that say what the information is, but you can define the tags however you want. And JSON is specifically designed for data interchange. And because it reflects the structure of the data in the programs, that makes it really easy. And then also because it's relatively compact, JSON is replacing gradually XML on the web as a container for data on web pages. If we're going to talk about coding and data science and the languages that are used, the first and foremost is are the reason for that is according to many standards, R is the language of data and data science. For example, take a look at this chart. This is a ranking based on a survey of data mining experts of the software that they use in doing their work. And R is right there at the top, R is first. And in fact, that's important because there's Python, which is usually taken hand in hand with R for data science. But R sees 50% more use than Python does, at least in this particular list. Now there's a few reasons for that popularity. Number one, R is free and its open source, both of which make things very easy. Second, R is specially developed for vector operations. That means it's able to go through an entire list of data without having to write for loops to go through. If you've ever had to write for loops and you know that that would be kind of disastrous having to do that with data analysis. Next, R has a fabulous community behind it. It's very easy to get help on things with our you Google it, you're going to end up in a place where you're going to be able to find good examples of what you need. And probably most importantly, R is very capable on its own, but there are 7000 packages actually as many more than that 7000 packages that add capabilities to R. Essentially, it can do anything.  That is, how do you actually do the coding and how do you get your results? R comes with its own IDE or interactive development environment. You can do that. Or if you're on a Mac or Linux, you can actually do R through the terminal through a command line. If you've installed R, you just type R and it starts up. There's also a very popular development environment called R studio. And that's actually the one that I use and I'll be using for all my examples. But another new competitor is Jupiter which very commonly used for Python is what I use for examples there. It works in a browser window even though it's locally installed. And our studio in Jupiter, there's pluses and minuses to each one of them. I'll mention them as we get to them. But no matter what interface you use, R's command line, you are typing lines of code in order to get the commands. Some people get really scared about that. But really, there's some advantages to that in terms of the replicability and really the accessibility, the transparency of your commands. So for instance, here's a short example of so commands in R. You can enter them into what's called the console. And that's just like one line at a time that's called an interactive way. Or you can save scripts and run bits and pieces of them selectively. That makes your life a lot easier. No matter how you do it, if you're familiar with programming other languages, then you can find that R is a little weird. It has an idiosyncratic model. It makes sense once you get used to it. But it is a different approach. And so it does take some adaptation if you're accustomed to programming in other languages. Now, once you do your programming to get your output, what you're going to get is graphs in a separate window, you're going to get text and numbers numerical output in the console. And no matter what you get, you can save the output to files. So that makes it portable, you can do it in other environments. But most importantly, I like to think of this, here's our box of chocolates where you never know what you're going to get. The beauty of R is in the packages that are available to expand its capabilities. Now there are two sources of packages for R. One goes by the name of CRAN, and that stands for the comprehensive R archive network. And that's at CRAN.RStudio.com. And what that does is it takes the 7000 or so different packages that are available, and it organizes them in topics that they call task views. And for each one, if they've done their homework, you have data sets that come along with that package, you have a manual in PDF format. And you can even have vignettes where they run through examples of how to do it. Another interface is called CRANTASIC and the exclamation point is part of the title. That's at CRANTASIC.org. And what this is is an alternative interface that links to CRAN. So if you find something like in CRANTASIC and you click on the link, it's going to open in CRAN. But the nice thing about CRANTASIC is it shows the popularity of packages, and it also shows how recently they were updated. And that can be a nice way of knowing that you're getting sort of the latest and greatest. Now from this very abstract presentation, we can say a few things about R. Number one, according to many, R is the language of data science. And it's a command line interface, you're typing lines of code. So that gives it both a strength and is a challenge for some people. But the beautiful thing is the thousands and thousands of packages of additional code and capability that are available for her that make it possible to do nearly anything in this statistical programming language. When talking about coding and data science in the languages, along with R, we need to talk about Python. Now, Python, the snakes, is a general purpose program that can do it all. And that's its beauty. If we go back to this survey of the software used by data mining experts, you see that Python's there. It's number three on the list. And what's significant about that is on this list, Python is the only general purpose programming language, it's the only one that can theoretically be used to develop any kind of application that you want. That gives it some special power compared to all these others, most of which are very specific to data science work. So the nice things about Python are number one, it's general purpose. It's also really easy to use. And if you have a Macintosh or a Linux computer, Python is built into it. Also, Python has a fabulous community around it with hundreds of thousands of people involved. And also Python has thousands of packages. Now, it actually has something like 70 or 80,000 packages. But in terms of ones that are specific to data, there are still thousands available. They give it some incredible capabilities. Now, a couple of things to know about Python. First is about versions. There are two versions of Python that are in wide circulation. There's 2.x. So that means like 2.5, 2.6. And there's 3.x 3.1 3.2. Version two and version three are similar. But they're not identical. And in fact, the problem is this, there's some compatibility issues where code that runs in one does not run in the other. And consequently, most people have to choose between one or the other. And what this leads to is that many people still use 2.x. I have to admit the examples that I use, I'm using 2.x, because so many of the data science packages are developed with that in mind. Now, let me say a few things about interfaces for Python. First, Python does come with its own interactive development and learning environment. They call it idle. You can also run it from the terminal or command line interface or any IDE that you have. Now, a very common and very good choice is Jupiter. Jupiter is a browser based framework for programming. And it was originally called I Python, and that served as its initial version. So a lot of times when people talk about I Python, what they're really talking about is Python in Jupiter. And the two are sometimes used interchangeably. One of the neat things you can do is there are two companies, there's continuum and thought, both of which have made special distributions of Python with hundreds and hundreds of packages preconfigured to make it very easy to work with data. I personally prefer continuum anaconda is the one that I use a lot of other people use it, but either one's going to work and it's going to get you up and running. And like I said with our no matter what interface you use, all of them are command line, you're typing line to code. Again, there are some tremendous strengths to that, but it can be intimidating to some people at first. In terms of the actual commands of Python, you have some examples here on the side. The important thing to remember is that it's a text interface. On the other hand, Python is familiar to millions of coders because it's very often a first programming language that people learn to do general purpose programming. And there are a lot of very simple adaptations for data that make it very powerful for data science work. So let me say something else again, data science loves Jupiter. And Jupiter is the browser based framework. It's a local installation, but you access it through a web browser that makes it possible to really do some excellent work in data science. There's a few reasons for this. When you're working in Jupiter, you get text output and you can use what's called Markdown as a way of formatting documents. You can get inline graphics where the graphics just show up directly beneath the code that you did it. Also, it's really easy to organize and present and to share analyses that are done in Jupiter, which makes it a strong contender for your choices in how you do data science programming. Another one of the beautiful things about Python like are is that there are thousands of packages available. In Python, there's one main repository. It goes by the name Pi Pi, which is for the Python package index. Right here, it says there's over 80,000 packages and seven or 8,000 of those are for data specific purposes. Now some of the packages that you'll get to be very familiar with are NumPy and SciPy, which are for scientific computing in general. Matplotlib, and a development of it called Seaborn are for data visualization and graphics. Pandas is the main package for doing statistical analysis. And for machine learning, almost nothing beats scikit learn. And when I go through hands on examples in Python, I will be using all of these as a way of demonstrating the power of the program for working with data. So in sum, we can say a few things. Number one, Python is a very popular program familiar to millions of people and that makes it a good choice. Second, of all the languages we use for data science on a frequent basis, this is the only one that's general purpose, which means it can be used for a lot of things other than processing data. And it gets its power like R does from having thousands of contributed packages, which greatly expand its capabilities, especially in terms of doing data science work. A choice for coding in data science, one of the languages that may not come immediately to mind when they think data science is SQL or SQL. And SQL is the language of databases. And we think, why do we want to work in SQL? Well, to paraphrase the famous bank robber Willie Sutton, who apparently explained why he robbed banks and said, because that's where the money is. The reason we would work with SQL in data science is because that's where the data is. And so let's take another look at our ranking of software among data mining professionals. And their SQL is third on the list. And also, of this list, it's the first database tool. Other tools in there, for instance, get much fancier and they're much newer and shinier, but SQL's been around for a while is very, very capable. Now there's a few things to know about SQL. By the way, you'll notice I'm saying SQL, even though that stands for something that stands for structured query language. SQL is a language, it's not an application, there's not a program SQL. It's a language that can be used in different applications. Primarily SQL is designed for what are called relational databases. And those are special ways of storing structured data that you can pull in, you can put things together, you can join them in special ways, you can get some summary statistics. And then what you usually do is you then export that data into your analytical application of choice. So the big word here is our DBMS. And that stands for relational database management system. And that's where you will usually see SQL as a query language being used. In terms of relational database management systems, there are a few very common choices. In the industrial world where people have some money to spend, there's Oracle database is a very common one. And Microsoft SQL or SQL server. In the open source world, two very common choices are My SQL, even though we generally say SQL when it's here, you generally say My SQL. And then another one is PostgreSQL. These are both open source free versions of the language. They're sort of dialects of each that make it possible for you to be working with databases and get your data out. The neat thing about them, no matter what you do, is that databases minimize data redundancy by using connected tables. Each table has rows and columns, and they store different levels or different of abstraction or measurement, which mean you only have to put the information in one place and then it can refer to lots of other tables. Makes it very easy to keep things organized and up to date. When you're looking at a way of working with a relational database management system, you get to choose in part between using a graphical user interface or GUI. And some of those include SQL developer and SQL server management studio to very common choices. And there are a lot of other ones like Toad and some other choices that are graphical interfaces for working with these databases. And there are also text based interfaces. So really, any command line interface, and any interactive development environment or programming tool is going to be able to do that. Now you can think of yourself being on the command deck of your ship and learn a few basic commands that are very important for working with SQL. They're just a handful of commands that can get you most of where you need to go. There is the select command where you're choosing the cases that you want to include from says which tables are you going to be extracting them from. Where is a way of specifying conditions and then order by obviously it's just a way of putting it all together. This works because usually when you're in a SQL database, you're just pulling out the information. You want to select it, you want to organize it. And then what you're going to do is you're going to send the data to your program of choice for further analysis like our Python or whatever. So in some, here's what we can say about SQL. Number one, as a language, it's generally associated with relational databases, which are very efficient and well structured ways of storing data. Just a handful of basic commands can be extremely useful when working with databases. You don't have to be a super ninja expert. Really a handful five, 10 commands will probably get you everything you need out of a SQL database. And then once you get the data organized, it's typically exported to some other program for analysis. When you talk about coding in any field, one of the languages or one of the groups of languages that come up most often are C, C plus plus and Java. Now these are extremely powerful applications and very frequently used for sort of professional production level coding. In data science, the place where you're going to see these languages most often is in the bedrock, the absolute fundamental layer that makes the rest of data science possible. So for instance, C and C plus plus sees from the 60s C plus plus is from the 80s. And they have extraordinarily wide usage. And their major advantage is that they're really, really fast. In fact, C is usually used as the benchmark for how fast is a language. They're also very, very stable, which makes them really well suited to production level code. And for instance, server use. What's really neat is that in certain situations, if time is really important of speeds important, then you can actually use C code in our or other statistical languages. Next is Java. Java is based on C plus plus. Its major contribution was the war or right once run anywhere. The idea that you are going to be able to develop code that is portable to different machined and different environments. And because of that, Java is actually the most popular computer programming language, overall against all tech situations. And the place where you would use these in data sciences, like I said, when time is of the essence, when something has to be fast, it has to get the job accomplished quickly, and it has to not break. Then these are the ones that you're probably going to use. The people who are going to use it are primarily going to be engineers. So the engineers and the developers, the software developers who deal with the inner workings of the algorithms in data science, or the back end of data science, the servers and the mainframes and the entire structure that makes analysis possible. In terms of analysts, people who are actually analyzing the data typically don't do hands on work with the foundational elements, they don't usually touch C or C plus plus, more the work is on the front end or closer to the high level languages like R or Python. In some C, C plus plus and Java form a foundational bedrock and the back end of data and data science. And they do this because they're very fast, and they are very reliable. On the other hand, given their nature, that work is typically reserved for the engineers who are working with the equipment that runs in the back that makes the rest of the analysis possible. I want to finish our extremely brief discussion of coding and data sciences and the languages that can be used by mentioning one other that's called bash. And bash really is a great example of old tools that have survived and are still being used actively and productively with new data. You can think of it this way. It's almost like typing on your typewriter. You're working at the command line, you're typing out code through a command line interface or CLI. Now, this method of interacting with computers practically goes back to the typewriter phase because it predates monitors. So before you even had a monitor, you would type in the code and it would print it out on a piece of paper. And the important thing to know about the command line is it's simply a method of interacting. It's not a language because lots of different languages can run at the command line. So for instance, it's important to talk about the concept of a shell. Now, computer science, a shell is a language or something that wraps around the computer to shell around the language that is the interaction level for the user to get things done at the lower levels that aren't really human friendly. On Mac computers and Linux, the most common is bash, which is short for born again shell. On Windows computers, the most common version is PowerShell. But whatever you do, there actually are a lot of choices. There's the born shell, there's the seashell, which is why I have a seashell right here. The seashell, there's fish for a friendly interactive shell and a whole bunch of other choices. But bash is the most common on Mac and Linux and PowerShell is the most common on Windows as a method of interacting with the computer at the command line level. Now there's a few things you need to know about this. First, you have a prompt of some kind and bash it's a dollar sign. And that just means type your command here. Then the other thing is you type one line at a time. It's actually amazing how much you can get done with what's called a one liner program by sort of piping things together. So one feeds into the other, you can run more complex commands if you use a script. And so you call a text document that has a bunch of things in it. And you can get much more elaborate analysis done. Now we have our tools here. In bash, we talk about utilities. And what these are are specific programs that accomplish specific tools. Bash really thrives on do one thing and do it very well. There are two general categories of utilities for bash. Number one is the built ins. These are the ones that come installed with it. And so you're able to use them at any time by simply calling in their name. Some of the most common ones are cat, which is for cat and eight. And that's to put information together. There's awk, which is its own interpreted programming language. But it's often used for text processing from the command line. By the way, the name awk comes from the initials of the people who created it. Then there's grep, which is for global search with a regular expression and print. It's a way of searching for information. And then there's said, which stands for stream editor. And its main use is to transform text, you can do an enormous amount with just these four utilities. A few more are head and tail that display the first or last 10 lines of the document sort and unique, which sort and count the number of unique answers in a document, WC, which is for word count, and print f, which formats the output that you get in your console. And while you can get a huge amount of work done with just this small number of built in utilities, there are also a wide range of installables or other command line utilities that you can add to bash or to whatever program you're using. So for instance, some really good ones that have been recently developed are jq, which is for pulling in JSON or JavaScript object notation data from the web. And then there's JSON to CSV, which is a way of converting JSON to CSV format, which is what a lot of statistical programs are going to be happier with. There's real, which allows you to run a wide range of commands from the statistical programming language are in the command line as part of bash. And then there's Big MLR. And this is a command line tool that allows you to access Big ML's machine learning servers through the command line. Normally, you do it through a web browser and it accesses their servers remote. It's an amazingly useful program. But to be able to just pull it up when you're in the command line is an enormous benefit. What's interesting is that even though you have all these opportunities, all these different utilities, you can do amazing things and that there still is active development of utilities for the command line. So let's say this in some, despite being in one sense as old as the dinosaurs, the command line survives because it is extremely well evolved and well suited to its purpose of working with data, the utilities, both the built in and the installables are fast and they are easy. And generally they do one thing and they do it very, very well. And then surprisingly, there is an enormous amount of very active development of command line utilities for these purposes, especially with data science. One critical task when you're coding in data science is to be able to find the things that you're looking for. And reg X, which is short for regular expressions is a wonderful way to do that. You can think of it as the supercharged method for finding needles in haystacks. Now, reg X tends to look a little cryptic. So for instance, here's an example of something that's designed to determine whether something is a valid email address. And it specifies what can go on the beginning, you have the at sign in the middle, then you've got a certain number of numbers and letters, and then you have to have a dot something at the end. And so this is a special kind of code for indicating what can go where. Now, regular expressions or reg X are really a form of pattern matching in text. And it's a way of specifying exactly what needs to be where what can vary and how much it can vary. And you can write both specific patterns, say I only want a one letter variation here, or very general like the email validator that I showed you. And the idea here is you can write this search pattern, your little wildcard thing, you can find the data. And then once you identify those cases, then you can export them into another program for analysis. So here's a short example of how it can work. What I've done is I've taken some text documents, they're actually the text to Emma and to Pygmalion two books I got off of Project Gutenberg. And this is the command grep caret l dot ve space asterisk dot txt. So when I'm looking for our lines in either of these books that start with L, then they have one character can be whatever. And then that's followed by ve. And then the dot txt means search for all of the text files in that particular folder. And what it found is lines that began with love and lived and lovely, and so on. Now in terms of the actual nuts and bolts of regular expressions, there are some certain elements. There are literals.  You type the letter L, you're looking for the letter L. There are also meta characters, which specify, for instance, things needs to go here, they're characters, but they actually really code that to give representations. There are also escape sequences, which are something that you use to say, well, normally this character is used as a variable, but I actually want to really look for a period as opposed to a placeholder. Then you have the entire search expression that you create. And then you have the target string, the thing that it's searching through. So let me give a few very short examples. This is the carrot. It's the sometimes called a hat or in French a circle flex. And what that means is you're looking for something that's at the beginning of the text that you're searching. So for example, you can have carrot and capital M. That means you need something that begins with a capital M. So, for instance, the word Mac true, it will find that. But if you have iMac, there's a capital M, but it's not the first thing. So that'll be false. It won't find that the dollar sign means you're looking for something that is at the end of the string. So for example, ing and then dollar sign, that'll find the word fling because it ends with ing. But it won't find the word flings because it actually ends with an s. And then the dot the period simply means we're looking for one letter and it can be anything. So for example, you can write a t period. And that will find data because it has an a t and then one letter after it. But it won't find flat because flat doesn't have anything after the a t. And so these are extremely simple examples of how it can work. Obviously, it gets more complicated. And the real power is when you start combining these bits and elements. Now, one interesting thing about this is you can actually treat this as a game. I love this website. It's called regx golf. And it's at regx.alf.nu. And what it does is it brings up lists of words, two columns, and your job is to write a regular expression in the top that matches all the words on the left column. And none of the words on the right. And that uses the fewest characters possible, you get a score. And it's a great way of learning how to do regular expressions and learning how to search in a way that's going to get you the data that you need for your projects. So in some regx or regular expressions help you find the right data for your project. They're very powerful and they're very flexible. Now on the other hand, they are cryptic, at least when you first look at them. But at the same time, it's like a puzzle and it can be a lot of fun if you practice it. And you see how you can find what you need. I want to thank you for joining me in coding and data science. And we'll wrap up this course by talking about some of the specific next steps that you can take for working in data science. The idea here is that you want to get some tools and you want to start working with those tools. Now, please keep in mind something that I've said at another time. Data tools and data science are related, they're important, but don't make the mistake of thinking that if you know the tools that you have done the same thing has actually conducted data science. That's not true.  What you need to remember is the relationship really is this data tools are an important part of data science, but data science itself is much bigger than just the tools. Now, speaking of tools, remember, there's a few kinds that you can use and that you might want to get some experience with these. Number one, in terms of apps or just specific built applications, Excel and Tableau are really fundamental for both getting the data from clients or doing some basic data browsing. And Tableau is really wonderful for interactive data visualization. I strongly recommend that you get very comfortable with both of those. In terms of code, it's a good idea to learn either R or Python or ideally to learn both because you can use them hand in hand. In terms of utilities, it's a great idea to learn how to work with bash the command line utility and to use regular expressions or reg X, you can actually use those in lots and lots of programs, regular expressions. And so they can have a very wide application. And then finally, data science requires some kind of domain expertise, you're going to need some sort of field experience, or intimate understanding of a particular domain and the challenges that come up and what constitutes workable answers and the kind of data that's available. Now, as you go through all of this, you don't need to build this monstrous list of things. Remember, you don't need everything, you don't need every tool, you don't need every function, you don't need every approach. Instead, remember, get what's best for your needs, and for your style. But no matter what you do, remember, tools or tools, there are means to an end. Instead, you want to focus on the goal of your data science project, whatever it is. And I can tell you really, the goal is meaning extracting meaning out of your data to make informed choices. In fact, I'll say a little more, the goal is always meaning. And so with that, I strongly encourage you get some tools, get started in data science and start finding meaning in the data that's around you. Welcome to mathematics and data science. I'm Barton Polson. And we're going to talk about how mathematics matters for data science. Now, you may be saying to yourself, why math? And computers can do it. I don't need to do it. And really fundamentally, I don't need math. I'm just here to do my work. Well, I'm here to tell you, no, you need math. That is, if you want to be a data scientist, and I assume that you do. So we're going to talk about some of the basic elements of mathematics really at a conceptual level and how they apply to data science. There are a few ways that math really matters to data science. Number one, it allows you to know which procedures to use and why so you can answer your questions in a way that's the most informative and most useful. Two, if you have a good understanding of math, then you know what to do when things don't work right, that you get impossible values or things won't compute. And that makes a huge difference. And then three, an interesting thing is that some mathematical procedures are easier and quicker to do by hand than by actually firing up the computer. And so for all three of these reasons, it's helpful to have at least a grounding in mathematics, if you're going to do work in data science. Now, probably the most important thing to start with is algebra. And there are three kinds of algebra that we want to mention. The first is elementary algebra, that's the regular x plus y. Then there's linear or matrix algebra, which looks more complex, but is conceptually simple and is used by computers to actually do the calculations. And then finally, I'm going to mention systems of linear equations where you have multiple equations simultaneously that you're trying to solve. Now there's more math than just algebra, a few other things that I'm going to cover in this course, a little bit of calculus, a little bit of big O or order, which has to do with the speed or the complexity of operations, a little bit of probability theory, and a little bit of Bayes or Bayes theorem, which is used for getting posterior probabilities and changes the way that you interpret the results of an analysis. And for the purposes of this course, I'm going to demonstrate the procedures by hand. Of course, you would use software to do this in the real world, but we're dealing with simple problems at conceptual levels. And really, the most important thing to remember is, even though a lot of people get put off by math, really, you can do it. And so in some, let's say these three things about math.  And interestingly, you can do a lot of it by hand, or you can use software computers to do the calculations as well. As we begin our discussion of the role of mathematics and data science, we'll of course begin with the foundational elements. And in data science, nothing is more foundational than elementary algebra. Now, I'd like to begin this with really just a little bit of history. In case you're not aware, the first book on algebra was written in 820 by Muhammad Ibn Musa al-Qarizmi. And it was called the Compendious Book on Calculation by Completion and Balancing. Actually, it was called this, which if you transliterate that comes out to this, but look at this word right here. That's the algebra, which means restoration. Any case, that's where it comes from. And for our concerns, there are several kinds of algebra that we're going to talk about. There's elementary algebra, there's linear algebra. And there are systems of linear equations, we'll talk about each of those in different videos. But to put it into context, let's take an example here of salaries. Now, this is actually based on real data from a survey of the salary of people employed in data science. And to give a simple version of it, the salary was equal to a constant. That's sort of an average value that everybody started with. And to that you added years and then you added some measure of bargaining skills and how many hours they worked per week. And that gave you a prediction. But because it wasn't exact, there's also some error to throw into it to get to the precise value that each person has. Now, if you want to abbreviate this, you can write it kind of like this s plus c plus y plus b plus h plus e, although it's more common to write it. And although it's more common to write it symbolically like this. And let's go through this equation very quickly. The first thing we have is outcome, we call that y, the variable y for person I, I stands for each case in our observations. So here's outcome y for person I. This letter right here is a Greek beta. And it represents the intercept or the average. That's way it has a zero because we don't multiply it times anything. But right next to it, we have the coefficient for variable one. So beta, which means a coefficient sub one for the first variable. And then we have variable one and then x one means variable one and then the I means it's the score on that variable for person I whoever we're talking about. Then we do the same thing for variables two and three. And then at the end, we have little epsilon here with an I for the error term for person I was says how far off the prediction was from their actual score. Now I'm going to run through some of these procedures and we'll see how they can be applied to data science. But for right now, let's just say this in some first off, algebra is vital to data science. It allows you to combine multiple scores, get a single outcome, do a lot of other manipulations. And really, the calculations are easy for one case at a time, especially when you're doing it by hand. The next step in mathematics for data science foundations is to look at linear algebra or an extension of elementary algebra. And depending on your background, you may know this by another name. And I like to think welcome to the matrix because it's also known as matrix algebra because we're dealing with matrices. Now let's go back to an example I gave in the last video about salary, where salary is equal to a constant plus years plus bargaining plus hours plus error. Okay, that's a way to write it out in words. And if you want to put it in symbolic form, it's going to look like this. Now, before we get started with matrix algebra, we need to talk about a few new words. Maybe you're familiar with them already. The first is scalar. And this means a single number. And then a vector is a single row or a single column of numbers that can be treated as like a collection. That usually means a variable. And then finally, a matrix consists of many rows and columns sort of a big rectangle of numbers. The plural that by the way is matrices. And the thing to remember is that machines love matrices. Now let's take a look at a very simple example of this. Here is a very basic representation of matrix algebra or linear algebra, where we're showing data on two people on four variables. So over here on the left, we have the outcomes for cases one and two are people one and two. And you put them in the square brackets to indicate that it's a vector or a matrix. Here on the far left, it's a vector because it's a single column of values. Next to that is a matrix that has here on the top the scores for case one, which I've written as x is x one is for variable one x two is for variable two. And the second subscript is indicated as for a person one. Below that are the scores for case two, the second person. And then over here and another vertical column are the regression coefficients. That's a beta there that we're using. And then finally, we've got a tiny little vector here at the end, which contains the error terms for cases one and two. Now, even though you would not do this by hand, it's kind of helpful to run through the procedure. So I'm going to show it to you by hand. And we're going to take two fictional people. This will be fictional person number one, we'll call her Sophie. We'll say that she's 28 years old. And we'll say that she has good bargaining skills of four on a scale of five, and that she works 50 hours a week, and that her salary is 118,000. Our second fictional person, we'll call him Lars, and we'll say that he's 34 years old, and then he has moderate bargaining skills, three out of five, works 35 hours per week and has a salary of $84,000. And so if we're trying to look at salaries, we can go back to our matrix representation that we had here with our variables indicated with their Latin and sometimes Greek symbols. And we're going to replace those variables with actual numbers. So we can get the salary for Sophie our first person. So let me plug in the numbers here. And let's start with the result here. Sophie salaries 118,000. And here's how these numbers all add up to get that. The first thing here is the intercept and we just multiply that times one. So that's sort of the starting point. And then we get this number 10, which actually has to do with years over 18. She's 28. So that's 10 years over 18. We multiply each year by 1395. Next is bargaining skills. She's got a four out of five. And for each step up, you get $5,900. By the way, these are real coefficients from study of survey of salary of data scientists. And then finally, hours per week. For each hour, you get $382. Now we can add those up and we can get a predicted value for her, but it's a little low. It's 30,000 low, which you may say, well, that's really messed up. Well, that's because there's like 40 variables in the equation including she might be the owner. And if she's the owner, yeah, she's going to make a lot more. And then we do a similar thing for the second case. But what's neat about matrix algebra or linear algebra is that you can use matrix notation. And this means the same stuff. And what we have here are these bolded variables that stand in for entire vectors or matrices. So for instance, this y a bold y stands for the vector of outcome scores. This bolded x is the entire matrix of values that each person has on each variable. This bolded beta is all of the regression coefficients. And then this bolded epsilon is the entire vector of error terms. And so it's really super compact way of representing the entire collection of data and coefficients that you use in predicting values. So in some, let's say this, first off, computers use matrices, they like to do linear algebra to solve problems. And it's conceptually simpler because you can put it all there in this tight formation. In fact, it's a very compact notation and allows you to manipulate entire collections of numbers pretty easily. And that's the major benefit of learning a little bit about linear or matrix algebra. Our next step in mathematics for data science foundations is systems of linear equations. And maybe you're familiar with this, but maybe you're not. And the idea here is there are times when you actually have many unknowns, and you're trying to solve for all of them simultaneously. And what makes this really tricky is a lot of these are interlocked. Specifically, that means x depends on y, but at the same time, y depends on x. What's funny about this is it's actually pretty easy to solve these by hand. And you can also use linear matrix algebra to do it. So let's take a little example here of sales. Let's imagine that you've got a company and you've sold 1000 iPhone cases of the not running around naked like in this picture, and that some of the cases sold for $20 and other sold for $5. You made a total of $5,900. And so the question is how many were sold at each price? Now, hopefully you were keeping your records, but you can also calculate it from this little bit of information. And to show you, I'm going to do it by hand. Now, we're going to start with this. We know that sales, the two price points x and y add up to 1000 total cases sold. And for revenue, we know that if you multiply a certain number times $20 and another number times $5, that it all adds up to $5,900. Between the two of those, we can figure out the rest. Let's start with sales.  And I'm going to do that by putting in this minus y on both sides. And then I can take that and I can subtract it. So I'm left with x is equal to 1000 minus y. Normally you solve for y, but I solve for x, you'll see why in just a second. Then we go to revenue. And we know from earlier that our sales of these two price points add up to $5,900 total. Well, what we're going to do is we're going to take this x that's right here and we're going to replace it with the equation that we just got, which is 1000 minus y. Then we multiply that through and we get 20,000 minus 20 y plus 5 y equals 5900. Well, we can subtract these two because they're on the same thing. So 20 y and we get 15 y. And then we subtract 20,000 from both sides. So there it is right there on the left. And that disappears. And then I get it over on the right side. And I do the math there and I get minus $14,100. Well, then I divide both sides by negative $15. And when we do that, we get y is equal to 940. Okay, so that's one of our values for sales. So let's go back to sales. We have x plus y equals 1000. We take the value that we just got 940, we stick that into the equation. And then we can solve for x, just subtract 940 from each side. There we go. We get x is equal to 60. So let's put it all together, just to recap what happened. What this tells us is that 60 cases were sold at $20 each. And that 940 cases were sold at $5 each. Now what's interesting about this is you can also do this graphically, we're going to draw it. So I'm going to graph the two equations. Here are the original ones we had this one predicts sales this one gives price. The problem is these really aren't in the canonical form for creating grass that needs to be y equals something else. So we're going to solve both of these for y. We subtract x from both sides there it is on the left we subtract that and then we have y is equal to minus x plus 1000. That's something that we can graph. Then we do the same thing for price. Let's divide by five all the way through that gets rid of that. And then we've got this 4x and then let's subtract 4x from each side. And what we're left with is minus 4x plus 1180. That's also something that we can graph. So here's the first line this indicates cases sold. It originally said x plus y equals 1000 but we rearranged it to y is equal to minus x plus 1000. And so that's the line we have here. And then we have another line which indicates earnings. And this one was originally written as $20 times x plus $5 times y equals $5900 total. We rearrange that to y equals minus 4x plus 1180. That's the equation for the line. And then the solution is right here at the intersection. There's our intersection and it's at 60 on the number of cases sold at $20 and 940 on the number of cases sold at $5. And that also represents the solution of these joint equations. And so it's a graphical way of solving a system of linear equations. So in some systems of linear equations allow us to balance several unknowns and find the unique solution. And in many cases it's easy to solve by hand. And it's really easy with linear algebra when you use software to do it at the same time. As we continue our discussion of mathematics and data science and the foundational principles, the next thing we want to talk about is calculus. And I'm going to give a little more history right here. The reason I'm showing you pictures of stones is because the word calculus is Latin for stone as in a stone used for tallying where people would actually have a little bag of stones and they would move them and they would use it to count sheep or whatever. And the system of calculus was formalized in the 1600s simultaneously independently by Isaac Newton and Gottfried Wilhelm Leibniz. And there are three reasons why calculus is important for data science. Number one, it's the basis of most of the procedures that we do. Things like least squares regression and probability distributions, they use calculus in getting those answers. Second one is if you're studying anything that changes over time. So if you're measuring quantities or rates that change over time, then you have to use calculus. Calculus is used in finding the maximum and the minimum of functions, especially when you're optimizing, which is something I'll show you separately. Also, it's important to keep in mind, there are two kinds of calculus. The first is differential calculus, which talks about rates of change at a specific time. It's also known as the calculus of change. The second kind of calculus is integral calculus. And this is where you're trying to calculate the quantity of something at a specific time, given the rate of change. And it's also known as the calculus of accumulation. So let's take a look at how this works. And we're going to focus on differential calculus. So I'm going to graph an equation here, I'm going to do y is equal to x squared, a very simple one. But it's a curve, which makes it harder to calculate things like the slope. So let's take a point here. That's at minus two, that's my little red dot, we have it x is equal to minus two. And because y is equal to x squared, if we want to get the y value, all we got to do is take that negative two in square and that gives us four. So that's pretty easy. So the coordinates for that red pointer minus two on x and plus four on y. Here's a harder question. What is the slope of the curve at that exact point? Well, it's actually a little tricky because the curve is always curvy and there's no flat part on it. But we can get the answer by getting the derivative of the function. Now, there are several different ways of writing this, I'm using the one that's easiest to type. And let's start by this, what we're going to do is the n here. And that is the squared part. So we had x squared. And you see that same n turns into the squared. And then we come over here and we put that same value two in right there. And we put the two in right here. And then we can do a little bit of subtraction two minus one is one and truthfully you can just ignore that and you get two x. That is the derivative. So what we have here is the derivative of x squared is two x. That means the slope at any given point of the curve is two x. So let's go back to what we had a moment ago. Here's our curve. Here's our point at x minus two. And so the slope is equal to two x. Well, we put in the minus two when we multiply it and we get minus four. So that is the slope at this exact point on the curve. Okay, what if we choose a different point, let's say we come over here to x is equal to three, while the slope is equal to two x. So that's two times three is equal to six. Great. And on the other hand, you might be saying to yourself, and why do I care about this? There's a reason that this is important. And what it is is that you can use these procedures to optimize decisions.  And I'm going to demonstrate that in the next video. But for right now in some, let's say this calculus is vital to practical data science. It's the foundation of statistics and it forms the core that's needed for doing optimization. In our discussion of mathematics and data science foundations, the last thing I want to talk about right here is calculus and how it relates to optimization. I'd like to think of this in other words as the place where math meets reality or it meets Manhattan or something. Now if you remember this graph I made in the last video, y is equal to x squared, that shows this curve here. And we have the derivative that the slope can be given by 2x. And so when x is equal to three, the slope is equal to six, fine. And this is where this comes into play. Calculus makes it possible to find values that maximize or minimize outcomes. And if you want to make something a little more concrete out of this, let's think of an example here. By the way, that's Cupid and Psyche. Let's talk about pricing for online dating. Let's assume you've created a dating service and you want to figure out how much can you charge for it that will maximize your revenue. So let's get a few hypothetical parameters involved. First off, let's say that subscriptions, annual subscriptions currently cost $500 a year, and you can charge that for a dating service. And let's say you sell 180 new subscriptions every week. On the other hand, based on your previous experience manipulating prices around, you have some data that suggests that for each $5 you discount from the price of $500, you will get three more sales. Also, because it's an online service, let's just make our lives a little simpler right now and assume that there is no increase in overhead. It's not really how it works, but we'll do it for now. And I'm actually going to show you how to do all this by hand. Now let's go back to price first. We have this $500 is the current annual subscription price. And you're going to subtract $5 for each unit of discount. That's what I'm giving D. So one discount is $5, two discounts is $10 and so on. And then we have a little bit of data about sales, that you're currently selling 180 new subscriptions per week, and that you will add three more for each unit of discount that you give. So what we're going to do here is we're going to find sales as a function of price. Now to do that, the first thing we have to do is get the Y intercept. So we have price here $500 is the current annual subscription price minus $5 times D. And what we're going to do is we're going to get the Y intercept by solving when does this equals zero? Okay, well, we take the 500 we subtract that from both sides. And then we end up with minus 5D is equal to minus 500 divide both sides by minus five. And we're left with D is equal to 100. That is, when D is equal to 100, X is zero. And that tells us how we can get the Y intercept. But to get that we have to substitute this value into sales. So we take D is equal to 100. And the intercept is equal to 180 plus three 180 is the number of new subscriptions per week.  So 180 times three times 100 is equal to 300. Add those together. And you get 480. And that is the Y intercept in our equations. So when we've discounted sort of price to zero, when prices zero, then the expected sales is 480. Of course, that's not going to happen in reality. But it's necessary for finding the slope of the line. And so now let's get the slope. The slope is equal to the change in Y on the Y axis divided by the change in X. One way we can get this is by looking at sales, we get our 180 new subscriptions per week, plus three for each unit of discount. And we take our information on price, $500 per year minus $5 for each unit of discount. And then we take these, the 3d and the 5d, and those will give us the slope. So it's plus three divided by minus five, and that's just minus 0.6. And so that is the slope of the line. Slope is equal to minus 0.6. And so what we have from this is sales as a function of price, where sales is equal to 480, because that's the Y intercept. When X is equal to zero when price is zero, minus 0.6 times price. So this isn't the final thing. Now what we have to do is we turn this into revenue. So there's another stage to this. Now revenue is equal to sales time the price, you know, how many things did you sell? And how much did it cost? Well, we can substitute in some information here. If we take sales, and we put it in as a function of price, because we just calculated that a moment ago, we get this, and then we do a little bit of multiplication, and then we get that revenue is equal to 480 times the price minus 0.6 times the square of the price. Okay, that's a lot of stuff going on there. What we're going to do now is we're going to get the derivative that's the calculus that we talked about. Well, the derivative of 48 in the price where price is sort of the X, the derivative is simply 480. And the minus 0.6 times the square of the price. Well, that's very similar to the thing we did with the curve. And what we end up with is 0.6 times two is equal to 1.2 times the price. This is the derivative of the original equation, we can solve that for zero now. And just in case you're wondering, why do we solve it for zero? Because that is going to give us the place when y is at a maximum. Now, we had a minus squared, so we have to invert the shape. And we're trying to look for this value right here when it's at the very tippy top of the curve, because that will indicate maximum revenue. Okay, so what we're going to do is we're going to solve for zero. Let's go back to our equation here. We want to find out when is that equal to zero? Well, we subtract 480 from each side. There we go. And we divide by minus 1.2 on each side. And this is our price for maximum revenue. So we've been charging $500 a week, but this says we'll have more total income if we charge $400 instead. And if you want to find out how many sales we can get, currently we have $480. And if you want to know what the sales volume is going to be for that, well, you take the 480, which is the hypothetical y intercept when the price is zero, but then we put in our actual price of $400, multiply that we get 240, do the subtraction and we get 240 total. So that would be 240 new subscriptions per week. So let's compare this. The current revenue is 180 new subscriptions per week at $500 per year. And that means that our current revenue is $90,000 per year. I know it sounds it sounds really good. But we can do better than that. Because the formula for maximum revenue is 240 times 400. When you multiply those you get 96,000. And so the improvements just the ratio of those two 96,000 divided by 90,000 is equal to 1.07. And what that means is a 7% increase in anybody would be thrilled to get a 7% increase in their business simply by changing the price and increasing the overall revenue. So let's summarize what we found here. If you lower the cost by 20% go from $500 per year to $400 per year, assuming all of our other information is correct, then you can increase sales by 33%. That's more than the 20 that you had. And that increases total revenue by 7%. And so we can optimize the price to get the maximum total revenue. And it has to do with this little bit of calculus and the derivative of a function. So in some calculus can be used to find the minimum and the maximum of functions, including prices. It allows for optimization. And that in turn allows you to make better business decisions. Our next topic in mathematics and data principles is something called big O. And if you're wondering what big O is all about, well, it is about time. Or you can think of it as how long does it take to do a particular operation? It's the speed of the operation. If you want to be really precise, the growth rate of a function, how much more it requires as you add elements is called its order. That's why it's big O. That's for order. And big O gives the rate of how things grow as a number of elements grows. And what's funny is there can be really surprising differences. Let me show you how it works with a few different kinds of growth rates or big O. First off, there's the ones that I say are sort of just on the spot, you can get stuff done right away. The simplest one is 01. And that is a constant order. And that's something that takes the same amount of time, no matter what, you can send out an email to 10,000 people, just hit one button. It's done. The number of elements, number of people, the number of operations, it just takes the same amount of time. Up from that is logarithmic where you take the number of operations, you get the logarithm of that. And you see, it's increased, but it's really only a small increase. And it tapers off really quickly. So an example is finding an item in a sorted array, not a big deal. Next one up from that now, this looks like a big change, but in the grand schemes, it's not a big change. This is a linear function, where each operation takes the same unit of time. And so if you have 50 operations, it takes 50 units of time. If you're storing 50 things, it takes 50 units of space. So find an item in an unsorted list, it's usually going to be linear time. Then we have the functions where I say, you know, you better just pack a lunch because it's going to take a little while. The best example of this is what's called log linear. That's where you take the number of items and you multiply that number times the log of the items. And an example of this is something called a fast Fourier transform, which is used for dealing, for instance, with sound or anything that's over time. You can see it takes a lot longer. If you got 30 elements, you're way up there at the top of this particular chart at 100 units of time or 100 units of space or every one to put it. And it looks like a lot. But really, that's nothing compared to the next set where I say, you know, you're just gonna be camping out, you might as well go home. That includes something like the quadratic, you square the number of elements. And see how that just kind of shoots straight up, that's quadratic growth. And so multiplying two n digit numbers. So if you're multiplying two numbers that each have 10 digits, it's going to take you that long, it's go