Welcome to Data Science and Introduction. I'm Barton Poulsen, and what we're going to do in this course is we're going to have a brief accessible and non technical overview of the field of data science. Now, some people when they hear data science, they start thinking things like data and think about piles of equations and numbers and then to throw on top of that science and think about people working in their lab and they start to say, That's not for me. I'm not really a technical person and that just seems much too techy. Well, here's the important thing to know. Well, a lot of people get really fired up about the technical aspects of data science. The important thing is that data science is not so much a technical discipline, but creative. And really, that's true. The reason I say that is because in data science you use tools that come from coding and statistics and from math. But you use those to work creatively with data. The idea is that there's always more than one way to solve a problem or answer a question. Most importantly, to get insight, because the goal, no matter how you go about it, is to get insight from your data. And what makes data science unique compared to so many other things is that you try to listen to all of your data, even when it doesn't fit in easily with your standard approaches and paradigms, you're trying to be much more inclusive in your analysis. And the reason you want to do that is because everything signifies everything carries meaning and everything can give you additional understanding and insight into what's going on around you. And so in this course, what we're trying to do is give you a map to the field of data science and how you can use it. And so now you have the map in your hands and you can get ready to get going with data science. Welcome back to data science and introduction. And we're going to begin this course by defining data science. That makes sense. But we're going to do it in kind of a funny way. The first thing I'm going to talk about is the demand for data science. So let's take a quick look. Now, data science can be defined in a few ways. I'm going to give you some short definitions. Take one on my definition is that data science is coding math and statistics in applied settings. That's a reasonable working definition. But if you want to be a little more concise, I've got take two on a definition that data science is the analysis of diverse data, or data that you didn't think would fit into standard analytic approaches. A third way to think about it is that data science is inclusive analysis. It includes all of the data, all of the information that you have in order to get the most insightful and compelling answer to your research questions. Now, you may say yourself, wait, That's it. Well, if you're not impressed, let me show you a few things. First off, let's take a look at this article. This says data scientists, the sexiest job of the 21st century. And please note that this is coming from Harvard Business Review. So this is an authoritative source. And it's the official source of this saying that data science is sexy. Now, again, you may be saying to yourself, sexy, I hardly think so. Well, oh yeah, it's sexy. And the reason data science is sexy is because first, it has rare qualities. And second, it has high demand. Let me say a little more about those. The rare qualities are that data science takes unstructured data than finds order, meaning and value in the data. Those are important, but they're not easy to come across. Second, high demand. Well, the reason it's in high demand is because data science provides insight into what's going on around you. And critically, it provides competitive advantage, which is a huge thing in business settings. Now, let me go back and say a little more about demand. Let's take a look at a few other sources. So for instance, the McKinsey Global Institute published a very well known paper, and you can get at it with this URL. And if you go to that web page, this is what's going to come up. And we're going to take a quick look at this one, the executive summary, the PDF that you can download. And if you open that up, you'll find this page. And let's take a look at the bottom right corner, two numbers here, I'm going to zoom in on those. The first one is they are projecting a need in the next few years for somewhere between 140 and 190,000 deep analytical talent positions. So this means actual practicing data scientists. That's a huge number. But almost 10 times as high is 1.5 million more data savvy managers will be needed to take full advantage of big data in the United States. So that's people who aren't necessarily doing the analysis, but have to understand it, who have to speak data. And that's one of the main purposes of this particular course is to help people who may or may not be the practicing data scientists, learn to understand what they can get out of data and some of the methods used to get there. Let's take a look at another article from LinkedIn. Here's a shortcut URL for it. And that will bring you to this web page, the 25 hottest job skills that got people hired in 2014. And take a look at number one here, statistical analysis and data mining very closely related to data science. And just to be clear, this was number one in Australia and Brazil, and Canada and France and India, and the Netherlands in South Africa and the United Arab Emirates and the United Kingdom, everywhere. And if you need a little more, let's take a look at Glassdoor, which published an article this year, 2016. And it's about the 25 best jobs in America. And look at number one right here, it's data scientists and we can zoom in on this information. It says there's going to be 1700 job openings with a median based salary of over 116,000 and fabulous career opportunities and job scores. So if you want to take all of this together, the conclusion you can reach is that data science pays. And I can show you a little more about that. So for instance, here's the list of the top 10 highest paying salaries that I got from US news, we have physicians or doctors, dentists and lawyers and so on. Now if we add data scientists to this list using data from O'Reilly.com, we have to push things around the side. And it goes in third with an average total salary not the base that we had in the other one but the total compensation of about $144,000 a year. That's extraordinary. So in sum, what do we get from all this? First off, we learned that there is a very high demand for data science. Second, we learned that there is a critical need for both specialists, those are the sort of practicing data scientists, and for generalists, people who speak the language and know what can be done. And of course, there's excellent pay and all together, this makes data science a compelling career alternative and a way of making you better at whatever you're doing. Back here in data science, we're going to continue our attempt to define data science by looking at something that's really well known in the field, the data science Venn diagram. Now if you want to, you can think of this in terms of what are the ingredients of data science. Well, we're going to first say thanks to Drew Conway, the guy who came up with this. And if you want to see the original article, you can go to this address. But what drew said is that data science is made of three things. And we can put them as overlapping circles because it's the intersection that's important. Here on the top left is coding or computer programming or as he calls it hacking. On the top right is stats or stats and mathematics or quantitative abilities in general. And on the bottom is domain expertise or intimate familiarity with a particular field of practice business or health or education or something like that. And the intersection here in the middle, that is data science. So it's the combination of coding and statistics and math and domain knowledge. Now let's say a little more about coding. The reason coding is important because it helps you gather and prepare the data, because a lot of the data comes from novel sources and it's not necessarily ready for you to gather. And it can be in very unusual formats. And so coding is important because it can require some real creativity to get the data from the sources to put it into your analysis. Now, a few kinds of coding that are important. For instance, there's statistical coding. A couple of major languages in this are our and Python to open source free programming languages are specifically for data Python's general purpose but well adapted to data. The ability to work with databases is important to the most common language there is SQL usually pronounced SQL which stands for structured query language, because that's where the data is.  Now most common language there is bash, which actually stands for born again shell. And then searching is important and regex or regular expressions. While there's not a huge amount to learn there it's a it's a small little field is sort of like super powered wildcard searching that makes it possible for you to both find the data and reformat it in ways that are going to be helpful for your analysis. Now let's say a few things about the math. You're going to need things like a little bit of probability some algebra, of course, regression very common statistical procedure, those things are important. And the reason you need the math is because that's going to help you choose the appropriate procedures to answer the question with the data that you have. And probably even more importantly, it's going to help you diagnose problems when things don't go as expected. And given that you're trying to do new things with new data in new ways, you're probably going to come across problems. And so the ability to understand the mechanics of what's going on is going to give you a big advantage. And the third element of the data science Venn diagram is some sort of domain expertise. Think of it as expertise in the field that you're in. Business settings are common. You need to know about the goals of that field, the methods that are used, the constraints that people come across. And it's important because whatever your results are, you need to be able to implement them well. Data science is very practical and it's designed to accomplish something. And your familiarity with a particular field of practice is going to make it that much easier and more impactful when you implement the results of your analysis. Now let's go back to our Venn diagram here just for a moment. Because this is a Venn, we also have these intersections of two circles at a time. At the top is machine learning. At the bottom right is traditional research. And on the bottom left is what drew Conway called the danger zone. Let me talk about each of these. It's off machine learning or ML. Now you think about machine learning. And the idea here is that it represents coding or statistical programming and mathematics without any real domain expertise. Sometimes these are referred to as black box models that you kind of throw data in. And you don't even necessarily have to know what it means or what language it's in and it'll just kind of crunch through it all and it'll give you some regularities. That can be very helpful. But machine learning is considered slightly different from data science because it doesn't involve the particular applications in a specific domain. Also, there's traditional research. This is where you have math or statistics and you have domain knowledge, often very intensive domain knowledge, but without the coding or programming. Now you can get away with that because the data that you use in traditional research is highly structured. It comes in rows and columns is typically complete. And it's typically ready for analysis. Doesn't mean your life is easy, because now you have to expand an enormous amount of effort in the method in designing the project and in the interpretation of the data. So still very heavy intellectual cognitive work, but it comes in a different place. And then finally, there's what Conway called the danger zone. And that's the intersection of coding and domain knowledge, but without math or statistics. Now, he says it's unlikely to happen. And that's probably true. On the other hand, I can think of some common examples where they're called word counts, where you take a large document or series of documents, and you count how often each word appears in there. That can actually tell you some important things. And also drawing maps and showing how things change across place and maybe across time. You don't necessarily have to have the math, but it can be very insightful and helpful. So let's think about a couple of backgrounds where people come from here. First is coding. You can have people who are coders who can do math, stats and business. So you get the three things. And this is probably the most common most of the people come from a programming background. On the other hand, there's also stats or statistics. And you can get statisticians who can code and who also can do business. That's less common, but it does happen. And finally, there's people who come into data science from a particular domain. These are for instance, business people who can code and do numbers. And they're the least common. But all of these are important to data science. And so in some, here's what we can take away. First, several fields make up data science. Second, diverse skills and backgrounds are important and their need is in data science. And third, there are many roles involved, because there's a lot of different things that need to happen. We'll say more about that in our next movie. The next step in our data science introduction and our definition of data science is to talk about the data science pathway. So I like to think of this as when you're working on a major project, you got to do one step at a time to get from here to there. In data science, you can take the various steps and can put them into a couple of general categories. First, there are the steps that involve planning. Second, there's the data prep. Third, there's the actual modeling of the data. And fourth, there's the follow up. And there are several steps within each of these, I'll explain each of them briefly. First, let's talk about planning. The first thing you need to do is you need to define the goals of your project. So you know how to use your resources well, and also so you know when you're done. Second, you need to organize your resources. So you might have data from several different sources, you might have different software packages, you might have different people, which gets us to the third one, you need to coordinate the people so they can work together productively. If you're doing a handoff, it needs to be clear who's going to do what and how their work is going to go together. And then really to state the obvious, you need to schedule the project so things can move along smoothly, you can finish in a reasonable amount of time. Next is the data prep where you're taking like food prep and getting the wrong ingredients ready. First, of course, is you need to get the data and it can come from many different sources and be in many different formats. You need to clean the data. And the sad thing is, this tends to be a very large part of any data science project. And that's because you're bringing in unusual data from a lot of different places. You also want to explore the data, that is really see what it looks like, how many people are in each group, what the shape of the distributions are like, what's associated with what. And you may need to refine the data. And that means choosing variables to include choosing cases to include or exclude making any transformations to the data you need to do. And of course, these steps kind of can bounce back and forth from one to the other. The third group is modeling or statistical modeling. This is where you actually want to create the statistical model. So for instance, you might do a regression analysis, or you might do a neural network. But whatever you do, once you create your model, you have to validate the model. You might do that with a holdout validation, you might do it really with a very small replication, if you can. You also need to evaluate the model. So once you know that the model is accurate, what does it actually mean and how much does it tell you. And then finally, you need to refine the model. So for instance, there may be variables you want to throw out there may be additional ones you want to include. You may want to again, transform some of the data, you may want to get it so it's easier to interpret and apply. And that gets us to the last part of the data science pathway. And that's follow up. And once you've created your model, you need to present the model, because it's usually work that's being done for a client could be in house could be a third party. But you need to take the insights that you got and share them in a meaningful way with other people. You also need to deploy the model. It's usually being done in order to accomplish something. So for instance, if you're working with an e-commerce site, you may be developing a recommendation engine that says people who bought this and this might buy this. You need to actually stick it on the website and see if it works the way you expected it to. Then you need to revisit the model, because a lot of times the data that you worked on is not necessarily all of the data. And things can change when you get out in the real world, or things just change over time. And so you have to see how well your model is working. And then just to be thorough, you need to archive the assets document what you have and make it possible for you or for others to repeat the analysis or develop off of it in the future. So those are the general steps of what I consider the data science pathway. And in some what we get from this is three things. First data science isn't just a technical field. It's not just coding. Things like planning and presenting and implementing are just as important. Also contextual skills, knowing how it works in a particular field, knowing how it will be implemented. Those skills matter as well. And then as you got from this whole thing, there's a lot of things to do. And if you go one step at a time, there'll be less backtracking, and you'll ultimately be more productive in your data science projects. We'll continue our definition of data science by looking at the roles that are involved in data science, the way that different people can contribute to it. That's because it tends to be a collaborative thing. And it's nice to be able to say that we're all together working together towards a single goal. So let's talk about some of the roles involved in data science and how they contribute to the projects. First off, let's take a look at engineers. These are people who focus on the back end hardware, for instance, the servers and the software that runs them. This is what makes data science possible. And it includes people like developers software developers or database administrators, and they provide the foundation for the rest of the work. Next, you can also have people who are big data specialists. These are people who focus on computer science and mathematics. And they may do machine learning algorithms as a way of processing very large amounts of data. And they often create what are called data products. So a thing that tells you what restaurant to go to or that says you might know these friends or provides ways of linking up photos. Those are data products, and those often involve a huge amount of very technical work behind them. There are also researchers. These are people who focus on domain specific research. So for instance, physics or genetics or whatever. And these people tend to have very strong statistics, and they can use some of the procedures and some of the data that comes from the other people like the big data researchers. But they focus on the specific questions. Also, in the data science realm, you'll find analysts, these are people who focus on the day to day tasks of running a business. So for instance, they might do web analytics like Google analytics, or they might pull data from a sequel database. And this information is very important and good for business. And so analysts are key to the day to day functioning of business. But you know, they may not exactly be data science proper, because most of the data they're they're working with is going to be pretty structured. Nevertheless, they play a critical role in business in general. And then speaking of business, you have the actual business people the men and women who organize and run businesses. These people need to be able to frame business relevant questions that can be answered with the data. Also, the business person manages the project and the efforts and the resources of others. And while they may not actually be doing the coding, they must speak data, they must know how the data works, what it can answer and how to implement it. You can also have entrepreneurs. So you might have, for instance, a data startup, they're starting their own little social network of their own little web search platform. An entrepreneur needs data and business skills. And truthfully, they have to be creative at every step along the way, usually because they're doing it all themselves at a smaller scale. Then we have in data science something known as the full stack unicorn. And this is a person who can do everything at an expert level. And they're called a unicorn because truthfully, they may not actually exist. I'll have more to say about that later. But for right now, we can sum up what we got out of this video by three things. Number one, data science is diverse. There's a lot of different people who go into it. And they have different goals for their work, and they bring in different skills and different experiences and different approaches. Also, they tend to work in very different contexts. An entrepreneur works in a very different place from a business manager works in a very different place from an academic researcher. But all of them are connected in some way to data science and make it a richer field. The last thing I want to say in data science and introduction where I'm trying to define data science is to talk about teams in data science. The idea here is that data science has many different tools and different people are going to be experts in each one of them. Now, you have for instance, coding, and you have statistics. Also, you have fields like design or business and management that are involved. And the question of course is, who can do all of it? Who's able to do all of these things at the level that we need? Well, that's where we get this saying I've mentioned it before, it's the unicorn. And just like in ancient history, the unicorn is a mythical creature with magical abilities. In data science, it works a little differently. It is a mythical data scientist with universal abilities. The trouble is, as we know from the real world, there's really no unicorns, animals, and there's really not many unicorns in data science. Really, there's just people. And so we have to find out how we can do the projects, even though we don't have this one person who can do everything for everybody. So let's take a hypothetical case just for a moment. I'm going to give you some fictional people. Here is my fictional person auto, who has strong visualization skills who has good coding, but has limited analytics or statistical ability. And if we graph his stuff out his ability, so here we got five things that we need to have happen. And for the project to work, they all have to happen at at least a level of eight on the zero to 10. If we take his coding ability, he's almost there. Statistics, not quite halfway. Graphics, yes, he can do that. And then business. All right, and project pretty good. So what you can see here is in only one of these five areas is auto sufficient on his own. On the other hand, let's pair him up with somebody else. Let's take a look at Lucy. And Lucy has strong business training has good tech skills, but has limited graphics. And so if we get her profile on the same thing that we saw, there's coding, pretty good statistics, pretty good graphics, not so much business, good and projects. Okay, now the important thing here is that we can make a team. So let's take our two fictional people auto and Lucy, and we can put together their abilities. Now actually have to change the scale here a little bit to accommodate the both of them. But our criterion still is at eight, we need a level of eight in order to do the project competently. And we combine them. Oh, look, coding is now past eight. Statistics is past eight graphics is way past business way past. And then the projects there too. And so when we combine their skills, we're able to get the level that we need for everything, or to put it another way we have now created a unicorn by team. And that makes it possible to do the data science project. So in some, you usually can't do data science on your own, that's a very rare individual. Or more specifically, people need people. And in data science, you have the opportunity to take several people and make collective unicorns so you can get the insight that you need in your project and you can get the things done that you want. In order to get a better understanding of data science, it can be helpful to look at contrasts between data science and other fields. Probably the most informative is with big data, because these two terms are actually often confused. It makes me think of situations where you have two things that are very similar, but not the same like we have here in the Piazza San Carlo in turn Italy. Part of the problem stems from the fact that data science and big data both have Venn diagrams associated with them. So for instance, Venn number one for data science is something we've seen already. We have three circles. And we have coding and we have math and we have some domain expertise that put together get data science. On the other hand, Venn diagram number two is for big data. It also has three circles. And we have the high volume of data, the rapid velocity of data and the extreme variety of data. Take those three v's together, you get big data. Now, we can also combine these two if we want in a third Venn diagram, we call it big data and data science. This time is just two circles with big data on the left and data science on the right. And the intersection there in the middle is big data science, which actually has a real term. But if you want to do a compare and contrast, it kind of helps to look at how you can have one without the other. So let's start by looking at big data without data science. So these are situations where you may have the volume or velocity of variety data, but don't need all the tools of data science. So we're just looking at the left side of the equation right now. Now, truthfully, this only works if you have big data without all three v's. Some say you have to have the volume velocity and variety for to count as big data. I basically say anything that doesn't fit into a standard machine is probably big data. I can think of a couple of examples here of things that might count as big data, but maybe don't count as data science. Machine learning, where you can have very large data sets and probably very complex doesn't require much domain expertise. So that may not be data science word counts, where you have an enormous amount of data. And it's actually a pretty simple analysis. Again, doesn't require much sophistication in terms of quantitative skills or even domain expertise. So maybe maybe not data science. On the other hand, to do any of these, you're going to need to have at least two skills, you're going to need to have the coding, and you will probably have to have some sort of quantitative skills as well. So how about data science without big data? That's the right side of this diagram. Well, to make that happen, you're probably talking about data with just one of the three v's from big data. So either volume or velocity or variety, but singly. So for instance, genetics data, you have a huge amount of data. And it comes in a very set structure, and it tends to come in at once. So you got a lot of volume. And it's a very challenging thing to work with, you have to use data science, but it may or may not count as big data. Similarly, streaming sensor data, where you have data coming in very quickly, but you're not necessarily saving it, you're just looking at these windows in it. That's a lot of velocity. And it's difficult to deal with it takes data science, the full skill set, but it may not require big data per se. Or facial recognition where you have enormous variety in the data, because you're getting photos or videos that are coming in. Again, very difficult to deal with requires a lot of ingenuity and creativity may or may not count as big data depending on how much of a stickler you are about definitions. Now, if you want to combine the two, we can talk about big data science. And in that case, we're looking right here at the middle. This is a situation where you have volume and velocity and variety in your data. And truthfully, if you have the three of those, you are going to need the full data science skill set, you're going to need coding and statistics and math, and you're going to have to have domain expertise, primarily because of the variety you're dealing with. But taken all together, you do have to have all of it. So in sum, here's what we get. Big data is not equal to is not identical to data science. Now there's common ground. And a lot of people who are good at big data are good at data science and vice versa, but they are conceptually distinct. On the other hand, there is the shared middle ground of big data science that unifies the two separate fields. Another important contrast you can make in trying to understand data science is to compare it with coding or computer programming. Now this is where you're trying to work with a machine and you're trying to talk to that machine to get it to do things. In one sense, you can think of coding as just giving task instructions how to do something. And it's a lot like a recipe when you're cooking. You get some sort of user input or other input. And then maybe you have if then logic and you get output from it to take an extremely simple example. If you're programming and Python version two, you write print and then quotes hello world and that will put the words hello world on the screen.  Very simple programming. Now coding and data gets a little more complicated. So for instance, there's word counts where you take a book or a whole collection of books, you take the words and you count how many there are in there. Now this is this is a conceptually simple task. And domain expertise and really math and statistics are not vital. But to make valid inferences and generalizations, in the face of variability and uncertainty in the data, you need statistics and by extension, you need data science. It might help to compare the two by looking at the tools of the respective trades. So for instance, there are tools for coding or generic computer programming. And there are tools that are specific to data science. So what I have right here is a list from the I triple e of the top 10 programming languages in 2015. And it starts at Java and C and goes down to shell. And some of these are also used for data science. So for instance, Python, and our and SQL are used for data science. But the other ones aren't major ones in data science. So let's in fact, take a look at a different list of most popular tools for data science. And you see that things move around a little bit now ours at the top, SQL's there, Python's there. But for me, what's the most interesting on this list is that Excel is number five, which would never be considered programming per se, but is in fact a very important tool for data science. And that's one of the ways that we can compare and contrast computer programming with data science. In sum, we can say this data science is not equal to coding, they're different things. On the other hand, they share some of the tools, and they share some practices, specifically when coding for data. On the other hand, there is one very big difference in that statistics, statistical ability is one of the major separators between general purpose programming and data science programming. When we talk about data science, and we're contrasting it with some fields, another field that a lot of people get confused and think they're the same thing is data science and statistics. Now, I'll tell you, there's a lot in common, but we can talk a little bit about the different focuses of each. And we also get into the issue of sort of definitionalism that data science is different, because we define it differently, even when there's an awful lot in common between the two. It helps to take a look at some of the things that go on in each field. So let's start here about statistics, put little circle here, and we'll put data science. And to borrow a term from Steven J. Gould, we can call these non overlapping magisteria Noma. So you think of them as separate fields that are sovereign unto themselves with nothing to do with each other. But you know, that doesn't seem right. And part of that is if we go back to the data science Venn diagram, you know, statistics is one part of it, there it is in the top corner. So now what do we do? What's the relationship? So it doesn't make sense to say these are totally separate areas. Maybe data science and statistics because they share procedures, maybe data science is a subset or a specialty of statistics more like this. But if data science were just a subset or specialty within statistics, then it would follow that all data scientists would first be statisticians. And interestingly, that's just not so. Say for instance, we take a look at the data science stars, the superstars in the field, we go to a rather intimidating article, it's called the world's seven most powerful data scientists from Forbes.com. And you can see the article if you go to this URL. There's actually more than seven people on the list because sometimes he brings them up in pairs. But let's check their degrees see what their academic training is in. If we take all the people on this list, we have five degrees in computer science, three in math, two in engineering, and one each in biology, economics, law, speech pathology, and one in statistics. And so that tells us, of course, that these major people in data science are not trained as statisticians, only one of them has formal training in that. So that gets us to the next question, where do these two fields, statistics and data science diverge because they seem like they should have a lot in common, but they don't have a lot in training. And specifically, we can look at the training, most data scientists are not trained formally as statisticians. Also in practice, things like machine learning and big data, which are central to data science, are not shared generally with most of statistics. And so they have separate domains there. And then there's the really important issue of context. Data scientists tend to work in different settings than statisticians. Specifically, data scientists very often work in commercial settings where they're trying to get recommendation engines or ways of developing a product that will make them money. So maybe instead of having data science as a subset of statistics, we can think of it more as these two fields have different niches, they both analyze data, but they do different things in different ways. So maybe it's fair to say they share they overlap, they both have analysis in common of data. But otherwise, they are ecologically distinct. So in some, what we can say here is that data science and statistics, both use data and they analyze it. But the people in each tend to come from different backgrounds, and they tend to function with different goals and contexts. And in that way, render them to be conceptually distinct fields, despite the apparent overlap. As we work to get a grasp on data science, there's one more contrast I want to make explicitly. And that's between data science and business intelligence or BI. The idea here is that business intelligence is data in real life. It's very, very applied stuff. The purpose of BI is to get data on internal operations on market competitors and so on, and make justifiable decisions, as opposed to just sitting in the bar and doing whatever comes to your mind. Now, data science is involved with this, except, you know, really, there's no coding in BI, there's using apps that already exist. And the statistics in business intelligence tend to be very simple. They tend to be counts and percentages and ratios. And so it's simple. The light bulb is simple. It just does this one job. There's nothing super sophisticated there. Instead, the focus in business intelligence is on domain expertise, and on really useful direct utility. It's simple, it's effective. And it provides insight. Now, one of the main associations with business intelligence is what are called dashboards or data dashboards, they look like this, it's a collection of charts and tables that go together to give you a very quick overview of what's going on your business. And while a lot of data scientists may let's say look down their nose upon dashboards, I'll say this, most of them are very well designed and you can learn a huge amount about user interaction, and the accessibility of information from dashboards. So really, where does data science come into this? What's the connection between data science and business intelligence? Well, data science can be useful to BI, in terms of setting it up identifying data sources and creating or setting up the framework for something like a dashboard or a business intelligence system. Also, data science can be used to extend it. Data science can help get past the easy questions and the easy data to get the questions that are actually most useful to you, even if they require really sometimes data that's hard to wrangle and work with. And also, there's an interesting interaction here that goes the other way. Data science practitioners can learn a lot about design from good business intelligence applications. So I strongly encourage anybody in data science to look at them carefully and see what they can learn. In some business intelligence or BI is very goal oriented. Data science perhaps prepares the data and sets up the form for business intelligence. But also data science can learn a lot about usability and accessibility from business intelligence. And so it's always worth taking a close look. Data science has a lot of really wonderful things about it. But it is important to consider some ethical issues. And I'll specifically call this do no harm in your data science projects. And for that, we can say thanks to Hippocrates, the guy who gave us the Hippocratic oath of do no harm. Let's specifically talk about some of the important ethical issues very briefly that come up in data science. Number one is privacy, that data tells you a lot about people and you need to be concerned about the confidentiality. If you have private information about people, their names, their social security numbers, their addresses, their credit scores, their health, that's private, that's confidential. And you shouldn't share that information unless they specifically gave you permission. Now, one of the reasons this presents a special challenge in data science, because we'll see later, a lot of the sources that are used in data science, we're not intended for sharing if you scrape data from a website or from PDFs, you need to make sure that it's okay to do that. But it was originally created without the intention of sharing. So privacy is something that really falls upon the analyst to make sure they're doing it properly. Next is anonymity. One of the interesting things we find is that it's really not hard to identify people and data. If you have a little bit of GPS data, and you know where a person was at four different points in time, you have about a 95% chance of knowing exactly who they are. You look at things like HIPAA, that's the Health Insurance Portability and Accountability Act. Before HIPAA, it was really easy to identify people for medical records. Since then, it has become much more difficult to identify people uniquely. That's an important thing for really people's well being. And then also proprietary data, if you're working for a client, a company, and they give you their own data, that data may have identifiers, you may know who the people are and they're not anonymous anymore. So anonymity may or may not be their major efforts to make data anonymous. But really, the primary thing is that even if you do know who they are, that you still maintain the privacy and confidentiality of the data. Next, there's the issue about copyright, where people try to lock down information. Now, just because something is on the web doesn't mean that you're allowed to use it. Scraping data from websites is a very common and a useful way of getting data for projects. You can get data from web pages from PDFs from images from audio from really a huge number of things. But again, the assumption that because it's on the web, it's okay to use it is not true. You always need to check copyright and make sure that it's acceptable for you to access that particular data. Next in our very ominous picture is data security. And the idea here is that when you go through all the effort to gather data to clean it up and prepare for an analysis, you've created something that's very valuable to a lot of people. And you have to be concerned about hackers trying to come in and steal the data, especially if the data is not anonymous and it has identifiers in it. And so there is an additional burden placed on the analyst to ensure to the best of their ability that the data is safe and cannot be broken into and stolen. And that can include very simple things like a person who is on the project but is no longer but took the data on a flash drive, you have to find ways to make sure that that can't happen as well. There's a lot of possibilities. It's tricky. But it's something that you have to consider thoroughly. Now, two other things that come up in terms of ethics, but don't usually get addressed in these conversations. Number one is potential bias. The idea here is that the algorithms or the formulas that are used in data science are only as neutral and bias free as the rules and the data that they get. And so the idea here is that if you have rules that address something that is associated with for instance, gender or age or race or economic standing, you might unintentionally be building in those factors, which say for instance for title nine, you're not supposed to be building those into the system without being aware of it. And an algorithm has this sheen of objectivity, and people can say they can place confidence in it without realizing that it's replicating some of the prejudices that may happen in real life. Another issue is overconfidence. And the idea here is that analyses are limited simplifications, they have to be that that's just what they are. And because of this, you still need humans in the loop to help interpret and apply this. The problem is when people run an algorithm to get out a number say to 10 decimal places and they say this must be true and treat it as written in stone, absolutely unshakable truth. When in fact, if the data were biased going in, if the algorithms were incomplete, if the sampling was not representative, you can have enormous problems and go down the wrong path with too much confidence in your own analyses. So once again, humility is in order when doing data science work. In sum, data science has enormous potential, but it also has significant risks involved in the projects. Part of the problem is that analyses can't be neutral, that you have to look at how the algorithms are associated with the preferences, prejudices and biases of the people who made them. And what that means is that no matter what, good judgment is always vital to the quality and success of a data science project. Data science is a field that is strongly associated with its methods or procedures. In this section of videos, we're going to provide a brief overview of the methods that are used in data science. Now, just as a quick warning, in this section, things can get kind of technical and that can cause some people to sort of freak out. But this course is a non technical overview. The technical hands on stuff is in the other courses. And it's really important to remember that tech is simply the means to doing data science. Insight or the ability to find meaning in your data. That's the goal tech only helps you get there. And so we want to focus primarily on insight and the tools and the tech as they serve to further that goal. Now there's a few general categories we're going to talk about again, with an overview for each of these. The first one is sourcing or data sourcing, that is how to get the data that goes into data science, the raw materials that you need. The second is coding that again is computer programming that can be used to obtain and manipulate and analyze the data. After that, a tiny bit of math that is the mathematics behind data science methods that really form the foundations of the procedures. And then stats, the statistical methods that are frequently used to summarize and analyze data, especially as applied to data science. And then there's machine learning ML. This is a collection of methods for finding clusters in the data for predicting categories or scores on interesting outcomes. And even across these five things, even then, the presentations aren't too techy crunchy, they're basically still friendly. And you know, really, that's the way it is. And so that is the overview of the overviews. In sum, we need to remember that data science includes tech, but data science is greater than tech, it's more than those procedures. And above all, that tech while important to data science is still simply a means to insight in data. The first step in discussing data science methods is to look at the methods of sourcing or getting data that's used in data science. You can think of this as getting the raw materials that go into your analysis. Now you've got a few different choices when it comes to this in data science. You can use existing data, you can use something called data APIs, you can scrape web data, or you can make data. We'll talk about each of those very briefly in a non technical manner. But right now, let me say something about existing data. This is data that already is at hand and it might be in house data. So if you work for a company, it might be your company records. Or you might have open data, for instance, many governments, many scientific organizations make their data available to the public. And then there's also third party data, this is usually data that you buy from a vendor, but it exists and it's very easy to plug it in and go. You can also use API's. Now that stands for application programming interface. And this is something that allows various computer applications to communicate directly with each other. It's like phones for your computer programs. It's the most common way of getting web data and the beautiful thing about it is it allows you to import that data directly into whatever programmer application you're using to analyze the data. Next is scraping data. And this is where you want to use data that's on the web, but they don't have an existing API. And what that means is usually data that's an HTML web tables and pages, maybe PDFs and you can do this either with using specialized applications for scraping data, or you can do it in a programming language like our Python and write the code to do the data scraping. Or another option is to make data and this lets you get exactly what you need. You can be very specific. And you can get what you need. You can do something like interviews or you can do surveys or you can do experiments. There's a lot of approaches. Most of them require some specialized training in terms of how to gather quality data. And that's actually important to remember because no matter what method you use for getting or making new data, you need to remember this one little aphorism you may have heard from computer science. It goes by the name of Gigo that actually stands for garbage in garbage out. And it means if you have bad data that you're feeding into your system, you're not going to get anything worthwhile any real insights out of it. Consequently, it's important to pay attention to metrics or methods for measuring and the meaning exactly what it is that they tell you. There's a few ways you can do this. For instance, you can talk about business metrics, you can talk about KPIs, which means key performance indicators also used in business settings, or smart goals, which is the way of describing the goals that are actionable and timely and so on. You can also talk about in a measurement sense classification accuracy. And I'll discuss each of those in a little more detail in a later movie. But for right now, in some we can say this data sourcing is important because you need to get the raw materials for your analysis. The nice thing is there's many possible methods, many ways that you can use to get the data for data science. But no matter what you do, it's important to check the quality and the meaning of the data so you can get the most insight possible out of your project. The next step we need to talk about in data science methods is coding. And I'm going to give you a very brief non technical overview of coding and data science. The idea here is that you're going to get in there and you are going to be king of the jungle, master of your domain, and make the data jump when you need it to jump. Now, if you remember when we talked about the data science van diagram in the beginning, coding is up here on the top left. And while we often think about sort of people typing lines of code, which is very frequent. It's more important to remember when we talk about coding or just computers in general, what we're really talking about here is any technology that lets you manipulate the data in the ways you need to perform the procedures you need to get the insight that you want out of your data. Now, there are three very general categories that we'll be discussing here on data lab. The first is apps. These are specialized applications or programs for working with data. The second is data or specifically data formats. There are special formats for web data. I'll mention those in a moment. And then code. There are programming languages that give you full control over what the computer does and how you interact with the data. Let's take a look at each one very briefly. In terms of apps, they're spreadsheets like Excel or Google Sheets. These are the fundamental data tools of probably the majority of the world. There are specialized applications like Tableau for data visualization, or SPSS, a very common statistical package in the social sciences and in business. And one of my favorite, JASP, which is a free open source analog of SPSS, which actually I think is a lot easier to use and replicate research with. And there are tons of other choices. Now, in terms of web data, it's helpful to be familiar with things like HTML and XML and JSON and other formats that are used to encapsulate data on the web.  And then there are actual coding languages. R is probably the most common along with Python, general purpose language, but it's been well adapted for data use. There's SQL, the structured query language for databases, and very basic languages like C and C++ and Java, which are used more in the back end of data science. And then there's bash, the most common command line interface, and regular expressions. And we'll talk about all of these in other courses here at Datalab. But remember this, tools are just tools. They're only one part of the entire data science process. There are means to the end. And the end, the goal is insight, you need to know where you're trying to go, and then simply choose the tools that help you reach that particular goal. That's the most important thing. So in sum, here's a few things. Number one, use your tools wisely. Remember, your questions need to drive the process, not the tools themselves. Also, I'll just mention that a few tools is usually enough, you can do an awful lot with Excel and R. And then the most important thing is focus on your goal and choose your tools and even your data to match the goal. So you can get the most useful insights from your data. The next step in our discussion of data science methods is mathematics. And I'm going to give a very brief overview of the math involved in data science. Now, the important thing to remember is that math really forms the foundation of what we're going to do. If you go back to the data science Venn diagram, we've got stats up here in the right corner, but really, it's math and stats or quantitative ability in general. But we'll focus on the math part right here. And probably the most important question is how much math is enough to do what you need to do, or to put it another way. Why do you need math at all? Because you've got a computer to do it. Well, I can think of three reasons you don't want to rely on just the computer, but it's helpful to have some sound mathematical understanding. Here they are. Number one, you need to know which procedures to use and why. So you have your question, you have your data, you need to have enough of an understanding to make an informed choice. That's not terribly difficult. Two, you need to know what to do when things don't work right. Sometimes you get impossible results. I know in statistics, you can get a negative adjusted R squared, that's not supposed to happen. And it's good to know the mathematics that go into calculating that so you can understand how something apparently impossible can work. Or you're trying to do a factor analysis or principal component to get a rotation that won't converge. It helps to understand what it is about the algorithm that's happening and why that won't work in that situation. And number three, interestingly, some procedures, some math is easier and quicker to do by hand than by firing up the computer. And I'll show you a couple of examples in later videos where that can be the case. Now, fundamentally, there's a nice sort of analogy here. Math is to data science. As for instance, chemistry is to cooking, kinesiology is to dancing and grammar is to writing. The idea here is that you can be a wonderful cook without knowing any chemistry. But if you know some chemistry, it's going to help. You can be a wonderful dancer without knowing kinesiology, but it's going to help. And you can probably be a good writer without having an explicit knowledge of grammar, but it's going to make a big difference. The same thing is true of data science, you will do it better if you have some of the foundational information. So the next question is what kinds of math do you need for data science? Well, there's a few answers to that. Number one is algebra. You need some elementary algebra. That's the basically simple stuff. You can have to do some linear or matrix algebra because that's the foundation of a lot of the calculations. And you can also have systems of linear equations where you're trying to solve several equations all at once. It's a tricky thing to do in theory, but this is one of the things that's actually easier to do by hand sometimes. Now there's more math. You can get some calculus, you can get some big O, which has to do with the order of a function, which has to do with sort of how fast it works. Probability theory can be important. And then Bayes theorem, which is a way of getting what's called a posterior probability can also be a really helpful tool for answering some fundamental questions in data science. So in sum. A little bit of math can help you make informed choices when planning your analyses. Very significantly, it can help you find the problems and fix them when things aren't going right. It's the ability to look under the hood that makes a difference. And then truthfully, some mathematical procedures like systems of linear equations that can even be done by hand sometimes faster than you can do with a computer. So you can save yourself some time and some effort and move ahead more quickly towards your goal of insight. Now data science wouldn't be data science and its methods without a little bit of statistics. So I'm going to give you a brief statistics overview here of how things work in data science. Now you can think of statistics as really an attempt to find order in case find patterns in an overwhelming mess sort of like trying to see the forest and the trees.  We recently had math and stats here in the top corner and we're going to go back to talking about stats in particular. What you're trying to do here, one thing is to explore your data. You can have exploratory graphics because we're visual people and it's usually easiest to see things. You can have exploratory statistics, a numerical exploration of the data and you can have descriptive statistics, which are the things that most people would have talked about when they took a statistics class in college if they did that. Next, there's inference. I've got smoke here because you can infer things about the wind and the air movement by looking at patterns and smoke. The idea here is that you're trying to take information from samples and infer something about a population. You're trying to go from one source to another. One common version of this is hypothesis testing. Another common version is estimation, sometimes called confidence intervals. There are other ways to do it, but all of these let you go beyond the data at hand to making larger conclusions. Now, one interesting thing about statistics is you're going to have to be concerned with some of the details and arranging things just so. For instance, you get to do something like feature selection that's picking variables that should be included or combinations. And there are problems that can come up. There are frequent problems and I'll address some of those in later videos. There's also the matter of validation. When you create a statistical model, you have to see if it actually is accurate. Hopefully you have enough data that you can have a holdout sample and do that, or you can replicate the study. Then there's the choice of estimators that you use, how you actually get the coefficients or the combinations in your model. And then there's ways of assessing how well your model fits the data. All of these are issues that I'll address briefly when we talk about statistical analysis at greater length. Now, I do want to mention one thing in particular here. And I just call this beware the trolls. There are people out there who will tell you that if you don't do things exactly the way they say to do it, that your analysis is meaningless, that your data is junk, and you've lost all your time. You know what, they're trolls. So the idea here is, don't listen to that. You can make enough of an informed decision on your own to go ahead and do an analysis that is still useful. Probably one of the most important things to think about in this is this wonderful quote from a very famous statistician that says, all models or all statistical models are wrong. But some are useful. And so the question isn't whether you're technically right or you have some sort of level of intellectual purity, but whether you've done something that is useful. That, by the way, comes from George Box. And I like to think of it basically as this as wave your flag, wave your do it yourself flag and just take pride in what you're able to accomplish, even when there are people who may be criticizing it.  And so in some statistics allow you to explore and describe your data, they allow you to infer things about the population. There's a lot of choices available, a lot of procedures. But no matter what you do, the goal is useful insight. Keep your eyes on that goal, and you will find something meaningful and useful in your data to help you in your own research and projects. Let's finish our data science methods overview by getting a brief overview of machine learning. Now I got to admit, when you say the term machine learning, people start thinking about something like the robot overlords are going to take over the world. That's not what it is. Instead, let's go back to our Venn diagram one more time. And in the intersection at the top between coding and stats is machine learning, or as it's commonly called is just ML. The goal of machine learning is to go and work in a data space. So you can, for instance, take a whole lot of data, we've got tons of books here. And then you can reduce the dimensionality that is take a very large scattered data set and try to find the most essential parts of that data. And then you can use these methods to find clusters within the data. Like goes with like, you can use methods like K means you can also look for anomalies or unusual cases that show up in the data space. Or if we go back to categories, again, I talked about like for like, you can use things like logistic regression or K nearest neighbors K and N, you can use naive Bayes for classification or decision trees or SVM, which is support vector machines, or artificial neural nets. Any of those will help you find the patterns and the clumping in your data. So you can get similar cases next to each other and get the cohesion that you need to make conclusions about these groups. Also, a major element of machine learning is predictions. You're going to point your way down the road. The most common approach here, the most basic is linear regression, multiple regression. There's also Poisson regression, which is used for modeling count or frequency data. And then there's the issue of ensemble models where you create several models and you take the predictions from each of those and you put them together to get an overall more reliable prediction. Now, I'll talk about each of these in a little more detail in later courses. But for right now, I mostly just want you to know that these things exist. And that's what we mean when we refer to machine learning. So in some machine learning can be used to categorize cases and to predict scores on outcomes. And there's a lot of choices, many choices and procedures available. But again, as I said with statistics, and I'll say again, many times after this, no matter what the goal is not that I'm going to do an artificial neural network or an SVM. The goal is to get useful insight into your data. Machine learning is a tool and use it to the extent that it helps you get that insight that you need. In the last several videos, I've talked about the role and data science of technical things. On the other hand, communicating is also central to the practice. And the first thing I want to talk about there is interpretability. The idea here is that you want to be able to lead people through a path on your data, you want to tell a data driven story. And that's the entire goal of what we're doing with data science. Now, another way to think about this is when you're doing your analysis, what you're trying to do is solve for value. You're making an equation, you take the data, you're trying to solve for value. The trouble is this, a lot of people get hung up on analysis, but they need to remember that analysis is not the same thing as value. Instead, I like to think of it this way, that analysis times story is equal to value. Now, please note, that's multiplicative, not additive. And so one consequence of that is when you go back to analysis times story equals value.  So instead of that, let's go back to this and say what we really want to do is we want to maximize the story so that we can maximize the value that results from our analysis. Again, maximum value is the overall goal here. The analysis, the tools, the tech are simply methods for getting to that goal. So let's talk about goals, for instance, analysis is goal driven, you're trying to accomplish something as specific. And so the story or the narrative or the explanation you give about your project should match those goals. If you're working for a client and they had a specific question that they wanted you to answer, then you have a professional responsibility to answer those questions clearly and unambiguously. So they know whether you said yes or no, and they know why you said yes or no. Now part of the problem here is the fact that the client isn't you and they don't see what you do. And as I show here, simply covering your face doesn't make things disappear. You have to worry about a few psychological abstractions. You have to worry about egocentrism. And I'm not talking about being vain. I'm talking about the idea that you think other people see and know and understand what you know. That's not true. Otherwise they wouldn't have hired you in the first place. And so you have to put it in terms that the client works with and that they understand. And you're going to have to get out of your own center in order to do that. Also, there's the idea of false consensus, the idea that well, everybody knows that. And again, that's not true. Otherwise they wouldn't have hired you. You need to understand that they're going to come from a different background with a different range of experience and interpretation. You're going to have to compensate for that. A funny little thing is the idea about anchoring. When you give somebody an initial impression, they use that as an anchor and then they adjust away from it. So if you're going to try to flip things over on their heads, watch out for giving a false impression at the beginning, unless you absolutely need to. But most importantly, in order to bridge the gap between the client and you, you need to have clarity and explain yourself at each step. You can also think about the answers. When you're explaining the project to the client, you might want to start in a very simple procedure, state the question that you're answering. Give your answer to that question. And if you need to qualify as needed, and then go in order top to bottom. So you're trying to make it as clear as possible what you're saying what the answer is and make it really easy to follow. Now, in terms of discussing your process, how you did this all, most of the time it's probably the case that they don't care. They just want to know what the answer is and that you used a good method to do that. So in terms of discussing process or the technical details, only when absolutely necessary, that's something to keep in mind. The process here is to remember that analysis, which means breaking something apart. This, by the way, is a mechanical typewriter broken into its individual components. Analysis means to take something apart. An analysis of data is an exercise in simplification. You're taking the overall complexity, sort of the overwhelmingness of the data, and you're boiling it down and finding the patterns that make sense and serve the needs of your client. Now, let's go to a wonderful quote from our friend Albert Einstein here, who said, Everything should be made as simple as possible, but not simpler. That's true in presenting your analysis. Or if you want to go see the architect and designer Ludwig Mies van der Rohe, who said less is more. It's actually Robert Browning who originally said that, but Mies van der Rohe popularized it. Or if you want another way of putting a principle that comes from my field, I'm actually a psychological researcher. They talk about being minimally sufficient just enough to adequately answer the question. If you're in commerce, you know about a minimal viable product is sort of the same idea with an analysis here, the minimum viable analysis. So here's a few tips. When you're giving a presentation, more charts, less text, great. And then simplify the charts, remove everything that doesn't need to be in there. Generally, you want to avoid tables of data because those are hard to read. And then one more time because I want to emphasize it less text again, charts tables can usually carry the message. And so let me give you an example here. I'm going to give a very famous data set Berkeley admissions. Now these are not stairs to Berkeley, but it gives the idea of trying to get into something that's far off and distant. Here's the data. This is graduate school admissions in 1973. So it's, you know, it's over 40 years ago. But the idea is that men and women were both applying for graduate school at the University of California at Berkeley. And what we found is that 44% of the men who applied were admitted that they're part in green and that of the women, only 35% were admitted when they applied. So really, at first glance, this is bias and it actually led to a lawsuit. It was it was a major issue. So what Berkeley then tried to do is find out, well, which programs are responsible for this bias. And then you got a very curious set of results. If you break the applications down by program, and here we're just calling them a through F six different programs. What you find actually is that in each of these male applicants are on the left, female applicants are on the right. If you look at program a, women actually got accepted at a higher rate. And the same is true for B. And the same is true for D. And the same is true for F. And so this is a very curious set of responses. And it's something that requires explanation. Now in statistics, this is known as Simpson's paradox. But here's the paradox bias may be negligible at the department level. And in fact, as we saw in for the departments, there was a possible bias in favor of women. And the problem is that women applied to more selective programs programs with lower acceptance rates. Now, some people stop right here and say, therefore, nothing's going on. Nothing's going on.  There are other questions that you can ask. And as producing a data driven story, this is stuff that you would want to do. So for instance, you may want to ask, why do the programs vary in overall class size? Why do the acceptance rates differ from one program to the other? So you may want to ask, why do the programs vary in overall class size? Why do the acceptance rates differ from one program to the other? Why do men and women apply to different programs? And you might want to look at things like the admissions criteria for each of the programs, the promotional strategies, how they advertise themselves to students. You might want to look at the kinds of prior education students have in each of the programs. And you really want to look at funding levels for each of the programs. And so really, you get one answer, it leads to more questions, maybe some more answers and more questions, and you need to address enough of this to provide a comprehensive overview and solution to it for your client. In sum, let's say this, stories give value to data analysis. And when you tell the story, you need to make sure that you are addressing your client's goals in a clear, unambiguous way. And the overall principle here is be minimally sufficient. Get to the point, make it clear, say what you need to, provide a comprehensive overview and solution to it for your client. In sum, let's say this, stories give value to data analyses. And when you tell the story, you need to make sure that you are addressing your client's goals in a clear, unambiguous way. And the overall principle here is be minimally sufficient. Get to the point, make it clear, say what you need to, but otherwise be concise and make your message clear. The next step in discussing data science and communicating is to talk about actionable insights or information that can be used productively to accomplish something. Now to give sort of a bizarre segue here, you look at a game controller, it may be a pretty thing, it may be a nice object. But remember, game controllers exist to do something, they exist to help you play the game, and to do it as effectively as possible. They have a function, they have a purpose. Same way, data is for doing. Now that's a paraphrase from one of my favorite historical figures. And this is William James, the father of American psychology and pragmatism and philosophy. And he has this wonderful quote, he said, my thinking is first and last and always for the sake of my doing. And the idea applies to analysis, your analysis and your data is for the sake of your doing. And so you're trying to get some sort of specific insight in how you should proceed. What you want to avoid is the opposite of this from one of my other favorite cultural heroes, the famous Yankees catcher, Yogi Berra, who said, we're lost, but we're making good time. And so the idea here is that frantic activity does not make up for a lack of direction. You need to understand what you're doing so you can reach the particular goal. And your analysis is supposed to do that. So when you're giving your analysis, you're going to try to point the way. Remember, why was the project conducted? The goal is usually to direct some kind of action reach some kind of goal for your client. And that the analysis should be able to guide that action in an informed way. One thing you want to do is you want to be able to give the next steps to your client, give the next steps, tell them what they need to do now. You want to be able to justify each of those recommendations with the data and your analysis. As much as possible, be specific, tell them exactly what they need to do. Make sure it's doable by the client that it's within their range of capability, and that each step should build on the previous step. Now, that being said, there is one really fundamental sort of philosophical problem here. And that's the difference between correlation and causation.  But your client doesn't simply want to know what's associated, they want to know what causes something, because if they're going to do something, that's an intervention is designed to produce a particular result. So really, how do you get from the correlation, which is what you have in the data to the causation, which is what your client wants? Well, there's a few ways to do that. One is experimental studies. These are randomized controlled trials. Now, that's theoretically the simplest path to causality, but it can be really tricky in the real world. There are quasi experiments. And these are methods, a whole collection of methods that use non randomized data, usually observational data, adjusted in particular ways to get an estimate of causal inference. Or there's the theory and experience. And this is research based theory and domain specific experience. And this is where you actually get to rely on your client's information. They can help you interpret the information, especially if they have greater domain expertise than you do. Another thing to think about are the social factors that affect your data. Now, you remember the data science Venn diagram, we've looked at it lots of times, it's got these three elements. Some people have proposed adding a fourth circle to this Venn diagram and we'll kind of put that in there and say that social understanding is also important critical really to valid data science. Now, I love that idea. And I do think that it's important to understand how things are going to play out. There's a few kinds of social understanding you want to be aware of your client's mission. You want to make sure that your recommendations are consistent with your client's mission. Also, that your recommendations are consistent with your client's identity, not just this is what we do, but this is really who we are. You need to be aware of the business context, sort of the competitive environment and the regulatory environment that they're working in, as well as the social context. And that can be outside of the organization, but even more often within the organization, your recommendations will affect relationships within the client's organization. And you're going to try to be aware of those as much as you can to make it so that your recommendations can be realized the way they need to be. So in sum, data science is a goal focused. And when you're focusing on that goal for your client, you need to give specific next steps that are based on your analysis and justifiable from the data. And in doing so, be aware of the social, political and economic context that gives you the best opportunity of getting something really useful out of your analysis. When you're working in data science and trying to communicate your results, presentation graphics can be an enormously helpful tool. Think of it this way, you were trying to paint a picture for the benefit of your client. Now when you're working with graphics, there can be a couple of different goals. It depends on what kind of graphics you're working with. There's the general category of exploratory graphics. These are ones that you were using as the analyst. And for exploratory graphics, you need speed and responsiveness. And so you get very simple graphics, this is a base histogram and R, and they can get a little more sophisticated. And this is done in GG plot, and then you can break it down a couple of histograms or you can make it a different way or make them see through or split them apart into small multiples. But in each case, this is done for the benefit of you as the analysts understanding the data. These are quick, they're effective now, they're not very well labeled, and they're usually for your insight, and then you do other things as a result of that. On the other hand, presentation graphics, which are for the benefit of your client, those need clarity, and they need a narrative flow. Now, let me talk about each of those characteristics very briefly. Clarity versus distraction. There are things that can go wrong in graphics. Number one is colors, colors can actually be a problem. Also, three dimensional or false third dimensions are nearly always a distraction. One that gets a little touchy for some people is interaction, we think of interactive graphics as really cool, great things to have. But you run the risk of people getting distracted by the interaction and start playing around with it. We're like, Oh, I press here, it does that. And that distracts from the message. So actually, it may be important to not have interaction. And then the same thing is true of animation. Flat static graphics can often be more informative because they have fewer distractions in them. Let me give you a quick example of how not to do things. Now, this is a chart that I made, I made it in Excel, and I did it based on some of the mistakes I've seen in graphics submitted to me when I teach. And I guarantee you everything in here I have seen in real life, just not necessarily combined all at once. Let's zoom in on this a little bit so we can see the full badness of this graphic. And let's see what's going on here, we've got a scale here that starts at eight goes to 28% and it's tiny doesn't even cover the range of the data. We've got this bizarre picture on the wall, we have no access lines on the walls. We come down here, the labels for educational levels are in alphabetical order instead of the more logical higher levels of education. Then we've got the data represented as cones, which are difficult to read and compare. And it's only made worse by the colors and the textures. You know, if you want to take an extreme this one for grad degrees doesn't even make it to the floor value of 8%. And this one for high school grad is cut off at the top at 28%. This by the way is a picture of a sheep and people do this kind of stuff and it drives me crazy. If you want to see a better chart with the exact same data, this is it right here. It's a straight bar chart, it's flat, it's as simple, it's as clean as possible. And this is better in many ways. Most effective here is that it communicates clearly, there's no distractions. It's a logical flow. This is going to get the point across so much faster. And I can give you another example of it. Here's a chart I showed previously about salaries for incomes. I have a list here, I've got data scientists in it. If I want to draw attention to it, I have the option of like putting a circle around it. And I can put a number next to it to explain it. That's one way to make it easy to see what's going on. But you don't even have to get fancy. You know, I just got out of pen and a post it note and I drew a bar chart of some real data about life expectancy. This tells the story as well that there is something terribly amiss in Sierra Leone. But now let's talk about creating narrative flow in your presentation graphics. To do this, I'm going to pull some charts from my most cited academic paper, which is called a third voice a review of empirical research on the psychological outcomes of restorative justice. Think of that as mediation for juvenile crimes, mostly juvenile. And this paper is interesting because really it's about 14 bar charts with just enough text to hold them together. And you can see there's a flow. The charts are very simple. This is judgments about whether the criminal justice system was fair. The two bars on the left are victims, the two bars on the right are offenders. And for each group on the left are people who participated in restorative justice or victim offender mediation or mediation for crimes. And for each set on the right are people who went through standard criminal procedures. It says court but it usually means plea bargaining. Anyhow, it's really easy to see that in both cases, restorative justice bar is higher. People were more likely to say it was fair. They also felt that they had an opportunity to tell their story. That's one reason they might think it's fair. They also felt the offender was held accountable more often. In fact, if you go to court on the offenders, that lines below 50%. And that's the offenders themselves making the judgment. Then you can go to forgiveness and apologies. And again, this is actually a simple thing to code. And you can see there's an enormous difference. In fact, one of the reasons there's such a big difference is because in standard court proceedings, the offender very rarely meets the victim.  When we take them out, we see a huge change. And then we can go to whether a person satisfied with the outcome. Again, we see an advantage for restorative justice, whether the victim is still upset about the crime. Now the bars are a little different. And whether they're afraid of revictimization, that's over a two to one difference. And then finally, recidivism for offenders are reoffending. And you see a big difference there. And so what I have here is a bunch of charts that are very, very simple to read. And they kind of flow in how they're giving the overall impression and then detailing it a little bit more. There's nothing fancy here. There's nothing interactive. There's nothing animated. There's nothing kind of flowing in 17 different directions. It's easy, but it follows a story and it tells a narrative about the data. And that should be your major goal with presentation graphics. In some presenting or the graphics they use for presenting are not the same as the graphics you use for exploring, they have different needs and different goals. But no matter what you're doing, be clear in your graphics and be focused in what you're trying to tell. And above all, create a strong narrative that gives a different level of perspective and answers questions as you go to anticipate a client's question and to give them the most reliable, solid information and the greatest confidence in your analysis. The final element of data science and communicating that I wanted to talk about is reproducible research. And you can think of it as this idea, you want to be able to play that song again. And the reason for that is data science projects are rarely one and done, rather they tend to be incremental, they tend to be cumulative, and they tend to adapt to the circumstances that they're working in. So one of the important things here and probably if you want to summarize it very briefly is this show your work. There's a few reasons for this. You may have to revise your research at a later date your own analyses, you may be doing another project and you want to borrow something from previous studies. More likely you'll have to hand it off to somebody else at a future point and they're going to have to be able to understand what you did. And then there's a very significant issue in both scientific and economic research of accountability. You have to be able to show that you did things in a responsible way and that your conclusions are justified. That's for clients, funding agencies, regulators, academic reviewers, any number of people. Now, you may be familiar with the concept of open data, but you may be less familiar with the concept of open data science. And that's more than open data. So for instance, I'll just let you know that there is something called the open data science conference and odse.com. And it meets three times a year in different places. And this is entirely, of course, devoted to open data science, using both open data, but making the methods transparent to people around them. One thing that can make this really simple is something called the open science framework, which is that OSF.io. It's a way of sharing your data and your research with an annotation of how you got through the whole thing with other people. It makes the research transparent, which is what we need. One of my professional organizations, the Association for Psychological Science has a major initiative on this called open practices, where they are strongly encouraging people to share their data as much as is ethically permissible and to absolutely share their methods before they even conduct a study. That's a way of getting rigorous intellectual honesty and accountability. Now, another step in all of this is to archive your data and make that information available, put it on the shelf. And what you want to do here is you want to archive all of your data sets, both the totally raw before you did anything with a data set and every step in process until your final clean data set. Along with that, you want to archive all of the code that you used to process and analyze the data. If you use a programming language like our Python, that's really simple. If you use a program like SPSS, you need to save the syntax files and then it can be done that way. And again, no matter what, make sure to comment liberally and explain yourself. Now, part of that is you need to explain your process, you know, because you're not just this lone person sitting on the sofa working by yourself. You're worth other people. And you need to explain why you did it the way that you did. You need to explain the choices, the consequences of those choices, the times that you had to backtrack and try it over again. This all also works into the principle of future proofing your work. You want to do a few things here. Number one, the data. You want to store the data in non proprietary formats like a CSV or comma separated values file, because anything could read CSV files. If you stored it in the proprietary SPSS dot save format, you might be in a lot of trouble when somebody tries to use it later and they can't open it. Also, their storage, you want to place all of your files in a secure accessible location like GitHub, it's probably one of the best choices. And then the code, you may want to use something like a dependency management package like packrat for our or virtual environment for Python as a way of making sure that the packages that you use that there are always versions that work, because sometimes things get updated and it gets broken. This is a way of making sure that the system that you have will always work. Overall, you can think of this to you want to explain yourself and a neat way to do that is to put your narrative in a notebook. Now you can have a physical lab book, but you can also do digital books, a really common one, especially if you're using Python is Jupiter with a why they're in the middle. The Jupiter notebooks are interactive notebooks. So here's a screenshot of a very simple one I made in Python. And you have titles, you have text, you have the graphics. If you're working in our you can do this with something called our markdown, which works in the same way you do it in our studio use markdown and you can annotate the whole thing, get more information about that at our markdown rstudio.com. And so for instance, here's an R analysis I did. And is you see the code on the left and you see the markdown version on the right, what's neat about this is that this little bit of code here this title and this text and this little bit of our code then is displayed as this formatted heading as this formatted text and this turns into the entire our output right there. It's a great way to do things. And then if you do our markdown, you actually have the option of uploading the document into something called our pubs. And that's an online document that can be made accessible to anybody. Here's the same document. And if you want to go see it, you can go to this address, it's kind of long. So I'm going to let you write that one down yourself. But in some here's what we have. You want to do your work and archive the information in a way that supports collaboration. Explain your choices, say what you did show how you did it. This allows you to future proof your work so it will work in other situations and for other people. And as much as possible, no matter how you do it, make sure to share your narrative so people understand your process. And they can see that your conclusions are justifiable, strong and reliable. Now something that I've mentioned several times when talking about data science, and I'll do it again in this conclusion, is that it's important to give people next steps. So I'm going to do that for you right now. If you're wondering what to do after having watched this very general overview course, I can give you a few ideas. Number one, maybe you want to start trying to do some coding in our or Python, we have courses for those. You might want to try doing some data visualization, one of the most important things that you can do. You may want to brush up on statistics and maybe some math that goes along with it. And you may want to try your hand at machine learning. All of these will get you up and rolling in the practice of data science. You can also try looking at data sourcing find the information that you're going to do. But no matter what happens, try to keep it in context. So for instance, data science going to be applied to marketing and sports and health and education and the arts and really a huge number of other things. And we will have courses here at data lab.cc that talk about all of those. You may also want to start getting involved in the community of data science. One of the best conferences that you can go to is O'Reilly strata, which means several times a year around the globe. There's also predictive analytics world, again, several times a year around the world. Then there's much smaller conferences. I love tapestry or tapestry conference calm, which is about storytelling in data science, and extract a one day conference about data stories that's put on by import IO, one of the great data sourcing applications that's available for scraping web data. If you want to start working with actual data, a great choice is to go to Kaggle.com. And they sponsored data science competitions, which actually have cash rewards. But there's also wonderful data sets you can work with there to find out how they work and compare your results to those of other people. And once you're feeling comfortable with that, you may actually try turning around and doing some service. Data kind.org is the premier organization for data science as humanitarian service. They do major projects around the world. I love their examples. There are other things you can do. There's an annual event called do good data. And then data lab dot CC will be sponsoring twice a year data lab surets, which are opportunities for people in the Utah area to work with local nonprofits on their data. But above all of this, I want you to remember this one thing. Data science is fundamentally democratic. It's something that everybody needs to learn to do in some way shape or form. The ability to work with data is a fundamental ability, and everybody would be better off by learning to work with data intelligently and sensitively. Or to put it another way, data science needs you. Thanks so much for joining me for this introductory course. I hope it's been good. And I look forward to seeing you in the other courses here at data lab dot CC. Welcome to data sourcing. I'm Barton Poulsen. And in this course, we're going to talk about data opus, or that's Latin for data needed. The idea here is that no data, no data science. And that is what we're going to do. It's a sad thing. So instead of leaving it that we're going to use this course to talk about methods for measuring and evaluating data, and methods for accessing existing data, and even methods for creating new custom data. Take those together. And it's a happy situation. At the same time, we'll do all of this still at an accessible, conceptual and non technical level, because the technical hands on stuff will happen in later other courses. But for now, let's talk data. For data sourcing, the first thing we want to talk about is a measurement. And within that category, we're going to talk about metrics. The idea here is that you actually need to know what your target is, if you want to have a chance to hit it. There's a few particular reasons for this. First off, data science is action oriented. The goal is to do something as opposed to simply understand something, which I say as an academic practitioner. Also, your goal needs to be explicit. And that's important because the goals can guide your effort. So you want to say exactly what you're trying to accomplish. So you know, when you get there. Also, goals exist for the benefit of the client, they can prevent frustration, they know what you're working on, they know what you have to do to get there. And finally, the goals and the metrics exist for the benefit of the analyst, because they help you use your time well, you know, when you're done, you know when you can move ahead with something. And that makes everything a little more efficient and a little more productive. Now, when we talk about this, the first thing you want to do is try to define success in your particular project or domain. Depending on where you are in commerce that can include things like sales or click through rates or new customers. In education, it can include scores on tests, it can include graduation rates or retention. In government and include things like housing and jobs. In research, it can include the ability to serve the people that you're trying to better understand. So whatever domain you're in, there will be different standards for success. And you're going to need to know what applies in your domain. Next are specific metrics or ways of measuring. Now, again, there are a few different categories here. There are business metrics, there are key performance indicators or KPIs. There are smart goals, that's an acronym. And there's also the issue of having multiple goals. I'll talk about each of those for just a second now. First off, let's talk about business metrics. If you're in the commercial world, there are some common ways of measuring success. A very obvious one is sales revenue. Are you making more money? Are you moving the merchandise? Are you getting sales? Also, there's the issue of lead generated new customers or new potential customers, because that then in turn is associated with future sales. There's also the issue of customer value or lifetime customer value. So you may have a small number of customers, but they all have a lot of revenue. And you can use that to really predict the overall profitability of your current system. And then there's churn rate, which has to do with, you know, losing and gaining new customers and having a lot of turnover. So any of these are potential ways of defining success and measuring it. These are potential metrics, there are others, but these are some really common ones. Now, I mentioned earlier, something called a key performance indicator, or KPI. KPIs come from David Parmenter, and he's got a few ways of describing them. He says a key performance indicator for business number one should be non financial. So not just the bottom line, but something else that might be associated with it or that measures the overall productivity of the association. They should be timely, for instance, weekly, daily or even constantly gathered information. They should have a CEO focus. So the senior management team are the ones who generally make the decisions that affect how the organization acts on the KPIs. They should be simple. So everybody in the organization, everybody knows what they are and knows what to do about them. They should be team based. So teams can take joint responsibility for meeting each one of the KPIs. They should have significant impact. What that really means is they should affect more than one important outcome. So you can do profitability and market reach or improve manufacturing time and fewer defects. And finally, an ideal KPI has a limited dark side. That means there's fewer possibilities for reinforcing the wrong behaviors and rewarding people for sort of exploiting the system. Next, there are smart goals, where smart stands for specific, measurable, assignable to a particular person, realistic meaning you can actually do it with the resources you have at hand, and time bound so you know when it can get done. So whenever you form a goal, you should try to assess it on each of these criteria. And that's a way of saying that this is a good goal to be used as a metric for the success of our organization. Now, the trick, however, is when you have multiple goals, multiple possible endpoints. And the reason that's difficult is because, well, it's easy to focus on one goal, if you're just trying to maximize revenue, or if you're just trying to maximize, you know, graduation rate, there's a lot of things you can do. It becomes more difficult when you have to focus on many things simultaneously, especially because some of these goals may conflict, the things that you do to maximize one may impair the other. And so when that happens, you actually need to start engaging in a deliberate process of optimization, you need to optimize. And there are ways you can do this. If you have enough data, you can do a mathematical optimization to find the ideal balance of efforts to pursue one goal, and the other goal at the same time. Now, this is a very general summary. And let me finish with this in some metrics or methods for measuring can help awareness of how well your organization is functioning and how well you're reaching your goals. There are many different methods available for defining success and measuring progress towards those things. The trick, however, comes when you have to balance efforts to reach multiple goals simultaneously, which can bring in the need for things like optimization. When talking about data sourcing and measurement, one very important issue has to do with the accuracy of your measurements. The idea here is that you don't want to have to throw away all your ideas, you don't want to waste effort. One way of doing this in a very quantitative fashion is to make a classification table. So what that looks like is this, you talk about, for instance, positive results, negative results. And in fact, let's start by looking at the top here, the middle two columns here talk about whether an event is present, whether your house is on fire, whether a sale occurs, whether you've got a tax evader, whatever. So that's whether a particular thing is actually happening or not. On the left here is whether the test or the indicator suggests that a thing is or is not happening. And then you have these combinations of true positives, where the test says it's happening and it really is. And false positive where the test says it's happening, but it's not. And then below that, true negatives where the test says it isn't happening. And that's correct. And then false negatives where the test says there's nothing going on, but there is in fact the event occurring. And then you start to get the column totals, the total number of events present or absent, and the row totals that talk about the test results. Now, from this table, what you get is four kinds of accuracy, or really four different ways of quantifying accuracy using different standards. And they go by these names, sensitivity, specificity, positive, predictive value, and negative predictive value. I'll show you very briefly how each of them works. sensitivity can be expressed this way. If there's a fire does the alarm ring, you want that to happen. And so that's a matter of looking at the true positives, and dividing that by the total number of alarms. So the test positive means there's an alarm, and the event present means there's a fire, you want to always to have an alarm when there's a fire. specificity, on the other hand, is sort of the flip side of this. If there isn't a fire, does the alarm stay quiet? This is where you're looking at the ratio of true negatives to total absent events where there's no fire and the alarm's not ringing. And that's what you want. Now, those are looking at columns, you can also go sideways across rows. So the first one there is positive predictive value, often just abbreviated as PPV. And we flip around the order a little bit. This one says if the alarm rings, was there a fire? So now you're looking at the true positives and dividing it by the total number of positives, total number of positives is anytime the alarm rings, true positives is because there was a fire. And negative predictive value or NPV says, if the alarm doesn't ring, does that in fact mean that there is no fire? Well, here you're looking at true negatives and dividing it by total negatives the time that it doesn't ring. And again, you want to maximize that so the true negatives account for all of the negatives the same way you want the true positives to account for all of the positives and so on. Now, you can put numbers on all of these going from 0% to 100%. And the idea is to maximize each one as much as you can. So in some from these tables, we get four kinds of accuracy, and there's a different focus for each one. But the same overall goal, you want to identify the true positives and true negatives, and avoid the false positives and false negatives. And this is one way of putting numbers on an index really, on the accuracy of your measurement. Now data sourcing may seem like a very quantitative topic, especially when we're talking about measurement. But I want to measure one important thing here. And that is the social context of measurement. The idea here really is that people are people, and they all have their own goals, and they're going their own ways. And we all have our own thoughts and feelings that don't always coincide with each other. And this can affect measurement. And so for instance, when you're trying to define your goals, and you're trying to maximize them, you want to look at things like, for instance, the business model, an organization's business model, the way they conduct their business, the way they make their money, is tied to its identity, and its reason to be. And if you make a recommendation that's contrary to their business model, that can actually be perceived as a threat to their core identity. And people tend to get freaked out in that situation. Also restrictions. So for instance, there may be laws, policies, and common practices, both organizationally and culturally, that may limit the ways that goals can be met. Now, most of these make a lot of sense. So the idea is you can't just do anything you want, you need to have these constraints. And when you make your recommendations, maybe you'll work creatively in them as long as you're still behaving legally and ethically. But you do need to be aware of these constraints. Next is the environment. And the idea here is that competition occurs both between organizations that company here is trying to reach a goal, but they're competing with company BU over there. But probably even more significantly, there is competition within the organization. This is really a recognition of office politics, and that when you as a consultant make a recommendation based on your analysis, you need to understand, you're kind of dropping a little football into the office and things are going to further one person's career, maybe to the detriment of another. And in order for your recommendations to have the maximum effectiveness, they need to play out well in the office. That's something that you need to be aware of as you're making your recommendations. Finally, there's the issue of manipulation. And sad truism about people is that any reward system, any reward system at all, will be exploited. And people will generally game the system. This happens, especially when you have a strong cutoff, you need to get at least 80% or you get fired. And people will do anything to make their numbers appear to be 80%. This happens an awful lot when you look at executive compensation systems, it looks a lot when you have very high stakes school testing. It happens in an enormous number of situations. And so you need to be aware of the risk of exploitation and gaming. Now, don't think then that all is lost, don't give up. You can still do really wonderful assessment, you can get good metrics, just be aware of these particular issues and be sensitive to them as you both conduct your research and as you make your recommendations. So in some social factors affect goals and they affect the way that you meet those goals. There are limits and consequences both on how you reach the goals and out really what the goal should be. And that when you're making advice on how to reach those goals, please be sensitive to how things play out with metrics and how people will adapt their behavior to meet the goals. That way, you can make something that's more likely to be implemented the way you meant, and more likely to predict accurately what can happen with your goals. When it comes to data sourcing, obviously, the most important thing is to get data. But the easiest way to do that, at least in theory, is to use existing data. Think of it as going to the bookshelf and getting the data that you have right there at hand. Now, there's a few different ways to do this. You can get in house data, you can get open data, and you can get third party data. Another nice way to think of that is proprietary, public and purchased data, the three P's I've heard it called. Let's talk about each of these a little bit more. So in house data, that's stuff that's already in your organization. What's nice about that is it can be really fast and easy. It's right there. And the format may be appropriate for the kind of software in the computer that you're using. If you're fortunate, there's good documentation, although sometimes when it's in house, people just kind of throw it together. So you have to watch out for that. And there's the issue of quality control. Now, this is true with any kind of data, but you need to pay attention with in house, because you don't know the circumstances necessarily under which people gathered the data and how much attention they were paying to something. There's also an issue of restrictions. There may be some data that while it's in house, you may not be allowed to use or you may not be able to publish the results or share the results with other people. So these are things that you need to think about when you're going to use in house data, in terms of how can you use it to facilitate your data science projects? Specifically, there are few pros and cons in house data, potentially quick, easy, free, hopefully standardized, maybe even the original team that conducted this study is still there. And you might have identifiers in the data, which make it easier for you to do an individual level analysis. On the con side, however, the in house data simply may not exist, maybe it's just not there, or the documentation may be inadequate. And of course, the quality may be uncertain, always true, but maybe something you have to pay more attention to when you're using in house data. Now, another choice is open data, like going to the library and getting something. This is prepared to data that's freely available consists of things like government data and corporate data and scientific data from a number of sources. Let me show you some of my favorite open data sources just so you know where they are and that they exist. Probably the best one is data.gov here in the US. That is the says right here, the home of the US government's open data. Or you may have a state level one. For instance, I'm in Utah, and we have data.utah.gov, also a great source of more regional information. If you're in Europe, you have open dash data dot Europa dot EU, the European Union Open Data portal. And then there are major nonprofit organizations. So the UN has UNICEF dot org slash statistics for their statistical and monitoring data. The World Health Organization has the global health observatory at who dot and slash GHO. And then there are private organizations that work in the public interest such as the Pew Research Center, which shares a lot of its data sets, and the New York Times, which makes it possible to use API to access a huge amount of the data of things they've published over a huge time span. And then two of the mother loads, there's Google, which at Google.com has public data, which is a wonderful thing. And then Amazon at aws.amazon.com data sets has gargantuan data set. So if you needed a data set that was like five terabytes in size, this is the place that you would go to get it. Now, there's some pros and cons to using this kind of open data. First is that you can get very valuable data sets that maybe cost millions of dollars to gather into process. And you can get a very wide range of topics and times and groups of people and so on. And often the data is very well formatted and well documented. There are, however, a few cons. Sometimes there's by a sample say, for instance, you only get people who have internet access. And that can mean, you know, not everybody. Sometimes the meaning of the data is not clear, or it may not mean exactly what you wanted to. A potential problem is that sometimes you may need to share your analysis. And if you're doing proprietary research, well, it's going to have to be open research instead. And so that can create a cramp with some of your clients. And then finally, there are issues with privacy and confidentiality. And in public data, that usually means that the identifiers are not there. And you're gonna have to work at a larger aggregate level of measurement. Another option is to use data from a third party. These go by the name data as a service or Das, you can also call them data brokers. And then thing about data brokers is they can give you an enormous amount of data on many different topics. Plus, they can save you some time and effort by actually doing some of the processing for you. And that can include things like consumer behaviors and preferences, they can get contact information, they can do marketing identity and finances, there's a lot of things. There's a number of data brokers of round. Here's a few of them. Axiom is probably the biggest one in terms of marketing data. There's also Nielsen, which provides data primarily for media consumption. And there's another organization, data sift, that's a smaller newer one. And there's a pretty wide range of choices. But these are some of the big ones. Now the thing about using data brokers, there's some pros and there's some cons. The pros are first that it can save you a lot of time and effort. It can also give you individual level data, which can be hard to get from open data. Open data is usually at the community level. They can give you information about specific consumers. They can even give you summaries and inferences about things like credit scores and marital status, possibly even whether a person gambles or smokes. Now the con is this, number one, it can be really expensive. I mean, this is a huge service, it provides a lot of benefit and is priced accordingly. Also, you still need to validate it, you still need to double check that it means what you think it means and that it works in with what you want. And probably a real sticking point here is the use of third party data is distasteful to many people. And so you have to be aware of that as you're making your choices. So in some as far as data sourcing existing data goes, obviously data science needs data. And there's the three P's of data sources proprietary and public and purchased. But no matter what source you use, you need to pay attention to quality and to the meaning and the usability of the data to help you along in your own projects. When it comes to data sourcing, a really good way of getting data is to use what are called API's. Now I like to think of these as the digital version of proof rocks mermaids, if you're familiar with the love song of J alpha proof rock by TS Eliot, he says, I have heard the mermaids singing each to each, that's TS Eliot. And I like to adapt that to say, API's have heard apps singing each to each and that's by me. Now, more specifically, when we talk about an API, what we're talking about is something called an application programming interface. And this is something that allows programs to talk to each other. It's most important use in terms of data science is it allows you to get web data, it allows your program to directly go to the web on its own, grab the data, bring it back in, almost as though it were local data. And that's a really wonderful thing. Now, the most common version of API's for data science are called rest API's that stands for representational state transfer, that's the software architectural style of the worldwide web. And it allows you to access data on webpages via HTTP, that's the hypertext transfer protocol that, you know, runs the web as we know it. And when you download the data, you usually get it in JSON format that stands for JavaScript, object notation. The nice thing about that is that's human readable, but it's even better for machines. Then you can take that information and you can send it directly to other programs. And the nice thing about rest API's is that they're what's called language agnostic, meaning any programming language can call a rest API can get data from the web and can do whatever it needs to with it. Now, there are a few kinds of API's that are really common. The first is what are called social API's. These are ways of interfacing with social networks. So for instance, the most common is Facebook, there's also Twitter, Google talk has been a big one and Four Square as well. And then SoundCloud, these are on lists of the most popular ones. And then there are also what are called visual API's, which are for getting visual data. So for instance, Google Maps is the most common, but YouTube is something that accesses YouTube on a particular website, or AccuWeather, which is for getting weather information, Pinterest for photos and Flickr for photos as well. So these are some really common API's and you can program your computer to pull in data from any of these services and sites and integrate it into your own website or here into your own data analysis. Now there's a few different ways you can do this. You can program it in our the statistical programming language, you can do it in Python. Also, you can even do it in the very basic bash command line interface. And there's a ton of other applications. Basically, anything can access an API one way or another. Now, I'd like to show you how this works in R. So I'm going to open up a script in our studio. And then I'm going to use it to get some very basic information from a web page. Let me go to our studio and show you how this works. I've opened up a script and our studio that allows me to do some data sourcing here. Now I'm just going to use a package called JSON light, I'm going to load that one up. And then I'm going to go to a couple of websites, I'm going to be getting historical data from formula one car races. And I'm going to be getting it from airgas.com. Now if we go to this page right here, I can just go straight to my browser right now. And this is what it looks like, it gives you the API documentation. So what you're doing for an API is you're just entering a web address. And to end that web address, it includes the information that you want. I'll go back to our here for a second. And if I want to get information about 1957 races in JSON format, I go to this address, I can skip over to that for a second. And what you see is it's kind of a big long mess here, but it is all labeled and it's clear to the computer, what's going on here. I'll go back to our. And so what I'm going to do is I'm going to save that URL into an object here in our, and then I'm going to use the command from JSON to read that URL and save it into our and which it is now done. And I'm going to zoom in on that so you can see what's happened. I've got this sort of mess of text, this is actually a list object in our and then I'm going to get just the structure of that object. So I'm going to do this one right here. And you can see that it's a list and it gives you the names of all the variables within each one of the lists. And what I'm going to do is I'm going to convert that list to a data frame by I went through the list and found exactly where the information I wanted was located, you have to use this big long statement here. That'll give me the names of the drivers. Let me zoom in on that again. There they are. And then I'm going to get just the column names for that bit of the data frame. And so what I have here is six different variables.  And when I do that, this is what I get. I'll zoom in on that again. And the first five people listed in this data set that I pulled in from 1957, are Juan Fangio makes sense, one of the greatest drivers ever, and other people who competed in that year. And so what I've done is by using this API call in our very simple thing to do, I was able to pull data off that web page in a structured format, and do a very simple analysis with it. And let's sum up what we've learned from all this. First off, APIs make it really easy to work with web data. They structure, they call it for you. And then they feed it straight into the programs for you to analyze. And they're one of the best ways of getting data and getting started in data science. When you're looking for data, another great way of getting data is through scraping and what that means is pulling information from webpages. I like to think of it as when data is hiding in the open. It's there, you can see it, but there's not an easy immediate way to get that data. Now, when you're dealing with scraping, you can get data in several different formats, you can get HTML text from webpages, you can HTML tables, the rows and columns that appear on webpages. You can scrape data from PDFs and you can scrape data from all sorts of media like images and video and audio. Now, we'll make one very important qualification before we say anything else. Pay attention to copyright and privacy. Just because something is on the web doesn't mean you're allowed to pull it out. Information gets copyrighted. And so when I use examples here, I make sure that this is stuff that's publicly available. And you should do the same when you're doing your own analyses. Now, if you want to scrape data, there's a couple of ways to do it. Number one is to use apps that are developed for this. So for instance, import dot io is one of my favorites. It's both a web page that sits address and it's a downloadable app. There's also a scraper wiki there's an application called tabula and you can even do scraping and Google Sheets, which I'll demonstrate in a second and Excel. Or if you don't want to use an app or if you want to do something that apps don't really let you do, you can code your scraper, you can do it directly in R or Python, or bash, or even Java or PHP. Now, what you're going to do is you're going to be looking for information on the web page. If you're looking for HTML text, what you're going to do is you're going to pull structured text from webpages similar to how a reader view works in a browser. It uses HTML tags on the web page to identify what's the important information. So that's things like body and h1 for header one and P for paragraph in the angle brackets. You can also get information from HTML tables, although this is a physical table of rows and columns I'm showing you. This also uses HTML table tags that's like table and TR for table row and TD for table data that's a cell. The trick is when you're doing this, you need the table number and sometimes you just have to find that through trial and error. Let me give you an example of how this works. Let's take a look at this Wikipedia page on the Iron Chef America competition. I'm going to go to the web right now and show you that one. So here we are in Wikipedia, Iron Chef America. And if you scroll down a little bit, you see we got a whole bunch of text here, we got our table of contents. And then we come down here, we have a table that lists the winners, the statistics for the winners. And let's say we want to pull that from this webpage into another program for us to analyze. Well, there's an extremely easy way to do this with Google Sheets. All we need to do is open up a Google Sheet and then sell a one of that Google Sheet, we paste in this formula. It's import HTML, then you give the web page, then you say that you're importing a table, you have to put that stuff in quotes and the index number for the table. I had to poke around a little bit to figure out that this one was table number two. So let me go to Google Sheets and show you how this works. Here I have a Google Sheet. And right now it's got nothing in it. But watch this, if I come here to this cell, and I simply paste in that information, all this stuff to sort of magically propagates into the sheet makes it extremely easy to deal with. And now I can for instance, save this as a CSV file, put in another program, lots of options. And so this is one way that I'm scraping the data from a web page, because I didn't use an API, but I just use a very simple one link command in Google Sheets to get the information. Now that was an HTML table, you can also scrape data from PDFs. You have to be aware of whether it's a native PDF, I call that a text PDF or a scanned or image PDF. And what it does with native PDFs is it looks for text elements. Again, those are like code that indicates this is text. And you can deal with raster images, that's pixel images or vector, which draws the lines. And that's what makes them infinitely scalable in many situations. And then in PDFs, you can deal with tabular data, but you probably have to use a specialized program like scraper wiki or tabula in order to get that. And then finally, media like images and video and audio, getting images is easy, you can download them in a lot of different ways. And then if you want to read data from them, say for instance, you have a heat map of a country, you can go through but you'll probably have to write a program that loops through the image pixel by pixel to read the data and then encode it numerically into your statistical program. Now that's my very brief summary. And let's summarize that. First off, if the data you're trying to get at doesn't have an existing API, you can try scraping. And you can use specialized apps for scraping, or you can write code in a language like our Python. But no matter what you do be sensitive to issues of copyright and privacy. So you don't get yourself in hot water. But instead, you make an analysis that can be of great use to you or to your client. The next step in data sourcing is making data. And specifically, we're talking about getting new data. I like to think of this as you're getting your hands on you're getting data de novo new data. So can't find the data that you need for your analysis. Well, one simple solution is do it yourself. And we're going to talk about a few general strategies used for doing that. Now, these strategies vary in a few dimensions. First off is the role. Are you passive and simply observing stuff that's happening already? Or are you active where you play a role in creating the situation to get the data? And then there's the QQ question. And that is, are you going to get quantitative or numerical data? Or are you going to get qualitative data, which usually means text, paragraph sentences, as well as things like photos and videos and audio? And also, how are you going to get the data? Do you want to get it online? Or do you want to get it in person? Now, there's other choices in these. But these are some of the big delineators of the different methods. When you look at those, you get a few possible options. Number one is interviews, and I'll say more about those. Another one is surveys. A third one is card sorting. And the fourth one is experiments, although I actually want to split experiments into two kinds of categories. The first one is laboratory experiments. And that's in person projects, where you shape the information or an experience for the participants as a way of seeing how that involvement changes their reactions.  And then there's also a B testing. This is automated online testing of two or more variations on a web page. It's a very, very simple kind of experimentation. There's actually very useful for optimizing websites. So in some from this very short introduction, make sure you can get exactly what you need get the data you need to answer your question. And if you can't find it somewhere, then make it. And as always, you have many possible methods, each of which have their own strengths and their own compromises. And we'll talk about each of those in the following sections. The first method of data sourcing where you're making new data that I want to talk about