Welcome to Introduction to Data Science. My name is Bill Howe and I'm the Director of Research for Scalable Data Analytics at the University of Washington E Science Institute and an Affiliate Assistant Professor in Computer Science and Engineering also at the University of Washington. So in this first segment, what I want to do is go through some examples of data science activities and projects from the recent past that I found interesting and use them to sort of wet your appetite for the concepts that we're going to learn in this course. Okay.  So this is a map of the Electoral College and each state is colored by the candidate that took the electoral votes and the numbers that represent how many electoral votes each state has. And so if you recall, what was interesting about this map at the time was that it was or that it led to a pretty significant discussion in the media about data science because Nate Silver of the 538 blog was able to predict this map perfectly before the election, alright. And you know that discussion in the media had talked a lot about you know what a genius Nate Silver was and mentioned the sophisticated mathematics he was using and how you know he's sort of a whiz with these things but what I thought was interesting about this was that Nate Silver would be the first one to tell you that the methods he was employing to make this prediction were actually pretty simple, right. And so he says here in a series of quotes from blog posts around that time, this first one from October 26 was the unification behind this ought to be very simple. Mr. Obama is maintaining leads in the polls in Ohio and other states that are sufficient for him to win 270 electoral votes.  The argument we're making is exceedingly simple. Here it is, Obama's ahead in Ohio, right, it's not a magic trick. And so then after the election on November 10th when he was shown to be right and got this sort of flawless prediction, the blog post that this last quote is taken from was describing why he started the 538 blog in the first place. And he says, look, the bar set by the competition was invitingly low. Someone could look like a genius simply by doing some fairly basic research into what really has predictive power in a political campaign. And so what really had predictive power in this case was the state polls themselves aggregated, right. So historically, the state polls aggregated did a pretty good job of predicting the outcome of the general election. And so that's what he did. Now there was some sophisticated work in quantifying the uncertainty and certainly in presenting these results. There's a lot of beautiful interactive visualizations that he created in order to sort of convey these ideas to the public. And that's one of the points I want to make about this is that getting the answer in some cases is the easy part. It's in interpreting the results and in convincing others of the result by presenting them, usually through visualizations, that can be the hard part. And this is one of the themes that we'll come back to throughout this course, okay. So just to summarize that though, I'm not sure I said this. That simple methods plus enough good data wins. That trumps more sophisticated methods in many cases. And that's another theme that we'll come back to. All right, so something else related to the campaign before we moved on to other topics was the system that the Obama campaign used for their data driven ground game, so to speak. The ability to sort of target specific categories of users. And so what they did was they built and maintained a really significantly size, a massive voter database. And used it to design these highly tailored messages to very, very specific groups. So the mother of two in a small town in Ohio who tweeted about the environment and mentioned organic vegetables on her Facebook page. And who had voted in 2008 and had registered on Obama's website but had never donated. Okay, she would get a message from Michelle Obama that highlighted Barack Obama's environmental policies, okay. And so in order to design these messages what you had to do is kind of do ad hoc hypothesis testing about what might work and what didn't. You had to kind of slice and dice this data at kind of interactive speeds. And this is another theme that we'll return to is the need for these kind of ad hoc interactive analysis. And the systems that use for this are pretty interesting too. This was a SQL database, a very fast one called Vertica. And we'll talk a little bit about what makes Vertica special, I hope, toward the end of the course. But it is a SQL database and so SQL sometimes gets a bad name in data science context as sort of the old guard that can't be possibly used for analytics and doesn't really make sense in today's era. But don't believe it, it has a role to play in many cases. And so here they did use Hadoop to do the aggregate generations of anything not real time, he says here. But for the speed of thought queries about the data, they used this Vertica database. Okay, and so we'll come back to systems in several segments from now. Okay, so moving on, this was around the same time. You know, when Hurricane Sandy made landfall, one of the things that struck me was the fact that visualizations of available data were starting to emerge in real time in response to the storm. And there's some very nice examples of people who used Twitter data to analyze the map or to produce a map of where the power was going out. And in this even simpler case, Joseph Ruel got public data from local weather stations and just from two different weather stations and just simply plotted them. And so this is the barometric pressure over the course of this few day period, two days I guess, in two locations, Atlantic City and Philadelphia. And so you can see this enormous dip is the storm passing through. And you can also see the time lag between Atlantic City and Philadelphia, and you can also see the intensity is probably a little bit higher given that the barometric drop is more significant in Atlantic City. Okay, so a couple things here. One is pulling data down from the web and repurposing it sort of in real time, or at least in short time, maybe not real time, to produce visualizations and then publishing those back on the web. I think this is very much the character of data science activities. In this particular example, there's not necessarily a large data set involved. But repurposing data that was collected for a different purpose is a theme that we'll come back to. And again, we see the ad hoc nature of this as well. Fine, so another plot here is wind speeds and they sort of peak out at 40 in Atlantic City, which is still the green. And you can see the Atlantic City is indeed more intense here. And the gray here is error bars. So again, another variance on the same data. Okay, so changing gears a little bit. This was a study, the title here is called The Expression of Emotions in 20th Century Books. And so what they were interested in is have the words that we choose to use in our collective literature changed over time. And eventually does that tell us something about sort of culture or civilization? I find the scientific inquiry sort of compelling. But what I think is most striking about this and why I wanted to include it as an example is that the methodology that they used here is pretty straightforward. You could do this yourself with not a significant background in either technology or in statistics or even in linguistics or anything. And so this is what they did, right? So the first step is kind of a doozy. This is take all the books written in the 20th century and digitize them. Well, that would be a non-starter. None of us could do that, but that's okay. Google's already done it for us, and they've made the data available at this URL. And so you can go check that out. What they've done is digitize the books, done the character recognition on it, and produced these ingram data sets. So these are tables of data where each row has an ingram and followed by the year and the counts of the number of times that ingram occurred. This has already been broken down and processed into a form that's digestible. Okay, and so what's an ingram? Well, it's pretty simple. A one gram is just a single word like yesterday. A five gram, an example here is the phrase analysis is often described as. Okay, and so in this study, they just ignored everything but the one grams.  So how do they do this? Well, you can imagine that certain words are charged with a particular mood or associated with joy or sadness or fear and so on. And you could also imagine that synonyms of those words might also be associated with that mood. And so this analysis sounds non-trivial and it is, but once again, that's already been done for you. There's a resource on the web called WordNet where they've done this kind of affect analysis. And so the authors of this paper were able to take the digitized books from Google, or already broken down into ingrams, and the affect scores from WordNet. And then do this calculation, which may sort of look intimidating if you're not used to staring at these mathematical expressions, but it's actually pretty simple. This is the count of a particular word in the set. The set being the set of WordNet words, which is not as big as all the words. Only some words are able to be scored as mood. And then you normalize by the count of the number of occurrences of the word the. So why do they do that? Well, you need to normalize over something in order to account for the fact that perhaps we just write more books in 2005 than we did in 1937, or we've been able to digitize more books. So we need to normalize by that total. Well, why not just normalize by the total number of words? Well, the reason is that the word the is a better indicator of pros than the total number of words. And this is because apparently we've also started to produce more sort of captions and figures and more sort of technical language and more sort of formula and more expressions, more non-pros utterances in these books. And therefore we can sort of skew the results. So they really want to capture that in our language when we actually write full of complete sentences, how often are these words being used, okay? And then you add those up and you divide by the total number of words in the set. And then there's one more transformation here that should look familiar to you if you sort of recall your high school statistics. And so you subtract the mean and divide by the standard deviation, okay? So this is normalizing with respect to a normal distribution, okay? But that's about it.  And they're big but they're not exceedingly big. They fit in memory on most of your laptops nowadays. So it's a significant computational task but nothing that requires a dupe. You could do this in a weekend if you had thought about it, okay? So I find that pretty compelling, fine. So these are the results. This is joy words minus sadness words. This is the Z score for joy and sadness. And you can see that there's sort of a big dip after World War II. And that's one of the points they make in the paper. And then you can see this sort of thing start to increase in the late 90s, okay? So I won't try to analyze this for the scientific value. I'll just present the results. What I think is maybe more interesting is this one. So this is now emotion words total minus random words total. And there's a sort of prominent downward slope over time. So what is this, what is this, what's going on here? Well, apparently you can make the argument that we're using fewer emotion words over time, okay? That said, there's a bit of an uptick in this red line and so what does that represent? Well, that's fear words. And you can imagine some of the reasons why there might be an increase in fear words since the 1980s, okay? So this is pretty fun though. This is a significant analysis that can be done just by taking these data sets that they didn't have to prepare themselves, all right? And then the other point I'll make about this, this is just a copy and paste of a segment of some of the papers that this paper cites. And I just was struck by the titles here. You know, quantitative analysis of culture using millions of digitized books, quantifying the evolutionary dynamics of language, frequency of word use predicts rates of lexical evolution through Indo-European history, song lyrics and linguistic markers. I mean, what strikes me about this is that, you know, linguistics, anthropology, history, culture, these studies are becoming hard sciences by the virtue of data-driven methods, right? So all science is becoming data science, right? And therefore data scientists have a lot of power in this regime. It's a great time to be, you know, a data geek. Okay. You know, there's data journalism as well. I mean, one point, I probably should have put a slide in here about this.  You're going to write algorithms that do this kind of an analysis, you know, word use analysis, look for email chains and dialogues, do sort of computational methods in order to analyze that material. And so now journalism itself is a computational enterprise. It's a data science problem, not a, or at least it's amenable to data science technique. So, you know, as a data scientist, the world is your oyster, all right? So let me pause there and we'll pick up with a couple more examples before moving on in the next segment. So let's go through a few more examples. So, if you were asked to decide how important a particular scientific paper was relative to other papers, how much you go about doing that? Well, one way to decide between, say, these two papers here, I'll mark one with a blue circle and one with a red circle, is to wait until other papers start to cite these and count up the number of citations. So here, you know, the paper marked with the blue circle has had four other papers, you know, authors have bothered to read this paper and cite it. So therefore it must have had more impact in the scientific community than the one marked with the red circle. But if we wait even longer, and that's indicated here by this darker blue color, if we wait even longer, you might get even more people citing this intermediate paper. And so maybe we can conclude that, well, over time, ultimately this one had more impact because it, you know, this paper was influenced by this paper and all of these papers were influenced by this one. So therefore, perhaps we change our answer and that this one is more important. So how do we decide between these two interpretations? Well, this problem looks a lot like the problem of judging the relative importance of pages on the web. And so one thing you can do is say, well, look, you know, a particular website is important if a lot of other important websites point to it. And so here, you can say a scientific paper is important if a lot of other important papers point to it. And this method that Google proposed and implemented and ultimately led to a pretty significant part of their success was this algorithm page rank. And so page rank did exactly this. You add up all the weights of your neighbors and give them to yourself and then pass on that weight to everybody that you link to. And you keep going with this until you reach some convergence condition and you end up with the relative importance of these pages. And so this is a method that comes up quite a bit. Whenever you have a graph, it makes sense to potentially run page rank algorithm on it even though it had nothing to do with ranking things on the web for which it was originally designed. Using the same data set, Carl Bergstrom and Martin Rosewall created this visualization. So ignore the importance question. Just think about the graph of the citation network and doing analytics on it. One kind of analysis you can do is judging importance. Another kind of analysis you can do is this. And so what this is is over time run a clustering algorithm, which I haven't described what that is, but it groups clusters of similar documents to and then map those clusters into the relative these fields. And so you can determine that with some confidence that this cluster represents medicine because it has the Lancet and it has other kinds of medical journals in it. And you can conclude that this cluster represents molecular and cell biology. What's pretty striking is that if you lay this out in this timeline like this, you can see that some fraction of the molecular and cell biology community and the neurology community started to combine to form a brand new field of science called neuroscience. And so just by doing this kind of analytics on this graph, right, to do this data science on this graph, you can uncover the emergence of new fields of science. And so I found that pretty striking. And they've gone on to do many other kinds of analysis on this same data set. And the overall field of this kind of meta-field of studying the scientific literature in order to draw inferences is called bibliometrics. This pen is a little wonky. Bibliometrics. Sorry for the bad view there. OK. So what is not amenable to data science? Well, you might think food. And this is a paper in a fairly respectable journal that applied some data-driven techniques to analyzing food pairing. OK. So what they did here is pretty interesting, right? So induce a graph on the ingredients by saying that if two ingredients appear together in a recipe, you draw an edge between them, a graph meaning vertices and edges. And if you're not familiar with graphs, you will be by the end of the course, but bear with me right now. So connect two ingredients if they appear together in some recipe. OK. Build that big graph. And now you can analyze it in ways that are similar to what we just talked about. You can look at the community structure. You can find the clusters within this graph and see if these clusters correspond to some of the well-known methods of food pairings. OK. And in some cases they do and in some cases they don't. And the authors sort of show that they've uncovered things that were not necessarily known but appear to be there in the data, right?  OK. So I thought that was kind of a fun example of an unusual use of data science in a respectable journal. All right. So another example is from the Last FM blog. They looked at the tags associated with the songs on Last FM and used it to do some simple analysis of the emergence of genres over time based on the popularity of the tags for songs coming from that time. And so you see things like, well, post-punk in red here came after punk in purple which you'd hoped the graph shows they'd expect. The other thing we also see is kind of a rise here in rock and roll over time and then maybe a bit of a dip more recently. So this is kind of interesting. But the other theme here we have is that we mentioned in the previous segment is repurposing data, right? So this data was collected simply to help with search, you know, find similar music and it's now being reused to sort of draw inferences about the emergence of entire genres. OK. So another example here that you may be familiar with, Google was able to show that by analyzing the search logs, the frequency of search terms, it could do a better job predicting the severity and the scope of flu outbreaks than the Centers for Disease Control. And by better here we mean essentially earlier, right? It was able to give more of a head start to the health community, OK? And also it was sort of more accurate. So how do they do this? Well, you know, when you're getting the flu, it turns out that you want to search for flu symptoms, terms associated with flu symptoms more often. And by watching that uptick, you can predict that the flu outbreak is coming, OK? So that's great. And they put in a, they, they, this worked and they sort of published a paper about it and then they put up this interactive visualization that allowed people to sort of analyze flu outbreak going forward. But just this year, you know, some folks showed that it didn't do a very good job in this last year. So a scientific hindsight shows that the Google Flu trends far overstated this year's flu season. The reason they think explains this is that there was lots of media attention associated with this year's flu season because it was a bit of an uptick. And so it got amplified. And so this caused people to search for flu-related terms more often, perhaps because they're worried about experiencing the symptoms, perhaps because they are trying to understand more about the flu outbreak, perhaps because they're searching for articles, perhaps because they're worried about their kids more. But it all was, it was a second-order effect based on the media attention on the problem, which led to skewed results and ultimately a wrong answer. And so the point here is this is great that they're repurposing data from the search engine to try to make predictions about something else, but it's, it's, it is biased data. And so you have to be careful with what you conclude from it, OK? So there are limits here, all right? So another example, also, also analyzing web search traffic that was done with perhaps a little more of a scientific rigor was done by some folks at Microsoft Research. And so here what you're looking for is side effects associated with particular drugs. And so these results are pretty striking. So what this graph is showing that is, over time, a set of users, it was around about a million that had, they had permission to sort of monitor their, their web search traffic, that when you searched for this drug in green, what percentage of the time did you also search for terms associated with hyperglycemia symptoms? And the answer is somewhere around 5%. For this other drug, it was somewhere around 4%. For the, in the background, for the average case, it was pretty close to 0%. If you searched for both of these drugs, the odds that you also searched for hyperglycemia symptoms went up to 10%. OK. So what's striking about this is that hyperglycemia is not a known side effect of these drugs, but it seems impossible to ignore from the web search data, right? There's just no reason to believe that this could be explained by coincidence. And they developed this argument more in the paper than I just have there. But so fine. So repurposing data, this is another example, a large data says they had to use a web search. And those are probably the two points I want to make about that, but a pretty fun one. OK. So the last example I'll give is a different take. There's more about prediction than data. But this from last October, if you recall, there were six Italian seismologists who were convicted of manslaughter for failing to predict a magnitude 6.3 earthquake in April 2009. And so while the locals were concerned about the seismic activity, the researchers were deemed to be just too reassuring about the verdict. And so the point I want to make here is that there's liability. I mean, so this scientific community was completely aghast that this happened, and I'm completely aghast that this happened, and pretty much everybody is, that you can imagine to hold researchers responsible for failing to predict something that is demonstrably and known to be impossible to predict, right? So there's no seismologists on the planet that would argue that earthquakes are even remotely predictable. And yet the courts sort of decided that somehow they, you know, because they got the wrong answer, it's bad. But it does sort of bring up the issue that when you make a prediction, there's a certain amount of weight you're going to put behind it, whether intentionally or not. And so understanding how confident you are about that prediction is sort of an important part of the game here. OK. So that was the last example. The themes that we saw come out here. We gave a couple of examples of graph analytics. We showed that databases are sort of useful in the Obama-Grounding case. We saw a lot of examples of visualization and communicating the results, interpreting the results. We saw some examples of using very large data sets, other examples that use very small data sets, and not everything's about big data. A couple of bullets that aren't on here. We talked about, you know, ad hoc interactive analysis is sort of not just faster but different. And so supporting that is important. And then we talked about repurposing data, right? So data collected perhaps by someone else for some other purpose, reusing that to draw inferences about something else. That's a pretty common theme here. OK. In the next couple of segments, we'll talk about how we organized this course and some of the design decisions we made in creating the material. So I want to talk about in this segment what this term data science actually means. So, you know, you'll see these quotes around the web. So in Fortune Magazine, they talk about data science being a hot new gig in tech and Hal Varian, who was Google's chief economist in the New York Times in 2009, which is a while ago now, talked about statistics being the next sexy job, and described it as the ability to take data, to understand it, process it, extract value from it and communicate it. That's going to be hugely important. Another person who's prolific in thinking and writing in this space is Mike Driscoll, who's the CEO of a company called MetaMarkets, and he talks about there, the data science, you know, as it's practiced is this, you know, as this colloquial view of it is a blend of red bullfueled hacking and espresso-inspired statistics. And maybe another quote of his is the data sciences that is the civil engineering of data, who, you know, whose acolytes possess this practical knowledge as well as a theoretical understanding. And so this balance between pragmatism and theory is something we'll come back to. So another perspective on this that you should be familiar with is this VIN diagram that made the rounds several years ago by Drew Conway. And what he's, his point was that data science is perhaps the mix of three different sort of areas. One is hacking skills, you know, programming expertise. Another is the academic view, you know, the math and statistics knowledge. And then the third that he added is this notion of substantive expertise. And what he meant by this is kind of a deep investment with the data. So if you think about your typical IT shop, they're typically building tools for other people to use to actually analyze the data, but they don't necessarily do the analysis themselves. And, you know, a data scientist, in contrast, may be a participant in tool building, but they're going to also sort of dive deeply into the data and do the analysis themselves. And that's one of the ways that I like to interpret Drew's blue bullet here of substantive expertise. Okay. And then he also sort of fills in the gaps between these, talking about that I'm not sure I'd agree that this is necessarily traditional research, but perhaps it is, is applying the statistical knowledge to a particular domain is where research comes in and that some deep theory plus some pragmatic programming makes you a machine learning expert. And then he sort of refers to this as the danger zone where you sort of know enough about the domain to be dangerous and you know enough hacking to be dangerous, but you don't know how to ground your analysis in proper theory. And, you know, some of my colleagues like to joke that computer scientists don't understand error bars, right? And that's going to be what they're referring to here. Okay. So fine. So what do data scientists actually do? Well, there's some more quotes here from EMC, a company who acquired a company called Green Plum and recently had a pretty significant initiative in data science, both related to a product called Green Plum as well as other products. So they need to find the nuggets of truth in data and then explain it to business leaders. And I like this, you know, and then explain it to business leaders. Something else we'll come back to that I mentioned previously is that communicating the results is a critical part of this process. It's not just getting the result. Okay. And then another view of this, which is sort of interesting, is that DJ Patil talks about data scientists tend to be hard scientists, maybe coming from a physics background who have a strong mathematical background and computing skills and, you know, come from a discipline in which survival depends on getting the most from the data. They're really used to sort of torturing the data to extract every last ounce of value out of it. This, you know, DJ has a applied math background, so he's maybe coming from that perspective. So Mike Driscoll also talks about the three sexy skills of data geeks, which are statistics, this thing data munging and will kind of, you know, this funny word munging, and we'll kind of return to this. You'll see a lot of this sort of colloquial language and I'll give my perspective on what I think that tells us about the state of the world in data science. And but what he means by this, as you can imagine, is sort of parsing data and scraping data from the web and converting to between different file formats efficiently and not getting hung up on these, the kind of friction that you deal with when you are working with large and heterogeneous data sets, right? So a data scientist is someone who's very comfortable in that environment and is able to sort of work nimbly, even when things aren't very clean. Okay.  The ability to communicate the results through visualization. All right. So another quote from Jeffrey Stanton who teaches a course in data science at Syracuse and has involved in one of the earlier programs in data science, talks about the emerging area of work concerned with the collection, preparation, analysis, visualization, management, and preservation of large collections of information. And I think one thing that's interesting about his perspective is it includes his word preservation. So while the, you know, preparation, the analytics, and the visualization are the three tenets that you see quite often. It's something that we like in this course as well. You know, Jeffrey goes one step further and talks about preservation. So even after you're done communicating the results, what do you do with the data long term? And that's part of the reason is he's got a background in library of science and information studies where they're very concerned with the curation of data. And so that's something we're probably not going to emphasize too much in this course, but it's definitely part of the overall data life cycle, if you will. Okay. So another quote from a thinker in this space is Hillary Mason, the chief scientist at Bitly. And so she says the data scientist is someone who can obtain scrub, explore, model, and interpret data. You know, blending, hacking, statistics, and machine learning. So we saw that before in Drew Conway's diagram, this blending. And data science is not only a depth at working with data, but can appreciate data itself as a first class product. And I think she's talking about there maybe is being able to organize the data and actually produce something that's usable by other people. Right? Okay. So having a quality data resource, a data asset that others can use to answer questions is one of the outputs of data science. And then she talks about, let's see, scrubbing. And we saw munging on the previous slide. And so you'll see some of these terms. You'll see data jiu-jitsu, people will say. The data scientist that I want to hire is really good at data jiu-jitsu. And we don't, in a couple of segments I'll talk about, or the next segment I think, I'll talk about what I think this means. Okay. So to summarize what we talked about, there's perhaps three overarching tasks involved in data science. And those are preparing to run some sort of statistical analysis, actually running that statistical analysis, and then interpreting and communicating the results. And this phase here, this preparing to run the model, is where we see all that data munging, cleaning, manipulating, integrating, and so on. All right. So another view that I like to take is that data science is really about data products. Producing data products that you may use yourself or that others may use. Okay. And so what do I mean by data products? Well, this could be data-driven application. So if you think about a spell checker, right, this is not just a piece of code. This is not just a piece of software that does something. This is only enabled by a dictionary of words and a dictionary of misspelled word, actually. Okay. And similarly, a machine translation where you can type a sentence in French and have it automatically translated into Arabic relies on not just clever algorithms, but rather an enormous corpus of French text and Arabic text. Okay. The second kind of data product is interactive visualizations. We saw this example with the Google Flu application. And there's a suite of visualizations on the web that I'm not going to show right now associated with the global burden of disease. And these are visualizations produced by the Institute for Health Metrics here at the University of Washington. And what I like about it is it's an example of where they've done a lot of research, but then the output was not just a research paper, although they wrote plenty of those as well. It was these interactive visualizations that allowed you to explore the data as well. And so this, I think, captures this notion of producing not just the answer or not just a paper in perhaps this case, but a data product that is usable by others. Okay. And then finally, another kind of data product might be an online database of some kind that others can actually use to conquery and answer their own questions. And so maybe there's not necessarily a visualization component, but the work that goes into producing these things, I would argue, is part of data science. And so this captures some of the enterprise data warehouse work and software and effort, of which there's a lot. And including business intelligence work and a couple of segments, I'll try to differentiate those two. Another example of an online database from science, as opposed to business, is the Sloan Digital Sky Survey, which I'll talk about in more detail in a couple of segments. So again, just to summarize here what's in red, data science is not just about building data products, it's about building data products, not just answering the questions once. And what data products are are assets, digital assets that empower others to use the data in new ways. And so they may help communicate results, for example, need civil abilities maps, or they may empower others to do their own kind of analysis, say with a data warehouse or with a visualization. All right. So let's talk a little bit about what distinguishes the term data science from other related fields. So one related field is business intelligence. Business intelligence systems are associated with a couple of concepts. One is a data warehouse, and the other is a set of dashboards or reports that consume data from the data warehouse and are used to answer particular questions. So both of these components require a lot of upfront effort to design and build, and are, therefore, not too adaptable when requirements change. OK. And so therefore a software stack designed for business intelligence may or may not be appropriate for any particular data science problems where changing requirements are considered the norm. And so a sort of warrants a new term is that business intelligence became associated with a particular approach to a particular set of problems, and a data science is in some sense broader. OK. The other point I like to make about business intelligence is that the BI engineers are not typically expected to consume their own data products and perform their own analysis and make the business decisions themselves. Usually they're building tools for others to make decisions with. As a data scientist, you'll be doing both. So what about statistics? Well, statistical methods are at the heart of what a data scientist does day to day. But a statistician will typically be comfortable with assuming that any data set they encounter will fit in main memory on a single machine. And this makes sense because the whole field was born out of the need to extract the most information possible from a very sparse, very expensive to collect, and typically, therefore, a very small data set. So if you only have 20 patients in the world with a particular disease, you can't just go find 20 more cheaply. So therefore, you need to come up with new mathematics to squeeze as much information as you can out of the 20 you already have. But that's not always the problem anymore. So as we shift from a data poor regime to a data rich regime, the set of challenges move from the need for new mathematics to squeeze information out of a data set to new engineering to even handle or process very, very large data sets. However, some of the methods, some of the models that you'll build are the same in both cases. So database experts, database programmers and administrators bring a lot of skills to the table to make them appropriate for data science tasks. But there's a focus on a particular data model, which is usually the relational data model. So this is rows and columns. So if you have data coming from sources that are as video or audio or even text or to some extent even graphs, nodes and edges, which we'll talk about, a relational database may or may not be the right tool. And even the concepts that transcend any particular database system may or may not be appropriate. And we'll sort of explore when and where it isn't appropriate as we get into the course. So visualization experts also bring a lot of skills at the table. But like statisticians, are historically less concerned with massive scale. Data that spans many hundreds of machines. And then finally, machine learning is perhaps the closest to data science. But here, and we'll try to make more of a point about this later, as a proportion of the time you'll spend on a data science problem, actually choosing the right model or algorithm, machine learning technique and applying it and running it is a fairly small fraction. Which we'll be spending much more time on is the preparation of the data, the manipulation of the data, the cleaning of the data, the wrangling of the data, as some have been saying. And for this, machine learning techniques are not particularly relevant. And so this falls back more towards the database managers, the database experts and database programmers. OK. So there's a lot of courses that could be considered data science courses. Some of them use data science in the name, some of the newer ones. Others have been around for a long time, but are obviously in the same space. And so I want to spend a little bit of time describing the dimensions by which you could describe these courses and then choose a particular point in this design space that we've used to motivate this course. So as a preface, let me show you this quote from Aaron Kimball, who's a CTO at WebData. And so he said to me that he worries that the data scientist's role is perhaps like the mythical webmaster of the 90s, that they were expected to do everything. The web manual, companies knew they needed to get on the internet in the mid-90s, but they didn't know how. And so they said, well, we'll hire a webmaster, problem solved. The webmaster will write all the content for the website, they'll do the design and manage the user experience, they'll write the code that will wire the website to the order fulfillment system in the back end, they'll actually structure the pages and do the navigation, they'll do the logging required to make sure that the site stays up all the time and has reasonably high availability, they'll design the schema to hold the data that will be served out through the website and so on. And so it wasn't really feasible that you're going to get this in a single person. And so instead the internet strategy became a broader team. Similarly, that might be what we see happening with data science, but here's what it means to me. The term data science tells me that if you're a database administrator and your skills are solely about relational databases, the current trend is you will need to learn more about unstructured data and statistical modeling. If you're a statistician, you will need to learn to deal with data that does not fit in memory. If you're a software engineer who's used to building systems and working with files directly, you'll need to learn some statistical modeling and how to communicate your results to your managers. You'll need to work with these data sets and actually use them to make decisions. And if you're a business analyst who is trained to make the decisions based on data, you're going to need to start understanding a little bit more about the algorithms and trade-offs, especially at scale. And for a couple of reasons. One is the cost change dramatically based on the technology you're picking. What's happening with cloud computing that we'll talk a little bit about and what's happening with these algorithms is that you might be able to get an answer, but it may cost more or less than it did five years ago. The other reason is that as we do more fly-by-wire business, meaning we trust algorithms more and more to make some decisions for us, they become these opaque black boxes. And if you don't understand what's going on inside that black box, you're bound to miss the results. And so it's no longer safe to just sort of throw your trust over the wall to some algorithm or to your staff that's running these algorithms. You may need to sort of understand, internalize the trade-offs and choosing one model versus another yourself. OK. So here are the dimensions by which I like to describe these different courses. The first one is breadth. And so I divide breadth into tools versus abstractions. And so every sophisticated course would prefer to cheat towards abstractions. You want concepts that transcend any particular implementation. However, what students are interested in is hands-on experience using tools they can use tomorrow at a job. And so you always have this tension between these two. All right. And so some examples here are hey-doop, which we'll talk about is an implementation of an abstraction called MapReduce. And then the MapReduce abstraction certainly transcends its particular implementation in hey-doop. And so here, as I'll mention in the next segment, I want to cheat towards abstraction whenever possible, but make sure that there are assignments that give you the hands-on skills that people are interested in. The next dimension here is depth. And so by depth, I intend the distinction between structural manipulation of data and statistical manipulation of data. And so here you can think about the relational algebra as a structural formalism, a formalism for manipulating data structurally, while the linear algebra is perhaps a formalism for manipulating data statistically. And so here, try to strike a balance, but I actually lean more towards structure. And I'm going to defend that position in the next segment. The next dimension you can think about is scale. And so here is a one-end is yes, it fits in main memory on a single machine versus what I'll call cloud, meaning that you might require hundreds of machines to work on it. And here I cheat towards cloud. And the reasons that I've already sort of described are that it's no longer safe to assume the data fits in main memory and to train people to work only with data of that size. The whole world changes when you start moving to even two machines, let alone 100. And to not give you some exposure to that change would not equip you to be an effective data scientist. And then the final dimension I use is sort of the target audience. Is this for hackers or more for analysts? And by hacker I mean you already have significant programming experience and you're looking to sort of round out your skills in some of the mathematics. Or are you more of a technology decision maker who is trying to bring a little bit of technical depth? And here I like to actually strike a balance.  So we're going to try to strike a balance between these two. All right. So here's the choices we sort of made in this course. So one is we cheat towards abstractions. We cheat towards structs. We definitely like large scale. And then I say we'll strike a balance. But we actually cheat towards the analysts side. We favor the fact that there are going to be analysts in the room who don't necessarily have significant experience. And I've already gotten a lot of questions from folks over email who say, hey, look, I haven't been doing programming day to day. Am I going to be able to take something away in this course? And I think the answer is yes. Although there will be some programming, so be ready. Welcome back to Introduction to Data Science. So in this segment I want to talk about these four dimensions that I introduced last time. And I want to justify the first three of them. And we'll talk about this one next time. And by justify what I mean is I want to explain why I've positioned the needles at the locations I have for this course. So the first point here is this dimension of tools versus abstractions. And this may seem sort of obvious that we want to focus on fundamental concepts as opposed to specific tools. But I can appreciate that people that are taking this course and many other courses really want sort of hands on experience. And we're definitely going to try to strike a balance. But let me try to motivate why I think it's important to sort of focus on this angle. And to do this, let me tell one particular story that you see happens sort of time and time again. So in this case, we're talking about sort of databases and what is currently going on in the NoSQL systems. So before 2004, you had the big three relational database vendors plus some open source solutions like MySQL and PostgreSQL. And then arguably a big event in 2004 was when Jeff Dean his colleagues published this paper on MapReduce at Google. And if you haven't heard of MapReduce, we'll talk about it at length. And if you have, bear with me. So this was great. What this will allow you to do was process very, very large data sets. And it sort of rebooted the database feature set. So it really stuck on, it really focused on just scale out parallelism. And that's it. None of the other features of databases. And this was exciting to a lot of people because they didn't have to sort of deal with the extra features that they didn't need in databases, nor pay the exorbitant license fees associated with databases. This seemed like, boy, this is the right solution. And it took a few years, but a few years later you had an open source implementation of the ideas in this paper called Hadoop led by some folks at Yahoo. Now even in the same year, one of the earliest and most successful projects within the Hadoop ecosystem was this system called PIG. And what PIG essentially was was a relational algebra programming environment for Hadoop. And if you haven't heard of relational algebra, don't worry. We'll talk about it. But let me convince you that you notice the word relational there. Relational algebra is the secret sauce within relational databases. And so a very early on project that was deemed necessary in the Hadoop community and was wildly successful was to have relational style programming on top of this non-relational system. Moreover, you had other competing projects like Dryad Link, which well Dryad and then Dryad Link, which is an interface to Dryad, which also provided a relational algebra-oriented programming environment for large scale out parallel data processing applications. Then you had people literally put the language SQL on top of Hadoop. So instead of just the underlying formalism, it literally had the programming language you needed. Then a bit later, you had indexing for Hadoop, which is another feature that databases have that we'll talk about. You had people talk about schemas and more sophisticated types of indexing, which is two other things that databases have. And then you now start to see transaction processing being a very hot, very important topic in no SQL systems. How to support concurrent access at very large scale transparently. And this slide is perhaps a little bit old. It's now 2013, of course, at the time of this recording. And the Spanner system from Google is an important system to look at that we'll talk a little bit about later. OK, so now this isn't to say that MapReduce was useless. We're going to talk about it at length and for a very good reason. It actually has some pretty important permanent contributions, three of which I mentioned here. One is it was the first system to really emphasize fault tolerance. And the idea here in a nutshell is that when you're working with 1,000 computers at one time for any length of time at all, a few minutes, a few hours, the odds of one of them failing in some way is extremely high. And so databases didn't typically have to worry about this, because first of all, they weren't running on thousands of computers at once. And second of all, they were under the assumption that your queries would typically be pretty fast. And so fault tolerance during query processing, so you don't lose all the work you started on when you're running a query, was something that the Hadoop and the MapReduce paper really sort of emphasized and has now been sort of accepted by the larger community. The other notion which is a little more subtle is this idea of schema on read. And what I mean by that is the way databases worked in the past and largely still work is we designed this thing called a schema, which is a particular structure for your data. And then your job is to fit your data into that schema. And until you do so, we don't really want to talk to you. The database has nothing to offer you until you're able to sort of fit into some sort of a schema. And the observation was that, well, look, a lot of data doesn't come pre-equipped with a schema. We don't have a schema just sort of lying around. We have to do something with it. And it's huge. It's many hundreds of terabytes or something. So what do we do? Well, one answer is you can use these MapReduce databases and say, hey, do for this. But having to say that before you're allowed to touch your data, you must load it into a database, that was kind of a non-starter for a lot of applications. And then finally, this idea of user-defined functions is something that most databases support. And it's the idea that you might want to do things outside of what you can do in a normal SQL query. You might want to write your own code and push it into the database. But the experience of having to sort of write and maintain and manage and use these things is not great. And that's why a lot of people put their logic inside the application, as opposed to pushing it down into the database, where arguably it could do more good, for reasons we'll talk about. And so I think MapReduce sort of argued that, look, you can give the Java programmers what they want, a Java programming environment, and let them write scalable systems without forcing them to kind of use this crazy user-defined function interface that databases offer. So what's my point over all this? Well, if we focus too much on tools, what you would get is a snapshot in time of what tools are important, as opposed to seeing that some of these features around databases, they sort of ebb and flow in their popularity. But they're sort of a permanent value in your reasoning about large-scale systems. Similarly, you might lose track of what's actually novel and what's actually new in the midst of the conversation about sort of relational databases versus no SQL systems. So I want to focus on these abstractions throughout the course whenever we can. Great. Now, OK. So finally, we're going to focus on the abstractions. What are the abstractions of data science? Well, it's not clear that people really know yet. And I'll give my case for this in the next slide. But the reason why I don't think we really know yet is do you see these words being used like data jujitsu and data wrangling and data munging? And this is the real skill of data scientists that they have to be able to wrangle data. What does that mean? So my translation of this is we don't really know what we're talking about yet. But that said, there's probably a few candidates we can consider here. So maybe everything's a matrix, and everything we want to do with data, it can be expressed in linear algebra.  If you're more of an object-oriented programmer, everything's an object, and we communicate between objects by sending messages back and forth through by calling methods. If you're more of a sysadmin type, then everything's a file, and we write bash scripts to process it. And if you're an R programmer, then maybe everything's a data frame, and we call functions in this library. And MATLAB, similarly, perhaps from MATLAB, everything's an array, or a matrix, or a vector, I guess, in their parlance, and everything's a function on that. So I think of all these possibilities, there are two that stand out as likely candidates as fundamental abstractions for data science. And those are the first two here. And the reason is that we see these abstractions appear over and over again, independent of particular tools. Now, relations and relational algebra are closely associated with databases. But as I argued a few slides ago, and as we'll see throughout the course, you see this come up time and time again. And you even see it in, say, object-oriented languages, and you see it in R and so on. So these are the two that we'll mainly focus on in this course. So now I want to motivate desktop scale versus cloud scale. And the argument here for desktop scale is that, well, data science is really about the functions and the statistics and the manipulation and the techniques. So therefore, we can push large scale data into a separate course or a separate category and really just focus on the math and the functions. And I think this is a bit of a mistake for a data science course. And the reason is that this is a fundamental limitation of a whole category of technologies. And R itself is included in that, although there's a lot of great work on how to sort of scale R up. But in its basic usage, what you do with R is you read a file, load the whole thing into main memory on one machine, and then call functions on that. And so if your data doesn't fit in main memory on one machine, you're kind of out of luck. Now, you can be clever and start to use indices to kind of limit the data that you need to access. And you can start to try to be parallel to take advantage of the fact that there's now four and six and eight and 12 cores in your computers you'll buy nowadays. But trying to be clever in doing that yourself overlooks the fact that a lot of these techniques are pretty well understood and already implemented in other systems. So being able to be cognizant of what other systems can do and take advantage of those flexibly and write your application in terms of these other systems that already do scale out is a critical skill in data science. And so the point being made in this slide that is somewhat out of date, although you can get the idea, is that simple tools that are available in every machine, such as grep, which if you haven't heard of grep, if you're a Windows user and don't use grep too often, this is essentially search a file for a particular pattern. But it searches it linearly, right? It looks at every single line of the file and checks for the pattern. And so you can do a linear scan of a megabyte in maybe a second and a gigabyte in a minute and so on. And so at very large scale data sets, you can't do this linear scan anymore. You have to search in a more, in a smarter way. And he sort of has some costs over here that are probably horribly out of date now. Fine. So the point is large scale data is not just bigger, it's different. It requires a different way of thinking about techniques and it requires a different stack of technologies and to ignore it's a mistake to ignore that in a data science course. And then this final dimension of hackers versus analysts. And again, what I mean here is, am I going to require sort of deep programming proficiency in order to participate in this data science class and in general in the data science kind of activities? And the answer is I don't think so. I think we need at least two types of people and really sort of a broad spectrum of people. And this isn't really my idea. This often quoted report from the McKinsey Global Institute, you'll see this quote time and again. But the people that use this quote tend to focus on this first part that talks about 140,000 to 190,000 people with deep analytical skills. But the second part of the quote is, well, you also need 1.5 million managers and analysts who know how to use the analysis of big data to make effective decisions. So this means that it won't be just the programmers who are working in this space. And I wanted to sort of think about how to design a course that could appeal and inform both categories of people. And this is my last side of this segment. The other reason why I think hackers versus analysts is that the line between them is kind of blurring nowadays and technology can actually help here. It doesn't require a PhD in computer science or a bachelor's degree in computer science even, in some cases, to manipulate large data sets. And in order to back up this claim, I'll give you an example from some of my work where we have done some work to try to make databases easier to use for, say, biologists. And this really nasty looking SQL query that if you squint closely, you can see that it's actually doing interval arithmetic over genetic sequences. This is a pretty tough query to understand for even experts. This was written by somebody that doesn't do any programming whatsoever. She doesn't write a line of pythons. She doesn't write a line of purls. She doesn't write a line of R. And she's able to write these SQL queries that process very large data sets. So the fact that if you understand what's going on and if you can think in terms of some of these abstractions and you understand your problem well enough, you can participate in the activity of manipulating large data sets and doing data science, even without a deep background in software engineering. And that's why I want to push this needle somewhere over this way. I probably put this in the middle, I suppose. I'm not so much trying to focus on only the analysts. I just want to make sure that they're included. OK. Next time we'll pick up with the last dimension. Welcome back. Last time we talked about three out of four of these dimensions in describing how we designed this course in data science.  And so this is the relative importance of data manipulation versus deeper mathematics. And you can see that I've sort of put the dial here a little bit to the left. And I'll try to motivate that in the next few minutes. All right. So we already saw one example of this in the first segment where I gave some examples of data science from recent history. And one of these was Nate Silver's prediction of the electoral college votes for the 2012 US presidential election. And if you recall, this prediction was accomplished by essentially taking the average of the state polls for each state. So it didn't really require a sophisticated statistical model. And yet it had massive impact. So a quote that I think sums this up a little bit comes from Aaron Kimball at a company called Weeby Data. He says 80% of analytics is really just sums and averages. And so what he means by this is if you can get these sums and averages right, if you can do it at any scale on any data that you might see, then you can always sort of build up more advanced techniques. Everything sort of boils down to just sums and averages. So I think this is a motivation for why focusing on data manipulation, which typically is associated with being able to express sums and averages. For example, what you can do in a database query, which we'll talk about in the next couple of lectures, that gets you a pretty long way. It gets you 80% of the problem. So another way of looking at this is that there's three main tasks involved with a data science project. There's preparing to run the model, running the actual statistical model, and then interpreting the results and communicating it. I got the animation out of order here so you can ignore that red. But the point here is again Aaron Kimball from a conversation with him is where I got this. 80% of work is really in this first step where gathering data and cleaning it and integrating it and restructuring it, transforming it, loading it and so on. So verifying all these verbs you see here, this is the hard part. And so actually running the model, or even choosing the model and then running it, doesn't tend to keep people up at night in practice. So and then the joke here is perhaps that the other 80% of the work, implying that there's sort of 160% of a normal task as in data science, is in this interpreting the results. So this is the visualization and the communication and the explanation of the results, okay. So this is another reason why I wanna focus in this course on data manipulation tasks that are associated with this first number one task, okay. You know another way of looking at this is a quote that's now really old, right? So this is 12 years old or so at the time of this recording from Doug Laney. And this is the document that first coined this notion of big data being the three of these of volume, velocity and variety. And we'll talk about that in a couple of segments. But he has this quote, you know, no greater barrier to effective data management will exist in the variety of incompatible data formats, not aligned data structures and inconsistent data semantics. So this, what the database community, you know, my community calls the data integration problem. This is the hard part. And so he was saying this back in 2001 and I would argue that it's still true today. This is the greatest barrier. And then in the context of this, he was talking about this notion of variety being harder than volume or velocity. And I'll explain more about those, what those V's mean in a couple of segments. All right, so another vignette here is something that we like to ask the scientists we work with. So these are, you know, astronomers, oceanographers and biologists. We ask them sort of informally how much time they spend quote handling data as opposed to quote doing science. Now, you know, we let them interpret these quotes however they want. But what we mean by doing science, you know, choosing a statistical method or designing a statistical model, they absolutely consider part of their science. And so what we mean by handling data is all the other crap, you know, the format conversions and so on. And so what do you think the most common answer is here? You can guess to yourself for a second, but they don't even blink. They say things like 90%. And so this number should, you know, give pause, right? This is taxpayer money that goes to federal funding agencies to come back to pay some postdoctoral fellow to spend 90% of her time doing something that she doesn't even consider science. And so this is why I think it's really important as a data scientist to focus on this problem. Now, you might say, well, that's just science. What about business? But as I will try to make the point throughout the course, there's an increasing alignment between what's going on in business and what's going in science. Okay, and we'll talk about that more in a couple of segments. All right, so if 90% of the problem is handling data, you know, boy, we ought to spend a lot of attention on that. All right, so, you know, another argument that sort of follows on the first slide that I gave is that structs, you know, the data manipulation platforms and databases in particular actually go a pretty long way to being able to express more advanced things. And this isn't just a matter of, oh, well, you can express anything if you have sums and averages. It's also even fairly advanced techniques. You can, there's an increasing amount of interest in figuring out how to get this stuff into the database. Okay, so this is a slide I'm taking from Christian Grant where he argues that, look, you know, if you consider databases versus statistical packages such as SAS or MATLAB or R or SPSS, you know, this is what they're doing now. They're downloading data to use in their favorite statistical package frequently under the assumption that, well, of course I have to, right? Of course that's the only thing that could possibly express this. Well, look, you know, most of these stat packages, the first thing you'll do is read the data off of disk and load it into memory and then start calling functions on it. Well, if increasingly data sets simply don't fit in memory on a single machine, certainly not on your laptop. And so you have a couple of choices here. Either you shift into some kind of fancy cluster version of the tools for which they exist for things like SPSS and MATLAB, although they're quite expensive. Or you sample the data so that you only can work with a subset that actually does fit in memory. And you'll see this to be very, very common, is that it's just a part of the course to take a sample of the data in order to be able to work with it efficiently. Right. But the point here is that this isn't really required if you use different packages. If you, in this case, the argument here is that if you can use databases, if you can figure out how to perform your task in the database, you'll get the scalability for free. Moreover, these toolkits don't necessarily have any kind of notion of parallelism, right? So even if it does fit in memory, every machine you buy nowadays has at least four cores and then probably more like eight and soon to be 12 and 16. So to take advantage of all those cores on your problem is something you're gonna be looking for in a package. And this is something that databases can do automatically. Most databases, not all. In fact, the ones you may be familiar with, MySQL and Postgres typically do not, but other databases will and we'll talk more about this. Okay, so you get parallelism for free if you can use a database and you get scalability beyond the size of main memory for free if you can use a database. And that's perhaps a big if and we'll talk about it. Okay, so let me give you an example here and you'll actually do this as part of a homework assignment. But can you express matrix multiplication in SQL and if you can, then I'd argue, well, hey, now any formula that you can express using matrix multiplication, you can perhaps express in SQL by doing this over and over again. Okay, and the answer is yes. And in fact, the simplest version of this is pretty straightforward. So if you haven't ever seen SQL before, don't worry, we'll talk a little bit more about this. But if you have, bear with me, imagine you have two matrices A and B. Oops, I'm using the wrong device here. Two matrices A and B. And what you want to do is find all the, you know, and the representation of each matrix here is as a row, excuse me, row ID, column ID and value. All right, so that's your relation. Now, this is a very inefficient relation if your matrix is dense. And I'll let you think about, well, I'll tell you why and you can think about it a little bit more as well, is that, you know, an implicit representation of this only has the, let's say you have five rows and five and six columns, then you only need the 30 values, five times six. But here you're doing, you have to do 30 row IDs plus 30 column IDs plus 30 values. So you sort of triple the size of your data relative to, you know, efficient main memory representations. So why would you do that? Well, it turns out that a lot of matrices in practice are sparse, and I put that word right up here at the top. In a sparse matrix, not all the cells actually have a value, and so you don't actually need to store them. And so this representation, in terms of, you know, explicitly having a row ID, a column ID and a value, turns out to be pretty efficient. Okay, and in fact, sparse matrix solvers, this is exactly the kind of representation they use internally. All right, so if you have a sparse matrix and if you encode it and if you represent it in a database, then expressing matrix multiply is not too bad. What you wanna do is find all the columns, you know, for each column number in the matrix A, find the corresponding row number in column B and then add up all the contributions to the new value. And I'll show a diagram of this after. In fact, you know what, let me skip going into too much detail about this right now because I'm gonna talk about this in detail in preparation for the homework where you'll do this. So right now, I guess the takeaway, what I want you to take away is that doing, you know, representing matrices inside of a database sounds very unusual. It's actually not the world's worst idea. And in part of the readings from this mad skills paper, you'll try to, you'll see why. So right now I just want you to take away that it can be done and it's not necessarily a terrible idea. Welcome back. So I wanna talk a little bit about how the term data science relates to other fields of science. And in particular, I wanna introduce this term E science which to a first approximation you can think of as equivalent to data science. So while the term E science is associated with astronomy and oceanography and biology, data science has been adopted more in business but they involve a lot of the same concepts. So let me tell you about what's going on in science. So for thousands of years, scientific inquiry has been empirical, right? You observe the natural world or in some cases maybe replicate the natural world in a controlled environment in the laboratory and make observations about that. In the last few hundreds of years, science has accepted theoretical models as a valid method of inquiry, one that is reinforcing empirical methods. So new theories suggest new experiments and the theories help explain the observed data you get from the experiments. In the last 50 years or so, high speed computation has emitted an entirely new method of scientific inquiry. You can simulate in the computer phenomenon that otherwise couldn't be, you couldn't study, you can't observe directly and you can't reproduce in the lab and even the theoretical models become too complex to solve analytically using essentially paper and pencil, right? But you can actually start from initial conditions and run the simulation to get a result. So this is maybe what goes on in the interior stars or the shift of tectonic plates or the evolution of the universe or the effects on the ecology from some species dying out and so on. So that's fine, so that's three methods of inquiry but in the last 10 years or so, there's been arguably a fourth method of scientific inquiry which is to acquire massive data sets from instruments or from simulations and then explore these data sets using new algorithms and new infrastructure. And so eScience is really about massive and complex data, data large enough to require automated or semi-automated analysis. You can't look at it, you can't inspect it directly. And so the relevant tools here are the same as those for data science. Databases, visualization, scale out computing, maybe the NoSQL systems, machine learning techniques, web services and so on, okay? So the way, this idea of the fourth paradigm, there's a book that's in the reading list that you can refer to here and there's a lot of, there's some other articles in the reading list that you can also refer to. The story's been told lots of ways. The way I like to talk about this story is that science has always been about asking questions but conventionally it was really about querying the world. You would sort of have data acquisition activities, experiments or field studies that were coupled to very specific hypotheses, right? You had the question in mind first and you went out and collected data. But eScience has really sort of shifted a bit where now you're kind of downloading data en masse, you're downloading the world first, putting into some sort of a representation in the computer and then querying that database to test your hypotheses. And so the data can be acquired independent of any specific hypotheses in some cases, okay? And this is due in part to the cost of data acquisition dropping precipitously thanks to advanced technology, right? So the telescopes you can build now that we'll talk about in the next couple of slides can acquire enormous amounts of data at very high resolution. And in the life sciences, you have sort of laboratory automation and you have high throughput sequencing. In oceanography, the sensors are getting cheaper. The models, thanks to Moore's law, and advances in computing, the simulations you can run are getting bigger and higher resolution and therefore producing larger and larger amounts of data and so on. And so the rate at which data can be produced has far outpaced the rate that we can analyze it or come up with the questions we need to ask about it, okay? And this suggested a new approach to science. So let me give you some examples. So we've said that eScience is driven by data more than by the computation, right? So some examples of the size of the data that's coming out. The Apache Point Telescope, that was the primary instrument for the Sloan Digital Sky Survey that we might refer to multiple times in this course, produced 80 terabytes of raw image data over a seven-year period. At the time, this is a pretty significant data size and even by many standards is still today. The next generation of this, the next generation project that's in the same sort of spirit as the Sloan Digital Sky Survey is the Large Synoptic Survey Telescope. So this guy can produce 40 terabytes per day and will do so for over a 10-year period. So in total, 100 plus petabytes and producing the same amount of data that Sloan Digital Sky Survey produced in over a total entire lifetime, it can produce that over every two days, okay? And so this is a pretty staggering amount of data and requires a pretty different approach. One thing I wanted to mention maybe about Sloan Digital Sky Survey, what they actually did here was to take the images, cook them, right? Extract the relevant objects from it, put all those objects into an off-the-shelf relational database. In fact, it was Microsoft SQL Server and critically host this database online and serve it out over the web. And this required a pretty significant investment in infrastructure, but as a result of doing this, of making all the data public and queryable, it became the most productive astronomy facility in history, right? So the number of papers that have been produced on this data is on the order of thousands and the original PIs of the project, the principal investigators of the project had sort of maybe on the order of 100 papers in mind for the data and the other 4,900 papers that have been written all came from external partners writing queries against this database. So just a wild, wild success. Now the problem is that the same technology stack and to some extent even the same approach is difficult to apply in this case of the large enough to survey telescope. The reason why this guy's producing so much more data is not just that it's much higher resolution and can perceive a much deeper field in the sky, but also because it's returning to the same point in the sky frequently every three days. And so this allows you to look at things that change over time. So asteroids, comets, you might get a supernova and so forth. Okay, and by comparing these images in the time series, there's all sorts of new questions you can ask, okay? So both because of the science that they're gonna do and because of the sheer scale and because some of the complexity of the details of how the data is acquired, the existing, the previous solution won't work. And so this is motivating a whole new area of research to study data management techniques and data analysis techniques to support this project, all right? So in life sciences, these high throughput sequencers are capable of producing terabytes per day when run continuously. And major labs that do this work, such as the Joint Genome Institute, have 25 to 100 of these machines running all the time, right? So this is spitting out an enormous amount of data for, I was gonna say, a variety of samples. So it's maybe individual organisms or it could even be samples from the environment where there's no one particular organism in there, but there's an entire population, all right? So for a variety of uses, these guys are able to spit out the data. In oceanography, the regional scale nodes of the NSF Ocean Observatories Initiative is a project led here at UW. Ocean Observatories Initiative is a multi-institutional partnership. The regional scale nodes part is run at the University of Washington. So this project is evolving 1,000 kilometers of fiber optic cable on the seafloor connecting thousands of instruments in chemical, physical, and biological, thousands of chemical, biological, and physical sensors, including live video from the seafloor to measure, to monitor volcanic activity. Okay, so again, the database, if not a relational database, the data sets and data infrastructure required to support this effort is significant, it's motivated a lot of new research. In the information space, there's a lot of science to be done directly on the web itself. And so just the web, you know, a single computer can read 30 to 35 megabytes per second from one disk. And so it would take about four months just to read the entire web. So new clusters of machines. So summing up a little bit, eScience about the analysis of data. So the automated or semi-automated extraction of knowledge from massive volumes of data. And so your main instrument for looking for answers are the algorithms and the technology as opposed to direct inspection. There's just too much of it to look at yourself. But it's not just a matter of volume. And as we'll talk about in the next segment, this is another link back to what's going on in business, right? There's this concept of big data and there's the three V's of big data that we'll talk about a little bit more next time. But let me just mention them here. Where volume, so the three V's are volume, variety, and velocity that you'll read about. And I'll give you where these terms came from in the next segment. The volume refers to just simply the number of rows or the number of bytes, right, the sheer scale. Variety is perhaps the number of columns or dimensions, but in science, for example, a lot of, in the life sciences in particular, you'll have experiments that involve accessing multiple public databases, as well as multiple sensors, your own data that you've collected and that of your colleagues. And the integration task of putting all this data together is a significant bottleneck, even when the actual scale of the data is not necessarily all that bad. Okay, so this is the complexity of the data. And then the velocity, you know, we saw with the large optic survey telescope that, although the scale itself was enormous, the fact that 40 terabytes are being collected every two days means that the infrastructure needs to keep up with that pace. And just transferring that data from the telescope facility to the data analysis facility is an engineering challenge. Okay, and you'll also see other V's here where things like veracity, you know, can we actually trust this data? Okay, so a bit more of that next time. So to summarize here, science is in the midst of a generational shift from a data poor enterprise where you can never, you know, there's never enough data to a data rich enterprise where there's so much of it you don't know what to do with it. And as a result, you know, data analysis has replaced data acquisition as the new bottleneck to discovery, right? So it's not that the cost of going out and getting the data, it's the cost of actually analyzing the data you might already have. So this is fine, but what does this have to do with business, which is probably where a lot of you are coming from and where your interests lie? Well, what we see is that business is beginning to look a lot like what's always been happening in science. So though, you know, businesses are requiring data aggressively and keeping it around indefinitely in case it becomes useful. They're beginning to hire people that have training and skill sets that look a lot like what's been important in science for a long time, especially in mathematical depth. And they're beginning to make decisions with this data that are very empirical. So we're always wanting to sort of back up every decision with a clear case based on data. And so for these reasons, I think that you can take the lessons learned in science and apply them in business and actually vice versa as well, because one thing where science is lagging behind businesses is in the adoption of technology. There's been proportionally a lot less spent on IT infrastructure in science than there has in business. And so there's this, it's a great time for this. There's this cross-pollination of ideas between both fields. Okay, and so going back to this first slide that I gave, eScience and data science have essentially everything in common. So we might use examples interchangeably between the two.  And I'm not too concerned with any sort of technical definition of the term because it probably doesn't exist. But I wanna arm you with some of the language that people use when they describe big data so that you can speak intelligently about it when asked. Okay, so probably the main thing to recognize is this notion of the three Vs of big data, which are volume, velocity, and variety. And we talked a little bit about this in a previous segment. So just to repeat, volume is the size of the data and measured in bytes or number of rows or number of objects or what have you, sort of the vertical dimension of the data. The velocity is, what I'll say here is the latency of the data processing relative to the demand for inactivity. And that's maybe a mouthful. But what I mean by that is, how fast is it coming based on how fast it needs to be consumed? And so there's a lot of applications for which interactive response times is increasingly important if not directly important, okay? And so when this becomes the bottleneck, when this becomes the challenge, then this velocity term starts to become pretty relevant. And then the one that I think is really pretty interesting to me and is near and dear to my heart with my research is this notion of variety. And so here the problem is, an increasing number of different data sources are being applied for any particular task. And so you need to pull out ASCII files as well as download data from the web, as well as pull data out of some database, as well as use some NoSQL system. And so on. And the integration of all these different data sources is a pretty significant problem and can end up occupying a lot of your time. And so I made this point a couple of segments ago about researchers who spend 90% of their time quote handling data. This is where a lot of that time is going. This notion of variety, okay? So all three of these are relevant in performing sort of data science tasks. All right, let me give you another notion and I'm gonna go back to use science examples and you've seen some of these before. But if you sort of make a plot of number of bytes on the y-axis versus number of data sources, maybe columns of data in a single table or columns data across multiple tables or numbers distinct data sources on the x-axis, you can sort of map out different fields of study or different problems and sort of see where they lie. And so typically astronomy has been challenged by the sheer volume of data. And so they're sort of up here. We're high on the y-axis, but you know, but the number of actual sources in astronomy is not too high. There's telescopes, there's these spectral images and then there's simulations of the galaxy. And so that's relatively few in say the ocean sciences and certainly in the life sciences, although I only show one example here, the variety is really more of a challenge. The actual sheer scale is not as high as the hundreds of petabytes that can be generated by these telescope infrastructures like the large synoptic survey telescope. But the number of different types of instruments you can use to acquire data is large and ever-growing, right? So you have these glider systems that will go out for months at a time and kind of porpoise through the water. You have autonomous underwater vehicles that are more for short-term missions. You know, there's oceanographic cruises where they deploy these conductivity, temperature and depth instruments that can take profiles of the water, right? So this is, you know, at a fixed x, y and a varying z and a varying t, a varying depth and a varying time while the gliders are sort of varying in all four dimensions. You have these simulations that are probably one of the largest sources of information. Right, so these can be mesoscales sort of at the order of the entire Northern Hemisphere or the old Eastern Pacific or they could be models of a particular bay or inlet or estuary connected to a river and connected to the open ocean, so a much smaller scale thing. So there's a lot of diversity there. And I say stations to mean these sort of fixed stations where there's a particular sensor deployed at one location and just measuring across time. ADCP is a acoustic Doppler or something profiler where they're using sound waves to record the time that the sound waves take to bounce off particulate matter in the ocean and they can therefore measure velocity. And so this gives you an entire profile of the velocities in the ocean. You can mount these on the seafloor pointing upwards or you can mount them on the bottom of boat pointing downwards and so on. And then there's satellite images that measure sort of sea color and wave breaking as well. Okay, so fine, so just a little more on the term big data. The notion that Mike Franklin at the University of Berkeley uses, which I like is that big data is really relative, right? It's any data that is expensive to manage and hard to extract value from. So it's not so much about a particular cutoff. What makes it big?  You know, not necessarily. It depends on what you're trying to do with it and it depends on what sort of resources and infrastructure you have to bring to bear on the problem. And so in some sense, difficult data is perhaps of what big data really means, right? It's not so much about big. It's about being challenging. Okay, and so this is really important to remember that big is relative. So let me give you a little bit of history of the term big data. There's the earliest notion I could find was from Eric Larson in 1989 where he says the keepers from Harper's Magazine eventually went into a book, the keepers of big data say they do it for consumers benefit, but data have a way of being used for purposes other than originally intended. So his point was not really about technology at all. It was just a notion that data is being collected for one purpose and being reused for another, which is a theme that I mentioned in the very first segment in this course and we'll come back to over and over again. And so I think he had it right in that sense. So his real point was that about consumer private data is starting to be commoditized, which was absolutely true and fairly prescient at the time since it's become a big issue now. But, and it's been perhaps especially impressive that given that this predates the rise of the internet and already sort of foreshadows very topical issues in big data such as ethics and privacy and sensitivity and so forth that we'll talk a little bit about. But this isn't quite what we mean by big data nowadays typically because it didn't have that technology aspect to it. It didn't talk about the challenge of actually managing these data sets. All right, so another point of reference is that more recent reports from some of these consulting firms get credit for these notion of three Vs but really the original source of this was a report from Gartner in 2001 written by a guy named Doug Laney. And so we talked about volume velocity variety which we've said but let me just give you a chance to look at these quotes, you know. So in volume it's, he's really talking about sort of business to business. When you think about 2001 this is around the dot com boom and so everyone was trying to figure out what this new era of technology was gonna get what the internet was really gonna give to them beyond just sort of putting up a webpage and serving it out to your customers. What, how are you gonna be able to interact with your supply chain or your vendors and so on? Okay, and so this is what we need by this notion of e-channels. But you know, up to 10x the quantity of data about an individual transaction may be collected. You know, absolutely true that this data exhaust to this point we've made a couple of times is giving rise to a larger scale of data being collected. You know, and on velocity, well it's increased the point of interaction speed, right? So this is that need for interactivity that didn't used to be so required but as the velocity of all business and all transactions sort of increase so do the constraints on the infrastructure used to process it. And then on variety, you know, I like this one a lot. So through 2003, 2004, right? So he was being sort of fairly conservative about how far out he wanted to predict. No greater barrier to effective data management will exist in the variety of incompatible data formats non-aligned data structures and inconsistent data semantics. And so this is great. This is, you know, he could have said this through 2015 and been arguably correct. This problem has not gone away, okay? All right, so another point in the history of this term, big data, there was a series of talks, a lot of work by John Mashe who was formerly the chief scientist at SGI who would talk about big data being the next wave of infra-stress. And so what he meant by infra-stress was what's gonna really drive the technology forward? Where are we gonna feel the pain? And his point was that the IO interfaces was where it was tough. So in particular, disc capacities were growing incredibly fast and still are and that the latencies are not keeping pace, right? So you can go down to a local store and buy a three terabyte drive for probably $200. But the rate at which you can pull data off that has essentially the same as it's been for many, many years. And so now it takes you hours to actually read every byte of data on that disk that you've stored. And so this is a problem because the actual analysis you can do over all those data, we can keep it and that's fairly cheap, but we can't actually do anything with it because the pipe is so small, okay? And so this is one of the arguments, he made several sort of very cogent arguments about where the bottlenecks are in the infrastructure, but they all sort of revolved around this idea that large big data and big data processing were gonna be the stress point. And so this is probably a pretty appropriate use of big data, although John Matthews has said that he's not sure that he deserves any credit for coining this term since it's a fairly generic term. And he was using it in one way and we use it now in a related way, but it's not necessarily that it captures everything we need today. And I probably agree with him. All right, so just another quote about big data sort of today where we're trying to capture exactly how it is being used. The necessity of grappling with big data and the desirability of unlocking information hidden within it is now a key theme in all the sciences. And I like this, arguably the key scientific theme of our times. And so we've talked about science and we talked about the fact that I think what's going on in science is more or less equivalent to what's going on in business. And so I think this is nice, right? It's really maybe the key problem across all fields is unlocking what's going on inside big data, right? Getting some, extracting value out of big data. So fine, the final point I wanna make is, where does all this big data come from? And we said this a little bit before, but one is data exhaust from customers, right? So we're actually tracking a lot more information about interactions with customers than we used to, right? It's not just about taking their order. It's about monitoring their clickstream that they use in order to get to that order. It's about not just about sending advertisements to them, but it's about watching how many clicks are on each advertisement and whether the number of clicks goes up or down to where that ad is placed on the webpage and so on. Another point that, especially true in science, but I think it's also true in business, and I'll give a couple of examples of this, is that the availability of new and pervasive sensors, right? We're actually able to get visibility on data sources that we previously couldn't. And I'll give a couple of examples of that in a second. And then I think, I mentioned this already, with the technology of data storage, right? Just this capacities of disks has gone up. The cost of storing a byte has gone down. And so we sort of have this ability to keep everything whether or not we need it, or at least there's a perceived ability to keep everything whether we're not and we need it. And so people are doing so, right? The other things that they would have otherwise thrown away they're starting to keep. And then scratching their heads thinking, boy, how might I use this to make predictions and make better business decisions? Okay. Okay, so just a couple of examples of sensors that may or may not lead to massive data, but just examples of where we're getting visibility on data sources that we didn't previously have.  And the reason is for forensics in the event of a crash, but they also record a lot of other information. And so insurance companies have similar devices that you can opt in, you can voluntarily plug in to reduce your insurance rates that track your speed, track other kinds of aspects of your driving habits. So this technology simply would have been hard to imagine 20, 30 years ago, but now that we have the technology, why not actually start collecting the data from it? Okay. And so while the purpose here is pretty clear, at least for the insurance companies and for these black boxes or crash forensics, you can imagine repurposing that data for other purposes. And in fact, that's what this article is about as well. Boy, is there a privacy risk here given that there's one purpose, they're being deployed for crash forensics but they might be used for other purposes. And for insurance companies, you can imagine how they would collect it with a particular actuarial risk model in mind, but may develop other models in the future given that now they have this data. Okay. And so I think this is really a theme of big data is that we're collecting new source of information independent of whether we know how we're gonna use them or not in the future. All right. So let me give you another couple of examples from research here by Schwedig Patel in the Computer Science and Engineering Department. And there are several related devices and here I just mentioned two of them, HydroSense and ElectroSense. And so both of these devices are intended for consumers to use to monitor their own resource consumption. And so instead of having to kind of rewire your house to monitor the consumption of every device in your house, every faucet and every shower and so on, you can just clamp this device on the main water line coming into your house and it will monitor the pressure changes associated with every individual device. So it can disambiguate that flow and recognize the signature that every device puts on the pressure changes. And so every time you turn on the shower in the upstairs bathroom or flush the toilet in the downstairs bathroom, this thing can tell you, can identify when that happens and give you a log of the events. And by analyzing that log, you can tell which device, where most of your water is going. Is it mostly going for showers? Is it mostly going for a dishwasher and so on? ElectroSense is very similar in that every electrical device in your house puts a recognizable load on the signal, on the power signal that can be read and disambiguate it to tell you where your energy is going. Okay, so these are just examples of new sensors that are coming on the market where data is being derived that otherwise wouldn't be able to be derived at all. Okay, in this segment, I want to talk about logistics of the course. So how we've organized this course is a guided tour of important trends along with a deep dive into specific topics. And then there's a set of hands-on assignments that are intended to deliver specific skills and experiences, and that's perhaps the most important part. Okay, and so overall the course is not, you know, the challenge here was to design a course that would be broad enough to cover the topics that we want and also inclusive enough that we didn't sort of have to dial it in for a very specific cohort. But the challenge then is that it's going to be very difficult for some people and others may find it, some aspects of it certainly routine. I'd be surprised if anybody finds the whole thing routine. If so, then I'd be surprised that you took this course. Okay, so the prerequisites here are pretty light because we are trying to cast such a wide net, but they're really important. So some prior programming experience in some language is going to be really critical. Then, you know, we're going to use terminology from basic college statistics or advanced high school statistics. So when I talk about linear regression, you should know what that means. You should also be able to sort of look at some visualization of data and be able to understand what it's telling you. Okay, and then perhaps the toughest one is that statistics. Perhaps the tough one is to have some exposure to databases and databases concepts. And, you know, if you're just starting out in college, that's not always an easy proficiency to have gained or experience to have gained. But, you know, it's not, we're going to couch a lot of the discussion in terms of databases and in the relationships to databases. And so some idea of what that means, what they are is going to be helpful. Okay, so to that end, one assignment will involve writing SQL. And if you've never written SQL before, but you understand database a little bit, you will probably be able to power through the assignment. If you're an expert in SQL, there's some parts of it that might still be interesting to you. Then two assignments will be required. We're involved writing Python. One optional assignment will involve sort of processing big data using Amazon web services. And here, you know, one of the reasons it's optional is that because of the varying skill sets, but another reason is that you'll have to pay out of pocket for the cloud resources. And the reason for that is there's, you know, 60,000 students who signed up for the course and we can't sort of pay for all of them. The good news is it will cost sort of less than $10 or so. Okay, and it's optional. So if you don't feel comfortable with that, you don't have to do it and you'll still get full credit for the course. All right. Then another assignment will involve competing in a Kaggle.com project, participating in a Kaggle.com competition using whatever you want. And so this may or may not involve any programming. A lot of valid assignment, you know, you can certainly compete by using Excel and other kinds of GUI tools. Okay. This last bullet probably isn't true, so let's just ignore that actually. Okay, so the learning objectives here is I really want people to come out of this course, being able to talk intelligently about the landscape of data science, concepts, tools, algorithms, and technologies. And this will be sort of a springboard for you to dive deeper into particular areas. So for example, machine learning. This is not a machine learning course, but you can dive deeper into machine learning by taking this course. This is not a database course, but you can dive deeper by taking a database course and so on. Okay. Then I also want to deliver some hands-on experience, manipulating data in order to level set people that don't have any programming experience and provide some specific experiences for those of you who do have programming experiences. For example, the first Python assignment will involve competing in some sentiment analysis using Twitter data. And so if you already know Python, the learning Python won't be much of a contribution to that assignment, but perhaps it's the first time that you've been able to work with the live Twitter stream. Okay. So the end result of this is we hope that you'll be sort of an advanced beginner in a variety of data science topics. And as I said, the tough part here is sort of how to do something more than just superficial access given that data science encompasses such a broad area as we've discussed. And so we think we put together a pretty good program, but you'll have to be the judge of that. Okay. All right, so the risk of belaboring this, the course of the philosophy here has been that the skills needed by the data scientists span a variety of different areas, statistics, programming databases, distributed systems, visualization, but the traditional organization of these topics is sort of vertical and is not ideal for becoming sort of introductory in data science, right? So in order to get introductory level knowledge in all these areas, what you end up having to do is take an introductory course in seven different areas or something. So a lot of different courses. Okay. And so our goal is to try to expose and simplify the links between these different areas, okay, as opposed to sort of narrowing our attention on what makes them unique, okay? All right, so, you know, after taking this course, you will not be an expert in statistics. You will not be an expert in machine learning, certainly. You will not emerge an expert in databases or even no SQL, nor will you sort of have programming versatility in all of these language. However, you will use all of these tools. You will understand the basic concepts of all these tools and you will have applied many of these tools, okay? The assignments will have online short quizzes during the lectures, of which you've already seen some, these finger exercise quizzes. There will be a set of the full length offline assignments, as I mentioned, and some of the assignments will be graded some of the programming assignments will be graded automatically. Some of the assignments that don't lend themselves to auto grading will be assessed using the peer assessment tools. So an example of that is you're gonna write up a description of your Kaggle solution, in addition to submitting your score for the Kaggle competition and other students are gonna sort of grade whether it's comprehensible or not, okay? So here's my background in one slide. So I have a bachelor's degree in industrial and systems engineering from Georgia Tech, but all the problems in industrial engineering tended to be about optimization and automation, which seemed to require software. So I sort of got more interested in computer science. So I spent a couple of years consulting with some big firms. Solmberg does oil field services and Siebel does customer relationship management software and you're probably familiar with Microsoft and Verizon and Deloitte is a managing consulting firm. And then I went back to grad school and got a PhD in computer science from working with oceanographers on query systems for large scale oceanographic models. And then I spent a couple of years working directly with oceanographers as kind of a data architect. And before coming to University of Washington where now I lead a group in scalable data analytics for the University of Washington E Science Institute and also an affiliate assistant professor of position in computer science and engineering. And so there's a bit of a mix of very practical kind of applied work as well as my research agenda. And so I think that this data science trend that's occurring sort of strikes close to home with me. I think it's a great time for it. Okay, this is a walkthrough of the first assignment for introductions of data science and here we'll be using Python to process some Twitter data. So the first thing to do is to go down to the link where you can get the class virtual machine and make sure you have that if you're gonna use it or to make sure you've installed Python if you are not using the virtual machine. Okay, and I encourage you to read this material about the assignment and about Python, especially if you're new to Python, but I'm gonna scroll right down to problem zero. And so some of these steps, problem zero included should be pretty straightforward for you if you've used Python, especially if you've used it extensively, but there'll be some steps in this assignment that should still give you a challenge. And if you're new to Python, this is a great way to get warmed up. So I've opened the virtual machine here and there were a couple of things I had to confirm to allow VMware to convert it to VMware format, but it wasn't too hard. You can also use virtual box, which some folks have had, have done. Okay, so here I am in the home directory and there's some things that have been automatically exported by VMware from the host machine. You can ignore those, but we have this data side course materials directory and I'm gonna go in to that. And the first thing to do every time you start an assignment is to do a get pull. Make sure you have all the most reasons, changes to get assignment. This allows us to fix bugs in the last minute and not have to worry about how to get you those changes. Okay, so I'm already up to date, so I can just get started. We can look at what's in. I'm gonna go to assignment one, which is what this assignment is. And I'm gonna scroll down to problem zero and see what it asks me to do. Okay, so the first thing here is to, we're gonna access some tweets through the public API that doesn't require any authentication. So this is pretty much identical to just going to the Twitter website and typing in a search term, in this case, Microsoft. And I'm gonna do that programmatically by copying this little snippet in. So we're gonna need a text editor for this and I typically wouldn't use a GUI text editor, but I'm going to, in this case, I think it might be the most democratic approach and this text editor called g-edit is already installed, so that's what I'm using here. So I've just pasted this in. You may or may not be able to paste into a virtual machine without some configuration. I've done that configuration, which amounts to, for VMware, amounts to installing VMware tools. You're not gonna go into that here, but hopefully you can do that. If not, then you can type it in and it shouldn't do too hard. That's really the only thing you're gonna have to be able to copy and paste. The rest you'll be typing anyway. Okay, so I'm gonna save that. I'm going to assignment one and name it print.py. And now I'm gonna go back over to the terminal window and I'm gonna run it. Now I'm choosing to not use a Python ID development environment. You certainly can if you want. I typically use this kind of a mechanism for a variety of reasons. One is just because it's a little bit more portable when I find myself in an environment that I don't have a lot of control over or that I can configure. Once I get sort of hooked on a development environment, it becomes awkward to not use it. So I sort of prefer to just not be hooked on any of them. But if you have one you like, by all means use it. Okay, so there I just ran print.py and I see a bunch of junk on the screen. So let's go back to the code and see if we can understand what this junk is. So what this line is doing, this is a response from the website, which is very simple, just a one-liner to open the URL and get whatever the website sends back to us as a string. And then this library parses that string because we're asserting that this is in a format called JSON and this JSON library knows how to understand that. And so what we get back is a Python object that has all this complicated structure in it. So all this complicated structure was there in a string and now we parse it into a Python complicated object. So what kind of complicated object? So let's say pi response. And we can check the type of pi response to understand what we're looking at and print that out. And it says it's a dict. Okay, so a dict maps keys to values and to access the keys in the dict we can use a method called keys. And I'll do that now. And so this object appears to represent a page of results. There's sort of a page thing that might be the page number. There's an indicator of the next page and so on, but what, it could be the original query you sent. And in fact, I'm actually somewhat guessing on some of the meaning of some of these. You can read the Twitter API documentation to try to sort this out or you can just manually inspect it. But this results key sounds intriguing. So let's take a look at that. So how do we actually access the value for a particular key? We use the square bracket, oops, excuse me. We use the square bracket notation and pass the key name as a string. And in Python, you can use double quotes or single quotes interchangeably. And so I've used double quotes here. So let's print that guy out. All right, so there's more junk. So the results is some complicated data structures. Let's do our standard trick here and see if we can understand what that is. So I'm gonna check the type of this thing and I'm gonna comment this out. And so comments, you can use a little hash character. And so that's a list. So what can we do with a list? Well, first thing I'm gonna do is put into another variable just to kind of keep myself sane. So these are now the results. And I'm going to print out the first element of the list. And so here we're using the same square bracket notation, but instead of passing a key name, which is what dictionaries support, I'm passing an index number, which is what lists support. Okay. And you can look at the Python documentation. There's some links in the assignment to learn about the methods supported by both the dictionary and the list type. Okay, so now we printed this guy and we get one element of the list, which itself looks like yet again, another complicated, that's gonna get annoying, another complicated object. And I happen to be able to get a feel for this. You can recognize that this is a dictionary because of these curly braces, but we don't necessarily need to guess. We can just check it out once again. So what is the type of this list element to dit? All right, same thing. Let's look at the keys of it. And notice I'm not caring too much about a lot of software engineering here. We're just trying to inspect some data that we pulled from the web. And this is something that comes up quite a bit, I think, in these data science tasks is you're looking at some data for the first time. And so it's not necessarily the time to get too obsessive about software engineering because there may be no one that's gonna look at this script besides you. You're just trying to get a feel for things. Okay, so there I've printed out the keys for this dictionary and I see some things I don't understand, but I would argue that perhaps text looks interesting. So let's take a look at the text of the tweet. We're presuming that this is indeed a tweet. And so now here, this is a little bit ugly. You know, I'm not saving the result as a variable. I'm just gonna print out directly, but this is okay. This expression, so what's the type of this expression? Well, that's a list and we know with a list we can access it with this square brackets and a number. So what's the type of this expression? Well, that's a dict and we know that we can access those with these square brackets and a key name. And so the type of this expression is whatever's in text and we haven't actually looked at that yet. So let's do that now. So that looks like a tweet. Okay, and we can try out another one. And that's a different tweet. And if we wanna loop over all tweets, we can do that too. We can say for I in, use this built in range function and now we can print out all the tweets from the first page, one per line, which is actually what the assignment asks for. Now for this one, I don't bother to turn anything in. That was just sort of a warm up, especially for folks who haven't really looked at Python before, but that's how you can kind of inspect a complicated data structure and try to make sense of it without doing too much work. So now we're gonna get access to the actual 1% live stream. So this data was again, just like going to the website and typing in a search term and looking at the results. But that's not a, we can't get a actual sample of all the tweets that are coming in. And so for that, you need to actually log in to your account and do a little bit of configuration. And the steps to do that configuration are here in the assignment. And we'll walk through that real quick right now. Okay, so we go to, by the way here, if you're not using the virtual machine, you will need to install the OAuth2 libraries and make sure you notice that. Okay, so if you first need to create a Twitter account, you don't already have one. And then once you do, you can navigate to this URL and you'll create a new application. I've got a couple here. You won't have those, but you can create one with that button. Okay, so I'm gonna name this, how about assignment one, just gonna name it walkthrough, why don't we do something more like what you might do. And this is a data science assignment Twitter application. And so what you're doing here is registering an application that's going to consume this Twitter feed. Now we're just gonna write a couple of scripts with it, but imagine you're building some sort of a website that was gonna do some analysis.  So in case you do something strange, they can be aware, have a little more information about what you're doing with it. Okay, so here you can put in basically any URL and this is again just a way to help provide some problems in case you do something untoward. I'm just gonna put in my homepage and you can essentially put in anything. Actually, yeah, and just to be clear, it actually literally can't be anything. It's not even gonna be sort of validated. Okay, so fine. So here's an agreement here and the capture and I'm gonna agree to that. Type in this guy and then create the output. Whoops, that's already been taken, which makes sense. So why don't we say data science could have been a nice complicated name and I don't care too much about it since we're not actually building an application with this. It's actually interesting. I didn't realize it had to be globally unique unless I've already done one. Okay, so now we have some information and some credentials here, but there's one more step. We actually have to authorize this application to be able to use the Twitter data and I'm not gonna go into the details of OAuth here. It's somewhat complicated but necessarily so to sort of support all the requirements we're trying to do to do web-based authentication. So you can read about it if you're curious but I'm gonna just sort of blindly follow the instructions in this walkthrough. Okay, so it's been created. It gives me this little green message but it takes a second to come back. So you're gonna sort of stare at this and nothing's gonna happen and I'm gonna refresh and there it is, okay. So don't be alarmed if it doesn't immediately come back. It takes a second. Okay, so now we have two sets of credentials here. The same called a consumer key and a consumer secret and access token and access token secret and we're just gonna plug these in blindly to this file, twitterstream.py. And right at the top of this, you've got placeholders here where you can put these. So let me put those in. So there's those and now the rest of this file is not something you need to edit. This is just manipulating the OAuth protocol to get the results. So I'm just gonna save that guy and then run it. Python, twitterstream and sure enough, I'm getting tweets from the website and this will just run forever constantly streaming down tweets but we don't wanna send it to the screen. We can't do anything with it. We would rather send to a file. So I'm gonna hit control C to cancel the stream and then I'm going to redirect this to an output file and I happen to know that the format is again this JSON format which stands for JavaScript object notation and so I'm gonna redirect it to that. I'm gonna hit return on that and it will sit there and silently blink at me forever. So I'm only gonna leave it running for maybe a little while. What I've asked for in the assignment here is to let it run for 10 minutes or so so you can get a fairly big data set. There's a step further down in the assignment that require a reasonable sized data set to really be meaningful and 10 minutes should be enough and if you wanna run it longer, you can. In an early version of the assignment, I actually tried to ask people to run it for hours and hours and hours but that's turned out to be a little bit unwieldy. So I pulled back from that. Okay, so that should be long enough. I'm gonna hit control C and I'm gonna look at what's going on here. So now I have a four megabyte file called output.json and I can look at that file, the first 20 lines of that file with this command and so that's what you're gonna turn in for this assignment is the output of head dot the first 20 lines of the file. So I'm gonna pick the output of this command and put it into turning.json and you can turn that in on the Coursera website and I'm not gonna walk through that step in the interest of time. Welcome back. I wanna talk a little bit about data models as a run up to talking about databases over the next couple of segments. So this is a data science course so we know we have data. So the first question to ask is just where is this stored? How do we store data? And so one way of interpreting this question is just to talk about technology, right? To what technology do we use? Well, we use magnetic media and we might use solid state drives more recently, right? And so both of these have the property that they persist even when the power goes up. The data is safe even when the power is not on. Okay, so non-volatile storage, right? And so a lot of the work in databases is all about how to work with data that's stored on these non-volatile storage media. But another way of interpreting the question is a little different, a little more of the logical way we store it, the organization of the data. And so we might ask, one way of interpreting this question is what is the data model we're using, okay? So it's not just bits on a disk or bits in a file, what do we do? Well, you know, in your personal computer or maybe in your computer you might store data sort of hierarchically arranged in these kind of nested folders, right? That's one organization of data. And so the data model here is kind of tree-like. Another way is rows and columns and this is what we'll talk a lot about in this course, right? So in this case it's an ASCII file and these are hits from a biological database, matches in a biological database for a particular sequence.  Maybe they look a little bit like rows and columns but maybe they don't. So you have sort of an embedded table within a spreadsheet and so on. So you need to sort of, the idea is to think about what data model is being applied whenever you think about data, right? So it could be a tree, it could be a table, it could be something, a grid like this unstructured, or it could be a graph. And we'll talk a little bit about that in the future. All right, so in general, what is a data model? Well, there's gonna be three components that you should remember, right? One is that there's gonna be some notion of a structure, right? So in the case of tables, it's rows and columns. There's gonna be some notion of constraints. What are the legal structures you're allowed to create? So for example, typically if you think about a tabular data model, you won't have all the rows will have the exact same number of columns, right? This is sort of a constraint. You might also have more constraints on the values themselves, such as this field must be this column, every value in this column is an integer. And you can even have other kinds of more semantic constraints such as every column in this, or every value in this column must be within a certain range of numbers because it represents, say, days of the year, okay? And the third one is the operations. And so this is sometimes thought of as independent of these three, but I really like to call the data model all three of these things, the structures, the constraints to define valid structures, valid instantiations of these structures, and then the operations that these structures support, okay? So let's see some examples. So your structures might be, as we mentioned, your rows and columns, nodes and edges if it's a graphic graph model. Key value pairs has been popular with the NoSQL movement. Just a sequence of bytes, that might be the structures you have if you're just working with bare files. And then the constraints you might imagine are all rows of the same number of columns as I mentioned, all values in one column are the same type. For a hierarchical view, you might have a child cannot have two parents, right? So this would define, you know, for example, one file cannot be in two folders at the same time in your data model of your file system, okay? And then the operations that are supported. Well, maybe you can look up the value given a key X in these key value pair data models. That's one of the primary operations is if I give you the key, you give me back the value. For a tabular data model, you might say, well, find me all the rows where a particular column has a particular value. In this case, you know, a column last name is equal to the value Jordan, okay? And with a file, there's not too many operations you can think about. It's to essentially get the next invites moved to another position within the file and then you can open and close the file. And that's about all the operations that are supported. Okay, so I think in any case you see data, especially on non-volatile storage, you can think about what operations are supported, what constraints there are over the structure and so on. And this gives you an idea of the data model. All right, so what is a database? Well, one definition that I think is pretty adequate is this one, which is a collection of information organized to afford efficient retrieval. All right, so this is a very pretty general definition. It doesn't say anything about tables, it doesn't say anything about relations. So when you think about database, don't necessarily assume relational database. It's perfectly adequate to talk about a database that has nothing to do with relations. All right, but it is not just a pile of data either. It's organized to afford efficient retrieval. So another view of a database is this idea of a schema. And so Jim Gray in this fourth paradigm book that we talked about in the e-science segment a little bit ago has this quote. When people use the word database, fundamentally what they are saying is the data should be self-describing and it should have a schema. That's really all the word database means. And so this goes back to this notion of a data model, right, there's a structure there and in fact, and some constraints and some operations. And all three of these are things you can intuit by looking at the data itself. It needs to be self-describing, right? So if I have a file of data that's organized into rows and columns, somewhere I'm able to inspect which columns it has and how many rows there are and so on, right? I need to be able to understand how to read this data just by looking at the data itself. And so in a database, for example, there'll be a catalog. There will be an explicit schema. So having this idea that the word database is synonymous with self-describing data, data equipped with a schema, that's a pretty common view and it's an important one to keep in mind as we go through this course. So let me give you another view of a database. Motivated by this question, why would I want one in the first place? Why would I want a database? What problem do they solve? Well, there's maybe four issues you might run into that putting your data into some kind of a database broadly defined can help you solve and these are the ones that I like to talk about. So one is sharing data, right? So, you know, once you have multiple users trying to access the same pile of data, some sort of infrastructure or interface to manage concurrent access starts to become required and this is something that all databases, all, I would argue, all real databases afford, okay? Another is, you know, the enforcement of a data model. You might say, well, look, I use a rows and columns data model or I use a hierarchical data model, but there's some sort of software that needs to enforce that. And so this is something that databases can do. Now, remember, the data model is not just the raw structure of parents and children, say, with trees or rows and columns. It's also higher level constraints such as, this must be a number from one to five or this must be a day of the week or it must be one of the customers that already exists in another table. So these kind of constraints are difficult to enforce in the application layer and I'll argue this more in a little bit. Okay, so the third reason might be scale, right? So we know we have a pile of data and we know we have sort of a data model floating around here, but once it gets over a certain size or we get a certain number of instances of this data model, we're gonna wanna use specialized algorithms to be a