Okay, welcome back to the technology room everybody We've been some interesting presentations so far, but we're running a little bit behind so we're going to get racing straight into the next one We've got Cameron Ball from Moodle HQ and he's going to give us a a rundown on the wide-eyed crazy functional programming Which covers a bit of philosophy and the age of parallel computing So any questions put in the forum if we can't get to them now we will get to them But for now it's over to you Cameron Yeah, good day. I'm Cameron from Moodle HQ. I work on the Moodle Cloud team and I'm here presenting What I think maybe the first talk at a Moodle Mood that's actually not about Moodle at all I mean, it's tangentially related. It's about programming I originally wrote this as a personal project and I presented it internally at Moodle HQ and it was so well received that Everyone was like hey, do you want to do it at the Mood? So yeah, why not? Here I am So let's get into it Right. So this first slide is intended to get your brain sort of bit activated. We got a bunch of fun little Facts and anecdotes here Let's go through a couple of them won't go through all of them It was intended to be so like a lobby while people join the chat and There's a good one all the back to the future happened in the past Maybe we'll just go through two or three of these to get the the brain fully fully engaged the cortex fully engorged and Everything is in the future. It's a very profound quote Now we'll go for one more that is a good one functional programming and procedural programming are owed to these two Pre-socratic philosophies. I encourage you to look those guys up because they're pretty cool All right So let's get into it. This is the talk called wide-eyed crazy functional programming And yeah, there's me Cameron middle-cloud developer and of course some fun animations because you cannot spell functional without Fun and I need this slide here just to let the confetti settle because otherwise the next slide chugs even more And so when I first presented this a few people curious about the title and I imagine a bunch of people out there are curious About the title. So I thought I'd come up with a slide to clarify what it means I'm gonna take it. There we go. So this is this is figure 1.1. What most people consider a typical Functional programming advocate bit of an eccentric crazy and they spell these nonsense words like monads applicative funk doors lambda calculus every few seconds and often they don't have very much hair and Of course before we get too far into it. I want to I want to add a bit of framing to the talk My goal here is actually not to teach you functional programming What I want to do is get you curious. I want to get you longing for more and You know every good talk has an obligatory quote and he is mine I think it really helps convey the framework I used when writing this lecture If you wish to build a ship do not divide the men into teams and send them to the forest to cut wood Instead teach them to long for the vast and endless sea. So that's my that's my goal here I want to get you to vast for the long and endless sea of not dysfunctional programming But I'm programming in general. There's there's lots out there to learn and you know Some people don't even bother learning about the other stuff there. So Yeah, and we can't forget attribution because that way everyone knows how cultured you are and how many important things you've read in your life And I actually don't know who said that. I'm not cultured. I'm Australian Really, sorry And with this framing in mind, I'd like to encourage you not to take any of the code examples in here too Seriously, there's not many They're mostly to demonstrate big picture ideas So what is this talk about I'm gonna commit a cardinal sin of teaching and I'm gonna tell you things It's not about It's not about Haskell and it's not about convincing you to stop using whichever language you like And it's not about mathematics When when trying to explain things there's lots of perspectives you can try slice that from and for this this talk I've gone for a very meta perspective Which is perspective itself As I said, it's not a talk about Haskell. It's about viewing programming through different perspectives And it's broken up into three sections. I think I think it's three sections I'm gonna pause for a bit at each section and when I do that, I'll encourage people to write questions in the forum I'm not gonna answer questions at the end of the talk because like I said, we're short for time And it's just better if I can go to the forums and answer them there So yeah, let's let's get into it What is functional programming? There we go profound it is programming with functions Really everything is a function Which does beg the question Come on change the slide. What is a function now? You know, I assume a bunch of you are watching our programmers and they've heard of functions before But functions in the functional programming context are a bit different What a function is is a mapping between two sets that associate every element of the first set to exactly one element of the second set This is different to functions in procedural languages and the the name function is a bit of a misnomer in that context I think they should have been called Procedures or subroutines and that's actually why those languages are part of the procedural programming paradigm a Function a true function is a mapping you can think of it similar to an array in PHP like you know if you have an array in PHP it associates the keys to values So say you have a key and no Jeff and it points to the string. Hello You can have another key Maybe chef that points to the string. Hello. That's fine, but you can't have Jeff point to two values and a pure function is exactly like that and in the same way a PHP array can't do anything except express a mapping a pure function can't do anything express a mapping you can't mutate state And as I just mentioned those are what's called pure functions And there's a good chance that I'll just say functions for the rest of the talk because I forget And this has some very interesting consequences in a functional in a purely functional programming language We have no equivalent constructs for for loops while loops go-tos variables and probably a bunch of other things There's little asterisk on the variables there because you can actually have variables, but it's in the Mathematical sense you can say like x equals 3 and that means they are one in the same You should be able to substitute x for 3 anywhere in the program And you can't change it unlike in a procedural language where you can reassign the value whenever you wish Now this this begs the question Who in their right mind would want to do that? So I'm here trying to convince you functional programming is awesome because it has less features That is true. It does, but it actually doesn't make any difference You can accomplish anything in a functional programming language that you can in a procedural language And yet why the best answer I can give to that question is learning is good And this is where it ties into Moodle. We're all about education. So we should be all on board with learning new things And it's brain plasticity idea Learning learning functional programming for someone who you know spent a lot of time doing procedural style programming It's gonna cause you to it's gonna require you to Wrestle your brain into shapes that it's never taken before which is a really good skill to develop it's gonna leave you with this plasticity to learn a great of a variety of things to encounter in life and It's also gonna make the maths cortex of the brain Very very buff.  Sorry programs written in the functional style are actually easier To communicate to people Then programs written in the procedural style, but of course that relies on the other person speaking the same language as you Right. So this is the section on paradigms I Was gonna pause here for questions, but I don't think anything I've said so far as particularly Controversial maybe it is but I'm gonna soldier on here just for the interest of time So in particular, I want to talk about the two most widely used programming paradigms And they are the two that are the most adults with each other actually So we have the declarative paradigm and the imperative paradigm and inside those we see sub paradigms And there are ones that sit outside of those like for example SQL is declarative language, but it is not functional and assembly is a Imperative Language, but it's not procedural. You can't you kind of procedures if you write assembly Although the vast majority of imperative languages are indeed procedural and then inside the inner boxes We see some examples of languages that are written with that paradigm in mind So for the purpose of the talk, it's probably okay to just think of declarative and functional It's interchangeable and imperative and procedural is interchangeable and there's a very good chance I'll mess that up as I go through the talk. So what's the purpose of this diagram? We're gonna do our first up of that perspective change Some people might have realized that those are both words used to describe human languages and that's relevant Because learning a new paradigm is beneficial in the same way learning a new human languages And so I've rewritten this as if I was a linguist and What those words actually mean so declarative a sentence which expresses a statement of fact something like he runs I like climbing ice is cold Taylor Swift is gonna go on a date with me one day I messaged her on my space a few years ago Just waiting for her to get back Taylor if you're watching my emails at the end of the video and then in imperative We have a sentence which expresses instructions or requests, you know, shut the door.  Don't hit on Taylor Swift and I Imagine you can see the similarities with with Imperative programming there like it's a clear set of instructions to follow And he is just if you remember nothing else This is this is how you remember them declarative is this is this imperative is do this And when you think about Programming as I explained with the functional paradigm. We lose our instructions word instruction words We lose the for while go to variable reassignment all that stuff and That brings us to one of the first big brain moments in the talk which is programming languages are human languages We do not write programs for computers. We write them for other people to read and So, okay Why is this declarative thing good because surely in both cases the human language case and the programming case? We learn something useful We we learn the ability to instruct people to do things like surely that's useful How do you do anything if you can't instruct people or computers? Here is one thing that we gain in the imperative style we gain Sorry, the declarative style we gain a temporality and that is Something being independent of or unaffected by time So a declarative program expresses what it actually does it expresses things in terms of relationships In an imperative paradigm thinking about the program means thinking about changes over time So think about your loops you got your variables in your loops and you've got to keep track of what's going on over time in your head But in a declarative paradigm you just think about relationships You look at something at what something is and how it relates to the other things around it And then you move on with your life and it's a bit of a weird idea But it does actually end up giving you more brain space to focus on solving the actual problem You're not tracking all these things going on all over the shop So why again? There's some reasons you can read them I'll say that some problems are simply very difficult to solve in the imperative way and That's especially true when you're talking about events that might be happening in parallel And may run at different speeds on different executions of the program Something that comes to mind is JavaScript if you ever use JavaScript in the days before promises You would have run into huge amounts of pain making requests to get data or something and you've got you know Variables to try and track which one finished first and this sort of thing you just you get yourself in a mess Really really quick and you can come up against data races just by virtue of the way the program is expressed And there's a little quote there that I really like constraints liberate liberties constraint. That's a fun one to think about and Yeah, at this point. I mentioned a lot of you people are thinking Enough of the philosophy and linguistics mumbo jumbo. Let's let's see something interesting. So here's a Code example. This is no particular programming language. It's one that I just came up with although it does look like Many languages. They're out there And this is a really interesting example because these two snippets of code behave differently depending on which paradigm you interpret them with So in the declarative paradigm this code the X equals X plus one will hang and not complete execution Whereas in the imperative paradigm it runs and And This code will run in the declarative paradigm, but it will hang in the imperative one and We'll explore that a bit more in the next slide So here we are we're gonna just interpret it using English So in the declarative paradigm we say X is the same as itself plus one which it doesn't make sense That's no good. It does does not compute, right? Whereas in the imperative paradigm because it's a set of instructions We say evaluate the thing on the right of the equal sign Then store it in the thing on the left of the equal sign and that's a list of instructions that can be followed and it makes perfect sense This one in the declarative style We read that as the repeat of X is X Appended to the repeat of X which if you think it through that's just an infinite list of X's Of whatever X happens to be in the imperative paradigm We evaluate that as we interpret that as to evaluate the repeat of X Just build wrong First evaluate the repeat of X. So there's obviously a recursion problem there We can it's defined in terms of itself and the instructions can never be followed because they just cascade out forever and Again, I'll just reiterate this programming languages are human languages the parallels. They're really really cool Imperative tells you what to do declarative tells you what things are and Here's some actual code that you can run if you have Haskell installed or something like that. This is Haskell syntax The double plus is the concatenation operator and this is the exact same as the definition I showed before and You read this as Three F's is the first three elements in an infinite list of F's and that's completely fine If you have an infinite list of F's of course, you can just take the first three. It's no problem It's a shame that the function name is take because that feels a bit like an instruction word, but you know, whatever no one's perfect by contrast in the JavaScript version we interpret this as first compute an infinite list of F's and Can't do that right, so That's pretty cool. I think because we can express what things are we can express an infinite list in in a declarative language And none of the declarative Stuff relies on tracking changes anywhere We just need to know what the thing is and how it relates to other things and we move on with our life And we've gained the ability to wield infinity, which is pretty cool That's something that you have to worry about a lot when programming things cascading out of control But now it's not a problem anymore Right, so this is the next section about promises. I'm gonna I'm doing pretty good for time So I'm gonna pause for two minutes if anyone wants to write questions in the forum Well, what I just spoke about is fresh in your mind. Please do so and after the talk I'll go through and I'll answer those while we wait I'll just go back to this guy and we can we can look at some of these little little quotes Yeah, if you can be disgruntled, can you also be gruntled? That's a good one There's more stars in our galaxy than there are atoms in the universe that one blew my mind when I first realized it and Point nine recurring is exactly equal to one a lot of people dislike that but it's true That's a really fun one about English spelling if you look at other words where those letters appear You can see that it produces the word fish And one of the facts is a complete fabrication and it would be very unfortunate if it was this one This is interesting. Yeah, if you sift through pie your credit card details are somewhere there And I'm pretty sure that's actually how scammers get credit card information They just sift through pie looking for valid credit card details. That's how I do it anyway And one seven two nine is the best number. I checked every number and that is the best one I'll probably wait for another 30 seconds and then move on. It's very strange not getting feedback during a talk I don't really know if people are ready to to proceed or or what And I see I actually ate a banana before doing this talk just to get a bit bit more strength buff brain We're back at the start of these. So, okay, I'm gonna continue back where we left off. We're in the section Oh, wait, hang on. No, we're not we're in the section about promises JavaScript promises That seems like a bit of an odd tangent. Why am I talking about JavaScript promises? But let's go with it and I'm gonna break the the cardinal rule of teaching again by telling you what promises are not Promises do not make things asynchronous There's a lot of misleading information out there that seems to indicate that they do You can have perfectly synchronous promises But the thing the thing you pass as a callback till promises not get like run in the background or anything like that So just wanted to get that out the way And this is this is what promises really are They solve these these problems they solve asynchronous request management error handling for computations which may fail and Probably a bunch more stuff that I forgot and so what are they a promise is a value which may or may not be available yet That's where the asynchronous part comes in that value might come in later wrapped in some additional context and Some functions to instantiate the wrapper in this case we've got promise resolve and promise dot reject and A rule for composing the wrapped values, which is the then method Now to have a quick JavaScript recap because I'm going to be using this notation That one at the top the X with the arrow is equivalent to that function definition And what is composition? It's a method for combining two things of the same type into another thing that is also of the same type And just a quick note on that. I'm sure a lot of you've come across function composition There's many different kinds of composition But function composition is very common, especially in programming However, in this example, we're talking about composition of promises in the sense that we combine two promises and get another promise Right. So promise composition The then method of a promise except the function is an argument and this function takes a value and returns a new promise The new promise combines the previous two using the composition rule that you pass into then So I hope it's not too small In this snippet above there are three promises So there's the first one We got just promise dot resolve some value and this is a completely synchronous promise because the values there immediately And then here this thing itself is a promise. It's the composition of the first one and then it uses the composition rule X plus And another value to produce a new promise using the value from the previous one And then the third promise is the composition of all three. So this Function here is using this composition rule with the promise that was composed back here So you end up with a promise with the value some value and another value and another one And this here still results in a promise. It's a slightly different. We have a reject in here And the way that works is any subsequent call to then just leaves you with the rejected promise So we just end up with a failed promise with the value. It's all gone wrong So this is what I mean when I was talking about Computations that may fail.  So, yeah, why did I show that who cares? What we're doing there is we're declaring how we want the promises to compose and we've come up with a system where the order That the promise has actually resolved their values in is no longer important And in other words, the timing aspect is gone and we can declare failures anywhere in the chain So those words are sure starting to sound familiar And this is one of my favorite things. This is uh JavaScript code and this is Haskell code and they look so similar So if I hover over this you can see the the symbols that correlate Um in Haskell we have this thing It's called it's called either it's called either type and instead of promise.resolve They have right and instead of promise.reject they have left but this example using just rights So promise.resolve we start here. That's the same Then this funny little arrow guy here. That's the same as then And this is just the Haskell way of making like an inline function promise.resolve And then we're doing string incatenation there and I mean just they look so similar you'd think one copied the other But it's not true. They it's an independent Thing that emerged in both languages Um And here's the the failure case. We've just chucked a left in there So in the top one we get a failed promise with onos and in the Haskell one we get a left with onos Um, so if you've ever used javascript promises You've been doing declarative style programming all along without even realizing it And yeah, we get some new superpowers here. We can write a temporal code Which explains why promises emerge in javascript It's a place where people are constantly dealing with requests that may finish in any order And so I think it's so cool That people using a language so far removed from Haskell ended up solving the problem in pretty much the exact same way And like I said, they're independent discoveries. There's no no plagiarism going on here But I don't think it's a coincidence that they appeared in javascript given the nature of of the language and the ecosystem Right, uh, this is the section on composition and I'm doing pretty well for time So, uh, if you've got questions about about that Feel free to go chuck them in in the forum and I guess we'll just go back to the little little dancing guy again I think we've seen all the facts, but whatever you can watch them again. Try find the fake one I should have bought a rubik's cube or something. I could like solve the rubik's cube while While waiting at least they would have been entertaining This one's cool 23 people in a room to ensure a 50% chance of a shared birthday Tim has put something in the chat. Um, he said it's much easier not to have bugs when your functions don't have side effects And yes, that's very true. Entire classes of bugs simply cannot exist In a purely functional programming language Give it another 30 seconds or so That's a fun one if you're into languages. Preservative means something different in many european languages For the european viewers All right, let's kick on Right, this is the section on composition I mentioned a bit in the previous section, but this is sort of composition in a more abstract philosophical sense So here's a quick explanation of what composition is Um, again, very abstract so say I have three things they can be anything but in this example I'm using coins and so you have some arrows in this case I'm using pipes coins and pipes because video games So if if you can go from there to there from a to b And if you can go from b to c Then the composition of those two things if it exists, which it's not guaranteed to goes from a to b Pretty straightforward I think like you can see from the diagram like obviously if this is a system where composition is possible Because you just go in that pipe in the other pipe and you're at c So yeah, of course if we can pose those things together we get a route from a to c Right who cares what's what's so great about composition What's great about it is that big problems can often be chopped up into smaller problems that are easier to solve And if we can be clever about how we like dice up the problems Um, we can come up with solutions to the individual pieces such that we can put the Solutions together and form a full solution And then this is the cool part you can infinitely speed up the work by simply adding more workers you get them to work in parallel, right? So i'm just going to go back to that diagram So yeah, maybe we want to go from a to c maybe that's our goal to go from a to c But that's like really hard for some reason But maybe someone out there worked at how to go from a to b and someone else worked at how to go from b to c Well, if we're working inside of a system where composition is possible, then we just solve the problem you just compose the two solutions together and you're done And uh, this is I think the natural way humans solve problems We love to you know divide problems up in our heads and then figure out the little bits and put it all back together Okay, here's some examples throughout history of where composition was employed The pyramids, you know, I had lots of people working in parallel To to move those blocks. It wouldn't make any sense to just get one guy to move every single block The large hadron collider is a great one I don't think there's a single person in the world who fully understands how that thing works That was a huge collaborative effort McDonald's Um, I think McDonald's was the first place to have staff make individual parts of a burger And then at the end compose the burger together and that way they could produce way more burgers instead of just having one One guy at the grill making the burgers and uh Moodle does it as well, uh, when we when we do projects. So let's have a think about how Moodle tackles projects um, so Do we get one mega awesome 10 times developer to bash out all the features a little bit like this? Uh, no, I don't know about you, but I wouldn't be keen to work with someone with that sort of attitude So the correct way to write software is of course with your friends as a team So this is why it's worth Spending the time to plan how to chop up the work And we need to make sure we're chopping the thing up into pieces that can be put together later And this is why uh at Moodle we spend all this time planning how we're going to do the work for the releases And um, yeah, this is a takeaway. I I believe composition is nature's greatest hack Uh, I wanted to try and convince you that it's fundamental to the way humans think We're amazing for our ability to do this. Uh, I don't know if there are other animals that break up problems this way And then solve them collaboratively. I think we're the only ones that do it As at its core the composition is about relationships and building complexity by Combining the relationships and hey that sounds familiar again, doesn't it that whole declarative thing that was about relationships So this brings me to One more near the end my final point more or less So when we write programs as an individual in this case, um, we naturally want to break up the problem That's why we have functions in programming in the first place. We don't write, you know, like one big program That's like a novel essentially We want to break the problem up into building blocks that we can put together and mix and match And the building blocks who use matter some compose better than others Um now because of all the restrictions the functional paradigm places on you The lack of mutation the lack of control flow functions in a pure language Naturally, uh, well, they're not well suited. They are they guaranteed to be composable. So I'm just going to go back to that composition slide so Say we have some function that goes from a to b and it has a side effect That means when you call b to c with the result from a to b That is not necessarily the same as calling the composition of those two functions a to c The whole point of the composition is that if you call a to b then b to c That the result should be the same as Calling the composition of them But if the if the database is mutated or something like that in one of the calls those two subsequent calls to the composition Or the one after the other version will have different results The point of composition is that they should be the same and in purely functional language They will be they're guaranteed to be there's no way you can mess it up Um, and I'll I want to make a comment that I think is my opinion I don't know the history of these things, but I'm I'm pretty sure procedures in imperative what we call functions. Um In like php for example are a short sighted attempt at composition It's like people wanted a way to have these building blocks They wanted a way to nicely put things together, but It doesn't quite work unless you're very disciplined and you try to keep the side effects Out of your functions, but as I said in pure language, you can't have the side effects You just cannot have them. It's guaranteed that you have composable building blocks And yeah final little box there. This is becoming super relevant in the age of parallel computing Um, which I'll touch on in a second Um, here's a little bit about why functional program is awesome The thing that I wanted to convince you of to begin with So by taking away these convenient but dangerous tools like the ability to just mutate state to modify the database, whatever Um, we're left with tools that are incredibly powerful But more difficult to wield In particular, we've got a temporality. We can express things independent of time We've got infinity. We we've managed to wrestle infinity And we've got composition the fundamental way human solve problems is now just available at all times. You don't have to worry And here's a couple of final big brain thoughts At Moodle we work across multiple time zones So it's you know, the idea of being able to solve problems in a time independent way seems a bit appealing Um, and Moore's law this one is particularly interesting for those that don't know that's the observation that the number of transistors in cpu's is doubling every year, but that's finishing Now it's um, it's coming to an end. We can't physically cram more Transistors in there And we're now seeing a shift to parallel computing because we you know, you want more you want to double the power You just add another box. I mean imperative paradigms Are not the right the right tool for programs that need to run across multiple computers. They're just not So either I think we'll see a shift to declarative style programming soon But that you know functional program has been saying that since the dawn of time Or we're going to shoot ourselves in the foot and make it hard real quick real fast And I hope we see the the paradigm shift because it's the definition of madness really, isn't it? Like if you do the same thing over and over again, but expect different results And another analogy that comes to mind when I think about these things is like the the frog in boiling water Well, he's not in boiling water. He's in cold water But the water temperature slowly rises so he doesn't notice and then eventually he's just cooked Where we're getting to boiling point. We got it. We got to jump out of that pot soon And here's a really fun one quantum mechanics might be so confusing because at a certain point The universe doesn't want to be chopped up anymore. Like, you know, we kept splitting things We split the atom and then we got like electrons and we started looking into those and then After a little while physicists were like, oh This stuff's starting to behave real weird. I don't like it And maybe it's the case that we just can't understand that our brains want things to be chopped up into small pieces But Does the universe have any obligation to actually be that way? I don't think so And um, yeah, that's that's it. Thank you so much for watching my talk and letting me share my passion for programming with you Um, if nothing else, I hope you're a little bit more curious about functional programming And even though it's not super relevant To Moodle or, you know, probably the kind of code that that you people write on a daily basis I really wouldn't encourage you to investigate it because learning more about it will help you be a better programmer If you have questions, please email me. I'm always always happy to talk about this and I would suggest going to the pub for a drink, but Well, uh, yeah, kovat 19 has other ideas about that And if you have ideas about how I can improve this presentation, as I said, it's a personal project Please get in touch. I I'll be very happy to To take on any any constructive criticism So, yeah, I'm gonna mute my mic now. Otherwise we'll get echoed from chris But thank you very much. I'll be on the forums answering answering your questions Well cam, what can I say? That was very fascinating and even a slightly bit entertaining as well So thanks for that. I learned a lot about how to make a hamburger and I also Found out how many credit cards you might find in a pie Um, I forgot to tell you that I did see Taylor Swift last night. She said to pass on your her regards. So regards Rado. All right, everyone. We've got a bit of a break now before the next one. So Head down to the cafe and have a coffee or a surveyser or whatever tickles your fancy And we'll see you back in the tech room At about 20 or 40 minutes. Thanks a lot


find for data analysis and help by applying to my controller. OK, so I'm interested in data analysis, which, to me, is the process where raw data comes in one end and understanding insight and knowledge come out the other. And to me, there really are three main sets of tools for doing data analysis. So before you can do data analysis, first of all, you have to get your data. And you need to get it into what I call a tidy form, a form that works well with the software, with the tools that you're using. Now, I have a little arrow here in any real data analysis. Often that arrow comes all the way around over here. Often the most challenging part of a data analysis is just getting it into a form that's easy to analyze. And once you've done that, there are three main sets of tools for doing data analysis, data transformation, where you are computing new variables as functions of existing variables, where you're doing basic summaries, filtering, all that kind of stuff. And then there are two main engines for gaining insight into data visualization and modeling. So visualizations are great because a visualization can surprise you. A visualization is also really helpful for refining your questions of the data. And often the first challenge of a data analysis is taking those vague, nebulous questions and making them sufficiently precise that you can answer them in a quantitative way. The problem with visualizations, however, is that they're fundamentally a human activity. Human eyes have to look at every single visualization. And that means visualizations fundamentally don't scale up. So to me, the complementary tool to visualizations are models. Think of this as statistical models, machine learning, data mining, whatever. But whenever you can make a question sufficiently precise that you can answer it with a handful of summary statistics or a simple algorithm or complicated algorithm even, I think of this as a model. So models are great because they don't need a human. They use a computer. And so they inherently scale much, much better than visualizations because it is much, much easier and cheaper to buy computers than it is to buy human brains. The problem with models, however, is that every model makes fundamental assumptions. And a model by its very nature cannot question those assumptions. And that means in some fundamental sense, a model cannot surprise you. A model can only tell you things that, in some sense, you already expect. So in any real analysis, you're going to have to use all of these tools multiple times. You might start with a visualization that will suggest a model. You fit that model, you transform the data to look at the residuals. You then visualize that again. Then now that you've removed the most powerful trend, you can see the subtler trends that remain. So in any real data analysis, you'll iterate through these tools many, many, many times. And a lot of my work developing R packages has been building tools to make these things easier. And kind of the first round of my work was reshape 2 for tidying data, plier for transforming data, and ggplot 2 for visualizing data. And what I want to talk about today is kind of the next generation of those tools. Tidy R for tidying data, dplyr for transforming data, and ggvis for visualizing data. And there are really three things that have had a profound impact in the way that I developed R packages. So the first one is RCBP. So RCBP is a tool that makes it very, very easy to write C++ code and call that from R. And this is really important because R is a language that is fundamentally designed to make humans effective. It is not designed to make computers efficient. So that means R is very expressive. You can do whatever you can imagine an R, but it is generally not the fastest of programming languages. C++ is fairly complementary to that. C++ is designed to have very, very high level performance. And any abstraction that's been added to C++ has never been added at the expense of sacrificing performance. So C++, it tends to take you much, much more time to write C++, sort of express an idea in C++ that would an R. But the end result is code that runs much, much more quickly. So this makes it possible for my packages that can now use C++ to expand to much larger data sets. As Kartik mentioned, another thing I've been working on is this advanced R program, this advanced R book. And this book has been great because before I wrote it, I thought I knew what advanced R was, but I didn't. And through the process of actually writing that book, it forced me to confront many of my misconceptions about the R language. It forced me to learn the areas of the language in which I didn't really understand. And one of those areas is this computing on the language or meta-programming, where you're using code to inspect and modify other code. And it turns out that is really important in R for creating fluent interfaces. Interfaces for doing interactive data analysis where the tools just recede into the background and let you focus on the challenges of the specific data that you're working on. The third thing that's had a profound impact and the thing I wanna focus on today is this kind of strange looking thing. This is called the pipe. This uses R's standard way of denoting an infix function, which is to surround it with percent signs. This is implemented in the Magrida package. And the idea of a pipe is very, very common in other places. If you've ever used the shell, pipes are really, really important for taking advantage of the Linux shell. F-sharp has a very similar idea. Clojure has similar macros that do similar things. Haskell has a similar thing in the form of Monads and their Arrow operator in Smalltalk and JavaScript and C++ have this idea of method change. It's a very, very old idea, but now you can access this idea very, very easily in R and it's hopefully I'm gonna persuade you. It allows you to make your code much more readable. And from my perspective, as well as making it easier for the user of the code, it also makes it easier for the author of the code. So today I'm gonna talk about pipe points. So let's start by, just I wanted to define a little bit what this operator does. So the rules are very simple. Basically, it takes a thing on the left-hand side and inserts it as the first argument on the right-hand side. And the way you pronounce this is then, so X, then F of Y is just the same as saying F of X and Y. Or if you don't want it to go into the first argument, you can use the dot as a placeholder. So this is gonna insert the thing on the left-hand side into the dot on the right-hand side. And we can join these things together. So X, then F of Y, then G of Z, G of Z, you should say, really, is equivalent to this sequence of function compositions. So this is really important because it allows us to return function composition. And function composition is a really solid, well-understood means of combining things. And it allows us to turn it into a sequence of operations, a sequence of things that look imperative so they're easier to understand. Now I wanted to motivate this with an example, which I am told should have cultural relevance to many of you, although it has none to me. We're gonna start with Fufu, which is a little bunny. And if you're gonna write what Fufu is doing in the terms of a sequence of functions, you're gonna write it like this, right? You're gonna start with the thing that happens first is the most internal function call. So if you wanna read the sequence of events, hop through, scoop up, pop on, then you need to read from inside out. You have to read from right to left. And that's challenging. And the other thing that's challenging is what are the two arguments to bop on? Well, it's the sequence of operations and this other object here. Now if we rewrite that with a pipe, this is much, much easier to see what's happening. We've turned from a sequence of function compositions into a thing that looks very imperative. So we hop through the forest, we scoop up the field, mouse, and we bop on the head. Now this is kind of surprisingly, I think, it's had a profound impact on the way that I and many other people write data analysis code because it allows you to write code that's much easier to read after the fact. And I think writing code that is easy to read is really, really important because every single project that you do involves collaboration between at least two people.  And future you will be very, very grateful to past you if you've written code that future you can understand. Now I find it seems like the older I get, the more future me and present me remember totally different things and understand totally different things. And it's really important to have a right code that I can actually understand in the future. And over time, in the past I've had these moments where I write some code and I think, wow, I am so smart and awesome. This is an incredibly awesome bit of code.  So the thing that's neat about this pipeline operator is you can apply it whenever you would normally do function composition. So it means you can use it with functions that were never written. You can use it with packages that were never written with this idea in mind. But it does work best when you can think about a pipeline of operations which means that the first argument to a function is typically the same type of thing as its output. So you're thinking about taking an object and transforming it repeatedly. And so today I'm gonna talk about three packages, this TIDER, the goal of a TIDER pipeline is to take a messy dataset and transform it through a sequence of steps into a TIDER dataset. Once you've got that TIDER dataset, you can use dplyr to manipulate it and ggvis to visualize it. A couple of other packages I've been working on recently is Arvis. That's a package that makes it very easy to express operations on HTML. So if you're doing web scraping, for example, Arvis makes it easy to express a sequence of transformations, a sequence of selections that gets you to exactly the piece that you're interested in from a web page. And similarly, lowliner is kind of like a pipeline to a list. So I wanna talk about three packages today. So TIDER, dplyr and ggvis. TIDER is the goal of this is to make this first step, this initial step where you take the data in whatever crazy format your collaborator has saved it in. And again, that collaborator might've been past you. You have no idea what they were thinking at the time. So the goal of TIDER is to take your messy data and put it into a tidy format. And the structure of a tidy format is actually really simple.  How many of you have heard of COD's third normal form before? So COD's third normal form is a, at some point I took a database class and I think I understood what it meant, but when I now look at the definition of it, I have absolutely no idea. But this is basically COD's third normal form. And this is really important for creating databases because it helps you, it helps ensure that you represent one fact in only one place. Because you never wanna represent one fact in multiple places because as soon as you do that, you now have the possibility that those facts are gonna record different values. And what do you do when you have inconsistencies? So the goal of TIDER data is to frame this really powerful idea and important idea from databases in a way that makes sense to people working with data. And this idea is very, very, very simple. And I'm not gonna kind of formally define what an observation is or what a variable is. It turns out that's pretty hard to do in practice, but you guys already know what those things are and your internal definitions are good. So I'm gonna illustrate this with a little example. So this data is some data on tuberculosis cases collected by the World Health Organization. And it's about what I'm gonna tell you about this data. Instead, I am going to give you a little challenge. What I want you to do is turn to your neighbor and I want you to brainstorm for a minute and see if you can figure out what are the variables in this data set. So I'm gonna give you a little hint. If is female, you is unknown and 1524 is shorthand for 15-24. So you've got one minute starting now. What are the variables in this data set? Okay, time's up. So does anyone wanna suggest one variable that's present in this data set? Country code, that's good spotting. So you might have recognized or realized, but this is the ISO2 letter country code and AD, I believe stands for Andorra. What other variables do we have here? There's another easy one, yeah, right? That's also in a column already. Is M04 a variable? No, right? M04 represents probably two variables, right? It's a value of two variables, one which is the sex, male or female, and the other is the age. So we have the year, we have the country, we have the sex, and we have the age. Are there any other variables in this data set? What do you think this represents? What does that number one represent? Probably the number of cases, right? So we have another variable that is the number of cases. So in this data set, we have five variables, two of them are represented in columns, two of them are kind of tangled up in the row names, and one of them is inside the cells. So this data in this form is hard to work with because you don't have a consistent way of referring to a variable because they're all muddled up in these different ways. So I'm gonna show you a little demo of how you might go about fixing that with the tidy R package. So one thing you'll notice here is that's a little bit different. Once I've read the data in, I'm just going to, if you've seen Arco before, I'm just gonna put this in a class called tableDF. Now all that does is it changes the behavior because the default behavior of R is when you print a large data set, it prints the first 10,000 rows, which is almost, I'm gonna go out on a limb and say that it is never useful. So what this does is just prints the first 10 rows, it fits all the columns that it can print on screen and it tells you how much data you have in total. Now if we're gonna convert messy data to tidy data, it turns out for any possible, pretty much any possible form of messiness, there's only really two operations that we need. The first one is a gather. So we need to gather all of the columns that are not currently variables and put them into two new columns, which I'm going to call demo short for demographic and end the number of cases. So if I do that, now I have four columns. ISO2 and year are variables, demo, that's kind of a variable. It's not actually one variable, it's two variables, right? We've still got age and gender muddled up in there and then in again is another variable. So we've taken all of those columns that weren't variables and we've turned them into key value pairs and chained them down the data set. So if we want to split that apart, there's a handy function called separate, which is not very exciting. All it's gonna do is take the first character and make that a six variable and take all the other characters and make that age. And now we have what I would call a tidy data set because each column is a variable. And again, this is important because when you model things, you're interested in the relationships between variables. When you plot things, you want to map the variables and the data to things that you can perceive. And now you have a standard way of doing that because variables are represented in the site, all the variables are represented in the same one. Now you might also do a few more things with this data if you're gonna make it a little bit easier to work with as a person. You might change ISO2 to country and you might arrange it. We'll talk about those operations a little bit later. Or we could actually do this all in one pipeline, right? We could say start with this name of the file, then load it, then turn it into a table DF, then gather it, then separate it, then arrange it, then, well, this should actually be renamed. Right, so we can do that whole sequence of operations in one pipeline should we want to, although generally you don't want to do very long pipelines because you want to check at each step, have you done the right thing? So TidyR provides a few other verbs that I'm not gonna talk about in general, but it should, for the vast majority of messy and strange data sets, it should allow you very quickly to get it into the right form. And the reason it is so, I think it is useful is because, first of all, it gives you these programming tools, but more importantly, it gives you these cognitive tools to think about how should you arrange your data and my advice for that is very, very simple. You always put the variables in the columns and you always, and then the rows form observations. And if you can follow that advice, your data is almost always gonna be easier to work with and are, or in any other programming language for that matter. If you'd like to learn more about this, you can Google for the package name. There's a package on CRAN, there's vignettes, there's also a paper called TidyData, which expands on this idea of TidyData in much more depth and talks about some of the other common cases of messiness. And there's also now a pretty cool cheat sheet, which just summarizes all the most important things. So TidyR helps you get your messy data into a tidy form. Often the first thing you're gonna do with that is to do some basic data manipulation.  And that's the goal of the de-plyer packages to make that as easy as possible. And it does that in sort of three ways. So first of all, when you're doing, for any operation you're doing in a data analysis, first of all, you have to think about what you want to do. What data manipulation do you wanna do to solve the problem that you have? Then you have to describe it precisely, and describe it precisely in such a way that a computer can understand it. In other words, you have to program it. And then finally, the computer has to go away and do it. So when you're thinking about, when I'm thinking about how can I make data manipulation easier, I'm kind of thinking about all these three things. How can I make it easier for you to think about data manipulation? How can I constrain the scope of the problem to give you some useful tools that solve the majority of problems? How can I make it easier for you to go from what's in your head into the computer? And then finally, how can we make the computer do it as quickly as possible? And so these are kind of the three goals of Deploy that make all three of these areas easier. And the way that Deploy makes it easier to think about data analysis is by kind of telling you there are five important verbs for working with a single table of data. So Select allows you to pick out the variables by their names. Filter allows you to pick observations based on some criteria to do with their values. Mutate allows you to add new variables based on existing variables. Summarize that you reduce down multiple values to a single value. And arrange that you reorder the rows. So my claim is that with these five verbs, plus the group by operator to be able to do this by group, this allows you to solve the vast majority of data manipulation operations. And that's actually really important because now when you go to do a data manipulation, you can now think, well, instead of the thousands of possible functions I have available to me, which one should I use? Now you can just say, well, which of these five should I use? That is assuming that you believe me that these are the right five. So I'm just gonna show those five verbs off. And the goal, one of the other goals here is to map functions to verbs. So you wanna be able to think about, you wanna be able to program the data manipulation in a very similar way to the way you think about it. So I'm gonna work with a slightly different data set. This is an R package called baby names, which contains basically the complete set of how many babies were given each name from each year from 1880 on. So again, this is a table DF object. And again, when you print it, you don't get all 1.7 million observations. Printed out, you just get the first 10, so you get some sense of what's going on there. So we have the year of the name, the sex, the name, how many children were given that name, and then the proportion of total number of birth. So this is telling us in 1880, there were 7,065 girls called Mary, and that's about 7% of all births. Well, this is actually not based on birth data, but based on social security data. So it's a little bit more complicated than that, but you can imagine it as births pretty easily.  So for example, we can say select all of the variables apart from proportion. We can treat variable names. Here I'm treating a variable name like it's a number. So I'm saying give me everything minus this variable, except this variable. We could pick out all of the variables between year and end. Now, in this data set, which is five variables, you don't really care about these operations. If you're working with a data set, it's not uncommon, particularly if you're working with government survey data, you might have a data set that has 800 variables in it. And the first part of any data analysis challenge is just to figure out which of those 800 variables do I need to answer the question at hand, and how can I pull them out as easily as possible? We can use filter to pick out rows that match some observations. So for example, I'm gonna filter out just to find all of the babies called Hadley, all the records about Hadley's. So you can see there's 155 records from 96 on. Or we could apply multiple conditions, give me the names of all of the, give me all of the records for 2013 for males. Right, so every single time I do this, I'm creating a new data frame. Now, when I select, if I just take a subset of the variables, because of the way the data frames work and are, I don't need to make a copy of those columns, I just kind of point to them where they were originally stored in the first data frame. When I filter them, I have to necessarily create a new data frame that just has those rows. Now, similarly, when I create a new variable with mutate, so for example, here I am just gonna do a simple transformation where I figure out the first letter of the name and the last letter of the name. This is also gonna create a new data frame. Again, it doesn't take up that much memory. The only memory it needs to allocate is for these new two new columns because these ones already exist in the old data frame. So mutate allows us to add new variables that are functions of existing variables. And then the last, well, we can use summarize, which is not that useful here, we can just use summarize to count. That's gonna give us a one row data frame here which shows in total how many baby names we have here. So that is 7333 million names in this database. Or we can finally, we can change the order of those rows. So those are just the basic verbs. Now the goal of these verbs is that I've pretty much told you almost everything you need to know about these things. Each of them is really simple to understand in isolation. And if you want to deal with a more complex problem, the way you deal with that is by combining simple operations together. So for example, you can combine any of these with a group by operation. So I'm gonna say take the baby names, group it by name and then summarize it. So now I'm using the same summarize code, but now because I have grouped it by name, I'm now gonna get the count of names, the count of babies with their name for each name. So you can see there are 56 R-barns and this is not the very many names start with AA. Or what we could do is we could take the baby names, we could group it by year and six. We could add a new variable called the rank which is just the min rank of the in and descending order. So basically what we're doing is we're creating a new variable that is gonna be one for the most popular name, two for the second most popular name, three for the third most popular and so on. And we're gonna do this inside the year and then finally just inside each combination of six and year and then just finally just to show you what that looks like. I'm just gonna look at the tail of the data frame, the first, the last six rows. And you can see so for example, in 2013, Zairie was an unpopular boy's name with only five names that was ranked, was tied with many other names at 2000, at 12,000 and 32. Now this data set for privacy reasons only contains names with at least five occurrences which is why N is no smaller than five. Yep. Oh, so when you group a data frame, it looks exactly the same. All that is done is it's built up sort of this index internally that says any other operation that you applied to this from now on is gonna be done by group. So if we wanna operate on some more complicated questions, we just chain these simple pieces together. So if we wanna see the most popular names of all time, we start with baby names, we group it up by name. Like grouping is a fundamentally statistical operation. Right, we're saying this is our unit of interest. Our unit of interest is the name for each name, we're gonna find out the total number of babies with that name, and then finally, we're gonna arrange it in descending order so we can see the most popular ones first. And you can see that for example, the most popular name of all time is James with over five million James's, five million John's, a bit under five million Roberts, Michaels, Mary's and so on. Or we could say how many Hadleys are there, which I'm interested in, because this is a very annoying trend. So we're gonna start with the baby names, we're gonna pick out all the Hadleys, we're gonna group it by sex, and then we're gonna count them. So you can see there are 15,000 female Hadleys and 70,000 male Hadleys, and we're gonna explore in a bit more detail how that's happening over time. So I'm doing the same thing, filtering it by Hadley. Now I'm breaking it down by year and sex. I wanna see the time course. Then I'm gonna summarize it again to count it. I'm gonna use a little trick from Tidia. I'm actually gonna make this data kind of untidy. I want one column for the number of men and one column for the number of females. I can use a function from that and then I'm gonna function from Tidia from that and then I'm gonna view the results which shows in this HTML table thing. So you can see early on Hadley was a, well it was never exactly a popular name, but there are only boys and no girls. And then you'll see we scroll down in time. In the 1960s we start to see it rise in the 70s of the popular for a girl's name and then recently it's been skyrocketing in popularity as a girl's name, which is very, very sad. Or we could do something like this. Again, break it down by year and sex. Add this ranking and now I wanna find all of the rows with a rank as one. So all of the most popular name for each sex in each year. Now we could very easily do the opposite, right? We could say for each name in what year was it the most popular? If we're gonna do that, we just group it by name now and we can see the year in which it was most popular. So because we can deal with complexity by combining these simple operations together, this makes it much easier to learn because you understand how each of these things works by itself to solve a more complicated problem. You just need to think about what's the sequence of these operations I need to apply. Now of course when you work with data you're not just typically just working with a single data frame or a single table of data. You're often working with multiple tables of data. So D-Plyer also comes with three families of methods for working with multiple tables. The first are what I call the mutating joins. So this is like a standard SQL join where the result is a new data frame that primarily has more columns on it. So if you want the intersection, you wanna see only the rows that match in both X and Y, you do an inner join. If you want all of the rows in X and the matching ones in all of the rows in Y and the matching ones in X, you do a left join, the complement has the right join and the full join goes to everything. There's also what I call the filtering joins. There's a semi join and anti join. These are kind of like these joins which you're probably familiar with already but these do not add any columns. All they use for is filtering. So semi join says give me all of the rows in this table that have a match in this other table. This turns out to be a really, really useful operation. The anti joins is the opposite. It says give me all of the rows in this table that did not have a match in this table. These are very, very useful when you're diagnosing what's gone wrong with your initial joins, why half of your data has now gone missing so you wanna find out why there isn't a match. Yep, now there's an option, you can join. So the way that Dplyr does these joins is by default. It does what's called a natural join. It assumes the variables you wanna join on have the same name in both tables, which is pretty common. You can override that if it's not the case for your data. And then finally, there's a sort of set or row based operations. So now these joins kind of think about tables side by side. These set operations think about tables on top of each other. Do you want all of the rows in common between the two? Do you want the columns that are in one but not the other? Do you want the rows that are in one but not the other? Or do you want all of the observations regardless of which table they're in? So really useful set of operations kind of cribbed from SQL, large VN house. Now one of the other things that's neat about Dplyr is that it is built to abstract away the details of how the data is stored. And that means you can work with local data frames, data tables, there's some experimental support for data cubes, but more interestingly, you can also have remote data sources. So one of the big bottlenecks when you're working with large data is that large data is expensive to move around. You don't want to move the data to the computation, you want to send the computation to the data. So what Dplyr allows you to do is work with tables and remote databases as if they are in your memory and your current computer and it does that by translating your R expressions into SQL and sending that query to the database. So there's a number of pluggable backends for that. So I think that's important because it abstracts away. A table of data is a table, no matter where it lives and you should be able to work with it in the same way without having to worry about the specific details of how it's stored. So a data cube is, so for example, when you think about a data frame or a table, normally sort of think about a sequence of columns, each column is the same type. A data cube, you're thinking about an array of data and memory where you have something that's basically all the values are the same type and it's wrapped around very densely in memory. There's some, so this tends to happen a lot. You tend to get data cubes, like if you imagine you have a grid of points, you have measured them at a set of time points, now you have sort of a three-dimensional data structure and now you have some kind of other covariate, maybe you've got a four-dimensional data structure. So you can, so if every combination of all of your variables is present, you can store it more efficiently and sometimes compute upon it more efficiently in a cube rather than thinking about a table. If you'd like to learn more about Deeply, you can Google for it, there's a mailing list, there's a GitHub page, there's a CRAN package, there's lots of vignettes to explain in more detail, all of the things I've skimmed over here. We end the cheat sheet now. I wanted to finish off by talking about GGViz, which is kind of the final piece of the puzzle, the visualization piece. And the GGViz is much like GGplot2, it's a grammar of graphics. It's not a typology of graphics where it lists all the name, it gives you a list of named graphics and allows you to choose between them. Instead, it gives you a set of small components that you can recombine to create whatever type of visualization you need. Now compared to GGplot2, GGViz is reactive. That means it is not creating a static plot, but it's a plot that changes over time, either as the data changes or you interact with it in some way. GGViz is also a pipeline to the way you create a complex visualization as by joining together many simple pieces. And finally, GGViz is fundamentally web graphics. It's of the web, it's not just something you put on the web, it's built with HTML and CSS and JavaScript and uses a JavaScript library called Vega to do the actual rendering. So I'm gonna show you a little demonstration of this, of what you can do with GGViz. Again, another dataset, this is a little smaller. This is data from the DEA on cocaine seizures. So I'm just gonna do a little scatter plot here. I'm putting weight on the x-axis and price on the y-axis. And when I first looked at this data, it took me a while to figure out what the units of the weight was because there was one measurement that had a weight of about 100 million. So I thought that was implausible, it was implausible that this would be measured in grams, but it turns out when I turned that many grams into pounds and googled for that many pounds plus cocaine, there was actually a seizure, which was an entire container ship full of cocaine. So here I'm just looking at a sub-seal of the data, you can see about the biggest seizure is about half a kilo. And what I've done here, the way I create a visualization is I take a dataset, I pipe it into this visualization function and I say, what do I wanna put on the x-axis? How do I wanna map my data to something I can perceive? What determines horizontal position? What determines vertical position? And here I haven't said how to draw it, so ggplot2 is, ggvis is intelligent enough to guess that if I have a continuous x and a continuous y, probably using points is a good way to do it. I could be more explicit about that as below and I could explicitly say, take that data, pipe it into a visualization, then add on a layer of points. Now we can do a similar thing if we just use a single continuous variable, then ggvis guesses that we might want a histogram and whenever you use a histogram, you always wanna modify the bin width. So here is a bin width broken down into five gram bins and we can see that there's hardly any data, more than 250 grams, so for the rest of this exploration, let's just focus on that. So I'm just gonna filter this down. This is a very, very common pattern, right? You do some visualization and now you need to do some data manipulation. So I'm just gonna plot that data now. And one thing that I kind of thought that maybe I saw when looking at this positive, it looks like maybe it's not quite a linear relationship, maybe it kind of curves off a little bit, maybe there's some sort of like bulk discount, so that as the weight of the cocaine increases, it gets like slightly relatively cheaper because you're buying in bulk. So we can make that easier to see first by layering on a layout of smooths. So I'm just gonna put a smooth curve through this data and then finally I could layer on some model predictions. So I'm just gonna fit a linear model to that data set and so I can have a nice reference. What's a straight line fit through this data versus a curved line and it looks like maybe there's a little bit of a difference out here, but of course there's not a lot of data out here. So it would probably be a little bit suspicious of this. We could look at another pair of variables instead. So weight versus potency. So potency is basically the percentage cocaine. So zero is there's no cocaine, it's all filler. 100% is all cocaine, no filler. So we've got a bit of a problem here that we've got a lot of over plotting. So it's kind of hard to see the trend. So again, I'm gonna layer on a smooth curve to that plot. And you see something here that I think that I personally thought was rather interesting and that is we see the sharp uptick in potency, right? It seems like smaller weights have higher potency and that's counterinsuative to me because I would have thought that like out here this, I would have thought this, I just kind of think of this is like the wholesale end and this is the retail end, right? And you'd expect wholesale to be relatively better and retail to be relatively more cut down. So I'm a little bit suspicious about this. So maybe it's just some peculiarity of the smooth that I've used or maybe there's something else hiding in here with all of this over plotting. So one thing you can do in GigiVis that you definitely cannot do in GigiPot 2 is instead of just messing around with the opacity by changing a value or changing the span of that smoother, what I can say is map the opacity to a slider, map the span of a smoother to a slider. So I'm gonna create something that's now has a little bit of interactivity. So I now get a slider and as I drag that slider, the plot updates. This is what I mean by GigiVis plots being fundamentally reactive. As the parameters change, so too does the plot. And similarly, as I change the span of that smoother, well, this isn't actually a very interesting, there you go. As I change the span of that smoother, I can maybe convince myself that this is a real pattern. It's not just an idiosyncrasy by chance that one parameter value happened to show this signal. So that's a fairly simple form of interactivity. I wanna show two other examples of somewhat more sophisticated reactivity. The first one is a grand tour. I'm just gonna pop out here. So the intuition behind the grand tour is very obvious. When you have a three-dimensional object to see what shape it is, you look at it from different angles. So why not do the same thing for a six-dimensional object? And it turns out the mathematics for doing that is basically the same in three dimensions as in six. It's just doing a little bit of matrix multiplication. So here I have a grand tour. So what happens is happening in R. I'm generating a random projection. I'm projecting the data and then I'm displaying it and I can control it as it's being piped to GigiVis here. And you can see here, I'm not gonna explain it in a huge amount of detail, but just to kind of skim by the code, you can see I'm using this package called tourR, which allows us to create grand tour objects. I'm setting up a few parameters. And this is all of the code that I need to create a custom interactive graphic. And what I'm doing, really what I'm doing here is now I'm creating this reactive object. So I'm creating a data frame that is not constant in time, but varies over time. It's implicitly a function of time. And every, and then I feed that into GigiVis. And GigiVis knows what to do when you have a data frame that changes over time. It updates itself as new data comes in. So I'm also illustrating here how you can embed these things in R Markdown documents. R Markdown is a nice way of combining code and narrative. So I write this in R Markdown. I can include some text, which would be nicely formatting. I can include some R code. And then when I render that, I get an interactive document. And this one is just showing linked brushing. So when I select points in one plot, I can see them in another. Now, people have been able to do linked brushing like this basically since the 1960s. This is not a huge advance in terms of the technology. What it is advance is in terms of the availability. Because anyone with a little bit of understanding of R code can create things like this. And because all this and does is run R code in a special way, you can also use it to do arbitrary things. Like I can show this little table, which I'm just gonna make it a little smaller. I can show a table of correlations, which just shows the correlations of the data points under the brush. So anything you can do in R, you can now combine with an interactive graphic to do interactively. And then finally, I've showed you all of these things on my computer. It's also trivial to publish them. So for example, I can show you them. So you can use my code if you have R, if you wanna share it with someone that does not have R. You can publish it to a web server. And then this is accessible to anyone who has a web browser, which is basically anyone these days. Now hopefully this is going to load and not prove me wrong. We'll give it a few seconds and I'm just gonna blame Berkeley's for internet service for any problems. Well, well, maybe we'll come back to this, but you can publish these things to a live server and then people can look at them interactively. They don't need to be running R. All the person who creates this needs to use R, but the people viewing it do not. If you'd like to learn more about this, Google for GgVis, there's a web page, it's available on CRAN, there's a mailing list, et cetera, et cetera. Lots of vignettes explaining how to use it. I wanted to finish by talking a little bit about what I kind of see still as the challenges of data analysis, particularly in R and kind of what my goals are future workers. And I think there really are, this arrow is still a huge problem. Getting your data is often one of the biggest bottlenecks and it's still a huge challenge. The other thing I think is particularly interesting is this sort of modeling and not just modeling, but the interaction between modeling and visualization. So given that you've looked at the visualization, how do you update your model? Given that you have a model, what insight does that give you into the next visualization you should look at? I think that's a really interesting and challenging area of research. There's been a little bit of work on this by David Roberts and he has written this broom package. What the broom package allows you to do is take any model object in R and turn it into a tidy data frame so that it is very easy to visualize with any of the tools that you're already familiar with. Modeling in R, I think, is absolutely one of our strong points and it is fairly consistent with the formula interface, but there are still a lot of inconsistencies and you are switching between various modeling families. You need to understand a lot of details and a lot of special cases, which makes modeling, I think, harder than it needs to be. So there are some interesting steps in the direction of making it easier like the Zellig package or Carrot. So my goal is I want to be able to provide or help to provide in R a fluent interface which allows you to go from having no data in R to having a final report or the sequence of tools that just disappear. They fade into the background. You're not fighting with them. They're there supporting you and they are effectively invisible because they just work. You're not constantly thinking, what's the name of this function and all of this function. That actually has the arguments and the other order of this function and how do I get the output from this function to the input of this other function. All of this kind of crap that is not really related to the challenge you're interested in which is how to make sense of your data. So personally for me this year, I am really heavily invested in making it easier to get data into R so making it faster to read files off disk, making it easier and more robust to connect to databases, to scrape stuff off web pages and to connect to web APIs. Thank you. Okay. Questions? Yep. The first question was, is my code available on GitHub? Now, yes, and here it is. I don't remember if I, it probably is. If you look at my GitHub repo, you'll probably find it and if you don't, feel free to shoot me an email and I can give it to you. So the question was, are there packages available to zoom in to graphics? There are some, there's some stuff based on SVG, there's some JavaScript ones. I know there is, I don't know their names off the top of my head. There was an older one called Play With. I think, I think you should be able to find them. One thing I guess, can I show you? Can I show my screen? One area that's, one package or website that's worth looking at, maybe I'll just try. Okay, I'll try that. So there's a new package called HTML Widgets. The goal of this is to make it very, very easy to, for R to talk to JavaScript libraries, often visualization libraries and it's worth looking at this website. There are some examples there and people extending, like if I was gonna go look for a interactive web thing that uses JavaScript, I would start by looking at the packages that use HTML Widgets. It does not, but there is a, you can add new backends to Deployer and there is a vignette that shows you how to do that. So it's possible. As long as there's a way to talk to site EB from R, then you can do it. So the question is have I ever, have I done any kind of formal evaluation of the usability of pipe methods versus non-pipe methods? I have not done any formal evaluation of basically anything I've ever done ever. But I think the anecdotal evidence I get is that it's really, so when I first created that pipe idea, I was, like it seemed like a really cool idea to me, but I have lots and lots of cool ideas that no one else can make heads or tails off. So I was initially used it pretty cautiously, but people really picked it up and really liked it and used it a lot. So the best metric I have is just my perceived visibility of people picking it up and people have picked it up really, really quickly and are using it and seem to really like it. And just that being able to put the verb at the start of each line is so nice because you can very, very easily scan a block of code and get the gist of what's going on. Okay, okay, good, thanks. So if you, this HTML widget is worth taking a look at, is a way, allows you to create interactive web graphics from R using a lot of the best JavaScript libraries. So if you wanna put stuff on a map, there's shiny, there's leaflet, you can see this is the R code and it generates this interactive web graphic. If you have time series data, there is digraphs which allows you to create interactive time series on the web. So they kind of see these as very like, I think one of the reasons that GG Plot 2 succeeded is because it took like the most common 80% of plotting tasks and provided a consistent interface to them. But that's not possible to do that unless there's a whole lot of other tools to do the other 20% of more specialized needs. So I see these JavaScript plotting libraries as a really, really important. They're designed for special cases. They have a lot of existing work put into them. They're extremely excellent. They're a great complement to what GGViz will get to eventually, which is a generic tool for making new types of interactive graphics. So the question is basically, I'll reframe it as how does stuff move into base R? And the question is basically stuff does not move into base R. And it probably never will. And I think that's actually okay. I think rather than trying to put more stuff into base R, it would actually be better to put less stuff in base R and really embrace the diverse ecosystem of R packages. And at the end of the day, having stuff in base R doesn't really make that much difference because installing packages is so easy anyway. I don't see that as a big concern. So the question, I really like this because I can just reframe it into a question that I wanna answer, the question is sort of about this, this 80-20 rule, like how do you deal with the fact that you've now got this like 20% elegant code and 20% icky code? I think one thing is that it's neat, the piping operator doesn't, you can use it with any function right now. So you still always can pipe stuff into functions that come from other packages. And I think there's a really interesting analogy between the piping operator and method chaining, which is if you've ever used the Pandas Data Manipulation Library in Python, it has a very, something that's very similar to the pipe idea, where you call a sequence of methods. So you chain, you have a pipeline of transformations achieved by calling a method after method after method. Now, there's the advantage of the pipe operator is that anyone can add a function into that chain. The disadvantage of method chaining is that the only person who can add a new method is the owner of that object. Now, there's downsize to the piping as well, but because these things are not, these functions that can go in a pipe are now functions that can be called anywhere. And that means that generally they need to have longer names because in order to be explicit, whereas if you're calling methods, they can be shorter. But I think that's something that's really powerful about this idea of pipelines, that you're not just limited to things that I've thought would be good in a pipeline. You can create your own functions that go in the pipeline and work really well there too. Yes, needless inconsistency. Yeah, so the question is about by use of non-standard valuation in R, and partly also about why is the syntax for dplyr and ggvis inconsistent? And basically, so using non-standard valuation, being able to evaluate an expression in R in an unusual way is really, really powerful in R because it allows you to create things that kind of break all the rules. And really, I think sort of an interesting journey for me is like how well have I understood how that works? And I've had at least I think three epiphanies where I'm like, yes, I finally understand how this works. And then two years later, I'm like, oh, I didn't really understand it at all. Now I finally understand it. And I'm reasonably certain I've had my final epiphany and I now actually really do understand how it all works. And I understand all the failure modes. And there was for about a year where I understood it and I was like, cool, now I can write code that reliably works for non-standard valuation. And that led to the creation of this lazy eval package. Now unfortunately, after finally understanding how it works and how to do it correctly in the vast majority of situations, I'm now kind of coming to the conclusion that you shouldn't do it in the first place. And instead you should use formulas which turn out to be one of the most powerful and useful components of R in terms of computing on the language or metaprogramming. They're not just useful for just defining model formulas but they're very, very useful in general. So unfortunately, D-Plyer kind of started on one side of that transformation in GGViz on the other and how we're gonna reconcile them to make them both work the same way is a difficult question because there's no way to make D-Plyer. There's no way to change D-Plyer without fundamentally breaking it to work in the same way as GGViz. So at some point there may be like a D-Plyer two which requires you to explicitly use formulas everywhere. I'm still struggling with that and how, what the right thing to do is there but for now it's just a small inconsistency between the two. So the question is basically what can you put in the columns of a data frame and all the examples I showed you just have very simple things like numbers or strings or logical. Now it turns out that in the regular data frames and base R you can actually put more complex objects in them. So it's totally valid to have a column of a data frame that contains a linear model for example. And so I think that idea of having more complex objects inside data frame columns is really, really powerful and I think there are certain circumstances we're having a column of a data frame where each value is another data frame can actually be really, really useful and really powerful and certainly putting models inside data frames is also a good idea. Now I, internet seems to stop working again but I've been kind of exploring that idea in this package called Lowliner which is thinking about how ideas from functional programming can kind of help us with that, that modeling part of the piece. Because often when you're, so I think a good example is cross validation, right? You take a whole random, you randomly sample your data and you randomly partition your data into like 80% training, 20% test. So now you have kind of two sets of data frames. You've got a whole lot of training data frames and you got a whole lot of test data frames. And now you want to fit to each training data frame you want to fit a model to it. So now you have a list of models and now you want to take that list of models and the list of test data sets and predict them and now you have a list of predictions and then finally you want to look at those predictions and compute the root mean square error or something and you end up with a vector of that. So I think you can do that inside a data frame and I think that's a really useful way of working and the goal is to get dplyr to the point where that, in Lowliner together to the point where that just feels natural and easy and the sort of each version of dplyr is stepped a little bit closer to that ideal and I think, and by and large the main reason you can't do this with data frames currently is that there are a few bugs in how data frames that contain complex objects get printed which basically makes them useless, sand or extremely annoying. One more, one more question? Okay. So let me, so the question was about the, there's some sort of more advanced forms of piping which I can actually happen to have a little picture of here. So there are a few restrictions about pipes. I'm not gonna kind of talk about them but these are like, these are pipelines that are useful and valid and there are some pipelines you just cannot construct for various reasons. Now it is possible to create a pipeline that has sort of a cycle in it like this where you do a sequence of operations and the final operation replaces the initial input and so that's pretty common, right? You do a sequence of operations and you use that to replace the thing you started with and there was this compound piping assignment operator in Margarita that lets you do sort of a little shortcut. I am fundamentally conflicted about whether that's a good idea or not. I mean, it's again really, really important when you are writing code to write code that's readable and that generally imply that if you wanna write code that is readable, you need to use a fairly standard vocabulary. You don't wanna use really esoteric words and I think at the moment this is fairly esoteric and I'm not sure whether like the amount of code it saves you is worth the cost of that additional knowledge. Now on the other hand, there's a thing that I think is really cool and that is often you apply the same pipeline to different input data sets and Margarita has this neat capability is if the input, the first thing into a pipeline is just the dot rather than executing the pipeline it creates a function that will execute the pipeline and that seems, I like this. I'm not sure again whether it's super useful but it seems really elegant to me that you've got some way of saying either create a specific pipeline that takes a specific data set and produces a specific result or make a generic pipeline which is just a fun function which takes some input and gives you some output. Thank you.


Hello there, my beautiful internet friends. Welcome back to my channel. Thanks for joining me here today for a very brief to the point update video. Many of you have been so very kind as to reach out to me and inquire how I was doing and if I was okay, and if something was up because I know that upload schedules don't really matter to most people. You may not care watching this, but I haven't uploaded a video in like a week and a half, almost two weeks now. That's very not like me or my schedule, so thank you for your concern everybody. Long and short of it is I've been sick for about four weeks now. I've gotten better for like a couple days at a time and gotten like a lot worse and then I've been bad for like a week and a half now and we didn't know what was going on and mostly I was just resting and trying to breathe issues around like coughing and a lot of chest pain, shortness of breath, trouble like actually getting air into my lungs and the inability to do anything except like sit still. Like I've been very useless. I've been doing a lot of absolutely nothing, but today I went in for a chest x-ray and apparently my lungs are hyper-extended, hyper-inflated, one of the two. Basically, I'm dealing with a really bad asthma exacerbation. I had a virus which triggered all of this, I guess, and now my lungs will not function properly, but the good news is that now that we know what's going on, we can properly treat it. So Brian's out getting medications and steroids right now and hopefully I will be able to actually breathe normally and talk for more than a few minutes at a time and sit up without getting dizzy and nauseous and the chest pain will go away. But I wanted to give you guys just a quick check-in update because it kind of disappeared off the face of social media for a while and I'm going to continue being disappeared until I'm actually better. It's funny. Occasionally, I've said this before, but occasionally like I listen to YouTube drama videos or what's going on in the community and a few weeks ago Tana Monju, Mungo, I don't know how to say her last name, made some comment about she was diagnosed with asthma and was basically joking about like, oh, it's nothing. It's just asthma. Everybody is asthma. Who cares? I'm not gonna take care of it because it's just asthma. Asthma is a pretty serious thing. It can get pretty bad. So if you have asthma, please take it seriously. I've never had a really bad case of it. Like I don't have the disease of asthma very severely. It's just like if I'm doing stuff, I have to take an inhaler, but I guess it can get bad pretty quickly, depending on the situation. Like if a virus exacerbates things. So I have the weekend to be on these medications and hopefully start feeling better. If I don't, then we have more serious steps after that. Hopefully get things working and functioning better and my lungs back up to speed. So I'll be a-okay. It's just it will probably take a little bit of time and I'm not gonna be doing much of anything until then and just focusing on taking care of my health and taking care of my little body and my little over-inflated lungs. So I hope you guys are doing okay.  Thanks for listening to me. I am really looking forward to being back sometime soon with actually cool content, cool videos that I'm excited to make for you guys, but those can't happen just yet as you know, you can't accomplish too much if you can't get enough air in your lungs. So with that being said, I'm gonna stop talking and post this video for you guys and say thank you again for all of your care and your concern and your support. It really means the world to me. You really have touched my life in so many ways and I thank you for it. Thanks guys. I will see you in an upcoming video and also for any more immediate updates. I'll post like an Instagram story to keep people updated, especially if this goes on for more than this weekend. So if you feel like it, check me out there under Footless Joe. Yeah, and I'll talk to you guys soon. I love you. I'm thinking about you and I will see you in the next video whenever that may be. Take care of yourselves. Bye guys.