Let's check your notebook now with the new... Yeah, mine is working, so... See, we solved the technical issues. So let's start with the opening session. First of all, welcome to Basel. So this is EuroPython 2019. It's the 18th EuroPython conference. So we've come a long, long way. We sold even more tickets than last year. We sold 1,200 tickets for the conference and 300 for the training days. We're new to EuroPython. Wow. Excellent. So who's the regular? Who's been to EuroPython before? Not that many. That's interesting. So for all the regulars, I'd like to ask you to help all the newcomers feel at home at EuroPython to maybe explain a bit how things work because every year, basically, we do more or less the same thing. And so if someone needs help, please do guide them, please help them and let them know how we do things. For example, what lightning talks are or what a poster session is or what open space is. That would be really nice of you because this is our conference and so we want to make it really nice for everyone. I'd also like to say a big thanks to our sponsors. The sponsor exhibit is just outside on the second floor.  What you typically don't realize is that the sponsors actually make the tickets affordable. So we'd have to charge you about if it were not for these companies to help us. So please give them a big hand. And now I'd like to hand over to Martin Christen. He's going to tell you a bit about Basel. Welcome to Basel. Welcome to Europas in 2019. I'm actually from Basel. I will tell more about Basel. Actually, in Switzerland, we do have four languages. So German, French, Italian and this Rhetoromanian language. And Basel, you can speak German, English, French, so it's no problem at all. You see here the picture of the Baselisk. That's a dragon or something like a dragon. It's a symbol of Basel and also the name Basel Basilea came from the Roman. Basel is a very old city. It was founded by Celts 120 BC. So really, really old. They didn't program Pison back then but I think they would have if they knew it. And the most important thing in Basel is the river Rhine. It actually separates Basel in two places, the lower and the higher part or the lower and higher, I could say. And you will see Basel also has two city centers on each side. I think you will enjoy that. Basel is a Pison city. I'm actually from the University of Applied Sciences and Arts, Northwestern Switzerland. That's where the tutorials yesterday and on Monday were happening. Those of you who are there, remember that certainly. I'm teaching Pison, so many students here, they know Pison and they spread the word how cool this language is. I'm also the organizer of Pi Basel and there is the big Geopison event every year in Basel. You can see that Pison and Geodata, you could say the Geospatial Data Science with Pison Conference. And there are also new trends like Pi Pharma because Basel is a pharma city so the pharma industry discovered how important Pison is in data science and discovered that it is really useful for that. And as I said before, Pison in education is very popular in Switzerland. More and more universities switch to Pison. It's a perfect language for many purposes. So I'm giving back the word to Mark.  It was a pleasure. It was really a pleasure. I like Pison, I do everything for Pison. So which brings me to the next slide who's making all this happen. It's actually just very few people. These are essentially the main organizers. Plus of course we have lots and lots of on-site volunteers. The people in the yellow T-shirts who are helping us enormously make this happen without the volunteers, the on-site volunteers that we have, this wouldn't be possible. All of the people that you see here do this for free. This is all volunteer jobs. No one gets paid for these things. So I think all of these people deserve a big hand from you as well. The Europe Pison Society is running the show. It's running Europe Pison. More recently we also started to support the European Pison community. So if you for example run a conference somewhere and you need a grant, you need some money to get you started or you need some other help, maybe organizational help, then please contact us. We are very open to helping you out. If you want to help us even more then of course you can earn one of these yellow T-shirts. And for that you only need to register on the website to become a volunteer and then please come to the registration desk and then we can find something for you to do or you can go to the volunteer app that we have to register there. This is the hitchhiker's guide to Europe Pison. Especially important is the Wi-Fi information in there because as you can see the username is not exactly something you can easily remember. We've tried to put these pieces of paper around the venues with all the data so hopefully you can get the information from there. If not you can go to the conference app. It has the information as well. And we'd like to encourage you to install the Attendify app. Again you can go to the website to learn how to use that. We have a telegram channel that's open to the public so anyone can register. If you want to for example arrange dinner or you want to meet some other people in the city then you can coordinate on that channel. You can also post images, post videos and so on there. Just please be aware that all this is subject to our code of conduct. I'm going to get to that later on. So you shouldn't post anything that could make people unhappy. Everything else is available at the conference desk. So if you have a question then you can just go down there and ask them. I tried to put these maps on the slide here. They're too small you can't really see them. If you have the app installed it has a section with the maps so you can find out where things are. We also put some signage outside which explains you where the different rooms are. So there are different monitors around the venue which show you things. Essentially we are using three floors. We're not using the first floor, we're just using the ground floor. We're using the second floor and the third floor. So actually I'm not sure. Did you get your bags already? Yes. No? Yes? No? Okay. So if you already received your bags so the people who went to the train days already received the bags for example. You'll have a booklet in there, a conference booklet which has all the information. The program in the booklet is not really up to date anymore because we made a few changes but most of the talks should still be fine. You should have a t-shirt. You should have the official EuroPython PewPew device. Did you get that already? Yes. Yeah? You'll probably also find some sponsor gadgets, coupons and other sponsor information in there. Coming to the PewPew device, this is it. This is how it looks like. It's a specially made device for EuroPython. It was designed by Radomir. I don't know whether he's here. Radomir? Okay. We had workshops for this device to teach you how to use it. It's programmable in Python. You can do lots of nice stuff with it. You can play games on it. Batteries are included. We have our own Python batteries for these devices. If you want to read up on how it's used, you can either go to Radomir and maybe he can help you a bit or you can go read the documentation there to figure out how it works. Essentially, you just put in the batteries, you turn it on, and it starts with showing the EuroPython, then you have to press a few buttons to go to the games. Right. The official part of the conference, of course, we have lots of talks. We have lots of poster sessions. We have keynotes. These are our keynotes. We're going to start with Lynn today. As you can see, we have four ladies, one gentleman there. So, diversity-wise, this is, I think, a good selection. We also have some other events that I would like to mention. We already had on the trading days all the great ones. We're going to have the poster session. I don't know exactly when it is. Maybe today or tomorrow. Then we have a pie data track. So, lots of talks in the pie data field. We have a recruitment session for those of you who are looking for a new job. We have an open space where you can arrange talks yourself. We're going to put up a whiteboard or something to manage this. We're going to have helpdesk where teams can have registered to then show you certain open source projects and help you with any questions that you might have. And, of course, on the weekend, we have the sprints and hackathons at the university again. So, just a few words about the open spaces. I'm not going to go into detail. Open space is essentially something that you arrange yourself. The room is called Delphi. It's at the level zero, so ground floor. And similarly, we have the lightning talks. How many of you don't know what a lightning talk is? Excellent, so I don't have to explain. So, it's first come, first serve. We're going to have them on Wednesday, Thursday, and Friday. We're going to set up a flip chart, probably near the registration desk where you can then mark your talks that you want to give. And then this is all going to be organized by Alexander. Where's Alexander? Alexander. There he is, down there. And so, these are usually probably the most exciting part of the conference. These are always fun and nice to go to. I already mentioned the conference app. I won't go into detail here. The only thing I want to mention is that you can actually rate the sessions. So, for each session, we have this rating bar in there. Whether it works out this year or not, I don't know. Last year it didn't work out, so maybe, you know. But it's definitely good to find the schedule. We're updating it regularly, and it works offline. Then we have a social event on Thursday. It's important. Please get your tickets. We only have a fixed number of seats. I think 400 we have, right? So, if you want to go to the social event, then please get your ticket in the ticket shop. We're going to have grilled sausages, baked potatoes. We're going to have the ticket includes two drinks, but of course you can buy more. We're going to have a very special Python music session in there. So, what's it called? Python de Travers. This is a group that's actually using Python to produce music. It's a bit experimental, so it's not the stuff that you get in discos. But it's going to be very interesting. We have a quiet room for those of you who want to take a break. It's also at the level zero ground floor. We have a code of conduct, like I mentioned. So, the very, very short version is be nice to each other, because this is our conference. We want to make it enjoyable for everyone and be professional and don't spam. This is more geared towards companies attending the conference who are not sponsors or even the companies who are sponsors, but because we don't want to, we don't want to spam people. If you want to read the full version of the code of conduct, you need to go to our website. The people on the right here, these are the code of conduct workgroup members. If you have any issues, please contact one of these people. It's probably best to contact them personally. If you don't see them, please go to the registration desk and they can then inform us and we will come to the registration desk to sort out any issues that you find. This applies to all the channels that we have, all the social channels, of course the conference, but also the social event and anything that happens around the conference. Right, so last slide, speaker guidelines. If you are a speaker, please do test your notebooks before you come to the session. You know, we should have done that as well, but we kind of forgot. After your talk, please do upload your slides. And there was one more thing I wanted to say about the catering. We have a special food point when you exit on the second level to the right from, you know, on that side, so it's the left on your side, for vegetarian food and vegan food and any other diets, I suppose that they can also serve other diets as well. If you want to have food specifically on one of those diets and please go there for all the others, I would recommend not going there because otherwise we're going to run out of vegetarian food and we don't want that. So we have plenty of other food points available for you to get your food. So that's it. Enjoy the conference. And have a good time. So now I'm going to pass over to Lin. Ah, you want to introduce her. Okay, excellent. Yeah, that's too complicated. I just need it for a minute. So I'm really happy to introduce you to the first keynote of this year's EuroPython. I'm very happy to welcome Lin. Lin has a distinguished career and user research, data mining and user experience. She holds two PhDs from Stanford and Cambridge. No, I misread this. Oh, a Masters in the PhD, sorry. Excuse me, I was, I was, excuse my... You got uninvited. No, sorry. But I think what I really like about her work is she has worked in the industry and in research and currently she's located in Lyon at a business school. All right, okay. And she's also a regular speaker at international conferences. So I'm really happy she's able to join us today and to tell you how to get your data genre de vivre. My French is, please excuse my French. Back, so please give Lin a warm welcome. Yeah, I don't have two PhDs. I also wasn't registered for the conference and my MacBook wasn't working with the projector. Let's see if we get through this. So I'm very happy to be here. This is always interesting to me to give talks. So I do a lot of NLP these days. I'm not going to talk about NLP as much today, but we're going to do some word-to-vec models. Right now I'm working on hate speech detection, toxic and offensive speech, and it's horribly depressing. That's also why I'm not going to talk about NLP.  So I was a teacher. I'm actually consulting now. That's where I'm doing the hate speech detection, but I'm still in Lena, so I'm just down the road from you guys by a train. When I do invited talks like this, I get to do what I want, right? So like I said, this is me doing self-care and hacking on the weekends, and probably I should develop some real-life hobbies rather than hacking, but I'm a nerd just like you guys. So this is the thing about hacking. It's really hard for those of us who program to separate out the programming for fun from the programming for work. Everybody I talk to is like, oh, yeah, I did the side project and I learned how to do blah, and now I can use it on my job. It's an especially American problem where we're always thinking about how to monetize our free time. Let's do less of that. Let's make more junk and let's have a good time doing it. This is one of the things I wanted to say to you guys. It's also a message to me. So I'm going to talk about some things I did for this talk that are just totally junk, but fun junk that I admit I learned a lot from and probably wouldn't have done them if I hadn't, but I was inspired to do them by wanting to do junk. So you guys might be familiar with Hieronymus Bosch's The Garden of Earthly Delights. This is this massive, massive beautiful painting with these three panes of it. The left side is sort of a Garden of Eden simple thing. In the middle we have whatever the hell is going on that has to do with like, stuff people on earth and maybe corruption and desire and talking fish. Who knows? And then over on the right we have the last judgment or sort of the end of the world, hell, whatever. There's a lot of analysis of this picture. It's a form of big data. I got really interested in this because of the Bosch bot on Twitter. The Bosch bot is this person who is posting quantum random selections of this big picture as a tiny little thumbnail.  The way it works is she posts this image that's a very, very zoomed in random segment. And then of course you want to go and click on it and you do and you get like the bigger context which is random. I mean so sometimes it's a good segment and sometimes it's not and then you favor it and you move on. So what I wondered as a data analyst is what are the details in this big picture that people faved the most? What did they like the most? So I made an app and these are the sort of stages I went through doing this. I'm going to, because we're a hacking audience I want you to know like the parts that were fun and the parts that were really hard but also because I'm learning while I'm hacking I'm pushing through the pain, right? There was some real pain on some of these projects. So first of all, there's this cool library, TWINT that you can actually, you don't need to use Twitter's API and it has a really nice command line interface to pull down tweets and you can even say like tweets from this user with only images in them which is what I did for this case. So I pulled down January 1st through May 5th of this year. I finished the project, finished for the current state in May. So that was a super awesome library that I hadn't used, it's really cool. Then I used pandas to load that data from JSON sort, filter, do some work on it and get the top 10 by likes. Pandas is always fun, totally awesome. The part that was interesting and hard and that I did not succeed at, talk about that in a second, is figuring out where in the giant picture that little segment came from. You can talk about that in a second. And then I used leaflet.js as a web app to display this. So you're going to see a lot of web stuff because I was sort of getting back to some web programming for these. So this is the app. Let me just quickly demo it so you can see. So what I did was with the top 10, I located them in the image and put a little marker there for where they were taken from the big picture. You can see at a glance, the thing I was most curious about is what panel are they coming from. The one on the right has an awful lot of action going on. It's not too surprising when you look at the actual details of what's going on in that panel. There's very little action over in the garden of Eden section. And then we have a bit, and I think there would be a bit more when I redo this and update it in the middle panel because there's so much stuff there. So I wanted to size by likes so you can see there's a big one here. This is the biggest one. And then I was like, okay, well, is it just a fact that the things that are older have more likes? But no, it's not true. So this one, at the time I did the scraping, was the most liked, but actually the newest as well. If we click on that, we zoom in and that's the picture people liked. Uh-huh. So I learned an awful lot about, like, Twitter from that. Now, if we look at, like, some other things like going on up here. So this one, this is like just burning in hell, like that little image, which is pretty cool too. There's one down here that I like. So one of the cool things about this picture is really zooming in, it pays off a lot. This is like big data that's totally fun, so we can sort of move around. There's so much detail in this picture, it's ridiculous. So this is the lowest level, zoom level. You can see probably, like, the paint fleck-drying effect here. This is one of the reasons localizing where the little segments came from in the big picture was a no-go for me. Okay, so this is big data. Big data is just data that's hard to deal with, right? It was slow to load the high-res image, which is 223 meg in one go in the browser, which is what I was going to do originally. And then if you want to move around, pan and zoom, it's basically ridiculous. Pillow-based Python tools won't load it. You actually get an error warning about decompression bombs, which I was like, what? So basically, you can disable that and get it to work with it anyway, but that shows you the size of this as a big image. But worse, the algorithms I tried to locate where that little snippet, the posted snippet were in the big image crashed my laptop several times. Now, obviously, I could have gone to a remote machine with more memory, et cetera, but the way they were failing and the kinds of matches I was getting when I changed resolutions, et cetera, were leading me to believe that they weren't going to work. If anyone is a really good expert at neural net template matching and feature detection, I would love to talk to you afterwards to see if I could figure this out. However, there wasn't out. This was a hack, right? I'm like, okay, solving this problem isn't the point of my exercise. It was what are the parts of the picture people like? So I just went into Photoshop and figured out where these segments were and like hand-coded a JSON file for the top 10. Obviously, that doesn't scale to a live app. And I talked to the Boschbot author and he or she has added the actual snapshot locations with top left corner, et cetera, to all of the posts now, just for me. So I have to make a live app with these after I'm done with this talk. So there will be a live app showing what the most recent likes and faves were and what section of the image they were. So that's kind of cool. I made a friend. So I said I used leaflet. So the big data part of this for web design, this was something I had been wondering about for a while. I sort of knew about it, but I hadn't made anything with it. You can think of a really big picture that you want to zoom and pan on as a map. And the tools for doing maps online are the tools for doing big image display like that. So what you do is you tile your image. You have sections of the big image at different resolutions and different size for all these different zoom layers and it theoretically happened seamlessly. I think my map left, I was running out of memory and that's why it wasn't so seamless when I was moving around. But essentially use something that's a map tool, GDAL to tiles, to make these layered directory folders for different zoom levels. And there's a lot online about how to do it. It's just interesting that we're essentially dealing with tools for maps. So this is me this morning in my hotel room trying to remember the bash script to show the size of each directory. So at the 0th level, which is the top view, there's only one image. And then as you zoom in, 7 being the lowest zoom level where you can see the little paint flex, we have the most images. And we only pull up the ones that we're looking at at that point which is how you get a smooth effect. So the other thing that's cool about treating this image like a map is that if you use a tool like LeafletJS which is for interactive map display online, it's an old established tool with lots and lots of UI features and controls. It's actually really easy to add stuff like when I zoom into a certain level replace it with the rectangle showing what the edges of that snippet were. When I zoom back out, replace it with a dot. And you get tool tips and things like that. So that was cool.  So this one, it's only because I was talking to the Bosch bot about how he or she did the segments of the image randomly that I learned why this was number one. At first I was like, well, obviously, that's just people like that picture, right? Actually, it's the pinned image on their account at the top of the account.  It's not just that. There was a celebrity who went and retweeted it. And so that's actually why it's number one. So when you're looking at faves on Twitter, you're always going to have these effects that are very famous, like, accelerates your liking. So it's a bit hard. She's a comic. She's actually a pretty funny follow. So just this past week, the pudding, they do a lot of interactive, interesting data visualizations. They did something similar. This is a TSNE layout with, using Mario Klingon's roster fairy to squareify the layout. They used a tool called OpenSeaDragon instead of Leaflet. So you get the same kinds of zoom, pan, move around. I don't have a mouse, so it's less than cool. But essentially these are very detailed, tiled images. I could probably plug in my same tiles. All they've got is the zoom in and out button. Because they didn't use Leaflet, they don't get the access to the same number of UI controls and things that I wanted to have. So even if I switched, I could do a lot of coding from hand to get some of the effects that I was building in the app. Okay, so that was a fun project that now has me promising to build a live, updating, real web app. Excellent. Word2vec toys. So one of these is a project I did last year for a conference that I then didn't go to because they had weird funding problems. But it led to the project after it, so I'm going to talk about this. So most of you are probably, at least, glancingly familiar with Word2vec, which has been all over the place for a few years. The concept in Word2vec is that we do an analysis of text and a lot of text, and we come up with essentially a matrix with a vector representation for each word in the document collection that we've analyzed. We do that by looking at sort of a window of context around the words. So what we do is we essentially encode in this matrix things about the relationship of this word to other words in the collection, which means that we can do math on that matrix like find me the most similar vectors using cosine similarity, for instance, and those are words that are related and that they've occurred in the same contexts. My source for these projects is Project Gutenberg, which is a great source for fair use text, lots and lots of text in lots of formats. You can even download and read it on your Kindle. You can get the plain text. There are libraries out there to deal with getting text out of this because it's a huge collection. The one that I settled on using is actually an R project, called Gutenberg from David Robinson because it allowed me to query by subject. So I do a lot of things by subject, like say ghost stories or poetry in this case or detective stories or whatever. You can do other queries by author by other types of metadata. Now Gutenberg updates its catalog fairly regularly, so you have to keep your local copy of the metadata up to date because their API, I don't believe they have an online API that you can query and do all of what we've done here. All right, so anyway, so I downloaded his code. I had to update all of the metadata tables because in January, in particular in January, there was a big change. And then one of the things his library does that some libraries don't do is it strips out the header text and the footer text, which is this sort of generic stuff on Gutenberg files. You don't want that in your language model. All right, so then if you want to make a word to VEC model, there's lots of info online about how to do it. The most popular way is Gensim. Gensim wants you to put in tokenized sentences. So you have essentially this is a sentence and it's tokens, so just the words as a list. And then your entire document collection is one of these for each document. Sentences generally. And then I'm giving you this particular code example because it shows how to save the model and then load the model. If you're going to do the kinds of things that I did where you use your model for other things, you need to be able to save and then reload it in some place like a Flask API. So one of the attributes of Word to VEC that a lot of people have heard about is that you can make these sort of 2D layouts that show you those relationships between words. So like I said, you can do cosine similarity and determine words that are similar to each other in the vector space. So you can also make these sort of explorable maps. Thanks to Peter Baumgartner for a gist that I modified to get this working quickly for this talk. So this is folk and fairy tales. And this is an interactive exploratory tool for looking at the space. So it's in plotly, which means that I can filter and sort and things better with a map. So if I roll over these, you can see what the words are and see possibly why they're grouped together. So this is things that have to do with time and distance probably. Fifth, sixth, fourth, eighth. Afterwards, week, before, short. Afterward, years, months, miles. So frequently you get parts of speech that are grouped similarly. Boy, it would be way better if I had a mouse. Over here we have things that are sayest, mayest, worst, willst, thyself. So apparently there was some really old fashioned text in this Gutenberg file. And it grouped all of those old fashioned helper verbs together. Over here, further, rather sooner, higher, sweeter, et cetera. Up here we have other languages. Words from other languages, which is strange and I could go and look at these. In this particular layout, color means that this word occurred more often in the text file. So his gist that I used to make this, I updated so that you can do your own with the word counts as well as the model. And it makes this cool, this cool map. Okay. So, like I said, you can find words that are related to each other with these models. So you can do the, so for polite, what are the most similar words? Courteous, friendly, cordial, professional, attentive, gracious, that looks like a good model. So my question was, what if we went from the closest word to polite, and then from courteous we looked for what the closest word is. It's not necessarily friendly because of the way these things work, right? And so what if we kept chaining from word to closest neighbor, word to closest neighbor in these models? And so I made an app. This is the word fire in a Gutenberg older poetry word-to-vec model. Closest word is burning, closest word to that, but it's a bit distant compared to others. It's folded. Lids, curls, silk, scarlet, glossy. And so if we keep doing that, this swoop is supposed to indicate that we're reading left to right. We get these words. Now if I change to a different model, this is a slightly different model that includes things that were released in January. We get different results. So the closest word for fire is flame, burning, crimson, purple, foam, gleaming. Now if I switch to folklore and fairy tales, the one I was just looking at in the big graph, we get hearth, and then pretty far away oven, pale, empty, dipped. So think about like Hansel and Gretel, things like that. The witch is going to put them in the oven or whatever. So these are nouns related to scary fire situations probably. Okay, and I don't think Nerd2Vec works very well. I'm going to talk about that in a second. This is quite different. Yeah, so Nerd2Vec, which is built on like Star Trek, Doctor Who, Star Wars, firing, shooting, throwing. Not so surprising. All right. So this is another example. Alison Parrish is the one who inspired me to do this originally because she had a poetry Gutenberg corpus, and then I made the word 2Vec and then built these apps with it. So what we're seeing here is the vertical distance. So the distance between one word and its next closest neighbor. And then the size here is essentially encoding the fact that there were loops. We don't want cycles in this list of words. It gets boring really fast because you see the same word repeated over and over. So I just sort of increment the count for a word and then go on to the next closest. So that we get an actual chain. And the color is just pretty in this case. I didn't say that as a database person, but it is. All right. So like I said, there was new public domain stuff in January 2019. So some of the things that were added were really cool things to work with, like Robert Frost. Robert Frost is a well-known American poet. His stuff all entered public domain. A bunch of other great fiction works that are essentially more modern in feel. So this is an interesting contrast here. Believe, the word believe in the older model pre-January. We had a model for poetry that went learn, grieve, suffrage, choose, understand, etc. And then after I updated the model with the things from January of this year that were just released, we get magazines in there somewhere, which is super weird. So now believe goes to suppose, believe. I didn't take out loops from the seed word. And then over there in the second column, magazines, satires, newspaper, etc. You can see it kind of goes off the rails sometimes over to the right where we get into really specific words like parliament. This isn't a poetry model too, which is weird. Yeah, so this is the source on Nerd2Vec. So Nerd2Vec is definitely very different in its properties. Now this brings me to my next project. So all of that was just sort of precursor to the project I actually wanted to make for this talk, which is that I'm really interested in creativity assist tools that help you build things using AI or machine learning or whatever, but with a human solidly in the loop and in control. So one of my inspirations with text games is things like blackout poetry. You have an entire text, but you only use certain words in it. That's Austin Cleon. This is cutout poetry or cutout poetry where it's different texts combined in order to make a poem by a human author. So this goes all the way back to the survey list some before. That's Timothy David Ray. This is an interesting case. This is Reznikov wrote poetry based on legal briefings. So he took legal documents and then put together segments to make poems that are fairly moving actually. So I built an app. So what I wanted to do was mess up poetry. I wanted to take good poetry and then play with it at the word level. The inspiration here was like, am I going to learn, am I going to make anything better? Probably not. But am I going to learn why they did what they did in great detail? Probably. So it's a really interesting close reading exercise to take someone's work and then mess with it. But I wanted to do it in a kind of principled way with lots of constraints. So what I wanted to do was for each word there that's in color, I'm loading up the list of the 10 closest words in a poetry model. What other poets might have used instead? So essentially this is an editable demo. So that's Robert Frost. I have a bunch of different models in here, word-to-vec models. I have the older original poetry one. I have up to 1923 poetry. So if I pick woods, I can pick a different word. So I'm going through the woods. Maybe I'm going through the fields instead. Whose fields these are, I think I know. His house is in the village. His table is in the village. See, this is not a good poem. But his table is in the castle, I kind of like. All right. So it turns out that just I'll get to the color in a second. It turns out that these longer poems are a bit harder to work with. Let me pick something shorter. Okay. So this is Amy Lowell. Your voice is like bells over roofs at dawn when a bird flies. Okay. You see how they aren't all the same part of speech. So there are word-to-vec models that are encoded for part of speech. So I could have done that. But instead I'm building a model that's just like, what did other poets do in this kind of context, right?  So at the point where you've done the second word, you might want to go back and fix the other one. This is the way this thing works, right? Okay. So these words about time are usually the easiest. Okay. Now, nerd-to-vec is weird. So remember Star Wars, Star Trek, Doctor Who, character, name, role, performance, vocal. Some of them are just like, what? Sorry. I don't know. So pillars, I don't know. So vampires have got to be at night. Notice there's sirens. One of these, you get sith, the sith lord. So, all right. So they're fun. In fact, it turns out that actually the really funnest ones are the haiku, for some reason. So with haiku, of course, you have a syllable count. So you have to decide, am I going to stick with the syllable count or not? So I usually try to stick with the syllable count. So old pond. Okay. Now look, here's a point about these older embeddings. They are not good at getting contexts of words that are very different. So if a word can appear in different meanings in different places, the modern transformer language models are much, much better at handling that. These old models aren't. This is an exact example of that. So pond. I'm in the nerd-to-vec model. Doctor Who, Amy Pond, is a character. If you're a Doctor Who fan. So is Rory.  So essentially the word pond in the nerd-to-vec model is about a person more than it is about ponds in the woods or whatever. In poetry, it's about ponds in the woods. But so here we essentially have a case where the word to vec sort of breaks down or this is a shitty model for poetry, which it definitely is. Okay. All right. So let's talk about this a little more. So in fact, the color here represents a normalized distance between the original word and its next closest relative. So essentially if you see something that's pinker, that means that the next relative here is further away from it than the next relative is in these cases. So that can mean that that's like an interestingly different word that isn't as common in these poems. I did a lot of work to get this color because I'm a data-vis person. I don't know if it was worth it and paid off in terms of actual use. That's one of the points about doing data-vis and interactive work is you aren't always sure you're going to get a payoff from the feature that you just spent a lot of time working on. So essentially this is just me explaining how this worked. D3 is what I used for data visualization on the web. The way this works, just so you understand, is that you essentially take a domain of numbers, which in my case are the distances between a word and its next closest neighbor, an array of all of those distances. And I get the minimum and I get the maximum. And those are my anchor points for the color scale. So the minimum is going to be the blue and the maximum is going to be the pink. So I could have recoded this, but I just used somebody's picture last night at 11 o'clock. So that's how the encoding to color works. So it also turned out that to get those scores, to get that color, I had to radically re-architect the app. This is another thing about data-vis. So this is the stack. So because I stupidly used React, the stack is incredibly deep here. We have the app at the top with the dropdowns for the model and the poem, and then we have the poem renderer. We have each line of the poem. We have words that are clickable that load up a little dialogue that have this list that you pick from and then you pass it back up to redraw the whole poem. So initially I called the API to get the closest word here, which is the obvious place. I click on a word, I get the list of closest words. Super simple.  So truthfully, like anyone here who's actually good at React, I wouldn't mind having a chat. Because React essentially is passing state is this. This is what my code looks like, too. I have like 18 functions. Yeah. I have like 18 functions called handle change. It's insane. All right. So like I said, the Basho haiku are actually the coolest. This is the original haiku, the first cold shower. Even the monkey seems to want a little coat of straw. I turned it in with one of the poetry models, the last wet shower. Even the basil seems to try a little cap of dirt, which I think is very haiku. But then the last hot meteor, even the lizard seems to need a little robe of guts, is the nerd to back version, which I thought is the so I laughed. All right. So the parts of this project. So making the word to back models is always easy with Gensium and always fun. When I decided to code the poem in React, so it's essentially because I've been doing data science in like server side Python for a little while now and haven't been making data of his apps. I was like, oh, let's just use the tool to draw. But in fact, this was a really complicated app to be my starter app in React. The learning curve was hard. The why isn't this updating and why is this updating too much was really hard. And then at the point where I decided I had to do the visualization by getting all of the distances up front and then passing them back down to the dialog list, that was really hard. The promises handling was hard. Putting the actual color on each button was super easy. D3 is fun and it's still fun in React. I was glad to see. And then at the end, like last week, I was like, oh, I'm gonna add a dropdown to change the poem, add a dropdown to change the word to back model so I can get nerd to back in there. That was actually kind of easy. So at that point, I'm like, maybe React is good for big projects where you need to make a little change. Okay. I think hooks would have helped, which is the latest way of handling all the state passing. So some related AI creativity work. So we're moving away from my projects to sort of the bigger picture. I really like that human in the loop working with a model or a tool or a learned representation. And for me, the interesting AI art is the art where I got to do an awful lot of the customization of the authoring, of the curation, not just throw up a big GAN model and then tweak one parameter. So obviously, if you're keeping up with the GP2 model, big language generation models, Hugging Faces, Write with Transformer is an example where you can write with the GP2 model. I still don't love it. I mean, I don't know if you've tried it, but essentially it's a great app, but all right. So essentially, we've got a starter text, and then we get options that the model generated based on what it was trained with. So Legolas and Gimli advanced on the orcs, raising their weapons with a herring work y. But the heroes were not to be defeated. So that's it. And then you start typing something. Hit tab, and then you get what it would complete with. A lot of times it's nonsensical. Most times it's nonsensical. Even GP2 isn't very good at big picture coherence and text. So for me, this is still not like a superb creativity tool. It's fun and funny, and I'll probably add a slide to my deck after that's one of my favorite stories I created with it, but it still has its issues. Okay. In the visual space, there's a lot more going on. Nvidia's Gauguin demo is pretty cool. So what you do is you draw with a pen that says, like, what are you drawing? Sky, tree, cloud, mountain or whatever. And then you tell it to turn it into a photo-realistic picture. And it does that based on what it's trained with. So this is like my not very good trees turned into weird trees by an ocean with some hills. So I like it because it's totally surreal, but it's obviously not photo-realism. Like, it's surreal because maybe because my input was bad or because of how it's putting it together. And then, like, you can edit it again and get more stuff going on, but it's still not superb. This is one of my favorites. Sorry about my Pinterest thing. This is one of my favorites, though. Ganbreeder is a very cool app. This person is updating it to do more things with it. The way it works is this. Those were my pictures, by the way. I'm obsessed with generating castles. I just want to build the coolest castle, fortress tower, whatever, and that's one of my obsessive goals. Okay, so if you go to the Ganbreeder page and you can see sort of random things other people have created or that have come out of the app, you can start from an image. And then what it does is it generates children related to that image, and you can edit the genes and you can see what went into this. Garden, spider, jellyfish, sea anemone, comic book, consomme, a soup, bell pepper. Okay, click on that. Get more children. Those are looking pretty, actually. Okay, I'm going to save that. Anyway, there's a bot that's posting cool ones from this and they're about to do major updates on the UI tools for this thing. It's going to be great. It's a super awesome tool. So this is a case of a human in the loop editing generated model pictures. I said I'm obsessed with castles, so as soon as he or she said they were updating it, I'm like, more castles, please. Because essentially there's one category for castles and they aren't sufficient for me to really make the castle of my dreams. All right. So that brings us to some of my next points. We're doing this stuff. Find, make, use, cool data sets, but be clever about it, right? There's so much interesting data out there and you can make your own data sets.  Jeremy Singer Vine's data is plural mailing list is awesome. He's a data journalist and there's a big archive of all of his data sets out there and it's a great newsletter with just a few short bullets in the description and he loves weird data sets. So that's super fun. I also obsessively collect links to data set collections. So my pinboard, which I update daily with my many open tabs from Twitter, there's a data set tag. Some of it isn't going to be obvious. It's things like game archives, things like that. So you're going to look and go, why is that a data set?  You can obviously make your own data sets. This is Josh Stevens, who's a great map maker. He did this now quite famous map of Bigfoot sightings in the U.S. Bigfoot, the vomitable snowman in the Yeti. So there's like public data about this and he turned it into this map that was all over in lots and lots of articles at the time. This wasn't for work, it was goofing off. And then he has over here a little like educational by variant explainer about population versus sightings of Bigfoot, which is super interesting. Okay, of Oz the Wizard. This is one of my favorite weird examples. So this is data you didn't know was data. So just like me editing poetry. So this is this guy, Matt Moosey, who alphabetized everything in the Wizard of Oz. All right, all right, all right, all right. Okay, yeah, so everything in here is alphabetized. So it's essentially alphabetical and then a temporal sequence. Decree, need the grade of degree. Illusion is not so loud. All right, so obviously he used code to do this. We could have done this, right? I'm kidding, I wouldn't have done this, but it's amazing. So one of the things about that exercise is that, as he said, his appreciation for the film increased. It's just like looking at the choice of a word in a poem and thinking, could I do better? Could I turn it into mine? When you look very closely at something using tools, you start to love how it was made. You start to love the data. Doing things with artistic works is even better because you learn a lot about the art. Another interesting AI art project. So Victor Divya did this African mask curation project. So he hand curated zillions of pictures of African masks and then used it to generate new African masks with a GAN. So there's a good write-up and an interesting demo that he made. This is a case of a passion project where you curate your own data set to do something interesting with it. I'm working on gargoyles and castles, obviously. Stay tuned. Anna Riddler. So probably Luba Elliott is going to talk about these projects tomorrow, so I'm not going into any detail, but Anna Riddler took pictures of tulips and has a huge display of all the tulips she used, and then she used them for the Netherlands to generate GAN images of tulips that have never existed. This is another hand curated beautiful collection. Helena Saren makes really interesting GAN art. She also curates and collects and does her own imagery that she feeds into the GAN to make her own artwork. I'm quite sure Luba will talk about this tomorrow. I love her style. It's very different from a lot of other GAN art.  When you're doing this kind of creative work and using data sets other people created or inspiration from other people or poems or code, give credit to your sources and the inspirations. And in particular, don't be this guy. Hopefully we all know about deep nude. This dude is like the algorithm only works with women because images of nude women are easier to find online.  He's not going to make a male version. And his argument was if I don't do it, someone else will do it in a year. What? Be smarter and be more interesting.  So the code was all over GitHub because he said it was too late. It was out there. I saw yesterday late last night that GitHub is taking it down. So good luck to them. But there's a lot of really offensive things you can do just because they're easy. It doesn't make them fun. So this is an example of sort of a tired metaphor, but it spoke to me. So this is my patio in Lyon. I have lots of flower pots on it. There's a particular flower that I really like that's essentially a weed that I cannot grow well in these pots. I don't know what the deal is. One seed fell off the edge. You can see it sort of down here. This is one plant. It gets almost no sun, no rain, and it fell on gravel, and it did that. That's one plant. I can't put it in a pot and have that success. But sometimes the stuff you just throw out there, the lands where you think it's shitty, it does the best. And that's one of the cool things about being creative and putting your stuff up there is that sometimes it will put a flower in a place full of weeds. I think you understand what I'm getting at. So Max Kraminski also has this reminder, if inexperienced creators are using your tool to churn out loads of half-baked garbage, your tool is a phenomenal success. So true. What we want is people to be making a lot of half-baked garbage and having a good time. So what I would say as a takeaway is if you find yourself playing with something you made and you're actually smiling or you're laughing at it and you're playing with it to check out something new, you've definitely succeeded at a fun weekend hack.   So I want to give some shout-out thanks to this is not all of the open-source projects I use, but the particular helps for this. The Bosch Bot was really helpful. Peter Baumgartner's just for making a U-Map display in Plotly was an amazing help. I used this dude's word-to-vec API code, which I could have written myself, but I saved a bunch of time weekend hack. Peter Gassner helped me a little with React, and this person helped a ton of people in a comment on the U-Map repo for how to actually get U-Map recognized in your Python project. I want to see how many people are super happy. I'm the one here. So anyway, that's my talk. Thank you. Thank you, Lynn. Thanks. That was a great keynote. I guess you have many questions maybe, but you're around until Thursday? Yeah, I think let's talk over coffee so we don't... Okay, thank you very much. So we have a short 15-minute coffee break, and we're going to continue at 10.30. So hi, everyone. My name is Olivier Grisel. I work at Inria in Paris, and I have some funding from the Inria Foundation. Maybe it doesn't really work this... Okay, no. So basically, I work as a software engineer to support the development of the scikit-learn project with a couple of colleagues at Inria and with more generally an international community of contributors around the world. So scikit-learn is a machine learning library in Python. So how many of you do not know about scikit-learn? I've not used scikit-learn in the past. Okay, a couple of new people. So for this presentation, what I would like to do is give a first introduction to other stuff that were also released in the past releases and some ongoing work. So first, scikit-learn and machine learning. So predictive modeling that we call also statistical machine learning is basically the process of using repeated events that are recorded in the database. So to keep a historical record and extract some statistical structures of those records, in order to turn them into some kind of executable model. So the end goal is to generate a program that will be able to predict the outcome of repeated events. So you can see that as an alternative to hard coding rules written by experts that know well about the process that is being modeled. But instead, we just use a record of data and hope that this executable model will be able to make good predictions on future events. So it's generally most useful for small predictions, a large number of fast predictions that do not have a big importance if you make an individual failure. But you want that, on average, your predictions are better than a random guess or some baseline rule that you could design quickly. And then you can be used to optimize many business processes or scientific discoveries and so on. So the general data flow is to start from the recorded data. So you have some acquisition process that could be based on mobile phones, cameras, microphones, or interaction with users via a web app. And then all those individual events, they are recorded in a database and the first step is to first find a numerical representation of those records. So those are the blue lines in this database and you have one specific column that you're interested in, the target variable that you want to predict. So this is the green one. Sometimes it's naturally present in your database. For instance, if you want to do forecast of something, you have the past information that is already there. Sometimes you have to collect human annotations. For instance, if you want to do image classification or translation of things like this, you need to have professional annotators to give you those labels. Once you have this, you can plug any kind of machine learning algorithms. So those are mathematical models implemented in Python or C++ or whatever. And the outcome of that process is a statistical model. So the statistical model is some kind of summary. It's a bunch of a couple of megabytes of parameters, for instance. That kind of summarizes the big training set that you used as an input. So typically the statistical models are a couple of megabytes or even smaller, whereas the training set could be possibly gigabytes or terabytes in some cases. Once you have this, you can deploy it. And basically, you don't need to recall all the training sets. You can just make a copy of that statistical model and deploy it on another server or on a mobile phone and record new data with the same data generative process, the same acquisition process and then execute the prediction algorithm by feeding the new test record and the model to get the predicted outcome. So one typical example of this is if you're a real estate agency, you can record the historical transactions for different housing types. And for each of those records, you collect numerical features or categorical features in different columns. So the number of rooms, it's an integer. The area in square meters is a continuous variable. The year is a specific kind of ordinal variable.  In this case, it's the price in your role in a specific column. And you can also have new records in your database where you don't have this target variable filled in because those are new customers that want to get an estimate for the house that they want to sell. And for those, by definition, you do not know the price of the future transaction. So here, the goal is to find the statistical relationships between the price and the descriptors in the historical data and then use that statistical relationship to predict some estimate of the price. So the names that we use for these are the features, the columns that we use for the input. The target variable is the output of the model. And then the records in that database are what we call in machine learning the samples. And those where we have labels and we can use for training the models are called the training set. And the new ones where we want to make predictions are the test set data. And once we collect the true value, we can compare the prediction with the true value and compute the accuracy of the machine learning model on the test set to do the generalization ability of the model, which is basically the quality of the model. So scikit-learn is a machine learning library that has hundreds of classical machine learning algorithms implemented in it. So those are the traditional families. And the goal is not to implement the latest state of the art, but more like to provide the data scientists with good baselines that they could apply on their data to build something useful out of it or to compare if they have new fancy ideas for better machine algorithms to compare that their new stuff is better than the traditional stuff. So it's an open source project using the BSD license, so pretty much anybody can use it. It's just that you have a disclaimer in case you have a bug, you take your responsibility. With a community of hundreds of contributors, now actually a pair of release, so I think it's more than 1,000 so far. And we have a team of core developers scattered around the globe in Australia and China, in France mostly around in RIA, in Germany and in the United States, in Columbia University, for instance. So most of the algorithms are meant to be used from a Python API and they are also sometimes implemented with the Python programming language with the help of numerical algebra, linear algebra routines made available by NumPy and SciPy. Sometimes the vector operations that are very efficient in NumPy are not sufficient to implement some algorithms, like for instance decision trees, where the bottleneck is not really in a matrix-matrix multiplication. So in those cases, when we have nested four loops in Python, that can be very inefficient, we have a compiled programming language which is called SITEN, which is basically an extension of the Python syntax that can be used to generate C code with types so that you can use a compiler to build a compiled extension for Python with a syntax that is very high-level and you still have NumPy operations and things like that readily available in SITEN. The main selling point of the scikit-learn machine learning library is the fact that we provide very mathematically heterogeneous models under a very homogeneous API. So data scientists can try to swap the models very easily because they all follow the same kind of API conventions, the fit method for training the model, and then the predict and transform method to apply the model to new data. And because all the models follow this standard API, we have standard tools to evaluate the quality of the model for classification, measuring the accuracy for regression, measuring the mean absolute error, and so on. Cross-validation procedures, model selection, hyperparameter tuning, and how to ensemble several models into a big model. And sometimes also to build pipelines, and so on with preprocessing. So it's a very active project. Nowadays, I think we are in the over 800,000 monthly active users browsing the documentation online. And basically it's doubling every two years approximately, so we estimate that it's around maybe a million of scikit-learn users or something like that. So what is new in scikit-learn O21? So if you look at the changelog, it's just a snippet. If you were to scroll the page and take screenshots, it would cover the walls basically. So I will just focus in a subset. And in particular, I will focus on the new gradient boosting implementation in scikit-learn. So gradient boosting is a very, very useful supervised classification model. It's based on decision trees that you train sequentially one after the other, such that second decision tree tries to correct the prediction errors of the first one. And you build an ensemble by sequentially refining the predictions using different trees. So in scikit-learn previously, we already had an implementation since approximately 80 years or something. But the way it was implemented was implementing the traditional exact method. But this traditional exact method was shown to be completely deprecated by a new approximate method implemented, for instance, in another C++ project called LightGBM by Microsoft and other libraries like XGBoost. But in particular, LightGBM could show that you could get really, really high performance by what we call beaning the data in a small number of buckets and computing histograms for the frequencies of when you see this data point with this value and compute frequencies. And this makes it possible when we train the decision trees to not have to do sorting operations that are very expensive in the traditional algorithm. So because we do that, we can get rid of sorting and we just do comparison operations and counting operations. And this is much, much more scalable. And furthermore, the beaning step itself can also improve a bit the quality by adding some regularization. So to implement this for scikit-learn, we first started to do it as some kind of prototype using the NUMBA framework, which is basically a just-in-time compiler for scientific programming, because I wanted to find a good opportunity to learn how to use NUMBA. And so this was implemented in this project, which is also open source, but just as this single algorithm implemented with NUMBA, and after that, once we could prove that we could reach this kind of performance with NUMBA, we decided to translate the code into scikit-learn, which is very similar to NUMBA. We just need to have additional type declarations, basically, so that it can be easily embedded in scikit-learn that already has a dependency on scikit-learn. And we didn't want to introduce a new dependency quickly. So maybe in the future, scikit-learn would use NUMBA, but it was easier for the short-term to just translate it. But we still plan to keep this PIGBM project around, so if you're interested in NUMBA and gradient-gustic trees, you can still use that directly. The two projects have basically the same performance, and when we measure on some benchmark datasets, for instance, the Higgs boson dataset, it's quite competitive with light GBM, which is very optimized C++ implementation. Sometimes it's even a bit faster. So the site translation was actually contributed by Nikolaou at Columbia University, and we are still working together to integrate the missing features. So this slide is slightly outdated. Now we have more losses. We can do classification with a multiclass. We still want to have new losses for quantile regression and so on to support sparse data, support missing values, which is almost ready to merge. And there are other features like this from light GBM that are not yet present in scikit-learn, so we're still working on that. Just to give you some intuition of how NUMBA makes it very easy to do high-performance algorithm in pure Python. So this is actually a snippet of the code for the beaning part, to go from continuous values to integer values in buckets. And so this is a very naive algorithm, but there is no... It's the traditional way to write beaning, and there is no obvious way to do better. But if you do that in Python, you see that you have two nested loops. It's very inefficient. But if you just import NUMBA and the ngToperator, and you decorate your function like this, then NUMBA will use LLVM to do a native version of this code and to compile it specifically for your platform. And this will run as fast as C++, basically. And furthermore, if you use the P-range function here, you see that the for loop will change the range operation with P-range. And automatically, by doing this, you get a parallel execution of the for loop. So you can do in different threads on the multi-core machines, the independent operations in different threads, and get a good speedup because of this. So with siton, it's very similar. It's just that we need to have additional types, so it wouldn't fit on the slide anymore, but the code would be very, very similar anyway. So the siton implementation is still flagged as experimental in scikit-learn. So because we plan to implement more features in it, so we know that we are going to need to change maybe the behavior of the default hyperparameters. So because of that, and we are very conservative in scikit-learn not to break our user's code, if you want to use the new Instagram of Gratin Bootstead classifier, you need to acknowledge in your source code that you are using an experimental feature and that for this specific model, we do not guarantee backward compatibility with the deprecation cycle. We will not break it for fun. We will probably just change the small behavior for the default hyperparameters. So we still need to implement sample weights, parse data, missing values is almost ready, capital recall variables and so on, but it's already usable for numerical values or if you preprocess your data you only have just numerical values. So from a performance point of view, if you benchmark this, this is a classification problem with, I think, 20 features and it's a synthetic classification problem and you see that on the x-axis you have the number of samples from hundreds of thousands to millions. I think the last point is 5 million so it could go to 10 millions, but on this laptop it was using too much memory in the end. You see that you have this kind of linear scalability, so this is a log-log plot on the y-axis, I missed the label, but it's the time of in seconds. So you see that for 50 million data points it takes approximately 10 seconds to train a typical ensemble of trees on this using this algorithm. So the blue line is scikit-learn, the orange line is light GBM so you see that for small data sets we have some override in scikit-learn because we do additional input validation, I guess, but then when you move to large data sets that last for more than one second we are very competitive with light GBM and significantly better than Exibust. I don't have exactly the same hyperparameters all Exibust and light GBM so it's hard to compare exactly but this is the kind of performance that you can expect. So I would like to switch to some interactive demo. So here I have a Jupyter notebook where I will show how to use scikit-learn basically in a typical use case and compare the different solvers and in particular focus on gradient boosted trees. So the data set... Oops, sorry, I was not at the beginning. So the data set that I'm going to use is what we call the California housing data set. So it's a small real estate data set where you have 20,000 samples so it's not very big and you have those features to describe a different housing type and the goal is to predict the price of the houses in different neighborhoods. See, it's all numeric data. So first we split the data set into a training set and a test set so that we can measure the quality on the test set and not cheat by just memorizing the prices that we have observed on the training set. So first we will just use a linear regression baseline so it's possible with scikit-learn to do linear regression like in Excel and it's very fast. You see it's 11 milliseconds and we can compute the error so it's the absolute error so I don't remember the unit but it must be several tens of thousands of dollars or something like that. And so you see that on the training set the error is slightly lower than on the test set which is kind of expected. It's easier to memorize than to generalize and you see this number so it's just a baseline. We can also do this kind of plot to compare the predictions on the x-axis to the true labels that we wanted to predict on the test set and you see that if you have a perfect model all the points should be on the diagonal. They should be the identity function basically. Here you see that there are many off-diagonal elements and especially for the large values our model tends to underpredict the large values. It goes off-diagonal. There is this kind of bias. So this is probably because we have this kind of distribution for the true labels so this is the distribution of the prices using a histogram and you see that there is this kind of censoring effect at 5 which means that basically housing above this price where censored were limited and so in the database just recorded 5 instead of more than 5. So this is an artifact of the training set and for linear models it can actually be a problem because the loss function that they used does not make this kind of assumption. So we can filter out this and we can also take the logarithm of that so that it's more like Gaussian distributed and see if our linear regression can perform better in this case and we also make a pipeline where we do some standard preprocessing linear regression on the data that has been preprocessed on the labels that have been preprocessed. So if we do that and we compute the scores again when we compute the scores we need to inverse the preprocessing that we did on Y but you see that it's significantly better already so just using a linear model let's say more correctly we can see that we already improved the predictions by quite a bit. So let's see if we can use a non-linear model to gain some more expressive power so in scikit-learn you can build complex pipelines where you do transformation of the data for instance in this case we will do polynomial feature extraction so we will keep the original numerical features and we will do also pairwise interactions and actually up to 5 degrees interactions between the numerical values and then after this feature expansion basically we will fit a final linear model but the full pipeline is no longer a linear model it's more complex so it's more expressive so this takes significantly slower because for 4 seconds instead of 11 milliseconds but you see that it's more and more diagonal you no longer see this kind of extreme deviation pessimism by the model so instead of doing feature engineering to turn a linear model into a non-linear pipeline what we can do is directly use a non-linear model like a neural network so it's kind of a generalized linear regression with built-in non-linear capabilities in it and if we do that it's also more expensive so in scikit-learn we have basic neural networks that you can use if you really want to use neural network I would advise you to rather use Keras or TensorFlow or PyTorch because you have more flexibility to design the architecture that you want but if you just want to use traditional MLP architecture you can use scikit-learn on the CPU and here you see that the accuracy is getting even slightly better than the feature extraction that we did manually with the pipeline so it's 0.21 on the test set and the training time starts to be slow it's 8 seconds now so we can do again this plot and you see that it's more and more diagonal it's still not perfect but it's slightly better than what we did before so finally the gradient boosting so whenever you have this kind of tabular data where you have some kind of Excel spreadsheet with different columns with different physical units like a number of rooms the criminality in the neighborhood whether or not it's close to a public transportation whether or not it's close to the ocean the GPS coordinates those are different columns with very different heterogeneous types physical units so this is what we call tabular structured data and for this kind of data typically neural networks are not significantly better than traditional machine learning and in particular decision trees and gradient boosting is very very competitive and most of the time significantly faster and less finicky to adjust so if you have this kind of tabular data this is the kind of machine learning algorithm that we recommend to try very quickly so it's always good to start with a linear model but then very quickly try this one to see if it's better so this is gradient boosting the traditional ones that were previously implemented in Scikit-learn so the exact algorithm so you see that it's taking a bit more time but six seconds and this is with the original data set without filtering the sensor stuff and you see that it's already much closer to the diagonal and the test error is even smaller than with the neural network and it's easier to find the good hyper parameters with this algorithm so also something that is interesting to note is that if you serialize the trained model it's not very big, it's 1.5 megabytes so it's easy to store on your disk and to deploy on servers to load in memory on many servers to run predictions on a compute farm or even in mobile phones and all models most of the time and if you make predictions you can time how to predict for a batch of 100 samples you see that to predict 100 houses it takes just a couple of hundreds of microseconds so less than 1 millisecond so they are also very fast to predict so this is a good feature for deploying this kind of model in production you can contrast this with the random forest which is another way to build ensembles of decision trees more traditional and maybe more popular in the past so from a training time point of view it's kind of equivalent from a test error point of view you see that it's very similar it's 0.19 and here it was 0.18.7 so it's very close typically gradient boosting tends to be slightly better than a random forest but the big difference is the size of the model that you get for a random forest you generally build much deeper trees so they take a lot more space in memory and furthermore they take a lot more time to predict you see that it's 100 milliseconds to do the same prediction so it's 100 times slower than the gradient boosting trees so this is why typically in production people tend to favor gradient boosting trees rather than a random forest you can reduce the inference cost basically so now in scikit-learn you also have this the new histogram based gradient boosting trees so you can do the same and what you will observe is that typically for this small dataset the training time is very similar because on small dataset it doesn't make a big difference the accuracy is very similar to the traditional gradient boosting algorithm the model size is slightly bigger because just for this run maybe it has built slightly more trees I don't know exactly but this is not very important and it's quite fast also to predict something that is very interesting is that if you use that on a dataset with millions of data points then the traditional method that I've demonstrated before would not work at all it would crash, it would use too much memory and it would be much too slow so one would have no problem with tens of millions of data points it would take between tens of seconds and a couple of minutes depending on the high parameters and something else that is very interesting with histogram of gradient boosting trees is the ability to do early stopping so remember that I said that we train the trees one after the other to try to fix the errors of the previous tree so when we do that we can keep on monitoring the train and validation error of the model so this is on the x-axis it's the number of trees that we put in the ensemble and on the y-axis it's the score function basically the higher the better the negative error in this case so you see that when you add more trees the training accuracy basically increases and the validation accuracy on the held out validation set is also increasing but at some point it's reaching a plateau it's making no longer good significant improvement so what early stopping does is basically computing this accuracy on the validation set after each tree such that whenever you detect that you are reaching this plateau you stop and by doing this here you see we stopped before reaching 100 trees so it's smaller than before we can build smaller models that are faster to train because we do not need to go to the end and faster to predict and smaller to store in memory or on a disk so it's very good to use early stopping in practice okay I will stop here for the demo so that I have some time left to talk about other things in Scikit-Lan so in the previous release of Scikit-Lan 020 that was published in September or October last year there were also a bunch of very cool features that some people might not necessarily know in this room and in particular one that I would like to highlight is the column transformer and there was a lot of effort done to make it much easier to do feature engineering typically on heterogeneously typed typed pandas data frames with columns with categorical variables columns with numerical variables different distributions and so on there were also other improvements but those are the ones that I would like to emphasize so for instance here we are using pandas to read a CSV file from the Titanic data set so it's the list of all the passengers of the Titanic with a specific column and to highlight whether or not the passengers have survived the Titanic or not and so we can introspect the different columns of this data frame and look at the data type the data type of the data frame whether or not they are integer or floating point values and if they are integer or floating point values we say that this is a numerical column numerical data if it's not integer or floating point for instance if it's a string, an object or something it's probably the label of a categorical variable at least it's a case on this data set so because of that we decided to split the columns into two groups the numerical ones and the categorical ones and then we can use this make column transformer method to define two different pipelines one that will apply for the numerical values and one that will apply for the categorical values so for the numerical values we will do a missing value imputation with the median whenever there is a missing value in a record in one of those numerical columns we compute the median for the non-missing ones in the same column and we put that instead we can also insert a new indicator column to say that we have done this imputation and then we will use the numerical preprocessing tool which is called Kbin's discretizer that we present in the next slide so this is the pipeline for numerical values and we can do a similar pipeline for categorical values but in this case the missing values we cannot compute the median because a category is just a name, a label so in that case we just fill in with a constant value which is missing and that's enough and then we use the one hot encoder dummy categorical value encoder and then we can combine the two pipelines using the column transformer and call that the resulting operation the preprocessor once we have this preprocessor we can make a final pipeline that stage the preprocessor first and the classifier second the logistic regression in this case and we can pass this full pipeline to a cross-validation procedure that will do the model evaluation of the full set of modeling decisions basically and if you do that you see the accuracy that you get in this one which is like a very good baseline for this data set and you see that the code that we have tried to do this level of flexible preprocessing is kind of limited and still very easy to understand so the numerical preprocessing that I mentioned is the cabins discretizer so this is interesting because on this there are three different data sets so the first data set you have the purple points and the green points that are arranged into overlapping folded half moons so the goal here is based on the position in this 2D space the position is the input for the model and the color of the dot is the output that we want to predict so basically the goal of the model is to generalize the color basically to all the possible locations in this space and on the first column you see that we fit a linear classifier which is basically a linear classification model so it will try to find a linear boundary into that 2D space to separate the two classes and when the two classes are overlapping like this you see that it's not possible to use a linear model for that for the last row you see that the two groups are approximately separable so in this case the linear model is optimal but for the other data sets the first two lines it's not possible to use a linear model to get good performance in this in this case but what we can do is use the K-Beans discretizer as a preprocessing stage before the linear model and in this case we will group values between ranges of possible values and output more features so we will generate a higher dimensional feature space and if we do a linear decision boundary in that higher dimensional feature space then it corresponds to the linear decision boundary in the original feature space and this is what we observe the second column is the combination of K-Beans discretizer with linear regression and you see that logistic regression and you see that the quality even on nonlinear problem is very good and this is a very fast model so you can compare to gradient boosting and support vector machines that are also nonlinear by nature so they can solve this classification problem quite easily and significantly faster and smaller to deploy so in this case it would make sense to just do this kind of preprocessing plus a linear model and that would be enough so there are other features that were introduced I will skip those I will just highlight that when we worked on that we also fixed a lot of things directly into the Python standard library for the serialization of large objects with non-py arrays so it's also important to keep in mind that the Python ecosystem is a community and when you work on your project sometimes you want to quickly fix and hack stuff on your project but sometimes it's better to make the investment to fix the problem upstream so that other projects can benefit from the solution so this we improve the PICON module over a span of two years and it's not going to be integrated in Python 3.8 so I would like to just go to the end and thanks to the partner of the INRIA Foundation who supported this work and thanks more generally the scikit-learn developer community and users community so you see pictures of last couple of international sprints in Paris, in Austin and in New York and thank you for your attention maybe we have a couple of minutes for one or two questions so there is a microphone here hello, thank you for the presentation are there slides somewhere and the iPython notebook available? they are already online but I need to tweet the URLs I will tweet them my Twitter is ogrizel ogri is here and probably the conference organizers will collect the slides and put them on the page of the website and the second question you introduced a few algorithms one after another in a iPython notebook and Random Forest was the last one and you said it's used in production very often but the Random Forest often overfits from experience the Random Forest can overfit but it's not necessarily the case typically in Random Forest the more trees you put in the Random Forest the lower you will overfit so by putting more trees you can decrease a bit the overfitting you can also limit the depth of the trees using max depth in scikit-learn and you can also do some kind of feature selection to remove the features that are not predictive and that will help also combat overfitting so sometimes you have a lot of noise in your data and there is no perfect solution but generally I would say that it's possible to reduce the overfitting with the Random Forest the main problem with the Random Forest is that there are much bigger models and they are slower to predict compared to gradient boosting like in the new scikit-learn or in XGBoost or in light GBM those models are smaller and faster to predict so this is why they tend to be favored in production Hi my name is Björn I have a question for your column transformer so basically it seems like you have now attached pandas so basically with sklearn you don't have to transform everything to like a numpy matrix before you input it and basically there was a tool called sklearn pandas this sklearn pandas was basically a big inspiration for the column transformer so we wanted to have something similar by default in scikit-learn that's really great because when I started to learn scikit-learn I was really a bit put off by that it makes it much easier to do feature engineering from original data that was loaded with pandas and in the future this could also probably be adapted to work with other kind of tabular data structures that's great Hi, thank you very much for the talk just a question regarding the support of categorical variables you said it's like ongoing is there a timeline? okay so there are two ways to support categorical variables either you do pre-processing like I showed with the column transformer and that will work for all the models in scikit-learn and you have much more flexibility on putting some business logic like filtering the rare events or stuff like that and we plan to improve so far we have one hot encoding for linear models on this kind of neural network we also have ordinal encoding which is better for decision trees but in the future we want to have impact coding that use the target variable to also find a good representation and better support for weird distributions of rare values and sometimes you could also have labels that are informative categorical category labels that have typos but are almost words and for this there are a third party project like dirty cats we plan to be used in scikit-learn pipeline to improve this and to do this kind of categorical variable pre-processing and then for decision trees it's possible directly inside the decision tree to deal with categorical variables so this is not yet implemented right now we are focusing on missing values and there are a couple of other stuff that have more priority like sample weights but I think this is after that in the coming months we plan to work on this Can this new algorithm work on GPU units? So for most of the scikit-learn solvers they depend on either an umpire and those libraries they do not support working with GPUs right now and furthermore some models will not really benefit from GPUs like for instance linear models they are memory-bound and they would not benefit from GPUs so there is no point in trying to use GPUs for them but there will be a presentation by NVIDIA later today and maybe another one on Rapids AI I don't know one on desk and one maybe another one on yeah I'm not sure you will see on the program or you can ask Peter here that basically provides similar estimators to scikit-learn compatible API it's not necessarily an exact drop-in replacement but basically similar features and some of them can really benefit from running on the GPU for decision trees for gradient boosting it's not always the case that running on the GPU will be faster so you have to keep in mind that it's not as for neural network that really like convolutional neural network really benefit from GPUs for decision trees and for linear models sometimes the CPU is good enough so yeah hi thanks for the very informative talk maybe you can quickly comment on regularization you mentioned L2 regularization in the prototype also planning to include the others like L1 so basically when we do the beaming preprocessing we are simplifying the representation of the data because we are decreasing the precision we are using 8-bit integers on 66 levels to represent the original values that were encoded in 64-bit float for instance and by doing so we reduce the complexity of the model and so this is a kind of regularization so for decision trees it can help a bit but it's not magical in any way so but sometimes you observe that by doing this approximate method you get better performance than the exact method so this might be a bit surprising but this is a regularization effect I don't know I think we should stop here because yeah we are running out of time okay thank you very much again hello everyone okay this works so today I'm going to talk about a new approach that we've been developing about how to handle relatively uncomfortable datasets I don't want to really use the term big data because everybody has their own definition of big and sometimes this is volume sometimes it's velocity sometimes it's complex structures that you need to maybe join 100 tables together to make one cohesive unit but I'm going to talk about handling volume on a simple machine that is probably not as big for example if you work at I don't know Google or Facebook and you have maybe a different idea of big data but data that is maybe making you uncomfortable with your data scientists that you want to work on your local laptop or on your local CPU at your employer so for example how many well I assume that most of the people here are data scientists or engineers or interacting with people like this so you have a few options here that I've listed so I think everybody can use a dataset that has about a million or so samples we can use the standard tools out of the books and we can even be sloppy with our programming here how about if we move to 10 million samples I think modern machines can handle this we have plenty of RAM now RAM is getting cheaper every day how about 100 million samples this maybe starts to get a little uncomfortable maybe we need to be really careful how we can do categoricals maybe we need to not load all the columns maybe we need to use binary format and how about a billion samples I'm talking about not going to the cloud just using your normal laptop or your normal workstation either at home or at work so this becomes a little bit of a problem especially if you have more than two columns let's say you have 50, 60 maybe 100 rows and you want to work and explore and build models and maybe even deploy these models in production and how about larger datasets so today I'm going to talk about a solution that we are developing and actively using actually to enable us to work with this kind of datasets without us being uncomfortable so I'm sure there are tools not I'm sure but I know that there are tools that you can go to the cloud and just rent a beastie machine or spin up a cluster but this already incurs additional costs time management and if you just want to explore your idea and make sure that this works and if it's worthy you don't always want to go to the cloud and sometimes I came from academia you cannot always justify to your employer oh give me this much time and some Amazon server and maybe this will work but I'm not sure so I'm Jovan, hi I used to be an astrophysicist in Groningen which is a small town in the north of the Netherlands now I'm a data scientist at CVL Labs where I work on DevOps pipelines and when I was an astronomer I met Martin Briddles and we were postdocs together in the same group and we were developing this VEX package because we needed it for our work as I'll show in a bit Jonathan who is a more experienced data scientist and he thought that what we're doing was really cool and helped us transition from academia to industry so making our package more standardized following pandas and cycle learning API and standards to basically enable all data scientists to use this not just a bunch of scientists in some little corner of the world and Mario is really using our tools to create cool dashboards without caring about scalability and data sets so me and Martin were really lucky when we started working together we worked on this data set that came from this gas satellite that the European Space Agency launched it was 20 years in the making and we were really really fortunate to be the first ones to explore it but the thing is it was kind of big for a bunch of academics it had over 1 billion stars and when you give stars to astronomers we want to plot them and see how the sky looks like and we tried to do that and this is what happened so our boss was not very happy because she spent 20 years trying to produce this data set and we couldn't even visualize it so what actually we thought about instead of just plotting points that will end up overlapping each other how about making histograms and when we were starting to make histograms we started to see structure the more data we added the more structure we could see and we figured out that actually you can build histograms really fast and you don't really need that much memory so this is the foundation of the VEX library really fast computations aggregate data that will now enable us to use as big data as we can fit in our hard drive so this is where my presentation ends and I will switch to doing actually a live demo I think the best way to showcase something is to basically show it how it works in practice so is this visible enough yes, no, okay so we took great inspiration from Pandas really Pandas showed us the way how a data firm library should look like and we really tried to follow their example in the way the API design and as we developed we basically needed more things to do our jobs we started to add machine learning capability and that's where we followed the scikit-learn convention so we're almost fully scikit-learn compatible so let's try to let me try to show you to demo VEX for you so VEX is built on few concepts these are memory mapable storage expression systems delayed computations and efficient algorithms so as I will go through this demo I will try to explain this concept and later on we'll see how this works in practice so for this demo I will use this very large dataset from the New York taxi yellow cab taxi company it basically stores all the trips that all the yellow cabs did in New York City between 2009-2015 I mean you can get the data until I think 2018 but something changed in their data system after 2015 so I decided to ignore that portion of the data but this is big enough so let me just show you how large this file is when I merge all this data into one file it is about 100 gigabytes and even though I have a very nice macbook but it's kind of off the shelf and I definitely don't have 100 gigabytes of RAM to play around with it but I will play around with it so let me open this dataset because this HDF5 allows memory mapping all what this means is that when I try to open a file when I opened it it doesn't really do anything it just points the operative system and says oh look the file is here and this is the structure there are a bunch of columns and a bunch of rows and when I want to display it all it does it gives me a preview so I get the first by default the first 5 and the last 5 rows for the entire thing I have over a million rows I can see them instantaneously because I actually do not need to see all of them I just want to see a preview so you can open in quotes 100 gigabyte file well instantly well we have you can do the dataframe.info gives you a little bit more info you can put custom metadata information you get information about the column structure, their type and again a preview so vex follows the standard dataframe API as I said showcased by pandas so for example you can see a specific column in this file let's say the trip distance and again even though I have a billion columns billion rows sorry I see this instantly because all I see because of the memory mapping I see just a preview the first and the last 5 examples data type so for example this is daytime this is the pickup daytime of each toxic ship in the dataframe but what happens when I do certain operations these are also instantaneous and this leads me to expressions so when I do certain certain expression by expression I mean a really mathematical expression so here I take a log of something and I add the square root of something else I do this on the fly what vex does is memorizes the expression so a formula that is required to generate the result and only evaluates it when needed this is kind of like a computational graph something what neural network libraries have been doing for quite some time so this is instant because all I need to do is actually calculate 10 values not the whole billion I will only calculate a whole billion when I actually need to know the whole billion for example calculating the mean or some other aggregate statistics and this expression is really the core the core system of vex so for example let's talk let's put ourselves in a position of a taxi driver or a CEO of a taxi company and we want to know what is the total amount divided by the trip distance this is nearly instantaneous because again I'm just calculating 10 values I can save this as if I would do I would do with a normal dataframe with pandas and here let's say fee over distance is what I call a virtual column because it doesn't store the entire output but it only stores the mathematical expression needed to generate this and now if I scroll to the far right I see this here so what happens if I want to calculate the mean well then I actually do use the full billion data set and because we use fast algorithms and everything is fully parallelized I can calculate the mean of the distance in I'd say 3 seconds so how about mean of fee over distance which is now a virtual expression so now I do this thing simultaneously where first I need to compute this fee over distance and then from that calculate the mean but oh there is a none and because this is fee divided by distance I see a none so probably there was 0 somewhere in the distance so let's do a filtering just as you would do in pandas let's select all the trip distances that are bigger than 0 and compute this and now everything is done for a billion rows and now we get the super fast 4 seconds billion rows, no memory used whatsoever so we can now do name selections name selections are really cool this is what hooked me onto vex you can specify a filter let's say trip distance bigger than something and bigger than 10 give it a name and then for every data that vex supports you can basically say what the selection is and get the result for this particular filter so you can imagine the flexibility where vex really shines as I show you with this skyplot slide at the beginning is bin statistics so we can count really efficiently as we saw so this is just a number of rows but we can count on a grid so on one grid counting this is basically we're creating a histogram we can say what the limits are and the shape of the grid basically the number of bins and then we can use standard plotting tools your favorite tool, modplotlib or whatever to plot this we can do this in two dimension so two dimensional grid and I'm doing this in real time for the whole billion sets of points so it took about 7 seconds and now I can visualize it and well you can kind of see New York and this is kind of Manhattan we offer wrappers around modplotlib so you can do this nicely and get a histogram really quickly with all the correct axes and labels and shapes and also for two dimensions and now you can see New York much nicer and you can see where people took their taxes from these are all the pickup locations and you can see all the streets it's a really nice graphic you can also pass the selections that I made earlier and how long would it take to make two images with billion points with different filters around three seconds so this really enables us to do interactive filtering, selections with basically a billion points so I'm going to stop there and this basically gives you the overall idea behind tex and how one would use it but what I would like to do in the next part is actually let's do a data science example because maybe this works really fast in a very curated because I was home and I was preparing to simulate this presentation but then I thought okay it would be more maybe convincing and informative if we actually do a quick data science project together on a big data center on my local machine in real time so now that the actual scary demo starts will be scary for me, fun for you and this is going to be a disclaimer so on on this demo I'm going to do some data science project with this texid data center in front of you and it's going to take as long as it takes I mean hopefully hopefully within the time that I have allocated so some steps may take a minute or two because I'm going to be using the full data set in real time and because I'm partly doing this to show you a nice data science project and partly to show Vex I'm not going to always make the best data science decisions in terms of cross-validation and maybe building the right features and so on okay so let's start oh by the way if you have any questions in the meantime please you can ask at the end but you can also stop me in the middle so I'm going to load the same data set again there 1.1 billion of samples I can print head and tail and see what the data frame contains we're going to go back to that later the data frame you can see here is roughly ordered by date so pick up date so I'm going to try to make a machine learning model out of this so I'm going to split it by time and here I've been really lazy so I'm just I know basically the instance is when 2015 starts and so I'm going to take that as a test set and everything before 2015 I'm going to use as a train set so I have this data set described and this is the longest part of this presentation so what it does for all these columns I think there are about 12 or 11 different data types so we have the vendor ID pick up date and drop of date number of passengers how the passengers are paid, trip distance where they were picked up from long into the latitude I'm actually not sure what this ratecode is and this other is whether they if they paid with a credit card whether the card was charged right away or whether the taxi had to go to basically a place with wifi and then the transaction occurred drop of longitude, latitude fair amount additional charges, taxes if there was any tip if they had any tools to pay and the total amount paid so for all of these columns with this data frame described basically computing the mean, median computing if there are any missing values mean max really gives you a nice overview of the data frame and this takes a bit of time because I'm doing it for a billion points but at the same time I can use this as an opportunity to show you how much RAM am I actually using so so far I'm only using 2 gigabytes even though my full file is over 100 gigabytes in size so from time to time I will switch here and you see that my CPU is running at full throttle everything is fully parallelized and it's going to take a bit of time maybe we can use this time if anybody has any questions so far or everything is clear or everybody is fully lost yes please, microphone please right at the start you said HDF5 allows you to memory map it you can memory map any file so does HDF5 is that specially designed to be easy with memory map applications well I don't know if that was the point initially but it allows us to do this I forgot to say we also support Arrow so the Apache Arrow format also allows you to do this and we're fully compatible so it just that I had this file a while ago so I used HDF5 but you can equally use Apache Arrow and actually with development of Apache Arrow it allows us to do more interesting things like store lists and dicks which are kind of tricky to store in HDF5 to make it memory mapable but HDF5 is not the only option so we actually would like to move more and more to Arrow as it gets better and more stable ok so this is done we calculated all the statistics for all the 11 columns in 2.5 minutes which is kind of impressive given that we have a billion points data frame and everything was done out of core so now we can do a little bit of a data science by exploring what everything is here so we don't really know much about this about this data set so let's go slowly over it first let's see if there are any missing values and this is basically what Pandas defined as a missing value in A so either noon P none or none here there is nothing there is one pickup latitude that's missing out of all this rate code I really don't know what it is I'm going to ignore this column for people that pay with credit cards I guess this is not important if they paid right away or they were charged a bit later drop of longitude and latitude and some of the tools are missing so ok I'm going to drop the missing values from the longitude from the drop of longitude and latitude and a pickup latitude and because every operation they do is a virtual column or it's an expression nothing gets down right away as you see I've dropped columns immediately but I'm just storing the information I'm storing the command saying drop missing values when you actually need to do some calculation for now just save the command ok so now I look at the number of passengers and I see the minimum number of passengers is zero so maybe somebody was shipping something that people were not traveling but the maximum is 255 so that's kind of tricky so if you're a data scientist and you're using pandas all the time probably your favorite method is value counts and I know it's mine and so it's really important to have a fast value count so let's do that and because Vex is really aimed at this progress bar so you can know you know more or less how long you're going to wait if you have ridiculously big data on a single computer at least so this is what we're doing now so ok 20 seconds and we see ok there most oh sorry this is the trip distance wrong thing let's do it again alright so mostly we had passengers we had taxi trips with what passengers 2, 5, 3, 4, 6 and after 6 we have many with zero so maybe taxis were used to also ship things and after that we have these spurious random numbers there were other mistakes or the taxi drivers were having fun taking notes so what I'm going to do is just take the taxi trips that had well some passengers and less than 7 so then we want to look at the trip distances we've already calculated this so this is just to show you the performance when we don't have a kind of categorical variable but the trip distances are just floats and it also works really fast so it took 20 seconds and we notice that there are lots of trip distances that are zero so that's kind of weird so let's make a plot we want to see the distribution of distances on the log scale and while there are even some negative distances and some very large distances and in fact if I just take the maximum this is a ridiculously big number in terms of miles this comes from the US so in fact this is 67 times the distance to the moon or half the distance to Mars so kind of large so probably fake so let's give us a limit so let's go from 0 to 20 and this is how the histogram looks like now and here I decided to let's say take all the trips within 10 miles it's kind of where the histogram flattens out I really want to focus on the core on the core of the taxi data set so what is the extent of New York this is the New York city cast but they really covered the whole New York I want to see so I'm going to plot the taxi trips the pickup locations but I really want to kind of play around so we have this widget with the help of iPad widgets and material design so this should be New York because we're really affected by some outliers so I can interactively zoom oops I don't have a mouse so I have to do this a bunch of times and this is not pre-computed as I'm zooming the grid the 2D histogram effectively is being computed on the fly you can see nice progress bar so I'm zooming, zooming, zooming you can kind of start to see New York and there it is and we can see that now I'm familiar with New York, I personally have not been this is Manhattan this is the rest of New York so really city caps covered mostly Manhattan rather than the full New York but that's fine so now we can zoom in even more if we want to explore these hotspots and if you look at Google Maps or some other map provider you can see these hotspots are actually either major hotels or train stations or intersections between buses and subways so this really allows you to interactively look at some plots and even for this big data without using much memory so then I played around with this and decided this will be my the edges of my box of New York I'm going to make a filter just like I normally would now I'm going to create some features I'm going to look at the mean speed so this is trip distance divided by the time it took and this is going to be in hours kind of a natural then I'm going to look at the trip duration in minutes so this is just the difference between the drop-off and pickup time in minutes and I'm going to look at this metric that we looked at fairly divided by distance because sometimes you can think of oh but more money is great yeah but maybe you need to travel somewhere very far and then you have expenses you lose time, you lose petrol and so on so really fairly divided by distance seems to be like a good metric to look at profit okay so now look at the histogram of trip durations I'm going to make another plot so now basically I'm computing on the fly this expression that I've just defined above and binning it let's check oh okay it's done so wow there are some really really long trips that make no sense I'm going to make another plot now a bit zooming in between 0 and 100 minutes and this is how it looks like so there is not really a clear cut-off here where I can say okay from here spurious results started to see but it's New York I highly doubt somebody would be a thousand minutes in a taxi so I honestly don't know how patient New Yorkers are but let's say I'll be generous and say between 0 and 2 hours so if somebody really wants to spend more than 2 hours in a taxi in New York then yeah my model will probably not take account into that so I'll do that filtering as well now I want to let's say I'm a CEO of a taxi company I want to see when they are the most profitable times so I'm going to just execute this so we don't wait while I explain I'm going to extract from the time date the hour of pickup same as you would do in pandas I'm going to extract the day of week the month here I subtract 1 so I get this original encoding by default so January is not 1 but 0 and then I will make a simple feature just to see to check if a taxi trip occurred on a weekend or not so basically if the pickup day was Saturday or Sunday make it a weekend otherwise in 0 so then I'm using this categorized feature method basically what it does is even though I have integers 1, 2, 3, 2, 6, 4 days and 2, 11, 4 months I'm using these categories so I can do easily beaning on-ins rather than trying to find splits and so on I can quickly plot because vex knows that these are our categories I can quickly plot a map of pickup time versus day of week so we can see that in the early hours on the working days there are not that many trips well in the evening as we go to the evening side there are more taxi trips and on Saturday and Sunday well this is actually Friday Thursday night because this is midnight and Saturday this is Friday night basically Saturday night we have the most most taxi trips we can do the same per month and for some reason Saturday in March is really popular for taxis in October I have not figured out why and in July and August there is not that many taxis maybe people are on holiday but anyway you can do this kind of one of grids really quickly took 30 seconds to bend this in real time basically so now I want to do some more drill down work I want to group by standard thing in Pandas by hour and see maybe if I am a taxi driver I want to see maybe I am a part-time taxi driver and I want to see when it is most worth it for me to do this work where can I get which times of hours of day I can get the most tip where I get this metric to maximize our shortest or longer strips where I can drive the fastest or slowest some ideas like this so this is the same way as you would do group by in Pandas and here I am just this is just a default plotting code so now we we are able to do this in just under just over a minute for a billion points so we can see mean tip amount people in the morning tend to tip a bit more people that travel just after midnight fair by distance probably this is affected by traffic jams around 3 or 4 when people leave work kind of makes sense mean trip duration or it is kind of inverse of trip speed that makes sense if you drive fast you get there faster and yeah makes sense in the early hours of the day 4 or 5 the taxis drive fastest because there is no traffic and the peak hours the traffic jam hours of the day well they earn more money but they drive slower we can do this same now per day of week and we see the speed is larger on Sunday I guess probably less traffic and if you are a taxi driver it is most worth it for you to do this on Thursday however I have not computed the error or standard deviation so I don't know how significant this thing is but keep in mind that this is computed for more than a billion points so maybe average is out density maps we already so we can do this I am just going to plot another density map so now it takes a bit longer because I have a lot of features and I have made a lot of filtering so we can see how the pickup density of New York looks like after we did all this filtering this looks like this I am going to skip this but oops for us maybe as taxi drivers we want to know where it is most worth it to pick up people so what I am doing here is I am actually picking up long into the latitude but instead of just simply displaying the counts I am trying to display the mean of fare by distance because if this value is maximum it is most worth it for me so I really now see the streets where I get the maximum of fare divided by distance so if I try to pick up passengers along these routes and this is one of the airports this is another airport and this is kind of a main vein of the city get the most profit so we have part of the exploration phase so now let's do a little bit of machine learning it's fun so we have a module in VEX called VEXML where we provide machine learning APIs that are fully out of core that can help us pre-process data very quickly so I have taken some inspiration from Kegel there was a similar competition to predict taxi trips durations for example or taxi trip tip amounts I can't remember so here are the ideas I'm going to make a custom function that calculates the arc distance between two points and to test it out we'll calculate the distance between BASEL and Utrecht where I work and this is in miles because it's an American data set and this is about right if you check Google you will see that this is but you have to be careful it's not the row distance but it's the actual arc distance so now I can add this as a virtual column in VEX and if I scroll far right this is a complicated expression it's a common cosine bunch of them are arithmetic it can take a while when we actually compute it so what VEX does is also it supports jitting it stands for just in time compiling and we can use NUMBA so basically this expression is translated into C so then when you execute it basically you get almost two times faster because all these low level computations are done in C we actually support CUDA as well so if you do instead of just NUMBA just CUDA you can use you have NVIDIA GPU you can use GPU for parallelized low level computations as well I cannot demo it because I have a Mac unfortunately so this is one virtual column that I'm adding I'm going to add another this is just to kind of calculate the angle another complicated expression that I have can also use git to speed up and now I can just display the head the data frame to make sure that I have all these new columns here that I've created and I can check how much memory I'm using 3GB so far nothing much that no other standard laptops can handle it so now I want to do PCA the reason why I'm doing PCA I'm going to execute this as well is if I look at the streets you can see that New York or most American cities are kind of building a grid and this is a tabular dataset and we learned from the scikit-learn talk earlier that really boosted trees really work well with tabular datasets so we kind of know that the first thing we're going to try is going to be some sort of a tree model and this is a grid so if the grid was like a chess board it would really help the tree splitting algorithm decide the best way to do splits faster so PCA does this it transforms the data well trying to align the data along the principal axis and we're hoping that because the map of New York is kind of like a grid it will align the city on more regular grid so everything is done here out of core it took about 30 seconds to compute and transform the PCA for the pickup location the pickup location and the drop of location and now I'm just spending a little bit of time generating plots I'm actually generating four plots all with a billion points in real time so it might take a little bit of time to do but I think it will really illustrate the idea behind doing this any questions at this point yes please so I have a question so maybe on the PCA you will get the same result just on a much smaller sub-sample do you have an efficient way to extract or run the algorithm on a sub-sample of the data yeah yeah so you can just slice your data frame as you normally would instead of slicing a sampling uniformly at random because maybe it has some time dependency you mean to sub-sample yeah randomly sub-sample we don't at the moment because randomly sub-sampling means generating in memory some index that you need to track and at the moment we really try to avoid doing this but you can we don't have an in-built method but I can add a prescription of how you would do this you will just need to pay the price of having one in memory in column okay so this is how it looks like and you see the city is now transformed into a more linear well the grid is transformed so it's not it's more along the principal axis if I was doing this for let's say to get optimal performance I would probably isolate Manhattan more and do the PCA just on this so I can get this street straight and out even more so then I want a problem for for some reason I want to use this is a feature of the payment type and I see that this is basically text and sometimes it's like all capital letters and sometimes it's it's mix of capital and non-capital letter so we're going to do value counts on this column but first I'm going to do string lower so that supports also string operations in a very fast way because we're using C Bidens to do all these operations in a fully parallel and extremely fast so I'm going to I'm trying to just get all the possible values for the payment type and from the documentation website of the taxi dataset I saw that these are these are the meanings behind well this is what the categories that they already had placed so there is something cash card I guess this is cash as well credit credit unknown and so on and because there are finite numbers of these things that I founded the dataset and very few documented types what I do I can just make a map just like you would do in Bandas and apply a map transformation really quickly and this is using a hash table in the background so the mapping goes extremely fast and it's also fully parallelized and it's fully out of core so now if I scroll to the right I see the PCA columns transformed and this type and once again I'm not really using so this is one of these two actually this one is from the previous notebook that I showed and this is the current one just because the question fits here how do you restrict memory usage can you run out of memory still how do you restrict memory usage I'm fully doing memory because everything is done in chunks so most operations support chunking so I don't have to hold anything in memory I hold as much as around my half I calculate the statistics that I need select the next chunk update the statistics and so on so there is no actually you don't if I had 8 gigabytes or would make no difference actually most of these things are CPU reading well CPU limited or SSD read speed limited I'm going to use some okay some estimator I'm going to use light GBM so in Vex we provide bindings to various this is some because I've not installed it properly or something so this is not a fork it's just binding to light GBM if you install it using conda or pip or from source it will just work so I'm using some default parameters I'm instant instantiating the wrapper it supports all the default the default parameters that you would use normally or if you use the secular API you don't have to use a wrapper you can just use the secular API and because light GBM is still not out of course so this I'm going to do I'm just going to take the first million samples so I can train this in real time and show it to you and this is me training it so it will take a second or two it's really fast for a million and I'm going to do a prediction so the booster has dot-predict method standard you get in-memory output of your predictions so this is I'm trying to predict the duration of the taxi trips but I have something transformed that usually scikit-learn uses for transformations of data in Vex what it means it adds a virtual column to the end of the data frame so you don't actually have to have it in memory you can only have it when you need it and this is really cool because now if you're suspicious maybe you want to add additional models to this it's quite simple because everything here we're just storing basically the model that generates the prediction and we can execute it anytime so we can let's say estimate this it doesn't matter these numbers are more or less meaningless because we've not done cross-validation but we can easily add a second estimator let's say we want to try a Lidia model just for fun but for a linear model we cannot use well trees work really well with label encoding and integers values but because they have ordinality in them let's one-hot encode all this payment type hour, day of month very quickly and we can also support we can also standard scale let's say the distance, direction treat duration and so on so we support a number of transformers exactly the full scikit-learn list because they have a big list, they've done a great job we support a subset of that that is fully out of core so it doesn't matter how many samples you have all this is if you're limited you can see my this is computer crunching numbers and this is the memory that I'm using only 3 gigabytes for the entire dataset and now once this is done I will print the head of the data frame and you can see all the columns that are added so basically when I do the fitting all it does, it memorizes the command and the meta parameters like mean, median, the number of unique categories that you have and only when you do transform it's when the actual transformation happens in chunks so that's how you get away from the memory problem as I said this is an honest demo so sometimes we have to wait for the good insight of the performance when you work with a billion size dataset how long typical operations that you may do will take and here it is computations were finished as in I scroll to the right and you see we have because I'm keeping everything in the same data frame I have nice labels of all the transformation I did so you can know at all times what is what and here are the standard scale things so we also support the transformers because they are quite complicated but we support a wrapper for all scikit-learn estimators so you can just install scikit-learn through KONDA or PEEP for your favorite way of styling it and you can use all scikit-learn estimator within BEX they will not be out of core yet but you can use them once you do your aggregation so here I'm just importing the linear model I'm going to select the features that I want and again I'm training only on 1 million samples now but when I do the transformation here I'm actually applying this train model on my full 1 billion data frame and here if when I scroll very much to the right I have the latest linear prediction but I also have the other prediction without wasting any memory well lots of columns right here from LightGBM so what this allows me to do is quick examples because all these are virtual columns I can just do a simple mean like this so this is the final part of the data frame of the demo so if you were bored until now and this is all kind of lots of data frames, expressions so on if I want to put this in production now I have to go back and leave the notebook that we all know and love with pipelines and transformers and all the rest but because basically what we're doing is creating a computational graph every operation that we saved so far is written, it's an expression it's a mathematical formula and that's what we call a state so we can simply save the state into a JSON format like this and now I'm going to, this is only going to take a second I'm going to open a new notebook new kernel, fully independent of the previous one I'm going to load I'm going to open the data set I'm going to open just the test set and you can see here that there is nothing applied it's just the raw data set that we start with the test set and all I need to do is load now this is like GBM loading load the state and all the operations that we've done so far will be transferred onto the test set so now everything is done for the first time so this is the test set I'm printing the 21 things, 21 elements and we see all the transformations PCA the one-hot encoding all the way to the final ensemble prediction so now we can also push this with our REST API so this is another way of creating a pipeline without actually explicitly creating a pipeline so in principle you don't even have to leave the Jupyter notebooks where you just can experiment, get immediate feedback and just try out more models and parameters I hope you enjoyed it, thank you very much now a few questions and we'll move on oh by the way this is all open source so for research, personal, commercial use we also run a consultancy but the software itself is fully open source and you can use and experiment with great talk, thank you I have a few questions about this last piece with the pipelines because you say you've stored all the computational steps one of those computational steps was for example to train and transform with a PCA so if I apply this same state to a new dataset, am I retraining a new PCA? no, because basically you do fit as if you do with a scikit-learn fit so you've memorized the eigenvalues, the eigenvectors all those things are stored so you're just applying the transformation to a new dataset as if you use scikit-learn so does the export then contain the scikit-learn pipeline? yes, so the export is just a JSON file that contains the orders of what you need to do but also these meta-values like means eigenvectors for example when we did the encoding the number of unique, well not the number but the unique categories all you need to do to transform the dataset that's great, thank you how does it compare to DASC? so do you mean DASC or DASC DataFrame? where are you? here just in front so do you mean DASC or DASC DataFrame? both how does it compare to DASC? if you mean DASC DASC is like noopy but distributed so maybe VEX will use DASC one day to become distributed so then they're not actually competing it's just so far we're using noopy for most of the computations but we have plans to maybe support DASC so you can run this on a cluster if you really want to go to hundr