Hey, I'm Vikrant. I'm a Python enthusiast. And currently, I'm building a startup in Singapore, basically trying to build a product catalog manager, and one of the largest global product repository. Product repository means product information for commerce. You might laugh that I'm trying to do something which Google gave up, but that's it. Before continuing this, you have read what I have wrote, the article details, right? That how I got into machine learning and what was the reason. So I thought that I will share my journey into machine learning with Python group here, right? So before starting machine learning, I think all of you are familiar with machine learning? No? Not really. So let's start with a very basic thing. If today, I say to you that I have a medicine which cures all diseases of this world. Will you believe me? Will you believe me? Oh, really? You will believe me? Very nice. But why I say this is because today, I see a lot of hype around machine learning and artificial intelligence everywhere, as if machine learning and artificial intelligence is a panacea of all the problems it can solve every problem of this world. There is one smart computer program running there. He will solve all the global problems for us. But obviously, machine learning is an advanced technology. Obviously, it's a very old technology. And previously, we couldn't do it because we don't have that much storage and computing power. Today, it is possible because we have storage and computing power. So in my view, it's more of an evolution than a revolution of what knowledge we have gathered across in mathematics, computational statistics, linear algebra, and applying them in modern problems and using the computing resources and storage to solve problems. Now in my talk, I will try to concentrate more on the realistic how I got into machine learning and how I use it for solving my problems. Probably it might help you to differentiate hype from reality. So let me come back to the definition of machine learning. What is machine learning? Machine learning is a subfield of computer science that gives computers the ability to learn without being programmed. That is basic definition of machine learning is. It's a kind of artificial intelligence in which computer programs use computational statistics, linear algebra, and through pattern recognition, try to predict something on an unseen data. In my case, what I wanted to do it is I want to learn from urban renewal authority data, which they release for rentals and purchases in Singapore and try to find if I have the address of the property or the location of the property, the square foot of the property, can I predict a reasonable rental for that property? Now, if you know machine learning is very good, but then machine learning is divided into three parts. One is supervised, unsupervised, and reinforcement, whether it is deep learning or whether it is linear regression and traditional machine learning, which you are talking about. In supervised learning, I train the program through a labeled data, and then it predicts for unseen data based on the training I have provided in this one. That is supervised learning. And this is the most common form of learning. Even AlphaGo required supervised learning initially. Then later on, they incorporated reinforcement learning. In unsupervised learning, it's still an evolving field. So right now, in unsupervised learning, if I don't have a historical labeled data, I try to predict still unseen data. It's mostly called clustering. It is mostly used for clustering. Then comes reinforcement learning. And reinforcement learning is quite hot these days, if you know Google has been using it. And I think in past meetups, there was one person who presented on reinforcement learning to learn the stock strategies. So in this one, what I want is I provide a reward function, and then I train the program to optimize that reward function. And based on that, then it chooses the strategy. So like a child learns about the work, we make our programs learn through reward functions. I think many of you might have read some literature on machine learning and a lot of things on this one. In my view, when I started my journey, I started my journey four years ago. I did not start with machine learning. My problem was my wife needed to organize one Asia Pacific conference. And she had one problem where she needed to make 1,100 people sit in one table in such a way that there is a cross department people. And for three days, no two person should sit together. That was the problem I needed to solve. They could solve it on an Excel file by arranging manually. There were seven people involved in this one. It would have taken them seven to eight days to do this. So what I did is I said, OK, I can help you with that one. What I did is I modeled the problem as a traveling salesman problem, prepared a matrix, and then I looked for algorithm to solve the traveling salesman problem. And the most common one is ant colony optimization. And the second one was simulated annealing. And I used simulated annealing because my data was a discrete data. The output of that one was an Excel file, which was not very good. I tell you, still she need to spend four hours to fine tune it. But instead of seven days, it's four, five hours of job of one person. So that's how I got interested in scikit, the Python library for scientific learning, actually, machine learning. Now, if you know basic machine learning, this are the following components of machine learning, any kind. And it covers up all gamut of machine learning and some part of unsupervised learning. First part is a performance component. What is a performance component? You know, in my case, I want to predict a rental price based on a house location and the square foot. That was my problem, right? So that will become my performance component. Now, target function. As I said to you that I define relationship between this and the existing data, how my rentals. Because I have the old rental data from Urban Railway and actual data from them. So how I use find a relationship between these parameters in this one, define a function. Then I use a training data. So I had a training data from Urban Renewal Authority, which I used. And using that data, I applied the function and tried to do this. Then the data representation. How to represent data? When I go into actual detail, I will tell you. But actually, in this problem, I needed to represent data as a matrix, as NumPy arrays. Are you familiar with NumPy arrays? So actually here, I represented it as a pandas data frame. And then from pandas data frame, I converted it into NumPy array and then applied it into the algorithm. That's what. Now, in any machine learning problem, the first five components are designed by human. So you can never take away the human component from machine learning. For AlphaGo, they needed 30 million sets of data and seven months of work by 30 top-notch scientists just to prepare the first version. So it is a lot of work, actually. Hypothesis space, the last part, is the hypothesis space where I'm trying to predict. So suppose in case of Google, they prepared a program to predict strategy for AlphaGo. The same program cannot be used for chess because it only solves the specific problem of hypothesis space. This was basically a brief introduction of machine learning. Rather than going into more theoretical aspect, I will go more towards the practical aspect of how I did it, right? Now, my problem statement I have already told you that given a URA data, I just wanted to fiddle with it and try to find out what will be the rent for a house given a certain location and this one. But before coming to that, my question was why Python? So first I will answer this question directly by this is the main reason why I always choose Python, the Zen of Python. It fits my philosophy, it fits my brain. Not every library is exactly following it, but this is one of the reasons I like it. I like beautiful is better than ugly, explicit is better than implicit, and I think all of you are aware, the people who work with Python should be aware of this Zen of Python. This is one reason. The second reason I choose it is there were very excellent data science libraries in Python, especially for data manipulation, like pandas, scikit-learn, and then they launched TensorFlow and Keras. Initially, when I started TensorFlow, I didn't know, I did not know all of this. I only knew scikit because I was looking for simulated annealing algorithm and that's how I started with it. The main problem, when you start with machine learning, the basic problem comes up is how to approach it. Now, my problem was I gave up my computer science like 19, 20 years ago, and since then being in a corporate job, I never need to do low-level mathematics. Now, going into machine learning, everybody faces the same problem, where to start and how to start. There are a lot of tutorials available, a lot of information available. I did one thing. The first thing is when I was solving that problem for traveling salesman, I realized one thing. If I can model a problem as a linear algebra matrix problem and define a result in a matrix, I can submit this data to the algorithm and then it will run and give me the results. So I brush up on my linear algebra a little bit using Khan's Academy and there was one course on edX. Very simple, very high-level, high-school level mathematics. I still do not understand all the internal details of mathematical details of each and every algorithm. So I still experiment with the models, I experiment with the data and see if it fits my problem, right? Now here, my past problem was this one. This time, my problem was that I needed to change the home. And there was a data. There are two websites in Singapore, PropertyGuru and 99.co. They are available where you can find all the available demand, which rental places are available, right? The problem is they are, the agents vary a lot, sometimes 30 to 40% from the actual rental data from URA, right? So I wanted to say whether what he's asking is reasonable or not. I wanted to do that. One easy way is I can go to URA website, see the property data and I can find out. So I go to URA data. Let me first show you the data. This is the, can everyone see this? This is actually URA data, property data. I do search by postal district. I do, because I live in district 15. So I'll just search for district 15 and here you can see. Get a connection, but okay. I can give you the raw data. I already have the raw data. I downloaded it. So actually this is the data. You get the, from the URA. What data you get is serial number, building project name, street name, postal district type, number of bedroom, but for non-landed only. And since I was looking for landed property, I don't get this data. I just get N-A-N-A-N-A. If you explore the data, then monthly gross rental and floor area, square, and lease commencement date. So this is the data I get from URA, right? So the first is I explore the data, right? So for exploring the data, I will use Python's pandas data toolkit. So import numpy, import path, import math, import pandas, and SQL. SQL learn is scikit-learn library of this one. What did I do here is just I have URA-landed data set, which I prepared and I call it as testing data because I combine multiple files. Oh, yes, yes, yes, yes. This is just a simple one. I was just looking at the data. So I first loaded the data into pandas data frame. It's a very simple. You go here, use the read CSV, and I just read CSV, loaded, and then I printed the first 10 head of the data. Now, when you want to define a machine learning model, first you need to find the relationship between what you want to predict and what you want to use for those predictions, right? There has to be certain relationship between them, right, core relationship. And sometimes the lesser, the number of features, the better it is. So now what did I do it is from the data, right? I removed the time dimension because I was not interested in time. Time was actually month by month data. That was month by month data. I removed the time and what did I do is this? Since I was just interested in building project name, street name, postal district type, floor area and monthly content. So what did I do it is I take the data and then created a new data set which contains only one, two, three, four, five, and six. So it created like an Excel file. If you take an analogy of an Excel file, from Excel file, I deleted few columns. I did some, this one, the problem which is there is that you need to clean up the data of it because the data you will get will never be cleaned up. There will be some cleaning job you need to do and sometimes actually 80% of your time goes in fitting the data properly and preparing the data. Only 20% of the time actually you spend on really testing with the algorithm and working with the algorithm. Right, so in this case when I got the data, I found out that postal district was actually a value which is a number, right? But actually it's a categorical data. District 15, 10, 12, 11, it's not a numerical data. Now, so first thing is I converted that number into a string. That's what I did here in the second part. And then I just described the data. Described, so now you can see here in the description actually it's on the upper part it is printing those 10, this one. And here it is giving you the 10. This is for district 15. So actually once you load the data into Pandas data frame you can do a lot of different analysis. You can generate a lot of different graphs, a lot of different this one just by single command. So here I just write the command your data.describe, right? It will describe me the data and it will automatically figure out which one is the numerical data, what is the standard mean, top 25%, 50%, 75% and max. So if you look at it, the maximum rental in Singapore in that was 54,000 per month. And below you see here is actually for district 15, the highest is 41,000, mean is 5,866. Because if you look at landed property in East Coast they are quite big actually. So this is just to explore the data, right? Now I wanted to apply this data for, I have got the data, right? I have analyzed the data. Now I need to prepare the data for the machine learning algorithms which are provided by Scikit-learn. Now how many of you are aware of how machine learning works? It's actually machine learning is not some magic. It's very simple actually that you have a function fx, right? And this function has many, many variables, right? You give some variable and you will get a result, right? But the problem is normally what you do it is that function can capture only single plane, x and y or x, y, z. But in machine learning through computational statistics I try to create a function which can give me the result which is very near to the actual result. And that's what they do. They optimize the function. Obviously there are a lot of complex mathematics involved but in reality it's just purely. I represent my feature data as a matrix. All the feature which I need to give it as an input then here is my result set. I pass on both of them to my algorithm and then for unseen data my algorithm will give me the result. But if any one of you have read a lot of material on the internet or place what they talk about is they talk about is that you prepare the model you cross validate the model you say what is the standard error of the model? But nobody talks about how can I save this model? How can I convert this model into an API which can be used by others? Because I have seen many, many example and nobody gives you end to end details actually. They only give you all very technical mostly talking about evaluating different algorithms and then giving you this is the result and this is the mean square error this is the square root error actually for that algorithm. So I wanted to go a bit more than this so I have done this one and this one. So now I go through one by one step by step. So in the first part is what I did was I, this is just exploring the data right? Now if you learn in machine learning machine learning requires one thing. Machine learning, oh, machine learning algorithms do not work with categorical data. This is the phenomenal problem right? I only have six columns in my file. How many I have? I only have six columns in my file which represent which postal district it is right? And even the square foot is not actually a correct thing it's a range right? So now my problem was this is a categorical data it is not acceptable for any of the machine learning algorithm they need actually numerical data. For that normally what you do it is either you can write a separate program you can write a separate program to convert your categorical data into a numerical data. How you do it is like this. Suppose I have a data. Postal code 15 monthly rental 6,000. Postal code 10 monthly rental 3,000 right? So this 15 and 12 will become columns and for one the value will become zero and for where it is applicable will become one. So this two value will become actually columnar data right? And this is what you need to do. You can do it programmatically or very good thing about pandas is it provides a very good function here. So what did I do it is? I did it and it did it. Now what I printed below I can show after I run get dummies I print two information for you to see actually how the data looks like. So if you converted to get dummies you can see here for each one monthly gross rental 4,300 7,000 for this 0,0,0,0,0,0 So this data is for actually so this one the second data is actually 4,000 to 4,500 square feet. It's not 4,500 to 5,000 square feet because it's zero. It's only one. So actually get dummies convert your categorical data into zeros and ones and zeros and ones and convert into numerical data. Just that's the step you need to do. And then it's very simple then after that. What did I do was what I'm doing it is basically I got the get dummies. I print two rows to check the data whether it is okay or not. In the third stage what I do it is basically this get data set category data set will be a numpy array. Once I got the numpy array this command will print the shape of the numpy array. How many rows and columns are there in that numpy array? Shape of numpy array. And head 10 will actually print the top 10 data of this URI. And then I also printed the shape of my actual pandas radio frame which I use. So if you look at below actually here is the data. This is my actual data 5297,6. Here if you see it will be, yeah. Here you see two rows 5652 column 5297.650. So as soon as I apply is the function six column become 652 columns. Now in 652 column one set is the result set which is basically actual monthly gross rental, right? So I need to remove this. So what I do it is now once I got the get data set I divided it into two parts x and y. Given this value this will be my result. So I divided it into two parts my data set. Here it's just a numpy array and I just use the numpy array facility to divide it into two parts. I will explain this a bit later, okay? So this I commented out. Now I simply from escaler I imported this elastic net and random forest regressor. Now if your problem is finding a numerical value then you will be using linear regression. If your problem is classification then you will not be using regression. You will be using clustering. K nearest neighbor or this one. The very classical problem for classification would be in spam, you spam and not spam. So you classify your mess into two parts. Basically that is a very classical classification problem. This is basically a regression problem because I want to find a numerical value here. So for them this are the two algorithms. Now what I wanted to do is basically I wanted to check which algorithm is actually good for this set of data which gives me least errors and this one, right? I want to evaluate before going further into details, right? So what did I do it is basically I define the models. Then what did I do it is I use scikit cross well score. I score. So what does it is basically is it divides the data into 10 parts, one part result, one part test and remaining part test and train data. So nine parts I use for training, one part I use for testing. And then it does it again and again, again and again, again and again. I define all those parameters. Cross validation means what it does is my data is 5297, correct? It will divide this into 10 parts, right? Now it will try to train the algorithm with the nine parts and then test against predicting the 10th part. Now the question comes up is it chooses first nine parts then it chooses the 10th part. It automatically divides it. I don't need to do it manually. If I'm writing all the machine learning by myself, I will write all this manually. But since Python provides all those facilities and all those libraries, I don't need to write all those code. I just use cross well function and run it. So if you see it, if I run it here, this is all the data and now you see. You see elastic net had average error 3388. Means 3,388 Singapore dollar error on an average, okay? Here it comes with random forest, 2095. So in your view, which one will be better? Random forest, right? And then what did I do it is, what did I do in the next part is, I don't go to the next part. I said you that I set this standard a scalar. You have 10 sets, right? Suppose you have an equation, 2x plus y equal to some result, right? 2x plus y equal to some result, right? X is 500 and y is only 10. X is 500 and y is only 10. So which will influence the final result more, x or y? But then a small variation will x will impact my result very heavily. So in order to scale the data to similar scale levels, I use this function standard scalar. It automatically converts the data into similar scale. And in, you know, Panda, actually if you look at scikit-learn, there are two kind of scalar operations you can do. One is the Robo Scalar, another is Standard Scalar. Now I show you, you saw this actually just now. If you look at it, average error is 338, right? Make the data standard scalar. You saw direct impact. My mean square error has reduced dramatically. This is how you need to do a lot of explorations and work to actually figure out which algorithm is best for me, okay? Now, okay, I have defined my algorithm, I have tested, okay, I say my elastic net is also good, random forest is also good, right? Now the problem comes up is I got the model. Now how can I use this model to predict my value, right? So here in scikit, all models provide you certain, some functions, right? There is a dot fit. Dot fit is basically used for training my model, okay? Once I have trained my model, I can use, if you look at here, right? I can use model dot predict to predict unseen value, and that is what I'm doing actually. So I take the URA data, I convert it into matrix, right? I do some transformation on the data, I run model dot fit, and then with model dot predict, I can actually predict it. But one thing you need to remember, whether you do model dot predict or model dot fit, in the older time, they can accept 1D array. They can accept 1D array. 1D array means what? Like, suppose you got district 15 from the user as an input. In district 15, 1500 square feet is what will be the price? What you need to do internally is convert this into a NumPy array, a data on which you need to predict, and then it will predict, actually. And here, what I'm doing it basically is, in the old version, actually it could, it could accept the 1D array. In the new version, they don't accept 1D array. So actually I need to just use the reshape, dot reshape, to give it the data. And actually you can see, currently the algorithm performs like this, if you look at the data. Actual data, here. Actually, average error is this one. Actually the actual original rental is 4,300. I make it bigger, if you look at it. Now you can see behind, right? Okay, so here if you can see, I just printed it for, basically I didn't have the time to finish the whole web front end and create an API, but it's very easy to do, once you have done this job, actually. Here, the actual rental was 4,300. The one predicted by my program is 4957. For random forest, it is 4,700, a margin of 10% error. I did not spend a lot of time in training and working with all the data, but you can have a fairly good predictions through this. Now the problem comes up is, you have trained the data, right? You have done this data. What you want to do is, you want to save this data, save this model, right? You have trained the model, you want to save this model, and then later on in another program, you just want to load the model, use it to predict, correct? That's the problem you would like to do, so that's something you can do very easily. Escaler, they provide JobLib. I just do one function, JobLib.dump, model one, ElasticNet model, just give the name of the whatever model you want to save. So it will save three files. One is actually the model information and the respective training data which was used to train this model. It saves together inside this one on your disk. Then what I do it is, okay, I show you here, is actually here. Now, I just give one example where I'm loading the random forest model from the saved model, JobLib.load, and then trying to predict using that model. The results are pretty quick. It doesn't need to go through training stage. So here you can see, and actually if you look at this result and the upper result is almost the same because it's the same model, I just loaded it. What you can do it is, in your practical way, when you work with building such kind of machine learning programs, what you will do it is, you will take your data, train your model, save your model, and then you will expose it as a web service. In the web service, you only get the input you want from the user. Now, one problem comes up is, I had six data, right? I have six parts. You can take away the gross floor, the actual rental. So there are five parts, right? Postal code, and building name, and the street name, correct, and the square foot, correct? These are the four parameters, five parameters I have. Now, users say, I only know district, in district, what are the price? If you want to solve this problem, how would you approach it? Can anybody answer? Actually, if you want to solve this problem, then actually, your problem is another problem. Because what you trained on is actually five features. You cannot get one feature and then predict. If you want to predict only on one feature, then you need to train your model on one feature. So actually, you will have seven or eight models behind, saved in your disk. Depending on the user query, you will invoke that specific model to predict. This is something, basically, I learned it through trial and error, because I did not find any blog post or anything explaining this one. And this was just for fun, you know, it's for my own home. So I did not use it anywhere. This was just for my own personal interest, I did it. Okay, this is done. Now, if I train it more, because I just use a very limited set of data, I did not optimize the data yet. But if I do that, the prediction will be more nearer. Currently, I took only 5,000 sets of data. And actually, URA data is around 200,000. Correct, correct, correct, correct. So it can be corrected, actually. I had to look for the exact parameter. My feeling here is, because the problem is, because you know the square foot. Square foot impacts the data very well, but I'm using it as a categorical data. You understand, in the prediction, why there is a positive bias is because if I have actual square foot data, right, it's a numerical value. Right now, my information, if I go through realists, and actually I subscribe to URA data realists and I get the data, I think my prediction will be more on both the sides. But you're right, there is mostly a positive bias, not negative bias. But if I change the training parameters, there was a negative bias. But then the MSC, MSC was higher. Mean square error was very high. So I change it to make it reasonably. I think negative bias is here, on the higher values. Here, here, here. Here there is a negative bias. So in sum, I find there is a negative bias, in sum there is a positive bias, but actual error is, as I said you, 2400. Okay, I worked with this one. So what did I do? It is I took the data, I converted into metrics, I dumbed the model, I used the model to predict. In this one, I use very traditional linear regression algorithms. There, I was just applying two. I did not use scalable vector machines, actually, SVR, SVR algorithm, because they took me at least 20 hours to finish one round. So I just didn't do it actually, exactly. Elastic net and random forest were pretty quick. And random forest give me reasonably good results, so I just relied on random forest to do my experiment. Now comes back is, then I was thinking, can I apply deep learning to this linear regression problem for housing data and try to see whether it works or is it better than my traditional approach or not. Since, again, this is again a demo. So I did not do a lot of time running on the epochs and all those optimizations which are needed to be done on TensorFlow to get the results, but I try to do it in a very simple way. How many of you are aware of deep learning? How does it work? Deep learning is actually a special branch of machine learning where you try to model your problem using similar structure as a brain. But actually what you do it is you have a simple function and you try to optimize that simple function and pass it on to layer from layer and try to optimize that function and then predict based on this. There is quite a lot of, you know, still we don't have a very sound mathematical theory proving why deep learning works. We do have some proofs, some explanations, some details, but we still in some areas we don't understand how it works or actually why it works. Actually the mathematical proof for linear regression and everything we have a pure mathematical proof how it works, how the data come, how we do a train data and we get the values. Anyways, let me reduce it here. So for this, I used a very simple way. Actually I'm not so proficient in deep learning so I looked at a lot of materials and then I find two or three tutorials which were talking about deep learning. They were quite dense so I just use a very simple Keras library. I use Keras documentation and based on this then I developed the model. Now, as I said you, in this one, this is my model. This is my neural net model in which I try to compile and I do mean squared error. This activation ReLU, kernel is a normal and input dimension 60, 51.  In this one, it's a simple neural network in which it's a simple one layer, one layer neural network where I define this dense function 651. Now 651 is actually the number of columns. So none input dimension. So my input dimensions are 651. Actually this one would be how many times I'm trying to use it to activate that function. This defines, if I make it 800, it will become a deep network topology. Sorry, wider topology. If I do 651, then in another layer I do 300, then I do 200, then it will become deep layer topology. In this one, activation function ReLU is actually defined in deep learning where you try to optimize this function. I have an example where in one of the blog, I was looking at an example in which they visually show how it optimizes the values. It tries to optimize the mean square error and the optimizer it uses is Adam. There is another optimizer which is called SGD, Stochastic Gradient Dissent. Obviously I don't understand all the mathematics behind it so you can consider me as just a user of this but I experiment with both of them to try to find out which will do a better prediction. So here I define my model, baseline model. And this is a single layer model. I can define deeper in other, they were there, I removed them because they take very long time to run. So model one is a baseline model. So I defined it as a function and then invoked it, right? Then what I did is x dot shape. This I defined earlier, x and y, if you know. Then what I did is just model one dot fit. The way to train a deep learning model is similar to the way you do machine learning model. Model dot fit, x comma y, epochs. Epochs means how many times I will be running it and what will be the batch size of the data it will take every time it runs. You need to experiment with these two to see which one will give you the most optimal result. You need to run it a lot of times to see what results it will give. So if you do 10 or 20, your result will be more varied. If you do 1000, 2000, then it will be better. But it takes pretty long time on my MacBook. So actually for the demo, I just keep it 150 but a 150 will also take a bit of time. Then what I do it is once I train the model I try to predict the data. But you say I predict the data, I did not test whether this model gives you what is the mean square error of this model in prediction. That's how you evaluate.  In this one, if I just do it, so it is using a TensorFlow backend by default. This is just printing the model details. This is the shape of the array X and Y. It will take a little bit of time to run. Did anybody of here tried using deep learning for linear regression? This was just my interest. This is the first time I also used. Because majority of the deep learning examples are mostly related to RNN or recurrent neural networks or concurrent neural networks talking about image recognition, working on image net data or mostly talking. There was not much examples on this part using it for linear regression. It will print the data, right? Now again, the same thing, right? You should be able to save this data and use the trained data later on, correct? Because you don't want to run it for 20 hours once you have trained the model. For that, actually here it provides, Keras provides a method called dot state. For this to save, you need to, it saves in H5-based format. So this and deep network tensor flow basic model, one dot H5 and it will just store it. And then what did I do? It is I used the same model, loaded it, and then predicted the render. These are the results from using linear regression in deep learning.  Now this is different from this one, correct? This is different from this one, right? And what I wanted to show you here is actually MSC. MSC is so high in this one because it's not optimized. I didn't do a lot of work on it. In the meantime, it will take a bit of time to run this one, although here if you look at this, now how to evaluate a deep learning model? For this, there is a function in Keras called Keras Regressor. That is what precisely I'm using here, actually. Keras Regressor, right? Now here, I change certain parameters to change, okay? I put epox as five, means only five times it will run. And every time, the batch size will be 1000. So then it finishes quickly. Although it will give me a very large MSC. The lower the batch size, the larger the number of epochs, the lower the MSC error. Mean square error, you can see the error. And this is extended deviation for using deep learning. After experimenting a lot with all these algorithms, for me, I found out that if I take URA data, the best way, the best algorithm so far has been random forest. Very reliable, very predictable results. At least I understand why and how it is doing it. Here, I save the data right upper side. Now, I load this data and run it. I'm about to finish, actually, for this one. Okay, here is the, in this data, you must have missed something. In this model, actually, you can't just take the URA data. Actually, you can combine URA data with the external data. If you know exactly the data and you can get the data at a similar time scale as your actual data. Give you an example, which I was thinking when I was working with this problem was, if you look at the housing rental or housing purchase in Singapore, the rental market is not driven by Singapore locals. Because here, the property ownership percent is actually 87.9, which is pretty high for a developed country. So here, the rental market is actually driven by either the PRs or the EPs or WP issued by the Singapore government, right? So if from MOM, I can take the EP data, right, and use it to train my model, my rental predictions will be a bit more aligned and will be able to predict based on how many number of issued pass, what will be the actual rental for that place. That's one. Second thing, purchase data in Singapore, property purchase is influenced by two factor. One is cyber, Singapore interbanking rate, which I can get the data, incorporate as one of the feature in this one, and then add this to my existing data as one of the columns, right? And then that becomes part of my machine learning. So that parameter will be covered up and cyber definitely impact the purchases of properties in the future. Because at present, this month, I change my cyber or there is a minor variation in the cyber, two months later, it will impact my property. So, you know, actually the thing is what I learned is that before coming into machine learning, the first thing you need to define it is the problem you want to solve. Then second, see what data is available for this problem, for you to work with, right? How this data correlate to the actual result you want to get it. You don't have to have a complete idea. As I said, you don't have to have 100% idea because as you can see, I can run various algorithms, I can evaluate the efficacy of an algorithm by seen data and unseen data. And the more data I get, the better the results get. I think this is it for my presentation and this is what I wanted to cover up today. Hope you find it useful. Obviously, in my next, I will try to develop a web service based on it and a user interface. So maybe next time, then I can cover up that part. But for time being, this is, any questions?


Well, hi there, I'm Sandy Alnok and today I'm going to show you a happy accident. I know some of you love to see that people like me screw up all the time, so this one has been recorded. And I'm going to be using Poinsettias from Colorado Craft Company from last week's brand new release. And I'm going to show you just one pedal so you can see how the pedal is done and then I'm going to zip through everything else because we want to get to the happy accident and the stuff that happened after that. So I'm starting by covering the whole pedal with a coat of just the base color, the lightest color. And then going in with a dark color, R89 is one of my favorites for using with reds, but you can use an RV99, RV69, it's just something really nice and dark. And then I'm using a lighter color than that right next to it so that I soften the edges of it. It's not just big strong lines. I want it to feel like it's one kind of mushy soft pedal because those pedals are really soft. And if you have too much hard line in there, it's going to look like it's, I don't know, shredded paper or something that can look really funky. And so then I stepped it up one layer a little bit lighter and so that's an R17 and I'm leaving some openings so that I only have one layer of the R14 showing. And then when I kind of try to smooth all that out so that I get something more uniform, I've got all of this nice blending going on underneath. So I've got the dimension carried through from that, but I could just kind of go over it real quickly with the R14 and it sort of ties the whole pedal together while still leaving all of that detail in there. So I'm going to zoom through the rest of coloring this one flower so you can watch it develop and watch how adding the contrasting color is what makes the definition between all those leaves. If you don't have enough contrast, that would be why your flowers don't look dimensional. So you want to have enough dimension in them. So I'm doing the dimension on each pedal first, doing the same process that I did with the first pedal and growing the colors little by little for each one. And they kind of start to smooth out. You can see they're all kind of coming to the point where the first pedal was. But watch what happens as I go in now with some darker colors. I'm going to take some RV9, no RV69 was what this was, and I'm adding some really strong like darkish shadows, not as dark as the R89, but darker shadows and then working my way across each one of the pedals because then each pedal feels like it's got a whole bunch of dimension going across the whole thing as well. That's just a lot of layers of color. I had this idea that I was going to do some faux watercolor, like making Copic marker look like watercolor. So I was going to put down a whole bunch of dark blues in here. These are all from the B9 family. And I wanted dark color right around the flowers and then merging toward lighter in the center portion. But I had to kind of work at it and as I said, it's lots of layering and layering of colors to try to get the color to go from dark around the edges to lighter in the middle, but not lighter in terms of light light, but lighter in terms of a medium kind of color. And I wasn't worried about the blending being perfect because I was going to put in these faux water droplets, almost like your paint kept making blooms and blossoms and that sort of thing. And using different B99, B97, B95, and just trying to make something look like there were all these little dots in there.   But I turned it over and look what I see on the back.  Well, that gave me the idea since I kind of couldn't figure out how I was going to rectify that flower. I wanted to see if I could push the color through to the other side by adding a lot of colorless blenders.  So then I put more on. It just started playing around with it. This is what I talk about when I say, don't let any piece of paper go by without learning something from it. If you can add something to it, just try and see what happens. If you soak it with colorless blender, then see what happens. Now you could also create beautiful pink flowers by doing this same technique because look how beautiful those pink flowers look as well. But on the other side, I was getting this really rich color and this is much richer than I had when I was working with just the straight marker color and using that on top and I'll show you at the end the two cards that I made and you'll see the difference between doing it this way and the other way. So I spritzed a little bit more, found a few areas that I needed to kind of go back and push a little bit more of that color through. And I'm just doing this on a craft assistant. It's kind of a craft mat, a black metal craft mat thing. I'll have a link to it in the doobly-doo if you're interested. But the edges of the flowers are really soft and then not all of it filled in completely. So I took my markers on the good side, quote unquote good side on the side I liked and started adding in more marker. So that ended up filling it in so I could put it on a card and I did decide to take a gold uniball gel pen and outline each one of the stamps just in a really loose way, not even trying to match the stamp lines at all. But when you look at the two cards that I made, the top one is not done with this bleeding through the paper, the bottom one is. And the color is much richer on the bottom one plus I get the shimmer and shine of those gold edges on it. But it was really fun to see the difference between whether you just color on the surface or whether you're pushing the color through to the other side and making it really soft and kind of artsy fartsy and cool. So I hope you enjoyed that. I hope you'll get a little something fun out of it. I do have a mixed media Copic glass that I will link in the doobly-doo as well if you're interested in more techniques like that, check out that class. It's a whole heck of a lot of fun. And I'll see you again really soon with another video. Bye-bye.


A couple of weeks ago, the Portland Police Department implemented a program where our uniform patrol officers that work the street 24-7365 are now trained and equipped with individual packets of Narcan. And that's four milligrams, and really what Narcan is for a lay person, call it an antidote for opioids and an opioid overdose. And now they've received training, they have a certification that would allow them to utilize Narcan. When they run into these situations, what Narcan looks like is a small packet like that, very similar to something you may find, a flonase as an example or something of that manner. This isn't a prop, this is something that I'm certified to carry, something I carry for duty every day. So it's something that our officers now carry as well when they're working the street. The volume of overdose situations that our officers find themselves in in this community is really overwhelming. The opioid epidemic has impacted our entire country, and certainly right here in our own backyard it's something that we see on a regular basis. So in the city of Portland, a beautiful spot where we live and work and raise families, we had 46 fatal overdoses in 2015. The vast majority of them did involve an opioid of one sort or another, whether that's heroin and fentanyl or some kind of a mix, and it's killing people right here in our community. And what we found is that we had a lot of fatal overdose situations and a lot of substance use disorder stemming from prescription medications. There's been a lot of conversations, there's been a lot of work around prescription meds. So now as that supply slowly got weaned off and cut back on by the medical profession and law enforcement, individuals made personal decisions and business decisions that say that if I'm going to buy an OxyContin as an example illegally on the open market off the street, that may be a dollar or a milligram. So you may buy one Oxy80, 80 milligrams, that costs you $80. When you could buy heroin for five, seven, ten dollars a bag, you know, if you suffer from a substance use disorder which is a brain disease and you're driven not to get sick, you're driven not to detox in your bedroom or in the jail, you're going to pursue something that's readily available and cheap. And that happens to be heroin. And unfortunately heroin is now laced with fentanyl and other things that are far more potent than the original heroin back in the day or even recently. And now we have these fatal overdose scenarios because people really don't know what they're using. You can use for ten years and know exactly how much you can take and how much you can use, but if the next batch that you get is cut with something that's incredibly strong, you may not survive. And that's what we're seeing, acts every walk of life, every walk of life. Because it's the high school kid that blew his knee out. It's the soccer mom that they got OxyCotin for getting their wisdom teeth out. Back injuries, you name it, they can happen to anybody at any time. So we're finding that this runs the gamut from incredibly young to those that are elderly based on definition. It's impacting everybody. You have to have prevention, you have to have treatment, and you have to have enforcement. If you just focus on any one of those three legs without equal disbursement of resources and time and energy, then you're gonna fail. So we do need to focus on prevention. We've got to be talking to our kids as early as possible, as often as possible. It does focus on treatment. You have to have options. You have to have detox beds, and you have to have residential opportunities. And medically assisted treatment is incredibly imperative to this equation. And we need to continue to enforce the laws as they exist.  You need to be punished for that. So it has to be all three legs. And there's only 16, 16, 16 detox beds in the entire state of Maine. Now it happens to be housed at the Milestone Foundation, which is an incredible organization, a block and a half from us, head to East here from the police department. But imagine having 16 beds for 1.3 million people. And they will tell you that they estimate that they turn away about 100 people a month that are seeking treatment. And they just don't have the space, they don't have the capacity. They don't have the ability to treat everybody. There's some great programs out there. There's some good people working hard on this issue. The Greater Portland Addiction Collaborative is an organization with the backbone is Mercy Hospital and Milestone is at the table, the Portland Recovery Community Center ourselves. There's a lot of good people there, community housing in Maine. And that's a process unlike anyone that I've seen across the state or the country for that matter. And it's really geared to be a comprehensive community based sustainable model. And we need more programs like that. And the issue is money, it's resources. It's really, and I talk about it a lot. It's really moving beyond the interest phase where people want to talk about this to the investment stage where people actually want to do something about this. If you know somebody that's suffering from an opioid related substance use disorder, then the first thing I'd ask you to do is get educated. Learn really what you're talking about. Find out what your resources are locally. The legislation was recently passed that if as a family member or friend, somebody that's in that world, you can be prescribed Narcan. So you'll have that life saving drug available to you if God forbid you need it in your loved one you find them in the house or in car or in any location.  And then I would say on top of that, to be involved moving forward, hold our elected officials responsible for legislation and for the support of this population and of all of us. So I think it's very, very important that people reach out. They find out who their recovery allies are in the community and there are plenty but you just got to figure out what works for you. What community are you in and what resources are available. I'll tell you, we didn't carry Narcan for a long time simply because we have a full time advanced life support service with our paramedics and our fire department. We have an incredible partnership and there's really good people over there. So traditionally we respond to overdoses together at the same time when we arrive. But I will say that over the last four days we've used Narcan four times. Saturday, Sunday, Monday, and then we used it again last night. So it's those scenarios that we want to make sure that officers have the tools necessary again to do that job. If you can try to save somebody's life, whether you're saving 30 seconds before Meku gets there or a minute or two minutes or there are going to be 20 minutes. We don't know and I'm not saying that we're directly saving lives because paramedics are right behind us, but I think we are. It makes sense that if we have an opportunity to use this, that it's given these victims, it's given our residents a leg up for survival. And I think it's important that we do it.


started. Okay. Okay. We're live. All right. Hello, everyone. Welcome to the Wikimedia Tech Talk series. Today, we have Dan Andrescu, who is a senior software engineer at the foundation in the analytics department, telling us a little bit about what's going on in the analytics department. What is it that they do there? I am really curious personally as to what is the work they do. I'm really excited to hear him tell us a little bit about that. And if you have any questions, please use IRC and the Wikimedia Office channel and direct your questions at me. And I'll make sure to pass them along to Dan. And this is just a quick early announcement. Next month, we're going to have Alex from SRE. He's going to talk about the deployment pipeline. All right. I'm going to start this and Dan, you're all good to go. I was just launching it and my computer kind of hangs in there for a second while I'm launching it. Everything good? Clear? You can all see the slides. Yes, we can see the slides. Very good. Cool. So welcome. You know a lot about how data flows through our system or are completely confused about what streams and lakes and clouds have to do with anything. You have come to the right place. I'm going to take you behind the scenes and show you how we see our data. I want to have some fun. And just basically, yeah, let's look at what it is that we're doing back here. So mostly, we maintain about 111 hosts. And these have access to 2.4 petabytes of distributed storage. And together, this is about a tenth of WMF resources. So I think WMF has something around 1,300 hosts and 15 petabytes. We're a team of six engineers and a manager. Two of us are site reliability engineers.  They're amazing at reading logs, like the best. One of us streams in Scala and does massive distributed joins on skewed data for breakfast. I'm going to show you some of that later. And three of us are getting really good at reading logs, because that's basically all we do. The three of us have more of a front-end development background, and we work on some of the UI tools. But also a lot of the layers in between have front-ends, like there's five layers of front-ends. So we got lost in there somewhere, yeah. And our wonderful manager keeps it all together, while also in her spare time hacking bot detection and differential privacy algorithms, which is great. So on top of all the infrastructure, we build and run systems. These are, there's five different kinds of systems. I've got them kind of catalog here. We handle lots of data coming in. We've got Kafka, Hadoop, et cetera that handle that. We aggregate and sanitize it. A bunch of custom code that does that. We serve slow answers to really big questions with Hive and Spark. We give fast answers to questions that are asked frequently with our fast query systems. And we make pretty dashboards and graphs and things like that. Most of these systems are out of the box as much as we can handle. And we integrate them with custom code. That custom code is, I'm going to show you a high-level view of it, but it's everything from elegant Python scripts to thousands of lines of complicated distributed Scala and programming. Going into detail on much of this is kind of, yeah, like we could do it. And there's talks that do this. These are slides, pictures from other slides. We're happy to do that at a time in the future where other folks can. But I wanted to go up a level for this talk and look at high-level how the data flows. So I like this kind of corny analogy about water, because I like water. If you've seen me running through the rain. Streams of data come in just like streams. They get filtered through algorithms that get rid of data we don't need, clean the data that we do need, aggregated, kind of like a wetland does. Now, this part of our system is called refinery, because my team refused to let me rename it to wetland, but you know, it's okay, you win some. And finally, the data gets stored in like this big, peaceful data repository we call the Data Lake, where you can have a nice relaxing time floating around fishing for the data that you need. And some of that data gets published for everybody in the world to work with and you can kind of think of that as evaporating to the cloud. And there you go. I've tied a pretty bow around our whole world here. Yeah, so what's all this water? I'm here to learn about data, not water, right? So let's look at what our platform does. It's basically what we're trying to do is build a platform that lets people ask questions and get good answers to those questions. And that means the most important part of this whole platform is the question, right? Like, what is your question? And we all work together, WMF and the movement work together to ask good questions. The strategy processes and these things are results of us asking questions. And so the C team, the researchers, product owners, data analysts, legal, really everybody gets together and tries to find out what we need to know to make, to evolve our movement. Once we have that, we can instrument, we can add measurements, right? And measurements, we call that instrumentation, instrumentation panels on planes. And for example, we maintain code, the analytics team maintains code that instruments our varnished front ends and we'll be moving that to Apache Traffic Server when the traffic team migrates the cache and we're there. But most of the instrumentation is done by teams developing user-facing products. So you all working on visual editor and putting in logging code in there, that's most of the instrumentation, most of the logic that does that. And right now we're like in a WMF-wide effort, we're helping improve instrumentation APIs and build more standardized instrumentation libraries. So this all can be easier and we can just get data flowing in. When we have data, it flows in through Kafka, which is a huge distributed streaming platform. We have a variety of different clusters. Two of them are maintained by the SRE team with a little help from us, used for things like the media week job queue. Those Kafka clusters may be useful in the future for updating, for example, the complex dependency graph. So when someone edits a template, notifying all the places that use that template that they need to update is going to have to happen via a stream of events and Kafka can help with that. A big cluster in the last cluster is used for analytics data. It's a big cluster where we're getting all of our data. So there we're trying, we're in the modern event platform effort. We're trying to standardize how the data comes in, that it all has a sort of standard schemas that evolve in a dependable stable way on top of it so that whenever someone comes in and looks at any topic of data there, they know what to expect. Once that data is coming in, we have to sort of handle it, right? So we bucket it, we aggregate it, we clean it, we do all these things that the wetland was doing, and this is the place where we have most of our custom code. So until now we have a lot of custom infrastructure and now this is where we have a lot of custom code. So this system is called refinery. It uses a job scheduler to react to different data. So when data comes in, it sort of sends a notice that it's ready, like one hour of data is ready, and we operate on that hour, we aggregate sanitizers. This is a place where, for example, we parse geolocation and user agent information, reconstruct and reshape faulty data, and much more. And we can talk about that more. We want to think about kind of long-term what is this data, what is it going to do once it lands in a storage system? So one of the first things we think about is long-term retention. We decide if to keep, what to keep, how to keep the data. So we can keep longitudinal data long-term. We can delete after 31 days, et cetera. We can sanitize stripping certain fields or making them less granular, like, for example, the user agent. We don't store that in raw. We get rid of after 30 days. We parse it and try to determine operating system and browser information, and we keep that around for longer, 90 days. And then we delete that as well and aggregate the data and store that long-term. We're also looking at, we're investigating differential privacy and other privacy-by-design techniques. Marcel and our team gave a good talk about this. It's Strata, there'll be links to his slides in my slides as well. Now, once we kind of have the data cleaned up, aggregated in a way that we can use it, we have to decide how and where to store it.  So if we store anything long-term, if we have huge data that's queried as a whole, so for example, how many editors edit articles that get fewer than 10 page views per day over the past three years? Processing that, like an answer to that question, means going over about 14 terabytes of data. And we replicate that three-fold, so it's not going to fit on any one machine, right? To query that, basically, we have the Hadoop cluster, which has a distributed file system called the Hadoop Distributive File System, HDFS, and we put that data up there, it replicates it three-fold, and so now you have 50-some nodes that have copies of this data all over the place. When you launch a job, you have a resource manager that tries to put your logic as close to the data you're trying to query as possible. So that's HDFS, that's why we store it there. If the data is not too complicated, so things like if you have a table, it doesn't have too many columns, like the classic SQL tables, but it has a lot of rows, billions and billions of rows, but not too many columns, and needs to be queried very fast, then we put that in Cassandra, and Cassandra can serve queries to that data. It doesn't do any aggregation or computation on top of it, but can serve the data very, very quickly. Basically, it's a big indexing system. This, for example, hosts data that serves the page of API, and we're going to go over the page of API as an example to this whole flow in a few minutes. If we have not super huge, but pretty huge data that's more complicated, so it has many columns, but none of those columns have too many unique values. For example, like putting all of the user IDs or all of the page or all of the article titles would be too complicated of a column. If it doesn't have that, then we put it into it, which is a place where you can slice and dice data, really big data, terabytes and terabytes, but not super complicated, super unique data. There you can slice and dice by different dimensions. You can hold a property constant, like, say, the browser, and look at how different metrics fluctuate over the other properties of the data set. For a specific browser, what countries, what mobile versus desktop sites, where is the data being accessed? Drew does this with a clever index that he's talked about in other talks. Finally, we have to decide how you present this data. Our audience is pretty varied. It includes researchers, both internal and external, people downloading the data and working on public data, product owners in the foundation, the communications team, data analysts, engineers, and community members. A few examples like analysts will log into their cluster via Jupyter Notebooks, which can use Python to run Spark or Hive and query data on HDFS. They have full access to basically the big, slow engines there. Community members will look at Wikistats, which is a pretty UI for top-line metrics. Some community members download huge dumps of analytics data and build tools and workflows on top of it, while engineers like Eric Bernhertson on the search team will use Pageview data to train models that improve search results. We support different ways of interfacing with the data, and the same is our infrastructure where we try to use out-of-the-box tools. We do the same here, Turnilo, SuperSat. All of these things come out of the box with lots of value, and we try to contribute to those open-source projects. At the end of the day, we have to develop custom code. I'm going to show you real quick a few examples of these interfaces and how you can look at this data. The most general-purpose dashboarding tool we have is called SuperSat. It basically can access everything on our Padoop system, all the data sets there. This is showing geo-editors. The data set that looks at what countries people are editing different projects from. This is showing some of the top-wiki's English comments and wiki data, and how many editors are coming to those projects as well as from what countries are they coming from. Basically, if you want a dashboard, a custom dashboard that gives you something that you want to look at regularly, and this is the tool to use. Turnilo is a place where you go and explore, so you slice and dice data. This, for example, shows traffic for the last 30 days broken down by the browser that is being used. If you're trying to find a particular pattern in Firefox or something like that, you come here, you can break it down. You can break it down further by versions and all things like that. You can see the dimensions on the left side that you can split and filter back on. We have this little tiny tool called custom dashboards. This is very useful for canned data sets that provide a lot of value. For example, we released operating system breakdowns and browser breakdowns, and these are widely used over the internet to determine, you know, we're a big site, so people want to know roughly what's the most popular browser on Wikipedia and stuff like that, and this data set helps them do that. This is the interface I was talking about before, the custom notebooks you can see here. Basically, you're just doing very, very simple Python, like literally putting a SQL statement and calling Spark SQL on it and transforming it to Pandas and, you know, doing everything all the data science you want from there, and this is basically running in your browser through a tunnel, through an SSH tunnel and doing this work on the cluster. So you have the full power 15-ohm cluster right here, and you can do anything you want. And finally, this is our community-facing interface, WikiStats. We basically built this so folks can celebrate each other's amazing work they can find here, you know, up-and-coming projects can find Rises and active editors, registered users, they can run campaigns and try to see if it moves and needles and stuff like that. So this is really focused towards the community and top-level metrics like, you know, stuff that the media cares about and our communications team wants to share. It's all in a nice, shareable format. Okay, so that's kind of an overview of our pipeline and I can walk you through a little example. I talked a lot about page views to kind of give you context about this example. There's many other data sets that flow through our system. You can find them all in our docs or you can ask us, any of us and but it helps to just give one concrete example. So here's we have the question, how many times does content on our sites get viewed by hopefully human users? It turns out it's pretty hard to determine who's human on the internet these days. Surprise. So we instrumented that by going to our front-end caches. We have a tool called Varnish Kafka and it sends an event to Kafka every time Varnish services a web request and the data flows into our Kafka platform. We get about 100,000 requests per second, something like that. We get more from other custom events but that's just page views. Sorry, that's just web requests. Web requests are the raw sort of unrefined data. Those become page views later as I'll explain. And we bucket them by hour in HDFS. Now HDFS has for this hour, these were the web requests that the Varnish layer saw. Now we look at that and we do a bunch of things. First, we check for gaps, could there be hardware failures and data loss and things like that and we check for gaps and we alarm if there's any gaps. Most of our systems have data quality alarms that say, hey, this job didn't do its thing and we look into it, et cetera. Yeah, and we also clean the data so we take out any raw information like IP address because we don't really need the IP and we don't want to keep it. We basically want to keep just, we keep it for operational purposes but we delete that after 31 days. What we keep is the location, like the geolocation of the IP. We keep that around that's useful for analytics and we parse the user agent like I was talking about before. So we generally make the data easier to use, aggregate it and then we aggregate it hourly by page title because also we don't care too much about each request separately. We care more about what articles are being accessed and by what user agents and stuff like that. So we aggregate that that reduces the size and it allows us to store it long-term, free of any sort of private identifiable information and allows us to query it faster and move it to the other parts of our flow that can be used. The decision to keep the data, what we sanitize what we keep, we basically evaluate privacy risks whether you can de-anonymize a user whether you can reconstruct browsing sessions from the data that we have and then we take action so we delete raw data after 31 days refined data after 90 days and investigate, we're looking at other privacy techniques that we can use to make the data safer because all we really care about is getting our answers. We don't care about keeping data around for the sake of data. And for page views specifically the data is of course stored in HDFS and drew it in different forms and showed to different people but I just wanted to talk about the page view API specifically because it's relatively simple data so there's not a lot of dimensions. There's whether it's accessed on the mobile site or whether the user agent was a spider or probably a human we have few columns and lots and lots of rows so lots of data. So Cassandra is the right place we store it there and it allows us to query it very very fast and so we built a very simple Node.js service based on the Node.js service template that we used to use using REST based and that's basically the API that lets people access data in Cassandra. So that's basically the page view API that's used from a whole bunch of tools a whole set of tools both internally using that to combine it with other information externally to have some info pages of articles on how many page views each page had and look at project level page views and stuff like that and it's also used in other interfaces that we have like wiki stats and dashiki and other other places. So that's pretty much the page view example I just wanted to point out one other small thing is there's a tool called Matomo which if you have a smaller site that gets less traffic like for example the wikipedia 15th birthday website that we put up you just get a little snippet of JavaScript stick it on there and you get some automated metrics without a lot of work so you get unique visitors and things like that. So that's our system overall I wanted to highlight a few things just to kind of give you context so if you think in terms of infrastructure, puppet code servers, things that are hard for our 2SRE engineers to work on these are the areas where they're doing the most work so Hadoop and HDFS is just huge it has a lot of moving parts Kafka as well so that's a lot of custom puppet code Cassandra is up there because it turns out Cassandra is very hard to operationalize and to maintain it's complicated you have to kind of know exactly how it serves data quickly to tune it properly and yeah Kerberos is another thing we're working on right now to harden the security on our cluster so that's involving a lot of work and that's roughly where the SRE work is going and the majority of our code so our application logic is that we write I don't think we have more than 100,000 lines of custom code but it's somewhere close to that so it's not a huge amount it's mostly tuning things out of the box but it's not a small amount it's not like a small code base it's mostly some of it is data transformation we're processing everything from the data sets flowing into our own data sets that we're importing from MediaWiki so for example we scoop all of all data from all MediaWiki databases and aggregate it and clean it up and present it in a way that's easy to query and ask questions about editorship and retention and stuff like that to just yeah like all the data transformation that I mentioned that's all happening in a big custom code base and some of that is pretty tricky but I want to show you one example at the end of the talk that kind of gives you an idea of distributed programming and why that's a little weirder than normal database programming and the other we have a lot of Python utilities that kind of glue everything together and call out to different APIs to delete sanitize data all that stuff and our front-end so that turns out it's pretty complex the UI has a lot of custom code to make things look pretty and feel good for the user so that's another bigger code base and I think one of the last things I want to mention about the overall flow is tech that you should keep in mind kind of as you think about how you could benefit from the analytics infrastructure and systems and experience that we have I was just talking to Subu who's working on porting the JavaScript parser to PHP and part of that work is testing testing the results of both parsers to make sure that they match and that could involve a lot of compute a lot of differences being made and that's not at all in any way analytics related but the compute capacity so the cluster could help that problem very much keep in mind are the data flowing in through Kafka if you want to react to things in real time send notifications for different reasons think about how we compute and aggregate data you can think about that notebooks if you want to come in to the cluster and run a big job and share the analysis with other people in a friendly Python set of commands that's available and if you want to publish data sets so for example we're working with Wikimedia Deutschland and other teams internally who want to publish media coms data so how many times images and videos are being accessed we don't have very good instrumentation on that but we do have some data that we're going to put in an API and evolve the quality of the data over time if you have data sets like that that could be useful to you and others you can think about that I think for me that's pretty much a victory as I'm going to declare victory for myself or having gone through and explained kind of the overall picture everything that flows through here kind of goes through roughly that there's a lot of nuance there's a lot of stuff I skipped for the interest of Brevity but if you have any questions so far I'm happy to take them I'll tell you I have after this an example problem that we solve in distributed systems so I can either take questions now about the overall picture or just go through the example and take questions at the end I'll wait for Subbu to make the call on that I think the stream is at least 30 seconds behind but I did have one question I think so the picture was pretty impressive you have a lot of systems in there and you mentioned how everything starts with a question so I was curious how does this whole workflow work suppose somebody has a question how much of it is a lot of custom code how does it work how do you is somebody sitting with analytics people trying to figure out how to piece it all together or is there some kind of a language that people write code in I don't quite really follow that that's a really good question and I completely forgot to talk about how we interface with people that these are the most important part who this infrastructure serves thank you for that so yeah basically when you formulating that question that's like I said a teamwork effort and instrumenting is usually you know on your team so if you have a product you're usually instrumenting that yourself because you know your code base you know all that stuff we help you in terms of helping you define the schema and we're working on that process to make it tighter and a whole bunch of other people to make sure the schema makes sense and it conforms to all the different policies we have and so but once you have instrumentation and a schema picked out then the data starts flowing in and that's you can for the most part hand it off to us to take in process in a way that makes sense we have standard ways of processing this data we decide together how you're going to store it how you're going to access it where you're going to do with it we can define so like if the raw data is not very useful we can define data sets that you build on top of that that can be useful or we can load it pretty much as is with simple modifications into Druid we have like a direct to Druid so one of the most common instrumentation is event logging on your feature say someone is instrumenting and they can go into that data set can go straight to Druid and they're updated hourly and you can query and slice and dice right there from Ternillo the interface to Druid so that's all pretty much out of the box but you don't really have to talk to us at all if you have data sets that you need more custom attention then we can help you to a certain extent we have a priority list and you have to get in line to help you shape that stuff and load it on whichever platform makes sense for you to query yeah, does that help answer or do you have Yeah, that helps answer the question I have more but I can follow up later Andre had a question on IRC Yeah Is asking are there plans to update redirect or do something about stats.wikimedia.org which looks very static but at least what makes it hard to find things Yeah, so this this tool, this is wikis stats 2 if you go to stats.wikimedia.org slash v2 you'll see this we're being very like cautious about the quality of it it's definitely totally okay to browse but we're just being very careful that it meets all the community expectations and build up with the old tool so the old tool might look static but it's actually like a collection of brilliant work by Eric Zockta that a lot of people find extremely useful and they've sort of gotten used to it over the past 12 years that it's been up so we're displacing that very carefully and right now this project this wikis stats 2 project is in alpha mode it's going to hit beta sometime in the next few weeks probably and after that we have a bunch of other changes that we want to do over the next year and it's going to be released next year but I think when it goes to beta we're just going to redirect it automatically input about that is welcome of course everybody here I think uses wikis stats to some extent so it's all your tool like let us know if you think it's ready to go to this new version and stuff like that we find that feedback useful okay another question another question okay great let's go and take a look at this little example problem and then I can answer any other questions at the end okay so I chose this because I ran into it and Joseph who's the scholar distributed system expert on our team he explained to me what the problem was and it was very frustrating for me because I'm coming from like a more traditional database background and I was so confused why this particular problem happened and here I'll set up the context so we import a bunch of media wiki data let's focus just on the revision table and the comment table so recently comments have been factored out of revision and they're now linked by an ID and we have to join revision and comment together to get those comments because some of them are useful for the reconstruction we do we don't collect all the data together so when we import we don't keep each database separate we just put it all in one place and add a column called wiki db that helps us analyze everything together it helps us do cross wiki joins and stuff like that and it's a lot you know more helpful for analytics purposes so when we join those two tables we're joining all of the revisions with all of the revisions from everywhere and the problem is basically that the data is skewed so a lot of revisions have the empty comment for example let's pretend like the first couple revisions have comment 80 a and b and revisions 3 through 10 billion have comment 80 x which happens to be the empty stream so when you join this table what happens does it just take a long time or you know it just fails it just completely burns to the ground in a spectacular fashion and even though we have a big cluster and plenty of capacity to process all the data what ends up happening is that you can't parallelize this particular join and I'll kind of explain why so the data is skewed and we have many many billions of revisions on one side trying to join to one record on the other side and when we join we use when we join in a distributed system we use a technique called map reduce right so we sort of break the problem down into two steps first step we have a bunch of mappers that are running looking at a big distributed cluster and going to the data and saying I want this part of the data so that's where the filtering would happen just like you would do map on you know javascript.map and you filter what you need and you send it to another worker called the reducer and then the reducer does whatever aggregation type work like you would call reduce in javascript that you take multiple streams from different mappers and you do some work on it and then you output and then you combine the output from a bunch of different reducers and that's your final output so that's kind of roughly what that looks like you know you have a bunch of data distributed all over the place you have mappers that know how to fetch it they go in and they find rev comment IDX and comment IDX and they know that this is going to be a join so they take rev comment IDX and comment IDX and hash it by the number of reducers and send the result send the row to the reducer that the hash maps do right so like you're trying to distribute a big data set of millions of records into end reducers you're going to hash mod end let's say so that it lands on one of those end reducers and the problem that you run into with skewed data so when you have lots and lots of repeated values is that all of the comment IDX will hash to the same reducer and that reducer just burns to the ground just you know it can't and if you have 10 billion records you wouldn't it wouldn't right it would be fine if you had a million 10 million it would be fine the problem is it has so many records that it can't even swap them out to disk as it's processing them because the operation is too slow think about if you have 10 billion records and it takes 10 milliseconds if you have just 1 billion records and it takes 10 milliseconds per record that's 4 months of computation so if you can't for example if you thought hey this is kind of dumb I'm just hashing the same value to the same reducer why don't I set up some server somewhere like a very fast lookup server like a reddit server where I tell it I hashed, I just keep a count a tally of how many times I hashed X to a particular number and when it gets to a billion or however many records one reducer can handle it does like a little collision detection and starts hashing to a different reducer well that conversation over the network would take a few milliseconds and if you add those milliseconds together you end up with like months of lag so with this size data you literally can't do it so what do you what do you do how do you solve that problem how do you take the load off of how do you de-concentrate that load very ugly and not at all generic there is no generic solution that we know that I know of but one way you can approach it is you can sort of add a fake ID to one side of the join that you randomly assign a number between one and K where the K is like some constant that you magically came up with that works for your cluster whatever your cluster size is and then that's on the side that has the many duplicate IDs and then on the side that you're joining to you want to duplicate those IDs with that new fake ID so that there is a target for all the joins so you duplicate that K times randomize you know one through K values for fake ID over 10 billion records on the left and the revision side and you assign you multiply the command ID X field by K on the common side so now when the join happens all of the random one through K values from revision will find a join on the common side and it will be the same value because you're just duplicating the common test and when it hashes it instead of hashing just comment ID X it now hashes fake ID 6 comment ID X fake ID K comment ID X and those are all different hash keys so they'll get evenly distributed so now you've managed to evenly distribute your work over your reducer cluster and you don't you know burn one of them to the ground anymore so yeah any any questions about that do I lose anyone on that I don't know if I did a good job of explaining because you know like if you don't if you just learn something like that you don't always do the best job of explaining so I'm happy to go over something again or fill in a gap can you explain why you picked this example what what you're trying to illustrate with this example maybe that will help yeah so this is this is something we do we we ran into this problem when when this refactor happened so what we do let me back up so we put together a data set called the media wiki history data set that's all editing metadata from all the databases looking at the logging table finding where users change their name where articles change titles where users were blocked and unblocked pages deleted and restored revisions reverted all of the different things that happen on the wikis and we put them all in one table literally like one table with many columns that explain everything that happened to that revision and give context so for example one row would have the number of edits that the user making that action has done up to that point in their life it would have the name of that user at that point in time and the name of that user currently today you know that's kind of what the data set looks like and we build this by doing a complex set of aggregations on top of the media wiki data so when there's a refactor uh in the media wiki schema that affects us pretty deeply and this was one of those examples where we had to join to the comment table all of a sudden and not so much now but especially in the middle of that migration we ended up seeing a lot of skew data because you know not all of the comments had come in so a lot of them were null a lot of the commentaries were null not migrated yet and we hit that that problem and I wanted to illustrate kind of the difference between working with big data and how that could have unexpected unexpected nuances versus you know if you're just doing a join on a couple of big tables in SQL it might just take a long time but it's not going to run out of memory or crash some particular worker because it's just you know churning over it and stopping to disk when it needs to join and stuff like that yeah so that's why I picked this example that's helpful thanks okay um yeah so we keep good docs on all of our on all of our systems and infrastructure and procedures uh and if we don't keep us honest and let us know we'll update the docs um but feel free to visit them they're on wikitec slash analytics um and yeah thank you ask me anything look on IRC okay this is the very beginning is this recorded I think so this is recorded on screen oh awesome this is very helpful um and will you share the slides as well yeah uh I have uh I noticed that on comments there are usually pdfs um but I have the slides with the notes as well I'm happy to share that um because I think that's where most of my talking is so I'll do both awesome thank you I'm going to uh further share this with other people in products that's awesome yeah I hope that it helps and I thought there was a little we usually talk about more specific systems or more specific algorithms so I thought this would be an interesting overview yeah I know seeing the high level view and how it all works together is super helpful um I'll make sure my team sees it definitely as well as other people okay you know if you have questions go have the rest of your day go have fun we're gonna we're gonna meet the new CTO in like a few minutes right so wait what there's a meet and greet with a one of the CTO candidates sorry not the new CTO one of the CTO candidates I hope I haven't like someone's mind backs yeah alright thanks Dan and okay thank you thank you all yeah thank you