he's a fresh perspective from his conversations with many of people like you and in the industry can you guys hear me? yes this is both a conversation and a thread that has been running for about 3 years so much of what I'm going to talk about is drawn up on our own Scribble's own journey building a feature engineering platform and talking to a lot of customers I can see a couple of my customers thank you thanks for giving us the opportunity to learn these are the set of questions that a fictitious customer of ours had and these are questions that a lot of you might be thinking about because there has been a very significant shift in the last year, year and half or so until recently the conversation was about can we build models, can we build interesting models in which we have confidence and so on now the conversation is shifting to can I make them trustworthy can I make them robust enough so that I can run them every day and the business can actually depend on all of them so these are some of the questions my customers were looking at what Uber is investing Uber has 400 data scientists and 25 people team only for their ML engineering platform and saying is that the kind of investment why is it so complex why is it so difficult to build and on and on the idea here is that we will give you a perspective on this cost and complexity and dive into the feature engineering one of the components of the platform hopefully you will walk away with at least partial answers to whether or not you should build and what would it be, how could it look so little bit about us we are an ML engineering company based in Bangalore and for about we've been in existence for about 3 years let me jump right into it, this is a slide from Achal's presentation, Achal is from Uber's Michelangelo team he gave a talk very much at this the venue about a year back and Michelangelo stands out in terms of its design as well as in terms of its sheer capability 5,000 models crunching petabytes of data and serving insane number of requests what has happened in the last one year much of this was built in house but in the last one here company after company all the names that we are familiar with and who are doing great work have been coming out and discussing their the architectures of the system the most recent one being the Relyard from Stripe my strong recommendation is that these designs the ideas underlying them reflect some of the cutting edge thinking and we have personally benefited great deal from that and you should also look at them for inspiration now the question is we know something about Michelangelo which is that it took 20-25 people over a period of many years to build it the question is what did they actually built we can talk in terms of technologies but can we separate out the hives and the kafkas of the world and look at the core ideas when we started looking at comparing all the ML platforms that organizations have built over a period of time and we started arriving at our own understanding but luckily for us google has already articulated some of this it was this is a picture from a paper from new new IPS conference from 2015 it was somewhat controversial because essentially what this picture showed is that all the stuff that we talk about that is the blog material all the nice charts that we have talked about what it is saying is that it is actually a small fraction of the effort that goes into models into real production what is the rest of the system doing if it is not crunching statistics and it is not scikit-learn code and when we looked at them very closely we found that the reason all of these things exist is that there is a degree of uncertainty associated with this ML code and all of these components are about achieving four objectives related to the same ML code the first one is speed this comes from the realization that nobody builds one model we are going to build more models every passing day if it is going to take a long time for you to build something then the cost of actually impacting the business is the value of impacting the business is much lower than the cost that goes in so one of the major purposes of all of these tools is to put more and more models in production you may not be doing 5000 models like Uber but you are definitely doing 25 30 models every organization that we have seen that can conceptualize one model can easily conceptualize 10 models because they have the scale the complexity and the data meet all of those requirements the second problem is correctness which is that it is not enough to put this ML code into production many of them are making very risky decisions we have we know of one entity which can whose ML model can effectively deny you a telecom connection a loan they can mark you as a fraud these are risky decisions that have a lot of impact and side effects in the world and it impacts real people so you want to know that you can actually believe this ML model that you are putting into production so bunch of the tooling is around I know that I have deployed the right one I know that it is performing well and I know that I can trust it a month from now two months from now five months the third one is evolution this has to reflect this reflects the idea that our understanding of what problem you are solving and how we are solving is actually incomplete when we put the model into production it will evolve as you see more corner cases as you get more data so you have to take the entire system along as you version these ML models over a period of time and the last one is of course scale which is the most talked about which is that all of these are computing large volumes of data so you need to have some sense of some control and scaling ability of this but the challenge that we have is that Uber's ML platform is very very specific to Uber how much of it is actually usable by us the same as the case with Stripe and these are all Unicorns which have very very unique problems and so we were thinking about what is it that we can learn from all of these and our own experience because we were building these ML engineering platforms so it turns out that there was this fantastic presentation from Gojek at Databricks conference in April which had one nice slide and you can see that there is a broad convergence there are three big problems that are being solved by the ML platforms the first one has to do with the training of the models itself you know all the scikit-learn for tracking all the versions of it tracking all the accuracy levels and so on and doing this in a scalable fashion especially if it involves deep learning and so on the second part has to do with once we have trained the models and they are actually in a database of some sort taking it and putting it into production and making sure that it is running all the time and we are able to monitor the drift and whatever other aspects of interest of the model the third piece which is the feature engine which is that today with every what we are finding is that in organization of organization people have stopped working on new algorithms the set of statistical methods that you use is reducing and it is standardizing more and more of the model behavior is determined by the underlying data sets so there is an entire piece that is about generating and managing and tracking those data sets broadly we call it a feature engineering now you can map this into direct products in the market already MLflow comes in mind there is a talk on MLflow later and the Databricks folks are also outside the Kubernetes and the Kubiflow and there is a buff on that as well the part that we felt was did not receive any attention is the whole part which involves the data preparation or the feature matrix generation that goes into these models so the rest of the four talk will be only about this left most piece and we can talk about other things later on and there is a buff later today on ML platforms there are some fantastic guys from Walmart, Swiggy and other places who are coming and talking about their ML platforms please join us for that now the question is you know what is feature engineering what is feature engineering right this is how ACME's data actually looks these are just transactions somebody bought some product of a certain time but what we really want is a large matrix of different columns columns like is this you know what percentage of the transactions go to the premium products or exotic products and so on right now the big change the reason you need full-fledged infrastructure and for this feature engineering is that if you look at what has been happening the first thing is that the set of features that you are computing has been exploding earlier we would have 20 different columns 20 attributes now thousands is something that people are easily imagining and these are changing continuously so depending upon the number of models and the kind of models and the methods that you are using the other thing that is happening is that new data is coming in new customers are coming in every day so whatever be your feature data feature matrix that you are computing you need to work at a fairly high high frequency and there are extreme cases also we have an automotive customer where they are building hundreds of these models it is not even feasible for them to go and manually feature engineer their system so there is a lot of use of AutoML and automatic feature development so you have to see until recently this whole feature everything was subsumed by the data scientists and it was a part of your scikit-learn code with every passing day as you are putting more models and as the cost of this activity is growing it is emerging as an activity in itself so today Michelangelo for example has a full-fledged team doing this and that may not be the case in most of your organizations but you should see this as a fairly standalone activity and you will understand over a period of time in next few slides on why now when you look at the featuring systems that have been built for all of these organizations what we find is that there are two big drivers to decide what will be the architecture or what will be the flavor of the system that you are going to build one has to do with what is the sheer scale at which you are operating if you are operating at petabytes that is an entirely different system you have to engineer it to the maximum the second part has to do with how closely tied are your modeling strategies with the feature matrix itself in deep learning you may not be able to decouple these and most of us fall into the top left category we are dealing with terabytes of data and we are dealing with standard ML modeling now what we find is that this is actually a large space there are many more that are coming one immediate subspace that we see even within ML is we had an IoT customer that has a billion rows with five columns the way you will process that is going to be very different from our retail customer that has a million rows and 60 columns the computational requirements as well as the interfaces that you provide are going to be very very different and this is just the beginning depending upon whether it is on the edge or in the core in real time versus not real time whether it is automatic versus non-automated this itself you will see a bunch of new discussion over the next two three years let me jump into what we have found over a period of time and so essentially the whole feature engineering platform or service whatever we are talking about takes the raw data transactions, logs, event files whatever it may be and then turning that into large matrices which are called the feature matrices and there may be even a repository of all of this because oftentimes you want to know features at a point in time three weeks back when I trained this model the features those are the kind of questions and when you look closely at this you will find that there are actually three sub-problems even within the feature engineering the core it is intuitive and we understand what it is that there has to be one piece which has to do all the processing of the raw data and generate the features and it looks a lot like DAG because this code tends to be fairly complex in one customer case this is an IoT customer we had 6000 lines of pandas code and the code would run for about 24-48 hours and it is A not possible to unit test any of this so it had to be broken up into small pieces they had to be organized in a certain way we will get into a little more of this the second part as the risk associated with this modeling is growing you want to actually know that it has computed correctly you want to be able to access and look at any data set that has been used to train any model because if the questions don't come today they are going to come tomorrow it may be from your CEO it could be from regulatory authorities it could be from anywhere the second part has to do with we actually believe what has been computed an equally important piece is you know this is computationally intensive it is also a very painful process you first want to know that your data is trustworthy to the last customer every customer's data sources are static in some way or other in the case of our retail customers gaps in data duplicates errors changes in formats all of this is a given we used to think that sensor data is a little more clean because it is being generated by machines but they have their own set of problems the sensor getting locked at a certain place even the manual process involving the sensors themselves is uncontrolled for example we had some sensors coming from a mall the staff would walk around and turn off the sensors right so the sensor is not a problem the process around the sensor is still problematic in all cases if one take away one obvious take away that you know I recommend this be paranoid paranoid at every step of the way you don't believe what has been given to you don't believe what has been given to you so having a lot of checks and balances through the entire life cycle is what is required and if you ask what is it that Uber and really hard and everybody else does it is a lot of error checking code all over the place that's one way of thinking about what they do now we will zoom into each one of these depending upon how much time there is I will give you some high level questions for you to think about how to build this so the important thing is that when I was talking about ACME unlike the flip cards of the world Uber's of the world most organizations are already burdened they don't have the ML engineers that are required ML engineers are half data scientists with a lot of system skill that's a very small intersection because they have to understand sufficiently about modeling to be able to build the right kind of systems and so every customer is facing resource limitations when they are thinking about building all of this and of course churn and staff is almost like a given and the amount of compute that they have to do everyday is also growing so you have to think very carefully about building this system because it is all some code it is all technical debt that you are undertaking right now so talk a little bit about features richness what do we want this is how ACME's data over a period of time looks we talked about the same two columns day one, day two computing and suddenly day three the version changes because we drop that old column and we get a new column and ACME had N pipelines each of which was generating N data sets like this and they are all going through versions so you can quickly notice that the number of combinations by the time a year of runs easily generated about 40,000 data sets and imagine tracking them who knows which model which version of which model you are trained with which of those 40,000 data sets right so the first problem that you will encounter if you will is how do I compute this reliably and in a fashion where I can I get some sense of confidence and essentially you are talking about a set of capabilities we all know pandas that are the core compute engines to generate the features themselves now what is what is missing in there's a bunch of this the first one is the ability to break up this complex code like I was saying our IoT customers manipulation feature engineering code was about 6000 lines of lines no way in hell you are going to be able to test that kind of code in one go so you need an ability to break up this complex multi-step process into individual modules each of which can be tested and they can all be stitched together with parameterization it is common that you will recompute all the features features for some areas and not others you will override some parameters and not others there's a bunch of the operations of the pipeline is an activity in itself so you need a set of capabilities around that the state management has to do with the fact that when there is this complex piece of code that is operating in a multi-step process you need data frames that move from one module to another module to another module and each of which is enriching, modifying, dropping is doing something interesting so the state management is again missing of course I talked about the pandas so whatever be your mechanism you are talking about getting this set of four capabilities and we can talk about what technology combination will get you these things but these are you know you have to think versions you have to think about documentation and lastly resource management the second part has to do with which came to us slightly later in the journey which is that it's actually laborious writing all of this code and our features keep growing shrinking, changing over a period of time and so over a period of time we have moved from pandas code like this which defined some of the features to more like a specification language each customer literally we define a new specification language where the data scientists or the engineers are able to express very compactly and one big thing is that the number of errors reduced if you know we all know that in pandas a single line of code can do a lot of heavy lifting so imagine 6000 lines of code and you figuring out how to make it fairly robust and specifications is probably the way to go so you should think in terms of how do I make it really easy for data scientists to say what needs to be computed and separate it out from how and keep the how very simple so that it becomes robust over a period of time remember the one reason this all matters is that the systems the production systems that we are talking about have to run every single day and there are real applications that depend on the outputs of it so you need to invest in the kinds of mechanisms that allow you over a period of time the ROI then we can talk about what are the different ways to specify the DSL Gojek has an open source called Feast it has its own specification language you may want to look at it the languages from the other platforms are not very clear but we know that there is something like Gojek so this will be you should expect that in the next 2-3 years there will be new open source new specification languages that are cross organization simply because there are some standard patterns here it is not that it is completely custom to every single organization so the this is an under appreciated capability that you need what we found is that in the case of our retail customer they actually had anthropologists who knew when you bought Haldiram's certain flavored Bujia they could tell a lot about you as you know individual family the culture the location and so on so there was a lot of tacit knowledge which the organization had they wanted to first ingest all of that information and be able to link it to features because then the features become really rich imagine they wanted in one case they had 15,000 SKUs and they want to tag every one of the 15,000 SKUs with aroma and don't ask me what is aroma of atta but you know they had ideas about the signal that was contained in it and we have found that in every one of those every one of the cases customer cases there is tacit knowledge so we will have to have a mechanism to ingest this third party information in the organization itself so please plan for it you can use you know third party integration or you can build one these are not complicated but today in our platform for example you know after the first couple of cases it was very obvious and we make it out of box now you will get one whether or not you use it this gets into more interesting part which is that you know I had so many pipelines they have so many runs over the past so much time each one of it generates a lot of data now where is all this data and what do you know about this data how did we arrive at it we are talking lineage we are talking auditability and all of those kind of things so we found that with all these mechanisms we were spending almost 20-30% of our time just concealing data where did a number come from so what happens is that invariably whenever you generate a number there will be somebody in the organization who will disagree with that right you said our return rate was 2.5 but your number is showing 3 you have to be able to defend those numbers it is going to happen in every organization because what we found is that the organizational processes are very in-discipline there are lots of there is a lot of stuff happening in excel a lot of stuff happening in emails and lots of conventional wisdom about what the facts are in a given organization so you will you should expect that if there are metrics that you are computing there are battles ahead to prove that this is actually the real metric right and you should have the mechanism to do it and so this gives you some some thinking right you know this is saying what are all the some of the important decisions that the pipeline has made before it arrived at the the feature set and this is exactly where it is and by the way there is checksum you are training a model you have to come back to me with the checksum so that I know that the training data set then from there we can go look at the logs audit logs and the dependencies and get to the ultimate even the last record that has been used to generate this so we found this to be incredibly useful it shaved off almost 20 30 percent of our time that goes in because a single ask about number can you will end up spending days getting to that underlying proof so of course lots of validation which is one of the things that was that we felt was missing in in pandas it does this manipulations of the data frame but how do you know that the output data frame is what you expected we are talking about even something as simple as counting the number of records it starts from there and then from there you are looking at the distributions of the if you are sampling for example the customers how does the distribution look compared to the original distribution and so on these are all checks that you need to build over a period of time whether you do it in the in the pipeline code or post that in audit and other search interfaces that is all you know you can decide what is the best way to do it so one of the things that we provided because invariably when a model is not performing well the first question is is it the model or is it the data right once the debugging of the code of the model is concerned it immediately comes to data now what has changed between today and yesterday why did the model give a different answer today compared to yesterday so there is a bunch of tooling around all of these things right figuring out the drift of both the model as well as the underlying data sets maybe the version of the code that generated the feature matrix itself has changed and that could be the source of the problem so we do not in our system for example you do not go and manipulate any of the code on the server the code directly gets deployed from github we actually track the git commit that is generating that output so that you can literally link it back to the code and see what has generated now all this is in many cases it is nice to have but what we have found is that this gives us immense confidence that we are actually computing correctly and that allows us to have a lot of good conversations with data scientists data scientists typically are overburden they have a lot of models to develop they keep moving from model to model it is hard for them to remember the choices that they have made and imagine somebody coming to the data scientist and saying okay three months back you scored you rejected the loan application of somebody can you tell me why you have rejected right that is that's a very long drawn out very painful process so all of this will help with some of those questions this is again this like I was saying you never build one model you are always building multiple models and second there is no customer of ours who doesn't have a very very aggressive data science roadmap they are all higher everybody we are happy to do the Good Samaritan thing if you are available please let us know we know fantastic customers where you can go help solve some of their problems so we are talking about how do you avoid reuse sorry how do you avoid duplication of the work right so one of the capabilities that you need as part of this platform and within your organization is to discover what exactly are you computing in your system as of today there are many pipelines on many different systems right what did it what are all these achieving so having a single place where I say ok this is the feature this was being computed by this and it is available this and by the way this is the statistical distribution then you can immediately say ok I have features that I can use for my model maybe I have to ask for new features and that's a nice conversation to have but it starts with do you even know what is happening in your current system so the what we have found over a period of time is that the first version of the model is you know really opens up a set of questions involving how are you going to nurture it over the next one year or two years right and you have to think through all of these different problems that we have talked about and different mechanisms that we have discussed now input correctness actually is somewhat my favorite because I hadn't appreciated how dirty and messy the data is get stuck here the modeling does not even begin before you have an understanding of what is in your data and whether it is complete and correct in most cases they don't even know what is in the data simply because the organization is very large in one customer case there was a transaction table that was coming in with 60 or 80 columns and this is these are from old Siemens Nixtof boss machines that these this company had deployed all over the place nobody actually documented what those columns are if it has sometimes it has 10 zeros sometimes it has 2 zeros so it took almost 2 to 3 months just to know what is in the data your data science has not even begun so one of the recommendation thinking I would strongly urge to you is that if you don't have a handle on your input data forget about the rest of them because you distrusting data is really important as a principle don't use it until you have verified and there are lots of stability issues sometimes it comes sometimes it doesn't come sometimes it comes with duplicates of format things we don't know whether that has correctness implications so we there is no I mean there are some open source libraries to check validate data but there is no obvious and a simple system that will apply in all contexts so we build in every one of the cases a health check that you know validates the data before it goes through any of this pipelines I mean remember that this is all happening after all the Kafka's and the data has reached the the data lake or your database or whatever is the storage mechanism the other checks beyond whatever they do this I already talked about there was a case when the models were getting messed up and we started looking at the the data and so we found that there was some significant change in a datetime column datetime column or totals column one of some column in the in the database and so we went back to the organization said tell me about this this column oh sir like you didn't realize in in 2013 we had a developer change that person changes semantics of that particular column earlier it meant this now it means this and that information is embedded in all their excels right the tacit knowledge and the business rules that were there and here we are struggling for almost 2 or 3 weeks trying to make sense out of the data the moment we knew what we had to go back and tell the customer and we are sorry you are 100 GB of transaction data from 2013 and 2014 cannot be used because they are not comparable and then we had to go off and do other stuff I can see that some of some smiles I think the it as data scientist in the data community as we progress it will help if we move from the fanciest part of the whole data science which is the model themselves to understanding it at a systems level and having end to end control because in most cases the models are useless if you if your data is messed up let me move on to some other stuff this is the key thing that you have to take away from there is economics of feature engineering it is upfront and it grows non-linearly with the number of features and each feature is very expensive right with these in mind you have to ask what additional value does each feature if a data scientist say I want to come up with a new model and the person wants a new feature to be computed you have to ask why? if I didn't have this feature what would be the accuracy with the feature what would be the accuracy does the ROI on the model justify the kinds of investment that you have to put in to keep this thing up and running everyday it should not surprise you that only one in about 50 models going to production right and I think that when we we start asking the hard questions only when we think in terms of the production right we know the famous Netflix case right where the recommendations they were giving a million dollars for improving the recommendations I believe that it did not going to production the complexity of the implementation of the alternative algorithm was way higher than the value of it that it could deliver fine so you want to ask a lot of questions how many models will I have what kinds of complexity will the features be simple or very complex and whether I have mechanisms like Google BigQuery where I can compactly represent some of these features or do I have to write a lot of spark code to be able to do all of this and of course how much volume are we talking about this goes back to again the economics of the features in all cases what we find is that beyond a certain scale you will need a feature engine platform simply because you cannot do it on your laptop anymore you cannot you have to be able to reuse across models every large organization that you know of has built their own ML platform overall and within that a feature engine platform so you want to think about what will be the roadmap and at what point you want to build your own and if you want to do it today we are early in the the development of this whole space if you want to do it today there are really only three options Gojek has open sourced tool co-authored by Google called Feast but it is very very GCP specific there is you can build your own but it takes a very significant amount of effort to stitch all of these very different systems that have not been designed for feature engineering as a end purpose in the last one there are wonderful companies like ours which can do it but commercial products have their own set of limitations costs and so on I expect that over the next one two years you will see lot more conversations, lot more open source lot more of our kind of companies one of the reasons we developed it early because we committed from a data scientist kind of doing a lot of modeling and then we realized that we need a ML platform of our own so we built one initially in my previous company from there he said no this is the first version this looks terrible let me go off and to the drawing board think about it and rebuild a version of it ourselves which we call the so you know the standard and benefits of open source versus stitching versus your own implementation or even a third party like ours my strong recommendation is that the ideas are not yet fully developed in this space people are still feeling their way through and much of those ideas are embedded in the product documentation from each of these companies you may want to look at them to actually read between the lines and ask what is their real objective right if I change the technology choice how will it look if I scale it down how will it look right and some fantastic thinking that is going on I will leave you with a quick take a base the key thing is that the difference between in my opinion ML engineering of which this is part and data engineering is that it is very very sensitive to the modeling process itself scalability versioning so the thinking that you need to bring to the table when it comes to ML platform design is different from that of the traditional data engineering right and we will have lot more conversation later today in the bof and of course some of us are around as well do you feel free to get in touch with us for any conversation around ML systems and productionization ok huge round of applause we can take one or two questions or one at the best and then we will be back after lunch for the combined Q&A so one question quick question so I saw that you moved from code to ML or some kind of language right ML, JSON I mean use any one it doesn't really matter what the specification I will just come down to the stage I guess it's easy yeah ok one last question yeah thanks so you mentioned that we should data validation after this data lake which makes all the sense because if we input like dirty data the model will not learn anything but I think this data validation would be more important after we do feature engineering because this feature engineering part is still a part where we can introduce a lot of bugs and if we are doing data validation before it then we might skip those errors and so on yes yes you are right it is not either or you do it before you do it during you do it afterwards it pays to be paranoid ok thank you ok we break for lunch we go back at 220 just one lost and found announcement there is an MI phone black colour as all mobile phones are and we have another one an iPhone 5s if anybody can unlock the phone they can take it so come prove your ownership it will be there at the at the AV desk both of them ok come and collect it from me hello hello audible go like few rows back yes yeah yeah so ok so I will just do the start only so what I am going to talk about is how do you build complex data pipelines using louder ok what I want to talk about is how do you build complex data pipelines which are evolutionary nature using contracts now ok is it fine what I am going to talk about is how do you build complex data pipelines which are evolutionary nature using contracts this one right this will happen ok ok this is better right is it audible ok Raja it's fine this is a better position this right that is a bit my time and in the middle of it you are like yeah also one more thing like if I would like take a quick short water break is it possible you can give a bottle here ok ok no because sometimes when you are going fast this is something you want to take a break you want to try just give me a minute it doesn't work from there it doesn't work from here you can't dream though yeah you have to be behind you yeah and this clicker stops working sometimes distance it's a USB bluetooth thing you can just use your yeah yeah open up your slipper stand up when the AV team is here so they might add the volume yeah I just have to make sure that breathing thing doesn't come because that's a this is better this is a good position actually check this when I come on stage this is a bit fun breathing thing ok all good right you can swing to change also no no no just press this button you will see the mouse button to use hello hello better I think it's in the corner enough yeah hello hello I think it's audible here ok sure sure thanks a lot hello so three kinds of patterns ingestion storage consumption patterns of millenium engines per second 14 TVs TV streams search thanks dude testing can everyone hear me breathing no excellent so this is a good angle for it yeah good ok hello we will start in about 10 minutes with the joint Q&A for data pipeline before that we will have the MI phone I think somebody is looking for it ok come and unlock the phone to take it he was making take all of them ok welcome back everyone hope you had a good lunch we are going to start the post lunch session with first of your announcements so flash talks are scheduled at 4.55 in audi 2 if you want to talk please come and give your names I don't think we have any names at the moment so people who are interested to talk about something that they are doing it's a 5 minute slot, maximum 7 talks no presentation required no laptop required unless you are doing a demo just come talk about something that you are doing get a chance to present what you are doing get a chance to get feedback from the community and start the conversations 455 flash talks will be in audi 2 if you brought the 5th elephant t-shirt then there is a t-shirt counter you can go and collect it there birds of feather session which are pretty much like what we are running here but even more informal and a chance to kind of interact talk about it, talk about your use cases, share those learnings will be starting off in audi 2 and audi 3 so look at the schedule and see if you want to go to any one of them please don't bring food and beverages into the auditorium and if you are interested in the tutorial tomorrow which is being run by Rajdeep Dua Salesforce on transmogrify kind of connect a little bit on feature engineering that the last talk we had on this session if you are really interested in that it will be in audi 2 from 11.10 tomorrow so have a look at the schedule get ready for that and if you are planning to attend that, that is tomorrow in audi 2 from 11.10 so before we go to the 2 talks that are scheduled we have this joint Q&A with the 3 speakers that were there we talked a lot about how do you start your data collection process from day 1 Kumar talked about that and start to get your data in Nitin talked about how we can start to prepare and ensure that those data systems are running so aiops and then Venkat talked about how do you ensure that data that you have got is managed in a meaningful way and transformed in a meaningful way into a feature data set and the process for doing the anatomy of that so we have so we have those 3 talks, really good ones around the entire life cycle or at least the initial life cycle of data systems I am going to open it up, we have got about 15 minutes for a combined Q&A if you have questions that got across that and you want to in the Q&A now is the time okay I think people are still hesitant let me kick it off with a few questions friends any questions or okay sure we have got a few I don't yeah hello so my question is for Kumar so you mentioned that you push your data directly into data warehouses in your V1 of data pipeline which is kind of not the norm with what I have seen outside also whether the data directly gets pushed to the data and then where it so can you give us an insight as to what pros and cons did you find because it did work before you moved to your V2 so the reason why we chose that anti-pattern is because we wanted the capability for a product manager or as a developer to kind of query data as real time as possible we did not if you remember we did not have any real time pipeline or real time dashboarding etc available in V1 because we didn't have time to build so what we wanted was as soon as our product or game is kind of downloaded by our first user we want to see what that first user did in the first hour of the gameplay and kind of if I have given the game to kind of a user on the street or somewhere if I am testing it on the field then I want to see the live data like what did they do how many clicks they made did they swipe too much did they touch a piece of the screen which they were not supposed to or what we wanted so to do this query we thought in terms of SQL itself and most of the stuff you will not know how do you want to see the data what's the right click stream etc that you can actually make real time charts about it so having that raw data available sooner was really helpful and so that we can query it in SQL so that's the primary reason why we pushed data directly into the data warehouse now obviously the core of it is very simple you can't keep doing this because your ingestion will take it all and any data warehouse you use behaves much better or much differently if you can kind of batch your ingestion in different intervals or different logic and at the same time if you are inserting it there and letting users kind of query directly that live data then obviously they are going to be fighting for the same resources right so you will have to figure that out what depending on your usage patterns or ingestion patterns we were able to do that and because in the beginning usage was far low there were a couple of users only who were doing this continuously but as we grew also right now also we allow that obviously we have to define the cues very strictly we have to apply some boundary rules query cutout times and all that stuff so that ingestion doesn't get starved for any reason and since we are able to do that for our use case we have been successfully running this but you should evaluate it for your own use cases any more questions yeah one here so I have a question to Nitin so yeah so like in one of your high level diagram I saw like you chose time series DB open time series DB so what was the reason for choosing that specific time series DB because currently in the market like few players are there some of them are like much more dominant like maybe droid or influx DB and that is the first question second is like how are we using time series DB in this specific context yeah sure so for us the time series use cases kind of at least the current part of the product are not very advanced the primary use cases we have because we do most of the aggregations in our streaming platform and the primary use cases which are and all of our logic the core logic of the product has to run in a very streaming way the time series databases themselves really power our UIs where we show trends of metrics and for most of those things it's a very look up heavy use case wherein I know I want to fetch data for a particular metric across a period of time so for that I think KV stores tend to behave in a good way we are though exploring droid as our next gen metric store because there are a lot of supporting related use cases which we want to power wherein we want to allow users to be able to look a slice and dice their data so that calls for really a read time aggregations which will kind of help which will kind of be something which will come out of the box with the system network so does that answer your question or looking for more details hold on just get the mic so yeah yeah so other than the roll up down what other things for which you basically put time series DB there so yeah so the time series database is right now storing the individual metrics it's also storing the baseline information for these individual metrics it is also storing a lot of our anomaly detection stuff anything which needs to be really be available for consumption on our portal is something which really lives in our time series database a lot of these other things because if you look at anomalies if you look at baselines there is a temporal angle to this and because the anomalies will be raised at a particular point in time so you want to really have a temporal view on a lot of these data elements and that's the primary use case for our time series database it's mainly powering our UI a lot so our processing itself is very real time and happens on stream okay we have one more question there yeah so my question is related to real time streaming one day so you had all immutable data but what about mutable data how real time is it possible or what would be the way you would go like if you're replicating mySQL data and you want less than a minute latency in your data lake or something that depends on how you want it when I meant immutable is primarily the way we want to use the data because if you allow for data to be mutable at some point in time and you have one use case in mind today trust me tomorrow there will be thousand use cases and since your warehouse cannot support this mutability of data like update or upserts for all of your historical data right you don't want to you add a column and I want to fill this column with this default value for my 2016 dataset hard to do at scale so we chose this that we wanted to be immutable and that is true for our use case that it was analytics events data that is going in but if it's some other kind of data maybe your payments and some order processing which has to kind of wait on some manual verification etc you will have to support it differently I will very highly suggest that in your traditional data platform mutable data set you should design such a way that they are very small and if you can live with not having mutability that is much easier to handle on the infra and the upside of it because you will not have to handle those scenarios or whether my SQL replication will finish under this or not whether they have to be in the same region same zones and then if I have to do a cross region then all the millisecond goes out of the window anyway okay any more questions well let me let me pose one question to Venkat while we wait yeah okay go ahead my question is to Venkat so lot of times when data scientists start building their ML models and before even they get into feature engineering there is a lot of trial and error that they want to do they want to see how things correlate how it varies across these categories and all so are you also exploring this area where you are directly querying big data and having a visualization layer on top of it where you can immediately see those results so typically for those kinds of things they already have mechanisms to do it for example ML flow will allow you to ML flow with combined with a bunch of other toolings allows them to explore the data even visualize and so on and then there are SageMaker and other data science studios our kind of work comes once it stabilizes a little bit right and then we are talking about sustaining it over a very long time the other angle here is that one of the side effects of building a feature engineering system is you actually generate enriched data sets to enable these kinds of exploration as well so we feed into the exploration tool but not actually substitute many of the existing tools just in the same way aren't we reinventing data modeling in like a just different term so like how does this differentiate from data lineage tools like TriFactor or Yeah, TriFactor is solving a slightly you know different problem if you will which has to do with let's say your fidelity and you have 20 different divisions and you want to know if a customer is there in two different divisions of your TriFactor of your fidelity then you will use those kinds of tool including there is a tamar and others so the way to think about this is that what has really happened is that it's not that ETL has not has gone away we still have the old informatica the newer age version of it whether it is dbt whether it is all tricks all of these are there and then there are these further systems that are in the in the same direction I think the need for a slightly different tooling comes about because the it is very very closely linked to the modeling the data science modeling for example if you didn't have the need to evolve this continuously right and if you didn't have the need for audit trails whether it is on the code of the data and so on some of these would go away right so my my suggestion and those are really solid products I mean they move petabytes of data this cannot substitute it has to solve a different problem okay one last question there and then yeah go ahead so first of all can I just his question a bit like so on the weekly yeah just ask the question yeah so for his one like we also have the same setup actually so we have data lake and then we have zeppelin which is more like dupiter for the big data right so we use spark jobs so you run your pie spark jobs or your skala written spark jobs and then whatever data you get you use zeppelin notebooks to basically visualize on top of it that is like what we use internally and my question is like what do what would you suggest as a tech stack or something for a cdc pipeline for your traditional art for moving data from traditional rdbms to your let's say big data systems right yeah I think which maybe have some perspective so like there's 200 options I think for this as well depends on the use case and the kind of data you have yeah so like in our case it's more the the sources are generally ideas so the issue doesn't come in while reading the data it's more like how do you manage the real time updates on the big data side like there are few options like kodu is there but it has certain limitations so like certain like on the big data side what would you suggest like as a storage engine where which can subject okay let me let me pause that question because I think it's a very broad question and we've gone down the tools track a bit I think what I would urge everyone to take away from these three talks is the mental model the anatomy or the cognitive model around how these things are done I think all the three speakers have tried to avoid tool specific discussion because they want you to go back and think around how to take these lessons that they have learned in a hard way and try and build it into your own tools right so continue the discussion outside because it sounds more tool specific than what the speakers were attending we'll stop now a huge round of applause for our three speakers there yeah thank you so much we okay we begin the two talks now around two sector talks so the first two talks in the morning were around algorithms and how people are applying it we had a good discussion on data pipeline data engineering feature engineering and the next set of two talks are talking a little bit about the harder things you know the successes the failures that we face around it so the first is around the software or the open source tools that we all love and the second is going to be about how to handle bad data right so I'm going to welcome Peter Wang from Anaconda and especially like to thank Anaconda Inc for sponsoring Peter's travel for the fifth elephant Peter is going to talk about his experience and journey in building Anaconda some of the tools that a lot of us use at least I personally use Anaconda a lot so welcome Peter and over to you a round of applause for him okay great thank you thank you very much it's great to be here I just flew in this morning so I might not have as much energy as I normally do and for that I apologize but try to keep it lively I'm Peter Wang I'm the founder of Anaconda and I'm currently the CTO Chief Technology Officer and today I'm going to take you through kind of the journey we went through at Anaconda both as a company as well as through this process of growing the data science community, growing the open source community around data science oh I have to turn this on there we go so my background actually my formal background is not in computer science but actually in physics and actually one of my colleagues from undergraduate here and we had a lot of fun back at Cornell and I was mucking around a lot with computers I've been coding for a long time but after I graduated I went into software development as a career and I only took three CS courses in college I had a lot of fun with them they were easier than the physics courses by far but essentially what happened is after after a number of years doing very complex I ended up doing consulting around the scientific Python stack in kind of the mid to late 2000s at a company called Enthought and built a lot of kind of early tools there and then around 2010 time frame I realized that the big data movement or revolution some people were calling it that the big data wave was going to drive a dramatic need for better data analysis tools than what was around at the time Python was starting to get organically adopted in a lot of different places and so I saw there was an opportunity to really push this scientific and numerical Python stuff that we had built so Travis, my co-founder, and I have been building a lot of these things we saw an opportunity to take that and really drive it in business so there it is so our company's mission from the very beginning has been to build better tools to help organizations harness the power of data science and data analytics and really put that power in the hands of people who are subject matter experts or domain experts so we make software that the data scientists use but also very importantly we provide software that businesses can use to empower those data scientists so many of you may be familiar with Anaconda primarily as the software distribution how many of you know Anaconda for that open source right so great so this kind of we call this a architecture diagram and this architecture diagram shows you the ecosystem of things that happen to make the Anaconda distribution what it is and it's a very unique piece of software actually in that it is cross language cross platform and multiple architecture so I mean you can go and run Jupyter notebooks to write R or Python you can use Jupyter notebooks on a Mac or on Linux or Windows you can use it on ARM or on IBM mainframes or on X86 so it crosses a lot of different matrices this makes our build farm incredibly complex but it means that for all of you who use Anaconda that you can write something and take it from one machine and run it on another machine or other set of machines without worrying too much about that and one of the key pieces of technology inside Anaconda that makes that possible all right is a tool called Conda which you may have experienced or used as a packaging manager and what it really is is an entire sort of framework for building open source libraries or building code in general a building in a way that's cross platform multi language potentially and most importantly built in a way that is sandboxable that sort of self-contained so it can be installed without having system administrator level privileges and this is actually one of the reasons why Anaconda is very popular in the windows world because many of our business users have windows machines at work they want to use so many of the Python libraries but their machines are locked down so Anaconda can install into user space and you can still install a lot of additional stuff but the fact that we do support multiple languages and we can put these together in these virtual environments that makes deployment easier it makes reproducible data science easier it actually is a much simpler thing to use than Docker and things like that and it's one of the reasons that our enterprise product is also very popular so our I'm going to stand right here so I don't have to keep doing this like punching my laptop so many people know us for our work in open source but then I always get the question well so how do you make money and people assume that we do consulting or training and things like that and we do a little bit of it but primarily the way we make money is we sell our commercial platform which is a completely different set of bits than the software that you download and use on your machines as Anaconda distribution and what the enterprise platform does is it uses underlying technologies like Kubernetes and Docker to allow data scientists to work in a corporate environment that's relatively locked down and still have easy one-click deployments of Jupyter notebooks, of dashboards of machine learning models APIs and allows the IT folks to govern and audit all of that so it's a very it's a very powerful product and it's used, you know, at a lot of really big companies it's also used in some smaller companies but when businesses are serious about managing the software supply chain that feeds into their machine learning that's when they call us up about Anaconda Enterprise but today's talk is not really going to be about Anaconda Enterprise it's about the overall journey that we went through and many of you may already know this and I apologize if this is a repeat but I think a lot of people don't know this and how many of you here are practicing data scientists? Show of hands. Great. How many of you have been practicing data science for more than five years? Not that many, right? So the journey takes us back to seven years ago my co-founder Travis and I we had some visions around 2011 2012 timeframe and what we realized was that the reason why the scientific Python tools were getting so heavily adopted in the 2000s as a displacement for MATLAB was because scientists wanted to write code like they wanted to express their ideas in code they didn't want to have to write a spec they give to a Java developer to code poorly they wanted to write the code themselves so they could then go and fix it when it was broken and they could also build it quickly without into production build a thing that they could use to drive some lab instruments or build a model that goes in trades on Wall Street so Python was getting picked up for these places we were getting called by everyone from the US Army and the military to the most prestigious hedge funds in New York and so what we realized across the board was domain experts, subject matter experts they wanted to manifest their ideas in code and Python was the weapon of choice for them and another point that we realized was that there were two things happening right at the turn of this decade which was that the big data wave was starting to really crescendo and reach its almost peak and that was going to completely blow up traditional data warehousing it was going to completely blow up traditional business intelligence tools people were struggling with excels like million row limit people were struggling with very expensive old tools like from Tipco and whatnot and the second of all cloud computing was starting to hit and it was mostly being adopted for web development but we saw the potential for data analytics the fact that people could go and rent a machine or a hundred or a thousand machines and get like a mini supercomputer on demand to crunch through a lot of vector math that was going to be a game changer on top of the big data thing so what we did actually in 2012 was a year after the jupiter notebook had been actually at the time it was called the ipython notebook but within the first year of it being released we created an online notebook service called wakari that you could just click on it go to wakari.io and you would have a running ipython notebook and it just worked how many of you actually used that or heard of it anybody here a few people good some old timers the problem is we were way ahead of the market nobody could spell ipython ipython hadn't renamed themselves jupiter yet it was way ahead of its time and it was also people would use it for free and no one would pay for it so we had to kind of move away from that product and figure out something else but we were on to something in terms of realizing that being able to turn on cloud resources tie them to interactive computation environments on top of large data sets this is not familiar to anyone right this is a very useful thing and then one other thing we realized was that what we were seeing as we're doing our consulting work using python that there were these different silos in business most of the business big corporate environments we went to and I have a very US centric view of this of course but maybe this kind of resonates with you all over here as well but we had business analysts who were used to using things like excel and bi tools like rapid miner and things like that we had developers right your traditional CS majors who were programmers java.net c++ etc etc and in the middle there was a small tier but growing there's small tier folks who would be doing things like predictive analytics using sass or they might be writing some matlab code as a quant at a hedge fund but more and more people in the middle were starting to use python to do a lot of those things and the goal here in the middle is not so much to just produce an insight or produce a report rather to build out machine infrastructure to build lots of insights and do lots of reports and actually do things with those so we saw that not only was this middle tier of folks going to be growing but additionally python as a language spanned all three of these things if you take some jupiter notebook that an analyst can read and maybe click on a few things and change a few things and have it still run that same notebook you can give to a c++ or java programmer and they would still kind of know what to do with it maybe they're going to you know complain about white space or something like that but they could it's going to be a lot easier for them than trying to understand a pile of r or matlab right and so we saw that python as a language that could cross all these different silos was itself that intrinsically was going to be another unifying connecting and a very important phenomenon so we started a company and this is sort of what it looked like when we first started out of cona that's my truck with some used furniture in it that's my co-founder travis and then brian van deven was actually the creator he was the first commit to cona he's also the creator of bokeh and I sort of created bokeh together this is our first strata conference and travis is pointing to our logo our first booth there our first office it was a very the company was called continuum analytics at the time it wasn't called anaconda and there's a fun story behind that actually so we created this company to push the use of python for data science because we really like python and so one of the things we did was we created this thing called pydata so I registered the domain name the very first pydata workshop and google and riley helped us put it together and it was at google's headquarters and mountain view it was a small workshop, 60 people but we had, it was quite a workshop because you had west teaching how to do time series in pandas you had francescal ted talking about optimizing numpy the late john hunter talking about advanced map plot lib jacob vonderbloss teaching you machine learning with scikit learn it was just Fernando teaching ipython and map plot lib, travis teaching advanced numpy this was an amazing cool event and it was only like $50 to register for it only happened once but based on the success of this we started doing a lot more pydata conferences and things like that and now numfocus runs pydata events around the world and at that pydata though a very important thing happened which is guido step by guido is still working at google and so he came over and he was talking to us and this is maybe more of a I think this conversation they're talking about something else but we actually have it on video where we approached guido that the scientific python community did and we said packaging is a huge pain in our butt can we do something about this can you convince python core dev to take packaging of c++ and fortran code seriously and guido was like you know what number one I don't care much about packaging that's why it's a second class concern in the language and second of all these cases are so exotic you should build something that works for you all and so we did and so we built anaconda something is not going right there because what we needed to do was we realized if we wanted to ship python for big data if we wanted to ship all these advanced tools like just-in-time compilers and all these other kinds of crazy things web frameworks for visualization we were going to have an even bigger bigger packaging problem that we already had just trying to ship numpy and sci-fi so we started the company to try to build all these great capabilities in the python space but to do any of that stuff okay this is I'm going to stand behind the podium here and do it this way in order to do any of these things we had to be able to get the software into people's hands and so we wrote a package manager called conda and we built I remember the moment I turned to travis at the meetup at that conference we decided to create our own distribution of python because this is going to be nuts to try to get people to install the stuff and because this is python for big data we should call it anaconda and that was the moment of inception really was in March 2012 and so by September we created the first version of the distribution we added window support which has been useful for some people I guess and we had all sorts of other fun things that we did and starting from that point you can see the that was an afflection point and it bent the adoption curve of python because prior to that point even though let's see that plotlib was probably 7 or 8 years old and sci-pi was 12 years old people were all the time coming up onto mailing lists and complaining I can't install this this compiler is blowing up this is not working this is an anaconda it made it easy for everyone to install all this stuff a lot of that stuff just went away so I think we really helped drive the adoption of python and to show you what that looks like now this is our weekly unique engagements on our anaconda repository so it's over a million unique IP addresses every week on a monthly basis it's about 2.5 to 3.5 million and to give you some sense of scale how many of you have heard of Tableau yes everyone rapid miner maybe some people write their data mining tool the entire Tableau user base is something on the order of half a million users the entire rapid miner user base is on the order of 440,000 people those are numbers from those companies themselves so we in our last trailing 12 months trailing 12 months unique for anaconda our repository, our download sites and everything else is 16 million so this is an absolute movement that you all are part of it's growing and open source is at the heart of that now packaging packaging was not the only problem we had back in 2012 number one people complained about python being very slow and in fact some people were looking at julia with great envy in fact the j and jupiter julia python r so we didn't want people to have julia envy so we used lvm and a couple other hacks and created a just in time compiler called numba which is now becoming quite popular as a way of very easily accelerating your python code without having to step into c++ or anything like that also people were doing a lot of r versus python stuff back then and there wasn't a way to do shiny for python people said I will walk away from r except we created a library that would suffice that particular need and then also there was this question of scale out and there was multi processing and there was a lot of other things that you could do with map reduce but it was clear that the world was going nuts over hadoop and map reduce and we needed something in the python space so it took us a few years to get there but we created a framework called desk which is a much nicer idiomatic way of doing scale up and scale out in python and these projects continue we created those projects and they continue to add features and get more stable and get wider adoption so numba now you have very nice gdb breakpoints from inside your python functions you can build dictionaries basically write python code with dictionaries and that all gets compiled to extremely fast height machine code and of course numba has always been an excellent way to easily target your code to gpu's as well as cpu's desk has been getting a lot of great interop with yarn and kubernetes also we're adding support for gpu's so we can load balance across them and more and more libraries in the python space like scikit-learn and pandas are keeping desk in mind as a way of doing scale out and parallelism and then on this side we've created the next gen set of things to do dashboarding in python you can use any number of viz libraries and give you interactive widgets and very nice dashboards either standalone or within jupiter notebooks and we've also added large data visualization support and we've just been through our work with some of the us army research funds and nasa and whatnot we've actually built some really wow can everyone hear me okay back there is this okay in the back can you hear me okay we've been building some really nice large data set visualization capabilities so I don't know how many of you have heard of data shader but we can actually render billions and billions of points in an interactive jupiter notebook on a laptop and these are some of the plots that we've done with those capabilities and we're still creating new ones so just last year we started a new project called intake which is a lightweight pythonic data catalog it's not as heavyweight as something like tamer it's actually not meant to be like you know immuta or tremio but it's something very simple for you to be able to run a little server that front ends a bunch of data and be able to connect to that server remotely and drive some computation on it so simple lightweight pythonic and it's starting to get some nice grassroots adoption I'd encourage everyone here to take a look at it if you have a chance but in doing all this stuff and this seems like a fairly linear sort of path we learned quite a few lessons right the growth of the company that we did from 2012 to now was during a time of great churn and great innovation in the data science and machine learning space we saw big data come and go then we saw the rise of data science and even as people were questioning the validity of the term data science we had machine learning rising even as people were googling what machine learning meant we had now AI so I don't even know what comes next quantum blockchains I guess but you know the last 5-7 years have been just layers upon layers of stuff and I kind of break it down like this so the first 3 years of the company we really spent in kind of this open source consulting bootstrap sort of mode and the industry was sort of peaking on big data data science was emerging and our biggest competitors was just our own potential like crashing into the ground and making a big crater we could we had failed to push Python for data science people could just go and do other things and also we were competing a lot with other things like Kadoop and Spark and R Python was not an established force many people to my face would question my sanity and pushing something weird like Python for the stuff by the time 2015 or 2016 world round those in the know they know that question was settled but we ourselves as a company were transforming into becoming a product based company so we took a fairly robust series A and then we had to rejigger from our services and consulting training model into trying to orient the product engineering group was building commercial enterprise grade product and now we brought in a sort of a seasoned veteran executive as CEO we've hired in a fairly veteran management team and together we're growing the company and trying to figure out in the space where there's so much noise how can we build well engineered products that do what our customers need but still manage to kind of keep on top of the hype cycle and also stay true to the covenants that we've made with our open source community and through all of this I mean these are some business challenges there's technical and landscape and marketing challenges but through all of this the biggest challenge has always been about people whether it was about the community at large and the data science personas who are trying to service or whether it was about individual library authors and innovators in the community whether it was about our own internal employee culture throughout the entire journey the hardest thing has always been about people even though we're a very deeply technical company we had plenty of technical challenges and business challenges it was always about people and one of my favorite quotes is from the not from the little prince but from the author of the little prince in a different book actually and he says there's but one veritable problem is that there's a lot of technical challenges and relationships throughout the entire time that I've been building the business throughout the entire time I've been trying to build pie data community this just constantly rings true for me and and you know it's kind of a touchy feely kind of of a point you know I can imagine some people in the audience are just like well okay whatever but I mean that's nice about there's ton of stuff going on saying that we need to be nice to each other doesn't help any of these things and in fact actually I think the challenge that we face really in the in the space we're not just building random web services we're actually building machine learning systems we're doing a lot of deep data driven analysis and we have to communicate these to sometimes less than savvy audiences and so our challenges even though they may appear to be technical you know that when we ask those engineering questions like what is the right answer what is in the right problem what is the right thing to do what technology is good is a turn as a progress how do I actually those of you who really want to do good work you're probably asking yourself how do I actually engineer from principle as opposed to just throwing something on a wall until it sticks and there's this great quote you know sort of paraphrasing Isaac Newton says scientists progress by standing on each other's shoulders and computer scientists progress by standing on each other's toes and there's a whole lot of toe stomping going on really in the industry although I guess if we go to quantum toes then we won't overlap but there will be less squealing but really you know our industry the computing industry in general lives in what I call the long shadow of the 70s so many of the things that we still use were invented in the 70s of course we still use some things invented since then but not that much actually if you're not careful sometimes you'll just download binaries on the web that machine learning libraries or open source libraries and they're compiled with the Pentium target which dates from the 90s they're compiled with very baseline instruction set architecture not optimized for any modern chip a lot of you probably are in shops that use a ton of Java, Java is very old very very old language at this point and the things that are really popular still like Excel it's basically the same thing as VisiCal and any of you who type DIR or make DIR into Windows DOS prompt you're using essentially a clone of CPM from 1974 and certainly many of us use Unix and these Linux primitives the Unix model all of that stuff dates from the early 70s I don't know how many of you drive a 1970s car but you use 1970s computer technology for the most part and so now on the one hand this is really great it means that we invented some really solid abstractions on the other hand it might mean we're really stuck so in the face of this how do we distinguish progress and churn one thing I can suggest I've been programming for quite some time but throughout my entire tenure as a developer I found very very few books that I could consistently and constantly recommend and the Pragmatic programmer is a book that I recommend without reservation all the time always if I could recommend one programming book to my children it would be this one and in it it lays out some of the basic design principles to build robust software systems now for data scientists there's an additional set of concerns but as the artifacts that we build in the data science community become more and more like software and less and less like a bunch of random data scripts then we need to consider these things keep these things in mind it's just absolutely a fantastic book but another book that augments this one really well is called the Clean Coder and the Clean Coder doesn't talk so much about design and programming technique but rather it talks quite a bit about the personal development that you must undergo your who you are as a person and the way you must look at your work if you want to be a professional developer I think this sort of carries over to data science in a way as well and some of these things again these feel like very touchy-feely kinds of things but these are absolutely critical if you actually want to build good teams that can go and actually answer those questions of what is progress versus what is churn one of these really important things at the end of it at the end of the day it's about continuous learning and it's about humility and in fact the things that I see whether it was the python community and the pi data community whether it's within my company the things that held us back the most was when the team failed to have trust with each other when egos get in the way the smartest developers the people who built things like conda solving extremely hard build problems they could sometimes be incredibly stubborn and would resist talking to other people who are building other amazing things and it was very frustrating for me as a manager as a founder to have a founding put the team of really smart people together and have them just be at dunderheads with each other so obviously very smart people both obviously mission aligned but they in those cases when they had that kind of conflict neither party really reached inside and figured out how to be vulnerable how to build trust with the other party how to actually express humility in a way that could allow the team together to progress so the two things that I found without this process the two things that you really need to have in order to have vulnerability you must have self confidence not arrogance but actual self confidence the confidence to say I know this and I don't know that and then of course the group and you can only say I don't know that if in a group you're not going to get kicked out of the group for saying that right for showing weakness and so one of the things that is incredibly important is as a manager as a team lead is to facilitate creating that kind of supportive structure where people can openly talk about what they did right what they did wrong what they know what they don't know and once you have that then you can actually progress and move forward so one of the core company values we have at Anaconda is we hire for people that have both ability and humility now humility doesn't mean false modesty humility simply means recognizing that you are probably not the smartest person in the room on everything you are probably pretty smart about a lot of things and your job therefore is to figure out what other people are smart about and what other people are not smart about so you all can be smarter together and that's what that humility is but it takes a group it takes a village to build that culture of humility and we have had a few painful exceptionally painful lessons when people fail to have humility and it cost us a year or more in terms of a product release cycle it cost us many good people leaving because people in the leadership positions didn't have humility this is a very painful and difficult thing to talk about because a lot of good people put their heart and soul into this stuff by the end of the day the failings were very simple I put them up on one slide and so I know when I was talking with some of the organizers here about what to talk about here at the fifth element and here in India in particular a concept that was brought up was that really certainly in Indian culture there tends to be a tendency to want to be very credentialed there's a tendency to really want to put your persona in a facade a very projected very strong and a very strong facade and the danger and the cost of that in settings such as a company or a technical team the cost of that is that everyone is then blind to your systemic risks for data teams in particular for data scientists in particular that is exceptionally dangerous because you all represent the brain of your organizations and we all know, we've all played with data you can find whatever you want in data you can tell whatever story you want to and if it's all about posturing it's all about making yourself proving that you're smarter than the other person you're going to end up really hurting your organization so on that note so what comes next so if we have done all these great things to kind of get data science in python to where it is now coming up next one of the things that I think is really really important to drive home is just how much hardware innovation will be coming down the pipe how many of you use GPUs in your work on a kind of regular basis anyone? not that many people how many of you are interested in using GPUs in your work a few more people how many of you would like to make tons more money doing deep learning alright there you go you're going to start using GPUs then how many of you are interested in IoT so now you're going to start running your machine learning models in things that are not x86 probably so this is just a quote from ARM but really I was shocked it was the last year I started taking stock of how many new kinds of chip architectures were coming out that we would have to support because again we have to build stuff to run on all these different platforms architectures so for me it's a bit of like a sobering thought as well but as of 2017 these were the chips that have been announced the kind of architectures we sort of figured we'd have to support people were talking about deep learning we knew we'd have to throw GPU plus CPU kind of workloads together but by the end of 2018 the number of chips that have been announced was way more and there's more you know this list just keeps growing and so what this means is that you know I put this up here just to sort of stress that many of the people in the IT space who have been doing software and hardware and infrastructure and system architecture maybe even doing this for a couple of decades you know what it's about throw a Linux kernel here put a network switch over there you kind of throw some data things around all of that is actually going to change quite dramatically and the innovation space that is going to be that we said at the intersection of that innovation space is going to grow by leaps and bounds in the coming years and there will be a tendency to say well gosh that's a lot to keep track of I mean look at all these chips I have to buy all of these to have to go qualify my algorithms all of these things no one's got time for all that so there's going to be this temptation to outsource the problem right oh here is this vendor they sell best in class yada yada pipeline we should buy that pipeline and the thing I would stress to all of you as decision makers influencers practitioners the thing I would recommend to all of you is to be very very clear with your management with your budget authorities that you cannot outsource innovation in the space if someone is selling you a predict button they're your competitor they're going to take your data and they're going to out compete you right and there's no big red solve button there's only a big red on me sort of button and generally don't want to pay for that the the thing that I think what we really have to do as an industry again going back to that concept of engineering on principle I think we need to move past the 1970s and 80s view of software infrastructure networking hardware we actually move to viewing the entirety of a computational system as an inference engine and looking at characterizing the qualities of that system managing it auditing it engineering it or debug ability and manageability that's a really really huge cognitive leap for a lot of businesses because they're so set in their ways now I know many people here work at startups younger companies that have a lot more flexibility but you're also under the gun to deliver much under much tighter timelines those of you who work at bigger companies start thinking about the start thinking about how can we build cross functional sort of inter interdisciplinary but across the different silos of management how can we actually start thinking about this overall thing is one unified holistic thing right now you know people talk a lot about model management and the the ugly truth is models consist of two things data plus software well most people are not really governing their software very well because they throw everything in a docker file and just pray that doesn't get hosed and then they're not governing their data very well because the data governance either it's good because it sits in a giant slow enterprise data warehouse or it's hardly governed hardly really you know managed because it's streaming through or you have a very ad hoc process with a bunch of S3 tables so there's a whole discipline that has to grow up around managing both of these in concert holistically and I think one of the takeaways here is the reason what's the difference between a machine learning model and a data science model how's that different than just regular software development this is a question I get asked quite a bit when I go and talk it you know in house at places and the way I characterize it is that traditional software development really puts data as an afterthought it has input to the system and no matter what values you get the correctness of the function is orthogonal to the values machine learning models are not that machine learning data science kinds of applications the correctness of the system is highly dependent on the values that go into it and that is a complete shift complete paradigm shift from how any of these things get managed in IT today and so two things I think that are going to happen in the next several years we're going to have to redefine the entire stack of governance management and audibility around these new computing systems the software development life cycle the application life cycle management sort of practices are going to have to evolve and lastly I think we're going to have to re democratize computing basically building the kinds of things we're building the natural end point of evolution for things like jupiter notebooks I think is actually serverless computing a new version perhaps of the python interpreter that doesn't require being grounded in a linux kernel running somewhere on somebody's cloud and thinking about how we can go beyond just python users writing a notebook but all of the excel users being able to write simple expressions inside code cells and actually going to a much broader use base for the python language that actually is an incredibly powerful concept and it sort of completes the full vision of what Travis and I were trying to do when we first started this whole thing about giving the people who understand the domain giving people who know how to ask the right questions giving them the power to answer those questions and of course all of this will require infrastructure on demand so if you don't have that then you can't even play this kind of game but I think these two things number one this wholesale redefinition of the entire stack of management and then two democratization of computing around a python centric a roughly python centric and certainly a very data oriented set of workflows these are two big trends that are going to happen over the next five to ten years without question thank you very much thank you Peter okay thank you Peter we have a joint Q&A with you and Chris after his talk so I think since we're nearly out of time we'll just hold the conversation hold the questions and the conversation and after Chris's talk we'll both get both of them to come and answer all your questions so stay stay here so we heard about the open source tooling the open source business the open source challenges I took a lot away from that conversation we have the next speaker Chris from simple and I think the other challenge when we work with data is around bad data right so you know I visited I visited a training session the last two days and even though I was talking about how to think using data or take data driven decision making I couldn't get the team to move and bad data now we don't have the data that is correct we can't actually make decisions because people don't believe on the data and I think every one of us probably working in data has had those conversations and challenges right so Chris is going to take us and I really like the talk title the final stage of grief about bad data is acceptance so hopefully we'll all be at acceptance at the end of it Chris hello hello that works are we good here okay so while they change the battery I'm Chris Duccio I'm the head of data at simple we're a pay later engine and so basically I'm going to leave a somewhat much more on the ground talk useful for day to day people who open up Python and crunch data every day so here's a question for the audience how many of you are happy with your data set does it answer every question you want answered does it does it work for you could it be a lot better who's really happy with it yeah I didn't think very I didn't expect to see very many hands on that so I've gone through many stages of grief I've tried constantly to get my data better and eventually I just sort of approached the idea that I'm going to accept that my data is bad and I'm going to see what I can do in spite of that so this talk is not about improving your data your labels are wrong a lot of stuff is missing and it's going to stay that way if you can fix things at the pipeline stage you absolutely should do that but in most cases you can't do very frequently to things beyond your control so this talk is about taking that into account and drawing correct inferences anyway it is a bit pessimistic but at the same time I do consider it realistic so the recipe I want to get across in this talk is instead of treating your data as a reflection of current reality and instead of treating it as something you want to use to predict the future all by itself instead treat your data as a reflection of an imperfect picture of what's actually out there in the world so I'll sort of talk about the present and the future and so normally speaking the way I see most people working myself included as often as I can is I'll do some data cleaning but apart from that I'll assume my data is correct let's say I got a credit score from a vendor I'll assume that credit score is true and I'll train a predictive model hope for the best and if the rocker you see is good enough I'll push it to production what I want to convey in this talk is there's another approach which you often have to get to is you get unfixably dirty data let's say 10% of your data is just missing for some reason or another or it's just wrong somehow what I want to what I want to suggest here is instead of treating your data as your input treat your data as the product of a random process build a model for how your data went from reality to what is in your data frame then you build a model to predict current reality from your data and then use the outputs of this model which are better reflection of the world outside your system as the inputs to your final model which is trying to predict the future so most of this talk will be going through examples of this in a little bit of detail you should treat this sort of as a recipe you can follow yourself none of these will actually apply to your problems but the ideas will hopefully suggest a way to work so missing data is a common problem and here's a problem I a situation I found myself in so I was building a product that was doing what is essentially a funnel analysis so we have a whole bunch of visitors that land on a website we ultimately want to take them to check out and we want to know the intermediate steps how many will add a product to a cart how many will click the purchase button how many will actually enter their credit card and how many will finally click the purchase button and it's called a funnel analysis because these diagrams if you were to draw them the other way looks kind of like a funnel as it gets narrower approaching the bottom so I started looking at my data just diving into what was available it was actually quite difficult to get this data there was more than a month of delay because I was first told SSH into this server and you can access it the server was actually running at 100% CPU and 100% disk it turns out to be the production server so then I asked them for could I just get access to the backups why are you giving me access to production they're like backups can't you just SSH in and do it there so I'm like do you mind if I take down production which obviously led to about a month and a month later I got actual backups so this is ultimately the data that came out oversimplified and so it looks pretty straight forward I have visitor ABC he entered his credit card at 12 and he purchased at 1 I have visitor DEF entered his credit card but decided not to purchase and then I have visitor Jkl who never entered his credit card but somehow managed to purchase I mean I would love to be able to do this on Amazon so let me remind you why it took me a month to get this data there was production server running at 100% everything there were no backups so just going in I kind of felt like maybe engineering best practices weren't being followed with this system so I inquired a little bit about the architecture so there's a JavaScript tracking pixel which tracks all these events, sends them to a single production server which is collecting quite a lot of data per second for some reason it's being put into hundreds of SQLite databases so there are easily hundreds of databases running per disk which for those of you who know how disks work let's say it's a spinning disk, a spinning piece of metal and each time you want to write to a different place it needs a spin, write spin somewhere else this is really not an optimal way to go traditional database will have one per disk at most and it will spin write a big chunk and essentially try to minimize disk seeks but if you have two disks two databases running on the same disk none of that works so I was just thinking is it possible that this data collection system itself is the reason for this is it possible that this system that I didn't write and have no control over maybe is losing some data so at this point what I want to do is I want to construct a probabilistic model of data loss so let's read our recipe a naive way to do this would be just take the data I have and I want to calculate a conversion rate which would be the fraction of people who make a purchase so I take the purchase time and if it's null I'll call that a false and if it's not null I'll call that a true so that's when a purchase happens and then I now have a boolean value which is either true or false I take it to mean and I will get the fraction of truths this is pandas code I'm sure it's slightly different in r but that's the basic idea and that will give me kind of an estimate of the conversion rate the idea I want to describe is I want to identify latent hidden variables which in this case are basically going to be what actually happened in reality how many people actually made a purchase how many people actually entered their credit card and additionally what fraction of data points is this server losing just due to all the odd things about its design and then the final model it's a very simple model it's just measuring a conversion rate but the point is what's going into this is a statistical estimate of the number of purchases made as opposed to just the count of purchases made by this tracking pixel so what we want to do is we want to model the generating process so the probability of a user of the visitor entering the credit card is some number which we don't necessarily know but that's one of the things we want to find out the probability of a purchase conditional on a user entering a credit card is another thing we want to know we also don't know it but these are the goals of the system we also want to know the probability that an event will be observed given that it occurred so if a person enters their credit card what are the odds this will be reported to us and as we kind of inferred from before that number is significantly less than one so the first two blue ones are interesting to customers and the red is interesting to us because that's sort of telling us what's the gap between what we built and what the system needs to be so here's some data reported to us summarized we had one lack unique visits 40,000 cases with a credit card entered but no purchase another 10,000 with a credit card entered and a purchase and 5,000 with no credit card entered but still a purchase so what we want to do is we want to back out what actually happened here this is a probabilistic model that comes more or less from basic arithmetic the fraction that entered the credit card isn't read because we don't know it but it's binomially distributed there's one lack visitors so that's the end of the binomial distribution and the probability is this unknown parameter the fraction we're going to enter their credit card the number we observed is again a binomial distributed variable coming from the number who entered their credit card which is unknown and the observation probability which is also unknown so in the second line the 40k is what we know literally in our data and everything on the right side is unknown and then as you work your way down this is basically ordinary probability theory just with a whole bunch of unknown variables that we want to find out so what we want to do from this is we want to actually find out the conversion rate so this is a probabilistic model it's based on some assumptions we're making about how the world works and more importantly how the data comes to us so pi mc is a great library when you have some complex probabilistic model and you want to come up with estimates from it this is if you notice this code is almost just this you write down your probabilistic model since it will typically be done in the Bayesian method you need to come up with some priors so for instance for the conversion rates I'm taking a uniform prior all conversion rates are equal equally likely, sorry not equal equally likely and then the rest of it is just describing our mathematical model in terms of both the things we know and the things we don't so pi mc is great if you want to find out models this way press solve it will do Markov chain Monte Carlo and it will give you probabilistic estimates and so the end result of this is if you do pi mc filly it will actually give you a range I'm just reporting the midpoints so with this implicit statistical model even though the conversion rate we can observe directly just by counting is 15% the actual conversion rate is 16.7% which is 11% higher than 15% and this comes from basically inferring that our rate of data loss is 10% and that means that we're actually having 10% more conversions approximately than we are reflected in the data so this is the key implicit modeling assumption we enter a credit card before you submit to skip this step so if this is false the whole thing breaks down but given this relationship in where our data comes from it kind of the rest of it falls out from simple probability so this is not as simple going to be as simple as just feed all your data into a deep neural network or into gradient boosting and expect something good to come out you really have to think about all the processes happening in the middle the key point is the data which is present and once you can come up with a better estimate of what's missing you can draw useful conclusions here's another problem that happens quite frequently data is just wrongly labeled formats are just different across different sources and you have to bring a whole bunch of sources into one so here's a problem I face that's simple this is my phone it's a fancy google pixel I'm mostly in Bangalore sometimes I would see in laws in Hyderabad um and so typically speaking simple should kind of expect my purchases to be in these locations they should expect my purchases to be happening on this kind of a phone suddenly simple gets a valid request to make a purchase from this device whatever the heck it is some weird looking Nokia thing in Jaipur I went to Jaipur once it was pretty probably won't go back does this seem right is it possible that maybe it's not actually me it's someone else who would like to make a purchase on my account most likely and so probably this transaction should be stopped someone should call the actual registered phone number confirm the purchase have you given your OTP to anyone on whatsapp we don't use whatsapp ask some probing questions of that nature so brilliant plan if devices don't match the historically used devices let's flag it as potential fraud escalate to the fraud calling team don't let the purchase happen call the customer up check if they've shared the password check if they've given out one time passwords that kind of thing so then I looked at my own data this is my device history across several different simple merchants and as you can see that capital Google capital pixel to excel is not equal to the other thing there's also a version with a plus which is some kind of url encoding there's all sorts of funny variations that happen like this and basically what's happened here is that these are all coming from different merchants who've implemented our system these merchants haven't all implemented it the same way there might be other steps of the pipeline somewhere in between the merchant and me that also change this stuff so there's a lot of people involved in this process there's the merchant partners and there's the business development team that's usually the one communicating with the merchants there's a product manager somewhere in the way who helps integrate merchants with simple there's the engineering team engineering team actually does this almost perfectly so getting this fixed would be a team effort I don't really believe that teamwork makes the dream work our merchants you probably you can go outside to our booth and see who they are mostly they don't care that much about stopping a small number of fraudulent purchases on our platform a fraudulent purchase on their platform is still a purchase they get paid we're out the money our business team really wants to close the next big sale like it's very reasonable for this not to be their top priority so getting something like this fixed is not going to happen quickly and quite likely may never happen and I'm just going to accept that instead I'm going to try to solve this problem with linear algebra which I actually enjoy a lot more than trying to wrangle up a whole bunch of people to finally feed me good data so sorry this is my phone I'm going to start with the data the strings I have in my data are a reflection of it but they are not actually reality so I'm going to model things this way the data is something that comes from reality but is an imperfect reflection of it my data set is of course going to be exactly what you see here it'll be my user ID there is a specific partner merchant where the device is being used to model and this is a function of my data but it is a function of reality but it's not true reality so since I like to solve problems with linear algebra I've actually been doing this since the number 8 days which Peter and a couple of others might remember I'm going to set up a matrix so my columns are going to be reported device strings so each column and a row will be a user so that first row means that particular user showed up with that Google pixel XL2 capitalized as you see on merchant A they did not show up with an iPhone X on A on merchant B they showed up with a Google pixel 2 the word XL missing on merchant C they never showed up and also on merchant B they did not show up with an iPhone 10 and actually the second guy didn't show up on merchant B he did show up on C so those are all reports and that last row there's something a little funny there is this person showed up with a pixel XL2 on A and an iPhone on A so this matrix is highly incomplete most users don't use most merchants so it's a matrix completion problem and we have to basically sort of infer what the NaNs are in the matrix the other observation we can make is that this matrix is low rank there are a lot of string each device generates a lot of different device strings but the rank of this matrix should essentially be the number of devices that are actually out there in the world so essentially if we go back to this matrix the Google pixel shows up as a 1 in 3 different columns in this oversimplified matrix so 1 0 1 1 0 is what generates the pixel 0 1 0 0 1 is what generates the iPhone and these are the things that should essentially be repeated so the rank of this matrix this matrix is essentially two different linearly independent vectors repeated a bunch of times so the rank of this matrix actually should be the number of devices not the number of device strings which is much larger so we have a matrix completion problem and it's very low rank if this seems familiar those of you who work in recommendation engines may have seen it because that's how a lot of recommendation engines work so can we use that to deal with our data problems basically take techniques from recommendations and solve this and the answer is yes we can so we know our matrix should look like this low rank approximation so we have our user vector tensor products are device vector and that essentially represents all the people who are using the same device and then there's an error term which will come from basically those weird fraud cases or a person who uses two phones and those should be uncommon so as I said this is kind of like a topic model the tensor products are our topics and the error term which in our case are attacks are directly analogous to you know if we think about topics let's say a person's major topic is violent action movies but also for some reason he likes the movie Notting Hill just one weird one off case that doesn't say he's in general interest in romantic comedies he's generally violent action movie guy but also he just likes that one movie for some idiosyncratic reason and then the user is a document and an attacker is a document that fits into multiple topics so if we so remember this matrix is highly very much non-square there's a lot more users than there are devices so what we can do is if we want to get a nice symmetric matrix we can just compute m adjoint m and if we look if we do some simple arithmetic on the ijth component here we discover this is the number of users who were seen having device i and also device j simultaneously and as per our model before this is so this is not strictly true this is stylistically true there's a bit more arithmetic that I'm suppressing because I don't want to make this take too long but essentially this matrix can be thought of as a bunch of topics which have size n n is roughly the number of people with a given device so there's a lot of people out there with a pixel there's a lot of people with the iphone and then there's error terms which are on the order of size one which will basically be how many people use a pixel and also a strange little Nokia thing how many people use an iphone 10 and also a highway 2600 the number of people who simultaneously use an iphone will be a much smaller than the number of people who strictly use one so what this means is that and by the way in this what I've done here is I've taken all the NaNs and I filled them in with zeros so what this means is that I should have a set of symmetric matrices and that dj tensor product dj that has eigenvector dj by definition and it's of size n I also have a bunch of weird one-off terms that I'm not interested in which are basically going to have a one or a two there so I can take this matrix and I can threshold it I can basically find some cutoff between the typical number of people who have a given device and anything below that I'm just going to set to zero so if I do that then what I can do is I can just find the eigenvectors of m adjoint m and after the thresholding these right singular these vectors are going to be the right singular vectors of m and therefore essentially it's going to be a bunch of ones or actually it's not going to be ones but there'll be non-zero entries in certain places these the non-zero entries mean that this device corresponds to these things and everything else is unrelated so essentially each dj is going to be zero, non-zero, non-zero a bunch of zeros, some more non-zeros and all the non-zeros are a device that's the same and then when we put this into production for any user we will map their device string to whichever vector has this as non-zero and then if we see another device string for that user that is not part of this non-zero in this vector that is potential fraud and worth flagging so how do we know this works so first thing we have some string patterns we should always start with the simple stuff and we did so for instance we know that pluses are url encodings of spaces so if we do that replacement and then try to check our data we know that Google plus pixel should be the same as Google pixel and this matrix technique when we don't do that as pre-processing first will reproduce the results it also tells us things like HMD global is Nokia which you can't really get from any string Howe is Honor and a few things of that nature which once I Google them it turns out there are just different words for the same manufacturer and then also as you would expect users with multiple devices are actually pretty rare which was kind of an implicit assumption that this model reproduces so this is a reasonable security feature it's hard for scammers to exploit because the only places they could exploit are devices that are rare and you have to find a user with a rare device in order to exploit them which dramatically reduces the pool of users you might want to attack and also keep in mind a lot of our attacks are some guy on whatsapp who took our logo and made it his whatsapp profile picture and he's like hey sir I'm from simple support in order to avoid adversely affecting your credit I'm going to send you an OTP send it back to me and then the user doesn't notice here's your OTP do not share it with anyone they ignore that second bit and then they're like hey who made this purchase so it's very rare so there are now the pool of users who could be attacked this way has dramatically reduced and this guy on whatsapp suddenly has to know you have a rare phone to even get away with this so ultimately this is a reasonable way to do it another use case here is the following you could imagine doing some string matching maybe some dictionaries to match up these devices what if the data were hashed to preserve customer privacy in that case we will have no way at all to do a string matching because we only have a hash of a we have a hash of a google plus pixel we also have a hash of a google space pixel we couldn't do string matching on the hashes anymore but a technique like this will still work these are sort of examples where data is messed up because of our pipeline in principle if we devoted some enough engineering efforts to fixing them then it could be resolved but now let's move on to places where your data is messed up and there's nothing even the best engineer in the world the best team could do anything about it so this is a fairly pervasive problem um I send an email today I'm interested in its conversion rate the number of people who open the email click a link in the email and then ideally later on make a purchase on some website use a coupon code something of that nature so I sent this email out just now I checked my stats and in the past 12 seconds nobody read it should I assume that my subject line sucks it would be a little unreasonable because how many of you just checked your email in the past 12 seconds probably not a lot on the other hand you know 30, 40 days later if nobody read it pretty much nobody ever will this is a delayed reaction problem if I measure too early I get a biased measurement of whether an event will occur and if I measure late I'm usually a day late in the dollar short sometimes you have to take action more recently so this is really pervasive like a simple we issue credit the data we're interested in is 30 days delinquent usually that's like a great metric that more or less tracks our business and it takes us 30 days to discover that so in May I ran an A-B test gathered some data tried to learn something and on July 10th is when I actually had the 30 day delinquency data to measure it um similarly when you're trading stocks you buy it you have to wait to see if it went up or down and you might have to make decisions in the meantime so here's a really concrete example you want to measure a conversion rate and I go back to conversion rates because the actual math problem the actual predictive problem at the end of the day is really simple the hard part is dealing with the dirty data the model at the end of the day is easy so let's look at the basics without delay if you have a conversion rate gamma the probability of gamma giving a conversion by a Bayes rule is you take your prior and you multiply it by gamma if a conversion did not occur instead of multiplying your prior by gamma you multiply it by one minus gamma and so if you had n visits and k conversions you could do recursion and you would basically take whatever your prior is a flat function one on the interval zero to one and you multiply it by gamma to the k one minus gamma to the n minus k and if you do this what you're getting is an estimate of your conversion rate in this case it's about 0.15% with a margin of error that comes out very clearly from the posterior distribution so this works great if there's no delays but again if you do this 20 seconds after sending an email I'm telling you much so what we want to do is we now have to add time into our modeling process and model how many people would have checked their email would have read this email if they ever are going to so our data set now instead of just being a count of count of email sent to count of conversions is a set of times the time between the first email that was sent the second email that was sent and the present and it's also the time between when you actually opened or clicked or whatever a conversion event is so this is sort of what I was describing before is that if you wait a very long time you have great information if you wait a short time you have that information and what do we do in between is the question so the solution to this problem is something called the survival model and there was a talk about this earlier today I hope at least some of you attended I'm not going to go into how to find this but the output of the survival model which describes assuming this event will be observed what's the probability it's observed before a certain time t I think in that talk they actually called it s of t and it was the probability of observing it after but this would just be one minus that so survival curve would look like this after I think according to this 8 there's like a 95% chance it's going to be observed after a time 2 days it'll be about .4 so if we go through the same process we did before doing Bayesian stats we discovered that the probability of a click before time t given a conversion rate is instead of just being the true conversion rate it's the true conversion rate times the probability of the click being observed before t and same thing with no click you just have to multiply the conversion rate by that survival curve if you do the same recursion you get a slightly more complicated function that is still not that hard to implement in Python and so if you use that function you can actually get unbiased albeit more uncertain estimate of the conversion rate so in this example here the green line so this simulated data I had true conversion rate of 25% the green curve which is fairly narrow is if I don't correct for delay bias and I just use the simple Bayesian formula to make my conversion rate and that's way off if I use the survival curve to correct for the bias what I can do here is I can get a correct estimate albeit with more uncertainty and as we wait longer of course the estimates converge to each other so this is the same simulation run but with more time between let's say send the email and measure the conversion rate so the key point here is that in the data we have first reflection of what the final email will be there's two important factors how good the email is and how much time has passed and if we ignore the way our data is generated which involves time we're not going to have the most accurate estimates of it but by incorporating time we can actually correct things so I'm going to close this talk with kind of an open problem that I'm constantly thinking about and never have a good solution for every solution I have involves spending a lot of money so you want to build a selection algorithm and the training data comes from the people or ads or whatever you've selected so you want to estimate the conversion rate of an advertisement the data you have about click-through rates comes from the ads you display it does not come from the ads you do not display when we issue loans we get performance data we lend money to we get no performance data from the people we reject same thing happens in education if you reject a student you don't know how you would have performed so this is sort of a really simple model we have some regressor and we have performance so you can think of the x-axis as being like a credit score and you have the y-axis being fraction of people who repay now we've trained this model which is the black line, that diagonal and it seems pretty accurate so we only want to approve the good people which are the red folks up here so we restrict those since we are interested in loan performance we will only issue loans to the people in the red and we get high performance and this goes great then we retrain our model and we get a new curve and suddenly this says that higher credit score is worse that's not a thing I want to put into production but this is a thing that will happen if you're not careful we've actually just by chopping off our data we've turned a positive correlation into a negative one and every so often on twitter in the news you discover some study coming out that discovers something weird and counterintuitive did you know that height is irrelevant to the performance of basketball players it turns out in the American NBA people who are 6 feet tall, people who are 7 feet tall equally good no correlation now you'll notice that there's basically no one in the NBA who is below 6 feet but usually when this is being reported no one mentions this is exactly the selection bias what's happened here is that there are really tall people who are maybe not as athletic so they get into the NBA because height helps there's also on the flip side there's really athletic people who are not quite as tall but they jump higher so both of these folks get into the NBA and the correlation reverses itself this was a recent one that went out about a year ago everybody loved it it said that GRE is useless in predicting the success of people who get into grad school and then a whole bunch of editorials were written suggesting let's stop using the GRE for grad school it doesn't work now it turns out that basically no one with a really bad GRE actually gets in and therefore they're excluded from the data when they are the generators of the correlation so this thing happens almost everywhere it was first known in the econometrics Heckman received a Nobel Prize for this, George Bourgeois was one of the first people to actually do anything with this and there's something called the Heckman correction which lets you mitigate this issue for linear regression which is what econometrics almost exclusively does fantastic methodology here's an open problem I would love it if anyone has ideas on how do you do this in gradient boosting machines that's what I use so I would be most interested there but even how would you use this in a neural network what Heckman did is almost exclusively based on linear algebra and normal distributions it can't directly translate but how can we solve this problem the way I solve this problem is when I issue loans I issue loans to a bunch of people I know are not going to pay just so that I get their training data this as you might expect costs my company a bunch of money we have to do it because we have no better way forward but if we could solve this problem it would be valuable to everyone in the world you have some ideas on this come find me at the simple booth I would love to make progress on this anyway that's where I'll leave it I'll just say that all of our data sucks and we just need to learn to deal with it okay thank you Chris and Peter can we have Peter also back now we're gonna have a joint Q&A with them if those who have questions we have a few mics yeah people who have questions for Chris I think they were both lovely talks on at least so we have one question there yeah so my question is to Peter so I have immense respect for Anaconda thank you for building that it has helped a lot of people and reduced the friction for a lot of people to get into this field so one of my questions is what events happened through your journey while you were getting python for people to use your product so what all events happened that accelerated your growth so did something specific happen which you know distributed your platform to a lot of more people maybe you gave a talk somewhere where a lot of people started talking about your platform that is the first question next is at the end of your presentation you talked about serverless deployments so let's say if I have a TensorFlow Lite model which I want to deploy on IoT devices where does Anaconda come into this whole picture so wow that's really loud okay the first I think the there's been a pretty steady organic growth for the usage of the Anaconda open source stuff and really there hasn't been any one thing that dramatically accelerated its uptake I think if one looks at the data there's always been a seasonality in synchronicity with the school years and that always said to us that data science has been brought in universities and especially when we look across India, China in particular almost more so than in the west there's that seasonality shows up so there hasn't been any single one thing but flipping that around if there is this latent sort of exponential growth of adoption latent in the space how do you not mess it up and that can also be part of the hidden thing that you don't see but there have been many things we've had to do to keep ourselves from breaking ourselves into jail and I'll give you two very interesting examples one of them is when Wes McKinney Justice Condo was really starting across the threshold for adoption by the core sort of PI data site by developers and it's not one homogeneous mass they all have different viewpoints Wes is the author of Pandas and he's a friend of mine but he wrote a blog post about how we really need to there's a fedora moment for conda as a community tool and you need to create essentially we need to somehow create a thing independent of anaconda that we can sort of rally around just like fedora is essentially red hat stuff without the red hat corporation control and it was an interesting blog post and I could see his points there but the community also recognized how much we were doing to support the community so just as a FYI our CDN and our servers we push out almost a petabyte of traffic every month that we pay for for everyone to use and that's not counting like university mirrors in china and other places and so when Wes advocated for that we worked with him and we also worked with other people who had an interest in buildings in the infrastructure that now eventually has become conda forge so we've added special things for conda forge into anaconda.org and that helped us we embraced the community and served their needs even though it was creating something that could potentially be a fork of what we do and embracing that we created a better healthier relationship and build more trust if we had said no no we got to shut this down and did all this kind of like closing things down that would have been even worse right so it was one thing and the second thing that really was important thing for us was when the they're now more and more commercial platforms that are trying to do the same thing that we do with our commercial platform and they use anaconda as an open source tool inside it and a few executives that we had hired in were very much of the strong opinion that we needed to basically put strong legal language in place to shut them down completely because we were sort of beating us them in sales and then the CEO myself we came out on the other side that we said no we don't want to do that we're going to basically out innovate our competition as opposed to try to dig a moat and if we had dug a moat and we put that stuff in there there's a chance we could have turned the community against us right so in both cases it was a community embracing innovation embracing adoption as opposed to trying to fight it and do it on its own terms oh yeah so the second question really quick we're not doing anything in particular support serverless just yet but it's somethin