ent called decisions operations and information technology which is a nice overlap of people from operations research industrial engineering computer science economics statistics so and the main and we're housing the business school there's the main idea is to actually work closely with business application rather than just building theory on its own which can actually sometimes happen in the cadmium departments so what i want to talk about today is using structural estimation methods which have their roots in economics and more than a concrete method they are more uh so to say uh guiding philosophy or a paradigm and the idea is actually to follow what the data generating process more closely like think deeper into what's really going on as it's not too different from the idea now popularized by ellen musk which is to think from first principles so that is something that i'm going to do today and these models actually take up there is no concrete formulation they take up various forms depending on the exact problem that you're studying so i do have a very concrete application in mind which is these bike sharing systems and so in general you would want to apply these methods when you don't just care about making predictions from certain inputs to certain outputs but you also want the discovered relationships to make some sort of a decision on top of that so with that let me just start so when i'm thinking about solving a particular problem especially in these kind of contexts i'm thinking of my model to have two main features in them so the first thing i would like to have my model to have is the idea of interpretability that is i do want to understand how my inputs are driving my outputs i am not really comfortable with using a black box model and the advantage of doing that is basically one it's very easy to communicate to managers and stakeholders what's really going on so that they are more comfortable in using the output of the model in making some decisions based on that setting up prices making some sort of a decision in terms of acquisition and things like that in addition catching of errors in our inconsistencies in your data becomes much more easier if your model is really transparent otherwise the things would just flow on without you ever noticing that there is something else there could exist many biases could exist in your data especially around some sort of a racial biases or gender-based biases and if your model is not transparent enough you might just let that pass through whereas if you actually have a good understanding of what's driving input to output you will actually be much more cognizant of what kind of biases exist in your data and finally once you have a very transparent view of what your model is you actually are more cognizant of what your limitations are and therefore you probably not apply to situations where you should not apply your model and that could be a good thing so that's the first thing i'm looking for in my model is my model should be interpretable it should make sense to me and the second thing which again builds up on avi stock is i would like my model to have causal relationship and so we have we are very familiar with this idea of correlation is not causation you have many funny examples running around internet this is one example where there is a huge correlation between people drowning after falling off a fishing boat and marriage rate in certain state and it would be actually foolish to say one thing drives the another and somehow try to influence one thing to make an impact on the other thing and that's the main idea and you want to establish causal relationship if you want to use your model to make any kind of decision making so this idea is well understood except what really happens in practice is we get so hung up on throwing thousands of models at data and sort of chasing this measure of accuracy and then finally seeing what really works that this idea somehow takes a backseat and we never really come to making sure is our model a causal model or is just an association and things can go really wrong if we are just willing to get ignore this particular thing so that will be the focus of the model building exercise that i'll do next so let me get to the application that i have in mind which is the bike share systems so just to get an overview of what these systems really look like so bikes i mean bicycles here it's a little bit different from how we use the term bike here but basically the idea is that you want a bike share system to be used by public as a public mode of transportation usually shared by different people so basically you have these kind of bikes which are sturdy enough to be used by many people typically you have these stations where there are these empty docks and the bikes can basically sit in those empty docks and so the way these would be used by people is basically you just go sign up have some sort of a daily monthly yearly subscription you then go to a particular station swipe out a bike ride it to any destination of your choice doesn't necessarily have to be the same place that you started with and that's it you finish your journey so that's how these systems typically work how people have you typically use them is for a lot of short trips going from one particular place to another particular place also one popular use is sort of this last mile connectivity that you are planning to take a long journey but you do the initial first mile using a bike share system go to a metro station take that metro tip and then again take a short bike trip to your work destination so that's typically how these systems are being used now what I'll talk about today will be in the context of bike share systems but a lot of that will just apply much more broadly to systems which are similar to these so in particular I'll be focusing on two features of the systems so the first is the location aspect what I'm thinking of in what is important in these systems is the location of a user and location of where the supply is say nearest bike is is quite an important factor in determining whether that match really happens okay so that's the location or the hyper local aspect of these systems and the second aspect is the on-demand nature of these systems which is you don't typically reserve these systems well in advance you just have a need for one of these let's say a bike trip right now and you then of an open an app go to the nearest station check out a bike right then and there and so this temporal on-demand nature is quite important in these systems and well if you think about other smart mobility systems be it the dockless bike share systems that are coming up now zoom car is launching the pedal system ulla has something so all of that have both these location and on-demand features if you think of uber if you think of ulla again very similar driver has a particular location I have a particular location and for the match to be happened they have to be close enough and they're at the same amount of time and again more broadly around delivery platforms be it swiggy big basket and so on all these kind of applications have very similar features so a lot of what I'll talk about today will apply just much more broadly in systems like this and even so the very concrete system that I'll be looking at is this belief system that I've worked with which is a huge bike share system in Parish so it was launched in 2007 it's actually one of the most popular bike share systems around the world also was one of the first modern system of its kind a lot of other systems have actually followed this system in how to design it so it has more than 1200 stations about 20,000 bikes hundreds of thousands of subscribers in terms of number of trips it has done over 173 million trips in last six years and has a huge impact has had a huge impact in terms of say the carbon impact and saved sort of a car journey trips being replaced by these bike share trips or a lot of health benefits as well where people actually bike instead of sitting in a cab and just driving to this place so that's actually one of the fascinating angles as well for me to be working with these kind of systems that there has to be there is this potential to make an impact in either a climate or a health sort of a effect to the community in addition to just doing data science and these kind of things so in particular when I think about these systems a lot of the concrete problems actually depend on understanding two main features of how users behave in this kind of systems so the first is it is important to think about how users think about distances that they have to walk to access one of the nearby station so that plays an important rule in understanding if I want to see how a user behaves if the distance to walk is 100 meters versus 500 meters how less likely are they to make that journey and subsequently use the system that will actually play an important role and the second aspect of the system which actually underlies a lot of problems that we can study about these is how people think about availability of a bikes which is say I go to a I have this preferred station somewhere and I usually go and check out a bike and do my subsequent journey but at this point in time the station does not have any bike over there so what do I do next do I actually substitute to a nearby station in which case I still make that journey using the system or I just abandon the system and do something else let's say order will our Uber and just not make a trip what how do users behave with respect to that availability feature and why the understanding do you think these two things are important is because if I think about a lot of questions around the system be it where should I how should I think about where where do I locate these stations well it's important to understand how people think about walking distances if people are very sensitive to walking even a few hundred meters then I actually need a lot of stations in the city whereas if people are okay walking 500 meters or even more then I don't need as many stations so that will actually guide the design of how dense I want my system to be once I have an understanding of how people think about walking distances or if I want to understand about how should I manage the inventory of the system how many bikes do I need how should I think of rebalancing bringing bikes from the full areas to the empty areas again what will guide those those algorithms or those strategies will be basically how people think about availabilities are people very sensitive to find not finding a bike at that station in which case I'll have to be very careful making sure the supply is there at each and every point in time whereas if people are okay not finding a bike once in a while and actually substitute to nearby stations well in that case I don't need to be so stringent about my inventory management policies so that will basically the agenda of my talk today I'll try and uncover how do people think about walking distances how do people think about bike availabilities and then use those primitive user level primitives to actually guide a lot of system design kind of questions for these systems and and it will turn out that actually these are a little bit non-trivial problems I'll first illustrate how traditional problems struggle with capturing a lot of realities for these for modeling these kind of features around users and how structural estimation methods can guide us into getting a better answer okay so the data set that I'll be using primarily comes from the belief system in Paris so this particular primary data set looks very simple I'll complement it with a few more data sets from other sources but basically the idea is that I observe each and every station at a very regular interval every two minutes in this case so what I have is this station ID a particular station 666 and every two minutes I observe what is the number of bikes those systems have so that's it that's the basic structure of the very simple data set that I'm starting with and once I have that I can uncover what is the demand at each of these stations at every two minute one bike was taken out three bikes were taken out and so on so that's the number of trips originating at each of the stations and also I can construct from that whether a station has any bikes or does not have any bikes so this binary measure of whether that station is sort of available for use for users or not really available so that will be an important factor as well in understanding the availability aspect so that's the basic structure of the data set now let's actually come back to a question which is we want to understand how people think about walking distances so another way to put that question is let's say this is my existing design I have four of these stations in my city blue station yellow station a pink station and a green station and say I want to move this blue station or I'm considering if I want to move this blue station by 100 meters and based on the data that I have collected so far in how people are using these stations what is the demand at each of these stations I want to say something about what will be the new demand once I move this blue station by 100 meters right so that will be another way to put this accessibility data so let's say I start in a very traditional way basically the dependent variable that I'm thinking of is demand at this station f and time t so that's something that I care about and I want to build them all what guides that demand okay so what I start doing is basically building in different features and how what could impact that demand so let's say I start with some neighborhood characteristics say if there is a metro nearby there is a is it the commercial location are there cafes nearby and so on so all of that should have some sort of impact on if a station is popular or not really popular I can include some weather characteristics whether it's raining whether it's a hot day what's really going on in terms of other humidity and so on so that could actually have a potentially an impact on the demand at the station but remember I also want to have some sort of notion about where the station exactly located the distance aspect in how that is driving users to actually use the system or not use the system so one way to think about accessibility here is I can include a measure for what is the distance to my nearest station so if the nearest station is very close well in that case typically the users who are coming to this station do not have to walk all that much so they might be willing to use this station more often so that could be one proxy for how I include the distance effect in my model which is distance to nearest station I can go a little bit crazy I can include distance to second nearest station third nearest station and so on right so those all of that would be kind of a proxy measures for including this distance effect in my model but then I take a step back and I think have I really captured how the demand is driven by the location of these systems well turns out not exactly because by including this distance measures let's say I'm focusing on this blue station over here and my measures say that well the nearest station is 100 meters away and the second nearest station is 200 meters away the design could still look either like this or like this in either case my features look exactly the same whereas we know just by looking at this that in this scenario this blue station would have more users coming to it just because there is no other station in this particular area so that spatial positioning of station is kind of not really captured in this distance measures that I'm including here so I can then well go ahead and try to construct some features which actually capture not just these distances but also some sort of a 2d measure of how they're positioned around this station but essentially what I'm really doing is trying to come up with a lot of proxy measures to capture this single notion of how people think about walking distances so it's kind of getting a little bit not that clean if I think of some other things which is I also want to model that well there is the metro station which is 300 meters away so that's the distance part but also has a lot of people coming per hour let's say there are more than thousand people using this station very regularly again to include that kind of feature in this particular model I'll have to come up with a lot of interactions building around the distance aspect and the number of people aspect and what's really going on here is I'm including a lot of proxy features a lot of interactions and my accessibility aspect is really getting scattered all over the place I'm not really getting a hold of the thing that I'm really after so let me actually start from a little bit different angle instead of using this particular approach which is what we are really familiar with having an output putting in some features let me actually start thinking about what really drives a particular unit demand at each of the station at time like what really happens to make a user use that particular station at a given time so think of this as one particular use case which is a user got out of a metro station and now is thinking how do I get to my work and he has a few choices which is either to order Ola and just not think about bike share system at all or he knows that there are a few stations which are a certain distance away from him and he can actually go and use one of the bikes from these stations and that could actually that is what generates a demand at the station so if we think about the demand generation process like this then we are still capturing this accessibility effect and the way that happens is basically say if I move this blue station away by 100 meters and now I think about this process again that there is this particular user who got out of this metro station and really now consider using this blue station well now the station has moved much further away so probably he'll be a little bit less willing to use that station so that might drive the demand at this blue station a little bit lower whereas there might be other users who are somewhere over here and they now may want to use this blue station just because it's like very close to them so actually by starting from this very user level paradigm I have in a much concise and neater way captured this accessibility effect so I'll dig deeper into have I really captured it all those problems around spatial positioning of stations and things like that do they really fit into this particular way of thinking about the demand generation process but that's the main idea is basically I am following a user instead of just putting in a demand ft and plugging in features to explain that particular demand so let me formalize this thing what I'll okay so before that actually one thing I do want to mention is I don't exactly have user location data right I don't know where the users are coming up and where are they now starting to think about okay there is a nearby station and I want to go there so that will become a little bit of a challenge but I'll get around that and one more thing to note is that a lot of applications these days are based on these apps which are able to track user location so in some cases you might have user location data and I'll talk a little bit later in if you have that data what kind of change the model required to incorporate that data and does it really solve the challenges that I'm talking about or are there still some challenges to think about so so let's start with this model building process so it's it's fairly simple design I'm still formalizing that intuition which is following the user so in the first step I'm allowing my users to originate at different points in the city so at every single point in the city which is this location l it's a pwl it basically captures the rate at which users are originating in my city and that I can make that as a function of lot of city characteristics right so I know that a lot of people will originate where there are metro locations with a lot of traffic where there are a lot of tourist locations where there are grocery stores cafes museums things like that I can also include some sort of a census data in how many people live there how many people work there what are the demographics and so all of that will basically guide the number of people originating at different locations of the city so these are not the people I'm not thinking about using the bike share system right now it's just people are originating and they could be potential users of my system and then there is a second aspect which is this neighborhood bike availability which is if people are more confident that they'll be able to find a bike then more people will sort of originate at those locations so that can play a little bit of a role in your density model so at this point these are the people who are originating and then once they've originated at a location now they start thinking about if I want to use this bike share system or if I want to do something else so to formalize that process I'm using a choice model which is which is not too complicated I'll just give you a brief primer on what that really means so what I'm saying is well there is this user and the way he's thinking about this system is he has a few options and for each of those options he writes down what is the value of using that option so that is what I call a utility and that can be function of what is the distance you have to walk and some other station level characteristics and once he has utility from all different options including the utility from an outside option which is say using an ola the option that gives him the highest value he basically goes and chooses that option so if that turns out to be this particular station this blue station well then you have a demand at this station and if it turns out something else well then in that case you don't have a demand okay so for a user this is a very deterministic process choosing an option which gives him the highest utility but then there are certain components of it which are only known to users but not known to us which are these error terms we typically have a standard distribution for those terms so essentially what we are going to do is from this choice model we are going to assign probabilities of a user look originating at a particular location li of using some station f at a location lf okay so what I'm going from here is writing down these particular probabilities of a user i using a station f at time given that he's originating at particular location li and all that distance aspect is basically embedded in there okay so what I've done is basically two things one I have created this model of where users are originating in different places in the city and the second is once they originate how do they think about using either my system at different locations or something else so my demand model is basically now very easy it's just a composite of these two things if I just integrate over all the locations in the city where are people originating and how they make a choice well then I have gotten a function which describes the demand at each particular station at a given time so let's dig a bit deeper into what this model really means has it really captured the distance thing that we are after or the by capability aspect that we are after so well so say if I think about a particular user and if there is a station which is just further away from where my you all my users are will that station dry have a smaller amount of demand well my model would say yes because there is this utility part in the model which depends on the distance to the station and stations that are further away will make them less attractive to users and therefore users will not show up there is the effect of density of stations captured in my model if there are a lot of stations in the city well then typically users would have to walk smaller distances to access these stations so again the utility model that I had there will make stations more attractive just because they have to walk smaller distances and so having a higher density of stations will make users use the system much more so that's again embedded directly in there without me having to do anything explicit and the third important thing is which is this basically this spatial positioning of station that is you have a station here and you have a station nearby here and you have a station nearby there does it have an impact on demand of this particular station again I did not have to do anything explicit it's just naturally in there because once I have a station that is nearby user always over there I find that station much more attractive than coming to this particular station so again the utility model just drives all of these dynamics very naturally without us having to really sit down and think of each of the corner cases and design these proxy feature variables to capture that so that's the sort of beauty of these structural models that I'm talking about once you follow the data generation process a lot of things just are included very naturally and the same thing about availability if a station is out of bikes well that does not show up in users choice it so very naturally users with their substitute to nearby station or use say OLA caps and again those ability aspects are being captured there but so let me actually come back to this particular aspect which is if I do have location data how would that player own in this model so it's great if you have the location data but then we have to be a little bit careful about what that location data actually means because remember the location we are after is a user it has particular location and he's considering whether to walk this particular distance to use the station from this particular place let's say so if you have this particular user whose journey looks like this that let's say he's at home right now and there is this first metro journey he's taking from metro station one to metro station two and after that he is considering going to this blue bike share station he can open his phone and check the app at two different places so one is after finishing that metro tip he can check his location or he can actually be more proactive he can check sort of what is the state bike share station status even before taking the metro journey so if you're not really sure if the location you're capturing either this or this this would actually be a bad data supply your model so this is something that you'll have to be a little bit careful about again not all the users are often checking the apps so you probably don't have location data on all the users and that could actually be a little bit problematic do you ignore those users what do you do about that missing data so that is something that you'll have to think about the most important aspect is well you might have location data on your current users but what about those users who are currently not using the system but if you were to add more stations or if you were to improve the inventory policies they would actually start using your system know nothing about them because they are not currently users but to make any sort of counterfactual analysis using this data that is if I were to add more stations who is who's coming to me you want to have some understanding of what are the locations of these non-users and that is actually very hard to get so even though you have the location data you haven't actually completely solved the problem and there are a few more problems sort of that comes along with that so this is something that I'm currently working on so like including how to actually exactly use some of the location data but that doesn't seem a straightforward process that's the main idea so what I'll do is I'll go back to assuming that there is no location data just because that's more concrete to work with and what I've shown you is a very bare one model but you can easily enhance that in any way you'd like so if you want to include let's say the type of commuters so some people are just commuters who actually know where they're going and their behave maybe differently compared to let's say tourists who might have very different preferences in terms of what they want to do with the system you can actually just tweak your density model or the utility model to have different parameters based on user type you can have based on intention of users be it a food run or some sort of exercise kind of trip or going to the nearest grocery store that can have different user level parameters based on that you can include different pricing data you can include heterogeneous parameters based on what kind of neighborhoods people are using from or what is the time of the day depending on that some of the parameters might change so all of that can be very naturally included because now you have a good understanding of what actually each of the parameters really mean okay this is about this user they're originating and so whatever you want to tweak has a very clear place in the model that's the main idea all right so before I get to estimating this model and making some sort of policy design kind of questions I want to think about my estimates am I going to get correct estimates that I'm after or they might be misleading in some manner and so that is what is called endogenetic economics and it depends on what is your data like it's it's generally a good exercise to understand what kind of variation you are using in your data so the first thing to note is that it's an existing system and then you are thinking of how do I change that system so whenever this system was built the stations were not just located randomly in the city there was some sort of a logic to placing stations in the city so more often than this you would have been this scenario where there are more stations in the more popular areas be it let's say the downtown area where there are many amenities around and so what how that would be a challenge to the data is basically if you look at these stations they have a high demand just because they are in more popular areas and typically users who have to access these stations have to walk smaller distances so if I just naively use this data and fit my model what I might conclude is that it is these smaller distances which are causing these stations to have high demand but that is actually not the case high demand is because these areas are just more popular to start with and that would actually bias my estimates if I'm not cognizant of what's really what really my data is and similarly about bike ability which is basically if you have stations which are really popular then a lot of bikes are checked out very fast and that would actually lower the fraction of time there are bikes available at that station so what you'd see in the raw data is stations with very low availability have actually high demand and that would be a wrong conclusion to make that it is the low ability which is driving a high demand and so you need to correct for this reverse causality from the dependent variable to the independent variable so we need to correct for these biases and so the way you can go about them is in more or less two ways so the first ideal way is to do some sort of a b testing that instead of just relying on raw data you want to do this control manipulation of the system features in this particular case it actually is a very expensive exercise to do this so if I'm thinking about the station location to do any sort of randomized experiment well I'll have to include some new stations at really some random places I'll have to allow some time to for users to figure out that there is a station incorporate that in their commute I'll have to change all my inventory rebalancing policies based on that change that I've made in the system and at least as a first step that can be a really expensive thing to do so what I have instead is some sort of a modeling solutions for this kind of variation that you want to use in the data and so what I have his first thing is just to control for all these features that might just bias your system which is basically you show in the station location example that I gave before that there are some popular areas and so you want to include what makes them really popular it is the metro stations etc etc so that your model attributes the high demand to presence of these other amenities rather than the smaller distances on top of that what you can also use is this idea of instrumental variables which are which the main idea is basically this that find some variable z which is outside your model that affects the independent variable x in that you have and once you have x in the model there is no place for including z in the model just because z has doesn't directly affect why doesn't directly affect why but the way it can be useful is let's say this z takes two values z low and z high that can actually make x takes value x low and x i and you can use that variation as some sort of a pseudo experiment to figure out how does x impact why so that's that's the main basic idea and then you can come up with many instrumental variables like in this example there are bikes coming in at that station which serve as a shock for the ability at that station and that can be used an instrument variable to figure out the effect of changing availability on the demand at the stations so this is the kind of optimization process that i used to figure out my estimates in interest of time i'll not go too much into it but basically what i'm using is this idea of generalized method of movements which is just the generalization of the kind of objective function that you have for linear regressions or maximum likelihood estimates where the idea is that i'm just imposing this condition that these instrumental variables should be orthogonal to my error terms that have so let me skip this a bit one more thing that you'll have to be cognizant of when you're using a model like this is even though it's great because it's very granular and it's following the user what's really driving the demand there could be a lot of computation that you're doing to actually make this model run and that is happening because what you have so i have this toy example here where i have 11 stations and these blue and red dots denote at every two minute level whether a station has bikes or does not have by so let me call this a system state a system state is basically this binary representation of zeros and ones for that particular time and given that these are fast-moving systems the system state can be changing over time very rapidly so no two time two minute time intervals look exactly same to one another so you will have to treat them differently and on top of that you have these lot of users who are originating at different locations in your model and so combination can be trillions or even more computations and on top of that if you're optimizing that system to figure out what your optimal parameters are that can be a whole lot of computations so what i do for that is basically exploit this idea that there is this location aspect which is important in these systems if i'm thinking of the station f6 here i know that if the station f1 is say two kilometers away then whether that station f1 has any bike available or does not have any bike available should not impact the demand at the station f6 so the way i can exploit that is basically as far as this station f6 is concerned i can just focus on the state of the stations that are nearby to this particular station and once i do that what i'll find is a lot of data points look exactly similar in terms of whether they have the bike available or not have the bike available and so that i basically build on this particular idea in terms of collapsing the data at each station level and that gives me actually a huge advantage like a thousand fold advantage in terms of the computations that i'm doing to estimate this particular model so i'm not going into too many details but they're present in the paper that is associated with the talk but that's the main idea using this location part right so now i am actually ready to estimate my model so what i have is a bunch of different estimates for my parameters i also do the validation which is keeping some test data some training data and what i actually see is not only this model is sort of more intuitive captures the process very well it actually performs much better than the other models that i were initially trying to construct beat regression models choice models or some variations of that so that's actually a good news i'm looking here at r square or mean square error kind of a thing so that gives me a little bit more confidence that this is a good approach to use in this particular setting the main power now is since i've understood this process in how the demand is being generated what i can now do is take my existing design and come up with a lot of counterfactual designs say i move stations around in any crazy i want crazy way i want i say change my inventory management policies so stations have different bike availabilities and what i can now do is just run my demand model like let my users come at different places and now make choices for this new perturbed system and since i know everything about how users are making those choices i've figured out all those parameters i can just predict what will be the demand for this newly configured system and i'm more confident because i focused on the causality part so that i know it is not just association but really if i change these locations this is how the system should behave and so i can exploit that quite a bit uh so the first thing i'm just showing you is some very first order understanding of what's really going on the system so if i focus on a single user i think about how they think about walking distances so what i just figured out is that uh they of course hate walking no one likes walking but the way that function is it it behaves in a little bit convex manner which is walking smaller distance is kind of okay so up to 300 meters the demand does not drop as much but after 300 meters or so users actually hate walking quite a bit in at least in this system so the demand decays quite rapidly so that gives some guidance that typically you want to be in the 300 meter range of the stations if i want to capture that particular user demand uh so that's just a very first have insight which i wouldn't have gotten so neatly from some of the other models that i was thinking of again i can just plot where are my use how much distance my users are walking and again note that i did not have this data to start with i had no idea where my users are but having this model i have sort of this backtraced what kind of distances my users are walking and i see again that a lot not a lot of them are walking more than 300 meters i can also trace out where are my where is my most of my demand coming from so it's good to know that a lot of it is coming from residential areas where supermarkets are where cafes are where metro stations are so that can again guide my design in where should i be locating more stations so that's just the first order insight i have done nothing fancy but just to exploit this particular model let me just illustrate what else you can do so so i have this use case here where so i'm starting with this hypothetical scenario that i have certain budget and that budget in this particular system would mean i can purchase some bikes or i can because that's kind of a huge capital cost in this kind of systems and i can purchase some docs to where these bikes should be placed and typically bikes and docs have a very proportional ratio like the number of docs would be let's say 1.5 times the number of bikes so i can think that i have money for let's say thousand bikes and once i have that i can think about distributing those thousand bikes in the city in let's say these two different ways so one is i can have say 20 stations in my city but each of the stations would have 50 bikes so large stations but very few stations so that's this design on the left or i can have a lot of stations in my city but each of the stations would be on average smaller in size so let's say 100 stations but each of the stations has only 10 bikes so the advantage of the system on the left is that each of the station is pooling in a lot of demand since they're bigger in size so we know that pooling has a lot of economies of scale and that would cancel out a lot of variability so basically what would happen is the system on the left each of the station would run out of bikes much less often so they would be good in terms of having higher availability of bikes but if the system on the right we know that would have lower availability just because each of the station is smaller in size but since there are more in number they would be very good in terms of having smaller distances for users so essentially what is happening is the system on the right is very good in terms of the accessibility part and the system on the left is very good for the availability aspect and now if i'm thinking well which of the design is good design because in general it is good to have both your system to have higher availability and higher accessibility but in this particular case it's really a trade-off you can really have more of one or more of another and to figure out what is a good design well it depends on how people think about these two aspects and well so far i have actually figured out how people think about these two aspects so what i can do is basically generate a lot of these scenarios and then use my this demand model that i know how about how people are sort of coming up in the system how people are making these different choices and that is what i exactly do i generate a lot of these system designs and i simulate my demand model and what i see is that this is where the status quo is this is what it is right now if i keep on adding a little bit more stations my demand actually goes up so this system becomes more accessible a little bit less available in terms of bike availability but still overall it is good for generating higher demand and after a while just it doesn't pay off that much so basically this simple exercise helps me reconfigure my system design by guiding me that i should be probably placing more stations in my city at least a little bit more than what the status quo is right now and so these are the kind of counterfactual designs you can do right away just because you've understood a lot about what is driving demand at the stations so that's just one illustration of it okay so let me conclude here and probably i'll have a few minutes for questions so just to wrap up what what i've basically shown you here is a little bit alternative paradigm to the typical traditional machining model that we have where i focus much more on having a model that is more interpretable more causal and i've shown you that it is more flexible in terms of including a lot of variations that you'd like in your model it you it is good when you not just want to make prescriptions predictions but also want to make some sort of prescriptions or decisions on top of the relationship that you already encountered in this particular case it turns out that there are huge improvement opportunities which this model is able to perfectly guide you and there is one more thing which i want to just showcase is that since you're now understood user primitives which is how users think about walking distances you can actually take that estimate and use it elsewhere so for example if there is this company which is running this bike share system and it wants to launch let's say motorbike sharing system there is no data whatsoever but still people are not going to think about walking differently just because it's a motorbike sharing station right so you can take this estimate plug into that model there would be still few more things to figure out how people how much prices are people willing to pay etc etc but at least these basic primitives can flow from one model to another whereas that would be a hard stretch to do if you have a random forest model because there is no understanding of how people think about walking distances so that's the kind of beauty these models come up with so so let me stop here and if there is time i would be happy to take some questions seems like there is time okay perfect yes please hi yeah so you are using you're focusing a model on dock bikes that have docking stations right yes so how much would the model change say if you have a dockless system and what would be the advantages and disadvantages of that so you mean advantages and disadvantages of the dockless system in general yes so okay so first how would the model apply to those systems there would be some differences there would be some similarities so you would still think similarly about users starting from a particular location and then having some distance to walk to so those kind of features will stay the same the availability will become a little bit different because now there are no more this idea of stations which have bikes so don't have bikes but just that there could be bikes scattered around and you would have to walk some distances so it would be more like a continuous rather than a disc right exactly yeah so instead of having these hubs there to be more dispersed so that's the first thing a second thing is how would i think that just goes beyond this model a little bit is how this is those dockless systems differ from these dock based systems so just for sort of information for everyone so the system that i'm looking at right now has these stations where bikes have to go at and what is happening right now is a lot of companies like ofo mo bike sort of ola are coming up with these designs where a lot of technology in the bike itself so you don't need these docks to park your bikes you can basically pick up a bike from anywhere by unlocking through your phone and drop them off literally anywhere on a footpath whatever you don't need a dock to dock for them to for your trip to end so the way that would differ is a little bit in terms of so model i said already how it would be differ and those designs i'm looking that looking into that quite a bit working with a few companies but basically the idea is that the sort of the ending part of the trip gets more convenient just because you can't leave you your bikes anywhere but also the fetching part of this whole thing becomes a little bit more difficult as well because instead of having this certainty that you have to go to the station and you will most certainly find a bike there now you almost every time have to open an app and sort of look around where a bike could be nearby me so that creates a little bit of differences in terms of how convenient that system is for a user or not but that kind of remains a little bit of an open question we have to see look at more data more closely thanks thank you yes we have a mic here i'll repeat the question that's okay so we have a question here as well so can we just have time for one question so please take it offline okay i'll do that yes yeah just a couple of observations you don't have user location data but obviously you can see superimpose the population density you know of the city on your to use your model and the second was your model should be biased on availability i'm sorry accessibility because you can always increase availability by adding more bikes so don't you think accessibility is more important or is that how your model is configured okay so two questions so first is you're saying i can always model sort of well people are originating just by looking at the population data so that is one aspect which is if people live somewhere there would be a more reason that people would start their journeys from that and that is part of my model except that that only accounts for like less than 50% of the trips because people would often start their trips not from basically where they live but also they would take a metro trip and start a journey from there or they would go to a nearby cafe at some time in the evening and then start a journey from there so all of those aspects actually play a substantial role in sort of driving the demand for this system so it has to be more than just the population data and so that's basically what's being built in in this particular system and the second question was about the availability part that you should always emphasize on the accessibility because availability is something that you can always tweak around well it costs money i mean these bikes especially in these dock based systems these don't come cheap like each of the bike in this wellyp system was about thousand dollars or so so increasing that number of bikes even though it seems like okay we should always be able to do it and sort of never allow any station to run our bikes that doesn't actually happen because it's just a very expensive investment to go about it. Does it make sense? All right well thank you so much and we'll look forward to the next talk. So we have boff sessions post lunch so we have the woman in data science boff at 2 p.m and the data science and production boff at 4 30 these boffs are in the first floor. So the next speaker is Nitin Hardinia Ji who has 30 minutes to go over quickly the recommendations and food discovery at Swiggy the right talk before lunch to make you all hungry. Nitin. I'm Nitin I'm currently a data scientist with Swiggy and I've been there for last one and a half year and today I'm going to talk about some of the experiments we have done with the food recommendations food discovery and in terms of experiments I will talk about some of the insights interesting insights and I'll also talk about the the challenges or the lesson learned in the process. So I hope it's a it will be interesting just talk because just before lunch it's it's kind of a bad timing I would say. So just quick show of hand how many have you actually ordered from Swiggy? Wow good marketing I guess. So I think I think people are aware of Swiggy and they do it do use regularly but I think just for completeness this is what Swiggy is I guess it's it's mostly these three major players that that actually drives the entire ecosystem you have millions of customers you have thousands of restaurant partners thousands of delivery partners and any problem that you're talking about if you're thinking about just from the one player it's actually going to impact the other two. So if you're thinking from the discovery and relevance problems and you're not thinking about restaurant as well as the delivery executive or the entire ecosystem it's not going to solve the problem. So at the same time you have a lot of the delivery side of stuff like predicting the the estimated times for the delivery how can we batch the orders or we can actually do the best assignment possible for the given order. All these problems are pretty tricky and I think today what I'm going to focus on is the first part of the story where I'll focus on the discovery as well as the recommendation side of the things. So let's take a step back and just formally define what is the recommendation system. So I think the idea of a recommendation system is just to come up with a utility function with even in the constrained environment like serviceability and availability if you have a list of items can you define this utility function that predicts the likelihood of a given customer and what's he gonna like or what's he gonna buy from the serviceable option. So and when I talk about items in this context it could be anything in context of movies or in context of Netflix it could be movies in context of let's say Spotify it will be those the songs and in the context of Swiggy it is one is the restaurant first view where you have we have the home feed you have a lot of recommendation going through on the home feed and then you also have some of these inline collections where those collection has been actually cut through the specific filtering criteria and then you have some of this discovery widgets where you can actually click on just pure vegetarian restaurant and those are also sort of personalized for your taste. Now we are also venturing into more of an item first world where in Bangalore at least we have released this new property called dish discovery where the idea is I think if you can from your historic behavior or from your implicit feedback if you can predict what is the main anchor dishes that you like and if I know that you generally like pizza versus let's say biryani if I show you a beautiful collection about those those pizzas rather than just you actually driving through multiple menus and then deciding what is the best pizza for you it's actually a great experience so we are venturing more from this to this and I think in the end it will be more of a generation of the entire app rather than just focusing on a specific list or a specific collection. We also have some recommendations at menu so we do recommend what are the best sellers what are the typical sold items in the recommended sections and also we have a traditional cross sell where you have a typical association rule mining base algorithm where but it's still smart enough to understand that if you have already ordered a main course then I'm sure that you're not going to add one more main course so it will suggest you some of the drinks or other desert place. If you take the abstract of idea of recommendation system it actually sort of a make my matchmaking between customer and what you're trying to sell so in in context of swiggy the items will be either restaurant first thing or it will be the exact food item and in some cases it could be the entire meal itself so though so that I think the secret ingredients is to understand your customer and if you can understand your catalog or the food items very well if you can solve these two then I think you can do the matchmaking much better so from the customer point of view you want to understand what is the taste profile of this customer whether he's a vegetarian whether he's a non-vegetarian what kind of time slot he generally orders whether he order for family whether he order for for a home or a work working lunch he generally like healthy stuff or he's focusing on let's say the speed similarly you want to understand from the restaurant view as well that what generally this restaurant means is it a taste profile that matters of that restaurant what it says is it like the cuisines or the dish that defines that restaurant or is it the price points or the CFT of that restaurant that talks about so if you understand these two parameters then I think it will be always additional features to the to the conventional models now still the big question is what is so unique or so niche about this recommendation because recommendation systems are not this is not the first company that they are working on recommendation system right so it has been solved across industry there are examples where people have solved it in a way that everybody experience the best possible recommendation that they have so I think one of the major or the most important aspects that actually differentiates Swiggy from some of the other recommendation systems are these major factors where I think that any hyper local business adds a lot more complexity to the system it it always we are always by I think constrained by the service will be or the ability of our system we do have restaurant stress or the entire stress on on the entire ecosystem as well and people do care about different elements like speed which are not there in the traditional recommendation system so think about this right of I know you like pizzas and you have a very specific joint that you really love now for some reason it's not available so the I think and that happens quite a lot in terms of Swiggy then what will be your the most relevant substitute to that and that's where I think the service will be constraints are very important in ours so we have to think about the the larger service ability and ability rather than just looking for a specific customer restaurant stress is one of the other angle where for example a restaurant like truffle is getting a lot more orders and it's really in the stress and at the same time people are keep on ordering from that and suddenly you have a lot more orders driving towards that and you you start pushing a lot more demands towards that same restaurant that what will happen I think eventually this restaurant gonna fail and then they will not be able to deliver the promises that we they have made and it will be very bad for the entire ecosystem so we do have some of these recommend real-time models that actually gives you the this restaurant factors like stress the prep times and the other things in the real-time and if a recommendation recommendation system can actually take some of these inputs and do some of the demand shaping based on these things it will be always helpful for the then the last consumer and people do care about speed in the in the online food delivery world so our faster options should always get some level of prevalence now the other factors that I'm talking about are not very specific to Swiggy and it's just very much very hard choices when you have to design a recommendation system so you you generally like let's say a biryani or I mean in the 99% cases you generally ordered biryanis in the past showing all the biryanis place in the world is it a good recommendation it might be accurate recommendation but there has been proven examples where adding some level of diversity into your recommendation system always helps so I would still go for getting some level of diversity with getting more Chinese points or adding some level of a Italian place into your listing or in the home feed discovery versus repeat is also one other other option where in general contained related websites like news websites or even your Netflix or Amazon's where I think there is a new original that has come on the Netflix so they they want to make I mean hopefully people will not once they have you have finished that original you will not keep on going on for the same original again and again now in the context of food delivery you still prefer the the restaurant that you have already ordered so the repeat behavior is much much more important in in the context here right so showing just the restaurant that you have already ordered maybe the top 10 restaurant that you have keep on ordering from is it a good recommendation or definitely yes I guess but still there there should be a balance between how much we want to do explore versus the exploit so I think this is again one of the tricky or the differentiator that that that we see across the industry freshness versus stability is another angle where you want to see that the recommendation you are getting on day today if they keep on changing very rapidly it doesn't give you the the the notion that you know you are not able to understand what why it's coming so the explainability or the scalability also has a has a significant importance at the same time freshness is also matters so if if you like pizzas and the real the new pizza place that that really excites you is not there in in your home feed gonna hurt your recommendation serendipitous is something I would say is is kind of a definition of discovery where some of these course from CNN they talk about that this is more of an end of era of search where you have explicit query you're looking for a pizza and get the pizzas now the real discovery comes when when a when a rare items actually finds you and that surprise you so the same case let's say in the netflix when i original that you feel that you know you started watching and you feel how many how these guys are able to get this or the in the case of song that the rare song that long tail that gets you then only you feel that you know this is the real discovery so the same context if you if i indian desert that you feel that you know whether it's available or not on on swiggy and that finds you that would be something really magical right uh for any recommendation system you need a lot more data and these are the different variety of data set that we are having we capture a lot of interaction on the app as well on the website we do have orders data or a typical transactional data we do have a very rich image data which is very specific to the indian foods and which sort of labeled as well for the for the specific categories and you also have this other matter enrichment like dish families cuisines which we have sort of generated and you we also capture a lot of unstructured data where if i if you generally order and then you you you ask for intra certain type of instructions as well as in the in the rating space where we generally capture the rating data as well and also ask for a feedback a lot of these things has been used for generating the different recommendation system then recommendation these are the two major philosophy one is more about if i can figure it out what who are the customers like you and this is pretty much the philosophy of collaborative filtering the other one is more on the generally people like similar items so if you can define the similarity very nicely then you obviously can do better recommendations and that comes from the contained side of the thing now collaborative filtering i think eventually you want to build this customer item matrix in case of this it could be either restaurant cuisine category or dish and then you have and some of these methods are defined for explicit data like ratings and coming from netflix challenge where the idea is to predict those blank boxes and if you can predict the rating of that then it will be a sort of a good recommendation you just sort the the recommendation based on the ratings now i think steven randall or korean they actually defined some of the methods for some of the similar methods of matrix factorization for implicit data and if you want to use this kind of implicit data like orders menu visits search and all these things it's it's actually kind of much fulfilling because there are specific papers where they talk about generating the explicit data is much more costly as well as it has its own bias while if you use the implicit data it in many cases it has actually outperform the explicit data so we tried out traditional matrix factorization methods like svds al s and we also did a few experiments with learning to rank and even in that we added some of these customer features as well as the content feature as a feature and then that actually improved the performance of the model now the problem with those methods of collaborative filtering is one is a very standard collab cold start problem and the other one is actually it's in some cases it's biased by popularity so idea was that how can how can we overcome these things right so that the traditional methods that that actually are the solution for that are the content best method where you're trying to define this notion of similarity and in the in the case of sugi how to define this so generally the content platforms like news and other websites they have a lot of maca like news articles they have maybe the titles and the genre in in case of movies you take that you build a nice backdrop of it and then you do some level of similarity calculation now in case of sugi we could have done two ways one okay so we could have done i think either go by the the the meta that restaurant partners or the the customer the items we have or we could have done something for of our own so what we did we actually take the order data at a restaurant level create a dummy document and then build a sort of a topic visualization out of this and somehow it's the demo is not working but if you click on one of the specific topic this actually captures the notion of the entire catalog what we have in sugi so if you zoom out in zoom in into one of the topic it talks about whether it's a biryani thing or a maybe a south indian thing so the idea is to project your restaurant into this and then you will have a decent vector out of it and then you can actually calculate the similarity notion that's again a more of a restaurant first kind of recommendation if you and this is how it will look like in picture right so you will have some level of notion for similarity and if you have already ordered from let's say truffle there is a good chance that cafe 12 will will be there on your listing you can have a restaurant first view based on the previous previous method but still if you want to do the recommendation at item level you have to know what is the standard taxonomy for for an individual item so you have to differentiate between somebody calling pizza as main course versus somebody is calling maybe a italian kind of thing so we define the standard taxonomy and these are the standard enums or the standards enums classes that we have defined and then we build some level of supervised model we have metadata related to some of these items which is either in terms of item names description this recipes and in some cases we also have the images so we did some level of image classification once you have this kind of set then it empowers you to do something that I was talking about in the initial slides where you can once you can filter your data on the cuts like a dish cuisine and all you can again go back to something like this and then you can create this matrix and then you can predict what are the most relevant cuisines or the dishes for you I'll just skip the word embedding part but I think this is so far the journey so far and where we have done a lot more work on the collaborative plus contained base and use ensemble of these two to generate some of the prediction which is currently the state is somewhere on the hybrid as well as on the learning to rank part while we are also looking into DNN based methods where generating the item embeddings product embeddings or the restaurant embeddings are obviously open area and also deep collaborative filtering and RNN based methods to generate the recommendation at a session level so can you capture the the real nuances on the on a specific session instead of just relying highly on the history and the final slide actually talks about the journey of the swiggy from the delivery or discovery platform so here you currently we highly rely on our home feed and some of the initial personalized collection we are moving towards lot more collections actually catered towards anchor theme and then slowly moving towards the generation of the entire app where lot of the person like all the different collection will be personalized towards your taste and slots and even the other context and finally if you can you do some level up real-time adjustments to make it much better I think we are done okay so this is the team that that we have and I think I'm available for the question we have some time for questions hello yeah I have one question hello hello yeah yeah go ahead yeah so my question is uh when you are showing when you are recommending restaurants or dishes what is the objective function you are maximizing for is it just the relevance for the user or like revenue generated for swiggy or is it a mix of these things or yeah I think definitely we are constrained by some of the other factors like service will be another so if if overall stress levels are decent so we have a way of identifying what is the stress for swiggy and if that that suffices then we are chasing the relevance as a major proxy and if if that is not the case let's say it's raining or it's we are not in the in the in the in the best of supply then we degrade our things and then we just look for a I think a general availability so the idea will be to optimize the entire system based on the efficiencies rather than just going for relevance but in in normal context which is where there is no stress there is no specific constraint generally the focus is always the relevance but we do try it out when we talk about l2r or learning to make better method where you can actually induce some of these other constraint into your objective functions so we tried out demand shippings towards certain things like fosters and in case of certain restaurant you do should not demand shape or you should demand shape in a much more holistic way so you do need to consider the restaurant view as well as the entire swiggy as a platform a question here yes so just want to understand what is your implicit signal and the second question is are you still using ALS for the factorization no so ALS was the first one but I think the the one I was talking about is is coming from this guy Stephen Randall I think it's a very a known publication and there you are using a wrap loss with I mean effectively it's a learning to rank method where you're doing a pair wise loss and ALS was obviously the the start kind of sorry I have one question about learning to rank so which model forms are you using within learning to rank there are many sub models within that right so can you repeat that by learning to rank which particular model forms do you mean I mean like a lambda mart or like rank net or what type of models are you using I think currently the models are pretty much on the matrix factorization based methods or some notion of these contained based methods what you're I think I don't my question is about the learning to rank models yes which you mentioned that you are exploring or you have been putting it in place I think we are just exploring the pair wise thing I think you're talking about I think maybe the list list wise methods so I think overall if you see the literature also the pair wise methods are there for a long time and they have been proven because that they are working for something list wise still I think I would say not not at the label where some of these they are for the research and other thing but I haven't come across a very standard method that that has been widely used but it's still I think that is the direction where people are hitting one more question on the data so when you're estimating any models like this and not forget about the last function yes you also need to think about what data do I use do I use those who just browse around but don't order but also those who have ordered something so for them the relevance will be different sure so there'll be these issues around you know the nature of search will be different and those who are just browsing it just click a ton of things and then go but then they don't book yeah right so have you looked into this did you what's your opinion about it did you try to account for all those in modeling I think the implicit data that we use is somewhat listed on this one so there is a long list but I think the idea is I think if you have attended the talk from Salish I think yesterday he was also talking about do you have tons of implicit data how you actually smartly take that and create a meaningful supervised label or meaningful implicit feedback is something that we should think about so though somebody has visited a menu but he never actually ordered from that specific restaurant should be ideally penalized at the same time if somebody is spending a lot more time on menu should give it we should give it a boost right or if somebody has already added to the card so the idea is that he actually likes that restaurant he liked the recommendation but he may be hung on some other factors like offers and all so you need to give some level of weightage to some of these things and that's where I think the real data will lie rather than just throwing everything at one go yeah so sorry last question my question is rather simple how do you evaluate your recommendation system whether it is working or not so that's a very standard ndcg matrix and so at least at a modeling step you can so if you think about these problems they started with a traditional matrix of rating prediction and if you can get the rating prediction right and do a basic like a root mean square error on that that was the initial start but then gradually people started hinging towards uh can you do a uc so you if you can think about it think it as a as a more of a classification model and then you can predict like then you can at least on your ensemble data you can do calculate a uc and all but when you are running this in the production like maybe a b system or something then the typical matrix will be either ndcg or a more conventional method that we talks about in in a larger group is how many people are actually ordering from the first 15 what you saw or how many people are actually not ordering from search so currently if our orders 60 percent of our orders let's say come from the feed and 30 percent people actually come from search if you can decrease so ideally that if your recommendations are very good people will not go for search though there is a behavioral changing as well but if you can reduce that then there is a good potential that you're whatever you're recommending to the customer is is a proxy like that you're getting right so overall at an online level you can always see the click through at a funnel of from the starting from the listing to the the card so that is some of the business object or business metrics but you can always define it in much more mathematical terms in in terms of ndcg or mean reciprocal ranks and yeah so one final question interest of time yeah we'll sure we'll break for lunch now and we'll see you in an hour thank you yes check check welcome back check quick announcements please do not bring any food inside the auditorium thank you we have a couple of bobs at two o'clock now we have the woman in data science boff starting in the first floor at 4 30 we have the data science in production boff also on the first floor evening at around 5 30 we have the open discussion on the fifth elephant agenda and community needs it's an open discussion that will happen again on the first floor right so most public discussions are centered around ai or ml saga and two underrated components are workflow and data explaining the data collection and quality with this dogma will be paul main sourcing on designing for data thank you uh good afternoon everybody i hope everyone had a good lunch uh certainly a good morning um a lot of thoughtful and provocative talks so thank you all for for continuing to be here my name is paul mine salson um and currently i'm a data scientist in residence at montane ventures which is an early stage vc fund here in india and i've spent most of my professional life as a data scientist but in the past year as an investor i've had the time and space to step back a little bit from the day to day work of data science and uh think about things at a little bit of a higher level and i've also gotten to see and talk to a lot of startups doing really fascinating and promising work in ai and ml and analytics um and i think the thing that characterizes the the best data science projects i've seen is the way a team approaches the foundation of its data um good quality data and a well-defined source of data i think are the critical factors to success and that's what i want to talk to you about today and the thesis i'm going to present uh is that quality is less uh less to do with data itself it's not uh a met a list of metrics or characteristics that you can rank it's not an intrinsic feature to the data it's more to do with how we look at it uh how we work to understand the data we have and how it came into its existence especially um that and so the quality comes from that symbiosis between the appropriateness of methods uh and the data that you have and i'm borrowing from the statistician george box here originally pointed out that all models are wrong but some are useful and i'm saying that something very similar applies to data and whether it's good data ends up being about how you approach it and i'll start with a story that motivates my message during world war two uh the the u.s. government uh put together and set up a small organization of statisticians and mathematicians in manhattan uh it called the statistical research group and the s.r.g worked on a variety of problems related to the war effort among these was a problem that the air force was facing taking off from england bombers were flying deep into northern europe uh and into europe to bomb strategic targets in germany they were also getting shot down and uh the air force was losing a lot of planes so the air force they decided leadership decided that they needed to up armor their planes but they also realized as with most problems we face i think uh that they faced the trade-off the more armor they put on the plane the heavier it would get uh which meant that that would decrease the distance that the planes could fly they could uh reach a fewer set of targets uh and they'd be closer to the line of running out of fuel so they decided they had an optimization problem and they turned to the s.r.g to help them with it and they said look uh we've been tracking all the planes that are coming back in and landing and that are flying these routes and we've created this data set of where they've been shot and where they've been hit where they're taking damage take it please and model it uh so that we can understand where we're getting hit and we'll focus our our armor there and there's a statistician on the team named abraham walled and he took on this problem and he looked at it and he thought about it for a bit and then he came back and he said he told the air force here on the engines this is where we're not getting hit this is where we don't seem to be taking damage and that's where we need to apply armor because you see he realized that the data he was seeing was this a sample of the world and there was key data missing namely the planes that were taking off getting hit and crashing and not making it back to your making it back to england right and so on those planes we we don't see them those those are not a part of our data set and so we can make the inference that missingness corresponds to fatal damage to planes and instead of focusing on armor our armor on where we're taking most fire we focus it on where the fire is doing the greatest damage and i think this characterizes story for me characterizes what i want to talk about and and to convey my message today i want to kind of walk through four stories from my own experience and i hope to persuade you that in data science work that you should be caring very deeply about what we'll call the data generating process to care enough about it to do the work that doesn't always get the glory and doesn't always feel like it's on the cutting edge to understand where your data is coming from and also to do the work of communicating this because i think both are necessary for for real success and i'm going to going to spend most of my time telling you stories about my encounters with interesting data generating processing experiences and these are not nicely wrapped stories they they're mostly open-ended there's no easy obvious clearly right answer this is not a sat or je exam because i think that's how life is mostly i'll be walking through a kind of a sample of experiences of discovery as i've experienced them and i'd like you to walk away thinking huh not not thinking okay in scenario a i apply x method and scenario b i apply y method but rather to kind of walk away and think huh how i see how i'm facing or might be facing something similar something analogous in my own work that deserves further thought that deserves attention or maybe even just sort of as a process of intellectual simulation that you see something about the examples i give and you can simulate in your own mind something that i seem to have overlooked or or how a line of approach might take you the distance towards a better solution so what is the data generating process one of my professors and statistics in graduate school used to describe it this way to convey kind of it's all the things that take data uh into the world and in from the world and into your data set and there are three general things that i like to think about uh they define it a little more concisely or a little more specifically um the first one which we often overlook and i think that's a little bit about what where you know chris's talk earlier this morning which i thought was quite good uh talked about it and probably sick of the things we learned in statistics class about sampling strategies or you know how marbles are selected from a jar they're actually really important but often overlooked statistical model of a data the statistical generated data generating process is often what we're trying to model when we set up a regression for example we're explaining some effect or some phenomena in in line you know using some other set we're explaining a y with a with a x matrix and then finally there's this data collection process which is often not what data uh which is often not what statisticians are talking about it's often more what we should be thinking about as say software engineers or developers or builders it's the routes and technical procedures by which data reach a database and that's data engineering so my first story picks up the thread from the statistical research group except we fast forward to india in 2015 and i think that we often overlook issues with our data generating process not because they're incredibly subtle or or hard to spot we have to be brilliant instead we overlook them because we're so excited about building solutions and finding a direct answer to our primary business problem in 2015 i was vice president of data science at housing.com in Mumbai and we were excited about building an excellent real estate experience so that instead of wasting your weekend you know crossing the city and traffic stuck just to see a flat that it turns out was obviously not right for you you should be able to find your next home from the comfort of your home and a core part of that effort was bringing as much real estate inventory online as possible in the summer of 2015 our ops team reached a million properties and that was a huge success for us registered online and up to the time i joined the data science lab had very little insight really no insight into that process that ops process and very little attention for it given all the other things we were thinking about for example we were really thinking a lot and very excited about overhauling our recommender system and we are intrigued by really intrigued by all the ways that conventional recommender systems for e-commerce or say media all on netflix weren't right for real estate and that's what had our mind share and that's where our data scientists were keen to start except we remember that we needed to start with thorough data exploration and our exploratory work led us to looking at this our data in this kind of way each of these plots shows two curves for a city in blue is the kind of properties that we that our users were searching for and in green were the properties that we had in our inventory for that city and in looking at these we realized we had a problem see in cities like Mumbai and Delhi you can see there's considerable area under the blue curve under what the demand curve that didn't follow within our supply curve which means we had users looking for kinds of flats that weren't in our inventory that weren't on our site and so it didn't really matter how sophisticated our recommender system would get if we didn't have the right inventory to recommend and these plots introduced a question for us that we needed to answer about our data is it representative of the market because if it is then our users are looking for flats that don't exist they're not available um and we need to educate them to that like you're looking for a particular kind of flatten band or west in Mumbai and it's just not there it's good for you to know that uh so you don't waste time on the other hand if our data isn't representative then there's something wrong with our data collection process and that deserves our attention as well so what is our data what was our data collection process well in each city and locality we hire data collector teams to go out and find properties to register and bring on the site and those collectors had been incentivized based on quantity get as many properties on the site as possible and demographically our data collectors tend to be young male often bachelors um and they were great at the job we'd given them getting inventory on the site but then we realized that uh potentially we hadn't been precise enough in our direction see in their rush to get inventory on the site uh they were going to brokers who they were familiar with who it was just it was like path of least resistance right and that means they tended to be going to flats like this on the left large lower end it's more affordable brokers who were kind of just renting out in volume and they were very happy to just put 12 sites on 12 flats on the site except those 12 flats are all basically the same right so there's very little value add from a user's perspective for looking through 12 flats that look exactly the same have the same layout and our collectors weren't nearly as much going to flats like this higher and more expensive lower because the brokers there were were not they're not so comfortable right they don't deal with large volume and um and the two worlds just don't really overlap so when we realized this we were like well we can actually change our data generating process we can say let's just build a feed our data collectors already have an app that uh that they use to collect so instead of just having it them go and and be incentivized based on raw numbers uh we can actually create just a simple api that says here's what we're looking for this month feed it to the app and then incentivize the brokers to do the extra work to break out of their comfort zone to do the things that can get us a more diverse set of inventory so our data generation generating process can often be influenced by uh social factors demographic factors things that we just wouldn't normally think about as a computer scientist but actually end up being incredibly important and we also had data generating process problems on the demand side at housing see we were ensuring that we collected as much data as possible about each flat uh and then we tried to make the search and filtering process for users as clean and delightfully user experience as possible to do that we needed to track and learn about how our users interacted with our next flat basically that meant using clickstream data across the user's product journey all the things they're doing so what is that uh what does that look like basically users uh get on the site and they they take a couple of actions and they sort of sequentially look through different um different uh different properties and we're learning something about the kind of properties they're looking at but we're making inferences based on the sequence in which they do that users user's preferences are conditional right so they'll trade things off they'll say closer to job i'll pay more for less if i'm further away i want to pay less i'll get more and so there's this event stream that you can kind of imagine so most of you are probably familiar with clickstream data but in case not your standard stuff that you know pretty much every website is tracking is things like okay the page a timestamp some kind of user i you know either an ip address or if you're logged in a user id um the url things like that but services like google analytics and others that kind of out of the box set this up will only take you so far if you want to really understand it and see more granular data about sequences where people are which filters they're selecting etc you end up getting into quite a more complex event data modeling and our product analytics team is part of the broader data group uh spent a lot of time working on this because they realized this is our data generating process and it's really important so it's worth the time and they would go to all they went to a lot of work mapping out the the flow that the site facilitated to sort of so we could track it and learn about it over time and this would be fine if we had a very stable uh product except we didn't because while housing was known for good data science at the time i like to think we were uh housing was also known at the time for really great design and the design team what they cared about most was how the how the product felt and which means they were willing to do all kinds of work to improve how it felt how it flowed how it was used and our front end team which had to implement all of that cared about performance they cared about making it fast making sure that that information is uh compressed or flowing smoothly and uh so what we realized was that all the tracking we set up when workflows are changing when the site's changing constantly those aren't because the teams the front end teams and design teams weren't really thinking about the data they were generating they weren't necessarily always taking the effort to go back in and remap or like work with the data team to understand if this event is no longer even possible on the site uh a drop in that event doesn't mean that people aren't doing it anymore it means they can't do it anymore and basically the lesson here is that uh when you have a data generating process that's changing very quickly it's very hard to make inferences uh from it you lose the longitudinal value of your data and it what it means is that this is a technological process by which data is getting into your database and if you're not paying attention to how and how that's set up uh then you're going to get data that's not appropriate for the questions you're asking of it now the two stories i've told from housing so far both illustrate how people it might be business partners it might be ops teams it might be front-end developers are somehow affecting an influencer data generating process and the data we work with and in my next story i kind of want to disrupt the tidy dichotomy between data and the world on the one hand and the data science team on the other uh many of you have heard the phrase that data is like the new oil right and housing we like to say no it's not really like oil it's more like soil because oil is a non-renewable resource right once you use it it's done but data is renewable you can use it you can you you can use it for multiple purposes and it can even increase in value over time but the issue is is that the things we do to it affect the data it generates new data we're affecting that uh and then over time there's new people coming onto the team and there's new ways data is being used and we still need to understand the things we did in the past to affect how our data has come into being so i'll talk about about our story so after i left housing i was chief data officer at a startup a fintech startup in Mumbai called PaySense PaySense does mobile loans through an app and we were working on a problem that's very common probably for everyone in this room you're all kind of familiar with it as a standard data science problem how to decide to give a person a loan and a lot of times we jump into to you know using building a model and we might uh you know use a linear regression which performs pretty well right that dris was talking about but what i want to talk about is is these these features that actually are going before we get to the model where the data is coming from and i'll talk about for instance a feature that we decided was important uh average monthly bank balance an average monthly bank balance uh you know is is this thing that represents uh deeper process so most people who are applying for a loan their lives are sick their financial lives are cyclical every month they make some money and every month they spend some money and what we're interested in knowing is what's the top and what's the bottom and what's the curve so really we're interested in knowing something like this normalized across day of the month here's where their balance tends to be and then we're going to reduce that right to just one number monthly average balance and you might think well that doesn't capture enough information about this curve you might want to know uh like monthly average range or variance or max or min or something like that and yeah you can add lots of features in there but we just need to understand the underlying process and then we realize that like this looks pretty pretty it looks like we've got enough data and we've kind of drawn a nice curve and we kind of understand what's happening but actually a lot of times the data looks more like this so we've not normalized anymore now or take say we have three months of data for observations of a person's bank balance and we can kind of see okay so in June and April it sort of we sort of see this uh this incline uh it looks like it might be representative and we kind of see a similar slope in May but it's it's lifted and it's somehow shifted rightward for some reason we could think about why this is happening but the point is is that it's not a neat curve uh and we need to figure out how to interpolate and do other things so we also need to ask ourselves where is this data coming from let's not jump into just like coming up with the right curve or coming up with the right number so where we're getting it from where we got it from was sms's financial sms's when you do a debit transaction when you do a credit transaction you get an sms and it contains a reference to your bank balance okay but not everyone receives the same rate of sms's not everyone some people delete them there's lots of different processes so let's just look at like a histogram of how many sms's we have for a given user and we kind of see well there's kind of a bit of a curve so we expect our users mostly to have around 20 to 50 sms's and then we might ask ourselves well let's describe it in a different way so we can also look at how much time because you can have a thousand messages that came in a week and that's really not conveying enough information to you right you care about over time so let's take the earliest date observation we see in the last date and we look at the span of time there and we can also draw a curve for that and we kind of see oh there's these peaks well by the way these peaks had to do early on in the process with the fact that we realized at first we were saying let's let's take 30 days let's and then deciding well one month is not going to be enough anyway so let's take 60 days so you can kind of see this that show up in the data has nothing to do with people's finance has nothing to do with how they do transactions has everything to do with how we set up our SDK and you can continue to kind of ask yourself how are you modeling this what is the underlying sms why do we expect people to have all their bank balance transactions being recorded on their phone so now we can say let's take the count of dates so like the number of dates covered but just like a thousand messages in one week is not sufficient similarly if we have an observation and then two months later we have another observation but like maybe only two observations in between if you look at it just in terms of the count of dates between the two it looks like a lot of dates but actually if you look at the count of unique dates it's quite low you have insufficient data so if you plot this together just very simply right we can kind of see ideally our our users would follow along this line if you have 40 message 40 days of messages you should have at least like around 40 or or high 30s of unique dates but of course that's not how all people are some have large spans of time but not a lot of good coverage and we can also kind of look at the density of of messages within those time spans all of this is just to sort of say that all of this work kind of needs to be done before we just jump in with a raw algorithm or like an equation and when we come up with that average balance we we're not just creating an average from observations right because if we have five balances across a month you don't average over five because there's all the dates for which you don't have observations so you you have to interpolate or you have to impute missing data so what do you do do you forward fill do you backfill so let's say I have an observation on the fifth of the month and it has a thousand rupees and then I see another bank balance on the tenth of the month and it's two thousand rupees now do I say all the days between between those do I do I use a thousand rupees or do I use two thousand rupees or do I split the difference it makes a difference because time spent at a different level of bank balance affects the credit that people are going to be able to get and your evaluation how whether they can repay that average bank balance is not just going to be used in the decision to give them a loan or not it's going to be used in the decision by a different model of how much to give them a loan a different model of when to market to them another model that's going to be asking you know how likely they are to be you know engaging in fraud and other data scientists are using it so we generated that data that then just becomes data in our database it's in the monthly average balance table or whatever table it's in but we affected its data generating process the decisions we made affected how the data ends up in our database okay so let's continue with the story of giving a person a loan and we take another feature and a lot of this you know so the stories we've looked at so far we've kind of thought about a single data generating process that kind of covers all of our users but this last story shows how sometimes figuring out our data quality involves recognizing where we have multiple data generating processes and we have where we have to some work to do to tell the difference so our next feature we might think about is monthly salary and when a person applies for a loan obviously their salary is a core feature of how much you decide to give them and well in the application process we rely on them to tell us to report on their salary but obviously we know that we can't just take that at face value there's a problem there and so there's another process that you can follow which is kind of logistically complex of sending a courier and collecting a paper that has a salary slip or an offer letter and then you kind of have to convert that into data and that's complex and expensive as well so we kind of want to know is can we validate salary more efficiently and more cleverly using our data so we say again turn to the SMS's because we observe that some of our SMS's seem to be telling us that when a person receives a credit you know a credit into their account associated with their salary then we've kind of observed a fairly trustworthy you know a little bit better or additional to the report they've given us and sometimes by the way we're really confident that that's a true SMS sometimes we think okay this is not it doesn't look like we're not sure it's a salary but we feel like it is right right here in red it doesn't say salary but we're pretty sure this is an employee of ebay and that they've received this uh this salary this month so we can say well let's use this and map it up against what people are reporting and so if we do that and we'll just log transform it so our distribution looks a little cleaner and we realize that yeah this kind of represents what we were hoping people there is some match there is some mapping between the two but we kind of can also suspect or what our intuition would tell us is that there's a lot more people telling us they make more than their SMS says than there are you know vice versa and we need to understand is what else is going on here and which ones can we trust and which one's not so let's think about how this data is being generated where is it coming from what is it well we can go all the way back to the beginning and we can say well how is salary itself generated what's the data generating process there well it's actually kind of a process of negotiation right and then you're like getting way off track and you're kind of thinking about like qualifications and and how a person gets a salary but it's important to at least think through that okay that comes up with the salary that you agree to with your company and that's usually a humanly meaningful number so you never agree to a salary of 37 thousand 237 that doesn't you like we like round numbers we like simple things but when we get our salary that's not exactly what we agreed to because the finance department gets their hands on our check and they do all kinds of deductions and things like that right and so that's a data generating process for what actually gets deposited into our account and we can kind of follow that through and we say okay that's our actual salary and then we have our user report how does that get generated what's the generating process there well a user is not looking at what their finance department gives them usually they don't even look at their payslip they're applying for the loan like say in the mall or as they're buying for something and so they're thinking of their humanly meaningful number they don't calculate they know their ctc is say 40 000 a month they don't know like that this tax and this debit and this blah blah blah so they're going to report that so to a certain extent okay so that's the user report and then we know that some users are going to exaggerate their salary that means they're just going to like bump it up a little bit because honestly they're going to get this bonus next month or when you calculate it out it's going they're going to get an increase within three months so that's really safe and then there's some that are like dishonest they're just telling us a lie they're making 25 but they say 60 because they think they'll get a better loan meanwhile on the other side we have our sms's we know that our salary is generated by a financial process and then we're getting a notification and that's generated by an sms whether the person deleted their s and all the things that we talked about before with balance and then also there's the this process where we're not even sure all the time where we say it's a salary sms whether it actually is okay so you know thinking about this we can kind of ask ourselves well we still know we have two sources of information what a person tells us and what their sms tells us and how do we look at the difference so we might take their income and and subtract what we've identified in the salary and we and we draw distribution for what the difference looks like and how big does it tend to be and it's kind of nice that um it's fairly closely approximate to to zero so it seems to be a good number of people who are somehow being honest we have a match but what we can and you might try and like you know run a regression to kind of come up with what the difference should be what's the bias because then you can kind of be a little bit smarter in the salary but you can also ask yourself is do we actually have multiple data generating processes here should we be estimating a bias that a user has as if it was a deterministic effect as if it's like just a coefficient and a regression and I don't think so I think because I think each person has a bias and that's that's not like uniform across the population so maybe we'd want to be modeling it that way but we can kind of take what's called a mixed distributions approach and we can say all right uh well we actually have our two populations here one is an error distribution where people just are off a little bit it's narrow and then bias distribution which covers when people exaggerate and it turns out that yeah we can kind of expect around 61% of our users to fall into the error distribution and we can assign each one based on a say like a likelihood ratio test and there's good estimation models where you can kind of choose the parameters of the distributions themselves you know model those as distributions but the point is now we can sort of say for some people we're going to like choose their true salary and for some we're going to adjust them based on sms's so the point is is that there's we don't have to apply just like one uniform probability model to all our data there's if we think about what's what itself is generating that the information we're looking at I think like you know this is a tweet from many years ago but I think it really represents well I think what a lot of data science work is it's this question of like we're just we're counting and we're figuring out what's our true population what's the relevant population for the data for the problem we're looking at and can we reproduce it and I'm gonna talk a little bit about that later so I also want to kind of quickly talk about the that second part of this because it's not just about knowing this or thinking about or working through it it's also about how do you why is it so important to actually talk to people about data generating processes especially your business partners especially your product managers and I think it comes down to something to my background like most of my research work was done in cognitive science and applying to the school and computation to how people make decisions and there's this thing called theory of mind which has to do with how we understand what's in another person's head and there's also this thing that comes from that called the curse of knowledge which represents the problem all humans have with knowing what they know knowing what someone else knows and like not telescoping or telegraphing what we know onto that person it's hard to remember that we know things that other people don't know that sounds weird it's usually shown through and you can kind of look and sort of see the humans aren't born with this capability it kind of is generated over time and so this is an experiment that's often done with young children and basically they're shown this series of events basically there's two young there's two two little two children once Sally has a basket and and as a box and Sally is a marble and she puts the marble in her basket and then she goes out for a walk she leaves the room and Anne takes the marble out of the basket and puts it into the box she moves it while Sally's out and that Sally comes back and she wants to play with her marble where will she look now we all know that she's going to look in the basket because that's where she put it but children of a particular age will ask I will say she's going to look at the box because they know it's in the box and they just kind of assume that Sally knows it's in the box as well we stopped doing this when we're children but there's still a lot of it that sort of stays with us it's still very hard to understand another way this shows up uh is is is uh in our language for example so I'm just going to do a quick experiment so let's say if I say highly likely actually can someone help me can you help me just just stand just stand and memorize the numbers I tell you so like if I say highly likely raise your hand in the room if you think I mean it's about 50 chance nobody right maybe two or three people uh now let's jump to 65 anyone 75 all right we're getting there 85 good good 99 okay so we're right around 85 or 90 remember that for highly likely yeah what about probably 50 okay a lot more 60 uh it's kind of going down actually okay so say 55 yeah you're remembering these numbers good all right probably not 50 45 35 25 15 that's weird we're kind of skewed on probably not somehow it's stronger than probably all right let's say 25 yeah and then finally highly unlikely let's skip jump we're not gonna start at 50 let's start at 30 30 25 18 15 sorry I'm minus from 100 uh yeah sorry no that's right uh 10 5 3 2 okay it's around 5 5 right okay well actually there's been work that's done I looked at this and it's it's pretty cool it's a nice visualization the point is is like if I say highly likely or probably not uh it's we don't really know what I'm saying I'm trying to be more statistical right I'm trying to be more database but there's a like quite a range on what people believe we're saying and this is what happens with our data like our business people get really excited they're like data science and AI and now we've got data and we've collected all this data and what's the answer and I know like oh I know his race and his sex and his income and I know everything it's not really true but we have to do the work to help them understand what are the limitations what are the appropriate questions that they should be asking of our data because it's kind of like turning the world into data as this process that's both really wonderful and letting us be more concise and more precise and more accurate but also loses all the things that Abhi was talking about this morning right about all the things that we pack into our knowledge uh that we forget to explain so right we have and and that's the best we can do or we can use it for a completely different purpose that it's more appropriate for and the final the final thing is how do we know we're using it in a good way and how do we know we're going through this process in a good way of communicating our data generating process and I think like coming from computational social science we in the last five to ten years there's been a lot of work that's been done in understanding the importance of reproducible research the fact that we shouldn't just believe a paper because it was published we need to understand what was happening behind it what were the methods being used what was the analysis done was it appropriate etc but reproducible is actually just the first step so here's a sort of a two by two that I like to think about we can think about our data and in our left column we have the same data and different data and then you think about our code or infrastructure or our algorithm and we can have the same and different and when we have the same data what that means is as a data scientist I do a problem I use some data I use some code and I come to a result now I might have messed it up somehow or overlook something I need someone else to be able to go in and do it again and come to the same result before we can really believe in it but that's not as robust as it could be if we use different data that's when it becomes replicable and if we use different code that's when it becomes robust it means yeah you use this use this linear regression but if I use logistic correction I couldn't have completely different outcomes there should be some resonance some similarity some consistency and finally uh when we when we're saying like we can use different data and we can use a different approach then we think okay now our model is generalizable we're really learning something about the world that we can believe and I think uh this is the kind of thing I like to think about how we kind of measure ourselves in terms of the appropriateness of what we're looking at and uh and that that's it uh thank you I don't know if we have time for um sorry you can sit down questions uh hello questions we have time for a couple of questions good because I think I ran over but I will be around I will be around and also doing a hello okay yeah uh there's a bof on data engineering we're going to talk more about actual methods and techniques and and and like technologies that you can use to kind of do these things all the work of data engineering uh tomorrow I'm pretty excited about that so if you're interested in this topic and you want to hear more like details please do come by that and anyway I'll be around if you have questions or thoughts or just want to chat cool thank you so people who are standing at the back uh the balcony is open upstairs so you can occupy the seats uh we've found a car key it's at the help desk the original owner is requested to contact the help desk and identify yourself and the car before you collect the keys a solar bof is starting in oddy 2 at 245 now anand Venkat Narayanan is here to tell us the entire tale on compromising a dollar six billion data acquisition project the aadhar case study at scale disabling security features and defeating the backend analytics is a hash fail hopefully our aadhar numbers are not for sale anand Venkat Narayanan so I would be talking I would be talking about the aadhar project and particularly on the technology angle about the real quality but not about anything else right fair enough so uh as part of this presentation uh you can see the blue links I've also linked all the articles public public research that we've done as part of that so if you have any questions you can go look at it right thank you huh it is on the talk funnel right uh about myself I exist so I haven't found a need to have another yet okay uh I'm a dependent for the petitioners in the supreme court of India in the ongoing cases so uh most of the technical arguments on the project has been drafted by me right uh I'm a security researcher and I do a lot of financial modeling and I also program that's basically my background so what is a big point I want to drive in today's talk uh if you're talking about a data acquisition system which is countrywide eventual consistency is a big weakness that's basically the point I want to keep driving towards okay and this part is important because why is eventually consistency is a weakness because it's it's going to be exploited and the way in which you think about aadhar uh it's much more easier if you think about it like an identity platform um and one platform that you are very very aware of is what we call as a currency note right so if you have a currency note you already know about what a platform is uh an identity document is something like a permanent currency note which does not have double spend problems right so if you historically look at all the bank scams and fake note scams you can basically think of currency notes as having read access and having write access okay so if you look at most of the common ATM frauds and sim card frauds and what kind of stuff it's basically about getting read access to your money and the guy basically comes and picks your pocket and takes some money and goes on right uh however uh fake currency scams where someone is basically printing notes is a lot harder okay because of the entire process involved and this is basically how the bank notes work the the way in which you can think about aadhar is is that what if you had great access to the entire database okay so I'm not going to talk about the leaks I'm not going to talk about uh all the various data leaks and it leaked there it leaked there I'm going to talk about only write access right so so what it what it really means by having write access is that if you are able to compromise the right process you basically have a permanent currency note that does not have double spend problems basically that's that's basically the impact of it enough uh this is again reiterate my point which says that if you're able to control what right into the database I mean you're you're basically a very rich person okay so to illustrate this problem of write access we have four existing case studies from the field which I call is the four horsemen so if in case you look at the x-men apocalypse usually records four horsemen before the apocalypse right that's basically the four horsemen studies all these are real these are not made up and we have extensive public documentation that these things happened and we also know about the impact right and the first one is what is called as an explorer because this happened on 2012 and this was probably the first template I've seen where people had write access to the database to control what is being written into it uh the second one is what is called as a 2016 lineage it started at 2016 it's still ongoing it has not been stopped and this I call as a patch maker who specialized from trading patches to get into the database uh this third one is is what is called as a fingerprint forger the fingerprint forger is primarily used not for creating entries but for updating existing entries using uh fake fingerprints right and the last one I heard is on uh deli which I call as a mixer who basically mixes all these techniques together and create what we call as a super write access okay and finally I will also tell you who's the apocalypse at the end of the talk right uh before we go further into the four horsemen I want you to understand about how the enrollment process itself works and the various checks and balances that is there in the system uh so if you look at uh the data acquisition there's a very specific reason why I call a data acquisition uh every enrollment that has happened in adar so far is a data acquisition uh there have been as per the official uh documentation by the adar authority 1.2 billion data acquisitions uh however they do not actually tell you the amount of rejects okay uh so if you consider the rejects it's much much more uh almost all the enrollment is done using what is called as an offline mode and the reason why it is called as an offline mode is because uh basically they wanted enrollments to be done even in places where there is no internet connectivity so so what they basically have is uh it is done through a software and a packet is created and it is uploaded later that's why we call it as an offline uh you know generation process that's why it takes a long time for you to get another maybe 16 days 90 days right so what does the enrollment really mean uh it contains three different uh set of documents right uh one is what we call as an xml uh which is basically your demographic data in an xml format right and then there's what is called as media uh which is basically your fingerprints and iris and the third is what is called as a derived uh documentation or your original poi poa scans right so these three things are basically what gets uh captured as part of the uh enrollment okay fair enough so if you look at this end-to-end process uh I want you to pay special attention uh towards data transfer to cdr with a pin drive okay so that is the part that you're to pay special attention towards it's it's very important for all that you're seeing today later right uh so if you if you look at it it just explains about uh how the whole thing works but for us the most important thing is uh there is devices hardware software and there's verification procedures and there's data transfer okay so as part of any data generation process or data capture process uh you have to ensure that there is a human uh checks that are performed on it so some of the checks that they have is an introducer so in case you are one of those people rare people who never had an existing id document before uh there has to be an introducer who would introduce you in the system right and there's also a document very fire and a supervisor and and the reason why uh these two gentlemen are very important uh is because you have to think about other as a derived identifier so the only new thing it actually adds into your existing documentation is the biometrics okay so you have supposed to submit your po a proof of identity proof of address and the date of birth uh and that's basically what gets into your ekvc etc and the only new thing that it is actually adding uh is the biometrics so uh in order to verify their derived IDs are actually real you need to have a I mean enrollment uh you need to have a document checker okay uh there is also a lot of incentives for successful generation this is basically for the operators uh so basically the the the structure works is by saying that if you generate uh if you capture data properly uh there is and you get another number successfully you get 50 bucks and if you make a mistake and there are a whole bunch of process errors etc etc there is a penalty for it so basic this is basically the structure that has been set up now uh the software checks is important because if you look at what they have done is they have rolled out an enrollment software and thus there are uh checks on the software about data capture and quality uh so if you look at the if you look at the software you can actually say the basic minimum biometric quality required for fingerprint analysis 52 percent just don't ask me why does not 80 percent okay uh and then uh there is uh onboarding for supervisor and operators using their own fingerprints and passcords and there is gps and parameter synchronization so I'll come back and tell you why you need uh operator and supervising onboarding using fingerprints because they figured out uh a long time ago that how do I know if I were an operator who's enrolling people how do I know this it is me who's enrolling but not some random person so you need username and passcords but that is it is done offline because you just basically take around in a mobile one and move around so they've wanted the guy's fingerprint okay so so the prerequisite for being an operator is that you need to have another number okay uh and uh when you actually enroll people you basically are given a username password and you have to put your fingerprint on it that's basically what we call as an onboarding okay the gps and parameter synchronization are important uh because uh unless until you put a gps how do you know the person whom you're recruiting is not in pakistan because it's offline okay enough that's really the reason why the gps was added in uh the parameter synchronization is fundamentally a method uh to update some parameters of the enrollment client uh from the backend uh it's usually about how much of data packets are pending how much of them are not uploaded how much of them uh fail checks etc etc uh so once you generate this uh enrollment uh before it even hits the deduplication there are a series of checks that are done on an enrollment packet and uh these are basically structure validation demographic checks operator checks is a guy is is a is a operator really blacklisted is allowed to enroll people then is a supervisor proper is it if it is a biometric exception you need a supervisor approval so is the guy really there and then uh there's an introduce a check and then finally there is a resident deduplication right so at this point of time is when you basically get another number so i will now touch upon what the explorer has found which is case tree number one so the first version of the explorer uh found a very interesting bug on the enrollment software so remember i told you that before an operator enrolls anyone he has to give us fingerprint okay so what the explorer figured out is that you can give any person's random fingerprint it's going to fail the first three times but the fourth time it's going to accept it okay so that's basically what it is right this is two zero one two right uh now let's go further so the other interesting bug that the end that the explorer had figured out is um so remember when it went to start on two zero one two uh they were using uh what is called as a document management agency which is heavily packered so basically you have a guy hang on basically you have a guy to whom you go and give your documents he will just put it on the printer take a scan and he also has original documents and all of them are piled up together and they're basically sent to the crdr for digitization okay so if you look at on two zero one two or even until last year you would see in any other center a big pile of documents right just don't ask me what happened to those documents those documents have their own history right but the key thing you have to remember is if you look at the process workflow uh basically from the start which is one to the end it's about 12 steps it takes three months a day's i mean three months is the best number that i've seen it takes three months for the documents to actually get digitized and go to the crdr okay so what happens during the time is it is like your passport uh you can apply for the call passport and basically they will give you the passport now and the police verification comes later right uh so what these guys do is when you basically make an enrollment give all these documents right uh if your biometric checks are done and all the structure validations are done you will get another number but probably in about two weeks or four weeks but the documents may take a long time for you to reach the crdr so ideally they're supposed to audit but they don't but that's a different story right so now here's the interesting part what is the other interventional consistency thing that i want to tell you about uh this is basically an email that was in uh that was basically sent to all the operators and just look at what they're saying any non-synced packets please take back up and upload to google drive okay right right this is 2017 so what do you mean by a non-synced packet a non-synced packet is basically the operator due to enrollment and they have pending documents and sometimes it so happens that they lose hard drives they miss it and whatever is left over they say please put it on google drive okay enough okay uh so the the interesting thing about the explorer is that he used these two weaknesses uh which is the the fourth time fingerprint luck over when it works and the long tail of documents uh he made 870 biometric exemptions at with missing documents of 30 000 okay so what happened was uh Yuri runs a back end and they figured out that there was a single operator called Muhammad Ali uh who made 30 000 enrollments in 20 days right uh which they thought was probably improbable and they said okay how can the guy who'd have made it they did computation say oh he must be enrolling 50 per hour oh that's not possible okay so let's go and figure out what he did and and he turned out that uh the operator Muhammad Ali has been uh deactivated long time ago but his fingerprint was invalidated but someone is basically using the fourth fourth fingerprint thing and just putting all this stuff okay right and because of the long tail uh they figured out they said okay let's go audit everything what this guy has done they went and figured out that of all the 30 000 enr